Free energy of neural network can predict accuracy after pruning

Physica A: Statistical Mechanics and its Applications. 2025. Vol. 681. Article 131085.

Surkov A., Sergei Koltcov, Ignatenko V., - Р. М., Ksenia Kupitman

Neural networks are powerful tools capable of achieving state-of-the-art performance across a wide range of tasks; however, their effectiveness often comes at the cost of extremely large numbers of parameters, which can hinder their deployment in resource-constrained environments. To address this issue, various pruning techniques have been proposed to reduce model size and complexity while preserving performance. In this study, we first propose a thermodynamic perspective for analyzing the behavior of neural networks during the pruning process based on magnitude-based weight pruning. Second, we demonstrate that by employing the thermodynamic concept of free energy, the selection procedure for the pruning level can be significantly simplified and accelerated. Thus, in this work, we propose a fast method for selecting the pruning threshold by computing the network’s free energy. We evaluate our method on classification tasks in the domains of natural language processing and computer vision, considering models such as multilayer perceptrons (MLP), encoder–decoder transformers, encoder-only transformers, pretrained transformers, VGG, ResNet, and DenseNet. Experimental results demonstrate that our approach provides a good approximation of the optimal pruning threshold for MLP and transformer-based models while significantly reducing the computational time (at least 70 times) compared to evaluating model accuracy.

Keywords: нейронные сети, neural networks, термодинамический подход, свободная энергия, Pruning, free energy, thermodynamic approach, прунинг

Publication based on the results of: Analysis of Human Interaction with Information and Improvement of Algorithms of Information Processing (2025)