Complex landscape of the cost function in a simple machine learning regression task

European Physical Journal: Special Topics. 2024. P. 1–8.

Soloviev I., Klinshov V.

We carry out detailed study of the phase space of a small neural network in a paradigmatic machine learning regression problem. In spite of its simplicity, the system phase space turns to be extremely complex with a plenty of local minima of the cost function. These minima differ in depth by several orders of magnitude, with the majority of them being significantly suboptimal. The deepest minima corresponding to optimal solutions are characterized by a peculiar form having both very steep and very flat directions. These features explain the difficulty of learning optimal solutions with standard machine learning methods.

Keywords: neural networks, machine learning