Implementation Challenges and Strategies for Hebbian Learning in Convolutional Neural Networks

Optical Memory and Neural Networks (Information Optics). 2023. Vol. 32. P. S252–S264.

A. V. Demidovskij, M. S. Kazyulina, Salnikov I. G., A. M. Tugaryov, A. I. Trutnev, S. V. Pavlov

Given the unprecedented growth of deep learning applications, training acceleration is becoming a subject of strong academic interest. Hebbian learning as a training strategy alternative to backpropagation presents a promising optimization approach due to its locality, lower computational complexity and parallelization potential. Nevertheless, due to the challenging optimization of Hebbian learning, there is no widely accepted approach to the implementation of such mixed strategies. The current paper overviews the 4 main strategies for updating weights using the Hebbian rule, including its widely used modifications—Oja’s and Instar rules. Additionally, the paper analyses 21 industrial implementations of Hebbian learning, discusses merits and shortcomings of Hebbian rules, as well as presents the results of computational experiments on 4 convolutional networks. Experiments show that the most efficient implementation strategy of Hebbian learning allows for acceleration and memory consumption when updating DenseNet121 weights compared to backpropagation. Finally, a comparative analysis of the implementation strategies is carried out and grounded recommendations for Hebbian learning application are formulated.

Keywords: hebbian learning, localized learning