On Perspectives of Zeroth-Order Methods for Large Language Models Pre-Training

P. 319–323.

Трутнев А. И., Demidovskij A.

Pre-training of large language models (LLMs) requires significant computing power and memory due to firstorder methods that are based on gradient calculation. As an alternative, zeroth-order (ZO) methods make it possible to approximate gradients solely on the forward passes and reduce memory consumption to the level of inference. Despite the active development of ZO methods in the tasks of LLM fine-tuning, their use for pre-training remains insufficiently studied. This paper presents an analysis of existing ZO methods in terms of their applicability to the task of pre-training. Experiments on the OLMo2-1B demonstrate that current ZO methods provide substantial computational advantages, reducing peak memory consumption by more than 73%. However, these benefits come with a significant degradation in optimization convergence, indicating that they cannot yet serve as direct replacements for firstorder optimization in large-scale LLM training. At the same time, experiments reveal that the combination of ZO methods with backpropagation (LoHO) can partially close this optimization gap and achieve consistently better convergence compared to conventional ZO estimators. Moreover, the improved perturbation strategy AGZO can improve the convergence behavior of pure ZO methods and achieve better downstream performance.

Keywords: artificial neural networks, zeroth-order optimization, pre-training, Natural language processing

In book

2026 10th Scientific School Dynamics of Complex Networks and their Applications

IEEE, 2026.