Generating and Debugging Java Code using LLMs based on Associative Recurrent Memory

Proceedings of the Institute for System Programming of the RAS. 2025. Vol. 37. No. 5. P. 173–182.

Василевский В. И., Alexandrov D.

Automatic code generation by large language models (LLMs) has achieved significant success, yet
it still faces challenges when dealing with complex and large codebases, especially in languages like Java. The
limitations of LLM context windows and the complexity of debugging generated code are key obstacles. This
paper presents an approach aimed at improving Java code generation and debugging. We propose using the
Associative Recurrent Memory Transformer (ARMT) model, which extends the context window and has
enhanced memory capabilities, to address two tasks: 1) selecting the most relevant snippets from the existing
codebase for generating new code; 2) selecting the most significant parts of stack traces and runtime data for
iterative debugging. This approach is integrated with an iterative debugging loop, embodied in our developing
system "JavaCapsule" (inspired by PyCapsule for Python), which includes compilation and test execution in a
controlled Docker environment using Gradle. It is expected that the proposed method will enhance the accuracy
and relevance of generated Java code, particularly in the context of large projects, and improve the automated
debugging process. Such benchmarks like JavaBench further underscore the need for such focused
advancements. This paper is an output of a research project implemented as part of the Basic Research Program
at the National Research University Higher School of Economics (HSE University).

Keywords: code generation, Java, генерация кода, Большие языковые модели (LLMs), Large language models (LLM)

Publication based on the results of: Formation and Research of Best Practices in the Development of Cloud and Mobile Applications (2025)