?
Long Context Benchmark for the Russian Language
P. 1–13.
Churin I., Apishev M., Tikhonova M., Shevelev D., Bulatov A., Kuratov Y., Averkiev S., Fenogenova A.
Recent progress in Natural Language Processing (NLP) has driven the creation of Large Language Models (LLMs) capable of tackling a vast range of tasks. A critical property of these models is their ability to handle large documents and process long token sequences, which has fostered the need for a robust evaluation methodology for long-text scenarios. To meet this requirement in the context of the Russian language, we present our benchmark consisting of 18 datasets designed to assess LLM performance in tasks such as information retrieval, knowledge extraction, machine reading, question answering, and reasoning. These datasets are categorized into four levels of complexity, enabling model evaluation across context lengths up to 128k tokens. To facilitate further research, we provide open-source datasets, a codebase, and a public leaderboard associated with the benchmark.
In book
Strube M., Braud C., Hardmeier C., Jessy Li J., Loaiciga S., Zeldes A., Li C. Suzhou: Association for Computational Linguistics, 2025.
Shavrina T., Fenogenova A., Emelyanov A. et al., , in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).: Association for Computational Linguistics, 2020. P. 4717–4726.
In this paper, we introduce an advanced Russian general language understanding evaluation benchmark – RussianSuperGLUE. Recent advances in the field of universal language models and transformers require the development of a methodology for their broad diagnostics and testing for general intellectual skills - detection of natural language inference, commonsense reasoning, ability to perform simple logical ...
Added: June 14, 2026
Zmitrovich D., Abramov A., Kalmykov A. et al., , in: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).: ELRA and ICCL, 2024.
Transformer language models (LMs) are fundamental to NLP research methodologies and applications in various languages. However, developing such models specifically for the Russian language has received little attention. This paper introduces a collection of 13 Russian Transformer LMs, which spans encoder (ruBERT, ruRoBERTa, ruELECTRA), decoder (ruGPT-3), and encoder-decoder (ruT5, FRED-T5) architectures. We provide a report ...
Added: June 14, 2026
Biryukova K., Chelnokova D., Erkenova J. et al., , in: Analysis of Images, Social Networks and Texts. AIST 2024Issue 2364.: Cham: Springer, 2024. P. 109–121.
Visual Question Answering is one of the essential parts of machine reasoning. Datasets are created to train a model to perform this task. However, there are only a few datasets for the Russian language. Moreover, existing sets may have strong biases, allowing models to score high without reasoning. In this paper, we adapt the idea ...
Added: June 14, 2026
Chervyakov A., Isaeva U., Emelyanov A. et al., , in: Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)Vol. 1.: Association for Computational Linguistics, 2026. P. 2114–2161.
Multimodal large language models (MLLMs) are currently at the center of research attention, showing rapid progress in scale and capabilities, yet their intelligence, limitations, and risks remain insufficiently understood. To address these issues, particularly in the context of the Russian language, where no multimodal benchmarks currently exist, we introduce MERA Multi, an open multimodal evaluation ...
Added: June 14, 2026
Association for Computational Linguistics, 2026.
Added: June 14, 2026
Chernogorskii F., Averkiev S., Kudraleeva L. et al., , in: Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)Vol. 4.: Association for Computational Linguistics, 2026. P. 622–638.
This paper introduces DRAGOn, method to design a RAG benchmark on a regularly updated corpus. It features recent reference datasets, a question generation framework, an automatic evaluation pipeline, and a public leaderboard. Specified reference datasets allow for uniform comparison of RAG systems, while newly generated dataset versions mitigate data leakage and ensure that all models ...
Added: June 13, 2026
Tikhonova M., Chekalina V., Chervyakov A. et al., , in: Proceedings of the Seventh Workshop on Teaching Natural Language Processing (TeachNLP 2026).: Association for Computational Linguistics, 2026. P. 41–44.
Modern LLM education is increasingly centered on system building: grounding generation with retrieval, enabling tool use, and deploying models under latency and cost constraints. We present an updated release of our open course on Transformer-based LLMs and multimodal models (Nikishina et al, 2024). The update introduces topics which became importance since the first edition, namely ...
Added: June 13, 2026
Kenneth E., Chung I., Kerboua I. et al., , in: Proceedings of the 13th International Conference on Learning Representations (ICLR 2025).: ICLR, 2025. P. 102004–102060.
Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more comprehensive evaluation, we introduce the Massive Multilingual Text Embedding Benchmark (MMTEB) - a large-scale, community-driven expansion of MTEB, covering over 500 quality-controlled evaluation tasks across 250+ languages. ...
Added: June 11, 2026
Снегирев А., Tikhonova M., Maksimova A. et al., , in: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language TechnologiesVol. 1: Volume 1: Long Papers.: Association for Computational Linguistics, 2025. P. 236–254.
Embedding models play a crucial role in Natural Language Processing (NLP) by creating text embeddings used in various tasks such as information retrieval and assessing semantic text similarity. This paper focuses on research related to embedding models in the Russian language. It introduces a new Russian-focused embedding model called ru-en-RoSBERTa and the ruMTEB benchmark, the ...
Added: June 11, 2026
Strube M., Braud C., Hardmeier C. et al., Suzhou: Association for Computational Linguistics, 2025.
Added: June 11, 2026
Rabat: Association for Computational Linguistics, 2026.
Added: May 19, 2026
Derkacheva A., Sakirkina M., Kraev G. et al., /. 2026.
Comprehensive data on natural hazards and their consequences are crucial for effective for risk assessment, adaptation planning, and emergency response. However, many countries face challenges with fragmented, inconsistent, and inaccessible data, particularly regarding local-scale events. To address this data gap in Russia, we developed an end-to-end processing pipeline that scrapes news from various online sources, ...
Added: April 28, 2026
Yalcin H., Demirhan D., Aracioglu B. et al., Technology in Society 2026 Vol. 84 Article 103094
This article comprehensively evaluates the critical role of FinTech in promoting carbon neutrality and green logistics practices in global supply chains. In our study, using bibliometric analysis, social network analysis and natural language processing (NLP) methods, we evaluate the potential of FinTech innovations to increase traceability, transparency and efficiency in supply chain processes. In this ...
Added: March 11, 2026
A.V. Demidovskij, Burmistrova E. O., E.I. Zharikov, Optical Memory and Neural Networks (Information Optics) 2025 Vol. 34 P. S166–S174
Large Language Models (LLMs) require a lot of computational resources for inference. That is why the latest advancements in hardware design may offer many possibilities for speeding the LLM up. For example, TPU optimize calculations on data, transformed into the Coordinate sparse tensor format. The SparseCore processing unit that performs the calculations is heavily tailored ...
Added: December 22, 2025
A. V. Demidovskij, A. I. Trutnev, Optical Memory and Neural Networks (Information Optics) 2025 Vol. 34 No. Suppl. 1 P. S16–S29
Large Language Models (LLMs) are widely employed across a broad range of applications due to their versatility and state-of-the-art performance. However, as usage scenarios grow, there is a pressing demand for task-specific adaptation of LLMs through fine-tuning. While full fine-tuning (FT) remains the most preferred in terms of quality, its high memory and computation requirements ...
Added: December 22, 2025
Малинов С. А., Галактика медиа: журнал медиа исследований 2025 Т. 7 № 4 С. 154–173
In recent years, artificial intelligence (AI) has been actively integrated into everyday human life. Its popularity continues to grow steadily, and companies increasingly employ AI to optimize and accelerate workflows. Ordinary users leverage large language models (LLMs) and multimodal AI systems to perform a wide range of tasks, including generating texts, images, and videos; planning ...
Added: December 7, 2025
Association for Computational Linguistics, 2025.
The book contains this year’s edition of the Conference on Empirical Methods in Natural
Language Processing! Importantly, it marks the 30th edition of EMNLP. With over 8,000 submissions,
more than 3,000 accepted papers, and thousands of attendees, we have come a long way from that first
workshop, which had 14 accepted papers. As the field looks ahead, Suzhou ...
Added: November 16, 2025
Ignatenko V., Surkov A., Zakharov V. et al., Sci 2025 Vol. 7 No. 3 Article 130
Large language models (LLMs) have recently demonstrated remarkable capabilities in natural language processing, mathematical reasoning, and code generation. However, their potential for solving differential equations—fundamental to applied mathematics, physics, and engineering—remains insufficiently explored. For the first time, we applied LLMs as translators from the textual form of an equation into the textual representation of its ...
Added: October 10, 2025
Grishina E., Gorbunov M., Rakhuba M., , in: Findings of the Association for Computational Linguistics: ACL 2025.: Association for Computational Linguistics, 2025. P. 26937–26949.
Large language models (LLMs) demonstrate impressive results in natural language processing tasks but require a significant amount of computational and memory resources. Structured matrix representations are a promising way for reducing the number of parameters of these models. However, it seems unrealistic to expect that weight matrices of pretrained models can be accurately represented by ...
Added: September 4, 2025
Vladimir Zakharov, Anton Surkov, Sergei Koltcov, Procedia Computer Science 2025 Vol. 258 P. 1169–1178
The rapid development of large language models (LLMs), including their successful application to solving mathematical problems requiring complex reasoning, presents a potential avenue for using LLMs in solving differential equations. While these equations are currently being solved successfully both numerically and via the symbolic approach, it is possible that fine-tuned LLMs, if they treat solving ...
Added: August 21, 2025
Matkin N., Smirnov A., Usanin M. et al., , in: 12th International Conference, AIST 2024, Bishkek, Kyrgyzstan, October 17–19, 2024, Revised Selected Papers.: Cham: Springer, 2025.
The labor market is undergoing rapid changes, with increasing demands on job seekers and a surge in job openings. Identifying essential skills and competencies from job descriptions is challenging due to varying employer requirements and the omission of key skills. This study addresses these challenges by comparing traditional Named Entity Recognition (NER) methods based on ...
Added: July 26, 2025
Bolshakova E. I., Семак В. В., Программные продукты и системы 2025 Т. 38 № 1 С. 5–16
The current state in the field of automatic term extraction from specialized natural language texts, including scientific and technical documents, is considered. Practical applications of methods and tools for extracting terms from texts include creation of terminological dictionaries, thesauri, and glossaries of problem oriented domains, as well as extraction of keywords and construction of subject ...
Added: July 2, 2025
Springer, 2024.
This book constitutes the refereed proceedings of the 12th International Conference on Analysis of Images, Social Networks and Texts, AIST 2024, held in Bishkek, Kyrgyzstan, during October 17–19, 2024.
The 16 full papers included in this book were carefully reviewed and selected from 70 submissions. They were organized in topical sections as follows: Natural Language Processing; Computer Vision; Data Analysis and Machine Learning; ...
Added: May 29, 2025