Cross-Lingual Bimodal Emotion Recognition with LLM-Based Label Smoothing

Big Data and Cognitive Computing. 2025. Vol. 9. No. 11. P. 1–38.

Ryumina Elena, Axyonov Alexandr, Abdulkadirov Timur, Koryakovskaya Darya, Ryumin D.

Bimodal emotion recognition based on audio and text is widely adopted in video-constrained real-world applications such as call centers and voice assistants. However, existing systems suffer from limited cross-domain generalization and monolingual bias. To address these limitations, a cross-lingual bimodal emotion recognition method is proposed, integrating Mamba-based temporal encoders for audio (Wav2Vec2.0) and text (Jina-v3) with a Transformer-based cross-modal fusion architecture (BiFormer). Three corpus-adaptive augmentation strategies are introduced: (1) Stacked Data Sampling, in which short utterances are concatenated to stabilize sequence length; (2) Label Smoothing Generation based on Large Language Model, where the Qwen3-4B model is prompted to detect subtle emotional cues missed by annotators, producing soft labels that reflect latent emotional co-occurrences; and (3) Text-to-Utterance Generation, in which emotionally labeled utterances are generated by ChatGPT-5 and synthesized into speech using the DIA-TTS model, enabling controlled creation of affective audio–text pairs without human annotation. BiFormer is trained jointly on the English Multimodal EmotionLines Dataset and the Russian Emotional Speech Dialogs corpus, enabling cross-lingual transfer without parallel data. Experimental results show that the optimal data augmentation strategy is corpus-dependent: Stacked Data Sampling achieves the best performance on short, noisy English utterances, while Label Smoothing Generation based on Large Language Model better captures nuanced emotional expressions in longer Russian utterances. Text-to-Utterance Generation does not yield a measurable gain due to current limitations in expressive speech synthesis. When combined, the two best performing strategies produce complementary improvements, establishing new state-of-the-art performance in both monolingual and cross-lingual settings.

Keywords: большая языковая модель, bimodal emotion recognition, cross-lingual modeling, cross-modal attention, large language model, mamba architecture, stacked data sampling, template-based utterance generation, label smoothing generation, бимодальное распознавание эмоций, кросс-лингвальное моделирование, межмодальное внимание, архитектура мамба, стратегия сэмплирования, генерация высказываний на основе шаблонов, сглаживание меток

Publication based on the results of: Analysis of Human Interaction with Information and Improvement of Algorithms of Information Processing (2025)