An Audio-Gated Cascade for Resource-Efficient B imodalEmotionRecognition

?

An Audio-Gated Cascade for Resource-Efficient B imodalEmotionRecognition

Ch. 3. P. 30–45.

Bimodal (audio–visual) emotion recognition systems typ-ically process speech and video in parallel with heavy neural back-bones, incurring a computational cost that makes them unavailable on mobile, embedded and edge platforms. In this paper we propose a two-stage cascade architecture that decouples the two modalities in time. A lightweight binary audio gate, implemented as a compact log-mel CNN, first decides whether an utterance is neutral; only when emo-tional speech is detected does a resource-intensive bimodal head clas-sify the specific emotion. The same log-mel spectrogram is reused by both the gate and the audio branch of the bimodal head, removing the need for large self-supervised audio encoders and shrinking the model to approximately 37.3 MB – almost ten times smaller than the clos-est bimodal competitor.

In book

Speech and Computer. 28th International Conference, SPECOM 2026, Ohrid, North Macedonia, September 17–18, 2026, Proceedings, Part II. LNCS, volume 16935

Springer, 2026.

Enhancing Emotion Recognition in Speech Based on Self-Supervised Learning: Cross-Attention Fusion of Acoustic and Semantic Features

Deeb B., Andrey V. Savchenko, Makarov I., IEEE Access 2026 Vol. 13 P. 56283–56295

Speech Emotion Recognition has gained considerable attention in speech processing and machine learning due to its potential applications in human-computer interaction, mental health monitoring, and customer service. However, state-of-the-art models for speech emotion recognition use many parameters, which leads to computational complexity. In this paper, we introduce a novel deep-learning model to enhance the accuracy ...

Added: June 16, 2026

A Bimodal Approach for Speech Emotion Recognition using Audio and Text

Verkholyak O., Dvoynikova A., Karpov A., Journal of Internet Services and Information Security 2021 No. 1 P. 80–96

This paper presents a novel bimodal speech emotion recognition system based on analysis of acoustic and linguistic information. We propose a novel decision-level fusion strategy that leverages both emotions and sentiments extracted from audio and text transcriptions of extemporaneous speech utterances. We perform experimental study to prove the effectiveness of the proposed methods using emotional ...

Added: April 24, 2026

Cross-Lingual Bimodal Emotion Recognition with LLM-Based Label Smoothing

Ryumina Elena, Axyonov Alexandr, Abdulkadirov Timur et al., Big Data and Cognitive Computing 2025 Vol. 9 No. 11 P. 1–38

Bimodal emotion recognition based on audio and text is widely adopted in video-constrained real-world applications such as call centers and voice assistants. However, existing systems suffer from limited cross-domain generalization and monolingual bias. To address these limitations, a cross-lingual bimodal emotion recognition method is proposed, integrating Mamba-based temporal encoders for audio (Wav2Vec2.0) and text (Jina-v3) ...

Added: November 12, 2025

CA-SER: Cross-Attention Feature Fusion for Speech Emotion Recognition

Deeb B., Savchenko A., Makarov I., , in: ECAI 2024. 27th European Conference on Artificial Intelligence, October 19 – 24 October 2024, Santiago de Compostela, Spain – Including 13th Conference on Prestigious Applications of Intelligent Systems (PAIS 2024).: IOS Press, 2024. P. 4479–4482.

In this paper, we introduce a novel tool for speech emotion recognition, CA-SER, that borrows self-supervised learning to extract semantic speech representations from a pre-trained wav2vec 2.0 model and combine them with spectral audio features to improve speech emotion recognition. Our approach involves a self-attention encoder on MFCC features to capture meaningful patterns in audio ...

Added: February 15, 2025

Speaker-Aware Training of Speech Emotion Classifier with Speaker Recognition

Savchenko L., Savchenko A., , in: Speech and Computer. 23rd International Conference, SPECOM 2021, St. Petersburg, Russia, September 27–30, 2021Vol. 12997.: St. Petersburg: Springer, 2021. Ch. 55 P. 614–625.

Added: September 24, 2021

On the Impact of Word Error Rate on Acoustic-Linguistic Speech Emotion Recognition: An Update for the Deep Learning Era

Sokolov A., / Series Computer Science "arxiv.org". 2021.

Text encodings from automatic speech recognition (ASR) transcripts and audio representations have shown promise in speech emotion recognition (SER) ever since. Yet, it is challenging to explain the effect of each information stream on the SER systems. Further, more clarification is required for analysing the impact of ASR's word error rate (WER) on linguistic emotion ...

Added: November 17, 2020