SSL-MEPR: A Semi-Supervised Multi-Task Cross-Domain Learning Framework for Multimodal Emotion and Personality Recognition

Machine Learning and Knowledge Extraction. 2026. No. 8. P. 1–41.

Ryumina E., Aksenov A., Koryakovskaya D., Abdulkhadirov T., Egorova A., Fedchin S., Zaburdaev A., Ryumin D.

The growing demand for personalized human-computer interaction calls for methods that jointly model emotional states and personality traits. However, large-scale multimodal corpora annotated for both tasks are still lacking. This challenge stems from integrating diverse, task-specific corpora with divergent modality informativeness and domain characteristics. To address it, we propose SSL-MEPR, a semi-supervised multi-task cross-domain learning framework for Multimodal Emotion and Personality Recognition, which enables cross-task knowledge transfer without jointly labeled data. SSL-MEPR employs a three-stage strategy, progressively integrating unimodal single-task, unimodal multi-task, and multimodal multi-task models. Key innovations include Graph Attention Fusion, task-specific query-based cross-attention, predict projectors, and guide banks, which enable robust fusion and effective use of semi-labeled data via a modified GradNorm method. Evaluated on MOSEI (emotion) and FIv2 (personality), SSL-MEPR achieves a mean Weighted Accuracy (mWACC) of 70.26 and a mean Accuracy (mACC) of 92.88 in single-task cross-domain settings, outperforming state-of-the-art methods. Multi-task learning reveals domain-induced misalignment in modality informativeness but still uncovers consistent psychological patterns: sadness correlates with lower personality trait scores, while happiness aligns with higher ones. This work establishes a new paradigm for extracting cross-task psychological knowledge from disjoint multimodal corpora, demonstrating that semi-supervised multi-task cross-domain learning can bridge annotation gaps while preserving theoretically grounded emotion-personality relationships.

Keywords: multi-task learning, multimodal emotion recognition, multimodal and multi-task knowledge extraction, multimodal personality traits assessment, cross-domain learning, semi-supervised learning, мультимодальное и многозадачное извлечение знаний, мультимодальное распознавание эмоций, мультимодальная оценка персональных качеств личности, междоменное обучение, многозадачное обучение, полуконтролируемое обучение