MM-PSYCHE: Multimodal Multitask Psychological Characteristic Estimation Through Cross-Domain Semi-Supervised Learning

IEEE Access. 2026. Vol. 14. P. 124759–124778.

Ryumina E., Aksenov A., Koryakovskaya D., Abdulkhadirov T., Egorova A., Fedchin S., Zaburdaev A., Salova I., Savchenko A., Ryumin D.

Psychological characteristic estimation from multimodal in-the-wild behavior is usually studied using separate corpora, each annotated for a single target task. Such annotation fragmentation limits cross-task learning and cross-domain generalization across affective, dispositional, and interactional phenomena. To address this problem, we use emotion, apparent personality trait, and ambivalence recognition as representative tasks and introduce MM-PSYCHE, a framework that unifies heterogeneous single-task corpora through cross-domain semi-supervised learning (SSL) with GradNorm-based adaptive task weighting. The framework jointly trains models for emotion, apparent personality trait, and ambivalence recognition. Its multimodal fusion model contains three task-aware components: 1) Task-Specific Projectors, which produce task-conditioned prediction embeddings and re-embed them into a hidden space; 2) Task-Specific Query-Based Cross-Attention Fusion, which uses the prediction embeddings as task-dependent queries and shared modality embeddings as keys and values; and 3) Guide Banks, which are learnable task-specific prototypes that anchor the final predictions. Experiments on the task-specific corpora show competitive supervised learning (SL) performance and more pronounced gains in cross-domain SSL, where heterogeneous corpora can be exploited without joint annotations. On the unseen Multimodal EmotionLines Dataset (MELD), multitask SSL improves generalization in emotion recognition, increasing the macro-F1 score by 7.5 percentage points (35.0% vs. 27.5%) over single-task SSL. The results indicate that cross-domain SSL is useful for fragmented multimodal corpora with heterogeneous psychological annotations, while also showing why single-task SL, single-task SSL, and multitask SSL should be evaluated separately. The source code is publicly available at https://github.com/LEYA-HSE/MM-PSYCHE.

Keywords: human-computer interaction, естественное человеко-машинное взаимодействие, affective computing, multimodal learning, multitask learning, аффективные вычисления, personality computing, semi-supervised learning, многозадачное обучение, полуконтролируемое обучение, мультимодальное обучение, psychological characteristic recognition, cross-domain adaptation, компьютерный анализ личности, распознавание психологических характеристик, кросс-доменная адаптация