3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark

Ch. 1353. P. 26625–26665.

Sviridov I., Miftakhova A., Tereshchenko A., Zubkova G., Blinov P., Savchenko A.

Though Large Vision-Language Models (LVLMs) are being actively explored in medicine, their ability to conduct complex real-world telemedicine consultations combining accurate diagnosis with professional dialogue remains underexplored. This paper presents 3MDBench (Medical Multimodal Multi-agent Dialogue Benchmark), an open-source framework for simulating and evaluating LVLM-driven telemedical consultations. 3MDBench simulates patient variability through temperament-based Patient Agent and evaluates diagnostic accuracy and dialogue quality via Assessor Agent. It includes 2996 cases across 34 diagnoses from real-world telemedicine interactions, combining textual and image-based data. The experimental study compares diagnostic strategies for widely used open and closed-source LVLMs. We demonstrate that multimodal dialogue with internal reasoning improves F1 score by 6.5% over non-dialogue settings, highlighting the importance of context-aware, information-seeking questioning. Moreover, injecting predictions from a diagnostic convolutional neural network into the LVLM’s context boosts F1 by up to 20%. Source code is available at https://github.com/univanxx/3mdbench.

Keywords: диалоговая система, dialogue systems, мультимодальные данные, искусственный интеллект в здравоохранении, large language model (LLM), medical artificial intelligence, Большие языковые модели (LLMs), multimodal dataset

In book

Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Association for Computational Linguistics, 2025.