A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysis
Journal:
arXiv
Published Date:
May 29, 2025
Abstract
Endoscopic procedures are essential for diagnosing and treating internal
diseases, and multi-modal large language models (MLLMs) are increasingly
applied to assist in endoscopy analysis. However, current benchmarks are
limited, as they typically cover specific endoscopic scenarios and a small set
of clinical tasks, failing to capture the real-world diversity of endoscopic
scenarios and the full range of skills needed in clinical workflows. To address
these issues, we introduce EndoBench, the first comprehensive benchmark
specifically designed to assess MLLMs across the full spectrum of endoscopic
practice with multi-dimensional capacities. EndoBench encompasses 4 distinct
endoscopic scenarios, 12 specialized clinical tasks with 12 secondary subtasks,
and 5 levels of visual prompting granularities, resulting in 6,832 rigorously
validated VQA pairs from 21 diverse datasets. Our multi-dimensional evaluation
framework mirrors the clinical workflow--spanning anatomical recognition,
lesion analysis, spatial localization, and surgical operations--to holistically
gauge the perceptual and diagnostic abilities of MLLMs in realistic scenarios.
We benchmark 23 state-of-the-art models, including general-purpose,
medical-specialized, and proprietary MLLMs, and establish human clinician
performance as a reference standard. Our extensive experiments reveal: (1)
proprietary MLLMs outperform open-source and medical-specialized models
overall, but still trail human experts; (2) medical-domain supervised
fine-tuning substantially boosts task-specific accuracy; and (3) model
performance remains sensitive to prompt format and clinical task complexity.
EndoBench establishes a new standard for evaluating and advancing MLLMs in
endoscopy, highlighting both progress and persistent gaps between current
models and expert clinical reasoning. We publicly release our benchmark and
code.
