SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence
Journal:
arXiv
Published Date:
Jun 3, 2025
Abstract
Foundation models have achieved transformative success across biomedical
domains by enabling holistic understanding of multimodal data. However, their
application in surgery remains underexplored. Surgical intelligence presents
unique challenges - requiring surgical visual perception, temporal analysis,
and reasoning. Existing general-purpose vision-language models fail to address
these needs due to insufficient domain-specific supervision and the lack of a
large-scale high-quality surgical database. To bridge this gap, we propose
SurgVLM, one of the first large vision-language foundation models for surgical
intelligence, where this single universal model can tackle versatile surgical
tasks. To enable this, we construct a large-scale multimodal surgical database,
SurgVLM-DB, comprising over 1.81 million frames with 7.79 million
conversations, spanning more than 16 surgical types and 18 anatomical
structures. We unify and reorganize 23 public datasets across 10 surgical
tasks, followed by standardizing labels and doing hierarchical vision-language
alignment to facilitate comprehensive coverage of gradually finer-grained
surgical tasks, from visual perception, temporal analysis, to high-level
reasoning. Building upon this comprehensive dataset, we propose SurgVLM, which
is built upon Qwen2.5-VL, and undergoes instruction tuning to 10+ surgical
tasks. We further construct a surgical multimodal benchmark, SurgVLM-Bench, for
method evaluation. SurgVLM-Bench consists of 6 popular and widely-used datasets
in surgical domain, covering several crucial downstream tasks. Based on
SurgVLM-Bench, we evaluate the performance of our SurgVLM (3 SurgVLM variants:
SurgVLM-7B, SurgVLM-32B, and SurgVLM-72B), and conduct comprehensive
comparisons with 14 mainstream commercial VLMs (e.g., GPT-4o, Gemini 2.0 Flash,
Qwen2.5-Max).
