Challenging Vision-Language Models with Surgical Data: A New Dataset and Broad Benchmarking Study
Journal:
arXiv
Published Date:
Jun 6, 2025
Abstract
While traditional computer vision models have historically struggled to
generalize to endoscopic domains, the emergence of foundation models has shown
promising cross-domain performance. In this work, we present the first
large-scale study assessing the capabilities of Vision Language Models (VLMs)
for endoscopic tasks with a specific focus on laparoscopic surgery. Using a
diverse set of state-of-the-art models, multiple surgical datasets, and
extensive human reference annotations, we address three key research questions:
(1) Can current VLMs solve basic perception tasks on surgical images? (2) Can
they handle advanced frame-based endoscopic scene understanding tasks? and (3)
How do specialized medical VLMs compare to generalist models in this context?
Our results reveal that VLMs can effectively perform basic surgical perception
tasks, such as object counting and localization, with performance levels
comparable to general domain tasks. However, their performance deteriorates
significantly when the tasks require medical knowledge. Notably, we find that
specialized medical VLMs currently underperform compared to generalist models
across both basic and advanced surgical tasks, suggesting that they are not yet
optimized for the complexity of surgical environments. These findings highlight
the need for further advancements to enable VLMs to handle the unique
challenges posed by surgery. Overall, our work provides important insights for
the development of next-generation endoscopic AI systems and identifies key
areas for improvement in medical visual language models.
