SAP-Bench: Benchmarking Multimodal Large Language Models in Surgical Action Planning
Journal:
arXiv
Published Date:
Jun 8, 2025
Abstract
Effective evaluation is critical for driving advancements in MLLM research.
The surgical action planning (SAP) task, which aims to generate future action
sequences from visual inputs, demands precise and sophisticated analytical
capabilities. Unlike mathematical reasoning, surgical decision-making operates
in life-critical domains and requires meticulous, verifiable processes to
ensure reliability and patient safety. This task demands the ability to
distinguish between atomic visual actions and coordinate complex, long-horizon
procedures, capabilities that are inadequately evaluated by current benchmarks.
To address this gap, we introduce SAP-Bench, a large-scale, high-quality
dataset designed to enable multimodal large language models (MLLMs) to perform
interpretable surgical action planning. Our SAP-Bench benchmark, derived from
the cholecystectomy procedures context with the mean duration of 1137.5s, and
introduces temporally-grounded surgical action annotations, comprising the
1,226 clinically validated action clips (mean duration: 68.7s) capturing five
fundamental surgical actions across 74 procedures. The dataset provides 1,152
strategically sampled current frames, each paired with the corresponding next
action as multimodal analysis anchors. We propose the MLLM-SAP framework that
leverages MLLMs to generate next action recommendations from the current
surgical scene and natural language instructions, enhanced with injected
surgical domain knowledge. To assess our dataset's effectiveness and the
broader capabilities of current models, we evaluate seven state-of-the-art
MLLMs (e.g., OpenAI-o1, GPT-4o, QwenVL2.5-72B, Claude-3.5-Sonnet, GeminiPro2.5,
Step-1o, and GLM-4v) and reveal critical gaps in next action prediction
performance.
