Multi-modal Representations for Fine-grained Multi-label Critical View of Safety Recognition
Journal:
arXiv
Published Date:
Jul 7, 2025
Abstract
The Critical View of Safety (CVS) is crucial for safe laparoscopic
cholecystectomy, yet assessing CVS criteria remains a complex and challenging
task, even for experts. Traditional models for CVS recognition depend on
vision-only models learning with costly, labor-intensive spatial annotations.
This study investigates how text can be harnessed as a powerful tool for both
training and inference in multi-modal surgical foundation models to automate
CVS recognition. Unlike many existing multi-modal models, which are primarily
adapted for multi-class classification, CVS recognition requires a multi-label
framework. Zero-shot evaluation of existing multi-modal surgical models shows a
significant performance gap for this task. To address this, we propose
CVS-AdaptNet, a multi-label adaptation strategy that enhances fine-grained,
binary classification across multiple labels by aligning image embeddings with
textual descriptions of each CVS criterion using positive and negative prompts.
By adapting PeskaVLP, a state-of-the-art surgical foundation model, on the
Endoscapes-CVS201 dataset, CVS-AdaptNet achieves 57.6 mAP, improving over the
ResNet50 image-only baseline (51.5 mAP) by 6 points. Our results show that
CVS-AdaptNet's multi-label, multi-modal framework, enhanced by textual prompts,
boosts CVS recognition over image-only methods. We also propose text-specific
inference methods, that helps in analysing the image-text alignment. While
further work is needed to match state-of-the-art spatial annotation-based
methods, this approach highlights the potential of adapting generalist models
to specialized surgical tasks. Code:
https://github.com/CAMMA-public/CVS-AdaptNet
