CLIP-RL: Surgical Scene Segmentation Using Contrastive Language-Vision Pretraining & Reinforcement Learning
Journal:
arXiv
Published Date:
Jul 6, 2025
Abstract
Understanding surgical scenes can provide better healthcare quality for
patients, especially with the vast amount of video data that is generated
during MIS. Processing these videos generates valuable assets for training
sophisticated models. In this paper, we introduce CLIP-RL, a novel contrastive
language-image pre-training model tailored for semantic segmentation for
surgical scenes. CLIP-RL presents a new segmentation approach which involves
reinforcement learning and curriculum learning, enabling continuous refinement
of the segmentation masks during the full training pipeline. Our model has
shown robust performance in different optical settings, such as occlusions,
texture variations, and dynamic lighting, presenting significant challenges.
CLIP model serves as a powerful feature extractor, capturing rich semantic
context that enhances the distinction between instruments and tissues. The RL
module plays a pivotal role in dynamically refining predictions through
iterative action-space adjustments. We evaluated CLIP-RL on the EndoVis 2018
and EndoVis 2017 datasets. CLIP-RL achieved a mean IoU of 81%, outperforming
state-of-the-art models, and a mean IoU of 74.12% on EndoVis 2017. This
superior performance was achieved due to the combination of contrastive
learning with reinforcement learning and curriculum learning.
