Text-Promptable Propagation for Referring Medical Image Sequence Segmentation
Journal:
arXiv
Published Date:
Feb 16, 2025
Abstract
Referring Medical Image Sequence Segmentation (Ref-MISS) is a novel and
challenging task that aims to segment anatomical structures in medical image
sequences (\emph{e.g.} endoscopy, ultrasound, CT, and MRI) based on natural
language descriptions. This task holds significant clinical potential and
offers a user-friendly advancement in medical imaging interpretation. Existing
2D and 3D segmentation models struggle to explicitly track objects of interest
across medical image sequences, and lack support for nteractive, text-driven
guidance. To address these limitations, we propose Text-Promptable Propagation
(TPP), a model designed for referring medical image sequence segmentation. TPP
captures the intrinsic relationships among sequential images along with their
associated textual descriptions. Specifically, it enables the recognition of
referred objects through cross-modal referring interaction, and maintains
continuous tracking across the sequence via Transformer-based triple
propagation, using text embeddings as queries. To support this task, we curate
a large-scale benchmark, Ref-MISS-Bench, which covers 4 imaging modalities and
20 different organs and lesions. Experimental results on this benchmark
demonstrate that TPP consistently outperforms state-of-the-art methods in both
medical segmentation and referring video object segmentation.
