CRISP-SAM2: SAM2 with Cross-Modal Interaction and Semantic Prompting for Multi-Organ Segmentation
Journal:
arXiv
Published Date:
Jun 29, 2025
Abstract
Multi-organ medical segmentation is a crucial component of medical image
processing, essential for doctors to make accurate diagnoses and develop
effective treatment plans. Despite significant progress in this field, current
multi-organ segmentation models often suffer from inaccurate details,
dependence on geometric prompts and loss of spatial information. Addressing
these challenges, we introduce a novel model named CRISP-SAM2 with CRoss-modal
Interaction and Semantic Prompting based on SAM2. This model represents a
promising approach to multi-organ medical segmentation guided by textual
descriptions of organs. Our method begins by converting visual and textual
inputs into cross-modal contextualized semantics using a progressive
cross-attention interaction mechanism. These semantics are then injected into
the image encoder to enhance the detailed understanding of visual information.
To eliminate reliance on geometric prompts, we use a semantic prompting
strategy, replacing the original prompt encoder to sharpen the perception of
challenging targets. In addition, a similarity-sorting self-updating strategy
for memory and a mask-refining process is applied to further adapt to medical
imaging and enhance localized details. Comparative experiments conducted on
seven public datasets indicate that CRISP-SAM2 outperforms existing models.
Extensive analysis also demonstrates the effectiveness of our method, thereby
confirming its superior performance, especially in addressing the limitations
mentioned earlier. Our code is available at:
https://github.com/YU-deep/CRISP_SAM2.git.
