Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection
Journal:
arXiv
Published Date:
Feb 22, 2025
Abstract
Zero-shot medical detection can further improve detection performance without
relying on annotated medical images even upon the fine-tuned model, showing
great clinical value. Recent studies leverage grounded vision-language models
(GLIP) to achieve this by using detailed disease descriptions as prompts for
the target disease name during the inference phase. However, these methods
typically treat prompts as equivalent context to the target name, making it
difficult to assign specific disease knowledge based on visual information,
leading to a coarse alignment between images and target descriptions. In this
paper, we propose StructuralGLIP, which introduces an auxiliary branch to
encode prompts into a latent knowledge bank layer-by-layer, enabling more
context-aware and fine-grained alignment. Specifically, in each layer, we
select highly similar features from both the image representation and the
knowledge bank, forming structural representations that capture nuanced
relationships between image patches and target descriptions. These features are
then fused across modalities to further enhance detection performance.
Extensive experiments demonstrate that StructuralGLIP achieves a +4.1\% AP
improvement over prior state-of-the-art methods across seven zero-shot medical
detection benchmarks, and consistently improves fine-tuned models by +3.2\% AP
on endoscopy image datasets.
