MAKE: Multi-Aspect Knowledge-Enhanced Vision-Language Pretraining for Zero-shot Dermatological Assessment
Journal:
arXiv
Published Date:
May 14, 2025
Abstract
Dermatological diagnosis represents a complex multimodal challenge that
requires integrating visual features with specialized clinical knowledge. While
vision-language pretraining (VLP) has advanced medical AI, its effectiveness in
dermatology is limited by text length constraints and the lack of structured
texts. In this paper, we introduce MAKE, a Multi-Aspect Knowledge-Enhanced
vision-language pretraining framework for zero-shot dermatological tasks.
Recognizing that comprehensive dermatological descriptions require multiple
knowledge aspects that exceed standard text constraints, our framework
introduces: (1) a multi-aspect contrastive learning strategy that decomposes
clinical narratives into knowledge-enhanced sub-texts through large language
models, (2) a fine-grained alignment mechanism that connects subcaptions with
diagnostically relevant image features, and (3) a diagnosis-guided weighting
scheme that adaptively prioritizes different sub-captions based on clinical
significance prior. Through pretraining on 403,563 dermatological image-text
pairs collected from education resources, MAKE significantly outperforms
state-of-the-art VLP models on eight datasets across zero-shot skin disease
classification, concept annotation, and cross-modal retrieval tasks. Our code
will be made publicly available at https: //github.com/SiyuanYan1/MAKE.
