Data-Efficient Generalization for Zero-shot Composed Image Retrieval
Journal:
arXiv
Published Date:
Mar 7, 2025
Abstract
Zero-shot Composed Image Retrieval (ZS-CIR) aims to retrieve the target image
based on a reference image and a text description without requiring
in-distribution triplets for training. One prevalent approach follows the
vision-language pretraining paradigm that employs a mapping network to transfer
the image embedding to a pseudo-word token in the text embedding space.
However, this approach tends to impede network generalization due to modality
discrepancy and distribution shift between training and inference. To this end,
we propose a Data-efficient Generalization (DeG) framework, including two novel
designs, namely, Textual Supplement (TS) module and Semantic-Set (S-Set). The
TS module exploits compositional textual semantics during training, enhancing
the pseudo-word token with more linguistic semantics and thus mitigating the
modality discrepancy effectively. The S-Set exploits the zero-shot capability
of pretrained Vision-Language Models (VLMs), alleviating the distribution shift
and mitigating the overfitting issue from the redundancy of the large-scale
image-text data. Extensive experiments over four ZS-CIR benchmarks show that
DeG outperforms the state-of-the-art (SOTA) methods with much less training
data, and saves substantial training and inference time for practical usage.
