FocalLens: Instruction Tuning Enables Zero-Shot Conditional Image Representations
Journal:
arXiv
Published Date:
Apr 11, 2025
Abstract
Visual understanding is inherently contextual -- what we focus on in an image
depends on the task at hand. For instance, given an image of a person holding a
bouquet of flowers, we may focus on either the person such as their clothing,
or the type of flowers, depending on the context of interest. Yet, most
existing image encoding paradigms represent an image as a fixed, generic
feature vector, overlooking the potential needs of prioritizing varying visual
information for different downstream use cases. In this work, we introduce
FocalLens, a conditional visual encoding method that produces different
representations for the same image based on the context of interest, expressed
flexibly through natural language. We leverage vision instruction tuning data
and contrastively finetune a pretrained vision encoder to take natural language
instructions as additional inputs for producing conditional image
representations. Extensive experiments validate that conditional image
representation from FocalLens better pronounce the visual features of interest
compared to generic features produced by standard vision encoders like CLIP. In
addition, we show FocalLens further leads to performance improvements on a
range of downstream tasks including image-image retrieval, image
classification, and image-text retrieval, with an average gain of 5 and 10
points on the challenging SugarCrepe and MMVP-VLM benchmarks, respectively.
