Prompt-Guided Mask Proposal for Two-Stage Open-Vocabulary Segmentation
Journal:
arXiv
Published Date:
Dec 13, 2024
Abstract
We tackle the challenge of open-vocabulary segmentation, where we need to
identify objects from a wide range of categories in different environments,
using text prompts as our input. To overcome this challenge, existing methods
often use multi-modal models like CLIP, which combine image and text features
in a shared embedding space to bridge the gap between limited and extensive
vocabulary recognition, resulting in a two-stage approach: In the first stage,
a mask generator takes an input image to generate mask proposals, and the in
the second stage the target mask is picked based on the query. However, the
expected target mask may not exist in the generated mask proposals, which leads
to an unexpected output mask. In our work, we propose a novel approach named
Prompt-guided Mask Proposal (PMP) where the mask generator takes the input text
prompts and generates masks guided by these prompts. Compared with mask
proposals generated without input prompts, masks generated by PMP are better
aligned with the input prompts. To realize PMP, we designed a cross-attention
mechanism between text tokens and query tokens which is capable of generating
prompt-guided mask proposals after each decoding. We combined our PMP with
several existing works employing a query-based segmentation backbone and the
experiments on five benchmark datasets demonstrate the effectiveness of this
approach, showcasing significant improvements over the current two-stage models
(1% ~ 3% absolute performance gain in terms of mIOU). The steady improvement in
performance across these benchmarks indicates the effective generalization of
our proposed lightweight prompt-aware method.
