What You Perceive Is What You Conceive: A Cognition-Inspired Framework for Open Vocabulary Image Segmentation
Journal:
arXiv
Published Date:
May 26, 2025
Abstract
Open vocabulary image segmentation tackles the challenge of recognizing
dynamically adjustable, predefined novel categories at inference time by
leveraging vision-language alignment. However, existing paradigms typically
perform class-agnostic region segmentation followed by category matching, which
deviates from the human visual system's process of recognizing objects based on
semantic concepts, leading to poor alignment between region segmentation and
target concepts. To bridge this gap, we propose a novel Cognition-Inspired
Framework for open vocabulary image segmentation that emulates the human visual
recognition process: first forming a conceptual understanding of an object,
then perceiving its spatial extent. The framework consists of three core
components: (1) A Generative Vision-Language Model (G-VLM) that mimics human
cognition by generating object concepts to provide semantic guidance for region
segmentation. (2) A Concept-Aware Visual Enhancer Module that fuses textual
concept features with global visual representations, enabling adaptive visual
perception based on target concepts. (3) A Cognition-Inspired Decoder that
integrates local instance features with G-VLM-provided semantic cues, allowing
selective classification over a subset of relevant categories. Extensive
experiments demonstrate that our framework achieves significant improvements,
reaching $27.2$ PQ, $17.0$ mAP, and $35.3$ mIoU on A-150. It further attains
$56.2$, $28.2$, $15.4$, $59.2$, $18.7$, and $95.8$ mIoU on Cityscapes,
Mapillary Vistas, A-847, PC-59, PC-459, and PAS-20, respectively. In addition,
our framework supports vocabulary-free segmentation, offering enhanced
flexibility in recognizing unseen categories. Code will be public.
