The Missing Point in Vision Transformers for Universal Image Segmentation
Journal:
arXiv
Published Date:
May 26, 2025
Abstract
Image segmentation remains a challenging task in computer vision, demanding
robust mask generation and precise classification. Recent mask-based approaches
yield high-quality masks by capturing global context. However, accurately
classifying these masks, especially in the presence of ambiguous boundaries and
imbalanced class distributions, remains an open challenge. In this work, we
introduce ViT-P, a novel two-stage segmentation framework that decouples mask
generation from classification. The first stage employs a proposal generator to
produce class-agnostic mask proposals, while the second stage utilizes a
point-based classification model built on the Vision Transformer (ViT) to
refine predictions by focusing on mask central points. ViT-P serves as a
pre-training-free adapter, allowing the integration of various pre-trained
vision transformers without modifying their architecture, ensuring adaptability
to dense prediction tasks. Furthermore, we demonstrate that coarse and bounding
box annotations can effectively enhance classification without requiring
additional training on fine annotation datasets, reducing annotation costs
while maintaining strong performance. Extensive experiments across COCO,
ADE20K, and Cityscapes datasets validate the effectiveness of ViT-P, achieving
state-of-the-art results with 54.0 PQ on ADE20K panoptic segmentation, 87.4
mIoU on Cityscapes semantic segmentation, and 63.6 mIoU on ADE20K semantic
segmentation. The code and pretrained models are available at:
https://github.com/sajjad-sh33/ViT-P}{https://github.com/sajjad-sh33/ViT-P.
