Segment Any-Quality Images with Generative Latent Space Enhancement
Journal:
arXiv
Published Date:
Mar 16, 2025
Abstract
Despite their success, Segment Anything Models (SAMs) experience significant
performance drops on severely degraded, low-quality images, limiting their
effectiveness in real-world scenarios. To address this, we propose GleSAM,
which utilizes Generative Latent space Enhancement to boost robustness on
low-quality images, thus enabling generalization across various image
qualities. Specifically, we adapt the concept of latent diffusion to SAM-based
segmentation frameworks and perform the generative diffusion process in the
latent space of SAM to reconstruct high-quality representation, thereby
improving segmentation. Additionally, we introduce two techniques to improve
compatibility between the pre-trained diffusion model and the segmentation
framework. Our method can be applied to pre-trained SAM and SAM2 with only
minimal additional learnable parameters, allowing for efficient optimization.
We also construct the LQSeg dataset with a greater diversity of degradation
types and levels for training and evaluating the model. Extensive experiments
demonstrate that GleSAM significantly improves segmentation robustness on
complex degradations while maintaining generalization to clear images.
Furthermore, GleSAM also performs well on unseen degradations, underscoring the
versatility of our approach and dataset.
