ELBO-T2IAlign: A Generic ELBO-Based Method for Calibrating Pixel-level Text-Image Alignment in Diffusion Models
Journal:
arXiv
Published Date:
Jun 11, 2025
Abstract
Diffusion models excel at image generation. Recent studies have shown that
these models not only generate high-quality images but also encode text-image
alignment information through attention maps or loss functions. This
information is valuable for various downstream tasks, including segmentation,
text-guided image editing, and compositional image generation. However, current
methods heavily rely on the assumption of perfect text-image alignment in
diffusion models, which is not the case. In this paper, we propose using
zero-shot referring image segmentation as a proxy task to evaluate the
pixel-level image and class-level text alignment of popular diffusion models.
We conduct an in-depth analysis of pixel-text misalignment in diffusion models
from the perspective of training data bias. We find that misalignment occurs in
images with small sized, occluded, or rare object classes. Therefore, we
propose ELBO-T2IAlign, a simple yet effective method to calibrate pixel-text
alignment in diffusion models based on the evidence lower bound (ELBO) of
likelihood. Our method is training-free and generic, eliminating the need to
identify the specific cause of misalignment and works well across various
diffusion model architectures. Extensive experiments on commonly used benchmark
datasets on image segmentation and generation have verified the effectiveness
of our proposed calibration approach.
