Deeper Diffusion Models Amplify Bias
Journal:
arXiv
Published Date:
May 23, 2025
Abstract
Despite the impressive performance of generative Diffusion Models (DMs),
their internal working is still not well understood, which is potentially
problematic. This paper focuses on exploring the important notion of
bias-variance tradeoff in diffusion models. Providing a systematic foundation
for this exploration, it establishes that at one extreme the diffusion models
may amplify the inherent bias in the training data and, on the other, they may
compromise the presumed privacy of the training samples. Our exploration aligns
with the memorization-generalization understanding of the generative models,
but it also expands further along this spectrum beyond ``generalization'',
revealing the risk of bias amplification in deeper models. Building on the
insights, we also introduce a training-free method to improve output quality in
text-to-image and image-to-image generation. By progressively encouraging
temporary high variance in the generation process with partial bypassing of the
mid-block's contribution in the denoising process of DMs, our method
consistently improves generative image quality with zero training cost. Our
claims are validated both theoretically and empirically.
