Boosting Resolution Generalization of Diffusion Transformers with Randomized Positional Encodings
Journal:
arXiv
Published Date:
Mar 24, 2025
Abstract
Resolution generalization in image generation tasks enables the production of
higher-resolution images with lower training resolution overhead. However, a
significant challenge in resolution generalization, particularly in the widely
used Diffusion Transformers, lies in the mismatch between the positional
encodings encountered during testing and those used during training. While
existing methods have employed techniques such as interpolation, extrapolation,
or their combinations, none have fully resolved this issue. In this paper, we
propose a novel two-dimensional randomized positional encodings (RPE-2D)
framework that focuses on learning positional order of image patches instead of
the specific distances between them, enabling seamless high- and low-resolution
image generation without requiring high- and low-resolution image training.
Specifically, RPE-2D independently selects positions over a broader range along
both the horizontal and vertical axes, ensuring that all position encodings are
trained during the inference phase, thus improving resolution generalization.
Additionally, we propose a random data augmentation technique to enhance the
modeling of position order. To address the issue of image cropping caused by
the augmentation, we introduce corresponding micro-conditioning to enable the
model to perceive the specific cropping patterns. On the ImageNet dataset, our
proposed RPE-2D achieves state-of-the-art resolution generalization
performance, outperforming existing competitive methods when trained at a
resolution of $256 \times 256$ and inferred at $384 \times 384$ and $512 \times
512$, as well as when scaling from $512 \times 512$ to $768 \times 768$ and
$1024 \times 1024$. And it also exhibits outstanding capabilities in
low-resolution image generation, multi-stage training acceleration and
multi-resolution inheritance.
