ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformer
Journal:
arXiv
Published Date:
Dec 10, 2024
Abstract
We present ACDiT, a novel Autoregressive blockwise Conditional Diffusion
Transformer, that innovatively combines autoregressive and diffusion paradigms
for modeling continuous visual information. By introducing a block-wise
autoregressive unit, ACDiT offers a flexible interpolation between token-wise
autoregression and full-sequence diffusion, bypassing the limitations of
discrete tokenization. The generation of each block is formulated as a
conditional diffusion process, conditioned on prior blocks. ACDiT is easy to
implement, as simple as creating a Skip-Causal Attention Mask (SCAM) on
standard diffusion transformer during training. During inference, the process
iterates between diffusion denoising and autoregressive decoding that can make
full use of KV-Cache. We show that ACDiT performs best among all autoregressive
baselines under similar model scales on image and video generation tasks. We
also demonstrate that benefiting from autoregressive modeling, pretrained ACDiT
can be transferred in visual understanding tasks despite being trained with the
diffusion objective. The analysis of the trade-off between autoregressive
modeling and diffusion demonstrates the potential of ACDiT to be used in
long-horizon visual generation tasks. We hope that ACDiT offers a novel
perspective on visual autoregressive generation and unlocks new avenues for
unified models.
