Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms
Journal:
arXiv
Published Date:
Feb 1, 2025
Abstract
Discrete diffusion models have emerged as a powerful generative modeling
framework for discrete data with successful applications spanning from text
generation to image synthesis. However, their deployment faces challenges due
to the high dimensionality of the state space, necessitating the development of
efficient inference algorithms. Current inference approaches mainly fall into
two categories: exact simulation and approximate methods such as
$\tau$-leaping. While exact methods suffer from unpredictable inference time
and redundant function evaluations, $\tau$-leaping is limited by its
first-order accuracy. In this work, we advance the latter category by tailoring
the first extension of high-order numerical inference schemes to discrete
diffusion models, enabling larger step sizes while reducing error. We
rigorously analyze the proposed schemes and establish the second-order accuracy
of the $\theta$-trapezoidal method in KL divergence. Empirical evaluations on
GPT-2 level text and ImageNet-level image generation tasks demonstrate that our
method achieves superior sample quality compared to existing approaches under
equivalent computational constraints.
