Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformers
Journal:
arXiv
Published Date:
May 28, 2025
Abstract
Diffusion transformers (DiT) have demonstrated exceptional performance in
video generation. However, their large number of parameters and high
computational complexity limit their deployment on edge devices. Quantization
can reduce storage requirements and accelerate inference by lowering the
bit-width of model parameters. Yet, existing quantization methods for image
generation models do not generalize well to video generation tasks. We identify
two primary challenges: the loss of information during quantization and the
misalignment between optimization objectives and the unique requirements of
video generation. To address these challenges, we present Q-VDiT, a
quantization framework specifically designed for video DiT models. From the
quantization perspective, we propose the Token-aware Quantization Estimator
(TQE), which compensates for quantization errors in both the token and feature
dimensions. From the optimization perspective, we introduce Temporal
Maintenance Distillation (TMD), which preserves the spatiotemporal correlations
between frames and enables the optimization of each frame with respect to the
overall video context. Our W3A6 Q-VDiT achieves a scene consistency of 23.40,
setting a new benchmark and outperforming current state-of-the-art quantization
methods by 1.9$\times$. Code will be available at
https://github.com/cantbebetter2/Q-VDiT.
