HumanDiT: Pose-Guided Diffusion Transformer for Long-form Human Motion Video Generation
Journal:
arXiv
Published Date:
Feb 7, 2025
Abstract
Human motion video generation has advanced significantly, while existing
methods still struggle with accurately rendering detailed body parts like hands
and faces, especially in long sequences and intricate motions. Current
approaches also rely on fixed resolution and struggle to maintain visual
consistency. To address these limitations, we propose HumanDiT, a pose-guided
Diffusion Transformer (DiT)-based framework trained on a large and wild dataset
containing 14,000 hours of high-quality video to produce high-fidelity videos
with fine-grained body rendering. Specifically, (i) HumanDiT, built on DiT,
supports numerous video resolutions and variable sequence lengths, facilitating
learning for long-sequence video generation; (ii) we introduce a prefix-latent
reference strategy to maintain personalized characteristics across extended
sequences. Furthermore, during inference, HumanDiT leverages Keypoint-DiT to
generate subsequent pose sequences, facilitating video continuation from static
images or existing videos. It also utilizes a Pose Adapter to enable pose
transfer with given sequences. Extensive experiments demonstrate its superior
performance in generating long-form, pose-accurate videos across diverse
scenarios.
