Unlock Pose Diversity: Accurate and Efficient Implicit Keypoint-based Spatiotemporal Diffusion for Audio-driven Talking Portrait
Journal:
arXiv
Published Date:
Mar 17, 2025
Abstract
Audio-driven single-image talking portrait generation plays a crucial role in
virtual reality, digital human creation, and filmmaking. Existing approaches
are generally categorized into keypoint-based and image-based methods.
Keypoint-based methods effectively preserve character identity but struggle to
capture fine facial details due to the fixed points limitation of the 3D
Morphable Model. Moreover, traditional generative networks face challenges in
establishing causality between audio and keypoints on limited datasets,
resulting in low pose diversity. In contrast, image-based approaches produce
high-quality portraits with diverse details using the diffusion network but
incur identity distortion and expensive computational costs. In this work, we
propose KDTalker, the first framework to combine unsupervised implicit 3D
keypoint with a spatiotemporal diffusion model. Leveraging unsupervised
implicit 3D keypoints, KDTalker adapts facial information densities, allowing
the diffusion process to model diverse head poses and capture fine facial
details flexibly. The custom-designed spatiotemporal attention mechanism
ensures accurate lip synchronization, producing temporally consistent,
high-quality animations while enhancing computational efficiency. Experimental
results demonstrate that KDTalker achieves state-of-the-art performance
regarding lip synchronization accuracy, head pose diversity, and execution
efficiency.Our codes are available at https://github.com/chaolongy/KDTalker.
