Video Creation by Demonstration
Journal:
arXiv
Published Date:
Dec 12, 2024
Abstract
We explore a novel video creation experience, namely Video Creation by
Demonstration. Given a demonstration video and a context image from a different
scene, we generate a physically plausible video that continues naturally from
the context image and carries out the action concepts from the demonstration.
To enable this capability, we present $\delta$-Diffusion, a self-supervised
training approach that learns from unlabeled videos by conditional future frame
prediction. Unlike most existing video generation controls that are based on
explicit signals, we adopts the form of implicit latent control for maximal
flexibility and expressiveness required by general videos. By leveraging a
video foundation model with an appearance bottleneck design on top, we extract
action latents from demonstration videos for conditioning the generation
process with minimal appearance leakage. Empirically, $\delta$-Diffusion
outperforms related baselines in terms of both human preference and large-scale
machine evaluations, and demonstrates potentials towards interactive world
simulation. Sampled video generation results are available at
https://delta-diffusion.github.io/.
