MotionStone: Decoupled Motion Intensity Modulation with Diffusion Transformer for Image-to-Video Generation
Journal:
arXiv
Published Date:
Dec 8, 2024
Abstract
The image-to-video (I2V) generation is conditioned on the static image, which
has been enhanced recently by the motion intensity as an additional control
signal. These motion-aware models are appealing to generate diverse motion
patterns, yet there lacks a reliable motion estimator for training such models
on large-scale video set in the wild. Traditional metrics, e.g., SSIM or
optical flow, are hard to generalize to arbitrary videos, while, it is very
tough for human annotators to label the abstract motion intensity neither.
Furthermore, the motion intensity shall reveal both local object motion and
global camera movement, which has not been studied before. This paper addresses
the challenge with a new motion estimator, capable of measuring the decoupled
motion intensities of objects and cameras in video. We leverage the contrastive
learning on randomly paired videos and distinguish the video with greater
motion intensity. Such a paradigm is friendly for annotation and easy to scale
up to achieve stable performance on motion estimation. We then present a new
I2V model, named MotionStone, developed with the decoupled motion estimator.
Experimental results demonstrate the stability of the proposed motion estimator
and the state-of-the-art performance of MotionStone on I2V generation. These
advantages warrant the decoupled motion estimator to serve as a general plug-in
enhancer for both data processing and video generation training.
