Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation
Journal:
arXiv
Published Date:
Jan 6, 2025
Abstract
We consider the task of Image-to-Video (I2V) generation, which involves
transforming static images into realistic video sequences based on a textual
description. While recent advancements produce photorealistic outputs, they
frequently struggle to create videos with accurate and consistent object
motion, especially in multi-object scenarios. To address these limitations, we
propose a two-stage compositional framework that decomposes I2V generation
into: (i) An explicit intermediate representation generation stage, followed by
(ii) A video generation stage that is conditioned on this representation. Our
key innovation is the introduction of a mask-based motion trajectory as an
intermediate representation, that captures both semantic object information and
motion, enabling an expressive but compact representation of motion and
semantics. To incorporate the learned representation in the second stage, we
utilize object-level attention objectives. Specifically, we consider a spatial,
per-object, masked-cross attention objective, integrating object-specific
prompts into corresponding latent space regions and a masked spatio-temporal
self-attention objective, ensuring frame-to-frame consistency for each object.
We evaluate our method on challenging benchmarks with multi-object and
high-motion scenarios and empirically demonstrate that the proposed method
achieves state-of-the-art results in temporal coherence, motion realism, and
text-prompt faithfulness. Additionally, we introduce \benchmark, a new
challenging benchmark for single-object and multi-object I2V generation, and
demonstrate our method's superiority on this benchmark. Project page is
available at https://guyyariv.github.io/TTM/.
