VFX Creator: Animated Visual Effect Generation with Controllable Diffusion Transformer
Journal:
arXiv
Published Date:
Feb 9, 2025
Abstract
Crafting magic and illusions is one of the most thrilling aspects of
filmmaking, with visual effects (VFX) serving as the powerhouse behind
unforgettable cinematic experiences. While recent advances in generative
artificial intelligence have driven progress in generic image and video
synthesis, the domain of controllable VFX generation remains relatively
underexplored. In this work, we propose a novel paradigm for animated VFX
generation as image animation, where dynamic effects are generated from
user-friendly textual descriptions and static reference images. Our work makes
two primary contributions: (i) Open-VFX, the first high-quality VFX video
dataset spanning 15 diverse effect categories, annotated with textual
descriptions, instance segmentation masks for spatial conditioning, and
start-end timestamps for temporal control. (ii) VFX Creator, a simple yet
effective controllable VFX generation framework based on a Video Diffusion
Transformer. The model incorporates a spatial and temporal controllable LoRA
adapter, requiring minimal training videos. Specifically, a plug-and-play mask
control module enables instance-level spatial manipulation, while tokenized
start-end motion timestamps embedded in the diffusion process, alongside the
text encoder, allow precise temporal control over effect timing and pace.
Extensive experiments on the Open-VFX test set demonstrate the superiority of
the proposed system in generating realistic and dynamic effects, achieving
state-of-the-art performance and generalization ability in both spatial and
temporal controllability. Furthermore, we introduce a specialized metric to
evaluate the precision of temporal control. By bridging traditional VFX
techniques with generative approaches, VFX Creator unlocks new possibilities
for efficient and high-quality video effect generation, making advanced VFX
accessible to a broader audience.
