OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions
Journal:
arXiv
Published Date:
Jun 29, 2025
Abstract
Existing feedforward subject-driven video customization methods mainly study
single-subject scenarios due to the difficulty of constructing multi-subject
training data pairs. Another challenging problem that how to use the signals
such as depth, mask, camera, and text prompts to control and edit the subject
in the customized video is still less explored. In this paper, we first propose
a data construction pipeline, VideoCus-Factory, to produce training data pairs
for multi-subject customization from raw videos without labels and control
signals such as depth-to-video and mask-to-video pairs. Based on our
constructed data, we develop an Image-Video Transfer Mixed (IVTM) training with
image editing data to enable instructive editing for the subject in the
customized video. Then we propose a diffusion Transformer framework, OmniVCus,
with two embedding mechanisms, Lottery Embedding (LE) and Temporally Aligned
Embedding (TAE). LE enables inference with more subjects by using the training
subjects to activate more frame embeddings. TAE encourages the generation
process to extract guidance from temporally aligned control signals by
assigning the same frame embeddings to the control and noise tokens.
Experiments demonstrate that our method significantly surpasses
state-of-the-art methods in both quantitative and qualitative evaluations.
Video demos are at our project page:
https://caiyuanhao1998.github.io/project/OmniVCus/. Our code will be released
at https://github.com/caiyuanhao1998/Open-OmniVCus
