Scaling RL to Long Videos
Journal:
arXiv
Published Date:
Jul 10, 2025
Abstract
We introduce a full-stack framework that scales up reasoning in
vision-language models (VLMs) to long videos, leveraging reinforcement
learning. We address the unique challenges of long video reasoning by
integrating three critical components: (1) a large-scale dataset,
LongVideo-Reason, comprising 52K long video QA pairs with high-quality
reasoning annotations across diverse domains such as sports, games, and vlogs;
(2) a two-stage training pipeline that extends VLMs with chain-of-thought
supervised fine-tuning (CoT-SFT) and reinforcement learning (RL); and (3) a
training infrastructure for long video RL, named Multi-modal Reinforcement
Sequence Parallelism (MR-SP), which incorporates sequence parallelism and a
vLLM-based engine tailored for long video, using cached video embeddings for
efficient rollout and prefilling. In experiments, LongVILA-R1-7B achieves
strong performance on long video QA benchmarks such as VideoMME. It also
outperforms Video-R1-7B and even matches Gemini-1.5-Pro across temporal
reasoning, goal and purpose reasoning, spatial reasoning, and plot reasoning on
our LongVideo-Reason-eval benchmark. Notably, our MR-SP system achieves up to
2.1x speedup on long video RL training. LongVILA-R1 demonstrates consistent
performance gains as the number of input video frames scales. LongVILA-R1 marks
a firm step towards long video reasoning in VLMs. In addition, we release our
training system for public availability that supports RL training on various
modalities (video, text, and audio), various models (VILA and Qwen series), and
even image and video generation models. On a single A100 node (8 GPUs), it
supports RL training on hour-long videos (e.g., 3,600 frames / around 256k
tokens).
