Mavors: Multi-granularity Video Representation for Multimodal Large Language Model
Journal:
arXiv
Published Date:
Apr 14, 2025
Abstract
Long-context video understanding in multimodal large language models (MLLMs)
faces a critical challenge: balancing computational efficiency with the
retention of fine-grained spatio-temporal patterns. Existing approaches (e.g.,
sparse sampling, dense sampling with low resolution, and token compression)
suffer from significant information loss in temporal dynamics, spatial details,
or subtle interactions, particularly in videos with complex motion or varying
resolutions. To address this, we propose $\mathbf{Mavors}$, a novel framework
that introduces $\mathbf{M}$ulti-gr$\mathbf{a}$nularity
$\mathbf{v}$ide$\mathbf{o}$ $\mathbf{r}$epre$\mathbf{s}$entation for holistic
long-video modeling. Specifically, Mavors directly encodes raw video content
into latent representations through two core components: 1) an Intra-chunk
Vision Encoder (IVE) that preserves high-resolution spatial features via 3D
convolutions and Vision Transformers, and 2) an Inter-chunk Feature Aggregator
(IFA) that establishes temporal coherence across chunks using transformer-based
dependency modeling with chunk-level rotary position encodings. Moreover, the
framework unifies image and video understanding by treating images as
single-frame videos via sub-image decomposition. Experiments across diverse
benchmarks demonstrate Mavors' superiority in maintaining both spatial fidelity
and temporal continuity, significantly outperforming existing methods in tasks
requiring fine-grained spatio-temporal reasoning.
