PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models
Journal:
arXiv
Published Date:
Dec 12, 2024
Abstract
Large Vision-Language Models (VLMs) have been extended to understand both
images and videos. Visual token compression is leveraged to reduce the
considerable token length of visual inputs. To meet the needs of different
tasks, existing high-performance models usually process images and videos
separately with different token compression strategies, limiting the
capabilities of combining images and videos. To this end, we extend each image
into a "static" video and introduce a unified token compression strategy called
Progressive Visual Token Compression (PVC), where the tokens of each frame are
progressively encoded and adaptively compressed to supplement the information
not extracted from previous frames. Video tokens are efficiently compressed
with exploiting the inherent temporal redundancy. Images are repeated as static
videos, and the spatial details can be gradually supplemented in multiple
frames. PVC unifies the token compressing of images and videos. With a limited
number of tokens per frame (64 tokens by default), spatial details and temporal
changes can still be preserved. Experiments show that our model achieves
state-of-the-art performance across various video understanding benchmarks,
including long video tasks and fine-grained short video tasks. Meanwhile, our
unified token compression strategy incurs no performance loss on image
benchmarks, particularly in detail-sensitive tasks.
