DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMs
Journal:
arXiv
Published Date:
Apr 23, 2025
Abstract
We present DyMU, an efficient, training-free framework that dynamically
reduces the computational burden of vision-language models (VLMs) while
maintaining high task performance. Our approach comprises two key components.
First, Dynamic Token Merging (DToMe) reduces the number of visual token
embeddings by merging similar tokens based on image complexity, addressing the
inherent inefficiency of fixed-length outputs in vision transformers. Second,
Virtual Token Unmerging (VTU) simulates the expected token sequence for large
language models (LLMs) by efficiently reconstructing the attention dynamics of
a full sequence, thus preserving the downstream performance without additional
fine-tuning. Unlike previous approaches, our method dynamically adapts token
compression to the content of the image and operates completely training-free,
making it readily applicable to most state-of-the-art VLM architectures.
Extensive experiments on image and video understanding tasks demonstrate that
DyMU can reduce the average visual token count by 32%-85% while achieving
comparable performance to full-length models across diverse VLM architectures,
including the recently popularized AnyRes-based visual encoders. Furthermore,
through qualitative analyses, we demonstrate that DToMe effectively adapts
token reduction based on image complexity and, unlike existing systems,
provides users more control over computational costs. Project page:
https://mikewangwzhl.github.io/dymu/.
