Shifting AI Efficiency From Model-Centric to Data-Centric Compression
Journal:
arXiv
Published Date:
May 25, 2025
Abstract
The rapid advancement of large language models (LLMs) and multi-modal LLMs
(MLLMs) has historically relied on model-centric scaling through increasing
parameter counts from millions to hundreds of billions to drive performance
gains. However, as we approach hardware limits on model size, the dominant
computational bottleneck has fundamentally shifted to the quadratic cost of
self-attention over long token sequences, now driven by ultra-long text
contexts, high-resolution images, and extended videos. In this position paper,
\textbf{we argue that the focus of research for efficient AI is shifting from
model-centric compression to data-centric compression}. We position token
compression as the new frontier, which improves AI efficiency via reducing the
number of tokens during model training or inference. Through comprehensive
analysis, we first examine recent developments in long-context AI across
various domains and establish a unified mathematical framework for existing
model efficiency strategies, demonstrating why token compression represents a
crucial paradigm shift in addressing long-context overhead. Subsequently, we
systematically review the research landscape of token compression, analyzing
its fundamental benefits and identifying its compelling advantages across
diverse scenarios. Furthermore, we provide an in-depth analysis of current
challenges in token compression research and outline promising future
directions. Ultimately, our work aims to offer a fresh perspective on AI
efficiency, synthesize existing research, and catalyze innovative developments
to address the challenges that increasing context lengths pose to the AI
community's advancement.
