ADAMIX: Adaptive Mixed-Precision Delta-Compression with Quantization Error Optimization for Large Language Models
Journal:
arXiv
Published Date:
Jun 5, 2025
Abstract
Large language models (LLMs) achieve impressive performance on various
knowledge-intensive and complex reasoning tasks in different domains. In
certain scenarios like multi-tenant serving, a large number of LLMs finetuned
from the same base model are deployed to meet complex requirements for users.
Recent works explore delta-compression approaches to quantize and compress the
delta parameters between the customized LLM and the corresponding base model.
However, existing works either exhibit unsatisfactory performance at high
compression ratios or depend on empirical bit allocation schemes. In this work,
we propose ADAMIX, an effective adaptive mixed-precision delta-compression
framework. We provide a mathematical derivation of quantization error to
motivate our mixed-precision compression strategy and formulate the optimal
mixed-precision bit allocation scheme as the solution to a 0/1 integer linear
programming problem. Our derived bit allocation strategy minimizes the
quantization error while adhering to a predefined compression ratio
requirement. Experimental results on various models and benchmarks demonstrate
that our approach surpasses the best baseline by a considerable margin. On
tasks like AIME2024 and GQA, where the norm of $\Delta \mathbf{W}$ is large and
the base model lacks sufficient ability, ADAMIX outperforms the best baseline
Delta-CoMe by 22.3% and 6.1% with 7B models, respectively.
