SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models
Journal:
arXiv
Published Date:
May 12, 2025
Abstract
Large Language Models (LLMs) present a critical trade-off between inference
quality and computational cost: larger models offer superior capabilities but
incur significant latency, while smaller models are faster but less powerful.
Existing serving strategies often employ fixed model scales or static two-stage
speculative decoding, failing to dynamically adapt to the varying complexities
of user requests or fluctuations in system performance. This paper introduces
\systemname{}, a novel framework that reimagines LLM inference as an adaptive
routing problem solved through multi-level speculative decoding. \systemname{}
dynamically constructs and optimizes inference "paths" (chains of models) based
on real-time feedback, addressing the limitations of static approaches. Our
contributions are threefold: (1) An \textbf{adaptive model chain scheduling}
mechanism that leverages performance profiling (execution times) and predictive
similarity metrics (derived from token distribution divergence) to continuously
select the optimal sequence of draft and verifier models, minimizing predicted
latency per generated token. (2) A \textbf{multi-level collaborative
verification} framework where intermediate models within the selected chain can
validate speculative tokens, reducing the verification burden on the final,
most powerful target model. (3) A \textbf{synchronized state management} system
providing efficient, consistent KV cache handling across heterogeneous models
in the chain, including precise, low-overhead rollbacks tailored for
asynchronous batch processing inherent in multi-level speculation. Preliminary
experiments demonstrate the validity of our method.
