Niyama : Breaking the Silos of LLM Inference Serving
Journal:
arXiv
Published Date:
Mar 28, 2025
Abstract
The widespread adoption of Large Language Models (LLMs) has enabled diverse
applications with very different latency requirements. Existing LLM serving
frameworks rely on siloed infrastructure with coarse-grained workload
segregation -- interactive and batch -- leading to inefficient resource
utilization and limited support for fine-grained Quality-of-Service (QoS)
differentiation. This results in operational inefficiencies, over-provisioning
and poor load management during traffic surges.
We present Niyama, a novel QoS-driven inference serving system that enables
efficient co-scheduling of diverse workloads on shared infrastructure. Niyama
introduces fine-grained QoS classification allowing applications to specify
precise latency requirements, and dynamically adapts scheduling decisions based
on real-time system state. Leveraging the predictable execution characteristics
of LLM inference, Niyama implements a dynamic chunking mechanism to improve
overall throughput while maintaining strict QoS guarantees. Additionally,
Niyama employs a hybrid prioritization policy that balances fairness and
efficiency, and employs selective request relegation that enables graceful
service degradation during overload conditions. Our evaluation demonstrates
that Niyama increases serving capacity by 32% compared to current siloed
deployments, while maintaining QoS guarantees. Notably, under extreme load, our
system reduces SLO violations by an order of magnitude compared to current
strategies.
