Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoC
Journal:
arXiv
Published Date:
Jun 30, 2025
Abstract
The proliferation of agentic Large Language Models (LLMs) on personal devices
introduces a new class of workloads characterized by a dichotomy of objectives.
Reactive tasks, initiated by users, demand immediate, low-latency responses,
while proactive tasks operate invisibly and prioritize throughput. Existing
on-device LLM engines, designed for isolated inferences, fail to efficiently
manage these concurrent and conflicting requests on consumer-grade
heterogeneous SoCs with CPU, integrated GPU, and NPU. This paper introduces
Agent.xpu, an efficient serving system for agentic LLM workloads on
memory-unified heterogeneous SoCs. With dedicated offline profiling, Agent.xpu
first constructs a heterogeneous execution graph, which fuses and chunks model
kernels for affinity-guided, elastic accelerator mapping with predictive kernel
annotation. At runtime, its online scheduler enables fine-grained, kernel-level
preemption to guarantee the responsiveness of reactive tasks. To maximize SoC
utilization, it adopts slack-aware kernel backfill to opportunistically append
proactive tasks, and mitigates NPU-iGPU contention via bandwidth-aware
dispatch. Evaluation on an Intel Core Ultra SoC shows that Agent.xpu achieves
4.6$\times$ lower latency for reactive tasks and sustains
1.6$\times$-6.8$\times$ higher throughput for proactive tasks compared to
state-of-the-art inference engines.
