CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory
Journal:
arXiv
Published Date:
Mar 4, 2025
Abstract
Large language models like GPT-4 are resource-intensive, but recent
advancements suggest that smaller, specialized experts can outperform the
monolithic models on specific tasks. The Collaboration-of-Experts (CoE)
approach integrates multiple expert models, improving the accuracy of generated
results and offering great potential for precision-critical applications, such
as automatic circuit board quality inspection. However, deploying CoE serving
systems presents challenges to memory capacity due to the large number of
experts required, which can lead to significant performance overhead from
frequent expert switching across different memory and storage tiers.
We propose CoServe, an efficient CoE model serving system on heterogeneous
CPU and GPU with limited memory. CoServe reduces unnecessary expert switching
by leveraging expert dependency, a key property of CoE inference. CoServe
introduces a dependency-aware request scheduler and dependency-aware expert
management for efficient inference. It also introduces an offline profiler to
automatically find optimal resource allocation on various processors and
devices. In real-world intelligent manufacturing workloads, CoServe achieves
4.5$\times$ to 12$\times$ higher throughput compared to state-of-the-art
systems.
