Minions: Cost-efficient Collaboration Between On-device and Cloud Language Models
Journal:
arXiv
Published Date:
Feb 21, 2025
Abstract
We investigate an emerging setup in which a small, on-device language model
(LM) with access to local data communicates with a frontier, cloud-hosted LM to
solve real-world tasks involving financial, medical, and scientific reasoning
over long documents. Can a local-remote collaboration reduce cloud inference
costs while preserving quality? First, we consider a naive collaboration
protocol where the local and remote models simply chat back and forth. Because
only the local model reads the full context, this protocol achieves a 30.4x
reduction in remote costs, but recovers only 87% of the performance of the
frontier model. We identify two key limitations of this protocol: the local
model struggles to (1) follow the remote model's multi-step instructions and
(2) reason over long contexts. Motivated by these observations, we study an
extension of this protocol, coined MinionS, in which the remote model
decomposes the task into easier subtasks over shorter chunks of the document,
that are executed locally in parallel. MinionS reduces costs by 5.7x on average
while recovering 97.9% of the performance of the remote model alone. Our
analysis reveals several key design choices that influence the trade-off
between cost and performance in local-remote systems.
