Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System
Journal:
arXiv
Published Date:
Jun 10, 2025
Abstract
Autonomous agents powered by multimodal large language models have been
developed to facilitate task execution on mobile devices. However, prior work
has predominantly focused on atomic tasks -- such as shot-chain execution tasks
and single-screen grounding tasks -- while overlooking the generalization to
compositional tasks, which are indispensable for real-world applications. This
work introduces UI-NEXUS, a comprehensive benchmark designed to evaluate mobile
agents on three categories of compositional operations: Simple Concatenation,
Context Transition, and Deep Dive. UI-NEXUS supports interactive evaluation in
20 fully controllable local utility app environments, as well as 30 online
Chinese and English service apps. It comprises 100 interactive task templates
with an average optimal step count of 14.05. Experimental results across a
range of mobile agents with agentic workflow or agent-as-a-model show that
UI-NEXUS presents significant challenges. Specifically, existing agents
generally struggle to balance performance and efficiency, exhibiting
representative failure modes such as under-execution, over-execution, and
attention drift, causing visible atomic-to-compositional generalization gap.
Inspired by these findings, we propose AGENT-NEXUS, a lightweight and efficient
scheduling system to tackle compositional mobile tasks. AGENT-NEXUS
extrapolates the abilities of existing mobile agents by dynamically decomposing
long-horizon tasks to a series of self-contained atomic subtasks. AGENT-NEXUS
achieves 24% to 40% task success rate improvement for existing mobile agents on
compositional operation tasks within the UI-NEXUS benchmark without
significantly sacrificing inference overhead. The demo video, dataset, and code
are available on the project page at https://ui-nexus.github.io.
