MAPLE: A Mobile Assistant with Persistent Finite State Machines for Recovery Reasoning
Journal:
arXiv
Published Date:
May 29, 2025
Abstract
Mobile GUI agents aim to autonomously complete user-instructed tasks across
mobile apps. Recent advances in Multimodal Large Language Models (MLLMs) enable
these agents to interpret UI screens, identify actionable elements, and perform
interactions such as tapping or typing. However, existing agents remain
reactive: they reason only over the current screen and lack a structured model
of app navigation flow, limiting their ability to understand context, detect
unexpected outcomes, and recover from errors. We present MAPLE, a state-aware
multi-agent framework that abstracts app interactions as a Finite State Machine
(FSM). We computationally model each UI screen as a discrete state and user
actions as transitions, allowing the FSM to provide a structured representation
of the app execution. MAPLE consists of specialized agents responsible for four
phases of task execution: planning, execution, verification, error recovery,
and knowledge retention. These agents collaborate to dynamically construct FSMs
in real time based on perception data extracted from the UI screen, allowing
the GUI agents to track navigation progress and flow, validate action outcomes
through pre- and post-conditions of the states, and recover from errors by
rolling back to previously stable states. Our evaluation results on two
challenging cross-app benchmarks, Mobile-Eval-E and SPA-Bench, show that MAPLE
outperforms the state-of-the-art baseline, improving task success rate by up to
12%, recovery success by 13.8%, and action accuracy by 6.5%. Our results
highlight the importance of structured state modeling in guiding mobile GUI
agents during task execution. Moreover, our FSM representation can be
integrated into future GUI agent architectures as a lightweight, model-agnostic
memory layer to support structured planning, execution verification, and error
recovery.
