EgoAgent: A Joint Predictive Agent Model in Egocentric Worlds
Journal:
arXiv
Published Date:
Feb 9, 2025
Abstract
This paper addresses the task of learning an agent model behaving like
humans, which can jointly perceive, predict, and act in egocentric worlds.
Previous methods usually train separate models for these three abilities, which
prevents them from learning from each other. In this paper, we propose a joint
predictive agent model, named EgoAgent, that simultaneously learns to represent
the world, predict future states, and take reasonable actions within a single
transformer. EgoAgent introduces two innovations to learn from the causal and
temporally intertwined nature of these abilities: (1) Interleaved sequential
modeling of states and actions with the causal attention mechanism, and (2) A
joint embedding-action-prediction architecture featuring temporal asymmetric
predictor-observer branches. Integrating these designs based on JEPA, EgoAgent
unifies these capabilities in a cohesive learning framework. Comprehensive
evaluations of EgoAgent on representative tasks such as image classification,
egocentric future state prediction, and 3D human motion prediction tasks
demonstrate the superiority of our method. The code and trained model will be
released for reproducibility.
