2024 NASA SUITS Report: LLM-Driven Immersive Augmented Reality User Interface for Robotics and Space Exploration
Journal:
arXiv
Published Date:
Jul 1, 2025
Abstract
As modern computing advances, new interaction paradigms have emerged,
particularly in Augmented Reality (AR), which overlays virtual interfaces onto
physical objects. This evolution poses challenges in machine perception,
especially for tasks like 3D object pose estimation in complex, dynamic
environments. Our project addresses critical issues in human-robot interaction
within mobile AR, focusing on non-intrusive, spatially aware interfaces. We
present URSA, an LLM-driven immersive AR system developed for NASA's 2023-2024
SUITS challenge, targeting future spaceflight needs such as the Artemis
missions. URSA integrates three core technologies: a head-mounted AR device
(e.g., HoloLens) for intuitive visual feedback, voice control powered by large
language models for hands-free interaction, and robot tracking algorithms that
enable accurate 3D localization in dynamic settings. To enhance precision, we
leverage digital twin localization technologies, using datasets like
DTTD-Mobile and specialized hardware such as the ZED2 camera for real-world
tracking under noise and occlusion. Our system enables real-time robot control
and monitoring via an AR interface, even in the absence of ground-truth
sensors--vital for hazardous or remote operations. Key contributions include:
(1) a non-intrusive AR interface with LLM-based voice input; (2) a ZED2-based
dataset tailored for non-rigid robotic bodies; (3) a Local Mission Control
Console (LMCC) for mission visualization; (4) a transformer-based 6DoF pose
estimator (DTTDNet) optimized for depth fusion and real-time tracking; and (5)
end-to-end integration for astronaut mission support. This work advances
digital twin applications in robotics, offering scalable solutions for both
aerospace and industrial domains.
