Visual Agentic AI for Spatial Reasoning with a Dynamic API
Journal:
arXiv
Published Date:
Feb 10, 2025
Abstract
Visual reasoning -- the ability to interpret the visual world -- is crucial
for embodied agents that operate within three-dimensional scenes. Progress in
AI has led to vision and language models capable of answering questions from
images. However, their performance declines when tasked with 3D spatial
reasoning. To tackle the complexity of such reasoning problems, we introduce an
agentic program synthesis approach where LLM agents collaboratively generate a
Pythonic API with new functions to solve common subproblems. Our method
overcomes limitations of prior approaches that rely on a static, human-defined
API, allowing it to handle a wider range of queries. To assess AI capabilities
for 3D understanding, we introduce a new benchmark of queries involving
multiple steps of grounding and inference. We show that our method outperforms
prior zero-shot models for visual reasoning in 3D and empirically validate the
effectiveness of our agentic framework for 3D spatial reasoning tasks. Project
website: https://glab-caltech.github.io/vadar/
