3D-Grounded Vision-Language Framework for Robotic Task Planning: Automated Prompt Synthesis and Supervised Reasoning
Journal:
arXiv
Published Date:
Feb 13, 2025
Abstract
Vision-language models (VLMs) have achieved remarkable success in scene
understanding and perception tasks, enabling robots to plan and execute actions
adaptively in dynamic environments. However, most multimodal large language
models lack robust 3D scene localization capabilities, limiting their
effectiveness in fine-grained robotic operations. Additionally, challenges such
as low recognition accuracy, inefficiency, poor transferability, and
reliability hinder their use in precision tasks. To address these limitations,
we propose a novel framework that integrates a 2D prompt synthesis module by
mapping 2D images to point clouds, and incorporates a small language model
(SLM) for supervising VLM outputs. The 2D prompt synthesis module enables VLMs,
trained on 2D images and text, to autonomously extract precise 3D spatial
information without manual intervention, significantly enhancing 3D scene
understanding. Meanwhile, the SLM supervises VLM outputs, mitigating
hallucinations and ensuring reliable, executable robotic control code
generation. Our framework eliminates the need for retraining in new
environments, thereby improving cost efficiency and operational robustness.
Experimental results that the proposed framework achieved a 96.0\% Task Success
Rate (TSR), outperforming other methods. Ablation studies demonstrated the
critical role of both the 2D prompt synthesis module and the output supervision
module (which, when removed, caused a 67\% TSR drop). These findings validate
the framework's effectiveness in improving 3D recognition, task planning, and
robotic task execution.
