Free-form language-based robotic reasoning and grasping
Journal:
arXiv
Published Date:
Mar 17, 2025
Abstract
Performing robotic grasping from a cluttered bin based on human instructions
is a challenging task, as it requires understanding both the nuances of
free-form language and the spatial relationships between objects.
Vision-Language Models (VLMs) trained on web-scale data, such as GPT-4o, have
demonstrated remarkable reasoning capabilities across both text and images. But
can they truly be used for this task in a zero-shot setting? And what are their
limitations? In this paper, we explore these research questions via the
free-form language-based robotic grasping task, and propose a novel method,
FreeGrasp, leveraging the pre-trained VLMs' world knowledge to reason about
human instructions and object spatial arrangements. Our method detects all
objects as keypoints and uses these keypoints to annotate marks on images,
aiming to facilitate GPT-4o's zero-shot spatial reasoning. This allows our
method to determine whether a requested object is directly graspable or if
other objects must be grasped and removed first. Since no existing dataset is
specifically designed for this task, we introduce a synthetic dataset
FreeGraspData by extending the MetaGraspNetV2 dataset with human-annotated
instructions and ground-truth grasping sequences. We conduct extensive analyses
with both FreeGraspData and real-world validation with a gripper-equipped
robotic arm, demonstrating state-of-the-art performance in grasp reasoning and
execution. Project website: https://tev-fbk.github.io/FreeGrasp/.
