DenseGrounding: Improving Dense Language-Vision Semantics for Ego-Centric 3D Visual Grounding
Journal:
arXiv
Published Date:
May 8, 2025
Abstract
Enabling intelligent agents to comprehend and interact with 3D environments
through natural language is crucial for advancing robotics and human-computer
interaction. A fundamental task in this field is ego-centric 3D visual
grounding, where agents locate target objects in real-world 3D spaces based on
verbal descriptions. However, this task faces two significant challenges: (1)
loss of fine-grained visual semantics due to sparse fusion of point clouds with
ego-centric multi-view images, (2) limited textual semantic context due to
arbitrary language descriptions. We propose DenseGrounding, a novel approach
designed to address these issues by enhancing both visual and textual
semantics. For visual features, we introduce the Hierarchical Scene Semantic
Enhancer, which retains dense semantics by capturing fine-grained global scene
features and facilitating cross-modal alignment. For text descriptions, we
propose a Language Semantic Enhancer that leverages large language models to
provide rich context and diverse language descriptions with additional context
during model training. Extensive experiments show that DenseGrounding
significantly outperforms existing methods in overall accuracy, with
improvements of 5.81% and 7.56% when trained on the comprehensive full dataset
and smaller mini subset, respectively, further advancing the SOTA in egocentric
3D visual grounding. Our method also achieves 1st place and receives the
Innovation Award in the CVPR 2024 Autonomous Grand Challenge Multi-view 3D
Visual Grounding Track, validating its effectiveness and robustness.
