I Speak and You Find: Robust 3D Visual Grounding with Noisy and Ambiguous Speech Inputs
Journal:
arXiv
Published Date:
Jun 17, 2025
Abstract
Existing 3D visual grounding methods rely on precise text prompts to locate
objects within 3D scenes. Speech, as a natural and intuitive modality, offers a
promising alternative. Real-world speech inputs, however, often suffer from
transcription errors due to accents, background noise, and varying speech
rates, limiting the applicability of existing 3DVG methods. To address these
challenges, we propose \textbf{SpeechRefer}, a novel 3DVG framework designed to
enhance performance in the presence of noisy and ambiguous speech-to-text
transcriptions. SpeechRefer integrates seamlessly with xisting 3DVG models and
introduces two key innovations. First, the Speech Complementary Module captures
acoustic similarities between phonetically related words and highlights subtle
distinctions, generating complementary proposal scores from the speech signal.
This reduces dependence on potentially erroneous transcriptions. Second, the
Contrastive Complementary Module employs contrastive learning to align
erroneous text features with corresponding speech features, ensuring robust
performance even when transcription errors dominate. Extensive experiments on
the SpeechRefer and peechNr3D datasets demonstrate that SpeechRefer improves
the performance of existing 3DVG methods by a large margin, which highlights
SpeechRefer's potential to bridge the gap between noisy speech inputs and
reliable 3DVG, enabling more intuitive and practical multimodal systems.
