IPDN: Image-enhanced Prompt Decoding Network for 3D Referring Expression Segmentation
Journal:
arXiv
Published Date:
Jan 9, 2025
Abstract
3D Referring Expression Segmentation (3D-RES) aims to segment point cloud
scenes based on a given expression. However, existing 3D-RES approaches face
two major challenges: feature ambiguity and intent ambiguity. Feature ambiguity
arises from information loss or distortion during point cloud acquisition due
to limitations such as lighting and viewpoint. Intent ambiguity refers to the
model's equal treatment of all queries during the decoding process, lacking
top-down task-specific guidance. In this paper, we introduce an Image enhanced
Prompt Decoding Network (IPDN), which leverages multi-view images and
task-driven information to enhance the model's reasoning capabilities. To
address feature ambiguity, we propose the Multi-view Semantic Embedding (MSE)
module, which injects multi-view 2D image information into the 3D scene and
compensates for potential spatial information loss. To tackle intent ambiguity,
we designed a Prompt-Aware Decoder (PAD) that guides the decoding process by
deriving task-driven signals from the interaction between the expression and
visual features. Comprehensive experiments demonstrate that IPDN outperforms
the state-ofthe-art by 1.9 and 4.2 points in mIoU metrics on the 3D-RES and
3D-GRES tasks, respectively.
