PGOV3D: Open-Vocabulary 3D Semantic Segmentation with Partial-to-Global Curriculum
Journal:
arXiv
Published Date:
Jun 30, 2025
Abstract
Existing open-vocabulary 3D semantic segmentation methods typically supervise
3D segmentation models by merging text-aligned features (e.g., CLIP) extracted
from multi-view images onto 3D points. However, such approaches treat
multi-view images merely as intermediaries for transferring open-vocabulary
information, overlooking their rich semantic content and cross-view
correspondences, which limits model effectiveness. To address this, we propose
PGOV3D, a novel framework that introduces a Partial-to-Global curriculum for
improving open-vocabulary 3D semantic segmentation. The key innovation lies in
a two-stage training strategy. In the first stage, we pre-train the model on
partial scenes that provide dense semantic information but relatively simple
geometry. These partial point clouds are derived from multi-view RGB-D inputs
via pixel-wise depth projection. To enable open-vocabulary learning, we
leverage a multi-modal large language model (MLLM) and a 2D segmentation
foundation model to generate open-vocabulary labels for each viewpoint,
offering rich and aligned supervision. An auxiliary inter-frame consistency
module is introduced to enforce feature consistency across varying viewpoints
and enhance spatial understanding. In the second stage, we fine-tune the model
on complete scene-level point clouds, which are sparser and structurally more
complex. We aggregate the partial vocabularies associated with each scene and
generate pseudo labels using the pre-trained model, effectively bridging the
semantic gap between dense partial observations and large-scale 3D
environments. Extensive experiments on ScanNet, ScanNet200, and S3DIS
benchmarks demonstrate that PGOV3D achieves competitive performance in
open-vocabulary 3D semantic segmentation.
