LIFT-GS: Cross-Scene Render-Supervised Distillation for 3D Language Grounding
Journal:
arXiv
Published Date:
Feb 27, 2025
Abstract
Our approach to training 3D vision-language understanding models is to train
a feedforward model that makes predictions in 3D, but never requires 3D labels
and is supervised only in 2D, using 2D losses and differentiable rendering. The
approach is new for vision-language understanding. By treating the
reconstruction as a ``latent variable'', we can render the outputs without
placing unnecessary constraints on the network architecture (e.g. can be used
with decoder-only models). For training, only need images and camera pose, and
2D labels. We show that we can even remove the need for 2D labels by using
pseudo-labels from pretrained 2D models. We demonstrate this to pretrain a
network, and we finetune it for 3D vision-language understanding tasks. We show
this approach outperforms baselines/sota for 3D vision-language grounding, and
also outperforms other 3D pretraining techniques. Project page:
https://liftgs.github.io.
