DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation
Journal:
arXiv
Published Date:
Jan 28, 2025
Abstract
Recent advancements in 3D content generation from text or a single image
struggle with limited high-quality 3D datasets and inconsistency from 2D
multi-view generation. We introduce DiffSplat, a novel 3D generative framework
that natively generates 3D Gaussian splats by taming large-scale text-to-image
diffusion models. It differs from previous 3D generative models by effectively
utilizing web-scale 2D priors while maintaining 3D consistency in a unified
model. To bootstrap the training, a lightweight reconstruction model is
proposed to instantly produce multi-view Gaussian splat grids for scalable
dataset curation. In conjunction with the regular diffusion loss on these
grids, a 3D rendering loss is introduced to facilitate 3D coherence across
arbitrary views. The compatibility with image diffusion models enables seamless
adaptions of numerous techniques for image generation to the 3D realm.
Extensive experiments reveal the superiority of DiffSplat in text- and
image-conditioned generation tasks and downstream applications. Thorough
ablation studies validate the efficacy of each critical design choice and
provide insights into the underlying mechanism.
