FreeInsert: Disentangled Text-Guided Object Insertion in 3D Gaussian Scene without Spatial Priors
Journal:
arXiv
Published Date:
May 2, 2025
Abstract
Text-driven object insertion in 3D scenes is an emerging task that enables
intuitive scene editing through natural language. However, existing 2D
editing-based methods often rely on spatial priors such as 2D masks or 3D
bounding boxes, and they struggle to ensure consistency of the inserted object.
These limitations hinder flexibility and scalability in real-world
applications. In this paper, we propose FreeInsert, a novel framework that
leverages foundation models including MLLMs, LGMs, and diffusion models to
disentangle object generation from spatial placement. This enables unsupervised
and flexible object insertion in 3D scenes without spatial priors. FreeInsert
starts with an MLLM-based parser that extracts structured semantics, including
object types, spatial relationships, and attachment regions, from user
instructions. These semantics guide both the reconstruction of the inserted
object for 3D consistency and the learning of its degrees of freedom. We
leverage the spatial reasoning capabilities of MLLMs to initialize object pose
and scale. A hierarchical, spatially aware refinement stage further integrates
spatial semantics and MLLM-inferred priors to enhance placement. Finally, the
appearance of the object is improved using the inserted-object image to enhance
visual fidelity. Experimental results demonstrate that FreeInsert achieves
semantically coherent, spatially precise, and visually realistic 3D insertions
without relying on spatial priors, offering a user-friendly and flexible
editing experience.
