Betsu-Betsu: Multi-View Separable 3D Reconstruction of Two Interacting Objects
Journal:
arXiv
Published Date:
Feb 19, 2025
Abstract
Separable 3D reconstruction of multiple objects from multi-view RGB images --
resulting in two different 3D shapes for the two objects with a clear
separation between them -- remains a sparsely researched problem. It is
challenging due to severe mutual occlusions and ambiguities along the objects'
interaction boundaries. This paper investigates the setting and introduces a
new neuro-implicit method that can reconstruct the geometry and appearance of
two objects undergoing close interactions while disjoining both in 3D, avoiding
surface inter-penetrations and enabling novel-view synthesis of the observed
scene. The framework is end-to-end trainable and supervised using a novel
alpha-blending regularisation that ensures that the two geometries are well
separated even under extreme occlusions. Our reconstruction method is
markerless and can be applied to rigid as well as articulated objects. We
introduce a new dataset consisting of close interactions between a human and an
object and also evaluate on two scenes of humans performing martial arts. The
experiments confirm the effectiveness of our framework and substantial
improvements using 3D and novel view synthesis metrics compared to several
existing approaches applicable in our setting.
