EscherNet++: Simultaneous Amodal Completion and Scalable View Synthesis through Masked Fine-Tuning and Enhanced Feed-Forward 3D Reconstruction
Journal:
arXiv
Published Date:
Jul 10, 2025
Abstract
We propose EscherNet++, a masked fine-tuned diffusion model that can
synthesize novel views of objects in a zero-shot manner with amodal completion
ability. Existing approaches utilize multiple stages and complex pipelines to
first hallucinate missing parts of the image and then perform novel view
synthesis, which fail to consider cross-view dependencies and require redundant
storage and computing for separate stages. Instead, we apply masked fine-tuning
including input-level and feature-level masking to enable an end-to-end model
with the improved ability to synthesize novel views and conduct amodal
completion. In addition, we empirically integrate our model with other
feed-forward image-to-mesh models without extra training and achieve
competitive results with reconstruction time decreased by 95%, thanks to its
ability to synthesize arbitrary query views. Our method's scalable nature
further enhances fast 3D reconstruction. Despite fine-tuning on a smaller
dataset and batch size, our method achieves state-of-the-art results, improving
PSNR by 3.9 and Volume IoU by 0.28 on occluded tasks in 10-input settings,
while also generalizing to real-world occluded reconstruction.
