Facial Appearance Capture at Home with Patch-Level Reflectance Prior
Journal:
arXiv
Published Date:
Jun 4, 2025
Abstract
Existing facial appearance capture methods can reconstruct plausible facial
reflectance from smartphone-recorded videos. However, the reconstruction
quality is still far behind the ones based on studio recordings. This paper
fills the gap by developing a novel daily-used solution with a co-located
smartphone and flashlight video capture setting in a dim room. To enhance the
quality, our key observation is to solve facial reflectance maps within the
data distribution of studio-scanned ones. Specifically, we first learn a
diffusion prior over the Light Stage scans and then steer it to produce the
reflectance map that best matches the captured images. We propose to train the
diffusion prior at the patch level to improve generalization ability and
training stability, as current Light Stage datasets are in ultra-high
resolution but limited in data size. Tailored to this prior, we propose a
patch-level posterior sampling technique to sample seamless full-resolution
reflectance maps from this patch-level diffusion model. Experiments demonstrate
our method closes the quality gap between low-cost and studio recordings by a
large margin, opening the door for everyday users to clone themselves to the
digital world. Our code will be released at https://github.com/yxuhan/DoRA.
