AsyncDSB: Schedule-Asynchronous Diffusion Schrödinger Bridge for Image Inpainting
Journal:
arXiv
Published Date:
Dec 11, 2024
Abstract
Image inpainting is an important image generation task, which aims to restore
corrupted image from partial visible area. Recently, diffusion Schr\"odinger
bridge methods effectively tackle this task by modeling the translation between
corrupted and target images as a diffusion Schr\"odinger bridge process along a
noising schedule path. Although these methods have shown superior performance,
in this paper, we find that 1) existing methods suffer from a
schedule-restoration mismatching issue, i.e., the theoretical schedule and
practical restoration processes usually exist a large discrepancy, which
theoretically results in the schedule not fully leveraged for restoring images;
and 2) the key reason causing such issue is that the restoration process of all
pixels are actually asynchronous but existing methods set a synchronous noise
schedule to them, i.e., all pixels shares the same noise schedule. To this end,
we propose a schedule-Asynchronous Diffusion Schr\"odinger Bridge (AsyncDSB)
for image inpainting. Our insight is preferentially scheduling pixels with high
frequency (i.e., large gradients) and then low frequency (i.e., small
gradients). Based on this insight, given a corrupted image, we first train a
network to predict its gradient map in corrupted area. Then, we regard the
predicted image gradient as prior and design a simple yet effective
pixel-asynchronous noise schedule strategy to enhance the diffusion
Schr\"odinger bridge. Thanks to the asynchronous schedule at pixels, the
temporal interdependence of restoration process between pixels can be fully
characterized for high-quality image inpainting. Experiments on real-world
datasets show that our AsyncDSB achieves superior performance, especially on
FID with around 3% - 14% improvement over state-of-the-art baseline methods.
