Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
Journal:
arXiv
Published Date:
Feb 9, 2025
Abstract
In image processing, solving inverse problems is the task of finding
plausible reconstructions of an image that was corrupted by some (usually
known) degradation model. Commonly, this process is done using a generative
image model that can guide the reconstruction towards solutions that appear
natural. The success of diffusion models over the last few years has made them
a leading candidate for this task. However, the sequential nature of diffusion
models makes this conditional sampling process challenging. Furthermore, since
diffusion models are often defined in the latent space of an autoencoder, the
encoder-decoder transformations introduce additional difficulties. Here, we
suggest a novel sampling method based on sequential Monte Carlo (SMC) in the
latent space of diffusion models. We use the forward process of the diffusion
model to add additional auxiliary observations and then perform an SMC sampling
as part of the backward process. Empirical evaluations on ImageNet and FFHQ
show the benefits of our approach over competing methods on various inverse
problem tasks.
