AnyRefill: A Unified, Data-Efficient Framework for Left-Prompt-Guided Vision Tasks
Journal:
arXiv
Published Date:
Feb 16, 2025
Abstract
In this paper, we present a novel Left-Prompt-Guided (LPG) paradigm to
address a diverse range of reference-based vision tasks. Inspired by the human
creative process, we reformulate these tasks using a left-right stitching
formulation to construct contextual input. Building upon this foundation, we
propose AnyRefill, an extension of LeftRefill, that effectively adapts
Text-to-Image (T2I) models to various vision tasks. AnyRefill leverages the
inpainting priors of advanced T2I model based on the Diffusion Transformer
(DiT) architecture, and incorporates flexible components to enhance its
capabilities. By combining task-specific LoRAs with the stitching input,
AnyRefill unlocks its potential across diverse tasks, including conditional
generation, visual perception, and image editing, without requiring additional
visual encoders. Meanwhile, AnyRefill exhibits remarkable data efficiency,
requiring minimal task-specific fine-tuning while maintaining high generative
performance. Through extensive ablation studies, we demonstrate that AnyRefill
outperforms other image condition injection methods and achieves competitive
results compared to state-of-the-art open-source methods. Notably, AnyRefill
delivers results comparable to advanced commercial tools, such as IC-Light and
SeedEdit, even in challenging scenarios. Comprehensive experiments and ablation
studies across versatile tasks validate the strong generation of the proposed
simple yet effective LPG formulation, establishing AnyRefill as a unified,
highly data-efficient solution for reference-based vision tasks.
