P3S-Diffusion:A Selective Subject-driven Generation Framework via Point Supervision
Journal:
arXiv
Published Date:
Dec 27, 2024
Abstract
Recent research in subject-driven generation increasingly emphasizes the
importance of selective subject features. Nevertheless, accurately selecting
the content in a given reference image still poses challenges, especially when
selecting the similar subjects in an image (e.g., two different dogs). Some
methods attempt to use text prompts or pixel masks to isolate specific
elements. However, text prompts often fall short in precisely describing
specific content, and pixel masks are often expensive. To address this, we
introduce P3S-Diffusion, a novel architecture designed for context-selected
subject-driven generation via point supervision. P3S-Diffusion leverages
minimal cost label (e.g., points) to generate subject-driven images. During
fine-tuning, it can generate an expanded base mask from these points, obviating
the need for additional segmentation models. The mask is employed for
inpainting and aligning with subject representation. The P3S-Diffusion
preserves fine features of the subjects through Multi-layers Condition
Injection. Enhanced by the Attention Consistency Loss for improved training,
extensive experiments demonstrate its excellent feature preservation and image
generation capabilities.
