Prompt-Free Conditional Diffusion for Multi-object Image Augmentation
Journal:
arXiv
Published Date:
Jul 8, 2025
Abstract
Diffusion models has underpinned much recent advances of dataset augmentation
in various computer vision tasks. However, when involving generating
multi-object images as real scenarios, most existing methods either rely
entirely on text condition, resulting in a deviation between the generated
objects and the original data, or rely too much on the original images,
resulting in a lack of diversity in the generated images, which is of limited
help to downstream tasks. To mitigate both problems with one stone, we propose
a prompt-free conditional diffusion framework for multi-object image
augmentation. Specifically, we introduce a local-global semantic fusion
strategy to extract semantics from images to replace text, and inject knowledge
into the diffusion model through LoRA to alleviate the category deviation
between the original model and the target dataset. In addition, we design a
reward model based counting loss to assist the traditional reconstruction loss
for model training. By constraining the object counts of each category instead
of pixel-by-pixel constraints, bridging the quantity deviation between the
generated data and the original data while improving the diversity of the
generated data. Experimental results demonstrate the superiority of the
proposed method over several representative state-of-the-art baselines and
showcase strong downstream task gain and out-of-domain generalization
capabilities. Code is available at
\href{https://github.com/00why00/PFCD}{here}.
