Exploring Disentangled and Controllable Human Image Synthesis: From End-to-End to Stage-by-Stage
Journal:
arXiv
Published Date:
Mar 25, 2025
Abstract
Achieving fine-grained controllability in human image synthesis is a
long-standing challenge in computer vision. Existing methods primarily focus on
either facial synthesis or near-frontal body generation, with limited ability
to simultaneously control key factors such as viewpoint, pose, clothing, and
identity in a disentangled manner. In this paper, we introduce a new
disentangled and controllable human synthesis task, which explicitly separates
and manipulates these four factors within a unified framework. We first develop
an end-to-end generative model trained on MVHumanNet for factor
disentanglement. However, the domain gap between MVHumanNet and in-the-wild
data produce unsatisfacotry results, motivating the exploration of virtual
try-on (VTON) dataset as a potential solution. Through experiments, we observe
that simply incorporating the VTON dataset as additional data to train the
end-to-end model degrades performance, primarily due to the inconsistency in
data forms between the two datasets, which disrupts the disentanglement
process. To better leverage both datasets, we propose a stage-by-stage
framework that decomposes human image generation into three sequential steps:
clothed A-pose generation, back-view synthesis, and pose and view control. This
structured pipeline enables better dataset utilization at different stages,
significantly improving controllability and generalization, especially for
in-the-wild scenarios. Extensive experiments demonstrate that our
stage-by-stage approach outperforms end-to-end models in both visual fidelity
and disentanglement quality, offering a scalable solution for real-world tasks.
Additional demos are available on the project page:
https://taited.github.io/discohuman-project/.
