MMGen: Unified Multi-modal Image Generation and Understanding in One Go
Journal:
arXiv
Published Date:
Mar 26, 2025
Abstract
A unified diffusion framework for multi-modal generation and understanding
has the transformative potential to achieve seamless and controllable image
diffusion and other cross-modal tasks. In this paper, we introduce MMGen, a
unified framework that integrates multiple generative tasks into a single
diffusion model. This includes: (1) multi-modal category-conditioned
generation, where multi-modal outputs are generated simultaneously through a
single inference process, given category information; (2) multi-modal visual
understanding, which accurately predicts depth, surface normals, and
segmentation maps from RGB images; and (3) multi-modal conditioned generation,
which produces corresponding RGB images based on specific modality conditions
and other aligned modalities. Our approach develops a novel diffusion
transformer that flexibly supports multi-modal output, along with a simple
modality-decoupling strategy to unify various tasks. Extensive experiments and
applications demonstrate the effectiveness and superiority of MMGen across
diverse tasks and conditions, highlighting its potential for applications that
require simultaneous generation and understanding.
