Enhancing image-based virtual try-on with Multi-Controlled Diffusion Models.
Journal:
Neural networks : the official journal of the International Neural Network Society
Published Date:
May 23, 2025
Abstract
Image-based virtual try-on technology digitally overlays clothing onto images of individuals, enabling users to preview how garments fit without physical trial, thus enhancing the online shopping experience. While current diffusion-based virtual try-on networks produce high-quality results, they struggle to accurately render garments with textual designs such as logos or prints which are widely prevalent in the real world, often carrying significant brand and cultural identities. To address this challenge, we introduce the Multi-Controlled Diffusion Models for Image-based Virtual Try-On (MCDM-VTON), a novel approach that synergistically incorporates global image features and local textual features extracted from garments to control the generation process. Specifically, we innovatively introduce an Optical Character Recognition (OCR) model to extract the text-style textures from clothing, utilizing the information gathered as text features. These features, in conjunction with the inherent global image features through a multimodal feature fusion module based on cross-attention, jointly control the denoising process of the diffusion models. Moreover, by extracting text information from both the generated virtual try-on results and the original garment images with the OCR model, we have devised a new content-style loss to supervise the training of diffusion models, thereby reinforcing the generation effect of text-style textures. Extensive experiments demonstrate that MCDM-VTON significantly outperforms existing state-of-the-art methods in terms of text preservation and overall visual quality.
