SPAST: Arbitrary style transfer with style priors via pre-trained large-scale model.

Journal: Neural networks : the official journal of the International Neural Network Society
Published Date:

Abstract

Given an arbitrary content and style image, arbitrary style transfer aims to render a new stylized image which preserves the content image's structure and possesses the style image's style. Existing arbitrary style transfer methods are based on either small models or pre-trained large-scale models. The small model-based methods fail to generate high-quality stylized images, bringing artifacts and disharmonious patterns. The pre-trained large-scale model-based methods can generate high-quality stylized images but struggle to preserve the content structure and cost long inference time. To this end, we propose a new framework, called SPAST, to generate high-quality stylized images with less inference time. Specifically, we design a novel Local-global Window Size Stylization Module (LGWSSM) to fuse style features into content features. Besides, we introduce a novel style prior loss, which can dig out the style priors from a pre-trained large-scale model into the SPAST and motivate the SPAST to generate high-quality stylized images with short inference time. We conduct abundant experiments to verify that our proposed method can generate high-quality stylized images and less inference time compared with the SOTA arbitrary style transfer methods.

Authors

  • Zhanjie Zhang
    College of Computer Science and Technology, Zhejiang University, No. 38, Zheda Road, Hangzhou 310000, China. Electronic address: cszzj@zju.edu.cn.
  • Quanwei Zhang
    Department of Aerospace and Mechanical Engineering , University of Notre Dame , Notre Dame , Indiana 46556 , United States.
  • Junsheng Luan
    College of Computer Science and Technology, Zhejiang University, No. 38, Zheda Road, Hangzhou 310000, China. Electronic address: l.junsheng121@zju.edu.cn.
  • Mengyuan Yang
    West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu 610041, China.
  • Yun Wang
    Department of Anesthesiology, Beijing Friendship Hospital, Capital Medical University, Beijing, 100050, People's Republic of China.
  • Lei Zhao
    Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China.