Beyond Language Modeling: An Exploration of Multimodal Pretraining

Journal: arXiv
Published Date:

Abstract

The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.

Authors

  • Shengbang Tong; David Fan; John Nguyen; Ellis Brown; Gaoyue Zhou; Shengyi Qian; Boyang Zheng; Théophane Vallaeys; Junlin Han; Rob Fergus; Naila Murray; Marjan Ghazvininejad; Mike Lewis; Nicolas Ballas; Amir Bar; Michael Rabbat; Jakob Verbeek; Luke Zettlemoyer; Koustuv Sinha; Yann LeCun; Saining Xie