OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models
Journal:
arXiv
Published Date:
Mar 11, 2025
Abstract
Recent advancements in unified multimodal understanding and visual generation
(or multimodal generation) models have been hindered by their quadratic
computational complexity and dependence on large-scale training data. We
present OmniMamba, the first linear-architecture-based multimodal generation
model that generates both text and images through a unified next-token
prediction paradigm. The model fully leverages Mamba-2's high computational and
memory efficiency, extending its capabilities from text generation to
multimodal generation. To address the data inefficiency of existing unified
models, we propose two key innovations: (1) decoupled vocabularies to guide
modality-specific generation, and (2) task-specific LoRA for
parameter-efficient adaptation. Furthermore, we introduce a decoupled two-stage
training strategy to mitigate data imbalance between two tasks. Equipped with
these techniques, OmniMamba achieves competitive performance with JanusFlow
while surpassing Show-o across benchmarks, despite being trained on merely 2M
image-text pairs, which is 1,000 times fewer than Show-o. Notably, OmniMamba
stands out with outstanding inference efficiency, achieving up to a 119.2 times
speedup and 63% GPU memory reduction for long-sequence generation compared to
Transformer-based counterparts. Code and models are released at
https://github.com/hustvl/OmniMamba
