EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Models
Journal:
arXiv
Published Date:
May 28, 2025
Abstract
Recent advancements have shown that the Mixture of Experts (MoE) approach
significantly enhances the capacity of large language models (LLMs) and
improves performance on downstream tasks. Building on these promising results,
multi-modal large language models (MLLMs) have increasingly adopted MoE
techniques. However, existing multi-modal MoE tuning methods typically face two
key challenges: expert uniformity and router rigidity. Expert uniformity occurs
because MoE experts are often initialized by simply replicating the FFN
parameters from LLMs, leading to homogenized expert functions and weakening the
intended diversification of the MoE architecture. Meanwhile, router rigidity
stems from the prevalent use of static linear routers for expert selection,
which fail to distinguish between visual and textual tokens, resulting in
similar expert distributions for image and text. To address these limitations,
we propose EvoMoE, an innovative MoE tuning framework. EvoMoE introduces a
meticulously designed expert initialization strategy that progressively evolves
multiple robust experts from a single trainable expert, a process termed expert
evolution that specifically targets severe expert homogenization. Furthermore,
we introduce the Dynamic Token-aware Router (DTR), a novel routing mechanism
that allocates input tokens to appropriate experts based on their modality and
intrinsic token values. This dynamic routing is facilitated by hypernetworks,
which dynamically generate routing weights tailored for each individual token.
Extensive experiments demonstrate that EvoMoE significantly outperforms other
sparse MLLMs across a variety of multi-modal benchmarks, including MME,
MMBench, TextVQA, and POPE. Our results highlight the effectiveness of EvoMoE
in enhancing the performance of MLLMs by addressing the critical issues of
expert uniformity and router rigidity.
