Astrea: A MOE-based Visual Understanding Model with Progressive Alignment
Journal:
arXiv
Published Date:
Mar 12, 2025
Abstract
Vision-Language Models (VLMs) based on Mixture-of-Experts (MoE) architectures
have emerged as a pivotal paradigm in multimodal understanding, offering a
powerful framework for integrating visual and linguistic information. However,
the increasing complexity and diversity of tasks present significant challenges
in coordinating load balancing across heterogeneous visual experts, where
optimizing one specialist's performance often compromises others' capabilities.
To address task heterogeneity and expert load imbalance, we propose Astrea, a
novel multi-expert collaborative VLM architecture based on progressive
pre-alignment. Astrea introduces three key innovations: 1) A heterogeneous
expert coordination mechanism that integrates four specialized models
(detection, segmentation, classification, captioning) into a comprehensive
expert matrix covering essential visual comprehension elements; 2) A dynamic
knowledge fusion strategy featuring progressive pre-alignment to harmonize
experts within the VLM latent space through contrastive learning, complemented
by probabilistically activated stochastic residual connections to preserve
knowledge continuity; 3) An enhanced optimization framework utilizing momentum
contrastive learning for long-range dependency modeling and adaptive weight
allocators for real-time expert contribution calibration. Extensive evaluations
across 12 benchmark tasks spanning VQA, image captioning, and cross-modal
retrieval demonstrate Astrea's superiority over state-of-the-art models,
achieving an average performance gain of +4.7\%. This study provides the first
empirical demonstration that progressive pre-alignment strategies enable VLMs
to overcome task heterogeneity limitations, establishing new methodological
foundations for developing general-purpose multimodal agents.
