MLaGA: Multimodal Large Language and Graph Assistant
Journal:
arXiv
Published Date:
Jun 3, 2025
Abstract
Large Language Models (LLMs) have demonstrated substantial efficacy in
advancing graph-structured data analysis. Prevailing LLM-based graph methods
excel in adapting LLMs to text-rich graphs, wherein node attributes are text
descriptions. However, their applications to multimodal graphs--where nodes are
associated with diverse attribute types, such as texts and images--remain
underexplored, despite their ubiquity in real-world scenarios. To bridge the
gap, we introduce the Multimodal Large Language and Graph Assistant (MLaGA), an
innovative model that adeptly extends LLM capabilities to facilitate reasoning
over complex graph structures and multimodal attributes. We first design a
structure-aware multimodal encoder to align textual and visual attributes
within a unified space through a joint graph pre-training objective.
Subsequently, we implement a multimodal instruction-tuning approach to
seamlessly integrate multimodal features and graph structures into the LLM
through lightweight projectors. Extensive experiments across multiple datasets
demonstrate the effectiveness of MLaGA compared to leading baseline methods,
achieving superior performance in diverse graph learning tasks under both
supervised and transfer learning scenarios.
