FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models
Journal:
arXiv
Published Date:
Jun 11, 2025
Abstract
Vision-Language Models (VLMs) have demonstrated remarkable capabilities in
cross-modal understanding and generation by integrating visual and textual
information. While instruction tuning and parameter-efficient fine-tuning
methods have substantially improved the generalization of VLMs, most existing
approaches rely on centralized training, posing challenges for deployment in
domains with strict privacy requirements like healthcare. Recent efforts have
introduced Federated Learning (FL) into VLM fine-tuning to address these
privacy concerns, yet comprehensive benchmarks for evaluating federated
fine-tuning strategies, model architectures, and task generalization remain
lacking. In this work, we present \textbf{FedVLMBench}, the first systematic
benchmark for federated fine-tuning of VLMs. FedVLMBench integrates two
mainstream VLM architectures (encoder-based and encoder-free), four fine-tuning
strategies, five FL algorithms, six multimodal datasets spanning four
cross-domain single-task scenarios and two cross-domain multitask settings,
covering four distinct downstream task categories. Through extensive
experiments, we uncover key insights into the interplay between VLM
architectures, fine-tuning strategies, data heterogeneity, and multi-task
federated optimization. Notably, we find that a 2-layer multilayer perceptron
(MLP) connector with concurrent connector and LLM tuning emerges as the optimal
configuration for encoder-based VLMs in FL. Furthermore, current FL methods
exhibit significantly higher sensitivity to data heterogeneity in
vision-centric tasks than text-centric ones, across both encoder-free and
encoder-based VLM architectures. Our benchmark provides essential tools,
datasets, and empirical guidance for the research community, offering a
standardized platform to advance privacy-preserving, federated training of
multimodal foundation models.
