Mitigating Group-Level Fairness Disparities in Federated Visual Language Models
Journal:
arXiv
Published Date:
May 3, 2025
Abstract
Visual language models (VLMs) have shown remarkable capabilities in
multimodal tasks but face challenges in maintaining fairness across demographic
groups, particularly when deployed in federated learning (FL) environments.
This paper addresses the critical issue of group fairness in federated VLMs by
introducing FVL-FP, a novel framework that combines FL with fair prompt tuning
techniques. We focus on mitigating demographic biases while preserving model
performance through three innovative components: (1) Cross-Layer Demographic
Fair Prompting (CDFP), which adjusts potentially biased embeddings through
counterfactual regularization; (2) Demographic Subspace Orthogonal Projection
(DSOP), which removes demographic bias in image representations by mapping fair
prompt text to group subspaces; and (3) Fair-aware Prompt Fusion (FPF), which
dynamically balances client contributions based on both performance and
fairness metrics. Extensive evaluations across four benchmark datasets
demonstrate that our approach reduces demographic disparity by an average of
45\% compared to standard FL approaches, while maintaining task performance
within 6\% of state-of-the-art results. FVL-FP effectively addresses the
challenges of non-IID data distributions in federated settings and introduces
minimal computational overhead while providing significant fairness benefits.
Our work presents a parameter-efficient solution to the critical challenge of
ensuring equitable performance across demographic groups in privacy-preserving
multimodal systems.
