FedMEKT: Distillation-based embedding knowledge transfer for multimodal federated learning.

Journal: Neural networks : the official journal of the International Neural Network Society
PMID:

Abstract

Federated learning (FL) enables a decentralized machine learning paradigm for multiple clients to collaboratively train a generalized global model without sharing their private data. Most existing works have focused on designing FL systems for unimodal data, limiting their potential to exploit valuable multimodal data for future personalized applications. Moreover, the majority of FL approaches still rely on labeled data at the client side, which is often constrained by the inability of users to self-annotate their data in real-world applications. In light of these limitations, we propose a novel multimodal FL framework, namely FedMEKT, based on a semi-supervised learning approach to leverage representations from different modalities. To address the challenges of modality discrepancy and labeled data constraints in existing FL systems, our proposed FedMEKT framework comprises local multimodal autoencoder learning, generalized multimodal autoencoder construction, and generalized classifier learning. Bringing this concept into the proposed framework, we develop a distillation-based multimodal embedding knowledge transfer mechanism which allows the server and clients to exchange joint multimodal embedding knowledge extracted from a multimodal proxy dataset. Specifically, our FedMEKT iteratively updates the generalized global encoders with joint multimodal embedding knowledge from participating clients through upstream and downstream multimodal embedding knowledge transfer for local learning. Through extensive experiments on four multimodal datasets, we demonstrate that FedMEKT not only achieves superior global encoder performance in linear evaluation but also guarantees user privacy for personal data and model parameters while demanding less communication cost than other baselines.

Authors

  • Huy Q Le
    Department of Computer Science and Engineering, Kyung Hee University, Yongin-si, 17104, Republic of Korea. Electronic address: quanghuy69@khu.ac.kr.
  • Minh N H Nguyen
    Digital Science and Technology Institute, The University of Danang-Vietnam-Korea University of Information and Communication Technology, Da Nang, 550000, Viet Nam. Electronic address: nhnminh@vku.udn.vn.
  • Chu Myaet Thwal
    Department of Computer Science and Engineering, Kyung Hee University, Yongin-si, 17104, Republic of Korea. Electronic address: chumyaet@khu.ac.kr.
  • Yu Qiao
    Department of English and American Studies, RWTH Aachen University, Aachen, North Rhine-Westphalia, Germany.
  • Chaoning Zhang
    Department of Artificial Intelligence, Kyung Hee University, Yongin-si, 17104, Republic of Korea. Electronic address: chaoningzhang1990@gmail.com.
  • Choong Seon Hong
    Department of Computer Science and Engineering, Kyung Hee University, Yongin-si, 17104, Republic of Korea. Electronic address: cshong@khu.ac.kr.