DFedGFM: Pursuing global consistency for Decentralized Federated Learning via global flatness and global momentum.
Journal:
Neural networks : the official journal of the International Neural Network Society
PMID:
39746248
Abstract
To tackle high communication costs and privacy issues in Centralized Federated Learning (CFL), Decentralized Federated Learning (DFL) is an alternative. However, a significant discrepancy exists between local updates and the expected global update, known as client drift, which arises from inconsistency and heterogeneous data. Previous research in the DFL field has focused on local information during client updates, without considering global information, which fails to alleviate the client drift issue. In this paper, we first rethink the local flatness and local momentum acceleration techniques in the two most popular DFL optimizers (DFedAvgM and DFedSAM) and propose a novel DFL algorithm, called DFedGFM to simultaneously explore the global flatness and global momentum to achieve global consistency. In contrast with DFedSAM which only seeks local flatness, DFedGFM leverages the estimation of global gradients better to identify the global flatness of the loss landscape. Compared to DFedAvgM, DFedGFM adopts global momentum to make the algorithm more robust to heterogeneous data. Theoretically, we prove that the O1T convergence rate for DFedGFM, and the results indicate that the convergence bound of the algorithm becomes tighter with better connectivity of the communication topology. Furthermore, we have experimentally verified the correctness of the theoretical results and extensive experiments demonstrate that our proposed algorithm outperforms existing DFL algorithms.