pFedSOP : Accelerating Training Of Personalized Federated Learning Using Second-Order Optimization
Journal:
arXiv
Published Date:
Jun 8, 2025
Abstract
Personalized Federated Learning (PFL) enables clients to collaboratively
train personalized models tailored to their individual objectives, addressing
the challenge of model generalization in traditional Federated Learning (FL)
due to high data heterogeneity. However, existing PFL methods often require
increased communication rounds to achieve the desired performance, primarily
due to slow training caused by the use of first-order optimization, which has
linear convergence. Additionally, many of these methods increase local
computation because of the additional data fed into the model during the search
for personalized local models. One promising solution to this slow training is
second-order optimization, known for its quadratic convergence. However,
employing it in PFL is challenging due to the Hessian matrix and its inverse.
In this paper, we propose pFedSOP, which efficiently utilizes second-order
optimization in PFL to accelerate the training of personalized models and
enhance performance with fewer communication rounds. Our approach first
computes a personalized local gradient update using the Gompertz function-based
normalized angle between local and global gradient updates, incorporating
client-specific global information. We then use a regularized Fisher
Information Matrix (FIM), computed from this personalized gradient update, as
an approximation of the Hessian to update the personalized models. This
FIM-based second-order optimization speeds up training with fewer communication
rounds by tackling the challenges with exact Hessian and avoids additional data
being fed into the model during the search for personalized local models.
Extensive experiments on heterogeneously partitioned image classification
datasets with partial client participation demonstrate that pFedSOP outperforms
state-of-the-art FL and PFL algorithms.
