FedsLLM: Federated Split Learning for Large Language Models over Communication Networks
Journal:
arXiv
Published Date:
Jul 12, 2024
Abstract
Addressing the challenges of deploying large language models in wireless
communication networks, this paper combines low-rank adaptation technology
(LoRA) with the splitfed learning framework to propose the federated split
learning for large language models (FedsLLM) framework. The method introduced
in this paper utilizes LoRA technology to reduce processing loads by dividing
the network into client subnetworks and server subnetworks. It leverages a
federated server to aggregate and update client models. As the training data
are transmitted through a wireless network between clients and both main and
federated servers, the training delay is determined by the learning accuracy
and the allocation of communication bandwidth. This paper models the
minimization of the training delay by integrating computation and communication
optimization, simplifying the optimization problem into a convex problem to
find the optimal solution. Additionally, it presents a lemma that describes the
precise solutions to this problem. Simulation results demonstrate that the
proposed optimization algorithm reduces delays by an average of 47.63% compared
to unoptimized scenarios.
