One Communication Round is All It Needs for Federated Fine-Tuning Foundation Models
Journal:
arXiv
Published Date:
Dec 5, 2024
Abstract
The recent advancement of large foundation models (FMs) has increased the
demand for fine-tuning these models on large-scale and cross-domain datasets.
To address this, federated fine-tuning has emerged as a solution, allowing
models to be fine-tuned on distributed datasets across multiple devices while
ensuring data privacy. However, the substantial parameter size of FMs and the
multi-round communication required by traditional federated fine-tuning
algorithms result in prohibitively high communication costs, challenging the
practicality of federated fine-tuning. In this paper, we are the first to
reveal, both theoretically and empirically, that the traditional multi-round
aggregation algorithms may not be necessary for federated fine-tuning large
FMs. Our experiments reveal that a single round of communication (i.e.,
one-shot federated fine-tuning) yields a global model performance comparable to
that achieved through multiple rounds of communication. Through rigorous
mathematical and empirical analyses, we demonstrate that large FMs, due to
their extensive parameter sizes and pre-training on general tasks, achieve
significantly lower training loss in one-shot federated fine-tuning compared to
smaller models. Our extensive experiments show that one-shot federated
fine-tuning not only reduces communication costs but also enables asynchronous
aggregation, enhances privacy, and maintains performance consistency with
multi-round federated fine-tuning for models larger than 1 billion parameters,
on text generation and text-to-image generation tasks. Our findings have the
potential to revolutionize federated fine-tuning in practice, enhancing
efficiency, reducing costs, and expanding accessibility for large-scale models.
This breakthrough paves the way for broader adoption and application of
federated fine-tuning across various domains.
