Parameter-Efficient Fine-Tuning for Foundation Models
Journal:
arXiv
Published Date:
Jan 23, 2025
Abstract
This survey delves into the realm of Parameter-Efficient Fine-Tuning (PEFT)
within the context of Foundation Models (FMs). PEFT, a cost-effective
fine-tuning technique, minimizes parameters and computational complexity while
striving for optimal downstream task performance. FMs, like ChatGPT, DALL-E,
and LLaVA specialize in language understanding, generative tasks, and
multimodal tasks, trained on diverse datasets spanning text, images, and
videos. The diversity of FMs guides various adaptation strategies for PEFT.
Therefore, this survey aims to provide a comprehensive overview of PEFT
techniques applied to diverse FMs and address critical gaps in understanding
the techniques, trends, and applications. We start by providing a detailed
development of FMs and PEFT. Subsequently, we systematically review the key
categories and core mechanisms of PEFT across diverse FMs to offer a
comprehensive understanding of trends. We also explore the most recent
applications across various FMs to demonstrate the versatility of PEFT,
shedding light on the integration of systematic PEFT methods with a range of
FMs. Furthermore, we identify potential research and development directions for
improving PEFTs in the future. This survey provides a valuable resource for
both newcomers and experts seeking to understand and use the power of PEFT
across FMs. All reviewed papers are listed at
\url{https://github.com/THUDM/Awesome-Parameter-Efficient-Fine-Tuning-for-Foundation-Models}.
