ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints
Journal:
arXiv
Published Date:
Jul 9, 2025
Abstract
Foundation models are pre-trained on large-scale datasets and subsequently
fine-tuned on small-scale datasets using parameter-efficient fine-tuning (PEFT)
techniques like low-rank adapters (LoRA). In most previous works, LoRA weight
matrices are randomly initialized with a fixed rank across all attachment
points. In this paper, we improve convergence and final performance of LoRA
fine-tuning, using our proposed data-driven weight initialization method,
ConsNoTrainLoRA (CNTLoRA). We express LoRA initialization as a domain shift
problem where we use multiple constraints relating the pre-training and
fine-tuning activations. By reformulating these constraints, we obtain a
closed-form estimate of LoRA weights that depends on pre-training weights and
fine-tuning activation vectors and hence requires no training during
initialization. This weight estimate is decomposed to initialize the up and
down matrices with proposed flexibility of variable ranks. With the proposed
initialization method, we fine-tune on downstream tasks such as image
generation, image classification and image understanding. Both quantitative and
qualitative results demonstrate that CNTLoRA outperforms standard and
data-driven weight initialization methods. Extensive analyses and ablations
further elucidate the design choices of our framework, providing an optimal
recipe for faster convergence and enhanced performance.
