Directional Gradient Projection for Robust Fine-Tuning of Foundation Models
Journal:
arXiv
Published Date:
Feb 21, 2025
Abstract
Robust fine-tuning aims to adapt large foundation models to downstream tasks
while preserving their robustness to distribution shifts. Existing methods
primarily focus on constraining and projecting current model towards the
pre-trained initialization based on the magnitudes between fine-tuned and
pre-trained weights, which often require extensive hyper-parameter tuning and
can sometimes result in underfitting. In this work, we propose Directional
Gradient Projection (DiGraP), a novel layer-wise trainable method that
incorporates directional information from gradients to bridge regularization
and multi-objective optimization. Besides demonstrating our method on image
classification, as another contribution we generalize this area to the
multi-modal evaluation settings for robust fine-tuning. Specifically, we first
bridge the uni-modal and multi-modal gap by performing analysis on Image
Classification reformulated Visual Question Answering (VQA) benchmarks and
further categorize ten out-of-distribution (OOD) VQA datasets by distribution
shift types and degree (i.e. near versus far OOD). Experimental results show
that DiGraP consistently outperforms existing baselines across Image
Classfication and VQA tasks with discriminative and generative backbones,
improving both in-distribution (ID) generalization and OOD robustness.
