Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting
Journal:
arXiv
Published Date:
May 30, 2025
Abstract
Vision foundation models pre-trained on massive data encode rich
representations of real-world concepts, which can be adapted to downstream
tasks by fine-tuning. However, fine-tuning foundation models on one task often
leads to the issue of concept forgetting on other tasks. Recent methods of
robust fine-tuning aim to mitigate forgetting of prior knowledge without
affecting the fine-tuning performance. Knowledge is often preserved by matching
the original and fine-tuned model weights or feature pairs. However, such
point-wise matching can be too strong, without explicit awareness of the
feature neighborhood structures that encode rich knowledge as well. We propose
a novel regularization method Proxy-FDA that explicitly preserves the
structural knowledge in feature space. Proxy-FDA performs Feature Distribution
Alignment (using nearest neighbor graphs) between the pre-trained and
fine-tuned feature spaces, and the alignment is further improved by informative
proxies that are generated dynamically to increase data diversity. Experiments
show that Proxy-FDA significantly reduces concept forgetting during
fine-tuning, and we find a strong correlation between forgetting and a
distributional distance metric (in comparison to L2 distance). We further
demonstrate Proxy-FDA's benefits in various fine-tuning settings (end-to-end,
few-shot and continual tuning) and across different tasks like image
classification, captioning and VQA.
