Final-Model-Only Data Attribution with a Unifying View of Gradient-Based Methods
Journal:
arXiv
Published Date:
Dec 5, 2024
Abstract
Training data attribution (TDA) is the task of attributing model behavior to
elements in the training data. This paper draws attention to the common setting
where one has access only to the final trained model, and not the training
algorithm or intermediate information from training. To serve as a gold
standard for TDA in this "final-model-only" setting, we propose further
training, with appropriate adjustment and averaging, to measure the sensitivity
of the given model to training instances. We then unify existing gradient-based
methods for TDA by showing that they all approximate the further training gold
standard in different ways. We investigate empirically the quality of these
gradient-based approximations to further training, for tabular, image, and text
datasets and models. We find that the approximation quality of first-order
methods is sometimes high but decays with the amount of further training. In
contrast, the approximations given by influence function methods are more
stable but surprisingly lower in quality.
