Adversarial Dependence Minimization
Journal:
arXiv
Published Date:
Feb 5, 2025
Abstract
Many machine learning techniques rely on minimizing the covariance between
output feature dimensions to extract minimally redundant representations from
data. However, these methods do not eliminate all dependencies/redundancies, as
linearly uncorrelated variables can still exhibit nonlinear relationships. This
work provides a differentiable and scalable algorithm for dependence
minimization that goes beyond linear pairwise decorrelation. Our method employs
an adversarial game where small networks identify dependencies among feature
dimensions, while the encoder exploits this information to reduce dependencies.
We provide empirical evidence of the algorithm's convergence and demonstrate
its utility in three applications: extending PCA to nonlinear decorrelation,
improving the generalization of image classification methods, and preventing
dimensional collapse in self-supervised representation learning.
