Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learning
Journal:
arXiv
Published Date:
Jan 13, 2025
Abstract
We derive explicit equations governing the cumulative biases and weights in
Deep Learning with ReLU activation function, based on gradient descent for the
Euclidean cost in the input layer, and under the assumption that the weights
are, in a precise sense, adapted to the coordinate system distinguished by the
activations. We show that gradient descent corresponds to a dynamical process
in the input layer, whereby clusters of data are progressively reduced in
complexity ("truncated") at an exponential rate that increases with the number
of data points that have already been truncated. We provide a detailed
discussion of several types of solutions to the gradient flow equations. A main
motivation for this work is to shed light on the interpretability question in
supervised learning.
