OUI Need to Talk About Weight Decay: A New Perspective on Overfitting Detection
Journal:
arXiv
Published Date:
Apr 24, 2025
Abstract
We introduce the Overfitting-Underfitting Indicator (OUI), a novel tool for
monitoring the training dynamics of Deep Neural Networks (DNNs) and identifying
optimal regularization hyperparameters. Specifically, we validate that OUI can
effectively guide the selection of the Weight Decay (WD) hyperparameter by
indicating whether a model is overfitting or underfitting during training
without requiring validation data. Through experiments on DenseNet-BC-100 with
CIFAR- 100, EfficientNet-B0 with TinyImageNet and ResNet-34 with ImageNet-1K,
we show that maintaining OUI within a prescribed interval correlates strongly
with improved generalization and validation scores. Notably, OUI converges
significantly faster than traditional metrics such as loss or accuracy,
enabling practitioners to identify optimal WD (hyperparameter) values within
the early stages of training. By leveraging OUI as a reliable indicator, we can
determine early in training whether the chosen WD value leads the model to
underfit the training data, overfit, or strike a well-balanced trade-off that
maximizes validation scores. This enables more precise WD tuning for optimal
performance on the tested datasets and DNNs. All code for reproducing these
experiments is available at https://github.com/AlbertoFdezHdez/OUI.
