Evaluating multiple models using labeled and unlabeled data
Journal:
arXiv
Published Date:
Jan 21, 2025
Abstract
It remains difficult to evaluate machine learning classifiers in the absence
of a large, labeled dataset. While labeled data can be prohibitively expensive
or impossible to obtain, unlabeled data is plentiful. Here, we introduce
Semi-Supervised Model Evaluation (SSME), a method that uses both labeled and
unlabeled data to evaluate machine learning classifiers. SSME is the first
evaluation method to take advantage of the fact that: (i) there are frequently
multiple classifiers for the same task, (ii) continuous classifier scores are
often available for all classes, and (iii) unlabeled data is often far more
plentiful than labeled data. The key idea is to use a semi-supervised mixture
model to estimate the joint distribution of ground truth labels and classifier
predictions. We can then use this model to estimate any metric that is a
function of classifier scores and ground truth labels (e.g., accuracy or
expected calibration error). We present experiments in four domains where
obtaining large labeled datasets is often impractical: (1) healthcare, (2)
content moderation, (3) molecular property prediction, and (4) image
annotation. Our results demonstrate that SSME estimates performance more
accurately than do competing methods, reducing error by 5.1x relative to using
labeled data alone and 2.4x relative to the next best competing method. SSME
also improves accuracy when evaluating performance across subsets of the test
distribution (e.g., specific demographic subgroups) and when evaluating the
performance of language models.
