Cost-Optimal Active AI Model Evaluation
Journal:
arXiv
Published Date:
Jun 9, 2025
Abstract
The development lifecycle of generative AI systems requires continual
evaluation, data acquisition, and annotation, which is costly in both resources
and time. In practice, rapid iteration often makes it necessary to rely on
synthetic annotation data because of the low cost, despite the potential for
substantial bias. In this paper, we develop novel, cost-aware methods for
actively balancing the use of a cheap, but often inaccurate, weak rater -- such
as a model-based autorater that is designed to automatically assess the quality
of generated content -- with a more expensive, but also more accurate, strong
rater alternative such as a human. More specifically, the goal of our approach
is to produce a low variance, unbiased estimate of the mean of the target
"strong" rating, subject to some total annotation budget. Building on recent
work in active and prediction-powered statistical inference, we derive a family
of cost-optimal policies for allocating a given annotation budget between weak
and strong raters so as to maximize statistical efficiency. Using synthetic and
real-world data, we empirically characterize the conditions under which these
policies yield improvements over prior methods. We find that, especially in
tasks where there is high variability in the difficulty of examples, our
policies can achieve the same estimation precision at a far lower total
annotation budget than standard evaluation methods.
