Improving Coverage in Combined Prediction Sets with Weighted p-values
Journal:
arXiv
Published Date:
May 17, 2025
Abstract
Conformal prediction quantifies the uncertainty of machine learning models by
augmenting point predictions with valid prediction sets, assuming
exchangeability. For complex scenarios involving multiple trials, models, or
data sources, conformal prediction sets can be aggregated to create a
prediction set that captures the overall uncertainty, often improving
precision. However, aggregating multiple prediction sets with individual
$1-\alpha$ coverage inevitably weakens the overall guarantee, typically
resulting in $1-2\alpha$ worst-case coverage. In this work, we propose a
framework for the weighted aggregation of prediction sets, where weights are
assigned to each prediction set based on their contribution. Our framework
offers flexible control over how the sets are aggregated, achieving tighter
coverage bounds that interpolate between the $1-2\alpha$ guarantee of the
combined models and the $1-\alpha$ guarantee of an individual model depending
on the distribution of weights. We extend our framework to data-dependent
weights, and we derive a general procedure for data-dependent weight
aggregation that maintains finite-sample validity. We demonstrate the
effectiveness of our methods through experiments on synthetic and real data in
the mixture-of-experts setting, and we show that aggregation with
data-dependent weights provides a form of adaptive coverage.
