PCS-UQ: Uncertainty Quantification via the Predictability-Computability-Stability Framework
Journal:
arXiv
Published Date:
May 13, 2025
Abstract
As machine learning (ML) models are increasingly deployed in high-stakes
domains, trustworthy uncertainty quantification (UQ) is critical for ensuring
the safety and reliability of these models. Traditional UQ methods rely on
specifying a true generative model and are not robust to misspecification. On
the other hand, conformal inference allows for arbitrary ML models but does not
consider model selection, which leads to large interval sizes. We tackle these
drawbacks by proposing a UQ method based on the predictability, computability,
and stability (PCS) framework for veridical data science proposed by Yu and
Kumbier. Specifically, PCS-UQ addresses model selection by using a prediction
check to screen out unsuitable models. PCS-UQ then fits these screened
algorithms across multiple bootstraps to assess inter-sample variability and
algorithmic instability, enabling more reliable uncertainty estimates. Further,
we propose a novel calibration scheme that improves local adaptivity of our
prediction sets. Experiments across $17$ regression and $6$ classification
datasets show that PCS-UQ achieves the desired coverage and reduces width over
conformal approaches by $\approx 20\%$. Further, our local analysis shows
PCS-UQ often achieves target coverage across subgroups while conformal methods
fail to do so. For large deep-learning models, we propose computationally
efficient approximation schemes that avoid the expensive multiple bootstrap
trainings of PCS-UQ. Across three computer vision benchmarks, PCS-UQ reduces
prediction set size over conformal methods by $20\%$. Theoretically, we show a
modified PCS-UQ algorithm is a form of split conformal inference and achieves
the desired coverage with exchangeable data.
