Robust Online Conformal Prediction under Uniform Label Noise
Journal:
arXiv
Published Date:
Jan 30, 2025
Abstract
Conformal prediction is an emerging technique for uncertainty quantification
that constructs prediction sets guaranteed to contain the true label with a
predefined probability. Recent work develops online conformal prediction
methods that adaptively construct prediction sets to accommodate distribution
shifts. However, existing algorithms typically assume perfect label accuracy
which rarely holds in practice. In this work, we investigate the robustness of
online conformal prediction under uniform label noise with a known noise rate,
in both constant and dynamic learning rate schedules. We show that label noise
causes a persistent gap between the actual mis-coverage rate and the desired
rate $\alpha$, leading to either overestimated or underestimated coverage
guarantees. To address this issue, we propose Noise Robust Online Conformal
Prediction (dubbed NR-OCP) by updating the threshold with a novel robust
pinball loss, which provides an unbiased estimate of clean pinball loss without
requiring ground-truth labels. Our theoretical analysis shows that NR-OCP
eliminates the coverage gap in both constant and dynamic learning rate
schedules, achieving a convergence rate of $\mathcal{O}(T^{-1/2})$ for both
empirical and expected coverage errors under uniform label noise. Extensive
experiments demonstrate the effectiveness of our method by achieving both
precise coverage and improved efficiency.
