Monty Hall and Optimized Conformal Prediction to Improve Decision-Making with LLMs
Journal:
arXiv
Published Date:
Dec 31, 2024
Abstract
Large language models (LLMs) are empowering decision-making in several
applications, including tool or API usage and answering multiple-choice
questions (MCQs). However, they often make overconfident, incorrect
predictions, which can be risky in high-stakes settings like healthcare and
finance. To mitigate these risks, recent works have used conformal prediction
(CP), a model-agnostic framework for distribution-free uncertainty
quantification. CP transforms a \emph{score function} into prediction sets that
contain the true answer with high probability. While CP provides this coverage
guarantee for arbitrary scores, the score quality significantly impacts
prediction set sizes. Prior works have relied on LLM logits or other heuristic
scores, lacking quality guarantees. We address this limitation by introducing
CP-OPT, an optimization framework to learn scores that minimize set sizes while
maintaining coverage. Furthermore, inspired by the Monty Hall problem, we
extend CP's utility beyond uncertainty quantification to improve accuracy. We
propose \emph{conformal revision of questions} (CROQ) to revise the problem by
narrowing down the available choices to those in the prediction set. The
coverage guarantee of CP ensures that the correct choice is in the revised
question prompt with high probability, while the smaller number of choices
increases the LLM's chances of answering it correctly. Experiments on MMLU,
ToolAlpaca, and TruthfulQA datasets with Gemma-2, Llama-3 and Phi-3 models show
that CP-OPT significantly reduces set sizes while maintaining coverage, and
CROQ improves accuracy over the standard inference, especially when paired with
CP-OPT scores. Together, CP-OPT and CROQ offer a robust framework for improving
both the safety and accuracy of LLM-driven decision-making.
