AI Debate Aids Assessment of Controversial Claims
Journal:
arXiv
Published Date:
Jun 2, 2025
Abstract
As AI grows more powerful, it will increasingly shape how we understand the
world. But with this influence comes the risk of amplifying misinformation and
deepening social divides-especially on consequential topics like public health
where factual accuracy directly impacts well-being. Scalable Oversight aims to
ensure AI truthfulness by enabling humans to supervise systems that may exceed
human capabilities--yet humans themselves hold different beliefs and biases
that impair their judgment. We study whether AI debate can guide biased judges
toward the truth by having two AI systems debate opposing sides of
controversial COVID-19 factuality claims where people hold strong prior
beliefs. We conduct two studies: one with human judges holding either
mainstream or skeptical beliefs evaluating factuality claims through
AI-assisted debate or consultancy protocols, and a second examining the same
problem with personalized AI judges designed to mimic these different human
belief systems. In our human study, we find that debate-where two AI advisor
systems present opposing evidence-based arguments-consistently improves
judgment accuracy and confidence calibration, outperforming consultancy with a
single-advisor system by 10% overall. The improvement is most significant for
judges with mainstream beliefs (+15.2% accuracy), though debate also helps
skeptical judges who initially misjudge claims move toward accurate views
(+4.7% accuracy). In our AI judge study, we find that AI judges with human-like
personas achieve even higher accuracy (78.5%) than human judges (70.1%) and
default AI judges without personas (69.8%), suggesting their potential for
supervising frontier AI models. These findings highlight AI debate as a
promising path toward scalable, bias-resilient oversight--leveraging both
diverse human and AI judgments to move closer to truth in contested domains.
