Decide less, communicate more: On the construct validity of end-to-end fact-checking in medicine
Journal:
arXiv
Published Date:
Jun 25, 2025
Abstract
Technological progress has led to concrete advancements in tasks that were
regarded as challenging, such as automatic fact-checking. Interest in adopting
these systems for public health and medicine has grown due to the high-stakes
nature of medical decisions and challenges in critically appraising a vast and
diverse medical literature. Evidence-based medicine connects to every
individual, and yet the nature of it is highly technical, rendering the medical
literacy of majority users inadequate to sufficiently navigate the domain. Such
problems with medical communication ripens the ground for end-to-end
fact-checking agents: check a claim against current medical literature and
return with an evidence-backed verdict. And yet, such systems remain largely
unused. To understand this, we present the first study examining how clinical
experts verify real claims from social media by synthesizing medical evidence.
In searching for this upper-bound, we reveal fundamental challenges in
end-to-end fact-checking when applied to medicine: Difficulties connecting
claims in the wild to scientific evidence in the form of clinical trials;
ambiguities in underspecified claims mixed with mismatched intentions; and
inherently subjective veracity labels. We argue that fact-checking should be
approached and evaluated as an interactive communication problem, rather than
an end-to-end process.
