Fact-Controlled Diagnosis of Hallucinations in Medical Text Summarization
Journal:
arXiv
Published Date:
May 31, 2025
Abstract
Hallucinations in large language models (LLMs) during summarization of
patient-clinician dialogues pose significant risks to patient care and clinical
decision-making. However, the phenomenon remains understudied in the clinical
domain, with uncertainty surrounding the applicability of general-domain
hallucination detectors. The rarity and randomness of hallucinations further
complicate their investigation. In this paper, we conduct an evaluation of
hallucination detection methods in the medical domain, and construct two
datasets for the purpose: A fact-controlled Leave-N-out dataset -- generated by
systematically removing facts from source dialogues to induce hallucinated
content in summaries; and a natural hallucination dataset -- arising
organically during LLM-based medical summarization. We show that general-domain
detectors struggle to detect clinical hallucinations, and that performance on
fact-controlled hallucinations does not reliably predict effectiveness on
natural hallucinations. We then develop fact-based approaches that count
hallucinations, offering explainability not available with existing methods.
Notably, our LLM-based detectors, which we developed using fact-controlled
hallucinations, generalize well to detecting real-world clinical
hallucinations. This research contributes a suite of specialized metrics
supported by expert-annotated datasets to advance faithful clinical
summarization systems.
