Can LLMs Support Medical Knowledge Imputation? An Evaluation-Based Perspective
Journal:
arXiv
Published Date:
Mar 29, 2025
Abstract
Medical knowledge graphs (KGs) are essential for clinical decision support
and biomedical research, yet they often exhibit incompleteness due to knowledge
gaps and structural limitations in medical coding systems. This issue is
particularly evident in treatment mapping, where coding systems such as ICD,
Mondo, and ATC lack comprehensive coverage, resulting in missing or
inconsistent associations between diseases and their potential treatments. To
address this issue, we have explored the use of Large Language Models (LLMs)
for imputing missing treatment relationships. Although LLMs offer promising
capabilities in knowledge augmentation, their application in medical knowledge
imputation presents significant risks, including factual inaccuracies,
hallucinated associations, and instability between and within LLMs. In this
study, we systematically evaluate LLM-driven treatment mapping, assessing its
reliability through benchmark comparisons. Our findings highlight critical
limitations, including inconsistencies with established clinical guidelines and
potential risks to patient safety. This study serves as a cautionary guide for
researchers and practitioners, underscoring the importance of critical
evaluation and hybrid approaches when leveraging LLMs to enhance treatment
mappings on medical knowledge graphs.
