Agent-Based Uncertainty Awareness Improves Automated Radiology Report Labeling with an Open-Source Large Language Model
Journal:
arXiv
Published Date:
Feb 2, 2025
Abstract
Reliable extraction of structured data from radiology reports using Large
Language Models (LLMs) remains challenging, especially for complex, non-English
texts like Hebrew. This study introduces an agent-based uncertainty-aware
approach to improve the trustworthiness of LLM predictions in medical
applications. We analyzed 9,683 Hebrew radiology reports from Crohn's disease
patients (from 2010 to 2023) across three medical centers. A subset of 512
reports was manually annotated for six gastrointestinal organs and 15
pathological findings, while the remaining reports were automatically annotated
using HSMP-BERT. Structured data extraction was performed using Llama 3.1
(Llama 3-8b-instruct) with Bayesian Prompt Ensembles (BayesPE), which employed
six semantically equivalent prompts to estimate uncertainty. An Agent-Based
Decision Model integrated multiple prompt outputs into five confidence levels
for calibrated uncertainty and was compared against three entropy-based models.
Performance was evaluated using accuracy, F1 score, precision, recall, and
Cohen's Kappa before and after filtering high-uncertainty cases. The
agent-based model outperformed the baseline across all metrics, achieving an F1
score of 0.3967, recall of 0.6437, and Cohen's Kappa of 0.3006. After filtering
high-uncertainty cases (greater than or equal to 0.5), the F1 score improved to
0.4787, and Kappa increased to 0.4258. Uncertainty histograms demonstrated
clear separation between correct and incorrect predictions, with the
agent-based model providing the most well-calibrated uncertainty estimates. By
incorporating uncertainty-aware prompt ensembles and an agent-based decision
model, this approach enhances the performance and reliability of LLMs in
structured data extraction from radiology reports, offering a more
interpretable and trustworthy solution for high-stakes medical applications.
