Review GIDE -- Restaurant Review Gastrointestinal Illness Detection and Extraction with Large Language Models
Journal:
arXiv
Published Date:
Mar 12, 2025
Abstract
Foodborne gastrointestinal (GI) illness is a common cause of ill health in
the UK. However, many cases do not interact with the healthcare system, posing
significant challenges for traditional surveillance methods. The growth of
publicly available online restaurant reviews and advancements in large language
models (LLMs) present potential opportunities to extend disease surveillance by
identifying public reports of GI illness. In this study, we introduce a novel
annotation schema, developed with experts in GI illness, applied to the Yelp
Open Dataset of reviews. Our annotations extend beyond binary disease
detection, to include detailed extraction of information on symptoms and foods.
We evaluate the performance of open-weight LLMs across these three tasks: GI
illness detection, symptom extraction, and food extraction. We compare this
performance to RoBERTa-based classification models fine-tuned specifically for
these tasks. Our results show that using prompt-based approaches, LLMs achieve
micro-F1 scores of over 90% for all three of our tasks. Using prompting alone,
we achieve micro-F1 scores that exceed those of smaller fine-tuned models. We
further demonstrate the robustness of LLMs in GI illness detection across three
bias-focused experiments. Our results suggest that publicly available review
text and LLMs offer substantial potential for public health surveillance of GI
illness by enabling highly effective extraction of key information. While LLMs
appear to exhibit minimal bias in processing, the inherent limitations of
restaurant review data highlight the need for cautious interpretation of
results.
