VaxGuard: A Multi-Generator, Multi-Type, and Multi-Role Dataset for Detecting LLM-Generated Vaccine Misinformation
Journal:
arXiv
Published Date:
Mar 12, 2025
Abstract
Recent advancements in Large Language Models (LLMs) have significantly
improved text generation capabilities. However, they also present challenges,
particularly in generating vaccine-related misinformation, which poses risks to
public health. Despite research on human-authored misinformation, a notable gap
remains in understanding how LLMs contribute to vaccine misinformation and how
best to detect it. Existing benchmarks often overlook vaccine-specific
misinformation and the diverse roles of misinformation spreaders. This paper
introduces VaxGuard, a novel dataset designed to address these challenges.
VaxGuard includes vaccine-related misinformation generated by multiple LLMs and
provides a comprehensive framework for detecting misinformation across various
roles. Our findings show that GPT-3.5 and GPT-4o consistently outperform other
LLMs in detecting misinformation, especially when dealing with subtle or
emotionally charged narratives. On the other hand, PHI3 and Mistral show lower
performance, struggling with precision and recall in fear-driven contexts.
Additionally, detection performance tends to decline as input text length
increases, indicating the need for improved methods to handle larger content.
These results highlight the importance of role-specific detection strategies
and suggest that VaxGuard can serve as a key resource for improving the
detection of LLM-generated vaccine misinformation.
