Truth, Trust, and Trouble: Medical AI on the Edge
Journal:
arXiv
Published Date:
Jul 1, 2025
Abstract
Large Language Models (LLMs) hold significant promise for transforming
digital health by enabling automated medical question answering. However,
ensuring these models meet critical industry standards for factual accuracy,
usefulness, and safety remains a challenge, especially for open-source
solutions. We present a rigorous benchmarking framework using a dataset of over
1,000 health questions. We assess model performance across honesty,
helpfulness, and harmlessness. Our results highlight trade-offs between factual
reliability and safety among evaluated models -- Mistral-7B,
BioMistral-7B-DARE, and AlpaCare-13B. AlpaCare-13B achieves the highest
accuracy (91.7%) and harmlessness (0.92), while domain-specific tuning in
BioMistral-7B-DARE boosts safety (0.90) despite its smaller scale. Few-shot
prompting improves accuracy from 78% to 85%, and all models show reduced
helpfulness on complex queries, highlighting ongoing challenges in clinical QA.
