Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategies
Journal:
arXiv
Published Date:
Mar 10, 2025
Abstract
The evaluation and improvement of medical large language models (LLMs) are
critical for their real-world deployment, particularly in ensuring accuracy,
safety, and ethical alignment. Existing frameworks inadequately dissect
domain-specific error patterns or address cross-modal challenges. This study
introduces a granular error taxonomy through systematic analysis of top 10
models on MedBench, categorizing incorrect responses into eight types:
Omissions, Hallucination, Format Mismatch, Causal Reasoning Deficiency,
Contextual Inconsistency, Unanswered, Output Error, and Deficiency in Medical
Language Generation. Evaluation of 10 leading models reveals vulnerabilities:
despite achieving 0.86 accuracy in medical knowledge recall, critical reasoning
tasks show 96.3% omission, while safety ethics evaluations expose alarming
inconsistency (robustness score: 0.79) under option shuffled. Our analysis
uncovers systemic weaknesses in knowledge boundary enforcement and multi-step
reasoning. To address these, we propose a tiered optimization strategy spanning
four levels, from prompt engineering and knowledge-augmented retrieval to
hybrid neuro-symbolic architectures and causal reasoning frameworks. This work
establishes an actionable roadmap for developing clinically robust LLMs while
redefining evaluation paradigms through error-driven insights, ultimately
advancing the safety and trustworthiness of AI in high-stakes medical
environments.
