Automating Expert-Level Medical Reasoning Evaluation of Large Language Models
Journal:
arXiv
Published Date:
Jul 10, 2025
Abstract
As large language models (LLMs) become increasingly integrated into clinical
decision-making, ensuring transparent and trustworthy reasoning is essential.
However, existing evaluation strategies of LLMs' medical reasoning capability
either suffer from unsatisfactory assessment or poor scalability, and a
rigorous benchmark remains lacking. To address this, we introduce
MedThink-Bench, a benchmark designed for rigorous, explainable, and scalable
assessment of LLMs' medical reasoning. MedThink-Bench comprises 500 challenging
questions across ten medical domains, each annotated with expert-crafted
step-by-step rationales. Building on this, we propose LLM-w-Ref, a novel
evaluation framework that leverages fine-grained rationales and LLM-as-a-Judge
mechanisms to assess intermediate reasoning with expert-level fidelity while
maintaining scalability. Experiments show that LLM-w-Ref exhibits a strong
positive correlation with expert judgments. Benchmarking twelve
state-of-the-art LLMs, we find that smaller models (e.g., MedGemma-27B) can
surpass larger proprietary counterparts (e.g., OpenAI-o3). Overall,
MedThink-Bench offers a foundational tool for evaluating LLMs' medical
reasoning, advancing their safe and responsible deployment in clinical
practice.
