OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmology
Journal:
arXiv
Published Date:
Feb 3, 2025
Abstract
Large language models (LLMs) have shown significant promise across various
medical applications, with ophthalmology being a notable area of focus. Many
ophthalmic tasks have shown substantial improvement through the integration of
LLMs. However, before these models can be widely adopted in clinical practice,
evaluating their capabilities and identifying their limitations is crucial. To
address this research gap and support the real-world application of LLMs, we
introduce the OphthBench, a specialized benchmark designed to assess LLM
performance within the context of Chinese ophthalmic practices. This benchmark
systematically divides a typical ophthalmic clinical workflow into five key
scenarios: Education, Triage, Diagnosis, Treatment, and Prognosis. For each
scenario, we developed multiple tasks featuring diverse question types,
resulting in a comprehensive benchmark comprising 9 tasks and 591 questions.
This comprehensive framework allows for a thorough assessment of LLMs'
capabilities and provides insights into their practical application in Chinese
ophthalmology. Using this benchmark, we conducted extensive experiments and
analyzed the results from 39 popular LLMs. Our evaluation highlights the
current gap between LLM development and its practical utility in clinical
settings, providing a clear direction for future advancements. By bridging this
gap, we aim to unlock the potential of LLMs and advance their development in
ophthalmology.
