Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics.

Journal: Journal of medical Internet research
Published Date: May 26, 2025

Abstract

Rigorous evaluation of generalist medical artificial intelligence (GMAI) is imperative to ensure their utility and safety before implementation in health care. Current evaluation strategies rely heavily on benchmarks, which can suffer from issues with data contamination and cannot explain how GMAI might fail (lacking explanatory power) or in what circumstances (lacking predictive power). To address these limitations, we propose a new methodology to improve the quality of GMAI evaluation using construct-oriented processes. Drawing on modern psychometric techniques, we introduce approaches to construct identification and present alternative assessment formats for different domains of professional skills, knowledge, and behaviors that are essential for safe practice. We also discuss the need for human oversight in future GMAI adoption.

Authors

  • Luning Sun
    Research Division of Clinical Pharmacology, First Affiliated Hospital with Nanjing Medical University, 300 Guangzhou Road, Nanjing, 210009, China. Electronic address: sunluning0521@aliyun.com.
  • Christopher Gibbons
    Oracle Health, Austin, TX, United States.
  • José Hernández-Orallo
    Valencian Research Institute for Artificial Intelligence (VRAIN), Universitat Politècnica de València, Valencia, Spain. jorallo@upv.es.
  • Xiting Wang
    Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, 100190, China.
  • Liming Jiang
    Department of Civil and Environmental Engineering, University of Massachusetts Lowell, United States.
  • David Stillwell
    The Psychometrics Centre, Cambridge Judge Business School, University of Cambridge, Cambridge, United Kingdom.
  • Fang Luo
    MOE Key Laboratory for Analytical Science of Food Safety and Biology, Fujian Provincial Key Laboratory of Analysis and Detection for Food Safety, College of Chemistry, Fuzhou University, Fuzhou, Fujian 350116, China; College of Biological Science and Engineering, Fuzhou University, Fuzhou, Fujian 350116, China. Electronic address: luofang0812@163.com.
  • Xing Xie
    Microsoft Research, China. Electronic address: xing.xie@microsoft.com.

Keywords

External Resources

Popular Topics
Recent Journals