ChatGPT Performs on the Chinese National Medical Licensing Examination.

Journal: Journal of medical systems
PMID: 37581690

Abstract

ChatGPT, a language model developed by OpenAI, uses a 175 billion parameter Transformer architecture for natural language processing tasks. This study aimed to compare the knowledge and interpretation ability of ChatGPT with those of medical students in China by administering the Chinese National Medical Licensing Examination (NMLE) to both ChatGPT and medical students. We evaluated the performance of ChatGPT in three years' worth of the NMLE, which consists of four units. At the same time, the exam results were compared to those of medical students who had studied for five years at medical colleges. ChatGPT's performance was lower than that of the medical students, and ChatGPT's correct answer rate was related to the year in which the exam questions were released. ChatGPT's knowledge and interpretation ability for the NMLE were not yet comparable to those of medical students in China. It is probable that these abilities will improve through deep learning.

Authors

  • Xinyi Wang
    School of Biological Sciences & Medical Engineering, Southeast University, Nanjing, China.
  • Zhenye Gong
    Department of Medical Education, Ruijin Hospital Affifiliated to Shanghai Jiao Tong University School of Medicine, 197 Ruijin Rd. II, Shanghai, 200025, China.
  • Guoxin Wang
    Department of Medical Education, Ruijin Hospital Affifiliated to Shanghai Jiao Tong University School of Medicine, 197 Ruijin Rd. II, Shanghai, 200025, China.
  • Jingdan Jia
    Department of Medical Education, Ruijin Hospital Affifiliated to Shanghai Jiao Tong University School of Medicine, 197 Ruijin Rd. II, Shanghai, 200025, China.
  • Ying Xu
    School of Biological and Food Engineering Changzhou University Changzhou Jiangsu China.
  • Jialu Zhao
    Department of Medical Education, Ruijin Hospital Affifiliated to Shanghai Jiao Tong University School of Medicine, 197 Ruijin Rd. II, Shanghai, 200025, China.
  • Qingye Fan
    Department of Medical Education, Ruijin Hospital Affifiliated to Shanghai Jiao Tong University School of Medicine, 197 Ruijin Rd. II, Shanghai, 200025, China.
  • Shaun Wu
    WORK Medical Technology Group LTD, Hangzhou, China.
  • Weiguo Hu
    Department of Urology, Beijing Tsinghua Changgung Hospital, School of Clinical Medicine, Tsinghua University, Beijing, China.
  • Xiaoyang Li
    Department of Thoracic Surgery, West China Hospital of Sichuan University, Chengdu, 610041, China.

Keywords

External Resources

Popular Topics
Recent Journals