Evaluating the Performance of Reasoning Large Language Models on Japanese Radiology Board Examination Questions.

Journal: Academic radiology
Published Date:

Abstract

RATIONALE AND OBJECTIVES: This study evaluates the performance, cost, and processing time of OpenAI's reasoning large language models (LLMs) (o1-preview, o1-mini) and their base models (GPT-4o, GPT-4o-mini) on Japanese radiology board examination questions.

Authors

  • Takeshi Nakaura
    Department of Diagnostic Radiology, Graduate School of Medical Sciences, Kumamoto University, Honjo 1-1-1, Kumamoto 860-8556, Japan (T.N., N.Y., N.K., Y.N., H.U., M.K., S.O., T.H.). Electronic address: kff00712@nifty.com.
  • Hiroto Takamure
    Department of Diagnostic Radiology, Graduate School of Medical Sciences, Kumamoto University, Honjo 1-1-1, Chuo-ku, Kumamoto-shi, Kumamoto 860-8556, Japan (T.N., H.T., N.K., K.S., N.Y., Y.N., H.U., M.K., T.H.).
  • Naoki Kobayashi
    Department of Diagnostic Radiology, Graduate School of Medical Sciences, Kumamoto University, Honjo 1-1-1, Kumamoto 860-8556, Japan (T.N., N.Y., N.K., Y.N., H.U., M.K., S.O., T.H.).
  • Kaori Shiraishi
    Department of Diagnostic Radiology, Graduate School of Medical Sciences, Kumamoto University.
  • Naofumi Yoshida
    Department of Diagnostic Radiology, Graduate School of Medical Sciences, Kumamoto University, Honjo 1-1-1, Kumamoto 860-8556, Japan (T.N., N.Y., N.K., Y.N., H.U., M.K., S.O., T.H.).
  • Yasunori Nagayama
    Department of Diagnostic Radiology, Graduate School of Medical Sciences, Kumamoto University, Honjo 1-1-1, Kumamoto 860-8556, Japan (T.N., N.Y., N.K., Y.N., H.U., M.K., S.O., T.H.).
  • Hiroyuki Uetani
    Department of Diagnostic Radiology, Graduate School of Medical Sciences, Kumamoto University, Honjo 1-1-1, Kumamoto 860-8556, Japan (T.N., N.Y., N.K., Y.N., H.U., M.K., S.O., T.H.).
  • Masafumi Kidoh
    Department of Diagnostic Radiology, Graduate School of Medical Sciences, Kumamoto University, Honjo 1-1-1, Kumamoto 860-8556, Japan (T.N., N.Y., N.K., Y.N., H.U., M.K., S.O., T.H.).
  • Yoshinori Funama
    Department of Medical Physics, Faculty of Life Sciences, Kumamoto University, Honjo 1-1-1, Kumamoto 860-8556, Japan (Y.F.).
  • Toshinori Hirai
    Department of Diagnostic Radiology, Graduate School of Medical Sciences, Kumamoto University, Honjo 1-1-1, Kumamoto 860-8556, Japan (T.N., N.Y., N.K., Y.N., H.U., M.K., S.O., T.H.).