Accuracy of Information and References Using ChatGPT-3 for Retrieval of Clinical Radiological Information.

Journal: Canadian Association of Radiologists journal = Journal l'Association canadienne des radiologistes
Published Date: Apr 20, 2023

Abstract

To assess the accuracy of answers provided by ChatGPT-3 when prompted with questions from the daily routine of radiologists and to evaluate the text response when ChatGPT-3 was prompted to provide references for a given answer. ChatGPT-3 (San Francisco, OpenAI) is an artificial intelligence chatbot based on a large language model (LLM) that has been designed to generate human-like text. A total of 88 questions were submitted to ChatGPT-3 using textual prompt. These 88 questions were equally dispersed across 8 subspecialty areas of radiology. The responses provided by ChatGPT-3 were assessed for correctness by cross-checking them with peer-reviewed, PubMed-listed references. In addition, the references provided by ChatGPT-3 were evaluated for authenticity. A total of 59 of 88 responses (67%) to radiological questions were correct, while 29 responses (33%) had errors. Out of 343 references provided, only 124 references (36.2%) were available through internet search, while 219 references (63.8%) appeared to be generated by ChatGPT-3. When examining the 124 identified references, only 47 references (37.9%) were considered to provide enough background to correctly answer 24 questions (37.5%). In this pilot study, ChatGPT-3 provided correct responses to questions from the daily clinical routine of radiologists in only about two thirds, while the remainder of responses contained errors. The majority of provided references were not found and only a minority of the provided references contained the correct information to answer the question. Caution is advised when using ChatGPT-3 to retrieve radiological information.

Authors

  • Matthias W Wagner
    Institute of Diagnostic and Interventional Radiology, University Hospital Zurich, Raemistrasse 100, 8091, Zurich, Switzerland.
  • Birgit B Ertl-Wagner
    From the Department of Radiology, University of Wisconsin Madison School of Medicine and Public Health, 600 Highland Dr, Madison, WI 53792 (D.A.B., M.L.S.); Department of Radiology, New York University, New York, NY (L.M.); Department of Musculoskeletal Radiology (M.A.B.) and Institute for Technology Assessment (E.F.H.), Massachusetts General Hospital, Boston, Mass; Department of Medical Imaging, Hospital for Sick Children, University of Toronto, Toronto, Canada (B.B.E.W.); Department of Radiology, University of California-San Diego, San Diego, Calif (K.J.F.); Department of Cancer Imaging, Division of Imaging Sciences & Biomedical Engineering, Kings College London, London, England (V.J.G.); Department of Radiology and Biomedical Imaging, University of California-San Francisco, San Francisco, Calif (C.P.H.); and Department of Radiology and Radiologic Science, The Johns Hopkins University School of Medicine, Baltimore, Md (C.R.W.).

Keywords

External Resources

Popular Topics
Recent Journals