Evaluating the effectiveness of biomedical fine-tuning for large language models on clinical tasks.

Journal: Journal of the American Medical Informatics Association : JAMIA
Published Date:

Abstract

OBJECTIVES: Large language models (LLMs) have shown potential in biomedical applications, leading to efforts to fine-tune them on domain-specific data. However, the effectiveness of this approach remains unclear. This study aims to critically evaluate the performance of biomedically fine-tuned LLMs against their general-purpose counterparts across a range of clinical tasks.

Authors

  • Felix J Dorfner
    Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA; Athinoula A. Martinos Center for Biomedical Imaging, Charlestown, MA, USA; Department of Radiology, Charité - Universitätsmedizin Berlin corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Hindenburgdamm 30, 12203, Berlin, Germany.
  • Amin Dada
    Institute for Artificial Intelligence in Medicine, University Hospital Essen (AöR), Girardetstraße 2, 45131 Essen, Germany.
  • Felix Busch
    Institute for Diagnostic and Interventional Radiology, TUM School of Medicine and Health, TUM University Hospital Rechts der Isar, Technical University of Munich, Munich, Germany.
  • Marcus R Makowski
    School of Medicine and Health, Department of Diagnostic and Interventional Radiology, Klinikum rechts der Isar, TUM University Hospital, Technical University of Munich, Munich, Germany.
  • Tianyu Han
    Physics of Molecular Imaging Systems, Experimental Molecular Imaging, RWTH Aachen University, Aachen, Germany. tianyu.han@pmi.rwth-aachen.de.
  • Daniel Truhn
    Department of Diagnostic and Interventional Radiology, University Hospital Düsseldorf, Düsseldorf, Germany (J.S., D.B.A., S.N.); Institute of Computer Vision and Imaging, RWTH University Aachen, Pauwelsstrasse 30, 52072 Aachen, Germany (J.S., D.M.); Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany (D.T., M.P., F.M., C.K., S.N.); and Faculty of Mathematics and Natural Sciences, Institute of Informatics, Heinrich Heine University Düsseldorf, Düsseldorf, Germany (S.C.).
  • Jens Kleesiek
    AG Computational Radiology, Abteilung Radiologie, Deutsches Krebsforschungszentrum (DKFZ), Im Neuenheimer Feld 280, 69120, Heidelberg, Deutschland. j.kleesiek@dkfz-heidelberg.de.
  • Madhumita Sushil
    Bakar Computational Health Sciences Institute, University of California, San Francisco, San Francisco, California, USA.
  • Lisa C Adams
    School of Medicine and Health, Department of Diagnostic and Interventional Radiology, Klinikum rechts der Isar, TUM University Hospital, Technical University of Munich, Munich, Germany.
  • Keno K Bressem
    School of Medicine and Health, Institute for Cardiovascular Radiology and Nuclear Medicine, German Heart Center Munich, TUM University Hospital, Technical University of Munich, Munich, Germany.