A multimodal generative AI copilot for human pathology.

Journal: Nature
Published Date: Jun 12, 2024

Abstract

Computational pathology has witnessed considerable progress in the development of both task-specific predictive models and task-agnostic self-supervised vision encoders. However, despite the explosive growth of generative artificial intelligence (AI), there have been few studies on building general-purpose multimodal AI assistants and copilots tailored to pathology. Here we present PathChat, a vision-language generalist AI assistant for human pathology. We built PathChat by adapting a foundational vision encoder for pathology, combining it with a pretrained large language model and fine-tuning the whole system on over 456,000 diverse visual-language instructions consisting of 999,202 question and answer turns. We compare PathChat with several multimodal vision-language AI assistants and GPT-4V, which powers the commercially available multimodal general-purpose AI assistant ChatGPT-4 (ref. ). PathChat achieved state-of-the-art performance on multiple-choice diagnostic questions from cases with diverse tissue origins and disease models. Furthermore, using open-ended questions and human expert evaluation, we found that overall PathChat produced more accurate and pathologist-preferable responses to diverse queries related to pathology. As an interactive vision-language AI copilot that can flexibly handle both visual and natural language inputs, PathChat may potentially find impactful applications in pathology education, research and human-in-the-loop clinical decision-making.

Authors

  • Ming Y Lu
    Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
  • Bowen Chen
    Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
  • Drew F K Williamson
    Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
  • Richard J Chen
    Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
  • Melissa Zhao
    University of California San Diego School of Medicine, La Jolla, CA, USA.
  • Aaron K Chow
    Department of Pathology, Wexner Medical Center, Ohio State University, Columbus, OH, USA.
  • Kenji Ikemura
    Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
  • Ahrong Kim
    Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
  • Dimitra Pouli
    Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
  • Ankush Patel
    Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, USA.
  • Amr Soliman
    Department of Pathology, Wexner Medical Center, Ohio State University, Columbus, OH, USA.
  • Chengkuan Chen
    Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
  • Tong Ding
    Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
  • Judy J Wang
    Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
  • Georg Gerber
    Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
  • Ivy Liang
    Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
  • Long Phi Le
    Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.
  • Anil V Parwani
    Department of Pathology, The Ohio State University Wexner Medical Centre, Columbus, OH, USA.
  • Luca L Weishaupt
    Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
  • Faisal Mahmood
    Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA. faisalmahmood@bwh.harvard.edu.

Keywords

External Resources

Popular Topics
Recent Journals