Federated benchmarking of medical artificial intelligence with MedPerf.

Journal: Nature machine intelligence
Published Date: Jul 17, 2023

Abstract

Medical artificial intelligence (AI) has tremendous potential to advance healthcare by supporting and contributing to the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving both healthcare provider and patient experience. Unlocking this potential requires systematic, quantitative evaluation of the performance of medical AI models on large-scale, heterogeneous data capturing diverse patient populations. Here, to meet this need, we introduce MedPerf, an open platform for benchmarking AI models in the medical domain. MedPerf focuses on enabling federated evaluation of AI models, by securely distributing them to different facilities, such as healthcare organizations. This process of bringing the model to the data empowers each facility to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status and real-world deployment, our roadmap and, importantly, the use of MedPerf with multiple international institutions within cloud-based technology and on-premises scenarios. Finally, we welcome new contributions by researchers and organizations to further strengthen MedPerf as an open benchmarking platform.

Authors

  • Alexandros Karargyris
    IHU Strasbourg, Strasbourg, France.
  • Renato Umeton
    Dana-Farber Cancer Institute, Boston, MA, USA.
  • Micah J Sheller
    Intel, Santa Clara, CA, USA.
  • Alejandro Aristizabal
    Factored, Palo Alto, CA, USA.
  • Johnu George
    Nutanix, San Jose, CA, USA.
  • Anna Wuest
    Dana-Farber Cancer Institute, Boston, MA, USA.
  • Sarthak Pati
    Perelman School of Medicine, Philadelphia, PA, USA.
  • Hasan Kassem
    University of Strasbourg, Strasbourg, France.
  • Maximilian Zenk
    German Cancer Research Center, Heidelberg, Germany.
  • Ujjwal Baid
    Perelman School of Medicine, Philadelphia, PA, USA.
  • Prakash Narayana Moorthy
    Intel, Santa Clara, CA, USA.
  • Alexander Chowdhury
    Dana-Farber Cancer Institute, Boston, MA, USA.
  • Junyi Guo
    Harvard T.H. Chan School of Public Health, Boston, MA, USA.
  • Sahil Nalawade
    Dana-Farber Cancer Institute, Boston, MA, USA.
  • Jacob Rosenthal
    Dana-Farber Cancer Institute, Boston, MA, USA.
  • David Kanter
    MLCommons, San Francisco, CA, USA.
  • Maria Xenochristou
    Stanford University, Stanford, CA, USA.
  • Daniel J Beutel
    University of Cambridge, Cambridge, UK.
  • Verena Chung
    Sage Bionetworks, Seattle, WA, USA.
  • Timothy Bergquist
    Sage Bionetworks, Seattle, WA, USA.
  • James Eddy
    Sage Bionetworks, Seattle, WA, USA.
  • Abubakar Abid
    Hugging Face, New York, NY, USA.
  • Lewis Tunstall
    Hugging Face, New York, NY, USA.
  • Omar Sanseviero
    Hugging Face, New York, NY, USA.
  • Dimitrios Dimitriadis
    Microsoft, Redmond, WA, USA.
  • Yiming Qian
    A*STAR, Singapore, Singapore.
  • Xinxing Xu
    A*STAR, Singapore, Singapore.
  • Yong Liu
    Department of Critical care medicine, Shenzhen Hospital, Southern Medical University, Guangdong, Shenzhen, China.
  • Rick Siow Mong Goh
    A*STAR, Singapore, Singapore.
  • Srini Bala
    Supermicro, San Jose, CA, USA.
  • Victor Bittorf
    Meta, Menlo Park, CA, USA.
  • Sreekar Reddy Puchala
    Dana-Farber Cancer Institute, Boston, MA, USA.
  • Biagio Ricciuti
    Dana-Farber Cancer Institute, Boston, MA, USA.
  • Soujanya Samineni
    Dana-Farber Cancer Institute, Boston, MA, USA.
  • Eshna Sengupta
    Dana-Farber Cancer Institute, Boston, MA, USA.
  • Akshay Chaudhari
    Stanford University, Stanford, CA, USA.
  • Cody Coleman
    Stanford University, Stanford, CA, USA.
  • Bala Desinghu
    Rutgers University, New Brunswick, NJ, USA.
  • Gregory Diamos
    Landing.AI, Palo Alto, CA, USA.
  • Debo Dutta
    Nutanix, San Jose, CA, USA.
  • Diane Feddema
    Red Hat, Raleigh, NC, USA.
  • Grigori Fursin
    cKnowledge, Paris, France.
  • Xinyuan Huang
    Cisco, San Jose, CA, USA.
  • Satyananda Kashyap
    IBM Research, San Jose, CA, USA.
  • Nicholas Lane
    University of Cambridge, Cambridge, UK.
  • Indranil Mallick
    Tata Medical Center, Kolkata, India.
  • Pietro Mascagni
    IHU Strasbourg, Strasbourg, France.
  • Virendra Mehta
    University of Trento, Trento, Italy.
  • Cassiano Ferro Moraes
    Write Choice, Florianópolis, Brazil.
  • Vivek Natarajan
    Google, Mountain View, CA, USA.
  • Nikola Nikolov
    Supermicro, San Jose, CA, USA.
  • Nicolas Padoy
    IHU Strasbourg, Strasbourg, France.
  • Gennady Pekhimenko
    University of Toronto, Toronto, Ontario, Canada.
  • Vijay Janapa Reddi
    Harvard University, Cambridge, MA, USA.
  • G Anthony Reina
    Intel, Santa Clara, CA, USA.
  • Pablo Ribalta
    NVIDIA, Santa Clara, CA, USA.
  • Abhishek Singh
    Department of Urology, Muljibhai Patel Urological Hospital, Nadiad, India.
  • Jayaraman J Thiagarajan
    Lawrence Livermore National Laboratory, Livermore, CA, USA.
  • Jacob Albrecht
    Sage Bionetworks, Seattle, WA, USA.
  • Thomas Wolf
    Hugging Face, New York, NY, USA.
  • Geralyn Miller
    Microsoft, Redmond, WA, USA.
  • Huazhu Fu
    A*STAR, Singapore, Singapore.
  • Prashant Shah
    Intel, Santa Clara, CA, USA.
  • Daguang Xu
    NVIDIA, Santa Clara, CA, USA.
  • Poonam Yadav
    Department of Microbiology, Chitwan Medical College, Tribhuvan University, Chitwan, Nepal.
  • David Talby
    John Snow Labs, Lewes, DE, USA.
  • Mark M Awad
    Dana-Farber Cancer Institute, Boston, MA, USA.
  • Jeremy P Howard
    fast.ai, San Francisco, CA, USA.
  • Michael Rosenthal
    Dana-Farber Cancer Institute, Boston, MA, USA.
  • Luigi Marchionni
    Weill Cornell Medicine, New York, NY, USA.
  • Massimo Loda
    Dana-Farber Cancer Institute, Boston, MA, USA.
  • Jason M Johnson
    Dana-Farber Cancer Institute, Boston, MA, USA.
  • Spyridon Bakas
    Perelman School of Medicine, Philadelphia, PA, USA.
  • Peter Mattson
    MLCommons, San Francisco, CA, USA.

Keywords

No keywords available for this article.

External Resources

Popular Topics
Recent Journals