Cost-effective instruction learning for pathology vision and language analysis.

Journal: Nature computational science
Published Date: Jun 19, 2025

Abstract

The advent of vision-language models fosters interactive conversations between artificial intelligence-enabled models and humans. However, applying these models in the clinic faces challenges related to large-scale training data as well as financial and computational resources. Here we propose CLOVER, a cost-effective instruction learning framework for conversational pathology. CLOVER trains a lightweight module and uses instruction tuning while freezing the parameters of the large language model. Instead of using costly GPT-4, we propose well-designed prompts on GPT-3.5 for building generation-based instructions, emphasizing the utility of pathological knowledge derived from the Internet source. We construct a high-quality set of template-based instructions in the context of digital pathology. Using two benchmark datasets, our findings reveal the strength of hybrid-form, pathological visual question-answer instructions. CLOVER outperforms baselines that possess 37 times more training parameters and exhibits few-shot capacity on an external clinical dataset. CLOVER could thus accelerate the adoption of rapid conversational applications in digital pathology.

Authors

  • Kaitao Chen
    Shanghai Artificial Intelligence Laboratory, Shanghai, China.
  • Mianxin Liu
  • Fang Yan
    College of Communication Engineering, Chongqing University, Chongqing, 400044, China.
  • Lei Ma
    School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, Sichuan, China. Electronic address: leima@wit.edu.cn.
  • Xiaoming Shi
    China CDC Key Laboratory of Environment and Population Health, National Institute of Environmental Health, Chinese Center for Disease Control and Prevention, Beijing 100021, People's Republic of China.
  • Lilong Wang
    Shanghai Artificial Intelligence Laboratory, Shanghai, China.
  • Xiaosong Wang
    Imaging Biomarkers and Computer-aided Diagnosis Lab, Radiology and Imaging Sciences, NIH Clinical Center, Bethesda, MD, 20892-1182, USA.
  • Lifeng Zhu
    The State Key Laboratory of Bioelectronics, Jiangsu Key Lab of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Sipailou 2, Nanjing, 210096, Jiangsu, China. lfzhulf@gmail.com.
  • Zhe Wang
    Department of Pathology, The Eighth Affiliated Hospital, Sun Yat-sen University, Shenzhen 518033, China.
  • Mu Zhou
  • Shaoting Zhang

Keywords

No keywords available for this article.

External Resources

Popular Topics
Recent Journals