MedS$^3$: Towards Medical Small Language Models with Self-Evolved Slow Thinking
Journal:
arXiv
Published Date:
Jan 21, 2025
Abstract
Medical language models (MLMs) have become pivotal in advancing medical
natural language processing. However, prior models that rely on pre-training or
supervised fine-tuning often exhibit low data efficiency and limited
practicality in real-world clinical applications. While OpenAI's o1 highlights
test-time scaling in mathematics, attempts to replicate this approach in
medicine typically distill responses from GPT-series models to open-source
models, focusing primarily on multiple-choice tasks. This strategy, though
straightforward, neglects critical concerns like data privacy and realistic
deployment in clinical settings. In this work, we present a deployable,
small-scale medical reasoning system, MedS3, designed for long-chain reasoning
in clinical tasks using a self-evolution paradigm. Starting with a seed dataset
of around 8,000 instances spanning five domains and 16 datasets, we prompt a
base policy model to perform Monte Carlo Tree Search (MCTS) to construct
rule-verifiable reasoning chains. Each reasoning step is assigned an evolution
rollout value, allowing verified trajectories to train the policy model and the
process reward model (PRM). During inference, the policy model generates
multiple responses, and the reward model selects the one with a newly proposed
PRM-guided Vote-Sum (P-VS) strategy. Experiments on eleven evaluation datasets
demonstrate that MedS3 outperforms not only the prior strongest medical model
by 6.59, but also 32B-level general reasoning models by 8.71 points. Code and
data are available at https://github.com/pixas/MedSSS.
