Process-Supervised Reward Models for Verifying Clinical Note Generation: A Scalable Approach Guided by Domain Expertise
Journal:
arXiv
Published Date:
Dec 17, 2024
Abstract
Process-supervised reward models (PRMs), which verify large language model
(LLM) outputs step-by-step, have achieved significant success in mathematical
and coding problems. However, their application to other domains remains
largely unexplored. In this work, we train a PRM to provide step-level reward
signals for clinical notes generated by LLMs from patient-doctor dialogues.
Guided by real-world clinician expertise, we carefully designed step
definitions for clinical notes and utilized Gemini-Pro 1.5 to automatically
generate process supervision data at scale. Our proposed PRM, trained on the
LLaMA-3.1 8B instruct model, outperformed both Gemini-Pro 1.5 and the vanilla
outcome-supervised reward model (ORM) in two key evaluations: (1) selecting
gold-reference samples from error-containing ones, achieving 98.8% accuracy
(versus 70.0% for the vanilla ORM and 93.8% for Gemini-Pro 1.5), and (2)
selecting physician-preferred notes, achieving 56.2% accuracy (compared to
37.5% for the vanilla ORM and 50.0% for Gemini-Pro 1.5). Additionally, we
conducted ablation studies to determine optimal loss functions and data
selection strategies, along with physician reader studies to explore predictors
of downstream Best-of-N performance. Our promising results suggest the
potential of PRMs to extend beyond the clinical domain, offering a scalable and
effective solution for diverse generative tasks.
