Embedding-Driven Diversity Sampling to Improve Few-Shot Synthetic Data Generation
Journal:
arXiv
Published Date:
Jan 20, 2025
Abstract
Accurate classification of clinical text often requires fine-tuning
pre-trained language models, a process that is costly and time-consuming due to
the need for high-quality data and expert annotators. Synthetic data generation
offers an alternative, though pre-trained models may not capture the syntactic
diversity of clinical notes. We propose an embedding-driven approach that uses
diversity sampling from a small set of real clinical notes to guide large
language models in few-shot prompting, generating synthetic text that better
reflects clinical syntax. We evaluated this method using the CheXpert dataset
on a classification task, comparing it to random few-shot and zero-shot
approaches. Using cosine similarity and a Turing test, our approach produced
synthetic notes that more closely align with real clinical text. Our pipeline
reduced the data needed to reach the 0.85 AUC cutoff by 40% for AUROC and 30%
for AUPRC, while augmenting models with synthetic data improved AUROC by 57%
and AUPRC by 68%. Additionally, our synthetic data was 0.9 times as effective
as real data, a 60% improvement in value.
