Development and validation of natural language processing pipelines to extract injury and surgery-related data elements from ACL reconstruction operative reports.

BACKGROUND: Over 100,000 patients undergo ACLR each year in the United States, providing rich electronic medical record data to improve outcomes, but manual chart review can be time-consuming and expensive. Natural language processing (NLP) methods have been successfully used to extract information for other orthopaedic operative procedures (e.g., hip and knee arthroplasties). Different subspecialities, however, use diverse terminology or "sublanguages" to describe clinical concepts and NLP models perform better concept extraction when trained on specialty-specific texts. Therefore, the purpose of this study was to develop and validate a reliable NLP pipeline to extract meaningful clinical data elements (e.g., graft type, meniscal involvement) from ACLR operative reports. METHODS: Operative reports for a training and test set were randomly selected based on surgeon volume and year from individuals who underwent ACLR within a single healthcare system between 2013 and 2021. Clinical Language Annotation, Modeling, and Processing Toolkit (CLAMP) was utilized to train a domain-specific model and build a pipeline for data extraction of clinically meaningful data elements, including injury-related factors and surgical factors. Relevancy metrics were calculated using values identified by a single clinician as the gold standard. RESULTS: Overall, the individuals selected for the training and test sets (n = 437 total) were 26.6 ± 10.9 years old, and 43.5% were female, similar to the age (26.7 ± 11.5 years) and proportion of females (43.0%) in the full data set (n = 5,818). Priority entities, including the side of surgery, ACL graft type, ACL procedure, and meniscal involvement, were identified with F1 scores between 0.87-1. CONCLUSIONS: F1 scores of 0.87-1 were attained for several priority entities using a modest set of annotated ACL operative reports to train the model and an NLP tool designed for non-expert use. The NLP pipelines have the potential to extract relevant operative information for a large cohort of patients to support clinical research, such as identifying predictors of subsequent surgery after ACLR. This work can also inform the planning of future studies using larger training sets and more advanced NLP methods.