Automated Prediction of Glasgow Coma Scale Scores From Unstructured Electronic Health Records Using Natural Language Processing: Development and Validation Study.

BACKGROUND: Multicenter electronic health records (EHRs) can support quality improvement and comparative effectiveness research in critical care. However, limitations of EHR-based research include challenges in abstracting key clinical variables, including a patient's level of consciousness. OBJECTIVE: This study aimed to develop a natural language processing model to predict Glasgow Coma Scale (GCS) scores from daily EHR notes. METHODS: The study included adult patients (aged ≥18 years) admitted to Mass General Brigham (MGB) hospitals (2017-2024) and patients from the Medical Information Mart for Intensive Care-III (MIMIC-III version 1.4; 2001-2012) database. A dataset of all patients from both institutions was split into training (70%) or hold-out test (30%) sets. Variables consisted of daily notes, age, sex, and admission type. A pooled ordinal regression model (ordinalNet) with an elastic net penalty was trained to predict the lowest daily level of consciousness across 3 classes of impairment: severe (GCS score 3-8), moderate (GCS score 9-12), and mild (GCS score 13-15), and a pooled linear model was trained to predict continuous GCS scores (3-15). Gold standard GCS was obtained from structured flowsheet data. External generalizability was assessed using a single-institution ordinal model trained on MGB and tested on MIMIC. Following post hoc calibration, the performance of the ordinal and linear models was evaluated on the hold-out test sets using the area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC) for the ordinal models and root mean square error and Pearson correlation coefficient for the linear models. RESULTS: The modeling cohort included 145,897 patients (MGB: n=123,257, MIMIC: n=22,640), with 1,446,965 days of hospitalization between training and testing sets; the average age was 62 (SD 18) years, and the sex distribution was balanced. The pooled ordinalNet achieved an AUROC of 0.96 (95% CI 0.96-0.96) and an AUPRC of 0.77 (95% CI 0.76-0.77). The single-institution ordinal model achieved an AUROC of 0.90 (95% CI 0.89-0.90) and an AUPRC of 0.80 (95% CI 0.79-0.80). The pooled linear model achieved a root mean square error of 2.30 (95% CI 2.30-2.30) and a correlation of 0.76 (95% CI 0.76-0.76). Predictions for severe GCS were driven by terms indicating unresponsiveness and critical interventions, moderate GCS by intermediate alertness descriptors, and mild GCS by mentions of normal or awake behavior. CONCLUSIONS: Pooled ordinal and linear models can accurately predict GCS from unstructured data and can support large-scale phenotyping of neurological assessments for future critical care research.