Evaluating the Influence of MCAT Scores on Medical Student Selection and Performance Using a Machine Learning Approach: Retrospective Cohort Study.

BACKGROUND: The Medical College Admission Test (MCAT) has been central to medical school admissions in North America, though its necessity in holistic selection processes remains debated. The COVID-19 pandemic's suspension of MCAT testing sessions allowed institutions to explore alternative admission criteria. Additionally, global challenges in test administration underscore the vulnerability of systems dependent on a single standardized test. OBJECTIVE: This study aimed to (1) assess whether including MCAT scores in admissions decisions improves the prediction of medical school performance compared with grade point average (GPA) and interview-based selection and (2) evaluate whether machine learning (ML) can generate viable MCAT score predictions when testing is unavailable. METHODS: We conducted a retrospective cohort study of 1898 applicants to the Lebanese American University School of Medicine (2009-2023). Among 349 admitted students with MCAT scores, we compared admission composites including vs excluding the MCAT in relation to academic outcomes (Med 1-4 final grades) and clinical performance (Med 1-2 objective structured clinical examination [OSCE] scores). Students were stratified into tertiles to examine tier-specific effects. To model an MCAT alternative, we trained an ensemble ML model (least absolute shrinkage and selection operator, kernel ridge, gradient boosting, elastic net, and light gradient boosting machine [LightGBM]) using data from 1583 applicants (2009-2020) to predict MCAT scores from cumulative GPA, core GPA, interview scores, merit points, and honors participation. The model was validated on 315 applicants admitted during the MCAT suspension (2021-2023), and its impact on admission rankings and performance correlations was evaluated. RESULTS: Including the MCAT in the admissions composite did not meaningfully improve prediction of overall academic performance (r=0.63, 95% CI 0.56-0.69 with MCAT vs r=0.62, 95% CI 0.55-0.68 without MCAT; P=.63). However, it significantly weakened prediction of clinical skills (OSCE: r=0.37, 95% CI 0.27-0.46 with MCAT vs r=0.48, 95% CI 0.40-0.56 without MCAT; P<.001). In tertile analyses, the MCAT modestly improved prediction among top-performing students but eliminated predictive validity in the bottom tertile (r=0.12, 95% CI -0.07 to 0.30; P=.21 vs r=0.28, 95% CI 0.10-0.44 without MCAT; P=.003). The ML model explained 36% of MCAT variance (R²=0.36 with 95% CI 0.32-0.41; root mean squared error ≈10% of score range). Incorporating predicted MCAT scores reduced correlations with subsequent performance across all outcomes. Although rankings based on predicted MCAT scores strongly correlated with original rankings (r=0.80, 95% CI 0.77-0.83), 6.3% (4/64) to 9.4% (6/64) of admission decisions would have changed. CONCLUSIONS: Within this single-institution study, MCAT scores provided limited incremental validity for predicting academic performance and reduced prediction of clinical skills. ML-based predictions of MCAT scores introduced sufficient error to affect admissions decisions, supporting a robust holistic admissions process during temporary MCAT disruptions while highlighting the need for validation across diverse institutional settings before broader generalization.