Assessing the Reliability of Large Language Models for Evaluation of Risk of Bias in Randomized Clinical Trials.

OBJECTIVE: Systematic reviews depend on rigorous risk-of-bias (RoB) assessments to ensure credibility, yet manual evaluation using the Cochrane RoB 2 tool is resource-intensive. While large language models (LLMs) offer potential for automation, their alignment with human judgment remains underexplored. This study evaluates the reliability of ChatGPT-4o, ChatGPT-5, and Claude 3.5 Sonnet in assessing RoB in randomized controlled trials (RCTs), comparing their agreement with human reviewers and internal consistency. STUDY DESIGN: We retrospectively analyzed 180 RCTs from systematic reviews published in the American Journal of Obstetrics and Gynecology (2021-2023) reporting complete human RoB 2 ratings. Each LLM processed full-text PDFs using a standardized prompt incorporating the complete RoB 2 algorithm. Model performance was evaluated against human benchmarks using Cohen's kappa and prevalence- and bias-adjusted kappa. Intramodel reliability was assessed across three independent runs to measure consistency. RESULTS: ChatGPT-5 consistently outperformed other models, achieving the highest agreement in randomization (Domain 1; 76%), missing outcome data (Domain 3; 80%), and outcome measurement (Domain 4; 76%). It showed moderate concordance for deviations from intended interventions (69%). However, all models struggled with selective reporting (Domain 5), where agreement dropped to 47 to 51%. For overall RoB judgments, ChatGPT-5 demonstrated superior concordance (60-62%, κ = 0.36-0.40) compared with ChatGPT-4o (45%) and Claude 3.5 Sonnet (43%). ChatGPT-5 also exhibited substantial to near-perfect internal consistency. CONCLUSION: Among the evaluated models, ChatGPT-5 most closely approximated human RoB 2 assessments and achieved superior internal consistency, suggesting it could serve as a practical first-pass tool to reduce reviewer burden. However, persistent limitations in detecting selective reporting-likely due to the inability to cross-reference external trial registries-highlight that expert human oversight remains essential for accurate evidence synthesis. KEY POINTS: · GPT-5, GPT-4o, and Claude evaluated 180 RCTs.. · GPT-5 outperformed GPT-4o and Claude models.. · Models struggled with selective reporting bias..