Assessment of OpenEvidence Responses to American Academy of Orthopaedic Surgeons Clinical Practice Guidelines for Anterior Cruciate Ligament Reconstruction.

INTRODUCTION/OBJECTIVES: General-purpose large language models (LLMs) have substantial limitations, including fabricated references and inconsistent concordance with evidence-based clinical practice guidelines (CPGs). OpenEvidence (OE) (Cambridge, MA) is a retrieval-augmented generation (RAG) platform designed to provide accurate, evidence-based, citation-supported answers. This study assessed the concordance, quality, and citation characteristics of OE responses to the 2022 American Academy of Orthopaedic Surgeons (AAOS) CPG for anterior cruciate ligament reconstruction (ACLR). It was hypothesized that OE would deliver relevant, accurate, clear, complete, and evidence-based responses, with low rates of hallucination and irrelevant citations. METHODS: Fifteen prompts reflecting the 2022 AAOS ACLR CPG were curated and categorized into three domains: (1) preoperative evaluation and diagnosis, (2) surgical timing and technique, and (3) rehabilitation and prevention. Each prompt was entered into OE with a new session to minimize context bias. Two blinded orthopaedic surgeons rated response quality on a 5-point Likert scale for relevance, accuracy, clarity, completeness, and evidence basis. Inter-rater reliability was assessed using intra-class correlation coefficients (ICCs). All citations were extracted and classified as relevant, irrelevant, or hallucinated, and journal sources and publication years were recorded. RESULTS: Inter-rater reliability was high across all domains (overall ICC 0.90; 95% confidence interval, 0.82-0.96). Median scores exceeded 4.0 for all domains, with perfect scores for evidence basis (5.0 [5.0-5.0]) and lowest scores for clarity (5.0 [4.0-5.0]). Across 15 prompts, OE generated 127 PubMed-indexed citations (mean 8.5 per response; range 6-11) with zero hallucinated and only 2.4% irrelevant citations. Citations spanned 29 journals from 1997-2025. CONCLUSION: OE produced highly relevant, accurate, clear, complete, and evidence-based responses to AAOS ACLR CPG prompts with trustworthy PubMed-indexed citations and no hallucinations. Its RAG architecture may offer a credible, citation-oriented adjunct to evidence-based decision support in orthopaedic surgery. LEVEL OF EVIDENCE: Level V, descriptive study.