Amount and certainty of evidence in Cochrane systematic reviews of interventions: a large-scale meta-research study

To quantify the amount and certainty of evidence in Cochrane systematic reviews of interventions, and to describe how this evidence has evolved over time. Large-scale meta-research study Cochrane Database of Systematic Reviews (search date April 8, 2025) Cochrane systematic reviews assessing interventions reporting “Summary of findings” tables. Data were automatically extracted using web scraping and a large language model, with quality control performed by humans on a random sample. We describe the certainty of evidence for each population-intervention-comparison-outcome (PICO) question reported in all Cochrane “Summary of findings” tables. When available, we compared the certainty of evidence between the initial version and the latest update. We identified 5,116 reviews that reported a “Summary of findings” table, containing 64,849 PICO questions. Overall, 24% (n = 15,768) of PICOs had no study included, 31% (n = 20,390) included only 1 study, 14% (n = 8,796) 2 studies, and 31% (n = 19,895) more than 2 studies. Nearly all PICOs (97%) only included randomized trials. The median [Q1-Q3] number of included participants was 123 [0–557]. The certainty of evidence was rated as high for 4% (n = 2,852), moderate for 16% (n = 10,574), low for 27% (n = 17,409), very low for 26% (n = 17,012), and not assessed for 26% (n = 17,002). Of the 7,461 PICO questions with an update (median time to update of 4.3 years [Q1-Q3: 2.6-6.4]), the number of included studies in the latest update remained the same for 63%; the certainty of evidence was unchanged for 71%; upgraded for 13% and downgraded for 15%. The amount and certainty of evidence is low and has not improved over time with review updates. These results question the efficiency of the research ecosystem. High quality up-to-date evidence synthesis is essential for decision-makers. Confidence in the evidence informing decision-making can be limited by the amount and quality of primary research on a specific research question This large-scale meta-research study analyzed all Cochrane “Summary of findings” tables (i.e., 64,849 population-intervention-comparison-outcome – PICO – questions), and found that about two thirds of the PICO questions were informed by two or fewer studies, with a median [Q1-Q3] of 123 [0–557] participants per PICO; the associated certainty of evidence was rated as high in only 4% of the cases. After an update of the review (i.e., 7,461 PICOs), 63% PICOs did not include additional studies, and 71% showed no change in certainty of evidence; upgrades and downgrades of certainty occurred at similar frequencies. These results question the efficiency of the research ecosystem.