AI-enabled GRADE: How the GRADE Working Group will use automation to rate the certainty of evidence of intervention effects.

OBJECTIVE: The Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group is developing GRADErater (GRADE Rating Automation Through Enhanced Reasoning), an official, automated tool for evaluation of the certainty of the evidence (CoE). In this article we describe the principles and methods underlying its development and state how the GRADE Working Group will use automation to rate the CoE of intervention effects. PRINCIPLES AND METHODS: We followed the GRADE methods for registering a project group on GRADE and AI. The project group established that the automated tool should be developed according to the following principles: (i) compliance with current GRADE guidance, (ii) transparency, (iii) human oversight, (iv) implementation of decision rules, (v) ease of use, (vi) understandability, and (vii) continuous improvement. We developed a set of decision rules to appraise each domain of the CoE in pairwise and network meta-analysis. These rules were developed based on official GRADE sources and validated by a GRADE experts. Based on these rules, we created a first version (https://gradeai.med.up.pt/) for GRADErater. We are now evaluating this version in terms of (i) its underlying rules, and (ii) its user interface. These assessments will allow for a refinement of the rules and of the first version. The modified version will be appraised and presented to the GRADE Working Group with input from internal and external interest-holders before we seek formal approval. Once launched, the automated tool will be continuously evaluated and refined by incorporating feedback from end-users. We will add new features (including generative AI-based functionalities), integrate this tool with GRADEpro, and develop versions in languages other than English. CONCLUSION: This project will follow a transparent methodology to create GRADErater, an official tool endorsed by the GRADE Working Group that will support humans in applying the most current GRADE methods to rate the CoE.