How does AI-generated assessment compare to expert assessment in simulation-based experiences: Analyzing AI-human interrater reliability.

BACKGROUND: In the health professions, artificial intelligence simulation-based experiences (AI-SBEs) are increasingly used to evaluate learner performance and provide feedback. Benefits of AI-SBEs include providing instantaneous feedback, allowing more practice opportunities than feasible using traditional methods, and reducing educator burden. However, methods for calibrating AI feedback are underdeveloped. METHODS: This study evaluated the feedback provided by a generative AI-based software program (SimConverse) when assessing simulated patient interactions using the Weight Sensitivity Instrument (WSI), a checklist used to assess learners' weight bias-related clinical behaviors. A two-phase interrater reliability (IRR) approach was used. First, iterative prompt engineering was guided by AI-human IRR results. Four authors and four volunteers generated 31 interactions in SimConverse. Interaction transcripts were downloaded, and another author with expertise in the WSI manually reviewed and scored the transcripts. Percent agreement and Gwet's AC1 were calculated to compare AI and expert assessments, and the research team reviewed feedback statements and agreement statistics to identify rubric instructions in need of revision. After revising the rubric in SimConverse, four authors produced 10 additional SBE interactions. Expert and AI assessments were explored. In phase two, the AI-SBE was piloted with 60 medical students in a larger intervention study. A random sample of 30 student transcripts were assessed for AI-human IRR, and students' comments about the AI-SBE were reviewed. RESULTS: Of the 11 WSI items, expert and AI agreement was consistently good or very good for 7 items across all rounds (Gwet's AC1=0.71 and higher). One item initially showed poor agreement (Gwet's AC1=0.56), but agreement improved after prompt engineering (final Gwet's AC1=0.89). The improved item assessed whether students made specific statements (asked if weight loss was a priority). Three other items had persistent poor agreement (final Gwet's AC1=0.42, 0.66, 0.64). These items required broader contextual assessment of the transcripts. Most open-ended student reactions to the AI-SBE were positive. CONCLUSIONS: Findings suggest that AI scoring and feedback within AI-SBEs requires iterative prompt engineering and educator/researcher guidance. Tuning was more successful for content-specific items than context-dependent items. Future work could explore best practices for prompt refinement, particularly for more complex and context-dependent rubric items.