Diagnosis, triage and response quality of five large language models on emergency medicine vignettes: a blinded comparison

Background: Emergency care requires appropriate triage and management as well as diagnosis. We compared five large language models on these tasks using the same clinical vignettes. Methods: Seventeen synthetic emergency department vignettes were submitted to Claude Fable 5, GPT-5.6 Sol, Grok 4.5, Gemini 3.6 Flash and Kimi K3 through consumer interfaces on 11-12 August 2026. Two emergency physicians independently assessed the 85 responses with model identities concealed. Outcomes were agreement with reference diagnoses and Canadian Triage and Acuity Scale (CTAS) levels, appropriateness and overall quality (1-5 scales), potential harm and preferred response. Results: Every model correctly named the reference diagnosis in all 17 cases. CTAS exact agreement ranged from 14/17 to 15/17 per model (p=0.938). Of 13 discordant assignments, nine were undertriage and four overtriage. Appropriateness and quality differed across models (both p<0.001): Kimi K3 had the highest mean scores (4.82 and 4.74) and Grok 4.5 the lowest (3.91 and 3.71); Grok 4.5 scored significantly lower than every other model on both scales after Holm adjustment. Nine responses were flagged for potential harm: four from Grok 4.5, two each from Gemini 3.6 Flash and GPT-5.6 Sol, one from Claude Fable 5 and none from Kimi K3 (p=0.119). Conclusion: In this selected set of synthetic vignettes, all five models correctly named the reference diagnosis, yet undertriage and potentially harmful omissions occurred and response quality differed between models. These findings do not establish any model's clinical safety.