Performance of large language models on the Japanese cardiovascular surgery board examination: a comparative analysis of eight contemporary AI models with educational implications.

OBJECTIVE: Large language models (LLMs) have shown strong performance on medical licensing and specialty examinations, but their utility in cardiovascular surgery certification and education remains unknown. METHODS: We evaluated eight LLMs (GPT-5, GPT-4o [OpenAI], Gemini-2.5Pro, Gemini-2.0, Gemma-3 [12B] [Google DeepMind], Claude-4, Claude-3 [Anthropic], and Llama-4 [Scout] [Meta AI]) using their official application programming interfaces as of September 2025. Examination items were obtained from the Japanese Cardiovascular Surgery Board (2021-2024; 523 questions). Texts were extracted from PDFs, images converted to JPEGs, and each question presented with a standardized Japanese prompt. Models produced three responses per item; final answers were determined by majority voting. Accuracy with 95% confidence intervals was calculated, and pairwise comparisons performed using McNemar's test. RESULTS: Across 523 items, GPT-5 achieved the highest accuracy (87.4%), followed by Gemini-2.5Pro (85.7%); their performance did not differ significantly. Claude-4 ranked third (80.3%), exceeding the passing threshold in some years. GPT-4o (65.6%) and Gemini-2.0 (58.5%) showed moderate accuracy, whereas Claude-3 (36.9%), Gemma-3 (40.9%), and Llama-4 (52.2%) scored lower. Pairwise testing confirmed a clear stratification: high (GPT-5, Gemini-2.5Pro), upper-intermediate (Claude-4), mid (GPT-4o, Gemini-2.0), and low (others). All models declined with image-based items, with top models reduced to around 70%. Accuracy remained stable across years, with GPT-5, Gemini-2.5Pro, and occasionally Claude-4, surpassing the pass threshold. CONCLUSIONS: Successive model generations, particularly GPT-5 and Gemini-2.5Pro, consistently achieved passing-level accuracy. These findings highlight substantial gains through model evolution and underscore the potential of LLMs as supplementary tools for specialty education, despite persistent limitations in image-based reasoning.