Benchmark Evaluation of Vision-Enabled Large Language Models in Mammography: A Case-Control Study.

OBJECTIVE: To compare the diagnostic performance of five contemporary vision-enabled large language models (LLMs) in mammography interpretation using a case-control benchmark design. METHODS: A total of 200 cases were analyzed using a quota-balanced case-control design (100 malignant, BI-RADS 4-5; 100 controls, BI-RADS 1-2). All models (ChatGPT-5T, ChatGPT-5I, ChatGPT-4o, Gemini 2.5 Pro, and Gemini 2.5 Flash) received standard mammography images. Models were evaluated for lesion detection, laterality, quadrant localization, lesion type, and malignant-benign discrimination. Performance was assessed using accuracy, sensitivity, specificity, balanced accuracy, Cohen's κ, and area under the receiver operating characteristic curve (AUC). Breast density was recorded for all cases, while histopathological subtype and lesion size were documented for malignant cases. RESULTS: Lesion detection accuracy ranged from 79% to 91%, with high malignant sensitivity (92%-100%) but lower specificity in normal cases (24%-82%). ChatGPT-5T performed best in laterality (83%; κ = 0.70) and quadrant localization (73%; κ = 0.62). Lesion type classification was modest (74%-85%). In malignant-benign discrimination, ChatGPT-5T achieved the highest balanced accuracy (87%), while ChatGPT-4o showed poor specificity (28%). The study cohort was predominantly composed of lower-density breasts, larger lesions, and invasive ductal carcinoma as the most common subtype. CONCLUSION: Vision-enabled LLMs demonstrated high sensitivity for breast lesion detection and classification tasks. More deliberative models, particularly ChatGPT-5T, consistently outperformed faster counterparts, especially in spatial reasoning tasks. However, reduced specificity in normal cases underscores that current LLM outputs should remain under radiologist supervision as supportive second-reader tools rather than standalone diagnostic systems. Given the quota-balanced design and cohort composition, results should be interpreted as comparative benchmarking rather than real-world diagnostic performance.