The Illusion of Precision: Artificial Intelligence Cannot Yet Reliably Predict Individual Outcomes in Infrapopliteal Chronic Limb-Threatening Ischemia.
Journal:
Annals of vascular surgery
Published Date:
Jul 14, 2026
Abstract
OBJECTIVE: The integration of Large Language Models (LLMs) into vascular surgery promises scalable clinical decision support, yet their prognostic reliability remains unproven. This study evaluates the accuracy, calibration, and stability of generative AI in predicting outcomes for Chronic Limb-Threatening Ischemia (CLTI). MATERIALS AND METHODS: In a retrospective single-center study, we evaluated 49 patients undergoing revascularization for CLTI. Two LLM architectures (GPT-5.5 and DeepSeek-V4) were tested using "Naive" (unstructured) and "Structured" (tabular) zero-shot prompting strategies to predict Major Adverse Events (MAE) at 6-month, 1-year, and 5-year horizons. Performance was assessed using Brier Scores for calibration, Sensitivity for discrimination, and Wasserstein Distance for stability. RESULTS: A critical dissociation between calibration and discrimination was observed. DeepSeek Naive achieved a superior Brier Score of 0.068 at 6 months, suggesting high probabilistic accuracy. However, this mathematical precision masked a failure in clinical utility: all models exhibited a Sensitivity of 0.00 at 6-month and 1-year horizons. The models adopted an "actuarial hedging" strategy, clustering predictions around the population mean to minimize mathematical error rather than identifying specific high-risk patients. While internal reproducibility (ICCs >0.500) and distributional stability across prompts were surprisingly high, this consistency proved deceptive, based entirely in non-discriminatory safety biases. CONCLUSION: While LLMs demonstrate semantic competence in medical discourse, they currently lack the pragmatic utility required for high-stakes prognostication. The observed "hallucinated calibration" and deceptive stability creates a dangerous illusion of reliability. Until issues of discriminatory failure are resolved, LLMs should remain restricted to administrative rather than predictive roles in patient decision making.
Authors
Keywords
No keywords available for this article.