EUVIMEDScopri EUVIMED
CrossrefScheda informativa

Evaluating the Performance of Large Language Models in Vascular Surgery Cases: Cross-Sectional Study

Asanka Wijetunga, Yunyi Wang, Chantell Yaghi, Mauro Vicaretti

JMIR Perioperative Medicine · 2026-11-06 · 9 · e91973-e91973

DOI: 10.2196/91973 ↗
AbstractDati bibliografici

Abstract

Abstract Background Large language models (LLMs) such as ChatGPT, Gemini, and Claude are increasingly used by clinicians, yet their accuracy, safety, and consistency in clinical cases remain poorly defined. Most studies assess LLMs using multiple-choice questions rather than free-response reasoning. Objective The primary objective of this study was to evaluate the performance and safety of 3 widely used LLMs in realistic vascular surgery scenarios in key domains such as diagnosis, investigation, and multidisciplinary and multimodal management. The secondary objective of this study was to evaluate the safety of the models’ outputs. Methods A total of 42 fictitious vascular cases across 5 major pathologies (acute limb ischemia [ALI], aortopathy, chronic limb ischemia, diabetic foot infection, and extracranial cerebrovascular disease) were independently entered into GPT-5, Gemini 2.5, and Claude Sonnet 4.5 using standardized prompts. Each model answered structured questions covering diagnosis, investigation, and management (both operative and nonoperative). Responses were scored out of 20 by a panel using a predefined rubric. Each plan’s overall safety was separately assessed. Comparative analyses used 2-tailed t tests, ANOVA, and multivariable logistic regression. Results Mean composite scores by model were 86.5% for ChatGPT, 83.5% for Gemini, and 88.0% for Claude, whereas scores by disease were 82.7% for ALI, 83.8% for aortopathy, 85.9% for chronic limb ischemia, 86.1% for diabetic foot infection, and 80.7% for extracranial cerebrovascular disease ( P >.05 in each case). Unsafe plans were generated in 11.9%, 23.8%, and 7.1% of cases by ChatGPT, Gemini, and Claude, respectively. On multivariable analysis, independent predictors of unsafe outputs were lower composite score (odds ratio [OR] 0.471, 95% CI 0.306‐0.722; P =.001), higher word count (OR 1.003, 95% CI 1.001‐1.005; P =.001), and ALI (OR 20.9, 95% CI 3.79-61.8; P

Conoscenze adatte al tuo lavoro.

Dall’inizio gratuito alla ricerca bibliografica personale, fino alla Research Suite completa.

Visualizza tariffe →

Sono inclusi tutti i testi integrali degli editori?

No. Citazioni, abstract e testi integrali sono contenuti diversi. La disponibilità dipende dalla fonte, dall’accesso aperto e dalle licenze esistenti. Un piano EUVIMED non sostituisce diritti aggiuntivi degli editori.

Visualizza tariffe →