EUVIMEDEUVIMED entdecken
CrossrefInfokarte

Large language models are unstable as autonomous medical evaluators across clinical tasks

Marcelo Carvalho e Silva, Fernando Carvalho e Silva, Renata de Paula Gomes Nunes de Castro, Alex Gonçalves, Artur Figueiredo Delgado

Journal of the American Medical Informatics Association · 2026-11-06

DOI: 10.1093/jamia/ocag190 ↗
AbstractBibliografische Angaben

Abstract

Abstract Objective Large language models (LLMs) are increasingly used as automated evaluators of medical text (“LLM-as-judge”). We tested whether LLM-evaluator performance validated on one medical task against a human reference standard remains stable on a structurally different task. Materials and methods We evaluated 10 LLMs (3 locally served open-source and 7 API models: Claude, GPT, DeepSeek) as autonomous judges on 3 tasks with human reference labels: LiveQA-Med answer quality (n = 300), PubMedQA research appraisal (n = 300), and Emergency Severity Index (ESI) triage on MIMIC-IV-Ext (n = 256). Each case was scored 3 times per model at temperature zero (25 680 judgments). Analyses used cluster-bootstrap confidence intervals, quadratic-weighted Cohen’s κ, and false-discovery-rate correction. Results Evaluator performance was unstable across tasks: no model ranked in the top 3 on all 3 tasks, and every model shifted by 3 or more rank positions between at least one pair of tasks. In the highest-stakes task, ESI level 1 (Resuscitation) cases were under-triaged in 89.1% of evaluations, in all 10 models (range 70.2%-96.9%). Best triage agreement only reached the human inter-rater range (Claude Opus quadratic-weighted κ = 0.60 [0.54, 0.66]). Locally served 7-27B models (no per-query charge) occupied the Pareto frontier on the text tasks but only its low-concordance end on triage. Discussion LLM-evaluator performance is unstable between medical tasks that differ in cognitive structure; performance validated on one task should not be assumed to transfer to another. Conclusion Task-specific validation against human reference labels is required before deployment, and emergency triage demands human oversight at the highest-acuity tier.

Wissen, das zu Ihrer Arbeit passt.

Vom kostenlosen Einstieg über die persönliche Recherche bis zur vollständigen Research Suite.

Tarife ansehen →

Sind alle Verlagsvolltexte enthalten?

Nein. Bibliografische Nachweise, Abstracts und Volltexte sind unterschiedliche Inhalte. Ob ein Volltext verfügbar ist, hängt von der Quelle, Open Access und Ihren vorhandenen Lizenzen ab. Ein EUVIMED-Tarif ersetzt keine zusätzlichen Verlagsrechte.

Tarife ansehen →