Large language models are unstable as autonomous medical evaluators across clinical tasks
Journal of the American Medical Informatics Association · 2026-11-06
DOI: 10.1093/jamia/ocag190 ↗Abstract
Abstract Objective Large language models (LLMs) are increasingly used as automated evaluators of medical text (“LLM-as-judge”). We tested whether LLM-evaluator performance validated on one medical task against a human reference standard remains stable on a structurally different task. Materials and methods We evaluated 10 LLMs (3 locally served open-source and 7 API models: Claude, GPT, DeepSeek) as autonomous judges on 3 tasks with human reference labels: LiveQA-Med answer quality (n = 300), PubMedQA research appraisal (n = 300), and Emergency Severity Index (ESI) triage on MIMIC-IV-Ext (n = 256). Each case was scored 3 times per model at temperature zero (25 680 judgments). Analyses used cluster-bootstrap confidence intervals, quadratic-weighted Cohen’s κ, and false-discovery-rate correction. Results Evaluator performance was unstable across tasks: no model ranked in the top 3 on all 3 tasks, and every model shifted by 3 or more rank positions between at least one pair of tasks. In the highest-stakes task, ESI level 1 (Resuscitation) cases were under-triaged in 89.1% of evaluations, in all 10 models (range 70.2%-96.9%). Best triage agreement only reached the human inter-rater range (Claude Opus quadratic-weighted κ = 0.60 [0.54, 0.66]). Locally served 7-27B models (no per-query charge) occupied the Pareto frontier on the text tasks but only its low-concordance end on triage. Discussion LLM-evaluator performance is unstable between medical tasks that differ in cognitive structure; performance validated on one task should not be assumed to transfer to another. Conclusion Task-specific validation against human reference labels is required before deployment, and emergency triage demands human oversight at the highest-acuity tier.
EUVIMED