EUVIMEDEUVIMED entdecken
CrossrefInfokarte

Item-writing flaws and answer-position bias in large language model-generated medical multiple-choice questions: a benchmark study of 15 models with an open corpus and a rule-based detector

Rohit Gundam

Journal of Educational Evaluation for Health Professions · 2026-11-05 · 23 · 36

DOI: 10.3352/jeehp.2026.23.36 ↗
AbstractBibliografische Angaben

Abstract

Purpose: Item-writing flaws in large language model (LLM)-generated medical multiple-choice questions have been assessed in small samples. This study developed an open benchmark of such items labeled for structural flaws, compared 15 models and 2 prompts, traced the source of answer-position bias, and evaluated LLM judges against physicians.Methods: Fifteen LLMs generated 12,000 vignette items across 8 specialties using a plain prompt and a prompt with 10 item-writing guidelines, plus 15,800 items in 5 experimental arms. A 14-rule detector labeled the corpus and 667 retired National Board of Medical Examiners (NBME) items; it was validated against 3,737 external human labels and 2 independent readers. Two physicians rated 100 items; 4 LLM judges assessed those items and 216 human-labeled items.Results: The detector found no format flaws, but 23.4% of plain-prompt items had a structural flaw, compared with 17.3% of NBME 5-option items. Excluding the longest-option rule reduced the gap from 6.1 to 1.2 percentage points. Guidelines halved the odds of a flaw (odds ratio, 0.50; 95% confidence interval, 0.45–0.56), reducing prevalence to 13.6%. Under the plain prompt, 7 models placed the key at position A in 61%–94% of items; removing the template’s example letter reduced the pooled proportion from 54% to 23%. Physicians found a content defect in 12%–13% of items (κ=0.59). The best LLM judge achieved agreement of κ=0.53–0.56 with the physicians on differently defined composites, a preliminary result.Conclusion: LLM-generated items avoided format flaws but contained answer cues and a position-A bias arising mostly from the output template. Guidelines reduced the flaws they specified without demonstrably improving content, which needs physician review.

Wissen, das zu Ihrer Arbeit passt.

Vom kostenlosen Einstieg über die persönliche Recherche bis zur vollständigen Research Suite.

Tarife ansehen →

Sind alle Verlagsvolltexte enthalten?

Nein. Bibliografische Nachweise, Abstracts und Volltexte sind unterschiedliche Inhalte. Ob ein Volltext verfügbar ist, hängt von der Quelle, Open Access und Ihren vorhandenen Lizenzen ab. Ein EUVIMED-Tarif ersetzt keine zusätzlichen Verlagsrechte.

Tarife ansehen →