Item-writing flaws and answer-position bias in large language model-generated medical multiple-choice questions: a benchmark study of 15 models with an open corpus and a rule-based detector
Journal of Educational Evaluation for Health Professions · 2026-11-05 · 23 · 36
DOI: 10.3352/jeehp.2026.23.36 ↗Abstract
Purpose: Item-writing flaws in large language model (LLM)-generated medical multiple-choice questions have been assessed in small samples. This study developed an open benchmark of such items labeled for structural flaws, compared 15 models and 2 prompts, traced the source of answer-position bias, and evaluated LLM judges against physicians.Methods: Fifteen LLMs generated 12,000 vignette items across 8 specialties using a plain prompt and a prompt with 10 item-writing guidelines, plus 15,800 items in 5 experimental arms. A 14-rule detector labeled the corpus and 667 retired National Board of Medical Examiners (NBME) items; it was validated against 3,737 external human labels and 2 independent readers. Two physicians rated 100 items; 4 LLM judges assessed those items and 216 human-labeled items.Results: The detector found no format flaws, but 23.4% of plain-prompt items had a structural flaw, compared with 17.3% of NBME 5-option items. Excluding the longest-option rule reduced the gap from 6.1 to 1.2 percentage points. Guidelines halved the odds of a flaw (odds ratio, 0.50; 95% confidence interval, 0.45–0.56), reducing prevalence to 13.6%. Under the plain prompt, 7 models placed the key at position A in 61%–94% of items; removing the template’s example letter reduced the pooled proportion from 54% to 23%. Physicians found a content defect in 12%–13% of items (κ=0.59). The best LLM judge achieved agreement of κ=0.53–0.56 with the physicians on differently defined composites, a preliminary result.Conclusion: LLM-generated items avoided format flaws but contained answer cues and a position-A bias arising mostly from the output template. Guidelines reduced the flaws they specified without demonstrably improving content, which needs physician review.
EUVIMED