Evaluating Large Language Model–Based Automated Scoring in a Voice-Based Virtual Standardized Patient Platform for Medical Students: Cross-Sectional Agreement Study
JMIR Medical Education · 24.09.2026 · 12 · e95578-e95578
DOI: 10.2196/95578 ↗Kurz gefasst
Querschnittstudie in der Zeitschrift JMIR Medical Education, online erschienen am 24.09.2026. Verfasst von Xiaoxing Gao und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Pulmonary and Critical Care Medicine, Peking Union Medical College Hospital, 1 Shuaifuyuan, Dongcheng District, Beijing, 100730, China,…. Hauptthemen laut MEDLINE: Educational Measurement, Students, Medical, Patient Simulation. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 27 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language model (LLM)–powered virtual standardized patients (VSPs) enable scalable clinical skills practice, but the validity of AI-generated scores relative to faculty ratings remains unclear.
Objective
This study aimed to assess agreement between LLM-generated and faculty ratings of history-taking and communication performance and to examine the influence of rater and case heterogeneity.
Methods
In this cross-sectional study, 92 fourth-year medical students completed one of three 15-minute voice-based VSP cases (fever, diarrhea, and cough). Ten blinded faculty raters scored performance (0‐100 points total; 0‐50 points per domain). AI scores were generated by DeepSeek-V3 using a calibrated prompt. Agreement was evaluated using mixed-effects models, intraclass correlation coefficients (ICC [2,1]), Spearman correlations, mean absolute error (MAE), Bland-Altman analysis, and variance partition coefficients (VPC).
Results
Median total scores were similar for AI and faculty (median 93.0, IQR 89.0-95.0 vs median 94.0, IQR 91.0-95.0). Rater variability accounted for 37% of residual variance in faculty total scores (VPC=0.37). AI total scores were positively associated with faculty total scores (β=0.37, 95% CI 0.26‐0.48; P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Pulmonary and Critical Care Medicine, Peking Union Medical College Hospital, 1 Shuaifuyuan, Dongcheng District, Beijing, 100730, China, 86 10 6915 5000.
- Department of General Internal Medicine, Peking Union Medical College Hospital, Beijing, China.
- Department of Nephrology, Peking Union Medical College Hospital, Beijing, China.
- Department of Rheumatology, Peking Union Medical College Hospital, Beijing, China.
- Department of Infectious Disease, Peking Union Medical College Hospital, Beijing, China.
- Department of Haematology, Peking Union Medical College Hospital, Beijing, China.
ORCID-Profile: Xiaoxing Gao, Xiaoming Huang, Rongrong Hu, Li Zhang, Huiting Liu, Bingqing Zhang, Chong Wei, Wei Qiu, Mengyu Zhang, Xuefeng Sun
Zitieren
APA
Gao, X., Huang, X., Hu, R., Zhang, L., Liu, H., Zhang, B., Wei, C., Qiu, W., Zhang, M., & Sun, X. (2026). Evaluating Large Language Model–Based Automated Scoring in a Voice-Based Virtual Standardized Patient Platform for Medical Students: Cross-Sectional Agreement Study. JMIR Medical Education, 12, e95578-e95578. https://doi.org/10.2196/95578
Vancouver
Gao X, Huang X, Hu R, Zhang L, Liu H, Zhang B, et al. Evaluating Large Language Model–Based Automated Scoring in a Voice-Based Virtual Standardized Patient Platform for Medical Students: Cross-Sectional Agreement Study. JMIR Medical Education. 2026;12:e95578-e95578. doi:10.2196/95578
BibTeX
@article{Gao2026,
author = {Gao, Xiaoxing and Huang, Xiaoming and Hu, Rongrong and Zhang, Li and Liu, Huiting and Zhang, Bingqing and Wei, Chong and Qiu, Wei and Zhang, Mengyu and Sun, Xuefeng},
title = {Evaluating Large Language Model–Based Automated Scoring in a Voice-Based Virtual Standardized Patient Platform for Medical Students: Cross-Sectional Agreement Study},
journal = {JMIR Medical Education},
year = {2026},
volume = {12},
pages = {e95578--e95578},
doi = {10.2196/95578}
}
EUVIMED