The Validity of Generative Artificial Intelligence in Evaluating Medical Students in Objective Structured Clinical Examination: Experimental Study
JMIR Formative Research · 2025
DOI: 10.2196/79465 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Formative Research, 2025. Verfasst von Masashi Yokose und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, Tochigi, Japan. Hauptthemen laut MEDLINE: Artificial Intelligence, Students, Medical, Educational Measurement, Clinical Competence. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 18 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
The Objective Structured Clinical Examination (OSCE) has been widely used to evaluate students in medical education. However, it is resource-intensive, presenting challenges in implementation. We hypothesized that generative artificial intelligence (AI) such as ChatGPT-4 could serve as a complementary assessor and alleviate the burden of physicians in evaluating OSCE.
Objective
By comparing the evaluation scores between generative AI and physicians, this study aims to evaluate the validity of generative AI as a complementary assessor for OSCE.
Methods
This experimental study was conducted at a medical university in Japan. We recruited 11 fifth-year medical students during the general internal medicine clerkship from April 2023 to December 2023. Participants conducted a mock medical interview with a patient experiencing abdominal pain and wrote patient notes. Four physicians independently evaluated the participants by reviewing medical interview videos and patient notes, while ChatGPT-4 was provided with interview transcripts and notes. Evaluations were conducted using the 6-domain rubric (patient care and communication, history taking, physical examination, patient notes, clinical reasoning, and management). Each domain was scored using a 6-point Likert scale, ranging from 1 (very poor) to 6 (excellent). Median scores were compared using the Wilcoxon signed-rank test, and the agreement between ChatGPT-4 and physicians was assessed using intraclass correlation coefficients (ICCs). All P values
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, Tochigi, Japan.
- Department of Internal Medicine, Yamagata Prefectural Kahoku Hospital, Yamagata, Japan.
ORCID-Profile: Masashi Yokose, Takanobu Hirosawa, Tetsu Sakamoto, Ren Kawamura, Yudai Suzuki, Yukinori Harada, Taro Shimizu
Zitieren
APA
Yokose, M., Hirosawa, T., Sakamoto, T., Kawamura, R., Suzuki, Y., Harada, Y., & Shimizu, T. (2025). The Validity of Generative Artificial Intelligence in Evaluating Medical Students in Objective Structured Clinical Examination: Experimental Study. JMIR Formative Research. https://doi.org/10.2196/79465
Vancouver
Yokose M, Hirosawa T, Sakamoto T, Kawamura R, Suzuki Y, Harada Y, et al. The Validity of Generative Artificial Intelligence in Evaluating Medical Students in Objective Structured Clinical Examination: Experimental Study. JMIR Formative Research. 2025. doi:10.2196/79465
BibTeX
@article{Yokose2025,
author = {Yokose, Masashi and Hirosawa, Takanobu and Sakamoto, Tetsu and Kawamura, Ren and Suzuki, Yudai and Harada, Yukinori and Shimizu, Taro},
title = {The Validity of Generative Artificial Intelligence in Evaluating Medical Students in Objective Structured Clinical Examination: Experimental Study},
journal = {JMIR Formative Research},
year = {2025},
doi = {10.2196/79465}
}
EUVIMED