EUVIMEDEUVIMED entdecken
Open Access

The Validity of Generative Artificial Intelligence in Evaluating Medical Students in Objective Structured Clinical Examination: Experimental Study

Masashi Yokose iD, Takanobu Hirosawa iD, Tetsu Sakamoto iD, Ren Kawamura iD, Yudai Suzuki iD, Yukinori Harada iD, Taro Shimizu iD

JMIR Formative Research · 2025

DOI: 10.2196/79465 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Formative Research, 2025. Verfasst von Masashi Yokose und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, Tochigi, Japan. Hauptthemen laut MEDLINE: Artificial Intelligence, Students, Medical, Educational Measurement, Clinical Competence. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 18 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The Objective Structured Clinical Examination (OSCE) has been widely used to evaluate students in medical education. However, it is resource-intensive, presenting challenges in implementation. We hypothesized that generative artificial intelligence (AI) such as ChatGPT-4 could serve as a complementary assessor and alleviate the burden of physicians in evaluating OSCE.

Objective

By comparing the evaluation scores between generative AI and physicians, this study aims to evaluate the validity of generative AI as a complementary assessor for OSCE.

Methods

This experimental study was conducted at a medical university in Japan. We recruited 11 fifth-year medical students during the general internal medicine clerkship from April 2023 to December 2023. Participants conducted a mock medical interview with a patient experiencing abdominal pain and wrote patient notes. Four physicians independently evaluated the participants by reviewing medical interview videos and patient notes, while ChatGPT-4 was provided with interview transcripts and notes. Evaluations were conducted using the 6-domain rubric (patient care and communication, history taking, physical examination, patient notes, clinical reasoning, and management). Each domain was scored using a 6-point Likert scale, ranging from 1 (very poor) to 6 (excellent). Median scores were compared using the Wilcoxon signed-rank test, and the agreement between ChatGPT-4 and physicians was assessed using intraclass correlation coefficients (ICCs). All P values

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceStudents, MedicalEducational MeasurementClinical CompetenceHumansJapanMaleFemaleReproducibility of ResultsGenerative Artificial Intelligence

Stichwörter der Autor:innen

ChatGPT-4OSCEgenerative artificial intelligencemedical educationmedical studentsobjective structured clinical examination

Autor:innen und Institutionen

  • Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, Tochigi, Japan.
  • Department of Internal Medicine, Yamagata Prefectural Kahoku Hospital, Yamagata, Japan.

ORCID-Profile: Masashi Yokose, Takanobu Hirosawa, Tetsu Sakamoto, Ren Kawamura, Yudai Suzuki, Yukinori Harada, Taro Shimizu

Zitieren

APA

Yokose, M., Hirosawa, T., Sakamoto, T., Kawamura, R., Suzuki, Y., Harada, Y., & Shimizu, T. (2025). The Validity of Generative Artificial Intelligence in Evaluating Medical Students in Objective Structured Clinical Examination: Experimental Study. JMIR Formative Research. https://doi.org/10.2196/79465

Vancouver

Yokose M, Hirosawa T, Sakamoto T, Kawamura R, Suzuki Y, Harada Y, et al. The Validity of Generative Artificial Intelligence in Evaluating Medical Students in Objective Structured Clinical Examination: Experimental Study. JMIR Formative Research. 2025. doi:10.2196/79465

BibTeX
@article{Yokose2025,
  author = {Yokose, Masashi and Hirosawa, Takanobu and Sakamoto, Tetsu and Kawamura, Ren and Suzuki, Yudai and Harada, Yukinori and Shimizu, Taro},
  title = {The Validity of Generative Artificial Intelligence in Evaluating Medical Students in Objective Structured Clinical Examination: Experimental Study},
  journal = {JMIR Formative Research},
  year = {2025},
  doi = {10.2196/79465}
}
Kostenlos starten →