Teaching Clinical Reasoning in Health Care Professions Learners Using AI-Generated Script Concordance Tests: Mixed Methods Formative Evaluation
JMIR Formative Research · 20.11.2025 · 9 · e76618-e76618
DOI: 10.2196/76618 ↗Kurz gefasst
Mixed-Methods-Studie in der Zeitschrift JMIR Formative Research, online erschienen am 20.11.2025. Verfasst von Alexandre Hudon und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Psychiatry and Addictology, Faculty of Medicine, Université de Montréal, Pavillon Roger-Gaudry, 2900 Bd Édouard-Montpetit Local L-315, …. Hauptthemen laut MEDLINE: Students, Medical, Educational Measurement, Clinical Reasoning, Artificial Intelligence. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
The integration of artificial intelligence (AI) in medical education is evolving, offering new tools to enhance teaching and assessment. Among these, script concordance tests (SCTs) are well-suited to evaluate clinical reasoning in contexts of uncertainty. Traditionally, SCTs require expert panels for scoring and feedback, which can be resource-intensive. Recent advances in generative AI, particularly large language models (LLMs), suggest the possibility of replacing human experts with simulated ones, though this potential remains underexplored.
Objective
This study aimed to evaluate whether LLMs can effectively simulate expert judgment in SCTs by using generative AI to author, score, and provide feedback for SCTs in cardiology and pneumology. A secondary objective was to assess students’ perceptions of the test’s difficulty and the pedagogical value of AI-generated feedback.
Methods
A cross-sectional, mixed methods study was conducted with 25 second-year medical students who completed a 32-item SCT authored by ChatGPT-4o (OpenAI). Six LLMs (3 trained on the course material and 3 untrained) served as simulated experts to generate scoring keys and feedback. Students answered SCT questions, rated perceived difficulty, and selected the most helpful feedback explanation for each item. Quantitative analysis included scoring, difficulty ratings, and correlations between student and AI responses. Qualitative comments were thematically analyzed.
Results
The average student score was 22.8 out of 32 (SD 1.6), with scores ranging from 19.75 to 26.75. Trained AI systems showed significantly higher concordance with student responses (ρ=0.64) than untrained models (ρ=0.41). AI-generated feedback was rated as most helpful in 62.5% of cases, especially when provided by trained models. The SCT demonstrated good internal consistency (Cronbach α=0.76), and students reported moderate perceived difficulty (mean 3.7, SD 1.1). Qualitative feedback highlighted appreciation for SCTs as reflective tools, while recommending clearer guidance on Likert-scale use and more contextual detail in vignettes.
Conclusions
This is among the first studies to demonstrate that trained generative AI models can reliably simulate expert clinical reasoning within a script-concordance framework. The findings suggest that AI can both streamline SCT design and offer educationally valuable feedback without compromising authenticity. Future studies should explore longitudinal effects on learning and assess how hybrid models (human and AI) can optimize reasoning instruction in medical education.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Psychiatry and Addictology, Faculty of Medicine, Université de Montréal, Pavillon Roger-Gaudry, 2900 Bd Édouard-Montpetit Local L-315, Montréal, QC, H3T 1J4, Canada, 1 514 343 6111.
- Department of Psychiatry, Institut universitaire en santé mentale de Montréal, Montreal, QC, Canada.
- Centre de recherche de l'Institut universitaire en santé mentale de Montréal, Montreal, QC, Canada.
- Department of Psychiatry, Institut national de psychiatrie légale Philippe-Pinel, Montreal, QC, Canada.
- Groupe Interdisciplinaire de recherche sur la cognition et le raisonnement professionnel (GIRCoPRo), Université de Montréal, Montreal, QC, Canada.
- Department of Pediatrics, Faculty of Medicine, Université de Montréal, Montreal, QC, Canada.
ORCID-Profile: Alexandre Hudon, Véronique Phan, Bernard Charlin, René Wittmer
Zitieren
APA
Hudon, A., Phan, V., Charlin, B., & Wittmer, R. (2025). Teaching Clinical Reasoning in Health Care Professions Learners Using AI-Generated Script Concordance Tests: Mixed Methods Formative Evaluation. JMIR Formative Research, 9, e76618-e76618. https://doi.org/10.2196/76618
Vancouver
Hudon A, Phan V, Charlin B, Wittmer R. Teaching Clinical Reasoning in Health Care Professions Learners Using AI-Generated Script Concordance Tests: Mixed Methods Formative Evaluation. JMIR Formative Research. 2025;9:e76618-e76618. doi:10.2196/76618
BibTeX
@article{Hudon2025,
author = {Hudon, Alexandre and Phan, Véronique and Charlin, Bernard and Wittmer, René},
title = {Teaching Clinical Reasoning in Health Care Professions Learners Using AI-Generated Script Concordance Tests: Mixed Methods Formative Evaluation},
journal = {JMIR Formative Research},
year = {2025},
volume = {9},
pages = {e76618--e76618},
doi = {10.2196/76618}
}
EUVIMED