ChatGPT ‐4 in Nursing Research: A Methodological Evaluation of Bias Risk in Randomized Controlled Trials
Journal of Nursing Scholarship · 2025-11 · 57 (6) · 1060-1067
DOI: 10.1111/jnu.70048 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Nursing Scholarship, online erschienen am 16.09.2025. Verfasst von Metin Tuncer und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Nursing Fundamentals, Gümüşhane University, Gümüşhane, Turkey. Hauptthemen laut MEDLINE: Randomized Controlled Trials as Topic, Nursing Research, Artificial Intelligence. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 37 Literaturangaben.
Abstract (Originalsprache)
Background
Conducting bias assessments in systematic reviews is a time‐consuming process that involves subjective judgments. The use of artificial intelligence (AI) technologies to perform these assessments can potentially save time and enhance consistency. Nevertheless, the efficacy of AI technologies in conducting bias assessments remains inadequately explored.
Aim
This study aims to evaluate the efficacy of ChatGPT‐4o in assessing bias using the revised Cochrane RoB2 tool, focusing on randomized controlled trials in nursing.
Methods
ChatGPT‐4o was provided with the RoB2 assessment guide in the form of a PDF document and instructed to perform bias assessments for the 80 open‐access RCTs included in the study. The results of the bias assessments conducted by ChatGPT‐4o for each domain were then compared with those of the meta‐analysis authors using Cohen's weighted kappa analysis.
Results
Weighted Cohen's kappa values showed better agreement in bias in the measurement of the outcome (D4, 0.22) and bias arising from the randomization process (D1, 0.20), while negative values in bias due to missing outcome data (D3, −0.12) and bias in the selection of the reported result (D5, −0.09) indicated poor agreement. The highest accuracy was observed in D5 (0.81), and the lowest in D1 (0.60). F1 scores were highest in bias due to deviations from intended interventions (D2, 0.74) and lowest in D3 (0.00) and D5 (0.00). Specificity was higher in D5 (0.93) and D3 (0.82), while sensitivity and precision were low in these domains.
Conclusions
The agreement between ChatGPT‐4o and the meta‐analysis studies in the same RCT assessments is generally low. This indicates that ChatGPT‐4o requires substantial enhancements before it can be used as a reliable tool for bias risk assessments. Clinical Relevance The AI–based tools have the potential to expedite bias assessment in systematic reviews. However, this study demonstrates that ChatGPT‐4o, in its current form, lacks sufficient consistency, indicating that such tools should be integrated cautiously and used under continuous human oversight, particularly in evidence‐based evaluations that inform clinical decision‐making.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Nursing Fundamentals, Gümüşhane University, Gümüşhane, Turkey.
- Department of Psychiatric Nursing, Dokuz Eylül University Faculty of Nursing, Izmir, Turkey.
ORCID-Profile: Metin Tuncer, Gülsüm Zekiye Tuncer
Zitieren
APA
Tuncer, M., & Tuncer, G. Z. (2025). ChatGPT ‐4 in Nursing Research: A Methodological Evaluation of Bias Risk in Randomized Controlled Trials. Journal of Nursing Scholarship, 57(6), 1060-1067. https://doi.org/10.1111/jnu.70048
Vancouver
Tuncer M, Tuncer GZ. ChatGPT ‐4 in Nursing Research: A Methodological Evaluation of Bias Risk in Randomized Controlled Trials. Journal of Nursing Scholarship. 2025;57(6):1060-1067. doi:10.1111/jnu.70048
BibTeX
@article{Tuncer2025,
author = {Tuncer, Metin and Tuncer, Gülsüm Zekiye},
title = {ChatGPT ‐4 in Nursing Research: A Methodological Evaluation of Bias Risk in Randomized Controlled Trials},
journal = {Journal of Nursing Scholarship},
year = {2025},
volume = {57},
number = {6},
pages = {1060--1067},
doi = {10.1111/jnu.70048}
}
EUVIMED