EUVIMEDDescoperă EUVIMED
CrossrefCard informativ

Safety-Oriented Evaluation of Large Language Models in Health Care: Guideline-Informed Systematic Review

Hikaru Matsuoka, Takayuki Takahashi, Takayuki Semitsu, Keiko Matsuoka, Ryoma Hayashi

Online Journal of Public Health Informatics · 2026-08-20 · 18 · e97240-e97240

DOI: 10.2196/97240 ↗
RezumatDate bibliografice

Rezumat

Abstract Background Large language models (LLMs) are rapidly emerging in health care, offering opportunities in decision support, education, and research, but raising critical concerns about safety, reliability, and ethics. Although several guidelines for trustworthy AI exist in business and technology, few systematic reviews have applied them to medical contexts. Objective This study aimed to conduct a systematic review of LLM research in health care, applying the AI Guidelines for Business as a framework across 11 domains, including safety, reliability, ethics, transparency, fairness, inclusiveness, privacy, security, robustness, data quality, and verifiability. Methods Following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 guidelines (retrospectively registered on the Open Science Framework; DOI 10.17605/OSF.IO/P4KSB), the PubMed, Scopus, Web of Science, arXiv, and IEEE Xplore databases were searched on January 15, 2025. Records were screened in 2 stages by 3 reviewers (with records retained only upon unanimous agreement). A total of 247 studies were included, of which 211 (85.4%) contributed quantitative values. Eligible studies were classified across 11 trustworthy AI domains. Heterogeneous metrics were summarized within metric families; when multiple models were evaluated, the mean across models was used as the primary estimate, with best, median, and primary-model sensitivity analyses. The LLM-assisted categorization (GPT-5 mini) was validated by using an automated internal consistency check, and 95% CIs were estimated by using cluster bootstrap on study-level values. Results Of the 25,156 records, 247 (1.0%) studies were included, and of these, 211 (85.4%) contributed quantitative values. Evaluation concentrated on accuracy (143/247, 57.9%) and fairness and inclusiveness (93/247, 37.7%), followed by data quality (47/247, 19.0%) and prevention of misinformation (44/247, 17.8%). Normalized performance was moderate to high (accuracy mean 0.73, 95% CI 0.7-0.76; data quality: 0.64; prevention of misinformation: 0.82). Selecting the best-performing model inflated domain means by up to 0.05. Privacy protection (2/247, 0.8%) and security assurance (0/247, 0.0%) were almost entirely absent. Domain assignments were recoverable from objective metric types in 98.9% of values (Cohen κ=0.985). Conclusions Current evaluations emphasize accuracy while underreporting privacy, security, robustness, explainability, and verifiability. This finding reflects gaps in reporting rather than demonstrated poor performance, underscoring the need for comprehensive, guideline-based, multidomain evaluation before deployment in high-stakes clinical settings.

Cunoștințe potrivite muncii dumneavoastră.

De la începutul gratuit și cercetarea bibliografică personală până la Research Suite complet.

Vezi tarifele →

Sunt incluse toate textele integrale ale editorilor?

Nu. Referințele bibliografice, rezumatele și textele integrale sunt conținuturi diferite. Disponibilitatea unui text integral depinde de sursă, de Open Access și de licențele pe care le aveți deja. Un abonament EUVIMED nu înlocuiește drepturile suplimentare ale editorilor.

Vezi tarifele →