EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Enhancing Large Language Models for Improved Accuracy and Safety in Medical Question Answering: Comparative Study

Dingqiao Wang iD, Jinguo Ye iD, Jingni Li iD, Jiangbo Liang iD, Qikai Zhang iD, Qiuling Hu iD, Caineng Pan iD, Dongliang Wang iD, Zhong Liu iD, Wen Shi iD, Mengxiang Guo iD, Fei Li iD und 2 weitere

JMIR Medical Education · 02.12.2025 · 11 · e70190

DOI: 10.2196/70190 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift JMIR Medical Education, online erschienen am 02.12.2025. Verfasst von Dingqiao Wang und 13 weiteren Autor:innen. Beteiligte Einrichtung: State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangdong Provincial Key Laboratory of Ophthalmology and …. Hauptthemen laut MEDLINE: Language, Physician-Patient Relations, Education, Medical. Die Arbeit wurde bisher 18-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) offer the potential to improve virtual patient-physician communication and reduce health care professionals’ workload. However, limitations in accuracy, outdated knowledge, and safety issues restrict their effective use in real clinical settings. Addressing these challenges is crucial for making LLMs a reliable health care tool.

Objective

This study aimed to evaluate the efficacy of Med-RISE, an information retrieval and augmentation tool, in comparison with baseline LLMs, focusing on enhancing accuracy and safety in medical question answering across diverse clinical domains.

Methods

This comparative study introduces Med-RISE, an enhanced version of a retrieval-augmented generation framework specifically designed to improve question-answering performance across wide-ranging medical domains and diverse disciplines. Med-RISE consists of 4 key steps: query rewriting, information retrieval (providing local and real-time retrieval), summarization, and execution (a fact and safety filter before output). This study integrated Med-RISE with 4 LLMs (GPT-3.5, GPT-4, Vicuna-13B, and ChatGLM-6B) and assessed their performance on 4 multiple-choice medical question datasets: MedQA (US Medical Licensing Examination), PubMedQA (original and revised versions), MedMCQA, and EYE500. Primary outcome measures included answer accuracy and hallucination rates, with hallucinations categorized into factuality (inaccurate information) or faithfulness (inconsistency with instructions) types. This study was conducted between March 2024 and August 2024.

Results

The integration of Med-RISE with each LLM led to a substantial increase in accuracy, with improvements ranging from 9.8% to 16.3% (mean 13%, SD 2.3%) across the 4 datasets. The enhanced accuracy rates were 16.3%, 12.9%, 13%, and 9.8% for GPT-3.5, GPT-4, Vicuna-13B, and ChatGLM-6B, respectively. In addition, Med-RISE effectively reduced hallucinations, with reductions ranging from 11.8% to 18% (mean 15.1%, SD 2.8%), factuality hallucinations decreasing by 13.5%, and faithfulness hallucinations decreasing by 5.8%. The hallucination rate reductions were 17.7%, 12.8%, 18%, and 11.8% for GPT-3.5, GPT-4, Vicuna-13B, and ChatGLM-6B, respectively.

Conclusions

The Med-RISE framework significantly improves the accuracy and reduces the hallucinations of LLMs in medical question answering across benchmark datasets. By providing local and real-time information retrieval and fact and safety filtering, Med-RISE enhances the reliability and interpretability of LLMs in the medical domain, offering a promising tool for clinical practice and decision support.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

LanguagePhysician-Patient RelationsEducation, MedicalHumansInformation Storage and RetrievalLarge Language Models

Stichwörter der Autor:innen

ChatGPThealth care communicationlarge language modelsmedical question answeringretrieval-augmented generation

Autor:innen und Institutionen

  • State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science, Guangdong Provincial Clinical Research Center for Ocul…
  • Department of Ophthalmology, Eighth Affiliated Hospital of Sun Yat-sen University, Shenzhen, Guangdong, China.
  • Guangzhou Women and Children's Medical Center, Guangzhou, China.

ORCID-Profile: Dingqiao Wang, Jinguo Ye, Jingni Li, Jiangbo Liang, Qikai Zhang, Qiuling Hu, Caineng Pan, Dongliang Wang, Zhong Liu, Wen Shi, Mengxiang Guo, Fei Li, Wei Du, Ying-Feng Zheng

Zitieren

APA

Wang, D., Ye, J., Li, J., Liang, J., Zhang, Q., Hu, Q., Pan, C., Wang, D., Liu, Z., Shi, W., Guo, M., Li, F., Du, W., & Zheng, Y. F. (2025). Enhancing Large Language Models for Improved Accuracy and Safety in Medical Question Answering: Comparative Study. JMIR Medical Education, 11, e70190. https://doi.org/10.2196/70190

Vancouver

Wang D, Ye J, Li J, Liang J, Zhang Q, Hu Q, et al. Enhancing Large Language Models for Improved Accuracy and Safety in Medical Question Answering: Comparative Study. JMIR Medical Education. 2025;11:e70190. doi:10.2196/70190

BibTeX
@article{Wang2025,
  author = {Wang, Dingqiao and Ye, Jinguo and Li, Jingni and Liang, Jiangbo and Zhang, Qikai and Hu, Qiuling and Pan, Caineng and Wang, Dongliang and Liu, Zhong and Shi, Wen and Guo, Mengxiang and Li, Fei and Du, Wei and Zheng, Ying-Feng},
  title = {Enhancing Large Language Models for Improved Accuracy and Safety in Medical Question Answering: Comparative Study},
  journal = {JMIR Medical Education},
  year = {2025},
  volume = {11},
  pages = {e70190},
  doi = {10.2196/70190}
}
Kostenlos starten →