Retrieval-Augmented Generation for Medical Question Answering on a Heart Failure Dataset: Performance Analysis
JMIR Formative Research · 2026
DOI: 10.2196/84932 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Formative Research, 2026. Verfasst von Shiran Zhang und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Mechanical & Industrial Engineering, Faculty of Applied Science & Engineering, University of Toronto, 27 King's College Circle, Toronto…. Hauptthemen laut MEDLINE: Heart Failure, Information Storage and Retrieval. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Themen: Herzinsuffizienz
Abstract (Originalsprache)
Background
The integration of retrieval-augmented generation (RAG) systems into the domain of medical question-answering (QA) presents a significant opportunity to enhance the effectiveness and accuracy of clinical support systems.
Objective
This study aimed to explore the design choices within the RAG framework and the use of large language model (LLM) classifiers to optimize medical QA systems, enhancing response quality for patient and caregiver queries of varying risk levels.
Methods
In total, we curated a dataset of 109 patient and caregiver questions related to heart failure (HF)—categorized into answerable (direct, fact-based queries), helpful deferral (general guidance or lifestyle advisory queries), and nonanswerable (out-of-scope or high-risk and medical intervention queries) types—along with relevant documents and a target answer for each question from the website The Heart Hub . Applying a system architecture leveraging RAG with a structured query taxonomy and robust classification mechanisms, this paper provided an empirical assessment for medical QA on a HF dataset and introduced a QA system pipeline design, providing a foundation for extended application across various medical fields. Specifically, we evaluated design choices in the initial retrieval stage of RAG and their impact on performance. We assessed final answer quality from the generation stage using popular passage scoring methods for QA, such as Recall-Oriented Understudy for Gisting Evaluation (ROUGE), BERTScore, and Intersection over Union score.
Results
The pipeline first uses an LLM-based classifier, achieving 65% accuracy for answerable and helpful deferral queries and 100% accuracy for identifying nonanswerable queries. In information retrieval, the BioMedical Contrastive Pre-trained Transformers (MedCPT) cross encoder performed best as a dense retrieval method, delivering an average of 93% recall @ 7 through ranked relevance scores to obtain the top documents with recall @ k denoting recall computed over the top-k retrieved items. For further retrieving snippets from such documents, its average performance was 72.5% for sentence-level snippets and 83% for paragraph-level snippets. A second LLM-based classifier, used to refine the generated responses, resulted in an overall reduction in ROUGE-1 recall by 13% and Bidirectional Encoder Representations from Transformers (BERT) precision by 11%. However, Intersection over Union scores, or the overlap between “gold answers” and system answers, increased by 24%, demonstrating enhanced alignment with ground truth responses. This also indicates the system’s improved ability to generate concise and accurate medical responses.
Conclusions
The implementation of a structured RAG framework paired with LLM classifiers for medical QA introduces a promising avenue for enhancing clinical decision support systems. By systematically analyzing the impact of query taxonomy, retrieval configurations, and response strategies, this approach clarifies the relative importance of each component within the medical RAG system using a HF dataset. Our findings provide actionable guidance on optimal design choices for maximizing retrieval and response accuracy; thus, informing the development of robust, scalable medical QA systems.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Mechanical & Industrial Engineering, Faculty of Applied Science & Engineering, University of Toronto, 27 King's College Circle, Toronto, ON, M5S 1A1, Canada, 1 416-978-2011.
- The Edward S. Rogers Sr. Department of Electrical and Computer Engineering, Faculty of Applied Science & Engineering, University of Toronto, Toronto, ON, Canada.
- Centre for Digital Therapeutics, Toronto General Hospital Research Institute, University Health Network, Toronto, ON, Canada.
- Institute of Health Policy, Management, and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada.
- Telfer School of Management, University of Ottawa, Ottawa, ON, Canada.
- School of Public Health Sciences, Faculty of Health, University of Waterloo, Waterloo, ON, Canada.
ORCID-Profile: Shiran Zhang, Evelyn Phan, Pedro Velmovitsky, Quynh Pham, Scott Sanner
Zitieren
APA
Zhang, S., Phan, E., Velmovitsky, P., Pham, Q., & Sanner, S. (2026). Retrieval-Augmented Generation for Medical Question Answering on a Heart Failure Dataset: Performance Analysis. JMIR Formative Research. https://doi.org/10.2196/84932
Vancouver
Zhang S, Phan E, Velmovitsky P, Pham Q, Sanner S. Retrieval-Augmented Generation for Medical Question Answering on a Heart Failure Dataset: Performance Analysis. JMIR Formative Research. 2026. doi:10.2196/84932
BibTeX
@article{Zhang2026,
author = {Zhang, Shiran and Phan, Evelyn and Velmovitsky, Pedro and Pham, Quynh and Sanner, Scott},
title = {Retrieval-Augmented Generation for Medical Question Answering on a Heart Failure Dataset: Performance Analysis},
journal = {JMIR Formative Research},
year = {2026},
doi = {10.2196/84932}
}
EUVIMED