Improving Retrieval Augmented Generation for Health Care by Fine-Tuning Clinical Embedding Models: Development and Evaluation Study
Journal of Medical Internet Research · 2026
DOI: 10.2196/82997 ↗Kurz gefasst
Evaluationsstudie in der Zeitschrift Journal of Medical Internet Research, 2026. Verfasst von Kamyar Arzideh und 13 weiteren Autor:innen. Beteiligte Einrichtung: Institute for Artificial Intelligence in Medicine,, University Hospital Essen, Essen, Germany. Hauptthemen laut MEDLINE: Information Storage and Retrieval, Electronic Health Records, Delivery of Health Care. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Embedding models are critical components of Retrieval Augmented Generation (RAG) systems for retrieving and searching unstructured medical data. However, existing models are predominantly trained on publicly available English datasets, limiting their effectiveness in non-English health care settings. More importantly, these models lack training on real-world clinical documents, leading to inaccurate context retrieval when integrated into RAG systems for health care applications. This gap is particularly pronounced in specialized medical documentation containing domain-specific terminology, abbreviations, and nuanced clinical language.
Objective
This retrospective study aimed to develop and validate embedding models specifically trained on real-world clinical documents from multiple medical specialties to improve medical information retrieval (IR) and RAG system performance in both German and English language contexts.
Methods
We fine-tuned embedding models, so-called sentence transformers, using the multilingual-e5-large architecture as a foundation. Training data consisted of approximately 11 million question-answer pairs synthetically generated from 400,000 diverse clinical documents from a large German tertiary hospital, spanning 163,840 patients and 282,728 clinical cases between 2018 and 2023. The SauerkrautLM-SOLAR-Instruct large language model generated medically relevant questions and corresponding answers for each document. The dataset was additionally pseudonymized and translated into English to aim for broader applicability. Models were evaluated in 2 distinct scenarios: IR using questions with multiple relevant passages, and RAG system performance in both cross-patient and patient-centered contexts.
Results
In the IR evaluation, the fine-tuned miracle model achieved a mAP@100 of 0.27, outperforming the multilingual-e5-large baseline (0.14) and state-of-the-art models such as bge-m3 (0.11). In the RAG evaluation, the model demonstrated robust performance comparable with the baseline in the constrained patient-centered scenario (BERTScore F1 0.781 vs 0.778) and showed moderate improvements in the unconstrained cross-patient setting (BLEURT 0.56 vs 0.53). Notably, the model trained on pseudonymized data achieved comparable retrieval performance (mAP@100 0.25) and the highest scores for patient-centered contextual precision (0.93). Performance gains were robust in the German dataset, while the translated English model demonstrated promising results as a proof of concept for cross-lingual transfer.
Conclusions
By leveraging a comprehensive real-world dataset spanning multiple medical specialties and using large language models for synthetic question generation, we successfully created and validated domain-specific embedding models. These models can improve medical IR in large-scale search spaces and perform competitively in constrained RAG applications. By publishing the models trained on pseudonymized data, other health care institutions can integrate or adapt these embedding models to their needs. This work establishes a reproducible framework for developing domain-specific clinical embedding models, with the potential to improve data retrieval in medical settings.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Institute for Artificial Intelligence in Medicine,, University Hospital Essen, Essen, Germany.
- Central IT Department, Data Integration Center, University Hospital Essen, Girardetstr. 2, Essen, 45131, Germany, 49 0231-77816.
- Institute for Transfusion Medicine, University Hospital Essen, Essen, Germany.
- Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital Essen, Essen, Germany.
- Center of Sleep and Telemedicine, University Hospital Essen, Ruhrlandklinik, Essen, Germany.
- Department of Hematology and Oncology, Ruhr-University Bochum, Marienhospital University Hospital, Bochum, Germany.
Zitieren
APA
Arzideh, K., Schäfer, H., Idrissi-Yaghir, A., Schmidt, C. S., Eryilmaz, B., Bahn, M., Turki, A. T., Pollok, O. B., Hartmann, E. M., Winnekens, P., Borys, K., Haubold, J., Nensa, F., & Hosch, R. (2026). Improving Retrieval Augmented Generation for Health Care by Fine-Tuning Clinical Embedding Models: Development and Evaluation Study. Journal of Medical Internet Research. https://doi.org/10.2196/82997
Vancouver
Arzideh K, Schäfer H, Idrissi-Yaghir A, Schmidt CS, Eryilmaz B, Bahn M, et al. Improving Retrieval Augmented Generation for Health Care by Fine-Tuning Clinical Embedding Models: Development and Evaluation Study. Journal of Medical Internet Research. 2026. doi:10.2196/82997
BibTeX
@article{Arzideh2026,
author = {Arzideh, Kamyar and Schäfer, Henning and Idrissi-Yaghir, Ahmad and Schmidt, Cynthia Sabrina and Eryilmaz, Bahadir and Bahn, Mikel and Turki, Amin T and Pollok, Olivia Barbara and Hartmann, Eva Maria and Winnekens, Philipp and Borys, Katarzyna and Haubold, Johannes and Nensa, Felix and Hosch, René},
title = {Improving Retrieval Augmented Generation for Health Care by Fine-Tuning Clinical Embedding Models: Development and Evaluation Study},
journal = {Journal of Medical Internet Research},
year = {2026},
doi = {10.2196/82997}
}
EUVIMED