EUVIMEDEUVIMED entdecken
CrossrefInfokarte

Cross-Site Stability of Embedding Model Rankings for Known-Item Retrieval From Clinical Notes: Two-Corpus Comparative Evaluation Study

Yngve Mikkelsen

JMIR AI · 2026-11-11 · 5 · e109305

DOI: 10.2196/109305 ↗
AbstractBibliografische Angaben

Abstract

Background Retrieval-augmented generation over clinical text depends on the embedding model chosen—a choice that interacts with documentation genre. Cross-site stability of embedding-model rankings has not been evaluated on real electronic health record (EHR) narratives with genre held constant; public benchmarks are predominantly single-source, and multi-institution evaluations compare models and query formulations within tasks rather than ranking stability across institutions. Objective The aim of the study is to test whether embedding model rankings transfer across 2 institutional EHR corpora, with genre held constant; to examine the contributions of model, site/corpus, and genre and their dependence on scoring; and to test whether the dominant contributor depends on model-panel composition. Methods We compared 13 embedding models on a known-item passage-to-source-document task, each query a deterministic 2-sentence excerpt from its target, in a 2×2 design crossing site/corpus (Beth Israel Deaconess Medical Center via MIMIC-IV-Note [Medical Information Mart for Intensive Care IV Clinical Notes Module] and University of California, San Francisco via ER-Reason) with genre (discharge summary and imaging report). Query sentences were removed from their targets before indexing to prevent verbatim overlap, notes were deduplicated after normalization, models used their recommended pooling, long documents were scored as overlapping chunks by maximum similarity, and cell sizes were matched (N=1235 per cell). Effectiveness was measured by mean reciprocal rank at 10, rank transfer by Kendall τ and selection regret, and variance by factorial decomposition (η2). Transfer, regret, and the contrastive-panel decomposition carry 95% patient-clustered bootstrap intervals. Results Among the 8 contrastive models, cross-site Kendall τ was 0.86 (95% CI 0.64-1.00) for discharge and 0.64 (95% CI 0.36-0.86) for imaging. The model×site interaction was small under every scoring choice (η2=0.005-0.029); the site/corpus main effect was negligible under best-chunk scoring (η2=0.000) but scoring-dependent (0.172 under mean-pooling). Cross-site selection regret was 0 for imaging and up to 0.033 for discharge (~10% of the destination’s best mean reciprocal rank; intervals 0-0.071). Adding 5 nonretrieval masked-language-model comparators moved the largest term to model (η2=0.874): the dominant-factor question is panel-sensitive, while the qualitative transfer conclusion was unchanged. Conclusions For known-item retrieval from clinical notes at 2 US academic medical centers, embedding-model rankings showed positive cross-site concordance, moderate for imaging and strong for discharge summaries—with intervals wide enough that direction, not magnitude, is the finding. Which factor dominates absolute effectiveness depends on the panel evaluated: primary contrastive-panel scoring favors genre, whereas adding nonretrieval comparators makes model choice dominant; so, claims about major determinants are panel-specific. Cross-site model selection incurred zero to modest performance loss. When documentation genre and retrieval configuration are matched, model-selection results from one institutional EHR corpus may be a useful starting point for another, although local confirmation remains warranted.

Wissen, das zu Ihrer Arbeit passt.

Vom kostenlosen Einstieg über die persönliche Recherche bis zur vollständigen Research Suite.

Tarife ansehen →

Sind alle Verlagsvolltexte enthalten?

Nein. Bibliografische Nachweise, Abstracts und Volltexte sind unterschiedliche Inhalte. Ob ein Volltext verfügbar ist, hängt von der Quelle, Open Access und Ihren vorhandenen Lizenzen ab. Ein EUVIMED-Tarif ersetzt keine zusätzlichen Verlagsrechte.

Tarife ansehen →