EUVIMEDObjevte EUVIMED
VergleichsstudieOpen access

Metastasis Extraction From Non–Small Cell Lung Cancer Clinical Notes Using Large Language Model–Based Classification: Retrospective Comparative Study

Sweta Balaji iD, Kiersten Campbell iD, Rou-Zhen Chen iD, Daniel Smith iD, Matthew A Reyna iD, Abeed Sarker iD, Joshua D Wallach iD, Ravi B Parikh iD, Selen Bozkurt iD

JMIR Cancer · 2026-10-16 · 12 · e99573

DOI: 10.2196/99573 ↗

At a glance

Vergleichsstudie in JMIR Cancer, published online on 2026-10-16. Written by Sweta Balaji and 8 co-authors. Institution: Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA, United States. Major MEDLINE topics: Carcinoma, Non-Small-Cell Lung, Lung Neoplasms. Cited 1 times so far (Crossref); 47 references. The full text is freely available in PubMed Central. Funded by: NLM NIH HHS.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Identification of metastasis status in non–small cell lung cancer (NSCLC) is a critical part of understanding disease prognosis, treatment courses, trial eligibility, and population-level cancer surveillance. However, metastasis status is inconsistently recorded in structured cancer registry fields, since manual abstraction of clinical notes is often an error-prone process. This challenge highlights an opportunity for leveraging large language models (LLMs) to conduct high-scale metastasis extraction from real-world clinical documentation.

Objective

We conducted a retrospective, multicohort comparative evaluation of 3 distinct LLMs for 2 independent classification tasks: overall metastasis presence at any site and brain/central nervous system (CNS) metastasis presence. We evaluated model performance on 2 independent NSCLC cohorts: one with registry labels and another with manual annotations. We further explored whether our methods could analyze clinical documentation and recover missing information in structured registry fields.

Methods

Patient cohorts were derived from the Winship Cancer Institute. Cohort 1 (n=579 patients; 24,887 notes across 69 note types; 2023-2025) used registry-linked metastasis fields as the reference standard. Cohort 2 (n=22 patients; 644 radiology notes; 2010-2021) was drawn from 2 completed randomized trials and used dual-annotated labels (Cohen κ: 0.93 overall metastasis; 0.88 CNS metastasis) as the reference standard. We fine-tuned the GatorTron-base encoder model for each independent binary classification task. We evaluated MedGemma-27B-text and Llama 3.1-70B using zero-shot prompting. A separate cohort of 675 patients with missing registry labels was used for an exploratory missingness-recovery analysis, validated against annotations of a random subsample. Our best-performing model was used to conduct further site-specific metastasis classification, based on available registry data.

Results

More than half (675/1260, 54%) of initially identified cohort 1 patients had missing or unknown registry metastasis labels. MedGemma demonstrated the best performance in overall metastasis classification (cohort 1: F1-score=0.80; cohort 2 patient-level: F1-score=1.0; cohort 2 note-level: F1-score=0.93). For brain/CNS metastasis, Llama3 performed best in both cohorts (cohort 1: F1-score=0.79; cohort 2 patient-level: F1-score=0.93; cohort 2 note-level: F1-score=0.86). The fine-tuned GatorTron model showed strong performance for classification of overall metastasis in cohort 1 (F1-score=0.72). Error analysis indicated that several model errors reflected incomplete registry labels, ambiguous clinical language, or missing documentation. In the exploratory recovery analysis, model predictions agreed with manual annotations at an accuracy of 0.90 and F1-score of 0.89. Additional site-specific metastasis classification by MedGemma demonstrated high performance for liver metastasis (F1-score=0.82) and bone metastasis (F1-score=0.79).

Conclusions

All models demonstrated relatively high performance. The zero-shot generative models were more robust to nuanced documentation and context-dependent brain/CNS metastasis extraction. The fine-tuned encoder model demonstrated strong classification performance but may have been limited by potential inaccuracies in the registry reference standards during model training. This study further demonstrated the potential of LLMs in recovering plausible structured labels from narrative text, complementing cancer registries for metastasis ascertainment.

Subjects

MeSH terms (MEDLINE)

Carcinoma, Non-Small-Cell LungLung NeoplasmsHumansLarge Language ModelsRetrospective StudiesMaleFemaleNeoplasm MetastasisRegistries

Author keywords

electronic health recordslarge language modelsmetastasisnatural language processingnon-small-cell lung cancer

Authors and institutions

  • Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA, United States.
  • Department of Epidemiology, Rollins School of Public Health, Emory University, Atlanta, GA, United States.
  • Department of IT and Research Informatics, Winship Cancer Institute, Atlanta, GA, United States.
  • Department of Hematology and Medical Oncology, School of Medicine, Emory University, Atlanta, GA, United States.

ORCID profiles: Sweta Balaji, Kiersten Campbell, Rou-Zhen Chen, Daniel Smith, Matthew A Reyna, Abeed Sarker, Joshua D Wallach, Ravi B Parikh, Selen Bozkurt

Funding

  • NLM NIH HHS

Cite

APA

Balaji, S., Campbell, K., Chen, R. Z., Smith, D., Reyna, M. A., Sarker, A., Wallach, J. D., Parikh, R. B., & Bozkurt, S. (2026). Metastasis Extraction From Non–Small Cell Lung Cancer Clinical Notes Using Large Language Model–Based Classification: Retrospective Comparative Study. JMIR Cancer, 12, e99573. https://doi.org/10.2196/99573

Vancouver

Balaji S, Campbell K, Chen RZ, Smith D, Reyna MA, Sarker A, et al. Metastasis Extraction From Non–Small Cell Lung Cancer Clinical Notes Using Large Language Model–Based Classification: Retrospective Comparative Study. JMIR Cancer. 2026;12:e99573. doi:10.2196/99573

BibTeX
@article{Balaji2026,
  author = {Balaji, Sweta and Campbell, Kiersten and Chen, Rou-Zhen and Smith, Daniel and Reyna, Matthew A and Sarker, Abeed and Wallach, Joshua D and Parikh, Ravi B and Bozkurt, Selen},
  title = {Metastasis Extraction From Non–Small Cell Lung Cancer Clinical Notes Using Large Language Model–Based Classification: Retrospective Comparative Study},
  journal = {JMIR Cancer},
  year = {2026},
  volume = {12},
  pages = {e99573},
  doi = {10.2196/99573}
}
Start for free →