EUVIMEDScopri EUVIMED
Open access

Practical Guide to Large Language Models for Information Extraction in Behavioral Health Notes: Tutorial

Diya Saha, Juliet B Edgcomb

JMIR Mental Health · 2026-09-24 · 13 · e97498-e97498

DOI: 10.2196/97498 ↗

At a glance

Journal article in JMIR Mental Health, published online on 2026-09-24. Written by Diya Saha and 1 co-authors. Institution: Department of Computer Science, University of California, Irvine, 6210 Donald Bren Hall, Irvine, CA, 92697, United States, 1 9498247427. Major MEDLINE topics: Large Language Models, Electronic Health Records, Natural Language Processing, Information Storage and Retrieval. 28 references. The full text is freely available in PubMed Central.

AbstractSubjectsDati bibliograficiCiteRelated studies

Abstract (original language)

Background

Mental health clinical notes contain decision-critical information often absent from structured electronic health record fields. Large language models (LLMs) can extract clinically relevant signals from narrative text; however, variability in output format, limited reproducibility, and inconsistent evaluation remain barriers to clinical deployment. Despite rapid advances in LLM-based information extraction, clear and reproducible guidance for interdisciplinary clinical teams is limited.

Objective

This tutorial aims to present a structured workflow for zero-shot information extraction from mental health clinical notes using locally deployed open-source LLMs. It aims to reduce barriers for clinicians and researchers with limited familiarity with natural language processing (NLP) or LLM-based pipelines. Each stage includes key decision points and examples. The workflow is illustrated on two tasks using synthetic notes: (1) detection of self-injurious thoughts and behaviors (SITB) in pediatric emergency department (ED) notes and (2) antipsychotic medication nonadherence detection in outpatient notes, using schema-constrained outputs and standardized evaluation.

Methods

We describe a five-stage zero-shot LLM pipeline: (1) infrastructure setup with local deployment via Ollama to prevent protected health information (PHI) transmission; (2) task definition specifying the clinical construct, output format, and evaluation; (3) dataset preparation using synthetic notes; (4) iterative prompt development using a hold-out development set with binary and Likert scale outputs constrained via JSON schemas; and (5) output parsing, normalization, and validation. We generated 300 synthetic notes per task using separate LLMs for generation and evaluation; 200 notes were used for evaluation, and 100 notes (50 positive and 50 negative) were used as a prompt-development set and excluded from final metrics. Evaluation used Large Language Model Meta AI (Llama) 3.2 and Llama 3.3 with deterministic decoding (temperature=0). Performance was assessed using accuracy, precision, recall, and F 1 -score; Likert thresholds were optimized using the Youden index with bootstrapped CIs.

Results

We demonstrated the pipeline’s functionality using 2 example behavioral health detection tasks. Across both examples, the more capable model (Llama 3.3) performed better than the lighter model used earlier in development (Llama 3.2), and we described how the pipeline’s evaluation and error-analysis steps work in practice. These examples also illustrated 2 useful design choices: requiring the model to output in a fixed format reduced errors, and using a graded rating scale, rather than a simple yes/no format, allowed the detection threshold to be adjusted based on clinical risk tolerance. These results are meant to show that the pipeline works as intended, not to serve as a benchmark of real-world accuracy.

Conclusions

A schema-driven, zero-shot LLM workflow can support reproducible extraction of clinically relevant information from narrative notes. Local deployment enables processing without transmitting PHI to external servers. This tutorial provides a transferable methodology for institutional adaptation and validation prior to clinical use. All prompts, code, and datasets are publicly available via Zenodo (European Organization for Nuclear Research [CERN]).

Subjects

MeSH terms (MEDLINE)

Large Language ModelsElectronic Health RecordsNatural Language ProcessingInformation Storage and RetrievalHumans

Author keywords

clinical noteselectronic health recordsinformation extractionlarge language modelsmedication adherencenatural language processingstructured outputssuicide riskzero-shot learning

Authors and institutions

  • Department of Computer Science, University of California, Irvine, 6210 Donald Bren Hall, Irvine, CA, 92697, United States, 1 9498247427.
  • Center for Community Health, UCLA Mental Health Informatics & Data Science (MINDS) Hub, University of California, Los Angeles, Los Angeles, CA, United States.

Cite

APA

Saha, D., & Edgcomb, J. B. (2026). Practical Guide to Large Language Models for Information Extraction in Behavioral Health Notes: Tutorial. JMIR Mental Health, 13, e97498-e97498. https://doi.org/10.2196/97498

Vancouver

Saha D, Edgcomb JB. Practical Guide to Large Language Models for Information Extraction in Behavioral Health Notes: Tutorial. JMIR Mental Health. 2026;13:e97498-e97498. doi:10.2196/97498

BibTeX
@article{Saha2026,
  author = {Saha, Diya and Edgcomb, Juliet B},
  title = {Practical Guide to Large Language Models for Information Extraction in Behavioral Health Notes: Tutorial},
  journal = {JMIR Mental Health},
  year = {2026},
  volume = {13},
  pages = {e97498--e97498},
  doi = {10.2196/97498}
}
Start for free →