Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study
Journal of Medical Internet Research · 2026
DOI: 10.2196/93328 ↗Kurz gefasst
Evaluationsstudie in der Zeitschrift Journal of Medical Internet Research, 2026. Verfasst von Clara Frydman-Gani und 9 weiteren Autor:innen. Beteiligte Einrichtung: Center for Neurobehavioral Genetics, Semel Institute for Neuroscience and Human Behavior, David Geffen School of Medicine, University of California, …. Hauptthemen laut MEDLINE: Large Language Models, Electronic Health Records, Natural Language Processing, Mental Disorders. Die Arbeit nennt 52 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIMH NIH HHS.
Abstract (Originalsprache)
Background
The accurate detection of clinical symptoms in electronic health records (EHRs) is pivotal for advancing large-scale genetic and longitudinal studies in psychiatry. Free-text clinical notes are an essential source of symptom-level information, particularly in psychiatry; however, the automated extraction of symptoms from clinical text remains challenging.
Objective
We aimed to evaluate the ability of open-source generative large language models (LLMs) to detect psychiatric symptom-level concepts from clinical text in Spanish and to develop a new fine-tuned, open-source LLM that can be shared without risk of exposing patient information.
Methods
We tested 11 open-source generative LLMs for their ability to detect 109 psychiatric concepts from clinical text, using annotated EHR notes from a psychiatric hospital in Colombia. The LLMs were evaluated both “out-of-the-box” and after fine-tuning and compared against a traditional natural language processing (tNLP) method developed from the same data. To develop a fine-tuned LLM that can be shared without risk of exposing patient information, we created a novel synthetic dataset, leveraging real concept annotations while generating fully synthetic clinical text, and fine-tuned a top-performing LLM on these data. Fine-tuned model generalizability was assessed through cross-site evaluation using data from a second independent hospital in Colombia.
Results
We show that while base LLM performance was poor to moderate (zero-shot: 0.2-0.6 macro-F1; few-shot: 0.2-0.74 macro-F1), it improved significantly after fine-tuning (0.75-0.86 macro-F1), with several fine-tuned LLMs outperforming the tNLP method. In total, 100 psychiatric concepts could be reliably detected (F1>0.8) using either a fine-tuned LLM or tNLP. Using the synthetic dataset, we developed Mistral-small-psych, a new LLM capable of detecting psychiatric concepts from Spanish clinical text with performance comparable to models trained on real EHR data (macro-F1=0.79) while remaining fully distributable. In cross-site evaluations, most LLMs generalized well, with small out-of-domain performance losses (macro-F1 decrease of 0.02-0.16).
Conclusions
This work provides the first large-scale, cross-institutional evaluation of state-of-the-art LLMs for psychiatric concept detection in Spanish EHRs, introduces a privacy-preserving pathway for building shareable clinical natural language processing models, and presents a new open model for extracting symptom-level psychiatric information from Spanish clinical text.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Center for Neurobehavioral Genetics, Semel Institute for Neuroscience and Human Behavior, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, United States.
- Research Group in Psychiatry (GIPSI), Department of Psychiatry, School of Medicine, Universidad de Antioquia, Medellin, Antioquia, Colombia.
- Department of Mental Health and Human Behavior, University of Caldas, Manizales, Caldas, Colombia.
- Department of Radiological Sciences, University of California, Los Angeles, Los Angeles, CA, United States.
ORCID-Profile: Clara Frydman-Gani, Alejandro Arias, Maria Perez-Vallejo, John Daniel Londoño Martínez, Johanna Valencia-Echeverry, Mauricio Castano Ramírez, Alex A T Bui, Nelson B Freimer, Carlos Lopez-Jaramillo, Loes M Olde Loohuis
Förderung
- NIMH NIH HHS
Zitieren
APA
Frydman-Gani, C., Arias, A., Perez-Vallejo, M., Londoño Martínez, J. D., Valencia-Echeverry, J., Castano Ramírez, M., Bui, A. A. T., Freimer, N. B., Lopez-Jaramillo, C., & Olde Loohuis, L. M. (2026). Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study. Journal of Medical Internet Research. https://doi.org/10.2196/93328
Vancouver
Frydman-Gani C, Arias A, Perez-Vallejo M, Londoño Martínez JD, Valencia-Echeverry J, Castano Ramírez M, et al. Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study. Journal of Medical Internet Research. 2026. doi:10.2196/93328
BibTeX
@article{FrydmanGani2026,
author = {Frydman-Gani, Clara and Arias, Alejandro and Perez-Vallejo, Maria and Londoño Martínez, John Daniel and Valencia-Echeverry, Johanna and Castano Ramírez, Mauricio and Bui, Alex A T and Freimer, Nelson B and Lopez-Jaramillo, Carlos and Olde Loohuis, Loes M},
title = {Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study},
journal = {Journal of Medical Internet Research},
year = {2026},
doi = {10.2196/93328}
}
EUVIMED