EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access

Large Language Models Can Enable Inductive Thematic Analysis of a Social Media Corpus in a Single Prompt: Human Validation Study

Michael S Deiner iD, Vlad Honcharov iD, Jiawei Li iD, Tim K Mackey iD, Travis C Porco iD, Urmimala Sarkar iD

JMIR Infodemiology · 29.08.2024 · 4 · e59641

DOI: 10.2196/59641 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift JMIR Infodemiology, online erschienen am 29.08.2024. Verfasst von Michael S Deiner und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Ophthalmology and Francis I Proctor Foundation, University of California San Francisco, San Francisco, CA, United States. Hauptthemen laut MEDLINE: Social Media. Die Arbeit wurde bisher 51-mal zitiert (Crossref) und nennt 48 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIDA NIH HHS, NCI NIH HHS, Agency for Healthcare Research and Quality..

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Manually analyzing public health–related content from social media provides valuable insights into the beliefs, attitudes, and behaviors of individuals, shedding light on trends and patterns that can inform public understanding, policy decisions, targeted interventions, and communication strategies. Unfortunately, the time and effort needed from well-trained human subject matter experts makes extensive manual social media listening unfeasible. Generative large language models (LLMs) can potentially summarize and interpret large amounts of text, but it is unclear to what extent LLMs can glean subtle health-related meanings in large sets of social media posts and reasonably report health-related themes.

Objective

We aimed to assess the feasibility of using LLMs for topic model selection or inductive thematic analysis of large contents of social media posts by attempting to answer the following question: Can LLMs conduct topic model selection and inductive thematic analysis as effectively as humans did in a prior manual study, or at least reasonably, as judged by subject matter experts?

Methods

We asked the same research question and used the same set of social media content for both the LLM selection of relevant topics and the LLM analysis of themes as was conducted manually in a published study about vaccine rhetoric. We used the results from that study as background for this LLM experiment by comparing the results from the prior manual human analyses with the analyses from 3 LLMs: GPT4-32K, Claude-instant-100K, and Claude-2-100K. We also assessed if multiple LLMs had equivalent ability and assessed the consistency of repeated analysis from each LLM.

Results

The LLMs generally gave high rankings to the topics chosen previously by humans as most relevant. We reject a null hypothesis (P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Social MediaHumansNatural Language Processing

Stichwörter der Autor:innen

COVID-19ClaudeGPTTwitterX formerly known as Twittergenerative large language modelgenerative pretrained transformerinductive content analysisinfodemiologysocial mediavaccine hesitancy

Autor:innen und Institutionen

  • Department of Ophthalmology and Francis I Proctor Foundation, University of California San Francisco, San Francisco, CA, United States.
  • Center for Vulnerable Populations, Zuckerberg San Francisco General Hospital, Department of Medicine, University of California San Francisco, San Francisco, CA, United States.
  • Division of General Internal Medicine, Zuckerberg San Francisco General Hospital, Department of Medicine, University of California San Francisco, San Francisco, CA, United States.
  • S-3 Research, LLC, San Diego, CA, United States.
  • Global Health Program, Department of Anthropology, University of California San Diego, La Jolla, CA, United States.
  • Departments of Ophthalmology, Epidemiology and Biostatistics, Global Health Sciences, and Francis I Proctor Foundation, University of California San Francisco, San Francisco, CA, United States.

ORCID-Profile: Michael S Deiner, Vlad Honcharov, Jiawei Li, Tim K Mackey, Travis C Porco, Urmimala Sarkar

Förderung

  • NIDA NIH HHS
  • NCI NIH HHS
  • Agency for Healthcare Research and Quality.

Zitieren

APA

Deiner, M. S., Honcharov, V., Li, J., Mackey, T. K., Porco, T. C., & Sarkar, U. (2024). Large Language Models Can Enable Inductive Thematic Analysis of a Social Media Corpus in a Single Prompt: Human Validation Study. JMIR Infodemiology, 4, e59641. https://doi.org/10.2196/59641

Vancouver

Deiner MS, Honcharov V, Li J, Mackey TK, Porco TC, Sarkar U. Large Language Models Can Enable Inductive Thematic Analysis of a Social Media Corpus in a Single Prompt: Human Validation Study. JMIR Infodemiology. 2024;4:e59641. doi:10.2196/59641

BibTeX
@article{Deiner2024,
  author = {Deiner, Michael S and Honcharov, Vlad and Li, Jiawei and Mackey, Tim K and Porco, Travis C and Sarkar, Urmimala},
  title = {Large Language Models Can Enable Inductive Thematic Analysis of a Social Media Corpus in a Single Prompt: Human Validation Study},
  journal = {JMIR Infodemiology},
  year = {2024},
  volume = {4},
  pages = {e59641},
  doi = {10.2196/59641}
}
Kostenlos starten →