EUVIMEDEUVIMED entdecken
CrossrefInfokarte

Evaluating large language models for automated REDCap support ticket triage and response

Andrew Carroll, Dawn Brown, Domenick Silvio, James Maszatics, David A Hanauer

JAMIA Open · 2026-05-05 · 9 (3)

DOI: 10.1093/jamiaopen/ooag073 ↗
AbstractBibliografische Angaben

Abstract

Abstract Objectives To evaluate the utility of large language models (LLMs) without and with retrieval-augmented generation (RAG) resources for automating Research Electronic Data Capture (REDCap) support at a large academic medical center, focusing on ticket complexity, categorization, and response drafting. Materials and Methods We randomly selected 90 REDCap support tickets from 6316 received in 2023, stratified by estimated complexity (low, moderate, high) using ticket metadata. Four LLM configurations (GPT-4o Default; GPT-4o + Vanderbilt REDCap documentation; GPT-4o + University of Michigan [UM] REDCap documentation; GPT-4o + combined documentation) were evaluated on 3 tasks per ticket: complexity assessment, category assignment, and email response drafting. Human REDCap experts provided gold-standard ratings and assessed email drafts. Results Compared to the gold standard, the GPT-4o Default environment most closely matched human complexity ratings, while specialized RAG environments tended to overestimate complexity. For ticket support categorization, all LLM configurations achieved high agreement with human judgments (Cohen’s kappa ≈0.85), correctly categorizing 81%-89% of tickets. In generating ticket responses, the GPT-4o + REDCap + UM environment provided the most correct answers, particularly for low and moderate complexity tickets, where about two-thirds of responses were substantially or fully correct. Discussion Large language models have potential to automate ticket triage and response drafting for routine REDCap support, with context augmentation improving response accuracy for lower complexity tickets. Conclusion Large language model-based support tools hold promise for reducing human workload for REDCap support tickets. Continued human oversight, iterative prompt engineering, and expanded local documentation are needed for reliable, effective deployment.

Wissen, das zu Ihrer Arbeit passt.

Vom kostenlosen Einstieg über die persönliche Recherche bis zur vollständigen Research Suite.

Tarife ansehen →

Sind alle Verlagsvolltexte enthalten?

Nein. Bibliografische Nachweise, Abstracts und Volltexte sind unterschiedliche Inhalte. Ob ein Volltext verfügbar ist, hängt von der Quelle, Open Access und Ihren vorhandenen Lizenzen ab. Ein EUVIMED-Tarif ersetzt keine zusätzlichen Verlagsrechte.

Tarife ansehen →