EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Escalation-Aware Agentic RAG for First-Line IT Incident Triage: Learned Abstention, Semantic Grounding Analysis and Human Escalation

Ioannis Mantzakos, Paris A. Mastorocostas, Christos Troussas, Phivos Mylonas

Computers · 04.11.2026 · 15 (11) · 802

DOI: 10.3390/computers15110802 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computers, online erschienen am 04.11.2026. Verfasst von Ioannis Mantzakos und 3 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models can produce remediation suggestions for IT incidents, but the evidence behind their suggestions and the technical correctness of their suggestions are unclear. We evaluate a prototype escalation-aware retrieval-augmented generation (RAG) system for supervised first-line incident triage, using IBM TechQA retrieval, a QLoRA specialized Llama-3-8B-Instruct model, explicit abstention, and Jira ticket escalation. The final reranker and decision thresholds were trained or calibrated on TechQA training questions, but previous choices of retrieval models and contexts used to train the adapter depended indirectly on development questions, so the overall development process is not fully leakage-free. On 205 missing-evidence TEST cases, correct abstention increased from 8.5% (Base) to 42.4% (QLoRA), and hazardous answering decreased from 91.5% to 57.6%. The gate had higher correct abstention (53.7% vs. 42.4%; a significant 11.2 percentage-point advantage, Holm p = 0.013) with no detectable difference in false abstention when operating at their own training-calibrated operating points. At retrospectively matched false abstention, QLoRA had an estimated advantage of +11.2 points over the gate. This different, TEST-threshold-selected contrast was exploratory and not statistically significant. In exploratory NLI analysis, 26.1% of citation-valid QLoRA answers were flagged as containing an unsupported step. NLI–human agreement did not meet the pre-specified acceptance criterion (κ = 0.324 < 0.5), so these estimates cannot be interpreted as validated semantic-grounding rates. In the paired Jira simulation, 64.1% of tickets were closed with a proposed resolution, including 54.9% of naturally unanswerable tickets. In an internal assessment by two research-team annotators, an estimated 14.2% of all incidents were automatically closed with fully correct proposed answers (not independently verified repairs) and potentially risky actions appeared in 24.0% (6/25) of sampled automatically closed missing-evidence tickets. Results support supervised triage and human escalation, not safe autonomous technical remediation.

Zitieren

APA

Mantzakos, I., Mastorocostas, P. A., Troussas, C., & Mylonas, P. (2026). Escalation-Aware Agentic RAG for First-Line IT Incident Triage: Learned Abstention, Semantic Grounding Analysis and Human Escalation. Computers, 15(11), 802. https://doi.org/10.3390/computers15110802

Vancouver

Mantzakos I, Mastorocostas PA, Troussas C, Mylonas P. Escalation-Aware Agentic RAG for First-Line IT Incident Triage: Learned Abstention, Semantic Grounding Analysis and Human Escalation. Computers. 2026;15(11):802. doi:10.3390/computers15110802

BibTeX
@article{Mantzakos2026,
  author = {Mantzakos, Ioannis and Mastorocostas, Paris A. and Troussas, Christos and Mylonas, Phivos},
  title = {Escalation-Aware Agentic RAG for First-Line IT Incident Triage: Learned Abstention, Semantic Grounding Analysis and Human Escalation},
  journal = {Computers},
  year = {2026},
  volume = {15},
  number = {11},
  pages = {802},
  doi = {10.3390/computers15110802}
}
Kostenlos starten →