EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

mt RAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems

Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, Marina Danilevsky

Transactions of the Association for Computational Linguistics · 29.07.2025 · 13 · 784-808

DOI: 10.1162/tacl.a.19 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 29.07.2025. Verfasst von Yannis Katsis und 9 weiteren Autor:innen. Die Arbeit wurde bisher 18-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on multi-turn RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation, is an important and often overlooked task with several additional challenges. We present mtRAG, an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. mtRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judge evaluation. Our human and automatic evaluations show that even state-of-the-art LLM RAG systems struggle on mtRAG. We demonstrate the need for strong retrieval and generation systems that can handle later turns, unanswerable questions, non-standalone questions, and multiple domains. mtRAG is available at https://github.com/ibm/mt-rag-benchmark.

Zitieren

APA

Katsis, Y., Rosenthal, S., Fadnis, K., Gunasekara, C., Lee, Y., Popa, L., Shah, V., Zhu, H., Contractor, D., & Danilevsky, M. (2025). mt RAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems. Transactions of the Association for Computational Linguistics, 13, 784-808. https://doi.org/10.1162/tacl.a.19

Vancouver

Katsis Y, Rosenthal S, Fadnis K, Gunasekara C, Lee Y, Popa L, et al. mt RAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems. Transactions of the Association for Computational Linguistics. 2025;13:784-808. doi:10.1162/tacl.a.19

BibTeX
@article{Katsis2025,
  author = {Katsis, Yannis and Rosenthal, Sara and Fadnis, Kshitij and Gunasekara, Chulaka and Lee, Young-Suk and Popa, Lucian and Shah, Vraj and Zhu, Huaiyu and Contractor, Danish and Danilevsky, Marina},
  title = {mt RAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2025},
  volume = {13},
  pages = {784--808},
  doi = {10.1162/tacl.a.19}
}
Kostenlos starten →