DemoRank: Selecting Effective Demonstrations for Large Language Models in Ranking Task
ACM Transactions on Information Systems · 31.03.2026 · 44 (3) · 1-25
DOI: 10.1145/3788867 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Information Systems, online erschienen am 02.03.2026. Verfasst von Wenhan Liu und 3 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 49 Literaturangaben. Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
Large Language Models (LLMs) have been proven to have strong zero-shot passage ranking capabilities. In-context learning effectively enhances LLM performance by providing few-shot demonstrations, opening avenues for further improving LLM’s ranking ability. However, existing studies usually retrieve the most similar demonstrations to the input, ignoring the demonstration dependencies and diversity, which is insufficient to inspire the LLM for assessing the current query-passage relevance. In this article, we propose a framework named DemoRank, which selects few-shot demonstrations by performing a novel dependency-aware reranking of the retrieved demonstrations. Considering the dependency, combining top-ranked demonstrations yields better results. Nevertheless, generating the training samples for such a dependency-aware demonstration reranker faces two challenges: (1) the traditional demonstration ranked list assumes demonstration independence, which cannot be used to train our reranker, and (2) obtaining the optimal demonstration ranked list from the retrieved set is NP-hard and inefficient. To overcome these challenges, we propose an approach to construct a kind of dependency-aware training samples efficiently and design a list-pairwise training approach for the optimization of the demonstration reranker. We conduct extensive experiments on a series of passage ranking datasets, and the results demonstrate the superior performance of our proposed DemoRank framework under various scenarios. Our code is publicly available at https://github.com/8421bcd/demorank .
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Liu, W., Zhu, Y., Dou, Z., & Zhou, Y. (2026). DemoRank: Selecting Effective Demonstrations for Large Language Models in Ranking Task. ACM Transactions on Information Systems, 44(3), 1-25. https://doi.org/10.1145/3788867
Vancouver
Liu W, Zhu Y, Dou Z, Zhou Y. DemoRank: Selecting Effective Demonstrations for Large Language Models in Ranking Task. ACM Transactions on Information Systems. 2026;44(3):1-25. doi:10.1145/3788867
BibTeX
@article{Liu2026,
author = {Liu, Wenhan and Zhu, Yutao and Dou, Zhicheng and Zhou, Yujia},
title = {DemoRank: Selecting Effective Demonstrations for Large Language Models in Ranking Task},
journal = {ACM Transactions on Information Systems},
year = {2026},
volume = {44},
number = {3},
pages = {1--25},
doi = {10.1145/3788867}
}
EUVIMED