The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates
Proceedings of the ACM on Human-Computer Interaction · 18.10.2025 · 9 (7) · 1-28
DOI: 10.1145/3757667 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Human-Computer Interaction, online erschienen am 16.10.2025. Verfasst von Giuseppe Russo und 4 weiteren Autor:innen. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 130 Literaturangaben. Gefördert von: Swiss National Science Foundation.
Abstract (Originalsprache)
Journals and conferences worry that peer reviews assisted by artificial intelligence (AI), in particular, large language models (LLMs), may negatively influence the validity and fairness of the peer-review system, a cornerstone of modern science. In this work, we address this concern with a study of the prevalence and impact of AI-assisted peer reviews in the context of the 2024 International Conference on Learning Representations (ICLR), a large and prestigious machine-learning conference. Our contributions are threefold. Firstly, we obtain a lower bound for the prevalence of AI-assisted reviews at ICLR 2024 using the closed- and open-source LLM detectors, estimating that at least 15.8% of reviews were written with AI assistance. Secondly, we estimate the impact of AI-assisted reviews on submission scores. Considering pairs of reviews with different scores assigned to the same paper, we find that in 53.4% of pairs, the AI-assisted review scores higher than the human review (p = 0.002; relative difference in probability of scoring higher: +14.4% in favor of AI-assisted reviews). Thirdly, we assess the impact of receiving an AI-assisted peer review on submission acceptance. In a matched study, submissions near the acceptance threshold that received an AI-assisted peer review were 4.9 percentage points (p = 0.024) more likely to be accepted than submissions that did not. Overall, we show that AI-assisted reviews are consequential to the peer-review process and offer a discussion on future implications of current trends.
Förderung
- Swiss National Science Foundation
Zitieren
APA
Russo, G., Ribeiro, M. H., Davidson, T. R., Veselovsky, V., & West, R. (2025). The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates. Proceedings of the ACM on Human-Computer Interaction, 9(7), 1-28. https://doi.org/10.1145/3757667
Vancouver
Russo G, Ribeiro MH, Davidson TR, Veselovsky V, West R. The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates. Proceedings of the ACM on Human-Computer Interaction. 2025;9(7):1-28. doi:10.1145/3757667
BibTeX
@article{Russo2025,
author = {Russo, Giuseppe and Ribeiro, Manoel Horta and Davidson, Tim Ruben and Veselovsky, Veniamin and West, Robert},
title = {The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates},
journal = {Proceedings of the ACM on Human-Computer Interaction},
year = {2025},
volume = {9},
number = {7},
pages = {1--28},
doi = {10.1145/3757667}
}
EUVIMED