ProjectEvalPlus: An Agentic Software Engineering Benchmark with Automatic Language Extension and User Simulated Evaluation
ACM Transactions on Software Engineering and Methodology · 30.11.2026 · 35 (11) · 1-19
DOI: 10.1145/3817119 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 04.11.2026. Verfasst von Kaiyuan Liu und 5 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Key R&D Program of China, National Science and Technology Major Program, National Natural Science Foundation of China.
Abstract (Originalsprache)
Large Language Models (LLMs) have shown impressive capabilities in Software Engineering (SWE) , yet evaluating their performance on realistic project-level SWE remains challenging. Existing benchmarks focus on isolated snippets, lacking project-level context, language extensibilities, and end-user reality. Thus, we introduce ProjectEvalPlus , a comprehensive user-interaction-simulation benchmark that extends automatic evaluation to multiple programming languages via Automatic Language Extensibility (ALE) , supports multi-agent scenario stages evaluation, and provides detailed feedback on logical and runtime errors. Through unified testing on Python, Java, and JavaScript, we demonstrate that ProjectEvalPlus effectively assesses LLM agents’ capabilities, reveals language-specific biases, and enables iterative cross-language adaptation. Our results highlight both the strengths and limitations of current LLM agents in practical software development, offering actionable insights for future multi-language, project-level software engineering evaluation.
Förderung
- National Key R&D Program of China
- National Science and Technology Major Program
- National Natural Science Foundation of China
- Guangdong S&T Program
- Shenzhen Science and Technology Program
- Major Key Project of PCL
Zitieren
APA
Liu, K., Pan, Y., Du, Y., Zhang, L., He, D., & Xiang, Y. (2026). ProjectEvalPlus: An Agentic Software Engineering Benchmark with Automatic Language Extension and User Simulated Evaluation. ACM Transactions on Software Engineering and Methodology, 35(11), 1-19. https://doi.org/10.1145/3817119
Vancouver
Liu K, Pan Y, Du Y, Zhang L, He D, Xiang Y. ProjectEvalPlus: An Agentic Software Engineering Benchmark with Automatic Language Extension and User Simulated Evaluation. ACM Transactions on Software Engineering and Methodology. 2026;35(11):1-19. doi:10.1145/3817119
BibTeX
@article{Liu2026,
author = {Liu, Kaiyuan and Pan, Youcheng and Du, Yexing and Zhang, Lei and He, Daojing and Xiang, Yang},
title = {ProjectEvalPlus: An Agentic Software Engineering Benchmark with Automatic Language Extension and User Simulated Evaluation},
journal = {ACM Transactions on Software Engineering and Methodology},
year = {2026},
volume = {35},
number = {11},
pages = {1--19},
doi = {10.1145/3817119}
}
EUVIMED