ClassEval-T: Evaluating Large Language Models in Class-Level Code Translation
Proceedings of the ACM on Software Engineering · 22.06.2025 · 2 (ISSTA) · 1421-1444
DOI: 10.1145/3728940 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 22.06.2025. Verfasst von Pengyu Xue und 11 weiteren Autor:innen. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 83 Literaturangaben. Gefördert von: National Natural Science Foundation of China, the Natural Science Foundation of Shandong Province, the Taishan Scholars Program.
Abstract (Originalsprache)
In recent years, Large Language Models (LLMs) have dramatically advanced the performance of automated code translation, making their computational accuracy score reach up to over 80% on many previous benchmarks. However, most code samples in these benchmarks are short, standalone, statement/method-level, and algorithmic, which is not aligned with practical coding tasks. Therefore, it is still unknown the actual capability of LLMs in translating code samples written for daily development. To achieve this, we construct a class-level code translation benchmark, ClassEval-T, and make the first attempt to extensively assess recent LLMs’ performance on class-level code translation. ClassEval-T is extended from ClassEval, a well-known class-level Python code generation benchmark consisting of multiple practical coding topics, such as database operation and game design, and diverse contextual dependencies (e.g., fields, methods, and libraries). It cost us 360 person-hours to accomplish the manual migration to Java and C++ with complete code samples and associated test suites. Subsequently, we design three translation strategies (i.e., holistic, min-dependency, and standalone) for class-level code translations and evaluate eight recent LLMs of commercial, general, and code kinds in diverse families and sizes on ClassEval-T. Experimental results demonstrate a remarkable performance drop compared with the most widely studied method-level code translation benchmark, and obvious discrepancies among LLMs appear, showing the effectivenes of ClassEval-T in measuring recent LLMs. Afterwards, we further discuss the usage scenarios for diverse translation strategies and LLMs’ ability to dependency awareness when translating class samples. Finally, 1,243 failure cases made by the best-performing LLM under test are thoroughly analyzed and categorized in this paper for practical guidance and future enlightenment.
Förderung
- National Natural Science Foundation of China
- the Natural Science Foundation of Shandong Province
- the Taishan Scholars Program
- the Research Grants Council of Hong Kong, and the Industry Research Project funds
Zitieren
APA
Xue, P., Wu, L., Yang, Z., Wang, C., Li, X., Zhang, Y., Li, J., Jin, R., Pei, Y., Shen, Z., Lyu, X., & Keung, J. W. (2025). ClassEval-T: Evaluating Large Language Models in Class-Level Code Translation. Proceedings of the ACM on Software Engineering, 2(ISSTA), 1421-1444. https://doi.org/10.1145/3728940
Vancouver
Xue P, Wu L, Yang Z, Wang C, Li X, Zhang Y, et al. ClassEval-T: Evaluating Large Language Models in Class-Level Code Translation. Proceedings of the ACM on Software Engineering. 2025;2(ISSTA):1421-1444. doi:10.1145/3728940
BibTeX
@article{Xue2025,
author = {Xue, Pengyu and Wu, Linhao and Yang, Zhen and Wang, Chengyi and Li, Xiang and Zhang, Yuxiang and Li, Jia and Jin, Ruikai and Pei, Yifei and Shen, Zhaoyan and Lyu, Xiran and Keung, Jacky Wai},
title = {ClassEval-T: Evaluating Large Language Models in Class-Level Code Translation},
journal = {Proceedings of the ACM on Software Engineering},
year = {2025},
volume = {2},
number = {ISSTA},
pages = {1421--1444},
doi = {10.1145/3728940}
}
EUVIMED