EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?

Bonan Kou, Shengmai Chen, Zhijie Wang, Lei Ma, Tianyi Zhang

Proceedings of the ACM on Software Engineering · 12.07.2024 · 1 (FSE) · 2261-2284

DOI: 10.1145/3660807 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 12.07.2024. Verfasst von Bonan Kou und 4 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 103 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large Language Models (LLMs) have recently been widely used for code generation. Due to the complexity and opacity of LLMs, little is known about how these models generate code. We made the first attempt to bridge this knowledge gap by investigating whether LLMs attend to the same parts of a task description as human programmers during code generation. An analysis of six LLMs, including GPT-4, on two popular code generation benchmarks revealed a consistent misalignment between LLMs’ and programmers’ attention. We manually analyzed 211 incorrect code snippets and found five attention patterns that can be used to explain many code generation errors. Finally, a user study showed that model attention computed by a perturbation-based method is often favored by human programmers. Our findings highlight the need for human-aligned LLMs for better interpretability and programmer trust.

Zitieren

APA

Kou, B., Chen, S., Wang, Z., Ma, L., & Zhang, T. (2024). Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?. Proceedings of the ACM on Software Engineering, 1(FSE), 2261-2284. https://doi.org/10.1145/3660807

Vancouver

Kou B, Chen S, Wang Z, Ma L, Zhang T. Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?. Proceedings of the ACM on Software Engineering. 2024;1(FSE):2261-2284. doi:10.1145/3660807

BibTeX
@article{Kou2024,
  author = {Kou, Bonan and Chen, Shengmai and Wang, Zhijie and Ma, Lei and Zhang, Tianyi},
  title = {Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?},
  journal = {Proceedings of the ACM on Software Engineering},
  year = {2024},
  volume = {1},
  number = {FSE},
  pages = {2261--2284},
  doi = {10.1145/3660807}
}
Kostenlos starten →