EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Can GPT-4 Replicate Empirical Software Engineering Research?

Jenny T. Liang, Carmen Badea, Christian Bird, Robert DeLine, Denae Ford, Nicole Forsgren, Thomas Zimmermann

Proceedings of the ACM on Software Engineering · 12.07.2024 · 1 (FSE) · 1330-1353

DOI: 10.1145/3660767 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 12.07.2024. Verfasst von Jenny T. Liang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 24-mal zitiert (Crossref) und nennt 71 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Empirical software engineering research on production systems has brought forth a better understanding of the software engineering process for practitioners and researchers alike. However, only a small subset of production systems is studied, limiting the impact of this research. While software engineering practitioners could benefit from replicating research on their own data, this poses its own set of challenges, since performing replications requires a deep understanding of research methodologies and subtle nuances in software engineering data. Given that large language models (LLMs), such as GPT-4, show promise in tackling both software engineering- and science-related tasks, these models could help replicate and thus democratize empirical software engineering research. In this paper, we examine GPT-4’s abilities to perform replications of empirical software engineering research on new data. We specifically study their ability to surface assumptions made in empirical software engineering research methodologies, as well as their ability to plan and generate code for analysis pipelines on seven empirical software engineering papers. We perform a user study with 14 participants with software engineering research expertise, who evaluate GPT-4-generated assumptions and analysis plans (i.e., a list of module specifications) from the papers. We find that GPT-4 is able to surface correct assumptions, but struggles to generate ones that apply common knowledge about software engineering data. In a manual analysis of the generated code, we find that the GPT-4-generated code contains correct high-level logic, given a subset of the methodology. However, the code contains many small implementation-level errors, reflecting a lack of software engineering knowledge. Our findings have implications for leveraging LLMs for software engineering research as well as practitioner data scientists in software teams.

Zitieren

APA

Liang, J. T., Badea, C., Bird, C., DeLine, R., Ford, D., Forsgren, N., & Zimmermann, T. (2024). Can GPT-4 Replicate Empirical Software Engineering Research?. Proceedings of the ACM on Software Engineering, 1(FSE), 1330-1353. https://doi.org/10.1145/3660767

Vancouver

Liang JT, Badea C, Bird C, DeLine R, Ford D, Forsgren N, et al. Can GPT-4 Replicate Empirical Software Engineering Research?. Proceedings of the ACM on Software Engineering. 2024;1(FSE):1330-1353. doi:10.1145/3660767

BibTeX
@article{Liang2024,
  author = {Liang, Jenny T. and Badea, Carmen and Bird, Christian and DeLine, Robert and Ford, Denae and Forsgren, Nicole and Zimmermann, Thomas},
  title = {Can GPT-4 Replicate Empirical Software Engineering Research?},
  journal = {Proceedings of the ACM on Software Engineering},
  year = {2024},
  volume = {1},
  number = {FSE},
  pages = {1330--1353},
  doi = {10.1145/3660767}
}
Kostenlos starten →