EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis

Nathan Cassee, Robert Feldt

ACM Transactions on Software Engineering and Methodology · 31.10.2026 · 35 (10) · 1-26

DOI: 10.1145/3799711 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 16.09.2026. Verfasst von Nathan Cassee und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 73 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Science Council, “Automated Boundary Testing for AI/ML models” and from WASP for the project “Bound-Miner.”.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In empirical software engineering (SE) research, researchers have considerable freedom to decide how to process data, what operationalizations to use, and which statistical model to fit. Gelman and Loken refer to this freedom as leading to a “garden of forking paths.” Although this freedom is often seen as an advantage, it also poses a threat to robustness and replicability: variations in analytical decisions, even when justifiable, can lead to divergent conclusions. To better understand this risk, we conducted a so-called multiverse analysis on a published empirical SE paper. The paper we picked is a Mining Software Repositories study, as MSR studies commonly use non-trivial statistical models to analyze post hoc, observational data. In the study, we identified nine pivotal analytical decisions—each with at least one equally defensible alternative—and systematically reran all the 3,072 resulting analysis pipelines on the original dataset. Interestingly, only six of these universes (

Förderung

  • Science Council
  • “Automated Boundary Testing for AI/ML models” and from WASP for the project “Bound-Miner.”

Zitieren

APA

Cassee, N., & Feldt, R. (2026). Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis. ACM Transactions on Software Engineering and Methodology, 35(10), 1-26. https://doi.org/10.1145/3799711

Vancouver

Cassee N, Feldt R. Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis. ACM Transactions on Software Engineering and Methodology. 2026;35(10):1-26. doi:10.1145/3799711

BibTeX
@article{Cassee2026,
  author = {Cassee, Nathan and Feldt, Robert},
  title = {Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis},
  journal = {ACM Transactions on Software Engineering and Methodology},
  year = {2026},
  volume = {35},
  number = {10},
  pages = {1--26},
  doi = {10.1145/3799711}
}
Kostenlos starten →