Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions
ACM Transactions on Computer-Human Interaction · 29.10.2024
DOI: 10.1145/3702320 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Computer-Human Interaction, online erschienen am 29.10.2024. Verfasst von Helena Vasconcelos und 4 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 70 Literaturangaben.
Abstract (Originalsprache)
Large-scale generative models have enabled the development of AI-powered code completion tools to assist programmers in writing code. Like all AI-powered tools, these code completion tools are not always accurate and can introduce bugs or even security vulnerabilities into code if not properly detected and corrected by a human programmer. One technique that has been proposed and implemented to help programmers locate potential errors is to highlight uncertain tokens. However, little is known about the effectiveness of this technique. Through a mixed-methods study with 30 programmers, we compare three conditions: providing the AI system's code completion alone, highlighting tokens with the lowest likelihood of being generated by the underlying generative model, and highlighting tokens with the highest predicted likelihood of being edited by a programmer. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits, and is subjectively preferred by study participants. In contrast, highlighting tokens according to their probability of being generated does not provide any benefit over the baseline with no highlighting. We further explore the design space of how to convey uncertainty in AI-powered code completion tools and find that programmers prefer highlights that are granular, informative, interpretable, and not overwhelming. This work contributes to building an understanding of what uncertainty means for generative models and how to convey it effectively.
Zitieren
APA
Vasconcelos, H., Bansal, G., Fourney, A., Liao, Q. V., & Vaughan, J. W. (2024). Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions. ACM Transactions on Computer-Human Interaction. https://doi.org/10.1145/3702320
Vancouver
Vasconcelos H, Bansal G, Fourney A, Liao QV, Vaughan JW. Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions. ACM Transactions on Computer-Human Interaction. 2024. doi:10.1145/3702320
BibTeX
@article{Vasconcelos2024,
author = {Vasconcelos, Helena and Bansal, Gagan and Fourney, Adam and Liao, Q. Vera and Vaughan, Jennifer Wortman},
title = {Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions},
journal = {ACM Transactions on Computer-Human Interaction},
year = {2024},
doi = {10.1145/3702320}
}
EUVIMED