EUVIMEDEUVIMED entdecken
Crossref

Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions

Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan

ACM Transactions on Computer-Human Interaction · 29.10.2024

DOI: 10.1145/3702320 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Computer-Human Interaction, online erschienen am 29.10.2024. Verfasst von Helena Vasconcelos und 4 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 70 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large-scale generative models have enabled the development of AI-powered code completion tools to assist programmers in writing code. Like all AI-powered tools, these code completion tools are not always accurate and can introduce bugs or even security vulnerabilities into code if not properly detected and corrected by a human programmer. One technique that has been proposed and implemented to help programmers locate potential errors is to highlight uncertain tokens. However, little is known about the effectiveness of this technique. Through a mixed-methods study with 30 programmers, we compare three conditions: providing the AI system's code completion alone, highlighting tokens with the lowest likelihood of being generated by the underlying generative model, and highlighting tokens with the highest predicted likelihood of being edited by a programmer. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits, and is subjectively preferred by study participants. In contrast, highlighting tokens according to their probability of being generated does not provide any benefit over the baseline with no highlighting. We further explore the design space of how to convey uncertainty in AI-powered code completion tools and find that programmers prefer highlights that are granular, informative, interpretable, and not overwhelming. This work contributes to building an understanding of what uncertainty means for generative models and how to convey it effectively.

Zitieren

APA

Vasconcelos, H., Bansal, G., Fourney, A., Liao, Q. V., & Vaughan, J. W. (2024). Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions. ACM Transactions on Computer-Human Interaction. https://doi.org/10.1145/3702320

Vancouver

Vasconcelos H, Bansal G, Fourney A, Liao QV, Vaughan JW. Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions. ACM Transactions on Computer-Human Interaction. 2024. doi:10.1145/3702320

BibTeX
@article{Vasconcelos2024,
  author = {Vasconcelos, Helena and Bansal, Gagan and Fourney, Adam and Liao, Q. Vera and Vaughan, Jennifer Wortman},
  title = {Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions},
  journal = {ACM Transactions on Computer-Human Interaction},
  year = {2024},
  doi = {10.1145/3702320}
}
Kostenlos starten →