EUVIMEDEUVIMED entdecken
Crossref

Generalizable Error Modeling for Human Data Annotation: Evidence From an Industry-Scale Search Data Annotation Program

Heinrich Peters, Alireza Hashemi, James Rae

Journal of Data and Information Quality · 30.09.2024 · 16 (3) · 1-15

DOI: 10.1145/3688394 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Data and Information Quality, online erschienen am 09.10.2024. Verfasst von Heinrich Peters und 2 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 27 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Machine learning (ML) and artificial intelligence (AI) systems rely heavily on human-annotated data for training and evaluation. A major challenge in this context is the occurrence of annotation errors, as their effects can degrade model performance. This paper presents a predictive error model trained to detect potential errors in search relevance annotation tasks for three industry-scale ML applications (music streaming, video streaming, and mobile apps). Drawing on data from an extensive search relevance annotation program, we demonstrate that errors can be predicted with moderate model performance (AUC=0.65-0.75) and that model performance generalizes well across applications (i.e., a global, task-agnostic model performs on par with task-specific models). In contrast to past research, which has often focused on predicting annotation labels from task-specific features, our model is trained to predict errors directly from a combination of task features and behavioral features derived from the annotation process, in order to achieve a high degree of generalizability. We demonstrate the usefulness of the model in the context of auditing, where prioritizing tasks with high predicted error probabilities considerably increases the amount of corrected annotation errors (e.g., 40% efficiency gains for the music streaming application). These results highlight that behavioral error detection models can yield considerable improvements in the efficiency and quality of data annotation processes. Our findings reveal critical insights into effective error management in the data annotation process, thereby contributing to the broader field of human-in-the-loop ML.

Zitieren

APA

Peters, H., Hashemi, A., & Rae, J. (2024). Generalizable Error Modeling for Human Data Annotation: Evidence From an Industry-Scale Search Data Annotation Program. Journal of Data and Information Quality, 16(3), 1-15. https://doi.org/10.1145/3688394

Vancouver

Peters H, Hashemi A, Rae J. Generalizable Error Modeling for Human Data Annotation: Evidence From an Industry-Scale Search Data Annotation Program. Journal of Data and Information Quality. 2024;16(3):1-15. doi:10.1145/3688394

BibTeX
@article{Peters2024,
  author = {Peters, Heinrich and Hashemi, Alireza and Rae, James},
  title = {Generalizable Error Modeling for Human Data Annotation: Evidence From an Industry-Scale Search Data Annotation Program},
  journal = {Journal of Data and Information Quality},
  year = {2024},
  volume = {16},
  number = {3},
  pages = {1--15},
  doi = {10.1145/3688394}
}
Kostenlos starten →