EUVIMEDEUVIMED entdecken
Crossref

Evaluating AI Models for Autograding Explain in Plain English Questions: Challenges and Considerations

Max Fowler, Chinedu Emeka, Binglin Chen, David H. Smith IV, Matthew West, Craig Zilles

ACM Transactions on Interactive Intelligent Systems · 31.12.2025 · 15 (4) · 1-29

DOI: 10.1145/3774752 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Interactive Intelligent Systems, online erschienen am 10.12.2025. Verfasst von Max Fowler und 5 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 81 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Code-reading ability has traditionally been under-emphasized in assessments as it is difficult to assess at scale. Prior research has shown that code-reading and code-writing are closely related skills; thus being able to assess and train code reading skills may be necessary for student learning. One way to assess code-reading ability is using Explain in Plain English (EiPE) questions, which ask students to describe what a piece of code does with natural language. Previous research deployed a binary (correct/incorrect) autograder using bigram models that performed comparably with human teaching assistants on student responses. With a dataset of 3,064 student responses from 17 EiPE questions, we investigated multiple autograders for EiPE questions. We evaluated methods as simple as logistic regression trained on bigram features, to more complicated Support Vector Machines (SVMs) trained on embeddings from Large Language Models (LLMs) to GPT-4. We found multiple useful autograders, most with accuracies in the \(86\!\!-\!\!88\%\) range, with different advantages. SVMs trained on LLM embeddings had the highest accuracy; few-shot chat completion with GPT-4 required minimal human effort; pipelines with multiple autograders for specific dimensions (what we call 3D autograders) can provide fine-grained feedback; and code generation with GPT-4 to leverage automatic code testing as a grading mechanism in exchange for slightly more lenient grading standards. While piloting these autograders in a non-major introductory Python course, students had largely similar views of all autograders, although they more often found the GPT-based grader and code-generation graders more helpful and liked the code-generation grader the most.

Zitieren

APA

Fowler, M., Emeka, C., Chen, B., IV, D. H. S., West, M., & Zilles, C. (2025). Evaluating AI Models for Autograding Explain in Plain English Questions: Challenges and Considerations. ACM Transactions on Interactive Intelligent Systems, 15(4), 1-29. https://doi.org/10.1145/3774752

Vancouver

Fowler M, Emeka C, Chen B, IV DHS, West M, Zilles C. Evaluating AI Models for Autograding Explain in Plain English Questions: Challenges and Considerations. ACM Transactions on Interactive Intelligent Systems. 2025;15(4):1-29. doi:10.1145/3774752

BibTeX
@article{Fowler2025,
  author = {Fowler, Max and Emeka, Chinedu and Chen, Binglin and IV, David H. Smith and West, Matthew and Zilles, Craig},
  title = {Evaluating AI Models for Autograding Explain in Plain English Questions: Challenges and Considerations},
  journal = {ACM Transactions on Interactive Intelligent Systems},
  year = {2025},
  volume = {15},
  number = {4},
  pages = {1--29},
  doi = {10.1145/3774752}
}
Kostenlos starten →