EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Human and AI Judgment in EFL Speaking Assessment: Psychometric Dependability, Feedback Quantity and Coverage, and Exploratory Teacher Perspectives

Yan Zhang, Jinyan Huang, Jun Yan

Journal of Intelligence · 09.11.2026 · 14 (11) · 296

DOI: 10.3390/jintelligence14110296 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Intelligence, online erschienen am 09.11.2026. Verfasst von Yan Zhang und 2 weiteren Autor:innen. Die Arbeit nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This study examined human-AI judgment in EFL speaking assessment as a measurement problem relevant to research on individual differences: how consistently do expert raters and repeated AI sessions preserve between-person differences in a complex performance? Thirty oral-performance samples from first-year Chinese university English majors were evaluated by four experienced EFL teachers and by ChatGPT-4o across four repeated sessions using the same holistic and accuracy-fluency-complexity framework. Score distributions were summarized descriptively; human-rater intraclass correlations and generalizability-theory G- and D-studies evaluated dependability. Feedback quantity and thematic coverage were summarized descriptively, and post-results interviews with the same four teacher-raters were analyzed as exploratory participant reflections. Teachers produced more dependable single ratings. Holistic phi coefficients were 0.54 for one ChatGPT-4o session and 0.65 for one teacher, increasing to 0.83 and 0.88 with four assessments. Analytic single-assessment coefficients ranged from 0.54 to 0.56 for ChatGPT-4o and 0.61 to 0.65 for teachers. Across holistic and analytic scoring, a phi coefficient of at least 0.80 was reached with four repeated ChatGPT-4o sessions but three independent teachers. ChatGPT-4o’s analytic dimensions showed limited differentiation from its holistic scoring: accuracy matched the holistic score in all 120 AI ratings, while fluency and complexity differed in only three and four ratings, respectively. ChatGPT-4o also produced more rubric-relevant comments descriptively, while the four teacher-raters emphasized human oversight for contextual and interpersonal interpretation. The study contributes to intelligence research at the level of psychometric measurement and human-AI judgment: assessor systems can differentially preserve observed individual differences in complex performance. EFL speaking was not treated as an intelligence test, and unmeasured cognitive mechanisms were not inferred from the score patterns.

Zitieren

APA

Zhang, Y., Huang, J., & Yan, J. (2026). Human and AI Judgment in EFL Speaking Assessment: Psychometric Dependability, Feedback Quantity and Coverage, and Exploratory Teacher Perspectives. Journal of Intelligence, 14(11), 296. https://doi.org/10.3390/jintelligence14110296

Vancouver

Zhang Y, Huang J, Yan J. Human and AI Judgment in EFL Speaking Assessment: Psychometric Dependability, Feedback Quantity and Coverage, and Exploratory Teacher Perspectives. Journal of Intelligence. 2026;14(11):296. doi:10.3390/jintelligence14110296

BibTeX
@article{Zhang2026,
  author = {Zhang, Yan and Huang, Jinyan and Yan, Jun},
  title = {Human and AI Judgment in EFL Speaking Assessment: Psychometric Dependability, Feedback Quantity and Coverage, and Exploratory Teacher Perspectives},
  journal = {Journal of Intelligence},
  year = {2026},
  volume = {14},
  number = {11},
  pages = {296},
  doi = {10.3390/jintelligence14110296}
}
Kostenlos starten →