When Do Large Language Models (LLMs) Struggle to Count Letters?
ACM Transactions on Intelligent Systems and Technology · 31.10.2026 · 17 (5) · 1-17
DOI: 10.1145/3818606 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Intelligent Systems and Technology, online erschienen am 23.07.2026. Verfasst von Tairan Fu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: FUN4DATE, Spanish Agencia Estatal de Investigacion, Chips Act Joint Undertaking project SMARTY.
Abstract (Originalsprache)
Large Language Models (LLMs) have achieved unprecedented performance on many complex tasks, being able, for example, to answer questions on almost any topic. However, they struggle with other simple tasks, such as counting the occurrences of letters in a word, as illustrated by the inability of many LLMs to count the number of “r” letters in “strawberry.” Several works have studied this problem and linked it to the tokenization used by LLMs, the intrinsic limitations of the attention mechanism, or to the lack of character-level training data. In this paper, we conduct an experimental study to evaluate when LLMs fail to count letters. In more detail, we study the relations between LLM errors when counting letters, including: (1) the frequency of the word and its components in the training dataset and (2) the complexity of the counting operation. We present a comprehensive analysis of the errors of LLMs when counting letter occurrences by evaluating a representative group of models over a large number of words. The results show a number of consistent trends in the models evaluated: (1) models are capable of recognizing the letters but not counting them; (2) the frequency of the word and tokens in the word does not have a significant impact on the LLM errors; (3) there is a positive correlation of letter frequency with errors; more frequent letters tend to have more counting errors; (4) the errors show a strong correlation with the number of letters or tokens in a word; and (5) the strongest correlation occurs with the number of letters with counts larger than one, with most models being unable to correctly count words in which letters appear more than twice. These results suggest that the problems of LLMs to counting letters are not related to the frequency of words or tokens in the training data but to the complexity of the counting operation. However, further studies are needed to build a better understanding of the limitations of LLMs in count the letters in a word.
Förderung
- FUN4DATE
- Spanish Agencia Estatal de Investigacion
- Chips Act Joint Undertaking project SMARTY
- OpenAI API research access program
Zitieren
APA
Fu, T., Ferrando, R., Conde, J., Arriaga, C., & Reviriego, P. (2026). When Do Large Language Models (LLMs) Struggle to Count Letters?. ACM Transactions on Intelligent Systems and Technology, 17(5), 1-17. https://doi.org/10.1145/3818606
Vancouver
Fu T, Ferrando R, Conde J, Arriaga C, Reviriego P. When Do Large Language Models (LLMs) Struggle to Count Letters?. ACM Transactions on Intelligent Systems and Technology. 2026;17(5):1-17. doi:10.1145/3818606
BibTeX
@article{Fu2026,
author = {Fu, Tairan and Ferrando, Raquel and Conde, Javier and Arriaga, Carlos and Reviriego, Pedro},
title = {When Do Large Language Models (LLMs) Struggle to Count Letters?},
journal = {ACM Transactions on Intelligent Systems and Technology},
year = {2026},
volume = {17},
number = {5},
pages = {1--17},
doi = {10.1145/3818606}
}
EUVIMED