Why LLMs Struggle to Count Letters

Before a language model sees any text, a tokenizer splits it into tokens. Most modern tokenizers use a method such as byte pair encoding, which learns common character sequences from a large body of text. Frequent words often become a single token, and rarer words are split into a few pieces. The model then receives each token as a number and turns it into a vector. It never receives the letters inside that token as separate inputs. So when you ask how many r's are in 'strawberry', the model is not scanning ten characters. It sees perhaps two or three chunks, and it has to recall, from patterns learned during training, which letters those chunks contain. That knowledge exists but it is indirect and fuzzy, like being asked how many times the letter e appears in a word you have only ever seen as a picture. The strawberry question became a widely shared example in 2024 precisely because the answer is obvious to people and surprisingly hard for chatbots. The same cause produces related errors: misspelling when asked to spell out loud, reversing words wrongly, failing letter-based constraints like acrostics or lipograms, and miscounting characters for length limits. Exact splits differ between tokenizers, so one model may get a given word right while another fails. A common misconception is that this shows models cannot reason or are simply careless. The limit is mostly in the input format, not in general reasoning. When a model is prompted to spell the word out first, each letter tends to become its own token, and counting becomes a far easier task. Newer reasoning models often do this spelling step on their own, which is one reason they get these questions right more often. For anything where exactness matters, running a short piece of code is still the most reliable method.