Why LLMs Struggle to Count Letters Quiz

Answers and explanations appear as you take the quiz.

1. What is the main reason language models miscount letters in a word?
   • They have too little training data about English
   • They process text as tokens, chunks of characters, rather than individual letters
   • They deliberately round numbers to save computation
   • They cannot store numbers larger than ten
2. What does a tokenizer produce from a piece of text?
   • A list of integer IDs, each standing for a chunk of text
   • A summary of the text's meaning
   • A translation into a universal language
   • An image of the text for the model to read
3. Which prompt change most helps a model count letters correctly?
   • Asking the question in all capital letters
   • Telling the model the answer is important
   • Asking it to spell the word out with separators before counting
   • Asking it to answer as quickly as possible
4. Byte pair encoding builds its vocabulary by doing what?
   • Assigning one token to every dictionary word
   • Repeatedly merging the most frequent adjacent pairs of symbols into new tokens
   • Splitting text only at spaces and punctuation
   • Randomly grouping characters into fixed-length chunks
5. Why might 'Strawberry' and ' strawberry' be handled differently by a model?
   • The model ignores capital letters completely
   • Only one of them is in the training data
   • Capitalization and leading spaces can change how the text is split into tokens
   • Models treat the first word of a sentence as a command
6. According to the guide, what is a common misconception about letter-counting failures?
   • That tokenizers exist at all
   • That they prove models cannot reason in general, when the limit is mostly the input format
   • That different models use different tokenizers
   • That code can count letters exactly
7. For a word game that needs exact letter counts every time, what is the most reliable approach?
   • Ask the model three times and take the most common answer
   • Use a larger model
   • Compute the counts with ordinary code and use the model for other parts
   • Ask the model to double-check its answer
8. Why do newer reasoning models often get the strawberry question right?
   • They were given a list of all letter counts
   • They tend to spell the word out step by step before answering
   • They use no tokenizer at all
   • They refuse to answer counting questions