Tokenization
Models read neither characters nor words; they read tokens, subword chunks produced by an algorithm like BPE that maps text to integer IDs. Tokenization sets how many tokens a piece of text costs (driving price, latency, and context usage), why models miscount letters or stumble on rare words, and why non-English text costs more. AI, ML, and GenAI engineer interviews probe it because token accounting is the first thing that bites a production LLM bill.
TL;DR: An LLM never touches raw text. A tokenizer cuts text into subword units (tokens) and assigns each an integer ID; the model works only with those IDs. Subword schemes like Byte Pair Encoding keep common words as single tokens and split rare ones into pieces, so no word is ever unknown, though token counts swing wildly by language and content. Since you pay, wait, and fill the context window per token, tokenization is the unit of account for every production LLM system.
Why models read tokens, not words or characters
Character-level models stretch sequences to impractical lengths; word-level models demand a huge vocabulary and still break on unseen words. Subword tokenization splits the difference: a fixed vocabulary (say 50k-200k entries) of frequent words and word fragments. Common words ("the", "model") stay single tokens; rarer or novel words ("tokenization", "antidisestablishment") fracture into known pieces. Each token maps to an integer ID, and every input is nothing but a sequence of those IDs.
The dominant scheme is Byte Pair Encoding (BPE): start from characters and repeatedly merge the most frequent adjacent pair into a new token, building up a vocabulary of useful chunks. Modern tokenizers operate on bytes (byte-level BPE), so any Unicode, emoji, or code is representable, there is never a true "unknown token", worst case a string falls back to individual bytes.
Why token accounting runs your bill
Everything downstream is measured in tokens, so tokenization is the unit of account:
- Cost. APIs charge per input + output token. A prompt that tokenizes into 1,200 tokens costs twice one that tokenizes into 600.
- Latency. Generation is per-token, and the prompt must be processed (prefill) before the first token appears, so token count drives time-to-first-token and total latency.
- Context budget. The context window is measured in tokens, so verbose or token-inefficient text fills it faster (see the context window).
A subtle, costly fact: non-English text and non-Latin scripts tokenize into far more tokens per unit of meaning, because the tokenizer was trained mostly on English. The same sentence in Hindi or Chinese can cost several times more tokens than in English, a real fairness and cost issue for global products (see multilingual models).
Why models "can't count letters"
A model that fails "how many r's in strawberry?" is not dumb, it never saw the letters. It saw the tokens for "straw" and "berry" (or similar), so character-level questions are genuinely hard for it. The same goes for reversing strings or arithmetic on digits: the tokenization hides the structure the task needs. Knowing this explains a whole class of "silly" LLM failures.
Why interviewers probe this
Token accounting is the first thing that surprises teams shipping LLMs: a feature that worked in a demo blows the budget at scale because prompts are token-heavy. A strong answer connects tokenization to the three things that bite, cost, latency, and context budget, and can explain why a multilingual product costs more and why letter-counting fails. It signals you think in the units the system actually runs on.
Common misconceptions
- "Tokens are words." Tokens are subword chunks; one word can be several tokens, and a token can span a space or punctuation.
- "There are unknown words." Byte-level BPE always represents any input (worst case, byte by byte), so there is no out-of-vocabulary failure.
- "Token count equals word count." A rough rule is ~0.75 words per token for English, but it varies by content and breaks badly for other languages and code.
- "All languages cost the same." Non-English text usually costs more tokens for the same meaning, because the tokenizer is English-centric.
Key takeaways
- A tokenizer maps text to subword token IDs; the model only ever processes those IDs.
- BPE (byte-level) builds the vocabulary by merging frequent pairs, so there is no true unknown token.
- Tokens are the unit of cost, latency, and context budget, the first lever in any LLM cost model.
- Tokenization explains miscounting letters, fumbling rare words, and the higher cost of non-English text.
Check yourself before an interviewer does. Answer from memory first.
The same prompt costs three times more tokens in Hindi than in English. What is the first-order explanation?