tokenization

Tokenization reads like plumbing until it accounts for your bill, your latency, and why the model stumbles on numbers and rare languages. The signal is understanding how subword tokenizers operate and the consequences that follow from them.

Multilingual models are uneven across languages, and most candidates cite only the obvious cause. The differentiator is the tokenizer: the same sentence can cost a non-English user several times as many tokens. Here is the answer.

When a drug name or a Hindi word splits into nine subword tokens, you pay in cost, context, and accuracy. Adding tokens is easy and adding them well is not. Here is the tradeoff.

Grammar-constrained decoding seems like a solved problem until tokenization stops lining up with the grammar. The signal is the token-vs-character boundary mismatch and how prefix automata and token healing address it.

The whiteboard question that spans the entire stack, from BPE to the sampler. What separates a strong answer is knowing that prefill and decode stress opposite hardware resources, and that nearly every serving decision you will ever make falls out of that one fact.

BPE powers GPT-style tokenizers, and coding it shows you grasp how subword vocabularies get constructed rather than just knowing they exist. What interviewers watch for is the repeated merge of the most common adjacent pair.

Every RAG system packs a prompt, and the packing bug is always the same one: the input fits the window exactly, so the model has nowhere left to answer. The arithmetic here is what separates a candidate who has shipped from one who has read about it.