subword

Tokenization reads like plumbing until it accounts for your bill, your latency, and why the model stumbles on numbers and rare languages. The signal is understanding how subword tokenizers operate and the consequences that follow from them.

BPE powers GPT-style tokenizers, and coding it shows you grasp how subword vocabularies get constructed rather than just knowing they exist. What interviewers watch for is the repeated merge of the most common adjacent pair.

A build-it-yourself check on subword tokenization. What matters is greedy longest-match encoding against a fixed vocabulary, the continuation-prefix convention, and how WordPiece diverges from BPE. The code follows.