positional encoding

Attention is permutation-invariant, so with no position information a transformer cannot detect word order. The signal is knowing why, and why the field shifted from absolute sinusoidal encodings to relative ones like RoPE and ALiBi that extrapolate to longer contexts.

Context windows are not arbitrary figures; they follow from attention's quadratic cost, KV-cache memory, and how the model was trained. What shows depth is naming all three causes and matching each to the technique that extends it.

Modern LLMs abandoned learned absolute positions for RoPE and ALiBi to cope with long context. The signal is recognizing that both encode RELATIVE position, and precisely how that permits length extrapolation past the trained window.

A model trained at 4K tokens can reach 128K without a full retrain. What matters is understanding why naive extrapolation breaks and how PI, NTK-aware scaling, and YaRN each rescale RoPE frequencies in different ways.

RoPE is why modern LLMs extrapolate to longer contexts, and it works by rotation, not addition. Coding it shows you truly understand how position enters attention. Here is the implementation.