rope

Attention is permutation-invariant, so with no position information a transformer cannot detect word order. The signal is knowing why, and why the field shifted from absolute sinusoidal encodings to relative ones like RoPE and ALiBi that extrapolate to longer contexts.

Modern LLMs abandoned learned absolute positions for RoPE and ALiBi to cope with long context. The signal is recognizing that both encode RELATIVE position, and precisely how that permits length extrapolation past the trained window.

RoPE is why modern LLMs extrapolate to longer contexts, and it works by rotation, not addition. Coding it shows you truly understand how position enters attention. Here is the implementation.