How do RoPE and ALiBi encode position, and why do they extrapolate better than learned positions?
Modern LLMs abandoned learned absolute positions for RoPE and ALiBi to cope with long context. The signal is recognizing that both encode RELATIVE position, and precisely how that permits length extrapolation past the trained window.
Updated Sep 2026 · Grounded in real GenAI, LLM, and AI/ML engineering interview loops and written to a senior-engineer editorial bar.
Modern LLMs abandoned learned absolute positions for RoPE and ALiBi to cope with long context. The signal is recognizing that both encode RELATIVE position, and precisely how that permits length extrapolation past the trained window.
Lead with where the obvious approach breaks, because that is the judgment they are screening for — most candidates jump straight to the happy path and lose the room.
Then walk the failure back through the pipeline in order, naming the one metric the customer's exec sponsor actually cares about before you propose the fix.
