long context

Context windows are not arbitrary figures; they follow from attention's quadratic cost, KV-cache memory, and how the model was trained. What shows depth is naming all three causes and matching each to the technique that extends it.

Full attention is O(n^2), so long context demands cheaper attention. What shows depth is naming the families (sparse, sliding-window, linear), what each surrenders, and why FlashAttention is not one of them.

Modern LLMs abandoned learned absolute positions for RoPE and ALiBi to cope with long context. The signal is recognizing that both encode RELATIVE position, and precisely how that permits length extrapolation past the trained window.

A model that claims a million-token window may not actually exploit it. The signal is knowing needle-in-a-haystack, where it fails, and how to probe genuine multi-fact reasoning across the entire context. Here is the answer.

Evict the oldest tokens to cap the KV cache and the model falls apart. Retain only the first few tokens and it keeps generating forever. What matters is grasping why those opening tokens behave as an attention sink.

A model trained at 4K tokens can reach 128K without a full retrain. What matters is understanding why naive extrapolation breaks and how PI, NTK-aware scaling, and YaRN each rescale RoPE frequencies in different ways.

At long context it is the KV cache, not the weights, that saturates the GPU. What matters is naming the levers (quantization, token eviction, head sharing, low-rank) and the quality cost of each.

Most teams reach for a token-dropping compressor and pay an accuracy tax for tokens they could have had for almost nothing. The ranking of levers matters more than any single technique, and the top of the list is not compression at all.

An emerging alternative to stuffing the window or building an index: keep the context outside the prompt and let the model write code to inspect it, recursing into the pieces. Knowing when it does not pay is most of the score.

Truncate, map-reduce, refine, retrieve, or stuff a long-context model. Each buys you something different and breaks differently. The candidates who score are the ones who pick based on the question being asked, not the document being read.

Video breaks a VLM on arithmetic before it breaks it on vision: one frame per second of a ten-minute clip is hundreds of thousands of image tokens. Frame selection is not preprocessing, it is the architecture, and the strongest systems index the audio first.

A 2025-2026 question that trips up people clinging to dogma in either direction. The signal is a cost, latency, accuracy, and scale tradeoff, plus knowing the 'lost in the middle' failure of long context. Here is the framework for deciding.

Late chunking reverses the usual order of operations to solve the lost-context problem with zero additional LLM calls. Everything hinges on which step runs first. Here is the answer.

Two models in a chain (ASR then summarization), a long-context problem buried inside, and a faithfulness bar where a fabricated action item carries real consequences. The signal is the pipeline plus how you keep a 3-hour transcript honest at thousands a day.

Two forces shape this design: documents longer than the context window, and summaries that must not lie. The strong answer pairs hierarchical map-reduce with claim-level grounding, then serves it cheaply at scale.

Summarizing a 200-page contract is not one LLM call: it is chunking, hierarchical reduction, and a faithfulness check so you never invent facts. Learn the map-reduce pattern, when long-context wins out, and how to evaluate summaries at scale.

At long context the KV cache, not the weights, dominates GPU memory and decode bandwidth. What counts is spotting it as the binding constraint and naming the levers that shrink it without wrecking quality.

Data, tensor, and pipeline parallelism each leave a single sequence's activations on one device, so training past 200k tokens hits a wall none of them can clear. The fourth axis shards the sequence itself, and the interview turns on the communication math.