context window
AI, ML & GenAI interview questions tagged context window, across every topic.
8 questions · 1 unlocked for you
Concepts behind "context window"
The curriculum that explains the ideas these questions test.
Foundational
TokenizationModels read neither characters nor words; they read tokens, subword chunks produced by an algorithm like BPE that maps text to integer IDs. Tokenization sets how many tokens a piece of text costs (driving price, latency, and context usage), why models miscount letters or stumble on rare words, and why non-English text costs more. AI, ML, and GenAI engineer interviews probe it because token accounting is the first thing that bites a production LLM bill.🧠 Foundations of LLMs & GenAI
Foundational
The Context WindowThe context window is the largest number of tokens a model can attend to at once, prompt plus generation. It is capped by attention's quadratic cost, the KV cache's linear memory growth, and the length the model trained on. A bigger window is neither free nor uniformly useful (models lose information in the middle), which is why retrieval often beats cramming everything into context. AI, ML, and GenAI engineer interviews probe it because it drives cost, latency, and the RAG-vs-long-context decision.🧠 Foundations of LLMs & GenAI
Core
Positional Encodings (RoPE and ALiBi)Attention is order-blind, so models inject token position separately. Modern LLMs rely on relative schemes: RoPE rotates query/key vectors by an angle proportional to position so the attention score hinges only on the offset between tokens, and ALiBi adds a distance penalty to attention scores. Both extrapolate to longer sequences far better than learned absolute positions, which is why RoPE-with-scaling is how context windows get extended. AI, ML, and GenAI engineer interviews probe it because it explains how long-context models are built.🧠 Foundations of LLMs & GenAISign in
Core
Retrieval vs Long ContextIf a whole document fits in a model's large context window, should you paste it, or retrieve only the relevant chunks? Long context is simpler but costly (quadratic attention), slower, and used unevenly (lost in the middle); retrieval is cheaper, faster, refreshes without retraining, and surfaces only what matters. The usual answer is retrieval for large, changing, or partially-relevant corpora, and long context for small, cohesive inputs. AI, ML, and GenAI engineer interviews test it because 'just use the big context window' is a common, costly oversimplification.🤖 Retrieval & AgentsSign in