← 🧠 Foundations of LLMs & GenAINEXT IN FOUNDATIONS OF LLMS & GENAILogits, Log-Probs, and Logit Bias→
Core
Causal Masking and Teacher Forcing
A causal mask adds a triangular block of large negative values to the attention scores before the softmax, so every position gets exactly zero attention weight on the future. That one trick lets you push a whole sequence through in a single forward pass and compute a loss at every position at once, which is teacher forcing, and it is the reason transformer training parallelizes while RNN training could not. AI, ML, and GenAI engineer interviews probe it because it explains the deepest asymmetry in LLMs: training is parallel over positions and generation is irreducibly serial, which is exactly why the KV cache exists.
a free account unlocks the core curriculum tier · no card
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
LLM & GenAI FundamentalsWhat is causal masking, and why does it let you train on every position in a sequence at once?→LLM & GenAI FundamentalsWhy do transformers scale attention scores by 1/√d_k, and what breaks if you skip it?→LLM & GenAI FundamentalsEncoder-only vs decoder-only vs encoder-decoder: when do you use each, and why are decoder-only models dominant?→ML Infrastructure & GPUsWhy is standard attention memory-bound, and how does FlashAttention fix it without changing the math?→Coding & DSAImplement multi-head self-attention from scratch in NumPy, with a causal mask.→LLM & GenAI FundamentalsWhy do transformers need positional encoding, and how do sinusoidal, RoPE, and ALiBi differ?→
COMPANIES THAT ASSUME THIS