← 🧠 Foundations of LLMs & GenAINEXT IN FOUNDATIONS OF LLMS & GENAIAttention Variants: MHA, MQA, and GQA→
Core
Attention and Self-Attention
Attention casts each token as a query, key, and value, scores every query against every key, softmaxes those scores into weights, and returns the weighted sum of values, so each token draws information from the others. Self-attention does this within one sequence. The all-pairs scoring is why cost grows with the square of sequence length, which then explains context limits, long-prompt expense, and the KV cache. AI, ML, and GenAI engineer interviews probe it because it ties architecture to cost and latency in one mental model.
a free account unlocks the core curriculum tier · no card
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
Coding & DSAImplement multi-head attention from scratch with a KV cache for decoding, then extend it to grouped-query attention.→LLM & GenAI FundamentalsWhat is causal masking, and why does it let you train on every position in a sequence at once?→LLM & GenAI FundamentalsWhy do transformers scale attention scores by 1/√d_k, and what breaks if you skip it?→LLM & GenAI FundamentalsEncoder-only vs decoder-only vs encoder-decoder: when do you use each, and why are decoder-only models dominant?→Coding & DSAImplement multi-head self-attention from scratch in NumPy, with a causal mask.→Coding & DSAImplement attention with a KV cache for autoregressive generation.→
COMPANIES THAT ASSUME THIS