Implement multi-head attention from scratch with a KV cache for decoding, then extend it to grouped-query attention.

Plain multi-head attention is rarely the whole exercise: the usual extension is to chain it into KV-cached decoding and GQA in one sitting. The locked answer walks the NumPy implementation, the equivalence check that proves your cache is correct, and the reshape bugs that silently corrupt attention.

Updated Sep 2026 · Grounded in real GenAI, LLM, and AI/ML engineering interview loops and written to a senior-engineer editorial bar.