decoding

Every LLM product tunes these knobs, and the interviewer wants to hear what each one does to the probability distribution and which setting suits which task. That includes why factual and creative tasks call for opposite settings.

How you decode from the model's probabilities shapes the output as much as the model itself. The signal is knowing the deterministic vs sampling methods and where each fits (factual vs creative). Here is the answer.

Token count is latency and dollars, not merely style. 'Be concise' hardly works, and max_tokens just cuts off mid-sentence. Here is how to genuinely shape length without amputating answers.

Neural text degeneration is a known failure of likelihood-maximizing decoding, not a broken model. The fixes are specific decode-time knobs. Here is which lever to reach for and what each one costs.

Generation has to end somewhere, and the wrong stop rule either truncates answers or wastes tokens on trailing garbage. Here is how the EOS token, stop strings, and max-token caps actually interact.

Beam search is the decoding algorithm everyone cites and few can code correctly. The signal is keeping k hypotheses by cumulative log-prob, adding logs (not multiplying probs), and handling completed sequences and length. Here is a correct implementation and the tradeoffs.

Coding the KV cache shows you understand why decode is efficient: you append the new token's K/V and reuse the rest rather than recomputing. Interviewers look for the append-and-attend-over-cache logic. The implementation follows.

Every chat model's 'temperature' and 'top_p' knobs come down to a few lines of logit surgery. Implementing them proves you understand decoding rather than just calling it. Here is the from-scratch version.