latency

Token count is latency and dollars, not merely style. 'Be concise' hardly works, and max_tokens just cuts off mid-sentence. Here is how to genuinely shape length without amputating answers.

Generation has to end somewhere, and the wrong stop rule either truncates answers or wastes tokens on trailing garbage. Here is how the EOS token, stop strings, and max-token caps actually interact.

Speculative decoding accelerates generation, and with the right verification rule it leaves the output distribution untouched. What matters is understanding how draft-model, self-drafting (Medusa/EAGLE), and lookahead approaches differ in where the guesses originate, and which of them are still exact.

Every request pays for the prompt and almost nobody has looked at where the tokens actually go. It is rarely the part you can read: thirty tool schemas and seven stale examples usually outweigh the system prompt, and the output side costs several times more per token.

RAG and agents turn expensive and slow quickly: retrieval plus reranking plus big-model calls, multiplied across agent steps. The signal is naming the dominant cost first, then the levers that genuinely move it. Here is the playbook.

This code runs on every single response, so it sits on the critical path and a heavyweight check here doubles your p95. The signal is in the ordering, the fail-open decision when the guardrail times out, and the fact that you cannot un-send a token you already streamed.

Voice is a major modality and a common applied-AI surface. The signal is the audio-to-text pipeline, why Whisper is resilient, and the cumulative latency budget that can make or break a real-time voice agent.

Two models landing at the same accuracy is common, and accuracy was rarely the deciding factor to begin with. The real tiebreakers are the ones a junior candidate leaves out. This is the checklist a staff engineer works through before shipping either one.

Transcribing in real time at scale probes streaming ASR, latency budgets, and GPU fleet management under heavy concurrency. The signal is chunked streaming inference with partial results, not batch transcription. Here is the design.

A voice assistant links STT, an LLM, and TTS under a punishing latency budget where each stage piles on delay. The signal is streaming and pipelining the stages plus turn-taking, not three blocking calls. Here is the design.

Every keystroke has to return ranked suggestions in tens of milliseconds, at search-engine QPS. The interesting part is the data structure and what you precompute, not the query path. Here is the design that holds up under the latency budget.

Predict, value, and bid on one impression in roughly 10ms, billions of times a day. The signal is the pCTR-to-bid pipeline under a tight latency budget, plus why calibration and budget pacing decide whether you make or lose money.

Machine translation is seq2seq generation delivered across hundreds of language pairs under strict latency and quality budgets. What matters is sidestepping the quadratic pair blowup, saving low-resource languages, and serving it cheaply. Here is the design.

Detecting and boxing objects at scale reduces to one driving tradeoff plus a handful of CV specifics most candidates fumble: the detector family, NMS, focal loss, and why mAP rather than accuracy is the metric. Here is the design.

Exact-match caching does little when no two prompts match. Semantic caching reuses answers across similar queries, and its entire danger is handing back a near-match that is subtly wrong. Here is how to build it safely.

Inline completion lives or dies on tail latency: a suggestion that lands after the developer keeps typing is useless. Learn how to assemble the context window, hide model latency behind speculation and caching, and keep a tight feedback loop on acceptance rate.

Moderating billions of items a day with an LLM on each one is too slow and too expensive. Learn the tiered funnel (cheap classifier then LLM then human), how to tune thresholds for precision versus recall, and how to stay ahead of adversaries.

A writing assistant has to feel instant as the cursor moves through a live document. Learn how to separate fast deterministic checks from LLM rewrites, stream suggestions, and keep edits stable so the text does not flicker while the user types.

When a page loads, an ad slot goes to auction across dozens of bidders and a winner is picked before the page finishes rendering, all in under 100ms. The design covers the auction mechanism, the brutal latency budget, budget pacing, and fraud. Here is how an exchange runs millions of auctions per second.

You cannot review a live stream after publishing, because it already reached the audience. The broadcast delay buffer is the design, and everything else (frame sampling, the audio channel, the cascade) is built to fit inside it. Here is how it fits together.

Every AI system spends quality to buy speed, whether or not the team admits it. What scores is knowing the exchange rate for each knob, taking the free wins first, and having already decided what you will sell when the system saturates.

The trap is leaping straight to 'add more GPUs.' The signal is profiling first, then reaching for the cheap, no-retrain levers in the correct sequence. Here is the diagnose-then-optimize playbook for p99.

Caching is one of the biggest LLM cost levers, but 'cache the response' is naive for a non-deterministic system. What counts is telling prompt (prefix) caching apart from semantic caching and knowing when each is safe.

Routing each request to the right model is one of the biggest LLM cost/latency levers in production. What counts is matching query difficulty to model capability and knowing when the cascade pattern beats a classifier.

A fraud model fed a feature an hour behind is half-blind, but recomputing everything in real time wastes money you don't need to spend. Freshness is a per-feature decision on a real cost curve. Here is how to manage it.

Decode runs sequentially and is memory-bound, so latency tricks count. What they grade is whether you can explain speculative decoding's draft-and-verify mechanism (and why it stays exact) along with the other levers and when each fits. This is the latency toolkit.

A newly booted replica is sluggish on its opening requests while weights load and kernels compile, which fights with reactive autoscaling. The signal is identifying where cold start comes from plus the warmup and provisioning remedies.

Users experience the p99, not the median, and the tail is where serving systems quietly break. The causes are queuing and batching effects, not a slow model. Here is how to flatten it.

Launching an LLM service on a guessed GPU count is how you get paged on day one. Capacity planning for LLMs differs from web services because tokens, not requests, are the unit. Here is how to size it.

Three batching strategies, three very different latency profiles. Choosing wrong leaves throughput or tail latency on the floor. Here is what each one costs and when to use it.

Forcing valid JSON sounds free, but the mask computation can stall every decode step. The fix is precompiling the grammar into a fast automaton. Here is where the cost hides and how to strip it out.

The number is bait. Interviewers use this to see whether you treat a benchmark delta as a decision or as a claim to be audited, and whether you can price latency, cost, and on-call burden against accuracy. Here is the answer that scores.