ttft
AI, ML & GenAI interview questions tagged ttft, across every topic.
5 questions · 0 unlocked for you
Concepts behind "ttft"
The curriculum that explains the ideas these questions test.
Foundational
Latency Budgets and StreamingLLM latency is not a single figure: time-to-first-token (driven by prefill and queueing) and inter-token latency (driven by decode) feel very different to users. Streaming tokens as they generate masks total latency by showing progress right away. Designing to a latency budget means splitting time across retrieval, model, and tools, tracking TTFT and tokens-per-second (not only end-to-end), and applying streaming, caching, and routing to meet it. AI, ML, and GenAI engineer interviews probe it because perceived latency makes or breaks LLM UX.⚙️ System Design for AI in Production
Core
Token Streaming: SSE, Chunking, and CancellationServer-Sent Events is the default transport for one-way token streams, and the interesting problems start after you pick it: an output guardrail that buffers the whole response destroys the time-to-first-token you paid a GPU for, you cannot send an HTTP error status after the 200 has flushed, and a client disconnect must actually cancel the GPU work or you keep generating tokens nobody will read. AI, ML, and GenAI interviews probe it because 'we stream the tokens' is one sentence and shipping it correctly is a design round.⚙️ System Design for AI in ProductionSign in
Advanced
Disaggregated Prefill/Decode and Prefix CachingLLM inference has two phases with opposite hardware profiles: prefill is compute-bound (it works through the whole prompt in parallel) while decode is memory-bandwidth bound (one token at a time). Running both on the same GPU pool makes them compete, so long prefills stall ongoing decodes and you miss either the time-to-first-token or the time-per-output-token SLO. Disaggregation places them on separate GPU pools and moves the KV cache between them, and prefix caching reuses KV for shared prompt prefixes. AI, ML, and GenAI engineer interviews probe it because it is the current frontier of serving architecture and a real latency-SLO tradeoff.🖥️ ML Infrastructure & Serving🔒 Premium