streaming

Agent latency is a step-count problem disguised as a speed problem. The winning move is removing round trips, not shaving milliseconds off each one, and knowing which step you can delete is the whole interview.

A classic that pays off the two-heap insight. A sorted list costs O(n) per insert; two balanced heaps give O(log n) insert and O(1) median. Here is the implementation and the rebalancing detail people get wrong.

A classic that checks whether you reach past sorting for the right structure. The signal is the heap solution (O(n log k)), the bucket-sort O(n) trick, and how it shifts for an unbounded stream. Here is the answer.

The textbook variance formula suffers catastrophic precision loss on streaming data and cannot update incrementally. Welford's one-pass update is the norm for online feature normalization. Here it is.

The classic recursive-descent exercise with an applied-AI twist: the JSON your model streams stays truncated mid-token for the whole generation. What matters is a clean strict parser plus a small repair layer, not a second parser. The code follows.

This code runs on every single response, so it sits on the critical path and a heavyweight check here doubles your p95. The signal is in the ordering, the fail-open decision when the guardrail times out, and the fact that you cannot un-send a token you already streamed.

The naive version splits on newlines and works right up until a TCP chunk lands mid-line. Then there is the error that arrives after a 200 OK, the user who closes the tab while you keep paying for tokens, and JSON you cannot parse until it closes.

Threshold alarms page you all weekend yet sleep through Monday's actual outage. What interviewers reward is stripping out seasonality first, matching the detector to the anomaly, and tuning against alert fatigue. Here is the answer that holds up in production.

A hard streaming-systems question: dedup at high throughput while keeping state bounded. What interviewers reward is a tiered state design (a probabilistic filter ahead of durable state) plus watermark-driven eviction. Here is the architecture that does not OOM.

A pipeline-design question that rewards fitting the architecture to the freshness requirement rather than chasing real-time for its own sake. What interviewers want is the latency-vs-complexity tradeoff and an understanding of why Kappa arose to kill Lambda's dual codebase. Here is the decision.

You cannot wait for all the data, and buffering an unbounded stream will OOM. The signal is windowed joins with watermarks for stream-stream, lookup joins for stream-table, and a clean account of late and out-of-order events.

At-least-once creates duplicates and at-most-once drops data; everyone asks for exactly-once yet few can explain how Kafka provides it. The signal is idempotent producers, transactional read-process-write, and the read-committed isolation that binds them together.

Events arrive minutes or days after they happened, and a window that has already closed reports wrong counts. The signal is event-time vs processing-time, watermarks with allowed lateness, and how you fix already-emitted aggregates.

Transcribing in real time at scale probes streaming ASR, latency budgets, and GPU fleet management under heavy concurrency. The signal is chunked streaming inference with partial results, not batch transcription. Here is the design.

A voice assistant links STT, an LLM, and TTS under a punishing latency budget where each stage piles on delay. The signal is streaming and pipelining the stages plus turn-taking, not three blocking calls. Here is the design.

Online models need features that are fresh and computed the same way training computed them. The classic failure is training-serving skew that quietly degrades the model. The signal is the online/offline split built on one shared definition. Here is the design.

A writing assistant has to feel instant as the cursor moves through a live document. Learn how to separate fast deterministic checks from LLM rewrites, stream suggestions, and keep edits stable so the text does not flicker while the user types.

Live speech-to-speech translation is a latency race across three models (recognize, translate, synthesize) as audio keeps streaming in. See the streaming pipeline, how to commit partial results without flip-flopping, and the tradeoff between latency and translation quality.

SSE versus WebSocket is the easy half. The half that separates candidates: a buffer-everything output filter throws away the time-to-first-token you paid a GPU to deliver, and you cannot send an HTTP error after the 200 has already flushed.

You cannot review a live stream after publishing, because it already reached the audience. The broadcast delay buffer is the design, and everything else (frame sampling, the audio channel, the cascade) is built to fit inside it. Here is how it fits together.

Not every prediction needs a low-latency endpoint, and reaching for one by default wastes money. What matters is matching the serving pattern to the latency and freshness requirement. Here is the decision and when you'd reverse it.

A fraud model fed a feature an hour behind is half-blind, but recomputing everything in real time wastes money you don't need to spend. Freshness is a per-feature decision on a real cost curve. Here is how to manage it.