gpu

The per-token sticker price masks the real decision. Self-hosting only pays off past a volume break-even most teams misjudge. Here is the back-of-envelope a staff engineer does on the whiteboard.

Limited GPUs, dozens of models, and every team demanding low latency for little money. The signal is whether you can shape that into a single governed serving fleet: continuous batching, KV cache, per-tenant quotas, and cost you can genuinely attribute.

A GPU-heavy generative system where sampling steps drive the bill and a single bad image becomes a headline. The signal is the diffusion serving pipeline, the cost levers on per-image GPU time, and a required two-sided safety layer.

A GPU-heavy generative-audio system with text/genre/lyric conditioning, minutes-long coherence, and copyright constraints. The signal is the generation pipeline plus the async, cost, and safety layer that gets it shippable.

Video generation is image generation plus the harsh constraint of temporal consistency, and it is the most GPU-intensive generative task around. The signal is diffusion over spacetime plus a serving design that holds up under multi-minute jobs.

Batch scoring a billion items is a throughput-and-cost problem, the reverse of low-latency serving. Every choice that harms latency works in your favor here. Here is how to design for dollars-per-million-documents.

GPU spend is typically the largest line item in an AI org, and this question tests whether you reason in utilization rather than just capacity. The signal is wringing out per-unit cost (utilization, right-sizing, spot, quantization) before scaling out.

A Deployment and a Service will serve a model, but GPUs upend every Kubernetes default: scheduling, probes, autoscaling signals, and rollouts. Here is the setup that survives production, and when KServe earns its complexity.

A hardware-literacy question, particularly at NVIDIA and the labs. What they grade is whether you understand throughput vs latency hardware, the memory hierarchy, and why matrix multiplication maps onto GPUs and TPUs. This is the architecture-aware answer.

Autoscaling GPUs is nothing like autoscaling web servers: GPUs are scarce and expensive, model loading is slow, and the right signal is not CPU. The signal is scaling on queue and GPU metrics, taming cold starts, and the scale-to-zero economics.

CPU-based autoscaling that suits a web tier quietly breaks on GPU inference: the signal is wrong, and replicas need minutes to warm. The interviewer wants the signals you genuinely scale on and how you mask the cold start.

Paying for accelerators that sit idle half the time is the most common waste in ML training, and the reflex to add more GPUs only makes it worse. The interviewer wants the profiling discipline that pinpoints what is starving them.

The counterintuitive reality of LLM serving: token generation is capped by how quickly you can read weights out of memory, not by arithmetic. Once that clicks, the entire optimization menu follows from one number.

Linear scaling is the marketing figure; the actual curve bends early for reasons rooted in physics, not bugs. Here is where the speedup leaks and how to recover it.

A 256-GPU job that freezes with no error at 3am is a special kind of pain. The causes come from a short, recurring list. Here is the systematic way to identify which one hit you.

Spot GPUs frequently run 60-90% cheaper, and they disappear with two minutes' notice. The savings hold only if a preemption costs you minutes rather than the whole run. Here is how to make that the case.

Naive 4-bit quantization can wreck quality, and the reflex to give up and serve fp16 leaves a large speedup unclaimed. The accuracy is usually recoverable. Here is the ladder of fixes.

Pinning one GPU per model leaves most of a fleet stranded on idle silicon. Sharing safely is a genuine systems problem with four distinct mechanisms, each suited to a different traffic shape. Here is how to choose.