Your model's p99 inference latency is too high. How do you bring it down without retraining?

TL;DR: Measure before you tune: determine whether the bottleneck is model compute, data and feature fetching, queuing, or network. Then reach for no-retrain levers in sequence: quantization (INT8/FP8) and a compiled runtime (TensorRT, ONNX Runtime), dynamic batching, caching, and only after that horizontal replicas. Aim at p99 in particular, since tail latency usually comes from queuing and batching effects rather than the median.

How to approach it. Demand measurement first; "add hardware" with no profile is precisely the wrong reflex the question is screening for. Break the latency budget into stages, list the levers cheapest-first, and note that p99 (tail) stems from different causes than the median.

A strong answer. Profile. Break end-to-end latency into stages: feature and data fetch, queue wait, model forward pass, post-processing, network. p99 is usually dominated by queuing and batching effects or a slow dependency, not raw model FLOPs, so optimizing the model when the bottleneck is a slow feature store wastes effort.

No-retrain levers, in order:

1. Quantization. Cast weights and activations to INT8 or FP8: often a large speedup for a small, validate-able quality hit. Post-training quantization needs no retraining.
2. Optimized runtime + graph compilation. Export to TensorRT or ONNX Runtime for kernel fusion, optimized kernels, and reduced overhead, frequently 2x or more with no model change.
3. Dynamic batching. Group concurrent requests so the GPU runs efficiently; raises throughput, but cap the batch wait so it does not blow the p99 budget (the classic throughput-vs-latency knob).
4. Caching. Cache results for repeated or near-duplicate inputs (and for LLMs, the KV cache and prompt caching); a meaningful fraction of production traffic repeats.
5. Distillation or a smaller model (mild retrain) only if the above is insufficient.
6. Horizontal replicas + autoscaling. Add capacity to cut queue wait once per-request cost is optimized, otherwise you just pay for inefficiency.

For LLMs specifically: continuous batching and PagedAttention (vLLM), speculative decoding, and serving a quantized model all attack tail latency directly.

The defensible order: measure, optimize per-request cost (quantize, compile, cache, batch), then scale out. Scaling first hides the problem and inflates cost.

Key takeaways

• Profile to find the bottleneck stage before touching the model.
• p99 is a queuing and batching problem more than a FLOPs problem.
• Optimize per-request cost before adding replicas, or you pay for the inefficiency.
• For LLMs, continuous batching, PagedAttention, and speculative decoding target the tail.

What interviewers probe next.

• "Why does batching raise throughput but risk p99?" Waiting to fill a batch adds latency to early-arriving requests; bound the wait or use continuous batching so it adapts to load.
• "Quantization quality risk?" Validate on your eval set; keep sensitive layers higher precision if accuracy drops, and prefer calibrated PTQ methods.
• "GPU utilization is 35%, latency still high, why?" Often CPU-bound preprocessing, data-loading, or small batch sizes starving the GPU; fix the pipeline, not the model.
• "When is horizontal scaling the right first move?" When per-request cost is already low and you are simply out of capacity (queue-bound), not when each call is wasteful.

Common mistakes.

• Adding GPUs or replicas before profiling, paying for inefficiency.
• Optimizing the model when the bottleneck is the feature store or network.
• Cranking batch size for throughput and blowing the p99 latency budget.
• Treating median and p99 as the same problem; tail latency is usually queuing.