capacity planning

The per-token sticker price masks the real decision. Self-hosting only pays off past a volume break-even most teams misjudge. Here is the back-of-envelope a staff engineer does on the whiteboard.

At a billion vectors the index choice is a memory budget question before it's a recall question. Flat search is off the table, HNSW may not fit, and PQ swaps recall for RAM. Here is the call a staff engineer makes on the whiteboard.

GPUs are scarce and costly, so multi-region HA for LLMs is not simply web-app HA with bigger machines. Capacity, routing, and failover all bend around the GPU constraint. Here is the design.

Launching an LLM service on a guessed GPU count is how you get paged on day one. Capacity planning for LLMs differs from web services because tokens, not requests, are the unit. Here is how to size it.

When each request thinks for 30,000 tokens, serving swings from compute-bound prefill to memory-bound decode, and the KV cache turns into the resource you genuinely schedule. The levers that governed chat traffic stop being the ones that count.

Raw tokens per second is gameable: crank the batch size and the dashboard looks great while every request misses its latency target. Goodput is the throughput that actually meets your SLOs, and it is the number you size and autoscale on.