rate limiting

Labs reach for this practical screen often, since it probes concurrency, time handling, and judgment inside 20 lines. The pitfall is a background thread that burns CPU. Below is the lazy-refill approach interviewers expect, along with the follow-ups.

The most job-shaped coding screen in AI, ML, and GenAI engineering: fan out N LLM calls without melting the rate limit or losing the batch to one bad request. What matters is the retry policy, not the async syntax. Below is the version that passes.

Hand every team an LLM API key and your bill turns into a mystery capped by a six-figure surprise at month end. Attributing, capping, and optimizing spend is a platform feature. Here is how to build it.

Every API platform needs one. Interviewers reward the candidate who picks the right algorithm (token bucket vs sliding window) for the burst behavior, then cracks the genuinely hard part: holding one shared limit across many servers without a per-request round trip to a central store.

LLM endpoints are unusually exposed: every call can spend real money and consume scarce GPU capacity, so abuse is a security and a financial problem at once. What lands is layered controls on tokens and cost, not just request count. Here is the defense.

An attacker doesn't have to breach your LLM to hurt you, only make it do expensive work. A few crafted prompts can pin your GPUs and run up the bill. The defense is not a bigger rate limit.