cost optimization

Most teams reach for a token-dropping compressor and pay an accuracy tax for tokens they could have had for almost nothing. The ranking of levers matters more than any single technique, and the top of the list is not compression at all.

Every request pays for the prompt and almost nobody has looked at where the tokens actually go. It is rarely the part you can read: thirty tool schemas and seven stale examples usually outweigh the system prompt, and the output side costs several times more per token.

Most candidates answer 'use the reasoning model for hard problems' and stop. The interviewer wants it framed as an eval and a budget problem: how you prove the extra thinking tokens paid off, and what you do when the model talks itself out of a correct answer.

RAG and agents turn expensive and slow quickly: retrieval plus reranking plus big-model calls, multiplied across agent steps. The signal is naming the dominant cost first, then the levers that genuinely move it. Here is the playbook.

GPU spend is typically the largest line item in an AI org, and this question tests whether you reason in utilization rather than just capacity. The signal is wringing out per-unit cost (utilization, right-sizing, spot, quantization) before scaling out.

Caching is one of the biggest LLM cost levers, but 'cache the response' is naive for a non-deterministic system. What counts is telling prompt (prefix) caching apart from semantic caching and knowing when each is safe.

Routing each request to the right model is one of the biggest LLM cost/latency levers in production. What counts is matching query difficulty to model capability and knowing when the cascade pattern beats a classifier.

An image is not one input, it is a large prompt. Once you internalize that image tokens dominate prefill, the cost levers stop being model choices and start being resolution, tiling, and cache placement.