caching
AI, ML & GenAI interview questions tagged caching, across every topic.
6 questions · 1 unlocked for you
Concepts behind "caching"
The curriculum that explains the ideas these questions test.
Foundational
The LLM GatewayAn LLM gateway is one proxy layer sitting between your application and one or more model providers. It consolidates the cross-cutting concerns every LLM app needs: routing and fallback across models/providers, caching, rate limiting, authentication, cost tracking, observability, and guardrails. By hiding providers behind a single interface, it also guards against vendor lock-in. AI, ML, and GenAI engineer interviews probe it because it forms the backbone of a production LLM platform and holds most operational controls.⚙️ System Design for AI in Production
Foundational
LLM Cost OptimizationLLM systems get expensive fast, and the cost model comes down mostly to tokens and number of model calls. The levers, in rough order of impact: route easy queries to cheaper/smaller models, cache repeated and similar requests, trim context (fewer, better chunks), use cheaper retrieval/reranking, and for agents cut unnecessary steps. The discipline is measuring cost per request and going after the dominant contributor. AI, ML, and GenAI engineer interviews probe it because cost is a primary production constraint and most teams overspend by defaulting to the biggest model on everything.⚙️ System Design for AI in Production
Foundational
Caching StrategiesA cache trades freshness for speed by holding a copy of hot data closer to the request. The strategy is the write/read pattern: cache-aside (app fills the cache on a miss), write-through (writes pass through the cache to the store), write-back (writes hit the cache and flush later). Eviction (LRU, LFU) and TTL govern what to keep, and cache stampede protection prevents a popular expired key from hammering the backing store. CDNs are caches at the network edge. AI, ML, and GenAI engineer interviews probe it because LLM responses, embeddings, and retrieval results are expensive enough that caching is a first-class design decision.⚙️ System Design for AI in Production