production
AI, ML & GenAI interview questions tagged production, across every topic.
4 questions · 0 unlocked for you
Concepts behind "production"
The curriculum that explains the ideas these questions test.
Foundational
The LLM GatewayAn LLM gateway is one proxy layer sitting between your application and one or more model providers. It consolidates the cross-cutting concerns every LLM app needs: routing and fallback across models/providers, caching, rate limiting, authentication, cost tracking, observability, and guardrails. By hiding providers behind a single interface, it also guards against vendor lock-in. AI, ML, and GenAI engineer interviews probe it because it forms the backbone of a production LLM platform and holds most operational controls.⚙️ System Design for AI in Production
Foundational
Latency Budgets and StreamingLLM latency is not a single figure: time-to-first-token (driven by prefill and queueing) and inter-token latency (driven by decode) feel very different to users. Streaming tokens as they generate masks total latency by showing progress right away. Designing to a latency budget means splitting time across retrieval, model, and tools, tracking TTFT and tokens-per-second (not only end-to-end), and applying streaming, caching, and routing to meet it. AI, ML, and GenAI engineer interviews probe it because perceived latency makes or breaks LLM UX.⚙️ System Design for AI in Production
Foundational
GuardrailsGuardrails are the runtime safety layer around an LLM: input checks (spotting prompt injection, off-topic or disallowed requests, PII) ahead of the model, and output checks (content safety, schema/format validation, grounding, PII/secret leakage) ahead of the user. They combine rules, classifiers, judge models, and validators, plus a defined fail-safe action when one trips. AI, ML, and GenAI engineer interviews probe it because 'add guardrails' is hand-wavy, and it is the concrete input/output checks plus fail-safe behavior that keep a deployment safe.⚙️ System Design for AI in Production
Foundational
Rate Limiting, Retries, and BackoffLLM systems rely on rate-limited, sometimes-failing providers, so resilient design is essential. Rate limiting (token bucket) shields your service and enforces per-tenant quotas; retries with exponential backoff and jitter absorb transient failures without hammering a struggling dependency; circuit breakers stop sending requests to a failing service so it can recover. AI, ML, and GenAI engineer interviews probe it because LLM calls are slow, expensive, and flaky, and naive retry logic turns a blip into an outage.⚙️ System Design for AI in Production
Foundational
Observability for LLM SystemsYou cannot run or improve an LLM system you cannot see. Observability means logging every request end to end, inputs, retrieved context, prompt and model version, output, tokens, latency, and cost, plus tracing multi-step agent/RAG flows and tracking quality signals. It grounds debugging, cost attribution, evaluation, and incident response. AI, ML, and GenAI engineer interviews probe it because LLM systems fail silently (a plausible-but-wrong answer throws no error), so visibility is what keeps them debuggable and trustworthy.⚙️ System Design for AI in Production
Foundational
LLM Cost OptimizationLLM systems get expensive fast, and the cost model comes down mostly to tokens and number of model calls. The levers, in rough order of impact: route easy queries to cheaper/smaller models, cache repeated and similar requests, trim context (fewer, better chunks), use cheaper retrieval/reranking, and for agents cut unnecessary steps. The discipline is measuring cost per request and going after the dominant contributor. AI, ML, and GenAI engineer interviews probe it because cost is a primary production constraint and most teams overspend by defaulting to the biggest model on everything.⚙️ System Design for AI in Production