AI Engineer Skills: What Interviews Actually Test

You need to be able to explain, without notes, what a token is and why it makes your cost model non-linear, what the KV cache is and why it dominates memory during decode, why prefill and decode have completely different performance characteristics, and what temperature and top-p actually do to the sampling distribution.

Then the things people get wrong under pressure: why a longer context window does not mean the model attends to the middle of it, when fine-tuning helps (style, format, a narrow domain vocabulary, latency by distilling into a smaller model) and when it is a bad answer to a knowledge problem that retrieval solves better, and what quantization costs you in practice.

The tell for a strong candidate is that they reason about the model mechanistically rather than magically. "The model got confused" is not a diagnosis. "The relevant passage was at position 40 of 60 retrieved chunks and the reranker was not scoring it above the boilerplate" is.

Chunking strategy, hybrid retrieval (BM25 plus dense, because lexical search still wins on identifiers, error codes, and product names), reranking, metadata filtering, and permission-aware retrieval so a user never sees a document they cannot open. Know recall@k and precision@k, know how to build the labeled set you measure them on, and know that if the right document is not in your top-k, no amount of prompting will save you.

The interview favorite: "your RAG system is returning plausible but wrong answers, triage it." The structured answer separates the layers. Is the document in the index at all (ingestion), is it retrieved (recall), is it ranked high enough to survive truncation (reranking), does the model use it when it is present (generation and prompt), and is the source itself stale or contradicted (data governance). Candidates who jump straight to "I would try a better embedding model" fail this.

Tool schema design, argument validation, error handling, retries with backoff and jitter, idempotency so a retried side effect does not fire twice, and stopping conditions so the loop cannot burn $40 answering one question. Then the design judgment: when a constrained workflow beats an autonomous loop (almost always, in production), how to decompose a task across agents without inventing a distributed system nobody asked for, and how to keep a human in the loop for irreversible actions.

Prompt injection is the security question that gets asked and rarely answered well. If your agent reads untrusted content and holds a tool that can act, the untrusted content can drive the tool. The mitigation is architectural (least privilege on tools, permission checks in code, confirmation gates on irreversible actions), not a sterner system prompt.

This is the skill that most separates candidates, and the one most people cannot demonstrate because they have never had to. You need a position on: how you build a golden set (start from real production failures, not synthetic happy paths), when an LLM judge is trustworthy (comparative judgments with a rubric, calibrated against human labels) and when it is not (absolute quality scores, anything where the judge shares the generator's blind spot), how you gate a merge on it, and how you know your offline metric predicts the online outcome.

Expect the follow-up: "your eval score went up and users complained more. What happened?" Good answers reach for Goodhart, for a golden set that has drifted away from the live distribution, or for an offline metric that measures the wrong thing (answer similarity when users care about task completion).

Continuous batching, KV cache management, PagedAttention and vLLM, speculative decoding, quantization tradeoffs, and the difference between throughput optimization and latency optimization (they pull against each other, and knowing which one the product needs is the actual skill). On the application side: streaming to hide latency, semantic caching, prompt compaction, and model routing so the frontier model only handles the requests that need it.

Have a number ready. Roughly what does a million tokens cost at the frontier versus at a small hosted model, roughly what does time-to-first-token look like when you have a large prompt to prefill, and what happens to both when you double the context. Approximate is fine. Vague is not.

ML system design shows up as an end-to-end product question: build semantic search over a company's documents, build a support agent, build a recommendation surface. They want to see you name the offline and online metrics before the architecture, handle the cold-start and the long tail, and design the feedback loop that improves the system after launch.

MLOps shows up as the boring questions that decide whether you are actually senior: how do you version a prompt, a model, and an index together so a rollback is possible, how do you detect drift when there is no label, how do you run a shadow deployment, and what does your CI do with a nondeterministic test.

And you still have to code. It is usually practical rather than puzzle-shaped: implement a rate limiter, write a chunker that respects document structure, parse a genuinely malformed file, wire retries around a flaky API. Python fluency is assumed. A working solution with the edge cases handled beats a clever one that ignores empty input.