What Does an AI Engineer Actually Do All Day?

Monday you are looking at traces from the weekend, because a support agent started answering a class of question confidently and wrongly, and the trace shows retrieval pulled a deprecated policy doc that nobody tombstoned when it was superseded. That is a data freshness bug wearing a hallucination costume, and most of the ones you chase are.

Tuesday you add fifteen examples of that failure to the golden set, fix the ingestion so superseded documents get filtered at query time rather than hoped away in the prompt, and rerun the suite. Retrieval recall moves; end-to-end answer quality moves less than you expected, which tells you something about where the remaining error actually lives.

Wednesday and Thursday are a feature: a new tool the agent can call, which means a schema, argument validation, an idempotency key so a retry does not double-charge anyone, a timeout, and a decision about what the model is allowed to do when the tool fails. Friday is cost. Someone noticed spend is up 40% and it turns out a prompt change quietly doubled the context you send on every call.

That rhythm (debug, measure, build, then pay for it) is the job. The proportions vary. The presence of all four does not.

Start with an eval, not a prompt. Before you write the feature, write down what a correct output looks like and how you will detect an incorrect one at scale. A hundred labeled examples with the real failure modes in them beats a thousand synthetic ones that all look like the happy path.

Then build the cheapest thing that could work. For retrieval, that is usually BM25 or hybrid search over well-chunked documents before you reach for a reranker or a knowledge graph. For agents, that is usually a constrained workflow with two or three tools before you reach for an open-ended planning loop, because open-ended loops are where latency and cost go to die.

Then iterate against the suite and keep an honest changelog of what moved the metric. This is the part candidates skip and interviewers ask about. When an interviewer says "you changed the chunk size, the prompt, and the reranker in one pass, and quality went up 4 points, what caused it?" the correct answer is that you do not know, and you should not have done that.

Then ship it behind a flag, watch the online metric, and expect the offline number to overstate the win. Offline evals are a proxy. The bar is whether the user's task got done.

Context: chunking, embedding, indexing, hybrid retrieval, reranking, freshness, and the deletion path. If a document is removed from the source system and still answers questions three weeks later, that is your bug.

Behavior: the prompt, the system message, structured output guarantees, tool schemas, the agent's control flow and stopping conditions, and what happens on an ambiguous request.

Quality: the golden set, the graders (exact-match where you can, LLM-as-judge where you must, human review for the sharp edges), the regression gate in CI, and the online telemetry that tells you whether any of it is real.

Economics and latency: model routing (a small model for classification, a frontier model for the hard 5%), caching, prompt compaction, streaming, batching, and knowing your cost per resolved task rather than your cost per token.

Safety: prompt injection defenses when the model reads untrusted content, PII handling, permission checks that happen in your code and not in the model's judgment, and the refusal path.

Not lines of code, and not model benchmark scores. The metrics that recur: task success rate on a held-out set, containment or deflection for support-style products, latency at p95 rather than the mean (the mean hides the tail that users actually feel), cost per successful outcome, and the rate of the specific failure your product cannot tolerate. For a legal or medical surface, that last one dominates everything.

A useful discipline: for every AI feature you ship, be able to name the one number you would resign over. If you cannot name it, you have not defined the product.

The data plumbing is most of it. Connectors, permissions mirroring, incremental sync, deduplication, and OCR that turns a table into a lie. Retrieval quality is an ingestion problem far more often than it is an embedding problem, and swapping the embedding model is the reflex that wastes the most time.

Evals rot. A golden set built six months ago encodes the failure modes of a model you no longer use. Budget for maintenance the way you budget for tests.

And nondeterminism changes on-call. You will get paged for a system that is technically up, returning 200s, and quietly wrong. Tracing (the full chain: query, retrieved documents, tool calls, model output) is not a nice-to-have. It is the only way to debug at all.