NVIDIA AI Engineer interview questions

Nearly every candidate can recite softmax(QKᵀ/√d_k)V. What the interviewer is after is the variance derivation and the precise training failure the scale heads off. This is the response that tells rote recall apart from real grasp.

PEFT is how everyone fine-tunes large models now. What matters is the low-rank insight behind LoRA, why it cuts memory so sharply, and what 4-bit QLoRA adds. Here is the answer that goes past 'it's efficient fine-tuning.'

The 2025-2026 landscape isn't 'bigger is better' anymore. What matters is knowing why small models and reasoning models exist, the test-time-compute tradeoff, and routing the right model to the right task. Here is the practitioner's view.

Fine-tune a model on your domain and it may lose the ability to do everything else. What earns the signal is explaining why shared weights cause it and listing the concrete mitigations that all boil down to one principle.

PEFT is a family, not only LoRA, and interviewers test whether you know how each one injects trainable parameters. The signal is where each method adds capacity and the latency and quality tradeoffs that result.

Synthetic data is how teams manufacture training examples when real data runs short, and it comes with sharp failure modes. What shows depth is pinning down when it helps, how to keep quality up, and why training on model output over successive generations narrows the distribution.

Attention gets all the attention, but the feed-forward layers hold most of a transformer's weights and handle much of the per-token 'knowledge' work. What shows depth is knowing what the FFN computes and why it dominates the parameter count.

Full attention is O(n^2), so long context demands cheaper attention. What shows depth is naming the families (sparse, sliding-window, linear), what each surrenders, and why FlashAttention is not one of them.

MQA and GQA shrink the KV cache, the thing that bottlenecks LLM serving. The signal is understanding they share key/value heads across query heads to trim memory and bandwidth, with GQA as the quality-preserving middle ground. Here is the answer.

Fine-tuning a 65B model once demanded a node of A100s. QLoRA compresses it onto one card with three specific tricks. The signal is knowing what gets quantized, what stays trainable, and why quality barely shifts. Here is the answer.

Distillation is not a single method. The signal is knowing when to match logits versus train on generated text, why on-policy distillation beats off-policy, and how reasoning models are distilled.

The proportions of web, code, books, and math in pretraining quietly set downstream skills. The signal is knowing how mixtures get chosen, why upsampling helps, and how methods like DoReMi automate it.

Continued pretraining adds domain knowledge that fine-tuning cannot, yet done carelessly it wrecks general ability. The signal is the replay ratio, learning-rate rewarming, and how you measure forgetting.

MQA and GQA cut the KV cache by sharing key/value heads. MLA goes another way: compress K and V into a low-rank latent and cache that instead. What matters is recognizing it as a cache trick rather than a head-sharing trick, and why it preserves quality.

Evict the oldest tokens to cap the KV cache and the model falls apart. Retain only the first few tokens and it keeps generating forever. What matters is grasping why those opening tokens behave as an attention sink.

A model trained at 4K tokens can reach 128K without a full retrain. What matters is understanding why naive extrapolation breaks and how PI, NTK-aware scaling, and YaRN each rescale RoPE frequencies in different ways.

At long context it is the KV cache, not the weights, that saturates the GPU. What matters is naming the levers (quantization, token eviction, head sharing, low-rank) and the quality cost of each.

In naive LLM serving most GPU memory bleeds away to KV-cache fragmentation. PagedAttention adapts virtual memory paging to win it back. What matters is explaining the fragmentation problem and how blocks resolve it.

Speculative decoding accelerates generation, and with the right verification rule it leaves the output distribution untouched. What matters is understanding how draft-model, self-drafting (Medusa/EAGLE), and lookahead approaches differ in where the guesses originate, and which of them are still exact.

Both compress an LLM to 4-bit weights, yet they choose what to protect in very different ways. What matters is GPTQ's error-correcting solve versus AWQ's activation-aware scaling, and the calibration each demands.

GGUF powers most local LLM runs, and its quant names look cryptic. What matters is decoding what Q4_K_M means and why mixed-precision k-quants outperform naive uniform quantization.

FP8 underpins modern H100-class training and serving. What matters is knowing the two FP8 variants, why a floating format beats INT8 on dynamic range, and where scaling still counts.

MoE trims compute but is awkward to serve: experts have to be sharded, tokens routed across devices, and batches balanced. What matters is the all-to-all communication and the load-imbalance problem, not the training story.

Speech LLMs either transcribe to text or model audio straight away as discrete tokens. The signal is the semantic-vs-acoustic token split and why end-to-end audio models outperform ASR-plus-LLM pipelines on latency and prosody.

Temperature 0 is not the same as deterministic, and the reason lives in the GPU kernels, not the sampler. What gets scored is naming the batch-invariance problem and knowing which fixes are real versus placebo.

Text prompts are a one-dimensional control surface, and cranking the guidance scale until the model obeys just burns your image. The candidates who pass name the right conditioning mechanism for the right kind of control.

The whiteboard question that spans the entire stack, from BPE to the sampler. What separates a strong answer is knowing that prefill and decode stress opposite hardware resources, and that nearly every serving decision you will ever make falls out of that one fact.

One triangle of negative infinity, added before the softmax, is what makes transformer training parallel and generation stubbornly serial. Interviewers want the mechanism and the asymmetry it creates, not a definition of attention read back to them.

Two defaults that every Llama-class model shares and almost no candidate can justify. One is a compute win that turned out to cost nothing in quality; the other is what makes a 60-layer stack converge at all.

Merging one adapter is a single matrix add. Merging four is where careers get interesting: the low-rank updates were each trained as if the others did not exist, and they collide. Here is what actually works and when to route instead.

Video breaks a VLM on arithmetic before it breaks it on vision: one frame per second of a ten-minute clip is hundreds of thousands of image tokens. Frame selection is not preprocessing, it is the architecture, and the strongest systems index the audio first.

A recurring ML-coding screen. The loop itself is easy; what interviewers watch for is k-means++ init, a genuine convergence criterion, and awareness of the failure modes. Here is the compact implementation plus the follow-ups interviewers reliably ask.

A deceptively easy ML-coding ask. Writing exp/sum is trivial; the signal is the max-subtraction trick and the log-sum-exp form that stop it overflowing. Here is the stable implementation and the reason the naive version breaks.

The from-scratch implementation that frontier labs and NVIDIA really do ask. Writing it shows you grasp shapes, the scale factor, the causal mask, and how heads split and recombine. Here is a correct, readable implementation plus the follow-ups.

Beam search is the decoding algorithm everyone cites and few can code correctly. The signal is keeping k hypotheses by cumulative log-prob, adding logs (not multiplying probs), and handling completed sequences and length. Here is a correct implementation and the tradeoffs.

The core operation beneath every embedding/RAG retrieval, posed as a coding exercise. The signal is vectorizing the similarity, normalizing correctly, and using a partial selection (argpartition) rather than a full sort. Here is the efficient implementation and the scaling follow-up.

Data loading is where training pipelines quietly bottleneck. The signal is a correct shuffle-then-batch iterator plus knowing why prefetching and parallel loading keep the GPU fed.

k-NN is quick to code and a solid check of vectorization and the lazy-learner tradeoff. Interviewers look for a tidy vectorized distance calculation plus knowing why naive prediction is O(n) and how to accelerate it. The implementation follows.

Coding conv2d shows you understand what a CNN layer really computes rather than just being able to name it. Interviewers watch for correct output-shape math, tidy stride and padding handling, and awareness of the im2col trick frameworks actually rely on.

LayerNorm appears in every transformer, and coding it verifies you know what it normalizes over and why. Interviewers look for normalization across the feature dimension per token, the learnable scale and shift, and the RMSNorm simplification modern LLMs took up.

Coding the KV cache shows you understand why decode is efficient: you append the new token's K/V and reuse the rest rather than recomputing. Interviewers look for the append-and-attend-over-cache logic. The implementation follows.

Coding BatchNorm verifies you know the train/inference difference, the part everyone forgets. Interviewers look for normalizing over the batch with running stats kept for inference. The implementation follows.

Coding Adam shows you know what .step() actually does: momentum combined with per-parameter adaptive rates and bias correction. What interviewers watch for is the two moments plus the bias-correction step.

Single-head attention shows up often; multi-head layers on the split-into-heads, attend-per-head, concatenate pattern. What gets scored is the reshape into heads plus a sharp explanation of why several heads help. Here is the implementation.

Dropout takes five lines, yet most candidates fumble two of them: turning it off at inference and rescaling the survivors so the expected activation holds constant. Inverted dropout is the convention they want to see.

RoPE is why modern LLMs extrapolate to longer contexts, and it works by rotation, not addition. Coding it shows you truly understand how position enters attention. Here is the implementation.

Many state-of-the-art training runs hold a shadow copy of the weights that averages the trajectory, then serve that rather than the final step. It's a few lines and a genuine accuracy gain. Here it is.

OpenAI's signature build screen: a credit ledger that issues GPU-hours, spends them, and lets unused grants lapse in issue order. The trap is charging against the right grant first. Here is the FIFO-by-expiry design that holds up through every follow-up.

A from-scratch favorite that probes linear algebra fluency. The signal is centering first, using SVD instead of forming the covariance matrix, and reading variance off the singular values. Here is the implementation and the details interviewers push on.

A build-it-yourself check on the EM algorithm and soft clustering. What matters is the pair of alternating steps (responsibilities, then weighted re-estimation), log-sum-exp for numerical stability, and seeing how GMM extends k-means. The code follows.

A build-it-yourself check on recurrent forward passes and backprop through time. What matters is the shared-weight recurrence, summing gradients over timesteps, and articulating the vanishing-gradient problem. The code follows.

A build-it-yourself check on gated recurrence. What matters is wiring the forget/input/output gates and candidate correctly, keeping cell state distinct from hidden state, and explaining why the additive cell path cures vanishing gradients. The code follows.

A build-it-yourself check on modern optimizers. What matters is the moment estimates, bias correction, and the single change that counts: AdamW decouples weight decay from the gradient rather than folding it into the loss. The code follows, plus why it wins.

A build-it-yourself check on how learning-rate schedules are actually computed. What matters is correct closed-form formulas, knowing when each schedule shines, and why decaying the rate aids convergence. Below is the implementation of the common ones.

A build-it-yourself check on how PyTorch actually works under the hood. What matters is assembling a computation graph during the forward pass, local derivatives per op, and a topological-order backward pass that accumulates gradients. Below is a minimal engine.

A build-it-yourself check on distributed input pipelines. What matters is partitioning data across workers with no overlap and no gaps, epoch-consistent shuffling with a shared seed, and handling the uneven-last-batch problem. The code follows.

The most frequent ML fundamentals question, and a subtle seniority check: anyone can repeat the definition, but can you break down the error and convert it into a concrete debugging plan?

Optimizer questions probe whether you grasp what each one adapts and the subtle AdamW fix that the whole field now relies on. The signal is the per-parameter adaptivity story plus why tying weight decay to Adam was a bug. Here is that answer.

SVMs sort the candidates who memorized 'maximize the margin' from those who can explain how a kernel delivers non-linear separation without ever touching the high-dimensional space. Here is the answer that lands the second signal.

Dimensionality questions probe linear-algebra intuition and practical judgment together. What interviewers reward is what PCA really does (project onto max-variance directions), why high dimensions hurt, and the honest catch: it is unsupervised, so it can discard the exact direction your label needs. Here is the answer.

EM is the classic latent-variable algorithm, and a GMM is how it appears in practice. What interviewers reward is the E-step/M-step alternation, why it is soft clustering where k-means is hard, and the honest caveat that it only reaches a local optimum. Here is the answer.

Backprop is the algorithm that makes deep learning trainable, and the interviewer wants the chain-rule mechanics, not just 'it computes gradients.' What interviewers reward is the forward-then-backward flow and why it is efficient. Here is the answer.

BatchNorm is one of the most-asked deep-learning questions, and the trap is the train/inference difference. What interviewers reward is what it normalizes, why it stabilizes and speeds training, and why it switches to running statistics at inference. Here is that answer.

This question connects why deep nets were hard to train with the cluster of tricks that solved it. What interviewers reward is the multiplicative-gradient cause and naming the real fixes: ReLU, He/Xavier init, residuals, normalization. Here is the answer.

CNNs remain foundational even as transformers rise, and this checks whether you grasp why convolution suits images. What interviewers reward is parameter sharing and local connectivity, what pooling buys, and how the receptive field grows. Here is the answer.

Patches as tokens, global attention from layer one, and a weaker inductive bias than a CNN. What interviewers reward is naming the data regime where each architecture wins and why. Here is the answer interviewers score highest.

Forward noising is fixed, reverse denoising is learned, and the training loss is a plain noise-prediction regression. What interviewers reward is why that beats a GAN's minimax and what the VAE and U-Net each do in latent space. Here is the answer.

Self-supervision is how modern models pretrain on unlabeled data, the engine behind LLMs and modern vision. The signal is the pretext-task idea and the genuine difference between contrastive and masked-prediction objectives.

Voice is a major modality and a common applied-AI surface. The signal is the audio-to-text pipeline, why Whisper is resilient, and the cumulative latency budget that can make or break a real-time voice agent.

Sequence models remain interview staples, particularly the gating that repaired RNNs and the reason attention took over. What matters is linking the vanishing-gradient narrative to the parallelism case that let transformers ride the scaling wave.

GANs shaped generative modeling for years, and interviewers look for the minimax game, the failure modes (mode collapse, instability), and the exact reason diffusion overtook them for images while GANs still lead on speed.

Autoencoders and VAEs sit beneath representation learning and generative modeling, including the VAE inside latent diffusion. What matters is the difference between a plain autoencoder and a variational one, and why sampling blocks backprop without the reparameterization trick.

Transfer learning is the reason you seldom train from scratch, and the real question is how much of the pretrained model to reuse versus adapt. What matters is a clean decision grid across data size and task similarity, plus knowing when a low LR guards against catastrophic forgetting.

The learning rate is the single hyperparameter that can NaN your run at step 50 or stall it indefinitely. Modern training schedules it rather than pinning it to one value. Interviewers want you to know precisely what warmup and decay each give you. Here is the answer.

A staple that tests whether you know why ReLU displaced sigmoid and how to pair the output activation with the loss. What matters is telling the vanishing-gradient story correctly. Here is the answer.

Everyone says 'gradient descent,' but the batch-size decision and momentum are the actual interview content. What earns credit is the noise-vs-cost tradeoff across batch/SGD/mini-batch and why momentum pays off.

The loss sets what the model optimizes, and the wrong choice silently sinks it. What counts is matching the loss to the task and data (outliers, imbalance), rather than reflexively reaching for MSE or cross-entropy.

Bad initialization makes deep nets fail to train at all, with activations that vanish or explode through depth. What matters is knowing why zero init is fatal and how to match Xavier or He to your activation function.

Dropout is the classic neural-net regularizer, but what matters is whether you can explain why zeroing activations forces redundancy, and the train-vs-inference scaling bug that trips most candidates. Here is the answer.

Data augmentation cheaply expands training data and regularizes models, but valid transforms differ by modality. What matters is the label-preserving constraint and why text is the hard one. Here is the answer.

Two inexpensive regularizers that cure overconfident classifiers. What proves you understand them is knowing that one softens the target while the other softens the input, and precisely why gentler signals produce calibrated models. Here is the answer.

'The model won't train' comes down to a short list of usual suspects, each with a distinctive loss-curve signature. The shape of the curve names the bug before you touch a single hyperparameter. Here is how to read it.

The notion that models, like students, learn better starting from easy examples is intuitive, and it sometimes works and sometimes does nothing. Knowing when it pays off is the real signal. Here is the honest answer.

Generative training breaks in distinctive ways: GANs collapse to a handful of outputs, diffusion samples come out noisy or blurry. The symptom points to which knob to turn. Here is the diagnosis.

Boxes and labels, not a single tag per image. The signal is picking two-stage versus one-stage from your latency and accuracy budget, knowing what NMS and anchors really do, and reporting mAP correctly. Here is the answer interviewers score highest.

Most CV systems break at the seams: a preprocessing mismatch between training and serving, not the model itself. The signal is naming every stage from ingestion through serving and the single consistency invariant that trips teams up. Here is the answer.

Cross-entropy is where you start, not where you finish. The signal is fitting the loss to the task structure: focal for detection's background flood, the IoU family for box overlap, Dice for imbalanced masks. Here is how to reason about it.

Semantic labels every pixel by class; instance pulls each object apart. The signal is knowing that the first cannot count overlapping objects, why Mask R-CNN adds a mask branch on RoIAlign, and what RoIAlign fixed. Here is the answer.

Most candidates can recite 'maximize the margin'. The dual is where you prove you really understand why only support vectors matter and where the kernel trick originates. Here is the derivation an interviewer wants.

A Gaussian mixture trained by EM has a well-known trap: one Gaussian shrinks onto a single point and the likelihood races to infinity. Understanding the cause, plus the three standard remedies, tells apart people who merely ran sklearn from those who grasp it.

Bayesian inference requires an intractable normalizing constant, and MCMC works around it. The signal is explaining the acceptance ratio, detailed balance, and why you can drop the constant entirely. Here is the answer.

The Kalman filter is optimal Bayesian tracking under linear-Gaussian assumptions, and it amounts to two steps repeated forever. The signal is explaining what the gain trades off and when the assumptions break. Here is the answer.

Raw gradient saliency maps are noisy and saturate. Integrated gradients cures both with two axioms and a path integral, yet the baseline choice quietly decides the answer. Here is what a careful candidate explains.

Anyone can say 'filter and dedup.' The signal is the funnel arranged by cost, the MinHash/LSH banding math, and recognizing that the shuffle across billions of documents is what actually runs up the bill, plus the benchmark decontamination people forget until their eval numbers get challenged.

Limited GPUs, dozens of models, and every team demanding low latency for little money. The signal is whether you can shape that into a single governed serving fleet: continuous batching, KV cache, per-tenant quotas, and cost you can genuinely attribute.

A GPU-heavy generative system where sampling steps drive the bill and a single bad image becomes a headline. The signal is the diffusion serving pipeline, the cost levers on per-image GPU time, and a required two-sided safety layer.

Transcribing in real time at scale probes streaming ASR, latency budgets, and GPU fleet management under heavy concurrency. The signal is chunked streaming inference with partial results, not batch transcription. Here is the design.

A GPU-heavy generative-audio system with text/genre/lyric conditioning, minutes-long coherence, and copyright constraints. The signal is the generation pipeline plus the async, cost, and safety layer that gets it shippable.

Video generation is image generation plus the harsh constraint of temporal consistency, and it is the most GPU-intensive generative task around. The signal is diffusion over spacetime plus a serving design that holds up under multi-minute jobs.

AV perception is safety-critical, real-time, multimodal ML: detect and track everything around the vehicle across multiple sensors. The signal is sensor fusion plus detection/tracking under a hard latency budget and a fail-safe bar. Here is the design.

Moving a cloud-API POC into a bank or defense network kills your model provider, your judge API, and your vendor dashboards in one stroke. Learn the signed-bundle supply chain, local evaluation loop, and zero-egress observability that make an air-gapped LLM stack actually operable.

The trap is leaping straight to 'add more GPUs.' The signal is profiling first, then reaching for the cheap, no-retrain levers in the correct sequence. Here is the diagnose-then-optimize playbook for p99.

At thousand-GPU scale, hardware failure is routine rather than rare, and a run that cannot resume burns weeks. The signal is checkpointing strategy, deterministic resume, and holding lost work to a minimum. Here is the fault-tolerant design.

Grid search is the wrong answer at scale. The signal is knowing why random beats grid, how Bayesian optimization and early-stopping schemes (Hyperband/ASHA) spend compute wisely, and how to parallelize without stragglers.

'Just set a seed' misses the point. Reproducibility means pinning the environment, the data, the code, and the hardware-level nondeterminism together. Here is the full checklist and where it springs leaks.

Four drift tests, four different assumptions. The weak answer lists them; the strong one knows which handles high-dimensional embeddings, which requires binning, and which returns a calibrated p-value. Here is how to choose.

A Deployment and a Service will serve a model, but GPUs upend every Kubernetes default: scheduling, probes, autoscaling signals, and rollouts. Here is the setup that survives production, and when KServe earns its complexity.

Interviewers here want concrete figures, not 'grab a bigger GPU.' Weight memory stays constant, the KV cache scales with traffic, and the order you pull levers in settles the outcome. This walks through the napkin math and the serving stack.

The interviewer is checking that you understand how a model too large for any one GPU still gets trained, and can run the optimizer-state memory math that justifies sharding. This covers the parallelism taxonomy and the 16-bytes-per-parameter figure.

Quantization is the opening move for shrinking and accelerating models, and the interviewer expects more than 'use fewer bits.' What they grade is whether you know what each precision level gains you, why outliers wreck naive quantization, and how you demonstrate quality survived. This lays that out.

A staple at hardware-aware teams. What they look for is grasping that attention's cost is memory traffic rather than FLOPs, and that FlashAttention is an exact, IO-aware reordering rather than an approximation. This is the answer that shows you reason about the memory hierarchy.

The KV cache is what makes LLM serving hard, and the interviewer wants the mechanics: what it holds, why it caps concurrency, and the tricks that shrink it. This is the answer that shows you grasp decode-time economics.

Decode runs sequentially and is memory-bound, so latency tricks count. What they grade is whether you can explain speculative decoding's draft-and-verify mechanism (and why it stays exact) along with the other levers and when each fits. This is the latency toolkit.

Mixed precision is routine at scale, and the interviewer wants the numerics: why FP16 needs loss scaling, why BF16 mostly does not, and what remains in FP32. What they grade is whether you understand dynamic range vs precision. This is that answer.

A hardware-literacy question, particularly at NVIDIA and the labs. What they grade is whether you understand throughput vs latency hardware, the memory hierarchy, and why matrix multiplication maps onto GPUs and TPUs. This is the architecture-aware answer.

Knowing the algorithms is only half the job. The rest is the serving stack that actually delivers throughput within a latency budget. What they grade is whether you match the server to the workload and name the four knobs that move the needle.

Distillation gives you a small fast model that retains most of a big model's quality. The signal is explaining why soft targets transfer more than hard labels, and knowing when to distill versus quantize versus prune. Here is the model-compression answer.

Kernel-level questions separate people who have written CUDA from people who have read about it. The signal is commanding the SIMT execution model and the three classic throughput killers, with a concrete fix for each. Here is the low-level answer.

Continuous batching is the biggest single throughput lever in modern LLM serving. The signal is explaining why static batching leaves the GPU idle on variable-length generation, and why PagedAttention is what makes the fix practical.

On-device AI is a genuine product surface (privacy, offline, latency), and it imposes hard constraints. The signal is the quantization plus format plus runtime stack and the tradeoffs you accept under tight memory and battery budgets.

Picking GPUs is a genuine applied decision, and 'get the biggest one' misses the point. The signal is knowing which spec binds your workload (VRAM, bandwidth, interconnect) and the train-versus-serve difference that flips the answer.

Autoscaling GPUs is nothing like autoscaling web servers: GPUs are scarce and expensive, model loading is slow, and the right signal is not CPU. The signal is scaling on queue and GPU metrics, taming cold starts, and the scale-to-zero economics.

Under load, a shared inference service has to decide whose request runs now. The signal is queuing with priorities, backpressure, and fairness wired into batching, not first-come-first-served until the service falls over.

FSDP and ZeRO let you train models too large for one GPU without splitting the compute. The signal is the ZeRO stages (what gets sharded at each level) and the communication you pay for it. Here is the answer.

Distributed training is bottlenecked by GPU-to-GPU communication, and these collectives are how the data travels. The signal is what each one does and which parallelism strategy relies on it. Here is the answer.

Activations, not just weights, can dominate training memory, and gradient checkpointing is the standard remedy. The signal is the precise trade: recompute activations in the backward pass rather than storing them. Here is the answer.

When a model is too large for one GPU you split the model itself, not just the data. The signal is telling tensor parallelism (split within a layer) apart from pipeline parallelism (split across layers) and matching each to the interconnect. Here is the answer.

LLM serving has its own metrics, and a single latency number hides the real bottleneck. The signal is separating prefill from decode and treating GPU utilization as a clue, not a verdict. Here is the diagnostic toolkit.

A PyTorch model in eager mode is not the fastest form for inference. The signal is knowing that ONNX buys portability while TensorRT-style compilers buy hardware-tuned speed, and precisely which optimizations get you there. Here is the answer.

LLM inference has two phases with opposite resource profiles, and co-locating them lets a long prompt stall everyone else's tokens. The signal is knowing why prefill and decode fight, and what separating them costs. Here is the answer.

These four get confused constantly, but they sit at different layers: data processing, distributed training, and general orchestration. The signal is fitting the tool to the workload instead of reaching for the one you know. Here is the answer.

Batch size is a training knob whose effects on speed, memory, and generalization are non-obvious. The signal is the large-batch tradeoffs and the learning-rate-scaling and gradient-accumulation tricks that make scaling actually work.

Almost nobody builds a serving stack by hand. What interviewers watch for is whether you know what frameworks give you (batching, multi-model, GPU scheduling) and why LLM-specific servers even exist when general ones batch already.

OOM is the wall you hit most often in deep learning, and 'buy a bigger GPU' is the weakest reply. What counts is naming the memory consumers, knowing which one dominates, and pairing the right lever with it.

nvidia-smi reading 100% can mask the fact that you are tapping only a fraction of the hardware's real compute. What matters is MFU (useful FLOPs vs peak) and the gap between 'the GPU is busy' and 'the GPU is efficient'.

Past FP16/BF16, FP8 sits on the next rung of the precision ladder, and the newest tensor cores compute on it natively. What counts is knowing the two formats, what FP8 really buys, and why dynamic range demands careful scaling.

At long context the KV cache, not the weights, dominates GPU memory and decode bandwidth. What counts is spotting it as the binding constraint and naming the levers that shrink it without wrecking quality.

Pruning strips redundant weights to make a model smaller and sometimes faster. The signal is the structured-versus-unstructured divide and why only structured pruning dependably cuts latency on commodity hardware.

Running one full fine-tuned model per customer grows GPU count linearly and drains the budget quickly. A method exists to pack hundreds of variants onto a single GPU without sacrificing batching. The interviewer wants to hear how.

CPU-based autoscaling that suits a web tier quietly breaks on GPU inference: the signal is wrong, and replicas need minutes to warm. The interviewer wants the signals you genuinely scale on and how you mask the cold start.

Paying for accelerators that sit idle half the time is the most common waste in ML training, and the reflex to add more GPUs only makes it worse. The interviewer wants the profiling discipline that pinpoints what is starving them.

The counterintuitive reality of LLM serving: token generation is capped by how quickly you can read weights out of memory, not by arithmetic. Once that clicks, the entire optimization menu follows from one number.

Linear scaling is the marketing figure; the actual curve bends early for reasons rooted in physics, not bugs. Here is where the speedup leaks and how to recover it.

A 256-GPU job that freezes with no error at 3am is a special kind of pain. The causes come from a short, recurring list. Here is the systematic way to identify which one hit you.

Naive 4-bit quantization can wreck quality, and the reflex to give up and serve fp16 leaves a large speedup unclaimed. The accuracy is usually recoverable. Here is the ladder of fixes.

Users experience the p99, not the median, and the tail is where serving systems quietly break. The causes are queuing and batching effects, not a slow model. Here is how to flatten it.

Pinning one GPU per model leaves most of a fleet stranded on idle silicon. Sharing safely is a genuine systems problem with four distinct mechanisms, each suited to a different traffic shape. Here is how to choose.

A serving box steady on short prompts falls over the instant a 30k-token request arrives, because KV-cache memory scales with sequence length times batch. Here is how to bound it without crashing.

Launching an LLM service on a guessed GPU count is how you get paged on day one. Capacity planning for LLMs differs from web services because tokens, not requests, are the unit. Here is how to size it.

One long prompt can freeze every other user's token stream for hundreds of milliseconds. Chunked prefill carves that prompt up so decode keeps flowing. Here is the mechanism and the knob that controls it.

A shared system prompt is re-prefilled on every request unless the server remembers it. Prefix caching skips that work, but only when the blocks align exactly. Here is the hashing and eviction machinery underneath.

Round-robin routing throws away a feature that doubles throughput: cache locality. Designing the router means picking a replica by KV state, not just least-loaded. Here is the two-layer design.

Three batching strategies, three very different latency profiles. Choosing wrong leaves throughput or tail latency on the floor. Here is what each one costs and when to use it.

GPU memory runs out well before you run out of conversations to cache. Offloading KV to CPU or NVMe buys capacity, but the interconnect can turn into the new bottleneck. Here is the bandwidth math.

FP8 serving is fast and usually fine, until it quietly degrades on the one workload your benchmark did not cover. What counts is knowing what FP8 breaks and how to validate it. Here is the recipe.

Forcing valid JSON sounds free, but the mask computation can stall every decode step. The fix is precompiling the grammar into a fast automaton. Here is where the cost hides and how to strip it out.

Most models do not fill a GPU, so one model per GPU wastes money. The three sharing mechanisms (MIG, MPS, swapping) differ widely in isolation and overhead. Here is how to pick.

Autoscaling reacts in minutes, but a burst lands in seconds, and the gap is where your tail latency dies. Absorbing bursts is about buffers and shedding, not just adding replicas. Here is the playbook.

No single parallelism dimension trains a 175B model by itself. The art is stacking three of them onto the right hardware so the expensive communication stays on the fast links. Here is how to size each axis.

ZeRO and FSDP both shard training state across data-parallel ranks, but the trick is in when each shard is gathered and freed. Get the memory accounting and the all-gather timing and you can size any run.

Full activation checkpointing saves memory but costs a flat 30% extra compute. Selective recomputation wins most of that back by recomputing only the cheap, memory-heavy operations. Here is how to pick what to recompute.

The collective communication in distributed training is pure overhead unless it runs while the GPU computes. Hiding it is the difference between 30% and 55% MFU. Here is how the overlap actually works and how you check it on a trace.

On a large training run a node will drop, and the interesting question is not whether but how many GPU-hours vanish when it does. Checkpoint cadence, sharded writes, and quick restart settle that.

When all-reduce is your bottleneck, the culprit is nearly always a misconfigured path: traffic on the wrong link, a dead NIC, or a topology NCCL failed to discover. Here is the methodical way to track down the missing bandwidth.

FP8 can nearly double training throughput over BF16, but with only a few mantissa bits the numerics leave no slack. Per-tensor scaling and a selective recipe are what get it to converge.

Data, tensor, and pipeline parallelism each leave a single sequence's activations on one device, so training past 200k tokens hits a wall none of them can clear. The fourth axis shards the sequence itself, and the interview turns on the communication math.

When each request thinks for 30,000 tokens, serving swings from compute-bound prefill to memory-bound decode, and the KV cache turns into the resource you genuinely schedule. The levers that governed chat traffic stop being the ones that count.

Most answers optimize the training run. For any model that gets used, inference dominates lifetime energy, and the biggest carbon lever is the serving stack. Measure first, then cut the thing that is actually large.

Anyone can list the engines. The signal is naming the one mechanism that distinguishes each (paged KV blocks, a radix prefix tree, ahead-of-time kernel compilation, quantized CPU inference) and picking from your traffic shape rather than a leaderboard.

Raw tokens per second is gameable: crank the batch size and the dashboard looks great while every request misses its latency target. Goodput is the throughput that actually meets your SLOs, and it is the number you size and autoscale on.

An image is not one input, it is a large prompt. Once you internalize that image tokens dominate prefill, the cost levers stop being model choices and start being resolution, tiling, and cache placement.

Sampling cost factors cleanly into steps, forward passes per step, and cost per step, and each factor has a different lever. Two of the biggest wins are free, and most teams take neither.

Federated learning trains across decentralized devices without pooling data, yet a malicious participant can poison the shared model. What lands is the privacy mechanism plus the outlier-resistant aggregation defenses. Here is the answer.

Encryption protects data at rest and in transit, but the instant you compute on it the data sits as cleartext in memory. Confidential computing closes that gap. The signal is the TEE concept, attestation, and the AI workloads that genuinely need it.

Federated learning transmits gradients, not raw data. The catch: a gradient is derived from the data, so it carries the data. The signal is knowing why 'we only share gradients' is not a privacy guarantee. Here is the answer.

A backdoored model acts perfectly until it meets a secret trigger, then flips. The signal is explaining why clean test accuracy never exposes it, and what detection actually buys you when the trigger is unknown.

Federated learning keeps data on-device, but you pay for it in accuracy, debuggability, and engineering complexity. The signal is naming when those costs are justified and when a simpler centralized pipeline with DP wins. Here is the answer.

Picking epsilon is the easy half. The hard half is the accuracy you just gave up, and the levers that get it back are not the ones most candidates reach for.

This is an entire interview round (OpenAI's project deep dive, Amazon's Tech Talk), not small talk. It is where they check whether you genuinely understand the work you claim. The signal is depth, your specific contribution, and the tradeoffs you can defend. Here is how to prepare and deliver it.

AI moves fast enough that learning speed is itself a core competency, and several companies ask about it directly. The signal is a concrete method for ramping fast plus a real, recent example of putting something new to use. Here is how to show learning agility rather than merely claim it.

Applied AI rarely waits for certainty, and companies (Amazon's Bias for Action, NVIDIA's speed) check whether you can act decisively without freezing or turning reckless. The signal is a reversible, well-reasoned call paired with a plan to validate. Here is the arc that connects.

At the frontier labs this is a genuine gate, not a pleasantry. They turn away strong engineers who cannot articulate real motivation. The signal is specific, honest alignment between what you want to build and what this company actually does.