LLM & GenAI Fundamentals

Nearly every candidate can recite softmax(QKᵀ/√d_k)V. What the interviewer is after is the variance derivation and the precise training failure the scale heads off. This is the response that tells rote recall apart from real grasp.

The most frequently asked applied GenAI question, and the one most candidates turn into a definition dump. What the interviewer wants is a decision framework with a default and the conditions that override it. Here is the call that earns points.

Alignment has become a baseline expectation even beyond 'safety' roles. The interviewer is looking for the three-stage RLHF pipeline and a sharp account of why DPO threw out the reward model completely. Here is the answer that leads with mechanism, not the buzzwords.

A tidy fundamentals question that trips up anyone who only knows GPT. The signal is pairing each architecture to its task through its attention pattern, and explaining why the field settled on decoder-only even for tasks that are not generation. Here is that answer.

Every customer wants to know how to stop the model inventing things. The weak reply is 'better prompts.' The signal is understanding why models hallucinate and stacking grounding, abstention, and measurement into a system you can defend.

Tokenization reads like plumbing until it accounts for your bill, your latency, and why the model stumbles on numbers and rare languages. The signal is understanding how subword tokenizers operate and the consequences that follow from them.

MoE explains why some frontier models carry huge parameter counts yet serve cheaply. The signal is the sparse-activation idea, the routing mechanism, and the operational cost (memory, load balancing) that the FLOP savings quietly conceal.

Attention is permutation-invariant, so with no position information a transformer cannot detect word order. The signal is knowing why, and why the field shifted from absolute sinusoidal encodings to relative ones like RoPE and ALiBi that extrapolate to longer contexts.

Every LLM product tunes these knobs, and the interviewer wants to hear what each one does to the probability distribution and which setting suits which task. That includes why factual and creative tasks call for opposite settings.

Evaluation is the toughest and most underrated part of shipping LLMs. The signal is understanding why public benchmarks mislead, why LLM-as-judge carries bias, and how to build a task-specific eval you can genuinely trust.

Embeddings are the foundation of modern NLP and retrieval. This tests whether you grasp the distributional idea behind them, not just 'words become vectors.' What matters is how training on context yields geometry that encodes meaning.

Perplexity is the classic LM metric. What matters is knowing it is just exponentiated cross-entropy, what it intuitively captures, and why low perplexity does not mean a good product.

PEFT is how everyone fine-tunes large models now. What matters is the low-rank insight behind LoRA, why it cuts memory so sharply, and what 4-bit QLoRA adds. Here is the answer that goes past 'it's efficient fine-tuning.'

In-context learning is the property that turned prompting into a paradigm; chain-of-thought is its most useful trick. What matters is knowing what 'learning' means here (no weight updates) and why making the model reason step by step actually helps. Here is the answer.

Two prompting strategies that purchase accuracy with extra compute. What matters is knowing how each explores the reasoning space, that they trade tokens and latency for reliability, and how to judge when the problem actually warrants it. Here is that answer.

ReAct is the pattern behind most tool-using agents. What matters is why interleaving reasoning with actions (and observing results) beats reasoning alone or acting alone. Here is the answer that ties the prompt pattern to real agent behavior.

RL alignment shifted from PPO to leaner methods, and DeepSeek-R1 put GRPO on the map. What matters is knowing what the value/critic model does in PPO and how GRPO replaces it. Here is the mechanism, not the acronyms.

The 2025-2026 landscape isn't 'bigger is better' anymore. What matters is knowing why small models and reasoning models exist, the test-time-compute tradeoff, and routing the right model to the right task. Here is the practitioner's view.

What separates a demo prompt from a production prompt is structure, versioning, and defense. What matters is treating the system prompt as a contract and templates as parameterized, testable artifacts. Here is the answer.

Complex tasks rarely fit one prompt, and free-text LLM output rarely drops into code cleanly. What matters is decomposing into chained steps and parsing each output. Here is the answer that ties prompting to real pipelines.

Jailbreaking is the adversarial sport of steering a model around its own safety training. What matters is naming the technique families and treating defense as a layered, never-finished arms race, not a one-time patch.

Instruction tuning converts a bare next-token predictor into a model that obeys instructions. What you are being tested on is slotting it correctly into the pretrain to SFT to alignment pipeline and stating precisely what it does and does not repair.

Fine-tune a model on your domain and it may lose the ability to do everything else. What earns the signal is explaining why shared weights cause it and listing the concrete mitigations that all boil down to one principle.

PEFT is a family, not only LoRA, and interviewers test whether you know how each one injects trainable parameters. The signal is where each method adds capacity and the latency and quality tradeoffs that result.

The classic automatic metrics for generated text. Interviewers want you to pair each with its task and identify the blind spot common to all three. Candidates who pass explain why none of them is the last word on quality.

Plenty of people cite benchmark numbers; almost nobody reads them carefully. What shows depth is explaining what each one measures and why leaderboard figures outrun real-world skill. The question the interviewer is saving is how you would spot contamination.

Fine-tuning lives or dies on data, not on hyperparameters. What shows depth is spelling out what makes a set good (quality, diversity, format, dedup) and defending why a few thousand clean examples outperform a million noisy ones.

Synthetic data is how teams manufacture training examples when real data runs short, and it comes with sharp failure modes. What shows depth is pinning down when it helps, how to keep quality up, and why training on model output over successive generations narrows the distribution.

LLM-as-judge is how you grade open-ended generation at scale, and G-Eval put a formal frame on it. What shows depth is naming the biases that make raw judge scores untrustworthy and demonstrating exactly how you harden them.

Offline metrics gate releases; online metrics reveal what actually happened with users. What shows depth is understanding why offline wins routinely fail to hold up online, and how the two feed each other.

'Eval is the new system design' for LLM apps, and starting with zero data is the real test. What shows depth is bootstrapping a golden set, choosing task-specific metrics, and running evals continuously as a regression gate.

Cross-attention is how a model attends from one sequence into another, the mechanism behind encoder-decoder models and multimodal conditioning. What shows depth is knowing exactly where the queries versus keys/values come from.

Attention gets all the attention, but the feed-forward layers hold most of a transformer's weights and handle much of the per-token 'knowledge' work. What shows depth is knowing what the FFN computes and why it dominates the parameter count.

Context windows are not arbitrary figures; they follow from attention's quadratic cost, KV-cache memory, and how the model was trained. What shows depth is naming all three causes and matching each to the technique that extends it.

Full attention is O(n^2), so long context demands cheaper attention. What shows depth is naming the families (sparse, sliding-window, linear), what each surrenders, and why FlashAttention is not one of them.

This is an architecture and business decision, not a religious one. What shows depth is weighing capability, cost, privacy, control, and operational burden, then committing to a call instead of declaring 'open is always better.'

RLAIF and Constitutional AI trade human preference labels for AI-generated ones to scale alignment. What shows depth is naming exactly what they substitute for human feedback, and the consistency-versus-bias tradeoff that swap introduces.

Reasoning models are the 2025-2026 frontier. The signal is understanding they are trained (largely via RL on verifiable rewards) to generate long internal reasoning, and that they scale along a second axis: spending more compute at inference.

Tool calling powers every agent. The signal is grasping that the model executes nothing: it emits a structured request your code runs, then you hand the result back. Here is the loop and the details that determine reliability.

Modern LLMs abandoned learned absolute positions for RoPE and ALiBi to cope with long context. The signal is recognizing that both encode RELATIVE position, and precisely how that permits length extrapolation past the trained window.

MQA and GQA shrink the KV cache, the thing that bottlenecks LLM serving. The signal is understanding they share key/value heads across query heads to trim memory and bandwidth, with GQA as the quality-preserving middle ground. Here is the answer.

DPO simplified RLHF, and a family of variants now trade off what data and reference model they require. The signal is knowing what each variant drops or alters (reference model, paired data, separate SFT stage). Here is the answer.

Scaling laws explain why bigger models trained on more data predictably improve, and Chinchilla changed how we allocate compute. The signal is the model-size-vs-data tradeoff and the fact that many models were under-trained. Here is the answer.

How you decode from the model's probabilities shapes the output as much as the model itself. The signal is knowing the deterministic vs sampling methods and where each fits (factual vs creative). Here is the answer.

BERT and GPT diverge on one decision made before a single weight is trained: what does each token get to see? That choice ripples into attention, use case, and why one family now dominates. Here is the answer.

Fine-tuning a 65B model once demanded a node of A100s. QLoRA compresses it onto one card with three specific tricks. The signal is knowing what gets quantized, what stays trainable, and why quality barely shifts. Here is the answer.

Prompting for JSON works until it doesn't, and at scale the tail breaks your parser. Constrained decoding makes malformed output literally impossible to sample. The signal is knowing exactly where in the loop the constraint takes hold. Here is the answer.

Multilingual models are uneven across languages, and most candidates cite only the obvious cause. The differentiator is the tokenizer: the same sentence can cost a non-English user several times as many tokens. Here is the answer.

Fuse several fine-tunes into one model by doing arithmetic on their weights, with no retraining and no data. The signal is knowing why it works and which method to pick when merges interfere.

A transformer emits one vector per token, but retrieval needs one vector per text. The signal is knowing the pooling options and the catch nearly everyone overlooks: the model has to be trained for whichever one you pick.

Chain-of-thought looks like the model showing its work, but it may not mirror the real computation. The signal is knowing CoT can be a post-hoc story, with evidence, and what that breaks for safety and oversight.

Preventing hallucinations is one job; catching the ones that slip through at runtime is a separate one. The signal is the detection toolkit, why each signal is imperfect, and how you merge them into an action.

Both are conditional compute, but one scales parameters and the other scales depth. The signal is knowing which axis each routes along and why that shifts the FLOP story. Here is the sharp version.

RAG and fine-tuning get cast as either/or. RAFT trains the model to be good at RAG itself. The tell is what goes into the training set: relevant docs mixed with distractors. Here is the answer.

Hand-tuned prompts break on the next model upgrade and fail to compose across pipeline stages. Frameworks like DSPy compile prompts against a metric instead. The signal is treating prompting as an optimized program. Here is the answer.

A model that claims a million-token window may not actually exploit it. The signal is knowing needle-in-a-haystack, where it fails, and how to probe genuine multi-fact reasoning across the entire context. Here is the answer.

Distillation gives you most of a frontier model's quality for a fraction of the serving cost. The signal is naming the three LLM-specific variants and knowing which one holds up against a closed API. Here is the answer.

The senior signal is fitting the method to what is genuinely missing rather than reaching for your favorite. Knowledge, behavior, or actions each steer you to a different tool. Here is the decision framework that scores highest.

A single trained embedding you can cut to any length and still use. The signal is the nested-prefix training objective and the coarse-to-fine retrieval win it enables at scale. Here is the answer.

Models are trained to be helpful, which quietly teaches them to never refuse. Restoring honest abstention is a system problem, not a prompt tweak. Here is the stack that actually shifts the refuse-when-unsure rate.

Token count is latency and dollars, not merely style. 'Be concise' hardly works, and max_tokens just cuts off mid-sentence. Here is how to genuinely shape length without amputating answers.

Alignment that piles on refusals and politeness often quietly erodes reasoning and coding. The alignment tax is real and measurable. Here is how to hold onto the safety gains without paying for them in capability.

Push any proxy hard enough and the model discovers the exploit, not the goal. Sycophancy, padding, and fake citations are reward hacking. Here is why it happens and the controls that actually hold.

Few-shot accuracy can swing on example order by itself. If your prompt is fragile to things that shouldn't matter, the fix is structural, not lucky example-hunting. Here is what genuinely reduces variance.

The model's training cutoff is frozen, but your dependencies shift weekly. Retraining is the wrong reflex. Here is how teams keep code suggestions current without touching the weights.

Neural text degeneration is a known failure of likelihood-maximizing decoding, not a broken model. The fixes are specific decode-time knobs. Here is which lever to reach for and what each one costs.

The per-token sticker price masks the real decision. Self-hosting only pays off past a volume break-even most teams misjudge. Here is the back-of-envelope a staff engineer does on the whiteboard.

Generation has to end somewhere, and the wrong stop rule either truncates answers or wastes tokens on trailing garbage. Here is how the EOS token, stop strings, and max-token caps actually interact.

CoT is not a blanket upgrade. On some tasks it does nothing or backfires, and the reason points to what to try instead. Here is the escalation ladder past 'let's think step by step'.

When a drug name or a Hindi word splits into nine subword tokens, you pay in cost, context, and accuracy. Adding tokens is easy and adding them well is not. Here is the tradeoff.

A model that scores 92% on your eval can still ship a product users hate, because the model is one component in a system. Teams that run only model evals get blindsided. Here is the distinction that matters.

For open-ended generation there's no ground-truth string to match, so accuracy is meaningless. The field relies on a layered mix of automatic, model-based, and human metrics. Here is how to assemble a credible eval.

A few thousand carefully chosen examples can beat a million scraped ones. The signal is knowing which filters matter, how you gauge example quality, and why diversity beats raw volume.

Push PPO hard enough and true quality peaks then declines while the reward keeps climbing. The signal is knowing the Gold-vs-proxy gap, the KL budget that bounds it, and how you actually measure when to stop.

GRPO drops the value model and estimates advantage from a group of samples per prompt. The signal is the exact mechanics, why the group baseline works, and the normalization traps that silently bias training.

DPO is simple and stable, but it can drive down the probability of chosen responses, overfit the preference margin, and magnify length bias. The signal is naming these and the variants that fix each.

Swap a learned reward model for a checker that returns right-or-wrong, and reward hacking largely vanishes. The signal is why verifiable rewards beat learned ones for math and code, and where they break.

Distillation is not a single method. The signal is knowing when to match logits versus train on generated text, why on-policy distillation beats off-policy, and how reasoning models are distilled.

A reward model can only match the quality of its labels, and human preference labels arrive noisy and inconsistent. The signal is measuring inter-annotator agreement and the concrete steps that raise label quality.

The proportions of web, code, books, and math in pretraining quietly set downstream skills. The signal is knowing how mixtures get chosen, why upsampling helps, and how methods like DoReMi automate it.

Continued pretraining adds domain knowledge that fine-tuning cannot, yet done carelessly it wrecks general ability. The signal is the replay ratio, learning-rate rewarming, and how you measure forgetting.

Pairwise asks which one wins, pointwise asks how good a single output is. The signal is knowing which is more reliable, how Elo/Bradley-Terry aggregates comparisons, and where each method quietly misleads.

When a benchmark leaks into pretraining, the score measures memorization rather than ability. The signal is the detection methods (n-gram overlap, canaries, perturbation tests) and why fresh held-out evals matter.

The same model can land ten points apart on MMLU depending on prompt format and scoring method. The signal is knowing the knobs (log-prob vs generation, few-shot, normalization) that make evals reproducible.

A reward model converts pairwise preferences into a scalar signal RLHF can optimize. The signal is the Bradley-Terry loss, the base-model and head choices, and how you validate it before trusting it.

By swapping costly human labels for an LLM's preferences, RLAIF scales cheaply but carries over the labeler model's biases. What matters is how AI feedback gets gathered and where it silently breaks down.

MQA and GQA cut the KV cache by sharing key/value heads. MLA goes another way: compress K and V into a low-rank latent and cache that instead. What matters is recognizing it as a cache trick rather than a head-sharing trick, and why it preserves quality.

Evict the oldest tokens to cap the KV cache and the model falls apart. Retain only the first few tokens and it keeps generating forever. What matters is grasping why those opening tokens behave as an attention sink.

A model trained at 4K tokens can reach 128K without a full retrain. What matters is understanding why naive extrapolation breaks and how PI, NTK-aware scaling, and YaRN each rescale RoPE frequencies in different ways.

At long context it is the KV cache, not the weights, that saturates the GPU. What matters is naming the levers (quantization, token eviction, head sharing, low-rank) and the quality cost of each.

In naive LLM serving most GPU memory bleeds away to KV-cache fragmentation. PagedAttention adapts virtual memory paging to win it back. What matters is explaining the fragmentation problem and how blocks resolve it.

Speculative decoding accelerates generation, and with the right verification rule it leaves the output distribution untouched. What matters is understanding how draft-model, self-drafting (Medusa/EAGLE), and lookahead approaches differ in where the guesses originate, and which of them are still exact.

Both compress an LLM to 4-bit weights, yet they choose what to protect in very different ways. What matters is GPTQ's error-correcting solve versus AWQ's activation-aware scaling, and the calibration each demands.

GGUF powers most local LLM runs, and its quant names look cryptic. What matters is decoding what Q4_K_M means and why mixed-precision k-quants outperform naive uniform quantization.

FP8 underpins modern H100-class training and serving. What matters is knowing the two FP8 variants, why a floating format beats INT8 on dynamic range, and where scaling still counts.

MoE trims compute but is awkward to serve: experts have to be sharded, tokens routed across devices, and batches balanced. What matters is the all-to-all communication and the load-imbalance problem, not the training story.

Candidates who cite one leaderboard number fail the follow-up. The interviewer wants you to name what SWE-bench measures, what it cannot (long-horizon multi-file refactors, real tool use, business logic), and which complementary benchmarks close those gaps.

Grammar-constrained decoding seems like a solved problem until tokenization stops lining up with the grammar. The signal is the token-vs-character boundary mismatch and how prefix automata and token healing address it.

VLMs such as GPT-4V and LLaVA attach a vision encoder to a language model through a projector. The signal is the image-tokens-as-prefix design, the alignment training, and the resolution and hallucination failure modes.

Speech LLMs either transcribe to text or model audio straight away as discrete tokens. The signal is the semantic-vs-acoustic token split and why end-to-end audio models outperform ASR-plus-LLM pipelines on latency and prosody.

Modern embedding models expect a task instruction prepended to the text, and a wrong prefix quietly wrecks retrieval. The signal is why instructions help and the asymmetric query/passage convention.

Temperature 0 is not the same as deterministic, and the reason lives in the GPU kernels, not the sampler. What gets scored is naming the batch-invariance problem and knowing which fixes are real versus placebo.

Autoregressive models commit one token at a time, left to right. Diffusion language models generate the whole sequence at once and refine it over several denoising steps, which buys parallel decoding and global editing but has not yet caught the frontier. The signal is knowing the tradeoff, not just the buzzword.

Text prompts are a one-dimensional control surface, and cranking the guidance scale until the model obeys just burns your image. The candidates who pass name the right conditioning mechanism for the right kind of control.

Everyone can name learning rate and epochs. The signal is knowing that LoRA wants a learning rate roughly 10x higher than full fine-tuning, why alpha/r is the only scaling that matters, and which curve tells you your rank is too big.

Your assistant scores 4.6 out of 5 on every individual turn and still loses the user by turn eight. Per-turn averages hide the failures that actually cancel subscriptions, and here is the session-level eval that catches them.

Most teams reach for a token-dropping compressor and pay an accuracy tax for tokens they could have had for almost nothing. The ranking of levers matters more than any single technique, and the top of the list is not compression at all.

A paraphrased prompt comes back with a training answer, verbatim. This is small-dataset SFT overfitting, not the pretraining memorization problem people confuse it with, and the fix has a strict order of operations.

Appending every turn is the default in tutorial code and it fails twice: on the context window and on cost. The signal is the layered assembly, the constraints you refuse to let the summarizer touch, and what you do the moment the user changes subject.

Four fusion families, and the choice between them is decided by one question most candidates never ask. Naming CLIP and LLaVA is table stakes; the score comes from knowing which one cannot reason and which one eats your context window.

The instinct is to blame the vision encoder. It is a token-budget problem, and the fix is to retrieve pages before you look at them. What separates a strong answer is knowing which question types page retrieval quietly cannot answer.

An emerging alternative to stuffing the window or building an index: keep the context outside the prompt and let the model write code to inspect it, recursing into the pieces. Knowing when it does not pay is most of the score.

Every LLM is one function: tokens in, a distribution over the next token out. The signal is what you say next, because the distance from that function to a useful assistant is three training stages, and almost every production complaint lands in exactly one of them.

The shift from one model per task to one model adapted many ways rewrote the job description. The interviewer is checking whether you know which half of the work got nearly free, which half did not move at all, and which problems classical ML still owns outright.

The whiteboard question that spans the entire stack, from BPE to the sampler. What separates a strong answer is knowing that prefill and decode stress opposite hardware resources, and that nearly every serving decision you will ever make falls out of that one fact.

One triangle of negative infinity, added before the softmax, is what makes transformer training parallel and generation stubbornly serial. Interviewers want the mechanism and the asymmetry it creates, not a definition of attention read back to them.

Two defaults that every Llama-class model shares and almost no candidate can justify. One is a compute win that turned out to cost nothing in quality; the other is what makes a 60-layer stack converge at all.

Most candidates answer with the 2015 ResNet answer and stop. The senior answer treats the residual stream as a shared communication bus, and that reframing is what makes logit attribution, the logit lens, and activation steering possible at all.

Definitions take thirty seconds. The interview is about the four things log-probs buy you in production: a confidence signal, banning tokens, guaranteeing valid output, and classification you can calibrate. Plus the tokenization gotcha that quietly breaks all of them.

Both work, so the decision is made by four numbers: how many examples you have, how far the capability has to move, how many variants you must serve, and how many GPUs you own. The serving axis is the one candidates never mention.

The follow-up every interviewer holds in reserve after you say you fine-tuned something. A task-metric win proves almost nothing on its own. Here is the five-part protocol that separates a shipped model from a demo.

Merging one adapter is a single matrix add. Merging four is where careers get interesting: the low-rank updates were each trained as if the others did not exist, and they collide. Here is what actually works and when to route instead.

The staged recipe is the easy half. The half that sinks real projects is data licensing, an evaluation bar that needs actual domain experts, and knowing which parts of the problem fine-tuning cannot solve at all.

Most staleness complaints are not model problems, and the engineer who says so first wins the round. Here is which changes belong in the index, which belong in the weights, and the lifecycle machinery that keeps months of refreshes from eroding the model.

Summarization is the one task where the model was handed everything it needed and still made something up. The signal is separating intrinsic from extrinsic hallucination, knowing why ROUGE cannot see either, and gating on entailment before you ship.

Truncate, map-reduce, refine, retrieve, or stuff a long-context model. Each buys you something different and breaks differently. The candidates who score are the ones who pick based on the question being asked, not the document being read.

A one-word rewrite moving your accuracy several points is not a prompting problem, it is an uncalibrated decision function. The fix is measuring the model's prior over labels and dividing it out, not another round of tinkering.

Three different systems are failing at once (the tokenizer, the model, and your pipeline) and most candidates blur them into one. Separating them tells you which fix buys the most, and the highest-leverage one is not the prompt.

Letting a model write your prompts works, right up until the optimizer memorizes your eval set and hands you an unreadable prompt that scores better and generalizes worse. The bounds are the whole answer.

The most repeated trick in prompt engineering, and the evidence for it is much thinner than the folklore. A screening question for whether you measure your prompts or inherit them from a thread of screenshots.

Every request pays for the prompt and almost nobody has looked at where the tokens actually go. It is rarely the part you can read: thirty tool schemas and seven stale examples usually outweigh the system prompt, and the output side costs several times more per token.

Naming RLHF is not an answer. The frame that scores is outer versus inner alignment, why Goodhart breaks any proxy objective you optimize hard enough, and why a classifier sitting at the boundary can constrain a model's behavior without changing what it is trying to do.

Video breaks a VLM on arithmetic before it breaks it on vision: one frame per second of a ten-minute clip is hundreds of thousands of image tokens. Frame selection is not preprocessing, it is the architecture, and the strongest systems index the audio first.

A VLM is three parts and the interview is entirely about which ones you train. The staged recipe, why the vision encoder almost always stays frozen, and the silent failure where your model learns to answer from the text prior and never looks at the image.

Most candidates answer 'use the reasoning model for hard problems' and stop. The interviewer wants it framed as an eval and a budget problem: how you prove the extra thinking tokens paid off, and what you do when the model talks itself out of a correct answer.