alignment
AI, ML & GenAI interview questions tagged alignment, across every topic.
15 questions · 1 unlocked for you
Concepts behind "alignment"
The curriculum that explains the ideas these questions test.
Foundational
What an LLM Is: Next-Token Prediction and the Training PipelineAn LLM is a function that maps a sequence of tokens to a probability distribution over the next token, called in a loop. Three stages turn that function into an assistant: pretraining on a huge corpus buys knowledge and fluency, supervised fine-tuning teaches it to answer rather than continue, and preference alignment teaches it which answer a human prefers. AI, ML, and GenAI engineer interviews probe this because capability comes from pretraining while behavior comes from post-training, and almost every production complaint is a behavior complaint.🧠 Foundations of LLMs & GenAI
Core
RLHF: Reinforcement Learning from Human FeedbackRLHF is how a raw next-token predictor turns into a helpful, harmless assistant. It runs in three stages: supervised fine-tuning on demonstrations, training a reward model on human preference comparisons, then optimizing the model against that reward (with a KL penalty to stay close to the base). It aligns the model to human preferences that resist specification as a loss. AI, ML, and GenAI engineer interviews probe it because it explains why instruct models behave well, where alignment data comes from, and the failure modes (reward hacking, sycophancy).🧠 Foundations of LLMs & GenAISign in
Core
Reward ModelsA reward model converts human preference comparisons into a scalar score for any response, the very signal RLHF chases. Trained on response pairs labeled by which one a human favored, it learns to rank instead of to write text. Its flaws are behind RLHF's failure modes: reward hacking (gaming the proxy) and going stale once the policy drifts off-distribution. AI, ML, and GenAI interviews probe it because it shows where the alignment signal originates and why it is exploitable, and the same idea carries over to LLM-as-judge evaluation.🧠 Foundations of LLMs & GenAISign in
Core
Constitutional AI and RLAIFRLAIF (RL from AI Feedback) swaps human preference labels for AI-generated ones, pushing alignment past the human-labeling bottleneck. Constitutional AI is Anthropic's particular version: the model critiques and revises its own outputs against a written set of principles (a constitution), producing the preference data from those principles. The upside is scalability, consistency, and explicit, editable values; the downside is the AI judge's own biases. AI, ML, and GenAI interviews probe it because it is how alignment scales and how values become explicit and auditable.🧠 Foundations of LLMs & GenAISign in
Core
DPO and Preference-Optimization VariantsDirect Preference Optimization aligns a model straight from preference pairs with a simple classification-style loss, bypassing RLHF's separate reward model and RL loop, which makes alignment far simpler and more stable. A family of variants then loosens DPO's requirements: SimPO drops the reference model, KTO drops the need for paired data, and ORPO folds SFT and alignment into one step. AI, ML, and GenAI interviews probe it because DPO is now the common way teams align open models, and the variants show you understand what each requirement buys.🧠 Foundations of LLMs & GenAISign in
Advanced
Policy Optimization: PPO and GRPOPPO and GRPO are the reinforcement-learning algorithms that optimize an LLM against a reward, the RL step in RLHF and in training reasoning models. PPO is the established workhorse, nudging the policy in small, clipped steps to stay stable; GRPO (used by DeepSeek-R1) removes PPO's separate value network and instead normalizes rewards within a group of samples, which is simpler and cheaper for LLMs. AI, ML, and GenAI interviews probe it because it explains how alignment and reasoning training actually run, and why RL on verifiable rewards scales.🧠 Foundations of LLMs & GenAI🔒 Premium