preference optimization
AI, ML & GenAI interview questions tagged preference optimization, across every topic.
3 questions · 1 unlocked for you
Concepts behind "preference optimization"
The curriculum that explains the ideas these questions test.
Core
RLHF: Reinforcement Learning from Human FeedbackRLHF is how a raw next-token predictor turns into a helpful, harmless assistant. It runs in three stages: supervised fine-tuning on demonstrations, training a reward model on human preference comparisons, then optimizing the model against that reward (with a KL penalty to stay close to the base). It aligns the model to human preferences that resist specification as a loss. AI, ML, and GenAI engineer interviews probe it because it explains why instruct models behave well, where alignment data comes from, and the failure modes (reward hacking, sycophancy).🧠 Foundations of LLMs & GenAISign in
Core
Reward ModelsA reward model converts human preference comparisons into a scalar score for any response, the very signal RLHF chases. Trained on response pairs labeled by which one a human favored, it learns to rank instead of to write text. Its flaws are behind RLHF's failure modes: reward hacking (gaming the proxy) and going stale once the policy drifts off-distribution. AI, ML, and GenAI interviews probe it because it shows where the alignment signal originates and why it is exploitable, and the same idea carries over to LLM-as-judge evaluation.🧠 Foundations of LLMs & GenAISign in
Core
DPO and Preference-Optimization VariantsDirect Preference Optimization aligns a model straight from preference pairs with a simple classification-style loss, bypassing RLHF's separate reward model and RL loop, which makes alignment far simpler and more stable. A family of variants then loosens DPO's requirements: SimPO drops the reference model, KTO drops the need for paired data, and ORPO folds SFT and alignment into one step. AI, ML, and GenAI interviews probe it because DPO is now the common way teams align open models, and the variants show you understand what each requirement buys.🧠 Foundations of LLMs & GenAISign in