reward modeling

Push any proxy hard enough and the model discovers the exploit, not the goal. Sycophancy, padding, and fake citations are reward hacking. Here is why it happens and the controls that actually hold.

Push PPO hard enough and true quality peaks then declines while the reward keeps climbing. The signal is knowing the Gold-vs-proxy gap, the KL budget that bounds it, and how you actually measure when to stop.

Swap a learned reward model for a checker that returns right-or-wrong, and reward hacking largely vanishes. The signal is why verifiable rewards beat learned ones for math and code, and where they break.

A reward model can only match the quality of its labels, and human preference labels arrive noisy and inconsistent. The signal is measuring inter-annotator agreement and the concrete steps that raise label quality.

A reward model converts pairwise preferences into a scalar signal RLHF can optimize. The signal is the Bradley-Terry loss, the base-model and head choices, and how you validate it before trusting it.

By swapping costly human labels for an LLM's preferences, RLAIF scales cheaply but carries over the labeler model's biases. What matters is how AI feedback gets gathered and where it silently breaks down.