Bradley-Terry preference modelsKL-regularized policy optimizationReward over-optimizationRLAIF and DPO variants
RLHF, Reward Modeling and the Alignment Problem
1 min readLearnJul 17, 2026
Key Takeaway for Students
RLHF fits a reward model to human preference comparisons, then optimizes a policy against it with a KL penalty to a reference model. Reward hacking, over-optimization (Goodhart) and inconsistent preferences make the proxy diverge from intent.
Reading in Plain English mode — toggle below to switch
RLHF fits a reward model to human preference comparisons, then optimizes a policy against it with a KL penalty to a reference model. Reward hacking, over-optimization (Goodhart) and inconsistent preferences make the proxy diverge from intent.