AI Research

Score Centering Stabilizes Off-policy Reinforcement Learning | AI Learning Platform

Inference OptimizationQuantizationCost OptimizationRLHF

Score Centering Stabilizes Off-policy Reinforcement Learning

Martin Marek, Max Ryabinin1 min readarXivSep 17, 2026

Key Takeaway for Students

Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily

30B parameteroutperforms methods based on importance sampling under quantization, woutperforms pure importance-sampling baselines in our staleness experi

Reading in Plain English mode — toggle below to switch

This paper addresses a critical challenge in applying RL to large language models, the training-inference mismatch (TIM), which often leads to instability and inefficiency. By proposing 'score centering' to mitigate drift, it offers a practical solution to stabilize RL training, potentially reducing computational costs and improving model performance in real-world deployments.

Read original on arXiv

Career Insight

high demand

Why it matters

Skills to learn

Implement RLHF algorithms (e.g., PPO, DPO)Debug and analyze training-inference mismatch issues in RL systemsApply score centering or similar stabilization techniques in RL trainingEvaluate the impact of quantization on RL performanceOptimize RL training for large-scale language models

Interview questions

Explain the concept of training-inference mismatch (TIM) in the context of RL for LLMs and why it leads to instability.
Describe how 'score centering' addresses the drift problem in RL under TIM. How does it compare to importance sampling-based methods?
Design a system to fine-tune a 30B parameter LLM using RLHF, considering potential TIM issues and how you would mitigate them in a production environment.

Relevant for ~24 months · generated by the enrichment pipeline