Exploration-Distillation (ExpDis): Decoupling Exploration from Optimization in RLVR
Exploration-Distillation (ExpDis) is a novel framework designed to improve reinforcement learning with verifiable rewards (RLVR) in language models. It addresses the challenge of encouraging models to discover new reasoning strategies without degrading overall performance, by separating the process of generating novel ideas from the process of refining those ideas.
Reading in Plain English mode — toggle below to switch
Exploration-Distillation (ExpDis) is a novel framework designed to improve reinforcement learning with verifiable rewards (RLVR) in language models. It addresses the challenge of encouraging models to discover new reasoning strategies without degrading overall performance, by separating the process of generating novel ideas from the process of refining those ideas.
Imagine a scientist (the explorer) who freely experiments with many wild ideas, some of which fail, but some lead to breakthroughs. A separate, meticulous engineer (the student) then takes only the successful, validated breakthroughs from the scientist and integrates them into robust, reliable systems. ExpDis mimics this by having 'explorer' policies generate diverse solutions, which are then filtered and taught to a 'student' policy, preventing the student from being polluted by failed explorations.
How ExpDis Works
- 11. **Explore with Novelty:** One or more 'explorer' policies are trained with a special reward bonus for generating novel solutions, encouraging them to try out many different approaches, even if some are incorrect.
- 22. **Filter for Quality:** The trajectories (sequences of actions and observations) generated by these explorers are then carefully filtered to keep only the correct and high-quality reasoning paths.
- 33. **Distill into Student:** These filtered, high-quality trajectories are used to train a separate 'student' policy. Crucially, the student policy is trained *without* any novelty bonus, focusing solely on optimizing for correctness and quality.
- 44. **Iterate and Refine:** This entire process of exploration, filtering, and distillation is repeated over several rounds. This iterative approach allows for aggressive exploration in one phase and stable optimization in another, leading to a more robust and capable final model.
Rₑₓ_pₗ_oᵣₑᵣ = R_vₑᵣᵢ_fᵢₐ_bₗₑ + λ N
$R_{explorer}$ is the total reward for the explorer policy. $R_{verifiable}$ is the standard reward based on correctness. $\lambda$ is a weighting factor, and $N$ is a novelty bonus that encourages the explorer to generate unique solutions.
Decoupling the search for novel solutions (exploration) from the refinement of correct solutions (optimization) allows LLMs to discover new reasoning strategies more effectively without sacrificing overall performance.