Semifactual Credit-Augmented Policy Optimization
Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning capabilities of large language models (LLMs), yet their predictions remain sensitive to task-irrelevant prompt features. We investigate this sensitivity through semifactual prompt interventions that preserve the underlying problem and its answer. Our analysis reveals substantial variation in token-
Reading in Plain English mode — toggle below to switch
Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning capabilities of large language models (LLMs), yet their predictions remain sensitive to task-irrelevant prompt features. We investigate this sensitivity through semifactual prompt interventions that preserve the underlying problem and its answer. Our analysis reveals substantial variation in token-
This research highlights limitations in current RLHF methods like GRPO, specifically regarding sensitivity to task-irrelevant prompt features. Understanding these sensitivities and developing methods to mitigate them is crucial for building more robust and reliable LLMs, directly impacting model quality and deployment strategies for ML engineers.