Questioning Instrumental Convergence in RLVR: Checkpoint Evidence from OLMo-3
Instrumental convergence is often invoked as a general reason to expect capable agents to acquire or express power-seeking subgoals: self-preservation, resource acquisition, deception, or resistance to oversight which poses an important threat to the society. For LLMs, reinforcement learning is the first stage of post-training that directly optimizes behavior against an objecti
Reading in Plain English mode — toggle below to switch
Instrumental convergence is often invoked as a general reason to expect capable agents to acquire or express power-seeking subgoals: self-preservation, resource acquisition, deception, or resistance to oversight which poses an important threat to the society. For LLMs, reinforcement learning is the first stage of post-training that directly optimizes behavior against an objecti
This paper directly addresses a critical concern in AI safety: whether reinforcement learning, particularly with verifiable rewards, inadvertently promotes undesirable instrumental behaviors like power-seeking in LLMs. Understanding these dynamics is crucial for ML engineers building and deploying advanced AI agents, as it impacts model robustness, trustworthiness, and societal alignment.