Paraphrase Fragility in Vision Language Reward Models (ROBORMBENCH)
This paper investigates a critical vulnerability in Vision-Language Models (VLMs) when used as reward functions for robotic learning: their fragility to paraphrased goal descriptions. It reveals that minor rephrasing of instructions can drastically alter predicted rewards for identical robot behaviors, even flipping success to failure. The research introduces ROBORMBENCH, a new benchmark to quantify this 'paraphrase fragility'.
Reading in Plain English mode — toggle below to switch
This paper investigates a critical vulnerability in Vision-Language Models (VLMs) when used as reward functions for robotic learning: their fragility to paraphrased goal descriptions. It reveals that minor rephrasing of instructions can drastically alter predicted rewards for identical robot behaviors, even flipping success to failure. The research introduces ROBORMBENCH, a new benchmark to quantify this 'paraphrase fragility'.
Imagine a complex mathematical proof. If you rephrase a lemma slightly, even if its logical meaning remains identical, a VLM acting as a 'proof checker' might suddenly declare the entire proof invalid or valid, despite no change in the underlying steps. This is similar to how VLMs struggle with semantically equivalent but lexically different instructions in robotics.
How Paraphrase Fragility Manifests
- 11. A robot performs a specific trajectory, e.g., 'pick up the red block'.
- 22. A VLM, acting as a reward model, evaluates this trajectory against the goal description and assigns a 'progress score' or reward.
- 33. The goal description is then paraphrased, e.g., 'grasp the crimson cube', while the robot's trajectory remains identical.
- 44. The VLM re-evaluates the *same* trajectory with the *new* description. Ideally, the reward should be invariant, but often, the score changes significantly, sometimes even reversing the success/failure judgment.
R(τ | G) ≈ R(τ | G')
R is the reward function, τ is a robot trajectory, G is the original goal description, and G' is a paraphrase of G. The ideal is that the reward for a trajectory given a goal should be approximately the same as the reward for the same trajectory given a semantically equivalent, paraphrased goal.
A robot is tasked with 'stacking two blocks'. If the instruction is rephrased to 'place one block atop another', a VLM reward model might assign a much lower reward to the exact same successful stacking behavior, indicating a failure of paraphrase invariance. This can lead to unreliable learning and deployment in real-world robotic systems.
Current Vision-Language Models used as reward functions in robotics exhibit significant paraphrase fragility, meaning minor linguistic variations in goal descriptions can lead to drastically different reward evaluations for identical robot behaviors, hindering reliable learning.