AD-WM: Action-Discriminative World Models for Counterfactual MPC
AD-WM introduces a novel world model designed for counterfactual Model Predictive Control (MPC). Unlike traditional world models that focus on predicting actual outcomes, AD-WM prioritizes distinguishing between the effects of different potential actions from the same state, leading to significantly improved planning performance in robotic tasks.
Reading in Plain English mode — toggle below to switch
AD-WM introduces a novel world model designed for counterfactual Model Predictive Control (MPC). Unlike traditional world models that focus on predicting actual outcomes, AD-WM prioritizes distinguishing between the effects of different potential actions from the same state, leading to significantly improved planning performance in robotic tasks.
Imagine a chess grandmaster planning moves. They don't just predict what will happen if they make one move; they mentally simulate and compare many different possible moves and their consequences to choose the best one. AD-WM does something similar for robots, learning to 'see' the distinct outcomes of various actions to make better decisions.
How AD-WM Works
- 11. **Latent Dynamics:** The model learns a compact representation (latent space) of the environment's state and how it changes over time.
- 22. **Action-Discriminative Training:** Instead of just predicting the next state, AD-WM is specifically trained to make sure that different actions lead to clearly distinguishable next states in its latent space.
- 33. **Inverse Dynamics Regularization:** It uses an auxiliary task to predict the action that led from one state to another, forcing the model to encode action information effectively.
- 44. **Conditional Mutual Information Objective:** A normalized recovery objective, motivated by conditional mutual information, further encourages the preservation of action-dependent differences.
- 55. **Counterfactual Planning:** During MPC, the model can then accurately compare the outcomes of various hypothetical actions from the current state, leading to better decision-making.
I(A; S' | S)
This represents the amount of information that the next state (S') provides about the action (A), given the current state (S). AD-WM aims to maximize this to make actions more discriminative.
In a robotic pick-and-place task, a standard world model might predict the robot's arm moving towards an object. AD-WM, however, would learn to clearly distinguish between the arm moving to grab the object versus moving to push it away, enabling the robot to choose the correct action more reliably, even in challenging 'hard-start' scenarios.
For effective model predictive control, world models need to do more than just predict factual transitions; they must explicitly learn to discriminate between the effects of different actions.