RPG: Guided Self-Improvement for Embodied Agents
RPG (Reconstruct, Practice, Go Real) is a novel framework that enables robotic systems to autonomously improve their manipulation capabilities without requiring model weight updates. It leverages offline data and simulation to diagnose failures, refine skills, and adapt system prompts for better real-world performance.
Reading in Plain English mode — toggle below to switch
RPG (Reconstruct, Practice, Go Real) is a novel framework that enables robotic systems to autonomously improve their manipulation capabilities without requiring model weight updates. It leverages offline data and simulation to diagnose failures, refine skills, and adapt system prompts for better real-world performance.
Imagine an athlete who watches videos of their past performances (offline dataset), practices specific moves in a simulated environment (simulation), identifies why they failed (diagnose failures), develops new techniques (new symbolic skills), refines existing ones, and adjusts their game plan (revises system prompt). This iterative self-improvement, testing changes before adopting them, is analogous to how RPG helps robots get better.
How RPG Improves Robot Performance
- 11. **Reconstruct Capabilities:** RPG analyzes an offline dataset of robot interactions to identify core manipulation capabilities.
- 22. **Practice in Simulation:** It constructs relevant practice tasks in a simulated environment, allowing the robot to repeatedly attempt and refine actions.
- 33. **Diagnose Failures:** During practice, RPG uses execution feedback, privileged simulator state, and dataset videos to pinpoint the causes of failures.
- 44. **Generate Improvements:** Based on diagnoses, RPG develops new symbolic skills, refines existing ones, and updates the system prompt that guides the robot's actions.
- 55. **Evaluate and Retain:** Candidate changes are tested in cross-task evaluations. Only validated improvements are merged and retained for future use.
- 66. **Go Real:** At deployment, a multimodal LLM uses the refined system prompt and skill library to coordinate perception and control for real-world tasks.
P_s_u_c_cₑ_s_s(t+1) > P_s_u_c_cₑ_s_s(t)
This indicates that the probability of success ($P_{success}$) at a given time step ($t+1$) is greater than at the previous time step ($t$), demonstrating continuous improvement.
In a pick-and-place task, RPG might observe a robot consistently dropping objects. Through simulation, it identifies that the grip strength is insufficient. It then generates a new skill for 'adjusting grip pressure' and updates the system prompt to prioritize this skill when lifting. After validation, the robot performs the task more reliably.
Autonomous self-improvement for embodied agents can be achieved by leveraging offline data and simulation for iterative diagnosis, skill refinement, and prompt revision, without needing to retrain underlying models.