UniWALM: Unified World-Action-Language Model for Embodied Robot Control
UniWALM is a novel unified model for embodied robot control that integrates video, action, and language into a single shared representation. It aims to mimic human ability to learn across these modalities, enabling robots to perform complex manipulation tasks and leverage language as a form of internal memory.
Reading in Plain English mode — toggle below to switch
UniWALM is a novel unified model for embodied robot control that integrates video, action, and language into a single shared representation. It aims to mimic human ability to learn across these modalities, enabling robots to perform complex manipulation tasks and leverage language as a form of internal memory.
Imagine a robot that not only sees and acts but also 'thinks' in words, describing its subtasks and remembering past actions internally. UniWALM is like giving a robot this inner monologue, where generating language helps it plan and recall, improving its ability to interact with the world.
How UniWALM Unifies Robot Intelligence
- 1UniWALM uses a single mixture-of-transformers backbone to process video, action, and language data simultaneously. Unlike previous systems that treat these as separate stages, UniWALM merges them into one unified learning process.
- 2It employs per-modality diffusion heads that co-denoise synchronously, meaning they decode subtask instructions, textual memory, and physical actions all in the same pass, eliminating the need for a separate planning stage.
- 3Crucially, the language generated by UniWALM is re-consumed in the next step, creating a 'language-as-memory' loop. This internal feedback mechanism allows the robot to remember and adapt without an external memory bank or planner.
- 4To ensure robust action generation, UniWALM introduces Flow Matching with Correction (FMC), a training objective that helps the model recover from perturbed action trajectories, enhancing reliability without increasing inference cost.
P(xₜ | xₜ_-₁, oₜ, lₜ)
This represents the probability of the robot's action $x_t$ at time $t$, conditioned on its previous action $x_{t-1}$, current observation $o_t$ (video), and internal language $l_t$ (memory/subtask).
In a 'Find-the-Item' task, UniWALM significantly improves success rates by generating internal textual memory. For instance, if a robot needs to find a 'red cup', its internal language might generate 'looking for red cup', 'found red object', 'grasping cup', which guides its actions and helps it recover from errors, leading to a success rate of 61.1% compared to 11.1% without this internal language.
By unifying language, perception, and action within a single model and leveraging generated language as internal memory, UniWALM achieves superior performance in complex embodied tasks without external planners or pre-trained language models.