Long-WAM: Scaling Context in Robot World-Action Models for Real-Time Control
Long-WAM is a model-system framework designed to extend the contextual understanding of robot world-action models, enabling them to process longer visual histories for real-time control. It addresses the challenge of maintaining responsiveness while incorporating extensive past observations, crucial for complex robotic tasks.
Reading in Plain English mode — toggle below to switch
Long-WAM is a model-system framework designed to extend the contextual understanding of robot world-action models, enabling them to process longer visual histories for real-time control. It addresses the challenge of maintaining responsiveness while incorporating extensive past observations, crucial for complex robotic tasks.
Imagine a chef trying to bake a complex cake. If they only remember the last 30 seconds of what they did, they'll struggle. Long-WAM is like giving the robot chef a much longer memory, allowing it to recall hours of previous steps and visual cues, making it much better at intricate, multi-stage tasks without slowing down.
How Long-WAM Extends Robot Memory
- 1Long-WAM first learns to predict future events from past visual and robot data without explicit action labels, focusing on causal relationships.
- 2It then adapts this learned 'history-to-future' structure to specific world-action tasks, ensuring that longer histories are effectively utilized.
- 3A key insight is that pretraining the video foundation autoregressively (AR) significantly enhances the benefit of longer context, unlike bidirectional pretraining.
- 4The system employs streaming observation encoding, asynchronous execution, and hardware-specific optimizations to maintain real-time performance even with extended context windows.
P(future | history) = P(future | past_observations, pastₐctions)
This formula represents the core idea of predicting future states or actions given a sequence of past observations and actions. Long-WAM focuses on making the 'history' component much longer and more effective.
In a dynamic cup stacking task, traditional robot control models (like Pi0.5 and Fast-WAM) failed in all 20 trials due to their limited understanding of the evolving environment. Long-WAM, with its extended context, achieved a 95% success rate by effectively remembering and interpreting the changing positions of cups and the robot's previous actions over a longer period.
Longer visual history significantly improves robot task success, but only if the underlying video foundation is pretrained autoregressively to effectively leverage that history.