Modality-Autoregressive World-Action Models
World-action models (WAMs) jointly model future observations and actions, typically predicting the future as RGB images. Other visual modalities such as depth, pretrained visual features, and point tracks can more efficiently capture geometric, semantic, and motion features. However, how best to combine these modalities within WAMs remains an open question. We introduce ModAR,
Reading in Plain English mode — toggle below to switch
World-action models (WAMs) jointly model future observations and actions, typically predicting the future as RGB images. Other visual modalities such as depth, pretrained visual features, and point tracks can more efficiently capture geometric, semantic, and motion features. However, how best to combine these modalities within WAMs remains an open question. We introduce ModAR,
This research advances world-action models (WAMs) by exploring multimodal autoregressive prediction, which is crucial for developing more robust and versatile AI agents. Understanding how to integrate diverse visual modalities like depth and pretrained features can lead to more efficient and capable embodied AI systems.