HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents
Agents built on vision language models (VLMs) now compete with learned policies on robot manipulation leaderboards. We ask how far a coding agent can go when it controls a robot instead of a computer, and where it stops. Our system, HexaShell, keeps the harness thin: it only turns the agent's commands into robot motion. The agent drives a robot with two arms through one command
Reading in Plain English mode — toggle below to switch
Agents built on vision language models (VLMs) now compete with learned policies on robot manipulation leaderboards. We ask how far a coding agent can go when it controls a robot instead of a computer, and where it stops. Our system, HexaShell, keeps the harness thin: it only turns the agent's commands into robot motion. The agent drives a robot with two arms through one command
This work explores the capabilities of VLM agents in robot control through code generation, highlighting a critical intersection of AI and robotics. For ML engineers, understanding how VLMs can be leveraged for real-world physical tasks, especially through iterative code refinement, is becoming increasingly important.