EyeRobot 2.0: Active Gaze for Precise Manipulation
EyeRobot 2.0 introduces an active gaze framework for robotic manipulation, inspired by human vision. It uses a single stereo camera to achieve fine-grained bimanual tasks by dynamically focusing its 'eyes' on specific points, significantly improving performance over passive systems.
Reading in Plain English mode — toggle below to switch
EyeRobot 2.0 introduces an active gaze framework for robotic manipulation, inspired by human vision. It uses a single stereo camera to achieve fine-grained bimanual tasks by dynamically focusing its 'eyes' on specific points, significantly improving performance over passive systems.
Imagine trying to thread a needle without moving your head or eyes, just staring straight ahead. It's incredibly hard! Humans constantly move their eyes to focus on the task at hand. EyeRobot 2.0 mimics this by actively swiveling its cameras to 'fixate' on critical areas, much like our fovea provides high-resolution vision where we look.
How EyeRobot 2.0 Works
- 11. **Active Visual Fixation (AVF):** The robot physically swivels its two stereo cameras to center its gaze on a 3D fixation point in the scene, creating foveated images where the center has higher visual resolution.
- 22. **Hierarchical Policy Training:** A low-level gaze servoing policy learns to direct the cameras to a goal object. A higher-level target selector policy then learns to emit these fixation goals based on the overall task progress.
- 33. **Reinforcement Learning:** Both policies are trained using reinforcement learning on real-world data. The gaze servoing policy uses a dense geometric reward, while the target selector co-trains with a gripper policy to discover effective fixation sequences.
- 44. **Fixation-Relative Canonicalization:** Gripper information is transformed into a coordinate frame relative to the current fixation point. This compacts the action distribution, making it easier for the robot to learn precise manipulation skills.
G = f(Iₗₑ_fₜ, Iᵣᵢ_g_hₜ, p_fᵢₓ)
The gaze servoing policy $G$ determines the camera movements based on the left and right stereo images ($I_{left}, I_{right}$) and the current fixation point ($p_{fix}$). This function aims to center the cameras on the target.
In a task like assembling small parts, EyeRobot 2.0 would first fixate on one part, then another, and finally on the point where they connect. This dynamic focusing allows it to perceive fine details crucial for precise grasping and placement, overcoming challenges like occlusions that plague static camera setups.
Active visual fixation, inspired by human vision, significantly enhances robotic manipulation precision and robustness by dynamically focusing computational resources on task-relevant visual information.