Workspace Models: Lightweight Robotic Memory via Saliency-Driven Supervision
This paper introduces 'workspace models,' a novel approach for equipping robots with efficient long-term memory for complex manipulation tasks. Instead of relying on computationally expensive Vision-Language Model (VLM) queries during operation, this method pre-processes task-relevant information during training, distilling it into a lightweight 'workspace token' for deployment.
Reading in Plain English mode — toggle below to switch
This paper introduces 'workspace models,' a novel approach for equipping robots with efficient long-term memory for complex manipulation tasks. Instead of relying on computationally expensive Vision-Language Model (VLM) queries during operation, this method pre-processes task-relevant information during training, distilling it into a lightweight 'workspace token' for deployment.
Imagine a student preparing for an exam. Instead of carrying all textbooks (full history) or constantly looking up every detail (in-the-loop VLM queries), they create a concise 'cheat sheet' (workspace token) during study time (train-time). This cheat sheet contains only the most crucial information needed to solve problems quickly during the exam (deployment), making them faster and often more accurate.
How Workspace Tokens Work
- 11. During training, a powerful VLM identifies critical past and present information necessary for a robot to complete a task.
- 22. This identified information is then compressed and distilled into a compact 'workspace token' using a set-reconstruction decoder loss function.
- 33. At deployment, the robot uses this lightweight workspace token as a direct replacement for raw observations, allowing it to access relevant memory without real-time VLM processing.
Lᵣₑ_c_oₙ = ||S - D(T)||²
This formula represents the loss function used to train the workspace token. $S$ is the set of salient information identified by the VLM, $T$ is the workspace token, and $D$ is the decoder that attempts to reconstruct $S$ from $T$. The goal is to minimize the difference between the original salient information and its reconstruction from the token.
Consider a robot tasked with assembling a complex object. Instead of re-analyzing every past action or visual input with a VLM each time it needs to recall a step, the workspace token stores a distilled representation of critical assembly stages, tool locations, or part orientations. This allows the robot to quickly access relevant 'memories' to guide its next action, leading to smoother and more efficient assembly.
By pre-distilling VLM-identified salient information into a lightweight latent memory (workspace token) during training, robots can achieve superior performance and efficiency in memory-intensive tasks at deployment, bypassing expensive real-time VLM queries.