Execution Before Memory: Diagnosing Memory Failures in Video World Models
A memory test for a video world model should say whether the scene was set, whether the asked change was carried out, and whether the last frame shows the result. Current tests fold these into one number, so a low score is easy to read as forgetting. When the change is not carried out, the hidden state transition cannot appear, so the test is not really a test of memory. We int
Reading in Plain English mode — toggle below to switch
A memory test for a video world model should say whether the scene was set, whether the asked change was carried out, and whether the last frame shows the result. Current tests fold these into one number, so a low score is easy to read as forgetting. When the change is not carried out, the hidden state transition cannot appear, so the test is not really a test of memory. We int
This research introduces a novel evaluation method for video world models, addressing a critical flaw in current memory tests. By separating scene-setting from instructed changes, it helps ML engineers diagnose whether models fail due to execution issues or true memory limitations, leading to more effective model development.