4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes
4DCodeBench introduces a new benchmark for evaluating AI agents on their ability to perform 4D inverse graphics by generating code. This involves reconstructing complex dynamic scenes from video into executable graphics programs. The benchmark aims to push the boundaries of AI in understanding and reproducing real-world physics through code.
Reading in Plain English mode — toggle below to switch
4DCodeBench introduces a new benchmark for evaluating AI agents on their ability to perform 4D inverse graphics by generating code. This involves reconstructing complex dynamic scenes from video into executable graphics programs. The benchmark aims to push the boundaries of AI in understanding and reproducing real-world physics through code.
Imagine an AI agent watching a movie and then being able to write a script (code) that, when executed, perfectly recreates all the visual effects, character movements, and physical interactions seen in the original film. 4DCodeBench tests how well AI can be this 'movie director' for dynamic scenes.
How 4D Inverse Graphics Works
- 1An AI agent observes a video of a dynamic scene, such as water flowing or objects deforming.
- 2The agent analyzes the visual information to infer the underlying physical processes, object geometries, and material properties.
- 3Based on this understanding, the agent generates an executable graphics program (code) that describes the scene's structure and dynamics.
- 4This generated code is then executed to render a new version of the scene, which is compared against the original video to evaluate the agent's reconstruction accuracy.
V = f(P)
V represents the observed video or visual data, and P represents the underlying scene parameters (geometry, physics, materials). The function f is the rendering process that generates visuals from parameters. Inverse graphics aims to find P given V.
Consider a video of a ball bouncing and deforming upon impact. An agent performing 4D inverse graphics would generate code that specifies the ball's material properties (elasticity), the physics of its collision, and its initial trajectory, allowing a graphics engine to simulate and render the same bouncing and deformation behavior.
Current AI models struggle to reliably reconstruct complex dynamic scenes, even if they are good at static scene reconstruction, highlighting a significant gap in understanding real-world physics.