PhysStream: Streaming Physics-Grounded Video Generation
PhysStream is a novel autoregressive model designed for physics-grounded video generation, allowing for fine-grained, interactive control over dynamic scenes. It addresses limitations of previous methods by integrating structured scene memory and sparse velocity-increment signals, enabling more physically consistent and controllable video synthesis.
Reading in Plain English mode — toggle below to switch
PhysStream is a novel autoregressive model designed for physics-grounded video generation, allowing for fine-grained, interactive control over dynamic scenes. It addresses limitations of previous methods by integrating structured scene memory and sparse velocity-increment signals, enabling more physically consistent and controllable video synthesis.
Imagine you're animating a complex scene with multiple objects, but instead of manually moving each object frame by frame, you just give them tiny nudges (velocity increments) and the system automatically ensures they move realistically, remembering where everything was before. PhysStream acts like this intelligent puppet master, understanding the physics of the scene.
How PhysStream Generates Physics-Consistent Videos
- 11. Bidirectional Model Pre-training: A bidirectional model is initially fine-tuned to understand how motion control signals affect video frames.
- 22. Causal Autoregressive Model Training: A causal autoregressive model is then trained, building upon the pre-trained model and incorporating structured scene memory (positional and object tracking maps) derived from previously generated frames.
- 33. Interactive Control: During generation, sparse velocity-increment signals are fed into the model, allowing users to interactively guide the motion of objects in the scene.
- 44. Scene Memory Integration: The model continuously updates its internal scene memory based on newly generated frames, ensuring physical consistency and coherence across the video.
Vₜ = f(Vₜ_-₁, Cₜ, Mₜ_-₁)
$V_t$ represents the video frame at time $t$, which is a function $f$ of the previous frame $V_{t-1}$, the control signals $C_t$ at time $t$, and the structured scene memory $M_{t-1}$ from the previous step.
PhysStream can generate videos of multi-object tabletop rigid-body scenes, where users can interactively adjust the velocity of objects mid-generation. For instance, one could simulate billiard balls colliding and rolling, with the ability to 'nudge' a ball in a specific direction during the simulation, and the system will maintain physical realism.
PhysStream introduces a novel approach to interactive video generation by combining structured scene memory with sparse, physics-grounded motion control signals, leading to significantly improved physical consistency and user controllability.