Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs
Video generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene. Here we present a model that dire
Reading in Plain English mode — toggle below to switch
Video generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene. Here we present a model that dire
This paper addresses a critical gap in video generation by enabling the co-generation of 3D motion and video for arbitrary rigs, moving beyond pixel-level output to actionable 3D assets. This capability is crucial for applications in gaming, animation, and virtual reality, where precise control over 3D character animation from video prompts is highly valuable.