Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs
This paper introduces a novel model that simultaneously generates video and corresponding 3D motion for any given skeletal rig, along with a camera pose that registers the motion within a 3D scene. Unlike prior methods limited to human skeletons or 2D pose triangulation, this approach enables animating diverse 3D assets directly from text prompts.
Reading in Plain English mode — toggle below to switch
This paper introduces a novel model that simultaneously generates video and corresponding 3D motion for any given skeletal rig, along with a camera pose that registers the motion within a 3D scene. Unlike prior methods limited to human skeletons or 2D pose triangulation, this approach enables animating diverse 3D assets directly from text prompts.
Imagine you're solving an inverse kinematics problem, but instead of just finding joint angles for a target pose, you're also generating the video frames that depict that motion, and the camera angle from which it's viewed. It's like having a director and an animator working in perfect sync, but both are AI.
How it Works: Decoupled Motion and Video Generation
- 1A dedicated 3D motion model learns skeletal movements from motion-only data, which is easier to acquire and augment than paired video. This model supports arbitrary skeletons, not just human ones.
- 2This motion model is connected to a pre-trained video generation model via a small set of cross-attention layers, allowing the two modalities to influence each other without tightly coupling their internal representations.
- 3The model learns to precisely locate each joint within the generated video frames. This joint localization information is then used to recover the camera parameters and accurately place the generated 3D motion within the virtual scene.
- 4Training data is augmented by using a generative video editor to place animated 3D assets into suitable scenes, ensuring the known motion remains valid. This, combined with human and robot video, yields a large dataset for training.
V, M, C = G(T, R)
V represents the generated Video, M is the co-generated 3D Motion, and C is the Camera pose. G is the generative model, which takes a Text prompt (T) and a 3D Rig (R) as input.
Decoupling 3D motion generation from video generation, while connecting them via cross-attention and joint localization, allows for robust animation of arbitrary rigs and scene-aware motion.