SNAP: Self-Supervised Geometric Representation Learning from Novel View Synthesis
This paper introduces SNAP, a self-supervised encoder-decoder transformer for learning robust 3D geometric representations from novel view synthesis (NVS). It argues that previous NVS methods fail to learn good representations due to overly expressive decoders and low-level pixel targets, proposing a new approach that addresses these issues.
Reading in Plain English mode — toggle below to switch
This paper introduces SNAP, a self-supervised encoder-decoder transformer for learning robust 3D geometric representations from novel view synthesis (NVS). It argues that previous NVS methods fail to learn good representations due to overly expressive decoders and low-level pixel targets, proposing a new approach that addresses these issues.
Imagine a sculptor trying to understand a 3D object by only looking at its 2D photographs. If they focus too much on perfectly recreating each photo (like an 'expressive decoder'), they might miss the underlying 3D shape. SNAP is like a sculptor who learns by trying to understand the object's fundamental geometry from different angles, rather than just copying pixels, leading to a deeper understanding of its form.
How SNAP Learns 3D Geometry
- 11. **Input Views:** SNAP takes multiple 2D images of a scene from different viewpoints as input.
- 22. **Encoder Processing:** A transformer-based encoder processes these images to extract rich, high-level features that capture the scene's underlying 3D structure.
- 33. **Latent Space Reconstruction:** Instead of reconstructing pixel-level images, SNAP's decoder reconstructs a compact, abstract representation (latent space) of the scene. This forces the encoder to learn more meaningful geometric features.
- 44. **Pose-Conditioned Decoding:** The decoder uses information about the camera's position and orientation (pose) to generate novel views from the learned latent representation, but with limited expressivity to prevent it from 'cheating' by memorizing pixel patterns.
- 55. **Self-Supervised Learning:** By comparing its generated novel views (or latent reconstructions) with actual views, SNAP refines its encoder to produce robust 3D geometric representations without explicit 3D supervision.
L = ||fₑnc(Iₜarget) - f_dec(fₑnc(I_source), Pₜarget)||²
L represents the loss function, which measures the difference between the encoded representation of a target image ($f_{enc}(I_{target})$) and the representation reconstructed by the decoder ($f_{dec}$) using the encoded source image ($f_{enc}(I_{source})$) and the target camera pose ($P_{target}$). The goal is to minimize this difference, forcing the encoder to learn good representations.
SNAP's learned representations can be used for various downstream tasks like visual localization (knowing where you are in a scene), depth estimation (how far objects are), and even robot manipulation, demonstrating its versatility and ability to generalize across different geometric challenges.
Restricting the decoder's expressivity and focusing on latent-space reconstruction, rather than pixel-space, forces the encoder to learn more transferable and robust 3D geometric representations.