Scr4tch

Abstract

Self-supervised motion masks

Scr4tch predicts a soft motion mask for every frame, learned without any mask, flow or motion labels.

Raw masks from the mask head, on videos where people and objects move independently of the camera. They are somewhat noisy, but follow walking people, cyclists, cars and hands, while static structure stays unmasked as the camera moves.

What the learned features encode

We freeze each backbone, remove its output heads and train the same light probe on top: two convolutional residual blocks on linearly projected features. DINOv2 processes each frame on its own and serves as a general-purpose reference.

InputGround truthScr4tchDINOv2CroCo v2VideoMAEV-JEPA 2

Scr4tch gives the cleanest masks and even catches small, distant moving objects. The video models fall below per-frame DINOv2: generic spatiotemporal features do not readily separate object motion from camera motion.

Camera estimation

Scr4tch decodes an explicit camera, with intrinsics and extrinsics, for every input frame.

Wild-SLAM Mocap

Relative translation accuracy within 5° (RTA@5) over all adjacent frame pairs on Wild-SLAM Mocap, a dynamic benchmark. Without camera labels or pretrained networks, Scr4tch is ahead of every self-supervised method. Supervised methods are shown for reference, and Scr4tch also surpasses the supervised MonST3R.

SelfSplat and PF3plat take ground-truth intrinsics as input; PF3plat and EgoMono4D build on pretrained networks. The full table, including the static DL3DV benchmark, is below.

A cart moves along with the camera and stays nearly fixed in the image. The Scr4tch trajectory aligns best with the ground truth (black), recovering the true camera motion despite the moving content.

Crowded casual video

When moving people fill most of the frame, little static background is left to pin down the camera. Each clip plays frame by frame while the cameras of Scr4tch and a baseline are laid against the reference trajectory. Drag the 3D view to rotate it.

Clip

Compare with

MonST3R NeoVerse E-RayZer

• Scr4tch
• MonST3R
• Reference

Drag to rotate

Pause

Clips from the raw training videos, which carry no camera labels; four of the ten hard clips we examined. They have no ground-truth cameras; each method is aligned to the dashed reference with one global Sim(3). MonST3R and NeoVerse are supervised; E-RayZer is a self-supervised 3D method.

Under unseen appearance

The stylized frame lies far outside the training distribution, and the person moves between the two views. Scr4tch still recovers the relative camera in agreement with the ground truth, which suggests it relies on scene structure rather than appearance statistics.

Disentangling camera and scene motion

Where prior work stops

Without compression, the camera stops steering the render

Latent renderings without motion tokens, along cameras interpolated between the context views t1 and t9. Trained without nested dropout, the renderings ignore the camera and only blend the two contexts. With it, scene content moves with the viewpoint. Removing nested dropout drops camera translation accuracy (RTA@5) from 0.592 to 0.011.

Compression needs augmentation, and the reverse

To test for appearance leakage, we bootstrap from stylized frames and render from clean context views. If the output picks up the stylized colors, the motion tokens are carrying target appearance.

Augmentation in training

On Off

Motion tokens kept

0 1 64

Camera and time, controlled separately

Every frame here is rendered from one 4D scene that Scr4tch predicted. Columns change only the camera, rows change only the time. Drag across the grid, use the sliders, or sweep one axis while the other stays fixed.

NVIDIA Dynamic. Each column uses the decoded camera of one input view, each row the time token of one input view. In Scr4tch the camera alone changes the viewpoint and the time token alone changes the scene. RayDer, also trained self-supervised on dynamic video, absorbs viewpoint changes into its dynamic state, so moving down a column shifts the view as well.

Method

One shared transformer encodes image patches together with per-view camera and time tokens and 64 global motion tokens. Training runs in two stages, and neither uses camera annotations, geometric pseudo-labels or pretrained geometric models.

1. Stage 1 Latent reconstruction pretraining

   • A bootstrap pass reads strongly augmented frames (grayscale with random channel weights, color jitter, blur) and infers camera, time and motion tokens.
   • The same transformer renders target views from clean context views, with target pixels set to zero.
   • Cross-split camera exchange, nested dropout and augmentation together block the shortcuts that would let tokens carry target appearance.

2. Stage 2 Explicit reconstruction alignment

   • Camera tokens are decoded into intrinsics and extrinsics and mapped back to tokens, so latent rendering becomes controllable by explicit cameras.
   • A mask head predicts soft motion masks that blend a static pass (cameras only) with a dynamic pass (plus motion and time). An asymmetric cost favors static explanations.
   • A context-only pass predicts pixel-aligned Gaussians; a light head deforms them toward each target time with 7-DoF residuals gated by the mask.
   • Camera gradient routing, photometric reprojection, rendered-depth distillation and single- and multi-view depth consistency ground the scene in geometry.

Full pipeline figure

Each of the N strongly augmented frames is spatially partitioned into disjoint splits A and B. A bootstrap pass predicts per-view motion masks and camera and time tokens, plus global motion tokens. Camera tokens inferred from complementary splits are exchanged and nested dropout is applied to the motion tokens. Three subsequent passes render target images from static latent features, dynamic latent features and a deformable Gaussian scene. Image-space losses on these predictions provide self-supervision.

Design details

Camera gradient routing

The static rendering uses undeformed Gaussians, which would otherwise be forced to fit moving objects and bend the cameras. With equal probability, it is supervised over the full image with detached cameras, or only over static regions with camera gradients on. On this sequence a cart moves with the camera; without routing it distorts the estimated trajectory (green), with routing the cameras align with the ground truth (black).

Context-only Gaussian pass

Input, one image ×3E-RayZerWithout context-only passWith context-only pass

Base Gaussians are predicted from clean context views only. If target views were present, jointly estimated camera conditioning could reveal that the viewpoint does not change, and reconstruction would succeed without meaningful depth.

Rendered-depth distillation

Top to bottom: inputs, raw depths without distillation, raw depths with it. Depths predicted by the Gaussian head get only indirect gradients through splatting and keep a patch-grid pattern. Distilling the rendered depths back into them transfers detail, down to poles and traffic signs.

Single- and multi-view depth consistency

Ground truthRendered imageRendered depth

Top row without, bottom row with the consistency loss. A wrong depth with compensating motion can still reconstruct the inputs, but fails from a new viewpoint. Single-view depth cannot tell whether an object moves, so it supervises moving regions, while multi-view rendered depth supervises static ones.

Novel view synthesis

Static scenes

Compare depth with

E-RayZer NoPoSplat SPFSplat

Scr4tch E-RayZer

Static scene from DL3DV, rendered from a view between two context views. Drag the divider: Scr4tch depths follow scene structure, while the other methods produce depths distorted by texture.

Dynamic scenes

Ground truthScr4tchC4GMoVieSNeoVerse

t₁t₁₀

NVIDIA Dynamic: each of the 12 input images comes from a different camera at a different time. Rendering other cameras at each input time, Scr4tch shows the correct scene state, with or without the umbrella, despite training on at most 9 frames and without the external geometric and motion supervision used by the baselines.

Casual YouTube

2,154 hours of casually captured monocular footage, collected for this work and used as raw video: no calibration, no annotations, no geometric preprocessing.

Footage

2,154 hours

Activities

Walking, cycling, first-person driving

Conditions

Sun, rain, snow and night, often with crowds and low light

Preprocessing

None

We release the URLs of all 1,684 videos, one per line. The videos themselves are not redistributed.

Casual YouTube URL list (.txt)

Ablations

• Without nested dropout the camera pathway collapses: RTA@5 falls to 0.011, while PSNR stays moderate because large background regions are still reconstructed.
• Camera gradient routing brings the largest gain in translation accuracy; depth consistency brings the largest rendering gain.
• Unprocessed Casual YouTube footage clearly improves camera estimation.

Citation

@article{wang2026scr4tch,
  title   = {Scr4tch: Self-Supervised 4D Reconstruction from Raw Video Alone},
  author  = {Wang, Ruoyu and Hu, Jiagao and Huang, Guichen and Wang, Chenyu
             and Wang, Fei and Zhou, Daiguo and Hou, Junhui},
  journal = {arXiv preprint arXiv:XXXX.XXXXX},
  year    = {2026}
}