A partial open-source replication of D4RT: Efficiently Reconstructing Dynamic Scenes One D4RT at a Time (Google DeepMind, CVPR 2026), built with the frozen VGGT-1B encoder and publicly available datasets.
D4RT is a unified transformer that takes a video and answers any query about it: "where is this pixel in 3D in every frame?", "what is the depth of this frame?", "give me the whole scene point cloud", "where was the camera?" — using a single decoder with no self-attention between queries.
This repository is a partial replication: it reimplements the paper's decoder, query system, multi-task losses, training loop, and benchmarks on top of the pretrained VGGT-1B encoder instead of the paper's proprietary encoder, trained on public datasets (Kubric MOVi-F for training, TAPVid-3D and Sintel depth for evaluation).
- Paper: https://arxiv.org/abs/2512.08924
- Project page: https://d4rt-paper.github.io
- Author contact: d4rt@msajjadi.com
# 1. Clone and install
git clone https://github.com/grairudolf/d4rt-replication.git && cd d4rt-replication
pip install -r requirements.txt
# 2. Install the VGGT encoder backbone
pip install git+https://github.com/facebookresearch/vggt.git
# 3. Put a trained checkpoint at ./checkpoints/best.pt (see Training below)
# 4. Run inference on any video
python inference.py --video my_video.mp4 --task depth
# 5. See the results
ls out/ # out/my_video_depth_0.png (colormap) + .npy (raw depths)Tasks: depth, point_track, point_cloud, extrinsics. Inputs can be an .mp4/.avi/.mov/.mkv, a folder of frames, or a pre-loaded .pt/.npz tensor. Point a --checkpoint explicitly if you do not use ./checkpoints/best.pt.
video (T frames, 3 x H x W)
|
v
+----------------------+ frozen pretrained backbone
| VGGT-1B (ViT) +-----> F_raw (B, T, P, 2048)
+----------------------+
|
v linear projection
Global Scene Representation F (1, T*P, D = 768)
|
| Q independent queries (u, v, t_src, t_tgt, t_cam) + 9x9 patch crop
v
+--------------------------+
| QueryEncoder | Fourier coords + learned timestep embeddings
| -> (1, Q, D) tokens |
+--------------------------+
|
v cross-attention only (NO self-attention between queries)
+--------------------------+
| Lightweight Decoder | 4 layers, trained from scratch
+--------------------------+
|
v
per-query outputs:
points_3d (Q, 3) visibility (Q, 1) confidence (Q, 1)
points_2d (Q, 2) surface_normal (Q, 3) motion_vector (Q, 3)
The central idea, inherited from the paper: queries decode independently. Because there is no self-attention across queries, arbitrary query sets (a track, a depth grid, a full point cloud, two cameras' point sets) batch into one forward pass — which is what makes dense tracking (Algorithm 1) cheap.
Synthetic videos with ground-truth 3D point trajectories, depth, and camera parameters.
# Option A: HuggingFace (recommended, ~1 GB)
pip install huggingface_hub
huggingface-cli download zbww/tapip3d-kubric --repo-type dataset --local-dir ./data/kubric
# Option B: original Kubric via TensorFlow Datasets
pip install tensorflow tensorflow-datasets
python -c "import tensorflow_datasets as tfds; tfds.load('movi_e', data_dir='gs://kubric-public/tfds')"Dataset page: https://huggingface.co/datasets/zbww/tapip3d-kubric — Kubric: https://github.com/google-research/kubric
4,000+ real-world videos with metric 3D point trajectories, used for the main 3D-tracking benchmark (minival split recommended).
git clone https://github.com/google-deepmind/tapnet.git && cd tapnet
pip install "git+https://github.com/google-deepmind/tapnet.git[tapvid3d_eval,tapvid3d_generation]"
# Generate only the minival split (all three data sources) into ./tapvid3d_dataset
./tapnet/tapvid3d/annotation_generation/generate_all.sh --split=minivalNOTE: the generation scripts download the raw DriveTrack / Panoptic Studio / Aria sources (tens of GB); Aria Digital Twin additionally requires accepting its licence terms on Project Aria Explorer. Individual per-source scripts exist too:
generate_adt.py,generate_pstudio.py,generate_drivetrack.py.
Each resulting *.npz contains images_jpeg_bytes, tracks_XYZ, visibility, queries_xyt, and fx_fy_cx_cy. Docs: https://github.com/google-deepmind/tapnet/tree/main/tapnet/tapvid3d
wget https://files.is.tue.mpg.de/jwulff/sintel/MPI-Sintel-depth-training-20150305.zip
unzip MPI-Sintel-depth-training-20150305.zip -d ./data/sintelpython train.py # trains a tiny synthetic model and asserts the loss dropspython - <<'PY'
from train_config import default_config
from train import D4RTTrainer
cfg = default_config()
cfg.update({
"synthetic": False,
"dataset": "./data/kubric",
"encoder": "auto", # VGGT if installed, else the tiny fallback
"batch_size": 4,
"T": 8,
"num_queries_per_video": 64,
"num_epochs": 50,
"checkpoint_dir": "./checkpoints",
})
D4RTTrainer(cfg).train()
PYCheckpoints (with the full config embedded) are written to checkpoints/last.pt / checkpoints/best.pt and resumed via the resume_from config key.
python colab_setup.py > colab_cell.py # or just open the fileCopy the printed COLAB_CELL into a notebook cell — it installs dependencies, mounts Google Drive, clones this repo, and resumes training from the latest checkpoint. Recommended runtime: A100 (or T4 for small experiments).
evaluate.py runs the official TAPVid-3D protocol (local-camera-frame 3D tracks, global-median depth rescaling, depth-relative Jaccard thresholds at 256 px), Sintel video-depth with scale / scale+shift alignment, and Algorithm 1 dense-tracking throughput. Paper numbers are hardcoded for the comparison table:
| Benchmark | Metric | D4RT (paper) | This replication |
|---|---|---|---|
| TAPVid-3D minival | 3D-AJ (w/ GT intrinsics) | 0.304 | pending trained checkpoint |
| TAPVid-3D minival | 3D-AJ (w/o GT intrinsics) | 0.257 | pending trained checkpoint |
| TAPVid-3D minival | APD3D (w/ GT intrinsics) | 0.410 | pending trained checkpoint |
| TAPVid-3D minival | APD3D (w/o GT intrinsics) | 0.345 | pending trained checkpoint |
| TAPVid-3D minival | OA | 0.875 | pending trained checkpoint |
| Sintel depth | AbsRel (S) | 0.171 | pending trained checkpoint |
| Sintel depth | AbsRel (SS) | 0.148 | pending trained checkpoint |
| Dense tracking | Algorithm 1 speedup vs naive | 18–300x | pending trained checkpoint |
Run the full suite on a trained checkpoint with run_full_eval(model, checkpoint_path, results_dir, tapvid3d_path=..., sintel_path=...) from evaluate.py; results land in results.json and a printed comparison table. python evaluate.py runs a synthetic self-check that verifies the metrics (perfect predictions -> 1.0) and the whole pipeline.
- Proprietary encoder. D4RT uses a Google-internal encoder pre-trained on DeepMind's synthetic data. This replication uses the publicly available, frozen VGGT-1B instead.
- Internal training data. The paper trains on a large proprietary dynamic-scene dataset; we train on public Kubric MOVi-F.
- Scale. The paper uses hundreds of A100 GPU-hours and a 1B+ parameter model; our training is far smaller.
- Unpublished details. Exact training schedule, loss weights, and data augmentation details are only described at a high level in the paper, so some hyperparameters are best-effort reconstructions.
The gap between this replication and the paper's numbers therefore measures the encoder + data contribution, not the decoder design. The decoder, query system, and benchmark harness are faithful reimplementations of the paper.
@inproceedings{sajjadi2026d4rt,
title={Efficiently Reconstructing Dynamic Scenes One D4RT at a Time},
author={Sajjadi, Mehdi S. M. and others},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2026}
}The frozen encoder is VGGT by Meta AI Research (https://github.com/facebookresearch/vggt), and the TAPVid-3D benchmark/metrics come from google-deepmind/tapnet (Koppula et al., https://arxiv.org/abs/2407.05921).
MIT (see LICENSE). Note that VGGT-1B, Kubric, TAPVid-3D, and Sintel each carry their own licenses; this repository redistributes none of their files.