One model, three outputs: video, depth maps, and camera paths—for every single frame. Built by researchers from Harvard, MIT, and a few other universities, UGG takes whatever you give it and fills in the rest: feed it a video and it recovers depth and camera motion, give it a single photo plus a camera path and it generates a video, or just write a sentence to get a moving 3D scene.
The project's intro video.
One idea, six different jobs
Usually, each of these tasks requires its own specialized model: one guesses depth, another tracks the camera, and a third generates video. UGG packs all of them under one roof. With the exact same model, you can:
Recover depth maps and camera paths from a video.
Generate video and depth from one photo plus a camera path.
Bridge the gap between a first and last image along a given camera path.
Generate video, depth, and camera motion simultaneously from text.
Build a scene from just a camera path or just a depth map (interestingly, it wasn't even trained directly on these two tasks).
A kitchen generated from text alone.
How can it be this flexible?
The trick is surprisingly clean. It formats image, depth, and camera data into the exact same shape (drawing the camera path as a "ray map" that looks just like an image) and feeds them all together into a single transformer. Here, noise acts like a mask: whatever you provide stays clean, whatever you want generated gets noised, and the model denoises it. Switching tasks simply means changing the noise pattern, not the model.
The model starts from Wan2.2-TI2V-5B and was trained on 50-frame clips. Its training data mixes real and synthetic datasets, including ones missing depth or camera labels—the model simply learns from whichever parts are labeled.
Another cool detail: because it's a generative model, it can take multiple guesses and pick the best one. It generates 16 candidate scenes and keeps whichever aligns best with the input images. This alone boosts depth and camera accuracy across every benchmark in the "two images only" setup.
A depth map alongside the input video.
Results (as reported by the authors)
On camera path estimation, its error is lower than Geo4D, RayDiffusion, and others across all five test sets. On ScanNet++, for instance, trajectory error dropped from 0.190 down to 0.087.
Even with only two images (the first and last frames), it posts the lowest camera error across all five benchmarks.
For camera-guided video generation from a single image, it achieves the best PSNR, LPIPS, and SSIM scores across all three datasets.
Joint training clearly paid off: when video generation was dropped from training, depth error on KITTI more than doubled (jumping from 0.152 to 0.322).
The code: just a few lines
The weights (30 GB) are up on Hugging Face and the code is on GitHub under the MIT license. Running it is straightforward:
from ugg import UGG
model = UGG.from_pretrained()
pred = model.video_to_4d("assets/bonn_balloon2/frames.mp4")
pred.rgb # video frames
pred.depth # depth map per frame
pred.c2w # camera path
pred.save("out/") # saves rgb.mp4, depth.mp4, cameras.mp4 ...There is a dedicated function for every task: image_to_4d, first_last_to_4d, text_to_4d, camera_to_4d, and depth_to_4d. Sample inputs for each live in the assets folder, and scripts/run_examples.py runs them all in one go. You will need a GPU with 80 GB of VRAM. One practical tip: when generating scenes, try a few different seeds and keep the best result; a descriptive caption also noticeably lifts quality.
Generating a new view from a camera path.
What is it good for? A few ideas
These are my own takeaways based on the model's outputs, not things I've personally tested:
Virtual tours from a couple of photos: hand it the first and last photo of a room, draw your own camera path, and get a walk-through video complete with depth.
VFX and animation workflows: extract both the camera path (for placing 3D assets) and depth maps (for masking and effects) from a standard clip without juggling separate tools.
Robotics and machine vision: a model that sees, understands depth, and tracks camera motion is a solid foundation for robotics and autonomous driving research (something the paper explicitly highlights as a motivation).
Synthetic training data: generate aligned image, depth, and camera triplets from text prompts to train your own downstream models.
Game and environment pre-vis: describe a room or corridor in text and see how it feels in motion before committing to 3D modeling.
Interior design and real-estate apps: snap a single photo of a space and generate smooth fly-through videos along different camera paths.
The paper
The UGG paper was accepted at NeurIPS 2026. The authors treat video generation and geometry prediction as two sides of the same coin, showing that combining them in a shared model improves both. All the numbers above come directly from the paper.
My take
What stands out to me is how it proves the idea that putting all these modalities side by side makes each piece stronger. The most fun way to test it is probably generating video from a single photo and a camera path, since you get immediate visual feedback. If I were diving in, I'd start with image_to_4d using one of the assets samples, then gradually tweak the camera path to see how far the model follows along.
