InfiniDepth Large

--- license: apache-2.0 datasets: - ritianyu/Hypersim base_model: - ritianyu/InfiniDepth pipeline_tag: depth-estimation tags: - vitl16 --- InfiniDepth: Arbitrary-Resolution and Fine-Grained Depth Estimation with Neural Implicit Fields Hao Yu* • Haotong Lin* • Jiawei Wang* • Jiaxin Li • Yida Wang • Xueyang Zhang • Yue Wang Xiaowei Zhou • Ruizhen Hu • Sida Peng --> ## 📢 News > **[2026-04]** 🎉 Training and evaluation code of InfiniDepth (RGB Only & Depth Sensor Augmentation) is available now! > **[2026-03]** 🎉 Inference code of InfiniDepth (RGB Only & Depth Sensor Augmentation) is available now! > **[2026-02]** 🎉 InfiniDepth has been accepted to CVPR 2026! Code coming soon! ## ✨ What can InfiniDepth do? InfiniDepth supports three practical capabilities for single-image 3D perception and reconstruction: | Capability | Input | Output | | --- | --- | --- | | Monocular & Arbitrary-Resolution Depth Estimation | RGB Image | Arbitrary-Resolution Depth Map | | Monocular View Synthesis | RGB Image | 3D Gaussian Splatting (3DGS) | | Depth Sensor Augmentation (Monocular Metric Depth Estimation) | RGB Image + Depth Sensor | Metric Depth + 3D Gaussian Splatting (3DGS) | ## ⚙️ Installation Please see [INSTALL.md](INSTALL.md) for manual installation. ## 🤗 Hugging Face Space Demo If you want to test InfiniDepth before running local CLI inference, start with the hosted demo: - Hugging Face Space: https://huggingface.co/spaces/ritianyu/InfiniDepth This repo also includes a Gradio Space entrypoint at `app.py`: - Input: RGB image (required), depth map (optional) - Task Switch: `Depth` / `3DGS` - Model Switch: `InfiniDepth` / `InfiniDepth_DepthSensor` ### Local run ```bash python app.py ``` ### Notes - In this demo, `InfiniDepth_DepthSensor` requires a depth map input; RGB-only inference should use `InfiniDepth`. - Supported depth formats in the demo upload: `.png`, `.npy`, `.npz`, `.h5`, `.hdf5`, `.exr`. ## 🚀 Inference ### Quick Command Index | If you want ... | Recommended command | | --- | --- | | Relative Depth from Single RGB Image | `bash example_scripts/infer_depth/courtyard_infinidepth.sh` | | 3D Gaussian from Single RGB Image | `bash example_scripts/infer_gs/courtyard_infinidepth_gs.sh` | | Metric Depth from RGB + Depth Sensor | `bash example_scripts/infer_depth/eth3d_infinidepth_depthsensor.sh` | | 3D Gaussian from RGB + Depth Sensor | `bash example_scripts/infer_gs/eth3d_infinidepth_depthsensor_gs.sh` | | Multi-View / Video Depth + Global Point Cloud | `bash example_scripts/infer_depth/waymo_multi_view_infinidepth.sh` | 1. Relative Depth from Single RGB Image ( inference_depth.py ) Use this when you want a relative depth map from a single RGB image and, optionally, a point cloud export. **Required input** - `RGB image` **Required checkpoints** - `checkpoints/depth/infinidepth.ckpt` - `checkpoints/moge-2-vitl-normal/model.pt` recover metric scale for point cloud export **Optional checkpoint** - `checkpoints/sky/skyseg.onnx` additional sky filtering **Recommended command** ```bash python inference_depth.py \ --input_image_path=example_data/image/courtyard.jpg \ --model_type=InfiniDepth \ --depth_model_path=checkpoints/depth/infinidepth.ckpt \ --output_resolution_mode=upsample \ --upsample_ratio=2 ``` Replace `example_data/image/courtyard.jpg` with your own image path. **For the example above, outputs are written to** - `example_data/pred_depth/` for the colorized depth map - `example_data/pred_pcd/` for the exported point cloud when `--save_pcd=True` **Example scripts** ```bash bash example_scripts/infer_depth/courtyard_infinidepth.sh bash example_scripts/infer_depth/camera_infinidepth.sh bash example_scripts/infer_depth/eth3d_infinidepth.sh bash example_scripts/infer_depth/waymo_infinidepth.sh ``` **Most useful options** | Argument | What it controls | | --- | --- | | `--output_resolution_mode` | Choose `upsample`, `original`, or `specific`. | | `--upsample_ratio` | Used when `output_resolution_mode=upsample`. | | `--output_size` | Explicit output size `(H,W)` when `output_resolution_mode=specific`. | | `--save_pcd` | Export a point cloud alongside the depth map. | | `--fx_org --fy_org --cx_org --cy_org` | Camera intrinsics in the original image resolution. | 2. 3D Gaussian + Novel-View Video from Single RGB Image ( inference_gs.py ) Use this when you want a 3D Gaussian export from a single RGB image and an optional novel-view video. **Required input** - `RGB image` **Required checkpoints** - `checkpoints/depth/infinidepth.ckpt` - `checkpoints/gs/infinidepth_gs.ckpt` - `checkpoints/moge-2-vitl-normal/model.pt` recover metric scale for 3D Gaussian export **Optional checkpoint** - `checkpoints/sky/skyseg.onnx` additional sky filtering **Recommended command** ```bash python inference_gs.py \ --input_image_path=example_data/image/courtyard.jpg \ --model_type=InfiniDepth \ --depth_model_path=checkpoints/depth/infinidepth.ckpt \ --gs_model_path=checkpoints/gs/infinidepth_gs.ckpt ``` Replace `example_data/image/courtyard.jpg` with your own image path. **For the example above, outputs are written to** - `example_data/pred_gs/InfiniDepth_courtyard_gaussians.ply` - `example_data/pred_gs/InfiniDepth_courtyard_novel_orbit.mp4` If `--render_size` is omitted, the novel-view video is rendered at the original input image resolution. **Example scripts** ```bash bash example_scripts/infer_gs/courtyard_infinidepth_gs.sh bash example_scripts/infer_gs/camera_infinidepth_gs.sh bash example_scripts/infer_gs/fruit_infinidepth_gs.sh bash example_scripts/infer_gs/eth3d_infinidepth_gs.sh ``` **Most useful options** | Argument | What it controls | | --- | --- | | `--render_novel_video` | Turn novel-view rendering on or off. | | `--render_size` | Output video resolution `(H,W)`. | | `--novel_trajectory` | Camera motion type: `orbit` or `swing`. | | `--sample_point_num` | Number of sampled points used for gaussian construction. | | `--enable_skyseg_model` | Enable sky masking before gaussian sampling. | | `--sample_sky_mask_dilate_px` | Dilate the sky mask before filtering. | > The exported `.ply` files can be visualized in 3D viewers such as [SuperSplat](https://superspl.at/). 3. Depth Sensor Augmentation (Metric Depth and 3D Gaussian from RGB + Depth Sensor) Use this mode when you have an RGB image plus metric depth from a depth sensor. **Required inputs** - `RGB image` - `Sparse depth` in `.png`, `.npy`, `.npz`, `.h5`, `.hdf5`, or `.exr` **Required checkpoints** - `checkpoints/depth/infinidepth_depthsensor.ckpt` - `checkpoints/moge-2-vitl-normal/model.pt` - `checkpoints/gs/infinidepth_depthsensor_gs.ckpt` **Required flags** - `--model_type=InfiniDepth_DepthSensor` - `--input_depth_path=...` **Metric Depth Inference Command** ```bash python inference_depth.py \ --input_image_path=example_data/image/eth3d_office.png \ --input_depth_path=example_data/depth/eth3d_office.npz \ --model_type=InfiniDepth_DepthSensor \ --depth_model_path=checkpoints/depth/infinidepth_depthsensor.ckpt \ --fx_org=866.39 \ --fy_org=866.04 \ --cx_org=791.5 \ --cy_org=523.81 \ --output_resolution_mode=upsample \ --upsample_ratio=1 ``` **3D Gaussian Inference Command** ```bash python inference_gs.py \ --input_image_path=example_data/image/eth3d_office.png \ --input_depth_path=example_data/depth/eth3d_office.npz \ --model_type=InfiniDepth_DepthSensor \ --depth_model_path=checkpoints/depth/infinidepth_depthsensor.ckpt \ --gs_model_path=checkpoints/gs/infinidepth_depthsensor_gs.ckpt \ --fx_org=866.39 \ --fy_org=866.04 \ --cx_org=791.5 \ --cy_org=523.81 ``` **Example scripts** ```bash bash example_scripts/infer_depth/eth3d_infinidepth_depthsensor.sh bash example_scripts/infer_depth/waymo_infinidepth_depthsensor.sh bash example_scripts/infer_gs/eth3d_infinidepth_depthsensor_gs.sh bash example_scripts/infer_gs/waymo_infinidepth_depthsensor_gs.sh ``` **Most useful options** | Argument | What it controls | | --- | --- | | `--fx_org --fy_org --cx_org --cy_org` | Strongly recommended when you know the sensor intrinsics. | | `--output_resolution_mode` | Output behavior for `inference_depth.py`. | | `--render_size` | Video resolution for `inference_gs.py`. | | `--output_ply_dir` | Custom output directory for gaussian export. | 4. Multi-View / Video Depth + Global Point Cloud ( inference_multi_view_depth.py ) Use this when you want sequence-level depth inference from an RGB image folder or video, plus per-frame aligned point clouds and one merged global point cloud. By default the script runs DA3 once on the whole sequence, then aligns each InfiniDepth depth map to the corresponding DA3 depth map before export. When you already know the camera intrinsics and extrinsics, you can instead provide them directly and skip DA3 entirely. **Required inputs** - `RGB image directory`, `single RGB image`, or `video` - `Sparse depth` directory / single file / depth video when `--model_type=InfiniDepth_DepthSensor` **Required checkpoints / dependencies** - `checkpoints/depth/infinidepth.ckpt` for RGB-only inference - `checkpoints/depth/infinidepth_depthsensor.ckpt` for RGB + depth sensor inference - `checkpoints/moge-2-vitl-normal/model.pt` recover metric scale for RGB-only frame inference - `depth-anything-3` installed in the current environment when using the default DA3-based sequence mode; default DA3 model is `depth-anything/DA3-LARGE-1.1` **Optional checkpoint** - `checkpoints/sky/skyseg.onnx` additional sky filtering **RGB-Only Multi-View / Video Command** ```bash python inference_multi_view_depth.py \ --input_path=example_data/multi-view/waymo/image \ --model_type=InfiniDepth \ --depth_model_path=checkpoints/depth/infinidepth.ckpt \ ``` **RGB + Depth Sensor Multi-View / Video Command** ```bash python inference_multi_view_depth.py \ --input_path=example_data/multi-view/waymo/image \ --input_depth_path=example_data/multi-view/waymo/depth \ --model_type=InfiniDepth_DepthSensor \ --depth_model_path=checkpoints/depth/infinidepth_depthsensor.ckpt \ ``` For video input, replace `--input_path` with a video file. When `--model_type=InfiniDepth_DepthSensor`, `--input_depth_path` can also be a depth video and must contain the same number of frames as the RGB input. **Explicit Camera-Parameter Multi-View Command** ```bash python inference_multi_view_depth.py \ --input_path=example_data/multi-view/waymo/image \ --camera_intrinsics_dir=/path/to/intrinsics \ --camera_extrinsics_dir=/path/to/extrinsics \ --model_type=InfiniDepth \ --depth_model_path=checkpoints/depth/infinidepth.ckpt \ ``` The explicit camera mode expects Waymo-style text files under `intrinsics/` and `extrinsics/`. Files are sorted lexicographically and matched one-to-one against the sorted RGB image list, so the number of camera files must exactly match the number of images. In this mode the script skips DA3 loading, DA3 cache export, DA3 RANSAC conditioning, and DA3 post scale alignment. This mode currently supports image inputs only, not video. **For the RGB-only example above, outputs are written to** - `example_data/multi-view/waymo/pred_sequence/image/frames/depth/` for aligned raw depth maps - `example_data/multi-view/waymo/pred_sequence/image/frames/depth_vis/` for colorized depth maps - `example_data/multi-view/waymo/pred_sequence/image/frames/pcd/` for per-frame aligned point clouds - `example_data/multi-view/waymo/pred_sequence/image/frames/meta/` for per-frame camera and alignment metadata - `example_data/multi-view/waymo/pred_sequence/image/da3/sequence_pose.npz` for cached DA3 predictions - `example_data/multi-view/waymo/pred_sequence/image/merged/sequence_merged.ply` for the merged global point cloud **Example scripts** ```bash bash example_scripts/infer_depth/waymo_multi_view_infinidepth.sh bash example_scripts/infer_depth/waymo_multi_view_infinidepth_depthsensor.sh bash example_scripts/infer_depth/waymo_multi_view_infinidepth_explicit_camera.sh ``` **Most useful options** | Argument |...