Instructions to use christianengel/WristDepth with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use christianengel/WristDepth with LeRobot:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
WristDepth: metric depth for robot wrist cameras
A fine-tune of Apple Depth Pro 2 B for robot wrist cameras at close manipulation range. Pass one RGB image and its horizontal focal length in pixels (f_px), and get camera-axis metric depth in meters. Unofficial; not affiliated with or endorsed by Apple.
Research use only. A Model Derivative under the Apple Machine Learning Research Model License: non-commercial research and academic use only. Apple Machine Learning Research Model is licensed under the Apple Machine Learning Research Model License Agreement. Modified by fine-tuning on procedural and multi-rig robotics wrist-camera data.
Watch it on a robot wrist
Side-by-side comparison on held-out DROID wrist cameras (PennPAL and TRI labs, excluded from training) with live stereo ground truth vs. Stock Depth Pro 2 vs. WristDepth on one fixed 0.08β1.5 m color scale. Download 23 s comparison Β· Download 15 s cut
A gripper carries a cube toward a cup, another clears a table, and a ReBot B601 picks up sugar cubes. The first two clips show qualitative relative depth colors because their camera focal lengths are unknown. The Blender cutaways show rendered training targets. Download the 15-second wipe demo
On the ReBot B601 (unseen during training), the middle panel shows measured RealSense D405 depth and the right panel shows WristDepth predicted from one RGB frame and its calibrated focal length. Both depth maps use the same 8β90 cm color scale; gray marks missing sensor returns. Download the sensor comparison
DROID clips (CC BY 4.0); LeRobot/ReBot footage by legalaspro, youliangtan, and prateeks-iu; Blender renders by Christian Engel. Source revisions, licenses and video descriptions Β· Try it on one wrist image (CPU) Β· Procedural dataset (98,180 Blender frames)
Held-out results (deployed CUDA FP16 ONNX)
All numbers below are measured on the exact downloadable onnx/model_fp16.onnx graph (CUDAExecutionProvider, single frame, no post-hoc alignment except where explicitly noted for uncalibrated relative depth). Mean absolute log-depth error (mae_log) averages per-frame error in either direction; lower is better. Scale is median predicted / reference depth (ideal 1.00).
Unseen labs & external rigs
| Evaluation set | Frames | Scope | v1 mae_log |
WristDepth mae_log |
v1 scale | WristDepth scale |
|---|---|---|---|---|---|---|
| DROID test | 392 | Unseen scenes | 0.249 | 0.115 | 0.94 | 1.00 |
| PennPAL | 1,235 | Excluded lab | 0.460 | 0.169 | 0.72 | 0.99 |
| HOPE-Video (Baxter / D415) | 160 | Unseen lab (10 scenes) | 0.179 | 0.129 | 0.86 | 0.92 |
| ReBot B601 / D405 | 96 | Unseen rig (12 episodes) | 0.505* | 0.517 | 1.67 | 1.61 |
*Legacy v1 emitted positive infinity on 5,524 pixels across 2 of 96 ReBot frames (95 valid reference pixels); WristDepth includes an inverse-depth guard and emits 0 nonfinite pixels. On ReBot's 15β25 cm manipulation band, mae_log drops from 0.513 (v1) to 0.288. On HOPE-Video, WristDepth improves 9 of 10 scenes over v1 (paired 95% CI [β0.079, β0.024]).
Within-rig held-out episodes & local tabletop tests
| Evaluation set | Frames | Scope | v1 mae_log |
WristDepth mae_log |
WristDepth scale |
|---|---|---|---|---|---|
| SO-101 private tabletop | 40 | Held-out frames | 0.193 | 0.190 | 1.10 |
| Local checkerboard placements | 6 | Disjoint placements | 0.373 (99 mm) | 0.024 (4.9 mm) | 0.99 |
| Paddy xArm7 | 100 | Held-out episode | 0.320 | 0.134 | 0.99 |
| YAM (Agent-as-Policy) | 1,009 | 20 held-out trials | 0.374 | 0.100 (23 mm) | 1.01 |
| Towa25 SO-101 | 115 | Held-out episode | 0.220 | 0.068 (15 mm) | 0.99 |
| mot-prog SO-101 (relative only) | 265 | 6 held-out episodes | 0.074 | 0.045 | β |
DROID and PennPAL inputs (640Γ352) were vertically centered on a black 640Γ480 canvas with focal length unchanged and scored strictly on the original valid pixels. Full JSON records: evaluation_onnx.json Β· evaluation_pytorch.json Β· final_confirmation.json.
Close-range tabletop & white dish geometry
- Synthetic glossy dishes (30 new Blender scenes, 59 frames): floor-above-table MAE dropped from 40.6 mm (v1) to 9.1 mm, and rim-to-floor relief MAE dropped from 29.4 mm (56.7% positive relief) to 14.7 mm (100% positive relief). synthetic_dish_geometry.json
- Real white plate floor above surrounding desk at 20.1 / 26.4 / 32.5 cm camera height (physical estimate ~5β15 mm): 46.2 / 54.8 / 73.3 mm (v1) β 17.7 / 10.4 / 3.0 mm (WristDepth).
- Real bare-wood tabletop signed height at 20.1 / 26.4 / 32.5 cm: +3.6 / +4.2 / β12.2 mm (v1) β β34.6 / β32.8 / β8.1 mm (WristDepth).
- Five reserved poses from the same physical desk session (mean absolute desk bias): 40.6 mm (v1) β 30.9 mm (WristDepth).
Shipped formats & quick start
This release ships two model artifacts:
model.safetensorsβ PyTorch weights (226/226tensors verified exact on reload viaDepthPro2.from_pretrained).onnx/model_fp16.onnxβ single-frame480Γ640FP16 ONNX graph with an inverse-depth finite guard (abs_rel = 0.000317,delta1 = 1.0000,0nonfinite pixels across 5.22M fixture pixels vs FP32 reference). parity_model_fp16.json Β· artifact-check.json Β· SHA256SUMS
(Multi-frame clip, streaming, and INT8 graphs from the earlier v1 archive are not shipped in this release.)
CPU one-image quick start
From protortyp/wristdepth:
uv run examples/predict_image.py wrist.png --fx-px 557.2
ONNX Runtime (CUDA)
onnx/model_fp16.onnx expects float32 RGB in [0, 1] with shape (1, 3, 480, 640) and float32 f_px (horizontal focal length in pixels at that image size), and outputs depth_meters:
import numpy as np
import onnxruntime as ort
ort.preload_dlls()
session = ort.InferenceSession(
"onnx/model_fp16.onnx", providers=["CUDAExecutionProvider"]
)
pixels = rgb.astype(np.float32).transpose(2, 0, 1)[None] / 255.0
depth_m = session.run(
["depth_meters"],
{"pixel_values": pixels, "f_px": np.asarray([fx_px], dtype=np.float32)},
)[0].squeeze()
PyTorch
pip install "git+https://github.com/apple-aiml-research/ml-depth-pro-2"
from functools import partial
import torch
from depth_pro_2 import DepthPro2, load_rgb
from depth_pro_2.model import to_normalized_tensor
model = DepthPro2.from_pretrained("christianengel/WristDepth", precision=torch.bfloat16)
model = model.to("cuda").eval()
transform = partial(to_normalized_tensor, device=torch.device("cuda"))
image, _ = load_rgb("wrist.jpg")
depth_m = model.infer(transform(image), f_px=557.2)["depth"]
Training, selection & procedural data ablation
The released checkpoint (crossrig_e_frame34560, training checkpoint SHA-256 cef749401d6b8219348f66ea3fdaa4a4e82a013afd51bd724e0f96689ee1da7c) was trained via a multi-stage curriculum starting from Apple Depth Pro 2 B (BF16 autocast with FP32 trainable weights, AdamW, EMA, gradient accumulation 8). Training combines procedural Blender tabletop renders from WristDepth-procedural (98,180 unique frames split by scene, pinned revision cc979fe037fb8ebb008a843cb3b41cec266a513e, CC BY 4.0) with real multi-rig training splits (DROID stereo excluding TRI and PennPAL, Paddy xArm7, YAM, Towa25 SO-101, mot-prog relative shape, local checkerboards/wood, and community replay). The final stage fine-tuned the decoder and last five encoder blocks (lr = 2.9e-5, encoder 0.08Γ, paired flip-consistency loss on 1 in 5 images; inference remains single-pass). Selection locked the #1 checkpoint across all 50 candidates solely on frozen development sets before running held-out audits and final confirmation.
Matched procedural-data ablation (two seeds)
To test whether substituting the updated Blender pools alone drives the gain at fixed compute, two matched seed pairs (33118, 33119, starting from v1 for 12,800 examples with identical real draws and optimization) compared pre-update procedural pools against replacing half of the synthetic draws with the updated pools (v5 seed 800 and seeds 900β907):
| Seed & arm | Generic dev (mae_log) |
64 held-out near synthetic | Checkerboard dev | Wood 6-pose / bright |
|---|---|---|---|---|
33118 pre-update |
0.2571 | 0.2857 | 0.2119 | 25.4 / 58.6 mm |
33118 with-update |
0.2571 | 0.2967 | 0.1981 | 25.5 / 65.7 mm |
33119 pre-update |
0.2544 | 0.2822 | 0.2029 | 23.5 / 77.5 mm |
33119 with-update |
0.2540 | 0.2911 | 0.1914 | 23.8 / 76.7 mm |
At this 12,800-example budget, swapping half of the synthetic draws is neutral on generic dev, consistently improves physical checkerboard dev in both seeds, and is ~0.01 higher on the 64 held-out near synthetic frames. The full model's large cross-rig gains reflect the combined multi-stage curriculum, multi-rig real supervision, near-field geometry losses, and longer training rather than the synthetic pool swap alone.
Limitations
- Close bare-wood scale bias: On two close-range bare-wood tabletop poses (
20.1and26.4 cm), the model places the desk ~33β35 mm too low even though it beats v1 across five reserved desk poses (30.9vs40.6 mmmean absolute bias) and achieves4.9 mmMAE on held-out checkerboards. For sub-centimeter real-robot height control on low-texture surfaces, anchor depth to a known tabletop plane or calibrate one per-camera scale offset. - Unseen camera / gripper shifts: On the completely unseen ReBot B601 / D405 rig, median scale is
1.61Γoverall (though1.07Γin the15β25 cmband). Rectified/undistorted RGB with accuratef_pxgives the best metric accuracy. - Public dataset scope: Only the procedural Blender renders are published in WristDepth-procedural. Real stereo labels, third-party datasets, and private desk captures are not redistributed.
Provenance & license
- Upstream weights: apple/DepthPro2-B (weights SHA-256
244ff85e0d9bde6e0f985de1c61420abc4c2110265b495c974556225ac2b60ed; exporter graph metadata records the byte-identical local snapshot281dc9883dea8fba21d5d78734291ca5d4f159d6). Licensed under the Apple Machine Learning Research Model License. - Code: protortyp/wristdepth (Apache 2.0).
- Procedural dataset: christianengel/WristDepth-procedural (CC BY 4.0).
Citation
If you use WristDepth or the procedural wrist-camera dataset in research, please cite both this release and Apple's Depth Pro 2 paper:
@misc{engel2026wristdepth,
title = {WristDepth: Metric Monocular Depth for Robot Wrist Cameras},
author = {Christian Engel},
year = {2026},
howpublished = {\url{https://huggingface.co/christianengel/WristDepth}},
}
@article{DepthPro2:2026:arxiv,
title = {Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth},
author = {Peiyun Hu and Marko Mihajlovic and Lyndon Duong and Lahav Lipson and Wei Dong and Ama\"{e}l Delaunoy and Stephan R. Richter and Vladlen Koltun},
journal = {arXiv preprint arXiv:2611.26603},
year = {2026},
}
- Downloads last month
- -
Model tree for christianengel/WristDepth
Base model
apple/DepthPro2-B