WristDepth: metric depth for robot wrist cameras

A fine-tune of Apple Depth Pro 2 B for robot wrist cameras at close manipulation range. Pass one RGB image and its horizontal focal length in pixels (f_px), and get camera-axis metric depth in meters. Unofficial; not affiliated with or endorsed by Apple.

Research use only. A Model Derivative under the Apple Machine Learning Research Model License: non-commercial research and academic use only. Apple Machine Learning Research Model is licensed under the Apple Machine Learning Research Model License Agreement. Modified by fine-tuning on procedural and multi-rig robotics wrist-camera data.

Watch it on a robot wrist

Side-by-side comparison on held-out DROID wrist cameras (PennPAL and TRI labs, excluded from training) with live stereo ground truth vs. Stock Depth Pro 2 vs. WristDepth on one fixed 0.08–1.5 m color scale. Download 23 s comparison Β· Download 15 s cut

A gripper carries a cube toward a cup, another clears a table, and a ReBot B601 picks up sugar cubes. The first two clips show qualitative relative depth colors because their camera focal lengths are unknown. The Blender cutaways show rendered training targets. Download the 15-second wipe demo

On the ReBot B601 (unseen during training), the middle panel shows measured RealSense D405 depth and the right panel shows WristDepth predicted from one RGB frame and its calibrated focal length. Both depth maps use the same 8–90 cm color scale; gray marks missing sensor returns. Download the sensor comparison

DROID clips (CC BY 4.0); LeRobot/ReBot footage by legalaspro, youliangtan, and prateeks-iu; Blender renders by Christian Engel. Source revisions, licenses and video descriptions Β· Try it on one wrist image (CPU) Β· Procedural dataset (98,180 Blender frames)

Held-out results (deployed CUDA FP16 ONNX)

All numbers below are measured on the exact downloadable onnx/model_fp16.onnx graph (CUDAExecutionProvider, single frame, no post-hoc alignment except where explicitly noted for uncalibrated relative depth). Mean absolute log-depth error (mae_log) averages per-frame error in either direction; lower is better. Scale is median predicted / reference depth (ideal 1.00).

Unseen labs & external rigs

Evaluation set Frames Scope v1 mae_log WristDepth mae_log v1 scale WristDepth scale
DROID test 392 Unseen scenes 0.249 0.115 0.94 1.00
PennPAL 1,235 Excluded lab 0.460 0.169 0.72 0.99
HOPE-Video (Baxter / D415) 160 Unseen lab (10 scenes) 0.179 0.129 0.86 0.92
ReBot B601 / D405 96 Unseen rig (12 episodes) 0.505* 0.517 1.67 1.61

*Legacy v1 emitted positive infinity on 5,524 pixels across 2 of 96 ReBot frames (95 valid reference pixels); WristDepth includes an inverse-depth guard and emits 0 nonfinite pixels. On ReBot's 15–25 cm manipulation band, mae_log drops from 0.513 (v1) to 0.288. On HOPE-Video, WristDepth improves 9 of 10 scenes over v1 (paired 95% CI [βˆ’0.079, βˆ’0.024]).

Within-rig held-out episodes & local tabletop tests

Evaluation set Frames Scope v1 mae_log WristDepth mae_log WristDepth scale
SO-101 private tabletop 40 Held-out frames 0.193 0.190 1.10
Local checkerboard placements 6 Disjoint placements 0.373 (99 mm) 0.024 (4.9 mm) 0.99
Paddy xArm7 100 Held-out episode 0.320 0.134 0.99
YAM (Agent-as-Policy) 1,009 20 held-out trials 0.374 0.100 (23 mm) 1.01
Towa25 SO-101 115 Held-out episode 0.220 0.068 (15 mm) 0.99
mot-prog SO-101 (relative only) 265 6 held-out episodes 0.074 0.045 β€”

DROID and PennPAL inputs (640Γ—352) were vertically centered on a black 640Γ—480 canvas with focal length unchanged and scored strictly on the original valid pixels. Full JSON records: evaluation_onnx.json Β· evaluation_pytorch.json Β· final_confirmation.json.

Close-range tabletop & white dish geometry

  • Synthetic glossy dishes (30 new Blender scenes, 59 frames): floor-above-table MAE dropped from 40.6 mm (v1) to 9.1 mm, and rim-to-floor relief MAE dropped from 29.4 mm (56.7% positive relief) to 14.7 mm (100% positive relief). synthetic_dish_geometry.json
  • Real white plate floor above surrounding desk at 20.1 / 26.4 / 32.5 cm camera height (physical estimate ~5–15 mm): 46.2 / 54.8 / 73.3 mm (v1) β†’ 17.7 / 10.4 / 3.0 mm (WristDepth).
  • Real bare-wood tabletop signed height at 20.1 / 26.4 / 32.5 cm: +3.6 / +4.2 / βˆ’12.2 mm (v1) β†’ βˆ’34.6 / βˆ’32.8 / βˆ’8.1 mm (WristDepth).
  • Five reserved poses from the same physical desk session (mean absolute desk bias): 40.6 mm (v1) β†’ 30.9 mm (WristDepth).

Shipped formats & quick start

This release ships two model artifacts:

  • model.safetensors β€” PyTorch weights (226/226 tensors verified exact on reload via DepthPro2.from_pretrained).
  • onnx/model_fp16.onnx β€” single-frame 480Γ—640 FP16 ONNX graph with an inverse-depth finite guard (abs_rel = 0.000317, delta1 = 1.0000, 0 nonfinite pixels across 5.22M fixture pixels vs FP32 reference). parity_model_fp16.json Β· artifact-check.json Β· SHA256SUMS

(Multi-frame clip, streaming, and INT8 graphs from the earlier v1 archive are not shipped in this release.)

CPU one-image quick start

From protortyp/wristdepth:

uv run examples/predict_image.py wrist.png --fx-px 557.2

ONNX Runtime (CUDA)

onnx/model_fp16.onnx expects float32 RGB in [0, 1] with shape (1, 3, 480, 640) and float32 f_px (horizontal focal length in pixels at that image size), and outputs depth_meters:

import numpy as np
import onnxruntime as ort

ort.preload_dlls()
session = ort.InferenceSession(
    "onnx/model_fp16.onnx", providers=["CUDAExecutionProvider"]
)
pixels = rgb.astype(np.float32).transpose(2, 0, 1)[None] / 255.0
depth_m = session.run(
    ["depth_meters"],
    {"pixel_values": pixels, "f_px": np.asarray([fx_px], dtype=np.float32)},
)[0].squeeze()

PyTorch

pip install "git+https://github.com/apple-aiml-research/ml-depth-pro-2"
from functools import partial
import torch
from depth_pro_2 import DepthPro2, load_rgb
from depth_pro_2.model import to_normalized_tensor

model = DepthPro2.from_pretrained("christianengel/WristDepth", precision=torch.bfloat16)
model = model.to("cuda").eval()
transform = partial(to_normalized_tensor, device=torch.device("cuda"))

image, _ = load_rgb("wrist.jpg")
depth_m = model.infer(transform(image), f_px=557.2)["depth"]

Training, selection & procedural data ablation

The released checkpoint (crossrig_e_frame34560, training checkpoint SHA-256 cef749401d6b8219348f66ea3fdaa4a4e82a013afd51bd724e0f96689ee1da7c) was trained via a multi-stage curriculum starting from Apple Depth Pro 2 B (BF16 autocast with FP32 trainable weights, AdamW, EMA, gradient accumulation 8). Training combines procedural Blender tabletop renders from WristDepth-procedural (98,180 unique frames split by scene, pinned revision cc979fe037fb8ebb008a843cb3b41cec266a513e, CC BY 4.0) with real multi-rig training splits (DROID stereo excluding TRI and PennPAL, Paddy xArm7, YAM, Towa25 SO-101, mot-prog relative shape, local checkerboards/wood, and community replay). The final stage fine-tuned the decoder and last five encoder blocks (lr = 2.9e-5, encoder 0.08Γ—, paired flip-consistency loss on 1 in 5 images; inference remains single-pass). Selection locked the #1 checkpoint across all 50 candidates solely on frozen development sets before running held-out audits and final confirmation.

Matched procedural-data ablation (two seeds)

To test whether substituting the updated Blender pools alone drives the gain at fixed compute, two matched seed pairs (33118, 33119, starting from v1 for 12,800 examples with identical real draws and optimization) compared pre-update procedural pools against replacing half of the synthetic draws with the updated pools (v5 seed 800 and seeds 900–907):

Seed & arm Generic dev (mae_log) 64 held-out near synthetic Checkerboard dev Wood 6-pose / bright
33118 pre-update 0.2571 0.2857 0.2119 25.4 / 58.6 mm
33118 with-update 0.2571 0.2967 0.1981 25.5 / 65.7 mm
33119 pre-update 0.2544 0.2822 0.2029 23.5 / 77.5 mm
33119 with-update 0.2540 0.2911 0.1914 23.8 / 76.7 mm

At this 12,800-example budget, swapping half of the synthetic draws is neutral on generic dev, consistently improves physical checkerboard dev in both seeds, and is ~0.01 higher on the 64 held-out near synthetic frames. The full model's large cross-rig gains reflect the combined multi-stage curriculum, multi-rig real supervision, near-field geometry losses, and longer training rather than the synthetic pool swap alone.

Limitations

  • Close bare-wood scale bias: On two close-range bare-wood tabletop poses (20.1 and 26.4 cm), the model places the desk ~33–35 mm too low even though it beats v1 across five reserved desk poses (30.9 vs 40.6 mm mean absolute bias) and achieves 4.9 mm MAE on held-out checkerboards. For sub-centimeter real-robot height control on low-texture surfaces, anchor depth to a known tabletop plane or calibrate one per-camera scale offset.
  • Unseen camera / gripper shifts: On the completely unseen ReBot B601 / D405 rig, median scale is 1.61Γ— overall (though 1.07Γ— in the 15–25 cm band). Rectified/undistorted RGB with accurate f_px gives the best metric accuracy.
  • Public dataset scope: Only the procedural Blender renders are published in WristDepth-procedural. Real stereo labels, third-party datasets, and private desk captures are not redistributed.

Provenance & license

Citation

If you use WristDepth or the procedural wrist-camera dataset in research, please cite both this release and Apple's Depth Pro 2 paper:

@misc{engel2026wristdepth,
  title        = {WristDepth: Metric Monocular Depth for Robot Wrist Cameras},
  author       = {Christian Engel},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/christianengel/WristDepth}},
}

@article{DepthPro2:2026:arxiv,
  title   = {Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth},
  author  = {Peiyun Hu and Marko Mihajlovic and Lyndon Duong and Lahav Lipson and Wei Dong and Ama\"{e}l Delaunoy and Stephan R. Richter and Vladlen Koltun},
  journal = {arXiv preprint arXiv:2611.26603},
  year    = {2026},
}
Downloads last month
-
Safetensors
Model size
96.5M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for christianengel/WristDepth

Finetuned
(3)
this model

Dataset used to train christianengel/WristDepth

Space using christianengel/WristDepth 1

Paper for christianengel/WristDepth