fanqi-robo/molmoact2_insert_gear_in_gripper_yam_fft

molmoact2 · fanqi-robo/insert_gear_in_gripper · variant yam_fft

Arm yam_fft (group baseline) of the MolmoAct2 baseline pair in experimental/lerobot_policy_molmoact2/modal/ (alpha-robotics, branch policy_test_fanqi), run with the same protocol as the π0.5 grid in experimental/lerobot_policy_pi05/.

• Description: base_fft started from allenai/MolmoAct2-BimanualYAM, the Ai2 checkpoint already fine-tuned on bimanual YAM data (same prompt strings, horizon 30).
• Hypothesis: Does the YAM-specific prior beat the generic base? pi0.5 has no counterpart (no YAM checkpoint).
• Base weights: allenai/MolmoAct2-BimanualYAM @ 8dcbed66 via policy.checkpoint_path (allenai/lerobot molmoact2-policy @ a4f15bf3)
• Dataset: fanqi-robo/insert_gear_in_gripper — all 50 episodes, no hold-out (copy of armnet/busybox_push_green_button with exact q01/q99 quantile stats; see its meta/quantile_fix.json)
• Trainable: train_mode_vlm=fft (action expert always trains); gradient_checkpointing=True
• Budget: global batch 32 (8/rank × 4 × H100) × 20000 steps; peak GPU memory 57783 MiB
• Joint frame: LeRobot v3 (dataset frame); no joint_signs/joint_offsets anywhere — evaluate WITHOUT any v2.1→v3 joint fix
• Published checkpoint: step 20000 (root of this repo, tag step-20000)
• Every saved step: fanqi-robo/molmoact2_insert_gear_in_gripper_yam_fft_step4000, fanqi-robo/molmoact2_insert_gear_in_gripper_yam_fft_step8000, fanqi-robo/molmoact2_insert_gear_in_gripper_yam_fft_step12000, fanqi-robo/molmoact2_insert_gear_in_gripper_yam_fft_step16000, fanqi-robo/molmoact2_insert_gear_in_gripper_yam_fft_step20000; full history (with optimizer state for the newest 2) on the Modal Volume molmoact2-busybox-ckpts
• W&B: https://wandb.ai/fanqi-robo-saferobotics/molmoact2_insert_gear_in_gripper

Evaluate

The only test is the real robot: submit fanqi-robo/molmoact2_insert_gear_in_gripper_yam_fft (or a _step<N> repo) on https://huggingface.co/spaces/armnet/armnet-eval (embodiment lerobot/so-101, task push_green_button, 20 rollouts, variation seed 42). Native horizon: execute the full 30-step chunk (n_action_steps=30).

Training config (lerobot-train --config_path=..., batch_size shown per rank)

policy:
  type: molmoact2
  checkpoint_path: allenai/MolmoAct2-BimanualYAM
  device: cuda
  action_mode: continuous
  train_mode_vlm: fft
  chunk_size: 30
  n_action_steps: 30
  image_keys:
  - observation.images.top
  - observation.images.left_wrist
  - observation.images.right_wrist
  setup_type: bimanual yam robotic arms in molmoact2
  control_mode: absolute joint pose
  model_dtype: bfloat16
  num_flow_timesteps: 8
  gradient_checkpointing: true
  normalize_gripper: true
  normalization_mapping:
    VISUAL: IDENTITY
    STATE: QUANTILES
    ACTION: QUANTILES
  optimizer_lr: 1.0e-05
  optimizer_vit_lr: 5.0e-06
  optimizer_connector_lr: 5.0e-06
  optimizer_action_expert_lr: 5.0e-05
  scheduler_warmup_steps: 200
  scheduler_decay_steps: 20000
  scheduler_decay_lr: 1.0e-06
  push_to_hub: false
  repo_id: fanqi-robo/molmoact2_insert_gear_in_gripper_yam_fft
  checkpoint_revision: 8dcbed66f2380e4393189c303ea72488eb9e63c2
dataset:
  repo_id: fanqi-robo/insert_gear_in_gripper
  episodes:
  - 0
  - 1
  - 2
  - 3
  - 4
  - 5
  - 6
  - 7
  - 8
  - 9
  - 10
  - 11
  - 12
  - 13
  - 14
  - 15
  - 16
  - 17
  - 18
  - 19
  - 20
  - 21
  - 22
  - 23
  - 24
  - 25
  - 26
  - 27
  - 28
  - 29
  - 30
  - 31
  - 32
  - 33
  - 34
  - 35
  - 36
  - 37
  - 38
  - 39
  - 40
  - 41
  - 42
  - 43
  - 44
  - 45
  - 46
  - 47
  - 48
  - 49
  image_transforms:
    enable: false
batch_size: 8
num_workers: 6
steps: 20000
save_freq: 4000
eval_freq: -1
log_freq: 20
seed: 1000
job_name: molmoact2_gear_yam_fft
output_dir: /root/outputs/train/molmoact2_gear_yam_fft
wandb:
  enable: true
  entity: fanqi-robo-saferobotics
  project: molmoact2_insert_gear_in_gripper
  mode: online
  disable_artifact: true