Upload EVO1 LIBERO Stage-2 70k checkpoint

.gitattributes

@@ -1,35 +1 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
 *.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,124 @@
+---
+language:
+  - en
+library_name: lerobot
+pipeline_tag: robotics
+tags:
+  - vision-language-action
+  - imitation-learning
+  - lerobot
+  - evo1
+  - libero
+inference: false
+license: other
+license_name: qwen
+license_link: https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/blob/main/LICENSE
+base_model:
+  - OpenGVLab/InternVL3-1B-hf
+datasets:
+  - lerobot/libero
+---
+
+# EVO1 · LIBERO (Stage 2, 70k)
+
+This is an EVO1 vision-language-action checkpoint trained on all four LIBERO suites with the
+two-stage LeRobot recipe: 5k steps of action-head-only training followed by joint VLM and action-head
+finetuning. This checkpoint was saved at step 70,000 of the 80k Stage-2 schedule and had the best overall
+result among the evaluated 10k, 40k, 60k, and 70k checkpoints.
+
+- EVO1: <https://github.com/MINT-SJTU/Evo-1>
+- Base VLM: [`OpenGVLab/InternVL3-1B-hf`](https://huggingface.co/OpenGVLab/InternVL3-1B-hf)
+- Dataset: [`lerobot/libero`](https://huggingface.co/datasets/lerobot/libero)
+
+## Results
+
+LeRobot LIBERO evaluation with seed `1000`, 50 episodes per task, 10 tasks per suite, 14-action
+replanning, 448×448 observations, and 2,000 episodes overall:
+
+| Suite          | Successful episodes |  Episodes | Success rate |
+| -------------- | ------------------: | --------: | -----------: |
+| LIBERO Spatial |                 485 |       500 |        97.0% |
+| LIBERO Object  |                 496 |       500 |        99.2% |
+| LIBERO Goal    |                 483 |       500 |        96.6% |
+| LIBERO-10      |                 469 |       500 |        93.8% |
+| **Overall**    |           **1,933** | **2,000** |   **96.65%** |
+
+These are single-seed results, not a multi-seed mean or confidence estimate.
+
+## Evaluation
+
+Install LeRobot with the EVO1 and LIBERO dependencies:
+
+```bash
+pip install "lerobot[evo1,libero] @ git+https://github.com/huggingface/lerobot.git@e40b58a8dfa9e7b86918c374791599d070518d11"
+```
+
+The following evaluates task 0 of LIBERO Spatial. Run task IDs 0–9 to reproduce the suite result.
+
+```bash
+export MUJOCO_GL=egl
+export PYOPENGL_PLATFORM=egl
+
+lerobot-eval \
+  --policy.path=zuoxingdong/evo1_libero \
+  --policy.vlm_model_name=OpenGVLab/InternVL3-1B-hf \
+  --policy.device=cuda \
+  --policy.use_amp=true \
+  --policy.vlm_dtype=bfloat16 \
+  --policy.use_flash_attn=false \
+  --policy.enable_gradient_checkpointing=false \
+  --policy.n_action_steps=14 \
+  --policy.num_inference_timesteps=32 \
+  --policy.postprocess_action_dim=7 \
+  --policy.binarize_gripper=true \
+  --policy.gripper_threshold=0.0 \
+  --policy.gripper_below_threshold_value=-1.0 \
+  --policy.gripper_above_threshold_value=1.0 \
+  --env.type=libero \
+  --env.task=libero_spatial \
+  --env.task_ids='[0]' \
+  --env.camera_name=agentview_image,robot0_eye_in_hand_image \
+  --env.camera_name_mapping='{agentview_image: image, robot0_eye_in_hand_image: image2}' \
+  --env.control_mode=relative \
+  --env.obs_type=pixels_agent_pos \
+  --env.observation_width=448 \
+  --env.observation_height=448 \
+  --env.init_states=true \
+  --env.episode_length=280 \
+  --env.render_mode=rgb_array \
+  --env.max_parallel_tasks=1 \
+  --eval.n_episodes=50 \
+  --eval.batch_size=1 \
+  --eval.use_async_envs=false \
+  --eval.recording=false \
+  --seed=1000 \
+  --output_dir=eval/evo1_libero/spatial-task-0
+```
+
+Use episode lengths 280 for `libero_spatial` and `libero_object`, 300 for `libero_goal`, and 520 for
+`libero_10`.
+
+## Training details
+
+- Dataset: all four suites from `lerobot/libero`, revision
+  `a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4`, with the standard enabled LeRobot image transforms.
+- Stage 1: randomly initialized action expert only, 5k steps.
+- Stage 2: VLM and action expert finetuning, 80k-step cosine schedule; this release is step 70k.
+- Both stages: global batch 128, learning rate 1e-5, warmup 1,000, weight decay 1e-3, dropout 0.2,
+  gradient clipping 1.0, chunk size 50, image size 448, and padded state/action dimensions 24.
+- Measured topology: two DDP processes with batch 64 per process on two H100 GPUs using bf16 AMP.
+- Base VLM source revision: `014c0583a0d4bedf29fbe2dbff4f865eb998e171`.
+- Model weight SHA-256: `c6ae0c7350d1810a064d16080137fdbc42d6dac5c6cf2cfb3874907efd959318`.
+
+## Limitations
+
+- Evaluation was run with one seed (`1000`).
+- The reported behavior depends on 14-action replanning and the documented LIBERO gripper postprocessing.
+- Loading the policy also downloads `OpenGVLab/InternVL3-1B-hf` unless it is already cached.
+
+## License
+
+The EVO1 code is MIT-licensed. InternVL3 is released under MIT, while its Qwen language component is
+covered by the [Qwen License](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/blob/main/LICENSE). This
+model is therefore published with `license: other` and Qwen attribution. The `lerobot/libero` dataset is
+Apache-2.0.

config.json

@@ -0,0 +1,103 @@
+{
+    "type": "evo1",
+    "n_obs_steps": 1,
+    "input_features": {
+        "observation.images.image": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                256,
+                256
+            ]
+        },
+        "observation.images.image2": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                256,
+                256
+            ]
+        },
+        "observation.state": {
+            "type": "STATE",
+            "shape": [
+                8
+            ]
+        }
+    },
+    "output_features": {
+        "action": {
+            "type": "ACTION",
+            "shape": [
+                7
+            ]
+        }
+    },
+    "device": "cuda",
+    "use_amp": true,
+    "use_peft": false,
+    "push_to_hub": false,
+    "repo_id": null,
+    "private": null,
+    "tags": null,
+    "license": null,
+    "pretrained_path": null,
+    "pretrained_revision": null,
+    "training_stage": "stage2",
+    "chunk_size": 50,
+    "n_action_steps": 50,
+    "max_state_dim": 24,
+    "max_action_dim": 24,
+    "max_views": 3,
+    "image_resolution": [
+        448,
+        448
+    ],
+    "empty_cameras": 0,
+    "postprocess_action_dim": null,
+    "binarize_gripper": false,
+    "gripper_index": 6,
+    "gripper_threshold": 0.5,
+    "gripper_below_threshold_value": 1.0,
+    "gripper_above_threshold_value": -1.0,
+    "normalization_mapping": {
+        "VISUAL": "IDENTITY",
+        "STATE": "MIN_MAX",
+        "ACTION": "MIN_MAX"
+    },
+    "vlm_model_name": "OpenGVLab/InternVL3-1B-hf",
+    "vlm_num_layers": 14,
+    "vlm_dtype": "float32",
+    "max_text_length": 1024,
+    "use_flash_attn": true,
+    "action_head": "flowmatching",
+    "embed_dim": 896,
+    "hidden_dim": 1024,
+    "state_hidden_dim": 1024,
+    "num_heads": 8,
+    "num_layers": 8,
+    "dropout": 0.2,
+    "num_inference_timesteps": 32,
+    "num_categories": 1,
+    "return_cls_only": false,
+    "enable_gradient_checkpointing": true,
+    "gradient_checkpointing_use_reentrant": false,
+    "finetune_vlm": true,
+    "finetune_language_model": true,
+    "finetune_vision_model": true,
+    "finetune_action_head": true,
+    "apply_training_stage_defaults": true,
+    "task_field": "task",
+    "embodiment_id_field": null,
+    "default_embodiment_id": 0,
+    "rtc_config": null,
+    "optimizer_lr": 1e-05,
+    "optimizer_betas": [
+        0.9,
+        0.999
+    ],
+    "optimizer_eps": 1e-08,
+    "optimizer_weight_decay": 0.001,
+    "optimizer_grad_clip_norm": 1.0,
+    "scheduler_warmup_steps": 1000
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c6ae0c7350d1810a064d16080137fdbc42d6dac5c6cf2cfb3874907efd959318
+size 3104659920

policy_postprocessor.json

@@ -0,0 +1,43 @@
+{
+  "name": "policy_postprocessor",
+  "steps": [
+    {
+      "registry_name": "unnormalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              24
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "IDENTITY",
+          "STATE": "MIN_MAX",
+          "ACTION": "MIN_MAX"
+        }
+      },
+      "state_file": "policy_postprocessor_step_0_unnormalizer_processor.safetensors"
+    },
+    {
+      "registry_name": "evo1_action_processor",
+      "config": {
+        "action_dim": 7,
+        "binarize_gripper": false,
+        "gripper_index": 6,
+        "gripper_threshold": 0.5,
+        "gripper_below_threshold_value": 1.0,
+        "gripper_above_threshold_value": -1.0
+      }
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cpu",
+        "float_dtype": "float32"
+      }
+    }
+  ]
+}

policy_postprocessor_step_0_unnormalizer_processor.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be872a1c074a41935944c6103ec3728a352ce63b50c647bd64a05c8fd7603182
+size 9216

policy_preprocessor.json

@@ -0,0 +1,76 @@
+{
+  "name": "policy_preprocessor",
+  "steps": [
+    {
+      "registry_name": "rename_observations_processor",
+      "config": {
+        "rename_map": {}
+      }
+    },
+    {
+      "registry_name": "to_batch_processor",
+      "config": {}
+    },
+    {
+      "registry_name": "evo1_pad_state_processor",
+      "config": {
+        "max_state_dim": 24
+      }
+    },
+    {
+      "registry_name": "evo1_pad_action_processor",
+      "config": {
+        "max_action_dim": 24
+      }
+    },
+    {
+      "registry_name": "normalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "observation.images.image": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              256,
+              256
+            ]
+          },
+          "observation.images.image2": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              256,
+              256
+            ]
+          },
+          "observation.state": {
+            "type": "STATE",
+            "shape": [
+              24
+            ]
+          },
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              24
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "IDENTITY",
+          "STATE": "MIN_MAX",
+          "ACTION": "MIN_MAX"
+        }
+      },
+      "state_file": "policy_preprocessor_step_4_normalizer_processor.safetensors"
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cuda",
+        "float_dtype": null
+      }
+    }
+  ]
+}

policy_preprocessor_step_4_normalizer_processor.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd8613329e21de83cf20de675fe4db6df3f6f083f01c4120a37047ff79a6f4d0
+size 9224

train_config.json

@@ -0,0 +1,253 @@
+{
+    "dataset": {
+        "repo_id": "lerobot/libero",
+        "root": null,
+        "episodes": null,
+        "image_transforms": {
+            "enable": true,
+            "max_num_transforms": 3,
+            "random_order": false,
+            "tfs": {
+                "brightness": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "brightness": [
+                            0.8,
+                            1.2
+                        ]
+                    }
+                },
+                "contrast": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "contrast": [
+                            0.8,
+                            1.2
+                        ]
+                    }
+                },
+                "saturation": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "saturation": [
+                            0.5,
+                            1.5
+                        ]
+                    }
+                },
+                "hue": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "hue": [
+                            -0.05,
+                            0.05
+                        ]
+                    }
+                },
+                "sharpness": {
+                    "weight": 1.0,
+                    "type": "SharpnessJitter",
+                    "kwargs": {
+                        "sharpness": [
+                            0.5,
+                            1.5
+                        ]
+                    }
+                },
+                "affine": {
+                    "weight": 1.0,
+                    "type": "RandomAffine",
+                    "kwargs": {
+                        "degrees": [
+                            -5.0,
+                            5.0
+                        ],
+                        "translate": [
+                            0.05,
+                            0.05
+                        ]
+                    }
+                }
+            }
+        },
+        "revision": "a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4",
+        "use_imagenet_stats": true,
+        "video_backend": "torchcodec",
+        "return_uint8": true,
+        "depth_output_unit": "mm",
+        "streaming": false,
+        "eval_split": 0.0
+    },
+    "env": null,
+    "policy": {
+        "type": "evo1",
+        "n_obs_steps": 1,
+        "input_features": {
+            "observation.images.image": {
+                "type": "VISUAL",
+                "shape": [
+                    3,
+                    256,
+                    256
+                ]
+            },
+            "observation.images.image2": {
+                "type": "VISUAL",
+                "shape": [
+                    3,
+                    256,
+                    256
+                ]
+            },
+            "observation.state": {
+                "type": "STATE",
+                "shape": [
+                    8
+                ]
+            }
+        },
+        "output_features": {
+            "action": {
+                "type": "ACTION",
+                "shape": [
+                    7
+                ]
+            }
+        },
+        "device": "cuda",
+        "use_amp": true,
+        "use_peft": false,
+        "push_to_hub": false,
+        "repo_id": null,
+        "private": null,
+        "tags": null,
+        "license": null,
+        "pretrained_path": null,
+        "pretrained_revision": null,
+        "training_stage": "stage2",
+        "chunk_size": 50,
+        "n_action_steps": 50,
+        "max_state_dim": 24,
+        "max_action_dim": 24,
+        "max_views": 3,
+        "image_resolution": [
+            448,
+            448
+        ],
+        "empty_cameras": 0,
+        "postprocess_action_dim": null,
+        "binarize_gripper": false,
+        "gripper_index": 6,
+        "gripper_threshold": 0.5,
+        "gripper_below_threshold_value": 1.0,
+        "gripper_above_threshold_value": -1.0,
+        "normalization_mapping": {
+            "VISUAL": "IDENTITY",
+            "STATE": "MIN_MAX",
+            "ACTION": "MIN_MAX"
+        },
+        "vlm_model_name": "OpenGVLab/InternVL3-1B-hf",
+        "vlm_num_layers": 14,
+        "vlm_dtype": "float32",
+        "max_text_length": 1024,
+        "use_flash_attn": true,
+        "action_head": "flowmatching",
+        "embed_dim": 896,
+        "hidden_dim": 1024,
+        "state_hidden_dim": 1024,
+        "num_heads": 8,
+        "num_layers": 8,
+        "dropout": 0.2,
+        "num_inference_timesteps": 32,
+        "num_categories": 1,
+        "return_cls_only": false,
+        "enable_gradient_checkpointing": true,
+        "gradient_checkpointing_use_reentrant": false,
+        "finetune_vlm": true,
+        "finetune_language_model": true,
+        "finetune_vision_model": true,
+        "finetune_action_head": true,
+        "apply_training_stage_defaults": true,
+        "task_field": "task",
+        "embodiment_id_field": null,
+        "default_embodiment_id": 0,
+        "rtc_config": null,
+        "optimizer_lr": 1e-05,
+        "optimizer_betas": [
+            0.9,
+            0.999
+        ],
+        "optimizer_eps": 1e-08,
+        "optimizer_weight_decay": 0.001,
+        "optimizer_grad_clip_norm": 1.0,
+        "scheduler_warmup_steps": 1000
+    },
+    "reward_model": null,
+    "output_dir": "outputs/train/evo1_libero_stage2_80k",
+    "job_name": "evo1_libero_stage2_80k",
+    "resume": false,
+    "seed": 1000,
+    "cudnn_deterministic": false,
+    "num_workers": 4,
+    "batch_size": 64,
+    "prefetch_factor": 2,
+    "persistent_workers": true,
+    "steps": 80000,
+    "env_eval_freq": 0,
+    "log_freq": 10,
+    "eval_steps": 0,
+    "max_eval_samples": 0,
+    "tolerance_s": 0.0001,
+    "save_checkpoint": true,
+    "save_freq": 10000,
+    "use_policy_training_preset": true,
+    "optimizer": {
+        "type": "adamw",
+        "lr": 1e-05,
+        "weight_decay": 0.001,
+        "grad_clip_norm": 1.0,
+        "betas": [
+            0.9,
+            0.999
+        ],
+        "eps": 1e-08
+    },
+    "scheduler": {
+        "type": "cosine_annealing_with_warmup",
+        "num_warmup_steps": 1000
+    },
+    "eval": {
+        "n_episodes": 50,
+        "batch_size": 50,
+        "use_async_envs": true,
+        "recording": false,
+        "recording_repo_id": null,
+        "recording_private": false
+    },
+    "wandb": {
+        "enable": false,
+        "disable_artifact": false,
+        "project": "lerobot",
+        "entity": null,
+        "notes": null,
+        "run_id": null,
+        "mode": null,
+        "add_tags": true
+    },
+    "peft": null,
+    "job": {
+        "target": null,
+        "image": "huggingface/lerobot-gpu:latest",
+        "timeout": "2d",
+        "detach": false,
+        "tags": []
+    },
+    "save_checkpoint_to_hub": false,
+    "sample_weighting": null,
+    "rename_map": {},
+    "checkpoint_path": null
+}