Long-WAM

Paper · Project and demos · Code

Long-WAM is a long-context world-action model: it uses a language instruction, robot state, and visual history to predict future dynamics and generate actions.

Model overview

• Learn to use history. LongLive2.0-Robot autoregressive video pretraining learns history-to-future dependencies before adaptation to robot actions.
• Predict, then act. IDM conditions action generation on observed history and predicted future latents, without decoding future RGB images.
• Scale temporal context. Longer observation windows capture motion and task progress; the history length is separate from the action horizon.

LeRobot supports both IDM and joint video/action co-denoising (COD), with native checkpoint loading, fine-tuning, and synchronous action-chunk inference. Wan backbones come from Diffusers and the UMT5 text encoder from Transformers.

Results

Reported lerobot-eval results for the published LeRobot-native IDM and COD checkpoints:

See Evaluation for example commands and episode limits. The commands were tested on NVIDIA GB200 with BF16 and synchronous inference.

Install

From a LeRobot source checkout:

pip install -e ".[longwam]"

Checkpoints

LeRobot-native checkpoints include full model weights and saved pre/post-processors. The tokenizer loads from google/umt5-xxl.

Both use P48 history (2.4 s at 20 Hz), two 224×224 camera views, 7-D actions, and an 8-D robot state. Use the checkpoint matching the desired denoising mode; original Long-WAM model.pt bundles cannot be loaded directly.

Data requirements

Use a LeRobot dataset with RGB cameras, observation.state, action, and task instructions. The released LIBERO camera keys are observation.images.image and observation.images.image2, in that order. Supply RGB tensors in [0, 1]; the processor resizes and center-crops each view, and the policy concatenates views horizontally. Keep camera order and action/state conventions aligned with the checkpoint.

Inference

from lerobot.policies import make_pre_post_processors
from lerobot.policies.longwam.modeling_longwam import LongWAMPolicy

checkpoint = "Efficient-Large-Model/Long-WAM-LIBERO-IDM-LeRobot"
policy = LongWAMPolicy.from_pretrained(checkpoint).to("cuda").eval()
preprocess, postprocess = make_pre_post_processors(
    policy.config,
    pretrained_path=checkpoint,
    preprocessor_overrides={"device_processor": {"device": "cuda"}},
)
policy.reset()
# observation contains both current camera images, observation.state, and task.
# Repeat on every control tick, including ticks that consume queued actions.
action = postprocess(policy.select_action(preprocess(observation)))

Call select_action on every control tick, including while an action chunk is executing, so the history stays current. predict_action_chunk returns [batch, prediction_steps, action_dim]; each call records the supplied current frame. reset() and drop_queued_actions() clear both history and queued actions. Generic async/RTC is not yet validated in this integration.

By default, inference_seed=1 restarts separate video/action noise generators with the same seed for each chunk, preserving released inference behavior. Set inference_seed=None in Python (--policy.inference_seed=null in the CLI) to use the advancing global RNG instead; reset() does not reseed that stream. Training always samples fresh noise.

Evaluation

Install the benchmark extra and evaluate a native checkpoint:

pip install -e ".[longwam,libero]"

lerobot-eval --policy.path=Efficient-Large-Model/Long-WAM-LIBERO-IDM-LeRobot \
  --policy.device=cuda --policy.dtype=bfloat16 \
  --env.type=libero --env.task=libero_spatial \
  --env.observation_height=256 --env.observation_width=256 \
  --env.episode_length=300 \
  --eval.batch_size=1 --eval.n_episodes=50 --seed=1

Use libero_object or libero_goal for the other short suites. For LIBERO-Long, set --env.task=libero_10 --env.episode_length=600. Replace IDM with COD in the checkpoint path to evaluate co-denoising.

History and action chunking

History duration is past_obs_steps / control_frequency. Early history is padded with the first observation. Use a checkpoint trained for the intended history; changing the window at inference is not context-scaling training.

Fine-tuning

lerobot-train --policy.path=Efficient-Large-Model/Long-WAM-LIBERO-IDM-LeRobot \
  --policy.device=cuda --policy.dtype=bfloat16 \
  --accelerator.mixed_precision=bf16 \
  --dataset.repo_id=YOUR_ORG/YOUR_DATASET \
  --output_dir=outputs/longwam --batch_size=1 --steps=30000 \
  --policy.push_to_hub=false

--policy.dtype selects parameter storage before loading; the native AdamW gradients and moments use that dtype. --accelerator.mixed_precision controls autocast, not FP32 master weights. The example uses BF16 for both, matching the original parameter dtype. For FP32 parameters/gradients/moments with BF16 autocast, use --policy.dtype=float32 --accelerator.mixed_precision=bf16; the frozen VAE and text encoder also use FP32, increasing memory. Choose this before loading: casting BF16-loaded weights to FP32 cannot recover lost precision.

The released gripper postprocessor converts the last action dimension from [0, 1] to signed commands. Use --policy.toggle_action_dimensions=[] if your fine-tuning data already uses the environment’s signed open/close convention. The active setting also applies to saved processors; explicit postprocessor_overrides take precedence. Saved normalization statistics are retained unless overridden, so align them with your fine-tuning data.

Training supervises future video and action chunks; inference receives only past/current observations. Adjust --batch_size to GPU memory and keep the checkpoint’s camera order, action/state definitions, and denoising mode.

Citation

@misc{huang2026longwamscalingcontextworldaction,
  title={Long-WAM: Scaling the Context of World-Action Models},
  author={Wei Huang and Bohan Zhang and Chenzhi Liu and Isabella Liu and
          Shuai Yang and Weian Mao and Luozhou Wang and Yicheng Xiao and
          Weifeng Lin and Qixin Hu and Bryan Chu and Sifei Liu and
          Linxi Fan and Xiaojuan Qi and Song Han and Yukang Chen},
  year={2026},
  eprint={2610.10528},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2610.10528},
}