← Benchmark results · Markdown source

RFC: stream MoE experts from disk in from_pretrained

Proposal for huggingface/transformers. Implementation: huggingface/transformers#50717 (proof of concept). Numbers: 2026-11-17, results page.

Feature request

Let from_pretrained keep MoE experts on disk and stream them through a bounded device cache, so MoE models whose experts do not fit in memory run for inference:

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.8-Flash-Next-FP8",
    device_map={"": "mps", "experts.*": "disk"},  # experts stream from disk, everything else on the device
    quantization_config=quantization_plan,        # per-layer quantizers, experts included (#50245)
    experts_offload={"cache_bytes": 12 << 30},    # device cache for experts; where these options live is open
)

Only the experts move: each token uses its top-k experts (10 of 512 per layer for Qwen3.8-Flash-Next), so they are read on demand and cached, while the dense layers load as usual.

Motivation

Qwen3.8-Flash-Next has 48 MoE layers of 512 experts: 121 GB of FP8 experts next to ~10 GB of dense weights. It does not load on a 32 GB machine, and Accelerate disk offload moves whole modules per forward, so every token would read every expert. Dedicated engines run it on a 32 GB Mac by streaming experts from the SSD (Strata).

On a 32 GB MacBook Pro M5, the prototype (draft PR plus a model-specific package for Strata's quantized experts and Metal kernels) against Strata on the same machine, same prompts and the same IQ3_XXS experts:

Transformers prototype Strata
Decode, 200 tokens (median of 4 prompts) 13.9 tok/s 14.8 1.06x
Prefill, 4K-token prompt 228 tok/s 220 1.0x
After a 6K-token document: prefill / decode 206 / 12.4 tok/s 222 / 13.7 1.08x / 1.1x

12 GiB expert cache, 8-bit dense layers. The remaining gap is mostly expert reads after a long prompt and one host round trip per MoE layer. On CUDA (one A10G, 24 GB), the same feature decodes at 5.8 tok/s with every expert in a pinned host tier (BF16 dense layers, 8 GiB device cache). Details and raw data: https://huggingface.co/spaces/tarekziade/moe-expert-streaming

Design

Hooks

What the feature changes outside integrations/experts_offload.py:

  1. from_pretrained (~25 lines in modeling_utils.py): after the model is built on meta, prepare_experts_offload(model, checkpoint_files, dtype, device, options) replaces each experts module's banks with the store and returns the checkpoint keys it takes over; they go into a new LoadStateDictConfig.skip_keys, which the safetensors loader skips. It refuses multi-device maps, GGUF files and quantizers other than the dense-only MetalConfig.
  2. Experts forward: the store registers an experts_offload implementation in ExpertsInterface and sets it as the config's _experts_implementation; modules decorated with @use_experts_implementation (gated, concatenated, untransposed, no bias) dispatch to it. No model code changes.
  3. MPS only: forward hooks on each decoder layer's sub-blocks record an event to commit queued work early (PyTorch's MPS stream otherwise waits for the next synchronization, and the GPU idles at each routing copy).

Scope

Inference on one device (MPS, CUDA or CPU). Training, multi-device maps and save_pretrained of a streamed model raise clear errors.

Open questions

  1. Options. With device_map={"experts.*": "disk"} selecting what streams, where do the cache budget and I/O settings go: a small ExpertsOffloadConfig, max_memory, or a generation-time setting?
  2. Quantized experts. Expressed through the quantization plan of #50245 (a quantizer per checkpoint layer, experts included), with the store caching whatever the plan produces; the draft still takes a quantized_experts directory.
  3. The loading hook. Is a skip_keys set in LoadStateDictConfig the right seam, or should the store plug in as a weight conversion operation / loader stage?
  4. Warm start. The draft can fill the cache with a routing profile's most used experts (preload_profile). Measured: 24% fewer misses and +2% decode on the first prompt after loading, no change after that (the LRU's own state is a better predictor than a profile). Keep it as an option, or drop it?

Draft implementation: #50717. It implements the device_map form above (experts_offload={...} carries the options), takes quantized experts from a quantized_experts directory until the quantization plan lands, and will follow the API settled here.