← Benchmark results · Markdown source
RFC: stream MoE experts from disk in from_pretrained
Proposal for huggingface/transformers. Implementation: huggingface/transformers#50717 (proof of concept). Numbers: 2026-11-17, results page.
Feature request
Let from_pretrained keep MoE experts on disk and stream them through a bounded device cache, so MoE models whose experts do not fit in memory run for inference:
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3.8-Flash-Next-FP8",
device_map={"": "mps", "experts.*": "disk"}, # experts stream from disk, everything else on the device
quantization_config=quantization_plan, # per-layer quantizers, experts included (#50245)
experts_offload={"cache_bytes": 12 << 30}, # device cache for experts; where these options live is open
)
Only the experts move: each token uses its top-k experts (10 of 512 per layer for Qwen3.8-Flash-Next), so they are read on demand and cached, while the dense layers load as usual.
Motivation
Qwen3.8-Flash-Next has 48 MoE layers of 512 experts: 121 GB of FP8 experts next to ~10 GB of dense weights. It does not load on a 32 GB machine, and Accelerate disk offload moves whole modules per forward, so every token would read every expert. Dedicated engines run it on a 32 GB Mac by streaming experts from the SSD (Strata).
On a 32 GB MacBook Pro M5, the prototype (draft PR plus a model-specific package for Strata's quantized experts and Metal kernels) against Strata on the same machine, same prompts and the same IQ3_XXS experts:
| Transformers prototype | Strata | ||
|---|---|---|---|
| Decode, 200 tokens (median of 4 prompts) | 13.9 tok/s | 14.8 | 1.06x |
| Prefill, 4K-token prompt | 228 tok/s | 220 | 1.0x |
| After a 6K-token document: prefill / decode | 206 / 12.4 tok/s | 222 / 13.7 | 1.08x / 1.1x |
12 GiB expert cache, 8-bit dense layers. The remaining gap is mostly expert reads after a long prompt and one host round trip per MoE layer. On CUDA (one A10G, 24 GB), the same feature decodes at 5.8 tok/s with every expert in a pinned host tier (BF16 dense layers, 8 GiB device cache). Details and raw data: https://huggingface.co/spaces/tarekziade/moe-expert-streaming
Design
- Store. One device pool of expert slots per expert layout, shared by the layers that use it; every layer gets the same number of slots, so mixed per-layer quantization works. LRU (or decaying LFU) per pool. Per-layer expert → slot tables live on the host; when every expert is resident, layers never synchronize with the host. Pool memory is taken at the first forward, not during loading.
- Reads. Each MoE layer copies its routing to the host once. Misses are read on I/O threads into recycled pinned buffers (one contiguous read and one non-blocking host-to-device copy per expert). Optional read-ahead: in decode, the next layer's best guessed experts (its router on this layer's input) read once this layer's reads are done and installed only when used; in long prefill chunks, all of the next layer's experts, installed behind the current layer's compute. Long prefill calls do not refresh the LRU and their installs go to its cold end, so a long prompt does not flush the experts decoding uses.
- Compute. Cached experts run on the pool through the experts kernels (
grouped_mmfor BF16,gather_qmmfor quantized experts); uncached ones run once from their read. - Host tier (optional). A pool of expert rows in pinned host memory between storage and the device cache, for discrete GPUs with more host than device memory.
- External experts. An
expert_source(layout + read) and anexpert_computecallable let a package stream experts in its own format (the Qwen3.8 package uses Strata's GGUF experts and ggml kernels this way) while the store keeps caching, reads and read-ahead.
Hooks
What the feature changes outside integrations/experts_offload.py:
from_pretrained(~25 lines inmodeling_utils.py): after the model is built on meta,prepare_experts_offload(model, checkpoint_files, dtype, device, options)replaces each experts module's banks with the store and returns the checkpoint keys it takes over; they go into a newLoadStateDictConfig.skip_keys, which the safetensors loader skips. It refuses multi-device maps, GGUF files and quantizers other than the dense-onlyMetalConfig.- Experts forward: the store registers an
experts_offloadimplementation inExpertsInterfaceand sets it as the config's_experts_implementation; modules decorated with@use_experts_implementation(gated, concatenated, untransposed, no bias) dispatch to it. No model code changes. - MPS only: forward hooks on each decoder layer's sub-blocks record an event to commit queued work early (PyTorch's MPS stream otherwise waits for the next synchronization, and the GPU idles at each routing copy).
Scope
Inference on one device (MPS, CUDA or CPU). Training, multi-device maps and save_pretrained of a streamed model raise clear errors.
Open questions
- Options. With
device_map={"experts.*": "disk"}selecting what streams, where do the cache budget and I/O settings go: a smallExpertsOffloadConfig,max_memory, or a generation-time setting? - Quantized experts. Expressed through the quantization plan of #50245 (a quantizer per checkpoint layer, experts included), with the store caching whatever the plan produces; the draft still takes a
quantized_expertsdirectory. - The loading hook. Is a
skip_keysset inLoadStateDictConfigthe right seam, or should the store plug in as a weight conversion operation / loader stage? - Warm start. The draft can fill the cache with a routing profile's most used experts (
preload_profile). Measured: 24% fewer misses and +2% decode on the first prompt after loading, no change after that (the LRU's own state is a better predictor than a profile). Keep it as an option, or drop it?
Draft implementation: #50717. It implements the device_map form above (experts_offload={...} carries the options), takes quantized experts from a quantized_experts directory until the quantization plan lands, and will follow the API settled here.