Running Qwen3.8-Flash-Next on a 32 GB Mac with Transformers
Transformers with streamed experts against Strata 0.1.60, both run on the same Apple M5 MacBook Pro (32 GB) with the same prompts
Strata runs Qwen3.8-Flash-Next
on a 32 GB Mac by streaming its experts from the SSD, since they are far too large to fit in memory. This is a
proof of concept that does the same with Transformers: from_pretrained with a new
experts_offload option, running on PyTorch MPS instead of a dedicated engine. It works, but it is behind:
this page measures by how much and where the time goes.
huggingface/transformers#50717 (the experts_offload PR) ·
RFC ·
tarekziade/moe-expert-streaming (Qwen3.8 parts, converter, benchmarks) ·
All the details (architecture, kernels, earlier measurements, cache-policy study, CUDA, known issues)
Head to head
| Same M5, same prompts | Transformers, GGUF experts | Transformers, 3-bit experts | Strata (IQ3_XXS) | Strata's lead |
|---|---|---|---|---|
| Decode, 200 tokens, median of 4 prompts (tok/s) | 13.92 | 11.59 / 11.56 | 14.8 | 1.06× |
| Prefill, 4,059-token prompt (tok/s) | 227.6 | 156.5 / 159.5 | 220 | 1.0× |
| Decode after a 6,062-token document (tok/s) | 12.41 | 8.75 / 8.94 / 8.44 | 13.7 | 1.1× |
| Prefill of that document (tok/s) | 205.6 | 158.2 / 157.6 / 157.2 | 222 | 1.08× |
| Routed experts | Strata's IQ2/IQ3 + IQ4_NL/Q2_0, 1.75 MB each, from its .experts file | 3-bit MLX affine, 2.15 MB each | IQ2/IQ3 + IQ4_NL/Q2_0, 1.75 MB each | |
| Dense layers (weights read per token) | Strata's own GGUF tensors (Q4_K–Q6_K, IQ4_XS, IQ3_S) + 8-bit hyper-connections, ~3.9 GB | 4–6-bit K-quants and IQ quants, BF16 hyper-connections in the GGUF, ~3.8 GB | ||
| Expert cache on the device | 12 GiB, LRU per size class, one guessed read per layer | 12 GiB, LRU | 12 GB, ranked by a routing profile, no eviction | |
Greedy, thinking off, no MTP and no speculative drafts for either, prompts in 1,024-token chunks for both, other apps closed (our prefill now uses 2,048-token chunks, see below). Strata: medians of three rounds of the whole protocol, with its one-read-per-expert copy of the experts and its default of skipping a missed expert ranked 7th–10th (turning that off costs it 1% at decode and 3% after the document). Transformers: decode is three runs, prefill and the document one run each. Quality is close but not identical: Strata's IQ3_XXS build is 1.6% above its Q8_0 in perplexity; our 8-bit dense layers match BF16 (+0.2%) and our 3-bit experts were not measured against 8-bit ones. Since 2026-11-16 our dense layers are Strata's own GGUF tensors (decode-path perplexity +1.4% over our 8-bit, same 94.5% top-1 agreement with BF16). The document rows use a 12 GiB expert cache. Transformers columns: two alternating runs each (GGUF experts: 2026-11-17, 3-bit: 2026-11-16 night), see "Strata's experts" below. Data: Strata report (client, tail skip off), Transformers reports.
Strata's experts in Transformers (2026-11-16, night)
The routed experts can now come straight from the .experts file Strata writes beside its GGUF (every
expert's gate, up and down blocks contiguous: one read per expert), in Strata's quantization: IQ2_XXS–IQ3_S gate/up,
IQ4_NL or Q2_0 down, 1.36–2.33 MB by layer, 1.75 MB on average against 2.15 MB for our 3-bit experts. Same 12 GiB cache,
more experts in it, fewer bytes per miss (branch gguf-experts):
- Cache: one slot pool per size class (10 for this model), each an LRU with the same number of slots per layer, like Strata's arena.
- Decode: ggml's
mul_mat_id(ggml-org/ggml-quantization) on the pools, 193–234 µs per layer against 230 µs for MLX's 3-bitgather_qmm; the Q2_0 down projections (30 layers, no Metal gemv in ggml) on our own gather-gemv kernel. - Prefill: dequantizing to bfloat16 and multiplying was 2.4× slower than the 3-bit kernel (68 tok/s). Now one
tile GEMM per projection over the pool, dequantizing the GGUF blocks inside the kernel (ggml's dequantizers) and
multiplying with Metal 4's
matmul2don the M5's matrix units, compiled from Python withtorch.mps.compile_shader(1.9× faster than a port of ggml's simdgroupkernel_mul_mm). The kernel reads the hidden states in place through a row index and a combine kernel sums each token's routes, instead of a gathered copy and anindex_add: one 2,048-token layer takes 47 ms (68 ms before, 336 ms with dequantize + matmul). Of those 47 ms, the matrix units take ~17 ms per projection pair, ~11 ms is loading and dequantizing that does not overlap them, and ~1/3 of the tile rows are padding (~40 routes per expert per chunk).
| M5 32 GB, alternating runs | 3-bit experts | GGUF experts |
|---|---|---|
| Decode, median of 4 prompts (tok/s) | 11.59 / 11.56 | 12.97 / 13.16 (+12–14%) |
| Misses / bytes read, prompts 2–4 | 17.0–17.4K / 36.5–37.5 GB | 13.1–14.3K / 22.8–24.9 GB |
| 4K prefill (tok/s) | 156.5 / 159.5 | 168.3 / 169.5 (+7%) |
| After a 6K document: prefill / decode (tok/s) | 158.2 / 8.75, 157.6 / 8.94, 157.2 / 8.44 | 158.2 / 9.70, 155.9 / 9.69 |
| Decode-path perplexity, 512 tokens (BF16: 3.3558) | 3.4088 | 3.3116 |
| Prefill-path perplexity, 1,024 tokens (BF16: 4.9007) | 4.9246 | 4.9684 |
| Device peak during the 4K prefill / after the document | 24.0 / 25.4 GiB | 20.3 / 22.5 GiB |
The gain matches a replay of the routing traces with Strata's sizes (+13% estimated). Read through the
OS page cache, as the 3-bit experts; with F_NOCACHE it was only +2–5%. Two earlier versions were killed by
the memory watchdog: pinned host buffers in 10 sizes (each cached at its peak by the allocator) and bfloat16 dequantized
banks; now one host buffer size, and the in-kernel dequantization needs no bank. Data:
reports.
2026-11-17, prefill kernels on top (GGUF experts, alternating runs, previous vs new): 4K prefill
172.3 / 167.2 -> 189.4 / 190.7 tok/s, decode 12.97 / 12.92 -> 13.40 / 13.19, after the 6K document 157 / 9.9 ->
179 / 9.8. Hyper-connection norm / act / mix kernels for any number of tokens (the fused down projection stays
single-token: MLX bf16 quantized_matmul is wrong for many rows on M5), 16-row tiles for each expert's
last routes, next-layer installs behind compute (a Gated DeltaNet output norm kernel was in that build but never ran:
it only took a SiLU gate, Qwen3.8's is a sigmoid). Data:
reports.
2026-11-17, Gated DeltaNet prefill kernels (previous vs new): 4K prefill 193.1 / 192.4 -> 212.2 / 225.6
tok/s (Strata: 220), decode 13.07 / 13.50 -> 13.40 / 13.63, after the 6K document 169 / 9.6 -> 186 / 9.5. One 2K-token
Gated DeltaNet layer 67 -> 28 ms: the output norm kernel with the sigmoid gate (9.3 -> 0.6 ms), our own delta-rule scan
after Strata's gdn_scan_r (ggml's kernel 32.8 ms -> 11.4 ms), the causal conv + SiLU as one bit-exact kernel
writing token-major rows, and a q/k L2 norm kernel. Prefill-path perplexity 4.9841, within the spread of the
implementations (PyTorch path 4.9964, ggml's 4.9761). Data:
reports; every kernel in use:
docs/kernels.md.
2026-11-17, decode after a long document (previous vs new): after the 6K document 9.82 / 9.73 -> 11.56 / 11.25 tok/s (Strata: 13.7), short prompts unchanged (13.86 / 13.41 -> 13.81 / 13.80). Past the indexer's 2,048-token budget the model built a mask over the whole cache and ran SDPA over every key (0.95 ms per layer at 6K keys); now a split-K flash decoding kernel reads only the selected blocks and the tail (0.12 ms), and the indexer's decode step is two kernels (0.47 -> 0.11 ms per layer, same blocks). Long prefill calls also leave their expert installs at the LRU's cold end (-7% misses). A replayed Strata-like static cache ranked by decode frequency missed 3× more than LRU after the document: routing has strong short-term locality. Data: reports.
Where the gap is
Decode: 85 ms per token against 68 ms
This breakdown is for the 3-bit experts (11.7 tok/s, 2026-11-16). With Strata's experts and the 2026-11-17 kernels a token takes ~73 ms (13.7 tok/s): the misses drop by ~18% and each reads ~19% fewer bytes; the routing round trips and the read waits below are still the gap.
Measured on the real model with the expert path replaced by a fake that maps routes to cache slots on the GPU (same kernels, no reads, no bookkeeping), and from the host's read-wait counters in normal runs:
| Part | ms / token | Notes |
|---|---|---|
| GPU compute, perfect cache, no host round trips | 47 | dense layers + cached experts: a ~21 tok/s ceiling for this design |
| 48 routing copies to the host | ~10 | ~200 µs each, also with MLX: Metal command-buffer completion on this Mac. Waiting faster (spin on pinned memory) saves only ~1 ms: the GPU idles while the host launches the work that follows each copy |
| Waiting for missed experts | 24–35 | ~55 of 480 expert lookups per token miss the 12 GiB cache; ~0.55 ms per miss, against 0.41 ms for one 2.15 MB read on this SSD (which tops out at 7.4–7.7 GB/s) |
Strata misses about as many experts (~58 per token) and reads about as much (100–140 MB per token, we read ~119 MB),
and it waits for those reads too: its own docs report 31.5 ms of SSD wait per token on the M4 after a 6K document. Its
code (stream.h,
stream.cpp,
mtl_rt.mm) shows where it saves time
around each wait:
- The handshake. A token is one Metal command buffer in 48 segments. After each layer's router the GPU signals a shared event and waits on it inside the same command buffer; the host spins on the event, does its turn and sets it, and the GPU resumes on work that is already encoded. We copy the routing to the host and then launch the rest of the layer: the GPU idles while the host launches (~10 ms per token).
- The reads. Missed experts are read from the SSD straight into the arena the GPU reads, in two pieces on six I/O threads, and the kernels look them up in a slot map in shared memory. We read into a pinned host buffer and copy it to the device.
- The read-ahead. The next layer's router logits are computed in the same GPU graph and read by the host from shared memory; its top 10 per token are queued at low priority and dropped if not started when the layer comes. Ours needs extra kernels, a copy and an install per guess, so it costs more than it hides and is off.
- Prompts. After each layer of a prompt chunk, every expert of the next layer that is not cached is streamed in behind the compute, at the cold end of the LRU so the decode-hot experts stay; the arena starts from a profile ranking. Our prefill read-ahead does the same first part, and since 2026-11-17 long prefill chunks also leave their installs at the LRU's cold end; a ranked starting arena replayed worse than LRU for us.
Strata's experts are also smaller (1.75 against 2.15 MB). Sources: its
tensor formats, per-token timing (STRATA_METAL_PROFILE=1),
SSD bytes, engine log;
ours: 2026-11-16 reports and scripts.
Prefill: 25.5 s for the 4K prompt against 18 s (now ~18.6 s)
2026-11-17: ~219 tok/s, even with Strata. Since the table below: Strata's experts on our tile-GEMM
kernels (Metal 4 matmul2d), then the Gated DeltaNet and hyper-connection prefill kernels (rows added at
the end). The SSD is not the limit here: page-cache bypass and 4 or 16 I/O threads made little difference.
A prompt chunk routes to nearly every expert of every layer, so each chunk reads nearly all of them again. Strata reads 120 GB at ~6 GB/s on average (6.7 GB/s peak): it is limited by the SSD. We used to read 137–145 GB (1,024-token chunks) at ~2.3 GB/s, because a layer's reads only started once its routing was known. The routed experts' own compute is small: one group of 64 experts (1,280 sorted rows) takes 3.7 ms in float32, ~5.7 s for the whole 4K prompt (microbenchmark; bfloat16 would save 1 ms per group but gives wrong results on the M5, dequantizing to bfloat16 first is 4× slower). What changed:
| 4K prefill, same Mac | tok/s | Read | Notes |
|---|---|---|---|
| 1,024-token chunks (before) | 68–77 | 137–145 GB | |
Gated DeltaNet prefill on ggml-gated-delta-net | +2.6% | 2.6× faster than the PyTorch reference per chunk (9.6 against 24.7 ms for 1,024 tokens); the whole layer is now ~4 s of a 4K prefill | |
| 2,048-token chunks | 89–96 | 86 GB | half the re-reads; 3,072 is slower and one 4,096-token chunk swaps on 32 GB |
+ read the next layer's experts during prefill (#50717 prefill_readahead_routes) | 103–106 | 114 GB | the SSD keeps reading while the GPU computes; ~28 GB of reads go unused; same output |
| + vectorized attention indexer | 158–160 | 114 GB | past 2,048 keys the model selects tokens with a Python loop and a host sync per query (1,355 ms per layer for the second 2K chunk); with a causal mask it is one scores matrix and one top-k (19.7 ms), same masks, same output |
+ Strata's GGUF experts, our tile GEMM on matmul2d (2026-11-16) | 168–170 | ~92 GB | GGUF blocks dequantized inside the kernel, rows read through an index, a combine kernel instead of a scatter |
| + 16-row tail tiles, next-layer installs behind compute, hyper-connection kernels for all tokens (2026-11-17) | 189–191 | ||
| + Gated DeltaNet prefill kernels: conv, q/k norm, delta-rule scan, gated norm (2026-11-17) | 212–226 | one 2K-token layer 67 → 28 ms |
Reading 114 GB at 7.1 GB/s takes 16 s; the rest of our 25.5 s is compute that does not overlap with the reads: Gated DeltaNet (~4 s), experts (~5.7 s), hyper-connections (~3.6 s), attention and the rest. Earlier per-module profile (1K chunks): data.
Decode after a long prompt: 8.5 against 13.7 tok/s (now ~12.4)
2026-11-17, later: clean pass on the
published code (decode 13.92, 4K prefill 227.6, after the document 205.6 / 12.41). Since ~11.4: one guessed expert of
the next layer read per layer, after the layer's own reads and installed only when used (+3%; reading every guess
competed with the real reads), the full-attention caches grown in place instead of by torch.cat
(1.4 ms per token at 6K), and a bit-exact 3-dispatch hyper-connection step. The SSD is bandwidth-bound (~0.35 ms
per expert read); the rest of the gap is the per-layer host round trip (~10 ms per token), which Strata avoids with
Metal shared events.
2026-11-17: the biggest cost was not the cache but attention. Past the indexer's 2,048-token budget the model ran SDPA over the whole cache under a boolean mask (0.95 ms per attention layer at 6K keys) plus ~35 small indexer ops; reading only the selected blocks (0.12 ms) and fusing the indexer (0.11 ms) gave most of the +16%. A new replay with Strata's experts in per-size pools found that leaving long-prefill installs at the LRU's cold end does help a little (-7% misses measured), and that a Strata-like static ranked cache would miss 3× more. The remaining gap is expert reads: ~61 misses per token after the document against ~42 after a short prompt.
After the 6K document, 82.5% of our expert lookups hit the 12 GiB cache, against 88.5% after a short prompt: the answer to a long document spreads over more experts. We first suspected the prefill flushing the cache, but replaying the routing trace says otherwise: starting the answer from the LRU state the prefill leaves (79.0% at 10 GiB), an empty cache (78.2%), or the experts the prompt used most (78.9%) changes little, and even the answer's own most-used experts as a fixed set reach only 81.2%. The 12 GiB cache now fits next to the 2K-chunk prefill (7.9 → 8.2 tok/s), and the indexer keeps its pooled keys between decode steps (8.5). Strata waits for its misses too; its lead comes from the cheaper work around each wait, as at short prompts.
What changed on 2026-11-16
Decode went from 9.6 to 11.7 tok/s on the same Mac (alternating A/B runs, apps closed):
| Change | Decode (tok/s) | Notes |
|---|---|---|
Dense layers from Strata's GGUF on ggml-org/ggml-quantization | 9.58 / 9.29 → 10.85 / 10.56 | +13–14%. The ggml gemv is correct in bfloat16 on the M5 and runs at 125–146 GB/s. Gated DeltaNet value heads come in llama.cpp's tiled order and are mapped back at load. 4K prefill 68 → 70 tok/s |
No expert read-ahead (prefetch_depth=0) | 11.04 / 11.15 / 11.09 → 11.80 / 11.78 / 11.58 | +5.6%. Guessing the next layers' experts hides ~9 ms of read waits per token but costs ~13 ms: guess routers, 2.7× the SSD reads (94 unused guesses per token), their device copies and evictions |
Tried, no gain or within noise: a larger expert cache with the freed dense memory (13.4 GiB: +1%, and the memory watchdog killed one run); writing the Gated DeltaNet output in GGUF order instead of a gather; spin-waiting on the routing copy instead of a blocking wait (bit-identical, no gain); queuing cached experts on the GPU before the routing copy and adding missed ones after their read (+1.3% with read-ahead, +3.5% mean without, noisy, not kept); a separate thread pool for reads the GPU waits on; bypassing the page cache and splitting one expert's read in parallel reads (both slower). A better next-layer predictor (least squares on the router's error, trained on a prefill trace) cuts critical misses 25.7 → 20.6 per token offline, but read-ahead as a whole does not pay on this Mac.
What would close it
- Less work around each miss (decode): fewer kernel launches between the routing copy and the next layer, and fewer copies per missed expert. Strata's two tools here, an event handshake inside an encoded command buffer and reads into GPU-visible memory without a copy, are not available from PyTorch on MPS. Running cached experts while the misses are read gave ~3%.
- Prefill compute (4K: even; after a document: ~1.16×): the hyper-connections, the Gated DeltaNet conv, delta rule and output norm now run as Metal kernels for all tokens; next is overlapping the remaining dense compute (attention, projections) with the expert reads, and the attention / indexer past 2,048 keys; the reads alone take 16 s against Strata's 18 s total.
- Smaller experts: done (above), +12–14% at decode, +7% at prefill. Since then: 16-row tiles for each expert's last routes (less padding) and the next layer's finished reads installed behind this layer's compute (+3%). Next: overlapping the weight decoding with the matrix units, and Strata's latest variant (weights decoded straight into the matrix units' inputs).
- Fewer host round trips (~10 ms per token): the cost is the GPU idling while the host launches what follows each routing copy; it shrinks with fewer launches per layer.
Files: transformers.patch (#50717 at 18abece3ab) ·
raw reports, converter and benchmark · details.