Running Qwen3.8-Flash-Next on a 32 GB Mac with Transformers

Transformers with streamed experts against Strata 0.1.60, both run on the same Apple M5 MacBook Pro (32 GB) with the same prompts

Strata runs Qwen3.8-Flash-Next on a 32 GB Mac by streaming its experts from the SSD, since they are far too large to fit in memory. This is a proof of concept that does the same with Transformers: from_pretrained with a new experts_offload option, running on PyTorch MPS instead of a dedicated engine. It works, but it is behind: this page measures by how much and where the time goes.

huggingface/transformers#50717 (the experts_offload PR) · RFC · tarekziade/moe-expert-streaming (Qwen3.8 parts, converter, benchmarks) · All the details (architecture, kernels, earlier measurements, cache-policy study, CUDA, known issues)

1.06×Strata's lead at decode: 14.8 against ~13.9 tok/s with its own GGUF experts (200-token answers to four different prompts; 1.26× with our 3-bit experts, 1.5× before the 2026-11-16 changes)
1.0×even at prefill: 220 against ~228 tok/s on a 4,059-token prompt (~158 with 3-bit experts; 3.2× before the 2026-11-16 changes)
1.1×its lead at decode after a 6,062-token document: 13.7 against ~12.4 tok/s (1.4× before the 2026-11-17 attention kernels, 1.9× before 2026-11-16); 1.08× at prefilling that document (222 against ~206)

Head to head

Same M5, same promptsTransformers, GGUF expertsTransformers, 3-bit expertsStrata (IQ3_XXS)Strata's lead
Decode, 200 tokens, median of 4 prompts (tok/s)13.9211.59 / 11.5614.81.06×
Prefill, 4,059-token prompt (tok/s)227.6156.5 / 159.52201.0×
Decode after a 6,062-token document (tok/s)12.418.75 / 8.94 / 8.4413.71.1×
Prefill of that document (tok/s)205.6158.2 / 157.6 / 157.22221.08×
Routed expertsStrata's IQ2/IQ3 + IQ4_NL/Q2_0, 1.75 MB each, from its .experts file3-bit MLX affine, 2.15 MB eachIQ2/IQ3 + IQ4_NL/Q2_0, 1.75 MB each
Dense layers (weights read per token)Strata's own GGUF tensors (Q4_K–Q6_K, IQ4_XS, IQ3_S) + 8-bit hyper-connections, ~3.9 GB4–6-bit K-quants and IQ quants, BF16 hyper-connections in the GGUF, ~3.8 GB
Expert cache on the device12 GiB, LRU per size class, one guessed read per layer12 GiB, LRU12 GB, ranked by a routing profile, no eviction

Greedy, thinking off, no MTP and no speculative drafts for either, prompts in 1,024-token chunks for both, other apps closed (our prefill now uses 2,048-token chunks, see below). Strata: medians of three rounds of the whole protocol, with its one-read-per-expert copy of the experts and its default of skipping a missed expert ranked 7th–10th (turning that off costs it 1% at decode and 3% after the document). Transformers: decode is three runs, prefill and the document one run each. Quality is close but not identical: Strata's IQ3_XXS build is 1.6% above its Q8_0 in perplexity; our 8-bit dense layers match BF16 (+0.2%) and our 3-bit experts were not measured against 8-bit ones. Since 2026-11-16 our dense layers are Strata's own GGUF tensors (decode-path perplexity +1.4% over our 8-bit, same 94.5% top-1 agreement with BF16). The document rows use a 12 GiB expert cache. Transformers columns: two alternating runs each (GGUF experts: 2026-11-17, 3-bit: 2026-11-16 night), see "Strata's experts" below. Data: Strata report (client, tail skip off), Transformers reports.

Strata's experts in Transformers (2026-11-16, night)

The routed experts can now come straight from the .experts file Strata writes beside its GGUF (every expert's gate, up and down blocks contiguous: one read per expert), in Strata's quantization: IQ2_XXS–IQ3_S gate/up, IQ4_NL or Q2_0 down, 1.36–2.33 MB by layer, 1.75 MB on average against 2.15 MB for our 3-bit experts. Same 12 GiB cache, more experts in it, fewer bytes per miss (branch gguf-experts):

M5 32 GB, alternating runs3-bit expertsGGUF experts
Decode, median of 4 prompts (tok/s)11.59 / 11.5612.97 / 13.16 (+12–14%)
Misses / bytes read, prompts 2–417.0–17.4K / 36.5–37.5 GB13.1–14.3K / 22.8–24.9 GB
4K prefill (tok/s)156.5 / 159.5168.3 / 169.5 (+7%)
After a 6K document: prefill / decode (tok/s)158.2 / 8.75, 157.6 / 8.94, 157.2 / 8.44158.2 / 9.70, 155.9 / 9.69
Decode-path perplexity, 512 tokens (BF16: 3.3558)3.40883.3116
Prefill-path perplexity, 1,024 tokens (BF16: 4.9007)4.92464.9684
Device peak during the 4K prefill / after the document24.0 / 25.4 GiB20.3 / 22.5 GiB

The gain matches a replay of the routing traces with Strata's sizes (+13% estimated). Read through the OS page cache, as the 3-bit experts; with F_NOCACHE it was only +2–5%. Two earlier versions were killed by the memory watchdog: pinned host buffers in 10 sizes (each cached at its peak by the allocator) and bfloat16 dequantized banks; now one host buffer size, and the in-kernel dequantization needs no bank. Data: reports.

2026-11-17, prefill kernels on top (GGUF experts, alternating runs, previous vs new): 4K prefill 172.3 / 167.2 -> 189.4 / 190.7 tok/s, decode 12.97 / 12.92 -> 13.40 / 13.19, after the 6K document 157 / 9.9 -> 179 / 9.8. Hyper-connection norm / act / mix kernels for any number of tokens (the fused down projection stays single-token: MLX bf16 quantized_matmul is wrong for many rows on M5), 16-row tiles for each expert's last routes, next-layer installs behind compute (a Gated DeltaNet output norm kernel was in that build but never ran: it only took a SiLU gate, Qwen3.8's is a sigmoid). Data: reports.

2026-11-17, Gated DeltaNet prefill kernels (previous vs new): 4K prefill 193.1 / 192.4 -> 212.2 / 225.6 tok/s (Strata: 220), decode 13.07 / 13.50 -> 13.40 / 13.63, after the 6K document 169 / 9.6 -> 186 / 9.5. One 2K-token Gated DeltaNet layer 67 -> 28 ms: the output norm kernel with the sigmoid gate (9.3 -> 0.6 ms), our own delta-rule scan after Strata's gdn_scan_r (ggml's kernel 32.8 ms -> 11.4 ms), the causal conv + SiLU as one bit-exact kernel writing token-major rows, and a q/k L2 norm kernel. Prefill-path perplexity 4.9841, within the spread of the implementations (PyTorch path 4.9964, ggml's 4.9761). Data: reports; every kernel in use: docs/kernels.md.

2026-11-17, decode after a long document (previous vs new): after the 6K document 9.82 / 9.73 -> 11.56 / 11.25 tok/s (Strata: 13.7), short prompts unchanged (13.86 / 13.41 -> 13.81 / 13.80). Past the indexer's 2,048-token budget the model built a mask over the whole cache and ran SDPA over every key (0.95 ms per layer at 6K keys); now a split-K flash decoding kernel reads only the selected blocks and the tail (0.12 ms), and the indexer's decode step is two kernels (0.47 -> 0.11 ms per layer, same blocks). Long prefill calls also leave their expert installs at the LRU's cold end (-7% misses). A replayed Strata-like static cache ranked by decode frequency missed 3× more than LRU after the document: routing has strong short-term locality. Data: reports.

Where the gap is

Decode: 85 ms per token against 68 ms

This breakdown is for the 3-bit experts (11.7 tok/s, 2026-11-16). With Strata's experts and the 2026-11-17 kernels a token takes ~73 ms (13.7 tok/s): the misses drop by ~18% and each reads ~19% fewer bytes; the routing round trips and the read waits below are still the gap.

Measured on the real model with the expert path replaced by a fake that maps routes to cache slots on the GPU (same kernels, no reads, no bookkeeping), and from the host's read-wait counters in normal runs:

Partms / tokenNotes
GPU compute, perfect cache, no host round trips47dense layers + cached experts: a ~21 tok/s ceiling for this design
48 routing copies to the host~10~200 µs each, also with MLX: Metal command-buffer completion on this Mac. Waiting faster (spin on pinned memory) saves only ~1 ms: the GPU idles while the host launches the work that follows each copy
Waiting for missed experts24–35~55 of 480 expert lookups per token miss the 12 GiB cache; ~0.55 ms per miss, against 0.41 ms for one 2.15 MB read on this SSD (which tops out at 7.4–7.7 GB/s)

Strata misses about as many experts (~58 per token) and reads about as much (100–140 MB per token, we read ~119 MB), and it waits for those reads too: its own docs report 31.5 ms of SSD wait per token on the M4 after a 6K document. Its code (stream.h, stream.cpp, mtl_rt.mm) shows where it saves time around each wait:

Strata's experts are also smaller (1.75 against 2.15 MB). Sources: its tensor formats, per-token timing (STRATA_METAL_PROFILE=1), SSD bytes, engine log; ours: 2026-11-16 reports and scripts.

Prefill: 25.5 s for the 4K prompt against 18 s (now ~18.6 s)

2026-11-17: ~219 tok/s, even with Strata. Since the table below: Strata's experts on our tile-GEMM kernels (Metal 4 matmul2d), then the Gated DeltaNet and hyper-connection prefill kernels (rows added at the end). The SSD is not the limit here: page-cache bypass and 4 or 16 I/O threads made little difference.

A prompt chunk routes to nearly every expert of every layer, so each chunk reads nearly all of them again. Strata reads 120 GB at ~6 GB/s on average (6.7 GB/s peak): it is limited by the SSD. We used to read 137–145 GB (1,024-token chunks) at ~2.3 GB/s, because a layer's reads only started once its routing was known. The routed experts' own compute is small: one group of 64 experts (1,280 sorted rows) takes 3.7 ms in float32, ~5.7 s for the whole 4K prompt (microbenchmark; bfloat16 would save 1 ms per group but gives wrong results on the M5, dequantizing to bfloat16 first is 4× slower). What changed:

4K prefill, same Mactok/sReadNotes
1,024-token chunks (before)68–77137–145 GB
Gated DeltaNet prefill on ggml-gated-delta-net+2.6%2.6× faster than the PyTorch reference per chunk (9.6 against 24.7 ms for 1,024 tokens); the whole layer is now ~4 s of a 4K prefill
2,048-token chunks89–9686 GBhalf the re-reads; 3,072 is slower and one 4,096-token chunk swaps on 32 GB
+ read the next layer's experts during prefill (#50717 prefill_readahead_routes)103–106114 GBthe SSD keeps reading while the GPU computes; ~28 GB of reads go unused; same output
+ vectorized attention indexer158–160114 GBpast 2,048 keys the model selects tokens with a Python loop and a host sync per query (1,355 ms per layer for the second 2K chunk); with a causal mask it is one scores matrix and one top-k (19.7 ms), same masks, same output
+ Strata's GGUF experts, our tile GEMM on matmul2d (2026-11-16)168–170~92 GBGGUF blocks dequantized inside the kernel, rows read through an index, a combine kernel instead of a scatter
+ 16-row tail tiles, next-layer installs behind compute, hyper-connection kernels for all tokens (2026-11-17)189–191
+ Gated DeltaNet prefill kernels: conv, q/k norm, delta-rule scan, gated norm (2026-11-17)212–226one 2K-token layer 67 → 28 ms

Reading 114 GB at 7.1 GB/s takes 16 s; the rest of our 25.5 s is compute that does not overlap with the reads: Gated DeltaNet (~4 s), experts (~5.7 s), hyper-connections (~3.6 s), attention and the rest. Earlier per-module profile (1K chunks): data.

Decode after a long prompt: 8.5 against 13.7 tok/s (now ~12.4)

2026-11-17, later: clean pass on the published code (decode 13.92, 4K prefill 227.6, after the document 205.6 / 12.41). Since ~11.4: one guessed expert of the next layer read per layer, after the layer's own reads and installed only when used (+3%; reading every guess competed with the real reads), the full-attention caches grown in place instead of by torch.cat (1.4 ms per token at 6K), and a bit-exact 3-dispatch hyper-connection step. The SSD is bandwidth-bound (~0.35 ms per expert read); the rest of the gap is the per-layer host round trip (~10 ms per token), which Strata avoids with Metal shared events.

2026-11-17: the biggest cost was not the cache but attention. Past the indexer's 2,048-token budget the model ran SDPA over the whole cache under a boolean mask (0.95 ms per attention layer at 6K keys) plus ~35 small indexer ops; reading only the selected blocks (0.12 ms) and fusing the indexer (0.11 ms) gave most of the +16%. A new replay with Strata's experts in per-size pools found that leaving long-prefill installs at the LRU's cold end does help a little (-7% misses measured), and that a Strata-like static ranked cache would miss 3× more. The remaining gap is expert reads: ~61 misses per token after the document against ~42 after a short prompt.

After the 6K document, 82.5% of our expert lookups hit the 12 GiB cache, against 88.5% after a short prompt: the answer to a long document spreads over more experts. We first suspected the prefill flushing the cache, but replaying the routing trace says otherwise: starting the answer from the LRU state the prefill leaves (79.0% at 10 GiB), an empty cache (78.2%), or the experts the prompt used most (78.9%) changes little, and even the answer's own most-used experts as a fixed set reach only 81.2%. The 12 GiB cache now fits next to the 2K-chunk prefill (7.9 → 8.2 tok/s), and the indexer keeps its pooled keys between decode steps (8.5). Strata waits for its misses too; its lead comes from the cheaper work around each wait, as at short prompts.

What changed on 2026-11-16

Decode went from 9.6 to 11.7 tok/s on the same Mac (alternating A/B runs, apps closed):

ChangeDecode (tok/s)Notes
Dense layers from Strata's GGUF on ggml-org/ggml-quantization9.58 / 9.29 → 10.85 / 10.56+13–14%. The ggml gemv is correct in bfloat16 on the M5 and runs at 125–146 GB/s. Gated DeltaNet value heads come in llama.cpp's tiled order and are mapped back at load. 4K prefill 68 → 70 tok/s
No expert read-ahead (prefetch_depth=0)11.04 / 11.15 / 11.09 → 11.80 / 11.78 / 11.58+5.6%. Guessing the next layers' experts hides ~9 ms of read waits per token but costs ~13 ms: guess routers, 2.7× the SSD reads (94 unused guesses per token), their device copies and evictions

Tried, no gain or within noise: a larger expert cache with the freed dense memory (13.4 GiB: +1%, and the memory watchdog killed one run); writing the Gated DeltaNet output in GGUF order instead of a gather; spin-waiting on the routing copy instead of a blocking wait (bit-identical, no gain); queuing cached experts on the GPU before the routing copy and adding missed ones after their read (+1.3% with read-ahead, +3.5% mean without, noisy, not kept); a separate thread pool for reads the GPU waits on; bypassing the page cache and splitting one expert's read in parallel reads (both slower). A better next-layer predictor (least squares on the router's error, trained on a prefill trace) cuts critical misses 25.7 → 20.6 per token offline, but read-ahead as a whole does not pay on this Mac.

What would close it

  1. Less work around each miss (decode): fewer kernel launches between the routing copy and the next layer, and fewer copies per missed expert. Strata's two tools here, an event handshake inside an encoded command buffer and reads into GPU-visible memory without a copy, are not available from PyTorch on MPS. Running cached experts while the misses are read gave ~3%.
  2. Prefill compute (4K: even; after a document: ~1.16×): the hyper-connections, the Gated DeltaNet conv, delta rule and output norm now run as Metal kernels for all tokens; next is overlapping the remaining dense compute (attention, projections) with the expert reads, and the attention / indexer past 2,048 keys; the reads alone take 16 s against Strata's 18 s total.
  3. Smaller experts: done (above), +12–14% at decode, +7% at prefill. Since then: 16-row tiles for each expert's last routes (less padding) and the next layer's finished reads installed behind this layer's compute (+3%). Next: overlapping the weight decoding with the matrix units, and Strata's latest variant (weights decoded straight into the matrix units' inputs).
  4. Fewer host round trips (~10 ms per token): the cost is the GPU idling while the host launches what follows each routing copy; it shrinks with fewer launches per layer.

Files: transformers.patch (#50717 at 18abece3ab) · raw reports, converter and benchmark · details.