← Back to the head-to-head and the gap analysis

Details: Qwen3.8-Flash-Next on a 32 GB Mac with Transformers

Experts streamed from disk through a bounded device cache · Apple M5 MacBook Pro, 32 GB unified memory, MPS · torch 2.15 · compared with Strata 0.1.60 run on the same Mac

Strata runs Qwen3.8-Flash-Next on a 32 GB Mac by streaming its experts from the SSD, since they are far too large to fit in memory. This page is a proof of concept that does the same with Transformers: from_pretrained with a new experts_offload option, running on PyTorch MPS instead of a dedicated engine.

~13.7 tok/sdecode, median of four different 200-token answers, cold cache first, with Strata's own IQ3_XXS experts and dense tensors. Strata on this same M5, same prompts: 14.8 (1.08×). With our 3-bit experts: ~11.6
~219 tok/sprefill of a 4K-token prompt in 2K chunks. Strata on this M5: 220 (even). 67–70 with the first 3-bit prototype
~11.4 tok/sdecode after a 6K-token document (prefill ~192). Strata: 13.7 / 222 (1.2× / 1.16×)
20–22.5 GiBdevice peak during the 4K prefill / after the document, 12 GiB expert cache included; experts are never loaded whole (24,576 of them, 1.75 MB each on average)

Updated 2026-11-17. Earlier sections keep their dated measurements (3-bit experts, 8-bit dense layers) as history; the current state is in the first tables, on the main page and in docs/kernels.md (every kernel, and which weights come from safetensors or from the GGUF).

Upstreaming expert streaming as from_pretrained(..., experts_offload=...): huggingface/transformers#50717 · RFC. The Qwen3.8-specific parts (disk-backed n-gram table, Metal decode kernels), the converter and every benchmark script live in tarekziade/moe-expert-streaming, a package on top of that PR.

Interactive architecture requires JavaScript. See the Architecture section below for the full model, storage and kernel details.

Results

Same protocol as Strata's Mac guide, decode tok/s on 200-token answers (greedy) and prefill tok/s on a 4,059-token prompt, OS page cache on, with one change: decode runs four different prompts once each (a story, Python code, a history explanation, French grammar), starting from a cold cache, instead of repeating the story. The expert cache carries over from one generation to the next, so a repeated prompt replays the same routing on a warm cache (91% hits instead of 89% for the story), and the story is among the fastest of the four: 10.7 tok/s, against 9.4–9.5 for the history and French answers, whose hit rate is 87% (3-bit experts, 2026-11-14). Other apps closed (an open browser costs ~0.7 tok/s on this Mac). Strata was installed and run on this same Mac with the same prompts (same tokenizer, same 200-token answers, the same 4,059- and 6,062-token prompts), thinking off, greedy, no MTP, no speculative drafts (0 proposed), no prompt reuse, a 12 GB expert arena and its one-read-per-expert copy of the experts, three rounds of the whole protocol (report, client). Its published M4 Mac mini numbers are kept for reference.

EngineExpertsDense layersExpert cacheDecode tok/sPrefill tok/s (4K)Perplexity
Strata 0.1.60 (this M5)IQ3_XXS (3.06 bits/weight), closest to our 3-bit expertspack quant12 GB16.3, 15.0, 14.4, 14.7
median 14.8 (3 rounds)
204, 221, 220
median 220
4.119 vs 4.052 for Q8_0 (+1.6%) ‡
Strata (published, M4 Mac mini 32 GB)IQ3_XXSpack quant11.5–14.7 GB9.265.8same build
Strata (published, M4 Mac mini 32 GB)Q2_0pack quant11.5–14.7 GB13.593.64.475 vs 4.052 for Q8_0 (+10.4%) ‡
Transformers (M5), 2026-11-17recommendedStrata's IQ3_XXS experts from its .experts file (IQ2/IQ3 gate/up, IQ4_NL or Q2_0 down, 1.75 MB each)Strata's GGUF tensors (Q4_K–Q6_K, IQ4_XS, IQ3_S) + 8-bit12 GiB (10 for the 4K prefill)medians 13.81, 13.80 (2 runs of the 4 prompts)212.2, 225.6 (2K chunks)4.984 §
Transformers (M5), 2026-11-163-bitsame GGUF tensors12 GiB (10)medians 11.59, 11.56156.5, 159.5 (2K chunks)4.925 §
Transformers (M5), 2026-11-143-bit (~3.5 bits/weight with scales and biases)8-bit12 GiB10.7, 10.7, 9.5, 9.4
median 10.0 (3 runs)
– †4.901
Transformers (M5), 2026-11-143-bit8-bit10 GiB–67.5, 69.2 (1K chunks, cold)4.901
Transformers (M5), earlier3-bit4-bit8 GiB6.4, 6.4 *77.1, 80.1 *5.355 (+9.3%)
Transformers (M5), earlier3-bitBF164 GiB5.0, 4.9 *72.7, 79.9 *4.901

§ Prefill-path perplexity on 1,024 tokens of Transformers' philosophy.md (6 GiB cache), the same text as the 4.901 measurements; with BF16 dense layers and 3-bit experts 4.9007. Kernel-to-kernel rounding moves it by up to ±0.4% (the PyTorch Gated DeltaNet path gives 4.9964 on the same build). Through the decode path (512 tokens), Strata's experts give 3.3116 and the 3-bit ones 3.4088, both with the GGUF dense tensors. The 2026-11-16/17 rows are alternating A/B runs; their reports are linked from the main page. Older rows: perplexity on 1,024 tokens of Transformers documentation with the same 3-bit experts; 8-bit dense layers pick the same top-1 token as BF16 at 95% of positions, 4-bit at 82%. 8 I/O threads. † a 4K prefill with a 12 GiB cache needs more free memory than this Mac had with its usual apps open (it ran at 60–70 tok/s on a quieter day); with 10 GiB it fits. Long prompts are prefilled in 1,024-token chunks, as Strata does on Macs under 96 GB. * measured before the read-pipeline and decode-kernel work, two runs of the same prompt (warm cache for the second). Neither Strata run uses MTP. Strata's decode rates are the third round's four prompts (the first round's story, on a cold arena: 15.9); the server's own timings agree with the client's within 1%. Through the decode path itself (teacher-forced, one token per step, 512 tokens), perplexity is 3.3558 with BF16 dense layers and 3.3629 with 8-bit ones (+0.2%, same top-1 token at 94.5% of positions; data, script). ‡ Strata's quality table, as quoted by ModelFit: perplexity relative to its own Q8_0 build, on text we do not know, so it is not comparable with our absolute numbers (ours compare dense-layer precisions with the same 3-bit experts).

Decode after a 6K-token document

Strata's third Mac row: 200 tokens decoded after reading a long document, so the attention layers see ~6K earlier tokens and the prefill has just churned the expert cache. Strata on this Mac now reads the document 1.16× faster and decodes after it 1.2× faster (3.4× and 1.9× on 2026-11-14).

EngineExpertsExpert cachePromptDecode tok/sPrefill tok/s
Strata 0.1.60 (this M5)IQ3_XXS12 GB6,062 tokens, 1K chunks13.6, 13.7, 13.8
median 13.7
222
Strata (published, M4 Mac mini 32 GB)IQ3_XXS11.5–14.7 GB6K document7.7–
Strata (published, M4 Mac mini 32 GB)Q2_011.5–14.7 GB6K document13.1–
Transformers (M5), 2026-11-17, GGUF denseStrata's IQ3_XXS12 GiB6,062 tokens, 2K chunks, cold11.56, 11.25197.5, 186.6
Transformers (M5), 2026-11-16, GGUF dense3-bit12 GiB6,062 tokens, 2K chunks, cold8.75, 8.94, 8.44158.2, 157.6, 157.2
Transformers (M5), 2026-11-14, 8-bit dense3-bit12 GiB6,062 tokens, 1K chunks, cold7.3 (1 run)64.5
Transformers (M5), 2026-11-14, 8-bit dense3-bit10 GiB6,062 tokens, 1K chunks, cold7.1 (1 run)64.5

2026-11-17: after the document, decode took 113 ms per token against 80 after a short prompt. Past the indexer's 2,048-token budget the model built a mask over the whole cache and ran SDPA over every key (0.95 ms per attention layer at 6K keys); a split-K flash decoding kernel now reads only the selected blocks and the tail (0.12 ms) and the indexer's decode step is two kernels (0.47 -> 0.11 ms per layer). The rest of the gap is expert reads: ~61 misses per token after the document against ~42 after a short prompt. Long prefill chunks now leave their expert installs at the LRU's cold end (-7% misses; see the cache-policy section).

2026-11-14: the document is the 4K prompt's synthetic harbour records, extended to 6K tokens; ask: summarize them. The 6K prefill peaks at 25.4 GiB of device memory with a 12 GiB cache (24.0 with 10 GiB), so it needs other apps closed. Decode drops from 10.0 to 7.3: the cache holds what the prefill touched (56% of lookups missed over the whole run; 292 GB read from the SSD) and full attention covers 6K tokens. Strata drops far less on the same Mac (14.8 to 13.7). The two cache sizes give the same answer for ~150 tokens, then different wording (the prefill groups experts differently, which changes the summation order). Reports: 12 GiB, 10 GiB.

A fair comparison

With both on the same Mac and the same prompts, and Transformers now running Strata's own IQ3_XXS experts and dense tensors, Strata is 1.08× faster at short decode, even at a 4K prefill and 1.2× faster at decode after a long document (with our 3-bit experts on 2026-11-14: 1.5×, ~3.2× and 1.9×). The gap was software. Strata publishes several expert quantizations; the one to compare with is the one closest in quality, not the fastest:

Strata buildSame top-1 token as Q8_0Perplexity (Q8_0: 4.052)Decode on the M4 mini (tok/s)Decode on this M5 (tok/s)
IQ3_XXS84.5%4.119 (+1.6%)9.214.8
Q2_080.6%4.475 (+10.4%)13.5not run

The two Macs

MachineChipCPUGPUMemory bandwidthUnified memorySSD
Strata's (Mac mini)M410 cores (4P + 6E)10 cores120 GB/s32 GBsize not stated
This one (MacBook Pro 14")M510 cores (4P + 6E)10 cores, Neural Accelerators in each153 GB/s32 GB1 TB, 7.1 GB/s measured

Apple's launch specs. Strata's guide says "M4 Mac mini 32 GB", which is the base M4 (the M4 Pro Mac mini comes with 24, 48 or 64 GB). The M5 has ~28% more memory bandwidth and its SSD reads 7.1 GB/s even for the 2.15 MB random reads experts need, so numbers measured on the two Macs are not directly comparable; the comparison above uses Strata run on this M5.

Architecture

Model: Qwen3.8-Flash-Next, 48 decoder layers, hidden size 2,560, vocabulary 248,320. Every layer has a token mixer and an MoE block, each wrapped in a 4-stream hyper-connection. Token mixers: 36 Gated DeltaNet (linear attention) layers and 12 full-attention layers (one every 4), whose attention is sparse: an indexer keeps at most 2,048 earlier tokens per query. MoE: 512 routed experts (top-10, intermediate size 640) plus one shared expert per layer. Layer 1 adds a per-layer n-gram embedding (320M rows × 160). MTP and vision are not loaded.

Where every weight lives

WeightsSizeStored asWhere
Routed experts (48 × 512), recommended~43 GBStrata's .experts file: its GGUF blocks (IQ2/IQ3 gate/up, IQ4_NL or Q2_0 down), 1.36–2.33 MB per expert by layer, one read eachSSD; 12 GiB in 10 device pools, one per expert size (LRU each)
Routed experts, 3-bit option49 GB3-bit MLX affine, group 64, 2.15 MB per expertSSD; 12 GiB in one device slot pool
N-gram embedding table51 GBFP8 (as in the checkpoint)SSD, read row by row (a few rows per token)
Large dense projections (Gated DeltaNet qkv/z/out, attention q/k/v/o, shared experts, lm_head), recommended~3.9 GBStrata's GGUF tensors as stored (Q4_K–Q6_K, IQ4_XS, IQ3_S); 8 tensors in Q2_0, which ggml's Metal kernels cannot multiply, stay 8-bitdevice
Other dense layers (Gated DeltaNet a/b, hyper-connection projections, indexer), and all of them in the 8-bit option~5.3 GB in the 8-bit option8-bit MLX affine (MetalConfig)device
Embeddings, norms, routers, conv weights, small gates~1.3 GBBF16device

Converted once from Qwen/Qwen3.8-Flash-Next-FP8 by convert_fp8.py (dense to BF16, experts to 3-bit, n-gram table kept FP8); MetalConfig quantizes the dense layers at load time. The recommended configuration then swaps in Strata's GGUF (gguf_dense=, gguf_experts=True); the converted checkpoint still provides the small layers, the n-gram table and the expert layout.

Kernels (recommended configuration)

Every kernel, the switches that turn each off, and which weights come from safetensors or from the GGUF: docs/kernels.md. Host code is pure Python; device work is Hub kernels or Metal source compiled with torch.mps.compile_shader.

PartDecode (one token)Prefill (2K-token chunks)
GGUF dense projectionsfused dequantize-gemv of ggml-org/ggml-quantizationits dequantize + torch.matmul
Other quantized linears (8-bit)quantized_matmul of kernels-community/mlx-quantization-metal-kernels v2 (float32 above ~30 rows on M5, see known issues)
Routed expertsggml's mul_mat_id on the size-class pools (up to 512 routes); our Q2_0 gather-gemv for the 30 layers whose down projection is Q2_0our tile GEMM on Metal 4's matmul2d (the M5's matrix units), dequantizing the GGUF blocks inside the kernel (ggml's dequantizers) and reading the hidden states through a row index, 32-row tiles plus 16-row tails; a combine kernel sums each token's routes
Gated DeltaNetone matmul for the four input projections; one kernel for the conv state, conv, SiLU and gates; one for the delta rule and the gated RMSNorm (~34 ops → 4 dispatches, bit-exact)conv + SiLU writing token-major rows (bit-exact); q/k L2 norm + k.q; the delta rule as a scan after Strata's gdn_scan_r (a lane per state column, the state in registers); the gated RMSNorm (sigmoid gate) as one kernel. 67 → 28 ms per 2K-token layer
Full attention (12 layers)past 2,048 keys: the indexer's query norm + rotary and block scores as two kernels, topk, then split-K flash decoding over the selected blocks and the tail only; below: SDPAthe indexer's token selection vectorized (one scores matrix and one topk instead of a Python loop per query), then SDPA under its mask
Hyper-connectionsgroup RMSNorm, SiLU, stream mixing: three kernels; the → 320 and → 4 projections fused into one quantized_matmulthe same three kernels for every token; separate projections (the fused one is wrong on M5 for many rows)
Routers, norms, embeddings, samplingPyTorch ops on MPS; the n-gram table read from disk row by row

With the 3-bit option, routed experts run through gather_qmm of the MLX kernel repository (sorted, 64 experts per call in prefill) and every dense layer through quantized_matmul.

Published kernels

Our Metal kernels are on the Kernel Hub, one repository each, and the package loads them from there (pinned to commits in kernels.lock). They are Metal source compiled at first use with torch.mps.compile_shader (Apple silicon, PyTorch MPS):

RepositoryUsed forKernelsLicense
tarekziade/gguf-mm-metalRouted experts, prefilltile GEMM over GGUF expert banks on Metal 4 matmul2d (blocks dequantized in the kernel, rows read through an index) + route combineMIT (ggml, Strata)
tarekziade/q2-0-metalRouted experts, decodeQ2_0 (ggml type 42) gather-gemv and dequantizeApache-2.0
tarekziade/gated-delta-net-decode-metalGated DeltaNet, decodeconv state + conv + SiLU + gates; delta rule + gated RMSNormApache-2.0
tarekziade/gdn-scan-metalGated DeltaNet, prefillcausal conv + SiLU, q/k L2 norm, the gated delta rule as a scanMIT (Strata)
tarekziade/qsa-decode-metalFull attention, decodesplit-K flash decoding over the selected blocks and the tail; indexer query norm/rotary and block scoresApache-2.0
tarekziade/gated-rms-norm-metalGated DeltaNet output normgated RMSNorm (SiLU or sigmoid gate)Apache-2.0
tarekziade/hyper-connection-metalHyper-connectionsgroup RMSNorm, SiLU, stream mixingApache-2.0

Load one with get_kernel("tarekziade/gdn-scan-metal", version=1, trust_remote_code=True) (kernels ≥ 0.18; the namespace is not a trusted kernel publisher). Several take the Transformers module itself (Qwen3.8's layer layout) rather than plain tensors.

Expert streaming

  1. Loading: from_pretrained(..., experts_offload={...}) builds the model without expert weights and loads everything else normally; quantization_config=MetalConfig(bits=8) quantizes the dense layers.
  2. Cache: preallocated device pools of expert slots with a per-layer table expert → slot kept on the host: one pool shared by all layers for the 3-bit experts (12 GiB hold ~6,000 of the 24,576), one per expert size for Strata's (10 sizes, 153 slots per layer on average, LRU each, gate/up and down in separate contiguous pools).
  3. Each MoE layer: the router's choices are copied to the host (the one host synchronization per layer), misses are read on 8 I/O threads into recycled pinned host buffers (one contiguous read per expert, through descriptors opened once), copied to free slots without blocking, and the layer sends its slot indices in one small copy and runs gather_qmm on the pool.
  4. Prediction (optional, prefetch_depth, off by default): the routers of the next layers run on the current hidden states and their guessed experts are read in the background. On the M5 the guesses cost more than they hid (decode 11.1 → 11.7 tok/s without them).
  5. Prefill: a 2K chunk touches nearly every expert of every layer. The next layer's experts are read while the current one computes (prefill_readahead_routes), and the finished reads are installed behind its compute. Chunks of 512 tokens or more leave their installs at the LRU's cold end and do not refresh hits: plain LRU evicts each expert just before the next chunk needs it, and leaves decode a cache of scan leftovers.

Decode speedups since the first published numbers (2026-11-14, 3-bit experts)

Each measured as alternating A/B pairs (4 prompts per run, 12 GiB cache), commits on the POC branch. Later changes (GGUF dense layers and experts, prefill kernels, decode after a long document) are on the main page and in the package README:

ChangeDecode tok/s (before → after)Output
Queue the shared expert after the routing copy (pinned buffer + event), so it runs while the host does the cache bookkeeping8.25 → 8.55 (+3.6%)identical
Commit queued MPS work early: record an event after each decoder-layer sub-block and after the expert kernels8.52 → 9.40 (+10.3%)identical
Hyper-connections: one 8-bit matmul for the 10240 → 320 and 10240 → 4 projections (the 4-row call alone took 49 µs, twice the 320-row one)9.48 → 10.03 (+5.8%)rounding differs; decode-path perplexity 3.3642 → 3.3629 (512 tokens), top-1 agreement 96.7%

The MPS commit issue. PyTorch's MPS stream encodes small kernels into a command buffer and commits it only at the next synchronization (or under memory pressure). Decode is made of small kernels and synchronizes once per MoE layer, so after each routing wait the GPU idled while the host encoded the experts and the next layer, then ran them only at the next wait. Launching 1,000 small matmuls and then doing 106 ms of host work takes 213 ms; recording an event after the launches (it commits without waiting) brings it to 109 ms (reproducer). With the experts replaced by zeros, the same fix takes decode from 67 to 56 ms per token. It would be best fixed in PyTorch; Transformers works around it with experts_offload={"mps_commit": True} (the default on MPS).

Do these changes help other MoEs?

Two of the three decode changes are generic, one targets a family of models, one is Qwen3.8-only:

ChangeApplies toChecked
Routing copy to a pinned buffer behind an eventevery streamed MoEOLMoE: identical output, never slower
Early MPS commits (mps_commit)every streamed MoE on MPS (hooks on each decoder layer's sub-blocks)OLMoE: +3% when streaming, more while the cache fills; Qwen3.8: +10%
Shared expert queued after the routing copyMoE blocks with gate, shared_expert and shared_expert_gate: Qwen2-MoE, Qwen3-Next, Qwen3.5-MoE, Qwen3-Omni (talker), Qwen3.8all five blocks have the same forward, which the override reproduces; for upstreaming it should be an explicit opt-in per model rather than a match on attribute names
Fused hyper-connection projectionsQwen3.8 only (its own kernel file)–

OLMoE-1B-7B (16 layers × 64 experts, 4-bit experts, 3.4 GB), same four prompts, 200 tokens, greedy, cold cache first; before is the branch just before the three changes (script, reports):

Expert cacheBefore (decode median tok/s)Now, commits offNowOutput
1 GiB (streaming, 85% hits; 2 runs each)29.0, 32.732.7, 32.133.2, 33.8identical text; perplexity 8.2953 (prefill), 8.2484 (decode path) in every run
4 GiB (all experts fit; 1 run each)60.2–73.2identical text

Streaming OLMoE gains ~3%, within run-to-run noise: its dense layers are small, so the GPU had little host encoding to wait for. With a cache that holds every expert, the gain comes from the first two answers, while the cache fills (66–67 against 47–48 tok/s); once every expert is resident and no layer synchronizes, both run at the same speed (79 / 84 against 72 / 85). Unrelated to these changes: with every expert resident, prefill takes a different matmul path (perplexity 8.2144 instead of 8.2953) and one of the four answers differs from the streamed one, before and after alike.

What limited decode (2026-11-14, 3-bit experts, 8-bit dense layers)

Since then the dense projections read Strata's smaller GGUF tensors, the experts are 1.75 MB instead of 2.15, and decode went from 10.0 to ~13.7 tok/s. At that time, the dense layers, measured by ablation: experts replaced by zeros, then each module category replays its first output instead of computing, which shows what it would save if it were free (data, script). Floor: reading the category's 8-bit weights once at the M5's 153 GB/s.

ms per token (dense part: 56 ms)MeasuredWeight-read floor
Gated DeltaNet (36 layers)21.913.6
Hyper-connections (97), before the projection fusion (~5 ms less after)14.54.2
Full attention (12 layers, with the indexer)6.84.0
lm_head5.74.2
Shared experts (48)2.61.5
Routers (48)2.10.4
PLE n-gram embedding (disk rows; its own host sync cost 10 ms before the commit fix)1.0–

Would a warmup or a smarter cache policy help? Not for decode

The routing of the four decode prompts (plus the story again) was logged and replayed through a cache simulator at the 12 GiB size (5,991 expert slots), counting only the experts the model asks for (no next-layer guesses, hence a few points below the measured hit rates). Belady's policy evicts the expert needed furthest in the future: no policy can beat it.

Simulated hit rateStoryPythonHistoryFrench
LRU, cold cache (current)86.2%86.0%82.1%81.3%
LFU86.6%84.7%79.5%80.1%
LRU, warmed by the previous prompt–86.9%82.9%82.5%
25% of the cache pinned to the experts most used by the other prompts, LRU for the rest83.8%84.3%78.8%79.4%
Same, 50% pinned80.4%81.9%77.3%76.3%
Belady (optimal)89.3%88.9%87.4%86.8%

One 200-token answer touches 10.7–12.6k distinct experts, about twice what the cache holds, and 10–13% of its lookups are first uses. For the story, the optimum at 12 GiB equals that first-use floor (89.3%), and LRU reaches it with a 1.5× cache. Consecutive prompts share only 51–72% of their experts, so warming up gains about one point, and pinning "popular" experts loses up to 10. New experts keep appearing through the answer (1.7 per layer and token at the start, 0.2 by token 175), so long conversations would gain a little from a better policy, but a 200-token benchmark would not. Replay output, simulator, routing trace.

After a long prefill (2026-11-17, Strata's experts, per-size pools). The same replay with the four story prompts followed by the 6K document, in misses per decode token (data): LRU 66.6 after the document and 42.5 on the stories; a static cache ranked by the stories' decode frequency, as Strata's arena is, 205 and 89 (77.7 and 48.9 with half of it static): routing has strong short-term locality, so LRU wins even on the prompts the ranking came from. What did help is the prefill: each 2K chunk cycles through nearly every expert, more than a pool holds, so LRU evicts each one just before the next chunk needs it. Leaving long-prefill installs at the LRU's cold end cut the 6K prefill's misses from 50.2K to 36.2K and the decode misses after it by 8% (measured: -7% misses, +0.6% decode).

CUDA (preview): one A10G

The same converted model also runs on an NVIDIA A10G (24 GB) through the same package (run, reports): BF16 dense layers, 3-bit experts in an 8 GiB device cache, run by a Triton kernel that unpacks the MLX format in registers (matches PyTorch at bfloat16 precision for 2-8 bits, check). The pod has 83.5 GiB of RAM and 22.8 CPUs; the experts sit on a network file system.

A10G, same 4 promptsDecode tok/sMedian
Experts read through the page cache2.8, 3.6, 4.5, 4.84.1
All experts preloaded in a pinned host tier (as Strata keeps them in RAM)5.8, 5.8, 5.7, 5.8 (repeat: 5.9, 5.8, 5.8, 5.8)5.8
Host tier filled on demand (no preload)2.9, 3.2, 4.1, 4.73.6

Quality: decode-path perplexity (teacher-forced, 512 tokens) is 3.4366 on CUDA (BF16 dense) vs 3.3558 on the Mac (BF16 dense); the two agree on the top-1 token 95.3% of the time. The host tier (host_cache_bytes, host_preload) is now in the PR. An answer that differed in an earlier run came from CUDA decode not being bit-deterministic run to run, not from the host tier.

Early numbers. Decode is held back by the dense layers, which run as plain PyTorch ops on CUDA: the Metal kernels (Gated DeltaNet step, hyper-connections) have no CUDA counterparts yet, and the per-expert prefill path makes a 4K prompt slow (17 tok/s).

Known issues

Files: transformers.patch (the diff of huggingface/transformers#50717 at 18abece3ab, the commit every 2026-11-16/17 number uses) · raw reports, converter and benchmark.