Clean benchmark pass: decode 13.92, 4K prefill 227.6, after 6K doc 205.6 / 12.41 (guessed reads, in-place caches, HC step)
Browse files- index.html +16 -9
- results/qwen38_local_m5/2026-11-17-clean-pass/decode.json +144 -0
- results/qwen38_local_m5/2026-11-17-clean-pass/long.json +58 -0
- results/qwen38_local_m5/2026-11-17-clean-pass/prefill.json +58 -0
- results/qwen38_local_m5/2026-11-17-clean-pass/summary.txt +3 -0
- rfc.html +6 -6
- rfc.md +4 -4
index.html
CHANGED
|
@@ -80,9 +80,9 @@
|
|
| 80 |
<a href="details.html"><b>All the details</b></a> (architecture, kernels, earlier measurements, cache-policy study, CUDA, known issues)</p>
|
| 81 |
|
| 82 |
<div class="kpis">
|
| 83 |
-
<div class="kpi"><b>1.
|
| 84 |
-
<div class="kpi"><b>1.0×</b><span>
|
| 85 |
-
<div class="kpi"><b>1.
|
| 86 |
</div>
|
| 87 |
|
| 88 |
<h2>Head to head</h2>
|
|
@@ -90,13 +90,13 @@
|
|
| 90 |
<table class="wrap">
|
| 91 |
<thead><tr><th>Same M5, same prompts</th><th>Transformers, GGUF experts</th><th>Transformers, 3-bit experts</th><th>Strata (IQ3_XXS)</th><th>Strata's lead</th></tr></thead>
|
| 92 |
<tbody>
|
| 93 |
-
<tr><td>Decode, 200 tokens, median of 4 prompts (tok/s)</td><td>13.
|
| 94 |
-
<tr><td>Prefill, 4,059-token prompt (tok/s)</td><td>
|
| 95 |
-
<tr><td>Decode after a 6,062-token document (tok/s)</td><td>
|
| 96 |
-
<tr><td>Prefill of that document (tok/s)</td><td>
|
| 97 |
<tr class="group"><td>Routed experts</td><td>Strata's IQ2/IQ3 + IQ4_NL/Q2_0, 1.75 MB each, from its <code>.experts</code> file</td><td>3-bit MLX affine, 2.15 MB each</td><td>IQ2/IQ3 + IQ4_NL/Q2_0, 1.75 MB each</td><td></td></tr>
|
| 98 |
<tr><td>Dense layers (weights read per token)</td><td colspan="2">Strata's own GGUF tensors (Q4_K–Q6_K, IQ4_XS, IQ3_S) + 8-bit hyper-connections, ~3.9 GB</td><td>4–6-bit K-quants and IQ quants, BF16 hyper-connections in the GGUF, ~3.8 GB</td><td></td></tr>
|
| 99 |
-
<tr><td>Expert cache on the device</td><td>12 GiB, LRU per size class</td><td>12 GiB, LRU</td><td>12 GB, ranked by a routing profile, no eviction</td><td></td></tr>
|
| 100 |
</tbody>
|
| 101 |
</table>
|
| 102 |
</div>
|
|
@@ -246,7 +246,14 @@
|
|
| 246 |
<p class="note">Reading 114 GB at 7.1 GB/s takes 16 s; the rest of our 25.5 s is compute that does not overlap with the
|
| 247 |
reads: Gated DeltaNet (~4 s), experts (~5.7 s), hyper-connections (~3.6 s), attention and the rest. Earlier per-module profile (1K chunks): <a href="results/qwen38_local_m5/prefill_profile.json">data</a>.</p>
|
| 248 |
|
| 249 |
-
<h3>Decode after a long prompt: 8.5 against 13.7 tok/s (now ~
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 250 |
<p class="note">2026-11-17: the biggest cost was not the cache but attention. Past the indexer's 2,048-token budget the
|
| 251 |
model ran SDPA over the whole cache under a boolean mask (0.95 ms per attention layer at 6K keys) plus ~35 small
|
| 252 |
indexer ops; reading only the selected blocks (0.12 ms) and fusing the indexer (0.11 ms) gave most of the +16%. A new
|
|
|
|
| 80 |
<a href="details.html"><b>All the details</b></a> (architecture, kernels, earlier measurements, cache-policy study, CUDA, known issues)</p>
|
| 81 |
|
| 82 |
<div class="kpis">
|
| 83 |
+
<div class="kpi"><b>1.06×</b><span>Strata's lead at decode: 14.8 against ~13.9 tok/s with its own GGUF experts (200-token answers to four different prompts; 1.26× with our 3-bit experts, 1.5× before the 2026-11-16 changes)</span></div>
|
| 84 |
+
<div class="kpi"><b>1.0×</b><span>even at prefill: 220 against ~228 tok/s on a 4,059-token prompt (~158 with 3-bit experts; 3.2× before the 2026-11-16 changes)</span></div>
|
| 85 |
+
<div class="kpi"><b>1.1×</b><span>its lead at decode after a 6,062-token document: 13.7 against ~12.4 tok/s (1.4× before the 2026-11-17 attention kernels, 1.9× before 2026-11-16); 1.08× at prefilling that document (222 against ~206)</span></div>
|
| 86 |
</div>
|
| 87 |
|
| 88 |
<h2>Head to head</h2>
|
|
|
|
| 90 |
<table class="wrap">
|
| 91 |
<thead><tr><th>Same M5, same prompts</th><th>Transformers, GGUF experts</th><th>Transformers, 3-bit experts</th><th>Strata (IQ3_XXS)</th><th>Strata's lead</th></tr></thead>
|
| 92 |
<tbody>
|
| 93 |
+
<tr><td>Decode, 200 tokens, median of 4 prompts (tok/s)</td><td>13.92</td><td>11.59 / 11.56</td><td class="best">14.8</td><td>1.06×</td></tr>
|
| 94 |
+
<tr><td>Prefill, 4,059-token prompt (tok/s)</td><td>227.6</td><td>156.5 / 159.5</td><td class="best">220</td><td>1.0×</td></tr>
|
| 95 |
+
<tr><td>Decode after a 6,062-token document (tok/s)</td><td>12.41</td><td>8.75 / 8.94 / 8.44</td><td class="best">13.7</td><td>1.1×</td></tr>
|
| 96 |
+
<tr><td>Prefill of that document (tok/s)</td><td>205.6</td><td>158.2 / 157.6 / 157.2</td><td class="best">222</td><td>1.08×</td></tr>
|
| 97 |
<tr class="group"><td>Routed experts</td><td>Strata's IQ2/IQ3 + IQ4_NL/Q2_0, 1.75 MB each, from its <code>.experts</code> file</td><td>3-bit MLX affine, 2.15 MB each</td><td>IQ2/IQ3 + IQ4_NL/Q2_0, 1.75 MB each</td><td></td></tr>
|
| 98 |
<tr><td>Dense layers (weights read per token)</td><td colspan="2">Strata's own GGUF tensors (Q4_K–Q6_K, IQ4_XS, IQ3_S) + 8-bit hyper-connections, ~3.9 GB</td><td>4–6-bit K-quants and IQ quants, BF16 hyper-connections in the GGUF, ~3.8 GB</td><td></td></tr>
|
| 99 |
+
<tr><td>Expert cache on the device</td><td>12 GiB, LRU per size class, one guessed read per layer</td><td>12 GiB, LRU</td><td>12 GB, ranked by a routing profile, no eviction</td><td></td></tr>
|
| 100 |
</tbody>
|
| 101 |
</table>
|
| 102 |
</div>
|
|
|
|
| 246 |
<p class="note">Reading 114 GB at 7.1 GB/s takes 16 s; the rest of our 25.5 s is compute that does not overlap with the
|
| 247 |
reads: Gated DeltaNet (~4 s), experts (~5.7 s), hyper-connections (~3.6 s), attention and the rest. Earlier per-module profile (1K chunks): <a href="results/qwen38_local_m5/prefill_profile.json">data</a>.</p>
|
| 248 |
|
| 249 |
+
<h3>Decode after a long prompt: 8.5 against 13.7 tok/s (now ~12.4)</h3>
|
| 250 |
+
<p class="note">2026-11-17, later: <a href="results/qwen38_local_m5/2026-11-17-clean-pass/">clean pass</a> on the
|
| 251 |
+
published code (decode 13.92, 4K prefill 227.6, after the document 205.6 / 12.41). Since ~11.4: one guessed expert of
|
| 252 |
+
the next layer read per layer, after the layer's own reads and installed only when used (+3%; reading every guess
|
| 253 |
+
competed with the real reads), the full-attention caches grown in place instead of by <code>torch.cat</code>
|
| 254 |
+
(1.4 ms per token at 6K), and a bit-exact 3-dispatch hyper-connection step. The SSD is bandwidth-bound (~0.35 ms
|
| 255 |
+
per expert read); the rest of the gap is the per-layer host round trip (~10 ms per token), which Strata avoids with
|
| 256 |
+
Metal shared events.</p>
|
| 257 |
<p class="note">2026-11-17: the biggest cost was not the cache but attention. Past the indexer's 2,048-token budget the
|
| 258 |
model ran SDPA over the whole cache under a boolean mask (0.95 ms per attention layer at 6K keys) plus ~35 small
|
| 259 |
indexer ops; reading only the selected blocks (0.12 ms) and fusing the indexer (0.11 ms) gave most of the +16%. A new
|
results/qwen38_local_m5/2026-11-17-clean-pass/decode.json
ADDED
|
@@ -0,0 +1,144 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "Qwen3.8-Flash-Next (from Qwen/Qwen3.8-Flash-Next-FP8)",
|
| 3 |
+
"experts": "3-bit MLX affine, group 64",
|
| 4 |
+
"cache_gib": 12.0,
|
| 5 |
+
"io_threads": 8,
|
| 6 |
+
"page_cache": "on",
|
| 7 |
+
"dense": "8-bit MetalConfig",
|
| 8 |
+
"device": "mps",
|
| 9 |
+
"host_cache_gib": 0.0,
|
| 10 |
+
"host_preload": false,
|
| 11 |
+
"gguf_dense": [
|
| 12 |
+
"~/.local/share/strata/Strata-data/models/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf"
|
| 13 |
+
],
|
| 14 |
+
"gguf_experts": true,
|
| 15 |
+
"prefetch_depth": 1,
|
| 16 |
+
"guess_limit": 1,
|
| 17 |
+
"preload_profile": null,
|
| 18 |
+
"prefill_chunk_size": null,
|
| 19 |
+
"torch": "2.15.0",
|
| 20 |
+
"platform": "macOS-26.5.1-arm64-arm-64bit",
|
| 21 |
+
"load_s": 6.17273679189384,
|
| 22 |
+
"device_after_load_bytes": 6580027392,
|
| 23 |
+
"runs": {
|
| 24 |
+
"story": [
|
| 25 |
+
{
|
| 26 |
+
"prompt": "Write a short story about a lighthouse keeper who finds a message in a bottle.",
|
| 27 |
+
"prompt_tokens": 29,
|
| 28 |
+
"new_tokens": 200,
|
| 29 |
+
"time_to_first_token_s": 2.8673544998746365,
|
| 30 |
+
"prefill_tokens_per_s": 10.113852333664326,
|
| 31 |
+
"decode_tokens_per_s": 14.775250527132199,
|
| 32 |
+
"total_s": 16.33582325000316,
|
| 33 |
+
"store": {
|
| 34 |
+
"hits": 88771,
|
| 35 |
+
"misses": 11382,
|
| 36 |
+
"evictions": 4042,
|
| 37 |
+
"not_admitted": 0,
|
| 38 |
+
"slots": 0,
|
| 39 |
+
"pools": 0,
|
| 40 |
+
"prefetched": 14842,
|
| 41 |
+
"prefetch_used": 11379,
|
| 42 |
+
"installed_ahead": 0,
|
| 43 |
+
"resident_bytes": 12816537600,
|
| 44 |
+
"peak_cache_bytes": 12816537600,
|
| 45 |
+
"read_bytes": 25956787200,
|
| 46 |
+
"read_calls": 14845,
|
| 47 |
+
"host_slots": 0,
|
| 48 |
+
"host_hits": 0,
|
| 49 |
+
"host_misses": 0
|
| 50 |
+
},
|
| 51 |
+
"text": "The fog had been thick for three days, a gray wool blanket that smothered the world beyond the glass of the lighthouse. Elias, a man whose skin had taken on the texture of the sea itself\u2014rough, salted, and weathered\u2014wiped the condensation from the lantern room window. He was sixty-two, and he had kept the light burning for thirty of those years. He knew the rhythm of the tides better than he knew the rhythm of his own heartbeat.\n\nIt was during the low tide, when the jagged rocks of the headland emerged like broken teeth, that he saw it.\n\nIt wasn\u2019t a bottle, not at first. It was a glint of green glass, half-buried in the silt near the base of the tower. Elias descended the spiral stairs, his boots heavy on the iron steps, and stepped out into the damp chill. The wind bit at his cheeks, carrying the scent of brine and decay."
|
| 52 |
+
},
|
| 53 |
+
{
|
| 54 |
+
"prompt": "Write a Python function that parses an ISO 8601 date string without using datetime, with tests.",
|
| 55 |
+
"prompt_tokens": 34,
|
| 56 |
+
"new_tokens": 200,
|
| 57 |
+
"time_to_first_token_s": 1.8601071669254452,
|
| 58 |
+
"prefill_tokens_per_s": 18.278516745999266,
|
| 59 |
+
"decode_tokens_per_s": 14.45962102801062,
|
| 60 |
+
"total_s": 15.62257020897232,
|
| 61 |
+
"store": {
|
| 62 |
+
"hits": 89753,
|
| 63 |
+
"misses": 10845,
|
| 64 |
+
"evictions": 10841,
|
| 65 |
+
"not_admitted": 0,
|
| 66 |
+
"slots": 0,
|
| 67 |
+
"pools": 0,
|
| 68 |
+
"prefetched": 15301,
|
| 69 |
+
"prefetch_used": 10838,
|
| 70 |
+
"installed_ahead": 0,
|
| 71 |
+
"resident_bytes": 7065600,
|
| 72 |
+
"peak_cache_bytes": 7065600,
|
| 73 |
+
"read_bytes": 26772121600,
|
| 74 |
+
"read_calls": 15308,
|
| 75 |
+
"host_slots": 0,
|
| 76 |
+
"host_hits": 0,
|
| 77 |
+
"host_misses": 0
|
| 78 |
+
},
|
| 79 |
+
"text": "```python\nimport re\n\n\ndef parse_iso8601(date_string: str) -> dict:\n \"\"\"\n Parse an ISO 8601 date string and return a dictionary with year, month, day,\n hour, minute, second, and timezone offset information.\n \n Supported formats:\n - YYYY-MM-DD\n - YYYY-MM-DDTHH:MM:SS\n - YYYY-MM-DDTHH:MM:SSZ\n - YYYY-MM-DDTHH:MM:SS+HH:MM\n - YYYY-MM-DDTHH:MM:SS-HH:MM\n - YYYY-MM-DDTHH:MM\n - YYYY-MM-DDTHH:MMZ\n - YYYY-MM-DDTHH:MM+HH:MM\n - YYYY-MM-DDTHH:MM-HH:MM\n \n Returns a dict with keys:\n - year: int\n - month: int\n "
|
| 80 |
+
},
|
| 81 |
+
{
|
| 82 |
+
"prompt": "Explain the causes of the French Revolution to a high school student.",
|
| 83 |
+
"prompt_tokens": 26,
|
| 84 |
+
"new_tokens": 200,
|
| 85 |
+
"time_to_first_token_s": 1.7635814580135047,
|
| 86 |
+
"prefill_tokens_per_s": 14.742727012613502,
|
| 87 |
+
"decode_tokens_per_s": 13.37062296240191,
|
| 88 |
+
"total_s": 16.646956792101264,
|
| 89 |
+
"store": {
|
| 90 |
+
"hits": 85662,
|
| 91 |
+
"misses": 14417,
|
| 92 |
+
"evictions": 14417,
|
| 93 |
+
"not_admitted": 0,
|
| 94 |
+
"slots": 0,
|
| 95 |
+
"pools": 0,
|
| 96 |
+
"prefetched": 18495,
|
| 97 |
+
"prefetch_used": 14408,
|
| 98 |
+
"installed_ahead": 0,
|
| 99 |
+
"resident_bytes": 0,
|
| 100 |
+
"peak_cache_bytes": 0,
|
| 101 |
+
"read_bytes": 32367974400,
|
| 102 |
+
"read_calls": 18504,
|
| 103 |
+
"host_slots": 0,
|
| 104 |
+
"host_hits": 0,
|
| 105 |
+
"host_misses": 0
|
| 106 |
+
},
|
| 107 |
+
"text": "To understand the French Revolution, imagine a pressure cooker. For years, heat (problems) was building up inside, but the lid (the monarchy) was sealed tight. Eventually, the pressure became too great, and the lid blew off.\n\nHere\u2019s a breakdown of the main causes, explained simply:\n\n### 1. Social Inequality: The \"Three Estates\" System\nFrance was divided into three social classes, called **Estates**:\n* **First Estate:** The Clergy (church leaders). They owned land and paid almost no taxes.\n* **Second Estate:** The Nobility (rich aristocrats). They held top government jobs, owned land, and also paid almost no taxes.\n* **Third Estate:** Everyone else (peasants, city workers, merchants, lawyers, doctors). This was about **97% of the population**. They paid almost all the taxes but had no political power.\n\n**Why it mattered:** The Third Estate felt"
|
| 108 |
+
},
|
| 109 |
+
{
|
| 110 |
+
"prompt": "Translate into French and explain the grammar: 'The children would have gone if it had not rained.'",
|
| 111 |
+
"prompt_tokens": 33,
|
| 112 |
+
"new_tokens": 200,
|
| 113 |
+
"time_to_first_token_s": 1.807607457973063,
|
| 114 |
+
"prefill_tokens_per_s": 18.256176059930688,
|
| 115 |
+
"decode_tokens_per_s": 13.314510919176882,
|
| 116 |
+
"total_s": 16.753706583054736,
|
| 117 |
+
"store": {
|
| 118 |
+
"hits": 87687,
|
| 119 |
+
"misses": 13183,
|
| 120 |
+
"evictions": 13183,
|
| 121 |
+
"not_admitted": 0,
|
| 122 |
+
"slots": 0,
|
| 123 |
+
"pools": 0,
|
| 124 |
+
"prefetched": 17492,
|
| 125 |
+
"prefetch_used": 13174,
|
| 126 |
+
"installed_ahead": 0,
|
| 127 |
+
"resident_bytes": 0,
|
| 128 |
+
"peak_cache_bytes": 0,
|
| 129 |
+
"read_bytes": 30599449600,
|
| 130 |
+
"read_calls": 17501,
|
| 131 |
+
"host_slots": 0,
|
| 132 |
+
"host_hits": 0,
|
| 133 |
+
"host_misses": 0
|
| 134 |
+
},
|
| 135 |
+
"text": "### Translation\n\n**Les enfants seraient partis s\u2019il n\u2019avait pas plu.**\n\n*(Note: If the children are all female, it would be \"Les enfants seraient parties...\")*\n\n---\n\n### Grammar Explanation\n\nThis sentence is a classic example of the **Third Conditional** in English, which describes a hypothetical situation in the past that did not happen. In French, this is expressed using the **Conditionnel Pass\u00e9** (Past Conditional) in the main clause and the **Plus-que-parfait** (Past Perfect) in the subordinate clause.\n\n#### 1. Main Clause: \"The children would have gone\"\n* **French:** *Les enfants seraient partis*\n* **Tense:** **Conditionnel Pass\u00e9** (Past Conditional)\n* **Structure:** Auxiliary verb *\u00eatre* in the **Conditionnel Pr\u00e9sent** + Past Participle.\n * *Serait* (conditional of *\u00eatre*) + *partis* ("
|
| 136 |
+
}
|
| 137 |
+
],
|
| 138 |
+
"prefill_4k": [],
|
| 139 |
+
"long_doc_6k": []
|
| 140 |
+
},
|
| 141 |
+
"decode_median": 13.915121995206265,
|
| 142 |
+
"device_peak_sampled_bytes": 19821068288,
|
| 143 |
+
"process_peak_rss_bytes": 8927035392
|
| 144 |
+
}
|
results/qwen38_local_m5/2026-11-17-clean-pass/long.json
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "Qwen3.8-Flash-Next (from Qwen/Qwen3.8-Flash-Next-FP8)",
|
| 3 |
+
"experts": "3-bit MLX affine, group 64",
|
| 4 |
+
"cache_gib": 12.0,
|
| 5 |
+
"io_threads": 8,
|
| 6 |
+
"page_cache": "on",
|
| 7 |
+
"dense": "8-bit MetalConfig",
|
| 8 |
+
"device": "mps",
|
| 9 |
+
"host_cache_gib": 0.0,
|
| 10 |
+
"host_preload": false,
|
| 11 |
+
"gguf_dense": [
|
| 12 |
+
"~/.local/share/strata/Strata-data/models/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf"
|
| 13 |
+
],
|
| 14 |
+
"gguf_experts": true,
|
| 15 |
+
"prefetch_depth": 1,
|
| 16 |
+
"guess_limit": 1,
|
| 17 |
+
"preload_profile": null,
|
| 18 |
+
"prefill_chunk_size": 2048,
|
| 19 |
+
"torch": "2.15.0",
|
| 20 |
+
"platform": "macOS-26.5.1-arm64-arm-64bit",
|
| 21 |
+
"load_s": 4.109033874934539,
|
| 22 |
+
"device_after_load_bytes": 6580027392,
|
| 23 |
+
"runs": {
|
| 24 |
+
"story": [],
|
| 25 |
+
"prefill_4k": [],
|
| 26 |
+
"long_doc_6k": [
|
| 27 |
+
{
|
| 28 |
+
"prompt_tokens": 6062,
|
| 29 |
+
"new_tokens": 200,
|
| 30 |
+
"time_to_first_token_s": 29.48017662507482,
|
| 31 |
+
"prefill_tokens_per_s": 205.6297042278869,
|
| 32 |
+
"decode_tokens_per_s": 12.406908215060852,
|
| 33 |
+
"total_s": 45.51962791709229,
|
| 34 |
+
"store": {
|
| 35 |
+
"hits": 104412,
|
| 36 |
+
"misses": 42692,
|
| 37 |
+
"evictions": 57763,
|
| 38 |
+
"not_admitted": 770,
|
| 39 |
+
"slots": 0,
|
| 40 |
+
"pools": 0,
|
| 41 |
+
"prefetched": 82483,
|
| 42 |
+
"prefetch_used": 65870,
|
| 43 |
+
"installed_ahead": 23185,
|
| 44 |
+
"resident_bytes": 12823603200,
|
| 45 |
+
"peak_cache_bytes": 12823603200,
|
| 46 |
+
"read_bytes": 144245657600,
|
| 47 |
+
"read_calls": 82490,
|
| 48 |
+
"host_slots": 0,
|
| 49 |
+
"host_hits": 0,
|
| 50 |
+
"host_misses": 0
|
| 51 |
+
},
|
| 52 |
+
"text": "Based on the records provided, it is **impossible to summarize the actual history** of the harbour town.\n\nHere is why:\n\n1. **Lack of Specific Data:** The notes provided are **meta-descriptions** (descriptions *about* the records) rather than the records themselves. Each note states that an entry \"describes one season in detail, with names, dates and the weather,\" but **none of the notes actually contain those names, dates, weather details, ship logs, or cargo lists.**\n2. **Repetitive Placeholder Text:** All 133 notes are identical in structure and content, differing only by the entry number. They function as placeholders or table of contents entries, not as historical data.\n3. **No Narrative Content:** There is no information about:\n * Specific ships or their voyages.\n * Specific storms or their impacts.\n * Specific cargo types or quantities.\n "
|
| 53 |
+
}
|
| 54 |
+
]
|
| 55 |
+
},
|
| 56 |
+
"device_peak_sampled_bytes": 23059480576,
|
| 57 |
+
"process_peak_rss_bytes": 10719444992
|
| 58 |
+
}
|
results/qwen38_local_m5/2026-11-17-clean-pass/prefill.json
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "Qwen3.8-Flash-Next (from Qwen/Qwen3.8-Flash-Next-FP8)",
|
| 3 |
+
"experts": "3-bit MLX affine, group 64",
|
| 4 |
+
"cache_gib": 10.0,
|
| 5 |
+
"io_threads": 8,
|
| 6 |
+
"page_cache": "on",
|
| 7 |
+
"dense": "8-bit MetalConfig",
|
| 8 |
+
"device": "mps",
|
| 9 |
+
"host_cache_gib": 0.0,
|
| 10 |
+
"host_preload": false,
|
| 11 |
+
"gguf_dense": [
|
| 12 |
+
"~/.local/share/strata/Strata-data/models/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf"
|
| 13 |
+
],
|
| 14 |
+
"gguf_experts": true,
|
| 15 |
+
"prefetch_depth": 1,
|
| 16 |
+
"guess_limit": 1,
|
| 17 |
+
"preload_profile": null,
|
| 18 |
+
"prefill_chunk_size": 2048,
|
| 19 |
+
"torch": "2.15.0",
|
| 20 |
+
"platform": "macOS-26.5.1-arm64-arm-64bit",
|
| 21 |
+
"load_s": 4.236552958143875,
|
| 22 |
+
"device_after_load_bytes": 6580027392,
|
| 23 |
+
"runs": {
|
| 24 |
+
"story": [],
|
| 25 |
+
"prefill_4k": [
|
| 26 |
+
{
|
| 27 |
+
"prompt_tokens": 4059,
|
| 28 |
+
"new_tokens": 32,
|
| 29 |
+
"time_to_first_token_s": 17.835860999999568,
|
| 30 |
+
"prefill_tokens_per_s": 227.57522050660174,
|
| 31 |
+
"decode_tokens_per_s": 10.077106253484079,
|
| 32 |
+
"total_s": 20.91214095777832,
|
| 33 |
+
"store": {
|
| 34 |
+
"hits": 25695,
|
| 35 |
+
"misses": 24069,
|
| 36 |
+
"evictions": 36017,
|
| 37 |
+
"not_admitted": 797,
|
| 38 |
+
"slots": 0,
|
| 39 |
+
"pools": 0,
|
| 40 |
+
"prefetched": 50889,
|
| 41 |
+
"prefetch_used": 42958,
|
| 42 |
+
"installed_ahead": 18889,
|
| 43 |
+
"resident_bytes": 10728243200,
|
| 44 |
+
"peak_cache_bytes": 10728243200,
|
| 45 |
+
"read_bytes": 89050598400,
|
| 46 |
+
"read_calls": 50889,
|
| 47 |
+
"host_slots": 0,
|
| 48 |
+
"host_hits": 0,
|
| 49 |
+
"host_misses": 0
|
| 50 |
+
},
|
| 51 |
+
"text": "Based on the records provided, it is **impossible to summarize the actual history** of the harbour town.\n\nHere is why:\n\n1. **"
|
| 52 |
+
}
|
| 53 |
+
],
|
| 54 |
+
"long_doc_6k": []
|
| 55 |
+
},
|
| 56 |
+
"device_peak_sampled_bytes": 20979105792,
|
| 57 |
+
"process_peak_rss_bytes": 10719248384
|
| 58 |
+
}
|
results/qwen38_local_m5/2026-11-17-clean-pass/summary.txt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
decode: "prefill_tokens_per_s": 10.113852333664326 "decode_tokens_per_s": 14.775250527132199 decode median 13.915121995206265
|
| 2 |
+
prefill: "prefill_tokens_per_s": 227.57522050660174 "decode_tokens_per_s": 10.077106253484079
|
| 3 |
+
long: "prefill_tokens_per_s": 205.6297042278869 "decode_tokens_per_s": 12.406908215060852
|
rfc.html
CHANGED
|
@@ -92,21 +92,21 @@
|
|
| 92 |
<tbody>
|
| 93 |
<tr>
|
| 94 |
<td>Decode, 200 tokens (median of 4 prompts)</td>
|
| 95 |
-
<td>13.
|
| 96 |
<td>14.8</td>
|
| 97 |
-
<td>1.
|
| 98 |
</tr>
|
| 99 |
<tr>
|
| 100 |
<td>Prefill, 4K-token prompt</td>
|
| 101 |
-
<td>
|
| 102 |
<td>220</td>
|
| 103 |
<td>1.0x</td>
|
| 104 |
</tr>
|
| 105 |
<tr>
|
| 106 |
<td>After a 6K-token document: prefill / decode</td>
|
| 107 |
-
<td>
|
| 108 |
<td>222 / 13.7</td>
|
| 109 |
-
<td>1.
|
| 110 |
</tr>
|
| 111 |
</tbody>
|
| 112 |
</table></div>
|
|
@@ -114,7 +114,7 @@
|
|
| 114 |
<h2>Design</h2>
|
| 115 |
<ul>
|
| 116 |
<li><strong>Store.</strong> One device pool of expert slots per expert layout, shared by the layers that use it; every layer gets the same number of slots, so mixed per-layer quantization works. LRU (or decaying LFU) per pool. Per-layer expert → slot tables live on the host; when every expert is resident, layers never synchronize with the host. Pool memory is taken at the first forward, not during loading.</li>
|
| 117 |
-
<li><strong>Reads.</strong> Each MoE layer copies its routing to the host once. Misses are read on I/O threads into recycled pinned buffers (one contiguous read and one non-blocking host-to-device copy per expert). Optional read-ahead: the next layer's
|
| 118 |
<li><strong>Compute.</strong> Cached experts run on the pool through the experts kernels (<code>grouped_mm</code> for BF16, <code>gather_qmm</code> for quantized experts); uncached ones run once from their read.</li>
|
| 119 |
<li><strong>Host tier (optional).</strong> A pool of expert rows in pinned host memory between storage and the device cache, for discrete GPUs with more host than device memory.</li>
|
| 120 |
<li><strong>External experts.</strong> An <code>expert_source</code> (layout + read) and an <code>expert_compute</code> callable let a package stream experts in its own format (the Qwen3.8 package uses Strata's GGUF experts and ggml kernels this way) while the store keeps caching, reads and read-ahead.</li>
|
|
|
|
| 92 |
<tbody>
|
| 93 |
<tr>
|
| 94 |
<td>Decode, 200 tokens (median of 4 prompts)</td>
|
| 95 |
+
<td>13.9 tok/s</td>
|
| 96 |
<td>14.8</td>
|
| 97 |
+
<td>1.06x</td>
|
| 98 |
</tr>
|
| 99 |
<tr>
|
| 100 |
<td>Prefill, 4K-token prompt</td>
|
| 101 |
+
<td>228 tok/s</td>
|
| 102 |
<td>220</td>
|
| 103 |
<td>1.0x</td>
|
| 104 |
</tr>
|
| 105 |
<tr>
|
| 106 |
<td>After a 6K-token document: prefill / decode</td>
|
| 107 |
+
<td>206 / 12.4 tok/s</td>
|
| 108 |
<td>222 / 13.7</td>
|
| 109 |
+
<td>1.08x / 1.1x</td>
|
| 110 |
</tr>
|
| 111 |
</tbody>
|
| 112 |
</table></div>
|
|
|
|
| 114 |
<h2>Design</h2>
|
| 115 |
<ul>
|
| 116 |
<li><strong>Store.</strong> One device pool of expert slots per expert layout, shared by the layers that use it; every layer gets the same number of slots, so mixed per-layer quantization works. LRU (or decaying LFU) per pool. Per-layer expert → slot tables live on the host; when every expert is resident, layers never synchronize with the host. Pool memory is taken at the first forward, not during loading.</li>
|
| 117 |
+
<li><strong>Reads.</strong> Each MoE layer copies its routing to the host once. Misses are read on I/O threads into recycled pinned buffers (one contiguous read and one non-blocking host-to-device copy per expert). Optional read-ahead: in decode, the next layer's best guessed experts (its router on this layer's input) read once this layer's reads are done and installed only when used; in long prefill chunks, all of the next layer's experts, installed behind the current layer's compute. Long prefill calls do not refresh the LRU and their installs go to its cold end, so a long prompt does not flush the experts decoding uses.</li>
|
| 118 |
<li><strong>Compute.</strong> Cached experts run on the pool through the experts kernels (<code>grouped_mm</code> for BF16, <code>gather_qmm</code> for quantized experts); uncached ones run once from their read.</li>
|
| 119 |
<li><strong>Host tier (optional).</strong> A pool of expert rows in pinned host memory between storage and the device cache, for discrete GPUs with more host than device memory.</li>
|
| 120 |
<li><strong>External experts.</strong> An <code>expert_source</code> (layout + read) and an <code>expert_compute</code> callable let a package stream experts in its own format (the Qwen3.8 package uses Strata's GGUF experts and ggml kernels this way) while the store keeps caching, reads and read-ahead.</li>
|
rfc.md
CHANGED
|
@@ -21,16 +21,16 @@ On a 32 GB MacBook Pro M5, the prototype (draft PR plus a model-specific package
|
|
| 21 |
|
| 22 |
| | Transformers prototype | Strata | |
|
| 23 |
|---|---|---|---|
|
| 24 |
-
| Decode, 200 tokens (median of 4 prompts) | 13.
|
| 25 |
-
| Prefill, 4K-token prompt |
|
| 26 |
-
| After a 6K-token document: prefill / decode |
|
| 27 |
|
| 28 |
12 GiB expert cache, 8-bit dense layers. The remaining gap is mostly expert reads after a long prompt and one host round trip per MoE layer. On CUDA (one A10G, 24 GB), the same feature decodes at 5.8 tok/s with every expert in a pinned host tier (BF16 dense layers, 8 GiB device cache). Details and raw data: https://huggingface.co/spaces/tarekziade/moe-expert-streaming
|
| 29 |
|
| 30 |
### Design
|
| 31 |
|
| 32 |
- **Store.** One device pool of expert slots per expert layout, shared by the layers that use it; every layer gets the same number of slots, so mixed per-layer quantization works. LRU (or decaying LFU) per pool. Per-layer expert → slot tables live on the host; when every expert is resident, layers never synchronize with the host. Pool memory is taken at the first forward, not during loading.
|
| 33 |
-
- **Reads.** Each MoE layer copies its routing to the host once. Misses are read on I/O threads into recycled pinned buffers (one contiguous read and one non-blocking host-to-device copy per expert). Optional read-ahead: the next layer's
|
| 34 |
- **Compute.** Cached experts run on the pool through the experts kernels (`grouped_mm` for BF16, `gather_qmm` for quantized experts); uncached ones run once from their read.
|
| 35 |
- **Host tier (optional).** A pool of expert rows in pinned host memory between storage and the device cache, for discrete GPUs with more host than device memory.
|
| 36 |
- **External experts.** An `expert_source` (layout + read) and an `expert_compute` callable let a package stream experts in its own format (the Qwen3.8 package uses Strata's GGUF experts and ggml kernels this way) while the store keeps caching, reads and read-ahead.
|
|
|
|
| 21 |
|
| 22 |
| | Transformers prototype | Strata | |
|
| 23 |
|---|---|---|---|
|
| 24 |
+
| Decode, 200 tokens (median of 4 prompts) | 13.9 tok/s | 14.8 | 1.06x |
|
| 25 |
+
| Prefill, 4K-token prompt | 228 tok/s | 220 | 1.0x |
|
| 26 |
+
| After a 6K-token document: prefill / decode | 206 / 12.4 tok/s | 222 / 13.7 | 1.08x / 1.1x |
|
| 27 |
|
| 28 |
12 GiB expert cache, 8-bit dense layers. The remaining gap is mostly expert reads after a long prompt and one host round trip per MoE layer. On CUDA (one A10G, 24 GB), the same feature decodes at 5.8 tok/s with every expert in a pinned host tier (BF16 dense layers, 8 GiB device cache). Details and raw data: https://huggingface.co/spaces/tarekziade/moe-expert-streaming
|
| 29 |
|
| 30 |
### Design
|
| 31 |
|
| 32 |
- **Store.** One device pool of expert slots per expert layout, shared by the layers that use it; every layer gets the same number of slots, so mixed per-layer quantization works. LRU (or decaying LFU) per pool. Per-layer expert → slot tables live on the host; when every expert is resident, layers never synchronize with the host. Pool memory is taken at the first forward, not during loading.
|
| 33 |
+
- **Reads.** Each MoE layer copies its routing to the host once. Misses are read on I/O threads into recycled pinned buffers (one contiguous read and one non-blocking host-to-device copy per expert). Optional read-ahead: in decode, the next layer's best guessed experts (its router on this layer's input) read once this layer's reads are done and installed only when used; in long prefill chunks, all of the next layer's experts, installed behind the current layer's compute. Long prefill calls do not refresh the LRU and their installs go to its cold end, so a long prompt does not flush the experts decoding uses.
|
| 34 |
- **Compute.** Cached experts run on the pool through the experts kernels (`grouped_mm` for BF16, `gather_qmm` for quantized experts); uncached ones run once from their read.
|
| 35 |
- **Host tier (optional).** A pool of expert rows in pinned host memory between storage and the device cache, for discrete GPUs with more host than device memory.
|
| 36 |
- **External experts.** An `expert_source` (layout + read) and an `expert_compute` callable let a package stream experts in its own format (the Qwen3.8 package uses Strata's GGUF experts and ggml kernels this way) while the store keeps caching, reads and read-ahead.
|