0xdfi/GLM-5.2-1M-context-NVFP4-4x-DGX-Spark

GLM-5.2 at 1M context on 4× DGX Spark — our NVFP4 long-context recipe

We wanted longer context than anything else could deliver for GLM-5.2 on this hardware — so we built our own recipe. Unpruned GLM-5.2 (744B / ~40B active) on a 4× NVIDIA DGX Spark (GB10, sm_121a) cluster, reaching up to 1,000,000 tokens of context by pairing an NVFP4 368-byte compact KV cache (58% the size of FP8) with every current leading speed-efficiency lever: B12X sparse-MLA attention, MTP-5 speculative decode, full CUDA graphs, Marlin atomic-add, and concurrency-aligned graph capture.

Every number below is read live from vLLM's own engine logger — measured, not modeled, not benchmark-gamed.

⚡ Speed per version (live-measured)

Version	What it adds	Peak single-stream decode	Multi-user aggregate (c4)	Prefill	Max context
v1	base NVFP4 long-context recipe	42.3 tok/s (30 @64K · 22 floor)	—	819 tok/s @64K	1M (DCP4)
v1.1	+ concurrency serving	42.2 tok/s	103.7 tok/s (4 users)	819 tok/s	1M

Single-stream peaks at 42 tok/s — the 4×-Spark hardware ceiling (independently confirmed by three rigs). v1.1 concurrency serves 4 simultaneous users at 103.7 tok/s aggregate (2.5×). Decode is MTP-acceptance-bound: 42 peak on structured content, ~30 at 64K, ~22 floor on adversarial content.

Operating-point matrix (measured 2026-07-20, full CUDA graphs, seqs=1). Two knobs trade off: prefill batch (bigger = faster prefill, less context — indexer scratch = ctx×batch) and DCP level (more sharding = slower prefill+decode, more context). Decode is batch-independent (acceptance × forward-rate).

DCP	max ctx (b512)	prefill b512 / b2048	peak decode	hardened in-use
DCP1 — fastest	375K	614 / ~820	~42	~340K
DCP2 — balanced	625K	512 / 746	~37	~560K
DCP4 — max ctx	1M	410 / 614	~38 short / ~30 deep	~900K

Max context scales sub-linearly with DCP (index-cache not sharded). No swap / no NVMe thrash at any point. Full sliding-scale matrices (context × prefill by batch, per DCP) at the top of the GitHub README.

vs the field: the best public 4×-Spark GLM-5.2 report is ~20–22 tok/s single-stream — and it pruned the model to get there. This runs unpruned at 42 peak / 104 aggregate.

---

⚡ Prefix caching — repeated prefixes skip prefill (on by default, v1.2)

Automatic prefix caching is enabled by default in the launcher. Requests sharing a leading prefix — multi-turn chat re-sending history, a shared system prompt across users, the same document re-queried — reuse cached KV and skip prefill for the shared span. Measured live (DCP2, identical ~25.6K-token prompt sent twice, single stream): 45.9s → 0.78s (≈59× faster), byte-identical output. Each hit saves ≈2s per 1,000 cached prefix tokens. Verified safe on this stack (B12X_MLA_SPARSE + nvfp4_ds_mla + DCP2 + MTP-5) — it does not raise peak KV memory (cached blocks are LRU-evictable). Set ENABLE_PREFIX_CACHING=0 only for clean prefill benchmarking. Helps when the varying text is at the end of the prompt; nothing for all-unique prompts.

---

Why it works

Lever	Effect
NVFP4 DS-MLA KV (368 B/record)	58% of FP8 → ~2× context per rank — the 1M enabler
B12X sparse-MLA + indexer	DeepSeek Sparse Attention on sm_121a (long-context backend)
MTP-5 speculative decode	2–3× decode on high-acceptance content
DCP 1/2/4	shards KV → trade decode speed for context depth
Full CUDA graphs (FULL_AND_PIECEWISE)	split at the indexer so cross-node collectives run piecewise → graphs work over RoCE
Marlin atomic-add + capture-size alignment	concurrency speed levers (credit: tonyd2wild Speed-Night)
TP4 + Ray + dual RoCE	4-node parallel — the ≈372 GB (INT4) model can't fit one 128 GB unit

Reproduce

Full serve command, parameterized launcher, benchmark harness, findings, and the concurrency patch are in the companion repo — including the exact image, env, and DCP1/2/4 configs:

➡️ https://github.com/0xdfi/GLM-5.2-1M-4x-DGX-Spark

Model checkpoint served: QuantTrio/GLM-5.2-Int4-Int8Mix (unpruned, stock — not redistributed here). Runtime: custom vLLM fork exp1sm121a368r4dtypefix.

Honest limitations

• Decode is modest by datacenter standards (~22–42 single-stream) — that's Spark's ~273 GB/s bus, not the stack. For raw tok/s, Spark is the wrong tool; for 1M context on cheap silicon, it's remarkable.
• 63% of every decode step is collective/launch overhead (measured) — the real remaining ceiling lever is an RDMA one-shot allreduce, not a better drafter.
• Custom fork required — stock/nightly vLLM doesn't serve 744B @ 1M on sm_121a today.
• Numbers are from one rig. Reproduce on yours before quoting.

Acknowledgments

Built on open community work — above all CosmicRaisins (the sm_121 sparse-MLA port + Triton kernels the whole stack depends on), tonyd2wild (the 200K recipe + Speed-Night optimization audit), QuantTrio (the checkpoint), zai-org (GLM-5.2), and ciprianveg / Zatz / back199640 / p33zy / aidendle94 / eugr / drowzeys / jasl plus the NVIDIA developer forum 374125 community. Full attribution in NOTICE. Apache-2.0.

Measured 2026-07-20 on 4× DGX Spark / GB10 sm_121a. Read from the live engine, not modeled.