OpensourceWTF/Hy3-oQ2e-MTPLX-streaming

Hy3 oQ2e MTPLX-streaming

2-bit Hunyuan 3.0 (295B-A21B MoE) for Apple Silicon, packaged for SSD-streamed serving with MTPLX. Weights stream from SSD under a configurable memory envelope, so the model runs on Macs that cannot hold it resident. Measured envelopes: 64, 80, and 88 GiB of weight budget on a 128 GB M5 Max.

Quantization: omlx oQe level-2 imatrix experts (2.44 bpw, 2.50 bpw effective with q8 residents), from mlx-community/Hy3-oQ2e.

Repo contents (~91 GiB)

files	size	what
model-resident-*.safetensors (2) + model.safetensors.index.json	8.7 GiB	attention, routers, shared experts, embeddings, dense layer 0 (q8-gs64); the index covers exactly these resident tensors
experts.bin + expert-manifest.json	75 GiB	all 15,168 routed expert records (2-bit gs128) as an aligned, hash-pinned bank; the manifest is sidecar-authoritative (references resident shards + the bank only)
expert-manifest.shards.json	—	provenance manifest pinning sha256 for the pre-repack safetensors shards and every expert record
mtp/layer80-bf16.safetensors, mtp/layer80-residents-q.safetensors	7.1 GiB	Hy3 MTP head (speculative decoding)
island-placement.json, route-census.json	—	per-machine seeds; MTPLX regenerates them on first serve

This is the MTPLX streaming layout: routed experts ship only in experts.bin, so nothing is downloaded twice. Download everything:

hf download OpensourceWTF/Hy3-oQ2e-MTPLX-streaming

Plain-MLX resident serving (128 GB Mac): use the upstream safetensors checkpoint at mlx-community/Hy3-oQ2e; this repo packages the same bytes for streaming.

llama.cpp: not served. MLX affine quantization does not convert to GGML blocks in either direction.

Measured decode (M5 Max, 614 GB/s)

Real long-code prefill (exact-token gated), greedy, decode 256, bf16 KV, 3 reps. Cell = decode tok/s, mode of record = faster of AR / K1 (MTP speculative depth 1).

weight envelope	1024 ctx	16k	64k
88 GiB (full residency)	48.0 K1	27.6 K1	— (KV exceeds the 112 GiB budget)
80 GiB	21.6 K1	15.9 AR	9.2 AR
64 GiB	11.1 K1	7.0 AR	4.0 AR

Short-prompt arm (320-token code prompt): K1 47.7 / 22.3 / 10.8 on 88/80/64 — K1 wins every envelope there.

Two regimes govern the mode choice. K1 wins at short context on every envelope, and at every context under full residency. AR wins at 16k+ on streaming envelopes: speculative verify multiplies expert-miss servicing under partial residency. Acceptance is 0.89–0.90 on code in every cell, so the crossover is a decode-cost effect, not an acceptance effect.

Quality (serving config: rq4 projections, bf16 KV)

eval	score
MBPP full-974	0.8004 (identical to the q8-resident reference)
HumanEval-164	0.8659 (q8 reference 0.8720, McNemar exact p = 1.0)

Serving notes

• Presets hy3-oq2e-rq4-{48..96} ship in the MTPLX repo (main branch).
• bf16 KV is required for MTP. kv4 was measured and reversed: acceptance collapses 0.898 → 0.125 and every MTP lane loses to AR.
• Recommended: K1 for short context, AR for 16k+ on streaming envelopes.
• Residents are pre-quantized q8-gs64 and load via the config-driven path; never requantize them at load.

Provenance and integrity

• Base model tencent/Hy3 (Apache-2.0); quantized bank mlx-community/Hy3-oQ2e rev 1979c306.
• Re-sharded here into resident/expert groups for selective download. Tensor bytes are unchanged; every tensor was sha256-verified at repack.
• expert-manifest.json (sidecar-authoritative, what the engine loads) pins the resident shards and experts.bin; it is what makes the expert-shard-free selective download servable.
• expert-manifest.shards.json pins sha256 for all 18 safetensors shards, all 15,168 expert records, and experts.bin.
• Two metadata corrections vs the published checkpoint (originals kept as *.orig-published): index total_size (overstated upstream by 345,252 B) and config num_nextn_predict_layers 0→1, which the bundled MTP head requires.

Not measured

32k context. 48/32 GiB envelopes (presets exist, never benchmarked). kv4 beyond the single 88 GiB × 16k cell. Prose-heavy workloads: MTP acceptance is content-dependent (0.52 measured on prose vs 0.89–0.90 on code); the speed table is a code-workload table.