GitHub: Build-scripts and docs  |  Forensics study  |  Runtime fork  |  Discussions

Scion-35B-A3B: ternary MoE experts + trained corrections

A full 35B-A3B MoE in one 11.3 GB GGUF for llama.cpp. Ternary expert banks plus small trained corrections, with no full-precision masters and no full-model QAT.

2.61 bpw | 11.34 GB (6.3x smaller than the BF16 reference) | best PPL of the 2-bit class | task retention at Q4-class level, at about half of Q4_K_M's size

The name comes from grafting. A scion is the shoot grafted onto a rootstock, and here the trained corrections are grafted onto a 2.125 bpw ternary body.

Performance

Measured with Strata, this project's own inference engine (a separate runtime from the llama.cpp fork the rest of this card documents), serving this exact file with the corrections attached at load, on one RX 7900 XT (20 GB, gfx1100):

	tok/s
Generation (warmed, greedy, 128 tokens per request)	118.0-118.2
Prompt read (38-token prompt)	480-482

• ~8.5 ms per generated token at context 4096, one card, the whole 11.3 GB file resident (a 400-token run measures 8.51 ms/token).
• Deterministic: eight consecutive requests of the fixed prompt reproduce the full generated text (reasoning plus answer) byte-for-byte, and a cold restart reproduces it again.
• Protocol: four requests of one fixed prompt, run 1 warms and runs 2-4 report, temperature=0, timings from the engine's own record. The embedded corrections are applied; the file is unmodified.

With the optional MTP drafter. Strata can also run a 96 MiB k=1 draft head (Scion-35B-A3B-mtp-strata.gguf, below) that proposes the next token from the target's own hidden state and hands it to the model in the two-row window the verify step needs anyway. It is opt-in and greedy requests only:

strata-q36 --model Scion-35B-A3B-PQ2_0-corr.gguf --max-context 4096 --chunk 512 \
    --vram-reserve-mib 3072 --mtp 1 --mtp-sidecar Scion-35B-A3B-mtp-strata.gguf

In a serve config (serve/server.py --config <file>) that same command line goes into args; add --mtp-period 4 for text that can loop and/or --mtp-backoff 8 as the generic guard (see the backstop bullet below). Like llama.cpp's own speculative decoding, the batched verify can flip a near-tie: on six free-running 400-token generations the drafted text first diverged from plain decoding at tokens 32, 41, 49, 64, 72 and 112, at positions whose top two logits differed by about 0.03 (one instance: 19.92149 for the plain pick against 19.89257 for the batched one, the next candidate 1.4 behind). One early flip cascades, so longer drafted generations are not the same text as plain ones. The fixed-prompt determinism check above does pass byte-for-byte with it enabled (its first 128 tokens contain no such tie), and the plain path is untouched. Treat the drafter as a speed option, not a bit-exact one. Same card, same day, same protocol as the row above:

Workload	plain	with the drafter	Draft acceptance
the fixed benchmark prompt (128 tokens)	117.9-118.3	141.5-142.3 (+20%)	76%
six technical prompts (400 tokens each)	117.3	155.1 (+32%)	94%
eight structured prompts (JSON, YAML, markdown tables)	117.3	126.7 (+8%)	59%
seventeen prompts outside its training mix	117.1	125.9 (+7%)	56-61%
repetitive text it used to lose the thread of (800 tokens)	116.8	155.6 (+33%)	96%

• Acceptance on the fixed prompt by generation length: 83% at 64 tokens, 76% at 128, 71% at 256, 70% at 400, 67% at 800.
• The head was fitted to this engine's own hidden states (the two-row verify path's, which is what the drafter actually reads). A head fitted to another runtime's hidden states reaches only 38% here, so the runtime's own drafter repo file and this file are not interchangeable.
• The loop case is no longer a loss. Before this head, the model's repeated-token runs were where the drafter accepted 0-13% and measured -24% against plain; the run is one period-4 stretch, so training on those degenerate stretches (a few windows, oversampled) took the same prompt to +33% at 96% acceptance. For text it has not seen, two free backstops remain: --mtp-period 4 drafts a confirmed period directly (the head call is skipped, the verify untouched; 132.3 on that run with the previous head), and --mtp-backoff 8 pauses speculation after a streak of rejected drafts (16 steps, then 32, 64 ... up to 128, any acceptance resets it; 112.1 on the same run). Neither costs anything on ordinary text - the rule fires only on confirmed periods and the backoff only on rejection streaks, both measured at identical window counts on ordinary prompts.
• Long-form coverage (the head on this card). A 60-prompt pass over the weakest long-form styles (dialogue, uncommon-language code, essays, reviews, emails, "begin now and do not stop early" factuals) retrained the head on the merged mix: long-form held-out acceptance 49.3% -> 50.7% (the earlier set 51.2% -> 52.4%, a fresh nine-prompt set 47.9% -> 49.4%), with round-4 prose (55.1% -> 56.2%), fresh structured (59.5% -> 60.3%) and technical (94.0% -> 94.5%) rising too and no set giving way. The rows above are within rounding of the updated head's.

Contents

Section	Subsections
Performance
Quickstart	Step 1 - runtime · Step 2 - weights · Step 3 - run · Build from source · Troubleshooting · For AI assistants
Highlights
Resources
Model Overview
Weight Representation	Memory Requirement · Shipped Components
Best Practices	Generation Parameters · System Prompt · Choosing Context and Offload
Speculative decoding
Benchmarks	Task Retention · Additional tasks · Reasoning and code · Distributional Fidelity · Full Grid
Use Cases
Limitations
Citation
License and Attribution

---

Quickstart

The one thing to know first: this file uses a container that only exists in the runtime fork. Stock/upstream llama.cpp (and the runtimes built on it, like LM Studio and Ollama) will refuse it with invalid ggml type 142. The Hub's auto-generated "Use this model" panel points at upstream binaries, so ignore it: you need sky-is-green/prism-ml-llama.cpp, branch moe-corr-runtime. A pre-built copy of that fork is below, so most people do not need to compile anything.

Step 1 - pick a runtime

Your machine	Use this	Notes
Windows x64 + NVIDIA	llama-v0.4.2-bin-win-cuda-12.4-x64.zip + cudart-llama-bin-win-cuda-12.4-x64.zip	Preferred for most drivers. CUDA 13.3 build also available if your driver is very new.
Windows x64, no GPU	windows-x64-cpu.zip	Slower, works everywhere. MTP still works.
Linux x64 + NVIDIA	linux-x64-cuda.tar.gz
Linux x64, no GPU	linux-x64-cpu.tar.gz
Linux x64 + AMD	linux-x64-hip.tar.gz	The verified path is a source build with -DGGML_HIP=ON; see below.
macOS Apple Silicon	macos-arm64-metal.tar.gz	Metal path is present but has not been benchmarked by the author.
Anything else / you want max control	Build from source	One command block below.

All release links point at the v0.4.2 release.

Do not use Vulkan. It is not a supported path for this model. The PQ2_0 kernels exist only for CPU, CUDA and Metal; a Vulkan build would leave the expert matmuls on the CPU, which defeats the point. See Limitations.

Step 2 - get the weights

# model body (11.3 GB)
hf download SkyIsNotGreen/Scion-35B-A3B Scion-35B-A3B-PQ2_0-corr.gguf --local-dir .

# optional k=1 speculative drafter (50 MB, ~1.2x faster)
hf download SkyIsNotGreen/Scion-35B-A3B-mtp-drafter Scion-35B-A3B-mtp-drafter.gguf --local-dir .

hf download is the current CLI; the older huggingface-cli download works the same.

Step 3 - run

# chat (the model thinks by default, so leave room for the trace)
./llama-cli -m Scion-35B-A3B-PQ2_0-corr.gguf \
    -ngl 99 -c 4096 -t 8 \
    --temp 0.6 --top-p 0.95 --top-k 20 \
    -p "Explain quantum computing in simple terms." -n 16384

# OpenAI-compatible server with a web UI
./llama-server -m Scion-35B-A3B-PQ2_0-corr.gguf -ngl 99 -c 4096 -t 8 --port 8080

# same, with the drafter auto-detected from the sidecar GGUF
./llama-server -m Scion-35B-A3B-PQ2_0-corr.gguf \
    -md Scion-35B-A3B-mtp-drafter.gguf -ngl 99 -c 4096 -t 8 --port 8080

• -ngl 99 offloads every layer (0 is CPU-only).
• -c is context up to 262144; start at 4096.
• -t should equal the physical core count (SMT siblings hurt CPU expert throughput).
• If VRAM is tight, keep -ngl 99 and offload experts with -ncmoe.
• On Windows, use llama-cli.exe / llama-server.exe, PowerShell or cmd.

Windows CUDA users: extract the main zip, then extract the three DLLs from the matching cudart zip (cudart64_12.dll, cublas64_12.dll, cublasLt64_12.dll) into the same folder as llama-server.exe / llama-cli.exe. Without them the binary will fail to start with a missing-DLL error.

Build from source (any platform)

git clone -b moe-corr-runtime https://github.com/sky-is-green/prism-ml-llama.cpp
cd prism-ml-llama.cpp
./verify-container-support.sh          # must print "RESULT: OK"
rm -rf build

# pick ONE backend flag:
cmake -B build -DGGML_CUDA=ON          # NVIDIA
# cmake -B build -DGGML_HIP=ON         # AMD (verified on gfx1100 / RX 7900 XT)
# cmake -B build                       # CPU only
cmake --build build -j --target llama-cli llama-server
# binaries land in build/bin/

On Windows, build with CMake + Ninja against MSVC (or the LLVM toolchain); a plain cmake -B build -DGGML_CUDA=ON after vcvarsall.bat x64 is enough.

Troubleshooting

Symptom	Cause	Fix
tensor '...' has invalid ggml type 142. should be in [0, 43)	You ran an upstream llama.cpp build, an old binary earlier on PATH, or a stale build/ dir	Use the fork binary from above; delete build/ and re-clone if compiling. Run ./verify-container-support.sh.
Missing cudart64_*.dll / cublas64_*.dll	CUDA runtime not bundled in the main zip	Extract the matching cudart-* zip next to the executables.
GPU not used / slow	Wrong build, or experts on CPU	Confirm ggml-cuda.dll loaded and -ngl 99; add -ncmoe only if VRAM is tight.
Out of VRAM at long context	KV cache growth	Lower -c, or offload experts with -ncmoe.
Hangs at start on old CPUs	CPU backend picked a variant the CPU lacks	Use the fork bundle (ships all CPU variants) or rebuild with -DGGML_NATIVE=OFF.

For AI assistants / agents helping a user

If you are an AI agent wiring Scion up for someone, the requirements are:

Runtime MUST be sky-is-green/prism-ml-llama.cpp, branch moe-corr-runtime.
  - Upstream llama.cpp, LM Studio, Ollama and llama.cpp wheels CANNOT load this file.
Model file: Scion-35B-A3B-PQ2_0-corr.gguf
  - Custom tensor type is PQ2_0 (ggml type id 142), group size 128. NOT upstream Q2_0 (type 42).
GPU backends that implement PQ2_0: CUDA, CPU, Metal. Vulkan does NOT.
Windows x64 + NVIDIA, no build required:
  1. Download llama-v0.4.2-bin-win-cuda-12.4-x64.zip and cudart-llama-bin-win-cuda-12.4-x64.zip from
     https://github.com/sky-is-green/prism-ml-llama.cpp/releases/tag/v0.4.2
  2. Extract both into ONE folder (the cudart DLLs must sit next to llama-server.exe / llama-cli.exe).
  3. Run:  llama-server.exe -m Scion-35B-A3B-PQ2_0-corr.gguf -ngl 99 -c 4096 -t <physical cores> --port 8080
Recommended sampling: --temp 0.6 --top-p 0.95 --top-k 20, and allow -n up to 16384 (reasoning trace).
Optional speedup: -md Scion-35B-A3B-mtp-drafter.gguf (drafter auto-detected, same context).
"invalid ggml type 142" == wrong (upstream) binary; use the fork.

---

Highlights

• One 11.34 GB file at 2.61 bpw (expert banks at 2.125 bpw). The BF16 reference of the same model is 71.07 GB; Q4_K_M is 21.71 GB and IQ2_M is 12.56 GB. It is the smallest published build of this model I have seen at this quality level.
• Task retention inside the Q4 and BF16 noise band: HellaSwag 400 gives 79.00% (BF16 81.25, Q4_K_M 80.00) and Winogrande gives 76.25% (BF16 76.00, Q4_K_M 76.00). It has the joint-best Winogrande row in the table and the best PPL of the 2-bit class (8.354 against IQ2_M 8.413 and Q2_K 8.473).
• Trained, not calibrated: rank-512 correction branches on the attention output and the MoE block output, plus router deltas, trained by output-KD against the BF16 teacher with the deployed quantizer in the loop (ternary Lloyd g128). No imatrix and no calibration corpus, which is what separates this build from the imatrix-calibrated quants on the chart.
• One file, no --lora: the corrections are embedded (adapter.embedded=true) and attached at load. There is no adapter plumbing.
• A k=1 speculative drafter ships alongside (Scion-35B-A3B-mtp-drafter, 50 MB): it drafts the next token from the model's own hidden state and gives 1.14-1.37x faster generation on llama.cpp's speculative path, with unchanged outputs. It needs the fork runtime (it is not a stock llama.cpp drafter).
• The gap is stated, not hidden: full-vocabulary KLD against BF16 is 0.269 mean, a strong 2-bit-class result but still behind Q4_K_M at 0.031. Tail-aware training was attempted at full scale and did not transfer (TAIL-EXPERIMENT-PLAN.md).

Resources

• GitHub sky-is-green/scion: the source of truth for this work. It carries the harness that produced the file, the full MoE write-up, the port decisions and the negative register.
• Bonsai 2 ternary forensics: the dense-model study this method grew out of (format recovery, the trained-weight residual, the calibration-artifact result).
• Runtime: sky-is-green/prism-ml-llama.cpp, branch moe-corr-runtime, a fork of Prism ML's llama.cpp with the PQ2_0 container, the ffn_moe_out virtual target and embedded-adapter support.
• Retention grid: this release against every community quant of the same model, same protocol.
• Discussions: questions, test reports and failures are all welcome.

Model Overview

Item	Specification
Base model	empero-ai/Qwen3.8-35B-A3B-Distill, a Qwen3.8-line reasoning distill built on Qwen/Qwen3.6-35B-A3B (Apache-2.0)
Parameters	34.9B total, about 3B active per token (256 experts, top-8 plus shared)
Architecture	qwen3_5_moe (qwen35moe in llama.cpp): 40 layers, hybrid linear and full attention, MoE feed-forward
Context length	262,144 tokens (inherited from the base model)
Weight format	Ternary PQ2_0 g128 expert banks (codes in {-1, 0, +1} plus one fp16 group scale per 128 weights), Q8_0 for the rest, embedded corrections in the legacy q1_0_g128 container (rank-512)
Low-bit coverage	Expert banks only; attention, embeddings and the head stay Q8_0, and norms, routers and the output stay F32
Deployed size	10.558 GiB / 11.337 GB (text only, single file)
Backends	llama.cpp fork; verified on CPU and ROCm/gfx1100. Pre-built Windows x64 CUDA binaries are on the v0.4.2 release but the CUDA path is not yet runtime-verified end-to-end. Metal untested. Vulkan unsupported (no PQ2_0 kernels)
License	Apache-2.0 (inherited from the base)

Weight Representation: ternary PQ2_0 + trained corrections

Each expert weight takes a value from {-1, 0, +1} with one shared FP16 scale per group of 128 weights: 2-bit slots at 2.125 bits per weight. The rest of the model is Q8_0 (attention, embeddings, LM head) and F32 (norms, routers, output). The corrections are rank-512 low-rank branches on the attention output and the MoE block output, plus exact router deltas. They ship in the compact legacy q1_0_g128 container (2-bit codes plus one fp16 group scale per 128) and are merged into the file. Effective overall: 2.61 bpw.

Memory Requirement

Format	bpw	Size	vs BF16
BF16 (reference)	16.38	71.07 GB	1.0x
Q8_0	8.72	37.80 GB	1.9x
Q4_K_M	5.01	21.71 GB	3.3x
IQ2_M	2.90	12.56 GB	5.7x
Scion-35B-A3B	2.61	11.34 GB	6.3x

Sizes are the published on-disk files of the same model, measured under one protocol (wikitext-2 PPL, KLD against BF16, HellaSwag and Winogrande 400; see Benchmarks).

Shipped Components

Component	Pack	Size	Residency
Language model (this repo)	PQ2_0 experts, Q8_0 rest, embedded corrections	11.34 GB	resident; the whole model
k=1 MTP drafter (separate repo)	fp16 head (fc1/gelu/fc2) over the model's own hidden + next-token embedding	50 MB	transient; ~1 ms/eval on GPU
Decision head (optional), in this repo under decision-head/	closed-form ridge + Platt (66 KB) over the model's own last-prompt-token hidden	66 KB	transient; one judge pass, ~0.3 s on one 7900 XT
Uncorrected body (not uploaded)	PQ2_0 experts and Q8_0 rest	10.46 GiB	for swap tests
Corrections (not uploaded)	rank-512 branches and router deltas (q1_0_g128)	98 MiB	for swap tests

The language model is the single released body file; the drafter is a separate 50 MB file in its own repo (Scion-35B-A3B-mtp-drafter). The two-file variant (body plus separate adapter) exists for reproducing the merge and swapping corrections at runtime. Ask in Discussions if you want it.

About the Hub's quant chip. The Hub parses file names, and until it knows Prism's container names it labels this file Q2_0 - the Q2_0 substring inside PQ2_0; the same happens on Prism ML's own PQ2_0 releases. The container is Prism's PQ2_0 (legacy name Q1_0_g128, type 142/43, identical byte layout): 2-bit codes with one fp16 group scale per 128 weights. It is not upstream llama.cpp's g64 Q2_0 (type 42). A PR adding PQ2_0 (plus PTQ1_0/Q4X160) to the Hub's label list is filed; once it ships, this chip reads PQ2_0. No single quant name fits the file anyway, because it is a mix: PQ2_0 expert banks, the embedded corrections in the legacy q1_0_g128 container, and Q8_0 for the rest (norms and routers in F32).

Decision head (optional): a correctness readout

A 66 KB closed-form head (ridge + Platt) that answers "is this candidate answer correct?" from the model's own last-prompt-token hidden state, read through one judge pass. It is not a second model. The study behind it (SkyIsNotGreen/decision-heads) measured AUC 0.999 on the frozen judge bench (a prompted self-judge: 0.903), 23 of this model's own 28 wrong answers caught at zero false rejects, and two uses that pay: best-of-3 accuracy at about half the pool cost (gating) and +7 tasks over majority on the held-out community set (veto). Run through the Strata engine, the same head reads Strata's states at AUC 0.9996 against 0.9988 where it was fit, and a refit in Strata's serving format is 1.0000, engine-vs-client parity 5.0e-7 over 378 judge passes.

Files. decision-head/ carries the fitted head: the .json is the exchange form (the study's schema), the .bin is the engine's runtime form for the Strata sidecar.

With the Strata q36 engine, either the engine sidecar:

STRATA_Q36_JEV_HEAD=decision-head/scion-35b-a3b-decision-head-q36.bin strata-q36 --serve --model ...
# an EMB request then answers:   EMB 2048 <file> <p_correct>

or the server route (any body Strata serves): put "decision_head": "<path>.json" in the run config,

curl -s localhost:8080/v1/verdict -d '{"task": "...", "candidate": "..."}'
# -> {"p_correct": 0.997, "threshold": 0.9, "accept": true, "tokens": 271, "ms": 318.4}

The full record (the arms table, the cross-engine transfer, the 176B results) is in the dataset's PORT-REPORT.md. Refit per body and per prompt format: this head belongs to this body and to the decision-function wording shipped with it.

Best Practices

Generation Parameters

Recommended values, from the base model card:

• temperature=0.6, top_p=0.95, top_k=20

This is a reasoning distill, and answers open with a long thinking segment. Allow generous max_new_tokens (for example -n 16384); a small cap ends generation mid-thought, before any answer.

Override the file's default. The GGUF's embedded general.sampling.temp is 1.0. Pass --temp 0.6 explicitly (or override the metadata) so loaders that honor the embedded defaults do not run too hot.

System Prompt

A simple prompt works, for example You are a helpful assistant. The base is a reasoning SFT distill and does not require a special system prompt.

Agentic / tool use

When Scion drives a shell/editor tool loop, keep its thinking trace in the conversation history. The model plans inside a thinking block and then acts; if the client drops the assistant's reasoning_content between turns, the template re-inserts an empty thinking block and the model can end a turn with a plan but no tool call (an empty turn that looks like looping). Either:

• return reasoning_content with each assistant message (the OpenAI-compatible way), or
• run the session with thinking disabled (chat_template_kwargs: {"enable_thinking": false}).

For tool use, sample at --temp 0.6 --top-p 0.95 --top-k 20 with a generous per-turn token budget, and do not force a tool call on every turn — let the model finish.

Choosing Context and Offload

• The weights fit a 12 GB card; about 16 GB is comfortable with context.
• -ngl 99 on a single card. When VRAM is tight, offload experts to CPU with -ncmoe, which is the VRAM-budget dial; the ternary container roughly halves the CPU-tail penalty compared to an f16 expert bank.
• Threads should equal physical cores (-t 8 on an 8C/16T CPU; SMT siblings collapse CPU expert throughput).
• Do not layer-split across two cards when one card fits; the proxy measurements showed a 40% generation loss.

Speculative decoding (k=1 drafter)

The drafter lives in its own repo: Scion-35B-A3B-mtp-drafter (50 MB). It is a small frozen-body head that predicts the model's own next token from its post-norm hidden state plus the next token's embedding; it reuses the model's own output_norm/output/token_embd, so it adds no second vocabulary projection. It is not a stock llama.cpp drafter: the fork's draft-mtp-sidecar implementation loads it (auto-detected from the GGUF) and runs it entirely against the target context, with no draft model.

hf download SkyIsNotGreen/Scion-35B-A3B-mtp-drafter Scion-35B-A3B-mtp-drafter.gguf --local-dir .

# server (no extra flags; the sidecar is auto-detected)
./build/bin/llama-server -m Scion-35B-A3B-PQ2_0-corr.gguf \
    -md Scion-35B-A3B-mtp-drafter.gguf -ngl 99 -c 4096 -t 8 --port 8080

# CLI
./build/bin/llama-cli -m Scion-35B-A3B-PQ2_0-corr.gguf \
    -md Scion-35B-A3B-mtp-drafter.gguf -ngl 99 -c 4096 -t 8

Measured on one RX 7900 XT (greedy):

Workload	Baseline	+ drafter	Speedup	Draft acceptance
Bench prompt, 400 tokens	12.41 ms/tok	9.07 ms/tok	1.37x	0.738
Open-ended CLI generation	49.0 tok/s	57.2 tok/s	1.17x	0.46-0.74
Server confirmation	-	-	1.14-1.21x	0.44-0.70

Teacher-forced acceptance: fineweb 0.483 / wikitext 0.360. Speedup is prompt-dependent - treat ~1.2x as the typical figure and 1.37x as the benchmark-prompt best. Drafting leaves the model's outputs unchanged apart from rare near-tie flips on batched verification.

Two drafter files, one per runtime. The numbers just above are the llama.cpp fork's, with the head the fork auto-detects. Strata's own engine runs its own draft head (Scion-35B-A3B-mtp-strata.gguf in this repo, 96 MiB) on the same model file, opt-in via --mtp 1 --mtp-sidecar - how to enable it and its numbers are in the Performance section: 141.3-141.7 tok/s against 117.9-118.3 plain on the fixed prompt (+20%, 76% acceptance), up to +32% on technical prompts, +7-8% on structured output and on prompts outside its training mix (long-form held-out acceptance 50.7%, up from 49.3%), and no measured loss left: repeated-token text, where the previous head accepted 0-13%, is now its best row at +33% and 96% acceptance. --mtp-period 4 (draft a confirmed period directly) and --mtp-backoff 8 (pause after a rejection streak) remain as free backstops for text the head has not seen. The two heads were trained against different runtimes' hidden states and are not interchangeable: in Strata, the fork's head accepts 38% and is slower than not drafting at all.

Benchmarks

Community protocol, the same for every row: wikitext-2 PPL (c512, 580 chunks); KLD against BF16 logits over 50 chunks (25.5k tokens); HellaSwag 400 and Winogrande 400 zero-shot (about +/-2% CI). BF16 and all quants were measured in one H100 session; this release was measured on the local card and cross-checked against the pod (IQ2_M KLD local vs pod: 0.2%).

Task Retention (400 tasks each)

Variant	Size	bpw	HellaSwag	Winogrande
BF16 (reference)	71.07 GB	16.38	81.25	76.00
Q4_K_M	21.71 GB	5.01	80.00	76.00
IQ2_M	12.56 GB	2.90	79.00	75.75
Q2_K	13.84 GB	3.19	76.50	73.25
Scion-35B-A3B	11.34 GB	2.61	79.00	76.25

Within +/-2% noise of Q4_K_M and BF16; tied with IQ2_M on HellaSwag and ahead of it on Winogrande at 1.2 GB less. Treat sub-1% differences as ties.

Additional tasks (re-measured 2026-09-30)

The 2026-09-27 grid covered HellaSwag and Winogrande. A later local run added the remaining standard multiple-choice tasks on the same release and the same fork build (one RX 7900 XT, llama-perplexity --multiple-choice, same seed):

Task	Scion-35B-A3B
ARC-Challenge (299)	57.19 +/-2.87
ARC-Easy (570)	80.00 +/-1.68
MMLU (2000, seed 1234)	40.15 +/-1.10
TruthfulQA (817)	32.56 +/-1.64
wikitext-2 PPL (c512, 580 chunks)	8.2731 +/-0.054

The fresh PPL uses --no-warmup and -np 8, so its level sits ~1% below the grid's 8.3539; the model is unchanged.

Reasoning and code tasks (2026-10-07)

Same release and fork build (one RX 7900 XT, llama-server). Sampled sets: GSM8K-300 (300 sampled from the official test set, seed 0), MATH-200 (200 sampled across the seven subjects, seed 0), HumanEval-164 (the full official test set). Programmatic checkers (final-answer number, \boxed{}, unit tests for code); single sample at --temp 0.7 --top-p 0.95 --top-k 20.

Task	pass@1, thinking	pass@1, no thinking	pass@3, no thinking
GSM8K (300)	88.7 +/-1.8	86.3 +/-2.0	95.3 +/-1.2
MATH (200)	60.0 +/-3.5	50.5 +/-3.5	70.5 +/-3.2
HumanEval (164)	84.1 +/-2.9	79.3 +/-3.2	91.5 +/-2.2

• Thinking runs used a 2048-token cap and no-thinking a 700-token cap; about 8% of generations in each arm reached the cap, so these are conservative against the recommended -n 16384.
• The no-thinking column is for reference; the model's intended mode is thinking (the base is a reasoning distill).
• These are sampled-set numbers under the stated protocol, not official full-set leaderboard figures. Uncertainties are 1σ binomial.

Distributional Fidelity: the known gap

Variant	PPL (lower better)	KLD mean vs BF16 (lower better)	KLD 99.9%
BF16	7.1595	-	-
Q4_K_M	7.2354	0.0314	1.141
IQ2_M	8.4133	0.1636	3.963
Q2_K	8.4729	0.1493	3.276
Scion-35B-A3B	8.3539	0.2694	4.758

The corrections improved mean token likelihood (PPL 11.60 uncorrected, 8.35 after) more than they improved the full-distribution tail. PPL ranks this build first of the 2-bit class and KLD ranks it last, and that disagreement is the research result: the training matches the teacher's top-50 logits, and the rest of the distribution is unconstrained. Tail-aware training was attempted at full scale (a tail-conditional KD term plus router bias plus a hard-window curriculum, the cur05 recipe) and did not transfer: on a 40-layer body the teacher puts ~98% of its mass inside the top-512 cache, so the tail terms are nearly inert, and the retrain ties this release on every task while PPL regresses ~2%. The gap stands, and the negative is recorded in the project's register.

Full Grid

All ten community quants plus BF16; up is better, left is smaller. Full table and method notes: QUANT-RETENTION-35B.md.

Full per-variant table

model	size GB	bpw	PPL	KLD mean	KLD 99.9%	HellaSwag	Winogrande
Scion-35B-A3B	11.34	2.61	8.354	0.269	4.758	79.00	76.25
IQ2_M	12.56	2.90	8.413	0.164	3.963	79.00	75.75
Q2_K	13.84	3.19	8.473	0.149	3.276	76.50	73.25
IQ3_M	16.34	3.77	7.520	0.057	1.225	80.00	74.25
Q3_K_M	17.66	4.07	7.432	0.058	1.611	79.75	76.00
IQ4_XS	19.63	4.53	7.264	0.022	0.636	80.75	75.75
Q4_K_M	21.71	5.01	7.235	0.031	1.141	80.00	76.00
Q5_K_M	25.35	5.84	7.273	0.015	0.717	80.50	76.25
Q6_K	29.21	6.73	7.153	0.008	0.360	80.75	74.75
Q8_0	37.80	8.72	7.160	0.004	0.209	80.25	75.50
BF16	71.07	16.38	7.160	-	-	81.25	76.00

Use Cases

• A 35B-A3B on one consumer GPU: 11.3 GB of weights fit a 20 GB card with room for context, and can be tiered further with CPU expert offload.
• Low-bit research and testing: a reference point for "ternary experts plus trained corrections, no full-precision masters, no full-model QAT". The build-scripts and every negative result are public.
• Local-first serving: the model was built for an offline assistant stack, so a single file with embedded corrections keeps deployment to one download and one set of flags.

Limitations

• KLD tail (stated above): distributional fidelity is strong-2-bit, not Q4-class. PPL, HellaSwag and Winogrande look Q4-class; KLD is where the gap lives. A full-scale tail-aware retrain was attempted and did not transfer.
• Text only: the base model has a vision tower, and this file carries no vision tensors (Q8_0 language path only).
• Reasoning distill: long thinking traces; budget max_new_tokens accordingly.
• Protocol caveats: the KLD figure is 50 chunks; task numbers are 400-task runs (about +/-2% CI); the 2-bit and 3-bit competitors are imatrix-calibrated on Wikipedia-like data, which flatters their wikitext KLD, and the corrections were trained on fineweb.
• Platform coverage: CPU and ROCm (gfx1100) verified. Windows x64 CUDA binaries are now published on the v0.4.2 release, but the CUDA path has not been runtime-verified end-to-end yet; reports welcome. Metal is present but untested. Vulkan is unsupported: ggml-vulkan has no PQ2_0 kernels, so the expert matmuls would fall back to CPU.
• The Hub chip says Q2_0: it is filename-derived; see the note above, and use the fork.
• Not affiliated with Prism ML, empero-ai, or Alibaba Cloud. It builds on Prism ML's engine work (fork and PQ2_0 container) and community GGUF conversions.

Citation

@misc{scion35b2026,
    title  = {Scion-35B-A3B: ternary MoE experts with trained corrections},
    author = {SkyIsNotGreen},
    year   = {2026},
    month  = {September},
    url    = {https://huggingface.co/SkyIsNotGreen/Scion-35B-A3B}
}

License and Attribution

Apache-2.0, inherited from the base model; see LICENSE and NOTICE. Base weights: empero-ai and Qwen / Alibaba Cloud (Apache-2.0); BF16 conversion: MrFuzzihead; container and kernels: Prism ML (MIT) with TAARDIS conventions (MIT); engine: llama.cpp (MIT).