L-Alchemyst/Step-5-Preview-GGUF

Step-5-Preview

Imatrix quants for ik_llama.cpp

NOTE: ik_llama.cpp is highly versatile and runs mainline llama.cpp quants such as existing GGUFs from bartowski, unsloth, mradermacher, Aes Sedai etc

IQ4_KS-mix-B uploading 🟡 — IQ3_KS-mix-B 🟡 uploading

Prefer the -B sets. Get IQ4_KS-mix-B/ or IQ3_KS-mix-B/, and step-preview-B.imatrix for your own quants — the older IQ3_KS-mix/ / IQ4_KS-mix/ sets and step-preview-A.imatrix are legacy.

Why the legacy sets are still here: the original IQ3_KS-mix/ and IQ4_KS-mix/ sets (calibrated with the older A imatrix, being relocated to legacy-A-imatrix/) are deprecated. They load and run, but are measurably worse than the -B sets (IQ4: +14.5% PPL vs +3.29%; IQ3: +23.5% PPL vs +11.50%) and require extra fix-up scripts to run on current builds. The A imatrix was collected from a source model with a broken indexer norm convention, which made the selector's scores noise during calibration — copying its invocation is fine, copying the file is not. New downloads should use the -B quants.

Repairing a legacy download (the -B sets need none of this): run gguf_meta_patch.py on shard 1 — it renames the arch step35 → step5 and adds the step5.sparse.* KVs, without which current builds will not load the file at all — then qnorm_fix.py on the last shard, which applies the indexer.q_norm.weight +1.0 convention (KBs of in-place writes; both scripts are plan / apply / verify, and qnorm_fix.py also reverts). If you converted from mainline yourself, run indexer_insert.py first to add the 184 indexer tensors.

General Specs

	Value
Architecture	Step-5 (MoE, 352 experts × 8 active + 1 shared, sparse GQA indexer)
Params	600.1B total / 27B active
Layers	92 (3 dense + 89 MoE) + 3 MTP
Attention	64 q heads / 4 kv, head_dim 192; sliding (512) × 3 → full, × 92
Context	1,048,576
Vocab	128,896

About the inference code

Step-5 uses a sparse-GQA indexer: a small auxiliary scoring network selects, per query, which cached key regions attend — everything else is masked out. This implementation follows the model's official reference semantics: CSA block compression of indexer keys, per-dim softmax(z)-weighted region summaries, and hard top-512-region selection with own-region force-append. Selection runs fused (scored and masked in-kernel), so the score matrix never materializes in the compute buffer.

The only environment flag is IK_SPARSE=1, which activates the sparse indexer on full-attention layers. Without it the model runs dense attention (correct, but slower at long context and not how the quants were calibrated). No other configuration is needed or supported — selection semantics are fixed in code.

This implementation is being upstreamed — open PR: ikawrakow/ik_llama.cpp#2737. Until it is merged, build from this repository's main branch.

PPL Analysis

Evaluation: ik_llama.cpp llama-perplexity on wiki.test.raw with the sparse indexer active (IK_SPARSE=1). Each quant table below is a matched-token PPL/KLD comparison against a Q8_0 logits base, so every ratio is computed against its own paired base. Both ctx 512 panels share one base (PPL = 1.582297 ± 0.005288) and both ctx 8192 panels share another (1.357235 ± 0.012489), which makes all four panels one comparable grid: two tiers × two regimes. For reference, a standalone Q8_0 run (ctx 512, 561 chunks, 287,232 tokens) gives 1.5812 ± 0.00528. Per-quant results live in each quant's tables under Quant Collection.

Imatrix

	Value
File	step-preview-B.imatrix (713 MB, 998/1000 tensor entries)
Corpus	ubergarm-imatrix-calibration-corpus-v02
Tokens	400K (50 chunks x 8192 tokens)
Context	8192
Runtime	ik_llama.cpp with sparse indexer active (IK_SPARSE=1)

The imatrix was collected with the sparse selection ACTIVE at deployment context, so activation statistics match how the model actually runs. Dense-mode calibration at ctx 8192 would measure the wrong distribution: with top-k 512 blocks the sparse path discards most keys at that length.

Download: step-preview-B.imatrix — reusable for any tier quant from the BF16 master.

Quant Collection

🟡 IQ4_KS-mix-B  333 GB / 310 GiB · 36 shards

Panel A — reference regime: ctx 512, full wiki.test panel (561 chunks, 287,232 tokens), matched against a Q8_0 logits base. This is the standard llama-perplexity comparison regime.

Metric	Value
PPL (quant)	1.634397 ± 0.005623
PPL (base ref, Q8_0)	1.582297 ± 0.005288
(PPL(Q)/PPL(base)) - 1	+3.29% ± 0.15%
KLD (mean)	0.100024 ± 0.000755
KLD (median)	0.003569
KLD (maximum)	9.930597
Same top-p	92.575% ± 0.069%
RMS Δp	12.803% ± 0.065%
Mean Δp	−1.023% ± 0.034%

Panel B — deployment regime: ctx 8192, 3 chunks of 8192 tokens (24,576 tokens), matched against a Q8_0 logits base collected at the same context. The imatrix was calibrated at ctx 8192 with the sparse indexer active, so this panel measures the quant in the regime it was calibrated for — activation statistics and attention selection both match deployment. Every metric improves. Sample is deliberately small (24,576 tokens vs 287,232), so the confidence intervals are ~3× wider than Panel A and cover a single contiguous corpus stretch; treat Panel A as the headline number and Panel B as the deployment-regime estimate.

Metric	Value
PPL (quant)	1.403946 ± 0.013595
PPL (base ref, Q8_0)	1.357235 ± 0.012489
(PPL(Q)/PPL(base)) - 1	+3.44% ± 0.40%
KLD (mean)	0.065653 ± 0.001948
KLD (median)	0.000503
KLD (maximum)	3.631622
Same top-p	94.831% ± 0.200%
RMS Δp	10.814% ± 0.222%
Mean Δp	−1.140% ± 0.097%

Reading the two panels together: absolute PPL is not comparable across panels — longer context lowers perplexity for both quant and base (base 1.582297 at ctx 512 vs 1.357235 at ctx 8192), which is why each panel quotes its own paired base and the ratios are what travel. The KLD gap between panels is partly context and partly content; measured on the same 24,576 tokens, the ctx 512 and ctx 8192 runs give KLD 0.0816 vs 0.0657 (−20%), so roughly a fifth of the improvement is the context regime itself. Per-chunk KLD through the 8192 run rises 0.0447 → 0.0561 → 0.0657 (cumulative), i.e. no depth collapse.

Quantization recipe

# Attention (all layers) blk\..*\.attn_q\.weight=q8_0 blk\..*\.attn_k\.weight=q8_0 blk\..*\.attn_v\.weight=q8_0 blk\..*\.attn_output\.weight=q8_0 blk\..*\.attn_gate\.weight=q8_0 # Indexer (full-attn layers) blk\..*\.indexer\.q\.weight=q8_0 blk\..*\.indexer\.k\.weight=q8_0 blk\..*\.indexer\.z\.weight=q8_0 blk\..*\.indexer\.w\.weight=q8_0 # Dense layers 0-2 + 91 blk\.(0|1|2|91)\.ffn_down\.weight=iq6_k blk\.(0|1|2|91)\.ffn_(gate|up)\.weight=iq5_ks blk\.(0|1|2|91)\.ffn_gate\.weight=iq5_ks # Shared experts blk\..*\.ffn_down_shexp\.weight=iq6_k blk\..*\.ffn_gate_shexp\.weight=iq6_k blk\..*\.ffn_up_shexp\.weight=iq6_k # Routed experts (uniform) # down one tier above up/gate blk\..*\.ffn_down_exps\.weight=iq4_k blk\..*\.ffn_(gate|up)_exps\.weight=iq4_ks # MTP / nextn tail # shared_head_head = draft lm_head # eh_proj = layer-input projector blk\..*\.nextn\.eh_proj\.weight=iq4_kss blk\..*\.nextn\.shared_head_head\.weight=iq5_ks # Embeddings / output token_embd\.weight=iq6_k output\.weight=iq6_k

🟡 IQ3_KS-mix-B  275 GB / 256 GiB · 30 shards — uploading

Panel A — reference regime: ctx 512, full wiki.test panel (561 chunks, 287,232 tokens), matched against a Q8_0 logits base. This is the standard llama-perplexity comparison regime.

Metric	Value
PPL (quant)	1.764302 ± 0.006521
PPL (base ref, Q8_0)	1.582297 ± 0.005288
(PPL(Q)/PPL(base)) - 1	+11.50% ± 0.24%
KLD (mean)	0.229590 ± 0.001509
KLD (median)	0.011254
KLD (maximum)	12.422327
Same top-p	88.419% ± 0.085%
RMS Δp	19.225% ± 0.079%
Mean Δp	−3.461% ± 0.050%

Panel B — deployment regime: ctx 8192, 3 chunks of 8192 tokens (24,576 tokens), matched against the same Q8_0 logits base used for the IQ4-B panel (paired base PPL 1.357235 ± 0.012489 is shared), so all four panels form one comparable grid: two tiers × two regimes. The imatrix was collected at ctx 8192 with the sparse indexer active — the regime this quant ships to run in — so Panel B is the context-matched measurement. As on the IQ4-B tier, every metric improves; on IQ4-B the context effect was isolated on identical tokens (0.0816 → 0.0657, −20%), so a part of the gain here is the context regime itself rather than calibration match. Sample is deliberately small (24,576 tokens vs 287,232) and covers a single contiguous corpus stretch, so confidence intervals are several times wider than Panel A; treat Panel A as the headline number.

Metric	Value
PPL (quant)	1.486197 ± 0.015380
PPL (base ref, Q8_0)	1.357235 ± 0.012489
(PPL(Q)/PPL(base)) - 1	+9.50% ± 0.66%
KLD (mean)	0.154146 ± 0.004044
KLD (median)	0.002003
KLD (maximum)	7.884445
Same top-p	91.762% ± 0.248%
RMS Δp	16.556% ± 0.269%
Mean Δp	−3.322% ± 0.146%

Per-chunk (cumulative) through the 8192 run: KLD 0.11076 → 0.12896 → 0.15415, same top-p 94.07% → 93.35% → 91.76% — a gentle drift with corpus content, no depth collapse. Tier ordering is preserved in both regimes: IQ4-B 0.0657 < IQ3-B 0.1541 at ctx 8192, IQ4-B 0.1000 < IQ3-B 0.2296 at ctx 512.

Quantization recipe

# Attention (all layers) blk\..*\.attn_q\.weight=q8_0 blk\..*\.attn_k\.weight=q8_0 blk\..*\.attn_v\.weight=q8_0 blk\..*\.attn_output\.weight=q8_0 blk\..*\.attn_gate\.weight=q8_0 # Indexer (full-attn layers) blk\..*\.indexer\.q\.weight=q8_0 blk\..*\.indexer\.k\.weight=q8_0 blk\..*\.indexer\.z\.weight=q8_0 blk\..*\.indexer\.w\.weight=q8_0 # Dense layers 0-2 + 91 blk\(0|1|2|91)\.ffn_down\.weight=iq6_k blk\(0|1|2|91)\.ffn_(gate|up)\.weight=iq5_ks # Shared experts blk\..*\.ffn_down_shexp\.weight=iq6_k blk\..*\.ffn_gate_shexp\.weight=iq5_ks blk\..*\.ffn_up_shexp\.weight=iq5_ks # Routed experts (uniform) # down one tier above up/gate blk\..*\.ffn_down_exps\.weight=iq4_ks blk\..*\.ffn_(gate|up)_exps\.weight=iq3_ks # MTP / nextn tail # shared_head_head = draft lm_head # eh_proj = layer-input projector blk\..*\.nextn\.eh_proj\.weight=iq4_kss blk\..*\.nextn\.shared_head_head\.weight=iq5_ks # Embeddings / output token_embd\.weight=iq6_k output\.weight=iq6_k

Making your own quants (conversion pipeline)

Convert with the Step5Model converter in ik_llama.cpp's convert_hf_to_gguf.py — it handles the sparse-indexer tensors and the norm-weight convention automatically. Do not use older third-party converter patches: they omit the indexer tensors and store the indexer q_norm zero-centered, which silently corrupts selection.

# 1. Convert HF -> BF16 GGUF
python convert_hf_to_gguf.py /path/to/Step5-Preview-HF \
  --outfile step5-preview-bf16.gguf --outtype bf16

# 2. Imatrix (sparse selection active, matching deployment)
./build/bin/llama-imatrix -m step5-preview-bf16.gguf \
  -f calibration-corpus.txt -o step5.imatrix \
  --gpu-layers 12 --ctx-size 8192

# 3. Quantize (IQK types are ik_llama.cpp-only)
./build/bin/llama-quantize --imatrix step5.imatrix \
  step5-preview-bf16.gguf step5-preview-IQ4_KS-mix-B.gguf IQ4_XS

Quick Start

Requires ik_llama.cpp PR #2737. Standard llama.cpp will not run these quants.

# Clone and build (fetches the PR branch) git clone https://github.com/ikawrakow/ik_llama.cpp cd ik_llama.cpp git fetch origin pull/2737/head:step5-pr && git checkout step5-pr cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON cmake --build build --config Release -j $(nproc)

# Download (main model) pip install huggingface_hub hf download L-Alchemyst/Step-5-Preview-GGUF --repo-type model --include "IQ4_KS-mix-B/*" --local-dir ./step5-iq4ks

# Optional: vision module (mmproj), required for image input hf download L-Alchemyst/Step-5-Preview-GGUF --repo-type model --include "mmproj-step5-vision-q8_0.gguf" --local-dir ./step5-iq4ks

Hybrid CPU+GPU: adjust -ot to your RAM/VRAM split — -cram is another adjustable flag. Export IK_SPARSE=1 to run the sparse indexer (the quants are calibrated in that regime).

export IK_SPARSE=1 ./build/bin/llama-server
-m ./step5-iq4ks/step5-preview-IQ4_KS-mix-B-00001-of-00036.gguf
--alias Alchemyst/Step-5
-ngl 999
-ot "blk.(2[1-9]|[3-8][0-9]|90).(ffn_up|ffn_gate|ffn_down)_exps=CPU"
-ub 4096 -b 4096
--flash-attn on
--ctx-checkpoints 5
--swa-compress
-c 80000
-cuda "offload-batch-size=128"
--spec-type mtp:n_max=1,p_min=0.0
-muge
-gr -ger
--merge-qkv
-ctk q8_0 -ctv q8_0
-khad -vhad
--jinja
-rtr
--threads 96 --threads-batch 128
--mmproj ./step5-iq4ks/mmproj-step5-vision-q8_0.gguf
--no-mmproj-offload
--port 8080

The --no-mmproj-offload flag keeps the vision module in system RAM instead of VRAM, trading a little image-processing speed for more GPU headroom.

CPU-only: use -ctk q8_0 and prefix with numactl -N ${SOCKET} -m ${SOCKET}.

Credits
Model: stepfun-ai
Runtime: ik_llama.cpp