rdtand/LFM2.5-8B-A1B-PrismaQuant-6.5bit-vllm

PrismaQuant LFM2.5-8B-A1B (mixed-precision, ~6.58 bpp)

Post-training mixed-precision quantization of LiquidAI/LFM2.5-8B-A1B, produced with PrismaQuant — a per-Linear format allocator that places each weight matrix at the cheapest format its KL-sensitivity allows, then exports to compressed-tensors.

Summary

Precision	compressed-tensors / mixed-precision
Average bits/param	~6.58 bpp (allocator post-cliff log knee)
On-disk size	8.35 GB
Quantized-Linear mix	NVFP4 (4b) ×27, FP8_E4M3 (8b) ×20, BF16 ×27
BF16 passthrough	short-conv in/out_proj, all RMSNorms, router + expert_bias, embed/lm_head, plus a few sensitive expert layers
end-KL vs BF16	0.131 nats/token (PrismaQuant harness)
ToolEvalBench (74 hard-mode)	70/100 (104/148) — identical to the BF16 source
vLLM serving	✅ loads + generates coherently (matches BF16)

Evaluation

ToolEvalBench agentic tool-call benchmark, 74 hard-mode scenarios (--temperature 0 --seed 1234 --timeout 180 --parallel 1), served through vLLM with the lfm2 tool-call parser:

	This (6.58 bpp NVFP4)	BF16 source
Score	70 / 100	70 / 100
Points	104 / 148	104 / 148

Quantization is capability-neutral here — the 4-bit MoE experts match the BF16 source on aggregate agentic tool use (and on Tool Selection, Restraint & Refusal, and Error Recovery the quantized model scores 100%). Per-category differences are within single-run greedy noise; the only category where the quant trails is Safety & Boundaries (20/26 vs 24/26).

The MoE experts (≈93% of params) carry the compression and are mostly NVFP4 (4-bit); sensitive paths stay FP8/BF16.

Serving (vLLM)

vllm serve <this-dir> --quantization compressed-tensors --trust-remote-code

Requires a vLLM build with Lfm2MoeForCausalLM and the LFM2 short-conv / linear-attention kernels (causal-conv1d, flash-linear-attention). Use the model's chat template — it's an instruct/reasoning model (raw-text completion will look degenerate; this is the base model's behavior, not a quantization artifact, and is identical to the BF16 source).

Method

• Probe — per-Linear diagonal-Fisher KL-sensitivity over calibration text.
• Cost — production-render-score: each (Linear, format) is rendered with its real export levers (GPTQ, joint-scale-opt, static-act-order) and scored by output MSE.
• Allocator — Pareto DP over per-Linear (format, cost) with a post-cliff log-error Kneedle pick; packed-MoE experts compete per-format.
• Calibration — diverse-v1, 8 samples × 1024 tokens.
• Built directly from the source per-expert checkpoint — PrismaQuant packs experts into the live module on the fly (no pre-pack).

Notes

• Quantization-only; weights are not further fine-tuned.
• end-KL (0.131 nats/token vs BF16) is the meaningful quality signal. Absolute raw-text perplexity is high/method-dependent (instruct model on raw text), not a clean quantization metric.
• The short-conv mixer Linears are kept BF16 (vLLM builds ShortConv without a quant_config), and NVFP4 experts are emitted under vLLM's canonical gate_proj/up_proj/down_proj scheme names so the FusedMoE scheme detector binds correctly.

Attribution: robert.tand@icloud.com