Cohere Transcribe 03-2026 โ€” MLX Mixed 2-bit/3-bit/4-bit

A dynamically quantized MLX build of CohereLabs/cohere-transcribe-03-2026: instead of one bit-width per section, every module gets a bit-width assigned from an empirical quantization-sensitivity map (Unsloth-style dynamic quantization, adapted to ASR). Most of the encoder runs at 2-bit; the empirically fragile groups (decoder, last encoder blocks) stay at 3/4-bit.

Compared to the uniform 3-bit/4-bit build, this checkpoint is 14% smaller and ~15% faster at nearly the same WER.

Key Metrics

Metric Value
Size 766 MB (vs 3.9 GB FP16 โ€” 5.2x smaller)
Effective bits/param 3.294
WER (LibriSpeech test-clean, full) 1.12%
WER (LibriSpeech test-other, full) 2.50%
Composite WER 1.81% (FP16 baseline: 1.55%)
RTFx (M4 Air, MLX) 22โ€“27x real-time

Method: sensitivity-guided bit allocation

  1. Sensitivity scan โ€” each (4-block, component) group of the 48-layer FastConformer encoder is fake-quantized to 2-bit alone (rest FP16) and scored by transcript divergence (CER) against the FP16 teacher on a calibration set.
  2. Greedy allocation โ€” groups above sensitivity thresholds are held at 4-bit / 3-bit; everything else drops to 2-bit (group size 64, affine). The decoder โ€” the single most sensitive component โ€” stays at 4-bit.
  3. The resulting per-module rule map ships in this repo as quant_bitmap.json for full reproducibility.

Uniform 2-bit costs +2.01 pt WER; this allocation recovers ~75% of that damage for only +0.4 effective bits.

Component Quantization
Encoder (most groups) 2-bit affine, group size 64
Encoder blocks 44โ€“47 ffn2/pointwise-conv 4-bit
Encoder groups with mid sensitivity 3-bit
Decoder + head 4-bit
Norms / embeddings / mel frontend FP16

1x1 Conv1d layers are converted to Linear equivalents to enable quantization.

Architecture

  • Base model: Cohere Transcribe 03-2026 (~2B params)
  • Encoder: FastConformer (48 layers, d=1280)
  • Decoder: Transformer (8 layers, d=1024)
  • Tokenizer: SentencePiece (16,384 tokens)

Usage

Requires mlx-audio installed from git main:

pip install "mlx-audio[stt] @ git+https://github.com/Blaizzy/mlx-audio.git"
# apply the loader patch shipped in this repo (fixes two upstream
# quantized-reload bugs; see mlx_audio_cohere_quant_patch.py)
from mlx_audio_cohere_quant_patch import apply_patch
apply_patch()

from mlx_audio.stt import load

model = load("MarkChen1214/cohere-transcribe-03-2026-MLX-Mixed-2bit3bit4bit")
result = model.generate(audio="audio.wav", language="en", punctuation=True)
print(result.text)

Eval Results (Full LibriSpeech, seed=42)

Dataset Samples Audio Hours WER RTFx
LibriSpeech test-clean 2,620 5.4h 1.12% 23.4x
LibriSpeech test-other 2,939 5.34h 2.50% 20.8x

Evaluated with the HF Open ASR Leaderboard methodology (Whisper EnglishTextNormalizer + jiwer).

License

MIT โ€” use it however you like, attribution appreciated.

The base model (CohereLabs/cohere-transcribe-03-2026) is Apache 2.0; its notices apply to the underlying weights.

Downloads last month
20
Safetensors
Model size
2B params
Tensor type
BF16
ยท
U32
ยท
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for MarkChen1214/cohere-transcribe-03-2026-MLX-Mixed-2bit3bit4bit

Quantized
(54)
this model

Collection including MarkChen1214/cohere-transcribe-03-2026-MLX-Mixed-2bit3bit4bit