Instructions to use MarkChen1214/cohere-transcribe-03-2026-MLX-Mixed-2bit3bit4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use MarkChen1214/cohere-transcribe-03-2026-MLX-Mixed-2bit3bit4bit with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download MarkChen1214/cohere-transcribe-03-2026-MLX-Mixed-2bit3bit4bit --local-dir cohere-transcribe-03-2026-MLX-Mixed-2bit3bit4bit
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Cohere Transcribe 03-2026 โ MLX Mixed 2-bit/3-bit/4-bit
A dynamically quantized MLX build of CohereLabs/cohere-transcribe-03-2026: instead of one bit-width per section, every module gets a bit-width assigned from an empirical quantization-sensitivity map (Unsloth-style dynamic quantization, adapted to ASR). Most of the encoder runs at 2-bit; the empirically fragile groups (decoder, last encoder blocks) stay at 3/4-bit.
Compared to the uniform 3-bit/4-bit build, this checkpoint is 14% smaller and ~15% faster at nearly the same WER.
Key Metrics
| Metric | Value |
|---|---|
| Size | 766 MB (vs 3.9 GB FP16 โ 5.2x smaller) |
| Effective bits/param | 3.294 |
| WER (LibriSpeech test-clean, full) | 1.12% |
| WER (LibriSpeech test-other, full) | 2.50% |
| Composite WER | 1.81% (FP16 baseline: 1.55%) |
| RTFx (M4 Air, MLX) | 22โ27x real-time |
Method: sensitivity-guided bit allocation
- Sensitivity scan โ each (4-block, component) group of the 48-layer FastConformer encoder is fake-quantized to 2-bit alone (rest FP16) and scored by transcript divergence (CER) against the FP16 teacher on a calibration set.
- Greedy allocation โ groups above sensitivity thresholds are held at 4-bit / 3-bit; everything else drops to 2-bit (group size 64, affine). The decoder โ the single most sensitive component โ stays at 4-bit.
- The resulting per-module rule map ships in this repo as
quant_bitmap.jsonfor full reproducibility.
Uniform 2-bit costs +2.01 pt WER; this allocation recovers ~75% of that damage for only +0.4 effective bits.
| Component | Quantization |
|---|---|
| Encoder (most groups) | 2-bit affine, group size 64 |
| Encoder blocks 44โ47 ffn2/pointwise-conv | 4-bit |
| Encoder groups with mid sensitivity | 3-bit |
| Decoder + head | 4-bit |
| Norms / embeddings / mel frontend | FP16 |
1x1 Conv1d layers are converted to Linear equivalents to enable quantization.
Architecture
- Base model: Cohere Transcribe 03-2026 (~2B params)
- Encoder: FastConformer (48 layers, d=1280)
- Decoder: Transformer (8 layers, d=1024)
- Tokenizer: SentencePiece (16,384 tokens)
Usage
Requires mlx-audio installed from git main:
pip install "mlx-audio[stt] @ git+https://github.com/Blaizzy/mlx-audio.git"
# apply the loader patch shipped in this repo (fixes two upstream
# quantized-reload bugs; see mlx_audio_cohere_quant_patch.py)
from mlx_audio_cohere_quant_patch import apply_patch
apply_patch()
from mlx_audio.stt import load
model = load("MarkChen1214/cohere-transcribe-03-2026-MLX-Mixed-2bit3bit4bit")
result = model.generate(audio="audio.wav", language="en", punctuation=True)
print(result.text)
Eval Results (Full LibriSpeech, seed=42)
| Dataset | Samples | Audio Hours | WER | RTFx |
|---|---|---|---|---|
| LibriSpeech test-clean | 2,620 | 5.4h | 1.12% | 23.4x |
| LibriSpeech test-other | 2,939 | 5.34h | 2.50% | 20.8x |
Evaluated with the HF Open ASR Leaderboard methodology (Whisper EnglishTextNormalizer + jiwer).
License
MIT โ use it however you like, attribution appreciated.
The base model (CohereLabs/cohere-transcribe-03-2026) is Apache 2.0; its notices apply to the underlying weights.
- Downloads last month
- 20
4-bit
Model tree for MarkChen1214/cohere-transcribe-03-2026-MLX-Mixed-2bit3bit4bit
Base model
CohereLabs/cohere-transcribe-03-2026