NVIDIA Nemotron 3.5 Bolt E4B ยท bf16 ยท for radiance
nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 in one 17.0 GiB file for the radiance inference engine on AMD RDNA4. The weights are as NVIDIA ships them, with NVIDIA's DFlash drafter, the vision tower and the audio encoder.
| File | nemotron-3.5-bolt-e4b-bf16.rad, 17.03 GiB |
| Needs | radiance 2.0.0 or newer; one card with 32 GB (tested on Radeon AI PRO R9700) and 8 GiB of free host memory |
| Format | weights bf16, as in the checkpoint; activations bf16; lm_head bf16 (tied to the token table) |
| Speculation | NVIDIA's DFlash drafter, 5 layers, int8 with one f16 scale per 128 weights, 7 draft tokens |
| Inputs | text ยท image ยท audio |
| Context | 65,536 tokens |
| Version | 2.0 (2026-11-15) |
Speed
Radeon AI PRO R9700 at 215 W, radiance 2.0.0, bf16 KV, 7 draft tokens, thinking off. BetterBench 0.6.1, full run, temperature 0.7. Tokens per second (median):
| 1 card | 2 cards | |
|---|---|---|
| decode, one stream | 161.9 | 261.6 |
| prefill at 24k tokens | 6,918 | 5,006 |
| 4 streams, total | 268 | 435 |
Quality
Distance to the bf16 original on the 80 MT-Bench questions, both turns. The text is the GPT-4 answers from lmsys/mt_bench_human_judgments, with the model's chat template and thinking off. The reference is the bf16 original in HF transformers, with fp32 arithmetic and the full vocabulary. radiance scores this file with --kld-ref, on the answer tokens only (42,283 tokens). radiance 2.0.0 (release image), bf16 KV, one card.
| mean KL | 99th percentile KL | top token the same | perplexity ratio | |
|---|---|---|---|---|
| this file | 0.00028 | 0.0035 | 99.5% | ร1.001 |
| version 1.1 of this file | 0.00028 | 0.00351 | 99.5% | ร1.001 |
| keptn/nemotron-3.5-bolt-e4b-a4 | 0.0537 | 0.705 | 93.6% | ร0.994 |
| keptn/nemotron-3.5-bolt-e4b-i8g | 0.00275 | 0.036 | 98.4% | ร1.000 |
Quick start
Start the model on one card. Save this as radiance.yaml:
server:
host: 0.0.0.0
port: 8000
models:
nemotron-3.5-bolt-e4b-bf16:
model: /models/nemotron-3.5-bolt-e4b-bf16.rad
tp: 1
kv-cache-dtype: bf16
max-model-len: 65536
max-num-seqs: 8
num-speculative-tokens: 7
mm-max-patches: 2520
vision-tower-placement: ram
temp: 1.0
top-p: 0.95
top-k: 64
chat-template-kwargs: {"enable_thinking":false}
The entry name nemotron-3.5-bolt-e4b-bf16 is the model id in requests. Then run:
hf download keptn/nemotron-3.5-bolt-e4b-bf16 nemotron-3.5-bolt-e4b-bf16.rad --local-dir ~/models
docker pull stilldeadcode/radiance:2.0.0
docker run -d --init --name nemotron-bolt --network host --group-add video --group-add render \
--device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
-v ~/models:/models:ro -v $PWD/radiance.yaml:/etc/radiance/radiance.yaml:ro \
-e ROCR_VISIBLE_DEVICES=0 \
stilldeadcode/radiance:2.0.0 --config /etc/radiance/radiance.yaml
| change | |
|---|---|
| no drafter | num-speculative-tokens: 0 |
| two cards | ROCR_VISIBLE_DEVICES=0,1, tp: 2, add p2p: on |
| four cards | ROCR_VISIBLE_DEVICES=0,1,2,3, tp: 4, add p2p: on |
| text only | mm-max-patches: 0 (no images and no audio), remove vision-tower-placement |
| thinking | off by default, as in NVIDIA's template; per request "chat_template_kwargs": {"enable_thinking": true} |
The drafter runs at 7 draft tokens only: it was trained on blocks of 8. Sampling is NVIDIA's (temperature 1.0, top_p 0.95, both thinking modes) and the checkpoint's top_k 64.
Memory and context
KV cache at max-model-len: 65536 and 8 sequences, bf16 KV, radiance 2.0.0, from radiance's startup report:
| KV cache | tokens of context in total | full-length sessions | |
|---|---|---|---|
| 1 card | 18.24 GiB | 524,304 | 8.0 |
| 2 cards | 9.12 GiB a card | 524,336 | 8.0 |
Images, audio and video
The same inputs go to radiance and to the bf16 original in HF transformers. Greedy, thinking off, 2 cards, radiance 1.15.1 (release image). There is one measurement for each vision or audio encoder, made with nemotron-3.5-bolt-e4b-bf16.rad.
| input | cases | same answer | mean KL | top token the same |
|---|---|---|---|---|
| images (DocVQA and AI2D, fixed selection) | 20 | 19/20 | 0.00761 | 69/70 |
| audio (LibriSpeech test-clean, 12 clips) | 12 | 12/12 | 0.00034 | 347/347 |
Word error rate on the audio clips: HF 6.37% ยท radiance 6.37%.
How it was made
- Downloaded
nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16(revision49843eed). - Converted NVIDIA's drafter from GGUF to safetensors with
drafter/gguf_to_safetensors.py. The GGUF isnvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash(revision0949a2eb);config.jsoncomes fromnvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16-DFlash-NVFP4(revision913d13c4). - Compared every drafter tensor with the NVFP4 release: cosine 0.989 to 0.994 for the matrices (4-bit noise), 1.000 for the norms.
- Built the container. The trunk, the vision tower and the audio encoder stay as in the checkpoint. The drafter is int8 with one f16 scale per 128 weights (round to nearest), as in radiance 2.0's own E4B recipes. Converted with radiance 2.0.0's
rad-convert.
python drafter/gguf_to_safetensors.py Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf \
NVIDIA-Nemotron-3.5-Bolt-E4B-BF16-DFlash-NVFP4 dflash-bf16
rad-convert NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 --recipe nemotron-3.5-bolt-e4b-bf16.recipe \
--draft-model dflash-bf16 --set quantization_config.quant_method=compressed-tensors \
-o nemotron-3.5-bolt-e4b-bf16.rad
Calibration: no calibration data.
Model card with: radiance-model-cards-framework 0.3 (codeberg.org/keptn42/radiance-model-cards-framework).
Limits
- radiance runs on AMD RDNA4 cards.
- The file needs radiance 2.0.0 or newer.
- Four cards are not measured on radiance 2.0.
- NVIDIA lists text as the only input of this model. The vision tower and the audio encoder are those of the checkpoint.
- The context is 65,536 tokens, the length NVIDIA trained.
Versions
| version | date | change |
|---|---|---|
| 2.0 | 2026-11-15 | for radiance 2.0: the drafter is int8, one f16 scale per 128 weights |
| 1.1 | 2026-10-25 | adds the audio encoder; the drafter is block FP8; radiance 1.13.0 to 1.17.0 |
Version 1.1 for radiance 1.13.0 to 1.17.0: hf download keptn/nemotron-3.5-bolt-e4b-bf16 nemotron-3.5-bolt-e4b-bf16.rad --revision v1.1 --local-dir ~/models. Its card at that revision has the command-line flags.
Credits
- Model and drafter: NVIDIA, post-trained from Google's Gemma 4 E4B (Apache 2.0).
- radiance: StillDeadcode (codeberg.org/StillDeadcode/radiance).
- Test sets: MT-Bench, DocVQA, AI2D, LibriSpeech.
License
OpenMDW License Agreement 1.1, the license of the base model. See LICENSE. The Gemma 4 base model is under the Apache License 2.0, see LICENSE.Apache-2.0. NOTICE names the sources and NVIDIA's terms.
- Downloads last month
- 189
Model tree for keptn/nemotron-3.5-bolt-e4b-bf16
Base model
google/gemma-4-E4B