NVIDIA Nemotron 3.5 Bolt E4B ยท bf16 ยท for radiance

nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 in one 17.0 GiB file for the radiance inference engine on AMD RDNA4. The weights are as NVIDIA ships them, with NVIDIA's DFlash drafter, the vision tower and the audio encoder.

File nemotron-3.5-bolt-e4b-bf16.rad, 17.03 GiB
Needs radiance 2.0.0 or newer; one card with 32 GB (tested on Radeon AI PRO R9700) and 8 GiB of free host memory
Format weights bf16, as in the checkpoint; activations bf16; lm_head bf16 (tied to the token table)
Speculation NVIDIA's DFlash drafter, 5 layers, int8 with one f16 scale per 128 weights, 7 draft tokens
Inputs text ยท image ยท audio
Context 65,536 tokens
Version 2.0 (2026-11-15)

Speed

Radeon AI PRO R9700 at 215 W, radiance 2.0.0, bf16 KV, 7 draft tokens, thinking off. BetterBench 0.6.1, full run, temperature 0.7. Tokens per second (median):

1 card 2 cards
decode, one stream 161.9 261.6
prefill at 24k tokens 6,918 5,006
4 streams, total 268 435

Quality

Distance to the bf16 original on the 80 MT-Bench questions, both turns. The text is the GPT-4 answers from lmsys/mt_bench_human_judgments, with the model's chat template and thinking off. The reference is the bf16 original in HF transformers, with fp32 arithmetic and the full vocabulary. radiance scores this file with --kld-ref, on the answer tokens only (42,283 tokens). radiance 2.0.0 (release image), bf16 KV, one card.

mean KL 99th percentile KL top token the same perplexity ratio
this file 0.00028 0.0035 99.5% ร—1.001
version 1.1 of this file 0.00028 0.00351 99.5% ร—1.001
keptn/nemotron-3.5-bolt-e4b-a4 0.0537 0.705 93.6% ร—0.994
keptn/nemotron-3.5-bolt-e4b-i8g 0.00275 0.036 98.4% ร—1.000

Quick start

Start the model on one card. Save this as radiance.yaml:

server:
  host: 0.0.0.0
  port: 8000
models:
  nemotron-3.5-bolt-e4b-bf16:
    model: /models/nemotron-3.5-bolt-e4b-bf16.rad
    tp: 1
    kv-cache-dtype: bf16
    max-model-len: 65536
    max-num-seqs: 8
    num-speculative-tokens: 7
    mm-max-patches: 2520
    vision-tower-placement: ram
    temp: 1.0
    top-p: 0.95
    top-k: 64
    chat-template-kwargs: {"enable_thinking":false}

The entry name nemotron-3.5-bolt-e4b-bf16 is the model id in requests. Then run:

hf download keptn/nemotron-3.5-bolt-e4b-bf16 nemotron-3.5-bolt-e4b-bf16.rad --local-dir ~/models
docker pull stilldeadcode/radiance:2.0.0
docker run -d --init --name nemotron-bolt --network host --group-add video --group-add render \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
    -v ~/models:/models:ro -v $PWD/radiance.yaml:/etc/radiance/radiance.yaml:ro \
    -e ROCR_VISIBLE_DEVICES=0 \
    stilldeadcode/radiance:2.0.0 --config /etc/radiance/radiance.yaml
change
no drafter num-speculative-tokens: 0
two cards ROCR_VISIBLE_DEVICES=0,1, tp: 2, add p2p: on
four cards ROCR_VISIBLE_DEVICES=0,1,2,3, tp: 4, add p2p: on
text only mm-max-patches: 0 (no images and no audio), remove vision-tower-placement
thinking off by default, as in NVIDIA's template; per request "chat_template_kwargs": {"enable_thinking": true}

The drafter runs at 7 draft tokens only: it was trained on blocks of 8. Sampling is NVIDIA's (temperature 1.0, top_p 0.95, both thinking modes) and the checkpoint's top_k 64.

Memory and context

KV cache at max-model-len: 65536 and 8 sequences, bf16 KV, radiance 2.0.0, from radiance's startup report:

KV cache tokens of context in total full-length sessions
1 card 18.24 GiB 524,304 8.0
2 cards 9.12 GiB a card 524,336 8.0

Images, audio and video

The same inputs go to radiance and to the bf16 original in HF transformers. Greedy, thinking off, 2 cards, radiance 1.15.1 (release image). There is one measurement for each vision or audio encoder, made with nemotron-3.5-bolt-e4b-bf16.rad.

input cases same answer mean KL top token the same
images (DocVQA and AI2D, fixed selection) 20 19/20 0.00761 69/70
audio (LibriSpeech test-clean, 12 clips) 12 12/12 0.00034 347/347

Word error rate on the audio clips: HF 6.37% ยท radiance 6.37%.

How it was made

  1. Downloaded nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 (revision 49843eed).
  2. Converted NVIDIA's drafter from GGUF to safetensors with drafter/gguf_to_safetensors.py. The GGUF is nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash (revision 0949a2eb); config.json comes from nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16-DFlash-NVFP4 (revision 913d13c4).
  3. Compared every drafter tensor with the NVFP4 release: cosine 0.989 to 0.994 for the matrices (4-bit noise), 1.000 for the norms.
  4. Built the container. The trunk, the vision tower and the audio encoder stay as in the checkpoint. The drafter is int8 with one f16 scale per 128 weights (round to nearest), as in radiance 2.0's own E4B recipes. Converted with radiance 2.0.0's rad-convert.
python drafter/gguf_to_safetensors.py Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf \
    NVIDIA-Nemotron-3.5-Bolt-E4B-BF16-DFlash-NVFP4 dflash-bf16
rad-convert NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 --recipe nemotron-3.5-bolt-e4b-bf16.recipe \
    --draft-model dflash-bf16 --set quantization_config.quant_method=compressed-tensors \
    -o nemotron-3.5-bolt-e4b-bf16.rad

Calibration: no calibration data.

Model card with: radiance-model-cards-framework 0.3 (codeberg.org/keptn42/radiance-model-cards-framework).

Limits

  • radiance runs on AMD RDNA4 cards.
  • The file needs radiance 2.0.0 or newer.
  • Four cards are not measured on radiance 2.0.
  • NVIDIA lists text as the only input of this model. The vision tower and the audio encoder are those of the checkpoint.
  • The context is 65,536 tokens, the length NVIDIA trained.

Versions

version date change
2.0 2026-11-15 for radiance 2.0: the drafter is int8, one f16 scale per 128 weights
1.1 2026-10-25 adds the audio encoder; the drafter is block FP8; radiance 1.13.0 to 1.17.0

Version 1.1 for radiance 1.13.0 to 1.17.0: hf download keptn/nemotron-3.5-bolt-e4b-bf16 nemotron-3.5-bolt-e4b-bf16.rad --revision v1.1 --local-dir ~/models. Its card at that revision has the command-line flags.

Credits

License

OpenMDW License Agreement 1.1, the license of the base model. See LICENSE. The Gemma 4 base model is under the Apache License 2.0, see LICENSE.Apache-2.0. NOTICE names the sources and NVIDIA's terms.

Downloads last month
189
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for keptn/nemotron-3.5-bolt-e4b-bf16