Instructions to use AXERA-TECH/Qwen3-VL-4B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AXERA-TECH/Qwen3-VL-4B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="AXERA-TECH/Qwen3-VL-4B-Instruct")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("AXERA-TECH/Qwen3-VL-4B-Instruct", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AXERA-TECH/Qwen3-VL-4B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AXERA-TECH/Qwen3-VL-4B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/Qwen3-VL-4B-Instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/AXERA-TECH/Qwen3-VL-4B-Instruct

SGLang

How to use AXERA-TECH/Qwen3-VL-4B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AXERA-TECH/Qwen3-VL-4B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/Qwen3-VL-4B-Instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AXERA-TECH/Qwen3-VL-4B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/Qwen3-VL-4B-Instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use AXERA-TECH/Qwen3-VL-4B-Instruct with Docker Model Runner:
```
docker model run hf.co/AXERA-TECH/Qwen3-VL-4B-Instruct
```

Qwen3-VL

This version of Qwen3-VL-2B-Instruct and Qwen3-VL-4B-Instruct have been converted to run on the Axera NPU using w8a16 quantization.

Compatible with Pulsar2 version: 5.0

Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo :

Pulsar2 Link, How to Convert LLM from Huggingface to axmodel

AXera NPU HOST LLM Runtime

Support Platform

AX650
- AX650N DEMO Board
- M4N-Dock(爱芯派Pro)
- M.2 Accelerator card

Image Process

Chips	input size	image num	image encoder	ttft(168 tokens)	w8a16	CMM	Flash
AX650	384*384	1	236 ms	907 ms	4.3 tokens/sec	7.3GiB	7.9GiB

Video Process

Chips	input size	image num	image encoder	ttft(600 tokens)	w8a16	CMM	Flash
AX650	384*384	8	778 ms	2442 ms	4.3 tokens/sec	7.3GiB	7.9GiB

The DDR capacity refers to the CMM memory that needs to be consumed. Ensure that the CMM memory allocation on the development board is greater than this value.

How to use

Download all files from this repository to the device

Demo Run

Image understand demo

start tokenizer server for image understand demo

axllm from https://github.com/AXERA-TECH/ax-llm

axllm run Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/

run image understand demo

input text

图片描述的内容是什么

input image

``` root@ax650:/mnt/sdcard/projects/llm# axllm run Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/ 09:45:26.953 INF Init:2700 | LLM init start tokenizer_type = 3 huggingface tokenizer mode = gpt2_byte_bpe 96% | ############################## | 30 / 31 [21.99s<22.73s, 1.36 count/s] init post axmodel ok,remain_cmm(5969 MB) 09:45:48.978 INF Init:3023 | max_token_len : 1023 09:45:48.978 INF Init:3026 | kv_cache_size : 1024, kv_cache_num: 1023 09:45:48.978 INF init_groups_from_model:1827 | prefill_token_num : 128 09:45:48.978 INF init_groups_from_model:2075 | decode grp: 0, gid: 0, max_token_len : 1023 09:45:48.978 INF init_groups_from_model:2079 | prefill grp: 0, gid: 1, history_cap: 0, total_cap: 128, symbolic_cap: 1 09:45:48.978 INF init_groups_from_model:2079 | prefill grp: 1, gid: 2, history_cap: 128, total_cap: 256, symbolic_cap: 128 09:45:48.978 INF init_groups_from_model:2079 | prefill grp: 2, gid: 3, history_cap: 256, total_cap: 384, symbolic_cap: 256 09:45:48.978 INF init_groups_from_model:2079 | prefill grp: 3, gid: 4, history_cap: 384, total_cap: 512, symbolic_cap: 384 09:45:48.978 INF init_groups_from_model:2086 | prefill_max_token_num : 512 09:45:48.978 INF Init:27 | LLaMaEmbedSelector use mmap 100% | ################################ | 31 / 31 [21.99s<21.99s, 1.41 count/s] embed_selector init ok 09:45:55.206 INF Init:1647 | Qwen token ids: vision_start=151652 image_pad=151655 video_pad=151656 audio_pad=-1 09:45:55.207 INF Init:1704 | VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32 09:45:55.207 INF Init:1710 | VisionModule deepstack enabled: layers=3 09:45:55.207 WRN Init:1713 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible) 09:45:55.207 INF load_config:554 | load config: 09:45:55.207 INF load_config:554 | { 09:45:55.207 INF load_config:554 | "enable_repetition_penalty": true, 09:45:55.207 INF load_config:554 | "enable_temperature": true, 09:45:55.207 INF load_config:554 | "enable_top_k_sampling": false, 09:45:55.207 INF load_config:554 | "enable_top_p_sampling": true, 09:45:55.207 INF load_config:554 | "penalty_window": 128, 09:45:55.207 INF load_config:554 | "repetition_penalty": 1.1, 09:45:55.207 INF load_config:554 | "temperature": 0.7, 09:45:55.207 INF load_config:554 | "top_k": 20, 09:45:55.207 INF load_config:554 | "top_p": 0.8 09:45:55.207 INF load_config:554 | } 09:45:55.207 INF Init:3138 | LLM init ok Commands: /q, /exit 退出 /reset 重置 kvcache /dd 删除一轮对话 /pp 打印历史对话 Ctrl+C: 停止当前生成 VLM enabled: after each prompt, input media path (empty = text-only). Use "video:" or "video:[:]" for video (default fps=2), "audio:" for audio.

prompt >> 图片描述的内容是什么 media >> /mnt/sdcard/projects/llm/Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/01.jpg 09:46:59.858 INF EncodeForContent:2344 | Qwen-VL pixel_values[0] bytes=884736 min=6 max=219 (w=384 h=384 tp=2 ps=16 sm=2) 09:47:00.019 INF EncodeForContent:2403 | vision cache store: /mnt/sdcard/projects/llm/Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/test.jpg 09:47:00.034 INF SetKVCache:3721 | decode_grpid:0 prefill_grpid:1 history_cap:0 total_cap:128 symbolic_cap:1 precompute_len:0 input_num_token:169 prefer_symbolic_group:0 09:47:00.034 INF SetKVCache:3743 | current prefill_max_token_num:512 09:47:00.060 INF SetKVCache:3759 | first run 09:47:00.093 INF Run:4287 | input token num : 169, prefill_split_num : 2 09:47:00.093 INF Run:4367 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=128 09:47:00.244 INF Run:4367 | prefill chunk p=1 history_len=128 grpid=3 kv_cache_num=256 input_tokens=41 09:47:00.442 INF Run:4680 | ttft: 348.99 ms 这09:47:00.442 INF Run:4722 | VLM decode positions: rope_start=37 dense_kv_start=169 是一张室内空间的图片，可能是客厅或餐厅区域。房间的墙壁为暖黄色调，搭配深色木质地板和一些绿色的装饰元素。空间内有多种家具和电器，包括一个大尺寸的平板电视、一个带有透明玻璃柜的橱柜以及一个圆形的餐桌。房间内的自然光线透过窗户洒进来，营造出温馨而现代的氛围。

此外，墙上的挂画、墙上的灯具和室内的设计风格都显示出一种简洁而现代的装修风格。

09:47:09.200 NTC Run:5095 | hit eos,decode avg 11.88 token/s 09:47:09.200 INF GetKVCache:3698 | precompute_len:274, remaining:238 (tracked)


#### Video understand demo

##### start tokenizer server for image understand demo

axllm run Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/


##### run video understand demo
- input text

描述视频的内容


- input video  

./video

root@ax650:/mnt/sdcard/projects/llm# axllm run Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/ 09:44:04.332 INF Init:2700 | LLM init start tokenizer_type = 3 huggingface tokenizer mode = gpt2_byte_bpe 96% | ############################## | 30 / 31 [4.15s<4.29s, 7.23 count/s] init post axmodel ok,remain_cmm(5969 MB) 09:44:08.515 INF Init:3023 | max_token_len : 1023 09:44:08.515 INF Init:3026 | kv_cache_size : 1024, kv_cache_num: 1023 09:44:08.515 INF init_groups_from_model:1827 | prefill_token_num : 128 09:44:08.515 INF init_groups_from_model:2075 | decode grp: 0, gid: 0, max_token_len : 1023 09:44:08.515 INF init_groups_from_model:2079 | prefill grp: 0, gid: 1, history_cap: 0, total_cap: 128, symbolic_cap: 1 09:44:08.515 INF init_groups_from_model:2079 | prefill grp: 1, gid: 2, history_cap: 128, total_cap: 256, symbolic_cap: 128 09:44:08.515 INF init_groups_from_model:2079 | prefill grp: 2, gid: 3, history_cap: 256, total_cap: 384, symbolic_cap: 256 09:44:08.515 INF init_groups_from_model:2079 | prefill grp: 3, gid: 4, history_cap: 384, total_cap: 512, symbolic_cap: 384 09:44:08.515 INF init_groups_from_model:2086 | prefill_max_token_num : 512 09:44:08.515 INF Init:27 | LLaMaEmbedSelector use mmap 100% | ################################ | 31 / 31 [4.15s<4.15s, 7.47 count/s] embed_selector init ok 09:44:08.913 INF Init:1647 | Qwen token ids: vision_start=151652 image_pad=151655 video_pad=151656 audio_pad=-1 09:44:08.913 INF Init:1704 | VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32 09:44:08.913 INF Init:1710 | VisionModule deepstack enabled: layers=3 09:44:08.913 WRN Init:1713 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible) 09:44:08.914 INF load_config:554 | load config: 09:44:08.914 INF load_config:554 | { 09:44:08.914 INF load_config:554 | "enable_repetition_penalty": true, 09:44:08.914 INF load_config:554 | "enable_temperature": true, 09:44:08.914 INF load_config:554 | "enable_top_k_sampling": false, 09:44:08.914 INF load_config:554 | "enable_top_p_sampling": true, 09:44:08.914 INF load_config:554 | "penalty_window": 128, 09:44:08.914 INF load_config:554 | "repetition_penalty": 1.1, 09:44:08.914 INF load_config:554 | "temperature": 0.7, 09:44:08.914 INF load_config:554 | "top_k": 20, 09:44:08.914 INF load_config:554 | "top_p": 0.8 09:44:08.914 INF load_config:554 | } 09:44:08.914 INF Init:3138 | LLM init ok Commands: /q, /exit 退出 /reset 重置 kvcache /dd 删除一轮对话 /pp 打印历史对话 Ctrl+C: 停止当前生成 VLM enabled: after each prompt, input media path (empty = text-only). Use "video:" or "video:[:]" for video (default fps=2), "audio:" for audio.

prompt >> 描述视频的内容 media >> /mnt/sdcard/projects/llm/Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/video/ 09:44:38.482 INF EncodeForContent:2344 | Qwen-VL pixel_values[0] bytes=884736 min=0 max=255 (w=384 h=384 tp=2 ps=16 sm=2) 09:44:38.643 INF EncodeForContent:2403 | vision cache store: /mnt/sdcard/projects/llm/Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/video/frame_0000.jpg 09:44:38.674 INF EncodeForContent:2344 | Qwen-VL pixel_values[0] bytes=884736 min=0 max=255 (w=384 h=384 tp=2 ps=16 sm=2) 09:44:38.834 INF EncodeForContent:2403 | vision cache store: /mnt/sdcard/projects/llm/Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/video/frame_0008.jpg 09:44:38.864 INF EncodeForContent:2344 | Qwen-VL pixel_values[0] bytes=884736 min=0 max=255 (w=384 h=384 tp=2 ps=16 sm=2) 09:44:39.023 INF EncodeForContent:2403 | vision cache store: /mnt/sdcard/projects/llm/Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/video/frame_0016.jpg 09:44:39.049 INF SetKVCache:3721 | decode_grpid:0 prefill_grpid:1 history_cap:0 total_cap:128 symbolic_cap:1 precompute_len:0 input_num_token:456 prefer_symbolic_group:0 09:44:39.049 INF SetKVCache:3743 | current prefill_max_token_num:512 09:44:39.075 INF SetKVCache:3759 | first run 09:44:39.076 INF Run:4287 | input token num : 456, prefill_split_num : 4 09:44:39.076 INF Run:4367 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=128 09:44:39.230 INF Run:4367 | prefill chunk p=1 history_len=128 grpid=3 kv_cache_num=256 input_tokens=128 09:44:39.401 INF Run:4367 | prefill chunk p=2 history_len=256 grpid=4 kv_cache_num=384 input_tokens=128 09:44:39.584 INF Run:4367 | prefill chunk p=3 history_len=384 grpid=4 kv_cache_num=384 input_tokens=72 09:44:39.795 INF Run:4680 | ttft: 719.15 ms 这09:44:39.795 INF Run:4722 | VLM decode positions: rope_start=324 dense_kv_start=456 是一段关于两只小松鼠的视频。

画面中，两只小松鼠正站在一块布满岩石和泥土的山坡上，它们身体前倾，相互靠近。一只小松鼠面向镜头，另一只则侧身对着镜头，它们似乎在进行一场亲密的互动或是在进行某种游戏。

背景是连绵起伏的山峦，植被茂盛，天空晴朗。整个场景呈现出一种自然和谐的氛围。

虽然图像有些模糊，但可以清晰地看到两只小松鼠的毛色——它们的毛色以灰白色为主，并带有棕色的条纹。它们的耳朵、眼睛以及尾巴都清晰可见。

整体来看，这是一段关于两只小松鼠在山林间嬉戏玩耍的视频，画面充满了生机与活力。

09:44:53.252 NTC Run:5095 | hit eos,decode avg 12.11 token/s 09:44:53.252 INF GetKVCache:3698 | precompute_len:620, remaining:-108 (tracked)