---
license: mit
language:
- en
- zh
base_model:
- Qwen/Qwen3-VL-2B-Instruct
- Qwen/Qwen3-VL-4B-Instruct
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- Qwen3-VL
- Qwen3-VL-2B-Instruct
- Qwen3-VL-4B-Instruct
- Int8
- VLM
---

# Qwen3-VL

This version of Qwen3-VL-2B-Instruct and Qwen3-VL-4B-Instruct have been converted to run on the Axera NPU using **w8a16** quantization. 

Compatible with Pulsar2 version: 5.0

## Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo : 

- https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct
- https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct

[Pulsar2 Link, How to Convert LLM from Huggingface to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/appendix/build_llm.html) 

[AXera NPU HOST LLM Runtime](https://github.com/AXERA-TECH/Qwen3-VL.AXERA) 


## Support Platform

- AX650
  - AX650N DEMO Board
  - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html)
  - [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html)

**Image Process**
|Chips| input size | image num | image encoder | ttft(168 tokens) | w8a16 | CMM | Flash |
|--|--|--|--|--|--|--|--|
|AX650| 384*384 | 1 | 236 ms | 907 ms | 4.3 tokens/sec| 7.3GiB | 7.9GiB |

**Video Process**
|Chips| input size | image num | image encoder |ttft(600 tokens) | w8a16 | CMM | Flash |
|--|--|--|--|--|--|--|--|
|AX650| 384*384 | 8  | 778 ms | 2442 ms | 4.3 tokens/sec| 7.3GiB | 7.9GiB |

The DDR capacity refers to the CMM memory that needs to be consumed. Ensure that the CMM memory allocation on the development board is greater than this value.

## How to use

Download all files from this repository to the device

### Demo Run

#### Image understand demo

##### start tokenizer server for image understand demo

axllm from https://github.com/AXERA-TECH/ax-llm

```
axllm run Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/
```

##### run image understand demo

- input text

```
图片描述的内容是什么
```

- input image

![](01.jpg)
```
root@ax650:/mnt/sdcard/projects/llm# axllm run Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/
09:45:26.953 INF Init:2700 | LLM init start
tokenizer_type = 3
huggingface tokenizer mode = gpt2_byte_bpe
 96% | ##############################   |  30 /  31 [21.99s<22.73s, 1.36 count/s] init post axmodel ok,remain_cmm(5969 MB)
09:45:48.978 INF Init:3023 | max_token_len : 1023
09:45:48.978 INF Init:3026 | kv_cache_size : 1024, kv_cache_num: 1023
09:45:48.978 INF init_groups_from_model:1827 | prefill_token_num : 128
09:45:48.978 INF init_groups_from_model:2075 | decode grp: 0, gid: 0, max_token_len : 1023
09:45:48.978 INF init_groups_from_model:2079 | prefill grp: 0, gid: 1, history_cap: 0, total_cap: 128, symbolic_cap: 1
09:45:48.978 INF init_groups_from_model:2079 | prefill grp: 1, gid: 2, history_cap: 128, total_cap: 256, symbolic_cap: 128
09:45:48.978 INF init_groups_from_model:2079 | prefill grp: 2, gid: 3, history_cap: 256, total_cap: 384, symbolic_cap: 256
09:45:48.978 INF init_groups_from_model:2079 | prefill grp: 3, gid: 4, history_cap: 384, total_cap: 512, symbolic_cap: 384
09:45:48.978 INF init_groups_from_model:2086 | prefill_max_token_num : 512
09:45:48.978 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  31 /  31 [21.99s<21.99s, 1.41 count/s] embed_selector init ok
09:45:55.206 INF Init:1647 | Qwen token ids: vision_start=151652 image_pad=151655 video_pad=151656 audio_pad=-1
09:45:55.207 INF Init:1704 | VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32
09:45:55.207 INF Init:1710 | VisionModule deepstack enabled: layers=3
09:45:55.207 WRN Init:1713 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
09:45:55.207 INF load_config:554 | load config:
09:45:55.207 INF load_config:554 | {
09:45:55.207 INF load_config:554 |     "enable_repetition_penalty": true,
09:45:55.207 INF load_config:554 |     "enable_temperature": true,
09:45:55.207 INF load_config:554 |     "enable_top_k_sampling": false,
09:45:55.207 INF load_config:554 |     "enable_top_p_sampling": true,
09:45:55.207 INF load_config:554 |     "penalty_window": 128,
09:45:55.207 INF load_config:554 |     "repetition_penalty": 1.1,
09:45:55.207 INF load_config:554 |     "temperature": 0.7,
09:45:55.207 INF load_config:554 |     "top_k": 20,
09:45:55.207 INF load_config:554 |     "top_p": 0.8
09:45:55.207 INF load_config:554 | }
09:45:55.207 INF Init:3138 | LLM init ok
Commands:
  /q, /exit  退出
  /reset     重置 kvcache
  /dd        删除一轮对话
  /pp        打印历史对话
Ctrl+C: 停止当前生成
VLM enabled: after each prompt, input media path (empty = text-only). Use "video:<frames_dir>" or "video:<video_file>[:<fps>]" for video (default fps=2), "audio:<file>" for audio.
----------------------------------------
prompt >> 图片描述的内容是什么
media >> /mnt/sdcard/projects/llm/Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/01.jpg
09:46:59.858 INF EncodeForContent:2344 | Qwen-VL pixel_values[0] bytes=884736 min=6 max=219 (w=384 h=384 tp=2 ps=16 sm=2)
09:47:00.019 INF EncodeForContent:2403 | vision cache store: /mnt/sdcard/projects/llm/Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/test.jpg
09:47:00.034 INF SetKVCache:3721 | decode_grpid:0 prefill_grpid:1 history_cap:0 total_cap:128 symbolic_cap:1 precompute_len:0 input_num_token:169 prefer_symbolic_group:0
09:47:00.034 INF SetKVCache:3743 | current prefill_max_token_num:512
09:47:00.060 INF SetKVCache:3759 | first run
09:47:00.093 INF Run:4287 | input token num : 169, prefill_split_num : 2
09:47:00.093 INF Run:4367 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=128
09:47:00.244 INF Run:4367 | prefill chunk p=1 history_len=128 grpid=3 kv_cache_num=256 input_tokens=41
09:47:00.442 INF Run:4680 | ttft: 348.99 ms
这09:47:00.442 INF Run:4722 | VLM decode positions: rope_start=37 dense_kv_start=169
是一张室内空间的图片，可能是客厅或餐厅区域。房间的墙壁为暖黄色调，搭配深色木质地板和一些绿色的装饰元素。空间内有多种家具和电器，包括一个大尺寸的平板电视、一个带有透明玻璃柜的橱柜以及一个圆形的餐桌。房间内的自然光线透过窗户洒进来，营造出温馨而现代的氛围。

此外，墙上的挂画、墙上的灯具和室内的设计风格都显示出一种简洁而现代的装修风格。

09:47:09.200 NTC Run:5095 | hit eos,decode avg 11.88 token/s
09:47:09.200 INF GetKVCache:3698 | precompute_len:274, remaining:238 (tracked)
```

#### Video understand demo

##### start tokenizer server for image understand demo

```
axllm run Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/
```

##### run video understand demo
- input text  

```
描述视频的内容
```

- input video  

./video  

```
root@ax650:/mnt/sdcard/projects/llm# axllm run Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/
09:44:04.332 INF Init:2700 | LLM init start
tokenizer_type = 3
huggingface tokenizer mode = gpt2_byte_bpe
 96% | ##############################   |  30 /  31 [4.15s<4.29s, 7.23 count/s] init post axmodel ok,remain_cmm(5969 MB)
09:44:08.515 INF Init:3023 | max_token_len : 1023
09:44:08.515 INF Init:3026 | kv_cache_size : 1024, kv_cache_num: 1023
09:44:08.515 INF init_groups_from_model:1827 | prefill_token_num : 128
09:44:08.515 INF init_groups_from_model:2075 | decode grp: 0, gid: 0, max_token_len : 1023
09:44:08.515 INF init_groups_from_model:2079 | prefill grp: 0, gid: 1, history_cap: 0, total_cap: 128, symbolic_cap: 1
09:44:08.515 INF init_groups_from_model:2079 | prefill grp: 1, gid: 2, history_cap: 128, total_cap: 256, symbolic_cap: 128
09:44:08.515 INF init_groups_from_model:2079 | prefill grp: 2, gid: 3, history_cap: 256, total_cap: 384, symbolic_cap: 256
09:44:08.515 INF init_groups_from_model:2079 | prefill grp: 3, gid: 4, history_cap: 384, total_cap: 512, symbolic_cap: 384
09:44:08.515 INF init_groups_from_model:2086 | prefill_max_token_num : 512
09:44:08.515 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  31 /  31 [4.15s<4.15s, 7.47 count/s] embed_selector init ok
09:44:08.913 INF Init:1647 | Qwen token ids: vision_start=151652 image_pad=151655 video_pad=151656 audio_pad=-1
09:44:08.913 INF Init:1704 | VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32
09:44:08.913 INF Init:1710 | VisionModule deepstack enabled: layers=3
09:44:08.913 WRN Init:1713 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
09:44:08.914 INF load_config:554 | load config:
09:44:08.914 INF load_config:554 | {
09:44:08.914 INF load_config:554 |     "enable_repetition_penalty": true,
09:44:08.914 INF load_config:554 |     "enable_temperature": true,
09:44:08.914 INF load_config:554 |     "enable_top_k_sampling": false,
09:44:08.914 INF load_config:554 |     "enable_top_p_sampling": true,
09:44:08.914 INF load_config:554 |     "penalty_window": 128,
09:44:08.914 INF load_config:554 |     "repetition_penalty": 1.1,
09:44:08.914 INF load_config:554 |     "temperature": 0.7,
09:44:08.914 INF load_config:554 |     "top_k": 20,
09:44:08.914 INF load_config:554 |     "top_p": 0.8
09:44:08.914 INF load_config:554 | }
09:44:08.914 INF Init:3138 | LLM init ok
Commands:
  /q, /exit  退出
  /reset     重置 kvcache
  /dd        删除一轮对话
  /pp        打印历史对话
Ctrl+C: 停止当前生成
VLM enabled: after each prompt, input media path (empty = text-only). Use "video:<frames_dir>" or "video:<video_file>[:<fps>]" for video (default fps=2), "audio:<file>" for audio.
----------------------------------------
prompt >> 描述视频的内容
media >> /mnt/sdcard/projects/llm/Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/video/
09:44:38.482 INF EncodeForContent:2344 | Qwen-VL pixel_values[0] bytes=884736 min=0 max=255 (w=384 h=384 tp=2 ps=16 sm=2)
09:44:38.643 INF EncodeForContent:2403 | vision cache store: /mnt/sdcard/projects/llm/Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/video/frame_0000.jpg
09:44:38.674 INF EncodeForContent:2344 | Qwen-VL pixel_values[0] bytes=884736 min=0 max=255 (w=384 h=384 tp=2 ps=16 sm=2)
09:44:38.834 INF EncodeForContent:2403 | vision cache store: /mnt/sdcard/projects/llm/Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/video/frame_0008.jpg
09:44:38.864 INF EncodeForContent:2344 | Qwen-VL pixel_values[0] bytes=884736 min=0 max=255 (w=384 h=384 tp=2 ps=16 sm=2)
09:44:39.023 INF EncodeForContent:2403 | vision cache store: /mnt/sdcard/projects/llm/Qwen3-VL-2B-Instruct-gptqmodel-4bit-ax650/video/frame_0016.jpg
09:44:39.049 INF SetKVCache:3721 | decode_grpid:0 prefill_grpid:1 history_cap:0 total_cap:128 symbolic_cap:1 precompute_len:0 input_num_token:456 prefer_symbolic_group:0
09:44:39.049 INF SetKVCache:3743 | current prefill_max_token_num:512
09:44:39.075 INF SetKVCache:3759 | first run
09:44:39.076 INF Run:4287 | input token num : 456, prefill_split_num : 4
09:44:39.076 INF Run:4367 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=128
09:44:39.230 INF Run:4367 | prefill chunk p=1 history_len=128 grpid=3 kv_cache_num=256 input_tokens=128
09:44:39.401 INF Run:4367 | prefill chunk p=2 history_len=256 grpid=4 kv_cache_num=384 input_tokens=128
09:44:39.584 INF Run:4367 | prefill chunk p=3 history_len=384 grpid=4 kv_cache_num=384 input_tokens=72
09:44:39.795 INF Run:4680 | ttft: 719.15 ms
这09:44:39.795 INF Run:4722 | VLM decode positions: rope_start=324 dense_kv_start=456
是一段关于两只小松鼠的视频。

画面中，两只小松鼠正站在一块布满岩石和泥土的山坡上，它们身体前倾，相互靠近。一只小松鼠面向镜头，另一只则侧身对着镜头，它们似乎在进行一场亲密的互动或是在进行某种游戏。

背景是连绵起伏的山峦，植被茂盛，天空晴朗。整个场景呈现出一种自然和谐的氛围。

虽然图像有些模糊，但可以清晰地看到两只小松鼠的毛色——它们的毛色以灰白色为主，并带有棕色的条纹。它们的耳朵、眼睛以及尾巴都清晰可见。

整体来看，这是一段关于两只小松鼠在山林间嬉戏玩耍的视频，画面充满了生机与活力。

09:44:53.252 NTC Run:5095 | hit eos,decode avg 12.11 token/s
09:44:53.252 INF GetKVCache:3698 | precompute_len:620, remaining:-108 (tracked)
```