--- license: mit language: - en - zh base_model: - OpenMOSS-Team/MOSS-Transcribe-Diarize pipeline_tag: automatic-speech-recognition library_name: transformers tags: - MOSS-Transcribe-Diarize - Int8 - ASR --- # MOSS-Transcribe-Diarize-0.9 This version of MOSS-Transcribe-Diarize-0.9 have been converted to run on the Axera NPU using **w8a16** quantization. Compatible with Pulsar2 version: 6.0 ## Convert tools links: For those who are interested in model conversion, you can try to export axmodel through the original repo : - https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize [Pulsar2 Link, How to Convert LLM from Huggingface to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/appendix/build_llm.html) ## Support Platform - AX650 - AX650N DEMO Board - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html) - [M.2 Accelerator card](https://docs.m5stack.com/zh_CN/ai_hardware/LLM-8850_Card) **Audio Process** |Chips| input size | audio length | audio encoder | ttft(476 tokens) | w8a16 | CMM | |--|--|--|--|--|--|--| |AX650| (1,80,300) | 30s | 588.471 ms | 1930.05 ms | 6.33 tokens/sec| 3.3GiB | The DDR capacity refers to the CMM memory that needs to be consumed. Ensure that the CMM memory allocation on the development board is greater than this value. PS: The model currently supports the input of up to 6 minutes of audio at a time. ## 模型下载（Hugging Face） 先创建模型目录并进入，然后下载到该目录： ```shell mkdir -p AXERA-TECH/MOSS-Transcribe-Diarize-0.9 cd AXERA-TECH/MOSS-Transcribe-Diarize-0.9 hf download AXERA-TECH/MOSS-Transcribe-Diarize-0.9 --local-dir . # structure of the downloaded files tree -L 3 . └── AXERA-TECH └── MOSS-Transcribe-Diarize-0.9 ├── config.json ├── infer_moss_axengine.py ├── model.embed_tokens.weight.bfloat16.bin ├── model.embed_tokens.weight.float32.bin ├── model.embed_tokens.weight.npy ├── moss_openai_api.py ├── post_config.json ├── preprocessor_config.json ├── __pycache__ ├── qwen3_p256_l0_together.axmodel ... ├── qwen3_p256_l9_together.axmodel ├── qwen3_post.axmodel ├── qwen3_tokenizer.txt ├── README.md ├── requirements.txt ├── tokenizer.json ├── utils ├── wav └── whisper_encoder_vq_adaptor.axmodel 3 directories, 42 files ``` ## Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board ```bash (base) root@ax650:~/wangli/MOSS-Transcribe-Diarize-0.9B# python infer_moss_axengine.py ./wav/2speakers_example.wav [INFO] Available providers: ['AxEngineExecutionProvider', 'AXCLRTExecutionProvider'] [LLM] slice_len=256 prefill_max_len=8448 max_seq_len=10240 [Audio] loading /root/wangli/MOSS-Transcribe-Diarize-0.9B/wav/2speakers_example.wav [Audio] duration=51.663s chunks=2 valid_audio_tokens=646 features=(2, 80, 3000) preprocess=16.421s [AXEngine] loading audio model: /root/wangli/MOSS-Transcribe-Diarize-0.9B/whisper_encoder_vq_adaptor.axmodel [INFO] Using provider: AxEngineExecutionProvider [INFO] Chip type: ChipType.MC50 [INFO] VNPU type: VNPUType.DISABLED [INFO] Engine version: 2.12.0s [INFO] Model type: 2 (triple core) [INFO] Compiler version: 6.0 48520c11 [Audio AXModel] embeddings=(2, 375, 1024) cost=1.153s [Prompt] tokens=751 audio_span=665 audio_embeddings=646 build=0.009s [AXEngine] loading LLM models through InferManager: /root/wangli/MOSS-Transcribe-Diarize-0.9B Detected prefixes: ['qwen3'], chosen: qwen3, layers: 28 Init InferenceSession: 0%| | 0/28 [00:00" or "video:[:]" for video (default fps=2), "audio:" for audio. ---------------------------------------- prompt >> 请将音频转写为文本，每一段需以起始时间戳和说话人编号（[S01]、[S02]、[S03]…）开头，正文为对应的语音内容，并在段末标注结束时间戳，以清晰标明该段语音范围。 media >> audio:./wav/2speakers_example.wav 14:09:58.946 INF EncodeBlocks:403 | MossTranscribeDiarizeVL audio encoded: chunks=2 valid_audio_tokens=646 14:09:58.948 INF Prepare:2188 | MOSS audio placeholder expanded: audio_pads=646 span_tokens=665 total_prompt_tokens=751 14:09:58.948 INF SetKVCache:2064 | decode_grpid:0 prefill_grpid:1 history_cap:0 total_cap:256 symbolic_cap:1 precompute_len:0 input_num_token:751 prefer_symbolic_group:0 14:09:58.948 INF SetKVCache:2086 | current prefill_max_token_num:8448 14:10:01.017 INF SetKVCache:2102 | first run 14:10:01.063 INF Run:257 | input token num : 751, prefill_split_num : 3 14:10:01.063 INF Run:337 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=256 14:10:01.605 INF Run:337 | prefill chunk p=1 history_len=256 grpid=3 kv_cache_num=512 input_tokens=256 14:10:02.996 INF Run:337 | prefill chunk p=2 history_len=512 grpid=4 kv_cache_num=768 input_tokens=239 14:10:04.851 INF Run:664 | ttft: 3787.44 ms [0.25][S01]嗯，那么今天我们就简单的进行一下那个新生招聘的。[5.28][5.32][S01]嗯，讨论吧，因为现在不是。[8.21][8.24][S01]马上就新生到校嘛。[10.21][10.23][S01]然后我们社团呢，也需要招聘一些新的社员。[13.72][13.78][S01]然后就今天就大概就讨论一下。[16.86][16.88][S01]嗯，怎么招聘的内容吧。[19.01][19.13][S01]嗯，我们就首先想一下那个。[21.51][21.53][S01]招新的地点在哪里吧。[23.54][24.14][S02]嗯，地点的话，我们现在可以有三个选择。[27.41][27.46][S02]嗯，第一个的话，我们可以选择在。[30.16][30.20][S02]操场，因为那儿。[32.01][32.38][S02]嗯，学生流动量也挺大的。[34.81][34.74][S01]操场的话。[36.13][36.17][S01]这这段时间太热了，我怕。[38.88][38.94][S01]那个人流量有点少。[40.88][40.91][S02]嗯，那我们还可以有第二个选择呀。[43.32][43.51][S02]嗯，我们可以在图书馆楼下。[46.38][46.41][S02]那里有一块可以遮阴的地方。[48.58][48.75][S01]哦，图书馆我觉得应该还可以吧。[51.66] 14:11:18.895 NTC Run:1090 | decode stop: reason=eos prompt_tokens=751 generated_tokens=473 output_max_token=-1 max_token_len=10240 dense_decode_start=751 decode_start=751 context_remaining=9016 avg 6.37 token/s 14:11:18.895 INF GetKVCache:2041 | precompute_len:1224, remaining:7224 (tracked) prompt >> /q 14:11:27.121 INF CheckCmmBalance:79 | [cmm-sentry] Deinit: CMM balanced (baseline 9963 MB, now 9963 MB) ``` ### Opneai api serve by axllm ```bash (base) root@ax650:~/wangli# axllm serve MOSS-Transcribe-Diarize-0.9B/ 14:13:00.114 INF run_server_mode:2191 | server request/queue timeout: 1800000 ms (config.server_timeout_ms, overridable via --server_timeout_ms) 14:13:00.116 INF Init:468 | LLM init start tokenizer_type = 3 huggingface tokenizer mode = gpt2_byte_bpe 96% | ############################## | 30 / 31 [5.25s<5.42s, 5.72 count/s] init post axmodel ok,remain_cmm(7073 MB) 14:13:05.369 INF Init:791 | max_token_len : 10240 14:13:05.369 INF Init:794 | kv_cache_size : 1024, kv_cache_num: 10240 14:13:05.369 INF init_groups_from_model:150 | prefill_token_num : 256 14:13:05.369 INF init_groups_from_model:398 | decode grp: 0, gid: 0, max_token_len : 10240 14:13:05.369 INF init_groups_from_model:402 | prefill grp: 0, gid: 1, history_cap: 0, total_cap: 256, symbolic_cap: 1 ... 14:13:05.370 INF init_groups_from_model:402 | prefill grp: 32, gid: 33, history_cap: 8192, total_cap: 8448, symbolic_cap: 8192 14:13:05.370 INF init_groups_from_model:409 | prefill_max_token_num : 8448 14:13:05.370 INF Init:27 | LLaMaEmbedSelector use mmap 100% | ################################ | 31 / 31 [5.25s<5.25s, 5.91 count/s] embed_selector init ok 14:13:05.654 INF init_whisper_audio_profile:161 | Whisper audio profile init ok: path=MOSS-Transcribe-Diarize-0.9B/whisper_encoder_vq_adaptor.axmodel duration=30.0s mel_frames=3000 tokens=375 out_dtype=fp32 14:13:05.654 INF Init:1077 | VisionModule init ok: type=MossTranscribeDiarizeVL, audio_pad=151671, audio_tokens_per_second=12.5000, time_marker_every_seconds=5, time_markers=1 14:13:05.654 INF load_config:554 | load config: 14:13:05.654 INF load_config:554 | { 14:13:05.654 INF load_config:554 | "enable_repetition_penalty": false, 14:13:05.654 INF load_config:554 | "enable_temperature": false, 14:13:05.654 INF load_config:554 | "enable_top_k_sampling": false, 14:13:05.654 INF load_config:554 | "enable_top_p_sampling": false, 14:13:05.654 INF load_config:554 | "penalty_window": 20, 14:13:05.654 INF load_config:554 | "repetition_penalty": 1.2, 14:13:05.654 INF load_config:554 | "temperature": 0.9, 14:13:05.654 INF load_config:554 | "top_k": 10, 14:13:05.654 INF load_config:554 | "top_p": 0.8 14:13:05.654 INF load_config:554 | } 14:13:05.654 INF Init:909 | LLM init ok Starting server on port 8000 with model 'MOSS-Transcribe-Diarize-0.9B'... API URLs: GET http://127.0.0.1:8000/health GET http://127.0.0.1:8000/v1/models POST http://127.0.0.1:8000/v1/chat/completions GET http://10.126.29.50:8000/health GET http://10.126.29.50:8000/v1/models POST http://10.126.29.50:8000/v1/chat/completions GET http://10.126.29.227:8000/health GET http://10.126.29.227:8000/v1/models POST http://10.126.29.227:8000/v1/chat/completions GET http://172.17.0.1:8000/health GET http://172.17.0.1:8000/v1/models POST http://172.17.0.1:8000/v1/chat/completions Aliases: GET http://127.0.0.1:8000/models POST http://127.0.0.1:8000/chat/completions GET http://10.126.29.50:8000/models POST http://10.126.29.50:8000/chat/completions GET http://10.126.29.227:8000/models POST http://10.126.29.227:8000/chat/completions GET http://172.17.0.1:8000/models POST http://172.17.0.1:8000/chat/completions OpenAI API Server starting on http://0.0.0.0:8000 Max concurrency: 1 Models: MOSS-Transcribe-Diarize-0.9B ``` #### Use axllm serve ```bash python moss_openai_api.py ../wav/2speakers_example.wav --endpoint chat --prompt "请将音频转写为文本，每一段需以起始时间戳和说话人编号（[S01]、[S02]、[S03]…）开头，正文为对应的语音内容，并在段末标注结束时间戳，以清晰标明该段语音范围。" [0.25][S01]嗯，那么今天我们就简单的进行一下那个新生招聘的。[5.28][5.32][S01]嗯，讨论吧，因为现在不是。[8.21][8.24][S01]马上就新生到校嘛。[10.21][10.23][S01]然后我们社团呢，也需要招聘一些新的社员。[13.72][13.78][S01]然后就今天就大概就讨论一下。[16.86][16.88][S01]嗯，怎么招聘的内容吧。[19.01][19.13][S01]嗯，我们就首先想一下那个。[21.51][21.53][S01]招新的地点在哪里吧。[23.54][24.14][S02]嗯，地点的话，我们现在可以有三个选择。[27.41][27.46][S02]嗯，第一个的话，我们可以选择在。[30.16][30.20][S02]操场，因为那儿。[32.01][32.38][S02]嗯，学生流动量也挺大的。[34.81][34.74][S01]操场的话。[36.13][36.17][S01]这这段时间太热了，我怕。[38.88][38.94][S01]那个人流量有点少。[40.88][40.91][S02]嗯，那我们还可以有第二个选择呀。[43.32][43.51][S02]嗯，我们可以在图书馆楼下。[46.38][46.41][S02]那里有一块可以遮阴的地方。[48.58][48.75][S01]哦，图书馆我觉得应该还可以吧。[51.66] ```