Instructions to use nxp/keyword-spotting-dscnn-imx with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT
How to use nxp/keyword-spotting-dscnn-imx with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Keyword Spotting - DS-CNN
Overview
Keyword spotting is a speech-recognition task that detects when one of a small set of command words is spoken. The source model takes a 1-second audio clip, converts it to an MFCC feature fingerprint, and classifies it into one of 12 classes (10 command words plus "Silence" and "Unknown"). It is part of the MLPerf Tiny benchmark suite, built around a depth-wise separable convolutional neural network (DS-CNN) architecture, and is published by MLCommons as a pre-quantized int8 TFLite model.
This repository contains that int8 model compiled for NXP i.MX 8M Plus, i.MX 93, i.MX 95, and i.MX 952
processors, along with the scripts used to regenerate these assets (recipe.sh) and an inference example
(example.py).
Key Features
- Depth-wise separable convolutional neural network (DS-CNN) for keyword spotting.
- Operates on MFCC feature fingerprints extracted from 1-second audio clips.
- Recognizes 12 classes (10 command words plus "Silence" and "Unknown").
- ~23,756 parameters, ~5 MOPS.
- int8 quantized (int8 input / int8 output), suitable for accelerated NPU inference on NXP i.MX platforms.
Model Description
This is a compiled version of the MLPerf Tiny keyword-spotting DS-CNN model for NXP i.MX 8M Plus, i.MX 93, i.MX 95, and i.MX 952 processors.
- Base Model: MLPerf Tiny keyword spotting
- Original Model Authors: MLCommons
- Original License: Apache-2.0
- Modified by: NXP
Modifications
This model is a derivative work with the following changes from the original:
- Quantization: none performed locally - the int8 TFLite model is downloaded directly from MLCommons, which already publishes it pre-quantized (int8 input / int8 output).
- Compilation (i.MX 93): compiled for the Arm Ethos-U65 NPU with
Vela, tags
lf-6.18.20_2.0.0andlf-6.18.37_2.1.0. - Compilation (i.MX 95 / i.MX 952): converted for the eIQ Neutron NPU with the eIQ Neutron SDK,
versions
3.1.3,3.2.1and3.2.3.
Original model available at: https://github.com/mlcommons/tiny/tree/master/benchmark/training/keyword_spotting.
Model Information
| Information | Value |
|---|---|
| Input shape | MFCC feature fingerprint (1, 49, 10, 1), int8 |
| Input example | example_input.wav (sound file from the Speech Commands dataset, CC BY 4.0) |
| Output shape | Vector of class scores (1, 12), int8 quantized; dequantize with the output tensor's scale / zero point, see example.py |
| Output example | Recognized word printed to stdout, e.g. Recognized word: Yes |
| FLOPS | 5 MOPS |
| Number of parameters | 23,756 |
| File size (int8) | 53 KB (assets/quantized_tflite_model/kws_ref_model.tflite) |
| Source framework | TensorFlow / Keras |
Labels
The model recognizes the following 12 classes:
Down, Go, Left, No, Off, On, Right, Stop, Up, Yes, Silence, Unknown
Supported Hardware and Software Configurations
All models run with the eIQ LiteRT (TensorFlow Lite) runtime and the corresponding NPU delegate (VX for i.MX 8M Plus, Ethos-U for i.MX 93, Neutron for i.MX 95 / i.MX 952). The same information is captured in release_manifest.json.
| Asset | Runtime | Precision | HW target | BSP & SDK versions | Link to asset |
|---|---|---|---|---|---|
| i.MX 8M Plus Quantized | LiteRT | INT8 | i.MX 8M Plus | - | Download |
| i.MX 93 Precompiled | LiteRT | INT8 | i.MX 93 | BSP lf-6.18.20_2.0.0, VELA_TAG = lf-6.18.20_2.0.0 | Download |
| i.MX 93 Precompiled | LiteRT | INT8 | i.MX 93 | BSP lf-6.18.37_2.1.0, VELA_TAG = lf-6.18.37_2.1.0 | Download |
| i.MX 95 Precompiled | LiteRT | INT8 | i.MX 95 | BSP lf-6.18.20_2.0.0, Neutron SDK v3.1.3 | Download |
| i.MX 95 Precompiled | LiteRT | INT8 | i.MX 95 | BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.1 | Download |
| i.MX 95 Precompiled | LiteRT | INT8 | i.MX 95 | BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.3 | Download |
| i.MX 952 Precompiled | LiteRT | INT8 | i.MX 952 | BSP lf-6.18.20_2.0.0, Neutron SDK v3.1.3 | Download |
| i.MX 952 Precompiled | LiteRT | INT8 | i.MX 952 | BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.1 | Download |
| i.MX 952 Precompiled | LiteRT | INT8 | i.MX 952 | BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.3 | Download |
The hardware-independent int8 model is also provided for host (x86) inference and as the input to the
compilers: assets/quantized_tflite_model/kws_ref_model.tflite (byte-identical to
assets/imx8mplus/kws_ref_model.tflite).
Limitations and Biases
This model inherits all limitations from the original MLPerf Tiny keyword-spotting model. Additional limitations:
- Task scope: Limited-vocabulary keyword spotting only - 10 command words plus "Silence" and "Unknown". It is not a general-purpose speech-recognition model.
- Input constraints: expects a single 1-second audio clip sampled consistently with the Speech Commands dataset; performance on noisier or differently-recorded audio is untested.
- Dataset bias: trained on the Speech Commands dataset. Detection quality may vary for accents, speaking rates, or recording conditions under-represented in that dataset.
Training
The model was trained by MLCommons on the Speech Commands dataset, an audio dataset of spoken words released under a Creative Commons BY 4.0 license. Training scripts are available on the MLCommons Tiny GitHub page. No local retraining is performed by NXP.
Download and Run
The recipe installs the eIQ Neutron SDK from the NXP eIQ PyPI Distribution Portal. Configure pip once to use it:
python -m pip config --user set global.index https://eiq.nxp.com/repository
python -m pip config --user set global.index-url https://eiq.nxp.com/repository
python -m pip config --user set global.trusted-host eiq.nxp.com
python -m pip config --user set global.extra-index-url https://pypi.org/simple
python -m pip config --user set global.find-links https://eiq.nxp.com/repository/
To download the quantized int8 model and compile the platform-specific variants, run:
bash recipe.sh
The script takes two optional positional arguments, the Vela tag and the eIQ Neutron SDK version
(defaulting to lf-6.18.20_2.0.0 and 3.2.3 respectively):
bash recipe.sh [IMX_ETHOS_U_VELA_TAG] [EIQ_NEUTRON_SDK_VERSION]
bash recipe.sh lf-6.18.37_2.1.0 3.2.3
The generated files are:
assets/quantized_tflite_model/kws_ref_model.tflite- the int8 model downloaded as-is from MLCommonsassets/imx93/<IMX_ETHOS_U_VELA_TAG>/kws_ref_model_vela.tflite- Vela-compiled for i.MX 93assets/imx95/<EIQ_NEUTRON_SDK_VERSION>/kws_ref_model_converted.tflite- Neutron-converted for i.MX 95assets/imx952/<EIQ_NEUTRON_SDK_VERSION>/kws_ref_model_converted.tflite- Neutron-converted for i.MX 952
Running the example
Install the dependencies from requirements.txt, then run example.py on a host PC. It loads a .wav
file, extracts an MFCC feature fingerprint, runs the DS-CNN, and prints the recognized word:
# Run on PC:
python example.py -m assets/quantized_tflite_model/kws_ref_model.tflite -i example_input.wav
example.py accepts the following arguments, all optional:
-m,--model: path to the TFLite model file (defaultassets/quantized_tflite_model/kws_ref_model.tflite)-i,--input: path to the input.wavfile (defaultexample_input.wav)-o,--output: optional path to write the recognized word as text (default: not written)
Please refer to the i.MX Machine Learning User Guide for details on how to run the models on each i.MX target with the NPU delegates.
License
This model is released under the Apache License 2.0 (see LICENSE), the same license as the original MLPerf Tiny keyword spotting model (see MODEL_LICENSE).
Citation
If you use this model, please cite both this work and the original model:
- Warden, Pete. "Speech commands: A dataset for limited-vocabulary speech recognition." arXiv preprint arXiv:1804.03209 (2018).
- Banbury, Colby, et al. "MLPerf Tiny Benchmark." arXiv preprint arXiv:2106.07597 (2021).
- Zhang, Yundong, et al. "Hello edge: Keyword spotting on microcontrollers." arXiv preprint arXiv:1711.07128 (2017).
@article{warden2018speech,
title={Speech commands: A dataset for limited-vocabulary speech recognition},
author={Warden, Pete},
journal={arXiv preprint arXiv:1804.03209},
year={2018}
}
@article{banbury2021mlperf,
title={Mlperf tiny benchmark},
author={Banbury, Colby and Reddi, Vijay Janapa and Torelli, Peter and Holleman, Jeremy and Jeffries, Nat and Kiraly, Csaba and Montino, Pietro and Kanter, David and Ahmed, Sebastian and Pau, Danilo and others},
journal={arXiv preprint arXiv:2106.07597},
year={2021}
}
@article{zhang2017hello,
title={Hello edge: Keyword spotting on microcontrollers},
author={Zhang, Yundong and Suda, Naveen and Lai, Liangzhen and Chandra, Vikas},
journal={arXiv preprint arXiv:1711.07128},
year={2017}
}
- Downloads last month
- 91