Keyword Spotting - DS-CNN

Overview

Keyword spotting is a speech-recognition task that detects when one of a small set of command words is spoken. The source model takes a 1-second audio clip, converts it to an MFCC feature fingerprint, and classifies it into one of 12 classes (10 command words plus "Silence" and "Unknown"). It is part of the MLPerf Tiny benchmark suite, built around a depth-wise separable convolutional neural network (DS-CNN) architecture, and is published by MLCommons as a pre-quantized int8 TFLite model.

This repository contains that int8 model compiled for NXP i.MX 8M Plus, i.MX 93, i.MX 95, and i.MX 952 processors, along with the scripts used to regenerate these assets (recipe.sh) and an inference example (example.py).

Key Features

  • Depth-wise separable convolutional neural network (DS-CNN) for keyword spotting.
  • Operates on MFCC feature fingerprints extracted from 1-second audio clips.
  • Recognizes 12 classes (10 command words plus "Silence" and "Unknown").
  • ~23,756 parameters, ~5 MOPS.
  • int8 quantized (int8 input / int8 output), suitable for accelerated NPU inference on NXP i.MX platforms.

Model Description

This is a compiled version of the MLPerf Tiny keyword-spotting DS-CNN model for NXP i.MX 8M Plus, i.MX 93, i.MX 95, and i.MX 952 processors.

Modifications

This model is a derivative work with the following changes from the original:

  • Quantization: none performed locally - the int8 TFLite model is downloaded directly from MLCommons, which already publishes it pre-quantized (int8 input / int8 output).
  • Compilation (i.MX 93): compiled for the Arm Ethos-U65 NPU with Vela, tags lf-6.18.20_2.0.0 and lf-6.18.37_2.1.0.
  • Compilation (i.MX 95 / i.MX 952): converted for the eIQ Neutron NPU with the eIQ Neutron SDK, versions 3.1.3, 3.2.1 and 3.2.3.

Original model available at: https://github.com/mlcommons/tiny/tree/master/benchmark/training/keyword_spotting.

Model Information

Information Value
Input shape MFCC feature fingerprint (1, 49, 10, 1), int8
Input example example_input.wav (sound file from the Speech Commands dataset, CC BY 4.0)
Output shape Vector of class scores (1, 12), int8 quantized; dequantize with the output tensor's scale / zero point, see example.py
Output example Recognized word printed to stdout, e.g. Recognized word: Yes
FLOPS 5 MOPS
Number of parameters 23,756
File size (int8) 53 KB (assets/quantized_tflite_model/kws_ref_model.tflite)
Source framework TensorFlow / Keras

Labels

The model recognizes the following 12 classes:

Down, Go, Left, No, Off, On, Right, Stop, Up, Yes, Silence, Unknown

Supported Hardware and Software Configurations

All models run with the eIQ LiteRT (TensorFlow Lite) runtime and the corresponding NPU delegate (VX for i.MX 8M Plus, Ethos-U for i.MX 93, Neutron for i.MX 95 / i.MX 952). The same information is captured in release_manifest.json.

Asset Runtime Precision HW target BSP & SDK versions Link to asset
i.MX 8M Plus Quantized LiteRT INT8 i.MX 8M Plus - Download
i.MX 93 Precompiled LiteRT INT8 i.MX 93 BSP lf-6.18.20_2.0.0, VELA_TAG = lf-6.18.20_2.0.0 Download
i.MX 93 Precompiled LiteRT INT8 i.MX 93 BSP lf-6.18.37_2.1.0, VELA_TAG = lf-6.18.37_2.1.0 Download
i.MX 95 Precompiled LiteRT INT8 i.MX 95 BSP lf-6.18.20_2.0.0, Neutron SDK v3.1.3 Download
i.MX 95 Precompiled LiteRT INT8 i.MX 95 BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.1 Download
i.MX 95 Precompiled LiteRT INT8 i.MX 95 BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.3 Download
i.MX 952 Precompiled LiteRT INT8 i.MX 952 BSP lf-6.18.20_2.0.0, Neutron SDK v3.1.3 Download
i.MX 952 Precompiled LiteRT INT8 i.MX 952 BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.1 Download
i.MX 952 Precompiled LiteRT INT8 i.MX 952 BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.3 Download

The hardware-independent int8 model is also provided for host (x86) inference and as the input to the compilers: assets/quantized_tflite_model/kws_ref_model.tflite (byte-identical to assets/imx8mplus/kws_ref_model.tflite).

Limitations and Biases

This model inherits all limitations from the original MLPerf Tiny keyword-spotting model. Additional limitations:

  • Task scope: Limited-vocabulary keyword spotting only - 10 command words plus "Silence" and "Unknown". It is not a general-purpose speech-recognition model.
  • Input constraints: expects a single 1-second audio clip sampled consistently with the Speech Commands dataset; performance on noisier or differently-recorded audio is untested.
  • Dataset bias: trained on the Speech Commands dataset. Detection quality may vary for accents, speaking rates, or recording conditions under-represented in that dataset.

Training

The model was trained by MLCommons on the Speech Commands dataset, an audio dataset of spoken words released under a Creative Commons BY 4.0 license. Training scripts are available on the MLCommons Tiny GitHub page. No local retraining is performed by NXP.

Download and Run

The recipe installs the eIQ Neutron SDK from the NXP eIQ PyPI Distribution Portal. Configure pip once to use it:

python -m pip config --user set global.index https://eiq.nxp.com/repository
python -m pip config --user set global.index-url https://eiq.nxp.com/repository
python -m pip config --user set global.trusted-host eiq.nxp.com
python -m pip config --user set global.extra-index-url https://pypi.org/simple
python -m pip config --user set global.find-links https://eiq.nxp.com/repository/

To download the quantized int8 model and compile the platform-specific variants, run:

bash recipe.sh

The script takes two optional positional arguments, the Vela tag and the eIQ Neutron SDK version (defaulting to lf-6.18.20_2.0.0 and 3.2.3 respectively):

bash recipe.sh [IMX_ETHOS_U_VELA_TAG] [EIQ_NEUTRON_SDK_VERSION]
bash recipe.sh lf-6.18.37_2.1.0 3.2.3

The generated files are:

  • assets/quantized_tflite_model/kws_ref_model.tflite - the int8 model downloaded as-is from MLCommons
  • assets/imx93/<IMX_ETHOS_U_VELA_TAG>/kws_ref_model_vela.tflite - Vela-compiled for i.MX 93
  • assets/imx95/<EIQ_NEUTRON_SDK_VERSION>/kws_ref_model_converted.tflite - Neutron-converted for i.MX 95
  • assets/imx952/<EIQ_NEUTRON_SDK_VERSION>/kws_ref_model_converted.tflite - Neutron-converted for i.MX 952

Running the example

Install the dependencies from requirements.txt, then run example.py on a host PC. It loads a .wav file, extracts an MFCC feature fingerprint, runs the DS-CNN, and prints the recognized word:

# Run on PC:
python example.py -m assets/quantized_tflite_model/kws_ref_model.tflite -i example_input.wav

example.py accepts the following arguments, all optional:

  • -m, --model: path to the TFLite model file (default assets/quantized_tflite_model/kws_ref_model.tflite)
  • -i, --input: path to the input .wav file (default example_input.wav)
  • -o, --output: optional path to write the recognized word as text (default: not written)

Please refer to the i.MX Machine Learning User Guide for details on how to run the models on each i.MX target with the NPU delegates.

License

This model is released under the Apache License 2.0 (see LICENSE), the same license as the original MLPerf Tiny keyword spotting model (see MODEL_LICENSE).

Citation

If you use this model, please cite both this work and the original model:

  • Warden, Pete. "Speech commands: A dataset for limited-vocabulary speech recognition." arXiv preprint arXiv:1804.03209 (2018).
  • Banbury, Colby, et al. "MLPerf Tiny Benchmark." arXiv preprint arXiv:2106.07597 (2021).
  • Zhang, Yundong, et al. "Hello edge: Keyword spotting on microcontrollers." arXiv preprint arXiv:1711.07128 (2017).
@article{warden2018speech,
  title={Speech commands: A dataset for limited-vocabulary speech recognition},
  author={Warden, Pete},
  journal={arXiv preprint arXiv:1804.03209},
  year={2018}
}

@article{banbury2021mlperf,
  title={Mlperf tiny benchmark},
  author={Banbury, Colby and Reddi, Vijay Janapa and Torelli, Peter and Holleman, Jeremy and Jeffries, Nat and Kiraly, Csaba and Montino, Pietro and Kanter, David and Ahmed, Sebastian and Pau, Danilo and others},
  journal={arXiv preprint arXiv:2106.07597},
  year={2021}
}

@article{zhang2017hello,
  title={Hello edge: Keyword spotting on microcontrollers},
  author={Zhang, Yundong and Suda, Naveen and Lai, Liangzhen and Chandra, Vikas},
  journal={arXiv preprint arXiv:1711.07128},
  year={2017}
}
Downloads last month
91
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collections including nxp/keyword-spotting-dscnn-imx

Papers for nxp/keyword-spotting-dscnn-imx