# Static quantization of gpt2-style models with ORTQuantizer

**URL:** <https://discuss.huggingface.co/t/static-quantization-of-gpt2-style-models-with-ortquantizer/50009>\
**Category:** 🤗Optimum\
**Created:** [August 8, 2023, 6:45pm UTC](https://discuss.huggingface.co/t/static-quantization-of-gpt2-style-models-with-ortquantizer/50009 "2023-08-08T18:45:01Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Imnimo](https://avatars.discourse-cdn.com/v4/letter/i/ecb155/32.png) [@Imnimo](https://discuss.huggingface.co/u/Imnimo)\
**Post date:** [August 8, 2023, 6:45pm UTC](https://discuss.huggingface.co/t/static-quantization-of-gpt2-style-models-with-ortquantizer/50009/1 "2023-08-08T18:45:02Z")

</div>

I have a gpt-2 style model that I’m trying to quantize using the Onnx quantizer. When exported to ONNX (using export=True argument to ORTModelForCausalLM.from\_pretrained) it has multiple model files, decoder\_model.onnx and decoder\_with\_past\_model.onnx. My understanding is that decoder\_with\_past-model is used for inference with cached keys and values, which is important for my application.

Trying to set up a quantizer for the entire export directory like this fails:

```auto
quantizer = ORTQuantizer.from_pretrained(save_directory)

```

```auto
RuntimeError: Found too many ONNX model files in 20M-onnx-static. ORTQuantizer does not support multi-file quantization. Please create separate ORTQuantizer instances for each model/file, by passing the argument `file_name` to ORTQuantizer.from_pretrained().

```

My understanding is that I can build a quantizer for each constituent model like this:

```auto
quantizer = ORTQuantizer.from_pretrained(save_directory,file_name="decoder_model.onnx")

```

I’m able to get this process to work for the regular decoder\_model - I just need to set up a dataset that produces batches of input\_ids and attention\_mask values, matching the input format of the decoder model, and run calibration. That part works great.

What I don’t understand is how to do the same process for the decoder\_with\_past\_model. Simply following the same procedure as the decoder\_model case produces this error:

```auto
ValueError: Model requires 14 inputs. Input Feed contains 2

```

Presumably, these 14 inputs include the cached keys and values? How can I figure out what the expected names and values of these extra inputs are? Is there a way to do this with the standard ORTQuantizer pipeline, or do I need to manually construct batches of cached activations?

---

<div class="post-metadata">

**Author:** ![regisss](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/regisss/32/22467_2.png) [@regisss](https://discuss.huggingface.co/u/regisss)\
**Post date:** [August 9, 2023, 5:56pm UTC](https://discuss.huggingface.co/t/static-quantization-of-gpt2-style-models-with-ortquantizer/50009/2 "2023-08-09T17:56:45Z")

</div>

Hi @Imnimo, for now quantizing decoder models with past key values is not trivial. It will be much easier soon when we have a single exported file.

In the meantime, 2 solutions:

1. Use dynamic quantization adding `is_static=False` in your quantization config.
2. If you want to perform static quantization as you tried to do, you can see [here](https://github.com/huggingface/optimum/blob/6162d5f7addac5f48e5db34b7737dd7950f0584d/optimum/exporters/onnx/base.py#L625) how the input names are defined and [there](https://github.com/huggingface/optimum/blob/6162d5f7addac5f48e5db34b7737dd7950f0584d/optimum/exporters/onnx/base.py#L562) how to generate dummy inputs. But it is not straightforward ☹

Hope that helps!

---

<div class="post-metadata">

**Author:** ![Imnimo](https://avatars.discourse-cdn.com/v4/letter/i/ecb155/32.png) [@Imnimo](https://discuss.huggingface.co/u/Imnimo)\
**Post date:** [August 9, 2023, 6:21pm UTC](https://discuss.huggingface.co/t/static-quantization-of-gpt2-style-models-with-ortquantizer/50009/3 "2023-08-09T18:21:59Z")

</div>

Thanks for the reply!

I had already gotten dynamic quantization working (very smooth, worked great!). It might turn out that the gains from static quantization are not worth the hassle at the moment, but I’ll poke around and see if I can get something working without too much trouble.

---

<div class="post-metadata">

**Author:** ![IdoAmit198](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/idoamit198/32/12110_2.png) [@IdoAmit198](https://discuss.huggingface.co/u/IdoAmit198)\
**Post date:** [September 18, 2023, 1:00pm UTC](https://discuss.huggingface.co/t/static-quantization-of-gpt2-style-models-with-ortquantizer/50009/4 "2023-09-18T13:00:59Z")

</div>

Hi, did you manage to get the static quantizaiton to work?  
I’m working on something similar right now too and having some issues with the static quantization (dynamic was easy indeed)
