# T5 inference performance

**URL:** <https://discuss.huggingface.co/t/t5-inference-performance/15494>\
**Category:** Models\
**Created:** [March 8, 2022, 9:53am UTC](https://discuss.huggingface.co/t/t5-inference-performance/15494 "2022-03-08T09:53:41Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![GenV](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/genv/32/12132_2.png) [@GenV](https://discuss.huggingface.co/u/GenV)\
**Post date:** [March 8, 2022, 9:53am UTC](https://discuss.huggingface.co/t/t5-inference-performance/15494/1 "2022-03-08T09:53:41Z")

</div>

Hello,  
I would like to improve the inference time of a finetuned T5-base (translation task). I am currently using the `.bin` file (`from_pretrained`) and a GPU. I have tried several approaches, such as ONNX and TensorRT. Having a `max_length=1024`, these approaches perform worse (and sometimes by a lot). Are there any techniques that could help? Thank you.

---

<div class="post-metadata">

**Author:** ![YannAgora](https://avatars.discourse-cdn.com/v4/letter/y/c0e974/32.png) [@YannAgora](https://discuss.huggingface.co/u/YannAgora)\
**Post date:** [March 8, 2022, 1:31pm UTC](https://discuss.huggingface.co/t/t5-inference-performance/15494/2 "2022-03-08T13:31:02Z")

</div>

Hey 👋,

Did you try quantization ?

There is an example for pegasus model [here](https://discuss.huggingface.co/t/pegasus-model-weights-compression-pruning/6381/9). I tried and it performed pretty well for summarization with an inference time decrease by 2x or 3x

---

<div class="post-metadata">

**Author:** ![GenV](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/genv/32/12132_2.png) [@GenV](https://discuss.huggingface.co/u/GenV)\
**Post date:** [March 8, 2022, 1:53pm UTC](https://discuss.huggingface.co/t/t5-inference-performance/15494/3 "2022-03-08T13:53:43Z")

</div>

Thanks @YannAgora. Can run with 2x or 3x on GPU?

---

<div class="post-metadata">

**Author:** ![YannAgora](https://avatars.discourse-cdn.com/v4/letter/y/c0e974/32.png) [@YannAgora](https://discuss.huggingface.co/u/YannAgora)\
**Post date:** [March 8, 2022, 2:05pm UTC](https://discuss.huggingface.co/t/t5-inference-performance/15494/4 "2022-03-08T14:05:35Z")

</div>

I haven’t tried on a GPU instance but I don’t see why it wouldn’t work.

---

<div class="post-metadata">

**Author:** ![GenV](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/genv/32/12132_2.png) [@GenV](https://discuss.huggingface.co/u/GenV)\
**Post date:** [March 8, 2022, 2:28pm UTC](https://discuss.huggingface.co/t/t5-inference-performance/15494/5 "2022-03-08T14:28:19Z")

</div>

@YannAgora I got this error using the code, adding `reconstructed_quantized_model.to("cuda")` for GPU inference:

`NotImplementedError: Could not run 'quantized::linear_dynamic' with arguments from the 'CUDA' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'quantized::linear_dynamic' is only available for these backends: [CPU, BackendSelect, Python, Named, Conjugate, Negative, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradLazy, AutogradXPU, AutogradMLC, Tracer, UNKNOWN_TENSOR_TYPE_ID, Autocast, Batched, VmapMode].`

I saw that GPU is not supported [here](https://discuss.pytorch.org/t/does-dynamic-quantization-support-gpu/119231).

---

<div class="post-metadata">

**Author:** ![YannAgora](https://avatars.discourse-cdn.com/v4/letter/y/c0e974/32.png) [@YannAgora](https://discuss.huggingface.co/u/YannAgora)\
**Post date:** [March 8, 2022, 2:41pm UTC](https://discuss.huggingface.co/t/t5-inference-performance/15494/6 "2022-03-08T14:41:43Z")

</div>

Oh ok I didn’t know that 😅