seq2seq decoding is inherently slow and using onnx is one obvious solution to speed it up. The onnxt5 package already provides one way to use onnx for t5. But if we export the complete T5 model to onnx, then we can’t use the past_key_values for decoding since for the first decoding step past_key_…

Here’s a self-contained notebook and small benchmark for summarization and translation tasks. [summ_benchmark] [translation_benchmark]

A question on twitter can’t we import T5Model from transformer and then may be pass it through a wrapper or decorator for onnx optimization. Or do we really need to change some ops in torch so that they can be onnx compatible? We could export the T5 as it is but then we can’t use past_key_values …

Thanks @valhalla . The reason of the doubt was, we have to make changes to some parts of the code to make it work as it is. And this sane issue is there in tensorflow too. Very nice post and super useful. Thanks for sharing:-) :+1:

What is causing the Onnx T5 model to be faster than the regular PyTorch T5 model?

Because of onnxruntime , which does several graph optimizations to speed-up inference. You could find more info in onnx runtime docs

you can quantize the model. As mentioned in the PyTorch doc PyTorch supports INT8 quantization compared to typical FP32 models allowing for a 4x reduction in the model size and a 4x reduction in memory bandwidth requirements. Hardware support for INT8 computations is typically 2 to 4 times faster …

hey @valhalla , does this onnx runtime solution work with mT5 also?

might not work as it is, but should be easy to adapt. I think replacing T5 related classes with MT5 should make it work, for example, T5ForConditionalGeneration with MT5ForConditionalGeneration etc

Hey @valhalla Hope you’re well. Firstly, I want to thank you for sharing this. I was trying to run the notebook you had shared to recreate the results you reported but kept running into an error. Would you be able to help me fix it? The only change to the notebook I had made was installed the sent…

Hi @rigoh5 , what transformer version are you using ? could you try with 3.1.0, I haven’t tested it with the latest version.

Speeding up T5 inference 🚀

🤗Transformers

kira March 15, 2021, 9:29am 14

thank you! @valhalla. created a new thread here.

Topic		Replies	Views
Boost inference speed of T5 models up to 5X & reduce the model size by 3X 🤗Transformers	2	5747	June 8, 2023
How to convert mT5 and ByT5 to ONNX format? 🤗Transformers	4	2202	December 22, 2021
Optimum & T5 for inference 🤗Optimum	18	6031	February 8, 2023
How can I use the ONNX model? 🤗Transformers	2	2174	January 29, 2024
Improving decoding speed by onnx conversion model Beginners	0	264	November 17, 2021

Speeding up T5 inference 🚀

Related topics