Hello, I just started to work with transformers and transformer pre trained models. And I’m facing a strange behaviour with t5-base pre trained model. I want to translate a text from english to french and everything works fine until my truncation issue. Here is my current code sample : def tran…

Hey! Did you find a solution to this? Pls let me know, I’m facing the exact same problem!

I think you need skip_special_tokens=True tokenizer.decode(output, skip_special_tokens=True)

T5 pretrained model truncate translation "large" text

sheoran95 March 16, 2023, 3:24am 3

Topic		Replies	Views
T5 Generates very short summaries 🤗Transformers	22	5797	September 11, 2020
Output truncation of summaries models 🤗Transformers	0	474	March 30, 2023
T5 for Q&A produces truncated sentence Beginners	0	308	December 6, 2022
T5 tokenizer's post-processor is suboptimal for truncated sequences for seq2seq finetuning 🤗Transformers	0	354	July 5, 2023
HuggingFace - Why does the T5 model shorten sentences? Models	2	784	April 28, 2024