I am using finetune.py script among the seq2seq examples to finetune for a QA task: export NQOPEN_DIR=/home/danielk/nqopen_csv export OUT=/home/danielk/fine_tune_t5_small python3 finetune.py \ --data_dir $NQOPEN_DIR \ --model_name_or_path t5-small --tokenizer_name t5-small \ --learning_rate=3e-4 …

I suspect that we have a bug here: https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune.py#L104-L106 We set decoder_start_token_id when the model is BART, but nothing is set for a T5 model. Thoughts @sshleifer @valhalla ?

Don’t think decoder_start_token_id is the issue, it will be set from config. Your command also looks reasonable. Debugging ideas: Try 1 GPU looked at the saved text_batch.json to make sure it makes sense. make sure command is same as it was for BART remove n_train=-1

@sshleifer I tried dropping n_train=-1 and used 1 GPU: python3 finetune.py \ --data_dir $NQOPEN_DIR \ --model_name_or_path t5-small --tokenizer_name t5-small \ --learning_rate=1e-4 \ --do_train \ --train_batch_size 128 \ --do_predict \ --max_source_length=12 \ --max_target_length=6 \ --eval_max_g…

For comparison, here is the output from text_batch.json file for BART (which seems to work well). { "input_ids": [ "<s> who sings does he love me with reba?</s><pad><pad><pad><pad><pad><pad><pad><pad>", "<s> what is the smallest prime number that is greater than 30?</s><pad><pad>…

we just print the shape of attention_mask for brevity. +Weird that the t5 version is missing special tokens, whereas bart has them. I suspect this is the problem. Seems like you are truncating more aggressively for T5.

we just print the shape of attention_mask for brevity. Okay, got it. Seems like you are truncating more aggressively for T5. Yeah, but I think it should cover over 90% of the distribution. +Weird that the t5 version is missing special tokens, whereas bart has them. I suspect this is the p…

There is no <s> (bos) token in T5. Also it might be useful to train the models and calculate the appropriate metric for the QA task rather than comparing ROUGE or loss, to see how it’s performing.

The appropriate metric is exact-match, which is less lenient than ROUGE. A TPU/TF trained T5-small scores 23% exact-match. A GPU/HF trained T5-small barely gets to 15% ROUGE (which is more relaxed than exact-match).

Visiting from your comment on github! I am just getting back to this project (finetuning t5) this weekend. I was planning to do similar investigations to what you’ve been doing here, since I also suspected something might be wrong with tokenizer / EOS/ masking. I haven’t run any new experiments ye…

@danyaljj Were you able to get things working?

Issue with finetuning a seq-to-seq model

🤗Transformers

brando August 11, 2022, 3:24pm 31

does this solve this? How to add all standard special tokens to my tokenizer and model?

Topic		Replies	Views
Encoder-Decoder model only generates bos_token's [<s><s><s>] Models	17	3335	December 6, 2022
Finetuning T5 problems Models	12	239	January 16, 2026
T5 Generates very short summaries 🤗Transformers	22	5804	September 11, 2020
T5 Finetuning Tips Models	48	58239	November 3, 2024
HF Trainer: HF trainer cause a problem while fine-tuning T5 (T5 doesn't generate eos token at proper point) 🤗Transformers	0	867	March 6, 2022

Issue with finetuning a seq-to-seq model

Related topics