Hi, I was going through the documentation and got a confusion trainer = Trainer( model=model, # the instantiated :hugs: Transformers model to be trained args=training_args, # training arguments, defined above train_dataset=train_dataset, # trainin…

For more context, he/she is talking about this page: https://huggingface.co/transformers/training.html I also got confused by this bit of the documentation, but I think this code expects datasets like the ones provided by Hugging Face’s NLP package . I think they are all based on Pytorch’s Dataset …

Trainer is to be used with PyTorch, so in this case the train_dataset needs to be a PyTorch dataset. TFTrainer would expect a TF dataset. The doc page is a bit unclear (types are right in the signature but too short/wrong in the enumeration). I’ll send a fix to this later today.

Type of dataset in Trainer class

Beginners

valhalla July 18, 2020, 9:23am 3

Hi @suyash21 this post has some explanation about the dataset expected by Trainer

Topic		Replies	Views
Dataset expected by Trainer Beginners	5	9228	September 28, 2020
Load torchtext.data.dataset.Dataset to Trainer Beginners	0	587	October 20, 2020
Technical clarification on the validation data vs. the training data in the trainer API 🤗Transformers	1	823	January 6, 2022
How can I fine tune with my own dataset? 🤗Transformers	0	400	May 3, 2022
Issues with Trainer class on custom dataset 🤗Transformers	3	7617	June 14, 2023

Type of dataset in Trainer class

Related topics