Using an IterableDataset for 1+ epochs in Trainer

[John6666]

Maybe this?

How to use Huggingface Trainer streaming Datasets without wrapping it with torchdata's IterableWrapper?

Using IterableDataset with Trainer - `IterableDataset' has no len()

Hi everyone, I have a large-ish dataset that I am loading with something like: dataset_train = load_dataset( 'json', data_files=..., split='train', streaming=True, ) def preprocess_function(examples): return tokenizer(examples["text"], examples["text_pair"], truncation=True) tokenized_dataset = dataset_train.map(preprocess_function, batched=True, batch_size=32) and I want to fine a text classification with: model = AutoModelForSequenceClassification.from_pretrained(pret…