How does one create a pytoch data loader using an interleaved hugging face dataset?

[brando]

HuggingFace dataset: each element in list of batch should be of equal size

I’m trying to use HuggingFace’s tokenizers and datasets with a PyTorch dataloader, like so: dataset = load_dataset( 'wikitext', 'wikitext-2-raw-v1', split='train[:5%]', # take only first 5% of the dataset cache_dir=cache_dir) tokenized_dataset = dataset.map( lambda e: self.tokenizer(e['text'], padding=True, ma…