Hey all, I’m trying to work on an example involving AI training using datasets that are bigger than both ram and system disk. This means I’m loading datasets with streaming=True. I have S3 configured as a filesystem at /home/ubuntu/infinity, which allows for storing data that is bigger than the sys…

It seems that it is necessary to convert it to a normal Dataset first. [image] 'datasets.iterable_dataset.IterableDataset' to 'datasets.dataset_dict.DatasetDict' 🤗Datasets how can i convert <class ‘datasets.iterable_dataset.IterableDataset’> into <class ‘datasets.dataset_d…

That seems kinda unsatisfiable for this, would I have to shard it or something? IE: collect up to n (where n>=50,000) records into an in-memory Dataset and then save/load those?

Hmm, it’s quite a problem when the data set is large…

How do you save an IterableDataset to disk?

🤗Datasets

John6666 November 15, 2024, 12:29am 2

It seems that it is necessary to convert it to a normal Dataset first.

Topic		Replies	Views
Load iterable dataset from disk Beginners	2	2244	July 21, 2022
Uploading a heavy dataset to Jean-Zay Intermediate	3	202	February 17, 2025
Use load dataset to load a sample of the dataset 🤗Datasets	3	1364	May 24, 2021
Recommended max size of dataset? 🤗Datasets	5	654	March 11, 2025
Question about streaming 🤗Datasets	3	607	April 25, 2023

How do you save an IterableDataset to disk?

Related topics