I am trying to load a dataset from the openx datasets using a command like this: dataset = datasets.load_dataset( “jxu124/OpenX-Embodiment”, “berkeley_gnm_cory_hall”, streaming=False, split=“train”, cache_dir=ds_root, trust_remote_code=True, ) But it’s failing on any argument configurations…

Character code errors still occur in 2024… Apparently there are cases where it can be avoided by explicitly specifying it at load time. If this does not work, there may be another cause. [image] Random utf-8 errors from dataset Intermediate I’ve moved on from that projec…

Thanks for the reply, @John6666 . Checking the dataset.load_dataset(), I don’t see any ‘encoding’ argument option. Setting it triggers such errors: ValueError: BuilderConfig OpenXConfig (name='berkeley_gnm_cory_hall', version=0.0.0, data_dir=None, data_files=None, description=None, features=None) d…

The referenced article is from 2023, so maybe the argument is obsolete… Anyway, I agree with you that just because the error message is UTF-8 related does not mean that it is a character code issue. I hope it’s a problem that can be avoided by modifying the options or code without changing the dat…

I’m sure you’ve done this long ago, but just a spell. pip uninstall datasets pip install git+https://github.com/huggingface/datasets.git This is an article from April of this year, but it seems that some people are using an unusually old datasets library, perhaps pulled in by some library dependen…

Thanks for the suggestions, @John6666 . I tried changing versions, but doesn’t look like working. Maybe it’s because the code is not well-maintained &/ dataset formats have become incompatible.

@lhoestq There is an occasional problem with some datasets not loading properly in the datasets library. On the surface this appears as a UTF-8 error, but the actual cause is unknown. Is this fixable?

We’re using PyArrow to load the data, maybe we can see with them how to improve it. Do you have a reproducible example ?

[image] jxu124/OpenX-Embodiment · Datasets at Hugging Face We’re on a journey to advance and democratize artificial intelligence through open source and open science. I would have to ask yashara to know exactly, but perhaps this data set can reproduce it. Als…

Do you have an example that doesn’t use trust_remote_code ? We stopped developing that option because it’s not great for obvious security reasons. Also note that datasets in WebDataset format are supported out-of-the-box so trust_remote_code / having a loading script is not needed in that case.

Do you have an example that doesn’t use trust_remote_code ? 2023, but his code doesn’t seem to trust_remote_code. [image] 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte Beginners Hello, there. I downloaded a dataset in Hub and saved it to a local…

Datasets.load_datasets fails

🤗Datasets

John6666 October 11, 2024, 2:00pm 11

Do you have an example that doesn’t use trust_remote_code ?

2023, but his code doesn’t seem to trust_remote_code.

from datasets import load_dataset
raw_datasets = load_dataset("roneneldan/TinyStories")
raw_datasets.save_to_disk("Tiny_Stories")
raw_datasets = load_dataset('text', data_dir = "Tiny_Stories")

Topic		Replies	Views
'utf-8' codec can't decode byte 0xff in position 0: invalid start byte Beginners	3	12499	August 23, 2023
Random utf-8 errors from dataset Intermediate	10	3992	December 8, 2023
UniDecodeError: 'charmap' codec can't decode byte from Load_dataset Beginners	0	88	December 5, 2024
'utf-8' codec can't decode byte 0xff in position 0: invalid start byte (dataset) Beginners	0	473	May 19, 2024
UTF-16 for datasets? 🤗Datasets	4	1593	June 21, 2023

Datasets.load_datasets fails

Related topics