Hello, I have a folder with .arrow files that I previously created using save_to_disk. When I now use upload_large_folder I see some .cache- files being pushed to the hub, that I don’t see in the directory i’m pushing in my local machine. Is this normal? Are they duplicates or is HF splitting the f…

It seems like it’s okay to delete it, but if you’re worried, call lhonestq. [image] How can I clean the dataset cache? 🤗Datasets Feel free to delete the download directory. Then all the other directories are named after the datasets, and contain the cache files…

cache files are the cache results of e.g. map() operations on the original data, you can delete them if you don’t need the map() results anymore :wink:

Duplicated cache- arrow files when uploading large folder?

🤗Datasets

John6666 March 28, 2025, 12:25pm 2

It seems like it’s okay to delete it, but if you’re worried, call lhonestq.

Topic		Replies	Views
Deleting Duplicate Saved Datasets 🤗Datasets	3	4800	September 7, 2022
How does cache work? 🤗Datasets	1	391	November 28, 2023
Load_dataset without saving cache files 🤗Datasets	1	1965	April 19, 2023
Cache size much larger than `dh cache scan` shows 🤗Hub	3	131	October 24, 2025
How can I clean the dataset cache? 🤗Datasets	4	14404	March 1, 2024

Duplicated cache- arrow files when uploading large folder?

Related topics