Hello everyone! I want to share multiple datasets in the same repo <my_username>/<my_repo_name>, each in its own folder. The datasets in each folder are already in sharded Arrow format (for best performance) and contain different splits, as usual. To read any of these datasets with load_dataset I w…

You can configure the subsets present in your dataset repository in YAML :slight_smile: see the docs at Manual Configuration See the GLUE dataset for example: nyu-mll/glue at main

If it’s already been converted to a Dataset class, is datasets.concatenate_dataset sufficient…? @lhoestq [image] Issue concatenating datasets 🤗Datasets I am trying to concatenate two datasets from datasets import load_dataset, concatenate_datasets movie = load_dataset("m…

@John6666 no because i dont want to concateneate the datasets! Each folder is a different dataset with different features. So do i need the arrow builder to tell HF how to load the different datasets from the subfolder?

Hmm… In that case, I thought that it would be easier for Hugging Face, which is based on one model per repo (and dataset), to work properly if the datasets with different structures were kept separate.:thinking: However, I think there was a way to merge datasets with different structures. Let’s wa…

Yeah, maybe. I’m hesitating to separate into different repos because the datasets are related. It’s not completely separate projects. Think of it as GLUE, which is a set of multiple datasets but they are all related to one objective or project, like shown here Create a dataset loading script

This is amazing! Thank you very much.

Sharing ArrowDataset with subfolders

🤗Datasets

John6666 March 10, 2025, 3:20pm 2

If it’s already been converted to a Dataset class, is datasets.concatenate_dataset sufficient…? @lhoestq

Topic		Replies	Views
[solved] How to load multiple arrow files into one dataset Beginners	4	3315	September 16, 2023
[urgent]Can you reconstruct datasets using the cache file (.arrow file)? 🤗Datasets	5	1126	August 27, 2021
How to combine local data files with an official 🤗 dataset 🤗Datasets	15	3709	April 7, 2021
How to merge two dataset objects? Beginners	7	46660	February 28, 2024
ArrowBasedBuilder versus GeneratorDBasedBuilder 🤗Datasets	4	441	February 8, 2025

Sharing ArrowDataset with subfolders

Related topics