Hi, I am trying to train a tokenizer and execute the following line of code: new_tokenizer = tokenizer.train_new_from_iterator(batch_iterator(), vocab_size=25000) Though, when I execute it, I get this error: KeyError: "Invalid key: slice(0, 1000, None). Please first select a split. For example: …

Hi ! Your batch_iterator must be iterating on a Dataset object, however it looks like you try to iterate over a DatasetDict (it maps split names to Dataset objects). To fix your code, you just have to replace dataset by dataset["train"] in your definition of batch_iterator. Let me know if that wor…

Hi, thanks for that - it works! I am now coming across this error, sadly: [image] [HELP] RuntimeError: CUDA error - when training my model? Beginners Hello everyone, I am encountering an error when training my language model from scratch, having trained a tokenizer befor…

But doesn’t that the trainer is only going to use the train dataset, because the iterator is only returning the train split?

KeyError: "Invalid key: slice(0, 1000, None). Please first select a split

Beginners

anon58275033 August 16, 2021, 9:24am 3

Hi, thanks for that - it works! I am now coming across this error, sadly:

Topic		Replies	Views
Unable to use custom dataset when training a tokenizer Beginners	2	408	August 11, 2021
KeyError: 'Invalid key. Only three types of key are available: (1) string, (2) integers for backend Encoding, and (3) slices for data subsetting.' Beginners	7	1104	June 3, 2024
Tokenizer from scratch Error TypeError: Can't convert None to PyString Beginners	1	1136	December 26, 2022
Tokenizer train_new_from_iterator hanging for several models 🤗Transformers	0	175	March 16, 2024
Getting error - trainer.train() 🤗Transformers	4	1049	June 3, 2024

KeyError: "Invalid key: slice(0, 1000, None). Please first select a split

Related topics