Difference betweeen DistilBertTokenizerFast and DistilBertTokenizer?

🤗Transformers
1

Hi everyone,

I’m trying to understand what is the difference between the DistilBertTokenizerFast and the DistilBertTokenizer.

From the documentation it looks like that the DistilBertTokenizerFast “Construct a “fast” DistilBERT tokenizer (backed by HuggingFace’s tokenizers library).” whereas the DistilBertTokenizer “Construct a DistilBERT tokenizer.”, but I don’t understand what that means.

This is what I found from the documentation: Tokenizer — transformers 4.5.0.dev0 documentation
To the best of my understanding the difference between a “fast” and a “non-fast” tokenizer is computation speed but there is no functional difference between them. Please correct me if I’m not the right direction.

Any help will be greatly appreciated.

Thank you,
Ayala

2

@ayalaall I also have the same thought as you.

I also want to know about this. Are they the same in terms of function?

3

FastTokenizers are implemented in Rust and are factors faster than the Python based tokenizers. Apart from that their encoding methods should behave the same. However, they are not functionally identical. From the docs:

When the tokenizer is a “Fast” tokenizer (i.e., backed by HuggingFace tokenizers library), this class provides in addition several advanced alignment methods which can be used to map between the original string (character and words) and the token space (e.g., getting the index of the token comprising a given character or the span of characters corresponding to a given token).

Typically you want to use the fast tokenizer if it is available.