Opcodeo Tokenizer

natankatz May 17, 2023, 6:24am 1

Is there a tokenizer for opcode sources? (a model will be even better)

Topic		Replies	Views
Generate tokenizer.json for Marian(Opus) MT 🤗Tokenizers	2	697	November 4, 2024
About the Tokenizers category 🤗Tokenizers	1	339	July 7, 2020
How can I get a list of word segmentation results for non-English string? 🤗Transformers	14	132	November 6, 2025
Custom Tokenizer for source code Beginners	0	474	March 4, 2022
Easy way to compare tokenizers 🤗Tokenizers	0	314	May 1, 2023