RoBERTa/BERT for Norwegian Currently, there is only a very limited amount of BERT-like models for Thai on the hub: Models – Hugging Face . For this project, the goal is to create a RoBERTa/BERT model for just the Norwegian language. Model A randomly initialized RoBERTa/BERT model Datasets One can …

I am also really interested in this one, or maybe even ALBERT. But the available dataset on OSCAR might not be enough. Using mC4 might add almost 100GB of clean-ish text. I have worked on Norwegian BERT-based models before , and we (the National Library of Norway) have a big corpus that unfortunately…

PReTrain RoBERTa from scratch in Norwegian

Flax/JAX Projects

saattrupdan June 28, 2021, 5:39pm 3

It could make sense to pretrain a common Scandinavian model instead, as our languages are so similar. I’ve made a project suggestion here: Scandinavian RoBERTa.

Topic		Replies	Views
PreTrain RoBERTa from scratch in Thai Flax/JAX Projects	3	695	July 2, 2021
PreTrain RoBERTa from scratch in Hindi Flax/JAX Projects	24	2162	December 10, 2021
Scandinavian RoBERTa Flax/JAX Projects	30	2178	July 15, 2021
Pretrain RoBERTa-large from scratch in Swedish Flax/JAX Projects	2	1102	July 5, 2021
RobIt : PreTrain RoBERTa-base from scratch in Italian Flax/JAX Projects	4	522	June 29, 2021

PReTrain RoBERTa from scratch in Norwegian

Related topics