NbAiLab/nb-roberta-tpu

NB-ROBERTA Training Code

This is the current training code for the planned nb-roberta models.

We are currently planning to run the following experiments:

Name	nb-roberta-base-old (C)
Corpus	NbAiLab/nb_bert
Pod size	v4-64
Batch size	62*4*8 = 1984 = 2k
Learning rate	3e-4 (RoBERTa article is using 6e-4 and bs=8k)
Number of steps	250k

Name	nb-roberta-base-ext (B)
Corpus	NbAiLab/nbailab_extended
Pod size	v4-64
Batch size	62*4*8 = 1984 = 2k
Learning rate	3e-4 (RoBERTa article is using 6e-4 and bs=8k)
Number of steps	250k

Name	nb-roberta-large-ext
Corpus	NbAiLab/nbailab_extended
Pod size	v4-64
Batch size	32*4*8 = 2024 = 1k
Learning rate	2-e4 (RoBERTa article is using 4e-4 and bs=8k)
Number of steps	500k

Name	nb-roberta-base-scandi
Corpus	NbAiLab/scandinavian
Pod size	v4-64
Batch size	62*4*8 = 1984 = 2k
Learning rate	3e-4 (RoBERTa article is using 6e-4 and bs=8k)
Number of steps	250k

Name	nb-roberta-large-scandi
Corpus	NbAiLab/scandinavian
Pod size	v4-64
Batch size	32*4*8 = 1024 = 1k
Learning rate	2-e4 (RoBERTa article is using 4e-4 and bs=8k)
Number of steps	500k

Calculations

Some basic that we used when estimating the number of training steps:

• The Scandinavic Corpus is 85GB
• The Scandinavic Corpus contains 13B words
• With a conversion factor of 2.3, this is estimated to around 30B tokens
• 30B tokens / (512 seq length * 3000 batch size) = 20.000 steps