gchhablani
commited on
Commit
·
5dd8688
1
Parent(s):
baaf453
Fix README
Browse files
README.md
CHANGED
@@ -12,7 +12,8 @@ datasets:
|
|
12 |
# MultiBERTs Seed 1 Checkpoint 20k (uncased)
|
13 |
Seed 1 intermediate checkpoint 20k MultiBERTs (pretrained BERT) model on English language using a masked language modeling (MLM) objective. It was introduced in
|
14 |
[this paper](https://arxiv.org/pdf/2106.16163.pdf) and first released in
|
15 |
-
[this repository](https://github.com/google-research/language/tree/master/language/multiberts). This
|
|
|
16 |
between english and English.
|
17 |
|
18 |
Disclaimer: The team releasing MultiBERTs did not write a model card for this model so this model card has been written by [gchhablani](https://hf.co/gchhablani).
|
@@ -81,7 +82,7 @@ The details of the masking procedure for each sentence are the following:
|
|
81 |
- In the 10% remaining cases, the masked tokens are left as is.
|
82 |
|
83 |
### Pretraining
|
84 |
-
The model was trained on 16 Cloud TPU v2 chips for two million steps with a batch size
|
85 |
of 256. The sequence length was set to 512 throughout. The optimizer
|
86 |
used is Adam with a learning rate of 1e-4, \\(\beta_{1} = 0.9\\) and \\(\beta_{2} = 0.999\\), a weight decay of 0.01,
|
87 |
learning rate warmup for 10,000 steps and linear decay of the learning rate after.
|
|
|
12 |
# MultiBERTs Seed 1 Checkpoint 20k (uncased)
|
13 |
Seed 1 intermediate checkpoint 20k MultiBERTs (pretrained BERT) model on English language using a masked language modeling (MLM) objective. It was introduced in
|
14 |
[this paper](https://arxiv.org/pdf/2106.16163.pdf) and first released in
|
15 |
+
[this repository](https://github.com/google-research/language/tree/master/language/multiberts). This is an intermediate checkpoint.
|
16 |
+
The final checkpoint can be found at [multiberts-seed-1](https://hf.co/multberts-seed-1). This model is uncased: it does not make a difference
|
17 |
between english and English.
|
18 |
|
19 |
Disclaimer: The team releasing MultiBERTs did not write a model card for this model so this model card has been written by [gchhablani](https://hf.co/gchhablani).
|
|
|
82 |
- In the 10% remaining cases, the masked tokens are left as is.
|
83 |
|
84 |
### Pretraining
|
85 |
+
The full model was trained on 16 Cloud TPU v2 chips for two million steps with a batch size
|
86 |
of 256. The sequence length was set to 512 throughout. The optimizer
|
87 |
used is Adam with a learning rate of 1e-4, \\(\beta_{1} = 0.9\\) and \\(\beta_{2} = 0.999\\), a weight decay of 0.01,
|
88 |
learning rate warmup for 10,000 steps and linear decay of the learning rate after.
|