Update README.md
Browse files
README.md
CHANGED
@@ -8,7 +8,7 @@ language: en
|
|
8 |
|
9 |
This is a more up-to-date version of the [original BERT](https://huggingface.co/bert-base-cased) and [original RoBERTa](https://huggingface.co/roberta-base).
|
10 |
In addition to being more up-to-date, it also tends to perform better than the original BERT on standard benchmarks.
|
11 |
-
We think it is
|
12 |
The original RoBERTa takes an order of magnitude more compute, although our model is also not that different in performance from RoBERTa on standard benchmarks.
|
13 |
Our model was trained on a cleaned October 2022 snapshot of Common Crawl and Wikipedia.
|
14 |
|
|
|
8 |
|
9 |
This is a more up-to-date version of the [original BERT](https://huggingface.co/bert-base-cased) and [original RoBERTa](https://huggingface.co/roberta-base).
|
10 |
In addition to being more up-to-date, it also tends to perform better than the original BERT on standard benchmarks.
|
11 |
+
We think it is fair to directly compare our model to the original BERT because our model was trained with about the same level of compute as the original BERT, and the architecture of BERT and RoBERTa are basically the same.
|
12 |
The original RoBERTa takes an order of magnitude more compute, although our model is also not that different in performance from RoBERTa on standard benchmarks.
|
13 |
Our model was trained on a cleaned October 2022 snapshot of Common Crawl and Wikipedia.
|
14 |
|