Create README.md

README.md

@@ -0,0 +1,54 @@
+---
+language: en
+
+---
+
+
+# OLM RoBERTa/BERT October 2022
+
+This is a more up-to-date version of the [original BERT](https://huggingface.co/bert-base-cased) and [original RoBERTa](https://huggingface.co/roberta-base).
+In addition to being more up-to-date, it also tends to perform better than the original BERT on standard benchmarks.
+We think it is more fair to directly compare our model to the original BERT because our model was trained with about the same level of compute as the original BERT, and the architecture of BERT and RoBERTa are basically the same.
+RoBERTa takes an order of magnitude more compute, although our model is also not that different in performance from RoBERTa on standard benchmarks.
+Our model was trained on a cleaned October 2022 snapshot of Common Crawl and Wikipedia.
+
+This model was created as part of the OLM project, which has the goal of continuously training and releasing models that are up-to-date and comparable in standard language model performance to their static counterparts.
+This is important because we want our models to know about events like COVID or 
+a presidential election right after they happen.
+
+## Intended uses
+
+You can use the raw model for text generation or fine-tune it to a downstream task.
+
+## How to use
+
+TODO
+
+## Dataset
+
+The model and tokenizer were trained with this [October 2022 cleaned Common Crawl dataset](https://huggingface.co/datasets/olm/olm-CC-MAIN-2022-40-sampling-ratio-0.15894621295) plus this [October 2022 cleaned Wikipedia dataset](https://huggingface.co/datasets/olm/olm-wikipedia-20221001).\
+The tokenized version of these concatenated datasets is [here](https://huggingface.co/datasets/olm/olm-october-2022-tokenized-512).\
+The datasets were created with this [repo](https://github.com/huggingface/olm-datasets).
+
+## Training
+
+The model was trained according to the OLM BERT/RoBERTa instructions at this [repo](https://github.com/huggingface/olm-training).
+
+## Evaluation results
+
+The model achieves the following results after being tuned on GLUE tasks:
+
+| Task | Metric   | Original BERT   | OLM RoBERTa Oct 2022 (Ours) |
+|:-----|:---------|----------------:|----------------------------:|
+|cola  |mcc       |**0.5889**       |0.2340                       |
+|sst2  |acc       |0.9181           |**0.9305**                   |
+|mrpc  |acc/f1    |**0.9182**/0.8923|0.8828/**0.9149**            |
+|stsb  |pear/spear|0.8822/0.8794    |**0.8943**/**0.8934**        |
+|qqp   |acc/f1    |0.9071/0.8748    |**0.9094**/**0.8781**        |
+|mnli  |acc/acc_mm|0.8400/0.8410    |**0.8599**/**0.8622**        |
+|qnli  |acc       |0.9075           |**0.9148**                   |
+|rte   |acc       |**0.6296**       |0.6253                       |
+|wnli  |acc       |0.4000           |**0.5042**                   |
+
+For both the original BERT and our model, we used the Hugging Face run_glue.py script [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification).
+For both models, we used the default fine-tuning hyperparameters and we averaged the results over five training seeds.