File size: 1,943 Bytes
a085148 ff95197 6c9dfaf a085148 ff91b3d a085148 0b3d3ea |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 |
---
language: is
widget:
- text: Má bjóða þér <mask> í kvöld?
- text: Forseti <mask> er ágæt.
- text: Súpan var <mask> á bragðið.
tags:
- roberta
- icelandic
- masked-lm
- pytorch
license: cc-by-4.0
datasets:
- mideind/icelandic-common-crawl-corpus-IC3
---
# IceBERT
IceBERT was trained with fairseq using the RoBERTa-base architecture. The training data used is shown in the table below.
| Dataset | Size | Tokens |
|------------------------------------------------------|---------|--------|
| Icelandic Gigaword Corpus v20.05 (IGC) | 8.2 GB | 1,388M |
| Icelandic Common Crawl Corpus (IC3) | 4.9 GB | 824M |
| Greynir News articles | 456 MB | 76M |
| Icelandic Sagas | 9 MB | 1.7M |
| Open Icelandic e-books (Rafbókavefurinn) | 14 MB | 2.6M |
| Data from the medical library of Landspitali | 33 MB | 5.2M |
| Student theses from Icelandic universities (Skemman) | 2.2 GB | 367M |
| Total | 15.8 GB | 2,664M |
If you find this model useful, please cite
```
@inproceedings{snaebjarnarson-etal-2022-warm,
title = "A Warm Start and a Clean Crawled Corpus - A Recipe for Good Language Models",
author = "Sn{\ae}bjarnarson, V{\'e}steinn and
S{\'\i}monarson, Haukur Barri and
Ragnarsson, P{\'e}tur Orri and
Ing{\'o}lfsd{\'o}ttir, Svanhv{\'\i}t Lilja and
J{\'o}nsson, Haukur and
Thorsteinsson, Vilhjalmur and
Einarsson, Hafsteinn",
booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation Conference",
month = jun,
year = "2022",
address = "Marseille, France",
publisher = "European Language Resources Association",
url = "https://aclanthology.org/2022.lrec-1.464",
pages = "4356--4366",
}
``` |