Update README.md

README.md

@@ -1,3 +1,32 @@
 ---
+tags:
+- pytorch
+- causal-lm
+metrics:
+- accuracy
+language:
+- sl
 license: apache-2.0
 ---
+
+# GPT-sl-base
+
+This model is a Slovene GPT model, based on the [bigscience workshop](https://github.com/bigscience-workshop/Megatron-DeepSpeed) fork of the Megatron. GPT-sl-base was trained on large Slovene corpora: Gigafida, KAS, slWaC, and MaCoCu.
+
+## Model architecture
+GPT-sl-base has about 110 million parameters. It consists of 12 transformer layers with a dimension of 768. It has 16 attention heads and can process sequences up to 1024 tokens in length.
+The tokenizer was trained on a smaller subset of the corpora, and has the vocabulary of 60k tokens.
+
+## Training
+The model was trained for about 20 epochs, a total of 390k steps or 102B tokens seen during training.
+
+|  Step  | Validation Perplexity |
+|:------:|:---------------------:|
+|  50000 | 26.801                |
+| 100000 | 25.574                |
+| 150000 | 24.773                |
+| 200000 | 24.099                |
+| 250000 | 23.336                |
+| 300000 | 22.607                |
+| 350000 | 22.329                |
+| 390000 | 22.293                |