|
--- |
|
tags: |
|
- pytorch |
|
- causal-lm |
|
metrics: |
|
- accuracy |
|
language: |
|
- sl |
|
license: apache-2.0 |
|
--- |
|
|
|
# GPT-sl-base |
|
|
|
This model is a Slovene GPT model, based on the [bigscience workshop](https://github.com/bigscience-workshop/Megatron-DeepSpeed) fork of the Megatron. GPT-sl-base was trained on large Slovene corpora: Gigafida, KAS, slWaC, and MaCoCu. |
|
|
|
## Model architecture |
|
GPT-sl-base has about 110 million parameters. It consists of 12 transformer layers with a dimension of 768. It has 16 attention heads and can process sequences up to 1024 tokens in length. |
|
The tokenizer was trained on a smaller subset of the corpora, and has the vocabulary of 60k tokens. |
|
|
|
## Training |
|
The model was trained for about 20 epochs, a total of 390k steps or 102B tokens seen during training. |
|
|
|
| Step | Validation Perplexity | |
|
|:------:|:---------------------:| |
|
| 50000 | 26.801 | |
|
| 100000 | 25.574 | |
|
| 150000 | 24.773 | |
|
| 200000 | 24.099 | |
|
| 250000 | 23.336 | |
|
| 300000 | 22.607 | |
|
| 350000 | 22.329 | |
|
| 390000 | 22.293 | |
|
|
