|
--- |
|
datasets: |
|
- wikitext-2-v1 |
|
- wikitext |
|
language: |
|
- en |
|
metrics: |
|
- perplexity |
|
- cross_entropy |
|
--- |
|
|
|
**metrics on 1024 context**: |
|
- valid_perplexity = 14.79 |
|
- valid_cross_entropy = 2.69 |
|
- train_perplexity = 13.77 |
|
- train_cross_entropy = 2.62 |
|
|
|
**metrics on 252 context**: |
|
- valid_perplexity = 17.35 |
|
|
|
**metrics on 378 context**: |
|
- valid_perplexity = 16.4 |
|
|
|
**metrics on 504 context**: |
|
- valid_perplexity = 15.86 |
|
|
|
**Dependence of the cross entropy loss on the length of the context for prediction** |
|
|
|
- x-axis*128 = context length |
|
- y-axis = cross entropy |
|
|
|
 |
