File size: 330 Bytes
113af5d fca7629 |
1 2 3 4 5 6 |
# GPT-2 (125M) 4k tokens
Fine-tuned GPT2 Smallest model on The Pile with a token length of 4k.
Weights are included and it follows Karpathy's nanoGPT implementation.
The model has been trained for ~1 million iterations with increasing batch size, ending at 32k.
The final loss is 3.9 which is probably due to 768 embedding size. |