Chrisneverdie commited on
Commit
a50bd8a
·
verified ·
1 Parent(s): b684761

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -12,7 +12,7 @@ tags:
12
 
13
  ## Model Overview
14
 
15
- OnlySportsLM is a 196 million parameter language model specifically designed and trained for sports-related natural language processing tasks. It is part of the larger OnlySports collection, which aims to advance domain-specific language modeling in the sports domain.
16
 
17
  ## Model Architecture
18
 
@@ -22,9 +22,9 @@ OnlySportsLM is a 196 million parameter language model specifically designed and
22
 
23
  ## Training
24
 
25
- - Dataset: OnlySports Dataset (subset of 315B tokens out of 600B total)
26
  - Training setup: 8 H100 GPUs
27
- - Optimizer: AdamW with weight decay of 0.1
28
  - Learning rate: Initially 6e-4, adjusted to 1e-4 due to observed loss spikes
29
  - Context length: 1024 tokens
30
 
 
12
 
13
  ## Model Overview
14
 
15
+ OnlySportsLM is a 196M language model specifically designed and trained for sports-related natural language processing tasks. It is part of the larger OnlySports collection, which aims to advance domain-specific language modeling in sports.
16
 
17
  ## Model Architecture
18
 
 
22
 
23
  ## Training
24
 
25
+ - Dataset: [OnlySports Dataset](https://huggingface.co/datasets/Chrisneverdie/OnlySports_Dataset) (subset of 315B tokens out of 600B total)
26
  - Training setup: 8 H100 GPUs
27
+ - Optimizer: AdamW
28
  - Learning rate: Initially 6e-4, adjusted to 1e-4 due to observed loss spikes
29
  - Context length: 1024 tokens
30