Update README.md
Browse files
README.md
CHANGED
@@ -14,9 +14,9 @@ Lil-Bevo is UT Austin's submission to the BabyLM challenge, specifically the *st
|
|
14 |
|
15 |
## TLDR:
|
16 |
- Unigram tokenizer trained on 10M BabyLM tokens plus MAESTRO dataset for a vocab size of 16k.
|
17 |
-
- `deberta-small-v3` trained on mixture of MAESTRO and 10M tokens for
|
18 |
- Model continues training for 50 epochs on 10M tokens with 128 sequence length.
|
19 |
-
- Model continues training for
|
20 |
- Model is trained with targeted linguistic masking for 10 epochs.
|
21 |
|
22 |
|
|
|
14 |
|
15 |
## TLDR:
|
16 |
- Unigram tokenizer trained on 10M BabyLM tokens plus MAESTRO dataset for a vocab size of 16k.
|
17 |
+
- `deberta-small-v3` trained on mixture of MAESTRO and 10M tokens for 5 epochs.
|
18 |
- Model continues training for 50 epochs on 10M tokens with 128 sequence length.
|
19 |
+
- Model continues training for 2 epochs on 10M tokens with 512 sequence length.
|
20 |
- Model is trained with targeted linguistic masking for 10 epochs.
|
21 |
|
22 |
|