Update README.md
Browse files
README.md
CHANGED
@@ -15,9 +15,8 @@ Lil-Bevo is UT Austin's submission to the BabyLM challenge, specifically the *st
|
|
15 |
## TLDR:
|
16 |
- Unigram tokenizer trained on 10M BabyLM tokens plus MAESTRO dataset for a vocab size of 16k.
|
17 |
- `deberta-small-v3` trained on mixture of MAESTRO and 10M tokens for 5 epochs.
|
18 |
-
- Model continues training for 50 epochs on 10M tokens with
|
19 |
-
- Model
|
20 |
-
- Model is trained with targeted linguistic masking for 10 epochs.
|
21 |
|
22 |
|
23 |
This README will be updated with more details soon.
|
|
|
15 |
## TLDR:
|
16 |
- Unigram tokenizer trained on 10M BabyLM tokens plus MAESTRO dataset for a vocab size of 16k.
|
17 |
- `deberta-small-v3` trained on mixture of MAESTRO and 10M tokens for 5 epochs.
|
18 |
+
- Model continues training for 50 epochs on 10M tokens with sequence length of 128.
|
19 |
+
- Model is trained for 2 epochs with targeted linguistic masking with sequence length of 512.
|
|
|
20 |
|
21 |
|
22 |
This README will be updated with more details soon.
|