|
--- |
|
license: mit |
|
language: |
|
- en |
|
tags: |
|
- babylm |
|
--- |
|
|
|
# Lil-Bevo |
|
|
|
Lil-Bevo is UT Austin's submission to the BabyLM challenge, specifically the *strict-small* track. |
|
|
|
[Link to GitHub Repo](https://github.com/venkatasg/Lil-Bevo) |
|
|
|
## TLDR: |
|
- Unigram tokenizer trained on 10M BabyLM tokens plus MAESTRO dataset for a vocab size of 16k. |
|
- `deberta-small-v3` trained on mixture of MAESTRO and 10M tokens for 5 epochs. |
|
- Model continues training for 50 epochs on 10M tokens with 128 sequence length. |
|
- Model continues training for 2 epochs on 10M tokens with 512 sequence length. |
|
- Model is trained with targeted linguistic masking for 10 epochs. |
|
|
|
|
|
This README will be updated with more details soon. |
