Fill-in the table of fine-tuning hyperparameters (#2)

- Fill-in the table of fine-tuning hyperparameters (1d34a113cc19630b8da65fe6bbc845b23c6cd857)


Co-authored-by: Jean-Loup Tastet <[email protected]>

README.md

@@ -8,41 +8,39 @@ language:
 
 Our submission to the `strict-small` track of the [BabyLM challenge](https://babylm.github.io/index.html).
 
-Baby Llama is a 58-million-parameter model, distilled from an ensemble consisting of LLaMA-360M and GPT2-705M, both trained on the `babylm_10M` dataset.
+Baby Llama is a 58M-parameter model, distilled from an ensemble consisting of LLaMA-360M and GPT2-705M, both trained on the `babylm_10M` dataset.
 
 See the associated paper (arXiv number **TBA**) for a detailed discussion of the training procedure and of the model performance.
 
-### Hyperparameters for the tasks requiring fine-tuning
+### Hyperparameters for the tasks that require fine-tuning
 
 When evaluating the model on the [tasks that require fine-tuning](https://github.com/babylm/evaluation-pipeline/tree/main#fine-tuning),
 we noticed that the [default hyperparameters](https://github.com/babylm/evaluation-pipeline/tree/main#hyperparameters)
 suggested by the BabyLM organizers lead to severe overfitting in a number of tasks.
 To avoid this issue, we have re-tuned those hyperparameters.
-
 The sets of hyperparameters selected for each task are listed in the table below.
-A star (*) indicates that the early-stopping criterion was triggered before the specified number of epochs was reached.
 
-| Task | Initial learning rate | Batch size | Maximum epochs | Patience | Evaluate every (steps) | Random seed |
+| Task | Maximum learning rate | Batch size | Maximum epochs | Patience | Evaluate every (steps) | Random seed |
 | ---- | ------------- | ---------- | -------- | -------- | ---------- | ---- |
-| CoLA | | | | | | |
-| SST-2 | | | | | | |
-| MRPC | | | | | | |
-| QQP | | | | | | |
-| MNLI | | | | | | |
-| MNLI-mm | | | | | | |
-| QNLI | | | | | | |
+| CoLA | 4e-5 | 64 | 3 | 10 | 20 | 12 |
+| SST-2 | 5e-5 | 64 | 6 | 10 | 200 | 12 |
+| MRPC | 3e-5 | 64 | 3 | 10 | 20 | 12 |
+| QQP | 4e-5 | 64 | 10 | 10 | 1000 | 12 |
+| MNLI | 5e-5 | 64 | 6 | 10 | 200 | 12 |
+| MNLI-mm |5e-5 | 64 | 6 | 10 | 200 | 12 |
+| QNLI | 5e-5 | 64 | 6 | 10 | 200 | 12 |
 | RTE | 5e-5 | 64 | 6 | 10 | 200 | 12 |
-| BoolQ | 3e-4 | 16 | 10* | 10 | 10 | 12 |
+| BoolQ | 3e-4 | 16 | 10 | 10 | 10| 12 |
 | MultiRC | 1e-4 | 64 | 7 | 10 | 1000 | 42 |
 | WSC | 5e-7 | 1 | 10 | 1000 | 2000 | 12 |
-| CR (Control) | | | | | | |
-| LC (Control) | | | | | | |
-| MV (Control) | | | | | | |
-| RP (Control) | | | | | | |
-| SC (Control) | | | | | | |
-| CR\_LC | | | | | | |
-| CR\_RTP | | | | | | |
-| MV\_LC | | | | | | |
-| MV\_RTP | | | | | | |
-| SC\_LC | | | | | | |
-| SC\_RP | | | | | | |
+| CR (Control) | 5e-5 | 64 | 10 | 10 | 100 | 12 |
+| LC (Control) | 1e-3 | 64 | 1 | 2 | 10 | 12 |
+| MV (Control) | 5e-5 | 64 | 6 | 10 | 200 | 12 |
+| RP (Control) | 1e-3 | 64 | 1 | 10 | 10 | 12 |
+| SC (Control) | 1e-3 | 64 | 2 | 10 | 10 | 12 |
+| CR\_LC | 1e-3 | 64 | 2 | 10 | 10 | 12 |
+| CR\_RTP | 5e-5 | 64 | 6 | 10 | 200 | 12 |
+| MV\_LC | 5e-5 | 64 | 6 | 10 | 200 | 12 |
+| MV\_RTP | 5e-5 | 64 | 6 | 10 | 200 | 12 |
+| SC\_LC | 1e-3 | 64 | 2 | 10 | 10 | 12 |
+| SC\_RP | 1e-3 | 64 | 2 | 10 | 10 | 12 |