Update README.md
Browse files
README.md
CHANGED
|
@@ -10,7 +10,7 @@ datasets:
|
|
| 10 |
- Anthropic/hh-rlhf
|
| 11 |
---
|
| 12 |
|
| 13 |
-
[Pythia-160m](https://huggingface.co/EleutherAI/pythia-
|
| 14 |
|
| 15 |
Checkpoints are also uploaded.
|
| 16 |
|
|
@@ -20,4 +20,50 @@ Fully reproducible finetuning code is available on [GitHub](https://github.com/l
|
|
| 20 |
|
| 21 |
See [Pythia-160m](https://huggingface.co/EleutherAI/pythia-410m) for model details [(paper)](https://arxiv.org/abs/2101.00027).
|
| 22 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
|
|
|
|
| 10 |
- Anthropic/hh-rlhf
|
| 11 |
---
|
| 12 |
|
| 13 |
+
[Pythia-160m](https://huggingface.co/EleutherAI/pythia-160m) supervised finetuned using TRLx library with the helpful subset of [Anthropic-hh-rlhf dataset](https://huggingface.co/datasets/Anthropic/hh-rlhf) for 1 epoch.
|
| 14 |
|
| 15 |
Checkpoints are also uploaded.
|
| 16 |
|
|
|
|
| 20 |
|
| 21 |
See [Pythia-160m](https://huggingface.co/EleutherAI/pythia-410m) for model details [(paper)](https://arxiv.org/abs/2101.00027).
|
| 22 |
|
| 23 |
+
hf (pretrained=lomahony/pythia-160m-helpful-sft), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 16
|
| 24 |
+
| Tasks |Version|Filter|n-shot| Metric | Value | |Stderr |
|
| 25 |
+
|--------------|------:|------|-----:|---------------|-------:|---|-------|
|
| 26 |
+
|arc_challenge | 1|none | 0|acc | 0.1894|± | 0.0115|
|
| 27 |
+
| | |none | 0|acc_norm | 0.2235|± | 0.0122|
|
| 28 |
+
|arc_easy | 1|none | 0|acc | 0.3889|± | 0.0100|
|
| 29 |
+
| | |none | 0|acc_norm | 0.3737|± | 0.0099|
|
| 30 |
+
|boolq | 2|none | 0|acc | 0.5346|± | 0.0087|
|
| 31 |
+
|hellaswag | 1|none | 0|acc | 0.2801|± | 0.0045|
|
| 32 |
+
| | |none | 0|acc_norm | 0.2949|± | 0.0046|
|
| 33 |
+
|lambada_openai| 1|none | 0|perplexity |439.3682|± |23.5771|
|
| 34 |
+
| | |none | 0|acc | 0.0984|± | 0.0041|
|
| 35 |
+
|openbookqa | 1|none | 0|acc | 0.1580|± | 0.0163|
|
| 36 |
+
| | |none | 0|acc_norm | 0.2260|± | 0.0187|
|
| 37 |
+
|piqa | 1|none | 0|acc | 0.5936|± | 0.0115|
|
| 38 |
+
| | |none | 0|acc_norm | 0.5865|± | 0.0115|
|
| 39 |
+
|sciq | 1|none | 0|acc | 0.5710|± | 0.0157|
|
| 40 |
+
| | |none | 0|acc_norm | 0.6290|± | 0.0153|
|
| 41 |
+
|wikitext | 2|none | 0|word_perplexity| 87.3261|± |N/A |
|
| 42 |
+
| | |none | 0|byte_perplexity| 2.3068|± |N/A |
|
| 43 |
+
| | |none | 0|bits_per_byte | 1.2059|± |N/A |
|
| 44 |
+
|winogrande | 1|none | 0|acc | 0.4878|± | 0.0140|
|
| 45 |
+
|
| 46 |
+
hf (pretrained=lomahony/pythia-160m-helpful-sft), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 16
|
| 47 |
+
| Tasks |Version|Filter|n-shot| Metric | Value | |Stderr |
|
| 48 |
+
|--------------|------:|------|-----:|---------------|--------:|---|-------|
|
| 49 |
+
|arc_challenge | 1|none | 5|acc | 0.2022|± | 0.0117|
|
| 50 |
+
| | |none | 5|acc_norm | 0.2270|± | 0.0122|
|
| 51 |
+
|arc_easy | 1|none | 5|acc | 0.3733|± | 0.0099|
|
| 52 |
+
| | |none | 5|acc_norm | 0.3746|± | 0.0099|
|
| 53 |
+
|boolq | 2|none | 5|acc | 0.5413|± | 0.0087|
|
| 54 |
+
|hellaswag | 1|none | 5|acc | 0.2770|± | 0.0045|
|
| 55 |
+
| | |none | 5|acc_norm | 0.2853|± | 0.0045|
|
| 56 |
+
|lambada_openai| 1|none | 5|perplexity |1644.8526|± |87.8870|
|
| 57 |
+
| | |none | 5|acc | 0.0491|± | 0.0030|
|
| 58 |
+
|openbookqa | 1|none | 5|acc | 0.1400|± | 0.0155|
|
| 59 |
+
| | |none | 5|acc_norm | 0.2200|± | 0.0185|
|
| 60 |
+
|piqa | 1|none | 5|acc | 0.5892|± | 0.0115|
|
| 61 |
+
| | |none | 5|acc_norm | 0.5854|± | 0.0115|
|
| 62 |
+
|sciq | 1|none | 5|acc | 0.5100|± | 0.0158|
|
| 63 |
+
| | |none | 5|acc_norm | 0.6020|± | 0.0155|
|
| 64 |
+
|wikitext | 2|none | 5|word_perplexity| 87.3261|± |N/A |
|
| 65 |
+
| | |none | 5|byte_perplexity| 2.3068|± |N/A |
|
| 66 |
+
| | |none | 5|bits_per_byte | 1.2059|± |N/A |
|
| 67 |
+
|winogrande | 1|none | 5|acc | 0.5178|± | 0.0140|
|
| 68 |
+
|
| 69 |
|