Update README.md
Browse files
README.md
CHANGED
|
@@ -22,3 +22,23 @@ It achieves the following results on the evaluation set:
|
|
| 22 |
- Gen Len: 12.0586
|
| 23 |
- Num Input Tokens Seen: 1931815668
|
| 24 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
- Gen Len: 12.0586
|
| 23 |
- Num Input Tokens Seen: 1931815668
|
| 24 |
|
| 25 |
+
## Quick eval
|
| 26 |
+
|
| 27 |
+
Quick eval for: `BEE-spoke-data/tFINE-900m-e16-d32-flan`
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
hf (pretrained=BEE-spoke-data/tFINE-900m-e16-d32-flan,trust_remote_code=True,dtype=bfloat16,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8
|
| 31 |
+
| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
|
| 32 |
+
|-------------|------:|----------------|-----:|-----------|---|-----:|---|------|
|
| 33 |
+
|boolq | 2|none | 0|acc |↑ |0.6700|± |0.0082|
|
| 34 |
+
|openbookqa | 1|none | 0|acc |↑ |0.1900|± |0.0176|
|
| 35 |
+
| | |none | 0|acc_norm |↑ |0.2980|± |0.0205|
|
| 36 |
+
|piqa | 1|none | 0|acc |↑ |0.6001|± |0.0114|
|
| 37 |
+
| | |none | 0|acc_norm |↑ |0.6072|± |0.0114|
|
| 38 |
+
|social_iqa | 0|none | 0|acc |↑ |0.4299|± |0.0112|
|
| 39 |
+
|tinyArc | 0|none | 25|acc_norm |↑ |0.3214|± | N/A|
|
| 40 |
+
|tinyGSM8k | 0|flexible-extract| 5|exact_match|↑ |0.0492|± | N/A|
|
| 41 |
+
| | |strict-match | 5|exact_match|↑ |0.0380|± | N/A|
|
| 42 |
+
|tinyHellaswag| 0|none | 10|acc_norm |↑ |0.4005|± | N/A|
|
| 43 |
+
|tinyMMLU | 0|none | 0|acc_norm |↑ |0.2857|± | N/A|
|
| 44 |
+
|winogrande | 1|none | 0|acc |↑ |0.4988|± |0.0141|
|