Update README.md
Browse files
README.md
CHANGED
|
@@ -124,3 +124,24 @@ def create_response(
|
|
| 124 |
pass
|
| 125 |
return response
|
| 126 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
pass
|
| 125 |
return response
|
| 126 |
```
|
| 127 |
+
|
| 128 |
+
### Model Performance Metrics
|
| 129 |
+
|
| 130 |
+
We present the results from various model benchmarks on the EleutherAI LLM Evaluation Harness for all models in the DLite family.
|
| 131 |
+
Model results are sorted by mean score, ascending, to provide an ordering. These metrics serve to further show that none of the DLite models are
|
| 132 |
+
state of the art, but rather further show that chat-like behaviors in LLMs can be trained almost independent of model size.
|
| 133 |
+
|
| 134 |
+
| model | openbookqa | arc_easy | winogrande | hellaswag | arc_challenge | piqa | boolq |
|
| 135 |
+
|:--------------|-------------:|-----------:|-------------:|------------:|----------------:|---------:|---------:|
|
| 136 |
+
| gpt2 | 0.164 | 0.438131 | 0.51618 | 0.289185 | 0.190273 | 0.628945 | 0.487156 |
|
| 137 |
+
| dlite-v2-124m | 0.174 | 0.44697 | 0.502762 | 0.291974 | 0.192833 | 0.631665 | 0.520183 |
|
| 138 |
+
| dlite-v1-124m | 0.17 | 0.462542 | 0.494081 | 0.293268 | 0.223549 | 0.622416 | 0.502446 |
|
| 139 |
+
| gpt2-medium | 0.186 | 0.490741 | 0.531176 | 0.333101 | 0.215017 | 0.676279 | 0.585933 |
|
| 140 |
+
| dlite-v2-355m | 0.206 | 0.493687 | 0.524073 | 0.334993 | 0.226109 | 0.670838 | 0.582263 |
|
| 141 |
+
| dlite-v1-355m | 0.216 | 0.507576 | 0.496448 | 0.338478 | 0.234642 | 0.664309 | 0.600306 |
|
| 142 |
+
| gpt2-large | 0.194 | 0.531566 | 0.553275 | 0.363971 | 0.216724 | 0.703482 | 0.604893 |
|
| 143 |
+
| dlite-774m-v2 | 0.212 | 0.539562 | 0.5588 | 0.365565 | 0.234642 | 0.700218 | 0.60367 |
|
| 144 |
+
| dlite-774m-v1 | 0.218 | 0.545875 | 0.562747 | 0.375124 | 0.250853 | 0.698041 | 0.614985 |
|
| 145 |
+
| gpt2-xl | 0.224 | 0.582912 | 0.583268 | 0.400418 | 0.25 | 0.708379 | 0.617737 |
|
| 146 |
+
| dlite-v1-1.5b | 0.226 | 0.588384 | 0.584846 | 0.401414 | 0.268771 | 0.708379 | 0.624159 |
|
| 147 |
+
| dlite-v2-1.5b | 0.226 | 0.59596 | 0.581689 | 0.40719 | 0.273891 | 0.705114 | 0.630887 |
|