mlabonne commited on
Commit
40c91f7
Β·
verified Β·
1 Parent(s): ff7c98d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -15
README.md CHANGED
@@ -121,9 +121,9 @@ model-index:
121
 
122
  # Daredevil-8B
123
 
124
- **tl;dr: It looks like a successful merge**
125
 
126
- Daredevil-8B is a merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
127
  * [nbeerbower/llama-3-stella-8B](https://huggingface.co/nbeerbower/llama-3-stella-8B)
128
  * [Hastagaras/llama-3-8b-okay](https://huggingface.co/Hastagaras/llama-3-8b-okay)
129
  * [nbeerbower/llama-3-gutenberg-8B](https://huggingface.co/nbeerbower/llama-3-gutenberg-8B)
@@ -134,6 +134,8 @@ Daredevil-8B is a merge of the following models using [LazyMergekit](https://col
134
  * [flammenai/Mahou-1.1-llama3-8B](https://huggingface.co/flammenai/Mahou-1.1-llama3-8B)
135
  * [KingNish/KingNish-Llama3-8b](https://huggingface.co/KingNish/KingNish-Llama3-8b)
136
 
 
 
137
  ## πŸ”Ž Applications
138
 
139
  It is a highly functional censored model. You might want to add `<end_of_turn>` as an additional stop string.
@@ -144,6 +146,14 @@ It is a highly functional censored model. You might want to add `<end_of_turn>`
144
 
145
  ## πŸ† Evaluation
146
 
 
 
 
 
 
 
 
 
147
  | Model | Average | AGIEval | GPT4All | TruthfulQA | Bigbench |
148
  | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------: | --------: | --------: | ---------: | --------: |
149
  | [**mlabonne/Daredevil-8B**](https://huggingface.co/mlabonne/Daredevil-8B) [πŸ“„](https://gist.github.com/mlabonne/080f9c5f153ea57a7ab7d932cf896f21) | **55.87** | **44.13** | **73.52** | **59.05** | **46.77** |
@@ -226,16 +236,3 @@ pipeline = transformers.pipeline(
226
  outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
227
  print(outputs[0]["generated_text"])
228
  ```
229
- # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
230
- Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_mlabonne__Daredevil-8B)
231
-
232
- | Metric |Value|
233
- |---------------------------------|----:|
234
- |Avg. |72.41|
235
- |AI2 Reasoning Challenge (25-Shot)|68.86|
236
- |HellaSwag (10-Shot) |84.50|
237
- |MMLU (5-Shot) |69.24|
238
- |TruthfulQA (0-shot) |59.89|
239
- |Winogrande (5-shot) |78.45|
240
- |GSM8k (5-shot) |73.54|
241
-
 
121
 
122
  # Daredevil-8B
123
 
124
+ Daredevil-8B is a mega-merge designed to maximize MMLU. On 27 May 24, it is the Llama 8B model with the **highest MMLU score**.
125
 
126
+ It is a merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
127
  * [nbeerbower/llama-3-stella-8B](https://huggingface.co/nbeerbower/llama-3-stella-8B)
128
  * [Hastagaras/llama-3-8b-okay](https://huggingface.co/Hastagaras/llama-3-8b-okay)
129
  * [nbeerbower/llama-3-gutenberg-8B](https://huggingface.co/nbeerbower/llama-3-gutenberg-8B)
 
134
  * [flammenai/Mahou-1.1-llama3-8B](https://huggingface.co/flammenai/Mahou-1.1-llama3-8B)
135
  * [KingNish/KingNish-Llama3-8b](https://huggingface.co/KingNish/KingNish-Llama3-8b)
136
 
137
+ Thanks to nbeerbower, Hastagaras, openchat, Kukedlc, cstr, flammenai, and KingNish for their merges. Special thanks to Charles Goddard and Arcee.ai for MergeKit.
138
+
139
  ## πŸ”Ž Applications
140
 
141
  It is a highly functional censored model. You might want to add `<end_of_turn>` as an additional stop string.
 
146
 
147
  ## πŸ† Evaluation
148
 
149
+ ### Open LLM Leaderboard
150
+
151
+ Here's the Open LLM Leaderboard sorted by MMLU score.
152
+
153
+ ![image/png](https://cdn-uploads.huggingface.co/production/uploads/61b8e2ba285851687028d395/6hrm6tD5GcIVpprgeFzmZ.png)
154
+
155
+ ### Nous
156
+
157
  | Model | Average | AGIEval | GPT4All | TruthfulQA | Bigbench |
158
  | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------: | --------: | --------: | ---------: | --------: |
159
  | [**mlabonne/Daredevil-8B**](https://huggingface.co/mlabonne/Daredevil-8B) [πŸ“„](https://gist.github.com/mlabonne/080f9c5f153ea57a7ab7d932cf896f21) | **55.87** | **44.13** | **73.52** | **59.05** | **46.77** |
 
236
  outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
237
  print(outputs[0]["generated_text"])
238
  ```