File size: 6,167 Bytes
de821b5 5ab7214 4c71592 de821b5 77974b7 b45bad5 77974b7 9253029 77974b7 de821b5 46e8d4d 0e496e2 46e8d4d 0b4902f 8572d46 0e496e2 de821b5 4c71592 de821b5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 |
---
language:
- en
license: apache-2.0
tags:
- text-generation-inference
- transformers
- unsloth
- mistral
- trl
- sft
base_model: alpindale/Mistral-7B-v0.2
---
# Mistral-7B-v0.2-OpenHermes
SFT Training Params:
+ Learning Rate: 2e-4
+ Batch Size: 8
+ Gradient Accumulation steps: 4
+ Dataset: teknium/OpenHermes-2.5 (200k split contains a slight bias towards rp and theory of life)
+ r: 16
+ Lora Alpha: 16
Training Time: 13 hours on A100
Prompt Template: ChatML
```
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What's the capital of France?<|im_end|>
<|im_start|>assistant
Paris.
```
## Quantizations
[GGUF](https://huggingface.co/macadeliccc/Mistral-7B-v0.2-OpenHermes-GGUF)
[AWQ](https://huggingface.co/macadeliccc/Mistral-7B-v0.2-OpenHermes-AWQ/)
### Evaluations
Thanks to Maxime Labonne for the evalution:
| Model |AGIEval|GPT4All|TruthfulQA|Bigbench|Average|
|-------------------------------------------------------------------------------------------|------:|------:|---------:|-------:|------:|
|[Mistral-7B-v0.2-OpenHermes](https://huggingface.co/macadeliccc/Mistral-7B-v0.2-OpenHermes)| 35.57| 67.15| 42.06| 36.27| 45.26|
### AGIEval
| Task |Version| Metric |Value| |Stderr|
|------------------------------|------:|--------|----:|---|-----:|
|agieval_aqua_rat | 0|acc |24.02|± | 2.69|
| | |acc_norm|21.65|± | 2.59|
|agieval_logiqa_en | 0|acc |28.11|± | 1.76|
| | |acc_norm|34.56|± | 1.87|
|agieval_lsat_ar | 0|acc |27.83|± | 2.96|
| | |acc_norm|23.48|± | 2.80|
|agieval_lsat_lr | 0|acc |33.73|± | 2.10|
| | |acc_norm|33.14|± | 2.09|
|agieval_lsat_rc | 0|acc |48.70|± | 3.05|
| | |acc_norm|39.78|± | 2.99|
|agieval_sat_en | 0|acc |67.48|± | 3.27|
| | |acc_norm|64.56|± | 3.34|
|agieval_sat_en_without_passage| 0|acc |38.83|± | 3.40|
| | |acc_norm|37.38|± | 3.38|
|agieval_sat_math | 0|acc |32.27|± | 3.16|
| | |acc_norm|30.00|± | 3.10|
Average: 35.57%
### GPT4All
| Task |Version| Metric |Value| |Stderr|
|-------------|------:|--------|----:|---|-----:|
|arc_challenge| 0|acc |45.05|± | 1.45|
| | |acc_norm|48.46|± | 1.46|
|arc_easy | 0|acc |77.27|± | 0.86|
| | |acc_norm|73.78|± | 0.90|
|boolq | 1|acc |68.62|± | 0.81|
|hellaswag | 0|acc |59.63|± | 0.49|
| | |acc_norm|79.66|± | 0.40|
|openbookqa | 0|acc |31.40|± | 2.08|
| | |acc_norm|43.40|± | 2.22|
|piqa | 0|acc |80.25|± | 0.93|
| | |acc_norm|82.05|± | 0.90|
|winogrande | 0|acc |74.11|± | 1.23|
Average: 67.15%
### TruthfulQA
| Task |Version|Metric|Value| |Stderr|
|-------------|------:|------|----:|---|-----:|
|truthfulqa_mc| 1|mc1 |27.54|± | 1.56|
| | |mc2 |42.06|± | 1.44|
Average: 42.06%
### Bigbench
| Task |Version| Metric |Value| |Stderr|
|------------------------------------------------|------:|---------------------|----:|---|-----:|
|bigbench_causal_judgement | 0|multiple_choice_grade|56.32|± | 3.61|
|bigbench_date_understanding | 0|multiple_choice_grade|66.40|± | 2.46|
|bigbench_disambiguation_qa | 0|multiple_choice_grade|45.74|± | 3.11|
|bigbench_geometric_shapes | 0|multiple_choice_grade|10.58|± | 1.63|
| | |exact_str_match | 0.00|± | 0.00|
|bigbench_logical_deduction_five_objects | 0|multiple_choice_grade|25.00|± | 1.94|
|bigbench_logical_deduction_seven_objects | 0|multiple_choice_grade|17.71|± | 1.44|
|bigbench_logical_deduction_three_objects | 0|multiple_choice_grade|37.33|± | 2.80|
|bigbench_movie_recommendation | 0|multiple_choice_grade|29.40|± | 2.04|
|bigbench_navigate | 0|multiple_choice_grade|50.00|± | 1.58|
|bigbench_reasoning_about_colored_objects | 0|multiple_choice_grade|42.50|± | 1.11|
|bigbench_ruin_names | 0|multiple_choice_grade|39.06|± | 2.31|
|bigbench_salient_translation_error_detection | 0|multiple_choice_grade|12.93|± | 1.06|
|bigbench_snarks | 0|multiple_choice_grade|69.06|± | 3.45|
|bigbench_sports_understanding | 0|multiple_choice_grade|49.80|± | 1.59|
|bigbench_temporal_sequences | 0|multiple_choice_grade|26.50|± | 1.40|
|bigbench_tracking_shuffled_objects_five_objects | 0|multiple_choice_grade|21.20|± | 1.16|
|bigbench_tracking_shuffled_objects_seven_objects| 0|multiple_choice_grade|16.06|± | 0.88|
|bigbench_tracking_shuffled_objects_three_objects| 0|multiple_choice_grade|37.33|± | 2.80|
Average: 36.27%
Average score: 45.26%
Elapsed time: 01:49:22
- **Developed by:** macadeliccc
- **License:** apache-2.0
- **Finetuned from model :** alpindale/Mistral-7B-v0.2
This mistral model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
|