File size: 5,612 Bytes
f52fdae 3586fc0 f52fdae b7c96e2 f52fdae |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 |
---
license: apache-2.0
tags:
- moe
- openchat
- hermes
- dolphin
- bagel
---
# 💫 Draco-8x7B
This is the model for Draco-8x7B. I used [this repo](https://bit.ly/weyaxi-moe-repo) to make this MOE model.
This model's experts are not using any merged models.
# 📚 Other branches (Number of Experts Per Token)
Other branches that this repository contains differ only slightly (from a git diff perspective) in terms of the number of experts per token.
Usually, a higher value for the number of experts per token will result in better performance, but it may also lead to increased inference time.
| Number of experts per token | Link of the branch |
| ---------------------------- | -------------------------------------------------------------------------------------------|
| 2 | [Main](https://huggingface.co/Weyaxi/Draco-8x7B/tree/main) |
| 3 | [3-experts-per-token](Link_To_3_Experts_Per_Token) |
| 4 | [4-experts-per-token](https://huggingface.co/Weyaxi/Draco-8x7B/tree/4-experts-per-token) |
| 6 | [6-experts-per-token](https://huggingface.co/Weyaxi/Draco-8x7B/tree/6-experts-per-token) |
| 8 | [8-experts-per-token](https://huggingface.co/Weyaxi/Draco-8x7B/tree/8-experts-per-token) |
# 💬 Prompt Template(s):
This model includes many models, so providing only one prompt template is not enough. You can use and try these prompt templates and decide which works best for you.
**Note:** The current chat template in the tokenizer config is set to [openchat-3.5-0106](https://huggingface.co/openchat/openchat-3.5-0106)'s chat template.
**Note 2:** It is also important to note that [jondurbin/bagel-dpo-7b-v0.1](https://huggingface.co/jondurbin/bagel-dpo-7b-v0.1) is using many prompt templates other than I provided. You can visit [jondurbin/bagel-dpo-7b-v0.1](https://huggingface.co/jondurbin/bagel-dpo-7b-v0.1) to learn more about this templates.
### GPT4 Correct
Used in [openchat/openchat-3.5-0106](https://huggingface.co/openchat/openchat-3.5-0106), [beowolx/CodeNinja-1.0-OpenChat-7B](https://huggingface.co/beowolx/CodeNinja-1.0-OpenChat-7B)
```
GPT4 Correct User: {user}<|end_of_turn|>GPT4 Correct Assistant: {asistant}<|end_of_turn|>
```
### ChatML:
Used in [teknium/OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B), [jondurbin/bagel-dpo-7b-v0.1](https://huggingface.co/jondurbin/bagel-dpo-7b-v0.1), [cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser](https://huggingface.co/cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser), [senseable/WestLake-7B-v2](https://huggingface.co/senseable/WestLake-7B-v2)
```
<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{asistant}<|im_end|>
```
### Math Alpaca
Used in [meta-math/MetaMath-Mistral-7B](https://huggingface.co/meta-math/MetaMath-Mistral-7B)
```
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Response: Let's think step by step.
```
# 🛠️ Yaml Config
<details><summary>See config</summary>
```yaml
base_model: openchat/openchat-3.5-0106
gate_mode: hidden
dtype: bfloat16
experts:
- source_model: openchat/openchat-3.5-0106
positive_prompts: # General (Mistral finetune)
- "chat"
- "assistant"
- "tell me"
- "explain"
- source_model: teknium/OpenHermes-2.5-Mistral-7B
positive_prompts: # General (Mistral finetune)
- "interact"
- "converse"
- "respond"
- "express"
- source_model: jondurbin/bagel-dpo-7b-v0.1
positive_prompts: # Science (Mistral finetune)
- "science"
- "biology"
- "chemistry"
- "physics"
- "Newton's laws"
- "scientific method"
- "periodic table"
- "photosynthesis process"
- source_model: meta-math/MetaMath-Mistral-7B
positive_prompts: # Math (Mistral finetune)
- "reason"
- "math"
- "mathematics"
- "solve"
- "count"
- source_model: cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser
positive_prompts: # Uncensored (Mistral finetune)
- "dolphin"
- "uncensored"
- "unbiased"
- "unfiltered"
- "unrestricted"
- "offensive"
- source_model: beowolx/CodeNinja-1.0-OpenChat-7B
positive_prompts: # Code (openchat-3.5-1210 finetune)
- "code"
- "script"
- "python"
- "javascript"
- "programming"
- "algorithm"
- source_model: senseable/WestLake-7B-v2
positive_prompts: # Roleplay (Unknown finetune)
- "storywriting"
- "write"
- "scene"
- "story"
- "character"
- "act as"
- "you are"
- source_model: snorkelai/Snorkel-Mistral-PairRM-DPO
positive_prompts: # Question Answering (? Mistral-7B-Instruct-v0.2 finetune ?)
- "what happens"
- "what is"
- "what can"
- "why"
- "who"
- "can a"
```
</details><br>
# 🔄 Quantizationed versions
Quantizationed versions of this model is available thanks to [TheBloke](https://hf.co/TheBloke).
##### GPTQ
- [TheBloke/Draco-8x7B-GPTQ](https://huggingface.co/TheBloke/Draco-8x7B-GPTQ)
##### GGUF
- [TheBloke/Draco-8x7B-GGUF](https://huggingface.co/TheBloke/Draco-8x7B-GGUF)
##### AWQ
- [TheBloke/Draco-8x7B-AWQ](https://huggingface.co/TheBloke/Draco-8x7B-AWQ)
If you would like to support me:
[☕ Buy Me a Coffee](https://www.buymeacoffee.com/weyaxi) |