|
--- |
|
license: apache-2.0 |
|
tags: |
|
- moe |
|
- openchat |
|
- hermes |
|
- dolphin |
|
- bagel |
|
model-index: |
|
- name: Draco-8x7B |
|
results: |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: AI2 Reasoning Challenge (25-Shot) |
|
type: ai2_arc |
|
config: ARC-Challenge |
|
split: test |
|
args: |
|
num_few_shot: 25 |
|
metrics: |
|
- type: acc_norm |
|
value: 65.02 |
|
name: normalized accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=PulsarAI/Draco-8x7B |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: HellaSwag (10-Shot) |
|
type: hellaswag |
|
split: validation |
|
args: |
|
num_few_shot: 10 |
|
metrics: |
|
- type: acc_norm |
|
value: 85.24 |
|
name: normalized accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=PulsarAI/Draco-8x7B |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: MMLU (5-Shot) |
|
type: cais/mmlu |
|
config: all |
|
split: test |
|
args: |
|
num_few_shot: 5 |
|
metrics: |
|
- type: acc |
|
value: 64.96 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=PulsarAI/Draco-8x7B |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: TruthfulQA (0-shot) |
|
type: truthful_qa |
|
config: multiple_choice |
|
split: validation |
|
args: |
|
num_few_shot: 0 |
|
metrics: |
|
- type: mc2 |
|
value: 62.65 |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=PulsarAI/Draco-8x7B |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: Winogrande (5-shot) |
|
type: winogrande |
|
config: winogrande_xl |
|
split: validation |
|
args: |
|
num_few_shot: 5 |
|
metrics: |
|
- type: acc |
|
value: 80.66 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=PulsarAI/Draco-8x7B |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: GSM8k (5-shot) |
|
type: gsm8k |
|
config: main |
|
split: test |
|
args: |
|
num_few_shot: 5 |
|
metrics: |
|
- type: acc |
|
value: 66.79 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=PulsarAI/Draco-8x7B |
|
name: Open LLM Leaderboard |
|
--- |
|
 |
|
|
|
# 💫 Draco-8x7B |
|
|
|
This is the model for Draco-8x7B. I used [this repo](https://bit.ly/weyaxi-moe-repo) to make this MOE model. |
|
|
|
This model's experts are not using any merged models. |
|
|
|
# 📚 Other branches (Number of Experts Per Token) |
|
|
|
Other branches that this repository contains differ only slightly (from a git diff perspective) in terms of the number of experts per token. |
|
|
|
Usually, a higher value for the number of experts per token will result in better performance, but it may also lead to increased inference time. |
|
|
|
| Number of experts per token | Link of the branch | |
|
| ---------------------------- | -------------------------------------------------------------------------------------------| |
|
| 2 | [Main](https://huggingface.co/Weyaxi/Draco-8x7B/tree/main) | |
|
| 3 | [3-experts-per-token](https://huggingface.co/Weyaxi/Draco-8x7B/tree/3-experts-per-token) | |
|
| 4 | [4-experts-per-token](https://huggingface.co/Weyaxi/Draco-8x7B/tree/4-experts-per-token) | |
|
| 6 | [6-experts-per-token](https://huggingface.co/Weyaxi/Draco-8x7B/tree/6-experts-per-token) | |
|
| 8 | [8-experts-per-token](https://huggingface.co/Weyaxi/Draco-8x7B/tree/8-experts-per-token) | |
|
|
|
|
|
# 💬 Prompt Template(s): |
|
|
|
This model includes many models, so providing only one prompt template is not enough. You can use and try these prompt templates and decide which works best for you. |
|
|
|
**Note:** The current chat template in the tokenizer config is set to [openchat-3.5-0106](https://huggingface.co/openchat/openchat-3.5-0106)'s chat template. |
|
|
|
**Note 2:** It is also important to note that [jondurbin/bagel-dpo-7b-v0.1](https://huggingface.co/jondurbin/bagel-dpo-7b-v0.1) is using many prompt templates other than I provided. You can visit [jondurbin/bagel-dpo-7b-v0.1](https://huggingface.co/jondurbin/bagel-dpo-7b-v0.1) to learn more about this templates. |
|
|
|
### GPT4 Correct |
|
|
|
Used in [openchat/openchat-3.5-0106](https://huggingface.co/openchat/openchat-3.5-0106), [beowolx/CodeNinja-1.0-OpenChat-7B](https://huggingface.co/beowolx/CodeNinja-1.0-OpenChat-7B) |
|
|
|
``` |
|
GPT4 Correct User: {user}<|end_of_turn|>GPT4 Correct Assistant: {asistant}<|end_of_turn|> |
|
``` |
|
|
|
### ChatML: |
|
|
|
Used in [teknium/OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B), [jondurbin/bagel-dpo-7b-v0.1](https://huggingface.co/jondurbin/bagel-dpo-7b-v0.1), [cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser](https://huggingface.co/cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser), [senseable/WestLake-7B-v2](https://huggingface.co/senseable/WestLake-7B-v2) |
|
|
|
``` |
|
<|im_start|>system |
|
{system}<|im_end|> |
|
<|im_start|>user |
|
{user}<|im_end|> |
|
<|im_start|>assistant |
|
{asistant}<|im_end|> |
|
``` |
|
|
|
### Math Alpaca |
|
|
|
Used in [meta-math/MetaMath-Mistral-7B](https://huggingface.co/meta-math/MetaMath-Mistral-7B) |
|
|
|
``` |
|
Below is an instruction that describes a task. Write a response that appropriately completes the request. |
|
|
|
### Instruction: |
|
{instruction} |
|
|
|
### Response: Let's think step by step. |
|
``` |
|
|
|
# 🛠️ Yaml Config |
|
|
|
<details><summary>See config</summary> |
|
|
|
```yaml |
|
base_model: openchat/openchat-3.5-0106 |
|
gate_mode: hidden |
|
dtype: bfloat16 |
|
|
|
experts: |
|
- source_model: openchat/openchat-3.5-0106 |
|
positive_prompts: # General (Mistral finetune) |
|
- "chat" |
|
- "assistant" |
|
- "tell me" |
|
- "explain" |
|
|
|
- source_model: teknium/OpenHermes-2.5-Mistral-7B |
|
positive_prompts: # General (Mistral finetune) |
|
- "interact" |
|
- "converse" |
|
- "respond" |
|
- "express" |
|
|
|
- source_model: jondurbin/bagel-dpo-7b-v0.1 |
|
positive_prompts: # Science (Mistral finetune) |
|
- "science" |
|
- "biology" |
|
- "chemistry" |
|
- "physics" |
|
- "Newton's laws" |
|
- "scientific method" |
|
- "periodic table" |
|
- "photosynthesis process" |
|
|
|
- source_model: meta-math/MetaMath-Mistral-7B |
|
positive_prompts: # Math (Mistral finetune) |
|
- "reason" |
|
- "math" |
|
- "mathematics" |
|
- "solve" |
|
- "count" |
|
|
|
- source_model: cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser |
|
positive_prompts: # Uncensored (Mistral finetune) |
|
- "dolphin" |
|
- "uncensored" |
|
- "unbiased" |
|
- "unfiltered" |
|
- "unrestricted" |
|
- "offensive" |
|
|
|
- source_model: beowolx/CodeNinja-1.0-OpenChat-7B |
|
positive_prompts: # Code (openchat-3.5-1210 finetune) |
|
- "code" |
|
- "script" |
|
- "python" |
|
- "javascript" |
|
- "programming" |
|
- "algorithm" |
|
|
|
- source_model: senseable/WestLake-7B-v2 |
|
positive_prompts: # Roleplay (Unknown finetune) |
|
- "storywriting" |
|
- "write" |
|
- "scene" |
|
- "story" |
|
- "character" |
|
- "act as" |
|
- "you are" |
|
|
|
- source_model: snorkelai/Snorkel-Mistral-PairRM-DPO |
|
positive_prompts: # Question Answering (? Mistral-7B-Instruct-v0.2 finetune ?) |
|
- "what happens" |
|
- "what is" |
|
- "what can" |
|
- "why" |
|
- "who" |
|
- "can a" |
|
``` |
|
|
|
</details><br> |
|
|
|
# 🔄 Quantizationed versions |
|
|
|
Quantizationed versions of this model is available thanks to [TheBloke](https://hf.co/TheBloke). |
|
|
|
##### GPTQ |
|
|
|
- [TheBloke/Draco-8x7B-GPTQ](https://huggingface.co/TheBloke/Draco-8x7B-GPTQ) |
|
|
|
##### GGUF |
|
|
|
- [TheBloke/Draco-8x7B-GGUF](https://huggingface.co/TheBloke/Draco-8x7B-GGUF) |
|
|
|
##### AWQ |
|
|
|
- [TheBloke/Draco-8x7B-AWQ](https://huggingface.co/TheBloke/Draco-8x7B-AWQ) |
|
|
|
If you would like to support me: |
|
|
|
[☕ Buy Me a Coffee](https://www.buymeacoffee.com/weyaxi) |
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) |
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_PulsarAI__Draco-8x7B) |
|
|
|
| Metric |Value| |
|
|---------------------------------|----:| |
|
|Avg. |70.89| |
|
|AI2 Reasoning Challenge (25-Shot)|65.02| |
|
|HellaSwag (10-Shot) |85.24| |
|
|MMLU (5-Shot) |64.96| |
|
|TruthfulQA (0-shot) |62.65| |
|
|Winogrande (5-shot) |80.66| |
|
|GSM8k (5-shot) |66.79| |
|
|
|
|
