---
license: apache-2.0
inference: false
---
# Given-MPT-7B

This is a merge of the following MPT-7B models:

- **g**orilla-llm/gorilla-mpt-7b-hf-v0
- **i**bm/mpt-7b-instruct2
- Teh**V**enom/MPT-7b-WizardLM_Uncensored-Storywriter-Merge
- **e**mozilla/mpt-7b-storysummarizer
- **n**omic-ai/gpt4all-mpt


## Model License

Apache 2.0


## Purpose

This model is for experimenting with merging and routing to expert layers.

# Test eval on only 10% of eval set

hf-causal (pretrained=Multi-Domain-Expert-Layers/given-mpt-7b,dtype=bfloat16,trust_remote_code=True), limit: 0.1, provide_description: False, num_fewshot: 0, batch_size: None
|                      Task                       |Version|  Metric   | Value |   |Stderr|
|-------------------------------------------------|------:|-----------|------:|---|-----:|
|arc_challenge                                    |      0|acc        | 0.4274|±  |0.0459|
|                                                 |       |acc_norm   | 0.3846|±  |0.0452|
|arc_easy                                         |      0|acc        | 0.7863|±  |0.0381|
|                                                 |       |acc_norm   | 0.7350|±  |0.0410|
|hellaswag                                        |      0|acc        | 0.5556|±  |0.0461|
|                                                 |       |acc_norm   | 0.8120|±  |0.0363|
|hendrycksTest-college_chemistry                  |      0|acc        | 0.3600|±  |0.0482|
|                                                 |       |acc_norm   | 0.3700|±  |0.0485|
|hendrycksTest-college_computer_science           |      0|acc        | 0.3400|±  |0.0476|
|                                                 |       |acc_norm   | 0.3600|±  |0.0482|
|hendrycksTest-college_mathematics                |      0|acc        | 0.2500|±  |0.0435|
|                                                 |       |acc_norm   | 0.2900|±  |0.0456|
|hendrycksTest-college_medicine                   |      0|acc        | 0.3675|±  |0.0448|
|                                                 |       |acc_norm   | 0.3162|±  |0.0432|
|hendrycksTest-college_physics                    |      0|acc        | 0.2451|±  |0.0428|
|                                                 |       |acc_norm   | 0.2941|±  |0.0453|
|hendrycksTest-computer_security                  |      0|acc        | 0.4800|±  |0.0502|
|                                                 |       |acc_norm   | 0.4400|±  |0.0499|
|hendrycksTest-conceptual_physics                 |      0|acc        | 0.2051|±  |0.0375|
|                                                 |       |acc_norm   | 0.1709|±  |0.0350|
|hendrycksTest-econometrics                       |      0|acc        | 0.2982|±  |0.0430|
|                                                 |       |acc_norm   | 0.2368|±  |0.0400|
|hendrycksTest-electrical_engineering             |      0|acc        | 0.3248|±  |0.0435|
|                                                 |       |acc_norm   | 0.3590|±  |0.0445|
|hendrycksTest-elementary_mathematics             |      0|acc        | 0.3333|±  |0.0438|
|                                                 |       |acc_norm   | 0.3162|±  |0.0432|
|hendrycksTest-formal_logic                       |      0|acc        | 0.3077|±  |0.0429|
|                                                 |       |acc_norm   | 0.3248|±  |0.0435|
|hendrycksTest-global_facts                       |      0|acc        | 0.3000|±  |0.0461|
|                                                 |       |acc_norm   | 0.2700|±  |0.0446|
|hendrycksTest-high_school_biology                |      0|acc        | 0.3675|±  |0.0448|
|                                                 |       |acc_norm   | 0.3077|±  |0.0429|
|hendrycksTest-high_school_chemistry              |      0|acc        | 0.2564|±  |0.0405|
|                                                 |       |acc_norm   | 0.2906|±  |0.0422|
|hendrycksTest-high_school_computer_science       |      0|acc        | 0.4100|±  |0.0494|
|                                                 |       |acc_norm   | 0.4400|±  |0.0499|
|hendrycksTest-high_school_european_history       |      0|acc        | 0.4359|±  |0.0460|
|                                                 |       |acc_norm   | 0.3590|±  |0.0445|
|hendrycksTest-high_school_geography              |      0|acc        | 0.3248|±  |0.0435|
|                                                 |       |acc_norm   | 0.3675|±  |0.0448|
|hendrycksTest-high_school_government_and_politics|      0|acc        | 0.3932|±  |0.0454|
|                                                 |       |acc_norm   | 0.3932|±  |0.0454|
|hendrycksTest-high_school_macroeconomics         |      0|acc        | 0.3333|±  |0.0438|
|                                                 |       |acc_norm   | 0.3248|±  |0.0435|
|hendrycksTest-high_school_mathematics            |      0|acc        | 0.2051|±  |0.0375|
|                                                 |       |acc_norm   | 0.2564|±  |0.0405|
|hendrycksTest-high_school_microeconomics         |      0|acc        | 0.3504|±  |0.0443|
|                                                 |       |acc_norm   | 0.4188|±  |0.0458|
|hendrycksTest-high_school_physics                |      0|acc        | 0.2650|±  |0.0410|
|                                                 |       |acc_norm   | 0.2906|±  |0.0422|
|hendrycksTest-high_school_psychology             |      0|acc        | 0.3761|±  |0.0450|
|                                                 |       |acc_norm   | 0.3419|±  |0.0440|
|hendrycksTest-high_school_statistics             |      0|acc        | 0.3077|±  |0.0429|
|                                                 |       |acc_norm   | 0.3504|±  |0.0443|
|hendrycksTest-high_school_us_history             |      0|acc        | 0.3333|±  |0.0438|
|                                                 |       |acc_norm   | 0.3333|±  |0.0438|
|hendrycksTest-high_school_world_history          |      0|acc        | 0.3333|±  |0.0438|
|                                                 |       |acc_norm   | 0.3419|±  |0.0440|
|hendrycksTest-human_aging                        |      0|acc        | 0.3761|±  |0.0450|
|                                                 |       |acc_norm   | 0.3162|±  |0.0432|
|hendrycksTest-human_sexuality                    |      0|acc        | 0.4274|±  |0.0459|
|                                                 |       |acc_norm   | 0.3761|±  |0.0450|
|hendrycksTest-international_law                  |      0|acc        | 0.4188|±  |0.0458|
|                                                 |       |acc_norm   | 0.4957|±  |0.0464|
|hendrycksTest-jurisprudence                      |      0|acc        | 0.3148|±  |0.0449|
|                                                 |       |acc_norm   | 0.4815|±  |0.0483|
|hendrycksTest-logical_fallacies                  |      0|acc        | 0.3504|±  |0.0443|
|                                                 |       |acc_norm   | 0.3675|±  |0.0448|
|hendrycksTest-machine_learning                   |      0|acc        | 0.3214|±  |0.0443|
|                                                 |       |acc_norm   | 0.2946|±  |0.0433|
|hendrycksTest-management                         |      0|acc        | 0.3786|±  |0.0480|
|                                                 |       |acc_norm   | 0.3495|±  |0.0472|
|hendrycksTest-marketing                          |      0|acc        | 0.5043|±  |0.0464|
|                                                 |       |acc_norm   | 0.4188|±  |0.0458|
|hendrycksTest-medical_genetics                   |      0|acc        | 0.3200|±  |0.0469|
|                                                 |       |acc_norm   | 0.4100|±  |0.0494|
|hendrycksTest-miscellaneous                      |      0|acc        | 0.5299|±  |0.0463|
|                                                 |       |acc_norm   | 0.4872|±  |0.0464|
|hendrycksTest-moral_disputes                     |      0|acc        | 0.3248|±  |0.0435|
|                                                 |       |acc_norm   | 0.3162|±  |0.0432|
|hendrycksTest-moral_scenarios                    |      0|acc        | 0.3248|±  |0.0435|
|                                                 |       |acc_norm   | 0.2479|±  |0.0401|
|hendrycksTest-nutrition                          |      0|acc        | 0.3675|±  |0.0448|
|                                                 |       |acc_norm   | 0.3932|±  |0.0454|
|hendrycksTest-philosophy                         |      0|acc        | 0.2991|±  |0.0425|
|                                                 |       |acc_norm   | 0.3504|±  |0.0443|
|hendrycksTest-prehistory                         |      0|acc        | 0.2821|±  |0.0418|
|                                                 |       |acc_norm   | 0.3248|±  |0.0435|
|hendrycksTest-professional_accounting            |      0|acc        | 0.2137|±  |0.0381|
|                                                 |       |acc_norm   | 0.2222|±  |0.0386|
|hendrycksTest-professional_law                   |      0|acc        | 0.3077|±  |0.0429|
|                                                 |       |acc_norm   | 0.2735|±  |0.0414|
|hendrycksTest-professional_medicine              |      0|acc        | 0.2991|±  |0.0425|
|                                                 |       |acc_norm   | 0.2650|±  |0.0410|
|hendrycksTest-professional_psychology            |      0|acc        | 0.3248|±  |0.0435|
|                                                 |       |acc_norm   | 0.3419|±  |0.0440|
|hendrycksTest-public_relations                   |      0|acc        | 0.3909|±  |0.0467|
|                                                 |       |acc_norm   | 0.3545|±  |0.0458|
|hendrycksTest-security_studies                   |      0|acc        | 0.3419|±  |0.0440|
|                                                 |       |acc_norm   | 0.2906|±  |0.0422|
|hendrycksTest-sociology                          |      0|acc        | 0.3761|±  |0.0450|
|                                                 |       |acc_norm   | 0.3162|±  |0.0432|
|hendrycksTest-us_foreign_policy                  |      0|acc        | 0.5000|±  |0.0503|
|                                                 |       |acc_norm   | 0.4100|±  |0.0494|
|hendrycksTest-virology                           |      0|acc        | 0.3932|±  |0.0454|
|                                                 |       |acc_norm   | 0.3248|±  |0.0435|
|hendrycksTest-world_religions                    |      0|acc        | 0.5299|±  |0.0463|
|                                                 |       |acc_norm   | 0.5128|±  |0.0464|
|truthfulqa_gen                                   |      1|bleurt_max |-0.8551|±  |0.0501|
|                                                 |       |bleurt_acc | 0.3590|±  |0.0445|
|                                                 |       |bleurt_diff|-0.1292|±  |0.0483|
|                                                 |       |bleu_max   |19.3738|±  |1.8461|
|                                                 |       |bleu_acc   | 0.3932|±  |0.0454|
|                                                 |       |bleu_diff  |-4.3883|±  |2.1748|
|                                                 |       |rouge1_max |41.8428|±  |2.6156|
|                                                 |       |rouge1_acc | 0.3162|±  |0.0432|
|                                                 |       |rouge1_diff|-8.8583|±  |2.7745|
|                                                 |       |rouge2_max |26.3956|±  |2.8311|
|                                                 |       |rouge2_acc | 0.2137|±  |0.0381|
|                                                 |       |rouge2_diff|-9.5287|±  |3.3258|
|                                                 |       |rougeL_max |39.5215|±  |2.5620|
|                                                 |       |rougeL_acc | 0.3162|±  |0.0432|
|                                                 |       |rougeL_diff|-8.5753|±  |2.8259|