Given-MPT-7B

This is a merge of the following MPT-7B models:

• gorilla-llm/gorilla-mpt-7b-hf-v0
• ibm/mpt-7b-instruct2
• TehVenom/MPT-7b-WizardLM_Uncensored-Storywriter-Merge
• emozilla/mpt-7b-storysummarizer
• nomic-ai/gpt4all-mpt

Model License

Apache 2.0

Purpose

This model is for experimenting with merging and routing to expert layers.

Test eval on only 10% of eval set

hf-causal (pretrained=Multi-Domain-Expert-Layers/given-mpt-7b,dtype=bfloat16,trust_remote_code=True), limit: 0.1, provide_description: False, num_fewshot: 0, batch_size: None

Task	Version	Metric	Value		Stderr
arc_challenge	0	acc	0.4274	±	0.0459
		acc_norm	0.3846	±	0.0452
arc_easy	0	acc	0.7863	±	0.0381
		acc_norm	0.7350	±	0.0410
hellaswag	0	acc	0.5556	±	0.0461
		acc_norm	0.8120	±	0.0363
hendrycksTest-college_chemistry	0	acc	0.3600	±	0.0482
		acc_norm	0.3700	±	0.0485
hendrycksTest-college_computer_science	0	acc	0.3400	±	0.0476
		acc_norm	0.3600	±	0.0482
hendrycksTest-college_mathematics	0	acc	0.2500	±	0.0435
		acc_norm	0.2900	±	0.0456
hendrycksTest-college_medicine	0	acc	0.3675	±	0.0448
		acc_norm	0.3162	±	0.0432
hendrycksTest-college_physics	0	acc	0.2451	±	0.0428
		acc_norm	0.2941	±	0.0453
hendrycksTest-computer_security	0	acc	0.4800	±	0.0502
		acc_norm	0.4400	±	0.0499
hendrycksTest-conceptual_physics	0	acc	0.2051	±	0.0375
		acc_norm	0.1709	±	0.0350
hendrycksTest-econometrics	0	acc	0.2982	±	0.0430
		acc_norm	0.2368	±	0.0400
hendrycksTest-electrical_engineering	0	acc	0.3248	±	0.0435
		acc_norm	0.3590	±	0.0445
hendrycksTest-elementary_mathematics	0	acc	0.3333	±	0.0438
		acc_norm	0.3162	±	0.0432
hendrycksTest-formal_logic	0	acc	0.3077	±	0.0429
		acc_norm	0.3248	±	0.0435
hendrycksTest-global_facts	0	acc	0.3000	±	0.0461
		acc_norm	0.2700	±	0.0446
hendrycksTest-high_school_biology	0	acc	0.3675	±	0.0448
		acc_norm	0.3077	±	0.0429
hendrycksTest-high_school_chemistry	0	acc	0.2564	±	0.0405
		acc_norm	0.2906	±	0.0422
hendrycksTest-high_school_computer_science	0	acc	0.4100	±	0.0494
		acc_norm	0.4400	±	0.0499
hendrycksTest-high_school_european_history	0	acc	0.4359	±	0.0460
		acc_norm	0.3590	±	0.0445
hendrycksTest-high_school_geography	0	acc	0.3248	±	0.0435
		acc_norm	0.3675	±	0.0448
hendrycksTest-high_school_government_and_politics	0	acc	0.3932	±	0.0454
		acc_norm	0.3932	±	0.0454
hendrycksTest-high_school_macroeconomics	0	acc	0.3333	±	0.0438
		acc_norm	0.3248	±	0.0435
hendrycksTest-high_school_mathematics	0	acc	0.2051	±	0.0375
		acc_norm	0.2564	±	0.0405
hendrycksTest-high_school_microeconomics	0	acc	0.3504	±	0.0443
		acc_norm	0.4188	±	0.0458
hendrycksTest-high_school_physics	0	acc	0.2650	±	0.0410
		acc_norm	0.2906	±	0.0422
hendrycksTest-high_school_psychology	0	acc	0.3761	±	0.0450
		acc_norm	0.3419	±	0.0440
hendrycksTest-high_school_statistics	0	acc	0.3077	±	0.0429
		acc_norm	0.3504	±	0.0443
hendrycksTest-high_school_us_history	0	acc	0.3333	±	0.0438
		acc_norm	0.3333	±	0.0438
hendrycksTest-high_school_world_history	0	acc	0.3333	±	0.0438
		acc_norm	0.3419	±	0.0440
hendrycksTest-human_aging	0	acc	0.3761	±	0.0450
		acc_norm	0.3162	±	0.0432
hendrycksTest-human_sexuality	0	acc	0.4274	±	0.0459
		acc_norm	0.3761	±	0.0450
hendrycksTest-international_law	0	acc	0.4188	±	0.0458
		acc_norm	0.4957	±	0.0464
hendrycksTest-jurisprudence	0	acc	0.3148	±	0.0449
		acc_norm	0.4815	±	0.0483
hendrycksTest-logical_fallacies	0	acc	0.3504	±	0.0443
		acc_norm	0.3675	±	0.0448
hendrycksTest-machine_learning	0	acc	0.3214	±	0.0443
		acc_norm	0.2946	±	0.0433
hendrycksTest-management	0	acc	0.3786	±	0.0480
		acc_norm	0.3495	±	0.0472
hendrycksTest-marketing	0	acc	0.5043	±	0.0464
		acc_norm	0.4188	±	0.0458
hendrycksTest-medical_genetics	0	acc	0.3200	±	0.0469
		acc_norm	0.4100	±	0.0494
hendrycksTest-miscellaneous	0	acc	0.5299	±	0.0463
		acc_norm	0.4872	±	0.0464
hendrycksTest-moral_disputes	0	acc	0.3248	±	0.0435
		acc_norm	0.3162	±	0.0432
hendrycksTest-moral_scenarios	0	acc	0.3248	±	0.0435
		acc_norm	0.2479	±	0.0401
hendrycksTest-nutrition	0	acc	0.3675	±	0.0448
		acc_norm	0.3932	±	0.0454
hendrycksTest-philosophy	0	acc	0.2991	±	0.0425
		acc_norm	0.3504	±	0.0443
hendrycksTest-prehistory	0	acc	0.2821	±	0.0418
		acc_norm	0.3248	±	0.0435
hendrycksTest-professional_accounting	0	acc	0.2137	±	0.0381
		acc_norm	0.2222	±	0.0386
hendrycksTest-professional_law	0	acc	0.3077	±	0.0429
		acc_norm	0.2735	±	0.0414
hendrycksTest-professional_medicine	0	acc	0.2991	±	0.0425
		acc_norm	0.2650	±	0.0410
hendrycksTest-professional_psychology	0	acc	0.3248	±	0.0435
		acc_norm	0.3419	±	0.0440
hendrycksTest-public_relations	0	acc	0.3909	±	0.0467
		acc_norm	0.3545	±	0.0458
hendrycksTest-security_studies	0	acc	0.3419	±	0.0440
		acc_norm	0.2906	±	0.0422
hendrycksTest-sociology	0	acc	0.3761	±	0.0450
		acc_norm	0.3162	±	0.0432
hendrycksTest-us_foreign_policy	0	acc	0.5000	±	0.0503
		acc_norm	0.4100	±	0.0494
hendrycksTest-virology	0	acc	0.3932	±	0.0454
		acc_norm	0.3248	±	0.0435
hendrycksTest-world_religions	0	acc	0.5299	±	0.0463
		acc_norm	0.5128	±	0.0464
truthfulqa_gen	1	bleurt_max	-0.8551	±	0.0501
		bleurt_acc	0.3590	±	0.0445
		bleurt_diff	-0.1292	±	0.0483
		bleu_max	19.3738	±	1.8461
		bleu_acc	0.3932	±	0.0454
		bleu_diff	-4.3883	±	2.1748
		rouge1_max	41.8428	±	2.6156
		rouge1_acc	0.3162	±	0.0432
		rouge1_diff	-8.8583	±	2.7745
		rouge2_max	26.3956	±	2.8311
		rouge2_acc	0.2137	±	0.0381
		rouge2_diff	-9.5287	±	3.3258
		rougeL_max	39.5215	±	2.5620
		rougeL_acc	0.3162	±	0.0432
		rougeL_diff	-8.5753	±	2.8259