metadata
license: apache-2.0
tags:
- not-for-all-audiences
- writing
- roleplay
- gguf
- gguf-imatrix
base_model:
- nakodanei/Blue-Orchid-2x7b
model_type: mixtral
quantized_by: Green-Sky
language:
- en
llama.cpp conversion of https://huggingface.co/nakodanei/Blue-Orchid-2x7b/
except for f16 and q8_0, every quant is using the merge.imatrix
merge.imatrix is a merge of kalomaze-group_10_merged.172chunks.imatrix and wiki.train.400chunks.imatrix, which took ~10min + ~20min to calulate on my machine.
full wiki.train would have taken 10h
for more info on imatrix handling see https://github.com/ggerganov/llama.cpp/pull/5302
ppl (512 wiki.test, 300chunks)
| quant | ppl (lower is better) |
|---|---|
| f16(baseline) | 5.8839 +/- 0.05173 |
| q8_0 | xxx |
| q5_k_m | xxx |
| q4_k_m | xxx |
| iq3_xxs | 6.1984 +/- 0.05475 |
| iq3_xxs(only-wiki) | 6.1796 +/- 0.05446 |
| iq3_xxs(only-kal) | 6.1984 +/- 0.05475 |
| q2_k | xxx |
| iq2_xs | xxx |
| iq2_xxs | xxx |