Hugging Face's logo

InferenceIllusionist
/
dolphin-2.9.1-mixtral-1x22b-iMat-GGUF

Transformers
GGUF
English
Generated from Trainer
axolotl
iMat
Inference Endpoints
imatrix
conversational
Model card Files Community
dolphin-2.9.1-mixtral-1x22b-iMat-GGUF / README.md
InferenceIllusionist's picture
InferenceIllusionist
Updated chunks size
80a45d6 verified
preview code
|
raw
history blame contribute delete
1.74 kB
metadata
base_model: cognitivecomputations/dolphin-2.9.1-mixtral-1x22b
datasets:
  - cognitivecomputations/Dolphin-2.9
  - teknium/OpenHermes-2.5
  - m-a-p/CodeFeedback-Filtered-Instruction
  - cognitivecomputations/dolphin-coder
  - cognitivecomputations/samantha-data
  - microsoft/orca-math-word-problems-200k
  - abacusai/SystemChat-1.1
  - Locutusque/function-calling-chatml
  - internlm/Agent-FLAN
language:
  - en
library_name: transformers
license: apache-2.0
quantized_by: InferenceIllusionist
tags:
  - generated_from_trainer
  - axolotl
  - iMat

dolphin-2.9.1-mixtral-1x22b-iMat-GGUF

Quantized from fp16.

  • Weighted quantizations were creating using fp16 GGUF and groups_merged-enhancedV2-TurboMini.txt in 228 chunks and n_ctx=512
  • This method of calculating the importance matrix showed improvements in some areas for Mistral 7b and Llama3 8b models, see above post for details
  • The enhancedv2-turbomini file appends snippets from turboderp's calibration data to the standard groups_merged.txt file
  • Repetition penalty 1.05-1.18 has worked well for these quants.

For a brief rundown of iMatrix quant performance please see this PR

All quants are verified working prior to uploading to repo for your safety and convenience.

Tip: Pick a file size under your GPU's VRAM while still allowing some room for context for best speed. You may need to pad this further depending on if you are running image gen or TTS as well.

Original model card can be found here