|
--- |
|
tags: |
|
- merge |
|
- gguf |
|
- llama3 |
|
- iMat |
|
license: llama3 |
|
--- |
|
<img src="https://i.imgur.com/P68dXux.png" width="400"/> |
|
|
|
# Llama3-ChatQA-1.5-8B-iMat-GGUF |
|
|
|
|
|
Quantized from fp16. |
|
* Weighted quantizations were creating using fp16 GGUF and [groups_merged-enhancedV2-TurboMini.txt](https://github.com/ggerganov/llama.cpp/discussions/5263#discussioncomment-9432658) in 234 chunks and n_ctx=512 |
|
* This method of calculating the importance matrix showed improvements in some areas for Mistral 7b and Llama3 8b models, see above post for details |
|
* The enhancedv2-turbomini file appends snippets from turboderp's calibration data to the standard groups_merged.txt file |
|
|
|
For a brief rundown of iMatrix quant performance please see this [PR](https://github.com/ggerganov/llama.cpp/pull/5747) |
|
|
|
<i>All quants are verified working prior to uploading to repo for your safety and convenience. </i> |
|
|
|
|
|
<b>Tip:</b> Pick a file size under your GPU's VRAM while still allowing some room for context for best speed. You may need to pad this further depending on if you are running image gen or TTS as well. |
|
|
|
BFloat16 model card can be found [here](https://huggingface.co/nvidia/Llama3-ChatQA-1.5-8B) |
