Quantized GGUF models
#4
by
MaziyarPanahi
- opened
Thanks for sharing this model with the community, I have quantized it in GGUF format if anyone is interested:
https://huggingface.co/MaziyarPanahi/merlinite-7b-GGUF
It's been quantized for a while already by our team: https://huggingface.co/ibm/merlinite-7b-GGUF
I did find that, but it only has Q4_K_M.gguf, I quantized it from 2 all the way to 8 bits. (for some hardware Q4 is heavy)