Q8 quantized version?

by lrq3000 - opened Mar 27

Mar 27

Hello,

Thank you very much for making this model, it works very well!

However the Q4 quantized version seems quite a bit more limited compared to the results you posted, especially on reasoning abilities. Would it be possible for you to publish a Q8_0 quantized version please?

lamhieu

Ghost X org Mar 28

Of course, I will be exporting Q8 quantized version soon.

lamhieu

Ghost X org Mar 28

hi @lrq3000 , I created Q8 quantized version but if you want to using on vllm should be using awq version instead of.

lrq3000

Mar 29

Thank you so much! It is awesome! The Q8 version is so much more powerful in terms of reasoning abilities, thank you for generating it! Thank you also for the AWQ version, I will check it out if I want to host it :-)

lamhieu

Ghost X org Mar 29

Thanks @lrq3000 , I will soon release a new version that promises many surprising improvements. Follow me to receive information as soon as possible if you want.

lrq3000

Mar 29

Wow I am very eager to see the new release :D I have now subscribed, keep up your awesome work!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment