Anyone able to run this on vLLM ?

by xfalcox - opened Jul 24

Jul 24

This is using bitsandbytes 8 bit quantization that is broken on vLLM at the moment.

Any chance to release it using GTPQ or any fixes to vLLM incoming?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment