Request: 4.5bpw model
#1
by
TreesPlay
- opened
Hi, I'm interested in a 4.5bpw model. I used to run the 5bpw model for Mistral-Small-2409 but since Mistral-Small-2501 has about 1.5B more parameters I'd like to run it with 4.5bpw. I am using 2080 Ti gpus which gives about 22GB VRAM minus the overhead of splitting the model between gpus.
Absolutely I can run one of those for you. Expect it by tomorrow. I'll post it here when it's ready. https://discord.com/channels/1238219753324281886/1332443910559105146
sleepdeprived3
changed discussion status to
closed
sleepdeprived3
changed discussion status to
open
sleepdeprived3
changed discussion status to
closed