parsing/integer
Can i ignore the parsing warning? Will the model load properly at 3.75bpw or does the config indeed need to be adjusted to 4bits?
You can safely ignore it. Huggingface seems to think having a non-integer bit size in the config is an error, which it isn't.
Great thanks, up and running. Whilst i got your attention, can i ask for your recommended generation parameters, preset etc?
The model seems to work well with the Alpaca or base Mistral prompt format, and I'm not sure which is better. Assuming you're using SillyTavern, here's the Alpaca story string/instruct template and Mistral one I used to use. If you use the Alpaca one, it likes to spit out ### Input:at the end of a reply, so you might have to add that string to the stopping strings.
As for samplers, I've been experimenting with this new preset with all my models and it works really well. It's pretty minimal and uses the new DRY repetition penalty with a bit of min_p, but is feels much better than other samplers and the older repetition penalty. Feel free to adjust the temperature if you want it more or less creative (at the cost of coherency).