cannot load int4
#2
by
babyta
- opened
ValueError: .to is not supported for 4-bit or 8-bit bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct dtype.
Hi,
You need to pass device_map=auto to the from_pretrained method in case you pass a BitsAndBytesConfig.