turboderp

/

c4ai-command-r-08-2024-exl3

like 4

I've run into a compatibility issue on Oobabooga, commit 8984e95. The traceback is as follows:

File "A:\LLaMa\text-generation-webui\modules\ui_model_menu.py", line 174, in load_model_wrapper

shared.model, shared.tokenizer = load_model(selected_model, loader)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "A:\LLaMa\text-generation-webui\modules\models.py", line 43, in load_model

output = load_func_map[loader](model_name)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "A:\LLaMa\text-generation-webui\modules\models.py", line 90, in ExLlamav3_HF_loader

return Exllamav3HF.from_pretrained(model_name)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "A:\LLaMa\text-generation-webui\modules\exllamav3_hf.py", line 247, in from_pretrained

return Exllamav3HF(pretrained_model_name_or_path)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "A:\LLaMa\text-generation-webui\modules\exllamav3_hf.py", line 77, in init

self.ex_model.load(**load_params)

File "A:\LLaMa\text-generation-webui\installer_files\env\Lib\site-packages\torch\utils_contextlib.py", line 116, in decorate_context

return func(*args, **kwargs)
       ^^^^^^^^^^^^^^^^^^^^^

File "A:\LLaMa\text-generation-webui\installer_files\env\Lib\site-packages\exllamav3\models\model.py", line 361, in load

for _ in f: pass

File "A:\LLaMa\text-generation-webui\installer_files\env\Lib\site-packages\exllamav3\models\model.py", line 333, in load_gen

yield from self._load_autosplit(

File "A:\LLaMa\text-generation-webui\installer_files\env\Lib\site-packages\exllamav3\models\model.py", line 157, in _load_autosplit

module.load(load_device)

File "A:\LLaMa\text-generation-webui\installer_files\env\Lib\site-packages\exllamav3\modules\embedding.py", line 40, in load

weight = self.config.stc.get_tensor(self.key + ".weight", self.device)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "A:\LLaMa\text-generation-webui\installer_files\env\Lib\site-packages\exllamav3\loader\safetensors.py", line 218, in get_tensor

assert numel * esize == bytesize, \
       ^^^^^^^^^^^^^^^^^^^^^^^^^

AssertionError: Incorrect size of model.embed_tokens.weight in user_data\models\turboderp_c4ai-command-r-08-2024-exl3_4.0bpw\model-00001-of-00003.safetensors

I was unable to find any hits on the particular error on Google or on the Oobabooga Discord. I'm guessing that this has something to do with a mismatch between the actual weights and the model specifications as found in model.safetensors.index.json. Hopefully, this may be quantisation related and you may know what it's about. I'm happy to provide details about my setup if they are needed.