Not working in TG-webui with ExLlama_HF
#1
by
cvinker
- opened
2023-11-05 14:13:30 INFO:Loading TheBloke_deepseek-coder-33B-instruct-GPTQ_gptq-8bit-128g-actorder_True...
2023-11-05 14:13:46 ERROR:Failed to load the model.
Traceback (most recent call last):
File "/workspace/text-generation-webui/modules/ui_model_menu.py", line 209, in load_model_wrapper
shared.model, shared.tokenizer = load_model(shared.model_name, loader)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/text-generation-webui/modules/models.py", line 84, in load_model
output = load_func_map[loader](model_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/text-generation-webui/modules/models.py", line 348, in ExLlama_HF_loader
return ExllamaHF.from_pretrained(model_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/text-generation-webui/modules/exllama_hf.py", line 174, in from_pretrained
return ExllamaHF(config)
^^^^^^^^^^^^^^^^^
File "/workspace/text-generation-webui/modules/exllama_hf.py", line 31, in __init__
self.ex_model = ExLlama(self.ex_config)
^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/text-generation-webui/installer_files/env/lib/python3.11/site-packages/exllama/model.py", line 889, in __init__
layer = ExLlamaDecoderLayer(self.config, tensors, f"model.layers.{i}", i, sin, cos)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/text-generation-webui/installer_files/env/lib/python3.11/site-packages/exllama/model.py", line 517, in __init__
self.self_attn = ExLlamaAttention(self.config, tensors, key + ".self_attn", sin, cos, self.index)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/text-generation-webui/installer_files/env/lib/python3.11/site-packages/exllama/model.py", line 304, in __init__
self.q_proj = Ex4bitLinear(config, self.config.hidden_size, self.config.num_attention_heads * self.config.head_dim, False, tensors, key + ".q_proj")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/text-generation-webui/installer_files/env/lib/python3.11/site-packages/exllama/model.py", line 154, in __init__
self.q4 = cuda_ext.ext_make_q4(self.qweight,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/text-generation-webui/installer_files/env/lib/python3.11/site-packages/exllama/cuda_ext.py", line 33, in ext_make_q4
return make_q4(qweight,
^^^^^^^^^^^^^^^^
RuntimeError: qweight and qzeros have incompatible shapes
ExLlama does not support 8-bit models - please see the table in the README which shows which files will or won't work with ExLlama
ExLlama does not support 8-bit models - please see the table in the README which shows which files will or won't work with ExLlama
Thank you, my bad, I have it working with Transformers.