Not working in TG-webui with ExLlama_HF

#1
by cvinker - opened
2023-11-05 14:13:30 INFO:Loading TheBloke_deepseek-coder-33B-instruct-GPTQ_gptq-8bit-128g-actorder_True...
2023-11-05 14:13:46 ERROR:Failed to load the model.
Traceback (most recent call last):
  File "/workspace/text-generation-webui/modules/ui_model_menu.py", line 209, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(shared.model_name, loader)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/text-generation-webui/modules/models.py", line 84, in load_model
    output = load_func_map[loader](model_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/text-generation-webui/modules/models.py", line 348, in ExLlama_HF_loader
    return ExllamaHF.from_pretrained(model_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/text-generation-webui/modules/exllama_hf.py", line 174, in from_pretrained
    return ExllamaHF(config)
           ^^^^^^^^^^^^^^^^^
  File "/workspace/text-generation-webui/modules/exllama_hf.py", line 31, in __init__
    self.ex_model = ExLlama(self.ex_config)
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/text-generation-webui/installer_files/env/lib/python3.11/site-packages/exllama/model.py", line 889, in __init__
    layer = ExLlamaDecoderLayer(self.config, tensors, f"model.layers.{i}", i, sin, cos)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/text-generation-webui/installer_files/env/lib/python3.11/site-packages/exllama/model.py", line 517, in __init__
    self.self_attn = ExLlamaAttention(self.config, tensors, key + ".self_attn", sin, cos, self.index)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/text-generation-webui/installer_files/env/lib/python3.11/site-packages/exllama/model.py", line 304, in __init__
    self.q_proj = Ex4bitLinear(config, self.config.hidden_size, self.config.num_attention_heads * self.config.head_dim, False, tensors, key + ".q_proj")
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/text-generation-webui/installer_files/env/lib/python3.11/site-packages/exllama/model.py", line 154, in __init__
    self.q4 = cuda_ext.ext_make_q4(self.qweight,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/text-generation-webui/installer_files/env/lib/python3.11/site-packages/exllama/cuda_ext.py", line 33, in ext_make_q4
    return make_q4(qweight,
           ^^^^^^^^^^^^^^^^
RuntimeError: qweight and qzeros have incompatible shapes

ExLlama does not support 8-bit models - please see the table in the README which shows which files will or won't work with ExLlama

ExLlama does not support 8-bit models - please see the table in the README which shows which files will or won't work with ExLlama

Thank you, my bad, I have it working with Transformers.

Sign up or log in to comment