bartowski

/

Llama-3.3-70B-Instruct-GGUF

like 38

You're likely on a very only llama cpp version and need to update

What tool are you using?

I can get the Q2 model to run but I cannot get the Q6 or Q8 to run on latest text-generation-webui :

llama_load_model_from_file: failed to load model
14:57:30-007887 ERROR Failed to load the model.
Traceback (most recent call last):
File "/home/charlton/oobabooga/text-generation-webui/modules/ui_model_menu.py", line 232, in load_model_wrapper
shared.model, shared.tokenizer = load_model(selected_model, loader)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/modules/models.py", line 93, in load_model
output = load_func_maploader
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/modules/models.py", line 278, in llamacpp_loader
model, tokenizer = LlamaCppModel.from_pretrained(model_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/modules/llamacpp_model.py", line 85, in from_pretrained
result.model = Llama(**params)
^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/llama_cpp/llama.py", line 369, in init
internals.LlamaModel(
File "/home/charlton/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/llama_cpp/_internals.py", line 56, in init
raise ValueError(f"Failed to load model from file: {path_model}")
ValueError: Failed to load model from file: models/Llama-3.3-70B-Instruct-Q8_0-bartowsky.gguf

Exception ignored in: <function LlamaCppModel.__del__ at 0x7fea506e0040>
Traceback (most recent call last):
File "/home/charlton/oobabooga/text-generation-webui/modules/llamacpp_model.py", line 33, in del
del self.model
^^^^^^^^^^
AttributeError: 'LlamaCppModel' object has no attribute 'model'

This is probably because my files are split and then stored in folders, I need to open a PR for tgwui to support my format

How are you supposed to recombine them ? I cat them together like everyone else does.

You're likely on a very only llama cpp version and need to update

What tool are you using?

@charltonh don't cat them together, just point to part 1 and let the llama.cpp loader handle the rest

That worked! Thanks.