johaanm

/

quantized-french1

like 0

I had this error OSError: johaanm/llama2-quantized1 does not appear to have a file named config.json. Checkout 'https://huggingface.co/johaanm/llama2-quantized1/main' for available files. when trying to load the model. As I now understand, the model is not yet converted to huggingface format.

My code is as follows:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch


model_id = "johaanm/llama2-quantized1"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_8bit= False,
    load_in_4bit=True,
    lm_int8_threshold= 6.0,
    llm_int8_skip_modules= None,
    llm_int8_enable_fp32_cpu_offload= False,
    llm_int8_has_fp16_weight= False,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

text = "What is one plus one? "
device = "cuda:0"

inputs = tokenizer(text, return_tensors="pt").to(device)
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))