How do I successfully merge adater weights to this base model correctly? And then siccessfulyl convert to GGUF
#94
by
uyiosa
- opened
This is my current work flow :
-Merge fine tuned apadter weights with base mistal model
-convert merge model to GGUF format using llama.cpp
For additional context:
- Repo containing fine-tuned model only contains the adapter_config.json and adapter_model.safetensors, The repo is named
Rmote6603/MedPrescription-FineTuning
*The python code I use to merge is attached to thread, this is most likely where I am doing something wrong I think.
*When the merged model is created in it's own directory it contains the following:
generation_config.json,
model.safetensors.index.json,
6 model-0000X-of-00006.safetensors (X represents numbers from 1 to 6)
*I had to manually add in the tokenizer.model
and tokenizer.json
from the base instruct model so could convert to GGUF with llama.cpp.
*I run python3 llama.cpp/convert-hf-to-gguf.py remote-hf --outfile remote2.gguf --outtype f16
.
Here is the merge code
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
peft_model_id = "Rmote6603/MedPrescription-FineTuning"
model = PeftModel.from_pretrained(base_model, peft_model_id)
merged_model = model.merge_and_unload()
local_save_path = "remote-hf"
# Save the merged model locally
merged_model.save_pretrained(local_save_path)
Does any part of my process look off? I'm getting wacky behaviors when I try to create the model in ollama
from the GGUF I get from this process.