|
--- |
|
library_name: transformers |
|
tags: |
|
- trl |
|
- sft |
|
license: apache-2.0 |
|
language: |
|
- en |
|
base_model: |
|
- meta-llama/Llama-3.1-8B-Instruct |
|
pipeline_tag: text-generation |
|
--- |
|
----------------------------------------------------------------------------------------------------- |
|
**Remeber this model is for illustration and knowlwdge Purpose. I have only used online freely available materials in whole process.** |
|
|
|
## Model Details |
|
This Model is Trained on Custum data related to Sales interactive conversations as Array of objects having Instruction and Response as Keys. |
|
-**Parameters:** ~8 Billion |
|
-**Quantization:** 4 Bit (Q-LORA) |
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
This is the model card of a 🤗 transformers model that has been pushed on the Hub. |
|
|
|
- **Trained by:** [vakodiya] [Viru Akodiya] |
|
- **Model type:** [Text-Generation] |
|
- **License:** [apache-2.0] |
|
- **Finetuned from model:** [meta-llama/Llama-3.1-8B-Instruct] |
|
|
|
|
|
### Training Data |
|
|
|
Training Data is specifically generated by me to train to my use case. |
|
It consits of Just 500 examples, so to increase dataset size, duplicated the original data and makes it 1000. |
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
- **Hardware Type:** [Kaggle's GPU T4X2] |
|
- **Time used:** [37 Minutes] |
|
- **Cloud Provider:** [Kaggle] |
|
----------------------------------------------------------------------------------------------------------- |
|
|
|
## INFERENCE (It will need GPU) |
|
------------------------------------------------------------------------------------------------------------ |
|
|
|
# Install Dependencies |
|
``` |
|
%%capture |
|
!pip install transformers accelerate bitsandbytes |
|
``` |
|
``` |
|
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig, pipeline, AutoConfig |
|
import torch |
|
``` |
|
--------------------------------------------------------------------------------------------------------- |
|
# Load model and Tokenizer |
|
``` |
|
model_name = "vakodiya/Llama-3-8B-instruct-4bit-salesbot" |
|
config = AutoConfig.from_pretrained(model_name) |
|
bnb_config = BitsAndBytesConfig( |
|
load_in_4bit=True, |
|
bnb_4bit_quant_type="nf4", |
|
bnb_4bit_use_double_quant=True, |
|
) |
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
model = AutoModelForCausalLM.from_pretrained( |
|
model_name, |
|
quantization_config=bnb_config, |
|
device_map="auto", |
|
torch_dtype=torch.bfloat16, |
|
) |
|
# Model evaluation mode |
|
model.eval() |
|
``` |
|
------------------------------------------------------------------------------------------------------- |
|
|
|
# Creating Inference Point |
|
``` |
|
def Trained_Llama3_1_inference(prompt): |
|
model.eval() |
|
conversation=[ |
|
{"role": "user", "content": prompt}, |
|
] |
|
input_ids = tokenizer.apply_chat_template(conversation, add_generation_prompt=True, return_tensors="pt", padding=True, truncation=True, return_attention_mask=True) |
|
if input_ids.shape[1] > 8192: |
|
input_ids = input_ids[:, -8192:] |
|
return "Input tokens more than 8k" |
|
inputs = input_ids.to(model.device) |
|
attention_mask = torch.ones_like(inputs, dtype=torch.long) |
|
final_prompt=tokenizer.decode(inputs[0]) |
|
outputs = model.generate(inputs, max_new_tokens=256, temperature=0.4,attention_mask=attention_mask,pad_token_id=tokenizer.pad_token_id) |
|
response = tokenizer.decode(outputs[0]) |
|
final_response= response.replace(final_prompt,"").replace('<|eot_id|>',"") # Exclude prompt from response |
|
return final_response |
|
``` |
|
------------------------------------------------------------------------------------------------------------------------------- |
|
# Invoking Inference |
|
``` |
|
Trained_Llama3_1_inference("What are qualities of good Sales-person ?") |
|
``` |
|
----------End of Inferece -------------------- |
|
|
|
---------------------------------------------------------------------------------------------------------------------------------- |
|
|
|
---------- Start of Training ----------------- |
|
|
|
#### Training (on Kaggle Notebook) |
|
This training is done on Kaggle Notebook enabling GPU(Required in quantized training/ inference). |
|
|
|
# Install Dependencies |
|
``` |
|
%%capture |
|
!pip install -U transformers[torch] datasets |
|
!pip install -q bitsandbytes trl peft accelerate |
|
!pip install flash-attn --no-build-isolation |
|
!pip install huggingface_hub |
|
``` |
|
|
|
------------------------------------------------------------------------------------------------------------------------------------------ |
|
# Import Modules |
|
``` |
|
from transformers import BitsAndBytesConfig, AutoTokenizer, AutoModelForCausalLM, TrainingArguments |
|
from trl import SFTTrainer |
|
from peft import LoraConfig |
|
from huggingface_hub import notebook_login |
|
import torch |
|
from huggingface_hub import login |
|
from datasets import Dataset |
|
from kaggle_secrets import UserSecretsClient |
|
import os |
|
``` |
|
------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
|
# Remember to generate a Token with edit access on HuggingFace and add it as secret in Kaggle Notebook |
|
``` |
|
hf_token = UserSecretsClient().get_secret("HF_TOKEN_LLAMA3") |
|
login(token = hf_token) |
|
os.environ["CUDA_VISIBLE_DEVICES"] = "0" # Use only GPU 0 |
|
``` |
|
|
|
------------------------------------------------------------------------------------------------------------------------------------------- |
|
# Remember to Customize your own data with at least 1000 examples. |
|
``` |
|
Data_examples = [{"instruction":"Who has taken oath as Prime minister of India in 2024", "response":" Shri Narendra Modi has taken oath as Prime minister of india on 9th June 2024. He is now become prime minister having 3 consecutive terms."}, |
|
...................................................................................,] |
|
``` |
|
|
|
------------------------------------------------------------------------------------------------------------------------------------------ |
|
# Process data to stringify only the `text` field |
|
``` |
|
processed_data = [] |
|
for example in Data_examples : |
|
processed_data.append({'text':f"{example['instruction']} \n {example['response']}"}) |
|
|
|
# Create a Dataset from the list of dictionaries |
|
dataset = Dataset.from_list(processed_data) |
|
|
|
# Split into train and test Data sets |
|
|
|
dataset = dataset.train_test_split(test_size=0.01) |
|
# Access train and test splits |
|
|
|
train_dataset = dataset['train'] |
|
test_dataset = dataset['test'] |
|
``` |
|
--------------------------------------------------------------------------------------------------------------------------------------- |
|
|
|
# Firstly add model to Kaggle notebook navigating to Add Input and Add LLama3.1 8 B in out Notebook |
|
|
|
``` |
|
model_path="/kaggle/input/llama-3.1/transformers/8b-instruct/2" # Change it according to your model path in Notebook |
|
trained_model_name = "Llama-3-8B-instruct-4bit-finetuned" |
|
output_dir = 'kaggle/working/' + trained_model_name |
|
``` |
|
---------------------------------------------------------------------------------------------------------------------------------------- |
|
## For 4 bit quantization (Q-LoRA) set Configs |
|
``` |
|
quantization_config = BitsAndBytesConfig( |
|
load_in_4bit=True, |
|
bnb_4bit_use_double_quant=True, |
|
bnb_4bit_quant_type="nf4", |
|
bnb_4bit_compute_dtype=torch.bfloat16,) |
|
|
|
peft_config = LoraConfig( |
|
r=16, |
|
lora_alpha=16, |
|
lora_dropout=0.1, |
|
bias="none", |
|
task_type="CAUSAL_LM", |
|
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], |
|
) |
|
``` |
|
----------------------------------------------------------------------------------------------------------------------------------------- |
|
# Load the Model and Tokenizer and set pad token |
|
``` |
|
tokenizer = AutoTokenizer.from_pretrained(model_path) |
|
model = AutoModelForCausalLM.from_pretrained( |
|
model_path, |
|
quantization_config=quantization_config, |
|
device_map="auto") |
|
|
|
# Use eos_token as pad_token |
|
tokenizer.pad_token = tokenizer.eos_token |
|
``` |
|
|
|
----------------------------------------------------------------------------------------------------------------------------------------- |
|
# Set Training configurations |
|
``` |
|
training_args = TrainingArguments( |
|
fp16=False, # specify bf16=True instead when training on GPUs that support bf16 else fp16 |
|
bf16=True, |
|
do_eval=True, |
|
eval_strategy="epoch", |
|
gradient_accumulation_steps=4, |
|
gradient_checkpointing=True, |
|
gradient_checkpointing_kwargs={"use_reentrant": False}, |
|
learning_rate=2.0e-05, |
|
log_level="info", |
|
logging_steps=5, |
|
logging_strategy="steps", |
|
lr_scheduler_type="cosine", |
|
max_steps=-1, |
|
num_train_epochs=1, # Number of times training will go through with same dataset. |
|
output_dir=output_dir, |
|
overwrite_output_dir=True, |
|
per_device_eval_batch_size=8, # You can reduce if out-of memory errors occurs |
|
per_device_train_batch_size=8, # You can reduce if out-of memory errors occurs |
|
report_to="none", # for skipping wandb logging |
|
save_strategy="no", |
|
save_total_limit=None, |
|
) |
|
``` |
|
-------------------------------------------------------------------------------------------------------------------------------------------- |
|
# Set-up Trainer (Supervised-fine-tuning) |
|
``` |
|
trainer = SFTTrainer( |
|
model=model, # Use above quantized model |
|
args=training_args, |
|
train_dataset=train_dataset, # If Training Fails Try to reduce Dataset Size |
|
eval_dataset=test_dataset, |
|
dataset_text_field="text", |
|
tokenizer=tokenizer, |
|
packing=False, # Setting it True will Reduce dataset size as it will exclude similar examples occuring repetitive |
|
peft_config=peft_config, |
|
max_seq_length=1024, |
|
) |
|
``` |
|
------------------------------------------------------------------------------------------------------------------------------------------------- |
|
# Note: It may take long Time to train model (several minutes to Hours) depending on your dataset size |
|
``` |
|
# To clear out cache for unsuccessful run |
|
torch.cuda.empty_cache() |
|
|
|
train_result = trainer.train() |
|
``` |
|
------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
|
# Save model in Notebook (in output_directory) |
|
``` |
|
trainer.save_model() |
|
``` |
|
------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
|
# Merge LoRA with the base model and save the merged model |
|
``` |
|
merged_model = trainer.model.merge_and_unload() |
|
merged_model.save_pretrained("merged_model",safe_serialization=True) |
|
tokenizer.save_pretrained("merged_model") |
|
``` |
|
--------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
|
# push merged model to the HuggingFace-hub (You must have logged in already) |
|
``` |
|
merged_model.push_to_hub("username/model_name") |
|
tokenizer.push_to_hub("username/model_name") |
|
``` |
|
------------------- End of Training and uploading trained model on our huggingface Space ---------------------------------- |
