phi-1_5-qlora-alpaca-instruction Model Card

Model Description

This model is a causal language model based on the microsoft/phi-1_5 and has been finetuned using QLORA technology on the vicgalle/alpaca-gpt4 dataset.

Fine-tuning Details

Base Model: microsoft/phi-1_5
Fine-tuning Dataset: vicgalle/alpaca-gpt4
Hardware: NVIDIA 3090ti
Training Duration: 8 hours
VRAM Consumption: Approx. 20 GB for 14 hours
Token Max Length: 2048
Model Size: 1.5billion + qlora weights merged

Hyperparameters

# Lora Configuration
config = LoraConfig(
    r=16,
    lora_alpha=16,
    target_modules=["Wqkv", "out_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# Training Hyperparameters
training_arguments = TrainingArguments(
        output_dir=f"{local_path}/output_dir",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=6,
        learning_rate=2e-4,
        lr_scheduler_type="cosine",
        evaluation_strategy = "steps",
        eval_steps=500,
        save_strategy="epoch",
        logging_steps=100,
        num_train_epochs=6,
        report_to = 'wandb',
        run_name = run_name
    )

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "nps798/phi-1_5-qlora-alpaca-instruction"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map={"": 0},
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    trust_remote_code=True
)

prompt= """Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Choose three places you would like to visit and explain why.

### Response:"""
inputs = tokenizer(prompt, return_tensors="pt", return_attention_mask=False)
outputs = model.generate(**inputs, max_length=500)
text = tokenizer.batch_decode(outputs)[0]
print(text)

License

Because the base model is microsoft phi-1.5b model, this fine-tuned model is provided under the MICROSOFT RESEARCH LICENSE and is meant for non-commercial use only.

Author

I am a medical doctor interested in ML/NLP field. If you have any advice, suggestions, or opportunities, or simply want to discuss the fascinating intersection of medicine and technology, please don't hesitate to reach out.