File size: 2,389 Bytes
73a3589 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 |
# phi-1_5-qlora-alpaca-instruction Model Card
## Model Description
This model is a causal language model based on the `microsoft/phi-1_5` and has been finetuned using QLORA technology on the `vicgalle/alpaca-gpt4` dataset.
## Fine-tuning Details
- **Base Model**: `microsoft/phi-1_5`
- **Fine-tuning Dataset**: `vicgalle/alpaca-gpt4`
- **Hardware**: NVIDIA 3090ti
- **Training Duration**: 8 hours
- **VRAM Consumption**: Approx. 20 GB for 14 hours
- **Token Max Length**: 2048
- **Model Size**: 1.5billion + qlora weights merged
### Hyperparameters
```python
# Lora Configuration
config = LoraConfig(
r=16,
lora_alpha=16,
target_modules=["Wqkv", "out_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
# Training Hyperparameters
training_arguments = TrainingArguments(
output_dir=f"{local_path}/output_dir",
per_device_train_batch_size=4,
gradient_accumulation_steps=6,
learning_rate=2e-4,
lr_scheduler_type="cosine",
evaluation_strategy = "steps",
eval_steps=500,
save_strategy="epoch",
logging_steps=100,
num_train_epochs=6,
report_to = 'wandb',
run_name = run_name
)
```
## Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "nps798/phi-1_5-qlora-alpaca-instruction"
model = AutoModelForCausalLM.from_pretrained(
model_name,
device_map={"": 0},
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
model_name,
trust_remote_code=True
)
prompt= """Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Choose three places you would like to visit and explain why.
### Response:"""
inputs = tokenizer(prompt, return_tensors="pt", return_attention_mask=False)
outputs = model.generate(**inputs, max_length=500)
text = tokenizer.batch_decode(outputs)[0]
print(text)
```
## License
Because the base model is microsoft phi-1.5b model, this fine-tuned model is provided under the MICROSOFT RESEARCH LICENSE and is meant for non-commercial use only.
## Author
I am a medical doctor interested in ML/NLP field.
If you have any advice, suggestions, or opportunities, or simply want to discuss the fascinating intersection of medicine and technology, please don't hesitate to reach out.
|