File size: 4,250 Bytes
f31a78d 73a3589 6f95c66 73a3589 effd735 23ee87a effd735 73a3589 6f95c66 f31a78d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 |
---
language:
- en
license: other
datasets:
- vicgalle/alpaca-gpt4
---
# phi-1_5-qlora-alpaca-instruction Model Card
## Model Description
This model is a causal language model based on the `microsoft/phi-1_5` and has been finetuned using QLORA technology on the `vicgalle/alpaca-gpt4` dataset.
## Fine-tuning Details
- **Base Model**: `microsoft/phi-1_5`
- **Fine-tuning Dataset**: `vicgalle/alpaca-gpt4`
- **Hardware**: NVIDIA 3090ti
- **Training Duration**: 14 hours
- **VRAM Consumption**: Approx. 20 GB
- **Token Max Length**: 2048
- **Model Size**: 1.5billion + qlora weights merged
### Hyperparameters
```python
# Lora Configuration
config = LoraConfig(
r=16,
lora_alpha=16,
target_modules=["Wqkv", "out_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
# Training Hyperparameters
training_arguments = TrainingArguments(
output_dir=f"{local_path}/output_dir",
per_device_train_batch_size=4,
gradient_accumulation_steps=6,
learning_rate=2e-4,
lr_scheduler_type="cosine",
evaluation_strategy = "steps",
eval_steps=500,
save_strategy="epoch",
logging_steps=100,
num_train_epochs=6,
report_to = 'wandb',
run_name = run_name
)
```
## Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "nps798/phi-1_5-qlora-alpaca-instruction"
model = AutoModelForCausalLM.from_pretrained(
model_name,
device_map={"": 0},
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
model_name,
trust_remote_code=True
)
prompt= """Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Choose three places you would like to visit and explain why.
### Response:"""
inputs = tokenizer(prompt, return_tensors="pt", return_attention_mask=False)
outputs = model.generate(**inputs, max_length=500)
text = tokenizer.batch_decode(outputs)[0]
print(text)
```
## Example
| Question | Original phi 1.5 | This fine-tuned model |
|----------|----------|----------|
| I'm planning to visit Taiwan soon. Can you recommend some local festivals or cultural events that I shouldn't miss? | There are many festivals and cultural events in Taiwan that you should definitely check out. Some popular ones include the Dragon Boat Festival, the Mid-Autumn Festival, and the Dragon Boat Festival Parade. You can also visit the Taiwan Museum of Natural History to learn more about the country's rich history and culture. | 1. Taipei City: This is the capital of Taiwan and is known for its modern architecture, shopping, and nightlife.2. Taipei Zoo: This is one of the largest zoos in the world, and it's home to a wide variety of animals, including tigers, elephants, and gorillas.3. Taipei Gardens: This is a large botanical garden that features a wide variety of plants, including tropical flowers, shrubs, and trees.4. Taipei Park: This is a large park that features a variety of attractions, including a zoo, a botanical garden, and a zoo.5. Taipei Aquarium: This is one of the largest aquariums in the world, and it features a wide variety of marine animals, including dolphins, whales, and sharks.6. Taipei International Airport: This is the largest airport in the world, and it's home to a wide variety of airlines, including American, British, and Japanese.7. Taipei International Museum: This is one of the largest museums in the world, and it features a wide variety of exhibits, including art, history, and science.8. Taipei International School: This is one of the largest schools in the world, and it's home to a wide variety of students, including children, teenagers, and adults. |
| Row2Col1 | Row2Col2 | Row2Col3 |
| Row3Col1 | Row3Col2 | Row3Col3 |
## License
Because the base model is microsoft phi-1.5b model, this fine-tuned model is provided under the MICROSOFT RESEARCH LICENSE and is meant for non-commercial use only.
## Author
I am a medical doctor interested in ML/NLP field.
If you have any advice, suggestions, or opportunities, or simply want to discuss the fascinating intersection of medicine and technology, please don't hesitate to reach out.
|