File size: 1,589 Bytes
df0614a 96917c1 df0614a 96917c1 beb591d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 |
---
license: apache-2.0
datasets:
- tatsu-lab/alpaca
---
This repo contains a low-rank adapter for LLaMA-7b
fit on the [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca) dataset.
This version of the weights was trained with the following hyperparameters:
- Epochs: 3 (load from best epoch)
- Batch size: 32
- Learning rate: 1e-4
- Lora _r_: 8
- lora_alpha : 16
- Lora target modules: q_proj, v_proj
That is:
```
python train_alpaca_lora.py \
--model_name_or_path decapoda-research/llama-7b-hf \
--data_path tatsu-lab/alpaca \
--output_dir work_dir_lora/ \
--num_train_epochs 3 \
--per_device_train_batch_size 4 \
--per_device_eval_batch_size 4 \
--gradient_accumulation_steps 8 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 500 \
--save_total_limit 5 \
--learning_rate 1e-4 \
--weight_decay 0. \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--model_max_length 2048 \
--logging_steps 1 \
--fp16 True
```
Instructions for running it can be found at https://github.com/jianzhnie/open-chatgpt.
### Citation
Please cite the repo if you use the data or code in this repo.
```
@misc{alpaca,
author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
title = {Stanford Alpaca: An Instruction-following LLaMA model},
year = {2023},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
}
``` |