File size: 2,036 Bytes
0bf81ad c8cb8d7 cccb915 0bf81ad c8cb8d7 0bf81ad c8cb8d7 0bf81ad c8cb8d7 0bf81ad c8cb8d7 0bf81ad c8cb8d7 231a56c 0bf81ad 231a56c c8cb8d7 0bf81ad c8cb8d7 0bf81ad c8cb8d7 0bf81ad c8cb8d7 0bf81ad c8cb8d7 0bf81ad c8cb8d7 0bf81ad c8cb8d7 41db283 c8cb8d7 41db283 c8cb8d7 0bf81ad c8cb8d7 0bf81ad c8cb8d7 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 |
---
library_name: peft
tags:
- generated_from_trainer
base_model: meta-llama/Meta-Llama-3-70B-Instruct
model-index:
- name: lora_Meta-Llama-3-70B_derta
results: []
license: apache-2.0
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# lora_Meta-Llama-3-70B_derta
This model is a fine-tuned version of [meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) on the [Evol-Instruct](https://huggingface.co/datasets/WizardLMTeam/WizardLM_evol_instruct_70k) and [BeaverTails](https://huggingface.co/datasets/PKU-Alignment/BeaverTails) dataset.
## Model description
Please refer to the paper [Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training](https://arxiv.org/abs/2407.09121) and GitHub [DeRTa](https://github.com/RobustNLP/DeRTa).
The model is continued train 100 steps with DeRTa on LLaMA3-70B-Instruct.
Input format:
```
[INST] Your Instruction [\INST]
```
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 8
- eval_batch_size: 1
- seed: 1
- distributed_type: multi-GPU
- num_devices: 8
- gradient_accumulation_steps: 2
- total_train_batch_size: 128
- total_eval_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- num_epochs: 2.0
The lora config is:
```
{
"lora_r": 96,
"lora_alpha": 16,
"lora_dropout": 0.05,
"lora_target_modules": [
"q_proj",
"v_proj",
"k_proj",
"o_proj",
"gate_proj",
"down_proj",
"up_proj",
"w1",
"w2",
"w3"
]
}
```
### Training results
### Framework versions
- PEFT 0.10.0
- Transformers 4.40.0
- Pytorch 2.2.0+cu118
- Datasets 2.10.0
- Tokenizers 0.19.1 |