Update README.md
Browse files
README.md
CHANGED
@@ -7,6 +7,8 @@ language:
|
|
7 |
base_model: mistralai/Mistral-7B-v0.1
|
8 |
pipeline_tag: text-generation
|
9 |
---
|
|
|
|
|
10 |
# zephyr-7b-sft-full-spin-iter1
|
11 |
|
12 |
This model is a self-play fine-tuned model at iteration 1 from [alignment-handbook/zephyr-7b-sft-full](https://huggingface.co/alignment-handbook/zephyr-7b-sft-full) using synthetic data based on on the [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) dataset.
|
@@ -23,16 +25,16 @@ This model is a self-play fine-tuned model at iteration 1 from [alignment-handbo
|
|
23 |
### Training hyperparameters
|
24 |
The following hyperparameters were used during training:
|
25 |
|
26 |
-
learning_rate: 5e-07
|
27 |
-
train_batch_size: 8
|
28 |
-
seed: 42
|
29 |
-
distributed_type: multi-GPU
|
30 |
-
num_devices: 8
|
31 |
-
total_train_batch_size: 64
|
32 |
-
optimizer: RMSProp
|
33 |
-
lr_scheduler_type: linear
|
34 |
-
lr_scheduler_warmup_ratio: 0.1
|
35 |
-
num_epochs: 2.0
|
36 |
|
37 |
## Citation
|
38 |
```
|
|
|
7 |
base_model: mistralai/Mistral-7B-v0.1
|
8 |
pipeline_tag: text-generation
|
9 |
---
|
10 |
+
see our paper in https://arxiv.org/abs/2401.01335
|
11 |
+
|
12 |
# zephyr-7b-sft-full-spin-iter1
|
13 |
|
14 |
This model is a self-play fine-tuned model at iteration 1 from [alignment-handbook/zephyr-7b-sft-full](https://huggingface.co/alignment-handbook/zephyr-7b-sft-full) using synthetic data based on on the [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) dataset.
|
|
|
25 |
### Training hyperparameters
|
26 |
The following hyperparameters were used during training:
|
27 |
|
28 |
+
- learning_rate: 5e-07
|
29 |
+
- train_batch_size: 8
|
30 |
+
- seed: 42
|
31 |
+
- distributed_type: multi-GPU
|
32 |
+
- num_devices: 8
|
33 |
+
- total_train_batch_size: 64
|
34 |
+
- optimizer: RMSProp
|
35 |
+
- lr_scheduler_type: linear
|
36 |
+
- lr_scheduler_warmup_ratio: 0.1
|
37 |
+
- num_epochs: 2.0
|
38 |
|
39 |
## Citation
|
40 |
```
|