Hugging Face's logo

thorirhrafn
/
gpt7b_domar_pretuned

PEFT
TensorBoard
Safetensors
Generated from Trainer
Model card Files Metrics Community
thorirhrafn commited on
Commit
cdda574
·
verified ·
1 Parent(s): 2b4b008

End of training

Browse files
Files changed (1) hide show
  1. README.md +136 -0
README.md ADDED
@@ -0,0 +1,136 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ library_name: peft
4
+ tags:
5
+ - generated_from_trainer
6
+ base_model: AI-Sweden-Models/gpt-sw3-6.7b
7
+ model-index:
8
+ - name: gpt7b_domar_pretuned
9
+ results: []
10
+ ---
11
+
12
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
13
+ should probably proofread and complete it, then remove this comment. -->
14
+
15
+ # gpt7b_domar_pretuned
16
+
17
+ This model is a fine-tuned version of [AI-Sweden-Models/gpt-sw3-6.7b](https://huggingface.co/AI-Sweden-Models/gpt-sw3-6.7b) on an unknown dataset.
18
+ It achieves the following results on the evaluation set:
19
+ - Loss: 1.5652
20
+
21
+ ## Model description
22
+
23
+ More information needed
24
+
25
+ ## Intended uses & limitations
26
+
27
+ More information needed
28
+
29
+ ## Training and evaluation data
30
+
31
+ More information needed
32
+
33
+ ## Training procedure
34
+
35
+ ### Training hyperparameters
36
+
37
+ The following hyperparameters were used during training:
38
+ - learning_rate: 0.0001
39
+ - train_batch_size: 2
40
+ - eval_batch_size: 2
41
+ - seed: 42
42
+ - gradient_accumulation_steps: 4
43
+ - total_train_batch_size: 8
44
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
45
+ - lr_scheduler_type: linear
46
+ - num_epochs: 3
47
+
48
+ ### Training results
49
+
50
+ | Training Loss | Epoch | Step | Validation Loss |
51
+ |:-------------:|:-----:|:-----:|:---------------:|
52
+ | 1.688 | 0.04 | 500 | 1.6383 |
53
+ | 1.6062 | 0.08 | 1000 | 1.6253 |
54
+ | 1.8011 | 0.12 | 1500 | 1.6183 |
55
+ | 1.7644 | 0.16 | 2000 | 1.6124 |
56
+ | 1.6247 | 0.19 | 2500 | 1.6081 |
57
+ | 1.6269 | 0.23 | 3000 | 1.6039 |
58
+ | 1.6635 | 0.27 | 3500 | 1.6009 |
59
+ | 1.6432 | 0.31 | 4000 | 1.5995 |
60
+ | 1.5819 | 0.35 | 4500 | 1.5959 |
61
+ | 1.682 | 0.39 | 5000 | 1.5935 |
62
+ | 1.6599 | 0.43 | 5500 | 1.5914 |
63
+ | 1.6944 | 0.47 | 6000 | 1.5896 |
64
+ | 1.6737 | 0.51 | 6500 | 1.5884 |
65
+ | 1.6297 | 0.55 | 7000 | 1.5872 |
66
+ | 1.5836 | 0.58 | 7500 | 1.5851 |
67
+ | 1.5836 | 0.62 | 8000 | 1.5840 |
68
+ | 1.6493 | 0.66 | 8500 | 1.5822 |
69
+ | 1.6649 | 0.7 | 9000 | 1.5806 |
70
+ | 1.6896 | 0.74 | 9500 | 1.5798 |
71
+ | 1.5696 | 0.78 | 10000 | 1.5794 |
72
+ | 1.6178 | 0.82 | 10500 | 1.5782 |
73
+ | 1.6603 | 0.86 | 11000 | 1.5774 |
74
+ | 1.674 | 0.9 | 11500 | 1.5761 |
75
+ | 1.6157 | 0.94 | 12000 | 1.5750 |
76
+ | 1.6451 | 0.97 | 12500 | 1.5748 |
77
+ | 1.5914 | 1.01 | 13000 | 1.5746 |
78
+ | 1.5879 | 1.05 | 13500 | 1.5741 |
79
+ | 1.5722 | 1.09 | 14000 | 1.5736 |
80
+ | 1.5923 | 1.13 | 14500 | 1.5731 |
81
+ | 1.7271 | 1.17 | 15000 | 1.5721 |
82
+ | 1.6395 | 1.21 | 15500 | 1.5717 |
83
+ | 1.6165 | 1.25 | 16000 | 1.5711 |
84
+ | 1.6374 | 1.29 | 16500 | 1.5712 |
85
+ | 1.6012 | 1.32 | 17000 | 1.5711 |
86
+ | 1.6343 | 1.36 | 17500 | 1.5700 |
87
+ | 1.549 | 1.4 | 18000 | 1.5698 |
88
+ | 1.5665 | 1.44 | 18500 | 1.5693 |
89
+ | 1.5952 | 1.48 | 19000 | 1.5686 |
90
+ | 1.6008 | 1.52 | 19500 | 1.5686 |
91
+ | 1.6026 | 1.56 | 20000 | 1.5682 |
92
+ | 1.6111 | 1.6 | 20500 | 1.5682 |
93
+ | 1.6922 | 1.64 | 21000 | 1.5678 |
94
+ | 1.5795 | 1.68 | 21500 | 1.5678 |
95
+ | 1.5179 | 1.71 | 22000 | 1.5678 |
96
+ | 1.6446 | 1.75 | 22500 | 1.5671 |
97
+ | 1.6107 | 1.79 | 23000 | 1.5670 |
98
+ | 1.5963 | 1.83 | 23500 | 1.5666 |
99
+ | 1.6309 | 1.87 | 24000 | 1.5663 |
100
+ | 1.6062 | 1.91 | 24500 | 1.5666 |
101
+ | 1.6382 | 1.95 | 25000 | 1.5663 |
102
+ | 1.5587 | 1.99 | 25500 | 1.5659 |
103
+ | 1.6027 | 2.03 | 26000 | 1.5660 |
104
+ | 1.5118 | 2.07 | 26500 | 1.5659 |
105
+ | 1.5388 | 2.1 | 27000 | 1.5662 |
106
+ | 1.6093 | 2.14 | 27500 | 1.5660 |
107
+ | 1.5602 | 2.18 | 28000 | 1.5660 |
108
+ | 1.5446 | 2.22 | 28500 | 1.5659 |
109
+ | 1.6267 | 2.26 | 29000 | 1.5657 |
110
+ | 1.5805 | 2.3 | 29500 | 1.5657 |
111
+ | 1.6263 | 2.34 | 30000 | 1.5656 |
112
+ | 1.6197 | 2.38 | 30500 | 1.5654 |
113
+ | 1.6596 | 2.42 | 31000 | 1.5653 |
114
+ | 1.5682 | 2.46 | 31500 | 1.5654 |
115
+ | 1.638 | 2.49 | 32000 | 1.5654 |
116
+ | 1.6036 | 2.53 | 32500 | 1.5654 |
117
+ | 1.5977 | 2.57 | 33000 | 1.5652 |
118
+ | 1.593 | 2.61 | 33500 | 1.5652 |
119
+ | 1.635 | 2.65 | 34000 | 1.5652 |
120
+ | 1.578 | 2.69 | 34500 | 1.5653 |
121
+ | 1.5745 | 2.73 | 35000 | 1.5653 |
122
+ | 1.585 | 2.77 | 35500 | 1.5652 |
123
+ | 1.6069 | 2.81 | 36000 | 1.5652 |
124
+ | 1.6137 | 2.84 | 36500 | 1.5652 |
125
+ | 1.6472 | 2.88 | 37000 | 1.5652 |
126
+ | 1.6273 | 2.92 | 37500 | 1.5652 |
127
+ | 1.6552 | 2.96 | 38000 | 1.5652 |
128
+
129
+
130
+ ### Framework versions
131
+
132
+ - PEFT 0.8.2
133
+ - Transformers 4.38.1
134
+ - Pytorch 2.2.0+cu118
135
+ - Datasets 2.17.1
136
+ - Tokenizers 0.15.2