End of training

README.md

@@ -16,14 +16,14 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
 The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 
 It achieves the following results on the evaluation set:
-- eval_enwikippl: 96.0
-- eval_frwikippl: 372.0
-- eval_zhwikippl: 128.0
-- eval_tinystoriesppl: 74.5
-- eval_loss: 0.7694
-- eval_runtime: 25.5363
-- eval_samples_per_second: 97.9
-- eval_steps_per_second: 12.257
+- eval_enwikippl: 216.0
+- eval_frwikippl: 876.0
+- eval_zhwikippl: 173.0
+- eval_tinystoriesppl: 178.0
+- eval_loss: 1.2693
+- eval_runtime: 25.4344
+- eval_samples_per_second: 98.292
+- eval_steps_per_second: 12.306
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -53,8 +53,8 @@ The following hyperparameters were used during training:
 - eval_batch_size: 8
 - seed: 42
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: constant
-- lr_scheduler_warmup_ratio: 0.2
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_ratio: 0.5
 - num_epochs: 1.0
 
 ### Resource Usage
@@ -64,69 +64,69 @@ Peak GPU Memory: 7.2012 GB
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | **teacher eval** |  | 43.25 | 61.25 |  |  |  |  | 11.6875 | 19.125 |
-| 0 | 0 | 1752346656768.0 | 132491151147008.0 | 20.4918 | 25.5082 | 98.008 | 12.271 | 5335154688.0 | 43705587204096.0 |
-| 1000 | 0.0162 | 274.0 | 1200.0 | 1.4194 | 25.4948 | 98.059 | 12.277 | 220.0 | 184.0 |
-| 2000 | 0.0323 | 200.0 | 680.0 | 1.2412 | 25.5013 | 98.034 | 12.274 | 163.0 | 151.0 |
-| 3000 | 0.0485 | 165.0 | 644.0 | 1.1185 | 25.5706 | 97.769 | 12.241 | 138.0 | 149.0 |
-| 4000 | 0.0646 | 146.0 | 576.0 | 1.0289 | 25.4758 | 98.132 | 12.286 | 114.5 | 134.0 |
-| 5000 | 0.0808 | 131.0 | 520.0 | 0.9689 | 25.4905 | 98.076 | 12.279 | 100.5 | 139.0 |
-| 6000 | 0.0970 | 117.0 | 456.0 | 0.9014 | 25.4949 | 98.059 | 12.277 | 92.5 | 140.0 |
-| 7000 | 0.1131 | 109.5 | 412.0 | 0.8654 | 25.5404 | 97.884 | 12.255 | 88.0 | 148.0 |
-| 8000 | 0.1293 | 104.0 | 414.0 | 0.8141 | 25.5188 | 97.967 | 12.265 | 80.0 | 131.0 |
-| 9000 | 0.1455 | 96.0 | 372.0 | 0.7694 | 25.5363 | 97.9 | 12.257 | 74.5 | 128.0 |
-| 10000 | 0.1616 | 90.0 | 372.0 | 0.7269 | 25.4962 | 98.054 | 12.276 | 71.5 | 125.0 |
-| 11000 | 0.1778 | 88.0 | 336.0 | 0.6943 | 25.4752 | 98.135 | 12.286 | 69.5 | 122.0 |
-| 12000 | 0.1939 | 84.5 | 336.0 | 0.6694 | 25.4772 | 98.127 | 12.285 | 66.5 | 134.0 |
-| 13000 | 0.2101 | 79.5 | 284.0 | 0.6458 | 25.4687 | 98.16 | 12.29 | 65.0 | 150.0 |
-| 14000 | 0.2263 | 78.5 | 298.0 | 0.6230 | 25.5385 | 97.891 | 12.256 | 62.75 | 150.0 |
-| 15000 | 0.2424 | 75.5 | 272.0 | 0.6116 | 25.5112 | 97.996 | 12.269 | 59.75 | 135.0 |
-| 16000 | 0.2586 | 76.0 | 260.0 | 0.6013 | 25.5149 | 97.982 | 12.267 | 59.0 | 162.0 |
-| 17000 | 0.2747 | 75.0 | 284.0 | 0.5878 | 25.5099 | 98.001 | 12.27 | 60.75 | 114.5 |
-| 18000 | 0.2909 | 74.5 | 260.0 | 0.5739 | 25.5029 | 98.028 | 12.273 | 60.75 | 141.0 |
-| 19000 | 0.3071 | 72.0 | 282.0 | 0.5736 | 25.5176 | 97.972 | 12.266 | 57.5 | 135.0 |
-| 20000 | 0.3232 | 72.0 | 264.0 | 0.5581 | 25.5059 | 98.016 | 12.272 | 55.5 | 129.0 |
-| 21000 | 0.3394 | 72.0 | 262.0 | 0.5565 | 25.5747 | 97.753 | 12.239 | 56.25 | 136.0 |
-| 22000 | 0.3556 | 73.5 | 252.0 | 0.5476 | 25.5063 | 98.015 | 12.271 | 54.25 | 116.5 |
-| 23000 | 0.3717 | 71.5 | 252.0 | 0.5455 | 25.4614 | 98.188 | 12.293 | 52.0 | 124.5 |
-| 24000 | 0.3879 | 72.0 | 260.0 | 0.5401 | 25.4593 | 98.196 | 12.294 | 53.5 | 136.0 |
-| 25000 | 0.4040 | 71.0 | 244.0 | 0.5351 | 25.4834 | 98.103 | 12.283 | 53.25 | 107.5 |
-| 26000 | 0.4202 | 68.5 | 227.0 | 0.5321 | 25.4557 | 98.21 | 12.296 | 51.25 | 94.0 |
-| 27000 | 0.4364 | 73.0 | 235.0 | 0.5286 | 25.4824 | 98.107 | 12.283 | 51.75 | 115.5 |
-| 28000 | 0.4525 | 70.0 | 248.0 | 0.5310 | 25.5236 | 97.949 | 12.263 | 50.5 | 117.5 |
-| 29000 | 0.4687 | 70.5 | 241.0 | 0.5233 | 25.4617 | 98.187 | 12.293 | 49.25 | 171.0 |
-| 30000 | 0.4848 | 70.0 | 228.0 | 0.5170 | 25.5019 | 98.032 | 12.274 | 54.0 | 106.0 |
-| 31000 | 0.5010 | 69.0 | 253.0 | 0.5187 | 25.5293 | 97.927 | 12.26 | 51.5 | 113.0 |
-| 32000 | 0.5172 | 73.5 | 243.0 | 0.5160 | 25.5033 | 98.027 | 12.273 | 53.5 | 85.0 |
-| 33000 | 0.5333 | 75.0 | 264.0 | 0.5181 | 25.516 | 97.978 | 12.267 | 52.5 | 87.5 |
-| 34000 | 0.5495 | 71.0 | 228.0 | 0.5141 | 25.4586 | 98.198 | 12.294 | 51.75 | 128.0 |
-| 35000 | 0.5657 | 69.5 | 241.0 | 0.5159 | 25.4876 | 98.087 | 12.28 | 52.0 | 142.0 |
-| 36000 | 0.5818 | 69.0 | 254.0 | 0.5107 | 25.497 | 98.051 | 12.276 | 52.75 | 111.0 |
-| 37000 | 0.5980 | 69.0 | 225.0 | 0.5010 | 25.5451 | 97.866 | 12.253 | 52.5 | 145.0 |
-| 38000 | 0.6141 | 70.0 | 220.0 | 0.5100 | 25.529 | 97.928 | 12.261 | 51.5 | 516.0 |
-| 39000 | 0.6303 | 67.0 | 235.0 | 0.5033 | 25.495 | 98.058 | 12.277 | 51.0 | 102.0 |
-| 40000 | 0.6465 | 67.5 | 231.0 | 0.5040 | 25.4679 | 98.163 | 12.29 | 48.5 | 95.0 |
-| 41000 | 0.6626 | 68.5 | 217.0 | 0.4893 | 25.4787 | 98.121 | 12.285 | 51.25 | 92.5 |
-| 42000 | 0.6788 | 68.5 | 211.0 | 0.4965 | 25.4674 | 98.165 | 12.29 | 50.5 | 143.0 |
-| 43000 | 0.6949 | 68.0 | 220.0 | 0.4998 | 25.4714 | 98.149 | 12.288 | 50.25 | 120.0 |
-| 44000 | 0.7111 | 69.5 | 224.0 | 0.4985 | 25.527 | 97.936 | 12.262 | 51.0 | 109.5 |
-| 45000 | 0.7273 | 69.0 | 230.0 | 0.5052 | 25.4809 | 98.113 | 12.284 | 50.75 | 109.5 |
-| 46000 | 0.7434 | 67.5 | 221.0 | 0.4921 | 25.4876 | 98.087 | 12.28 | 49.25 | 92.0 |
-| 47000 | 0.7596 | 69.0 | 230.0 | 0.4942 | 25.4768 | 98.128 | 12.286 | 49.5 | 164.0 |
-| 48000 | 0.7758 | 67.5 | 219.0 | 0.4892 | 25.4923 | 98.069 | 12.278 | 52.25 | 88.0 |
-| 49000 | 0.7919 | 70.0 | 247.0 | 0.4904 | 25.485 | 98.097 | 12.282 | 49.0 | 126.5 |
-| 50000 | 0.8081 | 66.0 | 219.0 | 0.4852 | 25.4685 | 98.161 | 12.29 | 48.0 | 120.0 |
-| 51000 | 0.8242 | 67.5 | 230.0 | 0.4926 | 25.5059 | 98.017 | 12.272 | 47.75 | 99.0 |
-| 52000 | 0.8404 | 67.5 | 224.0 | 0.4838 | 25.5222 | 97.954 | 12.264 | 50.25 | 144.0 |
-| 53000 | 0.8566 | 67.5 | 213.0 | 0.4853 | 25.5206 | 97.96 | 12.265 | 48.75 | 114.0 |
-| 54000 | 0.8727 | 64.5 | 228.0 | 0.4889 | 25.494 | 98.062 | 12.277 | 49.5 | 116.0 |
-| 55000 | 0.8889 | 65.5 | 231.0 | 0.4819 | 25.5118 | 97.994 | 12.269 | 49.25 | 113.5 |
-| 56000 | 0.9051 | 67.0 | 223.0 | 0.4872 | 25.4843 | 98.1 | 12.282 | 48.0 | 104.0 |
-| 57000 | 0.9212 | 68.5 | 234.0 | 0.4875 | 25.4685 | 98.16 | 12.29 | 49.75 | 131.0 |
-| 58000 | 0.9374 | 65.0 | 222.0 | 0.4854 | 25.4413 | 98.266 | 12.303 | 46.75 | 132.0 |
-| 59000 | 0.9535 | 65.5 | 211.0 | 0.4775 | 25.4688 | 98.159 | 12.29 | 49.5 | 132.0 |
-| 60000 | 0.9697 | 67.0 | 221.0 | 0.4806 | 25.4434 | 98.257 | 12.302 | 49.0 | 106.5 |
-| 61000 | 0.9859 | 65.5 | 216.0 | 0.4853 | 25.4536 | 98.218 | 12.297 | 50.25 | 109.5 |
-| 61875 | 1.0 | 62.75 | 222.0 | 0.4830 | 25.7335 | 97.15 | 12.163 | 49.25 | 204.0 |
+| 0 | 0 | 1606317768704.0 | 119297011613696.0 | 20.7344 | 25.4007 | 98.423 | 12.323 | 5066719232.0 | 19104014532608.0 |
+| 1000 | 0.0162 | 81408.0 | 958464.0 | 5.1263 | 25.4848 | 98.098 | 12.282 | 16384.0 | 1253376.0 |
+| 2000 | 0.0323 | 4032.0 | 47360.0 | 3.2154 | 25.5066 | 98.014 | 12.271 | 2368.0 | 256000.0 |
+| 3000 | 0.0485 | 1104.0 | 5792.0 | 2.3506 | 25.4684 | 98.161 | 12.29 | 724.0 | 8960.0 |
+| 4000 | 0.0646 | 632.0 | 3984.0 | 1.9871 | 25.4984 | 98.046 | 12.275 | 410.0 | 2368.0 |
+| 5000 | 0.0808 | 418.0 | 2096.0 | 1.6792 | 25.4486 | 98.237 | 12.299 | 310.0 | 572.0 |
+| 6000 | 0.0970 | 310.0 | 1384.0 | 1.5115 | 25.5353 | 97.904 | 12.258 | 241.0 | 284.0 |
+| 7000 | 0.1131 | 256.0 | 1040.0 | 1.3917 | 25.5364 | 97.899 | 12.257 | 203.0 | 205.0 |
+| 8000 | 0.1293 | 229.0 | 892.0 | 1.3296 | 25.4687 | 98.16 | 12.29 | 183.0 | 224.0 |
+| 9000 | 0.1455 | 216.0 | 876.0 | 1.2693 | 25.4344 | 98.292 | 12.306 | 178.0 | 173.0 |
+| 10000 | 0.1616 | 192.0 | 716.0 | 1.1774 | 25.4691 | 98.158 | 12.289 | 165.0 | 198.0 |
+| 11000 | 0.1778 | 255.0 | 824.0 | 1.0750 | 25.5641 | 97.793 | 12.244 | 296.0 | 169.0 |
+| 12000 | 0.1939 | 264.0 | 744.0 | 0.9937 | 25.5508 | 97.844 | 12.25 | 292.0 | 147.0 |
+| 13000 | 0.2101 | 231.0 | 608.0 | 0.9388 | 25.6185 | 97.586 | 12.218 | 192.0 | 158.0 |
+| 14000 | 0.2263 | 221.0 | 552.0 | 0.8904 | 25.5619 | 97.802 | 12.245 | 178.0 | 127.5 |
+| 15000 | 0.2424 | 221.0 | 608.0 | 0.8620 | 25.5139 | 97.986 | 12.268 | 145.0 | 128.0 |
+| 16000 | 0.2586 | 194.0 | 564.0 | 0.8357 | 25.4869 | 98.09 | 12.281 | 123.0 | 152.0 |
+| 17000 | 0.2747 | 143.0 | 468.0 | 0.8056 | 25.4845 | 98.099 | 12.282 | 88.0 | 131.0 |
+| 18000 | 0.2909 | 135.0 | 512.0 | 0.7808 | 25.6085 | 97.624 | 12.222 | 81.5 | 121.5 |
+| 19000 | 0.3071 | 130.0 | 484.0 | 0.8070 | 25.6119 | 97.611 | 12.221 | 80.5 | 123.5 |
+| 20000 | 0.3232 | 102.5 | 414.0 | 0.7258 | 25.5902 | 97.694 | 12.231 | 73.0 | 136.0 |
+| 21000 | 0.3394 | 92.5 | 370.0 | 0.6690 | 25.484 | 98.101 | 12.282 | 72.5 | 117.0 |
+| 22000 | 0.3556 | 89.5 | 320.0 | 0.6363 | 25.4959 | 98.055 | 12.277 | 72.0 | 109.5 |
+| 23000 | 0.3717 | 93.5 | 302.0 | 0.6073 | 25.5285 | 97.93 | 12.261 | 76.0 | 105.5 |
+| 24000 | 0.3879 | 93.5 | 306.0 | 0.5868 | 25.52 | 97.962 | 12.265 | 76.5 | 106.0 |
+| 25000 | 0.4040 | 126.5 | 336.0 | 0.5717 | 25.5413 | 97.881 | 12.255 | 108.0 | 132.0 |
+| 26000 | 0.4202 | 72.0 | 250.0 | 0.5578 | 25.5841 | 97.717 | 12.234 | 61.25 | 114.0 |
+| 27000 | 0.4364 | 87.5 | 262.0 | 0.5462 | 25.5236 | 97.948 | 12.263 | 63.75 | 116.0 |
+| 28000 | 0.4525 | 72.0 | 228.0 | 0.5431 | 25.5362 | 97.9 | 12.257 | 57.75 | 107.0 |
+| 29000 | 0.4687 | 75.5 | 247.0 | 0.5524 | 25.5375 | 97.895 | 12.256 | 56.75 | 105.5 |
+| 30000 | 0.4848 | 72.0 | 222.0 | 0.5459 | 25.5244 | 97.945 | 12.263 | 53.25 | 114.5 |
+| 31000 | 0.5010 | 71.5 | 230.0 | 0.5460 | 25.4562 | 98.208 | 12.296 | 54.25 | 151.0 |
+| 32000 | 0.5172 | 69.5 | 246.0 | 0.5391 | 25.5258 | 97.94 | 12.262 | 54.5 | 194.0 |
+| 33000 | 0.5333 | 70.5 | 242.0 | 0.5283 | 25.4905 | 98.076 | 12.279 | 53.75 | 117.5 |
+| 34000 | 0.5495 | 76.5 | 246.0 | 0.5147 | 25.4567 | 98.206 | 12.295 | 62.75 | 114.0 |
+| 35000 | 0.5657 | 68.0 | 216.0 | 0.5014 | 25.4939 | 98.063 | 12.277 | 50.25 | 87.0 |
+| 36000 | 0.5818 | 71.0 | 225.0 | 0.4986 | 25.4871 | 98.089 | 12.281 | 52.75 | 90.0 |
+| 37000 | 0.5980 | 79.5 | 231.0 | 0.4939 | 25.509 | 98.005 | 12.27 | 54.0 | 112.5 |
+| 38000 | 0.6141 | 75.5 | 232.0 | 0.4880 | 25.4997 | 98.04 | 12.275 | 58.75 | 116.5 |
+| 39000 | 0.6303 | 82.0 | 241.0 | 0.4879 | 25.519 | 97.966 | 12.265 | 61.75 | 123.0 |
+| 40000 | 0.6465 | 73.5 | 220.0 | 0.4759 | 25.4982 | 98.046 | 12.275 | 55.25 | 126.0 |
+| 41000 | 0.6626 | 70.5 | 212.0 | 0.4738 | 25.5094 | 98.003 | 12.27 | 55.5 | 102.0 |
+| 42000 | 0.6788 | 79.0 | 227.0 | 0.4699 | 25.527 | 97.935 | 12.262 | 59.25 | 82.5 |
+| 43000 | 0.6949 | 82.0 | 227.0 | 0.4613 | 25.5453 | 97.865 | 12.253 | 59.25 | 92.5 |
+| 44000 | 0.7111 | 78.0 | 210.0 | 0.4227 | 25.5427 | 97.875 | 12.254 | 61.5 | 79.0 |
+| 45000 | 0.7273 | 72.0 | 192.0 | 0.4068 | 25.4354 | 98.288 | 12.306 | 55.5 | 72.0 |
+| 46000 | 0.7434 | 72.0 | 185.0 | 0.3952 | 25.5194 | 97.965 | 12.265 | 56.0 | 53.0 |
+| 47000 | 0.7596 | 68.0 | 178.0 | 0.3928 | 25.5018 | 98.032 | 12.274 | 51.25 | 54.25 |
+| 48000 | 0.7758 | 71.5 | 183.0 | 0.3873 | 25.5018 | 98.032 | 12.274 | 54.5 | 63.0 |
+| 49000 | 0.7919 | 71.0 | 177.0 | 0.3853 | 25.5149 | 97.982 | 12.267 | 52.5 | 66.5 |
+| 50000 | 0.8081 | 71.0 | 174.0 | 0.3802 | 25.4786 | 98.122 | 12.285 | 54.75 | 61.75 |
+| 51000 | 0.8242 | 73.5 | 184.0 | 0.3797 | 25.4575 | 98.203 | 12.295 | 56.25 | 59.0 |
+| 52000 | 0.8404 | 75.5 | 187.0 | 0.3784 | 25.4298 | 98.31 | 12.308 | 57.5 | 68.5 |
+| 53000 | 0.8566 | 74.5 | 185.0 | 0.3741 | 25.5041 | 98.023 | 12.273 | 55.75 | 59.75 |
+| 54000 | 0.8727 | 73.0 | 178.0 | 0.3686 | 25.5431 | 97.874 | 12.254 | 54.75 | 56.25 |
+| 55000 | 0.8889 | 72.0 | 180.0 | 0.3649 | 25.447 | 98.244 | 12.3 | 55.0 | 55.0 |
+| 56000 | 0.9051 | 71.5 | 179.0 | 0.3636 | 25.5222 | 97.954 | 12.264 | 53.75 | 53.0 |
+| 57000 | 0.9212 | 71.5 | 176.0 | 0.3619 | 25.4887 | 98.083 | 12.28 | 54.25 | 50.5 |
+| 58000 | 0.9374 | 72.0 | 177.0 | 0.3605 | 25.4451 | 98.251 | 12.301 | 55.0 | 50.25 |
+| 59000 | 0.9535 | 72.0 | 177.0 | 0.3598 | 25.5041 | 98.023 | 12.273 | 54.5 | 50.5 |
+| 60000 | 0.9697 | 72.0 | 177.0 | 0.3589 | 25.5086 | 98.006 | 12.27 | 54.0 | 50.5 |
+| 61000 | 0.9859 | 72.0 | 177.0 | 0.3588 | 25.4042 | 98.409 | 12.321 | 54.5 | 50.5 |
+| 61875 | 1.0 | 72.0 | 177.0 | 0.3587 | 25.56 | 97.809 | 12.246 | 54.25 | 50.25 |
 
 ### Framework versions
 - Distily 0.2.0

logs/lr_scheduler_type=linear, warmup_ratio=0.5/events.out.tfevents.1724151237.f383272e719b

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d25613f58ba0775aca44b60d5e261053940f4379382de62faacfb2ea18a5672
+size 312