End of training

README.md

@@ -16,13 +16,13 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
 The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 
 It achieves the following results on the evaluation set:
-- eval_enwikippl: 234.9056
-- eval_frwikippl: 1372.0399
-- eval_zhwikippl: 581.4163
-- eval_loss: 1.3616
-- eval_runtime: 35.2036
-- eval_samples_per_second: 56.812
-- eval_steps_per_second: 7.102
+- eval_enwikippl: 215.7906
+- eval_frwikippl: 1306.3361
+- eval_zhwikippl: 583.5945
+- eval_loss: 1.2753
+- eval_runtime: 34.5544
+- eval_samples_per_second: 57.88
+- eval_steps_per_second: 7.235
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -45,7 +45,7 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- distillation_objective: MultiObjective(logits_weight=1, logits_loss_fn=(fn:kl_divergence_loss()), hs_weight=0.2, hs_loss_fn=(fn:kl_divergence_loss()), attn_weight=0, attn_loss_fn=(fn:soft_mse_loss()))
+- distillation_objective: MultiObjective(logits_weight=1, logits_loss_fn=(fn:kl_divergence_loss()), hs_weight=0.2, hs_loss_fn=(fn:soft_cross_entropy_loss()), attn_weight=0, attn_loss_fn=(fn:soft_mse_loss()))
 - train_embeddings: True
 - learning_rate: 4e-05
 - train_batch_size: 8
@@ -56,38 +56,38 @@ The following hyperparameters were used during training:
 - num_epochs: 1.0
 
 ### Resource Usage
-Peak GPU Memory: 8.0903 GB
+Peak GPU Memory: 8.0904 GB
 
 ### Eval-Phase Metrics
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | **teacher eval** |  | 30.2086 | 57.2728 |  |  |  |  | 18.1784 |
-| 0 | 0 | 54069.2930 | 57285.3438 | 7.1467 | 35.2037 | 56.812 | 7.102 | 54227.1016 |
-| 1000 | 0.0404 | 835.9614 | 5114.5200 | 2.0979 | 35.1654 | 56.874 | 7.109 | 20046.5859 |
-| 2000 | 0.0808 | 564.0770 | 3427.2224 | 1.8740 | 35.236 | 56.76 | 7.095 | 2653.2007 |
-| 3000 | 0.1212 | 453.5656 | 2948.4802 | 1.7420 | 35.0308 | 57.093 | 7.137 | 1432.7913 |
-| 4000 | 0.1616 | 389.6504 | 2661.5664 | 1.6437 | 35.0141 | 57.12 | 7.14 | 1010.8831 |
-| 5000 | 0.2020 | 338.8218 | 2219.8342 | 1.5658 | 35.0667 | 57.034 | 7.129 | 954.4720 |
-| 6000 | 0.2424 | 299.0258 | 1876.3831 | 1.4980 | 35.0748 | 57.021 | 7.128 | 1165.2183 |
-| 7000 | 0.2828 | 273.8413 | 1630.5182 | 1.4478 | 35.1356 | 56.922 | 7.115 | 1019.0151 |
-| 8000 | 0.3232 | 252.0468 | 1444.3036 | 1.3992 | 35.0678 | 57.032 | 7.129 | 876.0536 |
-| 9000 | 0.3636 | 234.9056 | 1372.0399 | 1.3616 | 35.2036 | 56.812 | 7.102 | 581.4163 |
-| 10000 | 0.4040 | 221.7531 | 1324.3276 | 1.3253 | 35.1731 | 56.862 | 7.108 | 536.0763 |
-| 11000 | 0.4444 | 205.0097 | 1190.9154 | 1.2854 | 35.2444 | 56.747 | 7.093 | 729.7837 |
-| 12000 | 0.4848 | 192.3177 | 1137.5690 | 1.2537 | 35.9424 | 55.645 | 6.956 | 630.9283 |
-| 13000 | 0.5253 | 178.7797 | 1029.4834 | 1.2186 | 35.669 | 56.071 | 7.009 | 711.3105 |
-| 14000 | 0.5657 | 171.1587 | 935.8806 | 1.1898 | 35.6704 | 56.069 | 7.009 | 608.0168 |
-| 15000 | 0.6061 | 163.3289 | 894.3995 | 1.1697 | 35.7337 | 55.97 | 6.996 | 592.7048 |
-| 16000 | 0.6465 | 161.3999 | 839.2896 | 1.1530 | 35.726 | 55.982 | 6.998 | 561.4263 |
-| 17000 | 0.6869 | 153.9090 | 816.5835 | 1.1376 | 35.6037 | 56.174 | 7.022 | 552.1324 |
-| 18000 | 0.7273 | 153.1935 | 798.4223 | 1.1286 | 38.2137 | 52.337 | 6.542 | 561.0514 |
-| 19000 | 0.7677 | 149.5150 | 771.6359 | 1.1168 | 35.6863 | 56.044 | 7.005 | 403.5265 |
-| 20000 | 0.8081 | 145.0661 | 746.0110 | 1.1048 | 35.6993 | 56.024 | 7.003 | 399.0254 |
-| 21000 | 0.8485 | 143.9999 | 710.9376 | 1.0945 | 35.6597 | 56.086 | 7.011 | 650.4333 |
-| 22000 | 0.8889 | 142.8749 | 695.1750 | 1.0892 | 35.7386 | 55.962 | 6.995 | 397.0057 |
-| 23000 | 0.9293 | 140.7277 | 699.8470 | 1.0831 | 35.4866 | 56.359 | 7.045 | 324.7243 |
-| 24000 | 0.9697 | 138.0864 | 705.5446 | 1.0736 | 35.6884 | 56.041 | 7.005 | 355.2115 |
-| 24750 | 1.0 | 137.5085 | 729.4709 | 1.0760 | 35.8247 | 55.827 | 6.978 | 660.5873 |
+| 0 | 0 | 55317.8945 | 54673.0039 | 5.9451 | 34.3869 | 58.162 | 7.27 | 59699.7266 |
+| 1000 | 0.0404 | 730.1941 | 4645.5654 | 1.9777 | 34.4565 | 58.044 | 7.256 | 11835.6895 |
+| 2000 | 0.0808 | 512.3346 | 3066.7913 | 1.7886 | 34.514 | 57.947 | 7.243 | 2242.3167 |
+| 3000 | 0.1212 | 429.5358 | 2881.4824 | 1.6761 | 34.5666 | 57.859 | 7.232 | 1026.3900 |
+| 4000 | 0.1616 | 373.7652 | 2655.5688 | 1.5906 | 34.614 | 57.78 | 7.223 | 763.8854 |
+| 5000 | 0.2020 | 325.4127 | 2080.4141 | 1.5114 | 34.571 | 57.852 | 7.231 | 852.0554 |
+| 6000 | 0.2424 | 283.6897 | 1739.0519 | 1.4385 | 34.537 | 57.909 | 7.239 | 678.3768 |
+| 7000 | 0.2828 | 258.3082 | 1566.5100 | 1.3779 | 34.3495 | 58.225 | 7.278 | 772.5022 |
+| 8000 | 0.3232 | 234.4317 | 1380.9691 | 1.3265 | 34.584 | 57.83 | 7.229 | 688.6912 |
+| 9000 | 0.3636 | 215.7906 | 1306.3361 | 1.2753 | 34.5544 | 57.88 | 7.235 | 583.5945 |
+| 10000 | 0.4040 | 198.0003 | 1188.5668 | 1.2276 | 34.6027 | 57.799 | 7.225 | 564.3576 |
+| 11000 | 0.4444 | 181.9449 | 1205.6156 | 1.1866 | 34.6424 | 57.733 | 7.217 | 871.1540 |
+| 12000 | 0.4848 | 168.2595 | 991.4434 | 1.1379 | 34.5542 | 57.88 | 7.235 | 533.5765 |
+| 13000 | 0.5253 | 158.1617 | 921.2160 | 1.1068 | 34.2816 | 58.34 | 7.293 | 551.3958 |
+| 14000 | 0.5657 | 150.2365 | 856.6271 | 1.0792 | 34.3516 | 58.221 | 7.278 | 696.3668 |
+| 15000 | 0.6061 | 144.5713 | 878.8344 | 1.0595 | 34.5245 | 57.93 | 7.241 | 519.5146 |
+| 16000 | 0.6465 | 139.2169 | 769.8428 | 1.0385 | 34.5444 | 57.896 | 7.237 | 485.0522 |
+| 17000 | 0.6869 | 137.1886 | 707.7368 | 1.0232 | 34.5043 | 57.964 | 7.245 | 714.2610 |
+| 18000 | 0.7273 | 133.8944 | 712.3927 | 1.0136 | 34.5983 | 57.806 | 7.226 | 653.7423 |
+| 19000 | 0.7677 | 130.7503 | 663.3331 | 1.0010 | 34.4715 | 58.019 | 7.252 | 561.5013 |
+| 20000 | 0.8081 | 129.0055 | 645.0233 | 0.9909 | 34.3899 | 58.157 | 7.27 | 516.6090 |
+| 21000 | 0.8485 | 127.4918 | 689.9504 | 0.9901 | 34.443 | 58.067 | 7.258 | 434.3951 |
+| 22000 | 0.8889 | 123.1615 | 682.8845 | 0.9756 | 34.5625 | 57.866 | 7.233 | 450.5237 |
+| 23000 | 0.9293 | 123.7943 | 689.0751 | 0.9766 | 35.1251 | 56.939 | 7.117 | 593.8933 |
+| 24000 | 0.9697 | 121.1035 | 699.7977 | 0.9661 | 34.5693 | 57.855 | 7.232 | 1115.5807 |
+| 24750 | 1.0 | 123.4295 | 640.9431 | 0.9586 | 34.5071 | 57.959 | 7.245 | 447.3463 |
 
 ### Framework versions
 - Distily 0.2.0

logs/distillation_objective=MultiObjective(logits_weight_1__logits_loss_fn_(fn_kl_divergence_loss())__hs_weight_0.2__hs_loss_fn_(fn_soft_cross_entropy_loss())__attn_weight_0__attn_loss_fn_(fn_soft_mse_loss/events.out.tfevents.1723650945.93d6cbb3ad53

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f7aae2c53d5bd169abd62499b89cea98cad5ae89ec0b5b38fef48010eb276a6
+size 253