End of training

README.md

@@ -16,14 +16,14 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
 The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 
 It achieves the following results on the evaluation set:
-- eval_enwikippl: 100.5
-- eval_frwikippl: 402.0
-- eval_zhwikippl: 202.0
-- eval_tinystoriesppl: 77.5
-- eval_loss: 0.8097
-- eval_runtime: 12.6234
-- eval_samples_per_second: 47.531
-- eval_steps_per_second: 11.883
+- eval_enwikippl: 280.0
+- eval_frwikippl: 1392.0
+- eval_zhwikippl: 2576.0
+- eval_tinystoriesppl: 207.0
+- eval_loss: 1.4458
+- eval_runtime: 12.6656
+- eval_samples_per_second: 47.372
+- eval_steps_per_second: 11.843
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -49,7 +49,7 @@ The following hyperparameters were used during training:
 - distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
 - train_embeddings: True
 - learning_rate: 0.0001
-- train_batch_size: 4
+- train_batch_size: 1
 - eval_batch_size: 4
 - seed: 42
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
@@ -64,17 +64,47 @@ Peak GPU Memory: 4.1856 GB
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | **teacher eval** |  | 43.75 | 61.75 |  |  |  |  | 11.8125 | 19.125 |
-| 0 | 0 | 1666447310848.0 | 185817465094144.0 | 19.8325 | 12.5128 | 47.951 | 11.988 | 8925478912.0 | 38482906972160.0 |
-| 1500 | 0.1010 | 832.0 | 5056.0 | 2.2173 | 12.5546 | 47.791 | 11.948 | 616.0 | 10752.0 |
-| 3000 | 0.2020 | 358.0 | 1552.0 | 1.6287 | 12.5223 | 47.914 | 11.979 | 288.0 | 324.0 |
-| 4500 | 0.3030 | 239.0 | 1008.0 | 1.3817 | 12.5744 | 47.716 | 11.929 | 200.0 | 185.0 |
-| 6000 | 0.4040 | 165.0 | 584.0 | 1.1057 | 12.6319 | 47.499 | 11.875 | 131.0 | 168.0 |
-| 7500 | 0.5051 | 127.0 | 486.0 | 0.9512 | 12.5972 | 47.63 | 11.907 | 99.0 | 198.0 |
-| 9000 | 0.6061 | 100.5 | 402.0 | 0.8097 | 12.6234 | 47.531 | 11.883 | 77.5 | 202.0 |
-| 10500 | 0.7071 | 82.5 | 324.0 | 0.6801 | 12.6184 | 47.549 | 11.887 | 66.0 | 129.0 |
-| 12000 | 0.8081 | 71.5 | 264.0 | 0.5949 | 12.5839 | 47.68 | 11.92 | 59.25 | 146.0 |
-| 13500 | 0.9091 | 69.0 | 245.0 | 0.5609 | 12.5927 | 47.647 | 11.912 | 57.25 | 117.0 |
-| 14850 | 1.0 | 68.5 | 240.0 | 0.5540 | 12.5567 | 47.783 | 11.946 | 56.5 | 114.5 |
+| 0 | 0 | 1821066133504.0 | 158329674399744.0 | 19.3254 | 12.6923 | 47.273 | 11.818 | 12079595520.0 | 98956046499840.0 |
+| 1500 | 0.0253 | 3616.0 | 28416.0 | 3.2424 | 12.6627 | 47.383 | 11.846 | 2560.0 | 606208.0 |
+| 3000 | 0.0505 | 836.0 | 5888.0 | 2.2048 | 12.6773 | 47.329 | 11.832 | 652.0 | 21376.0 |
+| 4500 | 0.0758 | 482.0 | 3152.0 | 1.8732 | 12.5614 | 47.766 | 11.941 | 352.0 | 580.0 |
+| 6000 | 0.1010 | 352.0 | 1520.0 | 1.5840 | 12.7029 | 47.233 | 11.808 | 272.0 | 330.0 |
+| 7500 | 0.1263 | 308.0 | 1272.0 | 1.5038 | 12.8562 | 46.67 | 11.668 | 241.0 | 294.0 |
+| 9000 | 0.1515 | 280.0 | 1392.0 | 1.4458 | 12.6656 | 47.372 | 11.843 | 207.0 | 2576.0 |
+| 10500 | 0.1768 | 238.0 | 936.0 | 1.3311 | 12.6721 | 47.348 | 11.837 | 185.0 | 286.0 |
+| 12000 | 0.2020 | 215.0 | 836.0 | 1.2527 | 12.6564 | 47.407 | 11.852 | 163.0 | 280.0 |
+| 13500 | 0.2273 | 197.0 | 772.0 | 1.2033 | 12.5546 | 47.791 | 11.948 | 150.0 | 266.0 |
+| 15000 | 0.2525 | 187.0 | 792.0 | 1.1701 | 12.5678 | 47.741 | 11.935 | 154.0 | 250.0 |
+| 16500 | 0.2778 | 171.0 | 704.0 | 1.1274 | 12.6162 | 47.558 | 11.889 | 150.0 | 152.0 |
+| 18000 | 0.3030 | 167.0 | 748.0 | 1.1101 | 12.5787 | 47.7 | 11.925 | 141.0 | 314.0 |
+| 19500 | 0.3283 | 164.0 | 684.0 | 1.1063 | 12.6722 | 47.348 | 11.837 | 142.0 | 254.0 |
+| 21000 | 0.3535 | 162.0 | 676.0 | 1.0552 | 12.6941 | 47.266 | 11.816 | 135.0 | 242.0 |
+| 22500 | 0.3788 | 146.0 | 628.0 | 1.0061 | 12.5812 | 47.69 | 11.923 | 120.0 | 226.0 |
+| 24000 | 0.4040 | 141.0 | 636.0 | 0.9924 | 12.5431 | 47.835 | 11.959 | 115.5 | 218.0 |
+| 25500 | 0.4293 | 138.0 | 776.0 | 0.9653 | 12.619 | 47.547 | 11.887 | 110.5 | 362.0 |
+| 27000 | 0.4545 | 129.0 | 572.0 | 0.9293 | 12.5466 | 47.822 | 11.955 | 113.0 | 245.0 |
+| 28500 | 0.4798 | 136.0 | 592.0 | 0.9387 | 12.658 | 47.401 | 11.85 | 117.5 | 204.0 |
+| 30000 | 0.5051 | 132.0 | 704.0 | 0.9436 | 12.5944 | 47.64 | 11.91 | 112.0 | 255.0 |
+| 31500 | 0.5303 | 120.0 | 596.0 | 0.9097 | 12.5827 | 47.684 | 11.921 | 104.5 | 262.0 |
+| 33000 | 0.5556 | 119.5 | 548.0 | 0.8672 | 12.5736 | 47.719 | 11.93 | 103.5 | 264.0 |
+| 34500 | 0.5808 | 114.0 | 544.0 | 0.8406 | 12.6869 | 47.293 | 11.823 | 95.0 | 300.0 |
+| 36000 | 0.6061 | 107.5 | 410.0 | 0.8157 | 12.6245 | 47.527 | 11.882 | 95.5 | 199.0 |
+| 37500 | 0.6313 | 102.5 | 478.0 | 0.8011 | 12.7459 | 47.074 | 11.769 | 92.0 | 312.0 |
+| 39000 | 0.6566 | 106.0 | 454.0 | 0.7952 | 12.6026 | 47.609 | 11.902 | 93.0 | 262.0 |
+| 40500 | 0.6818 | 102.5 | 448.0 | 0.7747 | 12.5828 | 47.684 | 11.921 | 85.5 | 249.0 |
+| 42000 | 0.7071 | 88.5 | 366.0 | 0.6942 | 12.5833 | 47.682 | 11.921 | 76.0 | 207.0 |
+| 43500 | 0.7323 | 79.5 | 326.0 | 0.6337 | 12.6842 | 47.303 | 11.826 | 66.5 | 160.0 |
+| 45000 | 0.7576 | 78.0 | 270.0 | 0.6070 | 12.684 | 47.304 | 11.826 | 65.5 | 159.0 |
+| 46500 | 0.7828 | 76.5 | 260.0 | 0.5937 | 12.5713 | 47.728 | 11.932 | 62.0 | 127.5 |
+| 48000 | 0.8081 | 76.0 | 272.0 | 0.5848 | 12.6783 | 47.325 | 11.831 | 63.5 | 127.0 |
+| 49500 | 0.8333 | 74.0 | 253.0 | 0.5771 | 12.5825 | 47.685 | 11.921 | 62.25 | 132.0 |
+| 51000 | 0.8586 | 72.5 | 252.0 | 0.5649 | 12.6897 | 47.283 | 11.821 | 59.5 | 101.0 |
+| 52500 | 0.8838 | 71.5 | 241.0 | 0.5513 | 12.6795 | 47.32 | 11.83 | 58.25 | 105.0 |
+| 54000 | 0.9091 | 71.0 | 238.0 | 0.5457 | 12.657 | 47.404 | 11.851 | 57.5 | 102.5 |
+| 55500 | 0.9343 | 70.5 | 237.0 | 0.5425 | 12.6411 | 47.464 | 11.866 | 56.75 | 95.5 |
+| 57000 | 0.9596 | 71.0 | 236.0 | 0.5401 | 12.772 | 46.978 | 11.744 | 56.75 | 92.0 |
+| 58500 | 0.9848 | 70.0 | 234.0 | 0.5385 | 12.5709 | 47.729 | 11.932 | 56.75 | 93.0 |
+| 59400 | 1.0 | 70.0 | 234.0 | 0.5385 | 12.5868 | 47.669 | 11.917 | 56.75 | 93.0 |
 
 ### Framework versions
 - Distily 0.2.0

logs/learning_rate=0.0001, per_device_train_batch_size=1, warmup_ratio=0.5/events.out.tfevents.1724090049.5f530b1cf724

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f6300dcd2830cc50f948eaec90998927281ce2243a86c284f290e188482e5b3
+size 28450748

logs/learning_rate=0.0001, per_device_train_batch_size=1, warmup_ratio=0.5/events.out.tfevents.1724100891.5f530b1cf724

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:efaed2b0c7f2777d6ef0aa02061db90808e6dbaa12ddb0c710e4ebec43cf2684
+size 588

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:17663af322b01be7624862bedcf89f09c8e2a817c5865a09ff38a06bd6f2f035
+oid sha256:974fc1972a602545efa3d6652649817b5c0627110efb26a9da1aafedbf6d789f
 size 248894656

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3b29132c596a46d946d4f33b45328cd83c25ea3fcfd94a27254be6474b0d74a3
+oid sha256:f9c4d041aa095b908edefa6f0a3a491706fee792245eaef3ec5562e85525f3c8
 size 1017899144