--- license: gemma base_model: google/gemma-2-9b tags: - trl - sft - generated_from_trainer model-index: - name: collapse_gemma-2-9b_hs2_accumulate_iter4_sftsd1 results: [] --- # collapse_gemma-2-9b_hs2_accumulate_iter4_sftsd1 This model is a fine-tuned version of [google/gemma-2-9b](https://huggingface.co/google/gemma-2-9b) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 0.9485 - Num Input Tokens Seen: 19153848 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 8e-06 - train_batch_size: 4 - eval_batch_size: 16 - seed: 1 - gradient_accumulation_steps: 32 - total_train_batch_size: 128 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: constant_with_warmup - lr_scheduler_warmup_ratio: 0.05 - num_epochs: 1 ### Training results | Training Loss | Epoch | Step | Validation Loss | Input Tokens Seen | |:-------------:|:------:|:----:|:---------------:|:-----------------:| | No log | 0 | 0 | 1.2335 | 0 | | 1.3147 | 0.0133 | 5 | 1.1832 | 254068 | | 1.0498 | 0.0265 | 10 | 1.0719 | 516552 | | 0.9485 | 0.0398 | 15 | 1.0240 | 772872 | | 0.702 | 0.0530 | 20 | 1.0216 | 1026756 | | 0.5628 | 0.0663 | 25 | 1.0325 | 1281388 | | 0.5407 | 0.0796 | 30 | 1.0239 | 1536964 | | 0.368 | 0.0928 | 35 | 1.0172 | 1794688 | | 0.3617 | 0.1061 | 40 | 1.0116 | 2049692 | | 0.4112 | 0.1194 | 45 | 1.0076 | 2297888 | | 0.3436 | 0.1326 | 50 | 1.0005 | 2555848 | | 0.3919 | 0.1459 | 55 | 0.9977 | 2811876 | | 0.3196 | 0.1591 | 60 | 0.9936 | 3062460 | | 0.3427 | 0.1724 | 65 | 0.9888 | 3309344 | | 0.3779 | 0.1857 | 70 | 0.9873 | 3558348 | | 0.3086 | 0.1989 | 75 | 0.9835 | 3812356 | | 0.3156 | 0.2122 | 80 | 0.9815 | 4067536 | | 0.3243 | 0.2254 | 85 | 0.9801 | 4320128 | | 0.353 | 0.2387 | 90 | 0.9767 | 4574252 | | 0.315 | 0.2520 | 95 | 0.9759 | 4824900 | | 0.2642 | 0.2652 | 100 | 0.9737 | 5082344 | | 0.2696 | 0.2785 | 105 | 0.9740 | 5335240 | | 0.3357 | 0.2918 | 110 | 0.9728 | 5593492 | | 0.2995 | 0.3050 | 115 | 0.9700 | 5844120 | | 0.2826 | 0.3183 | 120 | 0.9680 | 6100756 | | 0.3509 | 0.3315 | 125 | 0.9657 | 6349476 | | 0.2684 | 0.3448 | 130 | 0.9653 | 6605052 | | 0.3148 | 0.3581 | 135 | 0.9654 | 6859396 | | 0.2782 | 0.3713 | 140 | 0.9634 | 7111896 | | 0.288 | 0.3846 | 145 | 0.9626 | 7365476 | | 0.292 | 0.3978 | 150 | 0.9608 | 7623304 | | 0.3483 | 0.4111 | 155 | 0.9601 | 7884416 | | 0.2856 | 0.4244 | 160 | 0.9605 | 8142544 | | 0.3717 | 0.4376 | 165 | 0.9597 | 8395160 | | 0.258 | 0.4509 | 170 | 0.9583 | 8650004 | | 0.3976 | 0.4642 | 175 | 0.9593 | 8900404 | | 0.2696 | 0.4774 | 180 | 0.9585 | 9153460 | | 0.3276 | 0.4907 | 185 | 0.9584 | 9401692 | | 0.2893 | 0.5039 | 190 | 0.9571 | 9655896 | | 0.3125 | 0.5172 | 195 | 0.9546 | 9917980 | | 0.2823 | 0.5305 | 200 | 0.9554 | 10167660 | | 0.3795 | 0.5437 | 205 | 0.9572 | 10416596 | | 0.2921 | 0.5570 | 210 | 0.9555 | 10667880 | | 0.2822 | 0.5702 | 215 | 0.9554 | 10922988 | | 0.2741 | 0.5835 | 220 | 0.9562 | 11178148 | | 0.2576 | 0.5968 | 225 | 0.9548 | 11431204 | | 0.2774 | 0.6100 | 230 | 0.9549 | 11678968 | | 0.3431 | 0.6233 | 235 | 0.9550 | 11938440 | | 0.3598 | 0.6366 | 240 | 0.9552 | 12188988 | | 0.321 | 0.6498 | 245 | 0.9537 | 12450232 | | 0.3062 | 0.6631 | 250 | 0.9546 | 12702636 | | 0.3252 | 0.6763 | 255 | 0.9538 | 12953464 | | 0.2372 | 0.6896 | 260 | 0.9526 | 13213356 | | 0.2677 | 0.7029 | 265 | 0.9538 | 13471308 | | 0.2751 | 0.7161 | 270 | 0.9544 | 13719800 | | 0.2369 | 0.7294 | 275 | 0.9540 | 13971304 | | 0.2586 | 0.7426 | 280 | 0.9524 | 14218076 | | 0.2612 | 0.7559 | 285 | 0.9531 | 14475828 | | 0.3368 | 0.7692 | 290 | 0.9528 | 14736144 | | 0.2279 | 0.7824 | 295 | 0.9513 | 14986776 | | 0.2958 | 0.7957 | 300 | 0.9510 | 15241276 | | 0.3639 | 0.8090 | 305 | 0.9527 | 15496724 | | 0.3721 | 0.8222 | 310 | 0.9525 | 15751944 | | 0.3751 | 0.8355 | 315 | 0.9516 | 16010436 | | 0.3224 | 0.8487 | 320 | 0.9508 | 16264272 | | 0.3244 | 0.8620 | 325 | 0.9492 | 16514404 | | 0.2181 | 0.8753 | 330 | 0.9498 | 16771700 | | 0.3845 | 0.8885 | 335 | 0.9521 | 17022500 | | 0.2373 | 0.9018 | 340 | 0.9493 | 17281212 | | 0.219 | 0.9150 | 345 | 0.9513 | 17531932 | | 0.3645 | 0.9283 | 350 | 0.9508 | 17787156 | | 0.3015 | 0.9416 | 355 | 0.9487 | 18042308 | | 0.2971 | 0.9548 | 360 | 0.9504 | 18288852 | | 0.2889 | 0.9681 | 365 | 0.9493 | 18546880 | | 0.2924 | 0.9814 | 370 | 0.9493 | 18799684 | | 0.2359 | 0.9946 | 375 | 0.9483 | 19052580 | ### Framework versions - Transformers 4.44.0 - Pytorch 2.4.0+cu121 - Datasets 2.20.0 - Tokenizers 0.19.1