--- license: apache-2.0 base_model: tsavage68/UTI_M2_1000steps_1e7rate_SFT tags: - trl - dpo - generated_from_trainer model-index: - name: UTI2_M2_1000steps_1e8rate_05beta_CSFTDPO results: [] --- # UTI2_M2_1000steps_1e8rate_05beta_CSFTDPO This model is a fine-tuned version of [tsavage68/UTI_M2_1000steps_1e7rate_SFT](https://huggingface.co/tsavage68/UTI_M2_1000steps_1e7rate_SFT) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 0.6685 - Rewards/chosen: -0.0009 - Rewards/rejected: -0.0566 - Rewards/accuracies: 0.1900 - Rewards/margins: 0.0557 - Logps/rejected: -9.4872 - Logps/chosen: -4.5444 - Logits/rejected: -2.7046 - Logits/chosen: -2.7039 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 1e-08 - train_batch_size: 2 - eval_batch_size: 1 - seed: 42 - gradient_accumulation_steps: 2 - total_train_batch_size: 4 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 100 - training_steps: 1000 ### Training results | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen | |:-------------:|:-------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:| | 0.6931 | 0.3333 | 25 | 0.6941 | 0.0016 | 0.0029 | 0.0700 | -0.0013 | -9.3682 | -4.5393 | -2.7069 | -2.7061 | | 0.6849 | 0.6667 | 50 | 0.6964 | -0.0083 | -0.0028 | 0.1100 | -0.0055 | -9.3796 | -4.5591 | -2.7057 | -2.7049 | | 0.6934 | 1.0 | 75 | 0.6896 | -0.0050 | -0.0129 | 0.1300 | 0.0079 | -9.3998 | -4.5524 | -2.7063 | -2.7056 | | 0.6902 | 1.3333 | 100 | 0.6901 | -0.0010 | -0.0078 | 0.1400 | 0.0068 | -9.3896 | -4.5445 | -2.7066 | -2.7058 | | 0.6931 | 1.6667 | 125 | 0.6876 | -0.0018 | -0.0134 | 0.1500 | 0.0117 | -9.4008 | -4.5460 | -2.7054 | -2.7047 | | 0.6854 | 2.0 | 150 | 0.6873 | 0.0005 | -0.0127 | 0.1300 | 0.0132 | -9.3993 | -4.5414 | -2.7062 | -2.7055 | | 0.6833 | 2.3333 | 175 | 0.6804 | 0.0002 | -0.0268 | 0.1800 | 0.0269 | -9.4275 | -4.5421 | -2.7052 | -2.7044 | | 0.6668 | 2.6667 | 200 | 0.6804 | 0.0042 | -0.0233 | 0.1700 | 0.0275 | -9.4206 | -4.5340 | -2.7050 | -2.7043 | | 0.6881 | 3.0 | 225 | 0.6735 | -0.0007 | -0.0445 | 0.1900 | 0.0438 | -9.4629 | -4.5438 | -2.7047 | -2.7040 | | 0.6721 | 3.3333 | 250 | 0.6742 | 0.0043 | -0.0375 | 0.1900 | 0.0418 | -9.4490 | -4.5339 | -2.7051 | -2.7044 | | 0.6714 | 3.6667 | 275 | 0.6696 | 0.0020 | -0.0497 | 0.1900 | 0.0516 | -9.4733 | -4.5385 | -2.7056 | -2.7048 | | 0.6713 | 4.0 | 300 | 0.6671 | 0.0010 | -0.0558 | 0.2100 | 0.0567 | -9.4855 | -4.5405 | -2.7042 | -2.7035 | | 0.6563 | 4.3333 | 325 | 0.6701 | 0.0019 | -0.0493 | 0.2000 | 0.0512 | -9.4726 | -4.5387 | -2.7051 | -2.7043 | | 0.6715 | 4.6667 | 350 | 0.6705 | 0.0015 | -0.0473 | 0.2000 | 0.0488 | -9.4685 | -4.5394 | -2.7041 | -2.7033 | | 0.6551 | 5.0 | 375 | 0.6672 | 0.0053 | -0.0523 | 0.2000 | 0.0576 | -9.4785 | -4.5318 | -2.7052 | -2.7044 | | 0.6601 | 5.3333 | 400 | 0.6664 | 0.0025 | -0.0562 | 0.2100 | 0.0587 | -9.4863 | -4.5374 | -2.7046 | -2.7039 | | 0.665 | 5.6667 | 425 | 0.6682 | 0.0029 | -0.0517 | 0.2000 | 0.0546 | -9.4773 | -4.5367 | -2.7042 | -2.7035 | | 0.6409 | 6.0 | 450 | 0.6668 | 0.0053 | -0.0538 | 0.2000 | 0.0591 | -9.4816 | -4.5319 | -2.7047 | -2.7039 | | 0.6649 | 6.3333 | 475 | 0.6660 | 0.0086 | -0.0517 | 0.2000 | 0.0602 | -9.4773 | -4.5254 | -2.7050 | -2.7042 | | 0.6711 | 6.6667 | 500 | 0.6641 | 0.0013 | -0.0654 | 0.2000 | 0.0667 | -9.5048 | -4.5399 | -2.7048 | -2.7041 | | 0.6583 | 7.0 | 525 | 0.6654 | -0.0009 | -0.0620 | 0.2100 | 0.0611 | -9.4979 | -4.5442 | -2.7041 | -2.7033 | | 0.6565 | 7.3333 | 550 | 0.6646 | 0.0059 | -0.0577 | 0.2000 | 0.0636 | -9.4894 | -4.5307 | -2.7044 | -2.7037 | | 0.6666 | 7.6667 | 575 | 0.6661 | 0.0028 | -0.0590 | 0.2000 | 0.0618 | -9.4919 | -4.5369 | -2.7047 | -2.7039 | | 0.6817 | 8.0 | 600 | 0.6663 | 0.0025 | -0.0569 | 0.2000 | 0.0594 | -9.4877 | -4.5375 | -2.7046 | -2.7039 | | 0.6655 | 8.3333 | 625 | 0.6656 | 0.0029 | -0.0593 | 0.1900 | 0.0622 | -9.4926 | -4.5367 | -2.7050 | -2.7043 | | 0.6344 | 8.6667 | 650 | 0.6700 | 0.0013 | -0.0500 | 0.1900 | 0.0513 | -9.4740 | -4.5399 | -2.7051 | -2.7044 | | 0.6587 | 9.0 | 675 | 0.6667 | -0.0021 | -0.0602 | 0.2000 | 0.0581 | -9.4944 | -4.5466 | -2.7047 | -2.7040 | | 0.6364 | 9.3333 | 700 | 0.6650 | 0.0021 | -0.0606 | 0.2000 | 0.0627 | -9.4952 | -4.5384 | -2.7052 | -2.7044 | | 0.6623 | 9.6667 | 725 | 0.6685 | -0.0010 | -0.0580 | 0.1900 | 0.0570 | -9.4899 | -4.5444 | -2.7053 | -2.7045 | | 0.6824 | 10.0 | 750 | 0.6673 | 0.0000 | -0.0577 | 0.1900 | 0.0577 | -9.4894 | -4.5424 | -2.7050 | -2.7043 | | 0.6497 | 10.3333 | 775 | 0.6705 | -0.0015 | -0.0535 | 0.2000 | 0.0520 | -9.4809 | -4.5454 | -2.7046 | -2.7039 | | 0.6693 | 10.6667 | 800 | 0.6691 | -0.0010 | -0.0550 | 0.1900 | 0.0540 | -9.4839 | -4.5445 | -2.7046 | -2.7039 | | 0.6678 | 11.0 | 825 | 0.6670 | -0.0003 | -0.0609 | 0.1900 | 0.0605 | -9.4957 | -4.5431 | -2.7046 | -2.7039 | | 0.6551 | 11.3333 | 850 | 0.6685 | -0.0009 | -0.0566 | 0.1900 | 0.0557 | -9.4872 | -4.5444 | -2.7046 | -2.7039 | | 0.682 | 11.6667 | 875 | 0.6685 | -0.0009 | -0.0566 | 0.1900 | 0.0557 | -9.4872 | -4.5444 | -2.7046 | -2.7039 | | 0.6731 | 12.0 | 900 | 0.6685 | -0.0009 | -0.0566 | 0.1900 | 0.0557 | -9.4872 | -4.5444 | -2.7046 | -2.7039 | | 0.6377 | 12.3333 | 925 | 0.6685 | -0.0009 | -0.0566 | 0.1900 | 0.0557 | -9.4872 | -4.5444 | -2.7046 | -2.7039 | | 0.6802 | 12.6667 | 950 | 0.6685 | -0.0009 | -0.0566 | 0.1900 | 0.0557 | -9.4872 | -4.5444 | -2.7046 | -2.7039 | | 0.6716 | 13.0 | 975 | 0.6685 | -0.0009 | -0.0566 | 0.1900 | 0.0557 | -9.4872 | -4.5444 | -2.7046 | -2.7039 | | 0.6571 | 13.3333 | 1000 | 0.6685 | -0.0009 | -0.0566 | 0.1900 | 0.0557 | -9.4872 | -4.5444 | -2.7046 | -2.7039 | ### Framework versions - Transformers 4.41.2 - Pytorch 2.0.0+cu117 - Datasets 2.19.2 - Tokenizers 0.19.1