---
library_name: transformers
license: llama3
base_model: tsavage68/Na_L3_100steps_1e6rate_SFT
tags:
- trl
- dpo
- generated_from_trainer
model-index:
- name: Na_L3_1000steps_1e6rate_05beta_cSFTDPO
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Na_L3_1000steps_1e6rate_05beta_cSFTDPO

This model is a fine-tuned version of [tsavage68/Na_L3_100steps_1e6rate_SFT](https://huggingface.co/tsavage68/Na_L3_100steps_1e6rate_SFT) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.0000
- Rewards/chosen: 1.8884
- Rewards/rejected: -13.4648
- Rewards/accuracies: 1.0
- Rewards/margins: 15.3532
- Logps/rejected: -68.4313
- Logps/chosen: -21.1134
- Logits/rejected: -0.9549
- Logits/chosen: -0.8862

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-06
- train_batch_size: 2
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- training_steps: 1000

### Training results

| Training Loss | Epoch  | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 0.0           | 0.2667 | 50   | 0.0000          | 1.3180         | -9.6006          | 1.0                | 10.9185         | -60.7028       | -22.2542     | -0.9556         | -0.8883       |
| 0.0           | 0.5333 | 100  | 0.0000          | 1.5811         | -11.1723         | 1.0                | 12.7534         | -63.8463       | -21.7279     | -0.9556         | -0.8877       |
| 0.0           | 0.8    | 150  | 0.0000          | 1.7220         | -11.8689         | 1.0                | 13.5909         | -65.2395       | -21.4461     | -0.9552         | -0.8869       |
| 0.0           | 1.0667 | 200  | 0.0000          | 1.7657         | -12.2278         | 1.0                | 13.9935         | -65.9574       | -21.3588     | -0.9548         | -0.8865       |
| 0.0           | 1.3333 | 250  | 0.0000          | 1.7855         | -12.5076         | 1.0                | 14.2932         | -66.5170       | -21.3191     | -0.9541         | -0.8858       |
| 0.0           | 1.6    | 300  | 0.0000          | 1.8162         | -12.7234         | 1.0                | 14.5396         | -66.9486       | -21.2578     | -0.9548         | -0.8864       |
| 0.0           | 1.8667 | 350  | 0.0000          | 1.8322         | -12.8906         | 1.0                | 14.7228         | -67.2829       | -21.2257     | -0.9552         | -0.8868       |
| 0.0           | 2.1333 | 400  | 0.0000          | 1.8447         | -13.0321         | 1.0                | 14.8768         | -67.5660       | -21.2008     | -0.9541         | -0.8855       |
| 0.0           | 2.4    | 450  | 0.0000          | 1.8576         | -13.1430         | 1.0                | 15.0006         | -67.7877       | -21.1749     | -0.9542         | -0.8856       |
| 0.0           | 2.6667 | 500  | 0.0000          | 1.8705         | -13.2358         | 1.0                | 15.1063         | -67.9734       | -21.1491     | -0.9535         | -0.8847       |
| 0.0           | 2.9333 | 550  | 0.0000          | 1.8928         | -13.3108         | 1.0                | 15.2037         | -68.1234       | -21.1045     | -0.9551         | -0.8865       |
| 0.0           | 3.2    | 600  | 0.0000          | 1.8716         | -13.3744         | 1.0                | 15.2461         | -68.2506       | -21.1468     | -0.9545         | -0.8858       |
| 0.0           | 3.4667 | 650  | 0.0000          | 1.8898         | -13.4154         | 1.0                | 15.3052         | -68.3326       | -21.1106     | -0.9551         | -0.8865       |
| 0.0           | 3.7333 | 700  | 0.0000          | 1.8784         | -13.4316         | 1.0                | 15.3100         | -68.3650       | -21.1333     | -0.9548         | -0.8860       |
| 0.0           | 4.0    | 750  | 0.0000          | 1.8821         | -13.4603         | 1.0                | 15.3424         | -68.4222       | -21.1259     | -0.9551         | -0.8863       |
| 0.0           | 4.2667 | 800  | 0.0000          | 1.8839         | -13.4565         | 1.0                | 15.3404         | -68.4148       | -21.1223     | -0.9552         | -0.8862       |
| 0.0           | 4.5333 | 850  | 0.0000          | 1.8851         | -13.4374         | 1.0                | 15.3225         | -68.3765       | -21.1199     | -0.9552         | -0.8866       |
| 0.0           | 4.8    | 900  | 0.0000          | 1.8887         | -13.4633         | 1.0                | 15.3521         | -68.4284       | -21.1127     | -0.9547         | -0.8861       |
| 0.0           | 5.0667 | 950  | 0.0000          | 1.8879         | -13.4648         | 1.0                | 15.3527         | -68.4314       | -21.1144     | -0.9549         | -0.8862       |
| 0.0           | 5.3333 | 1000 | 0.0000          | 1.8884         | -13.4648         | 1.0                | 15.3532         | -68.4313       | -21.1134     | -0.9549         | -0.8862       |


### Framework versions

- Transformers 4.44.2
- Pytorch 2.4.0+cu121
- Datasets 2.21.0
- Tokenizers 0.19.1