---
license: mit
library_name: peft
tags:
- generated_from_trainer
base_model: microsoft/phi-2
model-index:
- name: fine-tuning-Phi2-with-webglm-qa-with-lora_9
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# fine-tuning-Phi2-with-webglm-qa-with-lora_9

This model is a fine-tuned version of [microsoft/phi-2](https://huggingface.co/microsoft/phi-2) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.1781

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 5
- total_train_batch_size: 10
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 60
- training_steps: 700
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.9045        | 0.16  | 10   | 1.6827          |
| 1.6798        | 0.32  | 20   | 1.5996          |
| 1.5222        | 0.48  | 30   | 1.4548          |
| 1.286         | 0.64  | 40   | 1.2269          |
| 1.0827        | 0.8   | 50   | 1.1089          |
| 0.9913        | 0.96  | 60   | 1.0021          |
| 0.881         | 1.13  | 70   | 0.8796          |
| 0.7673        | 1.29  | 80   | 0.7637          |
| 0.6315        | 1.45  | 90   | 0.6618          |
| 0.554         | 1.61  | 100  | 0.5964          |
| 0.5132        | 1.77  | 110  | 0.5487          |
| 0.4915        | 1.93  | 120  | 0.5030          |
| 0.4787        | 2.09  | 130  | 0.4705          |
| 0.4298        | 2.25  | 140  | 0.4451          |
| 0.4009        | 2.41  | 150  | 0.4099          |
| 0.3886        | 2.57  | 160  | 0.3889          |
| 0.3729        | 2.73  | 170  | 0.3674          |
| 0.3236        | 2.89  | 180  | 0.3527          |
| 0.3377        | 3.05  | 190  | 0.3407          |
| 0.3356        | 3.22  | 200  | 0.3261          |
| 0.3083        | 3.38  | 210  | 0.3121          |
| 0.2794        | 3.54  | 220  | 0.2992          |
| 0.2917        | 3.7   | 230  | 0.2926          |
| 0.2895        | 3.86  | 240  | 0.2879          |
| 0.2764        | 4.02  | 250  | 0.2782          |
| 0.2585        | 4.18  | 260  | 0.2732          |
| 0.2489        | 4.34  | 270  | 0.2678          |
| 0.2401        | 4.5   | 280  | 0.2591          |
| 0.2489        | 4.66  | 290  | 0.2573          |
| 0.2529        | 4.82  | 300  | 0.2501          |
| 0.2637        | 4.98  | 310  | 0.2455          |
| 0.255         | 5.14  | 320  | 0.2411          |
| 0.2266        | 5.31  | 330  | 0.2370          |
| 0.2209        | 5.47  | 340  | 0.2326          |
| 0.2311        | 5.63  | 350  | 0.2276          |
| 0.2203        | 5.79  | 360  | 0.2275          |
| 0.2048        | 5.95  | 370  | 0.2210          |
| 0.2133        | 6.11  | 380  | 0.2179          |
| 0.2045        | 6.27  | 390  | 0.2142          |
| 0.2053        | 6.43  | 400  | 0.2137          |
| 0.1898        | 6.59  | 410  | 0.2102          |
| 0.1897        | 6.75  | 420  | 0.2073          |
| 0.2141        | 6.91  | 430  | 0.2040          |
| 0.1872        | 7.07  | 440  | 0.2028          |
| 0.1938        | 7.23  | 450  | 0.1998          |
| 0.187         | 7.4   | 460  | 0.2004          |
| 0.1782        | 7.56  | 470  | 0.1973          |
| 0.1908        | 7.72  | 480  | 0.1967          |
| 0.1899        | 7.88  | 490  | 0.1912          |
| 0.1823        | 8.04  | 500  | 0.1912          |
| 0.1769        | 8.2   | 510  | 0.1915          |
| 0.1774        | 8.36  | 520  | 0.1909          |
| 0.1793        | 8.52  | 530  | 0.1890          |
| 0.1853        | 8.68  | 540  | 0.1880          |
| 0.1785        | 8.84  | 550  | 0.1861          |
| 0.1515        | 9.0   | 560  | 0.1845          |
| 0.1689        | 9.16  | 570  | 0.1845          |
| 0.1552        | 9.32  | 580  | 0.1836          |
| 0.1712        | 9.49  | 590  | 0.1828          |
| 0.1642        | 9.65  | 600  | 0.1818          |
| 0.1703        | 9.81  | 610  | 0.1806          |
| 0.1772        | 9.97  | 620  | 0.1804          |
| 0.1615        | 10.13 | 630  | 0.1796          |
| 0.1494        | 10.29 | 640  | 0.1801          |
| 0.1702        | 10.45 | 650  | 0.1798          |
| 0.1656        | 10.61 | 660  | 0.1787          |
| 0.1688        | 10.77 | 670  | 0.1782          |
| 0.1452        | 10.93 | 680  | 0.1780          |
| 0.1732        | 11.09 | 690  | 0.1782          |
| 0.1719        | 11.25 | 700  | 0.1781          |


### Framework versions

- PEFT 0.7.1
- Transformers 4.36.2
- Pytorch 2.0.0
- Datasets 2.15.0
- Tokenizers 0.15.0