File size: 1,966 Bytes
702849c b1ca4f7 702849c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
---
library_name: transformers
language:
- en
---
# Phi3-mini-128k-it ORPO model
[Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) fine-tuned on Text-to-SQL downstream task using [Odds Ratio Preference Optimization (ORPO)](https://arxiv.org/pdf/2403.07691).
## Details
A 4-bit quantized version of the [Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) model was used to fine-tuned on [zerolink/zsql-sqlite-dpo](https://huggingface.co/datasets/zerolink/zsql-sqlite-dpo).
Used PEFT to merge the trained adapters.
### Odds Ratio Preference Optimization (ORPO)
The goal of ORPO is to penalize the "rejected" samples, and increase the likelihood of "accepted" samples.This builds upon DPO but incorporates a ranking of preferences. This means we not only learn which outputs are preferred but also their relative ranking
#### Dataset
The model was fine-tuned on [zerolink/zsql-sqlite-dpo](https://huggingface.co/datasets/zerolink/zsql-sqlite-dpo) dataset.
Total entries in the dataset: 250,000
The dataset needs to be in the following format:
You need at least 3 columns:
- Schema
- Question
- Rejected
- Chosen
- Weight
For example:
- Schema: "CREATE TABLE table_name_56 (location TEXT, year INTEGER)"
- Question: "What location is previous to 1994?"
- Rejected: "SELECT location FROM table_name_56 WHERE year < 1994"
- Chosen: "SELECT "location" FROM "table_name_56" WHERE "year" < 1994"
- Weight: 0.056641
### Training Parameters
* QLoRA Parameters
- r = 16
- target_modules = ["q_proj", "k_proj", "v_proj", "o_proj","gate_proj", "up_proj", "down_proj",]
- lora_alpha = 16
- lora_dropout = 0
- bias = None
- random_state = 3407
* [ORPO Trainer](https://huggingface.co/docs/trl/main/en/orpo_trainer) Config
- num_epochs = 1
- max_steps = 30
- per_device_train_batch_size = 2
- gradient_accumulation_step = 4
- optim = "adamw_8it"
- lr_scheduler_type = "linear," |