|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
datasets: |
|
- jondurbin/truthy-dpo-v0.1 |
|
- snorkelai/Snorkel-Mistral-PairRM-DPO-Dataset |
|
--- |
|
Gazelle v0.2 is the mid-March release from [Tincans](https://tincans.ai) of a joint speech-language model. |
|
|
|
This repo contains an experimental DPO finetune. To our knowledge, this is the first multi-modal DPO finetune of a speech-language model - audio in, text out. |
|
|
|
The datasets used were [snorkelai/Snorkel-Mistral-PairRM-DPO-Dataset](https://huggingface.co/datasets/snorkelai/Snorkel-Mistral-PairRM-DPO-Dataset) (first iteration) and [jondurbin/truthy-dpo-v0.1](https://huggingface.co/datasets/jondurbin/truthy-dpo-v0.1?row=0). We trained for 2 epochs with max_lr=3e-4, batch size 32, 10 warmup steps, cosine decay. |
|
|
|
We can see some tell-tale signs of preference modeling at play, particularly longer replies, which don't exist in the base instruction-tuned model. Overall, we view the quality as being mixed and welcome experimentation but do not suggest production use. |
|
|
|
Please see [this notebook](https://github.com/tincans-ai/gazelle/blob/2939d7034277506171d61a7a1001f535426faa71/examples/infer.ipynb) for an inference example. |
