ToastyPigeon/ms3-roselily-instruct

This is a double fine-tuned version of Mistral Small 24B Base 2501.

Stage 1 was shoving 30M tokens of human-writen story content into it using completion training (ToastyPigeon/ms3-base-roselily), which is about half of my WIP Roselily dataset (~60M tokens total).

Stage 2 was teaching it instruct (this model) using a mix of public instruction following data and a private instruct dataset from ZeusLabs.

This model should accept (in theory) any of the following instruct formats:

Tekken v7

[SYSTEM_PROMPT]{system prompt}[/SYSTEM_PROMPT][INST]{user message}[/INST]{assistant response}</s>

ChatML

<|im_start|>system
{system prompt}<|im_end|>
<|im_start|>user
{user message}<|im_end|>
<|im_start|>assistant
{assistant response}<|im_end|>

Fizzpaca

### System:
{system prompt}

### Instruction:
{user message}

### Response:
{assistant response}</s>

The Tekken tokens were already in the tokenizer. unused special tokens #20 and 21 were repurposed for the ChatML tokens. Fizzpaca did not add any.

You may need to add both </s> and <|im_end|> as stop tokens for it to work properly with all formats.