| # Bigger Body 12b | |
|  | |
| A roleplay-focused pseudo full-finetune of Mistral Nemo Instruct. | |
| The successor to the Ink series. | |
| ## Testimonials | |
| > First impressions (temp 1, min-p .05-.1) | |
| > - It passes my silly logic tests (read: me trolling random characters) | |
| > - Haven't seen any slop yet | |
| > - Writes short and snappy replies | |
| > - ...yet not *too* short, like Mahou, and can write longer responses if the context warrants it | |
| > - Follows card formatting instructions | |
| > | |
| > If this holds up to 16K it will be constantly in the hopper alongside Mag-Mell for me. I'm biased towards shorter responses with smarts. :) | |
| \- Tofumagate | |
| ## Dataset | |
| The Bigger Body (referred to as Ink v2.1, because that's still the internal name) mix is absolutely disgusting. It's even more cursed than the original Ink mix. | |
| <details> | |
| <summary>(Public) Original Datasets</summary> | |
| <!-- Start Generation Here --> | |
| <ul> | |
| <li><a href="https://huggingface.co/datasets/Fizzarolli/limarp-processed">Fizzarolli/limarp-processed</a></li> | |
| <li><a href="https://huggingface.co/datasets/Norquinal/OpenCAI">Norquinal/OpenCAI</a> - <code>two_users</code> split</li> | |
| <li><a href="https://huggingface.co/datasets/allura-org/Celeste1.x-data-mixture">allura-org/Celeste1.x-data-mixture</a></li> | |
| <li><a href="https://huggingface.co/datasets/mapsila/PIPPA-ShareGPT-formatted-named">mapsila/PIPPA-ShareGPT-formatted-named</a></li> | |
| <li><a href="https://huggingface.co/datasets/allenai/tulu-3-sft-personas-instruction-following">allenai/tulu-3-sft-personas-instruction-following</a></li> | |
| <li><a href="https://huggingface.co/datasets/readmehay/medical-01-reasoning-SFT-json">readmehay/medical-01-reasoning-SFT-json</a></li> | |
| <li><a href="https://huggingface.co/datasets/LooksJuicy/ruozhiba">LooksJuicy/ruozhiba</a></li> | |
| <li><a href="https://huggingface.co/datasets/shibing624/roleplay-zh-sharegpt-gpt4-data">shibing624/roleplay-zh-sharegpt-gpt4-data</a></li> | |
| <li><a href="https://huggingface.co/datasets/CausalLM/Retrieval-SFT-Chat">CausalLM/Retrieval-SFT-Chat</a></li> | |
| <li><a href="https://huggingface.co/datasets/ToastyPigeon/fujin-filtered-instruct">ToastyPigeon/fujin-filtered-instruct</a></li> | |
| </ul> | |
| </details> | |
| ## Quants | |
| TODO! | |
| ## Recommended Settings | |
| Chat template: Mistral *v7-tekken* (NOT v3-tekken !!!! the main difference is that v7 has specific `[SYSTEM_PROMPT]` and `[/SYSTEM_PROMPT]` tags) | |
| Recommended samplers (not the be-all-end-all, try some on your own!): | |
| - Temp 1.25 / MinP 0.1 | |
| ## Hyperparams | |
| ### General | |
| - Epochs = 2 | |
| - LR = 1e-5 | |
| - LR Scheduler = Cosine | |
| - Optimizer = [Apollo-mini](https://github.com/zhuhanqing/APOLLO) | |
| - Optimizer target modules = `all_linear` | |
| - Effective batch size = 16 | |
| - Weight Decay = 0.01 | |
| - Warmup steps = 50 | |
| - Total steps = 920 | |
| ## Credits | |
| Humongous thanks to the people who created the data. I would credit you all, but that would be cheating ;) | |
| Big thanks to all Allura members for testing and emotional support ilya /platonic |