| license: apache-2.0 | |
| tags: | |
| - merge | |
| - mergekit | |
| - lazymergekit | |
| # TinyLlamaHerd-2x1.1B | |
| TinyLlamaHerd-2x1.1B is a merge of the following models using [mergekit](https://github.com/cg123/mergekit): | |
| ## 🧩 Configuration | |
| ```yaml | |
| base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T | |
| gate_mode: random | |
| dtype: bfloat16 | |
| experts: | |
| - source_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T | |
| positive_prompts: [""] | |
| - source_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T | |
| positive_prompts: [""] | |
| ``` |