Spaces:
Running
Running
Update README.md
Browse files
README.md
CHANGED
|
@@ -7,4 +7,45 @@ sdk: static
|
|
| 7 |
pinned: false
|
| 8 |
---
|
| 9 |
|
| 10 |
-
The merge crew is the mergiest crew.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
pinned: false
|
| 8 |
---
|
| 9 |
|
| 10 |
+
The merge crew is the mergiest crew. 😎
|
| 11 |
+
|
| 12 |
+
## Merge Crew Planning Document
|
| 13 |
+
https://docs.google.com/document/d/1fP2FIrCifWcLGdTBmqeogdCdZJOwxqPfEyO-HA76_qc/edit?usp=sharing
|
| 14 |
+
|
| 15 |
+
## Merging tutorial
|
| 16 |
+
https://huggingface.co/blog/mlabonne/merge-models
|
| 17 |
+
|
| 18 |
+
## Colab for merging
|
| 19 |
+
Lazy merge kit notebook for merging models.
|
| 20 |
+
https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing
|
| 21 |
+
|
| 22 |
+
## Model Merging Discord
|
| 23 |
+
Discord channel for discussions of model mergin.
|
| 24 |
+
https://discord.com/channels/905500133343518791/1202582325146161183
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
## Merge methods
|
| 28 |
+
When working you use a YAML file like the following
|
| 29 |
+
|
| 30 |
+
```
|
| 31 |
+
models:
|
| 32 |
+
- model: timpal0l/BeagleCatMunin
|
| 33 |
+
# No parameters necessary for base model
|
| 34 |
+
- model: bineric/NorskGPT-Mistral-7b
|
| 35 |
+
parameters:
|
| 36 |
+
density: 0.53
|
| 37 |
+
weight: 0.6
|
| 38 |
+
merge_method: dare_ties
|
| 39 |
+
base_model: timpal0l/BeagleCatMunin
|
| 40 |
+
parameters:
|
| 41 |
+
int8_mask: true
|
| 42 |
+
dtype: bfloat16
|
| 43 |
+
```
|
| 44 |
+
|
| 45 |
+
The dare_ties methods seems to perform better than other merging methods.
|
| 46 |
+
Learn more about merge ties here.
|
| 47 |
+
https://arxiv.org/pdf/2306.01708.pdf
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
|