Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
### TeeZee/GALAXY-XB-v1.03 ###
|
| 3 |
+
|
| 4 |
+
Experiment, can DUS be taken one or more steps further?
|
| 5 |
+
|
| 6 |
+
### Technical notes:
|
| 7 |
+
- model v03 finetuned on 50k entries from SlimOrca dataset
|
| 8 |
+
- 12 layers removed from both models, 4 more than in original paper but its 1/4 of all layers(48) as per original paper.
|
| 9 |
+
- base version of upstage/SOLAR-10.7B-v1.0 used for merge
|
| 10 |
+
- no finetuning done yet, this is just a merge, first step in DUS paper
|
| 11 |
+
- next step, if evaluation proves that its at least as 'smart' as base model, should be finetuning to 'recover' after merge
|
| 12 |
+
|
| 13 |
+
#To evaluate
|
| 14 |
+
- model performance after finetuning, did it recover initial permofmance loss after merge?
|