Update README.md
Browse files
README.md
CHANGED
@@ -12,9 +12,9 @@ model-index:
|
|
12 |
# stablelm-4e1t-2b-v0.1
|
13 |
|
14 |
|
15 |
-
This is a layer pruning experiment based off of
|
16 |
|
17 |
-
-
|
18 |
- layers selected using [BEE-spoke-data/fineweb-100k_en-med](https://hf.co/datasets/BEE-spoke-data/fineweb-100k_en-med)
|
19 |
- brief subsequent continued pretraining @ ctx 4096
|
20 |
- data: 10k rows of FineWeb (different than pruning data) + some curated data
|
|
|
12 |
# stablelm-4e1t-2b-v0.1
|
13 |
|
14 |
|
15 |
+
This is a layer pruning experiment based off of [stablelm-3b-4e1t](https://huggingface.co/stabilityai/stablelm-3b-4e1t):
|
16 |
|
17 |
+
- 10 layers pruned with [PruneMe](https://github.com/pszemraj/PruneMe/tree/upgrades)/MergeKit
|
18 |
- layers selected using [BEE-spoke-data/fineweb-100k_en-med](https://hf.co/datasets/BEE-spoke-data/fineweb-100k_en-med)
|
19 |
- brief subsequent continued pretraining @ ctx 4096
|
20 |
- data: 10k rows of FineWeb (different than pruning data) + some curated data
|