Update README.md
Browse files
README.md
CHANGED
@@ -24,9 +24,9 @@ language:
|
|
24 |
---
|
25 |
## Why prune?
|
26 |
|
27 |
-
Falcon-11B is still undertrained, as can be seen by this graph:
|
28 |
![image/png](https://cdn-uploads.huggingface.co/production/uploads/660c0a02cf274b3ab77dd6b7/QeaL9bOrPskustzFpjMUP.png)
|
29 |
-
This is why the choice is made
|
30 |
Note that \~1B of continued pre-training (\~1M rows of 1k tokens) is still required to restore the perplexity of this model in the desired language.
|
31 |
I'm planning on doing that for certain languages, depending on how much compute will be available.
|
32 |
|
|
|
24 |
---
|
25 |
## Why prune?
|
26 |
|
27 |
+
Even though [Falcon-11B](https://huggingface.co/tiiuae/falcon-11B) is trained on 5T tokens, it is still undertrained, as can be seen by this graph:
|
28 |
![image/png](https://cdn-uploads.huggingface.co/production/uploads/660c0a02cf274b3ab77dd6b7/QeaL9bOrPskustzFpjMUP.png)
|
29 |
+
This is why the choice is made to prune 50% of the layers.
|
30 |
Note that \~1B of continued pre-training (\~1M rows of 1k tokens) is still required to restore the perplexity of this model in the desired language.
|
31 |
I'm planning on doing that for certain languages, depending on how much compute will be available.
|
32 |
|