Update README.md
Browse files
README.md
CHANGED
@@ -8,16 +8,18 @@ pinned: false
|
|
8 |
short_description: CataLlama models official page
|
9 |
---
|
10 |
|
11 |
-
# CataLlama
|
12 |
|
13 |
**CataLlama is a fine-tune of Llama-3 8B on the Catalan language.**
|
14 |
|
15 |
-
CataLlama was trained on roughly **445 million new tokens** in three separate stages:
|
16 |
|
17 |
- **Language enhancement** with raw text - we could also call this "continued pre-training" at a very small scale.
|
18 |
- **Supervised fine-tuning** on instructions consisting of 70% Catalan Language and 30% English Language.
|
19 |
- **DPO fine-tuning** on preferences consisting of 70% Catalan language and 30% English Language.
|
20 |
|
|
|
|
|
21 |
**Note:** This model is not intended to beat benchmarks, but to demonstrate techniques for augmenting LLMs on new languages and preserve rare languages as part of our world heritage.
|
22 |
|
23 |
Three models and three respective datasets have been released.
|
|
|
8 |
short_description: CataLlama models official page
|
9 |
---
|
10 |
|
11 |
+
# CataLlama
|
12 |
|
13 |
**CataLlama is a fine-tune of Llama-3 8B on the Catalan language.**
|
14 |
|
15 |
+
CataLlama-v0.1 was trained on roughly **445 million new tokens** in three separate stages:
|
16 |
|
17 |
- **Language enhancement** with raw text - we could also call this "continued pre-training" at a very small scale.
|
18 |
- **Supervised fine-tuning** on instructions consisting of 70% Catalan Language and 30% English Language.
|
19 |
- **DPO fine-tuning** on preferences consisting of 70% Catalan language and 30% English Language.
|
20 |
|
21 |
+
CataLlama-v0.2 was trained on roughly **620 million new tokens** in a very similar manner to v0.1, except for the base model which is obtained via a merge.
|
22 |
+
|
23 |
**Note:** This model is not intended to beat benchmarks, but to demonstrate techniques for augmenting LLMs on new languages and preserve rare languages as part of our world heritage.
|
24 |
|
25 |
Three models and three respective datasets have been released.
|