update links to new naming scheme
Browse files
README.md
CHANGED
@@ -19,7 +19,7 @@ SEA-LION stands for _Southeast Asian Languages In One Network_.
|
|
19 |
|
20 |
## Model Details
|
21 |
### Base model
|
22 |
-
We perform instruction tuning in English and Indonesian on our [pre-trained SEA-LION-7B](https://huggingface.co/aisingapore/
|
23 |
|
24 |
### Benchmark Performance
|
25 |
We evaluated SEA-LION-7B-Instruct on the BHASA benchmark ([arXiv](https://arxiv.org/abs/2309.06085v2) and [GitHub](https://github.com/aisingapore/bhasa)) across a variety of tasks.
|
@@ -28,15 +28,15 @@ BHASA stands out amongst other evaluations for SEA languages for its holistic ap
|
|
28 |
|
29 |
The scores shown in the table below have been adjusted to only consider answers provided in the appropriate language.
|
30 |
|
31 |
-
| Model
|
32 |
|--------------------------------|---------|----------------|---------------|-------------------|-------------------|-------------------|-----------|--------------|
|
33 |
-
| SEA-LION-7B-Instruct-Research
|
34 |
-
| SEA-LION-7B-Instruct
|
35 |
-
| SeaLLM 7B v1
|
36 |
-
| SeaLLM 7B v2
|
37 |
-
| Sailor-7B
|
38 |
-
| Llama 2 7B Chat
|
39 |
-
| Mistral 7B Instruct v0.1
|
40 |
| GPT-4 | 73.60 | 74.14 | 63.96 | 69.38 | 67.53 | 18.71 | 83.20 | 96.00 |
|
41 |
|
42 |
### Usage
|
@@ -46,8 +46,8 @@ SEA-LION can be run using the 🤗 Transformers library
|
|
46 |
|
47 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
48 |
|
49 |
-
tokenizer = AutoTokenizer.from_pretrained("aisingapore/
|
50 |
-
model = AutoModelForCausalLM.from_pretrained("aisingapore/
|
51 |
|
52 |
prompt_template = "### USER:\n{human_prompt}\n\n### RESPONSE:\n"
|
53 |
prompt = """Apa sentimen dari kalimat berikut ini?
|
|
|
19 |
|
20 |
## Model Details
|
21 |
### Base model
|
22 |
+
We perform instruction tuning in English and Indonesian on our [pre-trained SEA-LION-7B](https://huggingface.co/aisingapore/sea-lion-7b), a decoder model using the MPT architecture, to create SEA-LION-7B-Instruct.
|
23 |
|
24 |
### Benchmark Performance
|
25 |
We evaluated SEA-LION-7B-Instruct on the BHASA benchmark ([arXiv](https://arxiv.org/abs/2309.06085v2) and [GitHub](https://github.com/aisingapore/bhasa)) across a variety of tasks.
|
|
|
28 |
|
29 |
The scores shown in the table below have been adjusted to only consider answers provided in the appropriate language.
|
30 |
|
31 |
+
| Model | QA (F1) | Sentiment (F1) | Toxicity (F1) | Eng>Indo (ChrF++) | Indo>Eng (ChrF++) | Summary (ROUGE-L) | NLI (Acc) | Causal (Acc) |
|
32 |
|--------------------------------|---------|----------------|---------------|-------------------|-------------------|-------------------|-----------|--------------|
|
33 |
+
| SEA-LION-7B-Instruct-Research | 24.86 | 76.13 | 24.45 | 52.50 | 46.82 | 15.44 | 33.20 | 23.80 |
|
34 |
+
| SEA-LION-7B-Instruct | **68.41** | **91.45** | 17.98 | 57.48 | 58.04 | **17.54** | 53.10 | 60.80 |
|
35 |
+
| SeaLLM 7B v1 | 30.96 | 56.29 | 22.60 | 62.23 | 41.55 | 14.03 | 26.50 | 56.60 |
|
36 |
+
| SeaLLM 7B v2 | 44.40 | 80.13 | **55.24** | 64.01 | **63.28** | 17.31 | 43.60 | **82.00** |
|
37 |
+
| Sailor-7B | 65.43 | 59.48 | 20.48 | **64.27** | 60.68 | 8.69 | 15.10 | 38.40 |
|
38 |
+
| Llama 2 7B Chat | 11.12 | 52.32 | 0.00 | 44.09 | 57.58 | 9.24 | 0.00 | 0.00 |
|
39 |
+
| Mistral 7B Instruct v0.1 | 38.85 | 74.38 | 20.83 | 30.60 | 51.43 | 15.63 | **53.10** | 50.80 |
|
40 |
| GPT-4 | 73.60 | 74.14 | 63.96 | 69.38 | 67.53 | 18.71 | 83.20 | 96.00 |
|
41 |
|
42 |
### Usage
|
|
|
46 |
|
47 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
48 |
|
49 |
+
tokenizer = AutoTokenizer.from_pretrained("aisingapore/sea-lion-7b-instruct", trust_remote_code=True)
|
50 |
+
model = AutoModelForCausalLM.from_pretrained("aisingapore/sea-lion-7b-instruct", trust_remote_code=True)
|
51 |
|
52 |
prompt_template = "### USER:\n{human_prompt}\n\n### RESPONSE:\n"
|
53 |
prompt = """Apa sentimen dari kalimat berikut ini?
|