Update README.md
Browse files
README.md
CHANGED
|
@@ -83,14 +83,14 @@ Breeze-7B-Instruct-64k-v0.1 can solve tasks such as question answering and summa
|
|
| 83 |
|
| 84 |
**Category ACC of TMMLU+ (5 shot)**
|
| 85 |
|
| 86 |
-
| Models | STEM | Social Science | Humanities | Other |
|
| 87 |
-
|
| 88 |
-
| Yi-34B
|
| 89 |
-
| Qwen-14B
|
| 90 |
-
| Yi-6B
|
| 91 |
-
| Qwen-7B
|
| 92 |
-
| **Breeze-7B-Base-v0.1**
|
| 93 |
-
| Mistral-7B-v0.1
|
| 94 |
|
| 95 |
|
| 96 |
|
|
@@ -123,8 +123,8 @@ Breeze-7B-Instruct-64k-v0.1 can solve tasks such as question answering and summa
|
|
| 123 |
|
| 124 |
**Category Score of MT-Bench-tw (0 shot)**
|
| 125 |
|
| 126 |
-
| Models | STEM |Extraction|Reasoning| Math | Coding | Roleplay| Writing |Humanities|
|
| 127 |
-
|
| 128 |
| gpt-3.5-turbo | | | | | | | | | |
|
| 129 |
| Yi-34B-Chat | | | | | | | | | |
|
| 130 |
| Qwen-14B-Chat | | | | | | | | | |
|
|
@@ -137,17 +137,17 @@ Breeze-7B-Instruct-64k-v0.1 can solve tasks such as question answering and summa
|
|
| 137 |
|
| 138 |
**Category ACC of TMMLU+ (0 shot)**
|
| 139 |
|
| 140 |
-
| Model | STEM | Social Science | Humanities | Other |
|
| 141 |
|-----------------------------------------------------|--------------|----------------|------------|------------|---------|
|
| 142 |
-
|
|
| 143 |
-
|
|
| 144 |
-
|
|
| 145 |
-
|
|
| 146 |
-
| **Breeze-7B-Instruct-
|
| 147 |
-
|
|
| 148 |
-
|
|
| 149 |
-
| Taiwan-LLM-13B-v2.0-chat | 27.74 | 33.69 | 27.03 | 29.43 |
|
| 150 |
-
| Taiwan-LLM-7B-v2.1-chat | 25.58 | 31.76 | 27.36 | 27.61 |
|
| 151 |
|
| 152 |
|
| 153 |
|
|
@@ -157,15 +157,15 @@ All inferences run on 2 RTX A6000 GPUs (using `vllm`, with a tensor-parallel siz
|
|
| 157 |
|
| 158 |
| Models | Inference Time (sec)|Estimated Max Input Length (Char)|
|
| 159 |
|--------------------------------------------------------------------|-------------------|--------------------------|
|
| 160 |
-
| Yi-6B
|
| 161 |
-
| **Breeze-7B-Instruct-v0.1**
|
| 162 |
-
| **Breeze-7B-Instruct-64k-v0.1**
|
| 163 |
-
| Qwen-7B
|
| 164 |
-
| Qwen-14B
|
| 165 |
-
| Mistral-7B-v0.1
|
| 166 |
-
| Taiwan-LLM-7B-v2.1-base
|
| 167 |
-
| Taiwan-LLM-13B-v2.0-base
|
| 168 |
-
| Yi-34B
|
| 169 |
|
| 170 |
## Long-context Performance
|
| 171 |
|
|
@@ -209,3 +209,14 @@ The suggested default `SYS_PROMPT` is
|
|
| 209 |
```txt
|
| 210 |
You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan.
|
| 211 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 83 |
|
| 84 |
**Category ACC of TMMLU+ (5 shot)**
|
| 85 |
|
| 86 |
+
| Models | STEM | Social Science | Humanities | Other | AVG |
|
| 87 |
+
|----------------------------------|--------------|----------------|------------|------------|-------|
|
| 88 |
+
| Yi-34B | 56.03 | 73.06 | 61.12 | 62.19 | 63.10 |
|
| 89 |
+
| Qwen-14B | 46.51 | 58.20 | 51.12 | 49.38 | 51.30 |
|
| 90 |
+
| Yi-6B | 41.14 | 57.77 | 50.22 | 49.39 | 49.63 |
|
| 91 |
+
| Qwen-7B | 28.25 | 47.80 | 43.14 | 42.17 | 42.84 |
|
| 92 |
+
| **Breeze-7B-Base-v0.1** | 35.74 | 46.08 | 40.29 | 39.27 | 40.35 |
|
| 93 |
+
| Mistral-7B-v0.1 | 33.01 | 42.23 | 35.86 | 37.63 | 36.93 |
|
| 94 |
|
| 95 |
|
| 96 |
|
|
|
|
| 123 |
|
| 124 |
**Category Score of MT-Bench-tw (0 shot)**
|
| 125 |
|
| 126 |
+
| Models | STEM |Extraction|Reasoning| Math | Coding | Roleplay| Writing |Humanities|AVG |
|
| 127 |
+
|-----------------------------------------------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|
|
| 128 |
| gpt-3.5-turbo | | | | | | | | | |
|
| 129 |
| Yi-34B-Chat | | | | | | | | | |
|
| 130 |
| Qwen-14B-Chat | | | | | | | | | |
|
|
|
|
| 137 |
|
| 138 |
**Category ACC of TMMLU+ (0 shot)**
|
| 139 |
|
| 140 |
+
| Model | STEM | Social Science | Humanities | Other | AVG |
|
| 141 |
|-----------------------------------------------------|--------------|----------------|------------|------------|---------|
|
| 142 |
+
| Yi-34B-Chat | 47.65 | 64.25 | 52.73 | 54.91 | 54.87 |
|
| 143 |
+
| Qwen-14B-Chat | 43.83 | 55.00 | 48.55 | 46.22 | 48.41 |
|
| 144 |
+
| Yi-6B-Chat | 37.80 | 51.74 | 45.36 | 44.25 | 44.79 |
|
| 145 |
+
| gpt-3.5-turbo | 41.56 | 46.72 | 36.73 | 42.03 | 41.76 |
|
| 146 |
+
| **Breeze-7B-Instruct-v0.1** | 37.41 | 46.81 | 42.06 | 40.16 | 41.61 |
|
| 147 |
+
| **Breeze-7B-Instruct-64k-v0.1** | 37.88 | 46.35 | 40.31 | 39.40 | 40.99 |
|
| 148 |
+
| Qwen-7B-Chat | 35.44 | 46.22 | 38.35 | 40.06 | 40.02 |
|
| 149 |
+
| Taiwan-LLM-13B-v2.0-chat | 27.74 | 33.69 | 27.03 | 29.43 | 29.47 |
|
| 150 |
+
| Taiwan-LLM-7B-v2.1-chat | 25.58 | 31.76 | 27.36 | 27.61 | 28.08 |
|
| 151 |
|
| 152 |
|
| 153 |
|
|
|
|
| 157 |
|
| 158 |
| Models | Inference Time (sec)|Estimated Max Input Length (Char)|
|
| 159 |
|--------------------------------------------------------------------|-------------------|--------------------------|
|
| 160 |
+
| Yi-6B | 10.62 | 5.2k |
|
| 161 |
+
| **Breeze-7B-Instruct-v0.1** | 10.74 | 11.1k |
|
| 162 |
+
| **Breeze-7B-Instruct-64k-v0.1** | 10.74 | 88.8k |
|
| 163 |
+
| Qwen-7B | 10.86 | 9.8k |
|
| 164 |
+
| Qwen-14B | 18.89 | 9.8k |
|
| 165 |
+
| Mistral-7B-v0.1 | 20.48 | 5.1k |
|
| 166 |
+
| Taiwan-LLM-7B-v2.1-base | 26.26 | 2.2k |
|
| 167 |
+
| Taiwan-LLM-13B-v2.0-base | 36.80 | 2.2k |
|
| 168 |
+
| Yi-34B | 43.71 | 4.5k |
|
| 169 |
|
| 170 |
## Long-context Performance
|
| 171 |
|
|
|
|
| 209 |
```txt
|
| 210 |
You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan.
|
| 211 |
```
|
| 212 |
+
|
| 213 |
+
## Citation
|
| 214 |
+
|
| 215 |
+
```
|
| 216 |
+
@article{breeze7b2024,
|
| 217 |
+
title={},
|
| 218 |
+
author={},
|
| 219 |
+
journal={arXiv},
|
| 220 |
+
year={2024}
|
| 221 |
+
}
|
| 222 |
+
```
|