mohan11111 commited on
Commit
d8fd35d
·
verified ·
1 Parent(s): 4e53bee

Upload folder using huggingface_hub

Browse files
README.md ADDED
@@ -0,0 +1,366 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: seallms
4
+ license_link: https://huggingface.co/SeaLLMs/SeaLLM-13B-Chat/blob/main/LICENSE
5
+ language:
6
+ - en
7
+ - zh
8
+ - vi
9
+ - id
10
+ - th
11
+ - ms
12
+ - km
13
+ - lo
14
+ - my
15
+ - tl
16
+ tags:
17
+ - multilingual
18
+ - sea
19
+ ---
20
+
21
+ <p align="center">
22
+ <img src="seal_logo.png" width="200" />
23
+ </p>
24
+
25
+ # *SeaLLM-7B-v2* - Large Language Models for Southeast Asia
26
+
27
+ # <strong style="color: red">BIG NEWS: <a href="https://huggingface.co/SeaLLMs/SeaLLM3-7B-Chat">SeaLLM3</a> is released with state-of-the-art performance of diverse tasks while specifically enhanced to be more trustworthy. Please consider using that latest model version.</strong>
28
+
29
+
30
+ <p align="center">
31
+ <a href="https://damo-nlp-sg.github.io/SeaLLMs/" target="_blank" rel="noopener">Technical Blog</a>
32
+ &nbsp;&nbsp;
33
+ <a href="https://huggingface.co/SeaLLMs/SeaLLM-7B-v2" target="_blank" rel="noopener"> 🤗 Tech Memo</a>
34
+ &nbsp;&nbsp;
35
+ <a href="https://huggingface.co/spaces/SeaLLMs/SeaLLM-7B" target="_blank" rel="noopener"> 🤗 DEMO</a>
36
+ &nbsp;&nbsp;
37
+ <a href="https://github.com/DAMO-NLP-SG/SeaLLMs" target="_blank" rel="noopener">Github</a>
38
+ &nbsp;&nbsp;
39
+ <a href="https://arxiv.org/pdf/2312.00738.pdf" target="_blank" rel="noopener">Technical Report</a>
40
+ </p>
41
+
42
+ We introduce [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2), the state-of-the-art multilingual LLM for Southeast Asian (SEA) languages 🇬🇧 🇨🇳 🇻🇳 🇮🇩 🇹🇭 🇲🇾 🇰🇭 🇱🇦 🇲🇲 🇵🇭. It is the most significant upgrade since [SeaLLM-13B](https://huggingface.co/SeaLLMs/SeaLLM-13B-Chat), with half the size, outperforming performance across diverse multilingual tasks, from world knowledge, math reasoning, instruction following, etc.
43
+
44
+ ### Highlights
45
+ * [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) achieves the **7B-SOTA** on the **Zero-shot CoT GSM8K** task with **78.2** score and outperforms GPT-3.5 in many GSM8K-translated tasks in SEA languages (🇨🇳 🇻🇳 🇮🇩 🇹🇭) as well as MGSM (🇨🇳 🇹🇭). It also surpasses GPT-3.5 in MATH CoT for Thai 🇹🇭.
46
+ * It scores competitively against GPT-3.5 in many zero-shot CoT commonsense benchmark, with **82.5, 68.3, 80.9** scores on Arc-C, Winogrande, and Hellaswag.
47
+ * It achieves **7.54** score on the 🇬🇧 **MT-bench**, it ranks 3rd place on the leaderboard for 7B category and is the most outperforming multilingual model.
48
+ * It scores **45.74** on the VMLU benchmark for Vietnamese 🇻🇳, and is the only open-source multilingual model that can be competitive to monolingual models ([Vistral-7B](https://huggingface.co/Viet-Mistral/Vistral-7B-Chat)) of similar sizes.
49
+
50
+
51
+ ### Release and DEMO
52
+
53
+ - DEMO: [SeaLLMs/SeaLLM-7B](https://huggingface.co/spaces/SeaLLMs/SeaLLM-7B).
54
+ - Technical report: [Arxiv: SeaLLMs - Large Language Models for Southeast Asia](https://arxiv.org/pdf/2312.00738.pdf).
55
+ - Model weights:
56
+ - [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2).
57
+ - [SeaLLM-7B-v2-gguf](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf).
58
+ - [SeaLLM-7B-v2-GGUF (thanks Lonestriker)](https://huggingface.co/LoneStriker/SeaLLM-7B-v2-GGUF). NOTE: use [seallm.preset.json](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf/blob/main/seallm.preset.json) to work properly.
59
+ - Run locally:
60
+ - [LM-studio](https://lmstudio.ai/):
61
+ - [SeaLLM-7B-v2-q4_0](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.q4_0.gguf) and [SeaLLM-7B-v2-q8_0](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.q8_0.gguf).
62
+ - LM-studio requires this [seallm.preset.json](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf/blob/main/seallm.preset.json) to set chat template properly.
63
+ - [ollama](https://ollama.ai/) `ollama run nxphi47/seallm-7b-v2:q4_0`
64
+ - [MLX for Apple Silicon](https://github.com/ml-explore/mlx): [mlx-community/SeaLLM-7B-v2-4bit-mlx](https://huggingface.co/mlx-community/SeaLLM-7B-v2-4bit-mlx)
65
+
66
+ <blockquote style="color:red">
67
+ <p><strong style="color: red">Terms of Use and License</strong>:
68
+ By using our released weights, codes, and demos, you agree to and comply with the terms and conditions specified in our <a href="https://huggingface.co/SeaLLMs/SeaLLM-Chat-13b/edit/main/LICENSE" target="_blank" rel="noopener">SeaLLMs Terms Of Use</a>.
69
+ </blockquote>
70
+
71
+ > **Disclaimer**:
72
+ > We must note that even though the weights, codes, and demos are released in an open manner, similar to other pre-trained language models, and despite our best efforts in red teaming and safety fine-tuning and enforcement, our models come with potential risks, including but not limited to inaccurate, misleading or potentially harmful generation.
73
+ > Developers and stakeholders should perform their own red teaming and provide related security measures before deployment, and they must abide by and comply with local governance and regulations.
74
+ > In no event shall the authors be held liable for any claim, damages, or other liability arising from the use of the released weights, codes, or demos.
75
+
76
+ > The logo was generated by DALL-E 3.
77
+
78
+
79
+ ### What's new since SeaLLM-13B-v1 and SeaLLM-7B-v1?
80
+
81
+ * SeaLLM-7B-v2 is continue-pretrained from [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-v0.1) and underwent carefully designed tuning with focus in reasoning.
82
+
83
+
84
+ ## Evaluation
85
+
86
+
87
+ ### Zero-shot CoT Multilingual Math Reasoning
88
+
89
+ [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) achieves with **78.2** score on the GSM8K with zero-shot CoT reasoning, making it the **state of the art** in the realm of 7B models. It also outperforms GPT-3.5 in the same GSM8K benchmark as translated into SEA languages (🇨🇳 🇻🇳 🇮🇩 🇹🇭). [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) also surpasses GPT-3.5 on the Thai-translated MATH benchmark, with **22.4** vs 18.1 scores.
90
+
91
+ ![fig_sea_math_side_by_side.png](fig_sea_math_side_by_side.png)
92
+
93
+
94
+ <details>
95
+ <summary>See details on English and translated GSM8K and MATH with zero-shot reasoning</summary>
96
+ <br>
97
+
98
+ | Model | GSM8K<br>en | MATH<br>en | GSM8K<br>zh | MATH<br>zh | GSM8K<br>vi | MATH<br>vi | GSM8K<br>id | MATH<br>id | GSM8K<br>th | MATH<br>th
99
+ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
100
+ | GPT-3.5 | 80.8 | 34.1 | 48.2 | 21.5 | 55 | 26.5 | 64.3 | 26.4 | 35.8 | 18.1
101
+ | Qwen-14B-chat | 61.4 | 18.4 | 41.6 | 11.8 | 33.6 | 3.6 | 44.7 | 8.6 | 22 | 6
102
+ | Vistral-7b-chat | 48.2 | 12.5 | | | 48.7 | 3.1 | | | |
103
+ | Qwen1.5-7B-chat | 56.8 | 15.3 | 40 | 2.7 | 37.7 | 9 | 36.9 | 7.7 | 21.9 |
104
+ | SeaLLM-7B-v2 | 78.2 | 27.5 | 53.7 | 17.6 | 69.9 | 23.8 | 71.5 | 24.4 | 59.6 | 22.4
105
+
106
+ </details>
107
+
108
+ Baselines were evaluated using their respective chat-template and system prompts ([Qwen1.5-7B-chat](https://huggingface.co/Qwen/Qwen1.5-7B-Chat/blob/main/tokenizer_config.json), [Vistral](https://huggingface.co/Viet-Mistral/Vistral-7B-Chat)).
109
+
110
+ #### Zero-shot MGSM
111
+
112
+ [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) also outperforms GPT-3.5 and Qwen-14B on the multilingual MGSM for Zh and Th.
113
+
114
+ | Model | MGSM-Zh | MGSM-Th
115
+ |-----| ----- | ---
116
+ | ChatGPT (reported) | 61.2 | 47.2
117
+ | Qwen-14B-chat | 59.6 | 28
118
+ | SeaLLM-7B-v2 | **64.8** | **62.4**
119
+
120
+
121
+ ### Zero-shot Commonsense Reasoning
122
+
123
+ We compare [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) with ChatGPT and Mistral-7B-instruct on various zero-shot commonsense benchmarks (Arc-Challenge, Winogrande and Hellaswag). We use the 2-stage technique in [(Kojima et al., 2023)](https://arxiv.org/pdf/2205.11916.pdf) to grab the answer. Note that we **DID NOT** use "Let's think step-by-step" to invoke explicit CoT.
124
+
125
+ | 0-shot reasoning | Arc-Challenge | Winogrande | Hellaswag
126
+ |-----| ----- | --- | -- |
127
+ | ChatGPT (reported) | 84.6* | 66.8* | 72.0*
128
+ | ChatGPT (reproduced)| 84.1 | 63.1 | 79.5
129
+ | Mistral-7B-Instruct | 68.1 | 56.4 | 45.6
130
+ | Qwen1.5-7B-chat | 79.3 | 59.4 | 69.3
131
+ | SeaLLM-7B-v2 | 82.5 | 68.3 | 80.9
132
+
133
+ Baselines were evaluated using their respective chat-template and system prompts ([Qwen1.5-7B-chat](https://huggingface.co/Qwen/Qwen1.5-7B-Chat/blob/main/tokenizer_config.json), [Mistral](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)).
134
+
135
+ ### Multilingual World Knowledge
136
+
137
+
138
+ We evaluate models on 3 benchmarks following the recommended default setups: 5-shot MMLU for En, 3-shot [M3Exam](https://arxiv.org/pdf/2306.05179.pdf) (M3e) for En, Zh, Vi, Id, Th, and zero-shot [VMLU](https://vmlu.ai/) for Vi.
139
+
140
+ | Model | Langs | En<br>MMLU | En<br>M3e | Zh<br>M3e | Vi<br>M3e | Vi<br>VMLU | Id<br>M3e | Th<br>M3e
141
+ |-----| ----- | --- | -- | ----- | ---- | --- | --- | --- |
142
+ | GPT-3.5 | Multi | 68.90 | 75.46 | 60.20 | 58.64 | 46.32 | 49.27 | 37.41
143
+ | Vistral-7B-chat | Mono | 56.86 | 67.00 | 44.56 | 54.33 | 50.03 | 36.49 | 25.27
144
+ | Qwen1.5-7B-chat | Multi | 61.00 | 52.07 | 81.96 | 43.38 | 45.02 | 24.29 | 20.25
145
+ | SeaLLM-7B-v2 | Multi | 61.89 | 70.91 | 55.43 | 51.15 | 45.74 | 42.25 | 35.52
146
+
147
+
148
+ VMLU reproduce script [here](https://github.com/DAMO-NLP-SG/SeaLLMs/blob/main/evaluation/vmlu/vmlu_run.py). Lm-eval was used to evaluate MMLU.
149
+ 0-shot VMLU scores for baselines were evaluated using their respective chat-template and system prompts ([Qwen1.5-7B-chat](https://huggingface.co/Qwen/Qwen1.5-7B-Chat/blob/main/tokenizer_config.json)).
150
+
151
+
152
+ ### MT-Bench
153
+
154
+ On the English [MT-bench](https://arxiv.org/abs/2306.05685) metric, SeaLLM-7B-v2 achieves **7.54** score on the MT-bench (3rd place on the leaderboard for 7B category), outperforms many 70B models and is arguably the only one that handles 10 SEA languages.
155
+
156
+ Refer to [mt_bench/seallm_7b_v2.jsonl](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2/blob/main/evaluation/mt_bench/seallm_7b_v2.jsonl) for the MT-bench predictions of SeaLLM-7B-v2, and [here](https://github.com/lm-sys/FastChat/issues/3013#issue-2118685341) to reproduce it.
157
+
158
+ | Model | Access | Langs | MT-Bench
159
+ | --- | --- | --- | --- |
160
+ | GPT-4-turbo | closed | multi | 9.32
161
+ | GPT-4-0613 | closed | multi | 9.18
162
+ | Mixtral-8x7b (46B) | open | multi | 8.3
163
+ | Starling-LM-7B-alpha | open | mono (en) | 8.0
164
+ | OpenChat-3.5-7B | open | mono (en) | 7.81
165
+ | **SeaLLM-7B-v2** | **open** | **multi (10+)** | **7.54**
166
+ | [Qwen-14B](https://huggingface.co/Qwen/Qwen-14B-Chat) | open | multi | 6.96
167
+ | [Llama-2-70B](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) | open | mono (en) | 6.86
168
+ | Mistral-7B-instuct | open | mono (en) | 6.84
169
+
170
+
171
+ ### Sea-Bench
172
+
173
+ Similar to MT-Bench, [Sea-bench](https://huggingface.co/datasets/SeaLLMs/Sea-bench) is a set of categorized instruction test sets to measure models' ability as an assistant that is specifically focused on 9 SEA languages, including non-Latin low-resource languages.
174
+
175
+ As shown, the huge improvements come from math-reasoning, reaching GPT-3.5 level of performance.
176
+
177
+ ![fig_sea_bench_side_by_side.png](fig_sea_bench_side_by_side.png)
178
+
179
+ Refer to [sea_bench/seallm_7b_v2.jsonl](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2/blob/main/evaluation/sea_bench/seallm_7b_v2.jsonl) for the Sea-bench predictions of SeaLLM-7B-v2.
180
+
181
+
182
+ ### Usage
183
+
184
+ #### Instruction format
185
+
186
+ ```python
187
+ prompt = """<|im_start|>system
188
+ You are a helpful assistant.</s><|im_start|>user
189
+ Hello world</s><|im_start|>assistant
190
+ Hi there, how can I help?</s>"""
191
+
192
+ # NOTE: previous commit has \n between </s> and <|im_start|>, that was incorrect!
193
+ # <|im_start|> is not a special token.
194
+ # Transformers chat_template should be consistent with vLLM format below.
195
+
196
+ # ! ENSURE 1 and only 1 bos `<s>` at the beginning of sequence
197
+ print(tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt)))
198
+
199
+ '<s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'system', '<0x0A>', 'You', '▁are', '▁a', '▁helpful', '▁assistant', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'user', '<0x0A>', 'Hello', '▁world', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'ass', 'istant', '<0x0A>', 'Hi', '▁there', ',', '▁how', '▁can', '▁I', '▁help', '?', '</s>']
200
+ """
201
+ ```
202
+
203
+ #### Using transformers's chat_template
204
+ ```python
205
+
206
+ from transformers import AutoModelForCausalLM, AutoTokenizer
207
+
208
+ device = "cuda" # the device to load the model onto
209
+
210
+ # use bfloat16 to ensure the best performance.
211
+ model = AutoModelForCausalLM.from_pretrained("SeaLLMs/SeaLLM-7B-v2", torch_dtype=torch.bfloat16, device_map=device)
212
+ tokenizer = AutoTokenizer.from_pretrained("SeaLLMs/SeaLLM-7B-v2")
213
+
214
+ messages = [
215
+ {"role": "system", "content": "You are a helpful assistant."},
216
+ {"role": "user", "content": "Hello world"},
217
+ {"role": "assistant", "content": "Hi there, how can I help you today?"},
218
+ {"role": "user", "content": "Explain general relativity in details."}
219
+ ]
220
+
221
+ encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
222
+ print(tokenizer.convert_ids_to_tokens(encodeds[0]))
223
+ # ['<s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'system', '<0x0A>', 'You', '▁are', '▁a', '▁helpful', '▁assistant', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'user', '<0x0A>', 'Hello', '▁world', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'ass', 'istant', '<0x0A>', 'Hi', '▁there', ',', '▁how', '▁can', '▁I', '▁help', '▁you', '▁today', '?', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'user', '<0x0A>', 'Ex', 'plain', '▁general', '▁rel', 'ativity', '▁in', '▁details', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'ass', 'istant', '<0x0A>']
224
+
225
+ model_inputs = encodeds.to(device)
226
+ model.to(device)
227
+
228
+ generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True, pad_token_id=tokenizer.pad_token_id)
229
+ decoded = tokenizer.batch_decode(generated_ids)
230
+ print(decoded[0])
231
+
232
+ ```
233
+
234
+ #### Using vLLM
235
+
236
+ ```python
237
+ from vllm import LLM, SamplingParams
238
+ TURN_TEMPLATE = "<|im_start|>{role}\n{content}</s>"
239
+ TURN_PREFIX = "<|im_start|>{role}\n"
240
+
241
+ # There is no \n between </s> and <|im_start|>.
242
+
243
+ def seallm_chat_convo_format(conversations, add_assistant_prefix: bool, system_prompt=None):
244
+ # conversations: list of dict with key `role` and `content` (openai format)
245
+ if conversations[0]['role'] != 'system' and system_prompt is not None:
246
+ conversations = [{"role": "system", "content": system_prompt}] + conversations
247
+ text = ''
248
+ for turn_id, turn in enumerate(conversations):
249
+ prompt = TURN_TEMPLATE.format(role=turn['role'], content=turn['content'])
250
+ text += prompt
251
+ if add_assistant_prefix:
252
+ prompt = TURN_PREFIX.format(role='assistant')
253
+ text += prompt
254
+ return text
255
+
256
+ sparams = SamplingParams(temperature=0.1, max_tokens=1024, stop=['</s>', '<|im_start|>'])
257
+ llm = LLM("SeaLLMs/SeaLLM-7B-v2", dtype="bfloat16")
258
+
259
+ message = "Explain general relativity in details."
260
+ prompt = seallm_chat_convo_format(message, True)
261
+ gen = llm.generate(prompt, sampling_params)
262
+
263
+ print(gen[0].outputs[0].text)
264
+ ```
265
+
266
+ #### Fine-tuning SeaLLM-7B-v2
267
+
268
+ Should follow the chat format and accurately mask out source tokens. Here is an example.
269
+
270
+ ```python
271
+ conversations = [
272
+ {"role": "system", "content": "You are helful assistant."},
273
+ {"role": "user", "content": "Hello world."},
274
+ {"role": "assistant", "content": "Hi there, how can I help?"},
275
+ {"role": "user", "content": "Tell me a joke."},
276
+ {"role": "assistant", "content": "Why don't scientists trust atoms? Because they make up everything."},
277
+ ]
278
+ def seallm_7b_v2_tokenize_multi_turns(tokenizer, conversations, add_assistant_prefix=False):
279
+ """
280
+ Inputs:
281
+ conversations: list of dict following openai format, eg
282
+ conversations = [
283
+ {"role": "system", "content": "You are helful assistant."},
284
+ {"role": "user", "content": "Hello world."},
285
+ {"role": "assistant", "content": "Hi there, how can I help?"},
286
+ {"role": "user", "content": "Tell me a joke."},
287
+ {"role": "assistant", "content": "Why don't scientists trust atoms? Because they make up everything."},
288
+ ]
289
+ add_assistant_prefix: whether to add assistant_prefix, only for inference decoding
290
+ Outputs:
291
+ tokenize_output_sample, {
292
+ "input_ids": ...
293
+ "token_type_ids": 1 if train and 0 if masked out (not train)
294
+ }
295
+ During training, need to create a labels, with masked-out tokens = -100 to avoid loss computations.
296
+ labels = sample['input_ids'].clone()
297
+ labels[sample['token_type_ids'] == 0] = -100
298
+ """
299
+ TURN_TEMPLATE = "<|im_start|>{role}\n{content}</s>"
300
+ TURN_PREFIX = "<|im_start|>{role}\n"
301
+ sample = None
302
+ assistant_prefix_len = None
303
+ for turn_id, turn in enumerate(conversations):
304
+ prompt = TURN_TEMPLATE.format(role=turn['role'], content=turn['content'])
305
+ turn_sample = tokenizer(
306
+ prompt, padding=False, truncation=False, verbose=False, add_special_tokens=False,
307
+ return_token_type_ids=True,
308
+ )
309
+ if turn['role'] == 'assistant':
310
+ if assistant_prefix_len is None:
311
+ assistant_prefix_len = len(tokenizer.encode(TURN_PREFIX.format(role=turn['role']), add_special_tokens=False))
312
+ turn_sample['token_type_ids'][assistant_prefix_len:] = [1] * (len(turn_sample['input_ids']) - assistant_prefix_len)
313
+ if sample is None:
314
+ sample = turn_sample
315
+ else:
316
+ for k in turn_sample.keys():
317
+ sample[k].extend(turn_sample[k])
318
+ if add_assistant_prefix:
319
+ assistant_prefix_sample = tokenizer(
320
+ TURN_PREFIX.format(role="assistant"), padding=False, truncation=False, verbose=False, add_special_tokens=False,
321
+ return_token_type_ids=True,
322
+ )
323
+ for k in sample.keys():
324
+ sample[k].extend(assistant_prefix_sample[k])
325
+ if tokenizer.add_bos_token:
326
+ sample['input_ids'] = [tokenizer.bos_token_id] + sample['input_ids']
327
+ sample['attention_mask'] = [1] + sample['attention_mask']
328
+ sample['token_type_ids'] = [sample['token_type_ids'][0]] + sample['token_type_ids']
329
+ return sample
330
+
331
+ # ! testing
332
+ sample = seallm_7b_v2_tokenize_multi_turns(tokenizer, conversations)
333
+ print(tokenizer.convert_ids_to_tokens(sample['input_ids']))
334
+ print(sample['token_type_ids'])
335
+ # ['<s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'system', '<0x0A>', 'You', '▁are', '▁hel', 'ful', '▁assistant', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'user', '<0x0A>', 'Hello', '▁world', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'ass', 'istant', '<0x0A>', 'Hi', '▁there', ',', '▁how', '▁can', '▁I', '▁help', '?', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'user', '<0x0A>', 'Tell', '▁me', '▁a', '▁joke', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'ass', 'istant', '<0x0A>', 'Why', '▁don', "'", 't', '▁scientists', '▁trust', '▁atoms', '?', '▁Because', '▁they', '▁make', '▁up', '▁everything', '.', '</s>']
336
+ # [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
337
+
338
+
339
+
340
+ ```
341
+
342
+
343
+ ## Acknowledgement to Our Linguists
344
+
345
+ We would like to express our special thanks to our professional and native linguists, Tantong Champaiboon, Nguyen Ngoc Yen Nhi and Tara Devina Putri, who helped build, evaluate, and fact-check our sampled pretraining and SFT dataset as well as evaluating our models across different aspects, especially safety.
346
+
347
+ ## Citation
348
+
349
+ If you find our project useful, we hope you would kindly star our repo and cite our work as follows: Corresponding Author: [l.bing@alibaba-inc.com](mailto:l.bing@alibaba-inc.com)
350
+
351
+ **Author list and order will change!**
352
+
353
+ * `*` and `^` are equal contributions.
354
+
355
+ ```
356
+ @article{damonlpsg2023seallm,
357
+ author = {Xuan-Phi Nguyen*, Wenxuan Zhang*, Xin Li*, Mahani Aljunied*,
358
+ Zhiqiang Hu, Chenhui Shen^, Yew Ken Chia^, Xingxuan Li, Jianyu Wang,
359
+ Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang,
360
+ Chaoqun Liu, Hang Zhang, Lidong Bing},
361
+ title = {SeaLLMs - Large Language Models for Southeast Asia},
362
+ year = 2023,
363
+ Eprint = {arXiv:2312.00738},
364
+ }
365
+ ```
366
+
config.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "seallm_dpo",
3
+ "architectures": [
4
+ "MistralForCausalLM"
5
+ ],
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 1,
8
+ "eos_token_id": 2,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 4096,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 14336,
13
+ "max_position_embeddings": 32768,
14
+ "model_type": "mistral",
15
+ "num_attention_heads": 32,
16
+ "num_hidden_layers": 32,
17
+ "num_key_value_heads": 8,
18
+ "rms_norm_eps": 1e-05,
19
+ "rope_theta": 10000.0,
20
+ "sliding_window": 4096,
21
+ "tie_word_embeddings": false,
22
+ "torch_dtype": "bfloat16",
23
+ "transformers_version": "4.37.0.dev0",
24
+ "use_cache": true,
25
+ "vocab_size": 48384
26
+ }
fig_sea_bench_side_by_side.png ADDED
fig_sea_math_side_by_side.png ADDED
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:618fca4435734f7eda752da105fd9b39214b9820aa0a6559db32b5bbcc2d86df
3
+ size 28711192234
seal_logo.png ADDED
special_tokens_map.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "</s>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": "<unk>",
17
+ "unk_token": {
18
+ "content": "<unk>",
19
+ "lstrip": false,
20
+ "normalized": false,
21
+ "rstrip": false,
22
+ "single_word": false
23
+ }
24
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4d88bdadaa2a065aa7c6e18a4b5999ce4c76cec14d9fea882102e7b4931d7ef0
3
+ size 779539
tokenizer_config.json ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": true,
3
+ "add_eos_token": false,
4
+ "added_tokens_decoder": {
5
+ "0": {
6
+ "content": "<unk>",
7
+ "lstrip": false,
8
+ "normalized": false,
9
+ "rstrip": false,
10
+ "single_word": false,
11
+ "special": true
12
+ },
13
+ "1": {
14
+ "content": "<s>",
15
+ "lstrip": false,
16
+ "normalized": false,
17
+ "rstrip": false,
18
+ "single_word": false,
19
+ "special": true
20
+ },
21
+ "2": {
22
+ "content": "</s>",
23
+ "lstrip": false,
24
+ "normalized": false,
25
+ "rstrip": false,
26
+ "single_word": false,
27
+ "special": true
28
+ }
29
+ },
30
+ "additional_special_tokens": [],
31
+ "bos_token": "<s>",
32
+ "clean_up_tokenization_spaces": false,
33
+ "eos_token": "</s>",
34
+ "legacy": true,
35
+ "model_max_length": 1000000000000000019884624838656,
36
+ "pad_token": "<unk>",
37
+ "sp_model_kwargs": {},
38
+ "spaces_between_special_tokens": false,
39
+ "tokenizer_class": "LlamaTokenizer",
40
+ "unk_token": "<unk>",
41
+ "use_default_system_prompt": false,
42
+ "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{{ bos_token }}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '</s>'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
43
+ }