NghiemAbe commited on
Commit
5a85665
1 Parent(s): 291f61d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +330 -13
README.md CHANGED
@@ -1,22 +1,339 @@
1
  ---
2
  language:
3
  - en
4
- license: apache-2.0
 
 
 
 
 
 
 
 
 
5
  tags:
6
- - text-generation-inference
7
- - transformers
8
- - unsloth
9
- - gemma
10
- - trl
11
- base_model: SeaLLMs/SeaLLM-7B-v2.5
12
  ---
13
 
14
- # Uploaded model
 
 
15
 
16
- - **Developed by:** NghiemAbe
17
- - **License:** apache-2.0
18
- - **Finetuned from model :** SeaLLMs/SeaLLM-7B-v2.5
19
 
20
- This gemma model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
21
 
22
- [<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  language:
3
  - en
4
+ - zh
5
+ - vi
6
+ - id
7
+ - th
8
+ - ms
9
+ - km
10
+ - lo
11
+ - my
12
+ - tl
13
+ license: other
14
  tags:
15
+ - multilingual
16
+ - sea
17
+ license_name: seallms
18
+ license_link: https://huggingface.co/SeaLLMs/SeaLLM-13B-Chat/blob/main/LICENSE
 
 
19
  ---
20
 
21
+ <p align="center">
22
+ <img src="seal_logo.png" width="200" />
23
+ </p>
24
 
25
+ # *SeaLLM-7B-v2.5* - Large Language Models for Southeast Asia
 
 
26
 
 
27
 
28
+ <p align="center">
29
+ <a href="https://damo-nlp-sg.github.io/SeaLLMs/" target="_blank" rel="noopener">Website</a>
30
+ &nbsp;&nbsp;
31
+ <a href="https://huggingface.co/SeaLLMs/SeaLLM-7B-v2.5" target="_blank" rel="noopener"> 🤗 Tech Memo</a>
32
+ &nbsp;&nbsp;
33
+ <a href="https://huggingface.co/spaces/SeaLLMs/SeaLLM-7B-v2.5" target="_blank" rel="noopener"> 🤗 DEMO</a>
34
+ &nbsp;&nbsp;
35
+ <a href="https://github.com/DAMO-NLP-SG/SeaLLMs" target="_blank" rel="noopener">Github</a>
36
+ &nbsp;&nbsp;
37
+ <a href="https://arxiv.org/pdf/2312.00738.pdf" target="_blank" rel="noopener">Technical Report</a>
38
+ </p>
39
+
40
+ 🔥<span style="color: #ff3860">[HOT]</span> SeaLLMs project now has a dedicated website - [damo-nlp-sg.github.io/SeaLLMs](https://damo-nlp-sg.github.io/SeaLLMs/)
41
+
42
+ We introduce [SeaLLM-7B-v2.5](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2.5), the state-of-the-art multilingual LLM for Southeast Asian (SEA) languages 🇬🇧 🇨🇳 🇻🇳 🇮🇩 🇹🇭 🇲🇾 🇰🇭 🇱🇦 🇲🇲 🇵🇭. It is the most significant upgrade since [SeaLLM-13B](https://huggingface.co/SeaLLMs/SeaLLM-13B-Chat), with half the size, outperforming performance across diverse multilingual tasks, from world knowledge, math reasoning, instruction following, etc.
43
+
44
+ ### Highlights
45
+ * [SeaLLM-7B-v2.5](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2.5) outperforms GPT-3.5 and achieves 7B SOTA on most multilingual knowledge benchmarks for SEA languages (MMLU, M3Exam & VMLU).
46
+ * It achieves 79.0 and 34.9 on GSM8K and MATH, surpassing GPT-3.5 in MATH.
47
+
48
+ ### Release and DEMO
49
+
50
+ - DEMO:
51
+ - [SeaLLMs/SeaLLM-7B-v2.5](https://huggingface.co/spaces/SeaLLMs/SeaLLM-7B-v2.5).
52
+ - [SeaLLMs/SeaLLM-7B | SeaLMMM-7B](https://huggingface.co/spaces/SeaLLMs/SeaLLM-7B) - Experimental multimodal SeaLLM.
53
+ - Technical report: [Arxiv: SeaLLMs - Large Language Models for Southeast Asia](https://arxiv.org/pdf/2312.00738.pdf).
54
+ - Model weights:
55
+ - [SeaLLM-7B-v2.5](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2.5).
56
+ - [SeaLLM-7B-v2.5-GGUF](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2.5-GGUF).
57
+ - Run locally:
58
+ - [LM-studio](https://lmstudio.ai/):
59
+ - [SeaLLM-7B-v2.5-q4_0-chatml](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2.5-GGUF/blob/main/seallm-7b-v2.5-chatml.Q4_K_M.gguf) with ChatML template (`<eos>` token changed to `<|im_end|>`)
60
+ - [SeaLLM-7B-v2.5-q4_0](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2.5-GGUF/blob/main/seallm-7b-v2.5.Q4_K_M.gguf) - must use SeaLLM-7B-v2.5 chat format.
61
+ - [MLX for Apple Silicon](https://github.com/ml-explore/mlx): [SeaLLMs/SeaLLM-7B-v2.5-mlx-quantized](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2.5-mlx-quantized)
62
+ - Previous models:
63
+ - [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2)
64
+ - [SeaLLM-7B-v1](https://huggingface.co/SeaLLMs/SeaLLM-7B-v1)
65
+
66
+ <blockquote style="color:red">
67
+ <p><strong style="color: red">Terms of Use and License</strong>:
68
+ By using our released weights, codes, and demos, you agree to and comply with the terms and conditions specified in our <a href="https://huggingface.co/SeaLLMs/SeaLLM-Chat-13b/edit/main/LICENSE" target="_blank" rel="noopener">SeaLLMs Terms Of Use</a>.
69
+ </blockquote>
70
+
71
+ > **Disclaimer**:
72
+ > We must note that even though the weights, codes, and demos are released in an open manner, similar to other pre-trained language models, and despite our best efforts in red teaming and safety fine-tuning and enforcement, our models come with potential risks, including but not limited to inaccurate, misleading or potentially harmful generation.
73
+ > Developers and stakeholders should perform their own red teaming and provide related security measures before deployment, and they must abide by and comply with local governance and regulations.
74
+ > In no event shall the authors be held liable for any claim, damages, or other liability arising from the use of the released weights, codes, or demos.
75
+
76
+ > The logo was generated by DALL-E 3.
77
+
78
+
79
+ ### What's new since SeaLLM-7B-v2?
80
+
81
+ * SeaLLM-7B-v2.5 was built on top of Gemma-7b, and underwent large scale SFT and carefully designed alignment.
82
+
83
+
84
+ ## Evaluation
85
+
86
+
87
+ ### Multilingual World Knowledge
88
+
89
+
90
+ We evaluate models on 3 benchmarks following the recommended default setups: 5-shot MMLU for En, 3-shot [M3Exam](https://arxiv.org/pdf/2306.05179.pdf) (M3e) for En, Zh, Vi, Id, Th, and zero-shot [VMLU](https://vmlu.ai/) for Vi.
91
+
92
+ | Model | Langs | En<br>MMLU | En<br>M3e | Zh<br>M3e | Vi<br>M3e | Vi<br>VMLU | Id<br>M3e | Th<br>M3e
93
+ |-----| ----- | --- | -- | ----- | ---- | --- | --- | --- |
94
+ | GPT-3.5 | Multi | 68.90 | 75.46 | 60.20 | 58.64 | 46.32 | 49.27 | 37.41
95
+ | Vistral-7B-chat | Mono | 56.86 | 67.00 | 44.56 | 54.33 | 50.03 | 36.49 | 25.27
96
+ | Qwen1.5-7B-chat | Multi | 61.00 | 52.07 | 81.96 | 43.38 | 45.02 | 24.29 | 20.25
97
+ | SailorLM | Multi | 52.72 | 59.76 | 67.74 | 50.14 | --- | 39.53 | 37.73
98
+ | SeaLLM-7B-v2 | Multi | 61.89 | 70.91 | 55.43 | 51.15 | 45.74 | 42.25 | 35.52
99
+ | SeaLLM-7B-v2.5 | Multi | 64.05 | 76.87 | 62.54 | 63.11 | 53.30 | 48.64 | 46.86
100
+
101
+
102
+ ### Zero-shot CoT Multilingual Math Reasoning
103
+
104
+ <!--
105
+ [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) achieves with **78.5** score on the GSM8K with zero-shot CoT reasoning, making it the **state of the art** in the realm of 7B models. It also outperforms GPT-3.5 in the same GSM8K benchmark as translated into SEA languages (🇨🇳 🇻🇳 🇮🇩 🇹🇭). [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) also surpasses GPT-3.5 on the Thai-translated MATH benchmark, with **28.4** vs 18.1 scores.
106
+
107
+ ![fig_sea_math_side_by_side.png](fig_sea_math_side_by_side.png)
108
+ -->
109
+
110
+ | Model | GSM8K<br>en | MATH<br>en | GSM8K<br>zh | MATH<br>zh | GSM8K<br>vi | MATH<br>vi | GSM8K<br>id | MATH<br>id | GSM8K<br>th | MATH<br>th
111
+ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
112
+ | GPT-3.5 | 80.8 | 34.1 | 48.2 | 21.5 | 55 | 26.5 | 64.3 | 26.4 | 35.8 | 18.1
113
+ | Qwen-14B-chat | 61.4 | 18.4 | 41.6 | 11.8 | 33.6 | 3.6 | 44.7 | 8.6 | 22 | 6.0
114
+ | Vistral-7b-chat | 48.2 | 12.5 | | | 48.7 | 3.1 | | | |
115
+ | Qwen1.5-7B-chat | 56.8 | 15.3 | 40.0 | 2.7 | 37.7 | 9 | 36.9 | 7.7 | 21.9 | 4.7
116
+ | SeaLLM-7B-v2 | 78.2 | 27.5 | 53.7 | 17.6 | 69.9 | 23.8 | 71.5 | 24.4 | 59.6 | 22.4
117
+ | SeaLLM-7B-v2.5 | 78.5 | 34.9 | 51.3 | 22.1 | 72.3 | 30.2 | 71.5 | 30.1 | 62.0 | 28.4
118
+
119
+
120
+ Baselines were evaluated using their respective chat-template and system prompts ([Qwen1.5-7B-chat](https://huggingface.co/Qwen/Qwen1.5-7B-Chat/blob/main/tokenizer_config.json), [Vistral](https://huggingface.co/Viet-Mistral/Vistral-7B-Chat)).
121
+
122
+ #### Zero-shot MGSM
123
+
124
+ [SeaLLM-7B-v2.5](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2.5) also outperforms GPT-3.5 and Qwen-14B on the multilingual MGSM for Thai.
125
+
126
+ | Model | MGSM-Zh | MGSM-Th
127
+ |-----| ----- | ---
128
+ | ChatGPT (reported) | 61.2 | 47.2
129
+ | Qwen-14B-chat | 59.6 | 28
130
+ | SeaLLM-7B-v2 | **64.8** | 62.4
131
+ | SeaLLM-7B-v2.5 | 58.0 | **64.8**
132
+
133
+
134
+ ### Sea-Bench
135
+
136
+ ![fig_sea_bench_side_by_side.png](fig_sea_bench_side_by_side.png)
137
+
138
+
139
+ ### Usage
140
+
141
+ **IMPORTANT NOTICE for using the model**
142
+
143
+ * `<bos>` must be at start of prompt, ff your code's tokenizer does not prepend `<bos>` by default, you MUST prepend <bos> into the prompt yourself, otherwise, it would not work!
144
+ * Repitition penalty (e.g: in llama.cpp, ollama, LM-studio) must be set to **1** , otherwise will lead to degeneration!
145
+
146
+ #### Instruction format
147
+
148
+ ```python
149
+ # ! WARNING, if your code's tokenizer does not prepend <bos> by default,
150
+ # You MUST prepend <bos> into the prompt yourself, otherwise, it would not work!
151
+
152
+ prompt = """<|im_start|>system
153
+ You are a helpful assistant.<eos>
154
+ <|im_start|>user
155
+ Hello world<eos>
156
+ <|im_start|>assistant
157
+ Hi there, how can I help?<eos>"""
158
+
159
+ # <|im_start|> is not a special token.
160
+ # Transformers chat_template should be consistent with vLLM format below.
161
+
162
+ # ! ENSURE 1 and only 1 bos `<bos>` at the beginning of sequence
163
+ print(tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt)))
164
+
165
+ """
166
+ ```
167
+
168
+ #### Using transformers's chat_template
169
+
170
+ Install the latest transformers (>4.40)
171
+
172
+ ```python
173
+
174
+ from transformers import AutoModelForCausalLM, AutoTokenizer
175
+
176
+ device = "cuda" # the device to load the model onto
177
+
178
+ # use bfloat16 to ensure the best performance.
179
+ model = AutoModelForCausalLM.from_pretrained("SeaLLMs/SeaLLM-7B-v2.5", torch_dtype=torch.bfloat16, device_map=device)
180
+ tokenizer = AutoTokenizer.from_pretrained("SeaLLMs/SeaLLM-7B-v2.5")
181
+
182
+ messages = [
183
+ {"role": "system", "content": "You are a helpful assistant."},
184
+ {"role": "user", "content": "Hello world"},
185
+ {"role": "assistant", "content": "Hi there, how can I help you today?"},
186
+ {"role": "user", "content": "Explain general relativity in details."}
187
+ ]
188
+
189
+ encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
190
+ print(tokenizer.convert_ids_to_tokens(encodeds[0]))
191
+
192
+ model_inputs = encodeds.to(device)
193
+ model.to(device)
194
+
195
+ generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True, pad_token_id=tokenizer.pad_token_id)
196
+ decoded = tokenizer.batch_decode(generated_ids)
197
+ print(decoded[0])
198
+
199
+ ```
200
+
201
+ #### Using vLLM
202
+
203
+ ```python
204
+ from vllm import LLM, SamplingParams
205
+ TURN_TEMPLATE = "<|im_start|>{role}\n{content}<eos>\n"
206
+ TURN_PREFIX = "<|im_start|>{role}\n"
207
+
208
+ def seallm_chat_convo_format(conversations, add_assistant_prefix: bool, system_prompt=None):
209
+ # conversations: list of dict with key `role` and `content` (openai format)
210
+ if conversations[0]['role'] != 'system' and system_prompt is not None:
211
+ conversations = [{"role": "system", "content": system_prompt}] + conversations
212
+ text = ''
213
+ for turn_id, turn in enumerate(conversations):
214
+ prompt = TURN_TEMPLATE.format(role=turn['role'], content=turn['content'])
215
+ text += prompt
216
+ if add_assistant_prefix:
217
+ prompt = TURN_PREFIX.format(role='assistant')
218
+ text += prompt
219
+ return text
220
+
221
+ sparams = SamplingParams(temperature=0.1, max_tokens=1024, stop=['<eos>', '<|im_start|>'])
222
+ llm = LLM("SeaLLMs/SeaLLM-7B-v2.5", dtype="bfloat16")
223
+
224
+ message = "Explain general relativity in details."
225
+ prompt = seallm_chat_convo_format(message, True)
226
+ gen = llm.generate(prompt, sampling_params)
227
+
228
+ print(gen[0].outputs[0].text)
229
+ ```
230
+
231
+ #### Fine-tuning SeaLLM-7B-v2.5
232
+
233
+ Should follow the chat format and accurately mask out source tokens. Here is an example.
234
+
235
+ ```python
236
+ conversations = [
237
+ {"role": "system", "content": "You are helful assistant."},
238
+ {"role": "user", "content": "Hello world."},
239
+ {"role": "assistant", "content": "Hi there, how can I help?"},
240
+ {"role": "user", "content": "Tell me a joke."},
241
+ {"role": "assistant", "content": "Why don't scientists trust atoms? Because they make up everything."},
242
+ ]
243
+ def seallm_7b_v25_tokenize_multi_turns(tokenizer, conversations, add_assistant_prefix=False):
244
+ """
245
+ Inputs:
246
+ conversations: list of dict following openai format, eg
247
+ conversations = [
248
+ {"role": "system", "content": "You are helful assistant."},
249
+ {"role": "user", "content": "Hello world."},
250
+ {"role": "assistant", "content": "Hi there, how can I help?"},
251
+ {"role": "user", "content": "Tell me a joke."},
252
+ {"role": "assistant", "content": "Why don't scientists trust atoms? Because they make up everything."},
253
+ ]
254
+ add_assistant_prefix: whether to add assistant_prefix, only for inference decoding
255
+ Outputs:
256
+ tokenize_output_sample, {
257
+ "input_ids": ...
258
+ "token_type_ids": 1 if train and 0 if masked out (not train)
259
+ }
260
+ During training, need to create a labels, with masked-out tokens = -100 to avoid loss computations.
261
+ labels = sample['input_ids'].clone()
262
+ labels[sample['token_type_ids'] == 0] = -100
263
+ """
264
+ TURN_TEMPLATE = "<|im_start|>{role}\n{content}<eos>\n"
265
+ TURN_PREFIX = "<|im_start|>{role}\n"
266
+ TURN_SUFFIX = "<eos>\n"
267
+ TURN_SUFFIX_TAKE = "<eos>"
268
+ sample = None
269
+ assistant_prefix_len = None
270
+ assistant_suffix_len = None
271
+ for turn_id, turn in enumerate(conversations):
272
+ prompt = TURN_TEMPLATE.format(role=turn['role'], content=turn['content'])
273
+ turn_sample = tokenizer(
274
+ prompt, padding=False, truncation=False, verbose=False, add_special_tokens=False,
275
+ return_token_type_ids=True,
276
+ )
277
+ if turn['role'] == 'assistant':
278
+ if assistant_prefix_len is None:
279
+ assistant_prefix_len = len(tokenizer.encode(TURN_PREFIX.format(role=turn['role']), add_special_tokens=False))
280
+ if assistant_suffix_len is None:
281
+ assistant_suffix_len = (
282
+ len(tokenizer.encode(TURN_SUFFIX.format(role=turn['role']), add_special_tokens=False)) -
283
+ len(tokenizer.encode(TURN_SUFFIX_TAKE, add_special_tokens=False))
284
+ )
285
+ turn_sample['token_type_ids'][assistant_prefix_len:-assistant_suffix_len] = [1] * (len(turn_sample['input_ids']) - assistant_prefix_len - assistant_suffix_len)
286
+ if sample is None:
287
+ sample = turn_sample
288
+ else:
289
+ for k in turn_sample.keys():
290
+ sample[k].extend(turn_sample[k])
291
+ if add_assistant_prefix:
292
+ assistant_prefix_sample = tokenizer(
293
+ TURN_PREFIX.format(role="assistant"), padding=False, truncation=False, verbose=False, add_special_tokens=False,
294
+ return_token_type_ids=True,
295
+ )
296
+ for k in sample.keys():
297
+ sample[k].extend(assistant_prefix_sample[k])
298
+ if tokenizer.add_bos_token:
299
+ sample['input_ids'] = [tokenizer.bos_token_id] + sample['input_ids']
300
+ sample['attention_mask'] = [1] + sample['attention_mask']
301
+ sample['token_type_ids'] = [sample['token_type_ids'][0]] + sample['token_type_ids']
302
+ return sample
303
+
304
+ # ! testing
305
+ sample = seallm_7b_v25_tokenize_multi_turns(tokenizer, conversations)
306
+ tokens = tokenizer.convert_ids_to_tokens(sample['input_ids'])
307
+ pairs = [(x, y) for x, y in zip(tokens, sample['token_type_ids'])]
308
+ print(pairs)
309
+
310
+ # source and special tokens is masked out (token_type 0), only assistant with <eos> is trained (token_type 1)
311
+ # [('<bos>', 0), ('<', 0), ('|', 0), ..., ('assistant', 0), ('\n', 0), ('Hi', 1), ('▁there', 1), (',', 1), ('▁how', 1), ('▁can', 1), ('▁I', 1), ('▁help', 1), ('?', 1), ('<eos>', 1), ('\n', 0), ('<', 0), ...
312
+
313
+ ```
314
+
315
+
316
+ ## Acknowledgement to Our Linguists
317
+
318
+ We would like to express our special thanks to our professional and native linguists, Tantong Champaiboon, Nguyen Ngoc Yen Nhi and Tara Devina Putri, who helped build, evaluate, and fact-check our sampled pretraining and SFT dataset as well as evaluating our models across different aspects, especially safety.
319
+
320
+ ## Citation
321
+
322
+ If you find our project useful, we hope you would kindly star our repo and cite our work as follows: Corresponding Author: [[email protected]](mailto:[email protected])
323
+
324
+ **Author list and order will change!**
325
+
326
+ * `*` and `^` are equal contributions.
327
+
328
+ ```
329
+ @article{damonlpsg2023seallm,
330
+ author = {Xuan-Phi Nguyen*, Wenxuan Zhang*, Xin Li*, Mahani Aljunied*, Weiwen Xu, Hou Pong Chan,
331
+ Zhiqiang Hu, Chenhui Shen^, Yew Ken Chia^, Xingxuan Li, Jianyu Wang,
332
+ Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang,
333
+ Chaoqun Liu, Hang Zhang, Lidong Bing},
334
+ title = {SeaLLMs - Large Language Models for Southeast Asia},
335
+ year = 2023,
336
+ Eprint = {arXiv:2312.00738},
337
+ }
338
+ ```
339
+