Update README.md

README.md

@@ -1,12 +1,14 @@
 ---
 license: cc-by-nc-sa-4.0
 ---
-# SEA-LION-Instruct-NC
+# SEA-LION-7B-Instruct-NC
 
 SEA-LION is a collection of Large Language Models (LLMs) which has been pretrained and instruct-tuned for the Southeast Asia (SEA) region.
 The size of the models range from 3 billion to 7 billion parameters.
 This is the card for the SEA-LION 7B Instruct (Non-Commercial) model.
 
+For more details on the base model, please refer to the [base model's model card](https://huggingface.co/aisingapore/sealion7b).
+
 SEA-LION stands for <i>Southeast Asian Languages In One Network</i>.
 
 
@@ -21,64 +23,39 @@ SEA-LION is built on the robust MPT architecture and has a vocabulary size of 25
 
 For tokenization, the model employs our custom SEABPETokenizer, which is specially tailored for SEA languages, ensuring optimal model performance.
 
-The training data for SEA-LION encompasses 980B tokens.
+The pre-training data for the base SEA-LION model encompasses 980B tokens.
+The model was then further instruction-tuned on <b>Indonesian data only</b>.
 
 - **Developed by:** Products Pillar, AI Singapore
 - **Funded by:** Singapore NRF
 - **Model type:** Decoder
 - **Languages:** English, Chinese, Indonesian, Malay, Thai, Vietnamese, Filipino, Tamil, Burmese, Khmer, Lao
-- **License:** MIT License
-
-
-## Training Details
+- **License:** CC BY-NC-SA 4.0 License
 
-### Data
+### Benchmark Performance
 
-SEA-LION was trained on 980B tokens of the following data:
+SEA-LION-7B-Instruct-NC performs better than other models of comparable size when tested on tasks in the Indonesian language.
 
-| Data Source               | Tokens | Percentage |
-|---------------------------|-------:|:----------:|
-| RefinedWeb - English      | 571.3B |     58.20% |
-| mC4 - Chinese             |  91.2B |      9.29% |
-| mC4 - Indonesian          |  14.7B |      1.50% |
-| mC4 - Malay               |   2.9B |      0.29% |
-| mC4 - Filipino            |   5.3B |      0.54% |
-| mC4 - Burmese             |   4.9B |      0.49% |
-| mC4 - Vietnamese          |  63.4B |      6.46% |
-| mC4 - Thai                |  21.6B |      2.20% |
-| mC4 - Lao                 |   1.1B |      0.12% |
-| mC4 - Khmer               |   3.9B |      0.40% |
-| mC4 - Tamil               |  10.2B |      1.04% |
-| the Stack - Python        |  41.8B |      4.26% |
-| the Stack - Javascript    |  55.6B |      5.66% |
-| the Stack - Shell         |   2.5B |      0.26% |
-| the Stack - SQL           |  12.8B |      1.31% |
-| the Stack - Markdown      |  26.6B |      2.71% |
-| RedPajama - StackExchange |  21.2B |      2.16% |
-| RedPajama - ArXiv         |  30.6B |      3.12% |
+We evaluated SEA-LION-7B-Instruct-NC on the [BHASA benchmark](https://arxiv.org/abs/2309.06085) and 
+compared it against [Llama-2-7B](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)
+and [Falcon-7B](https://huggingface.co/tiiuae/falcon-7b-instruct).
 
-### Infrastructure
+We only evaluated it on the Indonesian tasks as the model was only instruction-tuned in Indonesian.
 
-SEA-LION was trained using [MosaicML Composer](https://github.com/mosaicml/composer)
-on the following hardware:
+The scores shown in the tables below have been adjusted to only consider answers provided in the appropriate language.
 
-| Training Details     | SEA-LION 7B  |
-|----------------------|:------------:|
-| AWS EC2 p4d.24xlarge | 32 instances |
-| Nvidia A100 40GB GPU | 256          |
-| Training Duration    | 22 days      |
+For Natural Language Understanding (NLU) tasks, we tested the model on Sentiment Analysis (Sent) using the NusaX dataset, Question Answering (QA) using the TyDiQA dataset, and Toxicity Detection (Tox) using the Indonesian Multi-Label Hate Speech Detection dataset. The metrics used are F1 score for all three tasks.
 
+For Natural Language Generation (NLG) tasks, we tested the model on Machine Translation from English to Indonesian (MT-EN-ID) and from Indonesian to English (MT-ID-EN) using the FLORES-200 dataset, and Abstractive Summarization (AbsSum) using the XLSum dataset. The metrics used for Machine Translation are ChrF++ and COMET22, and ROUGE-L is used for Abstractive Summarization.
 
-### Configuration
+For Natural Language Reasoning (NLR) tasks, we tested the model on Natural Language Inference (NLI) using the IndoNLI lay dataset and on Causal Reasoning (Causal) using the XCOPA dataset. The metrics are accuracy for both tasks.
 
-| HyperParameter    | SEA-LION 7B        |
-|-------------------|:------------------:|
-| Precision         | bfloat16           |
-| Optimizer         | decoupled_adamw    |
-| Scheduler         | cosine_with_warmup |
-| Learning Rate     | 6.0e-5             |
-| Global Batch Size | 2048               |
-| Micro Batch Size  | 4                  |
+| Model Name               | Sent (F1) | QA (F1)  | Tox (F1) | MT-EN-ID (ChrF++)| (COMET22)| MT-ID-EN (ChrF++)| (COMET22)| AbsSum (ROUGE-L)| NLI (Acc) | Causal (Acc) |
+|--------------------------|-----------|----------|----------|------------------|----------|------------------|----------|-----------------|-----------|-------------|
+| sealion7b-instruct-nc    | **76.13** | 24.86    | **24.45**| **52.50**        | **86.97**| 46.82            | 81.34    | **15.44**       | **33.20** | **23.80**   |
+| Mistral-7B-Instruct-v0.1 | 73.66     | **26.08**| 18.60    | 31.08            | 55.29    | 51.20            | 82.38    | 14.41           | 29.20     | 11.00       |
+| Llama-2-7b-chat-hf       | 41.92     | 4.23     | 0.00     | 47.96            | 77.86    | **55.76**        | **86.08**| 4.59            | 0.00      | 0.00        |
+| falcon-7b-instruct       | 0.00      | 8.47     | 7.21     | 1.66             | 30.07    | 16.82            | 46.32    | 1.55            | 0.00      | 2.20        |
 
 
 ## Technical Specifications
@@ -102,6 +79,27 @@ We sample 20M lines from the training data to train the tokenizer.<br>
 The framework for training is [SentencePiece](https://github.com/google/sentencepiece).<br>
 The tokenizer type is Byte-Pair Encoding (BPE).
 
+### Example Usage
+
+```python
+# Please use transformers==4.34.1
+
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("aisingapore/sealion7b-instruct-nc", trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained("aisingapore/sealion7b-instruct-nc", trust_remote_code=True)
+
+prompt_template = "### USER:\n{human_prompt}\n\n### RESPONSE:\n"
+prompt = """Apa sentimen dari kalimat berikut ini?
+Kalimat: Buku ini sangat membosankan.
+Jawaban: """
+full_prompt = prompt_template.format(human_prompt=prompt)
+
+tokens = tokenizer(full_prompt, return_tensors="pt")
+output = model.generate(tokens["input_ids"], max_new_tokens=20, eos_token_id=tokenizer.eos_token_id)
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+
+```
 
 
 ## The Team
@@ -139,7 +137,7 @@ For more info, please contact us at [email protected]
 
 ## Disclaimer
 
-This the repository for the base model.
+This the repository for the non-commercial instruction-tuned model.
 The model has _not_ been aligned for safety.
 Developers and users should perform their own safety fine-tuning and related security measures.
 In no event shall the authors be held liable for any claim, damages, or other liability