|
--- |
|
license: mit |
|
--- |
|
# SEA-LION |
|
|
|
SEA-LION is a collection of Large Language Models (LLMs) which has been pretrained and instruct-tuned for the Southeast Asia (SEA) region. |
|
The size of the models range from 3 billion to 7 billion parameters. |
|
This is the card for the SEA-LION 3B base model. |
|
|
|
SEA-LION stands for <i>Southeast Asian Languages In One Network</i>. |
|
|
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
The SEA-LION model is a significant leap forward in the field of Natural Language Processing, |
|
specifically trained to understand the SEA regional context. |
|
|
|
SEA-LION is built on the robust MPT architecture and has a vocabulary size of 256K. |
|
|
|
For tokenization, the model employs our custom SEABPETokenizer, which is specially tailored for SEA languages, ensuring optimal model performance. |
|
|
|
The training data for SEA-LION encompasses 980B tokens. |
|
|
|
- **Developed by:** Products Pillar, AI Singapore |
|
- **Funded by:** Singapore NRF |
|
- **Model type:** Decoder |
|
- **Languages:** English, Chinese, Indonesian, Malay, Thai, Vietnamese, Filipino, Tamil, Burmese, Khmer, Lao |
|
- **License:** MIT License |
|
|
|
|
|
## Training Details |
|
|
|
### Data |
|
|
|
SEA-LION was trained on 980B tokens of the following data: |
|
|
|
| Data Source | Tokens | Percentage | |
|
|---------------------------|-------:|:----------:| |
|
| RefinedWeb - English | 571.3B | 58.20% | |
|
| mC4 - Chinese | 91.2B | 9.29% | |
|
| mC4 - Indonesian | 14.7B | 1.50% | |
|
| mC4 - Malay | 2.9B | 0.29% | |
|
| mC4 - Filipino | 5.3B | 0.54% | |
|
| mC4 - Burmese | 4.9B | 0.49% | |
|
| mC4 - Vietnamese | 63.4B | 6.46% | |
|
| mC4 - Thai | 21.6B | 2.20% | |
|
| mC4 - Lao | 1.1B | 0.12% | |
|
| mC4 - Khmer | 3.9B | 0.40% | |
|
| mC4 - Tamil | 10.2B | 1.04% | |
|
| the Stack - Python | 41.8B | 4.26% | |
|
| the Stack - Javascript | 55.6B | 5.66% | |
|
| the Stack - Shell | 2.5B | 0.26% | |
|
| the Stack - SQL | 12.8B | 1.31% | |
|
| the Stack - Markdown | 26.6B | 2.71% | |
|
| RedPajama - StackExchange | 21.2B | 2.16% | |
|
| RedPajama - ArXiv | 30.6B | 3.12% | |
|
|
|
### Infrastructure |
|
|
|
SEA-LION was trained using [MosaicML Composer](https://github.com/mosaicml/composer) |
|
on the following hardware: |
|
|
|
| Training Details | SEA-LION 3B | |
|
|----------------------|:------------:| |
|
| AWS EC2 p4d.24xlarge | 30 instances | |
|
| Nvidia A100 40GB GPU | 240 | |
|
| Training Duration | 14 days | |
|
|
|
|
|
### Configuration |
|
|
|
| HyperParameter | SEA-LION 3B | |
|
|-------------------|:------------------:| |
|
| Precision | bfloat16 | |
|
| Optimizer | decoupled_adamw | |
|
| Scheduler | cosine_with_warmup | |
|
| Learning Rate | 1.6e-4 | |
|
| Global Batch Size | 1200 | |
|
| Micro Batch Size | 5 | |
|
|
|
|
|
## Technical Specifications |
|
|
|
### Model Architecture and Objective |
|
|
|
SEA-LION is a decoder model using the MPT architecture. |
|
|
|
| Parameter | SEA-LION 3B | |
|
|-----------------|:-----------:| |
|
| Layers | 32 | |
|
| d_model | 2560 | |
|
| head_dim | 20 | |
|
| Vocabulary | 256000 | |
|
| Sequence Length | 2048 | |
|
|
|
|
|
### Tokenizer Details |
|
|
|
We sample 20M lines from the training data to train the tokenizer.<br> |
|
The framework for training is [SentencePiece](https://github.com/google/sentencepiece).<br> |
|
The tokenizer type is Byte-Pair Encoding (BPE). |
|
|
|
|
|
|
|
## The Team |
|
|
|
Lam Wen Zhi Clarence<br> |
|
Leong Wei Qi<br> |
|
Li Yier<br> |
|
Liu Darius<br> |
|
Lovenia Holy<br> |
|
Montalan Jann Railey<br> |
|
Ng Boon Cheong Raymond<br> |
|
Ngui Jian Gang<br> |
|
Nguyen Thanh Ngan<br> |
|
Ong Tat-Wee David<br> |
|
Rengarajan Hamsawardhini<br> |
|
Susanto Yosephine<br> |
|
Tai Ngee Chia<br> |
|
Tan Choon Meng<br> |
|
Teo Jin Howe<br> |
|
Teo Leslie<br> |
|
Teo Wei Yi<br> |
|
Tjhi William<br> |
|
Yeo Yeow Tong<br> |
|
Yong Xianbin<br> |
|
|
|
## Acknowledgements |
|
|
|
AI Singapore is a national programme supported by the National Research Foundation, Singapore and hosted by the National University of Singapore. |
|
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore. |
|
|
|
## Contact |
|
|
|
For more info, please contact us at [email protected] |
|
|
|
[Link to SEA-LION's GitHub repository](https://github.com/aisingapore/sealion) |
|
|
|
|
|
## Disclaimer |
|
|
|
This the repository for the base model. |
|
The model has _not_ been aligned for safety. |
|
Developers and users should perform their own safety fine-tuning and related security measures. |
|
In no event shall the authors be held liable for any claim, damages, or other liability |
|
arising from the use of the released weights and codes. |
