File size: 9,086 Bytes
a7c30c4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 |
---
language:
- id
- en
- jv
- su
license: llama2
library_name: transformers
tags:
- komodo
---
[](https://hf.co/QuantFactory)
# QuantFactory/komodo-7b-base-GGUF
This is quantized version of [Yellow-AI-NLP/komodo-7b-base](https://huggingface.co/Yellow-AI-NLP/komodo-7b-base) created using llama.cpp
# Original Model Card
# Model Card for Komodo-7B-Base
Komodo-7B-Base is a large language model that is developed through incremental pretraining and vocabulary expansion on top of Llama-2-7B-Base. This model can handle Indonesian, English and 11 regional languages of Indonesia.
**Disclaimer** : This is not an instruction-tuned model, further fine-tuning is needed for downstream tasks. For example, people usually utilize the [Alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca) dataset for further fine-tuning on top of Llama-2-7B-Base model. Hence, there is no prompt template for this model.
## Model Details
<h3 align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/638828121901766b88076aa1/eB0L_nmy3ZpwtGA6-vhbC.png" width="950" align="center">
</h3>
### Model Description
More details can be found in our paper: https://arxiv.org/abs/2403.09362
- **Developed by:** [Yellow.ai](https://yellow.ai/)
- **Model type:** Decoder
- **Languages:** English, Indonesian, Acehnese, Balinese, Banjarese, Buginese, Madurese, Minangkabau, Javanese, Dayak Ngaju, Sundanese, Toba Batak, Lampungnese
- **License:** llama2
## Usage Example
Since this is a gated model, you need to logged in to your HF account before using the model. Below is one way to do this. You can get the HF Token from your profile (Profile -> Settings -> Access Tokens)
```
import huggingface_hub
huggingface_hub.login("YOUR_HF_TOKEN")
```
Once you are logged in, you can start download and load the model & tokenizer. We wrote a custom decoding function for Komodo-7B, that's why we need to pass the `trust_remote_code=True`. The code also works without this parameter, but decoding process will not work as expected.
```
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
device = "cuda:0" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained("Yellow-AI-NLP/komodo-7b-base",trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Yellow-AI-NLP/komodo-7b-base",trust_remote_code=True)
model = model.to(device)
```
Then, you can try using the model.
```
full_prompt = "Candi borobudur adalah"
tokens = tokenizer(full_prompt, return_tensors="pt").to(device)
output = model.generate(tokens["input_ids"], eos_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(output[0], skip_special_tokens=True))
# Candi borobudur adalah candi yang terletak di Magelang, Jawa Tengah.
```
## Technical Specifications
### Model Architecture and Objective
Komodo-7B is a decoder model using the Llama-2 architecture.
| Parameter | Komodo-7B |
|-----------------|:-----------:|
| Layers | 32 |
| d_model | 4096 |
| head_dim | 32 |
| Vocabulary | 35008 |
| Sequence Length | 4096 |
### Tokenizer Details
Recognizing the importance of linguistic diversity, we focused on enhancing our language model's proficiency in both Indonesian and regional languages. To achieve this, we systematically expanded the tokenizer's vocabulary by identifying and incorporating approximately 2,000 frequently used words specific to Indonesian and 1,000 words for regional languages that were absent in the Llama-2 model.
The standard method for enhancing a vocabulary typically involves developing a new tokenizer and integrating it with the existing one. This technique has shown impressive results in projects like Chinese-LLaMA and Open-Hathi. The effectiveness of this strategy can be attributed to the significant linguistic distinctions between languages such as Chinese and Hindi when compared to English. In contrast, the Indonesian language employs the same Latin script as English, which presents a different set of challenges.
We tested the traditional method, as well as a new approach where we included the top n words (not tokens) from the Indonesian vocabulary. We discovered that with the new approach, we could achieve better fertility scores by adding around 3000 new vocabulary words. Adding more than 3000 words did not significantly improve the fertility score further, but it increased the size of the embedding matrix, leading to longer training times.
More details can be found in our paper: https://arxiv.org/abs/2403.09362
### Training Data
More details can be found in our paper: https://arxiv.org/abs/2403.09362
### Training Procedure
More details can be found in our paper: https://arxiv.org/abs/2403.09362
#### Preprocessing
More details can be found in our paper: https://arxiv.org/abs/2403.09362
## Evaluation & Results
Please note that the benchmarking values below are based on our SFT Model, Komodo-7B-Instruct, while here we only release the base model, Komodo-7B-base.
| Organization | Model Name | Indo MMLU | ID-EN | XCOPA-ID | Intent Classification | Colloquial Detection | NusaX-Senti | ID-Hate Speech | TydiQA-ID | Indosum | Average |
|--------------|--------------------|-----------|-------|----------|-----------------------|----------------------|-------------|----------------|-----------|---------|---------|
| OpenAI | GPT-3.5-turbo-0301 | 51.3 | 64.5 | 70.0 | 82.0 | 64.1 | 47.2 | 68.0 | 85.3 | 41.0 | 63.7 |
| OpenAI | GPT-3.5-turbo-0613 | 52.7 | 66.8 | 88.2 | 84.0 | 75.1 | 63.3 | 63.7 | 86.4 | 40.0 | 68.9 |
| OpenAI | GPT-3.5-turbo-1106 | 53.3 | 69.7 | 89.3 | 84.0 | 64.2 | 59.8 | 56.6 | 88.0 | 42.0 | 67.4 |
| OpenAI | GPT-4-preview-1106 | 69.8 | 78.0 | 98.3 | 89.0 | 92.7 | 66.1 | 73.4 | 72.0 | 33.0 | 74.7 |
| Meta | Llama-2-7B-Chat | 30.4 | 45.6 | 41.5 | 57.0 | 31.4 | 2.9 | 41.3 | 11.7 | 34.0 | 32.9 |
| Meta | Llama-2-13B-Chat | 32.0 | 61.7 | 38.0 | 59.0 | 31.1 | 58.7 | 57.2 | 71.9 | 40.0 | 50.0 |
| Google | Gemma-7B-it | 37.4 | 73.6 | 57.7 | 77.1 | 18.8 | 44.2 | 54.8 | 73.3 | 44.0 | 53.4 |
| Mistral | Mixtral-8x7B-v0.1-Instruct | 45.2 | 57.8 | 88.7 | 86.0 | 41.1 | 52.8 | 68.8 | 90.3 | 14.0 | 60.5 |
| AISingapore | Sealion-7B-Instruct-NC | 23.9 | 26.9 | 41.3 | 37.0 | 41.8 | 30.7 | 57.3 | 65.3 | 26.0 | 38.9 |
| Cohere | Aya-101-13B | 47.7 | 47.3 | 84.0 | 64.0 | 18.9 | 74.6 | 72.7 | 81.3 | 39.0 | 58.8 |
| MBZUAI | Bactrian-X-Llama-7B | 23.6 | 43.2 | 45.3 | 42.0 | 50.3 | 44.5 | 42.4 | 65.0 | 15.0 | 41.3 |
| Alibaba | Qwen-1.5-7B-chat | 40.0 | 56.0 | 29.5 | 85.0 | 41.8 | 58.7 | 63.9 | 51.22 | 29.0 | 50.6 |
| Yellow.ai | Komodo-7B-Instruct | 43.2 | 90.5 | 79.6 | 84.0 | 73.6 | 79.3 | 56.2 | 90.3 | 43.0 | 71.1 |
<h3 align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/638828121901766b88076aa1/CJkSjsVnC8MoMolIQ_Uv3.png" width="550" align="center">
</h3>
More details can be found in our paper: https://arxiv.org/abs/2403.09362
### Infrastructure
| Training Details | Komodo-7B |
|----------------------|:------------:|
| AWS EC2 p4d.24xlarge | 1 instances |
| Nvidia A100 40GB GPU | 8 |
| Training Duration | 300 hours |
## Citation
```
@misc{owen2024komodo,
title={Komodo: A Linguistic Expedition into Indonesia's Regional Languages},
author={Louis Owen and Vishesh Tripathi and Abhay Kumar and Biddwan Ahmed},
year={2024},
eprint={2403.09362},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```
## Model Card Authors
[Louis Owen](https://www.linkedin.com/in/louisowen/) <br>
[Vishesh Tripathi](https://www.linkedin.com/in/vishesh-tripathi/) <br>
[Abhay Kumar](https://www.linkedin.com/in/akanyaani/) <br>
[Biddwan Ahmed](https://www.linkedin.com/in/biddwan-ahmed-917333126/) <br>
|