Hugging Face's logo

nectec
/
Pathumma-llm-text-1.0.0

Text Generation
PyTorch
Safetensors
GGUF
Thai
Chinese
English
qwen2
chemistry
biology
finance
legal
code
medical
text-generation-inference
conversational
Inference Endpoints
Model card Files Community
19
SuperkingbasSKB commited on
Commit
786be0c
•
1 Parent(s): 34faf87

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +11 -10
README.md CHANGED
@@ -36,7 +36,7 @@ KeyError: 'qwen2'
36
  ```
37
  ## **Support Community**
38
 
39
- **https://discord.gg/JsezgAf7**
40
 
41
  ## **Implementation**
42
 
@@ -55,7 +55,7 @@ tokenizer = AutoTokenizer.from_pretrained("nectec/OpenThaiLLM-DoodNiLT-V1.0.0-Be
55
 
56
  prompt = "บริษัท A มีต้นทุนคงที่ 100,000 บาท และต้นทุนผันแปรต่อหน่วย 50 บาท ขายสินค้าได้ในราคา 150 บาทต่อหน่วย ต้องขายสินค้าอย่างน้อยกี่หน่วยเพื่อให้ถึงจุดคุ้มทุน?"
57
  messages = [
58
- {"role": "system", "content": "คุณคือ DoodNiLT Assistant จงตอบคำถามอธิบายเป็นภาษาไทย"},
59
  {"role": "user", "content": prompt}
60
  ]
61
  text = tokenizer.apply_chat_template(
@@ -68,20 +68,21 @@ model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
68
  generated_ids = model.generate(
69
  model_inputs.input_ids,
70
  max_new_tokens=4096,
71
- repetition_penalty=1.2
 
72
  )
73
  response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
74
  print(response)
75
  ```
76
 
77
  ## **Evaluation Performance**
78
- | Model | ONET | IC | TGAT | TPAT-1 | A-Level | Average (ThaiExam) | MMLU | M3Exam | M6Exam |
79
- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
80
- | OpenthaiLLM-Prebuilt-7B | **0.5493** | 0.6315 | **0.6307** | **0.4655** | **0.37** | **0.5294** | **0.7054** | **0.5705** | **0.596** |
81
- | SeaLLM-v3-7B | 0.4753 | **0.6421** | 0.6153 | 0.3275 | 0.3464 | 0.4813 | 0.7037 | 0.4907 | 0.4625 | 0.3666 |
82
- | llama-3-typhoon-v1.5-8B | 0.3765 | 0.3473 | 0.5538 | 0.4137 | 0.2913 | 0.3965 | 0.6451 | 0.4312 | 0.4125 |
83
- | Qwen-2-7B | 0.4814 | 0.621 | 0.6153 | 0.3448 | 0.3385 | 0.4802 | 0.7073 | 0.4949 | 0.4807 |
84
- | Meta-Llama-3.1-8B | 0.3641 | 0.2631 | 0.2769 | 0.3793 | 0.1811 | 0.2929 | 0.6591 | 0.4239 | 0.3583 |
85
 
86
 
87
  ## **Citation**
 
36
  ```
37
  ## **Support Community**
38
 
39
+ **https://discord.gg/3WJwJjZt7r**
40
 
41
  ## **Implementation**
42
 
 
55
 
56
  prompt = "บริษัท A มีต้นทุนคงที่ 100,000 บาท และต้นทุนผันแปรต่อหน่วย 50 บาท ขายสินค้าได้ในราคา 150 บาทต่อหน่วย ต้องขายสินค้าอย่างน้อยกี่หน่วยเพื่อให้ถึงจุดคุ้มทุน?"
57
  messages = [
58
+ {"role": "system", "content": "You are Pathumma LLM, created by NECTEC. Your are a helpful assistant."},
59
  {"role": "user", "content": prompt}
60
  ]
61
  text = tokenizer.apply_chat_template(
 
68
  generated_ids = model.generate(
69
  model_inputs.input_ids,
70
  max_new_tokens=4096,
71
+ repetition_penalty=1.1,
72
+ temperature = 0.4
73
  )
74
  response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
75
  print(response)
76
  ```
77
 
78
  ## **Evaluation Performance**
79
+ | Model | m3exam | thaiexam | xcopa | belebele | xnli | thaisentiment | XL sum | flores200 eng > th | flores200 th > eng | iapp | AVG(NLU) | AVG(MC) | AVG(NLG)
80
+ | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
81
+ | Pathumma-llm-text-1.0.0 | 55.02 | 51.32 | 83 | 77.77 | 40.11 | 41.29 | 16.9286253 | 26.54 | 51.88 | 41.28 |
82
+ | Openthaigpt1.5-7b-instruct | 54.01 | 52.04 | 85.4 | 79.44 | 39.7 | 50.24 | 18.11 | 29.09 | 29.58 | 32.49 | 63.70 | 53.03 | 27.32 |
83
+ | SeaLLMs-v3-7B-Chat | 51.43 | 51.33 | 83.4 | 78.22 | 34.05 | 39.57 | 20.27 | 32.91 | 28.8 | 48.12 | 58.81 | 51.38 | 32.53 |
84
+ | llama-3-typhoon-v1.5-8B | 43.82 | 41.95 | 81.6 | 71.89 | 33.35 | 38.45 | 16.66 | 31.94 | 28.86 | 54.78 | 56.32 | 42.89 | 33.06 |
85
+ | Meta-Llama-3.1-8B-Instruct | 45.11 | 43.89 | 73.4 | 74.89 | 33.49 | 45.45 | 21.61 | 30.45 | 32.28 | 68.57 | 56.81 | 44.50 | 38.23 |
86
 
87
 
88
  ## **Citation**