Synatra-7B-v0.3-dpo๐ง
Support Me
์๋ํธ๋ผ๋ ๊ฐ์ธ ํ๋ก์ ํธ๋ก, 1์ธ์ ์์์ผ๋ก ๊ฐ๋ฐ๋๊ณ ์์ต๋๋ค. ๋ชจ๋ธ์ด ๋ง์์ ๋์ จ๋ค๋ฉด ์ฝ๊ฐ์ ์ฐ๊ตฌ๋น ์ง์์ ์ด๋จ๊น์?
Wanna be a sponser? (Please) Contact me on Telegram AlzarTakkarsen
Model Details
Base Model
mistralai/Mistral-7B-Instruct-v0.1
Trained On
A100 80GB * 1
Instruction format
It follows ChatML format and Alpaca(No-Input) format.
Model Benchmark
KOBEST_BOOLQ, SENTINEG, WIC - ZERO_SHOT
EleutherAI/lm-evaluation-harness๋ฅผ ์ฌ์ฉํ์ฌ BoolQ, SentiNeg, Wic์ ์ธก์ ํ์ต๋๋ค.
Model | COPA | HellaSwag | BoolQ | SentiNeg |
---|---|---|---|---|
EleutherAI/polyglot-ko-12.8b | 0.7937 | 0.5954 | 0.4818 | 0.9117 |
Synatra-7B-v0.3-base | 0.6344 | 0.5140 | 0.5226 | NaN |
Synatra-7B-v0.3-dpo | 0.6380 | 0.4780 | 0.8058 | 0.8942 |
Ko-LLM-Leaderboard
On Benchmarking...
Implementation Code
Since, chat_template already contains insturction format above. You can use the code below.
from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" # the device to load the model onto
model = AutoModelForCausalLM.from_pretrained("maywell/Synatra-7B-v0.3-dpo")
tokenizer = AutoTokenizer.from_pretrained("maywell/Synatra-7B-v0.3-dpo")
messages = [
{"role": "user", "content": "๋ฐ๋๋๋ ์๋ ํ์์์ด์ผ?"},
]
encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")
model_inputs = encodeds.to(device)
model.to(device)
generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
Metric | Value |
---|---|
Avg. | 53.14 |
ARC (25-shot) | 62.8 |
HellaSwag (10-shot) | 82.58 |
MMLU (5-shot) | 61.46 |
TruthfulQA (0-shot) | 56.46 |
Winogrande (5-shot) | 76.24 |
GSM8K (5-shot) | 23.73 |
DROP (3-shot) | 8.68 |
- Downloads last month
- 6,169
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.