|
--- |
|
language: |
|
- en |
|
- ko |
|
license: cc-by-nc-4.0 |
|
datasets: |
|
- kyujinpy/KOR-gugugu-platypus-set |
|
base_model: |
|
- yanolja/KoSOLAR-10.7B-v0.2 |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
<div align="center"> |
|
<h1>🤗 KoSOLAR-v0.2-gugutypus-10.7B ☀️</h1> |
|
|
|
<a style="margin: 0px;" href="https://github.com/oneonlee/KoSOLAR-v0.2-gugutypus-10.7B"><img style="margin: 0.5em;" alt="GitHub" src="https://img.shields.io/badge/GitHub-181717.svg?style=flat&logo=GitHub"></a> |
|
<a style="margin: 0px;" href="https://huggingface.co/oneonlee/KoSOLAR-v0.2-gugutypus-10.7B"><img style="margin: 0.5em;" alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97-Models%20on%20Hub-yellow"></a> |
|
<a style="margin: 0px;" href="https://github.com/oneonlee/KoSOLAR-v0.2-gugutypus-10.7B/blob/main/LICENSE"><img style="margin: 0.5em;" alt="License: CC BY-NC 4.0" src="https://img.shields.io/badge/License-CC%20BY%2D%2DNC%204.0-blue.svg"></a> |
|
<a style="margin: 0px;" href="https://doi.org/10.57967/hf/1735"><img style="margin: 0.5em;" alt="DOI" src="https://img.shields.io/badge/DOI-10.57967%2Fhf%2F1735-blue"></a> |
|
|
|
<img src="logo.png" height=350, width=350> |
|
</div> |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Details |
|
|
|
**Model Developers** |
|
- DongGeon Lee ([oneonlee](https://huggingface.co/oneonlee)) |
|
|
|
**Model Architecture** |
|
- **KoSOLAR-v0.2-gugutypus-10.7B** is a instruction fine-tuned auto-regressive language model, based on the [SOLAR](https://huggingface.co/upstage/SOLAR-10.7B-v1.0) transformer architecture. |
|
|
|
**Base Model** |
|
- [yanolja/KoSOLAR-10.7B-v0.2](https://huggingface.co/yanolja/KoSOLAR-10.7B-v0.2) |
|
|
|
**Training Dataset** |
|
- [kyujinpy/KOR-gugugu-platypus-set](https://huggingface.co/datasets/kyujinpy/KOR-gugugu-platypus-set) |
|
|
|
|
|
--- |
|
|
|
|
|
## Model comparisons |
|
|
|
- **Ko-LLM leaderboard (2024/03/01)** [[link]](https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard) |
|
|
|
| Model | Average | Ko-ARC | Ko-HellaSwag | Ko-MMLU | Ko-TruthfulQA | Ko-CommonGen V2 | |
|
| ----------------------------------------- | ----------- | ------ | ------------ | ------- | ------------- | --------------- | |
|
| **oneonlee/KoSOLAR-v0.2-gugutypus-10.7B** | **51.17** | 47.78 | 58.29 | 47.27 | 48.31 | 54.19 | |
|
| [oneonlee/LDCC-SOLAR-gugutypus-10.7B](https://huggingface.co/oneonlee/LDCC-SOLAR-gugutypus-10.7B) | 49.45 | 45.9 | 55.46 | 47.96 | 48.93 | 49 | |
|
|
|
|
|
<br> |
|
|
|
- **(KOR) AI-Harness evaluation** [[link]](https://github.com/Beomi/ko-lm-evaluation-harness) |
|
|
|
|
|
| Tasks |Version|Filter|n-shot|Metric|Value | |Stderr| |
|
|-------------------------|-------|------|-----:|------|-----:|---|-----:| |
|
|KMMLU |N/A |none | 0|acc |0.3335|± |0.0475| |
|
|KMMLU |N/A |none | 5|acc |0.3938|± |0.0823| |
|
|KoBEST-HellaSwag | 0|none | 0|acc |0.4360|± |0.0222| |
|
|KoBEST-HellaSwag | 0|none | 5|acc |0.4420|± |0.0222| |
|
|KoBEST-BoolQ | 0|none | 0|acc |0.5064|± |0.0133| |
|
|KoBEST-BoolQ | 0|none | 5|acc |0.8583|± |0.0093| |
|
|KoBEST-COPA | 0|none | 0|acc |0.6040|± |0.0155| |
|
|KoBEST-COPA | 0|none | 5|acc |0.7610|± |0.0135| |
|
|KoBEST-SentiNeg | 0|none | 0|acc |0.5844|± |0.0248| |
|
|KoBEST-SentiNeg | 0|none | 5|acc |0.9471|± |0.0112| |
|
|
|
<br> |
|
|
|
- **(ENG) AI-Harness evaluation** [[link]](https://github.com/EleutherAI/lm-evaluation-harness) |
|
|
|
| Tasks |Version|Filter|n-shot|Metric|Value | |Stderr| |
|
|------------------|-------|------|-----:|------|-----:|---|-----:| |
|
|MMLU |N/A |none | 0|acc |0.5826|± |0.1432| |
|
|MMLU |N/A |none | 5|acc |0.5885|± |0.1285| |
|
|HellaSwag | 1|none | 0|acc |0.6075|± |0.0049| |
|
|HellaSwag | 1|none | 5|acc |0.6098|± |0.0049| |
|
|BoolQ | 2|none | 0|acc |0.8737|± |0.0058| |
|
|BoolQ | 2|none | 5|acc |0.8826|± |0.0056| |
|
|COPA | 1|none | 0|acc |0.8300|± |0.0378| |
|
|COPA | 1|none | 5|acc |0.9100|± |0.0288| |
|
|truthfulqa |N/A |none | 0|acc |0.4249|± |0.0023| |
|
|truthfulqa |N/A |none | 5|acc | - |± | - | |
|
|
|
|
|
--- |
|
|
|
## How to Use |
|
|
|
```python |
|
### KoSOLAR-gugutypus |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
import torch |
|
|
|
repo = "oneonlee/KoSOLAR-v0.2-gugutypus-10.7B" |
|
model = AutoModelForCausalLM.from_pretrained( |
|
repo, |
|
return_dict=True, |
|
torch_dtype=torch.float16, |
|
device_map='auto' |
|
) |
|
tokenizer = AutoTokenizer.from_pretrained(repo) |
|
``` |
|
|
|
--- |
|
|
|
## Citation |
|
``` |
|
@misc {donggeon_lee_2024, |
|
author = { {DongGeon Lee} }, |
|
title = { KoSOLAR-v0.2-gugutypus-10.7B (Revision 56841d5) }, |
|
year = 2024, |
|
url = { https://huggingface.co/oneonlee/KoSOLAR-v0.2-gugutypus-10.7B }, |
|
doi = { 10.57967/hf/1735 }, |
|
publisher = { Hugging Face } |
|
} |
|
``` |
|
|
|
--- |
|
|
|
## References |
|
- [yanolja/KoSOLAR-10.7B-v0.2](https://huggingface.co/yanolja/KoSOLAR-10.7B-v0.2) |
|
- [upstage/SOLAR-10.7B-v1.0](https://huggingface.co/upstage/SOLAR-10.7B-v1.0) |
|
- [kyujinpy/KOR-gugugu-platypus-set](https://huggingface.co/datasets/kyujinpy/KOR-gugugu-platypus-set) |
|
- [squarelike/OpenOrca-gugugo-ko](https://huggingface.co/datasets/squarelike/OpenOrca-gugugo-ko) |
|
- [kyujinpy/KOR-OpenOrca-Platypus-v3](https://huggingface.co/datasets/kyujinpy/KOR-OpenOrca-Platypus-v3) |
|
- [Open-Orca/OpenOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca) |
|
- [upstage/open-ko-llm-leaderboard](https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard) |
|
- [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) |
|
- [Beomi/ko-lm-evaluation-harness](https://github.com/Beomi/ko-lm-evaluation-harness) |
