README.md

---
library_name: transformers
license: apache-2.0
datasets:
- squarelike/OpenOrca-gugugo-ko
language:
- ko
- en
pipeline_tag: text-generation
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->
LLaMA2 7b 모델의 한국어 CP(Continual Pre-trained)모델인 https://huggingface.co/beomi/llama-2-ko-7b 모델의 instruction tuning 모델

transformers와 trl을 이용하여 QLoRA로 훈련 진행

Dataset은 https://huggingface.co/datasets/squarelike/OpenOrca-gugugo-ko

QLoRA 훈련 테스트를 위한 훈련 결과물


## Model Details

### Model Sources [optional]


## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

### Prompt template

```
"""
### instruction:
### intput:
### output:
"""
```


## Training Details

#### Training Hyperparameters

- **Training regime:** [bf16 mixed precision] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

- <PAD>토큰 추가 후 right 패딩사이드 지정하여 진행


- LoRA config
    ```
peft_config = LoraConfig(
    lora_alpha=16,
    lora_dropout=0.1,
    r=64,
    bias="none",
    task_type="CAUSAL_LM"
)
    ```

    

## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

link: https://github.com/Beomi/ko-lm-evaluation-harness

results/all/aeolian83/llama_ko_sft_gugugo_experi_01
|                                  |        0 |        5 |
|:---------------------------------|---------:|---------:|
| kobest_boolq (macro_f1)          | 0.588382 | 0.384051 |
| kobest_copa (macro_f1)           | 0.749558 | 0.778787 |
| kobest_hellaswag (macro_f1)      | 0.439247 | 0.439444 |
| kobest_sentineg (macro_f1)       | 0.448283 | 0.934415 |
| kohatespeech (macro_f1)          | 0.244828 | 0.371245 |
| kohatespeech_apeach (macro_f1)   | 0.337434 | 0.394607 |
| kohatespeech_gen_bias (macro_f1) | 0.135272 | 0.461714 |
| korunsmile (f1)                  | 0.254562 | 0.315907 |
| nsmc (acc)                       | 0.61248  | 0.84256  |
| pawsx_ko (acc)                   | 0.5615   | 0.5365   |


results/all/beomi/llama-2-ko-7b
|                                  |        0 |        5 |       10 |       50 |
|:---------------------------------|---------:|---------:|---------:|---------:|
| kobest_boolq (macro_f1)          | 0.612147 | 0.682832 | 0.713392 | 0.71622  |
| kobest_copa (macro_f1)           | 0.759784 | 0.799843 | 0.807907 | 0.829976 |
| kobest_hellaswag (macro_f1)      | 0.447951 | 0.460632 | 0.464623 | 0.458628 |
| kobest_sentineg (macro_f1)       | 0.3517   | 0.969773 | 0.977329 | 0.97481  |
| kohatespeech (macro_f1)          | 0.314636 | 0.383336 | 0.357491 | 0.366585 |
| kohatespeech_apeach (macro_f1)   | 0.346127 | 0.567627 | 0.583391 | 0.629269 |
| kohatespeech_gen_bias (macro_f1) | 0.204651 | 0.509189 | 0.471078 | 0.451119 |
| korunsmile (f1)                  | 0.290663 | 0.306208 | 0.304279 | 0.343946 |
| nsmc (acc)                       | 0.57942  | 0.84242  | 0.87368  | 0.8939   |
| pawsx_ko (acc)                   | 0.538    | 0.52     | 0.5275   | 0.5195   |