File size: 6,980 Bytes
5ff5f4c 45bbaf1 5ff5f4c 82312b0 5ff5f4c 45bbaf1 5ff5f4c 45bbaf1 21bfddc 45bbaf1 21bfddc 45bbaf1 21bfddc 5ff5f4c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 |
---
language:
- fi
license: apache-2.0
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
- sft
base_model: LumiOpen/Viking-7B
datasets:
- mpasila/Finnish-Alpaca-Small
---
This is a merge of [mpasila/Finnish-Alpaca-Small-LoRA-7B](https://huggingface.co/mpasila/Finnish-Alpaca-Small-LoRA-7B).
LoRA trained in 4-bit with 2k context using [LumiOpen/Viking-7B](https://huggingface.co/LumiOpen/Viking-7B/) as the base model for 1 epoch.
Dataset used is [mpasila/Finnish-Alpaca-Small](https://huggingface.co/datasets/mpasila/Finnish-Alpaca-Small).
Re-trained because I have no idea if I used the fully trained model or the partially trained model (of Viking-7B), since it apparently was just released. (After re-training the score lowered noticeably so I wonder if I screwed up something.)
### Prompt format: Alpaca
It uses Alpaca format but with a translated instruction at the start:
```
{
"instruction,output": "Alla on ohje, jossa kuvataan tehtävä. Kirjoita vastaus, joka täyttää pyynnön asianmukaisesti.\n\n### Instruction:\n%instruction%\n\n### Response:\n%output%",
"instruction,input,output": "Alla on ohje, jossa kuvataan tehtävä ja joka on yhdistetty kontekstia lisäävään syötteeseen. Kirjoita vastaus, joka täyttää pyynnön asianmukaisesti.\n\n### Instruction:\n%instruction%\n\n### Input:\n%input%\n\n### Response:\n%output%"
}
```
## Evaluation
| Model | Size | Type | FIN-bench (score) |
|-------|------|------|-------|
| **mpasila/Finnish-Alpaca-Small-7B** | 7B | Instruct | 0.3586 |
| [mpasila/Finnish-Alpaca-Tiny-V2-7B](https://huggingface.co/mpasila/Finnish-Alpaca-Tiny-V2-7B) | 7B | Instruct | **0.4654** |
| [mpasila/Alpacazord-Viking-7B](https://huggingface.co/mpasila/Alpacazord-Viking-7B) | 7B | Instruct | 0.4123 |
| [mpasila/NordicAlpaca-Finnish-V1-7B](https://huggingface.co/mpasila/NordicAlpaca-Finnish-V1-7B) | 7B | Instruct | 0.3891 |
| [mpasila/Finnish-Viking-Alpaca-V1-7B](https://huggingface.co/mpasila/Finnish-Viking-Alpaca-V1-7B) | 7B | Instruct | 0.3943 |
| [Finnish-NLP/llama-7b-finnish-instruct-v0.1](https://huggingface.co/Finnish-NLP/llama-7b-finnish-instruct-v0.1) | 7B | Instruct | 0.4365 |
| [Finnish-NLP/llama-7b-finnish-instruct-v0.2](https://huggingface.co/Finnish-NLP/llama-7b-finnish-instruct-v0.2) | 7B | Instruct | 0.3993 |
| [Finnish-NLP/llama-7b-finnish](https://huggingface.co/Finnish-NLP/llama-7b-finnish) | 7B | Base | 0.2350 |
| [LumiOpen/Viking-7B (1000B)](https://huggingface.co/LumiOpen/Viking-7B) | 7B | Base | 0.3721 |
| [HPLT/gpt-7b-nordic-prerelease](https://huggingface.co/HPLT/gpt-7b-nordic-prerelease) | 7B | Base | 0.3169 |
[Source](https://docs.google.com/spreadsheets/d/1rqJb9dQVihg-Z1_Ras1L_-wuzPg9xNzpdmM2x5HueeY/edit?usp=sharing)
#### FIN-bench scores:
| Task |Version| Metric |Value | |Stderr|
|------------------------------------------------|------:|---------------------|-----:|---|-----:|
|bigbench_analogies | 0|multiple_choice_grade|0.5923|± |0.0433|
|bigbench_arithmetic_1_digit_addition | 0|multiple_choice_grade|0.2700|± |0.0446|
|bigbench_arithmetic_1_digit_division | 0|multiple_choice_grade|0.4783|± |0.1065|
|bigbench_arithmetic_1_digit_multiplication | 0|multiple_choice_grade|0.2600|± |0.0441|
|bigbench_arithmetic_1_digit_subtraction | 0|multiple_choice_grade|0.2200|± |0.0416|
|bigbench_arithmetic_2_digit_addition | 0|multiple_choice_grade|0.1700|± |0.0378|
|bigbench_arithmetic_2_digit_division | 0|multiple_choice_grade|0.3600|± |0.0482|
|bigbench_arithmetic_2_digit_multiplication | 0|multiple_choice_grade|0.2000|± |0.0402|
|bigbench_arithmetic_2_digit_subtraction | 0|multiple_choice_grade|0.1300|± |0.0338|
|bigbench_arithmetic_3_digit_addition | 0|multiple_choice_grade|0.3100|± |0.0465|
|bigbench_arithmetic_3_digit_division | 0|multiple_choice_grade|0.2100|± |0.0409|
|bigbench_arithmetic_3_digit_multiplication | 0|multiple_choice_grade|0.1600|± |0.0368|
|bigbench_arithmetic_3_digit_subtraction | 0|multiple_choice_grade|0.2300|± |0.0423|
|bigbench_arithmetic_4_digit_addition | 0|multiple_choice_grade|0.3900|± |0.0490|
|bigbench_arithmetic_4_digit_division | 0|multiple_choice_grade|0.2300|± |0.0423|
|bigbench_arithmetic_4_digit_multiplication | 0|multiple_choice_grade|0.2100|± |0.0409|
|bigbench_arithmetic_4_digit_subtraction | 0|multiple_choice_grade|0.4500|± |0.0500|
|bigbench_arithmetic_5_digit_addition | 0|multiple_choice_grade|0.4800|± |0.0502|
|bigbench_arithmetic_5_digit_division | 0|multiple_choice_grade|0.0700|± |0.0256|
|bigbench_arithmetic_5_digit_multiplication | 0|multiple_choice_grade|0.1700|± |0.0378|
|bigbench_arithmetic_5_digit_subtraction | 0|multiple_choice_grade|0.5800|± |0.0496|
|bigbench_cause_and_effect_one_sentence | 0|multiple_choice_grade|0.6275|± |0.0684|
|bigbench_cause_and_effect_one_sentence_no_prompt| 0|multiple_choice_grade|0.6667|± |0.0667|
|bigbench_cause_and_effect_two_sentences | 0|multiple_choice_grade|0.5098|± |0.0707|
|bigbench_emotions | 0|multiple_choice_grade|0.3312|± |0.0373|
|bigbench_empirical_judgments | 0|multiple_choice_grade|0.3333|± |0.0476|
|bigbench_general_knowledge | 0|multiple_choice_grade|0.2857|± |0.0544|
|bigbench_hhh_alignment_harmless | 0|multiple_choice_grade|0.3793|± |0.0643|
|bigbench_hhh_alignment_helpful | 0|multiple_choice_grade|0.3559|± |0.0629|
|bigbench_hhh_alignment_honest | 0|multiple_choice_grade|0.3559|± |0.0629|
|bigbench_hhh_alignment_other | 0|multiple_choice_grade|0.5349|± |0.0770|
|bigbench_intent_recognition | 0|multiple_choice_grade|0.1546|± |0.0138|
|bigbench_misconceptions | 0|multiple_choice_grade|0.5448|± |0.0432|
|bigbench_paraphrase | 0|multiple_choice_grade|0.5300|± |0.0354|
|bigbench_sentence_ambiguity | 0|multiple_choice_grade|0.4333|± |0.0645|
|bigbench_similarities_abstraction | 0|multiple_choice_grade|0.6974|± |0.0530|
# Uploaded model
- **Developed by:** mpasila
- **License:** apache-2.0
- **Finetuned from model :** LumiOpen/Viking-7B
This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
|