Update README.md

README.md

@@ -1,3 +1,28 @@
 ---
 license: apache-2.0
+language:
+- pl
+tags:
+- qlora
+- instruct
 ---
+
+This repo contains a qlora adapter for [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1), fine-tuned on a private instructions dataset, exclusively in Polish.
+
+The fine-tuning took 1 hour on a single RTX 4090 with the following hyperparameters:
+
+* context length: 2048
+* batch_size: 16
+* learning_rate: 0.0001
+* lora_r: 64
+* lora_alpha: 16
+* lora_modules: all
+* lora_dropout: 0.0
+* weight_decay: 0.1
+* max_grad_norm: 0.3
+* double_quant, nf4
+* optimizer: paged_adamw_32bit (beta2: 0.999)
+
+During the training of the adapter, the instruction format was maintained. Each prompt should be surrounded by `[INST]` and `[\INST]` tokens to leverage instruction fine-tuning.
+
+This adapter allows the model to speak Polish more accurately than vanilla [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1).