yanolja/KoSOLAR-10.7B-v0.1-deprecated

Discord

If you're passionate about the field of Large Language Models and wish to exchange knowledge and insights, we warmly invite you to join our Discord server. It's worth noting that Korean is the primary language used in this server. The landscape of LLM is evolving rapidly, and without active sharing, our collective knowledge risks becoming outdated swiftly. Let's collaborate and drive greater impact together! Join us here: https://discord.gg/b27bAHg95m.

Caution

This model is DEPRECATED due to an issue with the tokenizer. A new, corrected version will be uploaded shortly. We strongly advise against fine-tuning this model until the updated version is available. Details for the new version will be provided in a separate model card.

yanolja/KoSOLAR-10.7B-v0.1

This model is a Korean vocabulary-extended version of upstage/SOLAR-10.7B-v1.0, specifically pre-trained on various Korean web-crawled datasets available on HuggingFace. Our approach was to expand the model's understanding of Korean by pre-training the embeddings for new tokens while preserving the original parameters of the base model.

Model Description

Most parameters of upstage/SOLAR-10.7B-v1.0 were kept frozen during our training process. Only the embeddings for the newly added Korean tokens in the embed_tokens layer and the lm_head layer were pre-trained. This approach allowed us to enhance the model's performance in Korean while maintaining its original English capabilities.

Intended Uses & Limitations

No instruction tuning has been performed on this model. We recommend further training for specific purposes with caution, as it was primarily enhanced for Korean language understanding.

Training and Evaluation Data

The model was pre-trained on various Korean web-crawled datasets openly available on HuggingFace.

Training Procedure

Clarification on "Pre-trained"

It's essential to understand what "pre-trained" means in the context of this model. While the base model was already pre-trained on a broad, non-task-specific corpus of data, we further pre-trained only the embeddings for the expanded Korean vocabulary. This means that we did not alter the other existing parameters from the base model at all. This approach ensures a robust understanding of both English and Korean.

Training Hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 4
• total_train_batch_size: 256
• total_eval_batch_size: 64
• optimizer: Adam with betas=(0.9, 0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 1

Training Results

upstage/SOLAR-10.7B-v1.0

Groups	Version	Filter	n-shot	Metric	Value		Stderr
kmmlu	N/A	none	0	acc	0.3004	±	0.0528
gsm8k	Yaml	get-answer	5	exact_match	0.5625	±	0.0137
hellaswag	Yaml	none	0	acc	0.6393	±	0.0048
mmlu	N/A	none	0	acc	0.6305	±	0.1452
truthfulqa	N/A	none	0	acc	0.4096	±	0.0467
winogrande	Yaml	none	0	acc	0.7443	±	0.0123

yanolja/KoSOLAR-10.7B-v0.1

Groups	Version	Filter	n-shot	Metric	Value		Stderr
kmmlu	N/A	none	0	acc	0.2948	±	0.0537
gsm8k	Yaml	get-answer	5	exact_match	0.5527	±	0.0137
hellaswag	Yaml	none	0	acc	0.6392	±	0.0048
mmlu	N/A	none	0	acc	0.6303	±	0.1411
truthfulqa	N/A	none	0	acc	0.3618	±	0.0472
winogrande	Yaml	none	0	acc	0.7459	±	0.0122

Framework Versions

• Transformers 4.37.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.16.0
• Tokenizers 0.15.0