Mistral-7B-Instruct-v0.1-LC-noSW

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.1 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.8686

Model description

This model is a fine-tuning of Mistral-7B-Instruct-v0.1. This FT was donne with full attention (removing the 4k SWA).

Intended uses & limitations

More information needed

Training and evaluation data

Data is a 9k sample from the RedPajama datset. The context is <=7200 with a decreasing exponential distribution of scale 1500.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 32
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 20
• training_steps: 300

Training results

Training Loss	Epoch	Step	Validation Loss
2.0233	0.18	50	1.8792
2.0805	0.36	100	1.8732
1.9414	0.55	150	1.8705
1.8302	0.73	200	1.8692
2.015	0.91	250	1.8686
1.894	1.09	300	1.8686

Framework versions

• Transformers 4.34.1
• Pytorch 2.0.0+cu117
• Datasets 2.14.6
• Tokenizers 0.14.1