c4ai-command-r-v01-SFT

This model is a fine-tuned version of CohereForAI/c4ai-command-r-v01-4bit on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 0.6709

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2.5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 0.03
• training_steps: 500

Training results

Training Loss	Epoch	Step	Validation Loss
1.5906	0.2326	10	1.5675
1.5542	0.4651	20	1.4713
1.3934	0.6977	30	1.3839
1.346	0.9302	40	1.2949
1.2605	1.1628	50	1.2154
1.1887	1.3953	60	1.1361
1.0784	1.6279	70	1.0520
1.0086	1.8605	80	0.9770
0.9247	2.0930	90	0.9134
0.8671	2.3256	100	0.8640
0.8391	2.5581	110	0.8284
0.7849	2.7907	120	0.8026
0.7458	3.0233	130	0.7818
0.7377	3.2558	140	0.7684
0.7112	3.4884	150	0.7544
0.7054	3.7209	160	0.7430
0.7029	3.9535	170	0.7331
0.657	4.1860	180	0.7263
0.675	4.4186	190	0.7189
0.6695	4.6512	200	0.7117
0.6418	4.8837	210	0.7058
0.6348	5.1163	220	0.7028
0.6414	5.3488	230	0.6981
0.612	5.5814	240	0.6951
0.6114	5.8140	250	0.6909
0.6149	6.0465	260	0.6876
0.5978	6.2791	270	0.6884
0.5955	6.5116	280	0.6839
0.6112	6.7442	290	0.6802
0.5841	6.9767	300	0.6794
0.5746	7.2093	310	0.6801
0.5849	7.4419	320	0.6773
0.5863	7.6744	330	0.6760
0.5738	7.9070	340	0.6748
0.5696	8.1395	350	0.6739
0.5632	8.3721	360	0.6747
0.5732	8.6047	370	0.6737
0.5512	8.8372	380	0.6725
0.5761	9.0698	390	0.6716
0.5471	9.3023	400	0.6727
0.5579	9.5349	410	0.6724
0.573	9.7674	420	0.6714
0.5459	10.0	430	0.6708
0.5677	10.2326	440	0.6710
0.5453	10.4651	450	0.6708
0.5638	10.6977	460	0.6708
0.5473	10.9302	470	0.6709
0.5553	11.1628	480	0.6709
0.5535	11.3953	490	0.6708
0.5409	11.6279	500	0.6709

Framework versions

• PEFT 0.10.1.dev0
• Transformers 4.40.0.dev0
• Pytorch 2.1.1+cu121
• Datasets 2.18.0
• Tokenizers 0.19.0