W2V2-Bert_DigitalUmuganda_Afrivoice_Shona_1hr_v2

This model is a fine-tuned version of facebook/w2v-bert-2.0 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.4556
• Model Preparation Time: 0.0127
• Wer: 0.3692
• Cer: 0.0718

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 500
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Model Preparation Time	Wer	Cer
7.2051	0.9778	22	6.1914	0.0127	1.0	0.9176
4.1072	2.0	45	3.1571	0.0127	1.0	0.9905
3.0778	2.9778	67	2.8928	0.0127	1.0	0.9418
2.6835	4.0	90	2.5263	0.0127	1.0	0.7968
2.0498	4.9778	112	1.1299	0.0127	0.9935	0.3234
0.6781	6.0	135	0.4324	0.0127	0.5731	0.0952
0.3809	6.9778	157	0.3307	0.0127	0.4004	0.0667
0.2728	8.0	180	0.2900	0.0127	0.3482	0.0571
0.2301	8.9778	202	0.2700	0.0127	0.3382	0.0549
0.1725	10.0	225	0.2791	0.0127	0.3510	0.0541
0.1417	10.9778	247	0.2745	0.0127	0.3643	0.0551
0.1158	12.0	270	0.3095	0.0127	0.3256	0.0534
0.1125	12.9778	292	0.2983	0.0127	0.3407	0.0555
0.0854	14.0	315	0.3183	0.0127	0.3505	0.0540
0.078	14.9778	337	0.3576	0.0127	0.3409	0.0549
0.0661	16.0	360	0.3444	0.0127	0.3550	0.0567
0.0736	16.9778	382	0.3750	0.0127	0.3570	0.0595
0.0584	18.0	405	0.3570	0.0127	0.3738	0.0583
0.0503	18.9778	427	0.4068	0.0127	0.3740	0.0612
0.0498	20.0	450	0.3769	0.0127	0.3545	0.0581
0.0549	20.9778	472	0.3819	0.0127	0.3620	0.0574
0.049	22.0	495	0.3945	0.0127	0.3632	0.0582

Framework versions

• Transformers 4.44.1
• Pytorch 2.2.0+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1