W2V2-Bert_DigitalUmuganda_Afrivoice_Shona_5hr_v1

This model is a fine-tuned version of facebook/w2v-bert-2.0 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.5836
• Wer: 0.3300
• Cer: 0.0669

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 500
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer
4.2445	1.0	110	2.8442	1.0	0.9853
1.099	2.0	220	0.2941	0.3722	0.0597
0.2699	3.0	330	0.2436	0.2851	0.0489
0.2172	4.0	440	0.2659	0.3090	0.0562
0.1991	5.0	550	0.2614	0.3125	0.0542
0.1848	6.0	660	0.2632	0.2933	0.0506
0.1544	7.0	770	0.2559	0.2831	0.0517
0.1371	8.0	880	0.2605	0.2746	0.0485
0.118	9.0	990	0.2669	0.3128	0.0502
0.0982	10.0	1100	0.2901	0.3135	0.0506
0.0832	11.0	1210	0.2899	0.2728	0.0477
0.0726	12.0	1320	0.2902	0.2791	0.0479
0.0619	13.0	1430	0.3287	0.2893	0.0477
0.0479	14.0	1540	0.3254	0.2664	0.0462
0.0405	15.0	1650	0.3244	0.3025	0.0473
0.0311	16.0	1760	0.3584	0.2753	0.0460
0.0279	17.0	1870	0.3913	0.2748	0.0474
0.0242	18.0	1980	0.3918	0.2678	0.0445
0.0235	19.0	2090	0.3669	0.2761	0.0475
0.0213	20.0	2200	0.3855	0.2631	0.0460
0.016	21.0	2310	0.4096	0.2748	0.0475
0.0154	22.0	2420	0.4276	0.2916	0.0488
0.0127	23.0	2530	0.3918	0.2649	0.0452
0.0115	24.0	2640	0.4195	0.2778	0.0472
0.0105	25.0	2750	0.4143	0.2726	0.0463
0.0086	26.0	2860	0.3923	0.2748	0.0468
0.0111	27.0	2970	0.4108	0.2708	0.0461
0.0107	28.0	3080	0.4169	0.2698	0.0469
0.0083	29.0	3190	0.4363	0.2659	0.0448
0.0085	30.0	3300	0.4340	0.2649	0.0459

Framework versions

• Transformers 4.44.1
• Pytorch 2.2.0+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1