w2v-bert-2.0-Fleurs_AMMI_AFRIVOICE_LRSC-ln-5hrs-v3

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.9406
• Wer: 0.2246
• Cer: 0.0716

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 8
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 80
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer
2.9645	0.9949	98	0.8966	0.7007	0.1938
0.6444	2.0	197	0.5263	0.3547	0.1095
0.4524	2.9949	295	0.4768	0.3064	0.0953
0.3671	4.0	394	0.4343	0.2827	0.0894
0.3124	4.9949	492	0.4295	0.2712	0.0888
0.2645	6.0	591	0.4370	0.2709	0.0873
0.2301	6.9949	689	0.4414	0.2643	0.0809
0.1917	8.0	788	0.4229	0.2525	0.0791
0.168	8.9949	886	0.4027	0.2495	0.0810
0.1428	10.0	985	0.4432	0.2583	0.0815
0.1299	10.9949	1083	0.4514	0.2624	0.0812
0.1087	12.0	1182	0.4549	0.2467	0.0762
0.0921	12.9949	1280	0.4699	0.2388	0.0755
0.0791	14.0	1379	0.5294	0.2340	0.0733
0.0749	14.9949	1477	0.5567	0.2490	0.0785
0.0666	16.0	1576	0.5504	0.2511	0.0823
0.0608	16.9949	1674	0.5643	0.2359	0.0747
0.049	18.0	1773	0.5892	0.2351	0.0738
0.0435	18.9949	1871	0.5814	0.2431	0.0757
0.0397	20.0	1970	0.6019	0.2494	0.0773
0.0351	20.9949	2068	0.6276	0.2517	0.0775
0.0298	22.0	2167	0.6176	0.2426	0.0763
0.0263	22.9949	2265	0.6573	0.2350	0.0743
0.0242	24.0	2364	0.6754	0.2369	0.0751
0.023	24.9949	2462	0.6835	0.2387	0.0768
0.019	26.0	2561	0.7021	0.2340	0.0737
0.0175	26.9949	2659	0.7082	0.2449	0.0758
0.0157	28.0	2758	0.7215	0.2291	0.0726
0.0136	28.9949	2856	0.7086	0.2391	0.0743
0.0117	30.0	2955	0.7628	0.2402	0.0769
0.0148	30.9949	3053	0.7502	0.2466	0.0777
0.011	32.0	3152	0.7490	0.2417	0.0753
0.0095	32.9949	3250	0.7873	0.2372	0.0742
0.0078	34.0	3349	0.8005	0.2347	0.0733
0.0068	34.9949	3447	0.8139	0.2408	0.0742
0.0098	36.0	3546	0.8041	0.2432	0.0767
0.006	36.9949	3644	0.8383	0.2323	0.0732
0.0052	38.0	3743	0.9155	0.2294	0.0735
0.0092	38.9949	3841	0.9501	0.2222	0.0716
0.0155	40.0	3940	0.8676	0.2316	0.0731
0.0053	40.9949	4038	0.8519	0.2302	0.0723
0.0041	42.0	4137	0.8310	0.2365	0.0752
0.0035	42.9949	4235	0.8959	0.2368	0.0731
0.003	44.0	4334	0.8903	0.2327	0.0730
0.0029	44.9949	4432	0.9306	0.2269	0.0715
0.0029	46.0	4531	1.0210	0.2346	0.0721
0.0055	46.9949	4629	1.0385	0.2362	0.0725
0.0041	48.0	4728	0.9412	0.2376	0.0726
0.0027	48.9949	4826	0.9144	0.2428	0.0737
0.0021	50.0	4925	0.9316	0.2520	0.0754
0.0024	50.9949	5023	0.9214	0.2300	0.0726
0.0011	52.0	5122	0.9172	0.2315	0.0731
0.0011	52.9949	5220	0.9218	0.2225	0.0715
0.0009	54.0	5319	0.9406	0.2246	0.0716

Framework versions

• Transformers 4.46.3
• Pytorch 2.1.0+cu118
• Datasets 3.1.0
• Tokenizers 0.20.3