w2v-bert-2.0-CV_Fleurs-lg-20hrs-v4

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.8219
• Wer: 0.3120
• Cer: 0.0655

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 4
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer
0.9353	1.0	1029	0.4773	0.5391	0.1166
0.3801	2.0	2058	0.3596	0.4311	0.0915
0.3008	3.0	3087	0.3586	0.4319	0.0903
0.252	4.0	4116	0.3576	0.3906	0.0854
0.2175	5.0	5145	0.3192	0.3979	0.0818
0.1932	6.0	6174	0.3230	0.3757	0.0810
0.1727	7.0	7203	0.3530	0.3784	0.0817
0.1568	8.0	8232	0.2885	0.3631	0.0749
0.1401	9.0	9261	0.3374	0.3719	0.0783
0.1237	10.0	10290	0.3464	0.3799	0.0791
0.1119	11.0	11319	0.3175	0.3537	0.0742
0.0993	12.0	12348	0.3254	0.3656	0.0768
0.0877	13.0	13377	0.3709	0.3759	0.0777
0.0791	14.0	14406	0.3799	0.3699	0.0777
0.0687	15.0	15435	0.3888	0.3568	0.0754
0.0599	16.0	16464	0.3878	0.3456	0.0749
0.0537	17.0	17493	0.3574	0.3765	0.0790
0.0476	18.0	18522	0.4234	0.3568	0.0762
0.0443	19.0	19551	0.4370	0.3587	0.0752
0.0387	20.0	20580	0.4226	0.3433	0.0723
0.0372	21.0	21609	0.4647	0.3613	0.0776
0.0327	22.0	22638	0.3934	0.3519	0.0727
0.029	23.0	23667	0.4096	0.3457	0.0732
0.0263	24.0	24696	0.4516	0.3584	0.0732
0.0264	25.0	25725	0.4486	0.3584	0.0747
0.0219	26.0	26754	0.4680	0.3668	0.0758
0.021	27.0	27783	0.5025	0.3746	0.0781
0.0213	28.0	28812	0.4752	0.3541	0.0751
0.0194	29.0	29841	0.4600	0.3759	0.0758
0.0189	30.0	30870	0.4395	0.3432	0.0732
0.0157	31.0	31899	0.4953	0.3593	0.0754
0.0162	32.0	32928	0.4672	0.3462	0.0732
0.0157	33.0	33957	0.4605	0.3518	0.0742
0.0137	34.0	34986	0.4626	0.3462	0.0727
0.0138	35.0	36015	0.4993	0.3460	0.0730
0.0116	36.0	37044	0.4761	0.3501	0.0734
0.0117	37.0	38073	0.4852	0.3437	0.0728
0.0119	38.0	39102	0.5100	0.3469	0.0739
0.0108	39.0	40131	0.4715	0.3393	0.0722
0.0094	40.0	41160	0.4893	0.3592	0.0734
0.0101	41.0	42189	0.4780	0.3480	0.0729
0.0103	42.0	43218	0.4702	0.3576	0.0738
0.0093	43.0	44247	0.4882	0.3456	0.0731
0.0084	44.0	45276	0.4869	0.3389	0.0710
0.0072	45.0	46305	0.5072	0.3456	0.0720
0.0075	46.0	47334	0.5266	0.3435	0.0715
0.0079	47.0	48363	0.4947	0.3409	0.0724
0.0075	48.0	49392	0.5071	0.3354	0.0707
0.0064	49.0	50421	0.5440	0.3328	0.0705
0.0061	50.0	51450	0.4619	0.3359	0.0706
0.0057	51.0	52479	0.4971	0.3333	0.0701
0.006	52.0	53508	0.5319	0.3470	0.0734
0.0058	53.0	54537	0.5140	0.3353	0.0724
0.0049	54.0	55566	0.5213	0.3297	0.0696
0.0049	55.0	56595	0.5396	0.3306	0.0692
0.0047	56.0	57624	0.4992	0.3304	0.0697
0.0044	57.0	58653	0.5217	0.3362	0.0705
0.0045	58.0	59682	0.5248	0.3291	0.0691
0.0038	59.0	60711	0.5136	0.3286	0.0698
0.0042	60.0	61740	0.4938	0.3313	0.0680
0.0037	61.0	62769	0.4934	0.3329	0.0693
0.0031	62.0	63798	0.5179	0.3287	0.0692
0.0036	63.0	64827	0.5062	0.3244	0.0676
0.0037	64.0	65856	0.4886	0.3225	0.0669
0.0029	65.0	66885	0.4942	0.3238	0.0689
0.003	66.0	67914	0.5234	0.3310	0.0680
0.0026	67.0	68943	0.5014	0.3256	0.0688
0.0024	68.0	69972	0.5699	0.3310	0.0691
0.0021	69.0	71001	0.5402	0.3223	0.0674
0.0017	70.0	72030	0.5824	0.3299	0.0703
0.0022	71.0	73059	0.5423	0.3254	0.0681
0.0016	72.0	74088	0.5683	0.3257	0.0673
0.0011	73.0	75117	0.6068	0.3293	0.0690
0.0014	74.0	76146	0.5651	0.3310	0.0681
0.0012	75.0	77175	0.5723	0.3268	0.0681
0.0013	76.0	78204	0.5953	0.3257	0.0666
0.0011	77.0	79233	0.5892	0.3218	0.0666
0.0014	78.0	80262	0.5929	0.3244	0.0679
0.0016	79.0	81291	0.5981	0.3175	0.0659
0.0014	80.0	82320	0.5880	0.3284	0.0681
0.0009	81.0	83349	0.6205	0.3176	0.0657
0.0006	82.0	84378	0.6295	0.3217	0.0660
0.0008	83.0	85407	0.5807	0.3246	0.0670
0.0007	84.0	86436	0.6224	0.3242	0.0666
0.0006	85.0	87465	0.6552	0.3182	0.0660
0.0005	86.0	88494	0.6757	0.3195	0.0670
0.0007	87.0	89523	0.5980	0.3223	0.0667
0.0005	88.0	90552	0.6357	0.3209	0.0667
0.0008	89.0	91581	0.6362	0.3194	0.0662
0.0005	90.0	92610	0.6596	0.3175	0.0666
0.0003	91.0	93639	0.6934	0.3186	0.0663
0.0002	92.0	94668	0.7177	0.3211	0.0671
0.0001	93.0	95697	0.7225	0.3181	0.0662
0.0001	94.0	96726	0.7272	0.3137	0.0656
0.0	95.0	97755	0.7488	0.3144	0.0658
0.0	96.0	98784	0.7746	0.3129	0.0656
0.0	97.0	99813	0.7903	0.3133	0.0657
0.0	98.0	100842	0.8061	0.3129	0.0656
0.0	99.0	101871	0.8173	0.3119	0.0655
0.0	100.0	102900	0.8219	0.3120	0.0655

Framework versions

• Transformers 4.46.0
• Pytorch 2.1.0+cu118
• Datasets 3.0.2
• Tokenizers 0.20.1