w2v-bert-2.0-CV_Fleurs-lg-5hrs-v4

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.9094
• Wer: 0.3815
• Cer: 0.0808

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 4
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer
1.9959	1.0	258	0.5475	0.6078	0.1338
0.5447	2.0	516	0.5650	0.5979	0.1310
0.4574	3.0	774	0.4544	0.5169	0.1122
0.3712	4.0	1032	0.4466	0.5049	0.1086
0.3187	5.0	1290	0.4438	0.4769	0.1004
0.276	6.0	1548	0.4189	0.4794	0.1057
0.235	7.0	1806	0.4696	0.4797	0.1004
0.2086	8.0	2064	0.4218	0.4603	0.0963
0.1883	9.0	2322	0.4250	0.4521	0.0976
0.1671	10.0	2580	0.4507	0.4586	0.0973
0.1466	11.0	2838	0.4738	0.4634	0.0970
0.1313	12.0	3096	0.4601	0.4411	0.0936
0.1181	13.0	3354	0.4736	0.4217	0.0891
0.1026	14.0	3612	0.4470	0.4278	0.0912
0.092	15.0	3870	0.4730	0.4620	0.0956
0.083	16.0	4128	0.5339	0.4482	0.0937
0.0744	17.0	4386	0.4855	0.4509	0.0944
0.0697	18.0	4644	0.5221	0.4375	0.0903
0.0602	19.0	4902	0.5148	0.4271	0.0894
0.056	20.0	5160	0.5518	0.4313	0.0898
0.05	21.0	5418	0.5374	0.4310	0.0912
0.0464	22.0	5676	0.5167	0.4265	0.0899
0.0453	23.0	5934	0.5782	0.4227	0.0881
0.0412	24.0	6192	0.5275	0.4353	0.0929
0.0369	25.0	6450	0.6112	0.4234	0.0919
0.0339	26.0	6708	0.6159	0.4164	0.0909
0.0316	27.0	6966	0.5938	0.4032	0.0845
0.0263	28.0	7224	0.5883	0.4094	0.0871
0.0268	29.0	7482	0.6013	0.4148	0.0871
0.0269	30.0	7740	0.6137	0.4218	0.0912
0.0234	31.0	7998	0.5840	0.4099	0.0873
0.0229	32.0	8256	0.6286	0.4041	0.0861
0.0205	33.0	8514	0.5923	0.3968	0.0859
0.0196	34.0	8772	0.6188	0.4050	0.0895
0.0191	35.0	9030	0.6255	0.4149	0.0872
0.0185	36.0	9288	0.5938	0.4104	0.0886
0.0163	37.0	9546	0.6004	0.4076	0.0864
0.0171	38.0	9804	0.6485	0.4010	0.0865
0.0163	39.0	10062	0.6360	0.4035	0.0851
0.0144	40.0	10320	0.6230	0.4107	0.0879
0.0133	41.0	10578	0.6123	0.4066	0.0878
0.0131	42.0	10836	0.6532	0.4037	0.0872
0.0122	43.0	11094	0.6613	0.4064	0.0865
0.0127	44.0	11352	0.6279	0.4023	0.0849
0.0115	45.0	11610	0.6950	0.3994	0.0845
0.01	46.0	11868	0.7085	0.3924	0.0849
0.0093	47.0	12126	0.6729	0.4131	0.0871
0.0106	48.0	12384	0.6983	0.4096	0.0892
0.0087	49.0	12642	0.6784	0.4139	0.0885
0.0093	50.0	12900	0.6546	0.4025	0.0851
0.0088	51.0	13158	0.6772	0.3998	0.0846
0.0086	52.0	13416	0.6763	0.3991	0.0863
0.0075	53.0	13674	0.6990	0.3952	0.0841
0.0062	54.0	13932	0.6648	0.3936	0.0832
0.0072	55.0	14190	0.7062	0.4115	0.0866
0.0066	56.0	14448	0.6819	0.4044	0.0868
0.0053	57.0	14706	0.7053	0.4044	0.0859
0.004	58.0	14964	0.6890	0.3966	0.0833
0.0038	59.0	15222	0.7095	0.4009	0.0850
0.005	60.0	15480	0.6999	0.3943	0.0850
0.0055	61.0	15738	0.7265	0.3958	0.0846
0.0043	62.0	15996	0.7267	0.3927	0.0837
0.0038	63.0	16254	0.7014	0.3868	0.0837
0.0026	64.0	16512	0.7609	0.3910	0.0835
0.0024	65.0	16770	0.7436	0.4052	0.0875
0.0044	66.0	17028	0.7610	0.3849	0.0827
0.0049	67.0	17286	0.7387	0.4077	0.0874
0.0057	68.0	17544	0.7030	0.3888	0.0833
0.0028	69.0	17802	0.7499	0.3971	0.0834
0.0027	70.0	18060	0.6728	0.3918	0.0835
0.0021	71.0	18318	0.7420	0.3884	0.0835
0.0024	72.0	18576	0.7596	0.3931	0.0843
0.0024	73.0	18834	0.7565	0.3874	0.0816
0.0019	74.0	19092	0.7420	0.3821	0.0814
0.0015	75.0	19350	0.7394	0.3845	0.0829
0.0012	76.0	19608	0.8261	0.3752	0.0814
0.0012	77.0	19866	0.7902	0.3849	0.0824
0.0008	78.0	20124	0.7845	0.3758	0.0804
0.001	79.0	20382	0.7995	0.3759	0.0809
0.0008	80.0	20640	0.7891	0.3844	0.0827
0.0028	81.0	20898	0.7151	0.3861	0.0823
0.0005	82.0	21156	0.7941	0.3850	0.0820
0.0005	83.0	21414	0.8362	0.3943	0.0836
0.0005	84.0	21672	0.8138	0.3809	0.0807
0.0013	85.0	21930	0.7675	0.3958	0.0837
0.001	86.0	22188	0.7725	0.3894	0.0828
0.0008	87.0	22446	0.7768	0.3907	0.0829
0.0004	88.0	22704	0.7767	0.3862	0.0817
0.0008	89.0	22962	0.7997	0.3849	0.0819
0.0	90.0	23220	0.8321	0.3819	0.0814
0.0	91.0	23478	0.8475	0.3820	0.0808
0.0	92.0	23736	0.8629	0.3815	0.0808
0.0	93.0	23994	0.8769	0.3808	0.0807
0.0	94.0	24252	0.8871	0.3808	0.0807
0.0	95.0	24510	0.9020	0.3821	0.0808
0.0	96.0	24768	0.9094	0.3815	0.0808

Framework versions

• Transformers 4.46.0
• Pytorch 2.1.0+cu118
• Datasets 3.0.2
• Tokenizers 0.20.1