w2v2_bert-Wolof-20-hours-kallaama-dataset

This model is a fine-tuned version of facebook/w2v-bert-2.0 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.4678
• Wer: 0.4571
• Cer: 0.2338

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 50

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer
2.4268	1.4953	400	1.6528	0.7331	0.3584
1.7	2.9907	800	1.7187	0.7569	0.3467
1.6858	4.4860	1200	1.6141	0.7141	0.3541
1.7488	5.9813	1600	1.6955	0.7369	0.3975
1.6371	7.4766	2000	1.5959	0.6903	0.3436
1.6326	8.9720	2400	1.6076	0.6882	0.3451
1.5492	10.4673	2800	1.5807	0.6799	0.3584
1.4962	11.9626	3200	1.5306	0.6343	0.3230
1.4136	13.4579	3600	1.5010	0.6151	0.3048
1.3613	14.9533	4000	1.5595	0.6145	0.3093
1.3313	16.4486	4400	1.4317	0.6016	0.3085
1.2857	17.9439	4800	1.3980	0.5948	0.3018
1.2162	19.4393	5200	1.3360	0.5698	0.2928
1.1641	20.9346	5600	1.3910	0.5779	0.2883
1.1238	22.4299	6000	1.3249	0.5558	0.2824
1.0654	23.9252	6400	1.3325	0.5566	0.2787
1.0303	25.4206	6800	1.3517	0.5490	0.2755
0.9952	26.9159	7200	1.2806	0.5800	0.3178
0.9415	28.4112	7600	1.2408	0.5354	0.2775
0.919	29.9065	8000	1.2844	0.5233	0.2675
0.8625	31.4019	8400	1.2743	0.5121	0.2624
0.8223	32.8972	8800	1.2668	0.5239	0.2697
0.7765	34.3925	9200	1.3101	0.5035	0.2619
0.7445	35.8879	9600	1.2016	0.5059	0.2712
0.7027	37.3832	10000	1.3119	0.4864	0.2512
0.6705	38.8785	10400	1.2881	0.4788	0.2503
0.6212	40.3738	10800	1.3907	0.4733	0.2431
0.6072	41.8692	11200	1.2896	0.4775	0.2431
0.5592	43.3645	11600	1.4088	0.4613	0.2332
0.5299	44.8598	12000	1.3308	0.4584	0.2362
0.4871	46.3551	12400	1.4151	0.4576	0.2364
0.4641	47.8505	12800	1.4593	0.4558	0.2327
0.4345	49.3458	13200	1.4678	0.4571	0.2338

Framework versions

• Transformers 4.44.2
• Pytorch 2.1.0+cu118
• Datasets 2.17.0
• Tokenizers 0.19.1