wikipedia_clm_13_full

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 3.9138

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 16
• eval_batch_size: 16
• seed: 13
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100000
• training_steps: 400000
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
6.8417	1.9399	4000	5.8853
5.4687	3.8797	8000	5.0961
4.7775	5.8196	12000	4.6049
4.3147	7.7595	16000	4.3387
4.0005	9.6993	20000	4.1758
3.7631	11.6392	24000	4.0702
3.5859	13.5790	28000	4.0086
3.4474	15.5189	32000	3.9848
3.3504	17.4588	36000	3.9548
3.2673	19.3986	40000	3.9505
3.1985	21.3385	44000	3.9164
3.1395	23.2784	48000	3.9289
3.0869	25.2182	52000	3.8992
3.0473	27.1581	56000	3.9205
3.0064	29.0980	60000	3.9049
2.9747	31.0378	64000	3.9138

Framework versions

• Transformers 4.45.2
• Pytorch 2.5.1+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1