license: apache-2.0
base_model: bert-base-cased
tags:
- generated_from_trainer
model-index:
- name: bert-finetuned-squad
results: []
bert-finetuned-squad
This model is a fine-tuned version of bert-base-cased on the SQUAD dataset.
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3
- mixed_precision_training: Native AMP
Training results
Step Training Loss 500 2.635500 1000 1.655900 1500 1.460800 2000 1.378100 2500 1.328600 3000 1.287900 3500 1.236900 4000 1.179500 4500 1.130300 5000 1.163700 5500 1.122700 6000 1.140600 6500 1.141300 7000 1.082100 7500 1.096400 8000 1.108300 8500 1.058300 9000 1.082500 9500 1.026400 10000 1.040700 10500 1.035200 11000 1.010700 11500 0.807700 12000 0.710500 12500 0.784300 13000 0.740100 13500 0.771600 14000 0.777200 14500 0.749000 15000 0.734800 15500 0.749500 16000 0.775600 16500 0.724300 17000 0.768300 17500 0.753600 18000 0.732900 18500 0.734200 19000 0.699800 19500 0.732600 20000 0.764600 20500 0.772900 21000 0.734000 21500 0.734000 22000 0.691000 22500 0.588700 23000 0.514800 23500 0.539000 24000 0.515900 24500 0.490800 25000 0.524200 25500 0.516200 26000 0.486200 26500 0.526000 27000 0.495300 27500 0.527600 28000 0.484800 28500 0.486300 29000 0.522200 29500 0.519200 30000 0.508800 30500 0.516700 31000 0.490600 31500 0.516100 32000 0.499500 32500 0.496100 33000 0.465300
Framework versions
- Transformers 4.38.2
- Pytorch 2.1.0+cu118
- Datasets 2.18.0
- Tokenizers 0.15.2