dgx1_w2v2_base_teacher_student_distillation_mozilla_epochs_100_batch_16_concatenate_datasets

This model is a fine-tuned version of on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 32.7379
  • Wer: 0.9997

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-05
  • train_batch_size: 16
  • eval_batch_size: 1
  • seed: 42
  • gradient_accumulation_steps: 128
  • total_train_batch_size: 2048
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_ratio: 0.2
  • num_epochs: 100
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Wer
81.1182 7.35 500 28.1891 0.9996
62.3932 14.7 1000 29.1354 0.9996
58.4025 22.06 1500 30.3676 0.9997
55.061 29.41 2000 30.5188 0.9996
52.9559 36.76 2500 32.1812 0.9996
51.1888 44.12 3000 32.7379 0.9997

Framework versions

  • Transformers 4.25.1
  • Pytorch 1.12.1
  • Datasets 2.8.0
  • Tokenizers 0.13.2
Downloads last month
2
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support