End of training

README.md

@@ -1,55 +1,56 @@
 ---
+language:
+- nl
 license: apache-2.0
 base_model: openai/whisper-small
 tags:
+- nl-asr-leaderboard
 - generated_from_trainer
 datasets:
-- common_voice_11_0
+- mozilla-foundation/common_voice_11_0
+metrics:
+- wer
 model-index:
-- name: whisper-nl-noise
-  results: []
+- name: Whisper Small NL - Noise
+  results:
+  - task:
+      name: Automatic Speech Recognition
+      type: automatic-speech-recognition
+    dataset:
+      name: Common Voice 11.0
+      type: mozilla-foundation/common_voice_11_0
+      config: nl
+      split: None
+      args: 'config: nl, split: test'
+    metrics:
+    - name: Wer
+      type: wer
+      value: 38.08532778355879
 ---
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment. -->
 
-# whisper-nl-noise
+# Whisper Small NL - Noise
 
-This model is a fine-tuned version of [openai/whisper-small](https://huggingface.co/openai/whisper-small) on the the (dutch) [mozilla/common-voice](https://commonvoice.mozilla.org/en/datasets) dataset (11.0). This dataset is augmented with various forms of background noise, retrieved from [pixabay](https://pixabay.com/sound-effects/search/car/)
+This model is a fine-tuned version of [openai/whisper-small](https://huggingface.co/openai/whisper-small) on the Common Voice 11.0 dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.6359
+- Wer: 38.0853
 
 ## Model description
 
-### Whisper (base)
-Whisper is a pre-trained model for automatic speech recognition (ASR) and speech translation. Trained on 680k hours of labelled data, Whisper models demonstrate a strong ability to generalise to many datasets and domains without the need for fine-tuning.
-
-Whisper was proposed in the paper [Robust Speech Recognition via Large-Scale Weak Supervision](https://arxiv.org/abs/2212.04356) by Alec Radford et al from OpenAI. The original code repository can be found [here](https://github.com/openai/whisper).
-
-### Whisper NL (noise)
-This current Whisper model is specifically fine-tuned on noisy Dutch data. It intends to have increased performance on this task. The generalizing aspects of the model will be lost in this process. It will nevertheless build upon the 680 hours of labeled data the base model already received during training. 
+More information needed
 
 ## Intended uses & limitations
 
-This model is specifically trained for (very) noise (Dutch) audio. It is expected that is performs worse on audio files which do not meet these criteria. 
-
-Noise is often bound to specific contexts and recordings. The model will therefore not generalize to all sorts and types of (car) noise.
+More information needed
 
 ## Training and evaluation data
 
-The [mozilla/common-voice](https://commonvoice.mozilla.org/en/datasets) dataset (11.0) was used. With the predefined 'train' and 'test' split. For reasons of time-management, only the first 5% of the test set was used.
+More information needed
 
 ## Training procedure
-The training procedure as outlined in the original [Huggingface blog](https://huggingface.co/blog/fine-tune-whisper) was used, see [here](https://colab.research.google.com/github/sanchit-gandhi/notebooks/blob/main/fine_tune_whisper.ipynb).
-
-The only alteration was made during preparation of the dataset (`prepare_dataset(batch)`). Within this step the data was [augmented](https://pytorch.org/audio/stable/tutorials/audio_data_augmentation_tutorial.html) to include various samples of background noise. These noises were sampled from various audio files and sources. For each audio track used from the common-voice dataset, a random noise sample was mixed with this audio track. The signal to noise ratio (SNR) varied (randomly) between -5, and 1 dB.
-
-$$ \mathrm{SNR} = {{P_{signal}} \over {P_{noise}}} $$
-
-$$ \mathrm{SNR_{dB}} = 10 \log _{10}\mathrm{SNR} $$
-
-This SNR range makes the noise disturbance, on average, quite invasive.
-The augmentation, does, however not alter the transcription of the audio track, these remain unchanged. 
-
-Whilst it would be beneficial to add the noise on the fly during training, to enable varying selections of noise for the same audio track, for efficiency reasons this strategy was not applied. Each audio tracks is therefore augmented with a single (randomly selected) noise track. The length of the audio track remained unchanged. If needed the noise track was repeated or truncated to meet the length of the audio track.
 
 ### Training hyperparameters
 
@@ -60,16 +61,21 @@ The following hyperparameters were used during training:
 - seed: 42
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: linear
-- lr_scheduler_warmup_steps: 2
-- training_steps: 20
+- lr_scheduler_warmup_steps: 200
+- training_steps: 2000
 - mixed_precision_training: Native AMP
 
 ### Training results
 
+| Training Loss | Epoch | Step | Validation Loss | Wer     |
+|:-------------:|:-----:|:----:|:---------------:|:-------:|
+| 0.3722        | 0.26  | 1000 | 0.6908          | 39.8543 |
+| 0.3779        | 0.53  | 2000 | 0.6359          | 38.0853 |
+
 
 ### Framework versions
 
-- Transformers 4.37.2
+- Transformers 4.39.0.dev0
 - Pytorch 2.1.0+cu121
 - Datasets 2.17.1
 - Tokenizers 0.15.2

generation_config.json

@@ -261,5 +261,5 @@
     "transcribe": 50359,
     "translate": 50358
   },
-  "transformers_version": "4.37.2"
+  "transformers_version": "4.39.0.dev0"
 }