metadata

language:
  - en
  - zh
  - de
  - es
  - ru
  - ko
  - fr
  - ja
  - pt
  - tr
  - pl
  - ca
  - nl
  - ar
  - sv
  - it
  - id
  - hi
  - fi
  - vi
  - iw
  - uk
  - el
  - ms
  - cs
  - ro
  - da
  - hu
  - ta
  - 'no'
  - th
  - ur
  - hr
  - bg
  - lt
  - la
  - mi
  - ml
  - cy
  - sk
  - te
  - fa
  - lv
  - bn
  - sr
  - az
  - sl
  - kn
  - et
  - mk
  - br
  - eu
  - is
  - hy
  - ne
  - mn
  - bs
  - kk
  - sq
  - sw
  - gl
  - mr
  - pa
  - si
  - km
  - sn
  - yo
  - so
  - af
  - oc
  - ka
  - be
  - tg
  - sd
  - gu
  - am
  - yi
  - lo
  - uz
  - fo
  - ht
  - ps
  - tk
  - nn
  - mt
  - sa
  - lb
  - my
  - bo
  - tl
  - mg
  - as
  - tt
  - haw
  - ln
  - ha
  - ba
  - jw
  - su
tags:
  - speech
  - audio
  - automatic-speech-recognition
license: apache-2.0

Whisper

OpenAI's Whisper

The Whisper model was proposed in Robust Speech Recognition via Large-Scale Weak Supervision by Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever.

Disclaimer: The team releasing Whisper wrote an official model card, which is available here. Content from this model card has been written by the Hugging Face team.

Intro

The first paragraphs of the abstract read as follows :

We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zeroshot transfer setting without the need for any finetuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.

The original code repository can be found here.

Model description

Whisper is an auto-regressive automatic speech recognition encoder-decoder model that was trained on 680 000 hours of 16kHz sampled multilingual audio. It was fully trained in a supervised manner, with multiple tasks :

English transcription
Any-to-English speech translation
Non-English transcription
No speech prediction

To each task corresponds a sequence of tokens that are given to the decoder as context tokens. The beginning of a transcription always starts with <|startoftranscript|> which is why the decoder_start_token is always set to tokenizer.encode("<|startoftranscript|>"). The following token should be the language token, which is automatically detected in the original code. Finally, the task is define using either <|transcribe|> or <|translate|>. In addition, a <|notimestamps|> token is added if the task does not include timestamp prediction.

Usage

To transcribe or translate audio files, the model has to be used along a WhisperFeatureExtractor.

Transcription

In the following example, the english only model is used. We set the decoder_input_ids accordingly.

English to english

The "<|en|>" token is used to specify that the speech is in english and should be transcribed to english

>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration
>>> from datasets import load_dataset
>>> import torch

>>> # load model and processor
>>> processor = WhisperProcessor.from_pretrained("openai/whisper-large")
>>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large")

>>> decoder_input_ids = processor.tokenizer.encode("<|startoftranscript|><|en|><|transcribe|><|notimestamps|>", return_tensors="pt")

>>> # load dummy dataset and read soundfiles
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")

>>> # tokenize
>>> input_features = processor(ds[0]["audio"]["array"], return_tensors="pt").input_features 
>>> # retrieve logits
>>> logits = model(input_features, decoder_input_ids).logits
>>> # take argmax and decode
>>> predicted_ids = torch.argmax(logits, dim=-1)
>>> transcription = processor.batch_decode(predicted_ids)
['<|en|><|transcribe|><|notimestamps|> Mr']

French to French

In order to obtain the full transcription, the generate() function is used. The following example demonstrates a french to french transcription.

>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration
>>> from datasets import load_dataset
>>> import torch

>>> # load model and processor
>>> processor = WhisperProcessor.from_pretrained("openai/whisper-large")
>>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large")

>>> decoder_input_ids = processor.tokenizer.encode("<|startoftranscript|><|fr|><|transcribe|><|notimestamps|>", return_tensors="pt")

>>> # load dummy dataset and read soundfiles
>>> ds = load_dataset("common_voice", "fr", split="test", streaming=True)
>>> ds = ds.cast_column("audio", datasets.Audio(sampling_rate=16_000))
>>> input_speech = next(iter(ds))["audio"]["array"]
>>> # tokenize
>>> input_features = processor(input_speech, return_tensors="pt").input_features 
>>> predicted_ids = model.generate(input_features, decoder_input_ids = decoder_input_ids, max_lenght = 460_000)
>>> transcription = processor.batch_decode(predicted_ids)
['<|startoftranscript|><|fr|><|transcribe|><|notimestamps|> Un vrai travail intéressant va enfin être mené sur ce sujet.<|endoftext|>']

>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens = True)
[' Un vrai travail intéressant va enfin être mené sur ce sujet.']

Translation

The "<|translate|>" is used as the first decoder input token to specify the transcription task.

French to English

>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration
>>> from datasets import load_dataset
>>> import torch

>>> # load model and processor
>>> processor = WhisperProcessor.from_pretrained("openai/whisper-large")
>>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large")

>>> decoder_input_ids = processor.tokenizer.encode("<|startoftranscript|><|fr|><|translate|><|notimestamps|>", return_tensors="pt")

>>> # load dummy dataset and read soundfiles
>>> ds = load_dataset("common_voice", "fr", split="test", streaming=True)
>>> ds = ds.cast_column("audio", datasets.Audio(sampling_rate=16_000))
>>> input_speech = next(iter(ds))["audio"]["array"]
>>> # tokenize
>>> input_features = processor(input_speech, return_tensors="pt").input_features 
>>> predicted_ids = model.generate(input_features, decoder_input_ids = decoder_input_ids, max_lenght = 460_000)
>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens = True)
[' A real interesting work will be done on this subject.']

Evaluation

This code snippet shows how to evaluate openai/whisper-large on LibriSpeech's "clean" and "other" test data.

>>> from datasets import load_dataset
>>> from transformers import WhisperForConditionalGeneration, WhisperProcessor
>>> import soundfile as sf
>>> import torch
>>> from jiwer import wer


>>> librispeech_eval = load_dataset("librispeech_asr", "clean", split="test")

>>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large").to("cuda")
>>> processor = WhisperProcessor.from_pretrained("openai/whisper-large")

>>> def map_to_pred(batch):
>>>     input_features = processor(batch["audio"]["array"], return_tensors="pt").input_features

>>>     with torch.no_grad():
>>>         logits = model(input_features.to("cuda")).logits

>>>     predicted_ids = torch.argmax(logits, dim=-1)
>>>     transcription = processor.batch_decode(predicted_ids, normalize = True)
>>>     batch['text'] = processor.tokenizer._normalize(batch['text'])
>>>     batch["transcription"] = transcription
>>>     return batch

>>> result = librispeech_eval.map(map_to_pred, batched=True, batch_size=1, remove_columns=["speech"])

>>> print("WER:", wer(result["text"], result["transcription"]))
0.030003583080317572

Training Data

Out of the 680 000 hours of training data, 65% are English language audio and their corresponding transcript. 18% of the remaining data is non-English audio and their corresponding translated transcript in english, while the remaining 17% are non english audio with their non english transcript. This non-English data represents 98 different languages.

As discussed in the accompanying paper, we see that performance on transcription in a given language is directly correlated with the amount of training data we employ in that language.

Performance and Limitations

Whisper shows strong robustness to accents, background noise, technical language, as well as zero shot translation from multiple languages into English; and that accuracy on speech recognition and translation is near the state-of-the-art level. The results are all highly correlated with the available training data in the studied language.

A common model hallucination (a predicted text that is not present in the audio) includes notably classic ending of youtube video : "Thank you for listening. Please subscribe to the channel".

The authors also mention that :

The sequence-to-sequence architecture of the model makes it prone to generating repetitive texts, which can be mitigated to some degree by beam search and temperature scheduling but not perfectly. Further analysis on these limitations are provided in the paper. It is likely that this behavior and hallucinations may be worse on lower-resource and/or lower-discoverability languages.

BibTeX entry and citation info

@misc{zhang2022opt,
      title={Robust Speech Recognition via Large-Scale Weak Supervision.}, 
      author={Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever},
      year={2022},
      url={https://cdn.openai.com/papers/whisper.pdf},
}