README.md

---
inference: false  # need to implement streaming API in HF
language:
  - en
thumbnail: null
tags:
  - automatic-speech-recognition
  - Transducer
  - Conformer
  - pytorch
  - speechbrain
license: apache-2.0
datasets:
  - speechcolab/gigaspeech
metrics:
  - wer
model-index:
  - name: Conformer-Transducer by SpeechBrain
    results:
      - task:
          name: Automatic Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: GigaSpeech
          type: gigaspeech
          split: test
          args:
            language: en
        metrics:
          - name: Test WER (non-streaming greedy)
            type: wer
            value: TBD
          - name: Test WER (960ms chunk size, 8 left context chunks)
            type: wer
            value: TBD
---

<iframe src="https://ghbtns.com/github-btn.html?user=speechbrain&repo=speechbrain&type=star&count=true&size=large&v=2" frameborder="0" scrolling="0" width="170" height="30" title="GitHub"></iframe>
<br/><br/>

# Conformer for Gigaspeech

This repository provides all the necessary tools to perform automatic speech
recognition from an end-to-end system pretrained on Gigaspeech (EN, XL split) within
SpeechBrain. For a better experience, we encourage you to learn more about
[SpeechBrain](https://speechbrain.github.io). 
The performance of the model in full context mode (no streaming) is the following:

| Release | Test WER | GPUs |
|:-------------:|:--------------:|:--------:|
| 24-02-26 | 11.00% | 4xA100 40GB |

TODO: validate the results

With streaming, the results with different chunk sizes on the test split are the following:

|       | full   | cs=32 (1280ms) | 24 (960ms) | 16 (640ms) | 12 (480ms) | 8 (320ms) |
|:-----:|:------:|:------:|:------:|:------:|:------:|:------:|
| full  | 11.00% | -      | -      | -      | -      | -      |
| 16    | -      | -      | -      | 11.70% | 11.84% | 12.14% |
| 8     | -      | -      | 11.50% | 11.72% | 11.88% | 12.28% |
| 4     | -      | 11.40% | 11.53% | 11.81% | 12.03% | 12.64% |
| 2     | -      | 11.46% | 11.67% | 12.03% | 12.43% | 13.25% |
| 1\*   | -      | 11.59% | 11.85% | 12.39% | 12.93% | 14.13% |

(\*: model was never explicitly trained with this setting)

When comparing configurations, bear in mind that the utterances of the test set are of a finite length.  
All WER values were evaluated using [this script](https://gist.github.com/asumagic/66aaf43156c31866cf2ebcffda46ea01) (until we integrate a nicer way to evaluate streaming WER in different conditions).

## Pipeline description

This ASR system is a Conformer model trained with the RNN-T loss (with an auxiliary CTC loss to stabilize training). The model operates with a unigram tokenizer.
Architecture details are described in the [training hyperparameters file](https://github.com/speechbrain/speechbrain/blob/develop/recipes/GigaSpeech/ASR/transducer/hparams/conformer_transducer.yaml).

Streaming support makes use of Dynamic Chunk Training. Chunked attention is used for the multi-head attention module, and an implementation of [Dynamic Chunk Convolutions](https://www.amazon.science/publications/dynamic-chunk-convolution-for-unified-streaming-and-non-streaming-conformer-asr) was used.  
The model was trained with support for different chunk sizes (and even full context), and so is suitable for various chunk sizes and offline transcription.

The system is trained with recordings sampled at 16kHz (single channel).
The code will automatically normalize your audio (i.e., resampling + mono channel selection) when calling `transcribe_file` if needed.

## Install SpeechBrain

First of all, please install SpeechBrain with the following command:

```
pip install speechbrain
```

Please notice that we encourage you to read our tutorials and learn more about
[SpeechBrain](https://speechbrain.github.io).

### Transcribing your own audio files (in English)

```python
from speechbrain.inference.ASR import StreamingASR
from speechbrain.utils.dynamic_chunk_training import DynChunkTrainConfig
asr_model = StreamingASR.from_hparams("speechbrain/asr-streaming-conformer-gigaspeech")
asr_model.transcribe_file(
    "speechbrain/asr-streaming-conformer-librispeech/test-en.wav",
    # select a chunk size of ~960ms with 4 chunks of left context
    DynChunkTrainConfig(24, 4),
    # disable torchaudio streaming to allow fetching from HuggingFace
    # set this to True for your own files or streams to allow for streaming file decoding
    use_torchaudio_streaming=False,
)
```

The `DynChunkTrainConfig` values can be adjusted for a tradeoff of latency, computational power and transcription accuracy. Refer to the streaming WER table to pick a value that is suitable for your usecase.

<details>
<summary>Commandline tool to transcribe a file or a live stream</summary>

**Decoding from a live stream using ffmpeg (BBC Radio 4):**

`python3 asr.py http://as-hls-ww-live.akamaized.net/pool_904/live/ww/bbc_radio_fourfm/bbc_radio_fourfm.isml/bbc_radio_fourfm-audio%3d96000.norewind.m3u8 --model-source=speechbrain/asr-streaming-conformer-gigaspeech --device=cpu -v`

**Decoding from a file:**

`python3 asr.py some-english-speech.wav --model-source=speechbrain/asr-streaming-conformer-gigaspeech --device=cpu -v`

```python
from argparse import ArgumentParser
import logging

parser = ArgumentParser()
parser.add_argument("audio_path")
parser.add_argument("--model-source", required=True)
parser.add_argument("--device", default="cpu")
parser.add_argument("--ip", default="127.0.0.1")
parser.add_argument("--port", default=9431)
parser.add_argument("--chunk-size", default=24, type=int)
parser.add_argument("--left-context-chunks", default=4, type=int)
parser.add_argument("--num-threads", default=None, type=int)
parser.add_argument("--verbose", "-v", default=False, action="store_true")
args = parser.parse_args()

if args.verbose:
    logging.getLogger().setLevel(logging.INFO)

logging.info("Loading libraries")

from speechbrain.inference.ASR import StreamingASR
from speechbrain.utils.dynamic_chunk_training import DynChunkTrainConfig
import torch

device = args.device

if args.num_threads is not None:
    torch.set_num_threads(args.num_threads)

logging.info(f"Loading model from \"{args.model_source}\" onto device {device}")

asr = StreamingASR.from_hparams(args.model_source, run_opts={"device": device})
config = DynChunkTrainConfig(args.chunk_size, args.left_context_chunks)

logging.info(f"Starting stream from URI \"{args.audio_path}\"")

for text_chunk in asr.transcribe_file_streaming(args.audio_path, config):
    print(text_chunk, flush=True, end="")
```
</details>

<details>
  <summary>Live ASR decoding from a browser using Gradio</summary>

We want to optimize some things around the model before we create a proper HuggingFace space demonstrating live streaming on CPU.

In the mean time, this is a simple hacky demo of live ASR in the browser using Gradio's live microphone streaming feature.

If you run this, please note:

- Modern browsers refuse to stream microphone input over an untrusted connection (plain HTTP), unless it is localhost. If you are running this on a remote server, you could use SSH port forwarding to expose the remote's port on your machine.
- Streaming using Gradio on Firefox seems to cause some issues. Chromium-based browsers seem to behave better.

Run using:

`python3 gradio-asr.py --model-source speechbrain/asr-streaming-conformer-gigaspeech --ip=localhost --device=cpu`

```python
from argparse import ArgumentParser
from dataclasses import dataclass
import logging

parser = ArgumentParser()
parser.add_argument("--model-source", required=True)
parser.add_argument("--device", default="cpu")
parser.add_argument("--ip", default="127.0.0.1")
parser.add_argument("--port", default=9431)
parser.add_argument("--chunk-size", default=24, type=int)
parser.add_argument("--left-context-chunks", default=4, type=int)
parser.add_argument("--num-threads", default=None, type=int)
parser.add_argument("--verbose", "-v", default=False, action="store_true")
args = parser.parse_args()

if args.verbose:
    logging.getLogger().setLevel(logging.INFO)

logging.info("Loading libraries")

from speechbrain.inference.ASR import StreamingASR, ASRStreamingContext
from speechbrain.utils.dynamic_chunk_training import DynChunkTrainConfig
import torch
import gradio as gr
import torchaudio
import numpy as np

device = args.device

if args.num_threads is not None:
    torch.set_num_threads(args.num_threads)

logging.info(f"Loading model from \"{args.model_source}\" onto device {device}")

asr = StreamingASR.from_hparams(args.model_source, run_opts={"device": device})
config = DynChunkTrainConfig(args.chunk_size, args.left_context_chunks)

@dataclass
class GradioStreamingContext:
    context: ASRStreamingContext
    chunk_size: int
    waveform_buffer: torch.Tensor
    decoded_text: str

def transcribe(stream, new_chunk):
    sr, y = new_chunk

    y = y.astype(np.float32)
    y = torch.tensor(y, dtype=torch.float32, device=device)
    y /= max(1, torch.max(torch.abs(y)).item())  # norm by max abs() within chunk & avoid NaN
    if len(y.shape) > 1:
        y = torch.mean(y, dim=1)  # downmix to mono

    # HACK: we are making poor use of the resampler across chunk boundaries
    # which may degrade accuracy.
    # NOTE: we should also absolutely avoid recreating a resampler every time
    resampler = torchaudio.transforms.Resample(orig_freq=sr, new_freq=asr.audio_normalizer.sample_rate).to(device)
    y = resampler(y)  # janky resample (probably to 16kHz)


    if stream is None:
        stream = GradioStreamingContext(
            context=asr.make_streaming_context(config),
            chunk_size=asr.get_chunk_size_frames(config),
            waveform_buffer=y,
            decoded_text="",
        )
    else:
        stream.waveform_buffer = torch.concat((stream.waveform_buffer, y))

    while stream.waveform_buffer.size(0) > stream.chunk_size:
        chunk = stream.waveform_buffer[:stream.chunk_size]
        stream.waveform_buffer = stream.waveform_buffer[stream.chunk_size:]

        # fake batch dim
        chunk = chunk.unsqueeze(0)

        # list of transcribed strings, of size 1 because the batch size is 1
        with torch.no_grad():
            transcribed = asr.transcribe_chunk(stream.context, chunk)
        stream.decoded_text += transcribed[0]

    return stream, stream.decoded_text

# NOTE: latency seems relatively high, which may be due to this:
# https://github.com/gradio-app/gradio/issues/6526

demo = gr.Interface(
    transcribe,
    ["state", gr.Audio(sources=["microphone"], streaming=True)],
    ["state", "text"],
    live=True,
)

demo.launch(server_name=args.ip, server_port=args.port)
```

</details>

### Inference on GPU

To perform inference on the GPU, add  `run_opts={"device":"cuda"}`  when calling the `from_hparams` method.

## Parallel Inference on a Batch

Currently, the high level transcription interfaces do not support batched inference, but the low-level interfaces (i.e. `encode_chunk`) do.  
We hope to provide efficient functionality for this in the future.

### Training

The model was trained with SpeechBrain `v1.0.2`.
To train it from scratch, follow these steps:
1. Clone SpeechBrain:
```bash
git clone https://github.com/speechbrain/speechbrain/
```
2. Install it:
```bash
cd speechbrain
pip install -r requirements.txt
pip install -e .
```

3. Follow the steps listed in the [README](https://github.com/speechbrain/speechbrain/blob/develop/recipes/GigaSpeech/ASR/transducer/README.md) for this recipe.

### Limitations
The SpeechBrain team does not provide any warranty on the performance achieved by this model when used on other datasets.

# **About SpeechBrain**
- Website: https://speechbrain.github.io/
- Code: https://github.com/speechbrain/speechbrain/
- HuggingFace: https://huggingface.co/speechbrain/


# **Citing SpeechBrain**
Please, cite SpeechBrain if you use it for your research or business.


```bibtex
@misc{speechbrainV1,
  title={Open-Source Conversational AI with SpeechBrain 1.0},
  author={Mirco Ravanelli and Titouan Parcollet and Adel Moumen and Sylvain de Langen and Cem Subakan and Peter Plantinga and Yingzhi Wang and Pooneh Mousavi and Luca Della Libera and Artem Ploujnikov and Francesco Paissan and Davide Borra and Salah Zaiem and Zeyu Zhao and Shucong Zhang and Georgios Karakasidis and Sung-Lin Yeh and Pierre Champion and Aku Rouhe and Rudolf Braun and Florian Mai and Juan Zuluaga-Gomez and Seyed Mahed Mousavi and Andreas Nautsch and Xuechen Liu and Sangeet Sagar and Jarod Duret and Salima Mdhaffar and Gaelle Laperriere and Mickael Rouvier and Renato De Mori and Yannick Esteve},
  year={2024},
  eprint={2407.00463},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2407.00463},
}
@misc{speechbrain,
  title={{SpeechBrain}: A General-Purpose Speech Toolkit},
  author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},
  year={2021},
  eprint={2106.04624},
  archivePrefix={arXiv},
  primaryClass={eess.AS},
  note={arXiv:2106.04624}
}
```