readme updated
Browse files- README.md +25 -38
- __pycache__/app.cpython-38.pyc +0 -0
- app.py +7 -1
- formula.png +0 -0
README.md
CHANGED
@@ -1,6 +1,6 @@
|
|
1 |
---
|
2 |
title: WER
|
3 |
-
emoji:
|
4 |
colorFrom: blue
|
5 |
colorTo: red
|
6 |
sdk: gradio
|
@@ -9,44 +9,31 @@ app_file: app.py
|
|
9 |
pinned: false
|
10 |
tags:
|
11 |
- evaluate
|
12 |
-
-
|
13 |
-
|
14 |
-
|
15 |
-
speech recognition system.
|
16 |
-
|
17 |
-
The general difficulty of measuring performance lies in the fact that the
|
18 |
-
recognized word sequence can have a different length from the reference word
|
19 |
-
sequence (supposedly the correct one). The WER is derived from the Levenshtein
|
20 |
-
distance, working at the word level instead of the phoneme level. The WER is a
|
21 |
-
valuable tool for comparing different systems as well as for evaluating
|
22 |
-
improvements within one system. This kind of measurement, however, provides no
|
23 |
-
details on the nature of translation errors and further work is therefore
|
24 |
-
required to identify the main source(s) of error and to focus any research
|
25 |
-
effort.
|
26 |
-
|
27 |
-
This problem is solved by first aligning the recognized word sequence with the
|
28 |
-
reference (spoken) word sequence using dynamic string alignment. Examination
|
29 |
-
of this issue is seen through a theory called the power law that states the
|
30 |
-
correlation between perplexity and word error rate.
|
31 |
-
|
32 |
-
Word error rate can then be computed as:
|
33 |
-
|
34 |
-
WER = (S + D + I) / N = (S + D + I) / (S + D + C)
|
35 |
-
|
36 |
-
where
|
37 |
-
|
38 |
-
S is the number of substitutions, D is the number of deletions, I is the
|
39 |
-
number of insertions, C is the number of correct words, N is the number of
|
40 |
-
words in the reference (N=S+D+C).
|
41 |
-
|
42 |
-
This value indicates the average number of errors per reference word. The
|
43 |
-
lower the value, the better the performance of the ASR system with a WER of 0
|
44 |
-
being a perfect score.
|
45 |
-
duplicated_from: evaluate-metric/wer
|
46 |
---
|
47 |
|
48 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
49 |
|
50 |
-
## Metric description
|
51 |
|
52 |
-
lorem ipsum
|
|
|
1 |
---
|
2 |
title: WER
|
3 |
+
emoji: 🖩
|
4 |
colorFrom: blue
|
5 |
colorTo: red
|
6 |
sdk: gradio
|
|
|
9 |
pinned: false
|
10 |
tags:
|
11 |
- evaluate
|
12 |
+
- wer
|
13 |
+
- neuralspace
|
14 |
+
- STT
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
15 |
---
|
16 |
|
17 |
+
## About this Demo
|
18 |
+
|
19 |
+
This demo was built as a part of NeuralSpace's VoiceAI blog on [Word Error Rate 101: Your Guide to STT Vendor Evaluation](voice.neuralspace.ai).
|
20 |
+
|
21 |
+
## What is WER?
|
22 |
+
|
23 |
+
WER or Word Error Rate is a metric used primarily in the field of speech recognition to measure the performance of an automatic speech recognition (ASR) system. WER calculates the minimum number of operations (substitutions, deletions, and insertions) required to change the system's transcription (prediction) into the reference transcription (truth), divided by the number of words in the reference.
|
24 |
+
|
25 |
+

|
26 |
+
|
27 |
+
WER can range from 0 to infinite. The closer the WER is to 0, the better. WER is often also represented as a percentage. It is usually calculated by just multiplying 100 to it. For example, a WER of 0.15 might also be represented as 15%.
|
28 |
+
|
29 |
+
WER is important because it provides:
|
30 |
+
* **Performance Measure**: It gives an objective measure of how well an ASR system is transcribing speech into text.
|
31 |
+
* **Comparison**: It allows for comparison between different ASR systems or versions of a system.
|
32 |
+
|
33 |
+
|
34 |
+
## References
|
35 |
+
|
36 |
+
* Python package to calculate WER: [jiwer](https://jitsi.github.io/jiwer/)
|
37 |
+
|
38 |
|
|
|
39 |
|
|
__pycache__/app.cpython-38.pyc
ADDED
Binary file (1.48 kB). View file
|
|
app.py
CHANGED
@@ -28,7 +28,13 @@ def compute(input):
|
|
28 |
return err
|
29 |
|
30 |
|
31 |
-
description = "
|
|
|
|
|
|
|
|
|
|
|
|
|
32 |
|
33 |
demo = gr.Interface(
|
34 |
fn=compute,
|
|
|
28 |
return err
|
29 |
|
30 |
|
31 |
+
description = """
|
32 |
+
To calculate WER:
|
33 |
+
|
34 |
+
* Type the `prediction` and the `truth` in the respective columns in the below calculator.
|
35 |
+
* You can insert multiple predictions and truths by clicking on the `New row` button.
|
36 |
+
* To calculate the WER after inserting all the texts, click on `Submit`.
|
37 |
+
"""
|
38 |
|
39 |
demo = gr.Interface(
|
40 |
fn=compute,
|
formula.png
ADDED
![]() |