Update README.md
Browse files
README.md
CHANGED
@@ -1,6 +1,6 @@
|
|
1 |
---
|
2 |
license: cc-by-nc-4.0
|
3 |
-
pipeline_tag:
|
4 |
---
|
5 |
# SimulSeamless
|
6 |
![ACL Anthology](https://img.shields.io/badge/anthology-brightgreen?logo=data%3Aimage%2Fsvg%2Bxml%3Bbase64%2CPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8%2BCjwhLS0gQ3JlYXRlZCB3aXRoIElua3NjYXBlIChodHRwOi8vd3d3Lmlua3NjYXBlLm9yZy8pIC0tPgo8c3ZnCiAgIHhtbG5zOnN2Zz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciCiAgIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIKICAgdmVyc2lvbj0iMS4wIgogICB3aWR0aD0iNjgiCiAgIGhlaWdodD0iNjgiCiAgIGlkPSJzdmcyIj4KICA8ZGVmcwogICAgIGlkPSJkZWZzNCIgLz4KICA8cGF0aAogICAgIGQ9Ik0gNDEuOTc3NTUzLC0yLjg0MjE3MDllLTAxNCBDIDQxLjk3NzU1MywxLjc2MTc4IDQxLjk3NzU1MywxLjQ0MjExIDQxLjk3NzU1MywzLjAxNTggTCA3LjQ4NjkwNTQsMy4wMTU4IEwgMCwzLjAxNTggTCAwLDEwLjUwMDc5IEwgMCwzOC40Nzg2NyBMIDAsNDYgTCA3LjQ4NjkwNTQsNDYgTCA0OS41MDA4MDIsNDYgTCA1Ni45ODc3MDgsNDYgTCA2OCw0NiBMIDY4LDMwLjk5MzY4IEwgNTYuOTg3NzA4LDMwLjk5MzY4IEwgNTYuOTg3NzA4LDEwLjUwMDc5IEwgNTYuOTg3NzA4LDMuMDE1OCBDIDU2Ljk4NzcwOCwxLjQ0MjExIDU2Ljk4NzcwOCwxLjc2MTc4IDU2Ljk4NzcwOCwtMi44NDIxNzA5ZS0wMTQgTCA0MS45Nzc1NTMsLTIuODQyMTcwOWUtMDE0IHogTSAxNS4wMTAxNTUsMTcuOTg1NzggTCA0MS45Nzc1NTMsMTcuOTg1NzggTCA0MS45Nzc1NTMsMzAuOTkzNjggTCAxNS4wMTAxNTUsMzAuOTkzNjggTCAxNS4wMTAxNTUsMTcuOTg1NzggeiAiCiAgICAgc3R5bGU9ImZpbGw6I2VkMWMyNDtmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2U6bm9uZTtzdHJva2Utd2lkdGg6MTIuODk1NDExNDk7c3Ryb2tlLWxpbmVjYXA6YnV0dDtzdHJva2UtbGluZWpvaW46bWl0ZXI7c3Ryb2tlLW1pdGVybGltaXQ6NDtzdHJva2UtZGFzaGFycmF5Om5vbmU7c3Ryb2tlLWRhc2hvZmZzZXQ6MDtzdHJva2Utb3BhY2l0eToxIgogICAgIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAsIDExKSIKICAgICBpZD0icmVjdDIxNzgiIC8%2BCjwvc3ZnPgo%3D&label=ACL&labelColor=white&color=red)
|
@@ -16,7 +16,7 @@ In the case of [💬 Inference using docker](#💬-inference-using-docker), use
|
|
16 |
`f1f5b9a69a47496630aa43605f1bd46e5484a2f4` for SimulEval.
|
17 |
|
18 |
## 🤖 Inference using your environment
|
19 |
-
|
20 |
[Fairseq Simultaneous Translation repository](https://github.com/facebookresearch/fairseq/blob/main/examples/speech_to_text/docs/simulst_mustc_example.md#inference--evaluation):
|
21 |
`${LIST_OF_AUDIO}` is the list of audio paths and `${TGT_FILE}` the segment-wise references in the
|
22 |
target language.
|
@@ -30,7 +30,7 @@ tokenizer used, for example, to evaluate German) or `char` (e.g., to evaluate ch
|
|
30 |
languages such as Chinese or Japanese).
|
31 |
|
32 |
The simultaneous inference of SimulSeamless is based on
|
33 |
-
[AlignAtt](ALIGNATT_SIMULST_AGENT_INTERSPEECH2023.md), thus the __f__ parameter (`${FRAME}`) and the
|
34 |
layer from which to extract the attention scores (`${LAYER}`) have to be set accordingly.
|
35 |
|
36 |
### Instruction to replicate IWSLT 2024 results ↙️
|
@@ -97,41 +97,6 @@ To set, `${TGT_LANG}`, `${FRAME}`, `${LAYER}`, `${BLEU_TOKENIZER}`, `${LATENCY_U
|
|
97 |
`${LIST_OF_AUDIO}`, `${TGT_FILE}`, `${SEG_SIZE}`, and `${OUT_DIR}` refer to
|
98 |
[🤖 Inference using your environment](#-inference-using-your-environment).
|
99 |
|
100 |
-
### Instruction to recreate the docker images
|
101 |
-
|
102 |
-
To recreate the docker images, follow the steps below.
|
103 |
-
|
104 |
-
1. Download SimulEval and this repository.
|
105 |
-
2. Create a `Dockerfile` with the following content:
|
106 |
-
```
|
107 |
-
FROM python:3.9
|
108 |
-
RUN pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113
|
109 |
-
ADD /SimulEval /SimulEval
|
110 |
-
WORKDIR /SimulEval
|
111 |
-
RUN pip install -e .
|
112 |
-
WORKDIR ../
|
113 |
-
ADD /fbk-fairseq /fbk-fairseq
|
114 |
-
WORKDIR /fbk-fairseq
|
115 |
-
RUN pip install -e .
|
116 |
-
RUN pip install -r speech_requirements.txt
|
117 |
-
WORKDIR ../
|
118 |
-
RUN pip install sentencepiece
|
119 |
-
RUN pip install transformers
|
120 |
-
|
121 |
-
ENTRYPOINT simuleval --standalone --remote-port 2024 \
|
122 |
-
--agent-class examples.speech_to_text.simultaneous_translation.agents.v1_1.simul_alignatt_seamlessm4t.AlignAttSeamlessS2T \
|
123 |
-
--model-size medium --num-beams 5 --user-dir fbk-fairseq/examples \
|
124 |
-
--target-language $TGTLANG --frame-num $FRAME --extract-attn-from-layer $LAYER --device $DEV \
|
125 |
-
--sacrebleu-tokenizer ${BLEU_TOKENIZER} --eval-latency-unit ${LATENCY_UNIT}
|
126 |
-
```
|
127 |
-
3. Build the docker image:
|
128 |
-
```
|
129 |
-
docker build -t simulseamless .
|
130 |
-
```
|
131 |
-
4. Save the docker image:
|
132 |
-
```
|
133 |
-
docker save -o simulseamless.tar simulseamless:latest
|
134 |
-
```
|
135 |
|
136 |
## 📍Citation
|
137 |
```bibtex
|
|
|
1 |
---
|
2 |
license: cc-by-nc-4.0
|
3 |
+
pipeline_tag: automatic-speech-recognition
|
4 |
---
|
5 |
# SimulSeamless
|
6 |
![ACL Anthology](https://img.shields.io/badge/anthology-brightgreen?logo=data%3Aimage%2Fsvg%2Bxml%3Bbase64%2CPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8%2BCjwhLS0gQ3JlYXRlZCB3aXRoIElua3NjYXBlIChodHRwOi8vd3d3Lmlua3NjYXBlLm9yZy8pIC0tPgo8c3ZnCiAgIHhtbG5zOnN2Zz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciCiAgIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIKICAgdmVyc2lvbj0iMS4wIgogICB3aWR0aD0iNjgiCiAgIGhlaWdodD0iNjgiCiAgIGlkPSJzdmcyIj4KICA8ZGVmcwogICAgIGlkPSJkZWZzNCIgLz4KICA8cGF0aAogICAgIGQ9Ik0gNDEuOTc3NTUzLC0yLjg0MjE3MDllLTAxNCBDIDQxLjk3NzU1MywxLjc2MTc4IDQxLjk3NzU1MywxLjQ0MjExIDQxLjk3NzU1MywzLjAxNTggTCA3LjQ4NjkwNTQsMy4wMTU4IEwgMCwzLjAxNTggTCAwLDEwLjUwMDc5IEwgMCwzOC40Nzg2NyBMIDAsNDYgTCA3LjQ4NjkwNTQsNDYgTCA0OS41MDA4MDIsNDYgTCA1Ni45ODc3MDgsNDYgTCA2OCw0NiBMIDY4LDMwLjk5MzY4IEwgNTYuOTg3NzA4LDMwLjk5MzY4IEwgNTYuOTg3NzA4LDEwLjUwMDc5IEwgNTYuOTg3NzA4LDMuMDE1OCBDIDU2Ljk4NzcwOCwxLjQ0MjExIDU2Ljk4NzcwOCwxLjc2MTc4IDU2Ljk4NzcwOCwtMi44NDIxNzA5ZS0wMTQgTCA0MS45Nzc1NTMsLTIuODQyMTcwOWUtMDE0IHogTSAxNS4wMTAxNTUsMTcuOTg1NzggTCA0MS45Nzc1NTMsMTcuOTg1NzggTCA0MS45Nzc1NTMsMzAuOTkzNjggTCAxNS4wMTAxNTUsMzAuOTkzNjggTCAxNS4wMTAxNTUsMTcuOTg1NzggeiAiCiAgICAgc3R5bGU9ImZpbGw6I2VkMWMyNDtmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2U6bm9uZTtzdHJva2Utd2lkdGg6MTIuODk1NDExNDk7c3Ryb2tlLWxpbmVjYXA6YnV0dDtzdHJva2UtbGluZWpvaW46bWl0ZXI7c3Ryb2tlLW1pdGVybGltaXQ6NDtzdHJva2UtZGFzaGFycmF5Om5vbmU7c3Ryb2tlLWRhc2hvZmZzZXQ6MDtzdHJva2Utb3BhY2l0eToxIgogICAgIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAsIDExKSIKICAgICBpZD0icmVjdDIxNzgiIC8%2BCjwvc3ZnPgo%3D&label=ACL&labelColor=white&color=red)
|
|
|
16 |
`f1f5b9a69a47496630aa43605f1bd46e5484a2f4` for SimulEval.
|
17 |
|
18 |
## 🤖 Inference using your environment
|
19 |
+
Set `--source`, and `--target` as described in the
|
20 |
[Fairseq Simultaneous Translation repository](https://github.com/facebookresearch/fairseq/blob/main/examples/speech_to_text/docs/simulst_mustc_example.md#inference--evaluation):
|
21 |
`${LIST_OF_AUDIO}` is the list of audio paths and `${TGT_FILE}` the segment-wise references in the
|
22 |
target language.
|
|
|
30 |
languages such as Chinese or Japanese).
|
31 |
|
32 |
The simultaneous inference of SimulSeamless is based on
|
33 |
+
[AlignAtt](https://github.com/hlt-mt/FBK-fairseq/blob/master/fbk_works/ALIGNATT_SIMULST_AGENT_INTERSPEECH2023.md), thus the __f__ parameter (`${FRAME}`) and the
|
34 |
layer from which to extract the attention scores (`${LAYER}`) have to be set accordingly.
|
35 |
|
36 |
### Instruction to replicate IWSLT 2024 results ↙️
|
|
|
97 |
`${LIST_OF_AUDIO}`, `${TGT_FILE}`, `${SEG_SIZE}`, and `${OUT_DIR}` refer to
|
98 |
[🤖 Inference using your environment](#-inference-using-your-environment).
|
99 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
100 |
|
101 |
## 📍Citation
|
102 |
```bibtex
|