2022-04-05 14:25:26,074 - speechbrain.core - INFO - Beginning experiment!
2022-04-05 14:25:26,075 - speechbrain.core - INFO - Experiment folder: results/wav2vec2_ctc_AMHARIC/1249
2022-04-05 14:25:26,762 - speechbrain.utils.superpowers - DEBUG - aiohttp==3.8.1
aiosignal==1.2.0
appdirs==1.4.4
async-timeout==4.0.2
attrs==21.4.0
audioread==2.1.9
azure-core==1.21.1
azure-storage-blob==12.9.0
bcrypt==3.2.0
black==19.10b0
certifi==2021.10.8
cffi==1.15.0
cfgv==3.3.1
charset-normalizer==2.0.10
click==8.0.3
cryptography==36.0.1
datasets==1.13.3
decorator==5.1.1
dill==0.3.4
distlib==0.3.4
entrypoints==0.3
ffmpeg==1.4
filelock==3.4.2
flake8==3.7.9
frozenlist==1.3.0
fsspec==2022.2.0
huggingface-hub==0.2.1
HyperPyYAML==1.0.0
identify==2.4.4
idna==3.3
isodate==0.6.1
joblib==1.1.0
librosa==0.8.1
llvmlite==0.38.0
mccabe==0.6.1
more-itertools==8.12.0
msrest==0.6.21
multidict==6.0.2
multiprocess==0.70.12.2
mutagen==1.45.1
nodeenv==1.6.0
numba==0.55.0
numpy==1.21.5
oauthlib==3.1.1
packaging==21.3
pandas==1.3.5
paramiko==2.10.3
pathspec==0.9.0
platformdirs==2.4.1
pluggy==0.13.1
pooch==1.5.2
pre-commit==2.17.0
py==1.11.0
pyarrow==7.0.0
pycodestyle==2.5.0
pycparser==2.21
pyflakes==2.1.1
PyNaCl==1.5.0
pyparsing==3.0.6
pytest==5.4.1
python-dateutil==2.8.2
pytz==2021.3
PyYAML==6.0
regex==2022.1.18
requests==2.27.1
requests-oauthlib==1.3.0
resampy==0.2.2
ruamel.yaml==0.17.20
ruamel.yaml.clib==0.2.6
sacremoses==0.0.47
scikit-learn==1.0.2
scipy==1.7.3
scp==0.14.4
sentencepiece==0.1.96
six==1.16.0
SoundFile==0.10.3.post1
threadpoolctl==3.0.0
tokenizers==0.10.3
toml==0.10.2
torch==1.10.1
torchaudio==0.10.1
tqdm==4.62.3
transformers==4.13.0
typed-ast==1.5.1
typing_extensions==4.0.1
urllib3==1.26.8
virtualenv==20.13.0
wcwidth==0.2.5
xxhash==3.0.0
yamllint==1.23.0
yarl==1.7.2


2022-04-05 14:25:26,922 - dvoice_prepare - INFO - Preparing CSV files for 7612 samples ...
2022-04-05 14:25:26,922 - dvoice_prepare - INFO - Creating csv lists in results/wav2vec2_ctc_AMHARIC/1249/save/train.csv ...
2022-04-05 14:25:27,322 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/train.csv successfully created!
2022-04-05 14:25:27,322 - dvoice_prepare - INFO - Number of samples: 7612 
2022-04-05 14:25:27,322 - dvoice_prepare - INFO - Total duration: 14.05 Hours
2022-04-05 14:25:27,382 - dvoice_prepare - INFO - Preparing CSV files for 1631 samples ...
2022-04-05 14:25:27,382 - dvoice_prepare - INFO - Creating csv lists in results/wav2vec2_ctc_AMHARIC/1249/save/dev.csv ...
2022-04-05 14:25:27,434 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/dev.csv successfully created!
2022-04-05 14:25:27,435 - dvoice_prepare - INFO - Number of samples: 1631 
2022-04-05 14:25:27,435 - dvoice_prepare - INFO - Total duration: 2.95 Hours
2022-04-05 14:25:27,477 - dvoice_prepare - INFO - Preparing CSV files for 1632 samples ...
2022-04-05 14:25:27,477 - dvoice_prepare - INFO - Creating csv lists in results/wav2vec2_ctc_AMHARIC/1249/save/test.csv ...
2022-04-05 14:25:27,546 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/test.csv successfully created!
2022-04-05 14:25:27,546 - dvoice_prepare - INFO - Number of samples: 1632 
2022-04-05 14:25:27,546 - dvoice_prepare - INFO - Total duration: 3.03 Hours
2022-04-05 14:25:27,552 - speechbrain.tokenizers.SentencePiece - INFO - Train tokenizer with type:char
2022-04-05 14:25:27,557 - speechbrain.tokenizers.SentencePiece - INFO - Extract wrd sequences from:results/wav2vec2_ctc_AMHARIC/1249/save/train.csv
2022-04-05 14:25:27,707 - speechbrain.tokenizers.SentencePiece - INFO - Text file created at: results/wav2vec2_ctc_AMHARIC/1249/save/train.txt
2022-04-05 14:25:27,870 - speechbrain.tokenizers.SentencePiece - INFO - ==== Loading Tokenizer ===
2022-04-05 14:25:27,870 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer path: results/wav2vec2_ctc_AMHARIC/1249/save/224_char.model
2022-04-05 14:25:27,870 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer vocab_size: 224
2022-04-05 14:25:27,870 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer type: char
2022-04-05 14:25:27,990 - speechbrain.core - INFO - Info: auto_mix_prec arg from hparam file is used
2022-04-05 14:25:27,991 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used
2022-04-05 14:25:30,936 - speechbrain.core - INFO - 318.8M trainable parameters in ASR
2022-04-05 14:25:30,940 - speechbrain.utils.checkpoints - INFO - Would load a checkpoint here, but none found yet.
2022-04-05 14:25:30,940 - speechbrain.utils.epoch_loop - INFO - Going into epoch 1
2022-04-05 14:37:08,995 - speechbrain.core - ERROR - Exception:
Traceback (most recent call last):
  File "/data/n.abdoumohamed/DVoice/speechbrain/recipes/DVoice/ASR/CTC/train2.py", line 366, in <module>
    asr_brain.fit(
  File "/data/n.abdoumohamed/DVoice/speechbrain/speechbrain/core.py", line 1034, in fit
    loss = self.fit_batch(batch)
  File "/data/n.abdoumohamed/DVoice/speechbrain/recipes/DVoice/ASR/CTC/train2.py", line 120, in fit_batch
    outputs = self.compute_forward(batch, sb.Stage.TRAIN)
  File "/data/n.abdoumohamed/DVoice/speechbrain/recipes/DVoice/ASR/CTC/train2.py", line 61, in compute_forward
    feats = self.modules.wav2vec2(wavs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data/n.abdoumohamed/DVoice/speechbrain/speechbrain/lobes/models/huggingface_wav2vec.py", line 254, in forward
    return self.extract_features(wav)
  File "/data/n.abdoumohamed/DVoice/speechbrain/speechbrain/lobes/models/huggingface_wav2vec.py", line 269, in extract_features
    out = self.model(wav)[0]
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1253, in forward
    encoder_outputs = self.encoder(
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 846, in forward
    layer_outputs = layer(
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 687, in forward
    hidden_states = hidden_states + self.feed_forward(self.final_layer_norm(hidden_states))
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 622, in forward
    hidden_states = self.intermediate_dense(hidden_states)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 103, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/functional.py", line 1848, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 11.91 GiB total capacity; 10.77 GiB already allocated; 25.25 MiB free; 10.94 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-05 21:06:45,197 - speechbrain.core - INFO - Beginning experiment!
2022-04-05 21:06:45,198 - speechbrain.core - INFO - Experiment folder: results/wav2vec2_ctc_AMHARIC/1249
2022-04-05 21:06:45,836 - speechbrain.utils.superpowers - DEBUG - aiohttp==3.8.1
aiosignal==1.2.0
appdirs==1.4.4
async-timeout==4.0.2
attrs==21.4.0
audioread==2.1.9
azure-core==1.21.1
azure-storage-blob==12.9.0
bcrypt==3.2.0
black==19.10b0
certifi==2021.10.8
cffi==1.15.0
cfgv==3.3.1
charset-normalizer==2.0.10
click==8.0.3
cryptography==36.0.1
datasets==1.13.3
decorator==5.1.1
dill==0.3.4
distlib==0.3.4
entrypoints==0.3
ffmpeg==1.4
filelock==3.4.2
flake8==3.7.9
frozenlist==1.3.0
fsspec==2022.2.0
huggingface-hub==0.2.1
HyperPyYAML==1.0.0
identify==2.4.4
idna==3.3
isodate==0.6.1
joblib==1.1.0
librosa==0.8.1
llvmlite==0.38.0
mccabe==0.6.1
more-itertools==8.12.0
msrest==0.6.21
multidict==6.0.2
multiprocess==0.70.12.2
mutagen==1.45.1
nodeenv==1.6.0
numba==0.55.0
numpy==1.21.5
oauthlib==3.1.1
packaging==21.3
pandas==1.3.5
paramiko==2.10.3
pathspec==0.9.0
platformdirs==2.4.1
pluggy==0.13.1
pooch==1.5.2
pre-commit==2.17.0
py==1.11.0
pyarrow==7.0.0
pycodestyle==2.5.0
pycparser==2.21
pyflakes==2.1.1
PyNaCl==1.5.0
pyparsing==3.0.6
pytest==5.4.1
python-dateutil==2.8.2
pytz==2021.3
PyYAML==6.0
regex==2022.1.18
requests==2.27.1
requests-oauthlib==1.3.0
resampy==0.2.2
ruamel.yaml==0.17.20
ruamel.yaml.clib==0.2.6
sacremoses==0.0.47
scikit-learn==1.0.2
scipy==1.7.3
scp==0.14.4
sentencepiece==0.1.96
six==1.16.0
SoundFile==0.10.3.post1
threadpoolctl==3.0.0
tokenizers==0.10.3
toml==0.10.2
torch==1.10.1
torchaudio==0.10.1
tqdm==4.62.3
transformers==4.13.0
typed-ast==1.5.1
typing_extensions==4.0.1
urllib3==1.26.8
virtualenv==20.13.0
wcwidth==0.2.5
xxhash==3.0.0
yamllint==1.23.0
yarl==1.7.2


2022-04-05 21:06:46,007 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/train.csv already exists, skipping data preparation!
2022-04-05 21:06:46,007 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/dev.csv already exists, skipping data preparation!
2022-04-05 21:06:46,007 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/test.csv already exists, skipping data preparation!
2022-04-05 21:06:46,007 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer is already trained.
2022-04-05 21:06:46,007 - speechbrain.tokenizers.SentencePiece - INFO - ==== Loading Tokenizer ===
2022-04-05 21:06:46,007 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer path: results/wav2vec2_ctc_AMHARIC/1249/save/224_char.model
2022-04-05 21:06:46,007 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer vocab_size: 224
2022-04-05 21:06:46,007 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer type: char
2022-04-05 21:06:46,145 - speechbrain.core - INFO - Info: auto_mix_prec arg from hparam file is used
2022-04-05 21:06:46,145 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used
2022-04-05 21:06:49,137 - speechbrain.core - INFO - 318.8M trainable parameters in ASR
2022-04-05 21:06:49,142 - speechbrain.utils.checkpoints - INFO - Would load a checkpoint here, but none found yet.
2022-04-05 21:06:49,142 - speechbrain.utils.epoch_loop - INFO - Going into epoch 1
2022-04-05 21:18:09,453 - speechbrain.core - ERROR - Exception:
Traceback (most recent call last):
  File "/data/n.abdoumohamed/DVoice/speechbrain/recipes/DVoice/ASR/CTC/train2.py", line 366, in <module>
    asr_brain.fit(
  File "/data/n.abdoumohamed/DVoice/speechbrain/speechbrain/core.py", line 1034, in fit
    loss = self.fit_batch(batch)
  File "/data/n.abdoumohamed/DVoice/speechbrain/recipes/DVoice/ASR/CTC/train2.py", line 120, in fit_batch
    outputs = self.compute_forward(batch, sb.Stage.TRAIN)
  File "/data/n.abdoumohamed/DVoice/speechbrain/recipes/DVoice/ASR/CTC/train2.py", line 61, in compute_forward
    feats = self.modules.wav2vec2(wavs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data/n.abdoumohamed/DVoice/speechbrain/speechbrain/lobes/models/huggingface_wav2vec.py", line 254, in forward
    return self.extract_features(wav)
  File "/data/n.abdoumohamed/DVoice/speechbrain/speechbrain/lobes/models/huggingface_wav2vec.py", line 269, in extract_features
    out = self.model(wav)[0]
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1253, in forward
    encoder_outputs = self.encoder(
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 846, in forward
    layer_outputs = layer(
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 687, in forward
    hidden_states = hidden_states + self.feed_forward(self.final_layer_norm(hidden_states))
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 626, in forward
    hidden_states = self.output_dense(hidden_states)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 103, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/functional.py", line 1848, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 11.91 GiB total capacity; 10.65 GiB already allocated; 17.25 MiB free; 10.95 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-06 10:21:42,642 - speechbrain.core - INFO - Beginning experiment!
2022-04-06 10:21:42,643 - speechbrain.core - INFO - Experiment folder: results/wav2vec2_ctc_AMHARIC/1249
2022-04-06 10:21:43,318 - speechbrain.utils.superpowers - DEBUG - aiohttp==3.8.1
aiosignal==1.2.0
appdirs==1.4.4
async-timeout==4.0.2
attrs==21.4.0
audioread==2.1.9
azure-core==1.21.1
azure-storage-blob==12.9.0
bcrypt==3.2.0
black==19.10b0
certifi==2021.10.8
cffi==1.15.0
cfgv==3.3.1
charset-normalizer==2.0.10
click==8.0.3
cryptography==36.0.1
datasets==1.13.3
decorator==5.1.1
dill==0.3.4
distlib==0.3.4
entrypoints==0.3
ffmpeg==1.4
filelock==3.4.2
flake8==3.7.9
frozenlist==1.3.0
fsspec==2022.2.0
huggingface-hub==0.2.1
HyperPyYAML==1.0.0
identify==2.4.4
idna==3.3
isodate==0.6.1
joblib==1.1.0
librosa==0.8.1
llvmlite==0.38.0
mccabe==0.6.1
more-itertools==8.12.0
msrest==0.6.21
multidict==6.0.2
multiprocess==0.70.12.2
mutagen==1.45.1
nodeenv==1.6.0
numba==0.55.0
numpy==1.21.5
oauthlib==3.1.1
packaging==21.3
pandas==1.3.5
paramiko==2.10.3
pathspec==0.9.0
platformdirs==2.4.1
pluggy==0.13.1
pooch==1.5.2
pre-commit==2.17.0
py==1.11.0
pyarrow==7.0.0
pycodestyle==2.5.0
pycparser==2.21
pyflakes==2.1.1
PyNaCl==1.5.0
pyparsing==3.0.6
pytest==5.4.1
python-dateutil==2.8.2
pytz==2021.3
PyYAML==6.0
regex==2022.1.18
requests==2.27.1
requests-oauthlib==1.3.0
resampy==0.2.2
ruamel.yaml==0.17.20
ruamel.yaml.clib==0.2.6
sacremoses==0.0.47
scikit-learn==1.0.2
scipy==1.7.3
scp==0.14.4
sentencepiece==0.1.96
six==1.16.0
SoundFile==0.10.3.post1
threadpoolctl==3.0.0
tokenizers==0.10.3
toml==0.10.2
torch==1.10.1
torchaudio==0.10.1
tqdm==4.62.3
transformers==4.13.0
typed-ast==1.5.1
typing_extensions==4.0.1
urllib3==1.26.8
virtualenv==20.13.0
wcwidth==0.2.5
xxhash==3.0.0
yamllint==1.23.0
yarl==1.7.2


2022-04-06 10:21:43,439 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/train.csv already exists, skipping data preparation!
2022-04-06 10:21:43,439 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/dev.csv already exists, skipping data preparation!
2022-04-06 10:21:43,439 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/test.csv already exists, skipping data preparation!
2022-04-06 10:21:43,439 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer is already trained.
2022-04-06 10:21:43,439 - speechbrain.tokenizers.SentencePiece - INFO - ==== Loading Tokenizer ===
2022-04-06 10:21:43,439 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer path: results/wav2vec2_ctc_AMHARIC/1249/save/224_char.model
2022-04-06 10:21:43,440 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer vocab_size: 224
2022-04-06 10:21:43,440 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer type: char
2022-04-06 10:21:43,580 - speechbrain.core - INFO - Info: auto_mix_prec arg from hparam file is used
2022-04-06 10:21:43,580 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used
2022-04-06 10:21:46,557 - speechbrain.core - INFO - 318.8M trainable parameters in ASR
2022-04-06 10:21:46,562 - speechbrain.utils.checkpoints - INFO - Would load a checkpoint here, but none found yet.
2022-04-06 10:21:46,562 - speechbrain.utils.epoch_loop - INFO - Going into epoch 1
2022-04-06 10:45:30,095 - speechbrain.core - ERROR - Exception:
Traceback (most recent call last):
  File "/data/n.abdoumohamed/DVoice/speechbrain/recipes/DVoice/ASR/CTC/train2.py", line 366, in <module>
    asr_brain.fit(
  File "/data/n.abdoumohamed/DVoice/speechbrain/speechbrain/core.py", line 1034, in fit
    loss = self.fit_batch(batch)
  File "/data/n.abdoumohamed/DVoice/speechbrain/recipes/DVoice/ASR/CTC/train2.py", line 120, in fit_batch
    outputs = self.compute_forward(batch, sb.Stage.TRAIN)
  File "/data/n.abdoumohamed/DVoice/speechbrain/recipes/DVoice/ASR/CTC/train2.py", line 61, in compute_forward
    feats = self.modules.wav2vec2(wavs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data/n.abdoumohamed/DVoice/speechbrain/speechbrain/lobes/models/huggingface_wav2vec.py", line 254, in forward
    return self.extract_features(wav)
  File "/data/n.abdoumohamed/DVoice/speechbrain/speechbrain/lobes/models/huggingface_wav2vec.py", line 269, in extract_features
    out = self.model(wav)[0]
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1253, in forward
    encoder_outputs = self.encoder(
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 846, in forward
    layer_outputs = layer(
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 682, in forward
    hidden_states, attn_weights, _ = self.attention(
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 586, in forward
    attn_probs = nn.functional.dropout(attn_weights, p=self.dropout, training=self.training)
  File "/home/n.abdoumohamed/.conda/envs/speechbrain/lib/python3.9/site-packages/torch/nn/functional.py", line 1169, in dropout
    return _VF.dropout_(input, p, training) if inplace else _VF.dropout(input, p, training)
RuntimeError: CUDA out of memory. Tried to allocate 46.00 MiB (GPU 0; 11.91 GiB total capacity; 10.61 GiB already allocated; 5.25 MiB free; 10.96 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-07 01:24:51,494 - speechbrain.core - INFO - Beginning experiment!
2022-04-07 01:24:51,498 - speechbrain.core - INFO - Experiment folder: results/wav2vec2_ctc_AMHARIC/1249
2022-04-07 01:24:52,086 - speechbrain.utils.superpowers - DEBUG - aiohttp==3.8.1
aiosignal==1.2.0
appdirs==1.4.4
async-timeout==4.0.2
attrs==21.4.0
audioread==2.1.9
azure-core==1.21.1
azure-storage-blob==12.9.0
bcrypt==3.2.0
black==19.10b0
certifi==2021.10.8
cffi==1.15.0
cfgv==3.3.1
charset-normalizer==2.0.10
click==8.0.3
cryptography==36.0.1
datasets==1.13.3
decorator==5.1.1
dill==0.3.4
distlib==0.3.4
entrypoints==0.3
ffmpeg==1.4
filelock==3.4.2
flake8==3.7.9
frozenlist==1.3.0
fsspec==2022.2.0
huggingface-hub==0.2.1
HyperPyYAML==1.0.0
identify==2.4.4
idna==3.3
isodate==0.6.1
joblib==1.1.0
librosa==0.8.1
llvmlite==0.38.0
mccabe==0.6.1
more-itertools==8.12.0
msrest==0.6.21
multidict==6.0.2
multiprocess==0.70.12.2
mutagen==1.45.1
nodeenv==1.6.0
numba==0.55.0
numpy==1.21.5
oauthlib==3.1.1
packaging==21.3
pandas==1.3.5
paramiko==2.10.3
pathspec==0.9.0
platformdirs==2.4.1
pluggy==0.13.1
pooch==1.5.2
pre-commit==2.17.0
py==1.11.0
pyarrow==7.0.0
pycodestyle==2.5.0
pycparser==2.21
pyflakes==2.1.1
PyNaCl==1.5.0
pyparsing==3.0.6
pytest==5.4.1
python-dateutil==2.8.2
pytz==2021.3
PyYAML==6.0
regex==2022.1.18
requests==2.27.1
requests-oauthlib==1.3.0
resampy==0.2.2
ruamel.yaml==0.17.20
ruamel.yaml.clib==0.2.6
sacremoses==0.0.47
scikit-learn==1.0.2
scipy==1.7.3
scp==0.14.4
sentencepiece==0.1.96
six==1.16.0
SoundFile==0.10.3.post1
threadpoolctl==3.0.0
tokenizers==0.10.3
toml==0.10.2
torch==1.10.1
torchaudio==0.10.1
tqdm==4.62.3
transformers==4.13.0
typed-ast==1.5.1
typing_extensions==4.0.1
urllib3==1.26.8
virtualenv==20.13.0
wcwidth==0.2.5
xxhash==3.0.0
yamllint==1.23.0
yarl==1.7.2


2022-04-07 01:24:52,319 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/train.csv already exists, skipping data preparation!
2022-04-07 01:24:52,319 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/dev.csv already exists, skipping data preparation!
2022-04-07 01:24:52,319 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/test.csv already exists, skipping data preparation!
2022-04-07 01:24:52,323 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer is already trained.
2022-04-07 01:24:52,323 - speechbrain.tokenizers.SentencePiece - INFO - ==== Loading Tokenizer ===
2022-04-07 01:24:52,323 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer path: results/wav2vec2_ctc_AMHARIC/1249/save/224_char.model
2022-04-07 01:24:52,323 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer vocab_size: 224
2022-04-07 01:24:52,323 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer type: char
2022-04-07 01:24:52,988 - speechbrain.core - INFO - Info: auto_mix_prec arg from hparam file is used
2022-04-07 01:24:52,988 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used
2022-04-07 01:24:56,199 - speechbrain.core - INFO - 318.8M trainable parameters in ASR
2022-04-07 01:24:56,215 - speechbrain.utils.checkpoints - INFO - Would load a checkpoint here, but none found yet.
2022-04-07 01:24:56,215 - speechbrain.utils.epoch_loop - INFO - Going into epoch 1
2022-04-07 01:39:30,997 - speechbrain.utils.train_logger - INFO - epoch: 1, lr_model: 1.00e+00, lr_wav2vec: 1.00e-04 - train loss: 2.16 - valid loss: 6.31e-01, valid CER: 17.94, valid WER: 61.08
2022-04-07 01:41:12,908 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+01-39-31+00
2022-04-07 01:41:13,081 - speechbrain.utils.epoch_loop - INFO - Going into epoch 2
2022-04-07 01:54:26,673 - speechbrain.utils.train_logger - INFO - epoch: 2, lr_model: 1.00e+00, lr_wav2vec: 1.00e-04 - train loss: 7.60e-01 - valid loss: 4.76e-01, valid CER: 13.59, valid WER: 49.70
2022-04-07 01:56:27,270 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+01-54-26+00
2022-04-07 01:56:28,698 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+01-39-31+00
2022-04-07 01:56:28,698 - speechbrain.utils.epoch_loop - INFO - Going into epoch 3
2022-04-07 02:09:43,082 - speechbrain.utils.train_logger - INFO - epoch: 3, lr_model: 1.00e+00, lr_wav2vec: 1.00e-04 - train loss: 5.88e-01 - valid loss: 4.04e-01, valid CER: 11.77, valid WER: 43.35
2022-04-07 02:14:21,215 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+02-09-43+00
2022-04-07 02:14:22,071 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+01-54-26+00
2022-04-07 02:14:22,071 - speechbrain.utils.epoch_loop - INFO - Going into epoch 4
2022-04-07 02:27:35,021 - speechbrain.utils.train_logger - INFO - epoch: 4, lr_model: 1.00e+00, lr_wav2vec: 1.00e-04 - train loss: 4.97e-01 - valid loss: 3.75e-01, valid CER: 10.61, valid WER: 39.29
2022-04-07 02:28:59,128 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+02-27-35+00
2022-04-07 02:29:00,277 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+02-09-43+00
2022-04-07 02:29:00,277 - speechbrain.utils.epoch_loop - INFO - Going into epoch 5
2022-04-07 02:42:13,500 - speechbrain.utils.train_logger - INFO - epoch: 5, lr_model: 1.00e+00, lr_wav2vec: 1.00e-04 - train loss: 4.29e-01 - valid loss: 3.71e-01, valid CER: 10.15, valid WER: 37.38
2022-04-07 02:47:20,435 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+02-42-13+00
2022-04-07 02:47:21,615 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+02-27-35+00
2022-04-07 02:47:21,615 - speechbrain.utils.epoch_loop - INFO - Going into epoch 6
2022-04-07 03:00:32,602 - speechbrain.utils.train_logger - INFO - epoch: 6, lr_model: 1.00e+00, lr_wav2vec: 1.00e-04 - train loss: 3.81e-01 - valid loss: 3.54e-01, valid CER: 9.54, valid WER: 35.23
2022-04-07 03:02:35,290 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+03-00-32+00
2022-04-07 03:02:35,924 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+02-42-13+00
2022-04-07 03:02:35,925 - speechbrain.utils.epoch_loop - INFO - Going into epoch 7
2022-04-07 03:15:54,749 - speechbrain.utils.train_logger - INFO - epoch: 7, lr_model: 1.00e+00, lr_wav2vec: 1.00e-04 - train loss: 3.39e-01 - valid loss: 3.41e-01, valid CER: 8.98, valid WER: 33.48
2022-04-07 03:17:35,622 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+03-15-54+00
2022-04-07 03:17:36,628 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+03-00-32+00
2022-04-07 03:17:36,628 - speechbrain.utils.epoch_loop - INFO - Going into epoch 8
2022-04-07 03:30:53,354 - speechbrain.nnet.schedulers - INFO - Changing lr from 1 to 0.8
2022-04-07 03:30:53,376 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.0001 to 9e-05
2022-04-07 03:30:53,520 - speechbrain.utils.train_logger - INFO - epoch: 8, lr_model: 1.00e+00, lr_wav2vec: 1.00e-04 - train loss: 3.08e-01 - valid loss: 3.57e-01, valid CER: 8.80, valid WER: 32.41
2022-04-07 03:33:47,026 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+03-30-53+00
2022-04-07 03:33:48,209 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+03-15-54+00
2022-04-07 03:33:48,209 - speechbrain.utils.epoch_loop - INFO - Going into epoch 9
2022-04-07 03:46:59,930 - speechbrain.utils.train_logger - INFO - epoch: 9, lr_model: 8.00e-01, lr_wav2vec: 9.00e-05 - train loss: 2.70e-01 - valid loss: 3.46e-01, valid CER: 8.47, valid WER: 31.33
2022-04-07 03:48:26,079 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+03-46-59+00
2022-04-07 03:48:26,901 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+03-30-53+00
2022-04-07 03:48:26,901 - speechbrain.utils.epoch_loop - INFO - Going into epoch 10
2022-04-07 04:01:43,511 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.8 to 0.64
2022-04-07 04:01:43,527 - speechbrain.nnet.schedulers - INFO - Changing lr from 9e-05 to 8.1e-05
2022-04-07 04:01:43,843 - speechbrain.utils.train_logger - INFO - epoch: 10, lr_model: 8.00e-01, lr_wav2vec: 9.00e-05 - train loss: 2.45e-01 - valid loss: 3.64e-01, valid CER: 8.30, valid WER: 30.31
2022-04-07 04:03:02,059 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+04-01-43+00
2022-04-07 04:03:02,966 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+03-46-59+00
2022-04-07 04:03:02,966 - speechbrain.utils.epoch_loop - INFO - Going into epoch 11
2022-04-07 04:16:16,977 - speechbrain.utils.train_logger - INFO - epoch: 11, lr_model: 6.40e-01, lr_wav2vec: 8.10e-05 - train loss: 2.17e-01 - valid loss: 3.43e-01, valid CER: 8.00, valid WER: 29.91
2022-04-07 04:17:41,125 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+04-16-16+00
2022-04-07 04:17:42,305 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+04-01-43+00
2022-04-07 04:17:42,305 - speechbrain.utils.epoch_loop - INFO - Going into epoch 12
2022-04-07 04:30:56,259 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.64 to 0.51
2022-04-07 04:30:56,276 - speechbrain.nnet.schedulers - INFO - Changing lr from 8.1e-05 to 7.3e-05
2022-04-07 04:30:56,372 - speechbrain.utils.train_logger - INFO - epoch: 12, lr_model: 6.40e-01, lr_wav2vec: 8.10e-05 - train loss: 1.98e-01 - valid loss: 3.68e-01, valid CER: 7.93, valid WER: 29.49
2022-04-07 04:33:36,676 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+04-30-56+00
2022-04-07 04:33:38,939 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+04-16-16+00
2022-04-07 04:33:38,939 - speechbrain.utils.epoch_loop - INFO - Going into epoch 13
2022-04-07 04:46:50,071 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.51 to 0.41
2022-04-07 04:46:50,283 - speechbrain.nnet.schedulers - INFO - Changing lr from 7.3e-05 to 6.6e-05
2022-04-07 04:46:50,494 - speechbrain.utils.train_logger - INFO - epoch: 13, lr_model: 5.12e-01, lr_wav2vec: 7.29e-05 - train loss: 1.75e-01 - valid loss: 3.94e-01, valid CER: 7.78, valid WER: 29.09
2022-04-07 04:49:19,543 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+04-46-50+00
2022-04-07 04:49:20,921 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+04-30-56+00
2022-04-07 04:49:20,921 - speechbrain.utils.epoch_loop - INFO - Going into epoch 14
2022-04-07 05:02:32,796 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.41 to 0.33
2022-04-07 05:02:32,805 - speechbrain.nnet.schedulers - INFO - Changing lr from 6.6e-05 to 5.9e-05
2022-04-07 05:02:32,852 - speechbrain.utils.train_logger - INFO - epoch: 14, lr_model: 4.10e-01, lr_wav2vec: 6.56e-05 - train loss: 1.58e-01 - valid loss: 3.94e-01, valid CER: 7.75, valid WER: 28.90
2022-04-07 05:04:12,624 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+05-02-32+00
2022-04-07 05:04:14,019 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+04-46-50+00
2022-04-07 05:04:14,019 - speechbrain.utils.epoch_loop - INFO - Going into epoch 15
2022-04-07 05:17:40,541 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.33 to 0.26
2022-04-07 05:17:40,556 - speechbrain.nnet.schedulers - INFO - Changing lr from 5.9e-05 to 5.3e-05
2022-04-07 05:17:40,606 - speechbrain.utils.train_logger - INFO - epoch: 15, lr_model: 3.28e-01, lr_wav2vec: 5.90e-05 - train loss: 1.37e-01 - valid loss: 4.13e-01, valid CER: 7.64, valid WER: 28.39
2022-04-07 05:19:24,936 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+05-17-40+00
2022-04-07 05:19:25,870 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+05-02-32+00
2022-04-07 05:19:25,870 - speechbrain.utils.epoch_loop - INFO - Going into epoch 16
2022-04-07 05:32:43,793 - speechbrain.utils.train_logger - INFO - epoch: 16, lr_model: 2.62e-01, lr_wav2vec: 5.31e-05 - train loss: 1.25e-01 - valid loss: 3.95e-01, valid CER: 7.48, valid WER: 27.94
2022-04-07 05:34:31,869 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+05-32-43+00
2022-04-07 05:34:32,562 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+05-17-40+00
2022-04-07 05:34:32,563 - speechbrain.utils.epoch_loop - INFO - Going into epoch 17
2022-04-07 05:47:42,780 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.26 to 0.21
2022-04-07 05:47:42,796 - speechbrain.nnet.schedulers - INFO - Changing lr from 5.3e-05 to 4.8e-05
2022-04-07 05:47:42,879 - speechbrain.utils.train_logger - INFO - epoch: 17, lr_model: 2.62e-01, lr_wav2vec: 5.31e-05 - train loss: 1.20e-01 - valid loss: 4.12e-01, valid CER: 7.36, valid WER: 27.73
2022-04-07 05:50:08,133 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+05-47-42+00
2022-04-07 05:50:11,233 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+05-32-43+00
2022-04-07 05:50:11,244 - speechbrain.utils.epoch_loop - INFO - Going into epoch 18
2022-04-07 06:03:30,233 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.21 to 0.17
2022-04-07 06:03:30,246 - speechbrain.nnet.schedulers - INFO - Changing lr from 4.8e-05 to 4.3e-05
2022-04-07 06:03:30,328 - speechbrain.utils.train_logger - INFO - epoch: 18, lr_model: 2.10e-01, lr_wav2vec: 4.78e-05 - train loss: 1.03e-01 - valid loss: 4.31e-01, valid CER: 7.44, valid WER: 27.69
2022-04-07 06:07:59,476 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+06-03-30+00
2022-04-07 06:08:02,491 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+05-47-42+00
2022-04-07 06:08:02,503 - speechbrain.utils.epoch_loop - INFO - Going into epoch 19
2022-04-07 06:21:15,279 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.17 to 0.13
2022-04-07 06:21:15,296 - speechbrain.nnet.schedulers - INFO - Changing lr from 4.3e-05 to 3.9e-05
2022-04-07 06:21:15,414 - speechbrain.utils.train_logger - INFO - epoch: 19, lr_model: 1.68e-01, lr_wav2vec: 4.30e-05 - train loss: 9.82e-02 - valid loss: 4.35e-01, valid CER: 7.28, valid WER: 27.08
2022-04-07 06:23:34,904 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+06-21-15+00
2022-04-07 06:23:36,249 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+06-03-30+00
2022-04-07 06:23:36,250 - speechbrain.utils.epoch_loop - INFO - Going into epoch 20
2022-04-07 06:36:49,810 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.13 to 0.11
2022-04-07 06:36:49,823 - speechbrain.nnet.schedulers - INFO - Changing lr from 3.9e-05 to 3.5e-05
2022-04-07 06:36:49,890 - speechbrain.utils.train_logger - INFO - epoch: 20, lr_model: 1.34e-01, lr_wav2vec: 3.87e-05 - train loss: 8.78e-02 - valid loss: 4.42e-01, valid CER: 7.27, valid WER: 27.18
2022-04-07 06:39:22,101 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+06-36-49+00
2022-04-07 06:39:22,945 - speechbrain.utils.epoch_loop - INFO - Going into epoch 21
2022-04-07 06:52:36,778 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.11 to 0.086
2022-04-07 06:52:36,782 - speechbrain.nnet.schedulers - INFO - Changing lr from 3.5e-05 to 3.1e-05
2022-04-07 06:52:36,852 - speechbrain.utils.train_logger - INFO - epoch: 21, lr_model: 1.07e-01, lr_wav2vec: 3.49e-05 - train loss: 8.27e-02 - valid loss: 4.72e-01, valid CER: 7.16, valid WER: 26.79
2022-04-07 06:55:10,567 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+06-52-36+00
2022-04-07 06:55:12,223 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+06-21-15+00
2022-04-07 06:55:12,905 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+06-36-49+00
2022-04-07 06:55:12,905 - speechbrain.utils.epoch_loop - INFO - Going into epoch 22
2022-04-07 07:08:24,492 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.086 to 0.069
2022-04-07 07:08:24,497 - speechbrain.nnet.schedulers - INFO - Changing lr from 3.1e-05 to 2.8e-05
2022-04-07 07:08:24,542 - speechbrain.utils.train_logger - INFO - epoch: 22, lr_model: 8.59e-02, lr_wav2vec: 3.14e-05 - train loss: 7.39e-02 - valid loss: 4.88e-01, valid CER: 7.03, valid WER: 26.28
2022-04-07 07:10:26,930 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+07-08-24+00
2022-04-07 07:10:29,254 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+06-52-36+00
2022-04-07 07:10:29,254 - speechbrain.utils.epoch_loop - INFO - Going into epoch 23
2022-04-07 07:23:43,676 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.069 to 0.055
2022-04-07 07:23:43,692 - speechbrain.nnet.schedulers - INFO - Changing lr from 2.8e-05 to 2.5e-05
2022-04-07 07:23:43,814 - speechbrain.utils.train_logger - INFO - epoch: 23, lr_model: 6.87e-02, lr_wav2vec: 2.82e-05 - train loss: 7.24e-02 - valid loss: 4.92e-01, valid CER: 6.95, valid WER: 26.03
2022-04-07 07:29:42,385 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+07-23-43+00
2022-04-07 07:29:44,066 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+07-08-24+00
2022-04-07 07:29:44,066 - speechbrain.utils.epoch_loop - INFO - Going into epoch 24
2022-04-07 07:43:02,440 - speechbrain.utils.train_logger - INFO - epoch: 24, lr_model: 5.50e-02, lr_wav2vec: 2.54e-05 - train loss: 6.76e-02 - valid loss: 4.80e-01, valid CER: 6.89, valid WER: 26.11
2022-04-07 07:48:09,873 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+07-43-02+00
2022-04-07 07:48:12,476 - speechbrain.utils.epoch_loop - INFO - Going into epoch 25
2022-04-07 08:01:25,251 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.055 to 0.044
2022-04-07 08:01:25,297 - speechbrain.nnet.schedulers - INFO - Changing lr from 2.5e-05 to 2.3e-05
2022-04-07 08:01:25,429 - speechbrain.utils.train_logger - INFO - epoch: 25, lr_model: 5.50e-02, lr_wav2vec: 2.54e-05 - train loss: 6.21e-02 - valid loss: 5.01e-01, valid CER: 6.88, valid WER: 25.88
2022-04-07 08:03:25,750 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+08-01-25+00
2022-04-07 08:03:27,595 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+07-43-02+00
2022-04-07 08:03:28,215 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+07-23-43+00
2022-04-07 08:03:28,215 - speechbrain.utils.epoch_loop - INFO - Going into epoch 26
2022-04-07 08:16:35,211 - speechbrain.utils.train_logger - INFO - epoch: 26, lr_model: 4.40e-02, lr_wav2vec: 2.29e-05 - train loss: 5.94e-02 - valid loss: 4.89e-01, valid CER: 6.90, valid WER: 26.08
2022-04-07 08:19:37,126 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+08-16-35+00
2022-04-07 08:19:39,166 - speechbrain.utils.epoch_loop - INFO - Going into epoch 27
2022-04-07 08:32:51,788 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.044 to 0.035
2022-04-07 08:32:51,994 - speechbrain.nnet.schedulers - INFO - Changing lr from 2.3e-05 to 2.1e-05
2022-04-07 08:32:52,086 - speechbrain.utils.train_logger - INFO - epoch: 27, lr_model: 4.40e-02, lr_wav2vec: 2.29e-05 - train loss: 5.54e-02 - valid loss: 5.09e-01, valid CER: 6.87, valid WER: 26.04
2022-04-07 08:35:48,324 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+08-32-52+00
2022-04-07 08:35:51,566 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+08-16-35+00
2022-04-07 08:35:51,582 - speechbrain.utils.epoch_loop - INFO - Going into epoch 28
2022-04-07 08:48:53,259 - speechbrain.utils.train_logger - INFO - epoch: 28, lr_model: 3.52e-02, lr_wav2vec: 2.06e-05 - train loss: 5.55e-02 - valid loss: 5.01e-01, valid CER: 6.76, valid WER: 25.67
2022-04-07 08:50:45,339 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+08-48-53+00
2022-04-07 08:50:46,363 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+08-32-52+00
2022-04-07 08:50:47,117 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+08-01-25+00
2022-04-07 08:50:47,117 - speechbrain.utils.epoch_loop - INFO - Going into epoch 29
2022-04-07 09:03:47,837 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.035 to 0.028
2022-04-07 09:03:47,840 - speechbrain.nnet.schedulers - INFO - Changing lr from 2.1e-05 to 1.9e-05
2022-04-07 09:03:47,887 - speechbrain.utils.train_logger - INFO - epoch: 29, lr_model: 3.52e-02, lr_wav2vec: 2.06e-05 - train loss: 5.44e-02 - valid loss: 5.10e-01, valid CER: 6.72, valid WER: 25.62
2022-04-07 09:06:30,053 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+09-03-47+00
2022-04-07 09:06:32,452 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+08-48-53+00
2022-04-07 09:06:32,452 - speechbrain.utils.epoch_loop - INFO - Going into epoch 30
2022-04-07 09:19:28,245 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.028 to 0.023
2022-04-07 09:19:28,266 - speechbrain.nnet.schedulers - INFO - Changing lr from 1.9e-05 to 1.7e-05
2022-04-07 09:19:28,342 - speechbrain.utils.train_logger - INFO - epoch: 30, lr_model: 2.81e-02, lr_wav2vec: 1.85e-05 - train loss: 5.01e-02 - valid loss: 5.29e-01, valid CER: 6.71, valid WER: 25.50
2022-04-07 09:21:20,512 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+09-19-28+00
2022-04-07 09:21:22,686 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+09-03-47+00
2022-04-07 09:21:22,696 - speechbrain.utils.checkpoints - INFO - Loading a checkpoint from results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+09-19-28+00
2022-04-07 09:21:42,982 - root - DEBUG - SaveableDataLoader was requested to load a checkpoint, but the DataLoader has already been iterated. The DataLoader file will be ignored. This is normal in evaluation, when a checkpoint is loaded just to retrieve the best model.
2022-04-07 09:24:40,071 - speechbrain.utils.train_logger - INFO - Epoch loaded: 30 - test loss: 5.24e-01, test CER: 6.57, test WER: 24.92
2022-04-07 09:25:00,438 - speechbrain.core - INFO - Beginning experiment!
2022-04-07 09:25:00,447 - speechbrain.core - INFO - Experiment folder: results/wav2vec2_ctc_AMHARIC/1249
2022-04-07 09:25:01,071 - speechbrain.utils.superpowers - DEBUG - aiohttp==3.8.1
aiosignal==1.2.0
appdirs==1.4.4
async-timeout==4.0.2
attrs==21.4.0
audioread==2.1.9
azure-core==1.21.1
azure-storage-blob==12.9.0
bcrypt==3.2.0
black==19.10b0
certifi==2021.10.8
cffi==1.15.0
cfgv==3.3.1
charset-normalizer==2.0.10
click==8.0.3
cryptography==36.0.1
datasets==1.13.3
decorator==5.1.1
dill==0.3.4
distlib==0.3.4
entrypoints==0.3
ffmpeg==1.4
filelock==3.4.2
flake8==3.7.9
frozenlist==1.3.0
fsspec==2022.2.0
huggingface-hub==0.2.1
HyperPyYAML==1.0.0
identify==2.4.4
idna==3.3
isodate==0.6.1
joblib==1.1.0
librosa==0.8.1
llvmlite==0.38.0
mccabe==0.6.1
more-itertools==8.12.0
msrest==0.6.21
multidict==6.0.2
multiprocess==0.70.12.2
mutagen==1.45.1
nodeenv==1.6.0
numba==0.55.0
numpy==1.21.5
oauthlib==3.1.1
packaging==21.3
pandas==1.3.5
paramiko==2.10.3
pathspec==0.9.0
platformdirs==2.4.1
pluggy==0.13.1
pooch==1.5.2
pre-commit==2.17.0
py==1.11.0
pyarrow==7.0.0
pycodestyle==2.5.0
pycparser==2.21
pyflakes==2.1.1
PyNaCl==1.5.0
pyparsing==3.0.6
pytest==5.4.1
python-dateutil==2.8.2
pytz==2021.3
PyYAML==6.0
regex==2022.1.18
requests==2.27.1
requests-oauthlib==1.3.0
resampy==0.2.2
ruamel.yaml==0.17.20
ruamel.yaml.clib==0.2.6
sacremoses==0.0.47
scikit-learn==1.0.2
scipy==1.7.3
scp==0.14.4
sentencepiece==0.1.96
six==1.16.0
SoundFile==0.10.3.post1
threadpoolctl==3.0.0
tokenizers==0.10.3
toml==0.10.2
torch==1.10.1
torchaudio==0.10.1
tqdm==4.62.3
transformers==4.13.0
typed-ast==1.5.1
typing_extensions==4.0.1
urllib3==1.26.8
virtualenv==20.13.0
wcwidth==0.2.5
xxhash==3.0.0
yamllint==1.23.0
yarl==1.7.2


2022-04-07 09:25:01,407 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/train.csv already exists, skipping data preparation!
2022-04-07 09:25:01,407 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/dev.csv already exists, skipping data preparation!
2022-04-07 09:25:01,407 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/test.csv already exists, skipping data preparation!
2022-04-07 09:25:01,413 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer is already trained.
2022-04-07 09:25:01,413 - speechbrain.tokenizers.SentencePiece - INFO - ==== Loading Tokenizer ===
2022-04-07 09:25:01,413 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer path: results/wav2vec2_ctc_AMHARIC/1249/save/224_char.model
2022-04-07 09:25:01,413 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer vocab_size: 224
2022-04-07 09:25:01,413 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer type: char
2022-04-07 09:25:01,591 - speechbrain.core - INFO - Info: auto_mix_prec arg from hparam file is used
2022-04-07 09:25:01,591 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used
2022-04-07 09:25:05,095 - speechbrain.core - INFO - 318.8M trainable parameters in ASR
2022-04-07 09:25:05,382 - speechbrain.utils.checkpoints - INFO - Loading a checkpoint from results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+09-19-28+00
2022-04-07 09:25:08,583 - speechbrain.utils.checkpoints - INFO - Loading a checkpoint from results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+09-19-28+00
2022-04-07 09:26:16,796 - speechbrain.utils.train_logger - INFO - Epoch loaded: 30 - test loss: 5.24e-01, test CER: 6.57, test WER: 24.92
2022-05-27 08:17:50,151 - speechbrain.core - INFO - Beginning experiment!
2022-05-27 08:17:50,151 - speechbrain.core - INFO - Experiment folder: results/wav2vec2_ctc_AMHARIC/1249
2022-05-27 08:17:54,273 - speechbrain.utils.superpowers - DEBUG - aiohttp==3.8.1
aiosignal==1.2.0
appdirs==1.4.4
async-timeout==4.0.2
attrs==21.4.0
audioread==2.1.9
audiosegment==0.23.0
azure-core==1.21.1
azure-storage-blob==12.9.0
bcrypt==3.2.0
black==19.10b0
certifi==2021.10.8
cffi==1.15.0
cfgv==3.3.1
charset-normalizer==2.0.10
click==8.0.3
cryptography==36.0.1
datasets==1.13.3
decorator==5.1.1
dill==0.3.4
distlib==0.3.4
entrypoints==0.3
ffmpeg==1.4
filelock==3.4.2
flake8==3.7.9
frozenlist==1.3.0
fsspec==2022.2.0
huggingface-hub==0.5.1
HyperPyYAML==1.0.1
identify==2.4.4
idna==3.3
isodate==0.6.1
joblib==1.1.0
librosa==0.8.1
llvmlite==0.38.0
mccabe==0.6.1
more-itertools==8.12.0
msrest==0.6.21
multidict==6.0.2
multiprocess==0.70.12.2
mutagen==1.45.1
nodeenv==1.6.0
numba==0.55.0
numpy==1.21.5
oauthlib==3.1.1
packaging==21.3
pandas==1.3.5
paramiko==2.10.3
pathspec==0.9.0
platformdirs==2.4.1
pluggy==0.13.1
pooch==1.5.2
pre-commit==2.17.0
py==1.11.0
pyarrow==7.0.0
pycodestyle==2.5.0
pycparser==2.21
pydub==0.25.1
pyflakes==2.1.1
PyNaCl==1.5.0
pyparsing==3.0.6
pytest==5.4.1
python-dateutil==2.8.2
pytz==2021.3
PyYAML==6.0
regex==2022.1.18
requests==2.27.1
requests-oauthlib==1.3.0
resampy==0.2.2
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.6
sacremoses==0.0.53
scikit-learn==1.0.2
scipy==1.7.3
scp==0.14.4
sentencepiece==0.1.96
six==1.16.0
SoundFile==0.10.3.post1
speechbrain==0.5.11
threadpoolctl==3.0.0
tokenizers==0.12.1
toml==0.10.2
torch==1.11.0
torchaudio==0.11.0
tqdm==4.62.3
transformers==4.18.0
typed-ast==1.5.1
typing_extensions==4.0.1
urllib3==1.26.8
virtualenv==20.13.0
wcwidth==0.2.5
webrtcvad==2.0.10
xxhash==3.0.0
yamllint==1.23.0
yarl==1.7.2
youtube-dl==2021.12.17


2022-05-27 08:19:01,150 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/train.csv already exists, skipping data preparation!
2022-05-27 08:19:01,151 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/dev.csv already exists, skipping data preparation!
2022-05-27 08:19:01,151 - dvoice_prepare - INFO - results/wav2vec2_ctc_AMHARIC/1249/save/test.csv already exists, skipping data preparation!
2022-05-27 08:19:01,160 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer is already trained.
2022-05-27 08:19:01,160 - speechbrain.tokenizers.SentencePiece - INFO - ==== Loading Tokenizer ===
2022-05-27 08:19:01,160 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer path: results/wav2vec2_ctc_AMHARIC/1249/save/224_char.model
2022-05-27 08:19:01,160 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer vocab_size: 224
2022-05-27 08:19:01,160 - speechbrain.tokenizers.SentencePiece - INFO - Tokenizer type: char
2022-05-27 08:19:01,425 - speechbrain.core - INFO - Info: auto_mix_prec arg from hparam file is used
2022-05-27 08:19:01,425 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used
2022-05-27 08:19:09,868 - speechbrain.core - INFO - 318.8M trainable parameters in ASR
2022-05-27 08:19:09,899 - speechbrain.utils.checkpoints - INFO - Loading a checkpoint from results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+09-19-28+00
2022-05-27 08:20:01,646 - speechbrain.utils.checkpoints - INFO - Loading a checkpoint from results/wav2vec2_ctc_AMHARIC/1249/save/CKPT+2022-04-07+09-19-28+00
2022-05-27 08:20:59,252 - speechbrain.utils.train_logger - INFO - Epoch loaded: 30 - test loss: 5.24e-01, test CER: 6.57, test WER: 24.92