"
- ]
- },
- "execution_count": 60,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "import IPython.display as ipd\n",
- "import numpy as np\n",
- "import random\n",
- "\n",
- "rand_int = random.randint(0, len(common_voice_train)-1)\n",
- "\n",
- "print(common_voice_train[rand_int][\"sentence\"])\n",
- "ipd.Audio(data=common_voice_train[rand_int][\"audio\"][\"array\"], autoplay=True, rate=16000)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "gY8m3vARHYTa"
- },
- "source": [
- "It seems like the data is now correctly loaded and resampled."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "1MaL9J2dNVtG"
- },
- "source": [
- "It can be heard, that the speakers change along with their speaking rate, accent, and background environment, etc. Overall, the recordings sound acceptably clear though, which is to be expected from a crowd-sourced read speech corpus.\n",
- "\n",
- "Let's do a final check that the data is correctly prepared, by printing the shape of the speech input, its transcription, and the corresponding sampling rate.\n",
- "\n",
- "**Note**: *You can click the following cell a couple of times to verify multiple samples.*"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "1Po2g7YPuRTx",
- "outputId": "63478ae9-2927-4ec1-c13c-41fb3754e18e"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Target text: ülke olimpiyatları yirmi ikinci sırada tamamladı\n",
- "Input array shape: (74880,)\n",
- "Sampling rate: 16000\n"
- ]
- }
- ],
- "source": [
- "rand_int = random.randint(0, len(common_voice_train)-1)\n",
- "\n",
- "print(\"Target text:\", common_voice_train[rand_int][\"sentence\"])\n",
- "print(\"Input array shape:\", common_voice_train[rand_int][\"audio\"][\"array\"].shape)\n",
- "print(\"Sampling rate:\", common_voice_train[rand_int][\"audio\"][\"sampling_rate\"])"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "M9teZcSwOBJ4"
- },
- "source": [
- "Good! Everything looks fine - the data is a 1-dimensional array, the sampling rate always corresponds to 16kHz, and the target text is normalized."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "k3Pbn5WvOYZF"
- },
- "source": [
- "Finally, we can leverage `Wav2Vec2Processor` to process the data to the format expected by `Wav2Vec2ForCTC` for training. To do so let's make use of Dataset's [`map(...)`](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=map#datasets.DatasetDict.map) function.\n",
- "\n",
- "First, we load and resample the audio data, simply by calling `batch[\"audio\"]`.\n",
- "Second, we extract the `input_values` from the loaded audio file. In our case, the `Wav2Vec2Processor` only normalizes the data. For other speech models, however, this step can include more complex feature extraction, such as [Log-Mel feature extraction](https://en.wikipedia.org/wiki/Mel-frequency_cepstrum).\n",
- "Third, we encode the transcriptions to label ids.\n",
- "\n",
- "**Note**: This mapping function is a good example of how the `Wav2Vec2Processor` class should be used. In \"normal\" context, calling `processor(...)` is redirected to `Wav2Vec2FeatureExtractor`'s call method. When wrapping the processor into the `as_target_processor` context, however, the same method is redirected to `Wav2Vec2CTCTokenizer`'s call method.\n",
- "For more information please check the [docs](https://huggingface.co/transformers/master/model_doc/wav2vec2.html#transformers.Wav2Vec2Processor.__call__)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "eJY7I0XAwe9p"
- },
- "outputs": [],
- "source": [
- "def prepare_dataset(batch):\n",
- " audio = batch[\"audio\"]\n",
- "\n",
- " # batched output is \"un-batched\"\n",
- " batch[\"input_values\"] = processor(audio[\"array\"], sampling_rate=audio[\"sampling_rate\"]).input_values[0]\n",
- " batch[\"input_length\"] = len(batch[\"input_values\"])\n",
- "\n",
- " with processor.as_target_processor():\n",
- " batch[\"labels\"] = processor(batch[\"sentence\"]).input_ids\n",
- " return batch"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "q6Pg_WR3OGAP"
- },
- "source": [
- "Let's apply the data preparation function to all examples."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 81,
- "referenced_widgets": [
- "012b09b271a842d0a7f1cd1260654c82",
- "6a40faf77bf445308534e7acada6bf0d",
- "a62ff1ee03dc47c9834b43720df991ea",
- "03d00b443bfd445d9286f64fd8556031",
- "557063bf861648f7aefacf71aa53ea9b",
- "21af0f39c0ce40a5846b980f98481940",
- "f6e0ea791e5a44abb5d5891bb3e254ad",
- "6323e35581be4b338a904266a5e85005",
- "6da7072e7f634a3aa2927fca1bcf38c0",
- "716dc4ab4fe74abab49f3c0f009357f5",
- "8a669878b8354867944e577860910001",
- "3aa317d2b42e4674b38dafa87b5599c5",
- "0a7c2124936345febe50288a476a12a6",
- "bdcf9380af5d4601b9b2de84b06ecb40",
- "b9c5532ffabb44078de8a6f76a4b8be6",
- "7732bb1317f940faa455f992f4fd2b26",
- "f28c64f34f9540b6b6bdc765c84e2307",
- "02016dcc44994c15b69e7f2c4dd0d9c4",
- "bbfc45e0cf5644f6bc24118fa9c6c098",
- "eca458b185ae4570b1ec1e77389c871e",
- "fe156fa95ee9410daff8f7c9a8b0d3a3",
- "a5e2e1fe3ab94cc6ae1a73cc1ab6f9a8"
- ]
- },
- "id": "-np9xYK-wl8q",
- "outputId": "00d6940a-a7bf-4128-896b-76bc289e5b7f"
- },
- "outputs": [
- {
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "012b09b271a842d0a7f1cd1260654c82",
- "version_major": 2,
- "version_minor": 0
- },
- "text/plain": [
- " 0%| | 0/3478 [00:00 Dict[str, torch.Tensor]:\n",
- " # split inputs and labels since they have to be of different lenghts and need\n",
- " # different padding methods\n",
- " input_features = [{\"input_values\": feature[\"input_values\"]} for feature in features]\n",
- " label_features = [{\"input_ids\": feature[\"labels\"]} for feature in features]\n",
- "\n",
- " batch = self.processor.pad(\n",
- " input_features,\n",
- " padding=self.padding,\n",
- " return_tensors=\"pt\",\n",
- " )\n",
- " with self.processor.as_target_processor():\n",
- " labels_batch = self.processor.pad(\n",
- " label_features,\n",
- " padding=self.padding,\n",
- " return_tensors=\"pt\",\n",
- " )\n",
- "\n",
- " # replace padding with -100 to ignore loss correctly\n",
- " labels = labels_batch[\"input_ids\"].masked_fill(labels_batch.attention_mask.ne(1), -100)\n",
- "\n",
- " batch[\"labels\"] = labels\n",
- "\n",
- " return batch"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "lbQf5GuZyQ4_"
- },
- "outputs": [],
- "source": [
- "data_collator = DataCollatorCTCWithPadding(processor=processor, padding=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "xO-Zdj-5cxXp"
- },
- "source": [
- "Next, the evaluation metric is defined. As mentioned earlier, the\n",
- "predominant metric in ASR is the word error rate (WER), hence we will use it in this notebook as well."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 49,
- "referenced_widgets": [
- "7c81059d35534b799623afe872095cac",
- "379128a308ab43ee9e785e234bb94049",
- "ff8e746cf8ef4f988617b8d4c0dd3c9c",
- "0204a7bc668746a68f6e5e5632753c1c",
- "84361b363998428a87b9e50b6625cca9",
- "c0919f3c8d594911815ce63273f7fe51",
- "69f2a9e473534148b96b2e9bd639a087",
- "5cd8c74ba2e845d498afdc1c25009767",
- "3f0c587f7b764d96837c8e16254aab12",
- "d3b5f5a0d35642e0b606ac63c38f88fa",
- "7ce84a97d72848c0967f732734afdd68"
- ]
- },
- "id": "9Xsux2gmyXso",
- "outputId": "18ceeb9e-1a0d-4ee8-f511-a12ad3608bf1"
- },
- "outputs": [
- {
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "7c81059d35534b799623afe872095cac",
- "version_major": 2,
- "version_minor": 0
- },
- "text/plain": [
- "Downloading: 0%| | 0.00/1.95k [00:00> m$.\n",
- "\n",
- "A logit vector $\\mathbf{y}_1$ contains the log-odds for each word in the vocabulary we defined earlier, thus $\\text{len}(\\mathbf{y}_i) =$ `config.vocab_size`. We are interested in the most likely prediction of the model and thus take the `argmax(...)` of the logits. Also, we transform the encoded labels back to the original string by replacing `-100` with the `pad_token_id` and decoding the ids while making sure that consecutive tokens are **not** grouped to the same token in CTC style ${}^1$."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "1XZ-kjweyTy_"
- },
- "outputs": [],
- "source": [
- "def compute_metrics(pred):\n",
- " pred_logits = pred.predictions\n",
- " pred_ids = np.argmax(pred_logits, axis=-1)\n",
- "\n",
- " pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id\n",
- "\n",
- " pred_str = processor.batch_decode(pred_ids)\n",
- " # we do not want to group tokens when computing the metrics\n",
- " label_str = processor.batch_decode(pred.label_ids, group_tokens=False)\n",
- "\n",
- " wer = wer_metric.compute(predictions=pred_str, references=label_str)\n",
- "\n",
- " return {\"wer\": wer}"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "Xmgrx4bRwLIH"
- },
- "source": [
- "Now, we can load the pretrained checkpoint of [Wav2Vec2-XLS-R-300M](https://huggingface.co/facebook/wav2vec2-xls-r-300m). The tokenizer's `pad_token_id` must be to define the model's `pad_token_id` or in the case of `Wav2Vec2ForCTC` also CTC's *blank token* ${}^2$. To save GPU memory, we enable PyTorch's [gradient checkpointing](https://pytorch.org/docs/stable/checkpoint.html) and also set the loss reduction to \"*mean*\".\n",
- "\n",
- "Because the dataset is quite small (~6h of training data) and because Common Voice is quite noisy, fine-tuning Facebook's [wav2vec2-xls-r-300m checkpoint](https://huggingface.co/facebook/wav2vec2-xls-r-300m) seems to require some hyper-parameter tuning. Therefore, I had to play around a bit with different values for dropout, [SpecAugment](https://arxiv.org/abs/1904.08779)'s masking dropout rate, layer dropout, and the learning rate until training seemed to be stable enough.\n",
- "\n",
- "**Note**: When using this notebook to train XLS-R on another language of Common Voice those hyper-parameter settings might not work very well. Feel free to adapt those depending on your use case."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 191,
- "referenced_widgets": [
- "1ca82f9af3ae423096661a1ffd8a165f",
- "ce429683f48349c083e5fc3253a897d7",
- "acc35dfdfca542828d1bb02cbde15dfe",
- "aa92ff120d9c4dd68ae941d7a1f33a81",
- "731a283141f642b9ab5bf9f7053381d2",
- "4246da00789f44e2abce952906bde6f1",
- "059588e34a88476eb6bbc1860ddc27ef",
- "6ae3b3139e6641fc9e62ae5744349add",
- "92c30a7c6102443cad2f6f03211c32e0",
- "ad95ae421231492e814238bde2ecb551",
- "136f973d9623421eb38e09c73c6b6810",
- "367c85905a024248bce6c620b822b2ee",
- "7ea310732a56432da1a095a8d30edc1c",
- "2d2b8b8d5774494aa73d872e9a4c10d9",
- "f4af6a0756e241e49e5d325f1bdb18b4",
- "2b7e52a9be904663ae74226691de6ebd",
- "771530ea22364f1ea8963e2d542fcf49",
- "1cdac9cda666466ca1e09f32661d40ed",
- "3ec9917890f4452883bcaeac444d056c",
- "e9531fcd48b74929bfdb45cd3f3315e7",
- "349c9e8b2a8846c1bff5a03d2a28066a",
- "de1e5c31eacf4fa59cfa51926354acca"
- ]
- },
- "id": "e7cqAWIayn6w",
- "outputId": "3e6cebea-78ef-45df-87b0-f63b78ba9644"
- },
- "outputs": [
- {
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "1ca82f9af3ae423096661a1ffd8a165f",
- "version_major": 2,
- "version_minor": 0
- },
- "text/plain": [
- "Downloading: 0%| | 0.00/1.53k [00:00 inspect -> Console tab and insert code*)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "VYYAvgkW4P0m"
- },
- "source": [
- "```javascript\n",
- "function ConnectButton(){\n",
- " console.log(\"Connect pushed\");\n",
- " document.querySelector(\"#top-toolbar > colab-connect-button\").shadowRoot.querySelector(\"#connect\").click()\n",
- "}\n",
- "setInterval(ConnectButton,60000);\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "7bGgLV2r0yvZ"
- },
- "source": [
- "Depending on what GPU was allocated to your google colab it might be possible that you are seeing an `\"out-of-memory\"` error here. In this case, it's probably best to reduce `per_device_train_batch_size` to 8 or even less and increase [`gradient_accumulation`](https://huggingface.co/transformers/master/main_classes/trainer.html#trainingarguments)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "background_save": true,
- "base_uri": "https://localhost:8080/",
- "height": 311
- },
- "id": "9fRr9TG5pGBl",
- "outputId": "122ea040-7b24-452a-c7d4-7e72e1c46973"
- },
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "The following columns in the training set don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.\n",
- "***** Running training *****\n",
- " Num examples = 3478\n",
- " Num Epochs = 30\n",
- " Instantaneous batch size per device = 16\n",
- " Total train batch size (w. parallel, distributed & accumulation) = 32\n",
- " Gradient Accumulation steps = 2\n",
- " Total optimization steps = 3270\n",
- "/usr/local/lib/python3.7/dist-packages/transformers/feature_extraction_utils.py:158: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:201.)\n",
- " tensor = as_tensor(value)\n",
- "/usr/local/lib/python3.7/dist-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:882: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
- " return (input_length - kernel_size) // stride + 1\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "\n",
- " \n",
- " \n",
- "
\n",
- " [3270/3270 4:41:48, Epoch 30/30]\n",
- "
\n",
- " \n",
- " \n",
- " | Step | \n",
- " Training Loss | \n",
- " Validation Loss | \n",
- " Wer | \n",
- "
\n",
- " \n",
- " \n",
- " \n",
- " | 400 | \n",
- " 3.765200 | \n",
- " 0.625533 | \n",
- " 0.693290 | \n",
- "
\n",
- " \n",
- " | 800 | \n",
- " 0.371600 | \n",
- " 0.405785 | \n",
- " 0.456337 | \n",
- "
\n",
- " \n",
- " | 1200 | \n",
- " 0.177900 | \n",
- " 0.412796 | \n",
- " 0.408028 | \n",
- "
\n",
- " \n",
- " | 1600 | \n",
- " 0.122600 | \n",
- " 0.399832 | \n",
- " 0.394137 | \n",
- "
\n",
- " \n",
- " | 2000 | \n",
- " 0.092700 | \n",
- " 0.408412 | \n",
- " 0.371259 | \n",
- "
\n",
- " \n",
- " | 2400 | \n",
- " 0.073700 | \n",
- " 0.394867 | \n",
- " 0.351956 | \n",
- "
\n",
- " \n",
- " | 2800 | \n",
- " 0.058100 | \n",
- " 0.369830 | \n",
- " 0.331120 | \n",
- "
\n",
- " \n",
- " | 3200 | \n",
- " 0.045300 | \n",
- " 0.359626 | \n",
- " 0.324175 | \n",
- "
\n",
- " \n",
- "
"
- ],
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "The following columns in the evaluation set don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.\n",
- "***** Running Evaluation *****\n",
- " Num examples = 1647\n",
- " Batch size = 8\n",
- "Saving model checkpoint to wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-400\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-400/config.json\n",
- "Model weights saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-400/pytorch_model.bin\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-400/preprocessor_config.json\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/preprocessor_config.json\n",
- "/usr/local/lib/python3.7/dist-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:882: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
- " return (input_length - kernel_size) // stride + 1\n",
- "The following columns in the evaluation set don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.\n",
- "***** Running Evaluation *****\n",
- " Num examples = 1647\n",
- " Batch size = 8\n",
- "Saving model checkpoint to wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-800\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-800/config.json\n",
- "Model weights saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-800/pytorch_model.bin\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-800/preprocessor_config.json\n",
- "/usr/local/lib/python3.7/dist-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:882: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
- " return (input_length - kernel_size) // stride + 1\n",
- "The following columns in the evaluation set don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.\n",
- "***** Running Evaluation *****\n",
- " Num examples = 1647\n",
- " Batch size = 8\n",
- "Saving model checkpoint to wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-1200\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-1200/config.json\n",
- "Model weights saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-1200/pytorch_model.bin\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-1200/preprocessor_config.json\n",
- "Deleting older checkpoint [wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-400] due to args.save_total_limit\n",
- "/usr/local/lib/python3.7/dist-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:882: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
- " return (input_length - kernel_size) // stride + 1\n",
- "The following columns in the evaluation set don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.\n",
- "***** Running Evaluation *****\n",
- " Num examples = 1647\n",
- " Batch size = 8\n",
- "Saving model checkpoint to wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-1600\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-1600/config.json\n",
- "Model weights saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-1600/pytorch_model.bin\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-1600/preprocessor_config.json\n",
- "Deleting older checkpoint [wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-800] due to args.save_total_limit\n",
- "/usr/local/lib/python3.7/dist-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:882: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
- " return (input_length - kernel_size) // stride + 1\n",
- "The following columns in the evaluation set don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.\n",
- "***** Running Evaluation *****\n",
- " Num examples = 1647\n",
- " Batch size = 8\n",
- "Saving model checkpoint to wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-2000\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-2000/config.json\n",
- "Model weights saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-2000/pytorch_model.bin\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-2000/preprocessor_config.json\n",
- "Deleting older checkpoint [wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-1200] due to args.save_total_limit\n",
- "/usr/local/lib/python3.7/dist-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:882: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
- " return (input_length - kernel_size) // stride + 1\n",
- "The following columns in the evaluation set don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.\n",
- "***** Running Evaluation *****\n",
- " Num examples = 1647\n",
- " Batch size = 8\n",
- "Saving model checkpoint to wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-2400\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-2400/config.json\n",
- "Model weights saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-2400/pytorch_model.bin\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-2400/preprocessor_config.json\n",
- "Deleting older checkpoint [wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-1600] due to args.save_total_limit\n",
- "/usr/local/lib/python3.7/dist-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:882: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
- " return (input_length - kernel_size) // stride + 1\n",
- "The following columns in the evaluation set don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.\n",
- "***** Running Evaluation *****\n",
- " Num examples = 1647\n",
- " Batch size = 8\n",
- "Saving model checkpoint to wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-2800\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-2800/config.json\n",
- "Model weights saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-2800/pytorch_model.bin\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-2800/preprocessor_config.json\n",
- "Deleting older checkpoint [wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-2000] due to args.save_total_limit\n",
- "/usr/local/lib/python3.7/dist-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:882: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
- " return (input_length - kernel_size) // stride + 1\n",
- "The following columns in the evaluation set don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.\n",
- "***** Running Evaluation *****\n",
- " Num examples = 1647\n",
- " Batch size = 8\n",
- "Saving model checkpoint to wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-3200\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-3200/config.json\n",
- "Model weights saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-3200/pytorch_model.bin\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-3200/preprocessor_config.json\n",
- "Deleting older checkpoint [wav2vec2-large-xls-r-300m-turkish-colab/checkpoint-2400] due to args.save_total_limit\n",
- "/usr/local/lib/python3.7/dist-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:882: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
- " return (input_length - kernel_size) // stride + 1\n",
- "\n",
- "\n",
- "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
- "\n",
- "\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "TrainOutput(global_step=3270, training_loss=0.5767540113641582, metrics={'train_runtime': 16918.4457, 'train_samples_per_second': 6.167, 'train_steps_per_second': 0.193, 'total_flos': 1.2873093995838829e+19, 'train_loss': 0.5767540113641582, 'epoch': 30.0})"
- ]
- },
- "execution_count": null,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "trainer.train()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "a9q4mgMZplr_"
- },
- "source": [
- "The training loss and validation WER go down nicely."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "4Ya7WEy0pd13"
- },
- "source": [
- "You can now upload the result of the training to the 🤗 Hub, just execute this instruction:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "background_save": true,
- "referenced_widgets": [
- "2d2f71d2c70d466cb9a0d3317fb3095e",
- "2e894a5b95cb489db8b27c6617fd9533"
- ]
- },
- "id": "ArG1Thf6NBWm",
- "outputId": "62ef1c3d-786c-4e25-f9c5-4020e71aa298"
- },
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Saving model checkpoint to wav2vec2-large-xls-r-300m-turkish-colab\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/config.json\n",
- "Model weights saved in wav2vec2-large-xls-r-300m-turkish-colab/pytorch_model.bin\n",
- "Configuration saved in wav2vec2-large-xls-r-300m-turkish-colab/preprocessor_config.json\n",
- "Several commits (2) will be pushed upstream.\n",
- "The progress bars may be unreliable.\n"
- ]
- },
- {
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "2d2f71d2c70d466cb9a0d3317fb3095e",
- "version_major": 2,
- "version_minor": 0
- },
- "text/plain": [
- "Upload file pytorch_model.bin: 0%| | 3.35k/1.18G [00:00 main\n",
- "\n",
- "Dropping the following result as it does not have all the necessary field:\n",
- "{'dataset': {'name': 'common_voice', 'type': 'common_voice', 'args': 'tr'}}\n",
- "To https://huggingface.co/patrickvonplaten/wav2vec2-large-xls-r-300m-turkish-colab\n",
- " fe76946..5f0d67b main -> main\n",
- "\n"
- ]
- },
- {
- "data": {
- "application/vnd.google.colaboratory.intrinsic+json": {
- "type": "string"
- },
- "text/plain": [
- "'https://huggingface.co/patrickvonplaten/wav2vec2-large-xls-r-300m-turkish-colab/commit/fe769461e4e2fb9534740e6c278a0cfabf268474'"
- ]
- },
- "execution_count": null,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "trainer.push_to_hub()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "RHIVc44_fY2N"
- },
- "source": [
- "You can now share this model with all your friends, family, favorite pets: they can all load it with the identifier \"your-username/the-name-you-picked\" so for instance:"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "5lWWIKyBpx1h"
- },
- "source": [
- "```python\n",
- "from transformers import AutoModelForCTC, Wav2Vec2Processor\n",
- "\n",
- "model = AutoModelForCTC.from_pretrained(\"patrickvonplaten/wav2vec2-large-xls-r-300m-tr-colab\")\n",
- "processor = Wav2Vec2Processor.from_pretrained(\"patrickvonplaten/wav2vec2-large-xls-r-300m-tr-colab\")\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "pmi1cX0fRBit"
- },
- "source": [
- "For more examples of how XLS-R can be fine-tuned, please take a look at the [official speech recognition examples](https://github.com/huggingface/transformers/tree/master/examples/pytorch/speech-recognition#examples)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "L8b8Qkoy3KyS"
- },
- "source": [
- "### Evaluation\n",
- "\n",
- "As a final check, let's load the model and verify that it indeed has learned to transcribe Turkish speech.\n",
- "\n",
- "Let's first load the pretrained checkpoint."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "background_save": true
- },
- "id": "R351I9IQp_9D",
- "outputId": "f2a2ee99-7db6-4962-e140-0107054102d3"
- },
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "loading configuration file wav2vec2-large-xls-r-300m-turkish-colab/config.json\n",
- "Model config Wav2Vec2Config {\n",
- " \"_name_or_path\": \"facebook/wav2vec2-xls-r-300m\",\n",
- " \"activation_dropout\": 0.0,\n",
- " \"apply_spec_augment\": true,\n",
- " \"architectures\": [\n",
- " \"Wav2Vec2ForCTC\"\n",
- " ],\n",
- " \"attention_dropout\": 0.0,\n",
- " \"bos_token_id\": 1,\n",
- " \"classifier_proj_size\": 256,\n",
- " \"codevector_dim\": 768,\n",
- " \"contrastive_logits_temperature\": 0.1,\n",
- " \"conv_bias\": true,\n",
- " \"conv_dim\": [\n",
- " 512,\n",
- " 512,\n",
- " 512,\n",
- " 512,\n",
- " 512,\n",
- " 512,\n",
- " 512\n",
- " ],\n",
- " \"conv_kernel\": [\n",
- " 10,\n",
- " 3,\n",
- " 3,\n",
- " 3,\n",
- " 3,\n",
- " 2,\n",
- " 2\n",
- " ],\n",
- " \"conv_stride\": [\n",
- " 5,\n",
- " 2,\n",
- " 2,\n",
- " 2,\n",
- " 2,\n",
- " 2,\n",
- " 2\n",
- " ],\n",
- " \"ctc_loss_reduction\": \"mean\",\n",
- " \"ctc_zero_infinity\": false,\n",
- " \"diversity_loss_weight\": 0.1,\n",
- " \"do_stable_layer_norm\": true,\n",
- " \"eos_token_id\": 2,\n",
- " \"feat_extract_activation\": \"gelu\",\n",
- " \"feat_extract_dropout\": 0.0,\n",
- " \"feat_extract_norm\": \"layer\",\n",
- " \"feat_proj_dropout\": 0.0,\n",
- " \"feat_quantizer_dropout\": 0.0,\n",
- " \"final_dropout\": 0.0,\n",
- " \"gradient_checkpointing\": false,\n",
- " \"hidden_act\": \"gelu\",\n",
- " \"hidden_dropout\": 0.0,\n",
- " \"hidden_size\": 1024,\n",
- " \"initializer_range\": 0.02,\n",
- " \"intermediate_size\": 4096,\n",
- " \"layer_norm_eps\": 1e-05,\n",
- " \"layerdrop\": 0.0,\n",
- " \"mask_feature_length\": 10,\n",
- " \"mask_feature_prob\": 0.0,\n",
- " \"mask_time_length\": 10,\n",
- " \"mask_time_prob\": 0.05,\n",
- " \"model_type\": \"wav2vec2\",\n",
- " \"num_attention_heads\": 16,\n",
- " \"num_codevector_groups\": 2,\n",
- " \"num_codevectors_per_group\": 320,\n",
- " \"num_conv_pos_embedding_groups\": 16,\n",
- " \"num_conv_pos_embeddings\": 128,\n",
- " \"num_feat_extract_layers\": 7,\n",
- " \"num_hidden_layers\": 24,\n",
- " \"num_negatives\": 100,\n",
- " \"pad_token_id\": 36,\n",
- " \"proj_codevector_dim\": 768,\n",
- " \"torch_dtype\": \"float32\",\n",
- " \"transformers_version\": \"4.11.3\",\n",
- " \"use_weighted_layer_sum\": false,\n",
- " \"vocab_size\": 39\n",
- "}\n",
- "\n",
- "loading weights file wav2vec2-large-xls-r-300m-turkish-colab/pytorch_model.bin\n",
- "All model checkpoint weights were used when initializing Wav2Vec2ForCTC.\n",
- "\n",
- "All the weights of Wav2Vec2ForCTC were initialized from the model checkpoint at wav2vec2-large-xls-r-300m-turkish-colab.\n",
- "If your task is similar to the task the model of the checkpoint was trained on, you can already use Wav2Vec2ForCTC for predictions without further training.\n",
- "loading feature extractor configuration file wav2vec2-large-xls-r-300m-turkish-colab/preprocessor_config.json\n",
- "Feature extractor Wav2Vec2FeatureExtractor {\n",
- " \"do_normalize\": true,\n",
- " \"feature_extractor_type\": \"Wav2Vec2FeatureExtractor\",\n",
- " \"feature_size\": 1,\n",
- " \"padding_side\": \"right\",\n",
- " \"padding_value\": 0.0,\n",
- " \"return_attention_mask\": true,\n",
- " \"sampling_rate\": 16000\n",
- "}\n",
- "\n",
- "Didn't find file wav2vec2-large-xls-r-300m-turkish-colab/tokenizer.json. We won't load it.\n",
- "loading file wav2vec2-large-xls-r-300m-turkish-colab/vocab.json\n",
- "loading file wav2vec2-large-xls-r-300m-turkish-colab/tokenizer_config.json\n",
- "loading file wav2vec2-large-xls-r-300m-turkish-colab/added_tokens.json\n",
- "loading file wav2vec2-large-xls-r-300m-turkish-colab/special_tokens_map.json\n",
- "loading file None\n",
- "Adding to the vocabulary\n",
- "Adding to the vocabulary\n"
- ]
- }
- ],
- "source": [
- "model = Wav2Vec2ForCTC.from_pretrained(repo_name).to(\"cuda\")\n",
- "processor = Wav2Vec2Processor.from_pretrained(repo_name)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "jD7TZ1YS3S_K"
- },
- "source": [
- "\n",
- "Now, we will just take the first example of the test set, run it through the model and take the `argmax(...)` of the logits to retrieve the predicted token ids."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "background_save": true
- },
- "id": "pax07TnL3WZn",
- "outputId": "867787ff-0cb7-41e9-f926-96f7b53e7134"
- },
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "It is strongly recommended to pass the ``sampling_rate`` argument to this function.Failing to do so can result in silent errors that might be hard to debug.\n"
- ]
- }
- ],
- "source": [
- "input_dict = processor(common_voice_test[0][\"input_values\"], return_tensors=\"pt\", padding=True)\n",
- "\n",
- "logits = model(input_dict.input_values.to(\"cuda\")).logits\n",
- "\n",
- "pred_ids = torch.argmax(logits, dim=-1)[0]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "7nkzSQu53Zs2"
- },
- "source": [
- "We adapted `common_voice_test` quite a bit so that the dataset instance does not contain the original sentence label anymore. Thus, we re-use the original dataset to get the label of the first example."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "background_save": true,
- "referenced_widgets": [
- "54097a6c744849128d7cc3da8aad6609",
- "b8827ecbec9e44f09216f62c1ee12840",
- "8728bb32478240b7abe576375b01e640",
- "19b8a530352a423794395979e6415bea",
- "5fc8ec04870e4bb7b63b20844c355ab6"
- ]
- },
- "id": "fe2AE-2xqKHx",
- "outputId": "1d8321b3-4f41-4d71-e74e-f33f32a7b261"
- },
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Using custom data configuration tr-ad9f7b76efa9f3a0\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Downloading and preparing dataset common_voice/tr (download: 592.09 MiB, generated: 2.89 MiB, post-processed: Unknown size, total: 594.98 MiB) to /root/.cache/huggingface/datasets/common_voice/tr-ad9f7b76efa9f3a0/6.1.0/f7a9d973839b7706e9e281c19b7e512f31badf3c0fdbd21c671f3c4bf9acf3b9...\n"
- ]
- },
- {
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "54097a6c744849128d7cc3da8aad6609",
- "version_major": 2,
- "version_minor": 0
- },
- "text/plain": [
- "0 examples [00:00, ? examples/s]"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "b8827ecbec9e44f09216f62c1ee12840",
- "version_major": 2,
- "version_minor": 0
- },
- "text/plain": [
- "0 examples [00:00, ? examples/s]"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "8728bb32478240b7abe576375b01e640",
- "version_major": 2,
- "version_minor": 0
- },
- "text/plain": [
- "0 examples [00:00, ? examples/s]"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "19b8a530352a423794395979e6415bea",
- "version_major": 2,
- "version_minor": 0
- },
- "text/plain": [
- "0 examples [00:00, ? examples/s]"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "5fc8ec04870e4bb7b63b20844c355ab6",
- "version_major": 2,
- "version_minor": 0
- },
- "text/plain": [
- "0 examples [00:00, ? examples/s]"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Dataset common_voice downloaded and prepared to /root/.cache/huggingface/datasets/common_voice/tr-ad9f7b76efa9f3a0/6.1.0/f7a9d973839b7706e9e281c19b7e512f31badf3c0fdbd21c671f3c4bf9acf3b9. Subsequent calls will reuse this data.\n"
- ]
- }
- ],
- "source": [
- "common_voice_test_transcription = load_dataset(\"common_voice\", \"tr\", data_dir=\"./cv-corpus-6.1-2020-12-11\", split=\"test\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "epu8kCQZ3h70"
- },
- "source": [
- "\n",
- "Finally, we can decode the example."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "background_save": true
- },
- "id": "K4xWqmk_qMn0",
- "outputId": "d9e40b3c-f02a-48a1-d081-6d7e8b37dcaf"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Prediction:\n",
- "ha ta küçük şeyleri için bir büyük biş şeylir koğoluyor ve yeneküçük şeyler için bir birmizi incilkiyoruz\n",
- "\n",
- "Reference:\n",
- "hayatta küçük şeyleri kovalıyor ve yine küçük şeyler için birbirimizi incitiyoruz.\n"
- ]
- }
- ],
- "source": [
- "print(\"Prediction:\")\n",
- "print(processor.decode(pred_ids))\n",
- "\n",
- "print(\"\\nReference:\")\n",
- "print(common_voice_test_transcription[0][\"sentence\"].lower())"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "HwhyoMml3oOT"
- },
- "source": [
- "Alright! The transcription can definitely be recognized from our prediction, but it is not perfect yet. Training the model a bit longer, spending more time on the data preprocessing, and especially using a language model for decoding would certainly improve the model's overall performance.\n",
- "\n",
- "For a demonstration model on a low-resource language, the results are quite acceptable however 🤗."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "uA4FF5fTZCK_"
- },
- "source": [
- "## Decoding with an n-gram language model"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "W2jMmdD6-54j",
- "outputId": "647599e9-b7b8-41ec-87d3-9012cfef6994"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "/content/drive/MyDrive/xls-r-ngram\n"
- ]
- }
- ],
- "source": [
- "%cd /content/drive/MyDrive/xls-r-ngram/"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "RZZmQONlQQAV",
- "outputId": "c5fa65f7-4d55-4c7b-c213-718db9a8ba0c"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "--2024-04-09 07:52:18-- https://kheafield.com/code/kenlm.tar.gz\n",
- "Resolving kheafield.com (kheafield.com)... 35.196.63.85\n",
- "Connecting to kheafield.com (kheafield.com)|35.196.63.85|:443... connected.\n",
- "HTTP request sent, awaiting response... 200 OK\n",
- "Length: 491888 (480K) [application/x-gzip]\n",
- "Saving to: ‘STDOUT’\n",
- "\n",
- "- 100%[===================>] 480.36K 222KB/s in 2.2s \n",
- "\n",
- "2024-04-09 07:52:20 (222 KB/s) - written to stdout [491888/491888]\n",
- "\n",
- "\u001b[0mCMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):\n",
- " Compatibility with CMake < 3.5 will be removed from a future version of\n",
- " CMake.\n",
- "\n",
- " Update the VERSION argument value or use a ... suffix to tell\n",
- " CMake that the project does not need compatibility with older versions.\n",
- "\n",
- "\u001b[0m\n",
- "-- The C compiler identification is GNU 11.4.0\n",
- "-- The CXX compiler identification is GNU 11.4.0\n",
- "-- Detecting C compiler ABI info\n",
- "-- Detecting C compiler ABI info - done\n",
- "-- Check for working C compiler: /usr/bin/cc - skipped\n",
- "-- Detecting C compile features\n",
- "-- Detecting C compile features - done\n",
- "-- Detecting CXX compiler ABI info\n",
- "-- Detecting CXX compiler ABI info - done\n",
- "-- Check for working CXX compiler: /usr/bin/c++ - skipped\n",
- "-- Detecting CXX compile features\n",
- "-- Detecting CXX compile features - done\n",
- "-- Could NOT find Eigen3 (missing: Eigen3_DIR)\n",
- "-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.74.0/BoostConfig.cmake (found suitable version \"1.74.0\", minimum required is \"1.41.0\") found components: program_options system thread unit_test_framework \n",
- "-- Found Threads: TRUE \n",
- "-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version \"1.2.11\") \n",
- "-- Found BZip2: /usr/lib/x86_64-linux-gnu/libbz2.so (found version \"1.0.8\") \n",
- "-- Looking for BZ2_bzCompressInit\n",
- "-- Looking for BZ2_bzCompressInit - found\n",
- "-- Looking for lzma_auto_decoder in /usr/lib/x86_64-linux-gnu/liblzma.so\n",
- "-- Looking for lzma_auto_decoder in /usr/lib/x86_64-linux-gnu/liblzma.so - found\n",
- "-- Looking for lzma_easy_encoder in /usr/lib/x86_64-linux-gnu/liblzma.so\n",
- "-- Looking for lzma_easy_encoder in /usr/lib/x86_64-linux-gnu/liblzma.so - found\n",
- "-- Looking for lzma_lzma_preset in /usr/lib/x86_64-linux-gnu/liblzma.so\n",
- "-- Looking for lzma_lzma_preset in /usr/lib/x86_64-linux-gnu/liblzma.so - found\n",
- "-- Found LibLZMA: /usr/lib/x86_64-linux-gnu/liblzma.so (found version \"5.2.5\") \n",
- "-- Looking for clock_gettime in rt\n",
- "-- Looking for clock_gettime in rt - found\n",
- "-- Configuring done (6.5s)\n",
- "-- Generating done (1.7s)\n",
- "-- Build files have been written to: /content/drive/MyDrive/xls-r-ngram/kenlm/build\n",
- "[ 0%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/bignum-dtoa.cc.o\u001b[0m\n",
- "[ 0%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/bignum.cc.o\u001b[0m\n",
- "[ 1%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/cached-powers.cc.o\u001b[0m\n",
- "[ 2%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/diy-fp.cc.o\u001b[0m\n",
- "[ 3%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/double-conversion.cc.o\u001b[0m\n",
- "[ 5%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/fast-dtoa.cc.o\u001b[0m\n",
- "[ 6%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/fixed-dtoa.cc.o\u001b[0m\n",
- "[ 8%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/strtod.cc.o\u001b[0m\n",
- "[ 8%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/chain.cc.o\u001b[0m\n",
- "[ 10%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/count_records.cc.o\u001b[0m\n",
- "[ 11%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/io.cc.o\u001b[0m\n",
- "[ 12%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/line_input.cc.o\u001b[0m\n",
- "[ 13%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/multi_progress.cc.o\u001b[0m\n",
- "[ 15%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/rewindable_stream.cc.o\u001b[0m\n",
- "[ 16%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/bit_packing.cc.o\u001b[0m\n",
- "[ 17%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/ersatz_progress.cc.o\u001b[0m\n",
- "[ 18%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/exception.cc.o\u001b[0m\n",
- "[ 20%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/file.cc.o\u001b[0m\n",
- "[ 21%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/file_piece.cc.o\u001b[0m\n",
- "[ 22%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/float_to_string.cc.o\u001b[0m\n",
- "[ 23%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/integer_to_string.cc.o\u001b[0m\n",
- "[ 25%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/mmap.cc.o\u001b[0m\n",
- "[ 26%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/murmur_hash.cc.o\u001b[0m\n",
- "[ 27%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/parallel_read.cc.o\u001b[0m\n",
- "[ 28%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/pool.cc.o\u001b[0m\n",
- "[ 30%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/read_compressed.cc.o\u001b[0m\n",
- "[ 31%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/scoped.cc.o\u001b[0m\n",
- "[ 32%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/spaces.cc.o\u001b[0m\n",
- "[ 33%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/string_piece.cc.o\u001b[0m\n",
- "[ 35%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/usage.cc.o\u001b[0m\n",
- "[ 36%] \u001b[32m\u001b[1mLinking CXX static library ../lib/libkenlm_util.a\u001b[0m\n",
- "[ 36%] Built target kenlm_util\n",
- "[ 37%] \u001b[32mBuilding CXX object util/CMakeFiles/probing_hash_table_benchmark.dir/probing_hash_table_benchmark_main.cc.o\u001b[0m\n",
- "[ 38%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/bhiksha.cc.o\u001b[0m\n",
- "[ 40%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/binary_format.cc.o\u001b[0m\n",
- "[ 41%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/config.cc.o\u001b[0m\n",
- "[ 42%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/lm_exception.cc.o\u001b[0m\n",
- "[ 43%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/model.cc.o\u001b[0m\n",
- "[ 45%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/quantize.cc.o\u001b[0m\n",
- "[ 46%] \u001b[32m\u001b[1mLinking CXX executable ../bin/probing_hash_table_benchmark\u001b[0m\n",
- "[ 46%] Built target probing_hash_table_benchmark\n",
- "[ 47%] \u001b[32mBuilding CXX object lm/filter/CMakeFiles/kenlm_filter.dir/arpa_io.cc.o\u001b[0m\n",
- "[ 48%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/read_arpa.cc.o\u001b[0m\n",
- "[ 50%] \u001b[32mBuilding CXX object lm/filter/CMakeFiles/kenlm_filter.dir/phrase.cc.o\u001b[0m\n",
- "[ 51%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/search_hashed.cc.o\u001b[0m\n",
- "[ 52%] \u001b[32mBuilding CXX object lm/filter/CMakeFiles/kenlm_filter.dir/vocab.cc.o\u001b[0m\n",
- "[ 53%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/search_trie.cc.o\u001b[0m\n",
- "[ 55%] \u001b[32m\u001b[1mLinking CXX static library ../../lib/libkenlm_filter.a\u001b[0m\n",
- "[ 55%] Built target kenlm_filter\n",
- "[ 56%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/sizes.cc.o\u001b[0m\n",
- "[ 57%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/trie.cc.o\u001b[0m\n",
- "[ 58%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/trie_sort.cc.o\u001b[0m\n",
- "[ 60%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/value_build.cc.o\u001b[0m\n",
- "[ 61%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/virtual_interface.cc.o\u001b[0m\n",
- "[ 62%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/vocab.cc.o\u001b[0m\n",
- "[ 63%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/common/model_buffer.cc.o\u001b[0m\n",
- "[ 65%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/common/print.cc.o\u001b[0m\n",
- "[ 66%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/common/renumber.cc.o\u001b[0m\n",
- "[ 67%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/common/size_option.cc.o\u001b[0m\n",
- "[ 68%] \u001b[32m\u001b[1mLinking CXX static library ../lib/libkenlm.a\u001b[0m\n",
- "[ 68%] Built target kenlm\n",
- "[ 70%] \u001b[32mBuilding CXX object lm/CMakeFiles/query.dir/query_main.cc.o\u001b[0m\n",
- "[ 71%] \u001b[32mBuilding CXX object lm/CMakeFiles/fragment.dir/fragment_main.cc.o\u001b[0m\n",
- "[ 72%] \u001b[32m\u001b[1mLinking CXX executable ../bin/fragment\u001b[0m\n",
- "[ 72%] Built target fragment\n",
- "[ 73%] \u001b[32mBuilding CXX object lm/CMakeFiles/build_binary.dir/build_binary_main.cc.o\u001b[0m\n",
- "[ 75%] \u001b[32m\u001b[1mLinking CXX executable ../bin/query\u001b[0m\n",
- "[ 75%] Built target query\n",
- "[ 76%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm_benchmark.dir/kenlm_benchmark_main.cc.o\u001b[0m\n",
- "[ 77%] \u001b[32m\u001b[1mLinking CXX executable ../bin/build_binary\u001b[0m\n",
- "[ 77%] Built target build_binary\n",
- "[ 78%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/adjust_counts.cc.o\u001b[0m\n",
- "[ 80%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/corpus_count.cc.o\u001b[0m\n",
- "[ 81%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/initial_probabilities.cc.o\u001b[0m\n",
- "[ 82%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/interpolate.cc.o\u001b[0m\n",
- "[ 83%] \u001b[32m\u001b[1mLinking CXX executable ../bin/kenlm_benchmark\u001b[0m\n",
- "[ 83%] Built target kenlm_benchmark\n",
- "[ 85%] \u001b[32mBuilding CXX object lm/filter/CMakeFiles/filter.dir/filter_main.cc.o\u001b[0m\n",
- "[ 86%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/output.cc.o\u001b[0m\n",
- "[ 87%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/pipeline.cc.o\u001b[0m\n",
- "[ 88%] \u001b[32m\u001b[1mLinking CXX executable ../../bin/filter\u001b[0m\n",
- "[ 88%] Built target filter\n",
- "[ 90%] \u001b[32mBuilding CXX object lm/filter/CMakeFiles/phrase_table_vocab.dir/phrase_table_vocab_main.cc.o\u001b[0m\n",
- "[ 91%] \u001b[32m\u001b[1mLinking CXX static library ../../lib/libkenlm_builder.a\u001b[0m\n",
- "[ 91%] Built target kenlm_builder\n",
- "[ 92%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/lmplz.dir/lmplz_main.cc.o\u001b[0m\n",
- "[ 93%] \u001b[32m\u001b[1mLinking CXX executable ../../bin/phrase_table_vocab\u001b[0m\n",
- "[ 93%] Built target phrase_table_vocab\n",
- "[ 95%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/count_ngrams.dir/count_ngrams_main.cc.o\u001b[0m\n",
- "[ 96%] \u001b[32m\u001b[1mLinking CXX executable ../../bin/lmplz\u001b[0m\n",
- "[ 96%] Built target lmplz\n",
- "[ 97%] \u001b[32m\u001b[1mLinking CXX executable ../../bin/count_ngrams\u001b[0m\n",
- "[ 97%] Built target count_ngrams\n"
- ]
- }
- ],
- "source": [
- "!wget -O - https://kheafield.com/code/kenlm.tar.gz | tar xz\n",
- "!mkdir kenlm/build && cd kenlm/build && cmake .. && make -j2\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "uME_zJUaQrlw",
- "outputId": "53b9e8b6-f3e7-42a0-cccf-96844293deac"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "build_binary filter\tkenlm_benchmark phrase_table_vocab\t query\n",
- "count_ngrams fragment\tlmplz\t\t probing_hash_table_benchmark\n"
- ]
- }
- ],
- "source": [
- "!ls kenlm/build/bin\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "npRCvP_sQyjF"
- },
- "outputs": [],
- "source": [
- "from transformers import AutoFeatureExtractor, AutoTokenizer, pipeline\n",
- "from datasets import Audio, Dataset, DatasetDict, load_dataset, load_metric\n",
- "\n",
- "import re\n",
- "import string\n",
- "import unidecode"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 860,
- "referenced_widgets": [
- "28b2dc34adf348e0b5dafa310c57f826",
- "a727d871ca474b0baa0a58b0ccd02ee3",
- "98d862eaabf54023b2f1305c6cab436b",
- "775df22308a94344a80e5a560ef768e0",
- "29465d2def27485cb7554b817601ee51",
- "9c876df7df4d499ea15f03288d92d303",
- "bae694008d9147cfb8ce0d60c6de1d48",
- "7b41651175464f2aa772a627d426a612",
- "5b306593954d43afbf06e5df6603648a",
- "0495bf9859c346d49249377233ac8acd",
- "29582652cf6f441ba69a422f61526c7c",
- "34ff5168c77f4faabcb35ff2bf6721a5",
- "df31d4419d794f5ea8fd3502f7b6a671",
- "f78e0d05b26b4ab9b506a9f1bbfb3ae4",
- "820999054bc744098a1e64246353468a",
- "2c74b94daea047a2975a96881a2f2b93",
- "97cd31f632d54fd793da8fe83222fa72",
- "93940c1d722349f98e215bddb698e661",
- "f00f3944409a4dad8755b465f8fc0409",
- "dffedff0181e4d958cbf1ab851be7bdc",
- "0e3574655f8448acb4c614c271f0bc5a",
- "a59e622148aa4963ab73ce835bcf7a22",
- "3c00e15e41104bb3b0474c5d15a7e3c5",
- "f6b7262455fd4e83b0a53de39fc81cc1",
- "01ba34d2dfb14e5e9c064a320dc6ba9b",
- "d83a8ffbf4ef465281d8519d432bb6a3",
- "ba1c82af79f54a119be6de163df40794",
- "4a2b25d243c74cb7bef3054b4c6e51d0",
- "210d015bd88d4e27b0619588a8656c6a",
- "ae2e810f51ca4f8f9afc7fab851ab755",
- "bf93b72434ae45c6a1ccafbc2bfdd84d",
- "1ec41ea5898b44a088b83857a2acc540",
- "e0488a4f58004a78b7df6655d6518ef8",
- "9bfa2700b6454d1eb4b01f1ada5ae0c7",
- "8ae2208c97564bd68ce3120ceee48207",
- "784580af0ef14d13bdf157e956ba2650",
- "72b63ca10e7e4a82999315342a10ee98",
- "47bcf08d66964cbc97c07c9f6a9ce7c3",
- "7a57dbcac48d4c3c9c67187e2cfc5826",
- "3658cb3637a644acbd6f9e0f79a61bde",
- "34278d6826a74ee0aa57ba0db5139384",
- "c1aeb1a87eb24c4e952a192ae2ed7a87",
- "679c9aff551e4a0eb7c0bad1e889ed80",
- "6c031abafe2c4072ab812e16d9bbbe80",
- "c056f785e72b4808ace510d815ed2ef6",
- "ef9830a8b1214f3cba4c207dd5adfebe",
- "bde1d5d9004f427fbc64b2244653328e",
- "882a78617e2949d4af5f9dd3683a024e",
- "702f788a0296474a8da0a90f12c95acf",
- "02f980bcc41d421aa150a8f57c422f28",
- "3f6d6ce6876c4d1486d7e02a92da4fd0",
- "9913f639042442308a9fbda1230ae716",
- "2e3d3ffab9214267bec9d0d0b2a351f6",
- "4dfbc256d8a04f2897f9b9522b8bbccd",
- "b4ff50380a6e4603aa4bd84018e3435d",
- "2a8a56ae255e485e950610db2a5440de",
- "cea7c8aa9c9f4ba3a887827e8d53e1a7",
- "71a71c475804431f9a15b4b4a16da917",
- "98551a90a93d47d7b83bc34cb3be4171",
- "d4a88a5241f04f159b76b245d96edba6",
- "c8aaa63bc35742db84eb5fac3fd47431",
- "3f3fab267c694ed8843b89d05e868d09",
- "606a1e22dd07406ab0cf9a655f593d39",
- "3459c121895c479bbb34becc62139772",
- "dc7a504ce77241a090636f69eb7cf689",
- "9693b79dc87b46fbbf2136df379d7243",
- "64b1f381ca29495eb050391410a3f9be",
- "d259904aa66c4abd836c1184379cc638",
- "ba3d11c972344c91a772848b625f1986",
- "7f09f2af0f054878912f6cfdaacd7280",
- "1c0f2eb6db78494d82dfe4231c18a186",
- "06d30c66375c40c19de44739bf04749f",
- "3cf3320281a148cab15db43d85f17c56",
- "6900dc72b64f4549b7ab2295fc82df3c",
- "8d42130cb8c649419285f937c11f1ca7",
- "22b08e38d2774e23bd6ed685aaa3325d",
- "9d92a1df1518487eb07929eecb3ff5c6",
- "58e5dd33cb034ce1821f49d3f881ae4a",
- "8fc270ccfef74a6683ca447d452ffae1",
- "a86ee875cabc4bb4b385f7a4f6ea3bea",
- "738fa9776faa450d86c284fd552f4495",
- "f6275bd987b74a488e028999bab678fd",
- "fb39efd6028040b4b6c9a122efef4e52",
- "ae1b4998b1de4dbea4822413da7a317f",
- "be51561e0d074258b44db0d8c17fb6a6",
- "445a6b206101449ea7ee7f1a197fc16e",
- "164bb32dc141469383b169f761e86f89",
- "148ec733c438472586b92583b5a33551",
- "585839d79bc74f378b07413bb09e0118",
- "ebf1cecb1da442f9b08569e89d193a9a",
- "ab10de32b94141ed9fe542a520c30df8",
- "274d029284724039a2ee938a04fdcc0d",
- "967deb198fdf445db6a29d4fd56f1a98",
- "6a92bc0ecf65468a9b6fd84b790c4c8c",
- "d11093621ef84fd8a30dffb849dfe410",
- "93717ea4078b4a7cb84e64d5fdb82b14",
- "9a77dcf81c7748c2923f6a90b29f85d4",
- "4eb3a5027a774be59dd87ece2136cb68",
- "f4134b064315427d89d0ca2a459982a0",
- "3ca06f5867264f91bb82ff28e0b4cb8d",
- "8c9b1ec5e84f45738932374e05db402a",
- "e6c7cf94d4934fecb8e11ad584926790",
- "1b624e661338414683fdc0fb3dc5d35e",
- "15aac68bb6c141e6901faaf17918e790",
- "612c3770319943069300bd833ec187bf",
- "b893141d3abe411aa0450ac7a73a1c13",
- "1375b35aefa94d53934444b0030d7e16",
- "71093adbfc874533b6c8a1973ba35ea6",
- "9ebbc43c1174436ba4d3f448c80e3006",
- "9ce8255f48104a348efd8a4215b414cb"
- ]
- },
- "id": "M1YRPnTZaFHW",
- "outputId": "4c760339-209b-4115-a5cf-0ba6370b5565"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stderr",
- "text": [
- "/usr/local/lib/python3.10/dist-packages/datasets/load.py:2516: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0.\n",
- "You can remove this warning by passing 'token=' instead.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/datasets/load.py:1461: FutureWarning: The repository for mozilla-foundation/common_voice_8_0 contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/mozilla-foundation/common_voice_8_0\n",
- "You can avoid this message in future by passing the argument `trust_remote_code=True`.\n",
- "Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.\n",
- " warnings.warn(\n"
- ]
- },
- {
- "output_type": "display_data",
- "data": {
- "text/plain": [
- "Downloading builder script: 0%| | 0.00/11.5k [00:00‘\", # after o/g indicate modified char\n",
- " unidecode.unidecode(batch[\"sentence\"]).lower()\n",
- " )\n",
- " )\n",
- " ) + \" \"\n",
- " return batch"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 49,
- "referenced_widgets": [
- "4b911fa55e3a42a69e3b541605ae718a",
- "a766ecd2e8a541f78d0a4254e5cf220f",
- "b85031c1f3c44b3ea83773cc89d1f998",
- "1c9791c7054f475da13c42e59b165abd",
- "7f98a486c3d14e1d96a5b8e7fc96b904",
- "0b18758131744afca3c6bf5abb780bab",
- "73d84ac66fef4483b1730d1896987a21",
- "df9b2e33aeae47a9952c6805e03351b6",
- "d0d48aafc16d4f98ab678ab10bb62872",
- "17947db63a384bf49204a568350acdeb",
- "362d9aa00bcf487c92c5ca887f024437"
- ]
- },
- "id": "4qTL2Fc0cbMq",
- "outputId": "aa5f0bcd-6033-4fee-bb6c-3653c778ee7d"
- },
- "outputs": [
- {
- "output_type": "display_data",
- "data": {
- "text/plain": [
- "Map: 0%| | 0/39456 [00:00 $ngram_data"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "po4zWZRjdOGc",
- "outputId": "d54de9c7-92c5-469b-c6f1-cbf35a8a18d9"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "\\data\\\n",
- "ngram 1=98004\n",
- "ngram 2=539023\n",
- "ngram 3=719386\n",
- "ngram 4=793206\n",
- "ngram 5=850589\n",
- "\n",
- "\\1-grams:\n",
- "-5.7507653\t\t0\n",
- "0\t\t-0.10007139\n",
- "-5.608028\tshartmidi\t-0.10007139\n",
- "-3.97965\tdeyman\t-0.12756462\n",
- "-3.4230683\tkeyingi\t-0.20062704\n",
- "-4.135141\tvoqealar\t-0.17204677\n",
- "-2.7541177\tbiz\t-0.29683134\n",
- "-3.5234666\tshunchaki\t-0.12191775\n",
- "-3.4540253\tqo‘l\t-0.18548343\n",
- "-3.53991\tberib\t-0.12553446\n",
- "-4.464714\tketadigan\t-0.10406738\n",
- "-2.817535\temas\t-0.23384908\n"
- ]
- }
- ],
- "source": [
- "!head -20 $ngram_data"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "55U0j6JCdbT8"
- },
- "outputs": [],
- "source": [
- "corrected_data = \"xls-r-uzbek-cv8/5gram_correct.arpa\"\n",
- "with open(ngram_data, \"r\") as read_file, open(corrected_data, \"w\") as write_file:\n",
- " has_added_eos = False\n",
- " for line in read_file:\n",
- " if not has_added_eos and \"ngram 1=\" in line:\n",
- " count=line.strip().split(\"=\")[-1]\n",
- " write_file.write(line.replace(f\"{count}\", f\"{int(count)+1}\"))\n",
- " elif not has_added_eos and \"\" in line:\n",
- " write_file.write(line)\n",
- " write_file.write(line.replace(\"\", \"\"))\n",
- " has_added_eos = True\n",
- " else:\n",
- " write_file.write(line)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "gVUKO_KDearL",
- "outputId": "e171bcbb-0862-4be6-e13f-4df86496f36d"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "\\data\\\n",
- "ngram 1=98005\n",
- "ngram 2=539023\n",
- "ngram 3=719386\n",
- "ngram 4=793206\n",
- "ngram 5=850589\n",
- "\n",
- "\\1-grams:\n",
- "-5.7507653\t\t0\n",
- "0\t\t-0.10007139\n",
- "0\t\t-0.10007139\n",
- "-5.608028\tshartmidi\t-0.10007139\n",
- "-3.97965\tdeyman\t-0.12756462\n",
- "-3.4230683\tkeyingi\t-0.20062704\n",
- "-4.135141\tvoqealar\t-0.17204677\n",
- "-2.7541177\tbiz\t-0.29683134\n",
- "-3.5234666\tshunchaki\t-0.12191775\n",
- "-3.4540253\tqo‘l\t-0.18548343\n",
- "-3.53991\tberib\t-0.12553446\n",
- "-4.464714\tketadigan\t-0.10406738\n"
- ]
- }
- ],
- "source": [
- "!head -20 $corrected_data"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "sSpNeQ_bel36",
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "outputId": "bb02cee7-c9cd-4b86-9b33-eb5dddc68191"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stderr",
- "text": [
- "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
- ]
- }
- ],
- "source": [
- "from transformers import AutoProcessor\n",
- "\n",
- "processor = AutoProcessor.from_pretrained(\"./xls-r-uzbek-cv8/\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "ms020VOvlmYr",
- "outputId": "fb9084c6-14dc-432c-8e52-e5244c73746c"
- },
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- "['|',\n",
- " 'a',\n",
- " 'b',\n",
- " 'c',\n",
- " 'd',\n",
- " 'e',\n",
- " 'f',\n",
- " 'g',\n",
- " 'h',\n",
- " 'i',\n",
- " 'j',\n",
- " 'k',\n",
- " 'l',\n",
- " 'm',\n",
- " 'n',\n",
- " 'o',\n",
- " 'p',\n",
- " 'q',\n",
- " 'r',\n",
- " 's',\n",
- " 't',\n",
- " 'u',\n",
- " 'v',\n",
- " 'w',\n",
- " 'x',\n",
- " 'y',\n",
- " 'z',\n",
- " '‘',\n",
- " '’',\n",
- " '[UNK]',\n",
- " '[PAD]',\n",
- " '',\n",
- " '']"
- ]
- },
- "metadata": {},
- "execution_count": 31
- }
- ],
- "source": [
- "vocab_dict =processor.tokenizer.get_vocab()\n",
- "labels = [k for k, v in sorted(vocab_dict.items(), key=lambda x: x[1])]\n",
- "labels"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "zUZ8MQ2rmFT6",
- "outputId": "93669dd7-9ca8-41ea-ca81-1b3f39031df1"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stderr",
- "text": [
- "WARNING:pyctcdecode.alphabet:Found entries of length > 1 in alphabet. This is unusual unless style is BPE, but the alphabet was not recognized as BPE type. Is this correct?\n"
- ]
- }
- ],
- "source": [
- "from pyctcdecode import build_ctcdecoder\n",
- "\n",
- "decoder = build_ctcdecoder(labels=labels, kenlm_model_path=corrected_data)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "PTNuGmy9nFRr"
- },
- "outputs": [],
- "source": [
- "from transformers import Wav2Vec2ProcessorWithLM\n",
- "\n",
- "processor_with_lm = Wav2Vec2ProcessorWithLM(\n",
- " feature_extractor=processor.feature_extractor,\n",
- " tokenizer=processor.tokenizer,\n",
- " decoder = decoder,\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "import shutil\n",
- "shutil.rmtree(\"xls-r-uzbek-cv8/language_model\")\n"
- ],
- "metadata": {
- "id": "vJGiEATd7vRG"
- },
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "2GuhuK11nlBe"
- },
- "outputs": [],
- "source": [
- "processor_with_lm.save_pretrained(\"xls-r-uzbek-cv8\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "jrV4hXy3oEJI",
- "outputId": "c5286483-8643-4831-d3cc-0ca129e17957"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "total 144M\n",
- "-rw------- 1 root root 143M Apr 9 08:12 5gram_correct.arpa\n",
- "-rw------- 1 root root 78 Apr 9 08:17 attrs.json\n",
- "-rw------- 1 root root 1.1M Apr 9 08:17 unigrams.txt\n"
- ]
- }
- ],
- "source": [
- "#!rm -r xls-r-uzbek-cv8/language_model/\n",
- "!ls -alh xls-r-uzbek-cv8/language_model/"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "GxIaO9JLAk8A",
- "outputId": "02930a5f-f20e-45b1-c599-b1f2d8ca6d58"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "Reading xls-r-uzbek-cv8/language_model/5gram_correct.arpa\n",
- "----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100\n",
- "****************************************************************************************************\n",
- "SUCCESS\n",
- "total 67M\n",
- "-rw------- 1 root root 66M Apr 9 08:17 5gram.bin\n",
- "-rw------- 1 root root 78 Apr 9 08:17 attrs.json\n",
- "-rw------- 1 root root 1.1M Apr 9 08:17 unigrams.txt\n"
- ]
- }
- ],
- "source": [
- "!kenlm/build/bin/build_binary xls-r-uzbek-cv8/language_model/5gram_correct.arpa xls-r-uzbek-cv8/language_model/5gram.bin\n",
- "!rm xls-r-uzbek-cv8/language_model/5gram_correct.arpa\n",
- "!ls -alh xls-r-uzbek-cv8/language_model/"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "WigwTkyWoJgg",
- "outputId": "abb946b9-8f9b-4b26-f747-fb23e9eef954"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "python3: can't open file '/content/drive/MyDrive/xls-r-ngram/./eval.py': [Errno 2] No such file or directory\n"
- ]
- }
- ],
- "source": [
- "!python ./eval.py --model_id . --dataset mozilla-foundation/common_voice_8_0 --config uz --split \"test\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "29B3FYPhyHST",
- "outputId": "b11cc6a3-4555-4c16-c97a-c64f8a460c6a"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "[Errno 2] No such file or directory: 'xls-r-uzbek-cv8/'\n",
- "/content\n"
- ]
- }
- ],
- "source": [
- "%cd xls-r-uzbek-cv8/"
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "!git remote remove origin\n"
- ],
- "metadata": {
- "id": "RHzKe8cX_4TG"
- },
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "code",
- "source": [
- "!git remote add origin https://huggingface.co/zohirjonsharipov/xls-r-uzbek-cv8\n"
- ],
- "metadata": {
- "id": "3O5oAnL__8FC"
- },
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "W8DN9SZ-Errc"
- },
- "outputs": [],
- "source": [
- "!git add eval.py preprocessor_config.json tokenizer_config.json language_model/"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "sLIBafqmEyq6"
- },
- "outputs": [],
- "source": [
- "!git add alphabet.json"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "WnQl8yyuFg8u",
- "outputId": "8e559cfd-2389-4281-d0ae-ed704436c590"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "On branch main\n",
- "Changes not staged for commit:\n",
- " (use \"git add ...\" to update what will be committed)\n",
- " (use \"git restore ...\" to discard changes in working directory)\n",
- "\t\u001b[31mmodified: .ipynb_checkpoints/eval-checkpoint.py\u001b[m\n",
- "\t\u001b[31mmodified: .ipynb_checkpoints/run_speech_recognition_ctc-checkpoint.py\u001b[m\n",
- "\t\u001b[31mmodified: added_tokens.json\u001b[m\n",
- "\t\u001b[31mmodified: log_mozilla-foundation_common_voice_8_0_uz_test[_500]_predictions.txt\u001b[m\n",
- "\t\u001b[31mmodified: log_mozilla-foundation_common_voice_8_0_uz_test[_500]_targets.txt\u001b[m\n",
- "\t\u001b[31mmodified: mozilla-foundation_common_voice_8_0_uz_test[_500]_eval_results.txt\u001b[m\n",
- "\t\u001b[31mmodified: requirements.txt\u001b[m\n",
- "\t\u001b[31mmodified: run_speech_recognition_ctc.py\u001b[m\n",
- "\t\u001b[31mmodified: runs/Jan27_22-59-08_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/1643325211.6916795/events.out.tfevents.1643325211.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.399095.1\u001b[m\n",
- "\t\u001b[31mmodified: runs/Jan27_22-59-08_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/events.out.tfevents.1643325211.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.399095.0\u001b[m\n",
- "\t\u001b[31mmodified: runs/Jan28_04-57-04_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/1643346306.8664992/events.out.tfevents.1643346306.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.541469.1\u001b[m\n",
- "\t\u001b[31mmodified: runs/Jan28_04-57-04_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/events.out.tfevents.1643346306.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.541469.0\u001b[m\n",
- "\t\u001b[31mmodified: runs/Jan30_19-35-25_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/1643572438.487491/events.out.tfevents.1643572438.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2037878.1\u001b[m\n",
- "\t\u001b[31mmodified: runs/Jan30_19-35-25_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/events.out.tfevents.1643572438.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2037878.0\u001b[m\n",
- "\t\u001b[31mmodified: runs/Jan31_00-08-55_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/1643588110.005454/events.out.tfevents.1643588110.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2141134.1\u001b[m\n",
- "\t\u001b[31mmodified: runs/Jan31_00-08-55_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/events.out.tfevents.1643588109.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2141134.0\u001b[m\n",
- "\t\u001b[31mmodified: runs/Jan31_05-52-36_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/1643608732.4243534/events.out.tfevents.1643608732.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2278718.1\u001b[m\n",
- "\t\u001b[31mmodified: runs/Jan31_05-52-36_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/events.out.tfevents.1643608732.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2278718.0\u001b[m\n",
- "\t\u001b[31mmodified: special_tokens_map.json\u001b[m\n",
- "\t\u001b[31mmodified: vocab.json\u001b[m\n",
- "\n",
- "Untracked files:\n",
- " (use \"git add ...\" to include in what will be committed)\n",
- "\t\u001b[31m5gram.arpa\u001b[m\n",
- "\t\u001b[31m5gram_correct.arpa\u001b[m\n",
- "\t\u001b[31muz_cv8_text.txt\u001b[m\n",
- "\n",
- "no changes added to commit (use \"git add\" and/or \"git commit -a\")\n"
- ]
- }
- ],
- "source": [
- "!git commit -m \"add ngram LM from train+other\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "R5x8nIucGJK3"
- },
- "outputs": [],
- "source": [
- "!git config --global user.name \"Zohirjon Sharipov\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "igL2FAM7F4gL"
- },
- "outputs": [],
- "source": [
- "!git config --global user.email \"zohirbeksharipov97@gmail.com\""
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "!git branch\n"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "I3KPnnxsDKRD",
- "outputId": "9647d5b4-48ed-4186-eeb2-c71f1222688c"
- },
- "execution_count": null,
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "* \u001b[32mmain\u001b[m\n"
- ]
- }
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "!git branch with_ngram_LM\n"
- ],
- "metadata": {
- "id": "pYxjfXGbDWSm"
- },
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "code",
- "source": [
- "!git checkout with_ngram_LM\n"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "VwX-ih5HDYSs",
- "outputId": "ef1cdf58-b3e5-445c-e239-e80eb45c46c6"
- },
- "execution_count": null,
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "M\t.ipynb_checkpoints/eval-checkpoint.py\n",
- "M\t.ipynb_checkpoints/run_speech_recognition_ctc-checkpoint.py\n",
- "M\tadded_tokens.json\n",
- "M\tlog_mozilla-foundation_common_voice_8_0_uz_test[_500]_predictions.txt\n",
- "M\tlog_mozilla-foundation_common_voice_8_0_uz_test[_500]_targets.txt\n",
- "M\tmozilla-foundation_common_voice_8_0_uz_test[_500]_eval_results.txt\n",
- "M\trequirements.txt\n",
- "M\trun_speech_recognition_ctc.py\n",
- "M\truns/Jan27_22-59-08_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/1643325211.6916795/events.out.tfevents.1643325211.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.399095.1\n",
- "M\truns/Jan27_22-59-08_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/events.out.tfevents.1643325211.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.399095.0\n",
- "M\truns/Jan28_04-57-04_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/1643346306.8664992/events.out.tfevents.1643346306.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.541469.1\n",
- "M\truns/Jan28_04-57-04_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/events.out.tfevents.1643346306.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.541469.0\n",
- "M\truns/Jan30_19-35-25_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/1643572438.487491/events.out.tfevents.1643572438.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2037878.1\n",
- "M\truns/Jan30_19-35-25_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/events.out.tfevents.1643572438.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2037878.0\n",
- "M\truns/Jan31_00-08-55_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/1643588110.005454/events.out.tfevents.1643588110.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2141134.1\n",
- "M\truns/Jan31_00-08-55_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/events.out.tfevents.1643588109.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2141134.0\n",
- "M\truns/Jan31_05-52-36_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/1643608732.4243534/events.out.tfevents.1643608732.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2278718.1\n",
- "M\truns/Jan31_05-52-36_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/events.out.tfevents.1643608732.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2278718.0\n",
- "M\tspecial_tokens_map.json\n",
- "M\tvocab.json\n",
- "Switched to branch 'with_ngram_LM'\n",
- "fatal: cannot exec '.git/hooks/post-checkout': Permission denied\n"
- ]
- }
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "!git add .\n",
- "!git commit -m \"add ngram LM from train+other\"\n"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "2N8fin8IDcRw",
- "outputId": "4b708554-2d24-4682-8a6d-eaba0bec3f7c"
- },
- "execution_count": null,
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "fatal: cannot exec '.git/hooks/post-commit': Permission denied\n",
- "[with_ngram_LM 6f8dcad] add ngram LM from train+other\n",
- " 23 files changed, 6000523 insertions(+), 7 deletions(-)\n",
- " mode change 100755 => 100644 .ipynb_checkpoints/eval-checkpoint.py\n",
- " mode change 100755 => 100644 .ipynb_checkpoints/run_speech_recognition_ctc-checkpoint.py\n",
- " create mode 100644 5gram.arpa\n",
- " create mode 100644 5gram_correct.arpa\n",
- " mode change 100755 => 100644 log_mozilla-foundation_common_voice_8_0_uz_test[_500]_predictions.txt\n",
- " mode change 100755 => 100644 log_mozilla-foundation_common_voice_8_0_uz_test[_500]_targets.txt\n",
- " mode change 100755 => 100644 mozilla-foundation_common_voice_8_0_uz_test[_500]_eval_results.txt\n",
- " mode change 100755 => 100644 run_speech_recognition_ctc.py\n",
- " rewrite runs/Jan27_22-59-08_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/1643325211.6916795/events.out.tfevents.1643325211.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.399095.1 (100%)\n",
- " rewrite runs/Jan27_22-59-08_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/events.out.tfevents.1643325211.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.399095.0 (100%)\n",
- " rewrite runs/Jan28_04-57-04_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/1643346306.8664992/events.out.tfevents.1643346306.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.541469.1 (100%)\n",
- " rewrite runs/Jan28_04-57-04_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/events.out.tfevents.1643346306.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.541469.0 (100%)\n",
- " rewrite runs/Jan30_19-35-25_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/1643572438.487491/events.out.tfevents.1643572438.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2037878.1 (100%)\n",
- " rewrite runs/Jan30_19-35-25_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/events.out.tfevents.1643572438.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2037878.0 (100%)\n",
- " rewrite runs/Jan31_00-08-55_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/1643588110.005454/events.out.tfevents.1643588110.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2141134.1 (100%)\n",
- " rewrite runs/Jan31_00-08-55_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/events.out.tfevents.1643588109.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2141134.0 (100%)\n",
- " rewrite runs/Jan31_05-52-36_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/1643608732.4243534/events.out.tfevents.1643608732.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2278718.1 (100%)\n",
- " rewrite runs/Jan31_05-52-36_job-0074bb36-c67f-4775-b1b6-176eb09b0ba4/events.out.tfevents.1643608732.job-0074bb36-c67f-4775-b1b6-176eb09b0ba4.2278718.0 (100%)\n",
- " rewrite special_tokens_map.json (100%)\n",
- " create mode 100644 uz_cv8_text.txt\n"
- ]
- }
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "YBb6acjCGMSt",
- "outputId": "a524741a-701e-4ff9-edf8-cebdd96941ef"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "fatal: cannot exec '.git/hooks/pre-push': Permission denied\n",
- "^C\n"
- ]
- }
- ],
- "source": [
- "!git push origin with_ngram_LM\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 1000,
- "referenced_widgets": [
- "f316ac23a859488a9ce156175dc3c2e5",
- "339d2fab53b64a518c6718188a3586bf",
- "c205fcf09e884eb1b210669703d2bf8f",
- "20785c23987842b18134e7f63bc1dadd",
- "5e78f8fc16554687ac5f727f89aa37e8",
- "0a18d6cbba0d4068b22eb8d93751c485",
- "d10474744e314057bd2f1a7a03d8a406",
- "69fe445d32be484592569f3336e0f0a6",
- "390897fbc79e4a688b724c2ef44c9432",
- "4aef6a137cf3447b93d1e97649b9b98d",
- "778f0ba4f6ad4f828893cb0925d69780",
- "28b1a9e448394bf098cf6c1f6fe0ccec",
- "fc0cd38fceea489dbce68fdaf09e829c",
- "5066bf4ceeb84381b50307cfc7f80980",
- "383eff90207a4f59a227e415cd5816d7",
- "287a0ba17432455aae024785577523b9",
- "f128ff19e6334ac58f33ad46bcf5adde",
- "35628323e2ab436e9cd2af93fca8189a",
- "276a1f1ce70649beba493c196d267adc",
- "d7affc19df15496ca9533910af631fa8",
- "7428e83e6ae245ada4547429a93353a1",
- "7222badd90a64c10bb8e2c8454307598",
- "c199a1877a4f4fad9e7352a50aca950d",
- "c66f4713a30c4511863d789342d3603c",
- "c98c2dd5708d4d848939e86a00d88dc6",
- "1c4500d0ccff4babb5fcffc89dadbda8",
- "fa3b6e5e530d447daf0ecb84d5d4956e",
- "225b2e2c8b06481ba13e50de189fedb8",
- "d9de77f6c0bd416db03513ab22c2f5c8",
- "ae811a7dfb7b4b0bae7bed91e849a272",
- "82fa5c91a8624391a2c16b8db4641592",
- "36fd01da41884bdca3a172fbcf772000",
- "e1f62138314b458687cd1c35aa133b39",
- "b5a80f0f38324ad7acf0f4ac7313de49",
- "d9bda94950f24d74818836effc8e4fd8",
- "a9cc24c2efa54689880054d0eba04b95",
- "015a02e42fd8479eb8c5dd974270e0ba",
- "dc6b228594e04ad09915a16aebe6a731",
- "6cb89fbfaf30421c87159c418f346c4d",
- "ac4225de38a34e408691cb3293297d03",
- "25a20c511c3942ee9b53d1f990f4f7b7",
- "67d2893013d841d893ca7bbb34f40e6c",
- "42f2d773781e44a395c9e32b83d118b1",
- "c960a1875987492eb221827d2d7774ef",
- "1027db5a3dd445eebcca28af6491762d",
- "aea199a2c41b4202a9dbce36bef8e35a",
- "139c32b0312c43c99ecd7d1b778cbf7c",
- "ebaf4da3cf244579a80e54c5c4969a23",
- "e9d3506fc9e945229a6a931e65dba1cd",
- "9cd8688dac4546acbac7935ee58325ba",
- "06f4f294c7504fdf9a1dc7c097bb1062",
- "566437009d904788b04da21c9a78e982",
- "a70fc94adb0f47ddad7dabfe8f6449fd",
- "a74474dc0c21463986ea55863591995e",
- "a7ce4661dd0d4316a7c3b2054acb607a",
- "3d4014296e204b6b9790f70933befaf3",
- "2c5c235b9ac94da4a269b65d7a754f72",
- "f63c5c2af67847fda3c4a05ddb182e42",
- "098ac5b85663415aa29c40a1290fb4c6",
- "916735a174434293b6c4ffd8dad61ac6",
- "85467b05feee490da6183fff1262ec11",
- "e770e00a39ad4e2e9948a31eff502ec6",
- "0b21cfd574b5486d8803520301486d11",
- "452c2f76f5b24ea68a22a0f89f5bd5ca",
- "78b37368449a4f909b5d6c5db75ff08a",
- "401f2b218fdc4b25841fea9872b6440e",
- "46da737e06c44c2c831e3ba536059c90",
- "e439ca8584ce41e39b91fe327cbccbec",
- "7f72c46624024f429d579428caa40c7c",
- "c25caf516590405fbb842a9a5ad095de",
- "7e9c82709be34110827b11a54b6780cb",
- "fa683080683448e69c396e4d5431a9ff",
- "9dd9e51fce54495891110ae98ab147e9",
- "e3435075dbec4d93935aa6025f5c1ef1",
- "0d75ab61bcde4e83b6720b92afaf9dca",
- "331f9e5eafb9493dbc958fd15b5fa30a",
- "bf3a42407e1042e299beab983ab6bdd2",
- "50a26a6fdd5f4095badaa472dc9a7799",
- "cedcea6b427e4955a48f71811ed4bff3",
- "1ed23fe4e12f4432aaa894cd7424360a",
- "0ca95af183f94190a72b02bf3e370abe",
- "45d4827aae9a44fea35be4233756c838",
- "b042fd5238c047f8a94ff3387ca27c90",
- "b761553b0655480a9ee4999b644c340e",
- "eac7f83fb5944d8a9924a42bfe6c6f75",
- "79c9f717d15f4cc5974bb7e7e863e016",
- "02b18f9952d041bc915803223739513f",
- "d56e7a7158a347dbb55e88924ceedef0",
- "ca8b3df6fa104d63b5b53a62dc8b9531",
- "1f03ffa9c65a479d91e5d78a975f2779",
- "af33d8c42c3547c1946f49735a013ac0",
- "816f4852c0f54163a1dc4e8a78afe909",
- "bad4919bdcc7438e928e83ffe717acb4",
- "88ea2bb60723446e88d78370ca8daece",
- "fcd3a035beb341c783928eac9597cb80",
- "63b5fd77f01444a082b97fab35fc2c34",
- "f569f9d3290b4cdb8b9c1b6c50c7c8ed",
- "b598171a01f8494ca4469b8998014e2a",
- "1861e12b5b6646d9940c33da79d262bd",
- "4e6bee0348b641228399488467407534",
- "b5733f20353d4a6fb2f01fb1852d2a13",
- "2d66dd2198cb4417b8bffff155587c29",
- "e0688ed3c2f549af80f45cf2d7001470",
- "792e8c08580b4e0d8a9a2c278ab096d5",
- "ee1bf17c1a9d4e27a69fb7340251150e",
- "348a50765c904eeea19ae5c3e39213da",
- "bc7724d5c2bb4d63ba80c2bb503e5c38",
- "a29cc061f4a54e3baf29ff2c99bcded4",
- "8c0edc5b8ba547829d20003b5d2a499a",
- "ce67287d749d45b0bd8b61d0dd5ae710",
- "7643a1976d844f3eb64a364e2333ea5d",
- "fd5f5e1224c342d994934934b751f482",
- "24da601521fc4cfc8e8b2550e3c9a4ff",
- "3fa24336e5964a9aa1fae1590da6e0d6",
- "49c5a0f77255449593c55ea3df837fd2",
- "6c217b9e30414a2aabec01b4416c5f4e",
- "74d133f23382433aa193769395c2a648",
- "f1de86c5c8ce4c90bce1640390ab5449",
- "6d725ecc270e4385a97feee19d3af108",
- "c30cf8cdbbae4b899c1b2b2cdf52417d",
- "021db95ea39a473fa7f81508679fe329",
- "8a951ee29c3148bf8ca3bf7265538346",
- "319e624dbc6347319847b022f6402890",
- "0a37fa4e7ae64d32903d51f6133662ee",
- "cc0b65d51a484a3687aa657e238dfabb",
- "9e0dffd91b154278a48f915cb0765847",
- "fd8feb9c03b6491bb815b70f18d8bea8",
- "bbde411fb01c492f857c0ac27672d06b",
- "b3e09bf7c325451186c224a61a43adbf",
- "65d44e7adecc4b9898f71e57fbcaec2c",
- "669ecea110fd462b82df158ed374808b",
- "c0937eb068be4fa3afecc3dd09a74f90",
- "4835210355f343a1a4116c1e00616946",
- "d8730f7d7e544c7788a5434ae9ae01d3",
- "3988006596f840379e033fe95095e204",
- "4c20cc41813841cd8d55a162bde021f8",
- "92831f3f308c4f8193b3ce3c70a6b0a7",
- "be0fe51a8e9044efac4c74c76e6901ec",
- "1003f117e73b451b8ed54bba35b2c3da",
- "0e3ee4a0e78545a6a9819d943af1173c",
- "622964bbf4f04a788d6be682209c93fe",
- "520aa82928ed4d7693f7db7387c85528",
- "627e167f33b3449ab26a813329d4ad38",
- "a9c7473923c44b5fbc6294927edc46d0",
- "22d90f43512b4801b7dbff7a5118d383",
- "01cf5627c3614d01aea9a1baa765407d",
- "9fddda4595fa404da837265d457f9a15",
- "623cc031f54f424bb4dbace4b3ee7e92",
- "5d85b6254732449383117709c76e64a7",
- "7b0eb0b12dec46bf80e70d4395b7e501",
- "6b1d8f8d77b1482f9366b212069c038c",
- "b23d9336851849238ef259f7b24bfd22",
- "635c198aeaec42c68517548243574c57",
- "b30030bbe82d49bf931e23cbf0f428eb",
- "6220d83318c34fdb91357da2a0540a4e",
- "a15da1926718415698636086568bf2dc",
- "bb50b56f51f2415c8b2312ef03958b48",
- "045265924e254824aa98c16ef72aec34",
- "4447340c82bb492ea94db957bb7247e4",
- "fe557fe2a11346de8718d6c1bbbeca18",
- "771052b9bf45415c8271f901c883b35e",
- "8962dc90ef8c41579d11a7a6971fca94",
- "fa785a5a9b75436385e3fadfdd9bc41a",
- "cf1089d7363944e39507184c31be6501",
- "203081ef9b6940bfa62d8203f267e1e8",
- "bcf99faf33ba4cf79da9a7337d79e752",
- "696c0bea89394652acc2318d0d8b5030",
- "036b34a7d10548bbb1b2e9c76a21c322",
- "cca3c69fcfc54986a1b2cb8717a74fce",
- "ebcab01e231c4bc9a710fda49c7f578c",
- "cc25c28da2644bd0a4524a2c56561788",
- "1998a7e652d44f8f841f6e9c0fea7c5a",
- "4d164c04d67840bc997f8b5e7b3210a5",
- "f75fe74ec2324e6e9543da2330bfdd5d",
- "5909005a84724f8fa82da3a33178c28f",
- "e9e0d582daa74443997992cd61658a8c",
- "bb2066381def4f8081fb57d81f69aba0",
- "26278139fdeb4ec2b37d2480a3076f9f",
- "7f9b79fe583f4dc8b0e5067c027d5d67",
- "4a4d1d103a1448219dde7cb1071a28ca",
- "544bfece9b3040cfbbf9cca9da09b0ee",
- "796671c340a946709cd8e76e085af3c3",
- "493b166cdb7a485687adc2da4ef411d2",
- "76125d0ad381477399c16a8666763a1c",
- "82f1bd221f2c4da8983fa4ec476058fa",
- "4955f50b60884ba996182a5947d27850",
- "0d76f65887b5492b91489f6ace8a6810",
- "fbbbc7bd23ff4d6ab35ae78801155db0",
- "edaa99542f024b669ce64b66de1db625",
- "4e30b50b41b8427e8e97a463cda5e62c",
- "63a3944f11244b97bfea8a812a0b6757",
- "162ee37d3785481e881576be343d47bb",
- "d286fbaa29ad416b8522e7369479579c",
- "a0fcc8435fca4926bdbe8ffbb56a92d2",
- "f1ad52b89ed14169ab0f6912c6628a62",
- "86c818b35d7941c69a9930bf4069961e",
- "c910706aa43d4383b63fed6bcbe7c5c4",
- "f23c017889cf4ad587278f3bac5336a6",
- "892c70c1da274a7f93f5cc11dfef9b99",
- "57f5208e79c543569782994a2d7c37e3",
- "210dc99e61ab44369df83741b33fba6c",
- "1551d881f5bb41318643f3edaf2f8ce9",
- "4a3cd47c7fd84e7b8a397138edf6674a",
- "c2da0bc59bab470d81c472d9bcf87866",
- "7c8dd7fbe1904f4d83d636f47f11ce33",
- "7a8ad06eee914a77b97ac58cfa2a9583",
- "d12bbb19c98340c48f41574ef3a61a11",
- "6cba85462dd04cf7bd730cc5826d0aaa",
- "b52fe506342844c489f85d1a2737127b",
- "17282bf0c49b4c54b1d1c127c2d0a214",
- "7378b46dbc7e4822a4ec54aae538b545",
- "28a9ad0d91e143ef981e534ddff38b9f",
- "df1d84fcdcd3413cb78c59e9b8f9b4ae",
- "87a11b5fef034c298eeebb642761aca4",
- "780d9b87520e4c9bbb49bcac29fdec24",
- "c83ff669c5644559b28490f63695c455",
- "b6cff2ae73dd46989931528ac6d80969",
- "8f567c9ca49342db8b89dbd66e6cf329",
- "62843d3925a8435c9e963c7468208b03",
- "548c01f0e16a45fda842f5e16553952d",
- "6b1826985fcd4cd8bf5784c3a18cef07",
- "558ad81105fd49459714c7f4a420dbca",
- "8ff1480376a64ee5b95594ff93c584cb",
- "4115a3a61c604a6bbbf4c2d1e64d6ff6",
- "7faf25ba200045259f3c017248ab5ae2",
- "ca8803b7a77d4dec864dcaaf00678017",
- "f0dc1595cdb2471484ef80713dcf31df",
- "fcd46bee99c84615b937111040c109a6",
- "b5dca6bd886241c18a80253d41a2d4ee",
- "0360f1614a384242acf28bbb1a7db46f",
- "5d7e18fe043d41c3a47c00ea09070c12",
- "dc1875ae4f9842f5849b26348431af1c",
- "c6a10842bafa46899fa74736e205c0d9",
- "964d9ff7a7d543e194b2ae216cbe1f6d",
- "aac5f3132fe747a8b71ec6a3049bf687",
- "d26926bff8254773add6fd9ef20fc36b",
- "9c516df762a14f81bb97009e3bf8b9ac",
- "06e5fe3c95154565bd8a282651b58b5a",
- "cf673abf422447d28cdb4f7df444f5c4",
- "395818877e6b40efb4f3be475bd2be1b",
- "335eb1e485ee430c91b9d81e573e8646",
- "d87f940f53444fdd984ad482961010a2",
- "2c1b2a97589645bdb6577d7dda67cd72",
- "f7f0f023bd364a018cbb724540bd1faf",
- "fd1b4cdc5714432082fd18897bad1d77",
- "bf7e0e1c96f7469fae325a7d3826fb71",
- "f9123bc103c148b79deed747139ffc2b",
- "26e95d4edd724b32bac1b20b93aead6f",
- "5b0270e27d3f40b5aacd37d6ed231b49",
- "9c9ae62f6c15413fb2b26267fe0db672",
- "f231195213fe4db596daf36d770cde3c",
- "f2b5c9b732ad40a687521a208b54caaf",
- "b5a6d09bad89438ab39c9902fc95fa54",
- "2c3fd2c352134e16982a38cdf639d9ca",
- "0f8931ea0d2140abb299c4dd2b93471f",
- "abbc919bde7443a3b52d60949bcd690f",
- "d64862f3ef424188b5c68c0e9b28eb2c",
- "d664a46434834af38980622a71197393",
- "b01f37adecce4017ac18e788cf106f04",
- "fad91057508e4a44925f289394022185",
- "e02bdac296e94b4f9742a3094382aaef",
- "6891d8c4bb2449109732cfeef6d7b253",
- "b02b45ecb2d043c9aae8355287f7897a",
- "fd573b6535804bb08e2ed41dd6094ed1",
- "eb403ea9e301445e9a878a410b72924f",
- "ae4e43159adc41d0b4706153936c2a22",
- "b42854f7fac441b99f7c71a90eef439d",
- "ef1283d895084f4f94722ba0e2cda412",
- "c323bfa53fcc4f9a9d005acd0b805760",
- "76fbc2af120c408a860942d5f8fd7e6c",
- "474cc0c4c21f401ea51619925554d42f",
- "cfe758d125704bd68f12be722819b9c7",
- "fc7967ceb16d47b7b1b204b77860f29e",
- "82dde95eb3d24dcf9d6061a32a244f25",
- "3ffb6979c38740698076993ecf3cbc85",
- "4c4cce0dc87844fda2dbbd4d38295618",
- "94642d1ae9714dafa6fb4cf8262e4067",
- "cb3ef1a647874f2c9af119649c95bfd7",
- "0584e7eaadc94cfe98c6065c53383c78",
- "f609df0e80f041c3bad7cf95dd14a42b",
- "79dc9086d3f54047a71e7fbd0e23560b",
- "e41cbfb0538240fc8b17b3e6187678f5",
- "8714f27c47894d738a2943f88a550bdd",
- "3fa94563ecc94689935074631e165ea3",
- "cdca5effb1f742af845d6f1c516da483",
- "6c7bfbfc7d5b46398c1857d9315a72e0",
- "e4fa79442be74d0d80e6fae85ef3ce2d",
- "2dfadf5e51be4cf6b5fa0078a9ba0c97",
- "649c69516d1e416d9128bd6637c48b93",
- "998a4c6ea7b74b57bb9709e91f29ad92",
- "755b82222aea454c9c75ab893d2f9061",
- "e3fe2e1a646b43e0a87fcdbc155b1678",
- "9f5e0318bf154c5ab27e042c49003f90",
- "e20d35644101435baaa4dd015c9d654b",
- "8437961316664c3d8ae83a8e83a63a91",
- "88f532d14ef84e7ca4a76f79694d6e86",
- "55523d655f884092b187e22931b8bee0",
- "bbc393717ec2498c8bbee947f2190669",
- "3d639de5599244ee8b59de81c327bb5f",
- "83e2010aa1654195a097421afd2407a6",
- "4f2862571fb6480287b8b34b54c0e80e",
- "7016c9d5e856495bba8c011cf4814577",
- "6ce0c8d492f742e6873e5741b98a300f",
- "ed3387989ea641abb3d431f7755c4e6f",
- "eec215c815964b8698e8dfcdf8d1f302",
- "1663fda7fcaa4001984f0aa0d7795899",
- "f4e7142505a8402b9e8de1cce0404ae2",
- "b1551d586b154398acacf8581047fe4b",
- "8073773bd9a142fa9cc12ea521c39d95",
- "8adec177300c4149ae6908aefc14cc32",
- "ee7601f179154ec9b1edf8dc421547dd",
- "850c36fd567f487c9cecd6f0e1f6a659",
- "123ccdddd0754f1ba5b9fb62e5854fca",
- "42240ae346254c4fb87a03f90c6d9e0d",
- "eb90d82aaf9f4f56bbd14395e5c2fca9",
- "c4d258f909cc4cd7ad353dfda45064be",
- "c9bb3ee52dfa4c11b782480caa8b8649",
- "e08311c69cd0402eab772520d310fb3b",
- "7f4c524057ed439bacdc75182a0d7065",
- "3caa5687ffbf4413b5cd119cc18b1722",
- "ea7043f048a2424b93e73edb74edb3ba",
- "4a32b0115d144a65988eec81a693938a",
- "1d702e749adc473dbae13dfa2d52523c",
- "4d20f0629ce2470c9de067fb5a3d4e0a",
- "070270b2dd4e476c8e6ce4d49f1782c2",
- "af7d8f27bbe24ef98099b028c5558e26",
- "e30dcc4a3d0d4b55969d195134e68280",
- "090d195731e44773962cdd679e862a23",
- "ac6ccadec7044f55b07a1ad024ad40b0",
- "0114ed884b2a4ce880f5b7418f23f36b",
- "52c46d17158e4b50839e00ce0944455d",
- "0018e31bc1614de9ae4ffe72aa45a1f6",
- "8746e69a09fe45478106a9f53781b14c",
- "1686858e8df742db91b639e52564322c",
- "a912249a32644b95a177d50a1572bcbf",
- "f39eef7149334c569a54c4e96e8fb18f",
- "149dc14dcc474fb89c7cd72d02b5072b",
- "6440c53ae329402fad473bc68abb0a96",
- "5be2f1ed8fdb4f2ea068f87fc3ec69f2",
- "161f2557577e4dc3ba9b3b4024897810",
- "0b5dc3861295498aa9644dce81845590",
- "08ee94562ab141df8c6836b61ab754c7",
- "bf5eb3b4c01e4a04b976447544b079f7",
- "9913d08dbfa042b182a9b752fcfc895d",
- "fb055a4945b5495e98601725c63c659c",
- "b8a1b69e60ea4fac85b4b6f8e1e0766c",
- "07e39f8ee38043b0998367ff04717b59",
- "cd4fded5f3384cf5a61765712abb78af",
- "b17ce6936a4541bdb7f9d8d47d3d336a",
- "bf56868ee0d549fabaf3b9690c8ac542",
- "5f65a5736ed6481990f4d7032f25cd16",
- "ae56dd0a32794e268d64d02ba8dfe32c",
- "5b34c6ef41b748b98a5186b40b75fc7b",
- "f8dfef8b602243ac83010601c5b04c13",
- "32f001296cfa40b1a3c4ffb5eacf6930",
- "07453aa36c744a33a915b064b1e86eb3",
- "8cd919792229445b9543da84533cc47c",
- "ce1a0db85e4443eeb5006e61e9cc1cd1",
- "1b519bc5ebf34239b48913992a5a8b5e",
- "6515b0f09c174c629bb3699ef6879280",
- "bc4377f558ae488ea336ff3c7dc440ec",
- "f7f83aafb1684fa593b5fad59bbdd58d",
- "87587e11cb4e45ceb677cd1306237c52",
- "b663a4f9ff7849f2b72127221b8b4c98",
- "958c958c5e0e4173bb91dbedb471cff4",
- "752b5408fc9546e380724529a6dc9474",
- "b93f35317cfd4826ae187d36cb84b4d4",
- "9eaa1afcbc514888ad4b67ed26240553",
- "5b3fc9a74beb4954a8fc470e2fa08095",
- "18f43565e2894dad9d99394a2b69303c",
- "8b1397bb460b431ea26e7dd8f9fdf009",
- "9b1e1d03f2474a069216596c07dd1a1f",
- "0f253f610d7f4cb59de6fc4213c506db",
- "7d5154b4249f4f92bf1d056b4f065d9a",
- "2c0e6080205f4f60b47df5420c5ccd33",
- "735a1063b43044808bfc067388f1f1f3",
- "1d48c7ee7d2f4ad2bb10f4f978ddfe1d",
- "ff06ca6dcd71408c9ac0c3faac7bd19b",
- "b167c654910245828d6fbdcc0d2162c0",
- "20b7c2b03df94e0d95a170798a6a9018",
- "478751b129124e3c9ca97ed948f0e6f5",
- "d01a66ed173746bc939cdb07a839aaf9",
- "73e2dc8b1e1a41ac94b9918f88d500fd",
- "dabca95c895a4d52b8534c47ef7d7368",
- "6c576e16b2e841e8b7f784c129a1807f",
- "9eb0b4e7af464d718f339eec242c0c3a",
- "090ab228cec149cf939b862fa4fe869f",
- "90fb128d34324eae8f57f7a0e3e4a75a",
- "53a4d93060544aaa9a6bea7b6b90173c",
- "6a337186c7504b94b22daa8a424c9db2",
- "00c98fa0678241fba30ada0c670987cd",
- "65a6307cca2b47799d2f87116a997e27",
- "1f531f949b614c9f84ffb29b946c11ff",
- "4c39faa37b2349c2b52fe1616d58f591",
- "49f3ea283eea49c887d17ec659b7ccc5",
- "51c2c44622da48079f5ebc7d4dbc709a",
- "efa0fdbca21d46e4823ea9479bb09cb5",
- "12144724d2f549b5a6a6bbda68a3a250",
- "9d4b4421006a4f9fb654cb9fe160fab1",
- "a1112a7f2a9c40ce9d7a0df0cd99a089",
- "99a13c7404b34056b15f25f9def574a3",
- "cc667df4d3cc4319954f59c5fe15f3a8",
- "a0af8ef063e644bfbb29a05f54ca900a",
- "78f0a961ca844fd69f4ea16c02d42824",
- "fada6325a0104111abbaf64126b18e65",
- "f7050ba54d8f44f9bf91ca839f1b72fb",
- "a8846a3bb9534c678d3711fef5f6fae7",
- "84c3d262815b45309e4ce9fd839a4cf1",
- "914c505716444440b2713d9418ff56a2",
- "7e7d287613da4681814b0a4a31ff1fc0",
- "55796f151eb1477b813c147f8265b03f",
- "c171b5c2158b425fbfd91e0171ab3271",
- "957f5435a6374fc2b3b431e6d551f05f",
- "2a6129f066ac46edadb195b4fdc8b8b0",
- "11dd4bc82f944d589d585cfc8a5b3ce2",
- "eae7a46eafdc4241bc28acf6a99dbca9",
- "eb98e7310b664e2e8beedeca8dae2560",
- "0ee929b55093452bb5bb4746c8a68434",
- "48822383cdf644e490c0e1971e076665",
- "9459c245d55d46838ff7411999bb1961",
- "80eca9e3398d44da896b6318ba0ebae3",
- "934713b7f4b7444f8c67f00a6cf2178b",
- "b1df6e592dd34434949747e3ced43d44",
- "8bdcca9371f6434e8c9ece6b4be2374b",
- "87dfa2eb2ec64d39847598ce750b3b20",
- "67e3d2d994c04cab8ea3830f1e45d2be",
- "bb77f6a2e85447669ecfcb22c02de224",
- "191eef9c3f244a58add7fa72742eac61",
- "bd2c517b5dab4da98406c6a6e8c0a5c5",
- "f9086d1d3435445cb1b1e9a0df5a304b",
- "a2475db0c8e64f14883921a377365005",
- "2f68e2cde7274a9ea05d71d77afcca04",
- "256da2fa9c1b44219c76dc5ca9697ae8",
- "235a4fa8fe2f41f99cc0ce2c37d3bae3",
- "ffff50eaf4fe4afcba80925b5912bd9b",
- "8560fb84e75b429d998987726f6353c2",
- "26a7b02c748644d98362748bcd41845e",
- "f1540bd240e541e1853a116ea7f3da08",
- "55eb4f52f4ca4f8e8a68abe3e8927a58",
- "44ac194fdb28433c80fb39c4fcd32730",
- "8eef1bd563434b879a45b1f8ad80c30d",
- "0d86cd7a52874425bc257c3c6dcbec9a",
- "936b7fbb9c36484ab788159f5feb85b9",
- "48035b91472c41268dbf1eb70c11ddb2",
- "6bb54a01194f4551aa60c62c286ac4f5",
- "8e35190d58b3452692253098940cd373",
- "d42728de03184eba9cdfb0f097a73442",
- "04da404404af483dac479e986dec4d49",
- "118fab51e26649199498f31f8c1cb6e7",
- "b468c21beab04cc4b2ddaf840c285aa8",
- "4bbe29246dfe4f7790aed2790e8cb680",
- "9ef39cafcb8a4db99800b47e7c175037",
- "379408926ab44d4db0a042d58783318b",
- "483e5b6cb04847e7a2f04d763d5db91f",
- "0e3f2b649a144eb491a04b6d8e31476d",
- "962789df25f54b61aa6ae3f5ba8cfc96",
- "cea7bd1e63074e3592b2a1b7d8e35fc4",
- "36813fd8f3364d499256cf4120dc770f",
- "3a45dc15c4c04512ad1da3cc800d00f3",
- "981473403ed44f2aa5e2291f650534ef",
- "58e3c45f5f754c49b28ff22da29d0117",
- "ef48db93005e4dcea8c3c13798499a37",
- "accb81bde5204c77ba61b80ab4092978",
- "5a9231c62b1d453a970299282f59c78f",
- "787c7d716d384177b70a5a8145d42185",
- "df139e3be253465eb325e9efea73fa6d",
- "1bcb9dc6c2ba4535a71914a3e0c9032a",
- "0c8741e8071d49a68da0f7ee8be058c4",
- "f853155f45844423832d2b6bef895235",
- "04b67d97fd794171b676ec2ce631eb5e",
- "6e21c8d9550a45ff8a97eb8b6714c200",
- "0b9ea926ccd24f059e111894ded9401a",
- "3fc7e20c6d0448f0abe57733cf43f44d",
- "83777a4b87924aa09d7d9593d90c2ed6",
- "a7042dd5d7de4fd3895f25800977c0ec",
- "34196051af324531b8680fc803c99c46",
- "3224cbe4c3994f40bea760b1484ed30a",
- "dab4f081ab08406fb6d944d0d85d3ddd",
- "6ca6acb0ce8e40e5a059c1bd8111a66c",
- "24c53946957b486e81369d005f6a0999",
- "86b8f5f51a4c41fabc10ae3d4f39f9da",
- "0db0b0a5a68a427494a648501379ed11",
- "14c60a09aecc4d26885bfb42e3f364c6",
- "3b4447dbc77d4295ace660f589bb5d1f"
- ]
- },
- "id": "nQtMcFzLGTuY",
- "outputId": "83ad8560-1581-43f9-cf66-88303d2a6c1c"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stderr",
- "text": [
- "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'Repository' (from 'huggingface_hub.repository') is deprecated and will be removed from version '1.0'. Please prefer the http-based alternatives instead. Given its large adoption in legacy code, the complete removal is only planned on next major release.\n",
- "For more details, please read https://huggingface.co/docs/huggingface_hub/concepts/git_vs_http.\n",
- " warnings.warn(warning_message, FutureWarning)\n",
- "Cloning https://huggingface.co/lucio/xls-r-uzbek-cv8 into local empty directory.\n",
- "WARNING:huggingface_hub.repository:Cloning https://huggingface.co/lucio/xls-r-uzbek-cv8 into local empty directory.\n"
- ]
- },
- {
- "output_type": "display_data",
- "data": {
- "text/plain": [
- "Download file pytorch_model.bin: 0%| | 8.00k/1.18G [00:00
Copy a token from your Hugging Face\ntokens page and paste it below.
Immediately click login after copying\nyour token or it might be stored in plain text in this notebook file.