Duplicate from Simonlob/TTS_AkylAI_2

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,128 @@
+---
+license: cc-by-nc-4.0
+inference: true
+tags:
+- mms
+- vits
+pipeline_tag: text-to-speech
+language:
+- ky
+---
+
+# Introduction
+
+This repository contains a text-to-speech (TTS) model fine-tuned on data consisting of sentences in the Kyrgyz language with audio examples voiced by a single speaker. The audio is provided at a sample rate of 16 kHz. The dataset comprises 3500 examples and 4 hours of audio. The model is based on the facebook/mms-tts-kir model pre-trained on the Kyrgyz language. The code for fine-tuning the model was based on the code from this [GitHub repository](https://github.com/ylacombe/finetune-hf-vits). Experimental findings concluded that the best results are achieved through two-stage fine-tuning:
+
+* Training with Learning Rate 1e-4 and 4 epochs,
+* Training with Learning Rate 9e-7 and 80 epochs.
+
+
+# MMS: Scaling Speech Technology to 1000+ languages
+
+The Massively Multilingual Speech (MMS) project expands speech technology from about 100 languages to over 1,000 by building a single multilingual speech recognition model supporting over 1,100 languages (more than 10 times as many as before), language identification models able to identify over [4,000 languages](https://dl.fbaipublicfiles.com/mms/misc/language_coverage_mms.html) (40 times more than before), pretrained models supporting over 1,400 languages, and text-to-speech models for over 1,100 languages. Our goal is to make it easier for people to access information and to use devices in their preferred language.  
+
+You can find details in the paper [Scaling Speech Technology to 1000+ languages](https://research.facebook.com/publications/scaling-speech-technology-to-1000-languages/) and the [blog post](https://ai.facebook.com/blog/multilingual-model-speech-recognition/).
+
+An overview of the languages covered by MMS can be found [here](https://dl.fbaipublicfiles.com/mms/misc/language_coverage_mms.html).
+
+##  Transformers
+
+MMS has been added to Transformers. For more information, please refer to [Transformers' MMS docs](https://huggingface.co/docs/transformers/main/en/model_doc/mms).
+
+[Click here](https://huggingface.co/models?other=mms) to find all MMS checkpoints on the Hub. 
+
+Checkout the demo here [![Open In HF Spaces](https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm-dark.svg)](https://huggingface.co/spaces/facebook/MMS) 
+
+## 
+
+
+# Inference
+
+
+The model takes Cyrillic text in the Kyrgyz language as input and preprocesses it by removing punctuation marks (periods, commas, colons, exclamation and question marks) as well as words written in Latin script. Therefore, it is not advisable to feed multiple sentences into the model at once as they will be vocalized without intonational pauses, indicating the end of one and the beginning of a new sentence. Words written in Latin script will be skipped in the generated speech.
+
+For example:
+```
+text = 'Кандай улут болбосун кыргызча жооп кайтарышыбыз керек.'
+```
+
+You can use this model by executing the code provided below.
+
+```
+import subprocess
+from transformers import pipeline
+from IPython.display import Audio
+import numpy as np
+import torch
+import scipy
+
+model_id = "Simonlob/simonlob_akylay"
+synthesiser = pipeline("text-to-speech", model_id) # add device=0 if you want to use a GPU
+
+```
+
+```
+text = 'Кандай улут болбосун кыргызча жооп кайтарышыбыз керек.'
+speech = synthesiser(text)
+```
+The output of the model looks as follows:
+```
+{'audio': array([[-1.7045566e-04,  8.9107212e-05,  2.8329418e-04, ...,
+          8.0898666e-08,  4.8763245e-06,  5.4663483e-06]], dtype=float32),
+ 'sampling_rate': 16000}
+```
+
+Listen to the result:
+
+```
+Audio(speech['audio'], rate=speech['sampling_rate'])
+```
+
+Save the audio as a file:
+
+```
+scipy.io.wavfile.write("<OUTPUT PATH>.wav", rate=speech["sampling_rate"], data=speech["audio"][0])
+```
+
+
+
+</details>
+
+## Model details
+
+- **Model type:** Text-to-speech model
+- **License:** CC-BY-NC 4.0 license
+- **Cite as:**
+
+      @article{pratap2023mms,
+        title={Scaling Speech Technology to 1,000+ Languages},
+        author={Vineel Pratap and Andros Tjandra and Bowen Shi and Paden Tomasello and Arun Babu and Sayani Kundu and Ali Elkahky and Zhaoheng Ni and Apoorv Vyas and Maryam Fazel-Zarandi and Alexei Baevski and Yossi Adi and Xiaohui Zhang and Wei-Ning Hsu and Alexis Conneau and Michael Auli},
+      journal={arXiv},
+      year={2023}
+      }
+
+
+## Credits
+
+
+- Facebook AI Research ([Official Space](https://huggingface.co/spaces/facebook/MMS))
+- Yoach Lacombe (Research) [GitHub](https://github.com/ylacombe/finetune-hf-vits)
+- The Cramer Project (Data collection and preprocessing)[Official Space](https://thecramer.com/), [Akyl_AI](https://github.com/Akyl-AI)
+- Amantur Amatov (Expert)
+- Timur Turatali (Expert, Research) [GitHub](https://github.com/golden-ratio)
+- Den Pavlov (Research, Data preprocessing and fine-tuning) [GitHub](https://github.com/simonlobgromov/finetune-hf-vits)
+- Ulan Abdurazakov (Environment Developer)
+- Nursultan Bakashov (CEO)
+
+
+## Additional Links
+
+- [Blog post](https://ai.facebook.com/blog/multilingual-model-speech-recognition/)
+- [Transformers documentation](https://huggingface.co/docs/transformers/main/en/model_doc/mms).
+- [Paper](https://arxiv.org/abs/2305.13516)
+- [GitHub Repository for fine tuning](https://github.com/ylacombe/finetune-hf-vits)
+- [GitHub Repository](https://github.com/facebookresearch/fairseq/tree/main/examples/mms#asr)
+- [Other **MMS** checkpoints](https://huggingface.co/models?other=mms)
+- MMS base checkpoints:
+  - [facebook/mms-1b](https://huggingface.co/facebook/mms-1b)
+  - [facebook/mms-300m](https://huggingface.co/facebook/mms-300m)

added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "<unk>": 38
+}

config.json

@@ -0,0 +1,108 @@
+{
+  "_name_or_path": "Simonlob/TTS_AkylAI_2",
+  "activation_dropout": 0.1,
+  "architectures": [
+    "VitsModelForPreTraining"
+  ],
+  "attention_dropout": 0.1,
+  "depth_separable_channels": 2,
+  "depth_separable_num_layers": 3,
+  "discriminator_kernel_size": 5,
+  "discriminator_period_channels": [
+    1,
+    32,
+    128,
+    512,
+    1024
+  ],
+  "discriminator_periods": [
+    2,
+    3,
+    5,
+    7,
+    11
+  ],
+  "discriminator_scale_channels": [
+    1,
+    16,
+    64,
+    256,
+    1024
+  ],
+  "discriminator_stride": 3,
+  "duration_predictor_dropout": 0.5,
+  "duration_predictor_filter_channels": 256,
+  "duration_predictor_flow_bins": 10,
+  "duration_predictor_kernel_size": 3,
+  "duration_predictor_num_flows": 4,
+  "duration_predictor_tail_bound": 5.0,
+  "ffn_dim": 768,
+  "ffn_kernel_size": 3,
+  "flow_size": 192,
+  "hidden_act": "relu",
+  "hidden_dropout": 0.1,
+  "hidden_size": 192,
+  "hop_length": 256,
+  "initializer_range": 0.02,
+  "layer_norm_eps": 1e-05,
+  "layerdrop": 0.1,
+  "leaky_relu_slope": 0.1,
+  "model_type": "vits",
+  "noise_scale": 0.667,
+  "noise_scale_duration": 0.8,
+  "num_attention_heads": 2,
+  "num_hidden_layers": 6,
+  "num_speakers": 1,
+  "posterior_encoder_num_wavenet_layers": 16,
+  "prior_encoder_num_flows": 4,
+  "prior_encoder_num_wavenet_layers": 4,
+  "resblock_dilation_sizes": [
+    [
+      1,
+      3,
+      5
+    ],
+    [
+      1,
+      3,
+      5
+    ],
+    [
+      1,
+      3,
+      5
+    ]
+  ],
+  "resblock_kernel_sizes": [
+    3,
+    7,
+    11
+  ],
+  "sampling_rate": 16000,
+  "segment_size": 8192,
+  "speaker_embedding_size": 0,
+  "speaking_rate": 1.0,
+  "spectrogram_bins": 513,
+  "torch_dtype": "float32",
+  "transformers_version": "4.38.1",
+  "upsample_initial_channel": 512,
+  "upsample_kernel_sizes": [
+    16,
+    16,
+    4,
+    4
+  ],
+  "upsample_rates": [
+    8,
+    8,
+    2,
+    2
+  ],
+  "use_bias": true,
+  "use_stochastic_duration_prediction": true,
+  "vocab_size": 38,
+  "wavenet_dilation_rate": 1,
+  "wavenet_dropout": 0.0,
+  "wavenet_kernel_size": 5,
+  "window_size": 4
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:744167b99051dd58b2b1f138f200b0cb20720e57dd209be9249b27512a7edf4c
+size 332160712

preprocessor_config.json

@@ -0,0 +1,11 @@
+{
+  "feature_extractor_type": "VitsFeatureExtractor",
+  "feature_size": 80,
+  "hop_length": 256,
+  "max_wav_value": 32768.0,
+  "n_fft": 1024,
+  "padding_side": "right",
+  "padding_value": 0.0,
+  "return_attention_mask": false,
+  "sampling_rate": 16000
+}

special_tokens_map.json

@@ -0,0 +1,16 @@
+{
+  "pad_token": {
+    "content": "ж",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,31 @@
+{
+  "add_blank": true,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "ж",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "38": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "is_uroman": false,
+  "language": "kir",
+  "model_max_length": 1000000000000000019884624838656,
+  "normalize": true,
+  "pad_token": "ж",
+  "phonemize": false,
+  "tokenizer_class": "VitsTokenizer",
+  "unk_token": "<unk>",
+  "verbose": false
+}

vocab.json

@@ -0,0 +1,40 @@
+{
+  " ": 14,
+  "-": 13,
+  "_": 16,
+  "а": 19,
+  "б": 30,
+  "в": 31,
+  "г": 35,
+  "д": 20,
+  "е": 8,
+  "ж": 0,
+  "з": 27,
+  "и": 4,
+  "й": 10,
+  "к": 26,
+  "л": 11,
+  "м": 36,
+  "н": 23,
+  "о": 9,
+  "п": 32,
+  "р": 24,
+  "с": 22,
+  "т": 21,
+  "у": 12,
+  "ф": 37,
+  "х": 28,
+  "ц": 1,
+  "ч": 33,
+  "ш": 25,
+  "ы": 5,
+  "ь": 34,
+  "э": 18,
+  "ю": 6,
+  "я": 7,
+  "ё": 29,
+  "ң": 17,
+  "ү": 15,
+  "ө": 2,
+  "–": 3
+}