Duplicate from amphion/Vevo

Co-authored-by: Xueyao Zhang <[email protected]>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,195 @@
+---
+license: cc-by-nc-4.0
+tags:
+- voice-conversion
+- text-to-speech
+- accent-conversion
+- emotion-conversion
+- style-transfer
+datasets:
+- amphion/Emilia-Dataset
+---
+
+# Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement
+
+[![arXiv](https://img.shields.io/badge/OpenReview-Paper-COLOR.svg)](https://openreview.net/pdf?id=anQDiQZhDP)
+[![hf](https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-model-yellow)](https://huggingface.co/amphion/Vevo)
+[![WebPage](https://img.shields.io/badge/WebPage-Demo-red)](https://versavoice.github.io/) [![readme](https://img.shields.io/badge/README-GitHub-blue)](https://github.com/open-mmlab/Amphion/blob/main/models/vc/vevo/README.md)
+
+We present our reproduction of [Vevo](https://openreview.net/pdf?id=anQDiQZhDP), a versatile zero-shot voice imitation framework with controllable timbre and style. We invite you to explore the [audio samples](https://versavoice.github.io/) to experience Vevo's capabilities firsthand.
+
+We have included the following pre-trained Vevo models at Amphion:
+
+- **Vevo-Timbre**: It can conduct *style-preserved* voice conversion.
+- **Vevo-Style**: It can conduct style conversion, such as *accent conversion* and *emotion conversion*.
+- **Vevo-Voice**: It can conduct *style-converted* voice conversion.
+- **Vevo-TTS**: It can conduct *style and timbre controllable* TTS.
+
+Besides, we also release the **content tokenizer** and **content-style tokenizer** proposed by Vevo. Notably, all these pre-trained models are trained on [Emilia](https://huggingface.co/datasets/amphion/Emilia-Dataset), containing 101k hours of speech data among six languages (English, Chinese, German, French, Japanese, and Korean).
+
+## Model Introduction
+
+We provide the following pre-trained models:
+
+
+| Model Name          | Description   |    
+|-------------------|-------------|
+| [Content Tokenizer](https://huggingface.co/amphion/Vevo/tree/main/tokenizer/vq32)      | Converting speech to content tokens. It is a single codebook VQ-VAE with a vocabulary size of 32. The frame rate is 50Hz.|
+| [Content-Style Tokenizer](https://huggingface.co/amphion/Vevo/tree/main/tokenizer/vq8192)      | Converting speech to content-style tokens. It is a single codebook VQ-VAE with a vocabulary size of 8192. The frame rate is 50Hz.|
+| [Vq32ToVq8192](https://huggingface.co/amphion/Vevo/tree/main/contentstyle_modeling/Vq32ToVq8192)      | Predicting content-style tokens from content tokens with an auto-regressive transformer (480M). |
+| [PhoneToVq8192](https://huggingface.co/amphion/Vevo/tree/main/contentstyle_modeling/PhoneToVq8192)      | Predicting content-style tokens from phone tokens with an auto-regressive transformer (740M). |
+| [Vq8192ToMels](https://huggingface.co/amphion/Vevo/tree/main/acoustic_modeling/Vq8192ToMels)      | Predicting mel-spectrogram from content-style tokens with a flow-matching transformer (330M). |
+| [Vocoder](https://huggingface.co/amphion/Vevo/tree/main/acoustic_modeling/Vocoder)      | Predicting audio from mel-spectrogram with a Vocos-based vocoder (250M). |
+
+You can download all pretrained checkpoints from [HuggingFace](https://huggingface.co/amphion/MaskGCT/tree/main) or use huggingface API.
+
+## Usage
+
+You can refer to our [recipe](https://github.com/open-mmlab/Amphion/blob/main/models/vc/vevo/README.md) at GitHub for more usage details. For example, to use Vevo-TTS, after you clone the Amphion github repository, you can use the script like:
+
+```python
+import os
+from huggingface_hub import snapshot_download
+
+from models.vc.vevo.vevo_utils import *
+
+
+def vevo_tts(
+    src_text,
+    ref_wav_path,
+    timbre_ref_wav_path=None,
+    output_path=None,
+    ref_text=None,
+    src_language="en",
+    ref_language="en",
+):
+    if timbre_ref_wav_path is None:
+        timbre_ref_wav_path = ref_wav_path
+
+    gen_audio = inference_pipeline.inference_ar_and_fm(
+        src_wav_path=None,
+        src_text=src_text,
+        style_ref_wav_path=ref_wav_path,
+        timbre_ref_wav_path=timbre_ref_wav_path,
+        style_ref_wav_text=ref_text,
+        src_text_language=src_language,
+        style_ref_wav_text_language=ref_language,
+    )
+
+    assert output_path is not None
+    save_audio(gen_audio, output_path=output_path)
+
+
+if __name__ == "__main__":
+    # ===== Device =====
+    device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
+
+    # ===== Content-Style Tokenizer =====
+    local_dir = snapshot_download(
+        repo_id="amphion/Vevo",
+        repo_type="model",
+        cache_dir="./ckpts/Vevo",
+        allow_patterns=["tokenizer/vq8192/*"],
+    )
+
+    content_style_tokenizer_ckpt_path = os.path.join(local_dir, "tokenizer/vq8192")
+
+    # ===== Autoregressive Transformer =====
+    local_dir = snapshot_download(
+        repo_id="amphion/Vevo",
+        repo_type="model",
+        cache_dir="./ckpts/Vevo",
+        allow_patterns=["contentstyle_modeling/PhoneToVq8192/*"],
+    )
+
+    ar_cfg_path = "./models/vc/vevo/config/PhoneToVq8192.json"
+    ar_ckpt_path = os.path.join(local_dir, "contentstyle_modeling/PhoneToVq8192")
+
+    # ===== Flow Matching Transformer =====
+    local_dir = snapshot_download(
+        repo_id="amphion/Vevo",
+        repo_type="model",
+        cache_dir="./ckpts/Vevo",
+        allow_patterns=["acoustic_modeling/Vq8192ToMels/*"],
+    )
+
+    fmt_cfg_path = "./models/vc/vevo/config/Vq8192ToMels.json"
+    fmt_ckpt_path = os.path.join(local_dir, "acoustic_modeling/Vq8192ToMels")
+
+    # ===== Vocoder =====
+    local_dir = snapshot_download(
+        repo_id="amphion/Vevo",
+        repo_type="model",
+        cache_dir="./ckpts/Vevo",
+        allow_patterns=["acoustic_modeling/Vocoder/*"],
+    )
+
+    vocoder_cfg_path = "./models/vc/vevo/config/Vocoder.json"
+    vocoder_ckpt_path = os.path.join(local_dir, "acoustic_modeling/Vocoder")
+
+    # ===== Inference =====
+    inference_pipeline = VevoInferencePipeline(
+        content_style_tokenizer_ckpt_path=content_style_tokenizer_ckpt_path,
+        ar_cfg_path=ar_cfg_path,
+        ar_ckpt_path=ar_ckpt_path,
+        fmt_cfg_path=fmt_cfg_path,
+        fmt_ckpt_path=fmt_ckpt_path,
+        vocoder_cfg_path=vocoder_cfg_path,
+        vocoder_ckpt_path=vocoder_ckpt_path,
+        device=device,
+    )
+
+    src_text = "I don't really care what you call me. I've been a silent spectator, watching species evolve, empires rise and fall. But always remember, I am mighty and enduring. Respect me and I'll nurture you; ignore me and you shall face the consequences."
+
+    ref_wav_path = "./models/vc/vevo/wav/arabic_male.wav"
+    ref_text = "Flip stood undecided, his ears strained to catch the slightest sound."
+
+    # 1. Zero-Shot TTS (the style reference and timbre reference are same)
+    vevo_tts(
+        src_text,
+        ref_wav_path,
+        output_path="./models/vc/vevo/wav/output_vevotts1.wav",
+        ref_text=ref_text,
+        src_language="en",
+        ref_language="en",
+    )
+
+    # 2. Style and Timbre Controllable Zero-Shot TTS (the style reference and timbre reference are different)
+    vevo_tts(
+        src_text,
+        ref_wav_path,
+        timbre_ref_wav_path="./models/vc/vevo/wav/mandarin_female.wav",
+        output_path="./models/vc/vevo/wav/output_vevotts2.wav",
+        ref_text=ref_text,
+        src_language="en",
+        ref_language="en",
+    )
+```
+
+## Citation
+
+If you use Vevo in your research, please cite the following papers:
+
+```bibtex
+@inproceedings{vevo,
+  author       = {Xueyao Zhang and Xiaohui Zhang and Kainan Peng and Zhenyu Tang and Vimal Manohar and Yingru Liu and Jeff Hwang and Dangna Li and Yuhao Wang and Julian Chan and Yuan Huang and Zhizheng Wu and Mingbo Ma},
+  title        = {Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement},
+  booktitle    = {{ICLR}},
+  publisher    = {OpenReview.net},
+  year         = {2025}
+}
+
+@article{amphion_v0.2,
+  title        = {Overview of the Amphion Toolkit (v0.2)},
+  author       = {Jiaqi Li and Xueyao Zhang and Yuancheng Wang and Haorui He and Chaoren Wang and Li Wang and Huan Liao and Junyi Ao and Zeyu Xie and Yiqiao Huang and Junan Zhang and Zhizheng Wu},
+  year         = {2025},
+  journal      = {arXiv preprint arXiv:2501.15442},
+}
+
+@inproceedings{amphion,
+    author={Xueyao Zhang and Liumeng Xue and Yicheng Gu and Yuancheng Wang and Jiaqi Li and Haorui He and Chaoren Wang and Ting Song and Xi Chen and Zihao Fang and Haopeng Chen and Junan Zhang and Tze Ying Tang and Lexiao Zou and Mingxuan Wang and Jun Han and Kai Chen and Haizhou Li and Zhizheng Wu},
+    title={Amphion: An Open-Source Audio, Music and Speech Generation Toolkit},
+    booktitle={{IEEE} Spoken Language Technology Workshop, {SLT} 2024},
+    year={2024}
+}
+```

acoustic_modeling/Vocoder/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7670d180569fdae986fbf94ede07d6fc4ce8bfcf406cd1aadbe33e08581b5f6a
+size 1020206416

acoustic_modeling/Vocoder/model_1.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:164c3db2a9f41a994ac1c7a7a57aa15aadb97dd3f94a4af9bef95d2f7edfad34
+size 69768280

acoustic_modeling/Vocoder/model_2.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:03785c5a3ed7181b22fae11a266d1438cb59333726dfbe8429f00d5543dd8c95
+size 180693296

acoustic_modeling/Vq8192ToMels/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:750f013ac1485855bfbe992ffec8ed5f625e6070b8bc52ce71a6f9ae0229c5c4
+size 1350803704

config.json

@@ -0,0 +1,5 @@
+{
+    "download_tracking": {
+        "query_files": ["config.json", "*.safetensors"]
+    }
+}

contentstyle_modeling/PhoneToVq8192/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:033248ccc7879456474134b6c77ea0d82a0e78ba7a09ba1ae7e1d99943d4eeaf
+size 2973225456

contentstyle_modeling/Vq32ToVq8192/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83e4984695487d6feeba9664c95375f9487e3326059b7239db1fe220e1d49b1d
+size 1925991368

tokenizer/vq32/hubert_large_l18_c32.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7d13c6f6e34a1ef43cd7d67cef9814df0ca99e9c5b4610c0a0115bb2b1d76045
+size 235174528

tokenizer/vq32/hubert_large_l18_c32.yaml

@@ -0,0 +1,30 @@
+bias: true
+code_dim: 1024
+codebook_num: 1
+codebook_size: 32
+dec_block_dilations:
+- 1
+- 1
+dec_block_kernel_size: 3
+dec_kernel_size: 3
+dec_ratios:
+- 1
+- 1
+dec_strides:
+- 1
+- 1
+decode_channels: 1024
+enc_block_dilations:
+- 1
+- 1
+enc_block_kernel_size: 3
+enc_kernel_size: 3
+enc_ratios:
+- 1
+- 1
+enc_strides:
+- 1
+- 1
+encode_channels: 1024
+input_channels: 1024
+output_channels: 1024

tokenizer/vq32/hubert_large_l18_mean_std.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84115dd2ad3a44526b8e75e173956e41805ba540704acd735f302889a83067b4
+size 8692

tokenizer/vq8192/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:660bd48b023e637a786a9c78f404cb979ef9a5d1c93ce24837e0bec942352c4d
+size 177183712