CosyVoice
Install
Clone and install
- Clone the repo
git clone https://github.com/modelscope/cosyvoice.git
- Install Conda: please see https://docs.conda.io/en/latest/miniconda.html
- Create Conda env:
conda create -n cosyvoice python=3.8
conda activate cosyvoice
pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com
Model download
We strongly recommand that you download our pretrained multi_lingual and mutli_emotion model.
If you are expert in this field, and you are only interested in training your own CosyVoice model from scratch, you can skip this step.
mkdir -p pretrained_models
git clone https://www.modelscope.cn/CosyVoice/multi_lingual_cosytts.git pretrained_models/multi_lingual_cosytts
git clone https://www.modelscope.cn/CosyVoice/multi_emotion_cosytts.git pretrained_models/multi_emotion_cosytts
Basic Usage
For zero_shot and sft inference, please use models in pretrained_models/multi_lingual_cosytts
from cosyvoice.cli.cosyvoice import CosyVoice
from cosyvoice.utils.file_utils import load_wav
import torchaudio
cosyvoice = CosyVoice('pretrained_models/multi_lingual_cosytts')
# sft usage
print(cosyvoice.list_avaliable_spks())
output = cosyvoice.inference_sft('hello, my name is Jack. What is your name?', 'aishuo')
torchaudio.save('sft.wav', output['tts_speech'], 22050)
# zero_shot usage
prompt_speech_22050 = load_wav('1089_134686_000002_000000.wav', 22050)
output = cosyvoice.inference_zero_shot('hello, my name is Jack. What is your name?', 'It would be a gloomy secret night.', prompt_speech_22050)
torchaudio.save('zero_shot.wav', output['tts_speech'], 22050)
For instruct inference, please use models in pretrained_models/multi_emotion_cosytts
from cosyvoice.cli.cosyvoice import CosyVoice
from cosyvoice.utils.file_utils import load_wav
import torchaudio
cosyvoice = CosyVoice('pretrained_models/multi_emotion_cosytts')
# instruct usage
prompt_speech_22050 = load_wav('1089_134686_000002_000000.wav', 22050)
output = cosyvoice.inference_instruct('hello, my name is Jack. What is your name?', 'It would be a gloomy secret night.', prompt_speech_22050, 'A serene woman articulates thoughtfully in a high pitch and slow tempo, exuding a peaceful and joyful aura.')
torchaudio.save('instruct.wav', output['tts_speech'], 22050)
Advanced Usage
For advanced user, we have provided train and inference scripts in examples/libritts/cosyvoice/run.sh.
You can get familiar with CosyVoice following this recipie.
Start web demo
You can use our web demo page to get familiar with CosyVoice quickly. We only support zero_shot/sft inference in web demo.
Please see the demo website for details.
python3 webui.py --port 50000 --model_dir pretrained_models/multi_lingual_cosytts
Build for deployment
Optionally, if you want to use grpc for service deployment, you can run following steps. Otherwise, you can just ignore this step.
cd runtime/python
docker build -t cosyvoice:v1.0 .
# change multi_lingual_cosytts to multi_emotion_cosytts if you want to use instruct inference
docker run -d --runtime=nvidia -v `pwd`/../../pretrained_models/multi_lingual_cosytts:/opt/cosyvoice/cosyvoice/runtime/pretrained_models -p 50000:50000 cosyvoice:v1.0
python3 client.py --port 50000 --mode <sft|zero_shot|instruct>