+ + +## Preliminaries +1. Install the environment: +```bash +bash ./scripts/EnvsSetup.sh +``` +2. Activate the conda environment: +```bash +conda activate WavJourney +``` + +3. (Optional) You can modify the default configuration in `config.yaml`, check the details described in the configuration file. +4. Pre-download the models (might take some time): +```bash +python scripts/download_models.py +``` + +5. Set the WAVJOURNEY_OPENAI_KEY in the environment variable for accessing [GPT-4 API](https://platform.openai.com/account/api-keys) [[Guidance](https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4)] +```bash +export WAVJOURNEY_OPENAI_KEY=your_openai_key_here +``` + +6. Set environment variables for using API services +```bash +# Set the port for the WAVJOURNEY service to 8021 +export WAVJOURNEY_SERVICE_PORT=8021 + +# Set the URL for the WAVJOURNEY service to 127.0.0.1 +export WAVJOURNEY_SERVICE_URL=127.0.0.1 + +# Limit the maximum script lines for WAVJOURNEY to 999 +export WAVJOURNEY_MAX_SCRIPT_LINES=999 +``` + + +7. Start Python API services (e.g., Text-to-Speech, Text-to-Audio) +```bash +bash scripts/start_services.sh +``` + +## Web APP + ```bash +bash scripts/start_ui.sh + ``` + +## Commandline Usage + ```bash + python wavjourney_cli.py -f --input-text "Generate a one-minute introduction to quantum mechanics" + ``` + + +## Kill the services +You can kill the running services via this command: + ```bash +python scripts/kill_services.py + ``` + +## (Advanced features) Speaker customization +You can add voice presets to WavJourney to customize the voice actors. Simply provide the voice id, the description and a sample wav file, and WavJourney will pick the voice automatically based on the audio script. Predefined system voice presets are in `data/voice_presets`. + +You can manage voice presets via UI. Specifically, if you want to add voice to voice presets. Run the script via command line below: +```bash +python add_voice_preset.py --id "id" --desc "description" --wav-path path/to/wav --session-id '' +``` +What makes for good voice prompt? See detailed instructions here. +## Hardware requirement +- The VRAM of the GPU in the default configuration should be greater than 16 GB. +- Operation system: Linux. + +## Citation +If you find this work useful, you can cite the paper below: + + @article{liu2023wavjourney, + title = {WavJourney: Compositional Audio Creation with Large Language Models}, + author = {Liu, Xubo and Zhu, Zhongkai and Liu, Haohe and Yuan, Yi and Huang, Qiushi and Liang, Jinhua and Cao, Yin and Kong, Qiuqiang and Plumbley, Mark D and Wang, Wenwu}, + journal = {arXiv preprint arXiv:2307.14335}, + year = {2023} + } + +[](https://www.buymeacoffee.com/liuxubo) + +## Appreciation +- [Bark](https://github.com/suno-ai/bark) for a zero-shot text-to-speech synthesis model. +- [AudioCraft](https://github.com/facebookresearch/audiocraft) for state-of-the-art audio generation models. + +## Disclaimer +We are not responsible for audio generated using semantics created by this model. Just don't use it for illegal purposes. + diff --git a/VoiceParser/__init__.py b/VoiceParser/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/VoiceParser/customtokenizer.py b/VoiceParser/customtokenizer.py new file mode 100644 index 0000000000000000000000000000000000000000..aa2e7d49bab149dfe3cb43db5502a4c5b40821c1 --- /dev/null +++ b/VoiceParser/customtokenizer.py @@ -0,0 +1,202 @@ +""" +Custom tokenizer model. +Author: https://www.github.com/gitmylo/ +License: MIT +""" + +import json +import os.path +from zipfile import ZipFile +from typing import Union + + +import numpy +import torch +from torch import nn, optim +from torch.serialization import MAP_LOCATION + + +class CustomTokenizer(nn.Module): + def __init__(self, hidden_size=1024, input_size=768, output_size=10000, version=0): + super(CustomTokenizer, self).__init__() + next_size = input_size + if version == 0: + self.lstm = nn.LSTM(input_size, hidden_size, 2, batch_first=True) + next_size = hidden_size + if version == 1: + self.lstm = nn.LSTM(input_size, hidden_size, 2, batch_first=True) + self.intermediate = nn.Linear(hidden_size, 4096) + next_size = 4096 + + self.fc = nn.Linear(next_size, output_size) + self.softmax = nn.LogSoftmax(dim=1) + self.optimizer: optim.Optimizer = None + self.lossfunc = nn.CrossEntropyLoss() + self.input_size = input_size + self.hidden_size = hidden_size + self.output_size = output_size + self.version = version + + def forward(self, x): + x, _ = self.lstm(x) + if self.version == 1: + x = self.intermediate(x) + x = self.fc(x) + x = self.softmax(x) + return x + + @torch.no_grad() + def get_token(self, x): + """ + Used to get the token for the first + :param x: An array with shape (N, input_size) where N is a whole number greater or equal to 1, and input_size is the input size used when creating the model. + :return: An array with shape (N,) where N is the same as N from the input. Every number in the array is a whole number in range 0...output_size - 1 where output_size is the output size used when creating the model. + """ + return torch.argmax(self(x), dim=1) + + def prepare_training(self): + self.optimizer = optim.Adam(self.parameters(), 0.001) + + def train_step(self, x_train, y_train, log_loss=False): + # y_train = y_train[:-1] + # y_train = y_train[1:] + + optimizer = self.optimizer + lossfunc = self.lossfunc + # Zero the gradients + self.zero_grad() + + # Forward pass + y_pred = self(x_train) + + y_train_len = len(y_train) + y_pred_len = y_pred.shape[0] + + if y_train_len > y_pred_len: + diff = y_train_len - y_pred_len + y_train = y_train[diff:] + elif y_train_len < y_pred_len: + diff = y_pred_len - y_train_len + y_pred = y_pred[:-diff, :] + + y_train_hot = torch.zeros(len(y_train), self.output_size) + y_train_hot[range(len(y_train)), y_train] = 1 + y_train_hot = y_train_hot.to('cuda') + + # Calculate the loss + loss = lossfunc(y_pred, y_train_hot) + + # Print loss + if log_loss: + print('Loss', loss.item()) + + # Backward pass + loss.backward() + + # Update the weights + optimizer.step() + + def save(self, path): + info_path = '.'.join(os.path.basename(path).split('.')[:-1]) + '/.info' + torch.save(self.state_dict(), path) + data_from_model = Data(self.input_size, self.hidden_size, self.output_size, self.version) + with ZipFile(path, 'a') as model_zip: + model_zip.writestr(info_path, data_from_model.save()) + model_zip.close() + + @staticmethod + def load_from_checkpoint(path, map_location: MAP_LOCATION = None): + old = True + with ZipFile(path) as model_zip: + filesMatch = [file for file in model_zip.namelist() if file.endswith('/.info')] + file = filesMatch[0] if filesMatch else None + if file: + old = False + data_from_model = Data.load(model_zip.read(file).decode('utf-8')) + model_zip.close() + if old: + model = CustomTokenizer() + else: + model = CustomTokenizer(data_from_model.hidden_size, data_from_model.input_size, data_from_model.output_size, data_from_model.version) + model.load_state_dict(torch.load(path, map_location=map_location)) + if map_location: + model = model.to(map_location) + return model + + + +class Data: + input_size: int + hidden_size: int + output_size: int + version: int + + def __init__(self, input_size=768, hidden_size=1024, output_size=10000, version=0): + self.input_size = input_size + self.hidden_size = hidden_size + self.output_size = output_size + self.version = version + + @staticmethod + def load(string): + data = json.loads(string) + return Data(data['input_size'], data['hidden_size'], data['output_size'], data['version']) + + def save(self): + data = { + 'input_size': self.input_size, + 'hidden_size': self.hidden_size, + 'output_size': self.output_size, + 'version': self.version, + } + return json.dumps(data) + + +def auto_train(data_path, save_path='model.pth', lload_model: Union[str, None] = None, save_epochs=1): + data_x, data_y = {}, {} + + if load_model and os.path.isfile(load_model): + print('Loading model from', load_model) + model_training = CustomTokenizer.load_from_checkpoint(load_model, 'cuda') + else: + print('Creating new model.') + model_training = CustomTokenizer(version=1).to('cuda') + save_path = os.path.join(data_path, save_path) + base_save_path = '.'.join(save_path.split('.')[:-1]) + + sem_string = '_semantic.npy' + feat_string = '_semantic_features.npy' + + ready = os.path.join(data_path, 'ready') + for input_file in os.listdir(ready): + full_path = os.path.join(ready, input_file) + try: + prefix = input_file.split("_")[0] + number = int(prefix) + except ValueError as e: + raise e + if input_file.endswith(sem_string): + data_y[number] = numpy.load(full_path) + elif input_file.endswith(feat_string): + data_x[number] = numpy.load(full_path) + + model_training.prepare_training() + epoch = 1 + + while 1: + for i in range(save_epochs): + j = 0 + for i in range(max(len(data_x), len(data_y))): + x = data_x.get(i) + y = data_y.get(i) + if x is None or y is None: + print(f'The training data does not match. key={i}') + continue + model_training.train_step(torch.tensor(x).to('cuda'), torch.tensor(y).to('cuda'), j % 50 == 0) # Print loss every 50 steps + j += 1 + save_p = save_path + save_p_2 = f'{base_save_path}_epoch_{epoch}.pth' + model_training.save(save_p) + model_training.save(save_p_2) + print(f'Epoch {epoch} completed') + epoch += 1 \ No newline at end of file diff --git a/VoiceParser/hubert_manager.py b/VoiceParser/hubert_manager.py new file mode 100644 index 0000000000000000000000000000000000000000..5f8445147a8997fdb54e1246e9a85af40342c748 --- /dev/null +++ b/VoiceParser/hubert_manager.py @@ -0,0 +1,33 @@ +import os.path +import shutil +import urllib.request + +import huggingface_hub + + +class HuBERTManager: + @staticmethod + def make_sure_hubert_installed(download_url: str = 'https://dl.fbaipublicfiles.com/hubert/hubert_base_ls960.pt', file_name: str = 'hubert.pt'): + install_dir = os.path.join('VoiceParser', 'hubert') + if not os.path.isdir(install_dir): + os.makedirs(install_dir, exist_ok=True) + install_file = os.path.join(install_dir, file_name) + if not os.path.isfile(install_file): + print('Downloading HuBERT base model') + urllib.request.urlretrieve(download_url, install_file) + print('Downloaded HuBERT') + return install_file + + + @staticmethod + def make_sure_tokenizer_installed(model: str = 'quantifier_hubert_base_ls960_14.pth', repo: str = 'GitMylo/bark-voice-cloning', local_file: str = 'tokenizer.pth'): + install_dir = os.path.join('VoiceParser', 'hubert') + if not os.path.isdir(install_dir): + os.makedirs(install_dir, exist_ok=True) + install_file = os.path.join(install_dir, local_file) + if not os.path.isfile(install_file): + print('Downloading HuBERT custom tokenizer') + huggingface_hub.hf_hub_download(repo, model, local_dir=install_dir, local_dir_use_symlinks=False) + shutil.move(os.path.join(install_dir, model), install_file) + print('Downloaded tokenizer') + return install_file \ No newline at end of file diff --git a/VoiceParser/model.py b/VoiceParser/model.py new file mode 100644 index 0000000000000000000000000000000000000000..1fd265241d43953a04578a43fe0248dd2233348a --- /dev/null +++ b/VoiceParser/model.py @@ -0,0 +1,102 @@ +import os +import json +import numpy as np + +import torch +import torchaudio +torchaudio.set_audio_backend("soundfile") # Use 'soundfile' backend + +from encodec import EncodecModel +from encodec.utils import convert_audio +from .hubert_manager import HuBERTManager +from .pre_kmeans_hubert import CustomHubert +from .customtokenizer import CustomTokenizer + +class VoiceParser(): + def __init__(self, device='cpu'): + model = ('quantifier_hubert_base_ls960_14.pth', 'tokenizer.pth') + + hubert_model = CustomHubert(HuBERTManager.make_sure_hubert_installed(), device=device) + quant_model = CustomTokenizer.load_from_checkpoint(HuBERTManager.make_sure_tokenizer_installed(model=model[0], local_file=model[1]), device) + encodec_model = EncodecModel.encodec_model_24khz() + encodec_model.set_target_bandwidth(6.0) + + self.hubert_model = hubert_model + self.quant_model = quant_model + self.encodec_model = encodec_model.to(device) + self.device = device + print('Loaded VoiceParser models!') + + + def extract_acoustic_embed(self, wav_path, npz_dir): + wav, sr = torchaudio.load(wav_path) + + wav_hubert = wav.to(self.device) + + if wav_hubert.shape[0] == 2: # Stereo to mono if needed + wav_hubert = wav_hubert.mean(0, keepdim=True) + + semantic_vectors = self.hubert_model.forward(wav_hubert, input_sample_hz=sr) + semantic_tokens = self.quant_model.get_token(semantic_vectors) + wav = convert_audio(wav, sr, self.encodec_model.sample_rate, 1).unsqueeze(0) + + wav = wav.to(self.device) + + with torch.no_grad(): + encoded_frames = self.encodec_model.encode(wav) + + codes = torch.cat([encoded[0] for encoded in encoded_frames], dim=-1).squeeze() + + codes = codes.cpu() + semantic_tokens = semantic_tokens.cpu() + + wav_name = os.path.split(wav_path)[1] + npz_name = wav_name[:-4] + '.npz' + npz_path = os.path.join(npz_dir, npz_name) + + np.savez( + npz_path, + semantic_prompt=semantic_tokens, + fine_prompt=codes, + coarse_prompt=codes[:2, :] + ) + + return npz_path + + + def read_json_file(self, json_path): + with open(json_path, 'r') as file: + data = json.load(file) + return data + + + def parse_voice_json(self, voice_json, output_dir): + """ + Parse a voice json file, generate the corresponding output json and npz files + Params: + voice_json: path of a json file or List of json nodes + output_dir: output dir for new json and npz files + """ + if isinstance(voice_json, list): + voice_json = voice_json + else: + # If voice_json is a file path (str), read the JSON file + voice_json = self.read_json_file(voice_json) + for item in voice_json: + wav_path = item['wav'] + npz_path = self.extract_acoustic_embed(wav_path=wav_path, npz_dir=output_dir) + item['npz'] = npz_path + del item['wav'] + + output_json = os.path.join(output_dir, 'metadata.json') + + with open(output_json, 'w') as file: + json.dump(voice_json, file, indent=4) + + + + + + + + diff --git a/VoiceParser/pre_kmeans_hubert.py b/VoiceParser/pre_kmeans_hubert.py new file mode 100644 index 0000000000000000000000000000000000000000..ca09cf7817a00130ac0d84ecc9e45c044f613360 --- /dev/null +++ b/VoiceParser/pre_kmeans_hubert.py @@ -0,0 +1,106 @@ +""" +Modified HuBERT model without kmeans. +Original author: https://github.com/lucidrains/ +Modified by: https://www.github.com/gitmylo/ +License: MIT +""" + +# Modified code from https://github.com/lucidrains/audiolm-pytorch/blob/main/audiolm_pytorch/hubert_kmeans.py + +from pathlib import Path + +import torch +from torch import nn +from einops import pack, unpack + +import fairseq + +from torchaudio.functional import resample + +from audiolm_pytorch.utils import curtail_to_multiple + +import logging +logging.root.setLevel(logging.ERROR) + + +def exists(val): + return val is not None + + +def default(val, d): + return val if exists(val) else d + + +class CustomHubert(nn.Module): + """ + checkpoint and kmeans can be downloaded at https://github.com/facebookresearch/fairseq/tree/main/examples/hubert + or you can train your own + """ + + def __init__( + self, + checkpoint_path, + target_sample_hz=16000, + seq_len_multiple_of=None, + output_layer=9, + device=None + ): + super().__init__() + self.target_sample_hz = target_sample_hz + self.seq_len_multiple_of = seq_len_multiple_of + self.output_layer = output_layer + + if device is not None: + self.to(device) + + model_path = Path(checkpoint_path) + + assert model_path.exists(), f'path {checkpoint_path} does not exist' + + checkpoint = torch.load(checkpoint_path, map_location=device) + load_model_input = {checkpoint_path: checkpoint} + model, *_ = fairseq.checkpoint_utils.load_model_ensemble_and_task(load_model_input) + + if device is not None: + model[0].to(device) + + self.model = model[0] + self.model.eval() + + @property + def groups(self): + return 1 + + @torch.no_grad() + def forward( + self, + wav_input, + flatten=True, + input_sample_hz=None + ): + device = wav_input.device + + if exists(input_sample_hz): + wav_input = resample(wav_input, input_sample_hz, self.target_sample_hz) + + if exists(self.seq_len_multiple_of): + wav_input = curtail_to_multiple(wav_input, self.seq_len_multiple_of) + + embed = self.model( + wav_input, + features_only=True, + mask=False, # thanks to @maitycyrus for noticing that mask is defaulted to True in the fairseq code + output_layer=self.output_layer + ) + + embed, packed_shape = pack([embed['x']], '* d') + + # codebook_indices = self.kmeans.predict(embed.cpu().detach().numpy()) + + codebook_indices = torch.from_numpy(embed.cpu().detach().numpy()).to(device) # .long() + + if flatten: + return codebook_indices + + codebook_indices, = unpack(codebook_indices, packed_shape, '*') + return codebook_indices \ No newline at end of file diff --git a/add_voice_preset.py b/add_voice_preset.py new file mode 100644 index 0000000000000000000000000000000000000000..70cbd91ae5afb25fdad3f6cb1a23bc4a96a34569 --- /dev/null +++ b/add_voice_preset.py @@ -0,0 +1,21 @@ +import argparse +import voice_presets + +def main(): + # Argument Parsing + parser = argparse.ArgumentParser(description="Add Voice Preset") + parser.add_argument("--id", required=True, help="ID of the voice") + parser.add_argument("--desc", required=True, help="Description of the voice") + parser.add_argument("--wav-path", required=True, help="Path to the .wav file") + parser.add_argument("--session-id", required=True, help="session_id, if set to '' then it's system voice presets") + args = parser.parse_args() + + if args.session_id: + print(voice_presets.add_session_voice_preset(args.id, args.desc, args.wav_path, args.session_id)) + else: + print(voice_presets.add_system_voice_preset(args.id, args.desc, args.wav_path)) + + + +if __name__ == "__main__": + main() diff --git a/code_generator.py b/code_generator.py new file mode 100644 index 0000000000000000000000000000000000000000..8a2192251d9df420a96430e602d5d115fac79047 --- /dev/null +++ b/code_generator.py @@ -0,0 +1,188 @@ +import os +import json5 +import utils + + +def check_json_script(data): + foreground_mandatory_attrs_map = { + 'music': ['vol', 'len', 'desc'], + 'sound_effect': ['vol', 'len', 'desc'], + 'speech': ['vol', 'text'] + } + background_mandatory_attrs_map = { + 'music': ['vol', 'desc'], + 'sound_effect': ['vol', 'desc'], + } + + def check_by_audio_type(audio, mandatory_attrs_map, audio_str): + if audio['audio_type'] not in mandatory_attrs_map: + raise ValueError('audio_type is not allowed in this layout, audio={audio_str}') + for attr_name in mandatory_attrs_map[audio['audio_type']]: + if attr_name not in audio: + raise ValueError(f'{attr_name} does not exist, audio={audio_str}') + + # Check json's format + for audio in data: + audio_str = json5.dumps(audio, indent=None) + if 'layout' not in audio: + raise ValueError(f'layout missing, audio={audio_str}') + elif 'audio_type' not in audio: + raise ValueError(f'audio_type missing, audio={audio_str}') + elif audio['layout'] == 'foreground': + check_by_audio_type(audio, foreground_mandatory_attrs_map, audio_str) + elif audio['layout'] == 'background': + if 'id' not in audio: + raise ValueError(f'id not in background audio, audio={audio_str}') + if 'action' not in audio: + raise ValueError(f'action not in background audio, audio={audio_str}') + if audio['action'] == 'begin': + check_by_audio_type(audio, background_mandatory_attrs_map, audio_str) + else: + if audio['action'] != 'end': + raise ValueError(f'Unknown action, audio={audio_str}') + else: + raise ValueError(f'Unknown layout, audio={audio_str}') + #except Exception as err: + # sys.stderr.write(f'PARSING ERROR: {err}, audio={json5.dumps(audio, indent=None)}\n') + # all_clear = False + + +def collect_and_check_audio_data(data): + fg_audio_id = 0 + fg_audios = [] + bg_audios = [] + # Collect all the foreground and background audio ids used to calculate background audio length later + for audio in data: + if audio['layout'] == 'foreground': + audio['id'] = fg_audio_id + fg_audios.append(audio) + fg_audio_id += 1 + else: # background + if audio['action'] == 'begin': + audio['begin_fg_audio_id'] = fg_audio_id + bg_audios.append(audio) + else: # ends + # find the backgound with the id, and update its 'end_fg_audio_id' + for bg_audio in bg_audios: + if bg_audio['id'] == audio['id'] and bg_audio['audio_type'] == audio['audio_type']: + bg_audio['end_fg_audio_id'] = fg_audio_id + break + + # check if all background audios are valid + for bg_audio in bg_audios: + if 'begin_fg_audio_id' not in bg_audio: + raise ValueError(f'begin of background missing, audio={bg_audio}') + elif 'end_fg_audio_id' not in bg_audio: + raise ValueError(f'end of background missing, audio={bg_audio}') + + if bg_audio['begin_fg_audio_id'] > bg_audio['end_fg_audio_id']: + raise ValueError(f'background audio ends before start, audio={bg_audio}') + elif bg_audio['begin_fg_audio_id'] == bg_audio['end_fg_audio_id']: + raise ValueError(f'background audio contains no foreground audio, audio={bg_audio}') + #except Exception as err: + # sys.stderr.write(f'ALIGNMENT ERROR: {err}, audio={bg_audio}\n') + # return None, None + + return fg_audios, bg_audios + + +class AudioCodeGenerator: + def __init__(self): + self.wav_counters = { + 'bg_sound_effect': 0, + 'bg_music': 0, + 'idle': 0, + 'fg_sound_effect': 0, + 'fg_music': 0, + 'fg_speech': 0, + } + self.code = '' + + def append_code(self, content): + self.code = f'{self.code}{content}\n' + + def generate_code(self, fg_audios, bg_audios, output_path, result_filename): + def get_wav_name(audio): + audio_type = audio['audio_type'] + layout = 'fg' if audio['layout'] == 'foreground' else 'bg' + wav_type = f'{layout}_{audio_type}' if layout else audio_type + desc = audio['text'] if 'text' in audio else audio['desc'] + desc = utils.text_to_abbrev_prompt(desc) + wav_filename = f'{wav_type}_{self.wav_counters[wav_type]}_{desc}.wav' + self.wav_counters[wav_type] += 1 + return wav_filename + + header = f''' +import os +import sys +import datetime + +from APIs import TTM, TTS, TTA, MIX, CAT, COMPUTE_LEN + + +fg_audio_lens = [] +wav_path = \"{output_path.absolute()}/audio\" +os.makedirs(wav_path, exist_ok=True) + +''' + self.append_code(header) + + fg_audio_wavs = [] + for fg_audio in fg_audios: + wav_name = get_wav_name(fg_audio) + if fg_audio['audio_type'] == 'sound_effect': + self.append_code(f'TTA(text=\"{fg_audio["desc"]}\", length={fg_audio["len"]}, volume={fg_audio["vol"]}, out_wav=os.path.join(wav_path, \"{wav_name}\"))') + elif fg_audio['audio_type'] == 'music': + self.append_code(f'TTM(text=\"{fg_audio["desc"]}\", length={fg_audio["len"]}, volume={fg_audio["vol"]}, out_wav=os.path.join(wav_path, \"{wav_name}\"))') + elif fg_audio['audio_type'] == 'speech': + npz_path = self.char_to_voice_map[fg_audio["character"]]["npz_path"] + npz_full_path = os.path.abspath(npz_path) if os.path.exists(npz_path) else npz_path + self.append_code(f'TTS(text=\"{fg_audio["text"]}\", speaker_id=\"{self.char_to_voice_map[fg_audio["character"]]["id"]}\", volume={fg_audio["vol"]}, out_wav=os.path.join(wav_path, \"{wav_name}\"), speaker_npz=\"{npz_full_path}\")') + fg_audio_wavs.append(wav_name) + self.append_code(f'fg_audio_lens.append(COMPUTE_LEN(os.path.join(wav_path, \"{wav_name}\")))\n') + + # cat all foreground audio together + self.append_code(f'fg_audio_wavs = []') + for wav_filename in fg_audio_wavs: + self.append_code(f'fg_audio_wavs.append(os.path.join(wav_path, \"{wav_filename}\"))') + self.append_code(f'CAT(wavs=fg_audio_wavs, out_wav=os.path.join(wav_path, \"foreground.wav\"))') + + bg_audio_wavs = [] + self.append_code(f'\nbg_audio_offsets = []') + for bg_audio in bg_audios: + wav_name = get_wav_name(bg_audio) + self.append_code(f'bg_audio_len = sum(fg_audio_lens[{bg_audio["begin_fg_audio_id"]}:{bg_audio["end_fg_audio_id"]}])') + self.append_code(f'bg_audio_offset = sum(fg_audio_lens[:{bg_audio["begin_fg_audio_id"]}])') + if bg_audio['audio_type'] == 'sound_effect': + self.append_code(f'TTA(text=\"{bg_audio["desc"]}\", volume={bg_audio["vol"]}, length=bg_audio_len, out_wav=os.path.join(wav_path, \"{wav_name}\"))') + elif bg_audio['audio_type'] == 'music': + self.append_code(f'TTM(text=\"{bg_audio["desc"]}\", volume={bg_audio["vol"]}, length=bg_audio_len, out_wav=os.path.join(wav_path, \"{wav_name}\"))') + else: + raise ValueError() + bg_audio_wavs.append(wav_name) + self.append_code(f'bg_audio_offsets.append(bg_audio_offset)\n') + self.append_code(f'bg_audio_wavs = []') + for wav_filename in bg_audio_wavs: + self.append_code(f'bg_audio_wavs.append(os.path.join(wav_path, \"{wav_filename}\"))') + + self.append_code(f'bg_audio_wav_offset_pairs = list(zip(bg_audio_wavs, bg_audio_offsets))') + self.append_code(f'bg_audio_wav_offset_pairs.append((os.path.join(wav_path, \"foreground.wav\"), 0))') + self.append_code(f'MIX(wavs=bg_audio_wav_offset_pairs, out_wav=os.path.join(wav_path, \"{result_filename}.wav\"))') + + + def init_char_to_voice_map(self, filename): + with open(filename, 'r') as file: + self.char_to_voice_map = json5.load(file) + + + def parse_and_generate(self, script_filename, char_to_voice_map_filename, output_path, result_filename='result'): + self.code = '' + self.init_char_to_voice_map(char_to_voice_map_filename) + + with open(script_filename, 'r') as file: + data = json5.load(file) + + check_json_script(data) + fg_audios, bg_audios = collect_and_check_audio_data(data) + self.generate_code(fg_audios, bg_audios, output_path, result_filename) + return self.code diff --git a/config.yaml b/config.yaml new file mode 100644 index 0000000000000000000000000000000000000000..429208544dda0efa028fd7fc6a75faf5b8506e89 --- /dev/null +++ b/config.yaml @@ -0,0 +1,17 @@ +AudioCraft: + # MusicGen + ttm_model_size: small # [small, medium, large] + # AudioGen + tta_model_size: medium # [medium] + +Text-to-Speech: + # Bark + speed: 1.05 + +Speech-Restoration: + # VoiceFixer + Enable: True + +Voice-Parser: + # HuBERT + device: 'cpu' diff --git a/convert_json_to_audio_gen_code.py b/convert_json_to_audio_gen_code.py new file mode 100644 index 0000000000000000000000000000000000000000..284cd1f0d4844c9bc5489999aa22b47aac598137 --- /dev/null +++ b/convert_json_to_audio_gen_code.py @@ -0,0 +1,30 @@ +import argparse +import os +import json5 +from pathlib import Path +from code_generator import AudioCodeGenerator + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--script", help="Path to the json script file") + parser.add_argument("--character-to-voice-map", help="Path to the character-to-voice mapping CSV file") + parser.add_argument( + "--path", + type=str, + default=".", + help="Path of all the output wav files to be created by the generated code, default: current path" + ) + args = parser.parse_args() + + if not os.path.isfile(args.script): + print(f"File {args.script} does not exist.") + return + + output_path = Path(args.path) + audio_code_generator = AudioCodeGenerator() + code = audio_code_generator.parse_and_generate(args.script, args.character_to_voice_map, output_path) + print(code) + +if __name__ == "__main__": + main() diff --git a/data/voice_presets/metadata.json b/data/voice_presets/metadata.json new file mode 100644 index 0000000000000000000000000000000000000000..2af50b0a9e10c92be7a4b30fd65a693be56214d0 --- /dev/null +++ b/data/voice_presets/metadata.json @@ -0,0 +1,272 @@ +{ + "Male1_En": { + "id": "Male1_En", + "desc": "A normal male adult voice, British accent; Language: English.", + "npz_path": "v2/en_speaker_1" + }, + "Male2_En": { + "id": "Male2_En", + "desc": "A normal male adult voice, American accent; Language: English.", + "npz_path": "v2/en_speaker_6" + }, + "Female1_En": { + "id": "Female1_En", + "desc": "A normal female adult voice, British accent; Language: English.", + "npz_path": "v2/en_speaker_9" + }, + "Female2_En": { + "id": "Female2_En", + "desc": "A normal female adult voice, American accent; Language: English.", + "npz_path": "v2/de_speaker_3" + }, + "News_Male_En": { + "id": "News_Male_En", + "desc": "A male voice of a news anchor, suitable for news scenarios; Language: English.", + "npz_path": "data/voice_presets/npz/news_male_speaker.npz" + }, + "News_Female_En": { + "id": "News_Female_En", + "desc": "A female voice of a news anchor, suitable for news scenarios; Language: English.", + "npz_path": "data/voice_presets/npz/news_male_speaker.npz" + }, + "News_Female_Out_En": { + "id": "News_Female_Out_En", + "desc": "A female voice of a off-site news reporter, suitable for news scenario; Language: English.", + "npz_path": "data/voice_presets/npz/news_female_speaker_outside.npz" + }, + "Child_En": { + "id": "Child_En", + "desc": "A small young boy voice; Language: English.", + "npz_path": "data/voice_presets/npz/child_boy.npz" + }, + "Old_Man_En": { + "id": "Old_Man_En", + "desc": "A voice of an old man; Language: English.", + "npz_path": "data/voice_presets/npz/elder_morgen.npz" + }, + "Male1_Zh": { + "id": "Male1_Zh", + "desc": "A normal male adult voice; Language: Chinese.", + "npz_path": "v2/zh_speaker_0" + }, + "Male2_Zh": { + "id": "Male2_Zh", + "desc": "A normal male adult voice; Language: Chinese.", + "npz_path": "v2/zh_speaker_1" + }, + "Female1_Zh": { + "id": "Female1_Zh", + "desc": "A normal female adult voice; Language: Chinese.", + "npz_path": "v2/zh_speaker_9" + }, + "Female2_Zh": { + "id": "Female2_Zh", + "desc": "A normal female adult voice; Language: Chinese.", + "npz_path": "v2/zh_speaker_4" + }, + "Male1_Fr": { + "id": "Male1_Fr", + "desc": "A normal male adult voice; Language: French.", + "npz_path": "v2/fr_speaker_0" + }, + "Male2_Fr": { + "id": "Male2_Fr", + "desc": "A normal male adult voice; Language: French.", + "npz_path": "v2/fr_speaker_8" + }, + "Female1_Fr": { + "id": "Female1_Fr", + "desc": "A normal female adult voice; Language: French.", + "npz_path": "v2/fr_speaker_5" + }, + "Female2_Fr": { + "id": "Female2_Fr", + "desc": "A normal female adult voice; Language: French.", + "npz_path": "v2/fr_speaker_1" + }, + "Male1_De": { + "id": "Male1_De", + "desc": "A normal male adult voice; Language: German.", + "npz_path": "v2/de_speaker_0" + }, + "Male2_De": { + "id": "Male2_De", + "desc": "A normal male adult voice; Language: German.", + "npz_path": "v2/de_speaker_1" + }, + "Female1_De": { + "id": "Female1_De", + "desc": "A normal female adult voice; Language: German.", + "npz_path": "v2/de_speaker_3" + }, + "Female2_De": { + "id": "Female2_De", + "desc": "A normal female adult voice; Language: German.", + "npz_path": "v2/de_speaker_8" + }, + "Male1_Hi": { + "id": "Male1_Hi", + "desc": "A normal male adult voice; Language: Hindi.", + "npz_path": "v2/hi_speaker_5" + }, + "Male2_Hi": { + "id": "Male2_Hi", + "desc": "A normal male adult voice; Language: Hindi.", + "npz_path": "v2/hi_speaker_8" + }, + "Female1_Hi": { + "id": "Female1_Hi", + "desc": "A normal female adult voice; Language: Hindi.", + "npz_path": "v2/hi_speaker_0" + }, + "Female2_Hi": { + "id": "Female2_Hi", + "desc": "A normal female adult voice; Language: Hindi.", + "npz_path": "v2/hi_speaker_3" + }, + "Male1_It": { + "id": "Male1_It", + "desc": "A normal male adult voice; Language: Italian.", + "npz_path": "v2/it_speaker_4" + }, + "Male2_It": { + "id": "Male2_It", + "desc": "A normal male adult voice; Language: Italian.", + "npz_path": "v2/it_speaker_5" + }, + "Female1_It": { + "id": "Female1_It", + "desc": "A normal female adult voice; Language: Italian.", + "npz_path": "v2/it_speaker_7" + }, + "Female2_It": { + "id": "Female2_It", + "desc": "A normal female adult voice; Language: Italian.", + "npz_path": "v2/it_speaker_9" + }, + "Male1_Ja": { + "id": "Male1_Ja", + "desc": "A normal male adult voice; Language: Japanese.", + "npz_path": "v2/ja_speaker_2" + }, + "Male2_Ja": { + "id": "Male2_Ja", + "desc": "A normal male adult voice; Language: Japanese.", + "npz_path": "v2/ja_speaker_6" + }, + "Female1_Ja": { + "id": "Female1_Ja", + "desc": "A normal female adult voice; Language: Japanese.", + "npz_path": "v2/ja_speaker_4" + }, + "Female2_Ja": { + "id": "Female2_Ja", + "desc": "A normal female adult voice; Language: Japanese.", + "npz_path": "v2/ja_speaker_5" + }, + "Male1_Ko": { + "id": "Male1_Ko", + "desc": "A normal male adult voice; Language: Korean.", + "npz_path": "v2/ko_speaker_1" + }, + "Male2_Ko": { + "id": "Male2_Ko", + "desc": "A normal male adult voice; Language: Korean.", + "npz_path": "v2/ko_speaker_2" + }, + "Female1_Ko": { + "id": "Female1_Ko", + "desc": "A normal female adult voice; Language: Korean.", + "npz_path": "v2/ko_speaker_0" + }, + "Female1_Ru": { + "id": "Female1_Ru", + "desc": "A normal female adult voice; Language: Russian.", + "npz_path": "v2/ru_speaker_5" + }, + "Female2_Ru": { + "id": "Female2_Ru", + "desc": "A normal female adult voice; Language: Russian.", + "npz_path": "v2/ru_speaker_6" + }, + "Male1_Ru": { + "id": "Male1_Ru", + "desc": "A normal male adult voice; Language: Russian.", + "npz_path": "v2/ru_speaker_3" + }, + "Male2_Ru": { + "id": "Male2_Ru", + "desc": "A normal male adult voice; Language: Russian.", + "npz_path": "v2/ru_speaker_4" + }, + "Female1_Es": { + "id": "Female1_Es", + "desc": "A normal female adult voice; Language: Spanish.", + "npz_path": "v2/es_speaker_8" + }, + "Female2_Es": { + "id": "Female2_Es", + "desc": "A normal female adult voice; Language: Spanish.", + "npz_path": "v2/es_speaker_9" + }, + "Male1_Es": { + "id": "Male1_Es", + "desc": "A normal male adult voice; Language: Spanish.", + "npz_path": "v2/es_speaker_6" + }, + "Male2_Es": { + "id": "Male2_Es", + "desc": "A normal male adult voice; Language: Spanish.", + "npz_path": "v2/es_speaker_7" + }, + "Female1_Tr": { + "id": "Female1_Tr", + "desc": "A normal female adult voice; Language: Turkish.", + "npz_path": "v2/tr_speaker_4" + }, + "Female2_Tr": { + "id": "Female2_Tr", + "desc": "A normal female adult voice; Language: Turkish.", + "npz_path": "v2/tr_speaker_5" + }, + "Male1_Tr": { + "id": "Male1_Tr", + "desc": "A normal male adult voice; Language: Turkish.", + "npz_path": "v2/tr_speaker_2" + }, + "Male2_Tr": { + "id": "Male2_Tr", + "desc": "A normal male adult voice; Language: Turkish.", + "npz_path": "v2/tr_speaker_3" + }, + "Male1_Pt": { + "id": "Male1_Pt", + "desc": "A normal male adult voice; Language: Purtuguese.", + "npz_path": "v2/pt_speaker_0" + }, + "Male2_Pt": { + "id": "Male2_Pt", + "desc": "A normal male adult voice; Language: Purtuguese.", + "npz_path": "v2/pt_speaker_1" + }, + "Female1_Pl": { + "id": "Female1_Pl", + "desc": "A normal female adult voice; Language: Polish.", + "npz_path": "v2/pl_speaker_4" + }, + "Female2_Pl": { + "id": "Female2_Pl", + "desc": "A normal female adult voice; Language: Polish.", + "npz_path": "v2/pl_speaker_6" + }, + "Male1_Pl": { + "id": "Male1_Pl", + "desc": "A normal male adult voice; Language: Polish.", + "npz_path": "v2/pl_speaker_5" + }, + "Male2_Pl": { + "id": "Male2_Pl", + "desc": "A normal male adult voice; Language: Polish.", + "npz_path": "v2/pl_speaker_7" + } +} \ No newline at end of file diff --git a/data/voice_presets/npz/child_boy.npz b/data/voice_presets/npz/child_boy.npz new file mode 100644 index 0000000000000000000000000000000000000000..6b539a8712b75e1f010b7193a0fcd0f7fb11fca8 Binary files /dev/null and b/data/voice_presets/npz/child_boy.npz differ diff --git a/data/voice_presets/npz/cnn_male_speaker.npz b/data/voice_presets/npz/cnn_male_speaker.npz new file mode 100644 index 0000000000000000000000000000000000000000..e2fbc1f557d2b89456d68c3cf43d3d92c7bb20eb Binary files /dev/null and b/data/voice_presets/npz/cnn_male_speaker.npz differ diff --git a/data/voice_presets/npz/elder_morgen.npz b/data/voice_presets/npz/elder_morgen.npz new file mode 100644 index 0000000000000000000000000000000000000000..7de1e16a9f2edd82350c4645e51819b7964c74ac Binary files /dev/null and b/data/voice_presets/npz/elder_morgen.npz differ diff --git a/data/voice_presets/npz/news_female_speaker.npz b/data/voice_presets/npz/news_female_speaker.npz new file mode 100644 index 0000000000000000000000000000000000000000..013dea952ecc7c34c601bfc8d25fc367cffb5584 Binary files /dev/null and b/data/voice_presets/npz/news_female_speaker.npz differ diff --git a/data/voice_presets/npz/news_female_speaker_outside.npz b/data/voice_presets/npz/news_female_speaker_outside.npz new file mode 100644 index 0000000000000000000000000000000000000000..2cd5f5964c778742ab200bf71f8bad83db6fdec3 Binary files /dev/null and b/data/voice_presets/npz/news_female_speaker_outside.npz differ diff --git a/data/voice_presets/npz/news_male_speaker.npz b/data/voice_presets/npz/news_male_speaker.npz new file mode 100644 index 0000000000000000000000000000000000000000..744b9afe7ca626c1e3f70adaea1a13c67bb0796c Binary files /dev/null and b/data/voice_presets/npz/news_male_speaker.npz differ diff --git a/examples/1.mp4 b/examples/1.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..ccef6d743419a62c247aa040090ceebf9d30800a Binary files /dev/null and b/examples/1.mp4 differ diff --git a/examples/2.mp4 b/examples/2.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..a4bb714f2ccd5fe5195ffb97bb753c89c34ea5be Binary files /dev/null and b/examples/2.mp4 differ diff --git a/examples/3.mp4 b/examples/3.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..038780067173a0b053714f8f9b9f18ab6acb7ff9 Binary files /dev/null and b/examples/3.mp4 differ diff --git a/examples/example1.wav b/examples/example1.wav new file mode 100644 index 0000000000000000000000000000000000000000..9f12805a840cf6d95161c2e6d2f6c485907ed3a8 Binary files /dev/null and b/examples/example1.wav differ diff --git a/examples/example2.wav b/examples/example2.wav new file mode 100644 index 0000000000000000000000000000000000000000..9f12805a840cf6d95161c2e6d2f6c485907ed3a8 Binary files /dev/null and b/examples/example2.wav differ diff --git a/examples/examples.py b/examples/examples.py new file mode 100644 index 0000000000000000000000000000000000000000..9a44a090cadc95d594eac3c2d47c1ac7a2c10466 --- /dev/null +++ b/examples/examples.py @@ -0,0 +1,87 @@ + +example1 = { + 'text': "An introduction to AI-assisted audio content creation.", + 'table_script': """ +| Audio Type | Layout | ID | Character | Action | Volume | Description | Length | +|--------------|------------|----|-----------|--------|--------|------------------------------------------------------------------|--------| +| music | background | 1 | N/A | begin | -35 | Inspirational technology-themed music | Auto | +| speech | foreground | N/A| Narrator | N/A | -15 | Welcome to the future of audio content creation. | Auto | +| sound_effect | foreground | N/A| N/A | N/A | -35 | Digital startup sound | 2 | +| speech | foreground | N/A| Narrator | N/A | -15 | With evolving technology, we are introducing AI-assisted tools for pristine audio production. | Auto | +| sound_effect | foreground | N/A| N/A | N/A | -35 | Keyboard typing noise | 3 | +| speech | foreground | N/A| Narrator | N/A | -15 | Imagine crafting audio content with the power of AI at your fingertips. | Auto | +| sound_effect | background | 2 | N/A | begin | -35 | Ambiance of a busy control room | Auto | +| speech | foreground | N/A| Narrator | N/A | -15 | Enhanced quality, efficient production and limitless creativity, all under one roof. | Auto | +| sound_effect | background | 2 | N/A | end | N/A | N/A | Auto | +| speech | foreground | N/A| Narrator | N/A | -15 | Unleash your potential with AI-assisted audio content creation. | Auto | +| music | background | 1 | N/A | end | N/A | N/A | Auto | + +""", + 'table_voice': """ +| Character | Voice | +|-------------|-----------| +| Narrator | News_Male_En | + +""", + 'wav_file': 'examples/1.mp4', +} + +example2 = { + 'text': "A couple dating in a cafe.", + 'table_script': """ +| Audio Type | Layout | ID | Character | Action | Volume | Description | Length | +|--------------|------------|----|-----------|--------|--------|-----------------------------------------------|--------| +| sound_effect | background | 1 | N/A | begin | -35 | Soft chattering in a cafe | Auto | +| sound_effect | background | 2 | N/A | begin | -38 | Coffee brewing noises | Auto | +| music | background | 3 | N/A | begin | -35 | Soft jazz playing in the background | Auto | +| speech | foreground | N/A| Man | N/A | -15 | It’s really nice to finally get out and relax a little, isn’t it? | Auto | +| speech | foreground | N/A| Woman | N/A | -15 | I know, right? We should do this more often. | Auto | +| sound_effect | background | 2 | N/A | end | N/A | N/A | Auto | +| speech | foreground | N/A| Man | N/A | -15 | Here’s your coffee, just as you like it. | Auto | +| speech | foreground | N/A| Woman | N/A | -15 | Thank you, it smells wonderful. | Auto | +| music | background | 3 | N/A | end | N/A | N/A | Auto | +| sound_effect | background | 1 | N/A | end | N/A | N/A | Auto | + +""", + 'table_voice': """ +| Character | Voice | +|-------------|-----------| +| Man | Male1_En | +| Woman | Female1_En | + +""", + 'wav_file': 'examples/2.mp4', +} + + +example3 = { + 'text': "A child is participating in a farting contest.", + 'table_script': """ +| Audio Type | Layout | ID | Character | Action | Volume | Description | Length | +|--------------|------------|----|-----------|--------|--------|------------------------------------------------------|--------| +| sound_effect | background | 1 | N/A | begin | -35 | Outdoor park ambiance, people chattering | Auto | +| music | background | 2 | N/A | begin | -35 | Light comedy theme music, quirky | Auto | +| speech | foreground | N/A| Host | N/A | -15 | Welcome to the annual Fart Competition. | Auto | +| speech | foreground | N/A| Host | N/A | -15 | Now, let’s welcome our youngest participant. | Auto | +| sound_effect | foreground | N/A| N/A | N/A | -35 | Clapping sound | 2 | +| speech | foreground | N/A| Child | N/A | -15 | Hi, I’m excited to be here. | Auto | +| sound_effect | foreground | N/A| N/A | N/A | -35 | Short, cartoonish duration of a fart sound | 4 | +| sound_effect | foreground | N/A| N/A | N/A | -35 | Audience laughing and applauding | 2 | +| speech | foreground | N/A| Host | N/A | -15 | Wow, that was impressive! Let’s give another round of applause! | Auto | +| sound_effect | foreground | N/A| N/A | N/A | -35 | Audience clapping and cheering | 3 | +| music | background | 2 | N/A | end | N/A | N/A | Auto | +| sound_effect | background | 1 | N/A | end | N/A | N/A | Auto | +""", + 'table_voice': """ +| Character | Voice | +|-------------|-----------| +| Host | Male1_En | +| Child | Child_En | + +""", + 'wav_file': 'examples/3.mp4', +} + + + +examples = [example1, example2, example3] \ No newline at end of file diff --git a/parse_voice.py b/parse_voice.py new file mode 100644 index 0000000000000000000000000000000000000000..9583f402cfb23aede18d421befd2508633b1d23c --- /dev/null +++ b/parse_voice.py @@ -0,0 +1,31 @@ +import os +import argparse +from VoiceParser.model import VoiceParser + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument('--wav-path', type=str, help="Path of a wav file") + parser.add_argument('--wav-dir', type=str, help="Directory of wav files") + parser.add_argument('--out-dir', type=str, help="Directory of output npz files") + args = parser.parse_args() + + if (args.wav_path is None and args.wav_dir is None) or (args.wav_path is not None and args.wav_dir is not None): + parser.error("Please provide either '--wav-path' or '--wav-dir', but not both.") + + out_dir = args.out_dir + + model = VoiceParser(device='cpu') + + if args.wav_path is not None: + model.extract_acoustic_embed(args.wav_path, out_dir) + print(f'Sucessfully parsed {args.wav_path}') + else: + wav_name_list = os.listdir(args.wav_dir) + for wav_name in wav_name_list: + wav_path = os.path.join(args.wav_dir, wav_name) + model.extract_acoustic_embed(wav_path, out_dir) + print(f'Sucessfully parsed {wav_path}') + + +if __name__ == '__main__': + main() diff --git a/pipeline.py b/pipeline.py new file mode 100644 index 0000000000000000000000000000000000000000..9c93b09568d5ada73175c8c3ea6ab8046878a05b --- /dev/null +++ b/pipeline.py @@ -0,0 +1,229 @@ +import datetime +import os +from string import Template +import openai +import re +import glob +import pickle +import time +import json5 +from retrying import retry +from code_generator import check_json_script, collect_and_check_audio_data +import random +import string + +import utils +import voice_presets +from code_generator import AudioCodeGenerator + +# Enable this for debugging +USE_OPENAI_CACHE = False +openai_cache = [] +if USE_OPENAI_CACHE: + os.makedirs('cache', exist_ok=True) + for cache_file in glob.glob('cache/*.pkl'): + with open(cache_file, 'rb') as file: + openai_cache.append(pickle.load(file)) + +def chat_with_gpt(prompt, api_key): + if USE_OPENAI_CACHE: + filtered_object = list(filter(lambda x: x['prompt'] == prompt, openai_cache)) + if len(filtered_object) > 0: + response = filtered_object[0]['response'] + return response + + try: + openai.api_key = api_key + chat = openai.ChatCompletion.create( + # model="gpt-3.5-turbo", + model="gpt-4", + messages=[ + { + "role": "system", + "content": "You are a helpful assistant." + }, + { + "role": "user", + "content": prompt + } + ] + ) + finally: + openai.api_key = '' + + if USE_OPENAI_CACHE: + cache_obj = { + 'prompt': prompt, + 'response': chat['choices'][0]['message']['content'] + } + with open(f'cache/{time.time()}.pkl', 'wb') as _openai_cache: + pickle.dump(cache_obj, _openai_cache) + openai_cache.append(cache_obj) + + return chat['choices'][0]['message']['content'] + + +def get_file_content(filename): + with open(filename, 'r') as file: + return file.read().strip() + + +def write_to_file(filename, content): + with open(filename, 'w') as file: + file.write(content) + + +def extract_substring_with_quotes(input_string, quotes="'''"): + pattern = f"{quotes}(.*?){quotes}" + matches = re.findall(pattern, input_string, re.DOTALL) + return matches + + +def try_extract_content_from_quotes(content): + if "'''" in content: + return extract_substring_with_quotes(content)[0] + elif "```" in content: + return extract_substring_with_quotes(content, quotes="```")[0] + else: + return content + +def maybe_get_content_from_file(content_or_filename): + if os.path.exists(content_or_filename): + with open(content_or_filename, 'r') as file: + return file.read().strip() + return content_or_filename + + + +# Pipeline Interface Guidelines: +# +# Init calls: +# - Init calls must be called before running the actual steps +# - init_session() is called every time a gradio webpage is loaded +# +# Single Step: +# - takes input (file or content) and output path as input +# - most of time just returns output content +# +# Compositional Step: +# - takes session_id as input (you have session_id, you have all the paths) +# - run a series of steps + +# This is called for every new gradio webpage + +def init_session(session_id=''): + def uid8(): + return ''.join(random.choices(string.ascii_lowercase + string.digits, k=8)) + + if session_id == '': + session_id = f'{datetime.datetime.now().strftime("%Y%m%d%H%M%S")}_{uid8()}' + # create the paths + os.makedirs(utils.get_session_voice_preset_path(session_id)) + os.makedirs(utils.get_session_audio_path(session_id)) + print(f'New session created, session_id={session_id}') + return session_id + +@retry(stop_max_attempt_number=3) +def input_text_to_json_script_with_retry(complete_prompt_path, api_key): + print(" trying ...") + complete_prompt = get_file_content(complete_prompt_path) + json_response = try_extract_content_from_quotes(chat_with_gpt(complete_prompt, api_key)) + json_data = json5.loads(json_response) + + try: + check_json_script(json_data) + collect_and_check_audio_data(json_data) + except Exception as err: + print(f'JSON ERROR: {err}') + retry_complete_prompt = f'{complete_prompt}\n```\n{json_response}```\nThe script above has format error(s). Return the fixed script.\n\nScript:\n' + write_to_file(complete_prompt_path, retry_complete_prompt) + raise err + + return json_response + +# Step 1: input_text to json +def input_text_to_json_script(input_text, output_path, api_key): + input_text = maybe_get_content_from_file(input_text) + text_to_audio_script_prompt = get_file_content('prompts/text_to_json.prompt') + prompt = f'{text_to_audio_script_prompt}\n\nInput text: {input_text}\n\nScript:\n' + complete_prompt_path = output_path / 'complete_input_text_to_audio_script.prompt' + write_to_file(complete_prompt_path, prompt) + audio_script_response = input_text_to_json_script_with_retry(complete_prompt_path, api_key) + generated_audio_script_filename = output_path / 'audio_script.json' + write_to_file(generated_audio_script_filename, audio_script_response) + return audio_script_response + +# Step 2: json to char-voice map +def json_script_to_char_voice_map(json_script, voices, output_path, api_key): + json_script_content = maybe_get_content_from_file(json_script) + prompt = get_file_content('prompts/audio_script_to_character_voice_map.prompt') + presets_str = '\n'.join(f"{preset['id']}: {preset['desc']}" for preset in voices.values()) + prompt = Template(prompt).substitute(voice_and_desc=presets_str) + prompt = f"{prompt}\n\nAudio script:\n'''\n{json_script_content}\n'''\n\noutput:\n" + write_to_file(output_path / 'complete_audio_script_to_char_voice_map.prompt', prompt) + char_voice_map_response = try_extract_content_from_quotes(chat_with_gpt(prompt, api_key)) + char_voice_map = json5.loads(char_voice_map_response) + # enrich char_voice_map with voice preset metadata + complete_char_voice_map = {c: voices[char_voice_map[c]] for c in char_voice_map} + char_voice_map_filename = output_path / 'character_voice_map.json' + write_to_file(char_voice_map_filename, json5.dumps(complete_char_voice_map)) + return complete_char_voice_map + +# Step 3: json to py code +def json_script_and_char_voice_map_to_audio_gen_code(json_script_filename, char_voice_map_filename, output_path, result_filename): + audio_code_generator = AudioCodeGenerator() + code = audio_code_generator.parse_and_generate( + json_script_filename, + char_voice_map_filename, + output_path, + result_filename + ) + write_to_file(output_path / 'audio_generation.py', code) + +# Step 4: py code to final wav +def audio_code_gen_to_result(audio_gen_code_path): + audio_gen_code_filename = audio_gen_code_path / 'audio_generation.py' + os.system(f'PYTHONPATH=. python {audio_gen_code_filename}') + +# Function call used by Gradio: input_text to json +def generate_json_file(session_id, input_text, api_key): + output_path = utils.get_session_path(session_id) + # Step 1 + print(f'session_id={session_id}, Step 1: Writing audio script based on text: {input_text} ...') + return input_text_to_json_script(input_text, output_path, api_key) + +# Function call used by Gradio: json to result wav +def generate_audio(session_id, json_script, api_key): + def count_lines(content): + # Split the string using the newline character and count the non-empty lines + return sum(1 for line in content.split('\n') if line.strip()) + + max_lines = utils.get_max_script_lines() + if count_lines(json_script) > max_lines: + raise ValueError(f'The number of lines of the JSON script has exceeded {max_lines}!') + + output_path = utils.get_session_path(session_id) + output_audio_path = utils.get_session_audio_path(session_id) + voices = voice_presets.get_merged_voice_presets(session_id) + + # Step 2 + print(f'session_id={session_id}, Step 2: Parsing character voice with LLM...') + char_voice_map = json_script_to_char_voice_map(json_script, voices, output_path, api_key) + # Step 3 + json_script_filename = output_path / 'audio_script.json' + char_voice_map_filename = output_path / 'character_voice_map.json' + result_wav_basename = f'res_{session_id}' + print(f'session_id={session_id}, Step 3: Compiling audio script to Python program ...') + json_script_and_char_voice_map_to_audio_gen_code(json_script_filename, char_voice_map_filename, output_path, result_wav_basename) + # Step 4 + print(f'session_id={session_id}, Step 4: Start running Python program ...') + audio_code_gen_to_result(output_path) + + result_wav_filename = output_audio_path / f'{result_wav_basename}.wav' + print(f'Done all processes, result: {result_wav_filename}') + return result_wav_filename, char_voice_map + +# Convenient function call used by wavjourney_cli +def full_steps(session_id, input_text, api_key): + json_script = generate_json_file(session_id, input_text, api_key) + return generate_audio(session_id, json_script, api_key) diff --git a/prompts/audio_script_to_character_voice_map.prompt b/prompts/audio_script_to_character_voice_map.prompt new file mode 100644 index 0000000000000000000000000000000000000000..3e3634613c0998a617d4ea70c90d814661c73d4f --- /dev/null +++ b/prompts/audio_script_to_character_voice_map.prompt @@ -0,0 +1,11 @@ +Given an audio script in json format, for each character appeared in the "character" attribute, you should map the character to a "voice type" according to the his/her lines and the voice type's features. Each character must be mapped to a different voice type, and each voice type must be from one of the following(each line in the format of "[voice_type_id]: [voice_type_description]"): +$voice_and_desc + +Output should be in the format of json, like: +''' +{ + "character_1": "voice_type_1", + "character_2": "voice_type_2", + ... +} +''' \ No newline at end of file diff --git a/prompts/audio_script_to_json.prompt b/prompts/audio_script_to_json.prompt new file mode 100644 index 0000000000000000000000000000000000000000..b7f867e2b309ad590da1dad8b2f44e0fc05f5f90 --- /dev/null +++ b/prompts/audio_script_to_json.prompt @@ -0,0 +1,74 @@ +Given an audio script, adapt it into a json file. You must go through each line of the script, and try your best to convert it to a json object or multiple json objects. + +Each json object represents an audio. There are three types of audios: sound effect, music, and speech. For each audio, there are two types of layouts: foreground and background. Foreground audios are played sequentially, and background audios are environmental sounds or music which are played while the foreground audios are being played. + +While going through each line of the script, you have choices as below: +- For character lines, you need to convert it to a speech audio. Note that a speech audio can only be foreground. Example: +From +``` +News Anchor: Good evening, this is BBC News. +``` +To +``` +{"audio_type": "speech", "layout": "foreground", "character": "News Anchor", "vol": -15, "text": "Good evening, this is BBC News."}, +``` +- For sound effects, you need to convert it to a sound_effect audio. Especially, you need to figure out its length according to the script's context, and put it into "len". Example: +From +``` +(SFX: Airport beeping sound) +``` +to +``` +{"audio_type": "sound_effect", "layout": "foreground", "vol": -35, "len": 2, "desc": "Airport beeping sound"}, +``` +- For music, you need to convert it to a music audio. Especially, you need to figure out its length according to the script's context, and put it into "len". Example: +From +``` +(SFX: Uplifting newsroom music) +``` +to +``` +{"audio_type": "music", "layout": "foreground", "vol": -35, "len": 10, "desc": "Uplifting newsroom music"}, +``` + +When a sound effect or music is environmental played in the background, you should set their layout to "background". You must give the background audio an unique id, and you must figure out the end of the background audio according to the context and indicate it explicitly. Example: +From +``` +... +(SFX: Airport ambiance, people walking) +Airport Announcer: Lades and Gentlemen, attentions please! +... +``` +to +``` +... +{"audio_type": "sound_effect", "layout": "background", "id":1, "action": "begin", "vol": -35, "desc": "Airport ambiance, people walking"}, +[foreground audio] +... +{"audio_type": "sound_effect", "layout": "background", "id":1, "action": "end"}, +... +``` + +When a line contains multiple sound effects and musics, you need to decompose it into multiple audios. Example: +From +``` +... +(SFX: A classy restaurant, low chatter, clinking silverware, jazz music playing) +... +``` +to +``` +... +{"audio_type": "sound_effect", "layout": "background", "id":1, "action": "begin", "vol": -35, "desc": "low chatter"}, +{"audio_type": "sound_effect", "layout": "background", "id":2, "action": "begin", "vol": -35, "desc": "clinking silverware"}, +{"audio_type": "music", "layout": "background", "id":3, "action": "begin", "vol": -35, "desc": "jazz music"}, +... +{"audio_type": "sound_effect", "layout": "background", "id":1, "action": "end"}, +{"audio_type": "sound_effect", "layout": "background", "id":2, "action": "end"}, +{"audio_type": "music", "layout": "background", "id":3, "action": "end"}, +... +``` + +The final json object contains a list of all the audio objects. + +Script: \ No newline at end of file diff --git a/prompts/script_to_json.prompt b/prompts/script_to_json.prompt new file mode 100644 index 0000000000000000000000000000000000000000..3cd5b9fd1e4261813a672a56932cb86c48d61503 --- /dev/null +++ b/prompts/script_to_json.prompt @@ -0,0 +1,58 @@ +Convert an audio script line to another format. Each line will be converted to a simple json format. Below are the examples of conversion of each line. + +Example line 1: +''' +[Background music 1 begins, -35dB: Uplifting newsroom music] +''' +convert to: +''' +{"voice_type": "back_ground_music", "id": 1, "state": "begin", "volume": -35, "desc": "Uplifting newsroom music"}, +''' +Example line 2: +''' +[Background music 1 ends] +''' +convert to: +''' +{"voice_type": "back_ground_music", "id": 1, "state": "end"}, +''' +Example line 3: +''' +[Background sound effect 2 begins, -35dB: Crowds cheering and arcade ambiance] +''' +convert to: +''' +{"voice_type": "back_ground_sound_effect", "id": 2, "state": "begin", "volume": -35, "desc": "Crowds cheering and arcade ambiance"}, +''' +Example line 4: +''' +[Background sound effect 2 ends] +''' +convert to: +''' +{"voice_type": "back_ground_sound_effect", "id": 2, "state": "end"}, +''' +Example line 5: +''' +News Anchor, -15dB: Good evening, this is BBC News. +''' +convert to: +''' +{"voice_type": "speech", "character": "News Anchor", "volume": -15, "desc": "Good evening, this is BBC News."}, +''' +Example line 6: +''' +[Sound effect, 3s, -15dB: Keyboard typing and mouse clicking] +''' +convert to: +''' +{"voice_type": "sound_effect", "length": 3, "volume": -15, "desc": "Keyboard typing and mouse clicking"}, +''' +Example line 7: +''' +[Sound music, 10s, -15dB: Uplifting newsroom music] +''' +convert to: +''' +{"voice_type": "music", "length": 10, "volume": -15, "desc": "Uplifting newsroom music"}, +''' diff --git a/prompts/text_to_audio_script.prompt b/prompts/text_to_audio_script.prompt new file mode 100644 index 0000000000000000000000000000000000000000..074e0f286802d4a3c2321f7b708db665555bd012 --- /dev/null +++ b/prompts/text_to_audio_script.prompt @@ -0,0 +1,34 @@ +I want you to act as a audio script writer. I'll give you an instruction which is a general idea and you will make it a short audio script. + +The script should follow the rules below: +- For dialogs, each line must contain the character's name, its volume in decibel (human voices are usually around -15dB) and the line, example: +''' +Darth Vader, -16dB: Luke, I'm your father. +''' +- For foreground sound effect, you must wrap the line with brackets and start with "Sound effect, ", and you should give the duration of the sound effect in seconds, and you should specify the volume you want in decibel(For foreground sound effects it's usually around -15dB), and you should give very detailed description of the sound effect, example: +''' +[Sound effect, 2s, -15dB: Airport beeping sound] +''' +- For foreground music, you must wrap the line with brackets and start with "Music, ", and you should give the duration of the music in seconds, and you should specify the volume you want in decibel(for foreground music it's usually around -15dB), and you should give very detailed description of the music, example: +''' +[Music, 10s, -15dB: 80's Rock and Roll music] +''' +- For background sound effects, you must wrap the line with brackets and start with "Background sound effect" followed by its id, and you must always explicitly indicate the start and end of the sound effects, and you should specify the volume you want in decibel(for background sound effect it's usually around -35dB), and you should give very detailed description of the sound effect, example: +''' +[Background sound effect 1 begins, -34dB: Airport ambiance, including footsteps, luggage rolling, and distant airplane engine] +... +[Background sound effect 1 ends] +''' +- For background music, you must wrap the line with brackets and start with "Background music" followed by its id, and you must always explicitly indicate the start and end of the music, and you should specify the volume you want in decibel(for background sound effect it's usually around -35dB), and you should give very detailed description of the music, example: +''' +[Background music 1 begins, -35dB: Uplifting newsroom music] +... +[Background music 1 ends] +''' +- For music and sound effect, you can not name the element outside these: +["Sound effect, ", +"Music, ", +"Background sound effect" followed by its id, +"Background music" followed by its id] +such as "Foreground sound effect", "Foreground music" is forbidden + diff --git a/prompts/text_to_json.prompt b/prompts/text_to_json.prompt new file mode 100644 index 0000000000000000000000000000000000000000..7b4e6ef47650c9504e8f4fcae415dde07cd877d1 --- /dev/null +++ b/prompts/text_to_json.prompt @@ -0,0 +1,33 @@ +I want you to act as an audio script writer. I'll give you input text which is a general idea and you will make it an audio script in json format. Instructions: +- Each line represents an audio. There are three types of audio: sound effects, music, and speech. For each audio, there are only two types of layouts: foreground and background. Foreground audios are played sequentially, and background audios are environmental sounds or music which are played while the foreground audios are being played. +- Sound effects can be either foreground or background. For sound effects, you must provide its layout, volume, length (in seconds), and detailed description of the real-world sound effect. Example: +''' +- The description of sound effects should not contain a specific person. +{"audio_type": "sound_effect", "layout": "foreground", "vol": -35, "len": 2, "desc": "Airport beeping sound"}, +''' +- Music can be either foreground or background. For music, you must provide its layout, volume, length (in seconds), and detailed description of the music. Example: +''' +{"audio_type": "music", "layout": "foreground", "vol": -35, "len": 10, "desc": "Uplifting newsroom music"}, +''' +- Speech can only be foreground. For speech, you must provide the character, volume, and the character's line. You do not need to specify the length of the speech. Example: +''' +{"audio_type": "speech", "layout": "foreground", "character": "News Anchor", "vol": -15, "text": "Good evening, this is BBC News. In today's breaking news, we have an unexpected turn of events in the political arena"}, +''' +- The description of speech should not contain anything other than the lines, such as actions, expressions, emotions etc. +- For background sound audio, you must specify the beginning and the end of background audio in separate lines to indicate when the audio begins and when it ends. Example for background sound effect (for background music it's similar): +''' +{"audio_type": "sound_effect", "layout": "background", "id":1, "action": "begin", "vol": -35, "desc": "Airport ambiance, people walking"}, +[foreground audio 1] +[foreground audio 2] +... +{"audio_type": "sound_effect", "layout": "background", "id":1, "action": "end"}, +''' +- Each background audio must have a unique id. +- You do not specify the length of the background audio. +- A background audio must be wrapped around at least one foreground audio. +- If a background sound effect has multiple sounds, please decompose it into multiple background sound effects. +- The description of speech can be multilingual, default is English. +- The description of sound effects and music must be in English. +- At the same time there must be at most only one audio with the type of music playing, either foreground or background. +- The volume of background sound effects/music is usually around -35 ~ -40 dB +- The output json must be a list as the root node containing all the audio nodes, and must be wrapped with triple quotes '''. diff --git a/scripts/EnvsSetup.sh b/scripts/EnvsSetup.sh new file mode 100644 index 0000000000000000000000000000000000000000..7324c83f4b9461b1ccffea55551be8cae95d98dc --- /dev/null +++ b/scripts/EnvsSetup.sh @@ -0,0 +1,7 @@ +conda env create -f Envs/WavJourney.yml && \ +conda env update -f Envs/Bark.yml && \ +conda env update -f Envs/AudioCraft.yml && \ +conda run --live-stream -n WavJourney pip install -U git+https://git@github.com/facebookresearch/audiocraft@c5157b5bf14bf83449c17ea1eeb66c19fb4bc7f0#egg=audiocraft && \ +conda run --live-stream -n WavJourney pip install -U --no-deps voicefixer==0.1.2 && \ +conda run --live-stream -n WavJourney pip install -U --no-deps numpy==1.21 && \ +conda run --live-stream -n WavJourney pip install -U --no-deps librosa==0.8.1 \ No newline at end of file diff --git a/scripts/download_models.py b/scripts/download_models.py new file mode 100644 index 0000000000000000000000000000000000000000..51dda13bea02b8453cf7958e34cc9571c208c0df --- /dev/null +++ b/scripts/download_models.py @@ -0,0 +1,32 @@ +import yaml +import os + +# Read the YAML file +with open('config.yaml', 'r') as file: + config = yaml.safe_load(file) + +# Extract values for each application +ttm_model_size = config['AudioCraft']['ttm_model_size'] +tta_model_size = config['AudioCraft']['tta_model_size'] + +# Download nltk +import nltk +nltk.download('punkt') + +# Downloading the TTS models +print('Step 1: Downloading TTS model ...') +os.system(f'conda run --live-stream -n WavJourney python -c \'from transformers import BarkModel; BarkModel.from_pretrained("suno/bark")\'') + +print('Step 2: Downloading TTA model ...') +os.system(f'conda run --live-stream -n WavJourney python -c \'from audiocraft.models import AudioGen; tta_model = AudioGen.get_pretrained("facebook/audiogen-{tta_model_size}")\'') + +print('Step 3: Downloading TTM model ...') +os.system(f'conda run --live-stream -n WavJourney python -c \'from audiocraft.models import MusicGen; tta_model = MusicGen.get_pretrained("facebook/musicgen-{ttm_model_size}")\'') + +print('Step 4: Downloading SR model ...') +os.system(f'conda run --live-stream -n WavJourney python -c \'from voicefixer import VoiceFixer; vf = VoiceFixer()\'') + +print('Step 5: Downloading VP model ...') +os.system(f'conda run --live-stream -n WavJourney python -c \'from VoiceParser.model import VoiceParser; vp = VoiceParser(device="cpu")\'') + +print('All models successfully downloaded!') diff --git a/scripts/kill_services.py b/scripts/kill_services.py new file mode 100644 index 0000000000000000000000000000000000000000..83d6e297f6989b94127c015266237bff0a8186d1 --- /dev/null +++ b/scripts/kill_services.py @@ -0,0 +1,11 @@ +import os + +# Extract values for each application +service_port = os.environ.get('WAVJOURNEY_SERVICE_PORT') + +# Execute the commands +os.system(f'kill $(lsof -t -i :{service_port})') + + + + diff --git a/scripts/start_service_and_ui.sh b/scripts/start_service_and_ui.sh new file mode 100644 index 0000000000000000000000000000000000000000..d3f8f40d9dfaca8e0f4ef97d1885515359528b62 --- /dev/null +++ b/scripts/start_service_and_ui.sh @@ -0,0 +1,2 @@ +conda run --live-stream -n WavJourney python -u services.py 2>&1 | tee services_logs/service.out & +conda run --live-stream -n WavJourney python -u ui_client.py 2>&1 | tee services_logs/wavejourney.out \ No newline at end of file diff --git a/scripts/start_services.sh b/scripts/start_services.sh new file mode 100644 index 0000000000000000000000000000000000000000..2347f327760ad9442f14eff8bb28924021dcfaba --- /dev/null +++ b/scripts/start_services.sh @@ -0,0 +1 @@ +nohup conda run --live-stream -n WavJourney python services.py > services_logs/service.out 2>&1 & diff --git a/scripts/start_ui.sh b/scripts/start_ui.sh new file mode 100644 index 0000000000000000000000000000000000000000..c869311af12b7619cf87327f5b1f05951d0b078e --- /dev/null +++ b/scripts/start_ui.sh @@ -0,0 +1 @@ +conda run --live-stream -n WavJourney python -u ui_client.py 2>&1 | stdbuf -oL tee services_logs/wavejourney.out \ No newline at end of file diff --git a/services.py b/services.py new file mode 100644 index 0000000000000000000000000000000000000000..18a96164dd0d15a05a2b26277a88a3682a0f6338 --- /dev/null +++ b/services.py @@ -0,0 +1,231 @@ +import os +import yaml +import logging +import nltk +import torch +import torchaudio +from torchaudio.transforms import SpeedPerturbation +from APIs import WRITE_AUDIO, LOUDNESS_NORM +from utils import fade, get_service_port +from flask import Flask, request, jsonify + +with open('config.yaml', 'r') as file: + config = yaml.safe_load(file) + +# Configure the logging format and level +logging.basicConfig( + level=logging.INFO, + format='%(asctime)s - %(levelname)s - %(message)s' +) + +# Create a FileHandler for the log file +os.makedirs('services_logs', exist_ok=True) +log_filename = 'services_logs/Wav-API.log' +file_handler = logging.FileHandler(log_filename, mode='w') +file_handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')) + +# Add the FileHandler to the root logger +logging.getLogger('').addHandler(file_handler) + + +""" +Initialize the AudioCraft models here +""" +from audiocraft.models import AudioGen, MusicGen +tta_model_size = config['AudioCraft']['tta_model_size'] +tta_model = AudioGen.get_pretrained(f'facebook/audiogen-{tta_model_size}') +logging.info(f'AudioGen ({tta_model_size}) is loaded ...') + +ttm_model_size = config['AudioCraft']['ttm_model_size'] +ttm_model = MusicGen.get_pretrained(f'facebook/musicgen-{ttm_model_size}') +logging.info(f'MusicGen ({ttm_model_size}) is loaded ...') + + +""" +Initialize the BarkModel here +""" +from transformers import BarkModel, AutoProcessor +SPEED = float(config['Text-to-Speech']['speed']) +speed_perturb = SpeedPerturbation(32000, [SPEED]) +tts_model = BarkModel.from_pretrained("suno/bark") +device = "cuda:0" if torch.cuda.is_available() else "cpu" +tts_model = tts_model.to(device) +tts_model = tts_model.to_bettertransformer() # Flash attention +SAMPLE_RATE = tts_model.generation_config.sample_rate +SEMANTIC_TEMPERATURE = 0.9 +COARSE_TEMPERATURE = 0.5 +FINE_TEMPERATURE = 0.5 +processor = AutoProcessor.from_pretrained("suno/bark") +logging.info('Bark model is loaded ...') + + +""" +Initialize the VoiceFixer model here +""" +from voicefixer import VoiceFixer +vf = VoiceFixer() +logging.info('VoiceFixer is loaded ...') + + +""" +Initalize the VoiceParser model here +""" +from VoiceParser.model import VoiceParser +vp_device = config['Voice-Parser']['device'] +vp = VoiceParser(device=vp_device) +logging.info('VoiceParser is loaded ...') + + +app = Flask(__name__) + + +@app.route('/generate_audio', methods=['POST']) +def generate_audio(): + # Receive the text from the POST request + data = request.json + text = data['text'] + length = float(data.get('length', 5.0)) + volume = float(data.get('volume', -35)) + output_wav = data.get('output_wav', 'out.wav') + + logging.info(f'TTA (AudioGen): Prompt: {text}, length: {length} seconds, volume: {volume} dB') + + try: + tta_model.set_generation_params(duration=length) + wav = tta_model.generate([text]) + wav = torchaudio.functional.resample(wav, orig_freq=16000, new_freq=32000) + + wav = wav.squeeze().cpu().detach().numpy() + wav = fade(LOUDNESS_NORM(wav, volumn=volume)) + WRITE_AUDIO(wav, name=output_wav) + + # Return success message and the filename of the generated audio + return jsonify({'message': f'Text-to-Audio generated successfully | {text}', 'file': output_wav}) + + except Exception as e: + return jsonify({'API error': str(e)}), 500 + + +@app.route('/generate_music', methods=['POST']) +def generate_music(): + # Receive the text from the POST request + data = request.json + text = data['text'] + length = float(data.get('length', 5.0)) + volume = float(data.get('volume', -35)) + output_wav = data.get('output_wav', 'out.wav') + + logging.info(f'TTM (MusicGen): Prompt: {text}, length: {length} seconds, volume: {volume} dB') + + + try: + ttm_model.set_generation_params(duration=length) + wav = ttm_model.generate([text]) + wav = wav[0][0].cpu().detach().numpy() + wav = fade(LOUDNESS_NORM(wav, volumn=volume)) + WRITE_AUDIO(wav, name=output_wav) + + # Return success message and the filename of the generated audio + return jsonify({'message': f'Text-to-Music generated successfully | {text}', 'file': output_wav}) + + except Exception as e: + # Return error message if something goes wrong + return jsonify({'API error': str(e)}), 500 + + +@app.route('/generate_speech', methods=['POST']) +def generate_speech(): + # Receive the text from the POST request + data = request.json + text = data['text'] + speaker_id = data['speaker_id'] + speaker_npz = data['speaker_npz'] + volume = float(data.get('volume', -35)) + output_wav = data.get('output_wav', 'out.wav') + + logging.info(f'TTS (Bark): Speaker: {speaker_id}, Volume: {volume} dB, Prompt: {text}') + + try: + # Generate audio using the global pipe object + text = text.replace('\n', ' ').strip() + sentences = nltk.sent_tokenize(text) + silence = torch.zeros(int(0.1 * SAMPLE_RATE), device=device).unsqueeze(0) # 0.1 second of silence + + pieces = [] + for sentence in sentences: + inputs = processor(sentence, voice_preset=speaker_npz).to(device) + # NOTE: you must run the line below, otherwise you will see the runtime error + # RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead. + inputs['history_prompt']['coarse_prompt'] = inputs['history_prompt']['coarse_prompt'].transpose(0, 1).contiguous().transpose(0, 1) + + with torch.inference_mode(): + # TODO: min_eos_p? + output = tts_model.generate( + **inputs, + do_sample = True, + semantic_temperature = SEMANTIC_TEMPERATURE, + coarse_temperature = COARSE_TEMPERATURE, + fine_temperature = FINE_TEMPERATURE + ) + + pieces += [output, silence] + + result_audio = torch.cat(pieces, dim=1) + wav_tensor = result_audio.to(dtype=torch.float32).cpu() + wav = torchaudio.functional.resample(wav_tensor, orig_freq=SAMPLE_RATE, new_freq=32000) + wav = speed_perturb(wav.float())[0].squeeze(0) + wav = wav.numpy() + wav = LOUDNESS_NORM(wav, volumn=volume) + WRITE_AUDIO(wav, name=output_wav) + + # Return success message and the filename of the generated audio + return jsonify({'message': f'Text-to-Speech generated successfully | {speaker_id}: {text}', 'file': output_wav}) + + except Exception as e: + # Return error message if something goes wrong + return jsonify({'API error': str(e)}), 500 + + +@app.route('/fix_audio', methods=['POST']) +def fix_audio(): + # Receive the text from the POST request + data = request.json + processfile = data['processfile'] + + logging.info(f'Fixing {processfile} ...') + + try: + vf.restore(input=processfile, output=processfile, cuda=True, mode=0) + + # Return success message and the filename of the generated audio + return jsonify({'message': 'Speech restored successfully', 'file': processfile}) + + except Exception as e: + # Return error message if something goes wrong + return jsonify({'API error': str(e)}), 500 + + +@app.route('/parse_voice', methods=['POST']) +def parse_voice(): + # Receive the text from the POST request + data = request.json + wav_path = data['wav_path'] + out_dir = data['out_dir'] + + logging.info(f'Parsing {wav_path} ...') + + try: + vp.extract_acoustic_embed(wav_path, out_dir) + + # Return success message and the filename of the generated audio + return jsonify({'message': f'Sucessfully parsed {wav_path}'}) + + except Exception as e: + # Return error message if something goes wrong + return jsonify({'API error': str(e)}), 500 + + +if __name__ == '__main__': + service_port = get_service_port() + # We disable multithreading to force services to process one request at a time and avoid CUDA OOM + app.run(debug=False, threaded=False, port=service_port) diff --git a/share_btn.py b/share_btn.py new file mode 100644 index 0000000000000000000000000000000000000000..a2a554cd8e85665c61ef6290289b3d363abf2e7c --- /dev/null +++ b/share_btn.py @@ -0,0 +1,74 @@ +community_icon_html = """""" + +loading_icon_html = """""" + +share_js = """async () => { + async function uploadFile(file){ + const UPLOAD_URL = 'https://huggingface.co/uploads'; + const response = await fetch(UPLOAD_URL, { + method: 'POST', + headers: { + 'Content-Type': file.type, + 'X-Requested-With': 'XMLHttpRequest', + }, + body: file, /// <- File inherits from Blob + }); + const url = await response.text(); + return url; + } + async function getInputVideoFile(videoEl){ + const res = await fetch(videoEl.src); + const blob = await res.blob(); + const videoId = Date.now() % 200; + const fileName = `sd-perception-${videoId}.mp4`; + return new File([blob], fileName, { type: 'video/mp4' }); + } + + async function audioToBase64(audioFile) { + return new Promise((resolve, reject) => { + let reader = new FileReader(); + reader.readAsDataURL(audioFile); + reader.onload = () => resolve(reader.result); + reader.onerror = error => reject(error); + + }); + } + const gradioEl = document.querySelector("gradio-app").shadowRoot || document.querySelector('body > gradio-app'); + const inputPromptEl = gradioEl.querySelector('#prompt-in textarea').value; + const outputVideoEl = gradioEl.querySelector('#output-video video'); + + let titleTxt = `WavJourney: ${inputPromptEl}`; + + const shareBtnEl = gradioEl.querySelector('#share-btn'); + const shareIconEl = gradioEl.querySelector('#share-btn-share-icon'); + const loadingIconEl = gradioEl.querySelector('#share-btn-loading-icon'); + if(!outputVideoEl){ + return; + }; + shareBtnEl.style.pointerEvents = 'none'; + shareIconEl.style.display = 'none'; + loadingIconEl.style.removeProperty('display'); + const outputVideo = await getInputVideoFile(outputVideoEl); + const urlOutputVideo = await uploadFile(outputVideo); + + const descriptionMd = ` +##### ${inputPromptEl} + +${urlOutputVideo} +`; + const params = new URLSearchParams({ + title: titleTxt, + description: descriptionMd, + }); + const paramsStr = params.toString(); + window.open(`https://huggingface.co/spaces/Audio-AGI/WavJourney/discussions/new?${paramsStr}`, '_blank'); + shareBtnEl.style.removeProperty('pointer-events'); + shareIconEl.style.removeProperty('display'); + loadingIconEl.style.display = 'none'; +}""" \ No newline at end of file diff --git a/ui_client.py b/ui_client.py new file mode 100644 index 0000000000000000000000000000000000000000..c4e13a2a6d09a70068cbcd2006bea82c0f52ec90 --- /dev/null +++ b/ui_client.py @@ -0,0 +1,632 @@ +import shutil +import json5 +import traceback + +import gradio as gr +from tabulate import tabulate + +import utils +import pipeline +from pipeline import generate_json_file, generate_audio +from voice_presets import load_voice_presets_metadata, add_session_voice_preset, \ + remove_session_voice_preset +from share_btn import community_icon_html, loading_icon_html, share_js + + + +VOICE_PRESETS_HEADERS = ['ID', 'Description'] +DELETE_FILE_WHEN_DO_CLEAR = False +DEBUG = False + + +def convert_json_to_md(audio_script_response): + audio_json_data = json5.loads(audio_script_response) + table = [[node.get(field, 'N/A') for field in ["audio_type", "layout", "id", "character", "action", 'vol']] + + [node.get("desc", "N/A") if node.get("audio_type") != "speech" else node.get("text", "N/A")] + + [node.get("len", "Auto") if "len" in node else "Auto"] + for i, node in enumerate(audio_json_data)] + + headers = ["Audio Type", "Layout", "ID", "Character", "Action", 'Volume', "Description", "Length" ] + + # Tabulate + table_txt = tabulate(table, headers, tablefmt="github") + return table_txt + + +def convert_char_voice_map_to_md(char_voice_map): + table =[[character, char_voice_map[character]["id"]] for character in char_voice_map] + headers = ["Character", "Voice"] + # Tabulate + table_txt = tabulate(table, headers, tablefmt="github") + return table_txt + + +def get_or_create_session_from_state(ui_state): + if 'session_id' not in ui_state: + ui_state['session_id'] = pipeline.init_session() + return ui_state['session_id'] + + +def generate_script_fn(instruction, _state: gr.State): + try: + session_id = get_or_create_session_from_state(_state) + api_key = utils.get_api_key() + json_script = generate_json_file(session_id, instruction, api_key) + table_text = convert_json_to_md(json_script) + except Exception as e: + gr.Warning(str(e)) + print(f"Generating script error: {str(e)}") + traceback.print_exc() + return [ + None, + _state, + gr.Button.update(interactive=False), + gr.Button.update(interactive=True), + gr.Button.update(interactive=True), + gr.Button.update(interactive=True), + ] + + _state = { + **_state, + 'session_id': session_id, + 'json_script': json_script + } + return [ + table_text, + _state, + gr.Button.update(interactive=True), + gr.Button.update(interactive=True), + gr.Button.update(interactive=True), + gr.Button.update(interactive=True), + ] + + +def generate_audio_fn(state): + btn_state = gr.Button.update(interactive=True) + try: + api_key = utils.get_api_key() + audio_path, char_voice_map = generate_audio(**state, api_key=api_key) + table_text = convert_char_voice_map_to_md(char_voice_map) + # TODO: output char_voice_map to a table + return [ + table_text, + gr.make_waveform(str(audio_path)), + btn_state, + btn_state, + btn_state, + btn_state, + ] + except Exception as e: + print(f"Generation audio error: {str(e)}") + traceback.print_exc() + gr.Warning(str(e)) + + return [ + None, + None, + btn_state, + btn_state, + btn_state, + btn_state, + ] + + +def clear_fn(state): + if DELETE_FILE_WHEN_DO_CLEAR: + shutil.rmtree('output', ignore_errors=True) + state = {'session_id': pipeline.init_session()} + return [gr.Markdown.update(value=''), + gr.Textbox.update(value=''), + gr.Video.update(value=None), + gr.Markdown.update(value=''), + gr.Button.update(interactive=False), + gr.Button.update(interactive=False), + state, gr.Dataframe.update(visible=False), + gr.Button.update(visible=False), + gr.Textbox.update(value=''), + gr.Textbox.update(value=''), + gr.File.update(value=None)] + + +def textbox_listener(textbox_input): + if len(textbox_input) > 0: + return gr.Button.update(interactive=True) + else: + return gr.Button.update(interactive=False) + + +def get_voice_preset_to_list(state: gr.State): + if state.__class__ == gr.State: + state = state.value + if 'session_id' in state: + path = utils.get_session_voice_preset_path(state['session_id']) + else: + path = '' + voice_presets = load_voice_presets_metadata( + path, + safe_if_metadata_not_exist=True + ) + dataframe = [] + for key in voice_presets.keys(): + row = [key, voice_presets[key]['desc']] + dataframe.append(row) + return dataframe + + +def df_on_select(evt: gr.SelectData): + print(f"You selected {evt.value} at {evt.index} from {evt.target}") + return {'selected_voice_preset': evt.index} + + +def del_voice_preset(selected_voice_presets, ui_state, dataframe): + gr_visible = gr.Dataframe.update(visible=True) + btn_visible = gr.Button.update(visible=True) + current_presets = get_voice_preset_to_list(ui_state) + if selected_voice_presets['selected_voice_preset'] is None or \ + selected_voice_presets['selected_voice_preset'][0] > len(current_presets) - 1: + gr.Warning('None row is selected') + return [current_presets, gr_visible, btn_visible, selected_voice_presets] + # Do the real file deletion + index = selected_voice_presets['selected_voice_preset'][0] + vp_id = dataframe['ID'][index] + remove_session_voice_preset(vp_id, ui_state['session_id']) + current_presets = get_voice_preset_to_list(ui_state) + gr.Dataframe.update(value=current_presets) + if len(current_presets) == 0: + gr_visible = gr.Dataframe.update(visible=False) + btn_visible = gr.Button.update(visible=False) + selected_voice_presets['selected_voice_preset'] = None + return [current_presets, gr_visible, btn_visible, selected_voice_presets] + + +def get_system_voice_presets(): + system_presets = load_voice_presets_metadata(utils.get_system_voice_preset_path()) + data = [] + for k, v in system_presets.items(): + data.append([k, v['desc']]) + # headers = ['id', 'description'] + # table_txt = tabulate(data, headers, tablefmt="github") + return data + + +def set_openai_key(key, _state): + _state['api_key'] = key + return key + + +def add_voice_preset(vp_id, vp_desc, file, ui_state, added_voice_preset): + if vp_id is None or vp_desc is None or file is None or vp_id.strip() == '' or vp_desc.strip() == '': + gr.Warning('please complete all three fields') + else: + count: int = added_voice_preset['count'] + # check if greater than 3 + session_id = get_or_create_session_from_state(ui_state) + file_path = file.name + print(f'session {session_id}, id {id}, desc {vp_desc}, file {file_path}') + # Do adding ... + try: + add_session_voice_preset(vp_id, vp_desc, file_path, session_id) + added_voice_preset['count'] = count + 1 + except Exception as exception: + print(exception) + traceback.print_exc() + gr.Warning(str(exception)) + + # After added + dataframe = get_voice_preset_to_list(ui_state) + df_visible = gr.Dataframe.update(visible=True) + del_visible = gr.Button.update(visible=True) + if len(dataframe) == 0: + df_visible = gr.Dataframe.update(visible=False) + del_visible = gr.Button.update(visible=False) + return [gr.Textbox.update(value=''), gr.Textbox.update(value=''), gr.File.update(value=None), + ui_state, added_voice_preset, dataframe, gr.Button.update(interactive=True), + df_visible, del_visible] + + +css = """ + a { + color: inherit; + text-decoration: underline; + } + .gradio-container { + font-family: 'IBM Plex Sans', sans-serif; + } + .gr-button { + color: white; + border-color: #000000; + background: #000000; + } + input[type='range'] { + accent-color: #000000; + } + .dark input[type='range'] { + accent-color: #dfdfdf; + } + .container { + max-width: 730px; + margin: auto; + padding-top: 1.5rem; + } + #gallery { + min-height: 22rem; + margin-bottom: 15px; + margin-left: auto; + margin-right: auto; + border-bottom-right-radius: .5rem !important; + border-bottom-left-radius: .5rem !important; + } + #gallery>div>.h-full { + min-height: 20rem; + } + .details:hover { + text-decoration: underline; + } + .gr-button { + white-space: nowrap; + } + .gr-button:focus { + border-color: rgb(147 197 253 / var(--tw-border-opacity)); + outline: none; + box-shadow: var(--tw-ring-offset-shadow), var(--tw-ring-shadow), var(--tw-shadow, 0 0 #0000); + --tw-border-opacity: 1; + --tw-ring-offset-shadow: var(--tw-ring-inset) 0 0 0 var(--tw-ring-offset-width) var(--tw-ring-offset-color); + --tw-ring-shadow: var(--tw-ring-inset) 0 0 0 calc(3px var(--tw-ring-offset-width)) var(--tw-ring-color); + --tw-ring-color: rgb(191 219 254 / var(--tw-ring-opacity)); + --tw-ring-opacity: .5; + } + #advanced-btn { + font-size: .7rem !important; + line-height: 19px; + margin-top: 12px; + margin-bottom: 12px; + padding: 2px 8px; + border-radius: 14px !important; + } + #advanced-options { + margin-bottom: 20px; + } + .footer { + margin-bottom: 45px; + margin-top: 35px; + text-align: center; + border-bottom: 1px solid #e5e5e5; + } + .footer>p { + font-size: .8rem; + display: inline-block; + padding: 0 10px; + transform: translateY(10px); + background: white; + } + .dark .footer { + border-color: #303030; + } + .dark .footer>p { + background: #0b0f19; + } + .acknowledgments h4{ + margin: 1.25em 0 .25em 0; + font-weight: bold; + font-size: 115%; + } + #container-advanced-btns{ + display: flex; + flex-wrap: wrap; + justify-content: space-between; + align-items: center; + } + .animate-spin { + animation: spin 1s linear infinite; + } + @keyframes spin { + from { + transform: rotate(0deg); + } + to { + transform: rotate(360deg); + } + } + #share-btn-container { + display: flex; padding-left: 0.5rem !important; padding-right: 0.5rem !important; background-color: #000000; justify-content: center; align-items: center; border-radius: 9999px !important; width: 13rem; + margin-top: 10px; + margin-left: auto; + } + #share-btn { + all: initial; color: #ffffff;font-weight: 600; cursor:pointer; font-family: 'IBM Plex Sans', sans-serif; margin-left: 0.5rem !important; padding-top: 0.25rem !important; padding-bottom: 0.25rem !important;right:0; + } + #share-btn * { + all: unset; + } + #share-btn-container div:nth-child(-n+2){ + width: auto !important; + min-height: 0px !important; + } + #share-btn-container .wrap { + display: none !important; + } + .gr-form{ + flex: 1 1 50%; border-top-right-radius: 0; border-bottom-right-radius: 0; + } + #prompt-container{ + gap: 0; + } + #generated_id{ + min-height: 700px + } + #setting_id{ + margin-bottom: 12px; + text-align: center; + font-weight: 900; + } +""" + +with gr.Blocks(css=css) as interface: + + gr.HTML( + """ +
+
+ """
+ )
+ gr.HTML(
+ """
+
+
+
+ + WavJourney: Compositional Audio Creation with LLMs +
+Due to the high user demand we are facing from our community, we will be offering free access to WavJourney for a few more days. You can also access WavJourney in this space later by providing your OPENAI_KEY.
+For faster inference without waiting in the queue, you can duplicate the space and upgrade to GPU (VRAM>16G) and provide OPENAI_KEY to access GPT-4 in settings.
+
+
+
+
Begin with a text prompt, and let WavJourney transform it into captivating audio content. Experience engaging audio storylines, personalized voices, lifelike speech, emotionally resonant musical compositions, and immersive sound effects! + #
+ # """ + # ) + + gr.Markdown(value='## WavJourney Pipeline:') + + gr.Markdown(value='Begin with a text prompt, and let WavJourney transform it into captivating audio content. Experience engaging audio storylines, personalized voices, lifelike speech, emotionally resonant musical compositions, and immersive sound effects!') + + gr.HTML( + """ +-
+
- Stage 0: (optional) add your customized voice preset for a more personalized audio creation experience. User also often shares presets in Discord. +
- Stage 1: generate the audio script based on the input text instruction (the default language is English, but you can actually type in your own language). +
- Stage 2: Select the suitable voice in the multilingual voice preset for each character in the audio script & generate audio. +
-
+
- You can share with the HuggingFace community by clicking the "Share to community" button. +
- You can share your generations to our Discord channel! +
- You can also share the voice presets (along with descriptions) you found in Discord. +
-
+
- You can use vague or specific descriptions or a combination of them. For example: "male speech about pizza" or "a man is saying: I love pizza!" +
- You can control the length of the audio script by simply adding the restriction. For example: "generate an audio script around 10-15 lines (max length has been set to 30)" +
- You can specify the language of the speaker. For example: "a boy is playing with a girl, boy's speech is in Chinese while girl's speech in Japanese" +
- Explore more prompting techniques by yourself! 🤗 +