metadata

license: mit
language:
  - en
pipeline_tag: audio-generation
tags:
  - audiocraft
  - audiogen
  - styletts2
  - shift
  - audeering
  - sound
  - audio-generation
  - text-to-speech
  - mimic3

Affective TTS / SoundScapes

SHIFT TTS tool
Analysis of emotionality #1
Soundscapes trees, water, castles via AudioGen
landscape2soundscape.py shows how to overlay TTS & sound to image and create video

Available Voices

Listen to available voices!

Flask API

git clone https://huggingface.co/dkounadis/artificial-styletts2

Create virtualenv

virtualenv --python=python3 ~/.envs/.my_env
source ~/.envs/.my_env/bin/activate
cd artificial-styletts2/
pip install -r requirements.txt

Start Flask

CUDA_DEVICE_ORDER=PCI_BUS_ID HF_HOME=./hf_home CUDA_VISIBLE_DEVICES=2 python api.py

Landscape 2 Soundscape

The following needs api.py to be already running on a tmux session.

# TTS & soundscape - overlay to .mp4
python landscape2soundscape.py

Videos / Examples

Video where Native voice is replaced with English TTS voice

Joint Application of D3.1 & D3.2

From an image and text create a video:


python tts.py --text sample.txt --image assets/image_from_T31.jpg

Landscape 2 Soundscape

# Loads image & text & sound-scene text and creates .mp4
python landscape2soundscape.py

For SHIFT demo / Collaboration with SMB

YouTube Videos

Live Demo - Paplay

Flask

CUDA_DEVICE_ORDER=PCI_BUS_ID HF_HOME=/data/dkounadis/.hf7/ CUDA_VISIBLE_DEVICES=4 python live_api.py

Client - Describe any sound with words and it will be played back to you.

python live_demo.py  # will ask text input & play soundscape

Simple Demo

CUDA_DEVICE_ORDER=PCI_BUS_ID HF_HOME=/data/dkounadis/.hf7/ CUDA_VISIBLE_DEVICES=4 python demo.py