README.md

---
license: mit
language:
- en
pipeline_tag: text-to-audio
tags:
- audiocraft
- audiogen
- styletts2
- shift
- audeering
- sound
- audio-generation
- text-to-speech
- mimic3
---


# Affective TTS / Soundscape

This repo is an expansion of the [SHIFT TTS tool](https://github.com/audeering/shift) with [foreign langs](https://huggingface.co/dkounadis/artificial-styletts2/blob/main/Utils/all_langs.csv) and audio sound generation via [AudioGen](https://huggingface.co/dkounadis/artificial-styletts2/discussions/3).
  - Analysis of emotions of TTS [#1](https://huggingface.co/dkounadis/artificial-styletts2/discussions/2)
  - `landscape2soundscape.py` generates soundscape, i.e. `trees, water, ..` & overlays TTS & creates video from image.

## Available Voices

<a href="https://audeering.github.io/shift/">Listen to available voices!</a>

## Flask API

<details>
<summary>
Create virtualenv
</summary>

Clone

```
git clone https://huggingface.co/dkounadis/artificial-styletts2
```
Install

```
virtualenv --python=python3 ~/.envs/.my_env
source ~/.envs/.my_env/bin/activate
cd artificial-styletts2/
pip install -r requirements.txt
```

</details>

Flask API

```
CUDA_DEVICE_ORDER=PCI_BUS_ID HF_HOME=./hf_home CUDA_VISIBLE_DEVICES=2 python api.py
```

## Landscape 2 Soundscape

The following needs `api.py` to be already running on a tmux session. 

```python
# TTS & soundscape - overlay to .mp4
python landscape2soundscape.py
```

For SHIFT demo / Collaboration with [SMB](https://www.smb.museum/home/)
  - YouTube Videos


[![01](uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____01_Schick_AII840_001.jpg)](https://youtu.be/SSi3gUO4GtY)

[![02](uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____02_Constable_AI555_001.jpg)](https://youtu.be/2YjxAPkdXIc)

[![03](uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____03_Schinkel_WS200-002.jpg)](https://youtu.be/BhMh02knkco)



[![05](uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____05_Blechen_FV40_001.jpg)](https://youtu.be/a3qk9S87v60)

[![06](uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____06_Menzel_AI900_001.jpg)](https://youtu.be/3M0y9OYzDfU)

[![07](uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____07_Courbet_AI967_001.jpg)](https://youtu.be/OBY666_By1k)

[![08](uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____08_Monet_AI1013_001.jpg)](https://youtu.be/gnGCYLcdLsA)

[![10](uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____10_Boecklin_967648_NG2-80_001_rsz.jpg)](https://www.youtube.com/watch?v=Y8QyYUgLaCg)

[![11](uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____11_Liebermann_NG4-94_001.jpg)](https://youtu.be/XDDzxDSrhb0)

[![12](uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____12_Slevogt_AII1022_001.jpg)](https://youtu.be/I3YYKiUzHpA)




# SoundScape Live (iterative) Demo - Paplay

Special Flask API for playing sounds live

```python
CUDA_DEVICE_ORDER=PCI_BUS_ID HF_HOME=/data/dkounadis/.hf7/ CUDA_VISIBLE_DEVICES=4 python live_api.py
```

Client - Describe any sound with words and it will be played back to you.

```python
python live_demo.py  # will ask text input & play soundscape
```

# SoundScape (basic) Demo

```python
CUDA_DEVICE_ORDER=PCI_BUS_ID HF_HOME=/data/dkounadis/.hf7/ CUDA_VISIBLE_DEVICES=4 python demo.py
```

##

# Audiobook

Convert `.docx` to audio `.wav` & `.mp4`. Via multiple voices. Listen to it in YouTube [male voice](https://www.youtube.com/watch?v=5-cpf7u18JE) / [female voice](https://www.youtube.com/watch?v=pzrLYCaWD2A)

```python
# uses Flask api.py
# download shift/assets/INCLUSION_IN_MUSEUMS_audiobook.docx
python audiobook.py 
```