{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "62a0cccb",
   "metadata": {},
   "source": [
    "# Pyaudio Microphone Streaming Examples\n",
    "\n",
    "A simple notebook that uses pyaudio to get the microphone audio and feeds this audio then to Silero VAD.\n",
    "\n",
    "I created it as an example on how binary data from a stream could be feed into Silero VAD.\n",
    "\n",
    "\n",
    "Has been tested on Ubuntu 21.04 (x86). After you installed the dependencies below, no additional setup is required."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64cbe1eb",
   "metadata": {},
   "source": [
    "## Dependencies\n",
    "The cell below lists all used dependencies and the used versions. Uncomment to install them from within the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57bc2aac",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install numpy==1.20.2\n",
    "#!pip install torch==1.9.0\n",
    "#!pip install matplotlib==3.4.2\n",
    "#!pip install torchaudio==0.9.0\n",
    "#!pip install soundfile==0.10.3.post1\n",
    "#!pip install pyaudio==0.2.11"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "110de761",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a647d8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import numpy as np\n",
    "import torch\n",
    "torch.set_num_threads(1)\n",
    "import torchaudio\n",
    "import matplotlib\n",
    "import matplotlib.pylab as plt\n",
    "torchaudio.set_audio_backend(\"soundfile\")\n",
    "import pyaudio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "725d7066",
   "metadata": {},
   "outputs": [],
   "source": [
    "model, utils = torch.hub.load(repo_or_dir='snakers4/silero-vad',\n",
    "                              model='silero_vad',\n",
    "                              force_reload=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c0b2ea7",
   "metadata": {},
   "outputs": [],
   "source": [
    "(get_speech_timestamps,\n",
    " save_audio,\n",
    " read_audio,\n",
    " VADIterator,\n",
    " collect_chunks) = utils"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9112603",
   "metadata": {},
   "source": [
    "### Helper Methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5abc6330",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Taken from utils_vad.py\n",
    "def validate(model,\n",
    "             inputs: torch.Tensor):\n",
    "    with torch.no_grad():\n",
    "        outs = model(inputs)\n",
    "    return outs\n",
    "\n",
    "# Provided by Alexander Veysov\n",
    "def int2float(sound):\n",
    "    abs_max = np.abs(sound).max()\n",
    "    sound = sound.astype('float32')\n",
    "    if abs_max > 0:\n",
    "        sound *= 1/32768\n",
    "    sound = sound.squeeze()  # depends on the use case\n",
    "    return sound"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5124095e",
   "metadata": {},
   "source": [
    "## Pyaudio Set-up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a845356e",
   "metadata": {},
   "outputs": [],
   "source": [
    "FORMAT = pyaudio.paInt16\n",
    "CHANNELS = 1\n",
    "SAMPLE_RATE = 16000\n",
    "CHUNK = int(SAMPLE_RATE / 10)\n",
    "\n",
    "audio = pyaudio.PyAudio()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b910c99",
   "metadata": {},
   "source": [
    "## Simple Example\n",
    "The following example reads the audio as 250ms chunks from the microphone, converts them to a Pytorch Tensor, and gets the probabilities/confidences if the model thinks the frame is voiced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d3d2c10",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_samples = 1536"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cb44a4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "stream = audio.open(format=FORMAT,\n",
    "                    channels=CHANNELS,\n",
    "                    rate=SAMPLE_RATE,\n",
    "                    input=True,\n",
    "                    frames_per_buffer=CHUNK)\n",
    "data = []\n",
    "voiced_confidences = []\n",
    "\n",
    "print(\"Started Recording\")\n",
    "for i in range(0, frames_to_record):\n",
    "    \n",
    "    audio_chunk = stream.read(num_samples)\n",
    "    \n",
    "    # in case you want to save the audio later\n",
    "    data.append(audio_chunk)\n",
    "    \n",
    "    audio_int16 = np.frombuffer(audio_chunk, np.int16);\n",
    "\n",
    "    audio_float32 = int2float(audio_int16)\n",
    "    \n",
    "    # get the confidences and add them to the list to plot them later\n",
    "    new_confidence = model(torch.from_numpy(audio_float32), 16000).item()\n",
    "    voiced_confidences.append(new_confidence)\n",
    "    \n",
    "print(\"Stopped the recording\")\n",
    "\n",
    "# plot the confidences for the speech\n",
    "plt.figure(figsize=(20,6))\n",
    "plt.plot(voiced_confidences)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3dda982",
   "metadata": {},
   "source": [
    "## Real Time Visualization\n",
    "\n",
    "As an enhancement to plot the speech probabilities in real time I added the implementation below.\n",
    "In contrast to the simeple one, it records the audio until to stop the recording by pressing enter.\n",
    "While looking into good ways to update matplotlib plots in real-time, I found a simple libarary that does the job. https://github.com/lvwerra/jupyterplot It has some limitations, but works for this use case really well.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05ef4100",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install jupyterplot==0.0.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1d4cdd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from jupyterplot import ProgressPlot\n",
    "import threading\n",
    "\n",
    "continue_recording = True\n",
    "\n",
    "def stop():\n",
    "    input(\"Press Enter to stop the recording:\")\n",
    "    global continue_recording\n",
    "    continue_recording = False\n",
    "\n",
    "def start_recording():\n",
    "    \n",
    "    stream = audio.open(format=FORMAT,\n",
    "                    channels=CHANNELS,\n",
    "                    rate=SAMPLE_RATE,\n",
    "                    input=True,\n",
    "                    frames_per_buffer=CHUNK)\n",
    "\n",
    "    data = []\n",
    "    voiced_confidences = []\n",
    "    \n",
    "    global continue_recording\n",
    "    continue_recording = True\n",
    "    \n",
    "    pp = ProgressPlot(plot_names=[\"Silero VAD\"],line_names=[\"speech probabilities\"], x_label=\"audio chunks\")\n",
    "    \n",
    "    stop_listener = threading.Thread(target=stop)\n",
    "    stop_listener.start()\n",
    "\n",
    "    while continue_recording:\n",
    "    \n",
    "        audio_chunk = stream.read(num_samples)\n",
    "    \n",
    "        # in case you want to save the audio later\n",
    "        data.append(audio_chunk)\n",
    "    \n",
    "        audio_int16 = np.frombuffer(audio_chunk, np.int16);\n",
    "\n",
    "        audio_float32 = int2float(audio_int16)\n",
    "    \n",
    "        # get the confidences and add them to the list to plot them later\n",
    "        new_confidence = model(torch.from_numpy(audio_float32), 16000).item()\n",
    "        voiced_confidences.append(new_confidence)\n",
    "    \n",
    "        pp.update(new_confidence)\n",
    "\n",
    "\n",
    "    pp.finalize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e398009",
   "metadata": {},
   "outputs": [],
   "source": [
    "start_recording()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}