![](\"images/sound.png\")
\n",
+ "\n",
+ "In this assignment you will learn to: \n",
+ "- Structure a speech recognition project\n",
+ "- Synthesize and process audio recordings to create train/dev datasets\n",
+ "- Train a trigger word detection model and make predictions\n",
+ "\n",
+ "Let's get started!\n",
+ "\n",
+ "## Important Note on Submission to the AutoGrader\n",
+ "\n",
+ "Before submitting your assignment to the AutoGrader, please make sure you are not doing the following:\n",
+ "\n",
+ "1. You have not added any _extra_ `print` statement(s) in the assignment.\n",
+ "2. You have not added any _extra_ code cell(s) in the assignment.\n",
+ "3. You have not changed any of the function parameters.\n",
+ "4. You are not using any global variables inside your graded exercises. Unless specifically instructed to do so, please refrain from it and use the local variables instead.\n",
+ "5. You are not changing the assignment code where it is not required, like creating _extra_ variables.\n",
+ "\n",
+ "If you do any of the following, you will get something like, `Grader Error: Grader feedback not found` (or similarly unexpected) error upon submitting your assignment. Before asking for help/debugging the errors in your assignment, check for these first. If this is the case, and you don't remember the changes you have made, you can get a fresh copy of the assignment by following these [instructions](https://www.coursera.org/learn/nlp-sequence-models/supplement/qHIve/h-ow-to-refresh-your-workspace)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Table of Contents\n",
+ "\n",
+ "- [Packages](#0)\n",
+ "- [1 - Data synthesis: Creating a Speech Dataset](#1)\n",
+ " - [1.1 - Listening to the Data](#1-1)\n",
+ " - [1.2 - From Audio Recordings to Spectrograms](#1-2)\n",
+ " - [1.3 - Generating a Single Training Example](#1-3)\n",
+ " - [Exercise 1 - is_overlapping](#ex-1)\n",
+ " - [Exercise 2 - insert_audio_clip](#ex-2)\n",
+ " - [Exercise 3 - insert_ones](#ex-3)\n",
+ " - [Exercise 4 - create_training_example](#ex-4)\n",
+ " - [1.4 - Full Training Set](#1-4)\n",
+ " - [1.5 - Development Set](#1-5)\n",
+ "- [2 - The Model](#2)\n",
+ " - [2.1 - Build the Model](#2-1)\n",
+ " - [Exercise 5 - modelf](#ex-5)\n",
+ " - [2.2 - Fit the Model](#2-2)\n",
+ " - [2.3 - Test the Model](#2-3)\n",
+ "- [3 - Making Predictions](#3)\n",
+ " - [3.1 - Test on Dev Examples](#3-1)\n",
+ "- [4 - Try Your Own Example! (OPTIONAL/UNGRADED)](#4)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "## Packages"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "deletable": false,
+ "editable": false
+ },
+ "outputs": [],
+ "source": [
+ "### v2.2"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "from pydub import AudioSegment\n",
+ "import random\n",
+ "import sys\n",
+ "import io\n",
+ "import os\n",
+ "import glob\n",
+ "import IPython\n",
+ "from td_utils import *\n",
+ "%matplotlib inline"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "## 1 - Data synthesis: Creating a Speech Dataset \n",
+ "\n",
+ "Let's start by building a dataset for your trigger word detection algorithm. \n",
+ "* A speech dataset should ideally be as close as possible to the application you will want to run it on. \n",
+ "* In this case, you'd like to detect the word \"activate\" in working environments (library, home, offices, open-spaces ...). \n",
+ "* Therefore, you need to create recordings with a mix of positive words (\"activate\") and negative words (random words other than activate) on different background sounds. Let's see how you can create such a dataset. \n",
+ "\n",
+ "\n",
+ "### 1.1 - Listening to the Data \n",
+ "\n",
+ "* One of your friends is helping you out on this project, and they've gone to libraries, cafes, restaurants, homes and offices all around the region to record background noises, as well as snippets of audio of people saying positive/negative words. This dataset includes people speaking in a variety of accents. \n",
+ "* In the raw_data directory, you can find a subset of the raw audio files of the positive words, negative words, and background noise. You will use these audio files to synthesize a dataset to train the model. \n",
+ " * The \"activate\" directory contains positive examples of people saying the word \"activate\". \n",
+ " * The \"negatives\" directory contains negative examples of people saying random words other than \"activate\". \n",
+ " * There is one word per audio recording. \n",
+ " * The \"backgrounds\" directory contains 10 second clips of background noise in different environments.\n",
+ "\n",
+ "Run the cells below to listen to some examples."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "IPython.display.Audio(\"./raw_data/activates/1.wav\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "IPython.display.Audio(\"./raw_data/negatives/4.wav\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": false
+ },
+ "outputs": [],
+ "source": [
+ "IPython.display.Audio(\"./raw_data/backgrounds/1.wav\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You will use these three types of recordings (positives/negatives/backgrounds) to create a labeled dataset."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "### 1.2 - From Audio Recordings to Spectrograms\n",
+ "\n",
+ "What really is an audio recording? \n",
+ "* A microphone records little variations in air pressure over time, and it is these little variations in air pressure that your ear also perceives as sound. \n",
+ "* You can think of an audio recording as a long list of numbers measuring the little air pressure changes detected by the microphone. \n",
+ "* We will use audio sampled at 44100 Hz (or 44100 Hertz). \n",
+ " * This means the microphone gives us 44,100 numbers per second. \n",
+ " * Thus, a 10 second audio clip is represented by 441,000 numbers (= $10 \\times 44,100$). \n",
+ "\n",
+ "#### Spectrogram\n",
+ "* It is quite difficult to figure out from this \"raw\" representation of audio whether the word \"activate\" was said. \n",
+ "* In order to help your sequence model more easily learn to detect trigger words, we will compute a *spectrogram* of the audio. \n",
+ "* The spectrogram tells us how much different frequencies are present in an audio clip at any moment in time. \n",
+ "* If you've ever taken an advanced class on signal processing or on Fourier transforms:\n",
+ " * A spectrogram is computed by sliding a window over the raw audio signal, and calculating the most active frequencies in each window using a Fourier transform. \n",
+ " * If you don't understand the previous sentence, don't worry about it.\n",
+ "\n",
+ "Let's look at an example. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "IPython.display.Audio(\"audio_examples/example_train.wav\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "x = graph_spectrogram(\"audio_examples/example_train.wav\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The graph above represents how active each frequency is (y axis) over a number of time-steps (x axis). \n",
+ "\n",
+ "![](\"images/spectrogram.png\")
\n",
+ "![](\"images/label_diagram.png\")
\n",
+ "| \n", + " **Overlap 1**\n", + " | \n", + "\n", + " False\n", + " | \n", + "
| \n", + " **Overlap 2**\n", + " | \n", + "\n", + " True\n", + " | \n", + "
| \n", + " **Segment Time**\n", + " | \n", + "\n", + " (2254, 3169)\n", + " | \n", + "
| \n", + " **sanity checks**:\n", + " | \n", + "\n", + " 0.0 1.0 0.0\n", + " | \n", + "
![](\"images/ones_reference.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Creating a training example\n",
+ "Finally, you can use `insert_audio_clip` and `insert_ones` to create a new training example.\n",
+ "\n",
+ "\n",
+ "### Exercise 4 - create_training_example\n",
+ "\n",
+ "Implement `create_training_example()`. You will need to carry out the following steps:\n",
+ "\n",
+ "1. Initialize the label vector $y$ as a numpy array of zeros and shape $(1, T_y)$.\n",
+ "2. Initialize the set of existing segments to an empty list.\n",
+ "3. Randomly select 0 to 4 \"activate\" audio clips, and insert them onto the 10 second clip. Also insert labels at the correct position in the label vector $y$.\n",
+ "4. Randomly select 0 to 2 negative audio clips, and insert them into the 10 second clip. \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# UNQ_C4\n",
+ "# GRADED FUNCTION: create_training_example\n",
+ "\n",
+ "def create_training_example(background, activates, negatives, Ty):\n",
+ " \"\"\"\n",
+ " Creates a training example with a given background, activates, and negatives.\n",
+ " \n",
+ " Arguments:\n",
+ " background -- a 10 second background audio recording\n",
+ " activates -- a list of audio segments of the word \"activate\"\n",
+ " negatives -- a list of audio segments of random words that are not \"activate\"\n",
+ " Ty -- The number of time steps in the output\n",
+ "\n",
+ " Returns:\n",
+ " x -- the spectrogram of the training example\n",
+ " y -- the label at each time step of the spectrogram\n",
+ " \"\"\"\n",
+ " \n",
+ " # Make background quieter\n",
+ " background = background - 20\n",
+ "\n",
+ " ### START CODE HERE ###\n",
+ " # Step 1: Initialize y (label vector) of zeros (≈ 1 line)\n",
+ " y = None\n",
+ "\n",
+ " # Step 2: Initialize segment times as empty list (≈ 1 line)\n",
+ " previous_segments = None\n",
+ " ### END CODE HERE ###\n",
+ " \n",
+ " # Select 0-4 random \"activate\" audio clips from the entire list of \"activates\" recordings\n",
+ " number_of_activates = np.random.randint(0, 5)\n",
+ " random_indices = np.random.randint(len(activates), size=number_of_activates)\n",
+ " random_activates = [activates[i] for i in random_indices]\n",
+ " \n",
+ " ### START CODE HERE ### (≈ 3 lines)\n",
+ " # Step 3: Loop over randomly selected \"activate\" clips and insert in background\n",
+ " for one_random_activate in random_activates:\n",
+ " # Insert the audio clip on the background\n",
+ " background, segment_time = None\n",
+ " # Retrieve segment_start and segment_end from segment_time\n",
+ " segment_start, segment_end = None\n",
+ " # Insert labels in \"y\" at segment_end\n",
+ " y = None\n",
+ " ### END CODE HERE ###\n",
+ "\n",
+ " # Select 0-2 random negatives audio recordings from the entire list of \"negatives\" recordings\n",
+ " number_of_negatives = np.random.randint(0, 3)\n",
+ " random_indices = np.random.randint(len(negatives), size=number_of_negatives)\n",
+ " random_negatives = [negatives[i] for i in random_indices]\n",
+ "\n",
+ " ### START CODE HERE ### (≈ 2 lines)\n",
+ " # Step 4: Loop over randomly selected negative clips and insert in background\n",
+ " for random_negative in random_negatives:\n",
+ " # Insert the audio clip on the background \n",
+ " background, _ = None\n",
+ " ### END CODE HERE ###\n",
+ " \n",
+ " # Standardize the volume of the audio clip \n",
+ " background = match_target_amplitude(background, -20.0)\n",
+ "\n",
+ " # Export new training example \n",
+ " file_handle = background.export(\"train\" + \".wav\", format=\"wav\")\n",
+ " \n",
+ " # Get and plot spectrogram of the new recording (background with superposition of positive and negatives)\n",
+ " x = graph_spectrogram(\"train.wav\")\n",
+ " \n",
+ " return x, y"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "deletable": false,
+ "editable": false
+ },
+ "outputs": [],
+ "source": [
+ "### THIS CELL IS NOT EDITABLE\n",
+ "\n",
+ "# UNIT TEST\n",
+ "def create_training_example_test(target):\n",
+ " np.random.seed(18)\n",
+ " x, y = target(backgrounds[0], activates, negatives, 1375)\n",
+ " \n",
+ " assert type(x) == np.ndarray, \"Wrong type for x\"\n",
+ " assert type(y) == np.ndarray, \"Wrong type for y\"\n",
+ " assert tuple(x.shape) == (101, 5511), \"Wrong shape for x\"\n",
+ " assert tuple(y.shape) == (1, 1375), \"Wrong shape for y\"\n",
+ " assert np.all(x > 0), \"All x values must be higher than 0\"\n",
+ " assert np.all(y >= 0), \"All y values must be higher or equal than 0\"\n",
+ " assert np.all(y <= 1), \"All y values must be smaller or equal than 1\"\n",
+ " assert np.sum(y) >= 50, \"It must contain at least one activate\"\n",
+ " assert np.sum(y) % 50 == 0, \"Sum of activate marks must be a multiple of 50\"\n",
+ " assert np.isclose(np.linalg.norm(x), 39745552.52075), \"Spectrogram is wrong. Check the parameters passed to the insert_audio_clip function\"\n",
+ "\n",
+ " print(\"\\033[92m All tests passed!\")\n",
+ "\n",
+ "create_training_example_test(create_training_example)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Set the random seed\n",
+ "np.random.seed(18)\n",
+ "x, y = create_training_example(backgrounds[0], activates, negatives, Ty)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Expected Output**\n",
+ "![](\"images/train_reference.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now you can listen to the training example you created and compare it to the spectrogram generated above."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "IPython.display.Audio(\"train.wav\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Expected Output**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "IPython.display.Audio(\"audio_examples/train_reference.wav\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Finally, you can plot the associated labels for the generated training example."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "plt.plot(y[0])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Expected Output**\n",
+ "![](\"images/train_label.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "### 1.4 - Full Training Set\n",
+ "\n",
+ "* You've now implemented the code needed to generate a single training example. \n",
+ "* We used this process to generate a large training set. \n",
+ "* To save time, we generate a smaller training set of 32 examples. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# START SKIP FOR GRADING\n",
+ "np.random.seed(4543)\n",
+ "nsamples = 32\n",
+ "X = []\n",
+ "Y = []\n",
+ "for i in range(0, nsamples):\n",
+ " if i%10 == 0:\n",
+ " print(i)\n",
+ " x, y = create_training_example(backgrounds[i % 2], activates, negatives, Ty)\n",
+ " X.append(x.swapaxes(0,1))\n",
+ " Y.append(y.swapaxes(0,1))\n",
+ "X = np.array(X)\n",
+ "Y = np.array(Y)\n",
+ "# END SKIP FOR GRADING"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You would like to save your dataset into a file that you can load later if you work in a more realistic environment. We let you the following code for reference. Don't try to run it into Coursera since the file system is read-only, and you cannot save files."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Save the data for further uses\n",
+ "# np.save(f'./XY_train/X.npy', X)\n",
+ "# np.save(f'./XY_train/Y.npy', Y)\n",
+ "# Load the preprocessed training examples\n",
+ "# X = np.load(\"./XY_train/X.npy\")\n",
+ "# Y = np.load(\"./XY_train/Y.npy\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "### 1.5 - Development Set\n",
+ "\n",
+ "* To test our model, we recorded a development set of 25 examples. \n",
+ "* While our training data is synthesized, we want to create a development set using the same distribution as the real inputs. \n",
+ "* Thus, we recorded 25 10-second audio clips of people saying \"activate\" and other random words, and labeled them by hand. \n",
+ "* This follows the principle described in Course 3 \"Structuring Machine Learning Projects\" that we should create the dev set to be as similar as possible to the test set distribution\n",
+ " * This is why our **dev set uses real audio** rather than synthesized audio. \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Load preprocessed dev set examples\n",
+ "X_dev = np.load(\"./XY_dev/X_dev.npy\")\n",
+ "Y_dev = np.load(\"./XY_dev/Y_dev.npy\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "## 2 - The Model\n",
+ "\n",
+ "* Now that you've built a dataset, let's write and train a trigger word detection model! \n",
+ "* The model will use 1-D convolutional layers, GRU layers, and dense layers. \n",
+ "* Let's load the packages that will allow you to use these layers in Keras. This might take a minute to load. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from tensorflow.keras.callbacks import ModelCheckpoint\n",
+ "from tensorflow.keras.models import Model, load_model, Sequential\n",
+ "from tensorflow.keras.layers import Dense, Activation, Dropout, Input, Masking, TimeDistributed, LSTM, Conv1D\n",
+ "from tensorflow.keras.layers import GRU, Bidirectional, BatchNormalization, Reshape\n",
+ "from tensorflow.keras.optimizers import Adam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "### 2.1 - Build the Model\n",
+ "\n",
+ "Our goal is to build a network that will ingest a spectrogram and output a signal when it detects the trigger word. This network will use 4 layers:\n",
+ " * A convolutional layer\n",
+ " * Two GRU layers\n",
+ " * A dense layer. \n",
+ "\n",
+ "Here is the architecture we will use.\n",
+ "\n",
+ "![](\"images/model.png\")
\n",
+ "| \n", + " **Total params**\n", + " | \n", + "\n", + " 523,329\n", + " | \n", + "
| \n", + " **Trainable params**\n", + " | \n", + "\n", + " 522,425\n", + " | \n", + "
| \n", + " **Non-trainable params**\n", + " | \n", + "\n", + " 904\n", + " | \n", + "