# [Math-Bot] Siamese LSTM:  Detecting duplicates

Author: Alin-Andrei Georgescu 2021

Welcome to my notebook! It explores the Siamese networks applied to natural language processing. The model is intended to detect duplicates, in other words to check if two sentences are similar.
The model uses "Long short-term memory" (LSTM) neural networks, which are an artificial recurrent neural networks (RNNs). This version uses a custom embedding layer with weigths based on GloVe pretrained vectors.

## Outline

- [Overview](#0)
- [Part 1: Importing the Data](#1)
    - [1.1 Loading in the data](#1.1)
    - [1.2 Converting a sentence to a tensor](#1.2)
    - [1.3 Understanding and building the iterator](#1.3)
- [Part 2: Defining the Siamese model](#2)
    - [2.1 Understanding and building the Siamese Network](#2.1)
    - [2.2 Implementing Hard Negative Mining](#2.2)
- [Part 3: Training](#3)
- [Part 4: Evaluation](#4)
- [Part 5: Making predictions](#5)

### Overview

General ideas:
- Designing a Siamese networks model
- Implementing the triplet loss
- Evaluating accuracy
- Using cosine similarity between the model's outputted vectors
- Working with Trax and Numpy libraries in Python 3

The LSTM cell's architecture (source: https://www.researchgate.net/figure/The-structure-of-the-LSTM-unit_fig2_331421650):

I will start by preprocessing the data, then I will build a classifier that will identify whether two sentences are the same or not.

I tokenized the data, then split the dataset into training and testing sets. I loaded pretrained GloVe word embeddings and built a sentence's vector by averaging the composing word's vectors. The model takes in the two sentence embeddings, runs them through an LSTM, and then compares the outputs of the two sub networks using cosine similarity.

This notebook has been built based on Coursera's Natural Language Processing Specialization.

# Part 1: Importing the Data

### 1.1 Loading in the data

First step in building a model is building a dataset. I used three datasets in building my model:
- the Quora Question Pairs
- edited SICK dataset
- custom Maths duplicates dataset

Run the cell below to import some of the needed packages.

```python
import os
import re
import nltk
from nltk.corpus import stopwords
nltk.download('punkt')
nltk.download('stopwords')
nltk.download('wordnet')

import numpy as np
import pandas as pd
import random as rnd

!pip install textcleaner
import textcleaner as tc

!pip install trax
import trax
from trax import layers as tl
from trax.supervised import training
from trax.fastmath import numpy as fastnp

from collections import defaultdict

# set random seeds
rnd.seed(34)
```

**Notice that in this notebook Trax's numpy is referred to as `fastnp`, while regular numpy is referred to as `np`.**

Now the dataset and word embeddings will get loaded and the data processed.

```python
data = pd.read_csv("data/merged_dataset.csv", encoding="utf-8")

N = len(data)
print("Number of sentence pairs: ", N)

data.head()

!wget -O data/glove.840B.300d.zip nlp.stanford.edu/data/glove.840B.300d.zip
!unzip -d data data/glove.840B.300d.zip
!rm data/glove.840B.300d.zip

!wget -O data/glove.6B.zip nlp.stanford.edu/data/glove.6B.zip
!unzip -d data data/glove.6B.zip
!rm data/glove.6B.zip
!rm data/glove.6B.200d.txt
!rm data/glove.6B.100d.txt
!rm data/glove.6B.50d.txt

# load the whole embedding into memory
embeddings_index = defaultdict(lambda: 1)
embeddings = dict()
index = 0

pad = "<PAD>"
embeddings[pad] = np.zeros(300)
embeddings_index[pad] = index
index += 1

unk = "<UNK>"
embeddings[unk] = np.asarray([0.22418134, -0.28881392, 0.13854356, 0.00365387, -0.12870757, 0.10243822, 0.061626635, 0.07318011, -0.061350107, -1.3477012, 0.42037755, -0.063593924, -0.09683349, 0.18086134, 0.23704372, 0.014126852, 0.170096, -1.1491593, 0.31497982, 0.06622181, 0.024687296, 0.076693475, 0.13851812, 0.021302193, -0.06640582, -0.010336159, 0.13523154, -0.042144544, -0.11938788, 0.006948221, 0.13333307, -0.18276379, 0.052385733, 0.008943111, -0.23957317, 0.08500333, -0.006894406, 0.0015864656, 0.063391194, 0.19177166, -0.13113557, -0.11295479, -0.14276934, 0.03413971, -0.034278486, -0.051366422, 0.18891625, -0.16673574, -0.057783455, 0.036823478, 0.08078679, 0.022949161, 0.033298038, 0.011784158, 0.05643189, -0.042776518, 0.011959623, 0.011552498, -0.0007971594, 0.11300405, -0.031369694, -0.0061559738, -0.009043574, -0.415336, -0.18870236, 0.13708843, 0.005911723, -0.113035575, -0.030096142, -0.23908928, -0.05354085, -0.044904727, -0.20228513, 0.0065645403, -0.09578946, -0.07391877, -0.06487607, 0.111740574, -0.048649278, -0.16565254, -0.052037314, -0.078968436, 0.13684988, 0.0757494, -0.006275573, 0.28693774, 0.52017444, -0.0877165, -0.33010918, -0.1359622, 0.114895485, -0.09744406, 0.06269521, 0.12118575, -0.08026362, 0.35256687, -0.060017522, -0.04889904, -0.06828978, 0.088740796, 0.003964443, -0.0766291, 0.1263925, 0.07809314, -0.023164088, -0.5680669, -0.037892066, -0.1350967, -0.11351585, -0.111434504, -0.0905027, 0.25174105, -0.14841858, 0.034635577, -0.07334565, 0.06320108, -0.038343467, -0.05413284, 0.042197507, -0.090380974, -0.070528865, -0.009174437, 0.009069661, 0.1405178, 0.02958134, -0.036431845, -0.08625681, 0.042951006, 0.08230793, 0.0903314, -0.12279937, -0.013899368, 0.048119213, 0.08678239, -0.14450377, -0.04424887, 0.018319942, 0.015026873, -0.100526, 0.06021201, 0.74059093, -0.0016333034, -0.24960588, -0.023739101, 0.016396184, 0.11928964, 0.13950661, -0.031624354, -0.01645025, 0.14079992, -0.0002824564, -0.08052984, -0.0021310581, -0.025350995, 0.086938225, 0.14308536, 0.17146006, -0.13943303, 0.048792403, 0.09274929, -0.053167373, 0.031103406, 0.012354865, 0.21057427, 0.32618305, 0.18015954, -0.15881181, 0.15322933, -0.22558987, -0.04200665, 0.0084689725, 0.038156632, 0.15188617, 0.13274793, 0.113756925, -0.095273495, -0.049490947, -0.10265804, -0.27064866, -0.034567792, -0.018810693, -0.0010360252, 0.10340131, 0.13883452, 0.21131058, -0.01981019, 0.1833468, -0.10751636, -0.03128868, 0.02518242, 0.23232952, 0.042052146, 0.11731903, -0.15506615, 0.0063580726, -0.15429358, 0.1511722, 0.12745973, 0.2576985, -0.25486213, -0.0709463, 0.17983761, 0.054027, -0.09884228, -0.24595179, -0.093028545, -0.028203879, 0.094398156, 0.09233813, 0.029291354, 0.13110267, 0.15682974, -0.016919162, 0.23927948, -0.1343307, -0.22422817, 0.14634751, -0.064993896, 0.4703685, -0.027190214, 0.06224946, -0.091360025, 0.21490277, -0.19562101, -0.10032754, -0.09056772, -0.06203493, -0.18876675, -0.10963594, -0.27734384, 0.12616494, -0.02217992, -0.16058226, -0.080475815, 0.026953284, 0.110732645, 0.014894041, 0.09416802, 0.14299914, -0.1594008, -0.066080004, -0.007995227, -0.11668856, -0.13081996, -0.09237365, 0.14741232, 0.09180138, 0.081735, 0.3211204, -0.0036552632, -0.047030564, -0.02311798, 0.048961394, 0.08669574, -0.06766279, -0.50028914, -0.048515294, 0.14144728, -0.032994404, -0.11954345, -0.14929578, -0.2388355, -0.019883996, -0.15917352, -0.052084364, 0.2801028, -0.0029121689, -0.054581646, -0.47385484, 0.17112483, -0.12066923, -0.042173345, 0.1395337, 0.26115036, 0.012869649, 0.009291686, -0.0026459037, -0.075331464, 0.017840583, -0.26869613, -0.21820338, -0.17084768, -0.1022808, -0.055290595, 0.13513643, 0.12362477, -0.10980586, 0.13980341, -0.20233242, 0.08813751, 0.3849736, -0.10653763, -0.06199595, 0.028849555, 0.03230154, 0.023856193, 0.069950655, 0.19310954, -0.077677034, -0.144811], dtype='float32')
embeddings_index[unk] = index
index += 1

f = open("data/glove.6B.300d.txt")
for line in f:
    values = line.split()
    word = values[0]
    coefs = np.asarray(values[1:], dtype="float32")
    embeddings_index[word] = index
    index += 1
    embeddings[word] = coefs
f.close()

vocab_size = index + 1
print("Loaded %s word vectors." % vocab_size)
```

Then I split the data into a train and test set. The test set will be used later to evaluate the model.

```python
N_dups = len(data[data.is_duplicate == 1])

# Take 90% of the duplicates for the train set
N_train = int(N_dups * 0.9)
print(N_train)

# Take the rest of the duplicates for the test set + an equal number of non-dups
N_test = (N_dups - N_train) * 2
print(N_test)

data_train = data[: N_train]
# Shuffle the train set
data_train = data_train.sample(frac=1)

data_test = data[N_train : N_train + N_test]
# Shuffle the test set
data_test = data_test.sample(frac=1)

print("Train set: ", len(data_train), "; Test set: ", len(data_test))

# Remove the unneeded data to some memory
del(data)
```

```python
S1_train_words = np.array(data_train["sentence1"])
S2_train_words = np.array(data_train["sentence2"])

S1_test_words = np.array(data_test["sentence1"])
S2_test_words = np.array(data_test["sentence2"])
y_test  = np.array(data_test["is_duplicate"])

del(data_train)
del(data_test)
```

Above, you have seen that the model only takes the duplicated sentences for training.
All this has a purpose, as the data generator will produce batches $([s1_1, s1_2, s1_3, ...]$, $[s2_1, s2_2,s2_3, ...])$, where $s1_i$ and $s2_k$ are duplicate if and only if $i = k$.

An example of how the data looks is shown below.

```python
print("TRAINING SENTENCES:\n")
print("Sentence 1: ", S1_train_words[0])
print("Sentence 2: ", S2_train_words[0], "\n")
print("Sentence 1: ", S1_train_words[5])
print("Sentence 2: ", S2_train_words[5], "\n")

print("TESTING SENTENCES:\n")
print("Sentence 1: ", S1_test_words[0])
print("Sentence 2: ", S2_test_words[0], "\n")
print("is_duplicate =", y_test[0], "\n")
```

The first step is to tokenize the sentences using a custom tokenizer defined below.

```python
# Create arrays
S1_train = np.empty_like(S1_train_words)
S2_train = np.empty_like(S2_train_words)

S1_test = np.empty_like(S1_test_words)
S2_test = np.empty_like(S2_test_words)
```

```python
def data_tokenizer(sentence):
    """Tokenizer function - cleans and tokenizes the data

    Args:
        sentence (str): The input sentence.
    Returns:
        list: The transformed input sentence.
    """

    if sentence == "":
        return ""

    sentence = tc.lower_all(sentence)[0]

    # Change tabs to spaces
    sentence = re.sub(r"\t+_+", " ", sentence)
    # Change short forms
    sentence = re.sub(r"\'ve", " have", sentence)
    sentence = re.sub(r"(can\'t|can not)", "cannot", sentence)
    sentence = re.sub(r"n\'t", " not", sentence)
    sentence = re.sub(r"I\'m", "I am", sentence)
    sentence = re.sub(r" m ", " am ", sentence)
    sentence = re.sub(r"(\'re| r )", " are ", sentence)
    sentence = re.sub(r"\'d", " would ", sentence)
    sentence = re.sub(r"\'ll", " will ", sentence)
    sentence = re.sub(r"(\d+)(k)", r"\g<1>000", sentence)
    # Make word separations
    sentence = re.sub(r"(\+|-|\*|\/|\^|\.)", " $1 ", sentence)
    # Remove irrelevant stuff, nonprintable characters and spaces
    sentence = re.sub(r"(\'s|\'S|\'|\"|,|[^ -~]+)", "", sentence)
    sentence = tc.strip_all(sentence)[0]

    if sentence == "":
        return ""

    return tc.token_it(tc.lemming(sentence))[0]
```

```python
for idx in range(len(S1_train_words)):
    S1_train[idx] = data_tokenizer(S1_train_words[idx])

for idx in range(len(S2_train_words)):
    S2_train[idx] = data_tokenizer(S2_train_words[idx])

for idx in range(len(S1_test_words)):
    S1_test[idx] = data_tokenizer(S1_test_words[idx])

for idx in range(len(S2_test_words)):
    S2_test[idx] = data_tokenizer(S2_test_words[idx])
```

### 1.2 Converting a sentence to a tensor

The next step is to convert every sentence to a tensor, or an array of numbers, using the word embeddings loaded above.

```python
# Converting sentences to arrays of integers
sentence_of_indexes = []

for i in range(len(S1_train)):
    for word in S1_train[i]:
        sentence_of_indexes += [embeddings_index[word]] if word in embeddings_index else [embeddings_index[unk]]

    S1_train[i] = sentence_of_indexes
    sentence_of_indexes = []

for i in range(len(S2_train)):
    for word in S2_train[i]:
        sentence_of_indexes += [embeddings_index[word]] if word in embeddings_index else [embeddings_index[unk]]

    S2_train[i] = sentence_of_indexes
    sentence_of_indexes = []

for i in range(len(S1_test)):
    for word in S1_test[i]:
        sentence_of_indexes += [embeddings_index[word]] if word in embeddings_index else [embeddings_index[unk]]

    S1_test[i] = sentence_of_indexes
    sentence_of_indexes = []

for i in range(len(S2_test)):
    for word in S2_test[i]:
        sentence_of_indexes += [embeddings_index[word]] if word in embeddings_index else [embeddings_index[unk]]

    S2_test[i] = sentence_of_indexes
    sentence_of_indexes = []
```

```python
print("FIRST SENTENCE IN TRAIN SET:\n")
print(S1_train_words[0], "\n")
print("ENCODED VERSION:")
print(S1_train[0],"\n")
del(S1_train_words)
del(S2_train_words)

print("FIRST SENTENCE IN TEST SET:\n")
print(S1_test_words[0], "\n")
print("ENCODED VERSION:")
print(S1_test[0])
del(S1_test_words)
del(S2_test_words)
```

Now, the train set must be split into a training/validation set so that it can be used to train and evaluate the Siamese model.

```python
# Splitting the data
cut_off = int(len(S1_train) * .8)
train_S1, train_S2 = S1_train[: cut_off], S2_train[: cut_off]
val_S1, val_S2 = S1_train[cut_off :], S2_train[cut_off :]
print("Number of duplicate sentences: ", len(S1_train))
print("The length of the training set is:  ", len(train_S1))
print("The length of the validation set is: ", len(val_S1))
```

### 1.3 Understanding and building the iterator

Given the compational limits, we need to split our data into batches. In this notebook, I built a data generator that takes in $S1$ and $S2$ and returned a batch of size `batch_size` in the following format $([s1_1, s1_2, s1_3, ...]$, $[s2_1, s2_2,s2_3, ...])$. The tuple consists of two arrays and each array has `batch_size` sentences. Again, $s1_i$ and $s2_i$ are duplicates, but they are not duplicates with any other elements in the batch.

The command `next(data_generator)` returns the next batch. This iterator returns a pair of arrays of sentences, which will later be used in the model.

**The ideas behind:**
- The generator returns shuffled batches of data. To achieve this without modifying the actual sentence lists, a list containing the indexes of the sentences is created. This list can be shuffled and used to get random batches everytime the index is reset.
- Append elements of $S1$ and $S2$ to `input1` and `input2` respectively.

```python
def data_generator(S1, S2, batch_size, pad=1, shuffle=False):
    """Generator function that yields batches of data

    Args:
        S1 (list): List of transformed (to tensor) sentences.
        S2 (list): List of transformed (to tensor) sentences.
        batch_size (int): Number of elements per batch.
        pad (int, optional): Pad character from the vocab. Defaults to 1.
        shuffle (bool, optional): If the batches should be randomnized or not. Defaults to False.
    Yields:
        tuple: Of the form (input1, input2) with types (numpy.ndarray, numpy.ndarray)
        NOTE: input1: inputs to your model [s1a, s2a, s3a, ...] i.e. (s1a,s1b) are duplicates
              input2: targets to your model [s1b, s2b,s3b, ...] i.e. (s1a,s2i) i!=a are not duplicates
    """

    input1 = []
    input2 = []
    idx = 0
    len_s = len(S1)
    sentence_indexes = [*range(len_s)]

    if shuffle:
        rnd.shuffle(sentence_indexes)

    while True:
        if idx >= len_s:
            # If idx is greater than or equal to len_q, reset it
            idx = 0
            # Shuffle to get random batches if shuffle is set to True
            if shuffle:
                rnd.shuffle(sentence_indexes)

        s1 = S1[sentence_indexes[idx]]
        s2 = S2[sentence_indexes[idx]]

        idx += 1

        input1.append(s1)
        input2.append(s2)

        if len(input1) == batch_size:
            # Determine max_len as the longest sentence in input1 & input 2
            max_len = max(max([len(s) for s in input1]), max([len(s) for s in input2]))
            # Pad to power-of-2
            max_len = 2 ** int(np.ceil(np.log2(max_len)))

            b1 = []
            b2 = []
            for s1, s2 in zip(input1, input2):
                # Add [pad] to s1 until it reaches max_len
                s1 = s1 + [pad] * (max_len - len(s1))
                # Add [pad] to s2 until it reaches max_len
                s2 = s2 + [pad] * (max_len - len(s2))

                # Append s1
                b1.append(s1)
                # Append s2
                b2.append(s2)

            # Use b1 and b2
            yield np.array(b1), np.array(b2)

            # reset the batches
            input1, input2 = [], []
```

```python
batch_size = 2
res1, res2 = next(data_generator(train_S1, train_S2, batch_size))
print("First sentences  :\n", res1, "\n Shape: ", res1.shape)
print("Second sentences :\n", res2, "\n Shape: ", res2.shape)
```

Now we can go ahead and start building the neural network, as we have a data generator.

# Part 2: Defining the Siamese model

### 2.1 Understanding and building the Siamese Network

A Siamese network is a neural network which uses the same weights while working in tandem on two different input vectors to compute comparable output vectors. The Siamese network model proposed in this notebook looks like this:

The sentences' embeddings are passed to an LSTM layer, the output vectors, $v_1$ and $v_2$, are normalized, and finally a triplet loss is used to get the corresponding cosine similarity for each pair of sentences. The triplet loss makes use of a baseline (anchor) input that is compared to a positive (truthy) input and a negative (falsy) input. The distance from the baseline (anchor) input to the positive (truthy) input is minimized, and the distance from the baseline (anchor) input to the negative (falsy) input is maximized. In math equations, the following is maximized:

$$\mathcal{L}(A, P, N)=\max \left(\|\mathrm{f}(A)-\mathrm{f}(P)\|^{2}-\|\mathrm{f}(A)-\mathrm{f}(N)\|^{2}+\alpha, 0\right)$$

$A$ is the anchor input, for example $s1_1$, $P$ the duplicate input, for example, $s2_1$, and $N$ the negative input (the non duplicate sentence), for example $s2_2$.

$\alpha$ is a margin - how much the duplicates are pushed from the non duplicates.

**The ideas behind:**
- Trax library is used in implementing the model.
- `tl.Serial`: Combinator that applies layers serially (by function composition) allowing the set up the overall structure of the feedforward.
- `tl.LSTM` The LSTM layer.
- `tl.Mean`: Computes the mean across a desired axis. Mean uses one tensor axis to form groups of values and replaces each group with the mean value of that group.
- `tl.Fn` Layer with no weights that applies the function f - vector normalization in this case.
- `tl.parallel`: It is a combinator layer (like `Serial`) that applies a list of layers in parallel to its inputs.

```python
def Siamese(vocab_size=vocab_size, d_model=128):
    """Returns a Siamese model.

    Args:
        vocab_size (int, optional): Length of the vocabulary. Defaults to len(vocab).
        d_model (int, optional): Depth of the model. Defaults to 128.
        mode (str, optional): "train", "eval" or "predict", predict mode is for fast inference. Defaults to "train".

    Returns:
        trax.layers.combinators.Parallel: A Siamese model.
    """

    global embedding_layer

    def normalize(x):  # normalizes the vectors to have L2 norm 1
        return x / fastnp.sqrt(fastnp.sum(x * x, axis=-1, keepdims=True))

    embedding_layer = tl.Embedding(vocab_size, d_model)

    s_processor = tl.Serial(                        # Processor will run on S1 and S2.
        embedding_layer,
        tl.LSTM(d_model),                          # LSTM layer
        tl.Mean(axis=1),                            # Mean over columns
        tl.Fn('Normalize', lambda x: normalize(x))  # Apply normalize function
    )  # Returns one vector of shape [batch_size, d_model].

    # Run on S1 and S2 in parallel.
    model = tl.Parallel(s_processor, s_processor)
    return model
```

Setup the Siamese network model.

```python
# Check the model
model = Siamese()
embedding_layer.weights = np.asarray(list(embeddings.values()))
print(model)
```

### 2.2 Implementing Hard  Negative Mining

Now it's the time to implement the `TripletLoss`.
As explained earlier, loss is composed of two terms. One term utilizes the mean of all the non duplicates, the second utilizes the *closest negative*. The loss expression is then:

\begin{align}
 \mathcal{Loss_1(A,P,N)} &=\max \left( -cos(A,P)  + mean_{neg} +\alpha, 0\right) \\
 \mathcal{Loss_2(A,P,N)} &=\max \left( -cos(A,P)  + closest_{neg} +\alpha, 0\right) \\
\mathcal{Loss(A,P,N)} &= mean(Loss_1 + Loss_2) \\
\end{align}

```python
def TripletLossFn(v1, v2, margin=0.25):
    """Custom Loss function.

    Args:
        v1 (numpy.ndarray): Array with dimension (batch_size, model_dimension) associated to S1.
        v2 (numpy.ndarray): Array with dimension (batch_size, model_dimension) associated to S2.
        margin (float, optional): Desired margin. Defaults to 0.25.

    Returns:
        jax.interpreters.xla.DeviceArray: Triplet Loss.
    """

    scores = fastnp.dot(v1, v2.T)       # pairwise cosine sim
    batch_size = len(scores)

    positive = fastnp.diagonal(scores)  # the positive ones (duplicates)
    negative_without_positive = scores - 2.0 * fastnp.eye(batch_size)

    closest_negative = fastnp.max(negative_without_positive, axis=1)
    negative_zero_on_duplicate = (1.0 - fastnp.eye(batch_size)) * scores
    mean_negative = fastnp.sum(negative_zero_on_duplicate, axis=1) / (batch_size - 1)

    triplet_loss1 = fastnp.maximum(0.0, margin - positive + closest_negative)
    triplet_loss2 = fastnp.maximum(0.0, margin - positive + mean_negative)
    triplet_loss = fastnp.mean(triplet_loss1 + triplet_loss2)

    return triplet_loss
```

```python
v1 = np.array([[0.26726124, 0.53452248, 0.80178373],[0.5178918 , 0.57543534, 0.63297887]])
v2 = np.array([[ 0.26726124,  0.53452248,  0.80178373],[-0.5178918 , -0.57543534, -0.63297887]])
TripletLossFn(v2,v1)
print("Triplet Loss:", TripletLossFn(v2,v1))
```

**Expected Output:**
```CPP
Triplet Loss: 0.5
```

```python
from functools import partial
def TripletLoss(margin=1):
    # Trax layer creation
    triplet_loss_fn = partial(TripletLossFn, margin=margin)
    return tl.Fn("TripletLoss", triplet_loss_fn)
```

# Part 3: Training

The next step is model training - defining the cost function and the optimizer, feeding in the built model. But first I will define the data generators used in the model.

```python
batch_size = 512
train_generator = data_generator(train_S1, train_S2, batch_size)
val_generator = data_generator(val_S1, val_S2, batch_size)
print("train_S1.shape ", train_S1.shape)
print("val_S1.shape   ", val_S1.shape)
```

Now, I will define the training step. Each training iteration is defined as an `epoch`, each epoch being an iteration over all the data, using the training iterator.

**The ideas behind:**
- Two tasks are needed: `TrainTask` and `EvalTask`.
- The training runs in a trax loop `trax.supervised.training.Loop`.
- Pass the other parameters to a loop.

```python
def train_model(Siamese, TripletLoss, lr_schedule, train_generator=train_generator, val_generator=val_generator, output_dir="trax_model/"):
    """Training the Siamese Model

    Args:
        Siamese (function): Function that returns the Siamese model.
        TripletLoss (function): Function that defines the TripletLoss loss function.
        lr_schedule (function): Trax multifactor schedule function.
        train_generator (generator, optional): Training generator. Defaults to train_generator.
        val_generator (generator, optional): Validation generator. Defaults to val_generator.
        output_dir (str, optional): Path to save model to. Defaults to "trax_model/".

    Returns:
        trax.supervised.training.Loop: Training loop for the model.
    """

    output_dir = os.path.expanduser(output_dir)

    train_task = training.TrainTask(
        labeled_data=train_generator,
        loss_layer=TripletLoss(),
        optimizer=trax.optimizers.Adam(0.01),
        lr_schedule=lr_schedule
    )

    eval_task = training.EvalTask(
        labeled_data=val_generator,
        metrics=[TripletLoss()]
    )

    training_loop = training.Loop(Siamese(),
                                  train_task,
                                  eval_tasks=[eval_task],
                                  output_dir=output_dir,
                                  random_seed=34)

    return training_loop
```

```python
train_steps = 1500
lr_schedule = trax.lr.warmup_and_rsqrt_decay(400, 0.01)
training_loop = train_model(Siamese, TripletLoss, lr_schedule)
training_loop.run(train_steps)
```

# Part 4:  Evaluation

To determine the accuracy of the model, the test set that was configured earlier is used. While the training used only positive examples, the test data, S1_test, S2_test and y_test, is setup as pairs of sentences, some of which are duplicates some are not.
This routine runs all the test sentences pairs through the model, computes the cosine simlarity of each pair, thresholds it and compares the result to  y_test - the correct response from the data set. The results are accumulated to produce an accuracy.

**The ideas behind:**
 - The model loops through the incoming data in batch_size chunks.
 - The output vectors are computed and their cosine similarity is thresholded.

```python
def classify(test_S1, test_S2, y, threshold, model, data_generator=data_generator, batch_size=64):
    """Function to test the model. Calculates some metrics, such as precision, accuracy, recall and F1 score.

    Args:
        test_S1 (numpy.ndarray): Array of S1 sentences.
        test_S2 (numpy.ndarray): Array of S2 sentences.
        y (numpy.ndarray): Array of actual target.
        threshold (float): Desired threshold.
        model (trax.layers.combinators.Parallel): The Siamese model.
        data_generator (function): Data generator function. Defaults to data_generator.
        batch_size (int, optional): Size of the batches. Defaults to 64.

    Returns:
        (float, float, float, float): Accuracy, precision, recall and F1 score of the model.
    """

    true_pos = 0
    true_neg = 0
    false_pos = 0
    false_neg = 0

    for i in range(0, len(test_S1), batch_size):
        to_process = len(test_S1) - i

        if to_process < batch_size:
            batch_size = to_process

        s1, s2 = next(data_generator(test_S1[i : i + batch_size], test_S2[i : i + batch_size], batch_size, shuffle=False))
        y_test = y[i : i + batch_size]

        v1, v2 = model((s1, s2))

        for j in range(batch_size):
            d = np.dot(v1[j], v2[j].T)
            res = d > threshold

            if res == 1:
                if y_test[j] == res:
                    true_pos += 1
                else:
                    false_pos += 1
            else:
                if y_test[j] == res:
                    true_neg += 1
                else:
                    false_neg += 1

    accuracy = (true_pos + true_neg) / (true_pos + true_neg + false_pos + false_neg)
    precision = true_pos / (true_pos + false_pos)
    recall = true_pos / (true_pos + false_neg)
    f1_score = 2 * precision * recall / (precision + recall)

    print("fn = " +  str(false_neg) + " fp = " + str(false_pos) + " tn = " + str(true_neg) + " tp = " + str(true_pos))

    return (accuracy, precision, recall, f1_score)
```

```python
print(len(S1_test))
```

```python
# Loading in the saved model
model = Siamese()
model.init_from_file("trax_model/model.pkl.gz")
# Evaluating it
accuracy, precision, recall, f1_score = classify(S1_test, S2_test, y_test, 0.7, model, batch_size=512)
print("Accuracy", accuracy)
print("Precision", precision)
print("Recall", recall)
print("F1 score", f1_score)
```

# Part 5: Making predictions

In this section the model will be put to work. It will be wrapped in a function called `predict` which takes two sentences as input and returns $1$ or $0$, depending on whether the pair is a duplicate or not.

But first, we need to embed the sentences.

```python
def predict(sentence1, sentence2, threshold, model, vocab, data_generator=data_generator, verbose=False):
    """Function for predicting if two sentences are duplicates.

    Args:
        sentence1 (str): First sentence.
        sentence2 (str): Second sentence.
        threshold (float): Desired threshold.
        model (trax.layers.combinators.Parallel): The Siamese model.
        vocab (collections.defaultdict): The vocabulary used.
        data_generator (function): Data generator function. Defaults to data_generator.
        verbose (bool, optional): If the results should be printed out. Defaults to False.

    Returns:
        bool: True if the sentences are duplicates, False otherwise.
    """

    s1 = data_tokenizer(sentence1)     # tokenize
    s2 = data_tokenizer(sentence2)     # tokenize

    S1 = [embeddings_index[word] for word in s1]  # encode s1
    S2 = [embeddings_index[word] for word in s2]  # encode s2

    S1, S2 = next(data_generator([S1], [S2], 1, vocab["<PAD>"]))

    v1, v2 = model((S1, S2))
    d = np.dot(v1[0], v2[0].T)
    res = d > threshold

    if verbose == True:
        print("S1  = ", S1, "\nS2  = ", S2)
        print("d   = ", d)
        print("res = ", res)

    return res
```

Now we can test the model's ability to make predictions.

```python
sentence1 = "I love running in the park."
sentence2 = "I like running in park?"
# 1 means it is duplicated, 0 otherwise
predict(sentence1 , sentence2, 0.7, model, verbose=True)
```

The Siamese network is capable of catching complicated structures. Concretely, it can identify sentence duplicates although the sentences do not have many words in common.


|
Name
               |
Date
 |
| ---------------------------------------------------| ------------------------------------- |
|
Diaaeldin SHALABY
   | 07.052021                            |

Hands-on AI II

Unit 2 — The Vanishing Gradient Problem (Assignment)

Authors: S. Lehner, J. Brandstetter, B. Schäfl

Date: 16-04-2021

This file is part of the "Hands-on AI II" lecture material. The following copyright statement applies to all code within this file.

Copyright statement:

This  material,  no  matter  whether  in  printed  or  electronic  form,  may  be  used  for personal  and non-commercial educational use only.  Any reproduction of this manuscript, no matter whether as a whole or in parts, no matter whether in printed or in electronic form, requires explicit prior acceptance of the authors.

Table of contents

Definition of Auxiliaries

Loading and visualizing

Downprojecting and interpreting

Loading and preparing

Training of a Neural Network

Constructing an FNN

Forward pass

Backward pass

Analyzing Gradients

Collecting and visualizing

Countermeasure and re-train

Comparing gradients

Deriving Derivatives

Case <tt>hardsigmoid</tt>

Case <tt>leaky_relu</tt>

How to use this notebook

This notebook is designed to run from start to finish. There are different tasks (displayed in orange boxes) which require your contribution (in form of code, plain text, ...). Most/All of the supplied functions are imported from the file u2_utils.py which can be seen and treated as a black box. However, for further understanding, you can look at the implementations of the helper functions. In order to run this notebook, the packages which are imported at the beginning of u2_utils.py need to be installed.

```python
# Import pre-defined utilities specific to this notebook.
import u2_utils as u2

# Import additional utilities needed in this notebook.
import math
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import torch

from typing import Dict, Sequence

# Setup Jupyter notebook (warning: this may affect all Jupyter notebooks running on the same Jupyter server).
u2.setup_jupyter()
```

Setting up notebook ... finished.

Module versions

As mentioned in the introductiory slides, specific minimum versions of Python itself as well as of used modules is recommended.

```python
u2.check_module_versions()
```

    Installed Python version: 3.8 (✓)
    Installed numpy version: 1.19.1 (✓)
    Installed pandas version: 1.1.3 (✓)
    Installed PyTorch version: 1.7.1 (✓)
    Installed scikit-learn version: 0.23.2 (✓)
    Installed scipy version: 1.5.0 (✓)
    Installed matplotlib version: 3.3.1 (✓)
    Installed seaborn version: 0.11.0 (✓)
    Installed PIL version: 8.0.0 (✓)

Definition of Auxiliaries

In this exercise you will be working with a data set composed of images of various handwritten digits. It is probably the most prominent data set in the domain of machine learning: the MNIST data set. The data set distinguishes ten different classes, one for each digit (zero to nine). For curious minds, more information regarding this data set can be found at:

    LeCun, Y., 1998. The MNIST database of handwritten digits. http://yann.lecun.com/exdb/mnist/.

Before analyzing and tackling the vanishing gradient problem, the data sets needs to be inspected.

    Execute the notebook until here and try to solve the following tasks:

Load the MNIST data set using the appropriate function as supplied by us.

Divide the data set between the training set and the test set in a ratio of $7:1$.

Visualize the MNIST training set in tabular form. What is the size of both subsets with respect to sample and feature counts?

```python
data_mnist = u2.load_mnist()
data_mnist
```

      PX_0
      PX_1
      PX_2
      PX_3
      PX_4
      PX_5
      PX_6
      PX_7
      PX_8
      PX_9
      ...
      PX_775
      PX_776
      PX_777
      PX_778
      PX_779
      PX_780
      PX_781
      PX_782
      PX_783
      digit

      0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      5

      1
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0

      2
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      4

      3
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1

      4
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      9

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      69995
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      2

      69996
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      3

      69997
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      4

      69998
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      5

      69999
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      6

70000 rows × 785 columns

```python
# Set default plotting style and random seed for reproducibility.
sns.set()
np.random.seed(seed=42)

# Split the Fashion-MNIST data set into training as well as test set and print their respective size.
data_mnist_train, data_mnist_test = u2.split_data(data_mnist, test_size=1.0 / 8.0)
print(f'{"Full data set is of size:":>27} {data_mnist.shape[0]:>5}')
print(f'Training subset is of size: {data_mnist_train.shape[0]:>5}')
print(f'{"Testing subset is of size:":>27} {data_mnist_test.shape[0]:>5}')
```

      Full data set is of size: 70000
    Training subset is of size: 61250
     Testing subset is of size:  8750

    Execute the notebook until here and try to solve the following tasks:

Reduce the dimensionality of the MNIST training set using PCA and visualize the downprojection.

Comment on the separability of the MNIST training set with respect to the downprojection.

```python
# Set default plotting style and random seed for reproducibility.
sns.set()
np.random.seed(seed=42)

data_mnist_pca = u2.apply_pca(data=data_mnist,n_components=2)
u2.plot_points_2d(data=data_mnist_pca, figsize=(14, 7))
```

    Execute the notebook until here and try to solve the following tasks:

Create a corresponding <tt>TensorDataset</tt> for the training as well as the test set.

Wrap the previously defined <tt>TensorDataset</tt> instances in separate <tt>DataLoader</tt> instances with a batch size of $64$ (shuffle the training data set).

Scale the features of the training as well as test set by a factor of $1\,/\,255$.

```python
# Set random seed for reproducibility.
np.random.seed(seed=42)
torch.manual_seed(seed=42)

# Create data loader for iterating the Fashion-MNIST training data set.
loader_mnist_train = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(
    torch.from_numpy(data_mnist_train.drop(columns=[r'digit']).values.astype(
        dtype=np.float32) / 255.0).unsqueeze(1).reshape(len(data_mnist_train), 28 * 28),
    torch.from_numpy(data_mnist_train[r'digit'].values.astype(dtype=np.long))
), batch_size=64, shuffle=True, drop_last=False)

# Create data loader for iterating the Fashion-MNIST test data set.
loader_mnist_test = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(
    torch.from_numpy(data_mnist_test.drop(columns=[r'digit']).values.astype(
        dtype=np.float32) / 255.0).unsqueeze(1).reshape(len(data_mnist_test), 28 * 28),
    torch.from_numpy(data_mnist_test[r'digit'].values.astype(dtype=np.long))
), batch_size=64, shuffle=False, drop_last=False)
```

    The following code snippet is taken from the accompanying exercise notebook. You do not need to modify it for this assignment.

```python
def train_and_evaluate(model: torch.nn.Module, optimizer: torch.optim.Optimizer,
                       device: torch.device, num_epochs: int,
                       loader_train: torch.utils.data.DataLoader,
                       loader_test: torch.utils.data.DataLoader) -> None:
    """
    Auxiliary function for training and evaluating a corresponding model.

    :param model: model instance to train and evaluate
    :param optimizer: optimizer to use for model training
    :param device: device to use for model training and evaluation
    :param num_epochs: amount of epochs for model training
    :param loader_train: data loader supplying the training samples
    :param loader_test: data loader supplying the test samples
    :return: None
    """
    for epoch in range(num_epochs):

        # Train model instance for one epoch.
        u2.train_network(
            model=model, data_loader=loader_train, device=device, optimizer=optimizer)

        # Evaluate current model instance.
        performance = u2.test_network(
            model=model, data_loader=loader_train, device=device)

        # Print result of current epoch to standard out.
        print(f'Epoch: {str(epoch + 1).zfill(len(str(num_epochs)))} ' +
              f'/ Loss: {performance[0]:.4f} / Accuracy: {performance[1]:.4f}')

    # Evaluate final model on test data set.
    performance = u2.test_network(model=model, data_loader=loader_test, device=device)
    print(f'\nFinal loss: {performance[0]:.4f} / Final accuracy: {performance[1]:.4f}')
```

Training of a Neural Network

Loading and inspecting a new data set is always an exciting moment, but even more exciting is the implementation of a corresponding neural network and applying it to said data set. Hence, in this section you will have to implement and train an appropriate neural network model and revisit your knowledge about the forward as well as the backward pass.

    Execute the notebook until here and try to solve the following tasks:

Implement a class <tt>FNN_0</tt> with the following architecture:

        Position
        Element
        Comment

            0
            input
            -

            1
            fully connected
            square weight matrix

            2
            sigmoid
            -

            2
            fully connected
            square weight matrix

            3
            sigmoid
            -

            4
            fully connected
            square weight matrix

            5
            sigmoid
            -

            6
            fully connected
            square weight matrix

            7
            sigmoid
            -

            8
            fully connected
            square weight matrix

            9
            sigmoid
            -

            10
            fully connected
            $10$ output features

            11
            output
            -

Create an instance of <tt>FNN_0</tt> as well as of a corresponding <tt>Adam</tt> optimizer with a learning rate of $0.0001$.

Print the resulting model and verify the architecture by inspecting the output.

Train an <tt>FNN_0</tt> network for $5$ epochs, print the training accuracy as well as the loss per epoch and report the final test set loss and accuracy.

```python
class FNN_0 (torch.nn.Module):

    def __init__(self):
        super(FNN_0, self).__init__()
        self.fc1 = torch.nn.Linear(28 * 28, 28 * 28)
        self.ac1 = torch.nn.Sigmoid()
        self.fc2 = torch.nn.Linear(self.fc1.out_features, self.fc1.out_features)
        self.ac2 = torch.nn.Sigmoid()
        self.fc3 = torch.nn.Linear(self.fc2.out_features, self.fc2.out_features)
        self.ac3 = torch.nn.Sigmoid()
        self.fc4 = torch.nn.Linear(self.fc3.out_features, self.fc3.out_features)
        self.ac4 = torch.nn.Sigmoid()
        self.fc5 = torch.nn.Linear(self.fc4.out_features, self.fc4.out_features)
        self.ac5 = torch.nn.Sigmoid()

        self.fc6 = torch.nn.Linear(self.fc5.out_features, 10)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.fc1(x)
        x = self.ac1(x)
        x = self.fc2(x)
        x = self.ac2(x)
        x = self.fc3(x)
        x = self.ac3(x)
        x = self.fc4(x)
        x = self.ac4(x)
        x = self.fc5(x)
        x = self.ac5(x)
        return self.fc6(x)
```

```python
# Set random seed for reproducibility.
np.random.seed(seed=42)
torch.manual_seed(seed=42)

# Create FNN_0 instance and the corresponding optimizer to use.
target_device = torch.device(r'cpu')
fnn_model = FNN_0().to(target_device)
optimizer = torch.optim.Adam(fnn_model.parameters(), lr=1e-4)

# Show the architecture of the FNN_0 model.
print(fnn_model)
```

    FNN_0(
      (fc1): Linear(in_features=784, out_features=784, bias=True)
      (ac1): Sigmoid()
      (fc2): Linear(in_features=784, out_features=784, bias=True)
      (ac2): Sigmoid()
      (fc3): Linear(in_features=784, out_features=784, bias=True)
      (ac3): Sigmoid()
      (fc4): Linear(in_features=784, out_features=784, bias=True)
      (ac4): Sigmoid()
      (fc5): Linear(in_features=784, out_features=784, bias=True)
      (ac5): Sigmoid()
      (fc6): Linear(in_features=784, out_features=10, bias=True)
    )

```python
# Set random seed for reproducibility.
np.random.seed(seed=42)
torch.manual_seed(seed=42)

# Train and evaluate FNN_0 instance on the MNIST training set.
train_and_evaluate(
    model=fnn_model,
    optimizer=optimizer,
    device=target_device,
    num_epochs=5,
    loader_train=loader_mnist_train,
    loader_test=loader_mnist_test)
```

    Epoch: 1 / Loss: 0.0149 / Accuracy: 0.6675
    Epoch: 2 / Loss: 0.0089 / Accuracy: 0.8292
    Epoch: 3 / Loss: 0.0067 / Accuracy: 0.8728
    Epoch: 4 / Loss: 0.0056 / Accuracy: 0.8949
    Epoch: 5 / Loss: 0.0053 / Accuracy: 0.8997

    Final loss: 0.0055 / Final accuracy: 0.8951

    Execute the notebook until here and try to solve the following tasks:

Write down a formula for the corresponding forward pass of <tt>FNN_0</tt>. Use the same notation as presented during the exercise.

    \begin{equation}
        \hat{y} = f\left(h_6(h_5(h_4(h_3(h_2(h_1(\mathbf{x};\mathbf{W}_1);\mathbf{W}_2);\mathbf{W}_3);\mathbf{W}_4);\mathbf{W}_5);\mathbf{W}_6 \right )
    \end{equation}

    Execute the notebook until here and try to solve the following tasks:

Write down a formula for the corresponding backward pass of <tt>FNN_0</tt>. Use the same notation as presented during the exercise.

    \begin{align*}
        \mathbf{W}_6 & \leftarrow \mathbf{W}_6 - \eta \frac{\partial L}{\partial \mathbf{W}_5} \\
        \mathbf{W}_5 & \leftarrow \mathbf{W}_5 - \eta \frac{\partial L}{\partial h_5}\frac{\partial h_5}{\partial \mathbf{W}_5} \\
        \mathbf{W}_4 & \leftarrow \mathbf{W}_4 - \eta \frac{\partial L}{\partial h_5}\frac{\partial h_5}{\partial h_4}\frac{\partial h_4}{\partial \mathbf{W}_4} \\
        \mathbf{W}_3 & \leftarrow \mathbf{W}_3 - \eta \frac{\partial L}{\partial h_5}\frac{\partial h_5}{\partial h_4}\frac{\partial h_4}{\partial h_3}\frac{\partial h_3}{\partial \mathbf{W}_3} \\
        \mathbf{W}_2 & \leftarrow \mathbf{W}_2 - \eta \frac{\partial L}{\partial h_5}\frac{\partial h_5}{\partial h_4}\frac{\partial h_4}{\partial h_3}\frac{\partial h_3}{\partial h_2}\frac{\partial h_2}{\partial \mathbf{W}_2} \\
        \mathbf{W}_1 & \leftarrow \mathbf{W}_1 - \eta \frac{\partial L}{\partial h_5}\frac{\partial h_5}{\partial h_4}\frac{\partial h_4}{\partial h_3}\frac{\partial h_3}{\partial h_2}\frac{\partial h_2}{\partial h_1}\frac{\partial h_1}{\partial \mathbf{W}_1} \\
    \end{align*}

Analyzing Gradients

Actually, the results of <tt>FNN_0</tt> do not look that bad. Is there really a problem with a vanishing gradient? This is exactly the point you're going to figure out in this exercise. As a first step, the gradients of a freshly initialized model needs to be collected and analyzed. Afterwards, in case of a vanishing gradient problem, countermeasures need to get deployed.

    The following code snippet is taken from the accompanying exercise notebook. You do not need to modify it for this assignment.

```python
def collect_gradients(model: torch.nn.Module, device: torch.device,
                      loader: torch.utils.data.DataLoader) -> Sequence[Dict[str, np.array]]:
    """
    Auxiliary function for collecting gradients of a corresponding model.

    :param model: model instance to be used for collecting gradients
    :param device: device to use for gradient collection
    :param loader: data loader supplying the samples used for collecting gradients
    :return: sequence of parameter names and gradients, averaged over all parameter elements
    """
    model_state = model.training
    model.train()
    model.zero_grad()

    # Iterating over the data set and computing the corresponding gradients.
    gradients = {}
    criterion = torch.nn.CrossEntropyLoss()
    for batch_index, (data, target) in enumerate(loader):
        data, target = data.float().to(device), target.long().to(device)
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()

        # Collecting the gradients from the current model.
        for name, parameter in model.named_parameters():
            if parameter.grad is not None:
                gradients.setdefault(name, []).append(parameter.grad.view(-1).abs().mean().item())
        model.zero_grad()

    # Reset model state and return collected gradients.
    model.train(mode=model_state)
    return gradients
```

    Execute the notebook until here and try to solve the following tasks:

Create a fresh instance of <tt>FNN_0</tt> and collect its gradients using the MNIST training set.

Visualize the gradients of each weight parameter accordingly.

Do the gradients vanish?

```python
fnn_model = FNN_0()
gradient = collect_gradients(
    model=fnn_model, device=target_device, loader=loader_mnist_train
)
```

```python
# Set default plotting style.
sns.set()

# Prepare collected gradients for plotting.

gradient_data = pd.DataFrame([
    v for k, v in sorted(gradient.items(), key=lambda _: _[0]) if r'weight' in k
]).transpose().rename(columns=lambda _: f'Layer {_}')
gradient_data = pd.melt(gradient_data, value_vars=gradient_data.columns)
gradient_data[r'Model'] = type(fnn_model).__name__

# Combine all gradients in a single data frame.
gradient_data = gradient_data.rename(
    columns={r'variable': r'Layer', r'value': r'Gradient Magnitude'})

# Define plotting figure and corresponding attributes.
fig, ax = plt.subplots(1, 1, figsize=(18, 7))
ax.set(yscale=r'log')

# Plot pre-processed gradients.
_ = sns.boxplot(x=r'Model', y=r'Gradient Magnitude', hue=r'Layer', data=gradient_data, ax=ax)
```

As shown by the graph, the gradient does vanish.

    Execute the notebook until here and try to solve the following tasks:

Assume a vanishing gradient. Apply the countermeasure presented during the accompanying exercise by implementing a corresponding <tt>FNN_1</tt>.

Create an instance of <tt>FNN_1</tt> as well as of a corresponding <tt>Adam</tt> optimizer with a learning rate of $0.0001$.

Print the resulting model and verify the architecture by inspecting the output.

Train an <tt>FNN_1</tt> network for $5$ epochs, print the training accuracy as well as the loss per epoch and report the final test set loss and accuracy.

```python
class FNN_1 (torch.nn.Module):

    def __init__(self):
        super(FNN_1, self).__init__()
        self.fc1 = torch.nn.Linear(28 * 28, 28 * 28)
        self.ac1 = torch.nn.ReLU()
        self.fc2 = torch.nn.Linear(self.fc1.out_features, self.fc1.out_features)
        self.ac2 = torch.nn.ReLU()
        self.fc3 = torch.nn.Linear(self.fc2.out_features, self.fc2.out_features)
        self.ac3 = torch.nn.ReLU()
        self.fc4 = torch.nn.Linear(self.fc3.out_features, self.fc3.out_features)
        self.ac4 = torch.nn.ReLU()
        self.fc5 = torch.nn.Linear(self.fc4.out_features, self.fc4.out_features)
        self.ac5 = torch.nn.ReLU()

        self.fc6 = torch.nn.Linear(self.fc5.out_features, 10)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.fc1(x)
        x = self.ac1(x)
        x = self.fc2(x)
        x = self.ac2(x)
        x = self.fc3(x)
        x = self.ac3(x)
        x = self.fc4(x)
        x = self.ac4(x)
        x = self.fc5(x)
        x = self.ac5(x)
        return self.fc6(x)
```

```python
# Set random seed for reproducibility.
np.random.seed(seed=42)
torch.manual_seed(seed=42)

# Create FNN_1 instance and the corresponding optimizer to use.
target_device = torch.device(r'cpu')
fnn_model = FNN_1().to(target_device)
optimizer = torch.optim.Adam(fnn_model.parameters(), lr=1e-4)

# Show the architecture of the FNN_1 model.
print(fnn_model)
```

    FNN_1(
      (fc1): Linear(in_features=784, out_features=784, bias=True)
      (ac1): ReLU()
      (fc2): Linear(in_features=784, out_features=784, bias=True)
      (ac2): ReLU()
      (fc3): Linear(in_features=784, out_features=784, bias=True)
      (ac3): ReLU()
      (fc4): Linear(in_features=784, out_features=784, bias=True)
      (ac4): ReLU()
      (fc5): Linear(in_features=784, out_features=784, bias=True)
      (ac5): ReLU()
      (fc6): Linear(in_features=784, out_features=10, bias=True)
    )

```python
# Set random seed for reproducibility.
np.random.seed(seed=42)
torch.manual_seed(seed=42)

# Train and evaluate FNN_1 instance on the MNIST training set.
train_and_evaluate(
    model=fnn_model,
    optimizer=optimizer,
    device=target_device,
    num_epochs=5,
    loader_train=loader_mnist_train,
    loader_test=loader_mnist_test)
```

    Epoch: 1 / Loss: 0.0032 / Accuracy: 0.9373
    Epoch: 2 / Loss: 0.0016 / Accuracy: 0.9679
    Epoch: 3 / Loss: 0.0009 / Accuracy: 0.9820
    Epoch: 4 / Loss: 0.0006 / Accuracy: 0.9877
    Epoch: 5 / Loss: 0.0009 / Accuracy: 0.9813

    Final loss: 0.0020 / Final accuracy: 0.9635

    Execute the notebook until here and try to solve the following tasks:

Create a fresh instance of <tt>FNN_1</tt> and collect its gradients using the MNIST training set.

Visualize the gradients of each weight parameter accordingly (include the gradient visualization of <tt>FNN_0</tt>).

Do the gradients vanish?

```python
fnn_model = FNN_1()
gradient = collect_gradients(
    model=fnn_model, device=target_device, loader=loader_mnist_train
)
```

```python
# Set default plotting style.
sns.set()

# Prepare collected gradients for plotting.

gradient_data = pd.DataFrame([
    v for k, v in sorted(gradient.items(), key=lambda _: _[0]) if r'weight' in k
]).transpose().rename(columns=lambda _: f'Layer {_}')
gradient_data = pd.melt(gradient_data, value_vars=gradient_data.columns)
gradient_data[r'Model'] = type(fnn_model).__name__

# Combine all gradients in a single data frame.
gradient_data = gradient_data.rename(
    columns={r'variable': r'Layer', r'value': r'Gradient Magnitude'})

# Define plotting figure and corresponding attributes.
fig, ax = plt.subplots(1, 1, figsize=(18, 7))
ax.set(yscale=r'log')

# Plot pre-processed gradients.
_ = sns.boxplot(x=r'Model', y=r'Gradient Magnitude', hue=r'Layer', data=gradient_data, ax=ax)
```

As shown by the graph above, diffrence between the gradient magnitude of each layer are much smaller now after applying ReLU activation layers instead of Sigmoid. Also the accuracies after each epoch and the final accuraccy have improved drastecally. Hence, we could say that this technique solved or at least minimized the vanishing gradient problem.

Deriving Derivatives

It is already known from the lecture as well as the exercise that activation functions are the primary culprit of the Vanishing Gradient Problem. Hence, it is important to know how the chosen activation functions activate the input and consequently what the derivative is.

    Execute the notebook until here and try to solve the following tasks (hint: have a look at the official PyTorch documentation):

Implement the <tt>hardsigmoid</tt> activation function as it was done for <tt>relu</tt> in the exercise.

Implement the derivative of the <tt>hardsigmoid</tt> activation function accordingly.

Find $3$ different inputs showing the value range of the <tt>hardsigmoid</tt> activation function.

Plot the <tt>hardsigmoid</tt> activation function indluding its derivative for the input range $[-6; 6]$.

```python
def hardsigmoid(x: float) -> float:
    """
    Compute the hardsigmoid function.

    :param x: the input on which to apply the hardsigmoid function
    :return: the result of the hardsigmoid function applied to its input
    """
    if x <= -3:
        return 0
    elif x >= 3:
        return 1
    else:
        return (x/6)+0.5

def hardsigmoid_d(x: float) -> float:
    """
    Compute the derivate of the hardsigmoid function.

    :param x: the input to the hardsigmoid function for computing its derivative
    :return: the derivative of the hardsigmoid function with respect to its input
    """
    if x <= -3:
        return 0
    elif x >= 3:
        return 0
    else:
        return (1/6)

# Crudly check the value range of the sigmoid function and its derivative.
print(f'hardsigmoid(-10): {hardsigmoid(-10.0):.4f} | hardsigmoid\'(-10): {hardsigmoid_d(-10.0):.4f}')
print(f'hardsigmoid(  0): {hardsigmoid(0.0):.4f} | hardsigmoid\'(  0): {hardsigmoid_d(0.0):.4f}')
print(f'hardsigmoid(+10): {hardsigmoid(10.0):.4f} | hardsigmoid\'(+10): {hardsigmoid_d(10.0):.4f}')
```

    hardsigmoid(-10): 0.0000 | hardsigmoid'(-10): 0.0000
    hardsigmoid(  0): 0.5000 | hardsigmoid'(  0): 0.1667
    hardsigmoid(+10): 1.0000 | hardsigmoid'(+10): 0.0000

```python
def plot_hardsigmoid_with_derivative(x_min: float = -6.0, x_max: float = 6.0, granularity: int = 100) -> None:
    """
    Plot the hardsigmoid function including its derivative.

    :param x_min: minimum value of the input value range
    :param x_max: maximum value of the input value range
    :param granularity: granularity controlling the stepsize of the input value range
    :return: None
    """
    data = np.linspace(x_min, x_max, granularity)

    fig, ax = plt.subplots(figsize=(14, 7))
    ax.spines[r'left'].set_position(r'center')
    ax.spines[r'right'].set_color(None)
    ax.spines[r'top'].set_color(None)

    plt.plot(data, tuple(map(hardsigmoid, data)), color=r'#307EC7', linewidth=3, label=r'hardsigmoid')
    plt.plot(data, tuple(map(hardsigmoid_d, data)), color=r'#accbe8', linewidth=3, label=r"hardsigmoid_d'")
    plt.locator_params(axis=r'y', nbins=6)
    plt.title(r'Hard Sigmoid function', fontsize=20)
    plt.legend(prop={'size': 15})
    plt.show()
```

```python
# Set default plotting style.
sns.set()

# Plot logistic function including its derivative.
plot_hardsigmoid_with_derivative()
```

    Execute the notebook until here and try to solve the following tasks (hint: have a look at the official PyTorch documentation):

Implement the <tt>leaky_relu</tt> activation function as it was done for <tt>relu</tt> in the exercise. Use a negative slope of $0.37$.

Implement the derivative of the <tt>leaky_relu</tt> activation function accordingly.

Find $3$ different inputs showing the value range of the <tt>leaky_relu</tt> activation function.

Plot the <tt>leaky_relu</tt> activation function indluding its derivative for the input range $[-6; 6]$.

```python
def leakyReLU(x: float) -> float:
    """
    Compute the leakyReLU function.

    :param x: the input on which to apply the leakyReLU function
    :return: the result of the leakyReLU function applied to its input
    """
    return x if x >= 0 else 0.01 * x

def leakyReLU_d(x: float) -> float:
    """
    Compute the derivate of the leakyReLU function.

    :param x: the input to the leakyReLU function for computing its derivative
    :return: the derivative of the leakyReLU function with respect to its input
    """
    return 1 if x >= 0 else 0.01

# Crudly check the value range of the sigmoid function and its derivative.
print(f'leakyReLU(-10): {hardsigmoid(-10.0):.4f} | leakyReLU\'(-10): {hardsigmoid_d(-10.0):.4f}')
print(f'leakyReLU(  0): {hardsigmoid(0.0):.4f} | leakyReLU\'(  0): {hardsigmoid_d(0.0):.4f}')
print(f'leakyReLU(+10): {hardsigmoid(10.0):.4f} | leakyReLU\'(+10): {hardsigmoid_d(10.0):.4f}')
```

    leakyReLU(-10): 0.0000 | leakyReLU'(-10): 0.0000
    leakyReLU(  0): 0.5000 | leakyReLU'(  0): 0.1667
    leakyReLU(+10): 1.0000 | leakyReLU'(+10): 0.0000

```python
def plot_leakyReLU_with_derivative(x_min: float = -6.0, x_max: float = 6.0, granularity: int = 100) -> None:
    """
    Plot the leakyReLU function including its derivative.

    :param x_min: minimum value of the input value range
    :param x_max: maximum value of the input value range
    :param granularity: granularity controlling the stepsize of the input value range
    :return: None
    """
    data = np.linspace(x_min, x_max, granularity)

    fig, ax = plt.subplots(figsize=(14, 7))
    ax.spines[r'left'].set_position(r'center')
    ax.spines[r'right'].set_color(None)
    ax.spines[r'top'].set_color(None)

    plt.plot(data, tuple(map(leakyReLU, data)), color=r'#307EC7', linewidth=3, label=r'leakyReLU')
    plt.plot(data, tuple(map(leakyReLU_d, data)), color=r'#accbe8', linewidth=3, label=r"leakyReLU_d'")
    plt.locator_params(axis=r'y', nbins=6)
    plt.title(r'Leaky ReLU function', fontsize=20)
    plt.legend(prop={'size': 15})
    plt.show()
```

```python
# Set default plotting style.
sns.set()

# Plot logistic function including its derivative.
plot_leakyReLU_with_derivative()
```


```python
from IPython.display import HTML

HTML('''
<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code for QISKit exercises."></form>''')
```

<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code for QISKit exercises."></form>

# 9. Quantum cryptography

The advent of quantum computation, which introduces the possibility of using quantum mechanics for information processing, gave rise to the following question: can quantum information be shared more securely than classical information?

In 1982, a very interesting property of quantum states was discovered [1,2]. This is the so-called "no-cloning theorem", which proved how the laws of quantum mechanics prohibit the copy of an unknown quantum state. Therefore, the no-cloning theorem assures us that qubits can hide the quantum information better than classical bits.

This has important implication for example for secure communications, where it allows for the sharing of private keys which cannot be eavesdropped by a third party. We consider the first protocol, the BB84 protocol, which exploits the quantum mechanical properties of qubits for secure exchange of a secret key between two parties.

## 9.1 No-cloning theorem

Let us prove the no-cloning theorem, the fact that an unknown quantum state cannot be copied.
First let us clearly state our problem:

We have a qubit in an unknown quantum state $\lvert \psi \rangle$ and we wish to copy his state on another qubit initilized to the state $\lvert s \rangle$. Therefore, we want to implement the following quantum gate:

\begin{equation}
U\lvert \psi \rangle \lvert s \rangle =\lvert \psi \rangle \lvert \psi \rangle
\end{equation}

Let us take the unknown quantum state to be

\begin{equation}
\lvert \psi \rangle  =\alpha \lvert 0 \rangle + \beta \lvert 1 \rangle
\end{equation}

where the amplitudes $\alpha$ and $\beta$ are unknown.
Therefore we have:

\begin{equation}
U\lvert \psi \rangle \lvert s \rangle =\lvert \psi \rangle \lvert \psi \rangle = (\alpha \lvert 0\rangle +\beta \lvert 1\rangle) (\alpha \lvert 0\rangle +\beta \lvert 1\rangle) = (\alpha^2 \lvert 0\rangle \lvert 0\rangle + \alpha \beta \lvert 0\rangle \lvert 1\rangle +   \beta \alpha \lvert 1 \rangle \lvert 0\rangle + \beta^2 \lvert 1\rangle \lvert 1\rangle)
\tag{1}
\end{equation}

Because of the linearity of operators, we can equivalently write:

\begin{equation}
U\lvert \psi \rangle \lvert s \rangle = U(\alpha \lvert 0\rangle + \beta \lvert 1\rangle )\lvert s\rangle = U(\alpha \lvert 0\rangle \lvert s\rangle + \beta \lvert 1\rangle \lvert s\rangle )=\alpha \lvert 00\rangle +\beta \lvert 11\rangle
\tag{2}
\end{equation}

Comparing Eqs. (1) and (2), one can see that we come to a contraddiction! Thus, the operation $U$ which copies an unknown quantum state of a qubit onto another qubit is not possible.

## 9.2 BB84 protocol

$$\text{1. BB84 protocol overview.}$$

In Ref. [3], the first protocol for the distribution of a secret quantum key between two parties is described.

First, let us assume that Alice and Bob may exchange qubits and classical information. Also, Alice can prepare a qubit in the $\lvert 0 \rangle$, $\lvert 1 \rangle$, $\lvert + \rangle = \frac{1}{\sqrt{2}} \left( \lvert 0 \rangle + \lvert 1 \rangle\right)$ and $\lvert - \rangle = \frac{1}{\sqrt{2}} \left( \lvert 0 \rangle - \lvert 1 \rangle\right)$ state, and Bob can measure in the standard (Z) $\left\{ \lvert 0 \rangle, \lvert 1 \rangle \right\}$ basis and in the Hadamard (H) $\left\{ \lvert + \rangle, \lvert - \rangle \right\} $ basis. Note that the two bases are non-orthogonal with respect to each other. Measuring in the $\left\{ \lvert + \rangle, \lvert - \rangle \right\} $ basis means that before the standard measurement in the $\left\{ \lvert 0 \rangle, \lvert 1 \rangle \right\} $ basis, Bob applies the Hadamard gate to the qubit. Thus

\begin{equation}
\lvert + \rangle =\frac{1}{\sqrt{2}}(\lvert 0\rangle +\lvert 1\rangle )
\end{equation}

gives $\lvert 0 \rangle$ when measured in the Hadamard basis, and

\begin{equation}
\lvert - \rangle =\frac{1}{\sqrt{2}}(\lvert 0\rangle -\lvert 1\rangle )
\end{equation}

gives $\lvert 1 \rangle$ when measured in the Hadamard basis.

The protocol then works in the following way. Alice picks the bit that she wants to transmit to Bob, either $0$ or $1$. She then prepares a qubit in the corresponding state $\lvert 0 \rangle$ or $\lvert 1 \rangle$, respectively. After that, she randomly decides whether or not to transform her qubit from the standard (Z) basis to the Hadamard (H) basis by applying or not the Hadamard gate her qubit, thus preparing the state $\lvert + \rangle$ or $\lvert - \rangle$.

Then Alice sends her first qubit to Bob. Bob receives Alice's qubit, selects one of the measurement bases at random and measures it. After that, Alice and Bob tell each other which basis they used through a classical communication channel.

In general, for every qubit Alice sends to Bob there are four possible scenarios:

Both Alice and Bob used the Hadamard basis.

They both used the standard basis.

Alice transformed to the Hadamard basis, and Bob measured in the standard basis.

Alice used the standard basis, and Bob the Hadamard basis.

When Alice and Bob agree on the same basis, they keep the transferred bit. When they disagree, they discard it. Thus, it is possible for Alice and Bob to securely communicate an $n$ bit private key using $2n$ qubits.

#### Example

For example, let us consider the case where Alice wants to send the bit $0$. She prepares her qubit in the $\lvert 0 \rangle$ state and then randomly selects whether or not she applies the Hadamard gate to it. Let's say she does apply the Hadamard gate to her qubit, obtaining the $\lvert + \rangle$ state.

Then, consider, the cased where Bob measures the qubit in the standard basis. After Bob's measurement, Alice and Bob communicate through the classical channel. Alice tells Bob that she  applied the Hadamard gate to her qubit and Bob tells Alice that he measured it in the standard basis. So, they abandon the first bit.

$$\text{2. Example of one application of the BB84 protocol. In this case, Alice and Bob will discard this bit.}$$

Next, Alice picks a second bit, $1$, encodes it into a qubit and selects at random whether to apply or not the Hadamard gate. Let us now assume that she does not apply the Hadamard gate. Thus, the qubit is in the state $\lvert 1\rangle $. Alice then sends her qubit to Bob. Bob selects at random one of his two measurement bases. Let us consider in this
case that he measures in the standard basis. As the qubit is in the state $\lvert 1\rangle $ the outcome of the measurement will be $1$. Thus, Bob chooses value $1$ for his second classical bit, the same as Alice did. Finally, Alice tells Bob that she did not apply the Hadamard gate, and Bob tells Alice that he measured in the standard basis. So, both Alice and Bob will use the bit with the value $1$ as the first bit in their secret key.

$$\text{3. Example of another application of the BB84 protocol.} \\  \text{In this case, Alice and Bob successfully communicate the value of a bit.}$$

###  QISKit: BB84 protocol

####  1) Show the communication of one bit

```python
from initialize import *
import random

#initialize quantum program
my_alg = initialize(circuit_name = 'bb84', qubit_number=1, bit_number=1, backend = 'local_qasm_simulator', shots = 1)

#add gates to the circuit

# Alice encodes the bit 1 into a qubit
my_alg.q_circuit.x(my_alg.q_reg[0])

# Alice randomly applies the Hadamard gate to go to the Hadamard basis
a = random.randint(0,1)
if a==1:
    my_alg.q_circuit.h(my_alg.q_reg[0])

# Bob randomly applies the Hadamard gate to go to the Hadamard basis
b = random.randint(0,1)
if b==1:
    my_alg.q_circuit.h(my_alg.q_reg[0])

my_alg.q_circuit.measure(my_alg.q_reg[0], my_alg.c_reg[0]) # measures first qubit

# print list of gates in the circuit
print('List of gates:')
for circuit in my_alg.q_circuit:
    print(circuit.name)

#Execute the quantum algorithm
result = my_alg.Q_program.execute(my_alg.circ_name, backend=my_alg.backend, shots= my_alg.shots)

#Show the results obtained from the quantum algorithm
counts = result.get_counts(my_alg.circ_name)

print('\nThe measured outcomes of the circuits are:',counts)

if a == b:
    print('Alice and Bob agree on the basis, thus they keep the bit')
else:
    print("Alice and Bob don't agree the same basis, thus they discard the bit")

```

    List of gates:
    x
    h
    measure

    The measured outcomes of the circuits are: {'0': 1}
    Alice and Bob don't agree the same basis, thus they discard the bit

    /anaconda/lib/python3.6/site-packages/qiskit/backends/local/qasm_simulator_cpp.py:89: DeprecationWarning: The behavior of getting statevector from simulators by setting shots=1 is deprecated and will be removed. Use the local_statevector_simulator instead, or place explicit snapshot instructions.
      DeprecationWarning)

## Problems

Alice wants to send Bob the following private key

\begin{equation}
101011
\end{equation}

She encodes those bits into the correspondig states of qubits and performs the gates H-H-I-I-I-H on each qubit. Bob measures the qubits in the following bases: Z-H-H-Z-H-H

Find the possible outcomes of Bob's measurements

Find the bits of the private key accepted by Alice and Bob

Imagine that a third party, Eve, intercepts Alice's qubits. She measures the intercepted qubit by randomly selecting either the Hadamard or the standard basis and then forwards the qubits to Bob.

Is it possible for Eve to find out the bit that Alice is sending to Bob without being discovered?

What is the probability that Eve successfully finds out the value of a bit?

Write a QISKit program for the transmission of a 1024 bits provate key between Alice and Bob

## References

[1] D. Dieks, Physics Letters A, 92, 271 (1982).

[2] W. K. Wootters and W. H. Zurek, Nature, 299.802 (1982).

[3] C. H. Bennett and G. Brassard, In Proceedings of IEEE International Conference on Computers, Systems and Signal Processing, volume 175, page 8. New York, 1984.