```
import torch, os, ipywidgets, json
import torch.nn as nn
import numpy as np
import pandas as pd
import seaborn as sns
import torch.nn.functional as F
import torchvision as tv
import matplotlib.pyplot as plt
from google.colab import drive
from google.colab import widgets as clwidgets
from tqdm import tqdm_notebook as tqdm
from torchvision.datasets import MNIST
from torch.utils.data import DataLoader
PI = torch.tensor(np.pi, dtype=torch.float32)
sns.set_style('whitegrid')
%matplotlib inline
```

```
save_path = '/content/gdrive/My Drive/Tutorials/InfoGAN/MNIST/'
drive.mount('/content/gdrive', force_remount=True)
os.makedirs(os.path.join(save_path, 'ckpt'), exist_ok=True)
```

# A quick tour on Information Maximising Generative Adversarial Networks (InfoGAN)
*Yen Yu* $\diamond$ *yen.yu@araya.org* $\diamond$ *yen.yu.10@ucl.ac.uk*

*17/Jun/2019*

## Background
* This Notebook traces my steps from my only knowing about GAN on a conceptual level to actually trying to learn the theory behind by implementing it.
* I intend this Notebook to be a tutorial accessible to those who wants to learn more about geneartive modelling (there is a plan to cover other GANs and VAE+Normalising Flows).

## Goal
* The goal of this Notebook is to to provide theoretical details of GAN just enough to motivate an implementation.
* The main focus will be on the implementation of InfoGAN
* To successfully train an InfoGAN on the MNIST dataset.

## References
[1] The origin GAN paper by Goodfellow

[2] The InfoGAN paper by Xi Chen

[3] Ferenc Huszar's [blog post](https://www.inference.vc/infogan-variational-bound-on-mutual-information-twice/)—very insightful

[4] [Useful GAN hacks (multiple authors)](https://github.com/soumith/ganhacks#authors)

## Classic GAN
### Theory
#### Layperson example

You are visiting a friend who is working in the packaging department of a factory. His/her job is to load the products onto a conveyer belt to initiate automatic packaging. You noticed, from the control room, that the conveyer belt has a layout resembling the shape of the letter Y. Products are loaded onto the branches on the top end and get merged into a single line at the junction in the middle. The packaging mechnism is located at the bottom end. An electronic gate at the junction controls which branch gets to pass. Whenever the indicator light on the gate shines red the left branch feeds into the bottom branch. And when a green light is shone, the opposite is the case.

You know the factory has a long-term partnership with a particular supplier who provides the standard products for this line. When the supply is short, the factory seeks additional supply from another supplier but thier quality can be a hit and miss.

Today, both branches are operating but you have no idea whether there has been a supply shortage. If there is a shortage, one of the branches must be carrying products of "somewhat questionable quality".

From the control room terminal, you see the gate camera view, along with its indicator light. Out of curiosity, you decide to challenge yourself to answer whether there *is* a shortage by finding out whether different lights signal a discernable impression on the product quality.

This time, you are positive because the non-standard products have a grey-ish tint.

Hopefully, on your next visit, knowing that there is still a shortage going on, you will find yourself having a hard time arriving at a definite conclusion. This can almost mean the additional supplier has boosted their quality to be on a par with the standard.

#### Remarks

* A Generative Adversarial Network (GAN) typically consists of a generator and a discriminator, both as trainable neural networks. The generator tries to generate samples to fool the discriminator, making it believe the generated samples are as good as the real ones. And the discriminator tries to unfool itself.
* The standard supplier provides the *real* data: $x_{real}$.
* The second supplier, which is the generator, provides the *fake* data: $x_{fake}$
* You, the discriminator, serve to differentiate the real from the fake. But you also help the fake fake better.
* The role of the indicator light $d$ will become clear in the following part.
* The layperson example lends itself to an information-theoretical interpretation as described below.

#### An information-theoretic perspective

Let us consider two random variables $x_{real}$ and $x_{fake}$, one representing data coming from a *real* distribution and another trying to mimic it. And $d \sim \mathrm{Bern}(0.5)$ being a mixing factor allows the following interaction:

$$
\begin{align}
x &= \left\{
    \begin{array}{l}
    x_{real},\; \hbox{ if } d=1 \\
    x_{fake},\; \hbox{ if } d = 0
    \end{array}
    \right.
\end{align}
$$

The task for a classic GAN is to make $x_{fake}$ as real as possible such that knowing $d$ provides minimal, if not none, insights into the actual identity of $x$. In the language of information theory, this is equivalent to saying

$$\min I(x : d)$$

The learning rule of the classic GAN can be derived from here by considering:

$$
\begin{align}
I(x : d) &= H(d) - H(d | x)  \\
    &= H(d) + \mathbb E_x \mathbb E_{d|x} \log p(d|x)  \\
    &= H(d) + \mathbb E_{d,x} \frac {\log p(d|x)} {\log q(d|x)} \log q(d|x)  \\
    &\ge H(d) + \mathbb E_{d,x} \log q(d|x)
\end{align}
$$

Here, an auxiliary density $q_{d|x}$ is introduced to create a lower bound on the mutual information. This bound is tight when $p(d|x) = q(d|x)$. Then the following holds:

$$
\begin{align}
I(x:d) &= H(d) + \max_{q_{d|x}} \mathbb E_{d,x} \log q(d|x)  \\
    &\ge H(d) + \max_\varphi \mathbb E_{d,x} \log q(d|x; \varphi)
\end{align}
$$

where the last line turns $q_{d|x}$ into a parametric family by assumption.

Then, one can expand the expectation to rewrite the expression since we knew $q$ is a coin-flip:

$$
\begin{align}
I(x:d) &\ge H(d) + \max_\varphi \left(
  \mathbb E_{x_{real}} \log q(1 | x_{real};\varphi) +
  \mathbb E_{x_{fake}} \log q(0 | x_{fake}; \varphi)
  \right)
\end{align}
$$

### Implementation
Following the derivation immediately above, it becomes clear that we can treat the approximate density $q(d=\{0, 1\} | x; \varphi)$ as a deep neural network that works as a discriminator ($\mathcal D(x; \varphi)$) . Specifically, we want this D network to take as input, say, an image tensor and output a scalar between 0 and 1. This D network should be able to tell $x$ apart by learning to assign 1 to $x_{real}$ and 0 to $x_{fake}$:
$$
\begin{align}
&= H(d) + \max_\varphi \left(
  \mathbb E_{x_{real}} \log \mathcal D(x_{real};\varphi) +
  \mathbb E_{x_{fake}} \log [1 - \mathcal D(x_{fake}; \varphi)]
  \right)
\end{align}
$$
On the other hand, $x_{fake}$ is something we have to create from nothing (well, from noise, to be exact). And we can assign this task to another neural network that we'd like to call the generator $\mathcal G(z; \vartheta)$, where $z$ is the noise vector. By replaceing $x_{fake}$ with $\mathcal D(z; \vartheta)$ we sidestep the need to actually evaluate the expectation $\mathbb E_{x_{fake}}$. Instead, we are now dealing with a much simpler $\mathbb E_z$, whcih can be Monte Carlo approximated:

$$
\begin{align}
&=H(d) + \max_\varphi \left(
  \mathbb E_{x_{real}} \log \mathcal D(x_{real};\varphi) +
  \mathbb E_{z} \log [1 - \mathcal D(\mathcal G(z; \vartheta); \varphi)]
  \right)
\end{align}
$$

where $\vartheta$ parametrises the generator network $\mathcal G$ and $z \sim N(0,1)$ is some Gaussian noise conventionally used by GAN.

The G network has to learn to fool the D network as the D network tries to tell the real and fake apart. This brings us back to our initial objective of minimising the mutual information. We can now write down the classic GAN objective:

$$
\begin{align}
\min_\vartheta \max_\varphi \left(
  \mathbb E_{x_{real}} \log \mathcal D(x_{real};\varphi) +
  \mathbb E_{z} \log [1 - \mathcal D(\mathcal G(z; \vartheta); \varphi)]
  \right)
\end{align}
$$

#### Task
To implement a classic GAN, we need to program the following:
* A discriminator network $\mathcal D$ that maps an input to a real number between 0 and 1 (as representing probability)
* A generator $\mathcal G$ that takes some noise vector of arbitrary dimension
* Write down the objective function

```
%%script false
# This is only a demonstration cell

# Taks 1: Program a discriminator network
class Discriminator(nn.Module):
    """A Discriminator network represents a function
    that maps an input to a scalar between (0, 1)."""
    def __init__(self):
        super(Discriminator, self).__init__()

        # The following contains the main component of the
        # desired function. It can take any valid architecture.
        # But for the sake of demonstration, I will only include
        # minimal numbers of network layers, assuming the inputs
        # are mini-batches of MNIST images which are monochromatic,
        # 28x28 images.
        self.layers = nn.Sequential(
            # NB. setting bias=False only because BN is used. The beta parameter
            # in a BN works just like a bias.
            nn.Conv2d(1, 64, (3, 3), stride=(2, 2), padding=1, bias=False),
            nn.BatchNorm2d(64),
            nn.LeakyReLU(0.1),
            nn.Conv2d(64, 128, (3, 3), stride=(2, 2), padding=1, bias=False),
            nn.BatchNorm2d(128),
            nn.LeakyReLU(0.1),
            nn.Conv2d(128, 1024, (7, 7), stride=(1, 1), padding=0, bias=False),
            nn.BatchNorm2d(1024)
            nn.LeakyReLU(0.1),
            nn.Conv2d(1024, 1, (1, 1), stride=(1, 1), padding=0, bias=True),
            nn.Sigmoid()
        )

    def forward(self, x):
        return self.layers(x)

# Task 2: Program a generator network
class Generator(nn.Module):
    """A Generator is a function that maps a latent code (usually a noise vector) of
    arbitrary dimension (here 74 is used) to a tensor that mimics real dataset."""
    def __init__(self):
        super(Generator, self).__init__()

        self.layers = nn.Sequential(
            nn.Conv2d(74, 1024, (1, 1), (1, 1), bias=False, padding=0),
            nn.LeakyReLU(0.1),
            nn.BatchNorm2d(1024),
            nn.ConvTranspose2d(1024, 128, (7, 7), (1, 1), bias=False, padding=0),
            nn.BatchNorm2d(128),
            nn.LeakyReLU(0.1),
            nn.ConvTranspose2d(128, 64, (4, 4), (2, 2), bias=False, padding=1),
            nn.BatchNorm2d(64),
            nn.ConvTranspose2d(64, 1, (4, 4), (2, 2), bias=True, padding=1),
            nn.Sigmoid())

        self.apply(initialise_weights)

    def forward(self, x):
        return self.layers(x)

# Task 3: Write down the GAN loss
class ClassicGANCriterion:
    def __call__(self, prob, label):
        prob = prob.view(-1)
        label = torch.zeros_like(prob).fill_(label)
        return F.binary_cross_entropy(prob, label)
```

## InfoGAN

The classic GAN generator places no restrictions on how the noise vector should be used to generate data. It can inadvertently come to ignore a big part of that noise vector by essentially map a range of value to the same output. As a result, the generator will fail to capture the true data distribution and quite often end up with one with very low entropy. Naturally, a generator behaviing like this will do a very poor job in fooling the discriminator. This failure mode is marked by high generator loss and 0 discriminator loss, otherwise known as "mode collapse" of GAN.

The authors of InfoGAN attempted to prevent mode collapse by introducing one regulariser as restriction. This restriction is such that any generator output, give a noise vector, has to retain maximum information about that vector. This is the same as saying:

$$
\max I(z, x_{fake})
$$

InfoGAN does this by allowing a partition in the noise vector. In classic GAN, a noise vector can be as simple as $z \sim N(\boldsymbol{0}, Id)$, where $\boldsymbol{0}$ is a zero vector and $Id$ is an identity matrix of arbitrary dimension. InfoGAN chooses to have $z = \{ c, z'\}$, where $c$ is a noise vector from a known distribution whose sufficient statistics must be recovered after the generation process (in the form of posterior estimates, but more on that later!). Whereas, $z'$ is the same old noise vector ($z$) used by classic GAN, meaning there will be no restrictions placed on $z'$.

We can then rewrite the mutual information above to only include $c$, i.e., $I(c, x_{fake})$. For InforGAN, the following is to be minimised

$$
I(d, x) - \lambda I(c, x_{fake})
$$
where $\lambda$ serves to adjust scale (entropy and mutual information are not scale-invariant).

### Implementation
To see how this can be implemented using neural network, we trace our steps in the previous section and attempt a similar work on the newly introduced regulariser. We have

$$
\begin{align}
I(c, x_{fake}) &= H(c) - H(c | x_{fake})  \\
&= H(c) + \mathbb E_{z\sim p(z)} \mathbb E_{c \sim p(c)} \log p(c | x_{fake}) \\
&= H(c) + \mathbb E_{z, c} \log p(c | \mathcal G(c, z))
\end{align}
$$

If we create a variational lower bound just like we did earlier we would have a difficult time evaluating the log-probability in the second term. However, by Law of Total Expectation, we can now see this expression in a different light:

$$
\begin{align}
\mathbb E_{z, c} \log p(c |\mathcal G(c, z)) &=
  \mathbb E_{z, c} \mathbb E_{c' \sim p(c'|x_{fake})} \log p(c' | \mathcal G (c, z)) \\
&= \mathbb E_{z, c} \mathbb E_{c' \sim p(c'|x_{fake})} \log \frac {p(c'|x_{fake})}{q(c'|x_{fake})} q(c'|x_{fake})  \\
&\ge \mathbb E_{z, c} \mathbb E_{c' \sim p(c'|x)} \log q(c'|x_{fake})
\end{align}
$$

This means we will use $c$ and $z$ to get our $x_{fake}$ through the generator $\mathcal G$. Then program a neural network as $q(c'|x_{fake})$—assuming a known family of probability distributions—that takes as input $x_{fake}$ and outputs the sufficient statistics of the approximate density $q_{c'|x_{fake}}$ (which is called a Recognition Model in the paper). There! We can work out the log-probability with much less effort.

#### Generator and utility objects
First, let us take care of the usual generator and we will get to the discriminator and recognition model after that. We have done this bit in the classic GAN section and there is nothing new in particular, only we have introduced a different normalisation module (which I might want to compare aginst BatchNorm later).

```
def initialise_weights(module):
    if isinstance(module, nn.Conv2d) or isinstance(module, nn.ConvTranspose2d):
        nn.init.xavier_normal_(module.weight.data)
        if isinstance(module.bias, torch.Tensor):
            module.bias.data.fill_(0.)

class GroupNorm2d(nn.Module):
    """Group Normalisation Layer."""
    def __init__(self, channels, groups, eps=1e-5):
        super(GroupNorm2d, self).__init__()

        self.gamma = nn.Parameter(torch.ones(1, channels, 1, 1))
        self.beta = nn.Parameter(torch.zeros(1, channels, 1, 1))
        self.num_groups = groups
        self.eps = eps

    def forward(self, x):
        N, C, H, W = x.size()
        G = self.num_groups

        x = x.view(N, G, -1)
        mean = x.mean(dim=2, keepdim=True)
        var = (x - mean).pow(2).sum(2, keepdim=True) / x.size(2)

        x = (x - mean) / (var + self.eps).sqrt()
        x = x.view(N, C, H, W)

        return x * self.gamma + self.beta

class Generator(nn.Module):
    """The Generator Network."""
    def __init__(self):
        super(Generator, self).__init__()
        # Network architecture follows the original InfoGAN paper (Chen et al.)
        self.layer = nn.Sequential(
            nn.Conv2d(74, 1024, (1, 1), (1, 1), bias=False, padding=0),
            nn.LeakyReLU(0.1),
            GroupNorm2d(1024, 32),
            nn.ConvTranspose2d(1024, 128, (7, 7), (1, 1), bias=False, padding=0),
            GroupNorm2d(128, 32),
            nn.LeakyReLU(0.1),
            nn.ConvTranspose2d(128, 64, (4, 4), (2, 2), bias=False, padding=1),
            GroupNorm2d(64, 32),
            nn.ConvTranspose2d(64, 1, (4, 4), (2, 2), bias=True, padding=1),
            nn.Tanh())

        self.apply(initialise_weights)

    def sample_latent(self, batch_size, device):
        # all dimension information is mentioned in the original InfoGAN paper
        # incompressible noise (z)
        z = torch.randn(batch_size, 62, device=device)

        # categorical latent code (cd)
        # c ~ Categ(K=10, p=0.1)
        index = torch.randint(0, 10, (batch_size,), device=device)
        cd = torch.zeros(batch_size, 10, device=device)
        cd[torch.arange(batch_size), index] = 1

        # continuous latent codes (cc)
        # c ~ Unif(-1., 1.)
        cc = torch.rand((batch_size, 2), device=device) * 2. - 1.

        noise = torch.cat([z, cd, cc], dim=1).view(-1, 74, 1, 1)
        return noise, index

    def forward(self, x):
        return self.layer(x)
```

#### Discriminator and Recognition model
Here, we have split the discriminator network into two parts (the "SharedNetwork" and "DiscriminatorEnd") and have one part shared with the Recognition model. This can be interpreted as using the Recognition model (which works as inferring the sufficient statistics of the seed noise vector) as an auxiliary task for the discriminator network. Auxiliary task tends to help obtain good representation. The same technique is also seen in reinforcement learning. We also attached another network to the discriminator to classify images from the true dataset which seems to help stablise training.

```
class SharedNetwork(nn.Module):
    """Network shared between the Discriminator and Recognition model."""
    def __init__(self):
        super(SharedNetwork, self).__init__()

        # Average pooling is used here in place of the usual max pooling or strides
        # Empirically, this avoids sparse gradients and tends to prevent mode collapse.
        self.net_base = nn.Sequential(
            nn.Conv2d(1, 64, (4, 4), (1, 1), bias=False, padding=0),
            nn.LeakyReLU(0.1),
            nn.AvgPool2d((2, 2)),
            GroupNorm2d(64, 32),
            nn.Conv2d(64, 128, (4, 4), (1, 1), bias=False, padding=0),
            nn.LeakyReLU(0.1),
            nn.AvgPool2d((2, 2)),
            GroupNorm2d(128, 32),
            nn.Conv2d(128, 1024, (4, 4), (1, 1), bias=False, padding=0),
            nn.LeakyReLU(0.1),
            GroupNorm2d(1024, 32))

        self.apply(initialise_weights)

    def forward(self, x):
        x = self.net_base(x)
        return x

class DiscriminatorEnd(nn.Module):
    """The Discriminator."""
    def __init__(self):

        super(DiscriminatorEnd, self).__init__()

        self.net_d = nn.Sequential(
            nn.Conv2d(1024, 1, (1, 1), (1, 1), bias=True, padding=0),
            nn.Sigmoid())

        self.apply(initialise_weights)

    def forward(self, x):
        return self.net_d(x).squeeze(3).squeeze(2)

class RecognitionEnd(nn.Module):
    """The Recognition model."""
    def __init__(self):
        super(RecognitionEnd, self).__init__()

        self.net_r = nn.Sequential(
            nn.Conv2d(1024, 128, (1, 1), (1, 1), bias=True, padding=0),
            nn.LeakyReLU(0.1),
            GroupNorm2d(128, 32))

        self.cat_r = nn.Conv2d(128, 10, (1, 1), (1, 1), bias=True, padding=0)
        self.gau_r = nn.Conv2d(128,  4, (1, 1), (1, 1), bias=True, padding=0)

        self.apply(initialise_weights)

    def forward(self, x):
        r = self.net_r(x)
        cat = self.cat_r(r).squeeze(3).squeeze(2)
        gau = self.gau_r(r).squeeze(3).squeeze(2)
        mean, logv = torch.chunk(gau, 2, dim=1)
        return cat, mean, logv

class ClassifierEnd(nn.Module):
    """Auxiliary Classification for the D Network."""
    # Introducing an auxiliary task whenever possible tends to help stabalise training.
    def __init__(self):
        super(ClassifierEnd, self).__init__()
        self.net_c = nn.Sequential(
            nn.Conv2d(1024, 10, (1, 1), (1, 1), bias=True, padding=0))

    def forward(self, x):
        logits = self.net_c(x).view(-1, 10)
        return F.softmax(logits, dim=1)
```

## Training InfoGAN to generate MNIST digits
### Loss functions

```
class NaNError(Exception):
    pass

class CriterionDiscriminator:
    """Binary Cross Entropy Loss."""
    def __call__(self, prob, label):
        prob = prob.view(-1)
        label = torch.zeros_like(prob).fill_(label)
        return F.binary_cross_entropy(prob, label)

class CriterionRecognitionCategorial:
    """Categorial Cross Entropy Loss"""
    def __call__(self, logits, index):
        return F.cross_entropy(logits, index)

class CriterionRecognitionNLL:
    """Negative log-likelihood for factorised Normal distribution."""
    def __call__(self, sample, mean, logv):
        sample = sample.squeeze()
        scale = (0.5 * logv).exp()
        normal = torch.distributions.Normal(mean, scale)
        # variance = logv.exp()
        # nll = (0.5 * torch.log(2 * PI).add(logv) +
        #        (sample - mean).pow(2).div(2 * variance + 1e-5))

        return - normal.log_prob(sample).sum(dim=1).mean()
```

### Setting up training
- Download the MNIST dataset and prepare data sampler
- Create new instances of network
- Define optimisers (for G and D networks, respectively)

```
%%capture
# Download the MNIST dataset
transforms = tv.transforms.Compose([
    tv.transforms.ToTensor(),
    tv.transforms.Normalize((0.5,), (0.5,))
])
mnist_data = {
    'train': MNIST('.', train=True,  transform=transforms, download=True),
    'test':  MNIST('.', train=False, transform=transforms, download=True)}
```

```
# Training settings and save names
checkpoint = 0
settings = {
    'batch_size': 128,
    'num_epochs': 300,
    'lr_g': 2e-4,
    'lr_d': 2e-4,
    'beta1': 0.5,
    'beta2': 0.999,
    'save_every': 50,
    'trained_d': 'trained_d.pth',
    'trained_g': 'trained_g.pth',
    'trained_r': 'trained_r.pth',
    'trained_c': 'trained_c.pth',
    'trained_shared': 'trained_shared.pth',
    'training_log': 'training_log.csv'}

# Pytorch DataLoader
data_loader = {
    'train': DataLoader(mnist_data['train'], batch_size=settings['batch_size'], shuffle=True, drop_last=True),
    'test':  DataLoader(mnist_data['test'],  batch_size=settings['batch_size'], shuffle=True, drop_last=True)}

# --- Create networks:
# One can potentially introduce an auxiliary classifier network as part of the D network.
# An auxiliary task, empirically, tends to help the stability of GAN training.
nets = {'G': Generator().cuda(), 'D': DiscriminatorEnd().cuda(), 'R': RecognitionEnd().cuda(),
        'Shared': SharedNetwork().cuda(), 'C': ClassifierEnd().cuda()}

# --- Optimisers:
# ADAM was chosen for G network and its hyperparameters (betas) follow that of DC-GAN
# SGD may be chosen for D network (empirically, this prevents discriminator mode collapse, i.e., D loss becomes zero)
optimiser = {
    'G': torch.optim.Adam([{'params': nets['G'].parameters()},
                           {'params': nets['R'].parameters()}],
                          lr=settings['lr_g'],
                          betas=[settings['beta1'], settings['beta2']]),
    'D': torch.optim.SGD([{'params': nets['D'].parameters()},
                          {'params': nets['C'].parameters()},
                          {'params': nets['Shared'].parameters()}],
                          lr=settings['lr_d'])}

# --- Loss functions:
# For D network, use binary cross entropy
# For Recognition (R) network, use categorical cross entropy for discrete latent codes;
# use negative Gaussian log-likelihood for continuous latent codes.
crit_d = CriterionDiscriminator()
crit_r_ce = CriterionRecognitionCategorial()
crit_r_nll = CriterionRecognitionNLL()

# Save the settings for later reference.
with open(os.path.join(save_path, 'notebook_settings.json'), 'w') as file:
    json.dump(settings, file, sort_keys=True)
```

```
%%script false  # Comment out this line to enable this cell

# Recover from previous checkpoint
checkpoint = 250
ckpt_path = os.path.join(save_path, 'ckpt')
nets['D'].load_state_dict(torch.load(os.path.join(ckpt_path, 'ckpt_d_{:03d}.pth'.format(checkpoint-1))))
nets['G'].load_state_dict(torch.load(os.path.join(ckpt_path, 'ckpt_g_{:03d}.pth'.format(checkpoint-1))))
nets['R'].load_state_dict(torch.load(os.path.join(ckpt_path, 'ckpt_r_{:03d}.pth'.format(checkpoint-1))))
nets['C'].load_state_dict(torch.load(os.path.join(ckpt_path, 'ckpt_c_{:03d}.pth'.format(checkpoint-1))))
nets['Shared'].load_state_dict(torch.load(os.path.join(ckpt_path, 'ckpt_shared_{:03d}.pth'.format(checkpoint-1))))
training_log = pd.read_csv(os.path.join(ckpt_path, 'ckpt_training_log_{:03d}.csv').format(checkpoint-1))
print('Recovered checkpoint: {}'.format(checkpoint))
```

### Main training script

```
%%script false

training_log = pd.DataFrame(columns=['epoch', 'phase', 'd_loss', 'g_loss', 'r_loss_cat', 'r_loss_gaus', 'c_loss'])
pgrid = clwidgets.Grid(1, 6)

# Main training script
for epoch in tqdm(range(checkpoint, settings['num_epochs'])):
    for phase in ['train']:
        for img_real, label in data_loader[phase]:
            i = len(training_log)
            img_real = img_real.to('cuda')
            label = label.to('cuda')
            noise, cat_index = nets['G'].sample_latent(settings['batch_size'], 'cuda')

            # Improve the Discriminator
            with torch.set_grad_enabled(phase == 'train'):
                optimiser['D'].zero_grad()
                shared_real = nets['Shared'](img_real)
                pr_real = nets['D'](shared_real)
                pr_auxc = nets['C'](shared_real)
                img_fake = nets['G'](noise)
                pr_fake = nets['D'](nets['Shared'](img_fake.detach()))

                loss_d = crit_d(pr_real, 1) + crit_d(pr_fake, 0)
                loss_c = F.cross_entropy(pr_auxc, label)

                if phase == 'train':
                    (loss_d + loss_c).backward()
                    optimiser['D'].step()

            # Improve the Generator and Recognition model
            if phase == 'train':
                optimiser['G'].zero_grad()
                shared_fake = nets['Shared'](img_fake)
                pr_fake = nets['D'](shared_fake)
                r_logit, r_mean, r_logv = nets['R'](shared_fake)

                loss_g = crit_d(pr_fake, 1)
                loss_cat = crit_r_ce(r_logit, cat_index)
                loss_gaus = crit_r_nll(noise[:, -2:], r_mean, r_logv)

                (loss_g + loss_cat + 0.1 * loss_gaus).backward()
                optimiser['G'].step()

                training_log.loc[i] = [epoch, phase, loss_d.item(), loss_g.item(), loss_cat.item(), loss_gaus.item(), loss_c.item()]

                # some sanity check for training
                if training_log.loc[i].isnull().any():
                    raise NaNError
            else:
                # if not in training phase, treat g-losses as missing
                training_log.loc[i] = [epoch, phase, loss_d.item(), np.nan, np.nan, np.nan, loss_c.item()]

    # keep a training checkpoint
    if (epoch + 1) % settings['save_every'] == 0:
        torch.save(nets['D'].state_dict(), os.path.join(save_path, 'ckpt', 'ckpt_d_{:03d}.pth'.format(epoch)))
        torch.save(nets['G'].state_dict(), os.path.join(save_path, 'ckpt', 'ckpt_g_{:03d}.pth'.format(epoch)))
        torch.save(nets['R'].state_dict(), os.path.join(save_path, 'ckpt', 'ckpt_r_{:03d}.pth'.format(epoch)))
        torch.save(nets['C'].state_dict(), os.path.join(save_path, 'ckpt', 'ckpt_c_{:03d}.pth'.format(epoch)))
        torch.save(nets['Shared'].state_dict(), os.path.join(save_path, 'ckpt', 'ckpt_shared_{:03d}.pth'.format(epoch)))
        training_log.to_csv(os.path.join(save_path, 'ckpt', 'ckpt_training_log_{:03d}.csv'.format(epoch)))

  # plot progress at the end of each epoch
    with pgrid.output_to(0, 0):
        pgrid.clear_cell()
        plt.figure(figsize=(3, 3))
        plt.imshow(img_real[0, 0].detach().cpu().numpy())
        plt.gca().set_title('Real Example ({})'.format(epoch))
        plt.gca().grid(False)
        plt.gca().set_axis_off()
    with pgrid.output_to(0, 1):
        pgrid.clear_cell()
        plt.figure(figsize=(3, 3))
        plt.imshow(img_fake[0, 0].detach().cpu().numpy())
        plt.gca().set_title('Fake Example ({})'.format(epoch))
        plt.gca().grid(False)
        plt.gca().set_axis_off()
    with pgrid.output_to(0, 2):
        pgrid.clear_cell()
        plt.figure(figsize=(3, 3))
        sns.lineplot(x='epoch', y='d_loss', data=training_log.tail(20000))
    with pgrid.output_to(0, 3):
        pgrid.clear_cell()
        plt.figure(figsize=(3, 3))
        sns.lineplot(x='epoch', y='g_loss', data=training_log.tail(20000))
    with pgrid.output_to(0, 4):
        pgrid.clear_cell()
        plt.figure(figsize=(3, 3))
        sns.lineplot(x='epoch', y='r_loss_cat', data=training_log.tail(20000))
    with pgrid.output_to(0, 5):
        pgrid.clear_cell()
        plt.figure(figsize=(3, 3))
        sns.lineplot(x='epoch', y='r_loss_gaus', data=training_log.tail(20000))

# save model and training log for later use
torch.save(nets['D'].state_dict(), os.path.join(save_path, settings['trained_d']))
torch.save(nets['G'].state_dict(), os.path.join(save_path, settings['trained_g']))
torch.save(nets['R'].state_dict(), os.path.join(save_path, settings['trained_r']))
torch.save(nets['C'].state_dict(), os.path.join(save_path, settings['trained_c']))
torch.save(nets['Shared'].state_dict(), os.path.join(save_path, settings['trained_shared']))
training_log.to_csv(os.path.join(save_path, settings['training_log']))
```

```
# %%script false
# load trained model
with open(os.path.join(save_path, 'notebook_settings.json'), 'r') as file:
    settings = json.load(file)

nets['D'].load_state_dict(torch.load(os.path.join(save_path, settings['trained_d'])))
nets['G'].load_state_dict(torch.load(os.path.join(save_path, settings['trained_g'])))
nets['R'].load_state_dict(torch.load(os.path.join(save_path, settings['trained_r'])))
nets['C'].load_state_dict(torch.load(os.path.join(save_path, settings['trained_c'])))
nets['Shared'].load_state_dict(torch.load(os.path.join(save_path, settings['trained_shared'])))
training_log = pd.read_csv(os.path.join(save_path, settings['training_log']))

# Download the MNIST dataset
transforms = tv.transforms.Compose([
    tv.transforms.ToTensor(),
    tv.transforms.Normalize((0.5,), (0.5,))
])
mnist_data = {
    'train': MNIST('.', train=True,  transform=transforms, download=True),
    'test':  MNIST('.', train=False, transform=transforms, download=True)}
```

```
noise, _ = nets['G'].sample_latent(1, 'cuda')
img_fake = nets['G'](noise)
img_real = mnist_data['test'][0][0].view(1, 1, 28, 28)
pgrid = clwidgets.Grid(1, 6)
with pgrid.output_to(0, 0):
    pgrid.clear_cell()
    plt.figure(figsize=(3, 3))
    plt.imshow(img_real[0, 0].detach().cpu().numpy())
    plt.gca().set_title('Real Example')
    plt.gca().grid(False)
    plt.gca().set_axis_off()
with pgrid.output_to(0, 1):
    pgrid.clear_cell()
    plt.figure(figsize=(3, 3))
    plt.imshow(img_fake[0, 0].detach().cpu().numpy())
    plt.gca().set_title('Generated Example')
    plt.gca().grid(False)
    plt.gca().set_axis_off()
with pgrid.output_to(0, 2):
    pgrid.clear_cell()
    plt.figure(figsize=(3, 3))
    sns.lineplot(x='epoch', y='d_loss', data=training_log)
with pgrid.output_to(0, 3):
    pgrid.clear_cell()
    plt.figure(figsize=(3, 3))
    sns.lineplot(x='epoch', y='g_loss', data=training_log)
with pgrid.output_to(0, 4):
    pgrid.clear_cell()
    plt.figure(figsize=(3, 3))
    sns.lineplot(x='epoch', y='r_loss_cat', data=training_log)
with pgrid.output_to(0, 5):
    pgrid.clear_cell()
    plt.figure(figsize=(3, 3))
    sns.lineplot(x='epoch', y='r_loss_gaus', data=training_log)
```

###Plot a number of random samples
Below shows 100 images, arranged into a 10x10 grid, generated by the GAN we just trained.

```
z = torch.randn(100, 62, 1, 1, device='cuda')
k = torch.zeros(100, 10, 1, 1, device='cuda')
k[torch.arange(100), torch.randint(0, 10, (100,))] = 1.
c = torch.stack(torch.meshgrid(torch.linspace(-1, 1, 10), torch.linspace(-1, 1, 10)), dim=2).view(100, 2, 1, 1).cuda()

with torch.no_grad():
    latc = torch.cat([z, k, c], dim=1)
    img_fake = nets['G'](latc)
    img_fake = img_fake.view(10, 10, 28, 28).permute(0, 2, 1, 3).contiguous().view(280, 280)
    img_fake = img_fake.cpu().numpy()
    plt.figure(figsize=(8, 8))
    plt.imshow(img_fake)
    plt.gca().grid(False)
    plt.gca().set_axis_off()
```

### Examining the categorical latent code
The 10x10 grid below shows how the continuous latent code affects the generated images. The continuous latent code has 2 dimensions, one varies from -1 to 1 along the horizontal axis and the other along the vertical axis. The noise vector $z$ is kept constant for each generation.

The slider on top of the plot controls the 1 position in the categorical latent code.

```
@ipywidgets.interact(index=(0, 9, 1))
def generate(index):
    k.fill_(0.)
    k[:, index] = 1

    with torch.no_grad():
        latc = torch.cat([z, k, c], dim=1)
        img_fake = nets['G'](latc)
        img_fake = img_fake.view(10, 10, 28, 28).permute(0, 2, 1, 3).contiguous().view(280, 280)
        img_fake = img_fake.cpu().numpy()
        plt.figure(figsize=(8, 8))
        plt.imshow(img_fake)
        plt.gca().grid(False)
        plt.gca().set_axis_off()
```

    interactive(children=(IntSlider(value=4, description='index', max=9), Output()), _dom_classes=('widget-interac…

```

```


+ This notebook is part of *Final exam review* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

```python
from sympy import init_printing, Matrix, symbols, sqrt, Rational
from sympy.solvers import solve
from numpy import matrix, transpose, sqrt, eye
from numpy.linalg import pinv, inv, det, svd, norm, eig
from scipy.linalg import pinv2
from warnings import filterwarnings
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

# Final examination review

## Previous examination questions

### Question 1

+ If A is a *m* × *n* matrix of rank *r* and the following holds
    + No solution
    $$ Ax=\begin{bmatrix}1\\0\\0\end{bmatrix} $$
    + One solution
    $$ Ax=\begin{bmatrix}0\\1\\0\end{bmatrix} $$

+ How many rows in this matrix?
    + *m* = 3

+ What is the rank?
    + If there are no solutions then *r* < *m*
    + If there is only a single solution then the nullspace has only the zero vector as so *r* = *n*

+ How many columns?
    + For one solution (as above) *r* = *n* and with *m* = 3 and *r* < *m* we have *r* = *n* < 3

+ Write down a matrix that fits the description above
$$ A=\begin{bmatrix}0&0\\1&0\\0&1\end{bmatrix} $$

+ True or False for the above
    + Determinant of ATA is same as determinant of AAT
        + False
    + ATA is invertible
        + If *r* = *n* (independent columns of A) then TRUE
    + AATA is positive definite
        + False (it is going to be 3 × 3, but still with only rank 2)

```python
A = Matrix([[0, 0], [1, 0], [0, 1]])
A
```

$$\left[\begin{matrix}0 & 0\\1 & 0\\0 & 1\end{matrix}\right]$$

```python
(A.transpose() * A).inv()
```

$$\left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$$

```python
(A.transpose() * A).det() == (A * A.transpose())
```

    False

```python
A * A.transpose()
```

$$\left[\begin{matrix}0 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$$

+ Prove that AT*y* = *c* has at least one solution for every *c* and in fact infinitely many solution for every *c*
    + It has at least one solution because the new number of rows (*n*) is equal to *r*
    + The dimension of the nullspace of AT is *m* - *r*, which in our example here would be > 0, thus infinitely many solutions

### Question 2

+ Suppose we have a matrix A with columns containing vectors *v*1, *v*2, and *v*3

+ Solve A**x** = *v*1 - *v*2 + *v*3
    + This is simple multiplication by columns
    $$ x=\begin{bmatrix}1\\-1\\1\end{bmatrix} $$

+ Suppose *v*1 - *v*2 + *v*3 = 0
    + Is the solution unique or are there more
        + Uniqueness means nothing in the nullspace except the zero vector, so in this cane the solutions are not unique
+ Suppose the columns are orthonormal (would be called *q*1, *q*2, *q*3)
    + What combination of *v*1 and *v*2 are closet to *v*3?
        + Zero for each *v*1 and *v*2

### Question 3

+ Consider the Markov matrix
$$ \begin{bmatrix}0.2&0.4&0.3\\0.4&0.2&0.3\\0.4&0.4&0.4\end{bmatrix} $$

+ Calculate the eigenvalues
    + The matrix is singular (note how ½ of columns 1 plys ½ of column 2 equals columns 3) so one eigenvalue will be zero
    + Another must be 1
    + The trace adds to 0.8 and so must the sums of the eigenvalues, thus the last eigenvalue is -0.2

+ If for the following the *u*(0) vector is as indicated, what would teh solution be after *k* steps?
$$ {u}_{k}={A}^{k};\quad u\left({0}\right)=\begin{bmatrix}0\\10\\0\end{bmatrix} $$
    + The following will hold
    $$ {u}_{k}={A}^{k};\quad u\left({0}\right)={c}_{1}{\lambda}_{1}^{k}{x}_{1}+{c}_{2}{\lambda}_{2}^{k}{x}_{2}+{c}_{3}{\lambda}_{3}^{k}{x}_{3} \\ {u}_{k}={A}^{k};\quad u\left({0}\right)=0+{c}_{2}\left({1}\right){x}_{2}+{c}_{3}\left({-0.2}\right)^{k}{x}_{3} $$
    + So at ∞ the only term that survives is *c*2*x*2
    + Indeed, the key eigenvalue in any Markov matrix is 1

+ Consider the eigenvector and calculate *u* at ∞
    + We already know that we have to use the λ = 1 eigenvalue
    + The distribution at ∞ will be as follows (see python code below)
    $$ u\left({\infty}\right)=\begin{bmatrix}3\\3\\4\end{bmatrix} $$

```python
A = Matrix([[0.2, 0.4, 0.3], [0.4, 0.2, 0.3], [0.4, 0.4, 0.4]])
A
```

$$\left[\begin{matrix}0.2 & 0.4 & 0.3\\0.4 & 0.2 & 0.3\\0.4 & 0.4 & 0.4\end{matrix}\right]$$

```python
A.eigenvects() # Looking for eigenvector of eigenvalue 1
# Have to distribute the totals into 10 (were 10 total intiallly)
```

$$\begin{bmatrix}\begin{pmatrix}-0.2, & 1, & \begin{bmatrix}\left[\begin{matrix}-1.0\\1.0\\0\end{matrix}\right]\end{bmatrix}\end{pmatrix}, & \begin{pmatrix}0, & 1, & \begin{bmatrix}\left[\begin{matrix}-0.5\\-0.5\\1.0\end{matrix}\right]\end{bmatrix}\end{pmatrix}, & \begin{pmatrix}1.0, & 1, & \begin{bmatrix}\left[\begin{matrix}0.75\\0.75\\1.0\end{matrix}\right]\end{bmatrix}\end{pmatrix}\end{bmatrix}$$

### Question 4

+ Calculate the projection onto the following line
$$ a=\begin{bmatrix}4\\-3\end{bmatrix} $$
    + The projection matrix is
    $$ P=\frac{{a}{a}^{T}}{{a}^{T}{a}} $$

```python
a = matrix([[4], [-3]]) # Using scipy
(a * transpose(a)) / (transpose(a) * a)
```

    matrix([[ 0.64, -0.48],
            [-0.48,  0.36]])

+ Consider the matrix with eigenvalues 0 and 3 and the following eigenvectors
$$ 0,\begin{bmatrix}1\\2\end{bmatrix}\quad 3,\begin{bmatrix}2\\1\end{bmatrix} $$
    + We use the following decomposition
    $$ A={S}{\Lambda}{S}^{-1} $$

```python
S = matrix([[1, 2], [2, 1]])
L = matrix([[0, 0], [0, 3]])
S_inv = inv(S)
```

```python
A = S * L * S_inv
A
```

    matrix([[ 4., -2.],
            [ 2., -1.]])

+ Give a 2 × 2 matrix A such that A ≠ BTB for any B
    + BT is always symmetric, so A can be any non-symmetric matrix

+ A matrix that has orthogonal eigenvectors, but is not symmetric
    + Any skew-symmetric matrix (transpose = negative of matrix)
    $$ \begin{bmatrix}0&1\\-1&0\end{bmatrix} $$
    + Any orthogonal matrix
    $$ \begin{bmatrix}\cos&-\sin\\\sin&\cos\end{bmatrix} $$

### Question 5

+ Consider the following system A**x**=**b**, with the least squares solution shown and calculate the projection of **b** onto the columnspace of A
$$ \begin{bmatrix} 1 & 0 \\ 1 & 1 \\ 1 & 2 \end{bmatrix}=\begin{bmatrix} c \\ d \end{bmatrix}=\begin{bmatrix} 3 \\ 4 \\ 1 \end{bmatrix}, \quad \begin{bmatrix} \hat { c }  \\ \hat { d }  \end{bmatrix}=\begin{bmatrix} \frac { 11 }{ 3 }  \\ -1 \end{bmatrix} $$
    + The least square solution is given, so simply multiply each entry by its column
    $$ \frac{ 11 }{ 3 } \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}-1\begin{bmatrix} 0 \\ 1 \\ 2 \end{bmatrix} $$

+ Calculate a different vector **b** such that all the least square solutions are zero
    + This requires **b** to be orthogonal to those columns, such as the following
    $$ \begin{bmatrix}1\\-2\\1\end{bmatrix} $$

### Question 6 (from recitation)

+ Consider then 3 × 3 matrix A, with λ1=1 and λ2=2 and the first two pivots *d*1=*d*2=1
$$ A=\begin{bmatrix}1&0&1\\0&1&1\\1&1&0\end{bmatrix} $$

+ Find λ3 and *d*3
    + The sum of the eigenvalues must equal the trace, thus λ3=-1
    + Constant multiples of a row subtracted from another won't change the determinant leaving *d*1×*d*2×*d*3=|A| (just watching out for singular matrices which will have a zero on the main diagonal; here though we have three non-zero eigenvalues, so the matrix is not-singular), leaving *d*3=-2 (product of eigenvalues is also the determinant of A)

+ Calculate the smallest *a*33 entry that will make positive semi-definite
    + For positive definite the eigenvalues must all be ≥ zero
    + The determinant must also be ≥ 0

```python
a33 = symbols('a33')
A = Matrix([[1, 0, 1], [0, 1, 1], [1, 1, a33]])
A
```

$$\left[\begin{matrix}1 & 0 & 1\\0 & 1 & 1\\1 & 1 & a_{33}\end{matrix}\right]$$

```python
A.det() # Thus a33 must be grteater tha or equal to 2
```

$$a_{33} - 2$$

+ Calculate the smallest values of *c* such that the following is still positive semi-definite
$$ A-cI $$
    + We can calculate the determinant using sympy (see below) or we can make use of the fact that adding a constant multiple of the identity matrix will only add that constant to each eigenvalue, leaving the eigenvectors intact
       $$ 1+c,\quad 2+c,\quad -1+c $$
   + Each must be ≥ 0, so the smallest value of *c* is 1

```python
c = symbols('c')
A = Matrix([[1, 0, 1], [0, 1, 1], [1, 1, 0]])
(A - c * eye(3))
```

$$\left[\begin{matrix}- 1.0 c + 1 & 0 & 1\\0 & - 1.0 c + 1 & 1\\1 & 1 & - 1.0 c\end{matrix}\right]$$

```python
(A - c * eye(3)).det() # From here we can calulcate the smallest value of c such
# that the determinant is still greater than or equal to zero
```

$$- 1.0 c^{3} + 2.0 c^{2} + 1.0 c - 2$$

```python
f = -c ** 3 + 2 * c ** 2 + c - 2
f
```

$$- c^{3} + 2 c^{2} + c - 2$$

```python
solve(f, c) # solve the equation f for the variable c
```

$$\begin{bmatrix}-1, & 1, & 2\end{bmatrix}$$

+ Consider now one of the starting vectors *u*0 below and with *u*k+1 = ½A*u*k calculate the limiting behavior of *u*k as *k* approaches ∞
$$ {u}_{0}=\begin{bmatrix}3\\0\\0\end{bmatrix},\quad\begin{bmatrix}0\\3\\0\end{bmatrix},\quad\begin{bmatrix}0\\0\\3\end{bmatrix} $$
    + Notice that ½ is a Markov matrix
    + We cannot be sure that there will be a steady state as there are zero entries in ½A
    + Multiplying a matrix by a constant scalar will not change the eigenvectors, but will change the eigenvalues by the same scalar multiple and we will have λ1=½ and λ2=1 and λ3=-½
    + We do have an eigenvalue of 1, so we will reach a steady-state
    + The eigenvector of λ2=1 is the following (see below)
    $$ \begin{bmatrix}1\\1\\1\end{bmatrix} $$
    + This already sums to 3, so will be *u*∞

```python

```


# What do word vectors represent?

On Monday we saw the result of running a word embedding algorithm on two collections. We "embed" words in a vector space, such that words that are subtitutable ("ship" and "boat") or that occur together ("Stop" and "thief!").

In today's work we will look at the intuition for how the properties of embedding vectors relate to properties we can directly observe in texts. They are really just representations of the words that occur near a given word.

What can we tell about a word from the words that occur near it?

```python
import numpy, sys, math
from IPython.display import display, clear_output, Markdown, Latex

from collections import Counter
```

```python
## Helper functions to nicely display numeric word scores

def show(sorted_words, n=20):
    markdown_table = "|Score | Word|\n|---:|:---|\n"
    for score, word in sorted_words[:n]:
        markdown_table += "|{:.3f}|{}|\n".format(score, word)
    display(Markdown(markdown_table))

def show_counter(counter, n=20):
    markdown_table = "|Count | Word|\n|---:|:---|\n"
    for word, score in counter.most_common(n):
        markdown_table += "|{}|{}|\n".format(score, word)
    display(Markdown(markdown_table))
```

First we'll read the texts. I've already split the tokens with Spacy and written the output to a file with one sentence per line, so punctuation will be included as distinct tokens.

While we read this, we'll also count the frequency of each word type in `all_counter`.

```python
text_filename = "../data/Sagas/sagas_en_split.txt"

sentences = []
all_counter = Counter()

with open(text_filename, encoding="utf-8") as reader:
    for line in reader:
        ## The file has already been tokenized, so we can split on whitespace
        tokens = line.strip().split()
        all_counter.update(tokens)

        sentences.append(tokens)
```

Let's start by looking at the context that words appear in. The next block defines a *key word in context* (KWIC) view.

```python
window_size = 5
```

```python
def keyword_in_context(query):
    table_markdown = "|left context|word|right context|\n|--:|--|:--|\n"
    for sentence in sentences:

        if not query in sentence:
            continue

        for i, word in enumerate(sentence):
            if word == query:
                start = max(i-window_size, 0)
                left_context = sentence[start:i]
                right_context = sentence[(i+1):(i+window_size+1)]
                table_markdown += "|{}|{}|{}|\n".format(" ".join(left_context), word, " ".join(right_context))

    display(Markdown(table_markdown))
```

### Part 1

I've given you an example, for *Shetland*, a chain of islands north of Scotland near the Orkney islands.

Add 10 additional cells, each with one call to the `keyword_in_context` function. Choose five pairs of words that you think might be similar (e.g. *Shetland* and *Orkneys*). Select a variety of parts of speech, such as nouns, verbs, adjectives, prepositions, and proper names.

Discuss what you notice about the similarities and differences between the contexts of these words.

**Answer here**

```python
keyword_in_context("Shetland")
```

```python
# add more `keyword_in_context` cells here
```

Now let's look at the distribution of words immediately preceding (*left* or *previous* context) and immediately following (*right* or *next* context) a word. This block creates two dictionaries, which map a string to the `Counter` of the words that follow that word and precede it, respectively.

```python
previous_context_counters = {} # count words that precede the key
next_context_counters = {} # count words that follow the key

for sentence in sentences:
    for i in range(len(sentence) - 1): # stop at the next-to-last token
        word = sentence[i]
        next_word = sentence[i+1]

        if not word in next_context_counters:
            next_context_counters[word] = Counter()
        if not next_word in previous_context_counters:
            previous_context_counters[next_word] = Counter()

        next_context_counters[word][next_word] += 1
        previous_context_counters[next_word][word] += 1
```

### Part 2

In the next code cell I'm demonstrating how to get the most frequent following words for a query word.

Use this function like a "predictive text" feature. Generate two Viking sentences of 10-20 words.
* In the first sentence, start with "Then" and pick the most frequent following word. Record your sentence, and comment on why always picking the most common word might not be a good idea.
* In the second sentence, start with "Then" but choose the next word based on both the frequency distribution and your artistic sensibilities.

**First sentence here**

**Comment on first sentence**

**Second sentence here**

Add cells to show previous *and* next context words for at least 10 more words. Use a selection of nouns, verbs, adjectives, prepositions, and names. These may be the same words you looked at before, but you may also want to add additional examples.

Discuss whether the words to the right or left of a word indicate its part of speech. Cite examples to support your argument. Are the two contexts equally informative for a given part of speech, and is that consistent across different parts of speech?

**Answer here**

```python
show_counter(next_context_counters["she"])
```

```python
show_counter(previous_context_counters["she"])
```

```python
## add cells here
```

Next we'll look at sums over the full five-word context window. This code creates one `Counter` for each word type, which adds up all the words that appear within the window around the word.

```python
word_context_counters = {}

for sentence in sentences:

    for i, word in enumerate(sentence):
        start = max(i-window_size, 0)
        left_context = sentence[start:i]
        right_context = sentence[(i+1):(i+window_size+1)]

        if not word in word_context_counters:
            word_context_counters[word] = Counter()

        word_context_counters[word].update(left_context)
        word_context_counters[word].update(right_context)
```

### Part 3

This next cell is an example showing output for the full context counts of a word, essentially adding up all the words you saw in the KWIC view earlier.

Show output for at least 10 words, from a mix of parts of speech.

Discuss how this view of a word's context differs from the single-previous-word and single-next-word context views we saw in Part 2.

**Answer here**

```python
show_counter(word_context_counters["Shetland"], 15)
```

```python
## add cells here
```

Finally, let's look at a way of comparing the word frequencies we actually observed to the word frequencies in the collection as a whole. We'll use a method called *pointwise mutual information*.

PMI is closely related to KL divergence. In this case, the two distributions we want to compare are the probability of context word $c$ *near* word $w$ and the probabilty of $c$ anywhere. The word *the* is common throughout the collection, so we expect to see it. This metric measures the ratio between the frequency that we actually saw it in the context and our expectation for any random context.

Notation:
* $N(c|w)$ is `word_context_counters[w][c]`
* $N(w)$ is `sum(word_context_counters[w].values()`
* $N(c)$ is `all_counter[c]`
* $N$ is `all_sum`

$$
\begin{align}
PMI(c, w) & = P(c, w) \log \frac{P(c,w)}{P(c)P(w)} \\
& = P(c, w) \log \frac{P(c|w)P(w)}{P(c)P(w)} \\
& = P(c, w) \log \frac{P(c|w)}{P(c)} \\
& \propto N(c|w) \log \frac{\frac{N(c|w)}{N(w)} }{ \frac{N(c)}{N}  } \\
& = N(c|w) \log \frac{N(c|w)N}{N(w)N(c)}
\end{align}$$

```python
def log_ratio(word):
    counter = word_context_counters[word]

    all_sum = sum(all_counter.values()) ## N
    word_sum = sum(counter.values())    ## N(w)

    comparisons = []
    for c in counter.keys():
        score = counter[c] * math.log((counter[c] * all_sum) / (word_sum * all_counter[c]))
        comparisons.append((score, c))

    return sorted(comparisons, reverse=True)

```

### Part 4

Compare results using this `log_ratio` function to the output of the `nearest` function used in Monday's notebook.

Provide some examples, and describe how they are similar or different from the output of the word embedding. If there are "missing" words in the output here that are close in the embedding space, show the `log_ratio` output for those words. Do the two words have similar context words? Describe whether this is true and mention examples.

**Answer here**

```python
## 'spae' is a Scots word for prophecy. Gunnhilda was the wife of Eric Bloodaxe.
## She was ordered to be drowned in a bog by King Harald Bluetooth, the namesake
## of the wireless standard. Think about that next time you put on some headphones.

show(log_ratio("queen"))
```

```python
## add cells with examples here
```

**Extra bonus for those interested** The embedding algorithm adds an additional step: subsampling the most frequent words. Here's code that generates this subsampling probability.

```python
sampling_probs = {}
all_sum = sum(all_counter.values())
for word in all_counter.keys():
    p_word = all_counter[word] / all_sum
    score = 1.0 / (10000 * p_word)
    sampling_probs[word] = math.sqrt(score) + score
```


```python
#from IPython.display import Image
#Image(filename='i_could_care_less.png')
#<div style="text-align:center">  </div>
from IPython.display import Image
from IPython.display import HTML
html1 = '<div style="text-align:center">  </div>'
HTML(html1)

#from IPython.display import Image
#from IPython.core.display import HTML
#Image(filename= "i_could_care_less.png",width=500, height=500)
```

# Word Representation With One-Hot Vectors

One-hot encoding is the most common, most basic way to turn a token into a vector. It consists in associating a unique integer index to every word, then turning this integer index $i$ into a binary vector of size $V$, which is the size of our vocabulary, that would be all-zeros except for the $i$-th entry, which would be 1.

```python
import numpy as np
samples = ['The cat sat on the mat.', 'The dog ate my homework.']
token_index = {}
for sample in samples:
    for word in sample.split():
        if word not in token_index:
            token_index[word] = len(token_index) + 1

sentence = "the cat ate the god"
seq = []
for word in sentence.split():
    one_hot = np.zeros((1,len(token_index)))
    if word not in token_index:
        seq.append(one_hot.squeeze())
    else:
        one_hot[0,token_index[word]-1] = one_hot[0,token_index[word]-1] + 1
        seq.append(one_hot.squeeze())
```

```python
seq
```

    [array([0., 0., 0., 0., 1., 0., 0., 0., 0., 0.]),
     array([0., 0., 0., 0., 0., 0., 1., 0., 0., 0.]),
     array([0., 0., 0., 0., 0., 0., 0., 1., 0., 0.]),
     array([0., 0., 0., 0., 1., 0., 0., 0., 0., 0.]),
     array([0., 1., 0., 0., 0., 0., 0., 0., 0., 0.])]

```python
import numpy as np
with open('cats100.test.txt') as file_handler:
    data = file_handler.read()

token_idx = {}
for word in data.split():
    if word not in token_idx:
        token_idx[word] = len(token_idx) + 1

sentence = "Geçtiğimiz zamanlarda kötü olaylar yaşandı"
one_hot_sentence = np.zeros((len(sentence.split()),len(token_idx)),dtype=np.int32)
for word_idx,word in enumerate(sentence.split()):
    one_hot_sentence[word_idx,token_idx[word] - 1] = 1
```

Figure is taken from Stanford cs-230 cheatsheet

 There are two major issues with this approach. First issue is the curse of dimensionality, which refers to all sorts of problems that arise with data in high dimensions. This requires exponentially large memory space. Most of the matrix is taken up by zeros, so useful data becomes sparse. Imagine we have a vocabulary of 50,000. (There are roughly a million words in English language.) Each word is represented with 49,999 zeros and a single one, and we need 50,000 squared = 2.5 billion units of memory space. Not computationally efficient.

Second issue is that every one hotted vector is orthogonal to each other. You cannot measure the
similarity (like cosine similarity) on this vectors.

Those vectors can be visualized in 3 or 2-dimensional space as an example. In $\mathbb{R}^3$ (in other words, vocabulary size of 3, $\mid V \mid=3$), the word vectors become our span set which is
$\left\{\begin{bmatrix} 1\\0\\0\end{bmatrix},
\begin{bmatrix} 0\\1\\0\end{bmatrix},
\begin{bmatrix} 0\\0\\1\end{bmatrix}\right\}$. The span set is linearly independent set. Therefore we cannot compute similarity metrics. The vectors $\vec{u_1}, \vec{u_2}, \vec{u_3}$ are orthogonal. Similarity metrics gives the result of zero. For example the cosine similarity:

$$ \text{cos-sim}(\vec{u_1},\vec{u_2}) = \frac{\langle \vec{u_1}, \vec{u_2} \rangle}{\lVert \vec{u_1} \lVert_2 \times \lVert \vec{u_2} \lVert_2 } = \frac{\sum_{i=0}^n u_{1_i} \times u_{2_i}}{\sqrt{\sum_{i=0}^n u_{1_i}^2} \times \sqrt{\sum_{i=0}^n u_{2_i}^2}} = \frac{1 \times 0 + 0 \times 1 + 0 \times 0}{1 + 1} = 0$$

So, how to deal with those problems?

# Lexical Semantics And Distributional Linguistics

Words that occur in similar contexts tend to have similar meanings. The link between similarity and words are distributed and similarity in what they mean is called distributional hypothesis or distributional semantics in the field of Computational Linguistics. So what can be counted when we say similar contexts? For example if you surf on the Wikipedia page of linguistics, the words in this page somehow related with each other in the context of linguistics. This was formulated firsty by Martin Joos (Description of Language Design, 1950), Zellig Harris (Distributional Structure, 1954), John Rupert Firth (Applications of General Linguistics, 1957).

Some words have similar meanings, for example word cat and dog similar. Also, words can be antonyms, for example hot and cold. And words have connotations (TR: çağrışım), for example happy->positive connotation and sad->negative connotation. Can you feel the similarity of words, [study, exam, night, FF]?

Also each word can have multiple meanings. The word mouse can refeer to the rodent or the cursor control device. We call each of these aspects of the meaning of mouse a word sense. In other words words can be polysemous (have multiple senses), which can lead us to make word interpretations difficult!

- Word Sense Disambiguation: "Mouse info" (person who types this into a web search engine, looking for a pet info or a tool?) (determining which sense of a word is being used in a particular context)

The word **similarity** is very useful in larger semantics tasks. Knowing how similar two words are can help in computing how similar the meaning of two phrases or sentences are, a very important component of natural language understanding tasks like **question answering**, **summarization** etc.

| Word1       | Word2       |  Similarity (0-10) |
| ----------- | ----------- |  -----------       |
| Vanish      | Disappear   |  9.8             |
| Behave      | Obey        |  7.3              |
| Belief      | Impression  |  5.95              |
| Muscle      | Bone        |  3.65              |
| Modest      | Flexible    |  0.98              |
| Hole        | Agreement   |  0.3              |

We should look at **word relatedness** , the meaning of two words can be related in ways other than similarity. One such class of connections is called word **relatedness**, also traditionally called word **association** in pysch.

- The word *cup* and *coffee*.
- The word *inzva* and *deep learning*.

Also words can affective meanings. Osgood et al. 1957, proposed that words varied along three important dimensions of affective meaning: *valence, arousal, dominance*.
- **valence**: the pleasantness of the stimulus.
- **arousal**: the intensity of emotion provoked by the stimulus.
- **dominance**: the degree of control exerted by the stimulus.

Examples:
- happy(1) $\uparrow$, satisfied(1) $\uparrow$; annoyed(1) $\downarrow$, unhappy(1)$\downarrow$
- excited(2) $\uparrow$, frenzied(2) $\uparrow$; relaxed(1) $\downarrow$, calm(2) $\downarrow$
- important(3)$\uparrow$, controlling(3)$\uparrow$; awed(3)$\downarrow$, influenced(3)$\downarrow$

*Question: does word embeddings has these dimesions?*

# Word Embeddings

How can we build a computational model that successfully deals with the different aspects of word meaning we saw above (word senses, word similarity, word relatedness, connotation etc.)?

**Instead of representing words with one-hot vector, sparse; with word embeddings we represents words with dense vectors**.

**The idea of vector semantics is thus to represent a word as a point in some multidimensional semantic space.** Vectors for representing words are generally called **embeddings**. We **LEARN**  this embeddings form an arbitrary context.

Figure is taken from this medium post

Main two advantages of this word embeddings are:

- Now we represent words with dense vectors, which leads low memory requirements.

Figure is taken from Stanford cs-230 cheatsheet

- We can now calculate similarity metrics on these vectors!

## Visualizing Word Embeddings

Word embeddings can be visualized with various dimensionality reduction/matrix factorization algorithms.

- left-figure source: [Zero-Shot Learning Through Cross-Modal Transfer (Socher et al., 2013, NeurIPS)](https://nlp.stanford.edu/~socherr/SocherGanjooManningNg_NIPS2013.pdf)
- right-figure source: [Word representations: A simple and general method for semi-supervised learning (Turian et al., 2010)](http://metaoptimize.s3.amazonaws.com/cw-embeddings-ACL2010/embeddings-mostcommon.EMBEDDING_SIZE=50.png)

- left-figure source: [Natural Language Processing (almost) from Scratch (Collobert et al., 2011)](https://arxiv.org/abs/1103.0398v1.pdf)
- right-figure source: [Bilingual Word Embeddings for Phrase-Based Machine Translation (Socher et al., 2013, EMNLP)](https://ai.stanford.edu/~wzou/emnlp2013_ZouSocherCerManning.pdf)

## Using Word Embeddings

### Named Entity Recognition (NER)
In Natural language processing, Named Entity Recognition (NER) is a process where a sentence or a chunk of text is parsed through to find entities that can be put under categories like names, organizations, locations, quantities, monetary values, percentages, etc. Traditional NER algorithms included only names, places, and organizations.

Since the embeddings can capture word senses etc., it is practical and beneficial to use word embeddings in NER task. Embeddings can capture entity informations while capturing word relations.

figure sources: [link](https://towardsdatascience.com/named-entity-recognition-ner-meeting-industrys-requirement-by-applying-state-of-the-art-deep-698d2b3b4ede)

### Transfer Learning

- To learn word embeddings, huge size of training data is always useful. For example GloVe is trained on 5 separate corpora:
    * 2010 Wikipedia dump with 1 billion tokens
    * 2014 Wikipedia dump with 1.6 billion tokens
    * Gigaword 5 which has 4.3 billion tokens
    * the combination Gigaword5 + Wikipedia2014, which has 6 billion tokens
    * 42 billion tokens of web data, from Common Crawl

- You can download pre-trained word embeddings online.
    * [GloVe pre-trained vectors](https://nlp.stanford.edu/projects/glove/)
        * Common Crawl (42B tokens, 1.9M vocab, uncased, 300d vectors, 1.75 GB download)
        * Common Crawl (840B tokens, 2.2M vocab, cased, 300d vectors, 2.03 GB download)
        * Twitter (2B tweets, 27B tokens, 1.2M vocab, uncased, 25d, 50d, 100d, & 200d vectors, 1.42 GB download)
    * [Hellinger PCA vectors](http://lebret.ch/words/)
    * [word2vec pre-trained vectors](https://wikipedia2vec.github.io/wikipedia2vec/pretrained/)
    * etc.

- Transfer embedding to new task with smaller training set.
    * Language Modelling
    * Predictive Typing
    * Spelling/Grammar Correction
    * Summarization
    * NMT
    * etc.

- Finetune the word embeddings with new data (if your training data relatively big).

### Dependency Parsing
A dependency parser analyzes the grammatical structure of a sentence, establishing relationships between "head" words and words which modify those heads. The figure below shows a dependency parse of a short sentence.

Syntactic Parsing or Dependency Parsing is the task of recognizing a sentence and assigning a syntactic structure to it. The most widely used syntactic structure is the parse tree which can be generated using some parsing algorithms. These parse trees are useful in various applications like grammar checking or more importantly it plays a critical role in the semantic analysis stage.

Dependency parsing is the task of analyzing the syntactic dependency structure of a given input sentence $S$. The output of a dependency parser is a dependency tree where the words of the input sentence are connected by typed dependency relations. Formally, the dependency parsing problem asks to create a mapping from the input sentence with words $S = w_0w_1...w_n$ (where $w_0$ is the ROOT) to its dependency tree graph $G$.

[A Fast and Accurate Dependency Parser using Neural Networks (Chen et al., 2014)](https://www.aclweb.org/anthology/D14-1082/)

right-figure source [CS224n Part IV](https://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes04-dependencyparsing.pdf)

### Representation Learning

Various tasks can be modeled with representations. Since words can be represented by embeddings, images can be (or even signals/audio) represented by a latent space $z$. Autoencoders is a way to learn latent representation of data.

## Semantic Properties Of Embeddings

Word embeddings have very important property: analogies. Analogy is another semantic property of embeddings that can capture relational meanings. Simply in words, analogy is to find **X**:

- **A is to B as C is to X**

For example **“woman is to queen as man is to X**. In this example **X** should be the word *king*.

Interestingly, such embeddings exhibit seemingly linear behaviour in analogies. This linear behaviour can be formulated as

- $w_a$ is to $w_a'$ as $w_b$ is to $w_b'$ $\rightarrow \rightarrow \rightarrow$ $w_a' - w_a + w_b \approx w_b'$

- vec('*queen*') - vec('*woman*') + vec('*man*') $\approx$ vec('*king*')

or

- vec('*queen*') - vec('*woman*') $\approx$ vec('*king*') - vec('*man*')

Another example:

- vec('*Paris*') - vec('*France*') $\approx$ vec('*Italy*') - vec('*Rome*')

- left-figure source: [Linguistic Regularities in Continuous Space Word Representations (Mikolov et al., 2013)](https://www.aclweb.org/anthology/N13-1090.pdf)
- mid-figure source: [Speech and Language Processing, Daniel Jurafsky, Third Edition](https://web.stanford.edu/~jurafsky/slp3/)
- right-figure source: [Efficient Estimation of Word Representations in Vector Space (Mikolov etl al., 2013)](https://arxiv.org/pdf/1301.3781.pdf)

Do vector embeddings capture syntactic relationships? Yes. Capture with linear behaviours? Yes.

- left-figure source: [Linguistic Regularities in Continuous Space Word Representations (Mikolov et al., 2013)](https://www.aclweb.org/anthology/N13-1090.pdf)
- right-figure source: [Speech and Language Processing, Daniel Jurafsky, Third Edition](https://web.stanford.edu/~jurafsky/slp3/)

For more formal definition of analogy? Check [Analogies Explained: Towards Understanding Word Embeddings (Allen et al., 2019)](https://arxiv.org/pdf/1901.09813.pdf)

## Evaluating Word Vectors

- Extrinsic Evaluation
    * This is the evaluation on a real task.
    * Can be slow to compute performance.
    * Unclear if the subsystem is the problem, or our system.
    * If replacing subsystem improves performance, the change is likely good.
    * NER, Question Answering etc.

- Intrinsic Evaluation
    * Fast to compute
    * Helps to understand subsystem
    * Needs positive correlation with real task to determine usefulness

SimLex-999 dataset (Hill et al., 2015) gives values on a scale from 0 to 10 by asking humans to judge how similar one word is to another. Other datasets for evaluating word vectors:
- WordSim 353
- TOEFL Dataset
- SCWS Dataset
- Word-in-Context (WiC)
- Miller & Charles Dataset
- Rubenstein & Goodenough Dataset
- Stanford Rare Word (RW)

Some evaluation metrics from [GloVe: Global Vectors for Word Representation (Pennington et al., 2014)](https://www.aclweb.org/anthology/D14-1162.pdf)

# Learning Word Embeddings

## Embedding Matrix
Embedding matrix can be represented as a single matrix $E \in \mathbb{R}^{d \times \mid V \mid}$ (or $E \in \mathbb{R}^{ \mid V \mid \times d }$, it does not even matter), where d is embedding size and $\mid V \mid$ is size of the vocabulary $V$.

$$E = \begin{bmatrix}
\text{hello} & \text{i} & \text{love} & \text{inzva} & \cdots & \text{sanctuary}\\
0.1 & 0.137 & -0.03 & -0.44 & \cdots & 0.36\\
-0.78 & -0.25 & 2.09 & 0.19 & \cdots & 0.32\\
-3.1 & 1.54 & -2.52 & 0.2 & \cdots & -1.51\\
1.13 & -0.78 & -0.56 & 0.95 & \cdots & 0.2112\\
\vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\
- 0.6 & 0.13 & 3.89 & -0.071 & -0.27 & 0.27
\end{bmatrix} \in \mathbb{R}^{d \times \mid V \mid}$$

But how to learn them?

## A Neural Probabilistic Language Model (Bengio et al., 2003)

*In neural language models, the prior context is represented by embeddings of the previous words*. Representing the prior context as embeddings, rather than by exact words as used in [n-gram](https://web.stanford.edu/~jurafsky/slp3/3.pdf) language models, allows neural language models to generalize to unseen data much better than n-gram language models.

For example in our training set we see this sentence,

    Today, after school, I am planning to go to cinema.

but we have never seen the word "concert" after the words "go to". In our test set we are trying to predict what comes after the prefix "Today, after school, I am planning to go to". An n-gram language model will predict "cinema" but not "concert". But a neural language model, which can make use of the fact that "cinema" and "concert" have similar embeddings, will be able to assign a reasonably high probability to "concert" as well as "cinema", merely because they have similar vectors.

In 2003 Bengio proposed a neural language model that can learns and uses embeddings to predict the next word in a sentence. Formally representation, for a sequence of words

$$x^{(1)}, x^{(2)}, ..., x^{(t)}$$

the probability distribution of the next word (output) is

$$p(x^{(t+1)} \mid x^{(1)}, x^{(2)}, ..., x^{(t)})$$

Bengio proposed a fixed-window neural Language Model which can be seen as the same approach of n-grams.

"<s> as the proctor started the clock</s> the students opened their ____."

We have a moving window at time $t$ with an embedding vector representing each of the window size previous words. For window size 3, words $w_{t-1}, w_{t-2}, w_{t-3}$. These 3 vectors are concatenated together to produce input x. The task is to predict $w_t$.

- left-figure source: [CS224n lecture 5](http://web.stanford.edu/class/cs224n/slides/cs224n-2021-lecture05-rnnlm.pdf)
- right-figure source: [Speech and Language Processing, Daniel Jurafsky, Third Edition](https://web.stanford.edu/~jurafsky/slp3/)

For this task, we are going to represent each of the $N$ prevuios words as a one-hot-vector of length $\mid V \mid$.

The forward equation for neural language model

- Input $x_i \in R^{1 \times \mid V \mid}$

- Learning word embeddings: $e = concat(x_1 E^T, x_2 E^T, x_3 E^T)$
    * $E \in \mathbb{R}^{d \times \mid V \mid}$
    * $e \in \mathbb{R}^{1 \times 3d}$

- $h = \sigma(e W^T + b_1)$
    * $W \in \mathbb{R}^{d_h \times 3d}$
    * $h \in \mathbb{R}^{1 \times d_h}$

- $z = h U^T + b_2$
    * $U \in \mathbb{R}^{\mid V \mid \times d_h}$
    * $z \in \mathbb{R}^{1 \times \mid V \mid}$

- $\hat{y} = softmax(z) \in \mathbb{R}^{1 \times \mid V \mid}$

Then the model is trained, at each word $w_t$, the negative log likelihood loss is:

$$ L = - \log p(w_t \mid w_{(t-1)}, w_{(t-2)}, ..., w_{(t-n+1)}) = softmax(z) $$

$$ \theta_{t+1} = \theta_t - \eta \frac{\partial - \log p(w_t \mid w_{(t-1)}, w_{(t-2)}, ..., w_{(t-n+1)})}{\partial \theta}$$

# word2vec

word2vec is proposed in the paper called [Distributed Representations of Words and Phrases and their Compositionality (Mikolov et al., 2013)](https://arxiv.org/abs/1310.4546). It allows you to learn the high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. Word2vec algorithm uses Skip-gram [Efficient Estimation of Word Representations in Vector Space
 (Mikolov et al., 2013)](https://arxiv.org/abs/1301.3781) model to learn efficient vector representations. Those learned word vectors has interesting property, words with semantic and syntactic affinities give the necessary result in mathematical similarity operations.

Suppose that you have a sliding window of a fixed size moving along a sentence: the word in the middle is the “target” and those on its left and right within the sliding window are the context words.

The skip-gram model is trained to predict the probabilities of a word being a context word for the given target.

For example consider this sentence,

         "A change in Quantity also entails a change in Quality"
Our target and context pairs for window size of 5:

| Sliding window (size = 5)       | Target word       |  Context |
| -----------                     | -----------       |  -----------       |
| \[A change in\]                 | a                 |  change, in            |
| \[A change in Quantity \]       | change        |  a, in, quantitiy             |
| \[A change in Quantity  also\]       | in  | a, change, quantitiy,also               |
| ...      |  ...      |  ...          |
| \[entails a change in Quality\]      | change    |  entails, a, in, Quality             |
| \[a change in Quality\]              | in   |  a, change, Quality             |
| \[change in Quality\]              | quality   |  change, in             |

Each context-target pair is treated as a new observation in the data.

For each position $t=1,..,T$ predict context words within a window of fixed size $m$, given center word $w_j$. In Skip-gram connections we have an objective to maximize, likelihood (or minimize log-likelihood):

$$\max \limits_{\theta} \prod_{\text{center}} \prod_{\text{context}} p(\text{context}|\text{center} ;\theta)$$

$$= \max \limits_{\theta} \prod_{t=1}^T \prod_{-c \leq j \leq c, j \neq c} p(w_{t+j}|w_t; \theta)$$

$$= \min \limits_{\theta} -\frac{1}{T} \prod_{t=1}^T \prod_{-c \leq j \leq c, j \neq c} p(w_{t+j}|w_t; \theta)$$

$$= \min \limits_{\theta} -\frac{1}{T} \sum_{t=1}^T \sum_{-c \leq j \leq c, j \neq c} \log p(w_{t+j}|w_t; \theta)$$

So, How can we calculate those probabilities? Softmax gives the normalized probabilities.

## Parameterization Of Skip-Gram Model

Let's say $w_t$ is our target word and $w_c$ is current context word. The softmax is defined as

$$p(w_c \mid w_t) = \frac{\exp(v_{w_c}^T v_{w_t})}{\sum_{i=0}^{\mid V \mid} \exp(v_{w_i}^T v_{w_t})}$$

maximizing this log-likelihood function under $v_{w_t}$ gives you the most likely value of the $v_{w_t}$ given the data.

$$ \frac{\partial}{\partial v_{w_t}}\cdot \log \frac{\exp(v_{w_c}^T v_{w_t})}{\sum_{i=0}^{\mid V \mid} \exp(v_{w_i}^T v_{w_t})}$$

$$ = \frac{\partial}{\partial v_{w_t}}\cdot \log \underbrace{\exp(v_{w_c}^T v_{w_t})}_{\text{numerator}} - \frac{\partial}{\partial v_{w_t}}\cdot \log \underbrace{\sum_{i=0}^{\mid V \mid} \exp(v_{w_i}^T v_{w_t})}_{\text{denominator}}$$

$$ \frac{\partial}{\partial v_{w_t}} \cdot v_{w_c}^T v_{w_t} =  v_{w_c} \; \; (\text{numerator})$$

Now, it is time to derive denominator.

$$\frac{\partial}{\partial v_{w_t}}\cdot \log \sum_{i=0}^{\mid V \mid} \exp(v_{w_i}^T v_{w_t}) = \frac{1}{\sum_{i=0}^{\mid V \mid} \exp(v_{w_i}^T v_{w_t})} \cdot \frac{\partial}{\partial v_{w_t}} \sum_{i=0}^{\mid V \mid} \exp(v_{w_i}^T v_{w_t})$$

$$ = \frac{1}{\sum_{i=0}^{\mid V \mid} \exp(v_{w_i}^T v_{w_t})} \cdot  \sum_{i=0}^{\mid V \mid} \frac{\partial}{\partial v_{w_t}} \cdot \exp(v_{w_i}^T v_{w_t})$$

$$ = \frac{1}{\sum_{i=0}^{\mid V \mid} \exp(v_{w_i}^T v_{w_t})} \cdot  \sum_{i=0}^{\mid V \mid} \exp(v_{w_i}^T v_{w_t}) \frac{\partial}{\partial v_{w_t}} v_{w_i}^T v_{w_t}$$

$$ = \frac{\sum_{i=0}^{\mid V \mid} \exp(v_{w_i}^T v_{w_t}) \cdot v_{w_i}}{\sum_{i=0}^{\mid V \mid} \exp(v_{w_i}^T v_{w_t})} \;\; (\text{denominator}) $$

To sum up,

$$\frac{\partial}{\partial w_t} \log p(w_c \mid w_t) = v_{w_c} - \frac{\sum_{j=0}^{\mid V \mid} \exp(v_{w_j}^T v_{w_t}) \cdot v_{w_j}}{\sum_{i=0}^{\mid V \mid} \exp(v_{w_i}^T v_{w_t})}$$

$$ = v_{w_c} - \sum_{j=0}^{\mid V \mid} \frac{\exp(v_{w_j}^T v_{w_t})}{\sum_{i=0}^{\mid V \mid} \exp(v_{w_i}^T v_{w_t})} \cdot v_{w_j}$$

$$ \underbrace{= v_{w_c} - \sum_{j=0}^{\mid V \mid} p(w_j \mid w_t) \cdot v_{w_j}}_{\nabla_{w_t}\log p(w_c \mid w_t)}$$

This is the observed representation subtract $\mathop{\mathbb{E}}[w_j \mid w_t]$.

## Negative Sampling (Noise Contrastive Estimation (NCE))

The Noise Contrastive Estimation (NCE) metric intends to differentiate the target word from noise samples using a logistic regression classifier [(Noise-contrastive estimation: A new estimation principle for unnormalized statistical models, Gutmann et al., 2010)](http://proceedings.mlr.press/v9/gutmann10a/gutmann10a.pdf).

In softmax computation, look at the denominator. The summation over $\mid V\mid$ is computationally expensive. The training or evaluation takes asymptotically $O(\mid V \mid)$. In a very large corpora, the most frequent words can easily occur hundreds or millions of times ("in", "and", "the", "a" etc.). Such words provides less information value than the rare words. For example, while the skip-gram model benefits from observing co-occurences of "inzva" and "deep learning", it benefits much less from observing the frequent co-occurences of "inzva" and "the".In a very large corpora, the most frequent words can easily occur hundreds or millions of times ("in", "and", "the", "a" etc.). Such words provides less information value than the rare words. For example, while the skip-gram model benefits from observing co-occurences of "inzva" and "deep learning", it benefits much less from observing the frequent co-occurences of "inzva" and "the".

For every training step, instead of looping over the entire vocabulary, we can just sample several negative examples! We "sample" from
a noise distribution $P_n(w)$ whose probabilities match the ordering of the frequency of the vocabulary.

Consider a pair $(w_t, w_c)$ of word and context. Did this pair come from the training data? Let’s denote by $p(D=1 \mid w_t,w_c)$ the probability that $(w_t, w_c)$ came from the corpus data. Correspondingly $p(D=0 \mid w_t,w_c)$ will be the probability that $(w_t, w_c)$ didn't come from the corpus data. First, let’s model $p(D=1 \mid w_t,w_c)$ with sigmoid:

$$p(D=1 \mid w_t,w_c) = \sigma(v_{w_c}^T v_{w_t}) = \frac{1}{1 + \exp(- v_{w_c}^T v_{w_t})}$$

Now, we build a new objective function that tries to maximize the probability of a word and context being in the corpus data if it indeed is, and maximize the probability of a word and context not being in the corpus data if it indeed is not. Maximum likelihood says:

$$ \max \prod_{(w_t, w_c) \in D} p(D=1 \mid w_t,w_c) \times \prod_{(w_t, w_c) \in D'} p(D=0 \mid w_t,w_c)$$

$$ = \max \prod_{(w_t, w_c) \in D} p(D=1 \mid w_t,w_c) \times \prod_{(w_t, w_c) \in D'} 1 - p(D=1 \mid w_t,w_c)$$

$$ = \max \sum_{(w_t, w_c) \in D} \log p(D=1 \mid w_t,w_c) + \sum_{(w_t, w_c) \in D'}  \log (1 - p(D=1 \mid w_t,w_c))$$

$$ = \max \sum_{(w_t, w_c) \in D} \log \frac{1}{1 + \exp(- v_{w_c}^T v_{w_t})} + \sum_{(w_t, w_c) \in D'} \log \left(1 - \frac{1}{1 + \exp(- v_{w_c}^T v_{w_t})}\right)$$

Note that $\frac{\exp(-x)}{(1 + \exp(-x))} \times \frac{\exp(x)}{\exp(x)} = \frac{1}{(1 + \exp(x))}$

$$ = \max \sum_{(w_t, w_c) \in D} \log \frac{1}{1 + \exp(- v_{w_c}^T v_{w_t})} + \sum_{(w_t, w_c) \in D'} \log \frac{1}{1 + \exp(v_{w_c}^T v_{w_t})}$$

Maximizing the likelihood is the same as minimizing the negative log likelihood:

$$L = - \sum_{(w_t, w_c) \in D} \log \frac{1}{1 + \exp(- v_{w_c}^T v_{w_t})} -  \sum_{(w_t, w_c) \in D'} \log \frac{1}{1 + \exp(v_{w_c}^T v_{w_t})}$$

Note that $D'$ is a "false" or "negative" corpus. Where we would have sentences like "the school is eaten by pilgrims". Unnatural sentences that should get a low probability of ever occurring. We can generate $D'$ on the fly by randomly sampling this negative from the word bank.

The Negative Sampling (NEG) proposed in the original word2vec paper. NEG approximates the binary classifier’s output with sigmoid functions as follows:

$$\begin{align}
p(d=1 \vert v_{w_c}, v_{w_t}) &= \sigma(v_{w_c}^T v_{w_t}) \\
p(d=0 \vert v_{w_c}, v_{w_t}) &= 1 - \sigma(v_{w_c}^T v_{w_t}) = \sigma(-v_{w_c}^T v_{w_t})
\end{align}$$

So the objective is

$$L = - [ \log \sigma(v_{w_c}^T v_{w_t}) +  \sum_{\substack{i=1 \\ \tilde{w}_i \sim Q}}^K \log \sigma(v_{\tilde{w}_i}^T v_{w_t})]$$

In the above formulation, ${v_{\tilde{w}_i} \mid i = 1 . . . K}$ are sampled from $P_n(w)$. How to define $P_n(w)$? In the word2vec paper $P_n(w)$ defined as

$$P_n(w_i) = 1 - \sqrt{\frac{t}{freq(w_i)}} \;\; t \approx 10^{-5}$$

This distribution assigns lower probability for lower frequency words, higher probability for higher frequency words.

Hence, this distribution is sampled form a unigram distribution $U(w)$ raised to the  $\frac{3}{4}$rd power.

The unigram distribuiton is defined as

$$P_n(w)= \left(\frac{U(w)}{Z}\right)^\alpha$$

Or just by Andrew NG's definition:

$$P_n(w_i) = \frac{freq(w_i)^\frac{3}{4}}{\sum_{j=0}^M freq(w_j)^\frac{3}{4}}$$

Raising the unigram distribution $U(w)$ to the power of $\alpha$ has an effect of smoothing out the distribution. It attempts to combat the imbalance between common words and rare words by decreasing the probability of drawing common words, and increasing the probability drawing rare words.

```python
import numpy as np
unig_dist  = {'inzva': 0.023, 'deep': 0.12, 'learning': 0.34, 'the': 0.517}
sum(unig_dist.values())
```

    1.0

```python
alpha      = 3 / 4
noise_dist = {key: val ** alpha for key, val in unig_dist.items()}
Z = sum(noise_dist.values())
noise_dist_normalized = {key: val / Z for key, val in noise_dist.items()}
noise_dist_normalized
```

    {'inzva': 0.044813853132981724,
     'deep': 0.15470428538870049,
     'learning': 0.33785130228003507,
     'the': 0.4626305591982827}

```python
sum(noise_dist_normalized.values())
```

    1.0

```python
K = 10
np.random.choice(list(noise_dist_normalized.keys()), size=K, p=list(noise_dist_normalized.values()))
```

    array(['the', 'the', 'deep', 'learning', 'the', 'learning', 'the',
           'inzva', 'the', 'learning'], dtype='<U8')

More on negative sampling? [link](https://www.aclweb.org/anthology/D17-1037.pdf).

Deriving negative sampling loss function is trivial. Left to you.

## Hierarchical Softmax

hierarchical softmax is proposed to make the sum calculation faster with the help of a binary tree structure. The model uses a binary tree to represent all words in the
vocabulary. The $\mid V \mid$ words must be leaf units of the tree.

```python

```