```python
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pylab as plt
import seaborn as sns
sns.set_context('notebook')

RANDOM_SEED = 20090425
```

---

# Comparing Two Groups with a Continuous or Binary Outcome

Statistical inference is a process of learning from incomplete or imperfect (error-contaminated) data. Can account for this "imperfection" using either a sampling model or a measurement error model.

### Statistical hypothesis testing

The *de facto* standard for statistical inference is statistical hypothesis testing. The goal of hypothesis testing is to evaluate a **null hypothesis**. There are two possible outcomes:

- reject the null hypothesis
- fail to reject the null hypothesis

Rejection occurs when a chosen test statistic is higher than some pre-specified threshold valuel; non-rejection occurs otherwise.

Notice that neither outcome says anything about the quantity of interest, the **research hypothesis**.

Setting up a statistical test involves several subjective choices by the user that are rarely justified based on the problem or decision at hand:

- statistical test to use
- null hypothesis to test
- significance level

Choices are often based on arbitrary criteria, including "statistical tradition" (Johnson 1999). The resulting evidence is indirect, incomplete, and typically overstates the evidence against the null hypothesis (Goodman 1999).

Most importantly to applied users, the results of statistical hypothesis tests are very easy to misinterpret.

### Estimation

Instead of testing, a more informative and effective approach for inference is based on **estimation** (be it frequentist or Bayesian). That is, rather than testing whether two groups are different, we instead pursue an estimate of *how different* they are, which is fundamentally more informative.

Additionally, we include an estimate of **uncertainty** associated with that difference which includes uncertainty due to our lack of knowledge of the model parameters (*epistemic uncertainty*) and uncertainty due to the inherent stochasticity of the system (*aleatory uncertainty*).

# An Introduction to Bayesian Statistical Analysis

Though many of you will have taken a statistics course or two during your undergraduate (or graduate education, most of those who have will likely not have had a course in *Bayesian* statistics. Most introductory courses, particularly for non-statisticians, still do not cover Bayesian methods at all. Even today, Bayesian courses (similarly to statistical computing courses!) are typically tacked onto the curriculum, rather than being integrated into the program.

In fact, Bayesian statistics is not just a particular method, or even a class of methods; it is an entirely **different paradigm** for doing statistical analysis.

> Practical methods for making inferences from data using probability models for quantities we observe and about which we wish to learn.
*-- Gelman et al. 2013*

A Bayesian model is described by parameters, uncertainty in those parameters is described using probability distributions.

All conclusions from Bayesian statistical procedures are stated in terms of **probability statements**

This confers several benefits to the analyst, including:

- ease of interpretation, summarization of uncertainty
- can incorporate uncertainty in parent parameters
- easy to calculate summary statistics

### Bayesian vs Frequentist Statistics: *What's the difference?*

Any statistical inferece paradigm, Bayesian or otherwise, involves at least the following:

1. Some **unknown quantities** about which we are interested in learning or testing. We call these *parameters*.
2. Some **data** which have been observed, and hopefully contain information about.
3. One or more **models** that relate the data to the parameters, and is the instrument that is used to learn.

### The Frequentist World View

- The **data** that have been observed are considered **random**, because they are realizations of random processes, and hence will vary each time one goes to observe the system.
- Model **parameters** are considered **fixed**. A parameter's true value is uknown and fixed, and so we *condition* on them.

In mathematical notation, this implies a (very) general model of the following form:

\\[f(y | \theta)\\]

Here, the model \\(f\\) accepts data values \\(y\\) as an argument, conditional on particular values of \\(\theta\\).

Frequentist inference typically involves deriving **estimators** for the unknown parameters. Estimators are formulae that return estimates for particular estimands, as a function of data. They are selected based on some chosen optimality criterion, such as *unbiasedness*, *variance minimization*, or *efficiency*.

> For example, lets say that we have collected some data on the prevalence of autism spectrum disorder (ASD) in some defined population. Our sample includes \\(n\\) sampled children, \\(y\\) of them having been diagnosed with autism. A frequentist estimator of the prevalence \\(p\\) is:

>

> $$\hat{p} = \frac{y}{n}$$
>

> Why this particular function? Because it can be shown to be unbiased and minimum-variance.

It is important to note that, in a frequentist world, new estimators need to be derived for every estimand that is introduced.

### The Bayesian World View

- Data are considered **fixed**. They used to be random, but once they were written into your lab notebook/spreadsheet/IPython notebook they do not change.
- Model parameters themselves may not be random, but Bayesians use probability distribtutions to describe their uncertainty in parameter values, and are therefore treated as **random**. In some cases, it is useful to consider parameters as having been sampled from probability distributions.

This implies the following form:

\\[p(\theta | y)\\]

This formulation used to be referred to as ***inverse probability***, because it infers from observations to parameters, or from effects to causes.

Bayesians do not seek new estimators for every estimation problem they encounter. There is only one estimator for Bayesian inference: **Bayes' Formula**.

## Bayes' Formula

Now that we have some probability under our belt, we turn to Bayes' formula. While frequentist statistics uses different estimators for different problems, Bayes formula is the **only estimator** that Bayesians need to obtain estimates of unknown quantities that we care about.

The equation expresses how our belief about the value of \\(\theta\\), as expressed by the **prior distribution** \\(P(\theta)\\) is reallocated following the observation of the data \\(y\\).

The innocuous denominator \\(P(y)\\) usuallt cannot be computed directly, and is actually the expression in the numerator, integrated over all \\(\theta\\):

\\[Pr(\theta|y) = \frac{Pr(y|\theta)Pr(\theta)}{\int Pr(y|\theta)Pr(\theta) d\theta}\\]

The intractability of this integral is one of the factors that has contributed to the under-utilization of Bayesian methods by statisticians.

### Priors

Once considered a controversial aspect of Bayesian analysis, the prior distribution characterizes what is known about an unknown quantity before observing the data from the present study. Thus, it represents the information state of that parameter. It can be used to reflect the information obtained in previous studies, to constrain the parameter to plausible values, or to represent the population of possible parameter values, of which the current study's parameter value can be considered a sample.

### Likelihood functions

The likelihood represents the information in the observed data, and is used to update prior distributions to posterior distributions. This updating of belief is justified becuase of the **likelihood principle**, which states:

> Following observation of \\(y\\), the likelihood \\(L(\theta|y)\\) contains all experimental information from \\(y\\) about the unknown \\(\theta\\).

Bayesian analysis satisfies the likelihood principle because the posterior distribution's dependence on the data is **only through the likelihood**. In comparison, most frequentist inference procedures violate the likelihood principle, because inference will depend on the design of the trial or experiment.

Remember from the density estimation section that the likelihood is closely related to the probability density (or mass) function. The difference is that the likelihood varies the parameter while holding the observations constant, rather than *vice versa*.

## Bayesian Inference, in 3 Easy Steps

Gelman et al. (2013) describe the process of conducting Bayesian statistical analysis in 3 steps.

### Step 1: Specify a probability model

As was noted above, Bayesian statistics involves using probability models to solve problems. So, the first task is to *completely specify* the model in terms of probability distributions. This includes everything: unknown parameters, data, covariates, missing data, predictions. All must be assigned some probability density.

This step involves making choices.

- what is the form of the sampling distribution of the data?
- what form best describes our uncertainty in the unknown parameters?

### Discrete Random Variables

$$X = \{0,1\}$$

$$Y = \{\ldots,-2,-1,0,1,2,\ldots\}$$

**Probability Mass Function**:

For discrete $X$,

$$Pr(X=x) = f(x|\theta)$$

***e.g. Poisson distribution***

The Poisson distribution models unbounded counts:

$$Pr(X=x)=\frac{e^{-\lambda}\lambda^x}{x!}$$

* $X=\{0,1,2,\ldots\}$
* $\lambda > 0$

$$E(X) = \text{Var}(X) = \lambda$$

```python
from pymc3 import Poisson

x = Poisson.dist(mu=1)
samples = x.random(size=10000)
```

```python
samples.mean()
```

    1.0115

```python
plt.hist(samples, bins=len(set(samples)));
```

### Continuous Random Variables

$$X \in [0,1]$$

$$Y \in (-\infty, \infty)$$

**Probability Density Function**:

For continuous $X$,

$$Pr(x \le X \le x + dx) = f(x|\theta)dx \, \text{ as } \, dx \rightarrow 0$$

***e.g. normal distribution***

$$f(x) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left[-\frac{(x-\mu)^2}{2\sigma^2}\right]$$

* $X \in \mathbf{R}$
* $\mu \in \mathbf{R}$
* $\sigma>0$

$$\begin{align}E(X) &= \mu \cr
\text{Var}(X) &= \sigma^2 \end{align}$$

```python
from pymc3 import Normal

y = Normal.dist(mu=-2, sd=4)
samples = y.random(size=10000)
```

```python
samples.mean()
```

    -2.027323774069744

```python
samples.std()
```

    3.969166857847105

```python
plt.hist(samples);
```

### Step 2: Calculate a posterior distribution

The mathematical form \\(p(\theta | y)\\) that we associated with the Bayesian approach is referred to as a **posterior distribution**.

> posterior /pos·ter·i·or/ (pos-tēr´e-er) later in time; subsequent.

Why posterior? Because it tells us what we know about the unknown \\(\theta\\) *after* having observed \\(y\\).

This posterior distribution is formulated as a function of the probability model that was specified in Step 1. Usually, we can write it down but we cannot calculate it analytically. In fact, the difficulty inherent in calculating the posterior distribution for most models of interest is perhaps the major contributing factor for the lack of widespread adoption of Bayesian methods for data analysis. Various strategies for doing so comprise this tutorial.

**But**, once the posterior distribution is calculated, you get a lot for free:

- point estimates
- credible intervals
- quantiles
- predictions

### Step 3: Check your model

Though frequently ignored in practice, it is critical that the model and its outputs be assessed before using the outputs for inference. Models are specified based on assumptions that are largely unverifiable, so the least we can do is examine the output in detail, relative to the specified model and the data that were used to fit the model.

Specifically, we must ask:

- does the model fit data?
- are the conclusions reasonable?
- are the outputs sensitive to changes in model structure?

## Estimation for one group

Before we compare two groups using Bayesian analysis, let's start with an even simpler scenario: statistical inference for one group.

For this we will use Gelman et al.'s (2007) radon dataset. In this dataset the amount of the radioactive gas radon has been measured among different households in all counties of several states. Radon gas is known to be the highest cause of lung cancer in non-smokers. It is believed to be more strongly present in households containing a basement and to differ in amount present among types of soil.

>  the US EPA has set an action level of 4 pCi/L. At or above this level of radon, the EPA recommends you take corrective measures to reduce your exposure to radon gas.

Let's import the dataset:

```python
radon = pd.read_csv('../data/radon.csv', index_col=0)
radon.head()
```

      idnum
      state
      state2
      stfips
      zip
      region
      typebldg
      floor
      room
      basement
      ...
      pcterr
      adjwt
      dupflag
      zipflag
      cntyfips
      county
      fips
      Uppm
      county_code
      log_radon

      0
      5081.0
      MN
      MN
      27.0
      55735
      5.0
      1.0
      1.0
      3.0
      N
      ...
      9.7
      1146.499190
      1.0
      0.0
      1.0
      AITKIN
      27001.0
      0.502054
      0
      0.832909

      1
      5082.0
      MN
      MN
      27.0
      55748
      5.0
      1.0
      0.0
      4.0
      Y
      ...
      14.5
      471.366223
      0.0
      0.0
      1.0
      AITKIN
      27001.0
      0.502054
      0
      0.832909

      2
      5083.0
      MN
      MN
      27.0
      55748
      5.0
      1.0
      0.0
      4.0
      Y
      ...
      9.6
      433.316718
      0.0
      0.0
      1.0
      AITKIN
      27001.0
      0.502054
      0
      1.098612

      3
      5084.0
      MN
      MN
      27.0
      56469
      5.0
      1.0
      0.0
      4.0
      Y
      ...
      24.3
      461.623670
      0.0
      0.0
      1.0
      AITKIN
      27001.0
      0.502054
      0
      0.095310

      4
      5085.0
      MN
      MN
      27.0
      55011
      3.0
      1.0
      0.0
      4.0
      Y
      ...
      13.8
      433.316718
      0.0
      0.0
      3.0
      ANOKA
      27003.0
      0.428565
      1
      1.163151

5 rows × 29 columns

Let's focus on the (log) radon levels measured in a single county (Hennepin).

Suppose we are interested in:

- whether the mean log-radon value is greater than 4 pCi/L in Hennepin county
- the probability that any randomly-chosen household in Hennepin county has a reading of greater than 4

```python
hennepin_radon = radon.query('county=="HENNEPIN"').log_radon
sns.distplot(hennepin_radon)
```

```python
hennepin_radon.shape
```

    (105,)

### The model

Recall that the first step in Bayesian inference is specifying a **full probability model** for the problem.

This consists of:

- a likelihood function(s) for the observations
- priors for all unknown quantities

The measurements look approximately normal, so let's start by assuming a normal distribution as the sampling distribution (likelihood) for the data.

$$y_i \sim N(\mu, \sigma^2)$$

(don't worry, we can evaluate this assumption)

This implies that we have 2 unknowns in the model; the mean and standard deviation of the distribution.

#### Prior choice

How do we choose distributions to use as priors for these parameters?

There are several considerations:

- discrete vs continuous values
- the support of the variable
- the available prior information

While there may likely be prior information about the distribution of radon values, we will assume no prior knowledge, and specify a **diffuse** prior for each parameter.

Since the mean can take any real value (since it is on the log scale), we will use another normal distribution here, and specify a large variance to allow the possibility of very large or very small values:

$$\mu \sim N(0, 10^2)$$

For the standard deviation, we know that the true value must be positive (no negative variances!). I will choose a uniform prior bounded from below at zero and from above at a value that is sure to be higher than any plausible value the true standard deviation (on the log scale) could take.

$$\sigma \sim U(0, 10)$$

We can encode these in a Python model, using the PyMC3 package, as follows:

```python
from pymc3 import Model, Uniform

with Model() as radon_model:

    μ = Normal('μ', mu=0, sd=10)
    σ = Uniform('σ', 0, 10)
```

> ## Software
> Today there is an array of software choices for Bayesians, including both open source software (*e.g.*, Stan, PyMC, JAGS, emcee) and commercial (*e.g.*, SAS, Stata). These examples can be replicated in any of these environments.

All that remains is to add the likelihood, which takes $\mu$ and $\sigma$ as parameters, and the log-radon values as the set of observations:

```python
with radon_model:

    y = Normal('y', mu=μ, sd=σ, observed=hennepin_radon)
```

Now, we will fit the model using a numerical approach called **variational inference**. This will estimate the posterior distribution using an optimized approximation, and then draw samples from it.

```python
from pymc3 import fit
with radon_model:

    samples = fit(random_seed=RANDOM_SEED).sample(1000)
```

    Average Loss = 136.48: 100%|██████████| 10000/10000 [00:05<00:00, 1937.16it/s]
    Finished [100%]: Average Loss = 136.37

```python
from pymc3 import plot_posterior

plot_posterior(samples, varnames=['μ'], ref_val=np.log(4), color='LightSeaGreen');
```

The plot shows the posterior distribution of $\mu$, along with an estimate of the 95% posterior **credible interval**.

The output

    83.1% < 1.38629 < 16.9%%

informs us that the probability of $\mu$ being less than $\log(4)$ is 83.1%% and the corresponding probability of being greater than $\log(4)$ is 16.9%.

> The posterior probability that the mean level of household radon in Henneprin County is greater than 4 pCi/L is 0.17.

### Prediction

What is the probability that a given household has a log-radon measurement larger than one? To answer this, we make use of the **posterior predictive distribution**.

$$p(z |y) = \int_{\theta} p(z |\theta) p(\theta | y) d\theta$$

where here $z$ is the predicted value and y is the data used to fit the model.

We can estimate this from the posterior samples of the parameters in the model.

```python
mus = samples['μ']
sigmas = samples['σ']
```

```python
radon_samples = Normal.dist(mus, sigmas).random()
```

```python
(radon_samples > np.log(4)).mean()
```

    0.463

> The posterior probability that a randomly-selected household in Henneprin County contains radon levels in excess of 4 pCi/L is 0.48.

### Model checking

But, ***how do we know this model is any good?***

Its important to check the fit of the model, to see if its assumptions are reasonable. One way to do this is to perform **posterior predictive checks**. This involves generating simulated data using the model that you built, and comparing that data to the observed data.

One can choose a particular statistic to compare, such as tail probabilities or quartiles, but here it is useful to compare them graphically.

We already have these simulations from the previous exercise!

```python
sns.distplot(radon_samples, label='simulated')
sns.distplot(hennepin_radon, label='observed')
plt.legend()
```

### Prior sensitivity

Its also important to check the sensitivity of your choice of priors to the resulting inference.

Here is the same model, but with drastically different (though still uninformative) priors specified:

```python
from pymc3 import Flat, HalfCauchy

with Model() as prior_sensitivity:

    μ = Flat('μ')
    σ = HalfCauchy('σ', 5)

    dist = Normal('dist', mu=μ, sd=σ, observed=hennepin_radon)

    sensitivity_samples = fit(random_seed=RANDOM_SEED).sample(1000)
```

    Average Loss = 123.98: 100%|██████████| 10000/10000 [00:04<00:00, 2201.42it/s]
    Finished [100%]: Average Loss = 123.94

```python
plot_posterior(sensitivity_samples, varnames=['μ'], ref_val=np.log(4), color='LightSeaGreen');
```

Here is the original model for comparison:

```python
plot_posterior(samples, varnames=['μ'], ref_val=np.log(4), color='LightSeaGreen');
```

## Two Groups with Continiuous Outcome

To illustrate how this Bayesian estimation approach works in practice, we will use a fictitious example from Kruschke (2012) concerning the evaluation of a clinical trial for drug evaluation. The trial aims to evaluate the efficacy of a "smart drug" that is supposed to increase intelligence by comparing IQ scores of individuals in a treatment arm (those receiving the drug) to those in a control arm (those recieving a placebo). There are 47 individuals and 42 individuals in the treatment and control arms, respectively.

```python
drug = pd.DataFrame(dict(iq=(101,100,102,104,102,97,105,105,98,101,100,123,105,103,100,95,102,106,
        109,102,82,102,100,102,102,101,102,102,103,103,97,97,103,101,97,104,
        96,103,124,101,101,100,101,101,104,100,101),
                         group='drug'))
placebo = pd.DataFrame(dict(iq=(99,101,100,101,102,100,97,101,104,101,102,102,100,105,88,101,100,
           104,100,100,100,101,102,103,97,101,101,100,101,99,101,100,100,
           101,100,99,101,100,102,99,100,99),
                            group='placebo'))

trial_data = pd.concat([drug, placebo], ignore_index=True)
sns.set()
trial_data.hist('iq', by='group');
```

Since there appear to be extreme ("outlier") values in the data, we will choose a Student-t distribution to describe the distributions of the scores in each group. This sampling distribution adds **robustness** to the analysis, as a T distribution is less sensitive to outlier observations, relative to a normal distribution.

The three-parameter Student-t distribution allows for the specification of a mean $\mu$, a precision (inverse-variance) $\lambda$ and a degrees-of-freedom parameter $\nu$:

$$f(x|\mu,\lambda,\nu) = \frac{\Gamma(\frac{\nu + 1}{2})}{\Gamma(\frac{\nu}{2})} \left(\frac{\lambda}{\pi\nu}\right)^{\frac{1}{2}} \left[1+\frac{\lambda(x-\mu)^2}{\nu}\right]^{-\frac{\nu+1}{2}}$$

the degrees-of-freedom parameter essentially specifies the "normality" of the data, since larger values of $\nu$ make the distribution converge to a normal distribution, while small values (close to zero) result in heavier tails.

Thus, the likelihood functions of our model are specified as follows:

$$\begin{align}
y^{(drug)}_i &\sim T(\nu, \mu_1, \sigma_1) \\
y^{(placebo)}_i &\sim T(\nu, \mu_2, \sigma_2)
\end{align}$$

As a simplifying assumption, we will assume that the degree of normality $\nu$ is the same for both groups.

### Exercise

Draw 10000 samples from a Student-T distribution (`StudentT` in PyMC3) with parameter `nu=3` and compare the distribution of these values to a similar number of draws from a Normal distribution with parameters `mu=0` and `sd=1`.

```python
from pymc3 import StudentT

t = StudentT.dist(nu=3).random(size=1000)
n = Normal.dist(0, 1).random(size=1000)
```

```python
sns.distplot(t, label='Student-T')
sns.distplot(n, label='Normal')
plt.legend()
plt.xlim(-10,10);
```

### Prior choice

Since the means are real-valued, we will apply normal priors. Since we know something about the population distribution of IQ values, we will center the priors at 100, and use a standard deviation that is more than wide enough to account for plausible deviations from this population mean:

$$\mu_k \sim N(100, 10^2)$$

```python
with Model() as drug_model:

    μ_0 = Normal('μ_0', 100, sd=10)
    μ_1 = Normal('μ_1', 100, sd=10)
```

Similarly, we will use a uniform prior for the standard deviations, with an upper bound of 20.

```python
with drug_model:
    σ_0 = Uniform('σ_0', lower=0, upper=20)
    σ_1 = Uniform('σ_1', lower=0, upper=20)
```

For the degrees-of-freedom parameter $\nu$, we will use an **exponential** distribution with a mean of 30; this allocates high prior probability over the regions of the parameter that describe the range from normal to heavy-tailed data under the Student-T distribution.

```python
from pymc3 import Exponential

with drug_model:
    ν = Exponential('ν_minus_one', 1/29.) + 1

```

```python
sns.distplot(Exponential.dist(1/29).random(size=10000), kde=False);
```

```python
from pymc3 import StudentT

with drug_model:

    drug_like = StudentT('drug_like', nu=ν, mu=μ_1, lam=σ_1**-2, observed=drug.iq)
    placebo_like = StudentT('placebo_like', nu=ν, mu=μ_0, lam=σ_0**-2, observed=placebo.iq)
```

Now that the model is fully specified, we can turn our attention to tracking the posterior quantities of interest. Namely, we can calculate the difference in means between the drug and placebo groups.

As a joint measure of the groups, we will also estimate the "effect size", which is the difference in means scaled by the pooled estimates of standard deviation. This quantity can be harder to interpret, since it is no longer in the same units as our data, but it is a function of all four estimated parameters.

```python
from pymc3 import Deterministic

with drug_model:

    diff_of_means = Deterministic('difference of means', μ_1 - μ_0)

    effect_size = Deterministic('effect size',
                        diff_of_means / np.sqrt((σ_1**2 + σ_0**2) / 2))

```

```python
with drug_model:

    drug_trace = fit(random_seed=RANDOM_SEED).sample(1000)
```

    Average Loss = 245.2: 100%|██████████| 10000/10000 [00:05<00:00, 1863.59it/s]
    Finished [100%]: Average Loss = 245.13

```python
plot_posterior(drug_trace[100:],
                varnames=['μ_0', 'μ_1', 'σ_0', 'σ_1', 'ν_minus_one'],
                color='#87ceeb');
```

```python
plot_posterior(drug_trace[100:],
          varnames=['difference of means', 'effect size'],
          ref_val=0,
          color='#87ceeb');
```

> The posterior probability that the mean IQ of subjects in the treatment group is greater than that of the control group is 0.99.

### Exercise

Load the `nashville_precip.txt` dataset. Build a model to compare rainfall in January and July.

- What's the probability that the expected rainfall in January is larger than in July?
- What's the probability that January rainfall exceeds July rainfall in a given year?

```python
nash_precip = pd.read_table('../data/nashville_precip.txt',
                            delimiter='\s+', na_values='NA', index_col=0)
nash_precip.head()
```

      Jan
      Feb
      Mar
      Apr
      May
      Jun
      Jul
      Aug
      Sep
      Oct
      Nov
      Dec

      Year

      1871
      2.76
      4.58
      5.01
      4.13
      3.30
      2.98
      1.58
      2.36
      0.95
      1.31
      2.13
      1.65

      1872
      2.32
      2.11
      3.14
      5.91
      3.09
      5.17
      6.10
      1.65
      4.50
      1.58
      2.25
      2.38

      1873
      2.96
      7.14
      4.11
      3.59
      6.31
      4.20
      4.63
      2.36
      1.81
      4.28
      4.36
      5.94

      1874
      5.22
      9.23
      5.36
      11.84
      1.49
      2.87
      2.65
      3.52
      3.12
      2.63
      6.12
      4.19

      1875
      6.15
      3.06
      8.14
      4.22
      1.73
      5.63
      8.12
      1.60
      3.79
      1.25
      5.46
      4.30

```python
# %load ../exercises/rainfall.py
```

## Two Groups with Binary Outcome

Now that we have seen how to generalize normally-distributed data to another distribution, we are equipped to analyze other data types. Binary outcomes are common in clinical research:

- survival/death
- true/false
- presence/absence
- positive/negative

> *Never, ever dichotomize continuous or ordinal variables prior to statistical analysis*

In practice, binary outcomes are encoded as ones (for event occurrences) and zeros (for non-occurrence). A single binary variable is distributed as a **Bernoulli** random variable:

$$f(x \mid p) = p^{x} (1-p)^{1-x}$$

Such events are sometimes reported as sums of individual events, such as the number of individuals in a group who test positive for a condition of interest. Sums of Bernoulli events are distributed as **binomial** random variables.

$$f(x \mid n, p) = \binom{n}{x} p^x (1-p)^{n-x}$$

The parameter in both models is $p$, the probability of the occurrence of an event. In terms of inference, we are typically interested in whether $p$ is larger or smaller in one group relative to another.

To demonstrate the comparison of two groups with binary outcomes using Bayesian inference, we will use a sample pediatric dataset. Data on 671 infants with very low (<1600 grams) birth weight from 1981-87 were collected at Duke University Medical Center. Of interest is the relationship between the outcome intra-ventricular hemorrhage (IVH) and predictor such as birth weight, gestational age, presence of pneumothorax and mode of delivery.

```python
vlbw = pd.read_csv('../data/vlbw.csv', index_col=0).dropna(axis=0, subset=['ivh', 'pneumo'])
vlbw.head()
```

      birth
      exit
      hospstay
      lowph
      pltct
      race
      bwt
      gest
      inout
      twn
      ...
      vent
      pneumo
      pda
      cld
      pvh
      ivh
      ipe
      year
      sex
      dead

      5
      81.593002
      81.598999
      2.0
      6.969997
      54.0
      black
      925.0
      28.0
      born at Duke
      0.0
      ...
      1.0
      1.0
      0.0
      0.0
      definite
      definite
      NaN
      81.594055
      female
      1

      6
      81.601997
      81.771004
      62.0
      7.189999
      NaN
      white
      940.0
      28.0
      born at Duke
      0.0
      ...
      1.0
      0.0
      0.0
      0.0
      absent
      absent
      absent
      81.602295
      female
      0

      13
      81.683998
      81.853996
      62.0
      7.179996
      182.0
      black
      1110.0
      28.0
      born at Duke
      0.0
      ...
      0.0
      1.0
      0.0
      1.0
      absent
      absent
      absent
      81.684448
      male
      0

      14
      81.689003
      81.877998
      69.0
      7.419998
      361.0
      white
      1180.0
      28.0
      born at Duke
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      absent
      absent
      absent
      81.689880
      male
      0

      16
      81.696999
      81.952004
      93.0
      7.239998
      255.0
      black
      770.0
      26.0
      born at Duke
      0.0
      ...
      1.0
      0.0
      0.0
      1.0
      absent
      absent
      absent
      81.698120
      male
      0

5 rows × 26 columns

To demonstrate binary data analysis, we will try to estimate the difference between the probability of an intra-ventricular hemorrhage for infants with a pneumothorax.

```python
pd.crosstab(vlbw.ivh, vlbw.pneumo)
```

      pneumo
      0.0
      1.0

      ivh

      absent
      359
      73

      definite
      45
      30

      possible
      6
      4

We will create a binary outcome by combining `definite` and `possible` into a single outcome.

```python
ivh = vlbw.ivh.isin(['definite', 'possible']).astype(int).values
x = vlbw.pneumo.astype(int).values
```

### Prior choice

What should we choose as a prior distribution for $p$?

We could stick with a normal distribution, but note that the value of $p$ is **constrained** by the laws of probability. Namely, we cannot have values smaller than zero nor larger than one. So, choosing a normal distribution will result in ascribing positive probability to unsupported values of the parameter. In many cases, this will still work in practice, but will be inefficient for calculating the posterior and will not accurately represent the prior information about the parameter.

A common choice in this context is the **beta distribution**, a continuous distribution with 2 parameters and whose support is on the unit interval:

$$ f(x \mid \alpha, \beta) = \frac{x^{\alpha - 1} (1 - x)^{\beta - 1}}{B(\alpha, \beta)}$$

- Support: $x \in (0, 1)$
- Mean: $\dfrac{\alpha}{\alpha + \beta}$
- Variance: $\dfrac{\alpha \beta}{(\alpha+\beta)^2(\alpha+\beta+1)}$

```python
from pymc3 import Beta

params = (5, 1), (1, 3), (5, 5), (0.5, 0.5), (1, 1)

fig, axes = plt.subplots(1, len(params), figsize=(14, 4), sharey=True)
for ax, (alpha, beta) in zip(axes, params):
    sns.distplot(Beta.dist(alpha, beta).random(size=10000), ax=ax, kde=False)
    ax.set_xlim(0, 1)
    ax.set_title(r'$\alpha={0}, \beta={1}$'.format(alpha, beta));
```

So let's use a beta distribution to model our prior knowledge of the probabilities for both groups. Setting $\alpha = \beta = 1$ will result in a uniform distribution of prior mass:

```python
with Model() as ivh_model:

    p = Beta('p', 1, 1, shape=2)
```

We can now use `p` as the parameter of our Bernoulli likelihood. Here, `x` is a vector of zeros an ones, which will extract the approproate group probability for each subject:

```python
from pymc3 import Bernoulli

with ivh_model:

    bb_like = Bernoulli('bb_like', p=p[x], observed=ivh)
```

Finally, since we are interested in the difference between the probabilities, we will keep track of this difference:

```python
with ivh_model:

    p_diff = Deterministic('p_diff', p[1] - p[0])
```

```python
with ivh_model:
    ivh_trace = fit(random_seed=RANDOM_SEED).sample(1000)
```

    Average Loss = 226.28: 100%|██████████| 10000/10000 [00:00<00:00, 13352.85it/s]
    Finished [100%]: Average Loss = 226.28

```python
plot_posterior(ivh_trace[100:], varnames=['p'], color='#87ceeb');
```

We can see that the probability that `p` is larger for the pneumothorax with probability one.

```python
plot_posterior(ivh_trace[100:], varnames=['p_diff'], ref_val=0, color='#87ceeb');
```

## References and Resources

- Goodman, S. N. (1999). Toward evidence-based medical statistics. 1: The P value fallacy. Annals of Internal Medicine, 130(12), 995–1004. http://doi.org/10.7326/0003-4819-130-12-199906150-00008
- Johnson, D. (1999). The insignificance of statistical significance testing. Journal of Wildlife Management, 63(3), 763–772.
- Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian Data Analysis, Third Edition. CRC Press.
- Kruschke, J.K. *Doing Bayesian Data Analysis, Second Edition: A Tutorial with R, JAGS, and Stan.* 2015. Academic Press / Elsevier.
- O'Shea M, Savitz D.A., Hage M.L., Feinstein K.A.: *Prenatal events and the risk of subependymal / intraventricular haemorrhage in very low birth weight neonates*. **Paediatric and Perinatal Epdiemiology** 1992;6:352-362

notebooks/2. Basic Bayesian Inference.ipynb


```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import matplotlib
import seaborn as sns
%matplotlib inline
```

```python
matplotlib.rcParams['figure.figsize'] = (10, 8) # set default figure size, 10in by 8in
```

This week, you will be learning about unsupervised learning. While supervised learning algorithms need labeled examples (x,y), unsupervised learning algorithms need only the input (x). You will learn about clustering—which is used for market segmentation, text summarization, among many other applications.

# Video W8 01: Unsupervised Learning

[YouTube Video Link](https://www.youtube.com/watch?v=PK5JsJZd1Uk&index=77&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW)

In an unsupervised learning problem, we are given data that does not have any labels associated with it.  So what we want
from unsupervised learning algorithms is to discover some sort of structure or organization or pattern in our data.  For example,
the easiest type of structure to understand is to try and find clusters in the data of items that appear related.  Such clusters
can be useful in many applications to identify and process the members of a cluster in some specific way, such as clusters of
different types customers and their buying habits.

Up to this point we have been studying supervised learning methods.  In supervised learning, for example to
perform a classification task, we are given a traing set of data, and all of the $m$ samples in the training set are
labeled:

\begin{equation}
\text{Training set:} \{ (x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), (x^{(3)}, y^{(3)}), \ldots, (x^{(m)}, y^{(m)}) \}
\end{equation}

Here the $y^{(m)}$ are the labels for the data.  For example if we have the data:

```python
x = np.array([[0.5, 0.5],
              [1.0, 0.5],
              [0.75, 1.0],
              [1.9, 0.25],
              [1.6, 0.75],
              [1.25, 1.25],
              [0.5, 1.6],
              [0.5, 2.25],
              [3.1, 1.1],
              [2.9, 1.5],
              [2.1, 2.1],
              [2.1, 2.75],
              [1.5, 3.1],
              [3.5, 1.9],
              [3.0, 2.1],
              [3.0, 3.0],
              [2.0, 3.5],
              [2.5, 3.5]])
y = np.array([0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])

neg_idx = np.where(y == 0)
pos_idx = np.where(y == 1)

# plot the example figure
plt.figure(figsize=(10,10))

# plot the points in our two categories, y=0 and y=1, using markers to indicated
# the category or output
neg_handle = plt.plot(x[neg_idx,0], x[neg_idx,1], 'bo', markersize=8, fillstyle='none', markeredgewidth=1, label='0 negative class')
pos_handle = plt.plot(x[pos_idx,0], x[pos_idx,1], 'rx', markersize=8, markeredgewidth=1, label='1 positive class')

# add some labels and titles
plt.axis([0, 4, 0, 4])
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
plt.title('Supervised Learning: Classification of labeled data');
plt.legend([neg_handle[0], pos_handle[0]], ['0 negative class', '1 positive class']);
```

Here the $y$ vector holds the binary classification labels, and the data we are given to train with is
in one of two classes, $0$ negative class or $1$ positive class.

For unsupervised learning we are given $m$ unlabeled samples of data to use:

\begin{equation}
\text{Training set:} \{ x^{(1)}, x^{(2)}, x^{(3)}, \ldots, x^{(m)} \}
\end{equation}

```python
# plot the example figure
plt.figure(figsize=(10,10))

# plot the points in our unlabeled data
plt.plot(x[:,0], x[:,1], 'ko', markersize=8, fillstyle='full')

# add some labels and titles
plt.axis([0, 4, 0, 4])
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
plt.title('Unsupervised Learning');
```

In unsupervised learning, we give the algorithm some data and we ask the algorithm to "find some structure"
in the data.

For example, given this data set, we might want the algorithm to find some likely clusters of the data, points that
may be similar or of related categories.

An algorithm that finds clusters is called a clustering algorithm.  The previous might have 2 clusters, or there
might be even 3 or 4 good clusters.

## Applications of Clustering

- marked segmentation
- social network analysis (coherent groups of people that form organically)
- organize computing clusters
- astronomical data analysis

# Video W8 02: K Means Algorithm

[YouTube Video Link](https://www.youtube.com/watch?v=6u19018FeHg&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW&index=78)

The K-means algorithm is an example of a clustering unsupervised learning algorithm.  It is probably the simplest clustering
algorithm, but it is still quite effective.  Thus it is still one of the most popular and most used clustering algorithms.

K-means is an iterative algorithm.  We start by specifying how many clusters (e.g. K clusters) we want the algorithm to
discover.  More formally, we can define the **K-means algorithm**

- Input:
  - $K$ (number of clusters)
  - Training set of $m$ inputs $\{x^{(1)}, x^{(2)}, \ldots, x^{(m)}\}$
- Where $x^{(i)} \in \mathbb{R}^n$ (we drop the $x_0 = 1$ convention)

And the **K-means algorithm** pseudocode

- Randomly initialize $K$ cluster centroids $\mu_1, \mu_2, \ldots, \mu_K \in \mathbb{R}^n$

- Repeat {
  - for $i = 1$ to $m$
    - $c^{(i)}$ := index (from 1 to $K$) of cluster centroid closest to $x^{(i)}$
  - for $k = 1$ to $K$
    - $\mu_k$ := average (mean) of points assigned to cluster $k$
- }

This basic algorithm for K-means clustering is really fairly simple, and it will help to understand it even further if we make
a quick and basic implementation of the algorithm in Python code.  First of all, we will read in a small simple set of
data that appears to be well separated into 2 clusters.  This dataset has $m = 32$ examples.  The dataset has only 2 features
$n = 2$, thus all of the points are in 2 dimensional space.

```python
#from sklearn.datasets import make_blobs
#X, y = make_blobs(n_samples = 10, n_features=2, centers=2, cluster_std = 0.8, center_box=(2, 5))
#np.savetxt('../data/lect-11-ex1data.csv', X, delimiter=',')
X = np.loadtxt('../../data/lect-11-ex1data.csv', delimiter=',')

plt.plot(X[:, 0], X[:, 1], 'go')
plt.xlabel(r'$x_1$', fontsize=20)
plt.ylabel(r'$x_2$', fontsize=20);
```

The first step in the K-means algorithm is to randomly initialize a set of centroids.  We usually initialze the centroids to be
within the ranges of the data set.  So for example, if we find the minimum and maximum values for the data for each of the dimensions,
we can use this to randomly initialize our centroids.  In this case, we are going to try and find $K = 2$ clusters, so we want to
create two centroids within the range of our data:

```python
# The number of clusters K we will find
K = 2

# The number of training data points, and the number of dimensions of our data set
m, n = X.shape

# randomly initialize K centroids
min_x1, max_x1 = min(X[:, 0]), max(X[:, 0])
min_x2, max_x2 = min(X[:, 1]), max(X[:, 1])
print(min_x1, max_x1)
print(min_x2, max_x2)

# create K centroids mu, where each point is randomly chosen within the range of the data
mu = np.zeros( (K, n) )
for k in range(K):
    mu[k, 0] = np.random.uniform(low = min_x1, high = max_x1)
    mu[k, 1] = np.random.uniform(low = min_x2, high = max_x2)

# visualize the original data, with our randomly chosen initial centroid points
plt.plot(X[:, 0], X[:, 1], 'go', label='training data')
plt.plot(mu[:,0], mu[:,1], 'rx', markersize=15, label='centroids')
plt.xlabel(r'$x_1$', fontsize=20)
plt.ylabel(r'$x_2$', fontsize=20)
plt.legend();
```

In the video, the first step in the iterative part of the K-means algorithm is to assign each of the training data points to
one of our $\mu$ clusters.  As shown in the video, we do this by calculating the distance between each data point and our
two centroids, and we assign the point to the closest centroid.  The measure used to calculate the distance can actually be
calculated in different ways.  The simplest is to use the eucledian distance.  And since the distance can be negative
depending on the order we evaluate points when calculating the distance, we usually take the square of the distance so that
all values are positive (e.g. we get the magnitude of the distance), and we can thus compare directly and find the minimum.

$$
\underset{k}{\textrm{min}} \;\; \| x^{(i)} - \mu_k  \|^2
$$

For example, lets calculate the distance between the first training data point and the two randomly generated centroids.  Keep
in mind that in Python, our arrays are indexed starting at 0, so the first training data example will be at $i = 0$.  Also, with
$K = 2$ cluster centroids, the $k$ clusters will range from $0$ to $1$.

Lets start by defining a function that will take 2 $n$ dimensional points, and calculate the square of the distance between
the two points:

```python
def distance(x, y):
    # calculate the square of the distance between 2 n dimensional points (passed as numpy arrays)
    # eucledian distance is sqrt( (x_1 - y_1)**2.0 + (x_2 - y_2)**2.0 ), but we then square this, so
    # we simply need the sum of the differences squared
    return np.sum( (x - y)**2.0 )
```

```python
# distance from the 0th training example and 0th cluster
i = 0
k = 0
print("%d training example: (%f, %f)" % (i, X[i,0], X[i,1]))
print("%d cluster centroid: (%f, %f)" % (k, mu[k,0], mu[k,1]))
print("square distance between input %d and cluster %d: %f" % (i, k, distance(X[i,:], mu[k,:])))

# distance from the 0th training example and 1th cluster
i = 0
k = 1
print("")
print("%d training example: (%f, %f)" % (i, X[i,0], X[i,1]))
print("%d cluster centroid: (%f, %f)" % (k, mu[k,0], mu[k,1]))
print("square distance between input %d and cluster %d: %f" % (i, k, distance(X[i,:], mu[k,:])))
```

    0 training example: (-3.500000, -5.000000)
    0 cluster centroid: (2.652079, 3.691993)
    square distance between input 0 and cluster 0: 113.398815

    0 training example: (-3.500000, -5.000000)
    1 cluster centroid: (2.742979, -0.253101)
    square distance between input 0 and cluster 1: 61.507844

By the way, the above function for calculating the distance basically does the same thing as calculating
the norm between the two point vectors

```python
def distance_norm(x, y):
    # calculate the square of the distance between 2 n dimensional points (passed as numpy arrays)
    # using the linear algebra vector norm to calculate the distance
    return np.linalg.norm(x - y)**2.0

```

```python
# distance from the 0th training example and 0th cluster
i = 0
k = 0
print("%d training example: (%f, %f)" % (i, X[i,0], X[i,1]))
print("%d cluster centroid: (%f, %f)" % (k, mu[k,0], mu[k,1]))
print("square distance between input %d and cluster %d: %f" % (i, k, distance_norm(X[i,:], mu[k,:])))

# distance from the 0th training example and 1th cluster
i = 0
k = 1
print("")
print("%d training example: (%f, %f)" % (i, X[i,0], X[i,1]))
print("%d cluster centroid: (%f, %f)" % (k, mu[k,0], mu[k,1]))
print("square distance between input %d and cluster %d: %f" % (i, k, distance_norm(X[i,:], mu[k,:])))
```

    0 training example: (-3.500000, -5.000000)
    0 cluster centroid: (2.652079, 3.691993)
    square distance between input 0 and cluster 0: 113.398815

    0 training example: (-3.500000, -5.000000)
    1 cluster centroid: (2.742979, -0.253101)
    square distance between input 0 and cluster 1: 61.507844

The first part of the iterative algorithm is to calculate such distances between each training data item and every centroid,
find the minimum, and assign the training data item to be in the cluster whose centroid it is closest too.  So for example,
we can determine the closest centroid for each training data point like this

```python
# This array will hold the index k of the cluster centroid each training data point is assigned too
c = np.zeros(m)

# for each training data point i
for i in range(m):
    # determine distance to cluster 0
    min_distance = distance_norm(X[i,:], mu[0, :])
    c[i] = 0
    # find out if any other cluster centroid k=1,...K is closer
    for k in range(1, K):
        another_distance = distance_norm(X[i,:], mu[k, :])
        if another_distance < min_distance:
            min_distance = another_distance
            c[i] = k

# the above loop represents the code needed to assign each point to the closest cluster mu.  Here were the
# clusters that each point was assigned to
for i in range(m):
    print("point x[%d] in cluster: %d" % (i, c[i]))

# lets visualize the resulting assignments of the points to the current cluster centroids
cluster_0 = np.where(c == 0)[0]
cluster_1 = np.where(c == 1)[0]

#plt.figure(figsize=(8,16))
ax = plt.gca()
ax.set_aspect('equal')

plt.plot(X[cluster_0, 0], X[cluster_0, 1], 'ro', label='assg cluster 0')
plt.plot(mu[0,0], mu[0,1], 'rx', markersize=15)
plt.plot(X[cluster_1, 0], X[cluster_1, 1], 'bo', label='assg cluster 1')
plt.plot(mu[1,0], mu[1,1], 'bx', markersize=15, label='centroids')
plt.xlabel(r'$x_1$', fontsize=20)
plt.ylabel(r'$x_2$', fontsize=20)
plt.legend(loc=2)
plt.axis([-5, 7, -7, 7]);
```

Once we have found which centroid each training data item is closest to, it is time to update the centroids.  We do this by calculating
new mu centroids which are simply the average of all of the points assigned to that centroid.  For example, we can use numpy vector
operations and the c array to find and average all of the points assigned to cluster $k = 0$

```python
cluster_0 = np.where(c == 0)[0]
print(X[cluster_0])
print(np.mean(X[cluster_0], axis=0))
```

    [[0.   3.8 ]
     [0.4  1.9 ]
     [0.5  2.  ]
     [0.55 2.3 ]
     [0.6  2.5 ]
     [0.55 3.  ]
     [0.5  4.2 ]
     [2.6  2.9 ]
     [2.7  5.  ]
     [3.   2.5 ]
     [6.   2.  ]]
    [1.58181818 2.91818182]

We can use the above idea to recalculate all $K$ centroids:

```python
# recalculate all cluster centroids
for k in range(K):
    cluster_pts = np.where(c == k)[0]
    mu[k] = np.mean(X[cluster_pts], axis=0)

# show the resulting new cluster centroids
print(mu)

# visualize the new centroid locations in relation to the assigned points in the clusters
cluster_0 = np.where(c == 0)[0]
cluster_1 = np.where(c == 1)[0]

#plt.figure(figsize=(8,16))
ax = plt.gca()
ax.set_aspect('equal')

plt.plot(X[cluster_0, 0], X[cluster_0, 1], 'ro', label='assg cluster 0')
plt.plot(mu[0,0], mu[0,1], 'rx', markersize=15)
plt.plot(X[cluster_1, 0], X[cluster_1, 1], 'bo', label='assg cluster 1')
plt.plot(mu[1,0], mu[1,1], 'bx', markersize=15, label='centroids')
plt.xlabel(r'$x_1$', fontsize=20)
plt.ylabel(r'$x_2$', fontsize=20)
plt.legend(loc=1)
plt.axis([-5, 7, -7, 7]);
```

The previous steps to randomly initialize a set of centroids, then repeatedly assign points to closest centroid and move the
centroids can easily be made into a function that performs the basic K-means algorithm.  We will leave this as an exercise
for the student for now to try and bring these pieces together.

## K-means for non-separated clusters

The previous example(s) had data that looked well separated.  However you can still run k-means clustering on
data that is not so clearnly separated.  Often analysis of a market to do market segmentation can benefit from
doing a k-means clustering even on data without real clear segmented groups.

# Video W8 03: Optimization Objective

[YouTube Video Link](https://www.youtube.com/watch?v=omcDeBY4lGE&index=79&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW)

As this video discusses, we can formally define a cost function and optimization objective for the K-means algorithm.
The cost function is high when the points in a cluster are far away from the cluster centroid, and it will be lower
when the points in a cluster are close to the cluster centroid:

$$
J(c^{(1)}, \ldots, c^{(m)}, \mu_1, \ldots, \mu_K) = \frac{1}{m} \sum_{i=1}^m \| x^{(i)} - \mu_{c^{(i)}} \|^2
$$

Thus for optimization we are trying to assign our points to clusters, which define the cluster centroids, that minimizes
this cost objective function:

$$
\underset{c^{(1)}, \ldots, c^{(m)}, \\ \mu_1, \ldots, \mu_K}{\textrm{min}} \;\; J(c^{(1)}, \ldots, c^{(m)}, \mu_1, \ldots, \mu_K)
$$

In words, k-means is trying to find parameters $c^{(i)}$ and $\mu_k$ that minimizes the sum of the squared distances
between each point and its assigned centroid.  It should be obvious from the previous pseudocode that we
assign the points to the centroid that it is closest too, thus we are minimizing the distance from each point to the
current set of centroids.  K-means is another example of a greedy algorithm.  But in this case, the next step
after cluster assignment of recalculating centroid locations based on the current assigned points in the cluster, can
be shown to lead to $\mu_k$ points that will end up having minimal overall summed up costs, once the algorithm
has converged.

So in other words, the cluster assignment step minimizes the cost function of the centroids with respect to
the cluster assignments, while holding the centroids fixed.   Then in the second step to move the centroids,
it chooses the values of $\mu_k$ that minimize the cost function with respect to the changing centroids.

# Video W8 04: Random Initialization

[YouTube Video Link](https://www.youtube.com/watch?v=wniLibHEE2Y&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW&index=80)

Above we showed simply picking $K$ random points within the range of the training data examples in order to randomly choose the
initial cluster centroids.  In this video, the instructor illustrates a different method, which usually works a bit better, and
it is actually a bit easier to understand.  If we want to discover $K$ cluster, we can simply choose $K$ of our input training
data points at random to be our initial centroids.  We know that by picking 2 of the input data points that the centroids will
automatically be within the range of the training data.  So for example, in Python, we could choose K points at random
to be our centroids like this:

```python
# choose the number of clusters we will be creating
K = 2

# this will choose 2 indexes in range 0 to m-1, that we will use as our initial points for the mu centroids
# NOTE: in the next function, the replace=False ensures that the choice() function will not pick the same random
# index.
random_pts = np.random.choice(m, size=K, replace=False)
print(random_pts)

mu = X[random_pts]
print(mu)
```

    [10 12]
    [[-1.7 -1.7]
     [-0.2 -1.8]]

This method is the recommended way to choose the initial K centroid methods, and is what will normally be used
by a K-means library like for example the scikit-learn K-means implementation.

The optimization cost function defined by K-means is not guaranteed to have only 1 global minimum (unlike
some previous cost functions we defined).  Thus when you run a K-means with random initial points, the
clusters the algorithm finds and converges on can be different depending on the random starting locations
picked.  Thus K-means is not deterministic, you can get different clustering results each time you run
the algorithm.

One solution to this is to run K-means multiple times with different starting random initializations.  At the
end of K-means clustering, once the algorithm has converged, you can find the final cost of the discovered
clusters.  Usually if you are hitting local minimum when clustering, if you run multiple times you can compare
the final costs of the multiple runs, and usually the lower or lowest final costs achieved will be the better
clusterings of your data.

Somewhat backwards from what you might intuitively expect, local optimization tend to be more of a problem when
the number of clusters K you want to determine is relatively small, say from 2 to 10 as a rule of thumb.  For these
number of clusters it is usually a good idea to run 50 to 1000 or so K-means clustering attempts, keep track
of the final cost of each, and examine/use the one that achieved the lowest cost at the end.

However when you are trying to create larger number of clusters, often the minimum that exist are going to
be all relatively close to the same, so one clustering, even if different from another one found, will have
a similar overall cost.  Thus when trying to determine a large number of clusters/segments it is not as useful
or necessary to run multiple times to watch out for local minima results.

# Video W8 05: Choosing the Number of Clusters

[YouTube Video Link](https://www.youtube.com/watch?v=izCbbMbRWHw&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW&index=81)

The most common method is still to choose the number of clusters you want by hand.  Sometimes the problem you are trying to
cluster natually lends itself to a particular number of clusters you desire (e.g. we want to design t-shirts for 3 sizes, S, M and L).  Other times, you can do some visualization of the data, and get a rough idea of how many there might be, but often there can
be different interpretations of this.

It is often genuinly ambiguous the number of clusters in a data set.  Even if/when you can visualize, there will
often be ambiguity, and different clustering can be supported based on the needs of the application.

There are some things you can do to try and help to algorithmically pick a good size for K for your clustering.

In the Elbow method, you compute the cluster for $K=1, 2, 3, ... N$, and look at the final cost function $J$ achieved
for each $K$ clustering size.  If you plot the cost, often there will be some point where the cost changes from going
down rapidly to going down much slower.  Often the "Elbow" of this curve, or somewhere around it, will be a good
number of cluster for the data you have.

However it is possible to get a much more ambiguous result, where there is no apparent elbow to your graph.
In practice this will often be the case.  It can be worth a shot, but as often as not you won't get a good idea
from this of what might be a good K size, thus you will have to result to other means.

Another method is really application driven.  If you have some metric downstream for evaluating the effectiveness
of your application, you can then compare that metric when you try different clustering values K, and use the
clustering/segmentation that works best for the application domain.

# K-Means Clustering with Scikit-Learn

`Scikit-Learn` has clustering algorithms in the `sklearn.cluster` sublibrary.

We can get a basic clustring of the made up data set we had previously using $K = 2$
clusters like this:

```python
# fit a clustering estimator to the made up date
from sklearn.cluster import KMeans

cluster = KMeans(n_clusters=2);
cluster.fit(X)

# display/access the results, like the final cluster centers, the labels of the data, and the final cost
centers = cluster.cluster_centers_
print(centers)
labels = cluster.labels_
print(labels)
cost = cluster.inertia_
print(cost)

```

    [[-1.96875 -2.49375]
     [ 1.99375  2.28125]]
    [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
    139.75249999999997

```python
# visualize the final cluster labels found
cluster_0 = np.where(labels == 0)[0]
cluster_1 = np.where(labels == 1)[0]

plt.plot(X[cluster_0, 0], X[cluster_0, 1], 'ro', label='assg cluster 0')
plt.plot(centers[0,0], centers[0,1], 'rx', markersize=15)
plt.plot(X[cluster_1, 0], X[cluster_1, 1], 'bo', label='assg cluster 1')
plt.plot(centers[1,0], centers[1,1], 'bx', markersize=15, label='centroids')
plt.xlabel(r'$x_1$', fontsize=20)
plt.ylabel(r'$x_2$', fontsize=20)
plt.legend(loc=2)
plt.axis([-5, 7, -7, 7]);
```

## Check for Local Minima

In this case for our made up data the clusters are pretty well separated, and local minima are not so easy
to find.  Lets run the cluster 1000 times, and keep the worst and best K-means clusters we find based on the
cost (inertia) measure.

By default `scikit-learn` actually performs the clustering 10 times (controlled by the `n_init`) parameter.
If we want to see different results, we should set `n_init = 1` so that only 1 clustering is performed.

```python
N = 1000

clusters = []
cost = np.empty(N)

# now perform different clusterings, checking for ones that improve or make things worse
for n in range(N):
    cluster = KMeans(n_clusters=2, n_init=1)
    cluster.fit(X)
    clusters.append(cluster)
    cost[n] = cluster.inertia_
```

```python
# in this case, we are never seeing any solution other than the 1 minimum that is discovered
print(cost.min())
print(cost.max())
```

    139.75249999999997
    139.75249999999997

## Determine K number of clusters

Likewise lets try and illustrate the Elbow method, and fit the data with clusters of size $K = 1 \cdots 10$
Again the data looks like 2 clusters is pretty optimal, so we won't get a very useful result here.

```python
MAX_K = 10

clusters = []
cost = np.empty(MAX_K+1)

# perform clusterings for K ranging from 2 to MAX_K.  If we were having local minima issues, we might also want
# to run k-means multiple times for each K and find/choose the best cost achieved for the elbow graph
for k in range(2, MAX_K+1):
    cluster = KMeans(n_clusters=k)
    cluster.fit(X)
    clusters.append(cluster)
    cost[k] = cluster.inertia_

# visualize the resulting costs as a function of K clustering size
plt.plot(np.arange(2,MAX_K+1), cost[2:])
plt.plot(np.arange(2,MAX_K+1), cost[2:], 'bo')
plt.xlabel('K (clustering size)')
plt.ylabel('J (cost or inertia) achieved')
plt.title("Elbow plot of cost as a function of K clustering size");
```

# K-Means Clustering on Iris Data

Using the simple data set does not give a great example of the potential for local minima and choosing K.
Here we perform the previous again, but use the iris data set.  The iris data set is 4 dimensional, and we will
use all 4 dimensions.  We will try clustering into 3 cluster, which is of course the number of categories
we have for the original iris data set.

```python
from sklearn import datasets

iris = datasets.load_iris()
X = iris.data
```

```python
# create a K-means clustering with K=3
cluster = KMeans(n_clusters=3);
cluster.fit(X)

# display/access the results, like the final cluster centers, the labels of the data, and the final cost
centers = cluster.cluster_centers_
print(centers)
labels = cluster.labels_
print(labels)
cost = cluster.inertia_
print(cost)

```

    [[5.9016129  2.7483871  4.39354839 1.43387097]
     [5.006      3.428      1.462      0.246     ]
     [6.85       3.07368421 5.74210526 2.07105263]]
    [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 2 2 2 2 0 2 2 2 2
     2 2 0 0 2 2 2 2 0 2 0 2 0 2 2 0 0 2 2 2 2 2 0 2 2 2 2 0 2 2 2 0 2 2 2 0 2
     2 0]
    78.851441426146

```python
# visualize resulting clusters on the 2 dimensions of Petal length/width
cluster_0 = np.where(labels == 0)[0]
cluster_1 = np.where(labels == 1)[0]
cluster_2 = np.where(labels == 2)[0]

plt.plot(X[cluster_0, 0], X[cluster_0, 1], 'ro', label='assg cluster 0')
plt.plot(centers[0,0], centers[0,1], 'rx', markersize=15)
plt.plot(X[cluster_1, 0], X[cluster_1, 1], 'bo', label='assg cluster 1')
plt.plot(centers[1,0], centers[1,1], 'bx', markersize=15, label='centroids')
plt.plot(X[cluster_2, 0], X[cluster_2, 1], 'go', label='assg cluster 2')
plt.plot(centers[2,0], centers[2,1], 'gx', markersize=15, label='centroids')
plt.xlabel(r'petal width', fontsize=20)
plt.ylabel(r'petal height', fontsize=20)
plt.legend();
```

## Check for Local Minima

```python
N = 1000

clusters = []
cost = np.empty(N)

# now perform different clusterings, checking for ones that improve or make things worse
for n in range(N):
    cluster = KMeans(n_clusters=3, n_init=1)
    cluster.fit(X)
    clusters.append(cluster)
    cost[n] = cluster.inertia_
```

```python
# For the iris data
print(cost.min())
print(cost.max())
print(np.unique(cost))
```

    78.851441426146
    145.4526917648503
    [ 78.85144143  78.85566583 142.7540625  145.45269176]

```python
sns.distplot(cost);
```

In this case, it looks like we usually discover 3 or 4 unique mimina.  Most of the time we get a cost of a bit
over 78.  But sometimes we get 142 or 145, which are probably not as optimal clusterings.  Lets plot one of
the 145 cost clusterings:

```python
len(clusters)
clusters[318]
```

    KMeans(n_clusters=3, n_init=1)

```python
cluster_num = np.where(cost > 140)[0][0]
print(cluster_num)
cluster = clusters[cluster_num]

centers = cluster.cluster_centers_
print(centers)
labels = cluster.labels_
print(labels)
cost = cluster.inertia_
print(cost)

```

    4
    [[5.19375    3.63125    1.475      0.271875  ]
     [6.31458333 2.89583333 4.97395833 1.703125  ]
     [4.73181818 2.92727273 1.77272727 0.35      ]]
    [0 2 2 2 0 0 2 0 2 2 0 0 2 2 0 0 0 0 0 0 0 0 0 0 2 2 0 0 0 2 2 0 0 0 2 0 0
     0 2 0 0 2 2 0 0 2 0 2 0 0 1 1 1 1 1 1 1 2 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1]
    142.7540625000001

```python
# visualize resulting clusters on the 2 dimensions of Petal length/width
cluster_0 = np.where(labels == 0)[0]
cluster_1 = np.where(labels == 1)[0]
cluster_2 = np.where(labels == 2)[0]

plt.plot(X[cluster_0, 0], X[cluster_0, 1], 'ro', label='assg cluster 0')
plt.plot(centers[0,0], centers[0,1], 'rx', markersize=15)
plt.plot(X[cluster_1, 0], X[cluster_1, 1], 'bo', label='assg cluster 1')
plt.plot(centers[1,0], centers[1,1], 'bx', markersize=15, label='centroids')
plt.plot(X[cluster_2, 0], X[cluster_2, 1], 'go', label='assg cluster 2')
plt.plot(centers[2,0], centers[2,1], 'gx', markersize=15, label='centroids')
plt.xlabel(r'petal width', fontsize=20)
plt.ylabel(r'petal height', fontsize=20)
plt.legend();
```

This is definitely not a good clustering, it has broken up the smaller separate group (which were the easier
to classify Virginica samples) into 2, and group the other 2 into 1 big cluster.

## Determine K number of Clusters

```python
MAX_K = 10

clusters = []
cost = np.empty(MAX_K+1)

# perform clusterings for K ranging from 2 to MAX_K.  If we were having local minima issues, we might also want
# to run k-means multiple times for each K and find/choose the best cost achieved for the elbow graph
for k in range(2, MAX_K+1):
    cluster = KMeans(n_clusters=k, n_init=1)
    cluster.fit(X)
    clusters.append(cluster)
    cost[k] = cluster.inertia_

# visualize the resulting costs as a function of K clustering size
plt.plot(np.arange(2,MAX_K+1), cost[2:])
plt.plot(np.arange(2,MAX_K+1), cost[2:], 'bo')
plt.xlabel('K (clustering size)')
plt.ylabel('J (cost or inertia) achieved')
plt.title("Elbow plot of cost as a function of K clustering size");
```

```python
import sys
sys.path.append("../../src") # add our class modules to the system PYTHON_PATH

from ml_python_class.custom_funcs import version_information
version_information()
```

                  Module   Versions
    --------------------   ------------------------------------------------------------
             matplotlib:   ['3.3.0']
                  numpy:   ['1.18.5']
                 pandas:   ['1.0.5']
                seaborn:   ['0.10.1']


```python
# This mounts your Google Drive to the Colab VM.
from google.colab import drive
drive.mount('/content/drive')

# TODO: Enter the foldername in your Drive where you have saved the unzipped
# assignment folder, e.g. 'cs231n/assignments/assignment1/'
FOLDERNAME = 'CS231N/assignment/assignment2/'
assert FOLDERNAME is not None, "[!] Enter the foldername."

# Now that we've mounted your Drive, this ensures that
# the Python interpreter of the Colab VM can load
# python files from within it.
import sys
sys.path.append('/content/drive/My Drive/{}'.format(FOLDERNAME))

# This downloads the CIFAR-10 dataset to your Drive
# if it doesn't already exist.
%cd /content/drive/My\ Drive/$FOLDERNAME/cs231n/datasets/
!bash get_datasets.sh
%cd /content/drive/My\ Drive/$FOLDERNAME
```

    Mounted at /content/drive
    /content/drive/My Drive/CS231N/assignment/assignment2/cs231n/datasets
    /content/drive/My Drive/CS231N/assignment/assignment2

# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train. One idea along these lines is batch normalization, proposed by [1] in 2015.

To understand the goal of batch normalization, it is important to first recognize that machine learning methods tend to perform better with input data consisting of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features. This will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance, since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, they propose to insert into the network layers that normalize batches. At training time, such a layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```python
# Setup cell.
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams["figure.figsize"] = (10.0, 8.0)  # Set default size of plots.
plt.rcParams["image.interpolation"] = "nearest"
plt.rcParams["image.cmap"] = "gray"

%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """Returns relative error."""
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print(f"  means: {x.mean(axis=axis)}")
    print(f"  stds:  {x.std(axis=axis)}\n")
```

```python
# Load the (preprocessed) CIFAR-10 data.
data = get_CIFAR10_data()
for k, v in list(data.items()):
    print(f"{k}: {v.shape}")
```

    X_train: (49000, 3, 32, 32)
    y_train: (49000,)
    X_val: (1000, 3, 32, 32)
    y_val: (1000,)
    X_test: (1000, 3, 32, 32)
    y_test: (1000,)

# Batch Normalization: Forward Pass
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network.
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))

# Means should be close to zero and stds close to one.
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])

# Now means should be close to beta and stds close to gamma.
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

    Before batch normalization:
      means: [ -2.3814598  -13.18038246   1.91780462]
      stds:  [27.18502186 34.21455511 37.68611762]

    After batch normalization (gamma=1, beta=0)
      means: [5.32907052e-17 7.04991621e-17 1.85962357e-17]
      stds:  [0.99999999 1.         1.        ]

    After batch normalization (gamma= [1. 2. 3.] , beta= [11. 12. 13.] )
      means: [11. 12. 13.]
      stds:  [0.99999999 1.99999999 2.99999999]

```python
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

    After batch normalization (test-time):
      means: [-0.03927354 -0.04349152 -0.10452688]
      stds:  [1.01531428 1.01238373 0.97819988]

# Batch Normalization: Backward Pass
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```python
# Gradient check batchnorm backward pass.
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)

# You should expect to see relative errors between 1e-13 and 1e-8.
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.7029261167605239e-09
    dgamma error:  7.420414216247087e-13
    dbeta error:  2.8795057655839487e-12

# Batch Normalization: Alternative Backward Pass
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\...\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
& \mu=\frac{1}{N}\sum_{k=1}^N x_k  &  v=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2 \\
& \sigma=\sqrt{v+\epsilon}         &  y_i=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hard part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```python
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

    dx difference:  6.284600172572596e-13
    dgamma difference:  0.0
    dbeta difference:  0.0
    speedup: 2.03x

# Fully Connected Networks with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

**Hint:** You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`.

```python
np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.
for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            normalization='batchnorm')

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
  if reg == 0: print()
```

    Running check with reg =  0
    Initial loss:  2.261195509168841
    W1 relative error: 1.10e-04
    W2 relative error: 3.89e-06
    W3 relative error: 8.12e-10
    b1 relative error: 2.22e-03
    b2 relative error: 5.55e-09
    b3 relative error: 5.81e-10
    beta1 relative error: 6.82e-09
    beta2 relative error: 2.40e-09
    gamma1 relative error: 1.83e-08
    gamma2 relative error: 2.82e-09

    Running check with reg =  3.14
    Initial loss:  6.996533219149863
    W1 relative error: 1.98e-06
    W2 relative error: 2.28e-06
    W3 relative error: 2.23e-08
    b1 relative error: 5.55e-09
    b2 relative error: 2.22e-08
    b3 relative error: 7.06e-10
    beta1 relative error: 6.32e-09
    beta2 relative error: 4.34e-09
    gamma1 relative error: 5.61e-09
    gamma2 relative error: 5.28e-09

# Batch Normalization for Deep Networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```python
np.random.seed(231)

# Try training a very deep net with batchnorm.
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

    Solver with batch norm:
    (Iteration 1 / 200) loss: 2.340974
    (Epoch 0 / 10) train acc: 0.107000; val_acc: 0.115000
    (Epoch 1 / 10) train acc: 0.313000; val_acc: 0.265000
    (Iteration 21 / 200) loss: 2.039345
    (Epoch 2 / 10) train acc: 0.396000; val_acc: 0.280000
    (Iteration 41 / 200) loss: 2.047471
    (Epoch 3 / 10) train acc: 0.484000; val_acc: 0.316000
    (Iteration 61 / 200) loss: 1.739554
    (Epoch 4 / 10) train acc: 0.525000; val_acc: 0.318000
    (Iteration 81 / 200) loss: 1.246973
    (Epoch 5 / 10) train acc: 0.595000; val_acc: 0.335000
    (Iteration 101 / 200) loss: 1.354766
    (Epoch 6 / 10) train acc: 0.638000; val_acc: 0.331000
    (Iteration 121 / 200) loss: 1.014049
    (Epoch 7 / 10) train acc: 0.673000; val_acc: 0.323000
    (Iteration 141 / 200) loss: 1.135644
    (Epoch 8 / 10) train acc: 0.682000; val_acc: 0.305000
    (Iteration 161 / 200) loss: 0.652241
    (Epoch 9 / 10) train acc: 0.781000; val_acc: 0.336000
    (Iteration 181 / 200) loss: 0.782712
    (Epoch 10 / 10) train acc: 0.749000; val_acc: 0.318000

    Solver without batch norm:
    (Iteration 1 / 200) loss: 2.302332
    (Epoch 0 / 10) train acc: 0.129000; val_acc: 0.131000
    (Epoch 1 / 10) train acc: 0.283000; val_acc: 0.250000
    (Iteration 21 / 200) loss: 2.041970
    (Epoch 2 / 10) train acc: 0.316000; val_acc: 0.277000
    (Iteration 41 / 200) loss: 1.900473
    (Epoch 3 / 10) train acc: 0.373000; val_acc: 0.282000
    (Iteration 61 / 200) loss: 1.713156
    (Epoch 4 / 10) train acc: 0.390000; val_acc: 0.310000
    (Iteration 81 / 200) loss: 1.662209
    (Epoch 5 / 10) train acc: 0.434000; val_acc: 0.300000
    (Iteration 101 / 200) loss: 1.696059
    (Epoch 6 / 10) train acc: 0.535000; val_acc: 0.345000
    (Iteration 121 / 200) loss: 1.557987
    (Epoch 7 / 10) train acc: 0.530000; val_acc: 0.304000
    (Iteration 141 / 200) loss: 1.432189
    (Epoch 8 / 10) train acc: 0.628000; val_acc: 0.339000
    (Iteration 161 / 200) loss: 1.034116
    (Epoch 9 / 10) train acc: 0.654000; val_acc: 0.342000
    (Iteration 181 / 200) loss: 0.905794
    (Epoch 10 / 10) train acc: 0.712000; val_acc: 0.328000

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```python
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch Normalization and Initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train eight-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```python
np.random.seed(231)

# Try training a very deep net with batchnorm.
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
    print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
    bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

    bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    bn_solver.train()
    bn_solvers_ws[weight_scale] = bn_solver

    solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    solver.train()
    solvers_ws[weight_scale] = solver
```

    Running weight scale 1 / 20
    Running weight scale 2 / 20
    Running weight scale 3 / 20
    Running weight scale 4 / 20
    Running weight scale 5 / 20
    Running weight scale 6 / 20
    Running weight scale 7 / 20
    Running weight scale 8 / 20
    Running weight scale 9 / 20
    Running weight scale 10 / 20
    Running weight scale 11 / 20
    Running weight scale 12 / 20
    Running weight scale 13 / 20
    Running weight scale 14 / 20
    Running weight scale 15 / 20
    Running weight scale 16 / 20
    Running weight scale 17 / 20
    Running weight scale 18 / 20
    Running weight scale 19 / 20
    Running weight scale 20 / 20

```python
# Plot results of weight scale experiment.
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers_ws[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

  best_val_accs.append(max(solvers_ws[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs. weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs. weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs. weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the weight initialization scale affect models with/without batch normalization differently, and why?

## Answer:

Batch normalization is less sensisive to the weight initialization

# Batch Normalization and Batch Size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```python
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)

    # Try training a very deep net with batchnorm.
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

    No normalization: batch size =  5
    Normalization: batch size =  5
    Normalization: batch size =  10
    Normalization: batch size =  50

```python
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:

If the batch size is too small, the accuracy can goes down because it cannot represent the whole dataset.

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:

Batch: 1

Layer: 2

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_forward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization.

# Simulate the forward pass for a two-layer network.
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(D3)
beta = np.zeros(D3)

# Means should be close to zero and stds close to one.
print('After layer normalization (gamma=1, beta=0)')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0])

# Now means should be close to beta and stds close to gamma.
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

    Before layer normalization:
      means: [-59.06673243 -47.60782686 -43.31137368 -26.40991744]
      stds:  [10.07429373 28.39478981 35.28360729  4.01831507]

    After layer normalization (gamma=1, beta=0)
      means: [ 4.81096644e-16 -7.40148683e-17  2.22044605e-16 -5.92118946e-16]
      stds:  [0.99999995 0.99999999 1.         0.99999969]

    After layer normalization (gamma= [3. 3. 3.] , beta= [5. 5. 5.] )
      means: [5. 5. 5. 5.]
      stds:  [2.99999985 2.99999998 2.99999999 2.99999907]

```python
# Gradient check batchnorm backward pass.
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

# You should expect to see relative errors between 1e-12 and 1e-8.
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.4336158494902849e-09
    dgamma error:  4.519489546032799e-12
    dbeta error:  2.276445013433725e-12

# Layer Normalization and Batch Size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```python
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:

2. If the dimension is very small, then the mean and variance are not representable for the data.