Probabilistic Programming
=====
and Bayesian Methods for Hackers
========

##### Version 0.1

Original content created by Cam Davidson-Pilon

Ported to [Pyro](http://pyro.ai/) by Carlos Souza (souza@gatech.edu), with the help from [Pyro community](https://forum.pyro.ai/).
___

Welcome to *Bayesian Methods for Hackers*. The full Github repository is available at [github/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers](https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers). The other chapters can be found on the project's [homepage](https://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/). We hope you enjoy the book, and we encourage any contributions!

Chapter 1
======
***

The Philosophy of Bayesian Inference
------

> You are a skilled programmer, but bugs still slip into your code. After a particularly difficult implementation of an algorithm, you decide to test your code on a trivial example. It passes. You test the code on a harder problem. It passes once again. And it passes the next, *even more difficult*, test too! You are starting to believe that there may be no bugs in this code...

If you think this way, then congratulations, you already are thinking Bayesian! Bayesian inference is simply updating your beliefs after considering new evidence. A Bayesian can rarely be certain about a result, but he or she can be very confident. Just like in the example above, we can never be 100% sure that our code is bug-free unless we test it on every possible problem; something rarely possible in practice. Instead, we can test it on a large number of problems, and if it succeeds we can feel more *confident* about our code, but still not certain.  Bayesian inference works identically: we update our beliefs about an outcome; rarely can we be absolutely sure unless we rule out all other alternatives.

### The Bayesian state of mind

Bayesian inference differs from more traditional statistical inference by preserving *uncertainty*. At first, this sounds like a bad statistical technique. Isn't statistics all about deriving *certainty* from randomness? To reconcile this, we need to start thinking like Bayesians.

The Bayesian world-view interprets probability as measure of *believability in an event*, that is, how confident we are in an event occurring. In fact, we will see in a moment that this is the natural interpretation of probability.

For this to be clearer, we consider an alternative interpretation of probability: *Frequentist*, known as the more *classical* version of statistics, assume that probability is the long-run frequency of events (hence the bestowed title). For example, the *probability of plane accidents* under a frequentist philosophy is interpreted as the *long-term frequency of plane accidents*. This makes logical sense for many probabilities of events, but becomes more difficult to understand when events have no long-term frequency of occurrences. Consider: we often assign probabilities to outcomes of presidential elections, but the election itself only happens once! Frequentists get around this by invoking alternative realities and saying across all these realities, the frequency of occurrences defines the probability.

Bayesians, on the other hand, have a more intuitive approach. Bayesians interpret a probability as measure of *belief*, or confidence, of an event occurring. Simply, a probability is a summary of an opinion. An individual who assigns a belief of 0 to an event has no confidence that the event will occur; conversely, assigning a belief of 1 implies that the individual is absolutely certain of an event occurring. Beliefs between 0 and 1 allow for weightings of other outcomes. This definition agrees with the probability of a plane accident example, for having observed the frequency of plane accidents, an individual's belief should be equal to that frequency, excluding any outside information. Similarly, under this definition of probability being equal to beliefs, it is meaningful to speak about probabilities (beliefs) of presidential election outcomes: how confident are you candidate *A* will win?

Notice in the paragraph above, I assigned the belief (probability) measure to an *individual*, not to Nature. This is very interesting, as this definition leaves room for conflicting beliefs between individuals. Again, this is appropriate for what naturally occurs: different individuals have different beliefs of events occurring, because they possess different *information* about the world. The existence of different beliefs does not imply that anyone is wrong. Consider the following examples demonstrating the relationship between individual beliefs and probabilities:

- I flip a coin, and we both guess the result. We would both agree, assuming the coin is fair, that the probability of Heads is 1/2. Assume, then, that I peek at the coin. Now I know for certain what the result is: I assign probability 1.0 to either Heads or Tails (whichever it is). Now what is *your* belief that the coin is Heads? My knowledge of the outcome has not changed the coin's results. Thus we assign different probabilities to the result.

-  Your code either has a bug in it or not, but we do not know for certain which is true, though we have a belief about the presence or absence of a bug.

-  A medical patient is exhibiting symptoms $x$, $y$ and $z$. There are a number of diseases that could be causing all of them, but only a single disease is present. A doctor has beliefs about which disease, but a second doctor may have slightly different beliefs.

This philosophy of treating beliefs as probability is natural to humans. We employ it constantly as we interact with the world and only see partial truths, but gather evidence to form beliefs. Alternatively, you have to be *trained* to think like a frequentist.

To align ourselves with traditional probability notation, we denote our belief about event $A$ as $P(A)$. We call this quantity the *prior probability*.

John Maynard Keynes, a great economist and thinker, said "When the facts change, I change my mind. What do you do, sir?" This quote reflects the way a Bayesian updates his or her beliefs after seeing evidence. Even — especially — if the evidence is counter to what was initially believed, the evidence cannot be ignored. We denote our updated belief as $P(A |X )$, interpreted as the probability of $A$ given the evidence $X$. We call the updated belief the *posterior probability* so as to contrast it with the prior probability. For example, consider the posterior probabilities (read: posterior beliefs) of the above examples, after observing some evidence $X$:

1\. $P(A): \;\;$ the coin has a 50 percent chance of being Heads. $P(A | X):\;\;$ You look at the coin, observe a Heads has landed, denote this information $X$, and trivially assign probability 1.0 to Heads and 0.0 to Tails.

2\.   $P(A): \;\;$  This big, complex code likely has a bug in it. $P(A | X): \;\;$ The code passed all $X$ tests; there still might be a bug, but its presence is less likely now.

3\.  $P(A):\;\;$ The patient could have any number of diseases. $P(A | X):\;\;$ Performing a blood test generated evidence $X$, ruling out some of the possible diseases from consideration.

It's clear that in each example we did not completely discard the prior belief after seeing new evidence $X$, but we *re-weighted the prior* to incorporate the new evidence (i.e. we put more weight, or confidence, on some beliefs versus others).

By introducing prior uncertainty about events, we are already admitting that any guess we make is potentially very wrong. After observing data, evidence, or other information, we update our beliefs, and our guess becomes *less wrong*. This is the alternative side of the prediction coin, where typically we try to be *more right*.

### Our Bayesian framework

We are interested in beliefs, which can be interpreted as probabilities by thinking Bayesian. We have a *prior* belief in event $A$, beliefs formed by previous information, e.g., our prior belief about bugs being in our code before performing tests.

Secondly, we observe our evidence. To continue our buggy-code example: if our code passes $X$ tests, we want to update our belief to incorporate this. We call this new belief the *posterior* probability. Updating our belief is done via the following equation, known as Bayes' Theorem, after its discoverer Thomas Bayes:

\begin{align}
 P( A | X ) = & \frac{ P(X | A) P(A) } {P(X) } \\\\[5pt]
& \propto P(X | A) P(A)\;\; (\propto \text{is proportional to })
\end{align}

The above formula is not unique to Bayesian inference: it is a mathematical fact with uses outside Bayesian inference. Bayesian inference merely uses it to connect prior probabilities $P(A)$ with an updated posterior probabilities $P(A | X )$.

##### Example: Mandatory coin-flip example

Every statistics text must contain a coin-flipping example, I'll use it here to get it out of the way. Suppose, naively, that you are unsure about the probability of heads in a coin flip (spoiler alert: it's 50%). You believe there is some true underlying ratio, call it $p$, but have no prior opinion on what $p$ might be.

We begin to flip a coin, and record the observations: either $H$ or $T$. This is our observed data. An interesting question to ask is how our inference changes as we observe more and more data? More specifically, what do our posterior probabilities look like when we have little data, versus when we have lots of data.

Below we plot a sequence of updating posterior probabilities as we observe increasing amounts of data (coin flips).

```python
%matplotlib inline

import torch
import pyro
import pyro.distributions as dist
from pyro.infer import MCMC, NUTS, HMC
from IPython.core.pylabtools import figsize
import numpy as np
from matplotlib import pyplot as plt
```

```python
n_trials = [0, 1, 2, 3, 4, 5, 8, 15, 50, 500]
x = np.linspace(0, 1, 100)
data = pyro.sample('coin_toss', pyro.distributions.Bernoulli(0.5), [n_trials[-1]])
```

```python
figsize(11, 9)

# For the already prepared, I'm using Binomial's conj. prior.
for k, N in enumerate(n_trials):
    sx = plt.subplot(len(n_trials)/2, 2, k+1)
    plt.xlabel("$p$, probability of heads") \
        if k in [0, len(n_trials)-1] else None
    plt.setp(sx.get_yticklabels(), visible=False)
    heads = data[:N].sum()
    d = pyro.distributions.Beta(1 + heads, 1 + N - heads)
    y = torch.exp(d.log_prob(torch.tensor(x)))
    plt.plot(x, y, label="observe %d tosses,\n %d heads" % (N, heads))
    plt.fill_between(x, 0, y, color="#348ABD", alpha=0.4)
    plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)

    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight=True)

plt.suptitle("Bayesian updating of posterior probabilities",
             y=1.02,
             fontsize=14)

plt.tight_layout()
```

The posterior probabilities are represented by the curves, and our uncertainty is proportional to the width of the curve. As the plot above shows, as we start to observe data our posterior probabilities start to shift and move around. Eventually, as we observe more and more data (coin-flips), our probabilities will tighten closer and closer around the true value of $p=0.5$ (marked by a dashed line).

Notice that the plots are not always *peaked* at 0.5. There is no reason it should be: recall we assumed we did not have a prior opinion of what $p$ is. In fact, if we observe quite extreme data, say 8 flips and only 1 observed heads, our distribution would look very biased *away* from lumping around 0.5 (with no prior opinion, how confident would you feel betting on a fair coin after observing 8 tails and 1 head?). As more data accumulates, we would see more and more probability being assigned at $p=0.5$, though never all of it.

The next example is a simple demonstration of the mathematics of Bayesian inference.

##### Example: Bug, or just sweet, unintended feature?

Let $A$ denote the event that our code has **no bugs** in it. Let $X$ denote the event that the code passes all debugging tests. For now, we will leave the prior probability of no bugs as a variable, i.e. $P(A) = p$.

We are interested in $P(A|X)$, i.e. the probability of no bugs, given our debugging tests $X$. To use the formula above, we need to compute some quantities.

What is $P(X | A)$, i.e., the probability that the code passes $X$ tests *given* there are no bugs? Well, it is equal to 1, for a code with no bugs will pass all tests.

$P(X)$ is a little bit trickier: The event $X$ can be divided into two possibilities, event $X$ occurring even though our code *indeed has* bugs (denoted $\sim A\;$, spoken *not $A$*), or event $X$ without bugs ($A$). $P(X)$ can be represented as:

\begin{align}
P(X ) & = P(X \text{ and } A) + P(X \text{ and } \sim A) \\\\[5pt]
 & = P(X|A)P(A) + P(X | \sim A)P(\sim A)\\\\[5pt]
& = P(X|A)p + P(X | \sim A)(1-p)
\end{align}

We have already computed $P(X|A)$ above. On the other hand, $P(X | \sim A)$ is subjective: our code can pass tests but still have a bug in it, though the probability there is a bug present is reduced. Note this is dependent on the number of tests performed, the degree of complication in the tests, etc. Let's be conservative and assign $P(X|\sim A) = 0.5$. Then

\begin{align}
P(A | X) & = \frac{1\cdot p}{ 1\cdot p +0.5 (1-p) } \\\\
& = \frac{ 2 p}{1+p}
\end{align}
This is the posterior probability. What does it look like as a function of our prior, $p \in [0,1]$?

```python
figsize(12.5, 4)
p = torch.linspace(0, 1, 50)
plt.plot(p, 2 * p / (1 + p), color="#348ABD", lw=3)
plt.scatter(0.2, 2*(0.2)/1.2, s=140, c="#348ABD")
plt.xlim(0, 1)
plt.ylim(0, 1)
plt.xlabel("Prior, $P(A) = p$")
plt.ylabel("Posterior, $P(A|X)$, with $P(A) = p$")
plt.title("Are there bugs in my code?");
```

We can see the biggest gains if we observe the $X$ tests passed when the prior probability, $p$, is low. Let's settle on a specific value for the prior. I'm a strong programmer (I think), so I'm going to give myself a realistic prior of 0.20, that is, there is a 20% chance that I write code bug-free. To be more realistic, this prior should be a function of how complicated and large the code is, but let's pin it at 0.20. Then my updated belief that my code is bug-free is 0.33.

Recall that the prior is a probability: $p$ is the prior probability that there *are no bugs*, so $1-p$ is the prior probability that there *are bugs*.

Similarly, our posterior is also a probability, with $P(A | X)$ the probability there is no bug *given we saw all tests pass*, hence $1-P(A|X)$ is the probability there is a bug *given all tests passed*. What does our posterior probability look like? Below is a chart of both the prior and the posterior probabilities.

```python
figsize(12.5, 4)
colours = ["#348ABD", "#A60628"]

prior = [0.20, 0.80]
posterior = [1./3, 2./3]
plt.bar([0, .7], prior, alpha=0.70, width=0.25,
        color=colours[0], label="prior distribution",
        lw="3", edgecolor=colours[0])

plt.bar([0+0.25, .7+0.25], posterior, alpha=0.7,
        width=0.25, color=colours[1],
        label="posterior distribution",
        lw="3", edgecolor=colours[1])

plt.xticks([0.20, .95], ["Bugs Absent", "Bugs Present"])
plt.title("Prior and Posterior probability of bugs present")
plt.ylabel("Probability")
plt.legend(loc="upper left");
```

Notice that after we observed $X$ occur, the probability of bugs being absent increased. By increasing the number of tests, we can approach confidence (probability 1) that there are no bugs present.

This was a very simple example of Bayesian inference and Bayes rule. Unfortunately, the mathematics necessary to perform more complicated Bayesian inference only becomes more difficult, except for artificially constructed cases. We will later see that this type of mathematical analysis is actually unnecessary. First we must broaden our modeling tools. The next section deals with *probability distributions*. If you are already familiar, feel free to skip (or at least skim), but for the less familiar the next section is essential.

_______

## Probability Distributions

**Let's quickly recall what a probability distribution is:** Let $Z$ be some random variable. Then associated with $Z$ is a *probability distribution function* that assigns probabilities to the different outcomes $Z$ can take. Graphically, a probability distribution is a curve where the probability of an outcome is proportional to the height of the curve. You can see examples in the first figure of this chapter.

We can divide random variables into three classifications:

-   **$Z$ is discrete**: Discrete random variables may only assume values on a specified list. Things like populations, movie ratings, and number of votes are all discrete random variables. Discrete random variables become more clear when we contrast them with...

-   **$Z$ is continuous**: Continuous random variable can take on arbitrarily exact values. For example, temperature, speed, time, color are all modeled as continuous variables because you can progressively make the values more and more precise.

- **$Z$ is mixed**: Mixed random variables assign probabilities to both discrete and continuous random variables, i.e. it is a combination of the above two categories.

### Discrete Case
If $Z$ is discrete, then its distribution is called a *probability mass function*, which measures the probability $Z$ takes on the value $k$, denoted $P(Z=k)$. Note that the probability mass function completely describes the random variable $Z$, that is, if we know the mass function, we know how $Z$ should behave. There are popular probability mass functions that consistently appear: we will introduce them as needed, but let's introduce the first very useful probability mass function. We say $Z$ is *Poisson*-distributed if:

$$P(Z = k) =\frac{ \lambda^k e^{-\lambda} }{k!}, \; \; k=0,1,2, \dots $$

$\lambda$ is called a parameter of the distribution, and it controls the distribution's shape. For the Poisson distribution, $\lambda$ can be any positive number. By increasing $\lambda$, we add more probability to larger values, and conversely by decreasing $\lambda$ we add more probability to smaller values. One can describe $\lambda$ as the *intensity* of the Poisson distribution.

Unlike $\lambda$, which can be any positive number, the value $k$ in the above formula must be a non-negative integer, i.e., $k$ must take on values 0,1,2, and so on. This is very important, because if you wanted to model a population you could not make sense of populations with 4.25 or 5.612 members.

If a random variable $Z$ has a Poisson mass distribution, we denote this by writing

$$Z \sim \text{Poi}(\lambda) $$

One useful property of the Poisson distribution is that its expected value is equal to its parameter, i.e.:

$$E\large[ \;Z\; | \; \lambda \;\large] = \lambda $$

We will use this property often, so it's useful to remember. Below, we plot the probability mass distribution for different $\lambda$ values. The first thing to notice is that by increasing $\lambda$, we add more probability of larger values occurring. Second, notice that although the graph ends at 15, the distributions do not. They assign positive probability to every non-negative integer.

```python
figsize(12.5, 4)

lambda_ = [1.5, 4.25]

d = dist.Poisson(torch.tensor(lambda_))
x = torch.arange(16).float().view(-1, 1)
y = torch.exp(d.log_prob(x))

colours = ["#348ABD", "#A60628"]
plt.bar(x.flatten(), y[:, 0], color=colours[0],
        label="$\lambda = %.1f$" % lambda_[0], alpha=0.60,
        edgecolor=colours[0], lw="3")

plt.bar(x.flatten(), y[:, 1], color=colours[1],
        label="$\lambda = %.1f$" % lambda_[1], alpha=0.60,
        edgecolor=colours[1], lw="3")

plt.xticks(x.flatten().numpy() + 0.4, x.flatten().numpy().astype(int))
plt.legend()
plt.ylabel("probability of $k$")
plt.xlabel("$k$")
plt.title("Probability mass function of a Poisson random variable; differing \
$\lambda$ values");
```

### Continuous Case
Instead of a probability mass function, a continuous random variable has a *probability density function*. This might seem like unnecessary nomenclature, but the density function and the mass function are very different creatures. An example of continuous random variable is a random variable with *exponential density*. The density function for an exponential random variable looks like this:

$$f_Z(z | \lambda) = \lambda e^{-\lambda z }, \;\; z\ge 0$$

Like a Poisson random variable, an exponential random variable can take on only non-negative values. But unlike a Poisson variable, the exponential can take on *any* non-negative values, including non-integral values such as 4.25 or 5.612401. This property makes it a poor choice for count data, which must be an integer, but a great choice for time data, temperature data (measured in Kelvins, of course), or any other precise *and positive* variable. The graph below shows two probability density functions with different $\lambda$ values.

When a random variable $Z$ has an exponential distribution with parameter $\lambda$, we say *$Z$ is exponential* and write

$$Z \sim \text{Exp}(\lambda)$$

Given a specific $\lambda$, the expected value of an exponential random variable is equal to the inverse of $\lambda$, that is:

$$E[\; Z \;|\; \lambda \;] = \frac{1}{\lambda}$$

```python
lambda_ = [0.5, 1]
d = dist.Exponential(torch.tensor(lambda_))

x = torch.linspace(0, 4, 100).view(-1, 1)
y = torch.exp(d.log_prob(x))
```

```python
for i, c in enumerate(colours):
    plt.plot(x.flatten(), y[:, i], lw=3,
             color=c, label="$\lambda = %.1f$" % lambda_[i])
    plt.fill_between(x.flatten(), y[:, i], color=c, alpha=.33)

plt.legend()
plt.ylabel("PDF at $z$")
plt.xlabel("$z$")
plt.ylim(0,1.2)
plt.title("Probability density function of an Exponential random variable;\
 differing $\lambda$");
```

### But what is $\lambda \;$?

**This question is what motivates statistics**. In the real world, $\lambda$ is hidden from us. We see only $Z$, and must go backwards to try and determine $\lambda$. The problem is difficult because there is no one-to-one mapping from $Z$ to $\lambda$. Many different methods have been created to solve the problem of estimating $\lambda$, but since $\lambda$ is never actually observed, no one can say for certain which method is best!

Bayesian inference is concerned with *beliefs* about what $\lambda$ might be. Rather than try to guess $\lambda$ exactly, we can only talk about what $\lambda$ is likely to be by assigning a probability distribution to $\lambda$.

This might seem odd at first. After all, $\lambda$ is fixed; it is not (necessarily) random! How can we assign probabilities to values of a non-random variable? Ah, we have fallen for our old, frequentist way of thinking. Recall that under Bayesian philosophy, we *can* assign probabilities if we interpret them as beliefs. And it is entirely acceptable to have *beliefs* about the parameter $\lambda$.

##### Example: Inferring behaviour from text-message data

Let's try to model a more interesting example, one that concerns the rate at which a user sends and receives text messages:

>  You are given a series of daily text-message counts from a user of your system. The data, plotted over time, appears in the chart below. You are curious to know if the user's text-messaging habits have changed over time, either gradually or suddenly. How can you model this? (This is in fact my own text-message data. Judge my popularity as you wish.)

```python
figsize(12.5, 3.5)
count_data = torch.from_numpy(np.loadtxt("data/txtdata.csv"))
n_count_data = len(count_data)
plt.bar(np.arange(n_count_data), count_data, color="#348ABD")
plt.xlabel("Time (days)")
plt.ylabel("count of text-msgs received")
plt.title("Did the user's texting habits change over time?")
plt.xlim(0, n_count_data);
```

Before we start modeling, see what you can figure out just by looking at the chart above. Would you say there was a change in behaviour during this time period?

How can we start to model this? Well, as we have conveniently already seen, a Poisson random variable is a very appropriate model for this type of *count* data. Denoting day $i$'s text-message count by $C_i$,

$$ C_i \sim \text{Poisson}(\lambda)  $$

We are not sure what the value of the $\lambda$ parameter really is, however. Looking at the chart above, it appears that the rate might become higher late in the observation period, which is equivalent to saying that $\lambda$ increases at some point during the observations. (Recall that a higher value of $\lambda$ assigns more probability to larger outcomes. That is, there is a higher probability of many text messages having been sent on a given day.)

How can we represent this observation mathematically? Let's assume that on some day during the observation period (call it $\tau$), the parameter $\lambda$ suddenly jumps to a higher value. So we really have two $\lambda$ parameters: one for the period before $\tau$, and one for the rest of the observation period. In the literature, a sudden transition like this would be called a *switchpoint*:

$$
\lambda =
\begin{cases}
\lambda_1  & \text{if } t \lt \tau \cr
\lambda_2 & \text{if } t \ge \tau
\end{cases}
$$

If, in reality, no sudden change occurred and indeed $\lambda_1 = \lambda_2$, then the $\lambda$s posterior distributions should look about equal.

We are interested in inferring the unknown $\lambda$s. To use Bayesian inference, we need to assign prior probabilities to the different possible values of $\lambda$. What would be good prior probability distributions for $\lambda_1$ and $\lambda_2$? Recall that $\lambda$ can be any positive number. As we saw earlier, the *exponential* distribution provides a continuous density function for positive numbers, so it might be a good choice for modeling $\lambda_i$. But recall that the exponential distribution takes a parameter of its own, so we'll need to include that parameter in our model. Let's call that parameter $\alpha$.

\begin{align}
&\lambda_1 \sim \text{Exp}( \alpha ) \\\
&\lambda_2 \sim \text{Exp}( \alpha )
\end{align}

$\alpha$ is called a *hyper-parameter* or *parent variable*. In literal terms, it is a parameter that influences other parameters. Our initial guess at $\alpha$ does not influence the model too strongly, so we have some flexibility in our choice.  A good rule of thumb is to set the exponential parameter equal to the inverse of the average of the count data. Since we're modeling $\lambda$ using an exponential distribution, we can use the expected value identity shown earlier to get:

$$\frac{1}{N}\sum_{i=0}^N \;C_i \approx E[\; \lambda \; |\; \alpha ] = \frac{1}{\alpha}$$

An alternative, and something I encourage the reader to try, would be to have two priors: one for each $\lambda_i$. Creating two exponential distributions with different $\alpha$ values reflects our prior belief that the rate changed at some point during the observations.

What about $\tau$? Because of the noisiness of the data, it's difficult to pick out a priori when $\tau$ might have occurred. Instead, we can assign a *uniform prior belief* to every possible day. This is equivalent to saying

\begin{align}
& \tau \sim \text{DiscreteUniform(1,70) }\\\\
& \Rightarrow P( \tau = k ) = \frac{1}{70}
\end{align}

So after all this, what does our overall prior distribution for the unknown variables look like? Frankly, *it doesn't matter*. What we should understand is that it's an ugly, complicated mess involving symbols only a mathematician could love. And things will only get uglier the more complicated our models become. Regardless, all we really care about is the posterior distribution.

We next turn to [Pyro](https://pyro.ai/), a Python library for performing Bayesian analysis that is undaunted by the mathematical monster we have created.

## Introducing our first hammer: Pyro

Pyro is a Python library for programming Bayesian analysis. It is intended for data scientists, statisticians, machine learning practitioners, and scientists. Since it is built on the PyTorch stack, it brings the runtime benefits of PyTorch to Bayesian analysis. These include write-once run-many (ability to run your development model in production) and speedups via state-of-the-art hardware (GPUs and TPUs).

Since Pyro is relatively new, the Pyro community is actively developing documentation,
especially docs and examples that bridge the gap between beginner and hacker. One of this book's main goals is to solve that problem, and also to demonstrate why TFP is so cool.

We will model the problem above using Pyro. This type of programming is called *probabilistic programming*, an unfortunate misnomer that invokes ideas of randomly-generated code and has likely confused and frightened users away from this field. The code is not random; it is probabilistic in the sense that we create probability models using programming variables as the model's components.

B. Cronin [[4]](#scrollTo=nDdph0r1ABCn) has a very motivating description of probabilistic programming:

>   Another way of thinking about this: unlike a traditional program, which only runs in the forward directions, a probabilistic program is run in both the forward and backward direction. It runs forward to compute the consequences of the assumptions it contains about the world (i.e., the model space it represents), but it also runs backward from the data to constrain the possible explanations. In practice, many probabilistic programming systems will cleverly interleave these forward and backward operations to efficiently home in on the best explanations.

Because of the confusion engendered by the term *probabilistic programming*, I'll refrain from using it. Instead, I'll simply say *programming*, since that's what it really is.

Pyro code is easy to read. The only novel thing should be the syntax. Simply remember that we are representing the model's components ($\tau, \lambda_1, \lambda_2$ ) as variables.

```python
def model(data):
    alpha = 1.0 / data.mean()
    lambda_1 = pyro.sample("lambda_1", dist.Exponential(alpha))
    lambda_2 = pyro.sample("lambda_2", dist.Exponential(alpha))

    tau = pyro.sample("tau", dist.Uniform(0, 1))
    lambda1_size = (tau * data.size(0) + 1).long()
    lambda2_size = data.size(0) - lambda1_size
    lambda_ = torch.cat([lambda_1.expand((lambda1_size,)),
                         lambda_2.expand((lambda2_size,))])

    with pyro.plate("data", data.size(0)):
        pyro.sample("obs", dist.Poisson(lambda_), obs=data)
```

```python
kernel = NUTS(model, jit_compile=True, ignore_jit_warnings=True, max_tree_depth=3)
posterior = MCMC(kernel, num_samples=5000, warmup_steps=500)
posterior.run(count_data);
```

    Sample: 100%|██████████| 5500/5500 [00:43, 126.11it/s, step size=2.45e-01, acc. prob=0.802]

```python
hmc_samples = {k: v.detach().cpu().numpy() for k, v in posterior.get_samples().items()}
lambda_1_samples = hmc_samples['lambda_1']
lambda_2_samples = hmc_samples['lambda_2']
tau_samples = (hmc_samples['tau'] * count_data.size(0) + 1).astype(int)
```

```python
figsize(12.5, 10)
#histogram of the samples:

ax = plt.subplot(311)
ax.set_autoscaley_on(False)

plt.hist(lambda_1_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label="posterior of $\lambda_1$", color="#A60628", density=True)
plt.legend(loc="upper left")
plt.title(r"""Posterior distributions of the variables
    $\lambda_1,\;\lambda_2,\;\tau$""")
plt.xlim([15, 30])
plt.xlabel("$\lambda_1$ value")

ax = plt.subplot(312)
ax.set_autoscaley_on(False)
plt.hist(lambda_2_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label="posterior of $\lambda_2$", color="#7A68A6", density=True)
plt.legend(loc="upper left")
plt.xlim([15, 30])
plt.xlabel("$\lambda_2$ value")

plt.subplot(313)
w = 1.0 / tau_samples.shape[0] * np.ones_like(tau_samples)
plt.hist(tau_samples, bins=n_count_data, alpha=1,
         label=r"posterior of $\tau$",
         color="#467821", weights=w, rwidth=2.)
plt.xticks(np.arange(n_count_data))

plt.legend(loc="upper left")
plt.ylim([0, .75])
plt.xlim([35, len(count_data)-20])
plt.xlabel(r"$\tau$ (in days)")
plt.ylabel("probability");
```

### Interpretation

Recall that Bayesian methodology returns a *distribution*. Hence we now have distributions to describe the unknown $\lambda$s and $\tau$. What have we gained? Immediately, we can see the uncertainty in our estimates: the wider the distribution, the less certain our posterior belief should be. We can also see what the plausible values for the parameters are: $\lambda_1$ is around 18 and $\lambda_2$ is around 23. The posterior distributions of the two $\lambda$s are clearly distinct, indicating that it is indeed likely that there was a change in the user's text-message behaviour.

What other observations can you make? If you look at the original data again, do these results seem reasonable?

Notice also that the posterior distributions for the $\lambda$s do not look like exponential distributions, even though our priors for these variables were exponential. In fact, the posterior distributions are not really of any form that we recognize from the original model. But that's OK! This is one of the benefits of taking a computational point of view. If we had instead done this analysis using mathematical approaches, we would have been stuck with an analytically intractable (and messy) distribution. Our use of a computational approach makes us indifferent to mathematical tractability.

Our analysis also returned a distribution for $\tau$. Its posterior distribution looks a little different from the other two because it is a discrete random variable, so it doesn't assign probabilities to intervals. We can see that near day 45, there was a 50% chance that the user's behaviour changed. Had no change occurred, or had the change been gradual over time, the posterior distribution of $\tau$ would have been more spread out, reflecting that many days were plausible candidates for $\tau$. By contrast, in the actual results we see that only three or four days make any sense as potential transition points.

### Why would I want samples from the posterior, anyways?

We will deal with this question for the remainder of the book, and it is an understatement to say that it will lead us to some amazing results. For now, let's end this chapter with one more example.

We'll use the posterior samples to answer the following question: what is the expected number of texts at day $t, \; 0 \le t \le 70$ ? Recall that the expected value of a Poisson variable is equal to its parameter $\lambda$. Therefore, the question is equivalent to *what is the expected value of $\lambda$ at time $t$*?

In the code below, let $i$ index samples from the posterior distributions. Given a day $t$, we average over all possible $\lambda_i$ for that day $t$, using $\lambda_i = \lambda_{1,i}$ if $t \lt \tau_i$ (that is, if the behaviour change has not yet occurred), else we use $\lambda_i = \lambda_{2,i}$.

```python
figsize(12.5, 5)
# tau_samples, lambda_1_samples, lambda_2_samples contain
# N samples from the corresponding posterior distribution
N = tau_samples.shape[0]
expected_texts_per_day = np.zeros(n_count_data)
for day in range(0, n_count_data):
    # ix is a bool index of all tau samples corresponding to
    # the switchpoint occurring prior to value of 'day'
    ix = day < tau_samples
    # Each posterior sample corresponds to a value for tau.
    # for each day, that value of tau indicates whether we're "before"
    # (in the lambda1 "regime") or
    #  "after" (in the lambda2 "regime") the switchpoint.
    # by taking the posterior sample of lambda1/2 accordingly, we can average
    # over all samples to get an expected value for lambda on that day.
    # As explained, the "message count" random variable is Poisson distributed,
    # and therefore lambda (the poisson parameter) is the expected value of
    # "message count".
    expected_texts_per_day[day] = (lambda_1_samples[ix].sum()
                                   + lambda_2_samples[~ix].sum()) / N

plt.plot(range(n_count_data), expected_texts_per_day, lw=4, color="#E24A33",
         label="expected number of text-messages received")
plt.xlim(0, n_count_data)
plt.xlabel("Day")
plt.ylabel("Expected # text-messages")
plt.title("Expected number of text-messages received")
plt.ylim(0, 60)
plt.bar(np.arange(len(count_data)), count_data, color="#348ABD", alpha=0.65,
        label="observed texts per day")

plt.legend(loc="upper left");
```

Our analysis shows strong support for believing the user's behavior did change ($\lambda_1$ would have been close in value to $\lambda_2$ had this not been true), and that the change was sudden rather than gradual (as demonstrated by $\tau$'s strongly peaked posterior distribution). We can speculate what might have caused this: a cheaper text-message rate, a recent weather-to-text subscription, or perhaps a new relationship. (In fact, the 45th day corresponds to Christmas, and I moved away to Toronto the next month, leaving a girlfriend behind.)

##### Exercises

1\.  Using `lambda_1_samples` and `lambda_2_samples`, what is the mean of the posterior distributions of $\lambda_1$ and $\lambda_2$?

```python
# Your code here.
```

2\.  What is the expected percentage increase in text-message rates? `hint:` compute the mean of `lambda_1_samples/lambda_2_samples`. Note that this quantity is very different from `lambda_1_samples.mean()/lambda_2_samples.mean()`.

```python
# Your code here.
```

3\. What is the mean of $\lambda_1$ **given** that we know $\tau$ is less than 45.  That is, suppose we have been given new information that the change in behaviour occurred prior to day 45. What is the expected value of $\lambda_1$ now? (You do not need to redo the PyMC3 part. Just consider all instances where `tau_samples < 45`.)

```python
# Your code here.
```

### References

-  [1] Gelman, Andrew. N.p.. Web. 22 Jan 2013. [N is never large enough](http://andrewgelman.com/2005/07/31/n_is_never_larg).
-  [2] Norvig, Peter. 2009. [The Unreasonable Effectiveness of Data](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf).
- [3] Salvatier, J, Wiecki TV, and Fonnesbeck C. (2016) Probabilistic programming in Python using PyMC3. *PeerJ Computer Science* 2:e55 <https://doi.org/10.7717/peerj-cs.55>
- [4] Jimmy Lin and Alek Kolcz. Large-Scale Machine Learning at Twitter. Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (SIGMOD 2012), pages 793-804, May 2012, Scottsdale, Arizona.
- [5] Cronin, Beau. "Why Probabilistic Programming Matters." 24 Mar 2013. Google, Online Posting to Google . Web. 24 Mar. 2013. <https://plus.google.com/u/0/107971134877020469960/posts/KpeRdJKR6Z1>.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunss.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsx.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsi.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunso.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 130%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 22pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```python

```


```python
from IPython.core.display import HTML
HTML("""
<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>
""")
```

<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>

# *Circuitos Elétricos I - Semana 11*

### A integral de Laplace

Seja $f(t)$ uma função definida no intervalo $0\leq t \leq \infty$, com $t$ e $f(t)$ reais, então a função $F(s)$, definida pela integral de Laplace

$$\large
\begin{equation}
F(s)=\mathcal{L}\{f(t)\}=\int_{0}^{\infty} f(t) e^{-s t} dt,\;\; s \in \mathbb{C},
\end{equation}
$$

é conhecida como a transformada de Laplace de $f(t)$.

Para informações sobre como utilizar o Sympy para o cálculo da transformada de Laplace:

https://dynamics-and-control.readthedocs.io/en/latest/1_Dynamics/3_Linear_systems/Laplace%20transforms.html

```python
import matplotlib.pyplot as plt
import numpy as np
import sympy as sp
from utils import round_expr, symdisp, symplot

from sympy.polys.partfrac import apart

# temp workaround
import warnings
from matplotlib import MatplotlibDeprecationWarning
warnings.filterwarnings('ignore', category=MatplotlibDeprecationWarning)

plt.rcParams['figure.figsize'] = 6, 4
plt.rcParams['legend.fontsize'] = 13
plt.rcParams['lines.linewidth'] = 2
plt.rcParams['axes.grid'] = False
```

```python
# transformada de Laplace
def L(f,t,s):
    return sp.laplace_transform(f, t, s, noconds=True)

# transformada inversa de Laplace
def invL(F,s,t):
    return sp.re(sp.inverse_laplace_transform(F, s, t, noconds=True))

# funções para auxílio na expansão em frações parciais
def adjustCoeff(expr):
    coeff = expr.as_numer_denom()
    c0 = sp.poly(coeff[1].cancel()).coeffs()[0]

    return (coeff[0].cancel()/c0)/(coeff[1].cancel()/c0)

def partFrac(expr, Ndigits):
    expr = expr.cancel()
    expr = apart(adjustCoeff(expr), s, full=True).doit()

    return sp.N(expr, Ndigits)

sp.init_printing()
```

#### Definindo algumas variáveis simbólicas de interesse

```python
t, s  = sp.symbols('t, s')
a     = sp.symbols('a', real=True, positive=True)
omega = sp.symbols('omega', real=True)
```

## Gere sua tabela de transformadas

```python
func = [1,
         t,
         sp.exp(-a*t),
         t*sp.exp(-a*t),
         t**2*sp.exp(-a*t),
         sp.sin(omega*t),
         sp.cos(omega*t),
         1 - sp.exp(-a*t),
         sp.exp(-a*t)*sp.sin(omega*t),
         sp.exp(-a*t)*sp.cos(omega*t),
         ]
func
```

```python
Fs = [L(f,t,s) for f in func]
Fs
```

### Problema 1

Não existe nenhuma energia armazenada no circuito da figura a seguir no momento em que a fonte de corrente é ligada.

a. Determine $I_a(s)$ e $I_b(s)$.\
b. Determine $i_a(t)$ e $i_b(t)$.\
c. Determine $V_a(s)$, $V_b(s)$ e $V_c(s)$.\
d. Determine $v_a(t)$, $v_b(t)$ e $v_c(t)$.

a. Determinando $I_a(s)$ e $I_b(s)$:

```python
I2, I3, s = sp.symbols('I2, I3, s')

# define os sistema de equações
eq1 = sp.Eq((4 + s)*I2 - 2*I3, 4)
eq2 = sp.Eq(-2*s*I2 + s*(4 + s)*I3, 8)

# resolve o sistema
soluc = sp.solve([eq1, eq2],[I2, I3], dict=True)
soluc

I2 = [sol[I2] for sol in soluc]
I3 = [sol[I3] for sol in soluc]

I1 = 4/s
I2 = I2[0]
I3 = I3[0]

print('Correntes de malha no domínio de Laplace: \n')
symdisp('I_1(s) =', I1, 'As')
symdisp('I_2(s) =', I2, 'As')
symdisp('I_3(s) =', I3, 'As')
```

    Correntes de malha no domínio de Laplace:

$\displaystyle I_1(s) =\frac{4}{s}\;As$

$\displaystyle I_2(s) =\frac{4 s + 8}{s^{2} + 6 s}\;As$

$\displaystyle I_3(s) =\frac{16}{s^{2} + 6 s}\;As$

```python
# Calculando Ia
Ia = I1-I2
Ia = Ia.simplify()

symdisp('I_a(s) =', Ia, 'As')
```

$\displaystyle I_a(s) =\frac{16}{s \left(s + 6\right)}\;As$

```python
# Calculando Ib
Ib = I2

symdisp('I_b(s) =', Ib, 'As')
```

$\displaystyle I_b(s) =\frac{4 s + 8}{s^{2} + 6 s}\;As$

b. Determinando $i_a(t)$ e $i_b(t)$

```python
symdisp('I_a(s) =', Ia.apart(), 'As')
```

$\displaystyle I_a(s) =- \frac{8}{3 \left(s + 6\right)} + \frac{8}{3 s}\;As$

```python
t = sp.symbols('t',real=True)

ia = invL(Ia.apart(),s,t)

symdisp('i_a(t) =', ia, 'A')
```

$\displaystyle i_a(t) =\frac{8 \left(e^{6 t} - 1\right) e^{- 6 t} \theta\left(t\right)}{3}\;A$

```python
symdisp('I_b(s) =', Ib.apart(), 'As')
```

$\displaystyle I_b(s) =\frac{8}{3 \left(s + 6\right)} + \frac{4}{3 s}\;As$

```python
ib = invL(Ib,s,t)

symdisp('i_b(t) =', ib, 'A')
```

$\displaystyle i_b(t) =\frac{4 \left(e^{6 t} + 2\right) e^{- 6 t} \theta\left(t\right)}{3}\;A$

c. Determinando $V_a(s)$, $V_b(s)$ e $V_c(s)$.

```python
Va = (100/s)*I2
Vb = (100/s)*(I3-I2)
Vc = (100/s)*(I1-I3)

symdisp('V_a(s) =', Va.simplify(), 'Vs')
symdisp('V_b(s) =', Vb.simplify(), 'Vs')
symdisp('V_c(s) =', Vc.simplify(), 'Vs')
```

$\displaystyle V_a(s) =\frac{400 \left(s + 2\right)}{s^{2} \left(s + 6\right)}\;Vs$

$\displaystyle V_b(s) =\frac{400 \left(2 - s\right)}{s^{2} \left(s + 6\right)}\;Vs$

$\displaystyle V_c(s) =\frac{400 \left(s + 2\right)}{s^{2} \left(s + 6\right)}\;Vs$

```python
symdisp('V_a(s) =', Va.apart(), 'Vs')
```

$\displaystyle V_a(s) =- \frac{400}{9 \left(s + 6\right)} + \frac{400}{9 s} + \frac{400}{3 s^{2}}\;Vs$

```python
symdisp('V_b(s) =', Vb.apart(), 'Vs')
```

$\displaystyle V_b(s) =\frac{800}{9 \left(s + 6\right)} - \frac{800}{9 s} + \frac{400}{3 s^{2}}\;Vs$

```python
symdisp('V_c(s) =', Vc.apart(), 'Vs')
```

$\displaystyle V_c(s) =- \frac{400}{9 \left(s + 6\right)} + \frac{400}{9 s} + \frac{400}{3 s^{2}}\;Vs$

d. Determinando $v_a(t)$, $v_b(t)$ e $v_c(t)$.

```python
va = ((-400/9)*sp.exp(-6*t) + (400/9) + (400/3)*t)*sp.Heaviside(t)

symdisp('v_a(t) =', round_expr(va,2), 'V')
```

$\displaystyle v_a(t) =\left(133.33 t + 44.44 - 44.44 e^{- 6 t}\right) \theta\left(t\right)\;V$

```python
vb = ((800/9)*sp.exp(-6*t) - (800/9) + (400/3)*t)*sp.Heaviside(t)

symdisp('v_b(t) =', round_expr(vb,2), 'V')
```

$\displaystyle v_b(t) =\left(133.33 t - 88.89 + 88.89 e^{- 6 t}\right) \theta\left(t\right)\;V$

```python
vc = va

symdisp('v_c(t) =', round_expr(vc,2), 'V')
```

$\displaystyle v_c(t) =\left(133.33 t + 44.44 - 44.44 e^{- 6 t}\right) \theta\left(t\right)\;V$

```python
# plota funções no domínio do tempo
intervalo = np.arange(-4, 10, 0.1)
symplot(t, [va, vb, vc], intervalo, ['va(t)','vb(t)','vc(t)'])
```

Pergunta: estas soluções fazem sentido para o circuito analisado?

### Problema 2

Não existe nenhuma energia armazenada no circuito da figura a seguir no momento em que a fonte de tensão é conectada.

a. Determine $V_0(s)$.\
b. Determine $v_0(t)$.

```python
Ia, Ib, s = sp.symbols('Ia, Ib, s')

# define o sistema de equações
eq1 = sp.Eq(10*Ia + (s + 250/s)*Ib, 35/s)
eq2 = sp.Eq(Ia - (1 + 0.4*s)*Ib, 0)

# resolve o sistema
soluc = sp.solve([eq1, eq2],[Ia, Ib], dict=True)
soluc

Ia = [sol[Ia] for sol in soluc]
Ib = [sol[Ib] for sol in soluc]

Ia = Ia[0]
Ib = Ib[0]
```

```python
symdisp('I_a(s) =', Ia.simplify(), 'As')
```

$\displaystyle I_a(s) =\frac{14.0 s + 35.0}{5.0 s^{2} + 10.0 s + 250.0}\;As$

```python
symdisp('I_b(s) =', Ib.simplify(), 'As')
```

$\displaystyle I_b(s) =\frac{7.0}{s^{2} + 2.0 s + 50.0}\;As$

```python
V0 = 35/s - 2*Ia

symdisp('V_0(s) =', V0.simplify(), 'Vs')
```

$\displaystyle V_0(s) =\frac{147.0 s^{2} + 280.0 s + 8750.0}{s \left(5.0 s^{2} + 10.0 s + 250.0\right)}\;Vs$

```python
symdisp('V_0(s) =', partFrac(V0, 2), 'Vs')
```

$\displaystyle V_0(s) =\frac{-2.8 - 0.59 i}{s + 1.0 + 7.0 i} + \frac{-2.8 + 0.59 i}{s + 1.0 - 7.0 i} + \frac{35.0}{s}\;Vs$

```python
raizes = np.roots([1, 2, 50, 0])
raizes
```

    array([-1.+7.j, -1.-7.j,  0.+0.j])

```python
K    = sp.symbols('K')
σ, ω = sp.symbols('σ, ω', real=True)

j = sp.I

F = K/(s + σ + j*ω) + sp.conjugate(K)/(s + σ - j*ω)

symdisp('F(s) =', F)
```

$\displaystyle F(s) =\frac{K}{s + σ + i ω} + \frac{\overline{K}}{s + σ - i ω}\; $

```python
symdisp('f(t) =', invL(F,s,t))
```

$\displaystyle f(t) =\left(2 \sin{\left(t ω \right)} \operatorname{im}{\left(K\right)} + 2 \cos{\left(t ω \right)} \operatorname{re}{\left(K\right)}\right) e^{- t σ} \theta\left(t\right)\; $

```python
v0 = (35 + sp.exp(-t)*(-5.6*sp.cos(7*t)-1.2*sp.sin(7*t)))*sp.Heaviside(t)

symdisp('v_0(t) =', v0)
```

$\displaystyle v_0(t) =\left(\left(- 1.2 \sin{\left(7 t \right)} - 5.6 \cos{\left(7 t \right)}\right) e^{- t} + 35\right) \theta\left(t\right)\; $

```python
# plota funções no domínio do tempo
intervalo = np.arange(-4, 10, 0.05)
symplot(t, v0, intervalo, 'v0(t)')
```

Jupyter notebooks/.ipynb_checkpoints/Circuitos Eletricos I - Semana 11.2-checkpoint.ipynb


```python
import holoviews as hv
hv.extension('bokeh')
hv.opts.defaults(hv.opts.Curve(width=500),
                 hv.opts.Points(width=500),
                 hv.opts.Image(width=500, colorbar=True, cmap='Viridis'))
```

```python
import numpy as np
import scipy.linalg
```

# Estimador lineal óptimo

Un **estimador** es un sistema diseñado para **extraer información** a partir de una **señal**

- La señal contiene **información y ruido**
- La señal es representada como una secuencia de **datos**

Tipos de estimador

- **Filtro:** Estimo el valor actual de mi señal acentuando o eliminando una o más características
- **Predictor:** Estimo el valor futuro de mi señal

En esta lección estudiaremos estimadores lineales y óptimos

- Lineal: La cantidad estimada es una función lineal de la entrada
- Óptimo: El estimador es la mejor solución posible de acuerdo a un criterio

Para entender los fundamentos de los estimadores óptimos es necesario introducir el concepto de proceso aleatorio. Luego estudiaremos uno de los estimadores óptimos más importantes: El filtro de Wiener

## Proceso aleatorio o proceso estocástico

Un proceso estocástico es una **colección de variables aleatorias** indexadas tal que forman una secuencia. Se denotan matemáticamente como un conjunto $\{U_k\}$, con $k=0, 1, 2, \ldots, N$. El índice $k$ puede representar tiempo, espacio u otra variable independiente.

La siguiente figura muestra tres realizaciones u observaciones de un proceso estocástico con cuatro elementos

Existen muchos fenómenos cuya evolución se modela utilizando procesos aleatorios. Por ejemplo

- Los índices bursatiles
- El comportamineto de un gas dentro de un contenedor
- Las vibraciones de un motor eléctrico
- El área de una célula durante un proceso de organogénesis

A continuación revisaremos algunas de las propiedades de los procesos aleatorios

**Momentos de un proceso estocástico**

Un proceso aleatorio $U_n = (u_n, u_{n-1}, u_{n-2}, \ldots, u_{n-L})$ se describe a través de sus momentos estadísticos. Si consideramos una caracterízación de segundo orden necesitamos definir

- Momento central o media: Describe el valor central del proceso

$$
\mu(n) = \mathbb{E}[U_n]
$$

- Segundo momento o correlación: Describe la dispersión de un proceso

$$
r_{uu}(n, n-k) = \mathbb{E}[U_n U_{n-k}]
$$

- Segundo momento centrado o covarianza

$$
\begin{align}
c_{uu}(n, n-k) &= \mathbb{E}[(U_n-\mu_n) (U_{n-k}- \mu_{n-k})] \nonumber \\
&= r(n,n-k) - \mu_n \mu_{n-k} \nonumber
\end{align}
$$

- Correlación cruzada entre dos procesos

$$
r_{ud}(n, n-k) = \mathbb{E}[U_n D_{n-k}]
$$

**Proceso estacionario y ergódico**

En esta lección nos vamos a centrar en el caso simplificado donde el **proceso es estacionario**, matemáticamente esta propiedad significa que

$$
\mu(n)  = \mu, \forall n
$$

y

$$
r_{uu}(n, n-k)  = r_{uu}(k), \forall n
$$

es decir que los momentos estadísticos se mantienen constantes en el tiempo (no depende de $n$).

Otra simplificación que utilizaremos es que el proceso sea **ergódico**,

$$
\mathbb{E}[U_n] = \frac{1}{N} \sum_{n=1}^N u_n
$$

es decir que podemos reemplazar el valor esperado por la media muestral en el tiempo

**Densidad espectral de potencia**

La densidad espectral de potencia o *power spectral density* (PSD) mide la distribución en frecuencia de la potencia del proceso estocástico. Su definición matemática es

$$
\begin{align}
S_{uu}(f) &= \sum_{k=-\infty}^{\infty} r_{uu}(k) e^{-j 2\pi f k} \nonumber \\
&= \lim_{N\to\infty} \frac{1}{2N+1} \mathbb{E} \left [\left|\sum_{n=-N}^{N} u_n e^{-j 2\pi f n} \right|^2 \right]
\end{align}
$$

que corresponde a la transformada de Fourier de la correlación (caso estacionario)

La PSD y la correlación forman un par de Fourier, es decir que uno es la transformada de Fourier del otro.

## Filtro de Wiener

El filtro de Wiener fue publicado por Norbert Wiener en 1949 y es tal vez el ejemplo más famoso de un estimador lineal óptimo.

:::{important}

Para diseñar un estimador óptimo necesitamos un **criterio** y **condiciones** (supuestos). Luego el estimador será **óptimo según dicho criterio y bajo los supuestos considerados**. Por ejemplo podríamos suponer un escenario donde el ruido es blanco o donde el proceso es estacionario.

:::

A continuación describiremos en detalle este filtro y explicaremos como se optimiza. Luego se veran ejemplos de aplicaciones.

### Notación y arquitectura del filtro de Wiener

El filtro de Wiener es un sistema de tiempo discreto con estructura FIR y $L+1$ coeficientes. A continuación se muestra un esquema del filtro de Wiener

Del esquema podemos reconocer los elementos más importantes de este filtro

- Los coeficientes del filtro: $h_0, h_1, h_2, \ldots, h_{L}$
- La señal de entrada al filtro: $u_0, u_1, u_2, \ldots$
- La señal de salida del filtro: $y_0, y_1, y_2, \ldots$
- La señal de respuesta "deseada" u objetivo: $d_0, d_1, d_2, \ldots$
- La señal de error: $e_0, e_1, e_2, \ldots$

Al ser un filtro FIR la salida del filtro está definida como

$$
y_n = \sum_{k=0}^{L} h_k u_{n-k},
$$

es decir la convolución entre la entrada y los coeficientes. Luego la señal de error es

$$
e_n = d_n - y_n = d_n - \sum_{k=0}^{L} h_k u_{n-k}
$$

que corresponde a la diferencia entre la señal objetivo y la señal de salida.

A continuación veremos que se ajustan los coeficientes del filtro en base al criterio de optimalidad.

### Ajuste del filtro de Wiener

El criterio más común para aprender o adaptar el filtro de Wiener es el **error medio cuadrático** o *mean square error* (MSE) entre la respuesta deseada y la salida del filtro.

Asumiendo que $u$ y $d$ son secuencias de valores reales podemos escribir el MSE como

$$
\begin{align}
\text{MSE} &= \mathbb{E}\left [e_n^2 \right] \nonumber \\
&= \mathbb{E}\left [(d_n - y_n)^2 \right] \nonumber \\
&= \mathbb{E}\left [d_n^2 \right]  - 2\mathbb{E}\left [ d_n y_n \right] + \mathbb{E}\left [ y_n^2 \right] \nonumber
\end{align}
$$

donde $\sigma_d^2 = \mathbb{E}\left [d_n^2 \right]$ es la varianza de la señal deseada y $\sigma_y^2 = \mathbb{E}\left [ y_n^2 \right]$ es la varianza de nuestro estimador

:::{note}

Minimizar el MSE implica acercar la salida del filtro a la respuesta deseada

:::

En este caso, igualando la derivada del MSE a cero, tenemos

$$
\begin{align}
\frac{d}{d h_j} \text{MSE} &= -2\mathbb{E}\left[ d_n \frac{d y_n}{d h_j}  \right]  + 2 \mathbb{E}\left[ y_n \frac{d y_n}{d h_j}    \right]  \nonumber \\
&= -2\mathbb{E}\left[ d_n u_{n-j} \right]  + 2 \mathbb{E}\left[ y_n u_{n-j}    \right]  \nonumber \\
&= -2\mathbb{E}\left[ d_n u_{n-j} \right]  + 2 \mathbb{E}\left[ \sum_{k=0}^{L} h_k u_{n-k}  u_{n-j} \right] \nonumber \\
&= -2\mathbb{E}\left[ d_n u_{n-j} \right]  + 2 \sum_{k=0}^{L} h_k \mathbb{E}\left[ u_{n-k}  u_{n-j} \right] = 0 \nonumber \end{align}
$$

Si despejamos y repetimos para $j=0, \ldots, L$ obtenemos el siguiente sistema de ecuaciones

$$
\begin{align}
\begin{pmatrix}
r_{uu}(0) & r_{uu}(1) & r_{uu}(2) & \ldots & r_{uu}(L) \\
r_{uu}(1) & r_{uu}(0) & r_{uu}(1) & \ldots & r_{uu}(L-1) \\
r_{uu}(2) & r_{uu}(1) & r_{uu}(0) & \ldots & r_{uu}(L-2) \\
\vdots & \vdots & \vdots & \ddots &\vdots \\
r_{uu}(L) & r_{uu}(L-1) & r_{uu}(L-2) & \ldots & r_{uu}(0) \\
\end{pmatrix}
\begin{pmatrix}
h_0  \\
h_1  \\
h_2  \\
\vdots  \\
h_L \\
\end{pmatrix} &=
\begin{pmatrix}
r_{ud}(0)  \\
r_{ud}(1)  \\
r_{ud}(2) \\
\vdots  \\
r_{ud}(L) \\
\end{pmatrix} \nonumber \\
R_{uu} \textbf{h} &= R_{ud},
\end{align}
$$

que se conoce como las **ecuaciones de Wiener-Hopf**. Además $R_{uu}$ se conoce como matriz de auto-correlación.

Asumiendo que $R_{uu}$ es no-singular, es decir que su inversa existe, la **solución óptima en el sentido de mínimo MSE** es

$$
\textbf{h}^{*} = R_{uu} ^{-1} R_{ud}
$$

Notar que por construcción la matriz $R_{uu}$ es simétrica y hermítica. Por lo que el sistema puede resolverse de forma eficiente con  $\mathcal{O}(L^2)$ operaciones usando la [recursión de Levison-Durbin](https://en.wikipedia.org/wiki/Levinson_recursion)

:::{warning}

Para llegar a la solución impusimos dos condiciones sobre la salida deseada y la entrada: (1) tienen media cero y (2) son estacionarias en el sentido amplio (es decir la correlación solo depende del retardo $m$).

:::

- Si la primera condición no se cumpliera, podría restarse la media previo al entrenamiento del filtro
- Si la segunda condición no se cumple conviene usar otro método como los que veremos en las lecciones siguientes

## Aplicaciones del filtro de Wiener

### Regresión  o identificación de sistema

En regresión buscamos encontrar los coeficientes $h$ a partir de tuplas $(X, Y)$ tal que

$$
Y = h^T X + \epsilon,
$$

donde $X \in \mathbb{R}^{N\times D}$ son las variables dependientes (entrada), $Y \in \mathbb{R}^N$ es la  variable dependiente (salida) y $\epsilon$ es ruido

Para entrenar el filtro

1. Asumimos que hemos observado N muestras de $X$ e $Y$
1. A partir de $u=X$ construimos $R_{uu}$
1. A partir de $d=Y$ construimos $R_{ud}$
1. Finalmente recuperamos $\textbf{h}$ usando $R_{uu} ^{-1} R_{ud}$
1. Con esto podemos interpolar $Y$

**Ejemplo** Sea por ejemplo una regresión de tipo polinomial donde queremos encontrar $h_k$ tal que

$$
\begin{align}
d_i &= f_i + \epsilon \nonumber \\
&= \sum_{k=1}^L h_k u_i^k + \epsilon \nonumber
\end{align}
$$

```python
np.random.seed(12345)
u = np.linspace(-2, 2, num=30)
f = 0.25*u**5 - 2*u**3 + 5*u # Los coeficientes reales son [0, 5, 0, -2, 0, 1/4, 0, 0, 0, ...]
d = f + np.random.randn(len(u))
```

```python
hv.Points((u, d), kdims=['u', 'd'])
```

Implementemos el filtro como una clase con dos métodos públicos `fit` (ajustar) y `predict` (predecir). El filtro tiene un argumento, el número de coeficientes $L$

```python
class Wiener_polynomial_regression:

    def __init__(self, L: int):
        self.L = L
        self.h = np.zeros(shape=(L+1,))

    def _polynomial_basis(self, u: np.ndarray) -> np.ndarray:
        U = np.ones(shape=(len(u), self.L))
        for i in range(1, self.L):
            U[:, i] = u**i
        return U

    def fit(self, u: np.ndarray, d: np.ndarray):
        U = self._polynomial_basis(u)
        Ruu = np.dot(U.T, U)
        Rud = np.dot(U.T, d[:, np.newaxis])
        self.h = scipy.linalg.solve(Ruu, Rud, assume_a='pos')[:, 0]

    def predict(self, u: np.ndarray):
        U = self._polynomial_basis(u)
        return np.dot(U, self.h)
```

:::{note}

La función `scipy.linalg.solve(A, B)` retorna la solución del sistema de ecuaciones lineal `Ax = B`. El argumento `assume_a` puede usarse para indicar que `A` es simétrica, hermítica o definido positiva.

:::

Los solución de un sistema con `10` coeficientes es:

```python
regressor = Wiener_polynomial_regression(10)
regressor.fit(u, d)
print(regressor.h)
```

¿Cómo cambia el resultado con L?

```python
uhat = np.linspace(np.amin(u), np.amax(u), num=100)
yhat = {}
for L in [2, 5, 15]:
    regressor = Wiener_polynomial_regression(L)
    regressor.fit(u, d)
    yhat[L] = regressor.predict(uhat)
```

```python
p = [hv.Points((u, d), kdims=['u', 'd'], label='data').opts(size=4, color='k')]
for L, prediction in yhat.items():
    p.append(hv.Curve((uhat, prediction), label=f'L={L}'))
hv.Overlay(p).opts(legend_position='top')
```

:::{note}

Si $L$ es muy pequeño el filtro es demasiado simple. Si $L$ es muy grande el filtro se puede sobreajustar al ruido

:::

### Predicción a futuro

En este caso asumimos que la señal deseada es la entrada en el futuro

$$
d_n = \{u_{n+1}, u_{n+2}, \ldots, u_{n+m}\}
$$

donde $m$ es el horizonte de predicción. Se llama *predicción a un paso* al caso particular $m=1$.

El largo del filtro $L$ define la cantidad de muestras pasadas que usamos para predecir. Por ejemplo un sistema de predicción a un paso con $L+1 = 3$ coeficientes:

$$
h_0 u_n +  h_1 u_{n-1} + h_2 u_{n-2}= y_n = \hat u_{n+1} \approx u_{n+1}
$$

Para entrenar el filtro

1. Asumimos que la señal ha sido observada y que se cuenta con $N$ muestras para entrenar
1. Podemos formar una matriz cuyas filas son $[u_n, u_{n-1}, \ldots, u_{n-L}]$ para $n=L,L+1,\ldots, N-1$
1. Podemos formar un vector $[u_N, u_{N-1}, \ldots, u_{L+1}]^T$ (caso $m=1$)
1. Con esto podemos formar las matrices de correlación y obtener $\textbf{h}$
1. Finalmente usamos $\textbf{h}$ para predecir el futuro no observado de $u$

Nuevamente implementamos el filtro como una clase. Esta vez se utiliza la función de numpy `as_strided` para formar los vectores de "instantes pasados"

```python
from numpy.lib.stride_tricks import as_strided

class Wiener_predictor:

    def __init__(self, L: int):
        self.L = L
        self.h = np.zeros(shape=(L+1,))

    def fit(self, u: np.ndarray):
        U = as_strided(u, [len(u)-self.L+1 , self.L+1],
                       strides=[u.strides[0], u.strides[0]])
        Ruu = np.dot(U[:, :self.L].T, U[:, :self.L])
        Rud = np.dot(U[:, :self.L].T, U[:, self.L][:, np.newaxis])
        self.h = scipy.linalg.solve(Ruu, Rud, assume_a='pos')[:, 0]

    def predict(self, u: np.ndarray, m: int=1):
        u_pred = np.zeros(shape=(m+self.L, ))
        u_pred[:self.L] = u
        for k in range(self.L, m+L):
            u_pred[k] = np.sum(self.h*u_pred[k-self.L:k])
        return u_pred[self.L:]
```

Para la siguiente señal sinusoidal ¿Cómo afecta $L$ a la calidad del predictor lineal?

Utilizaremos los primeros 100 instantes para ajustar y los siguientes 100 para probar el predictor

```python
np.random.seed(12345)
t = np.linspace(0, 10, num=200)
u = np.sin(2.0*np.pi*0.5*t) + 0.25*np.random.randn(len(t))
N_fit = 100

yhat = {}
for L in [10, 20, 30]:
    predictor = Wiener_predictor(L)
    predictor.fit(u[:N_fit])
    yhat[L] = predictor.predict(u[N_fit-L:N_fit], m=100)

```

```python
p = [hv.Points((t, u), ['instante', 'u'], label='Datos').opts(color='k')]
for L, prediction in yhat.items():
    p.append(hv.Curve((t[N_fit:], prediction), label=f'L={L}'))
hv.Overlay(p).opts(legend_position='top')
```

:::{note}

Si $L$ es muy pequeño el filtro es demasiado simple. Si $L$ es muy grande el filtro se puede sobreajustar al ruido

:::

### Eliminar ruido blanco aditivo

En este caso asumimos que la señal de entrada corresponde a una señal deseada (información) que ha sido contaminada con ruido aditivo

$$
u_n = d_n + \nu_n,
$$

adicionalmente asumimos que
- el ruido es estacionario en el sentido amplio y de media cero $\mathbb{E}[\nu_n] = 0$
- el ruido es blanco, es decir no tiene correlación consigo mismo o con la señal deseada

$$
r_{\nu d}(k) = 0, \forall k
$$

- el ruido tiene una cierta varianza $\mathbb{E}[\nu_n^2] = \sigma_\nu^2, \forall n$

Notemos que en este caso $R_{uu} = R_{dd} + R_{\nu\nu}$ y $R_{ud} = R_{dd}$, luego

la señal recuperada es $\hat d_n = h^{*} u_n$ y el filtro es

$$
\vec h^{*} = \frac{R_{dd}}{R_{dd} + R_{\nu\nu}}
$$

y su respuesta en frecuencia

$$
H(f) = \frac{S_{dd}(f)}{S_{dd}(f) + S_{\nu\nu}(f)}
$$

es decir que
- en frecuencias donde la $S_{dd}(f) > S_{\nu\nu}(f)$, entonces $H(f) = 1$
- en frecuencias donde la $S_{dd}(f) < S_{\nu\nu}(f)$, entonces $H(f) = 0$

```python

```


## Data-driven Design and Analyses of Structures and Materials (3dasm)

## Lecture 5

### Miguel A. Bessa | M.A.Bessa@tudelft.nl  | Associate Professor

**What:** A lecture of the "3dasm" course

**Where:** This notebook comes from this [repository](https://github.com/bessagroup/3dasm_course)

**Reference for entire course:** Murphy, Kevin P. *Probabilistic machine learning: an introduction*. MIT press, 2022. Available online [here](https://probml.github.io/pml-book/book1.html)

**How:** We try to follow Murphy's book closely, but the sequence of Chapters and Sections is different. The intention is to use notebooks as an introduction to the topic and Murphy's book as a resource.
* If working offline: Go through this notebook and read the book.
* If attending class in person: listen to me (!) but also go through the notebook in your laptop at the same time. Read the book.
* If attending lectures remotely: listen to me (!) via Zoom and (ideally) use two screens where you have the notebook open in 1 screen and you see the lectures on the other. Read the book.

**Optional reference (the "bible" by the "bishop"... pun intended 😆) :** Bishop, Christopher M. *Pattern recognition and machine learning*. Springer Verlag, 2006.

**References/resources to create this notebook:**
* [Car figure](https://korkortonline.se/en/theory/reaction-braking-stopping/)

Apologies in advance if I missed some reference used in this notebook. Please contact me if that is the case, and I will gladly include it here.

## **OPTION 1**. Run this notebook **locally in your computer**:
1. Confirm that you have the 3dasm conda environment (see Lecture 1).

2. Go to the 3dasm_course folder in your computer and pull the last updates of the [repository](https://github.com/bessagroup/3dasm_course):
```
git pull
```
3. Open command window and load jupyter notebook (it will open in your internet browser):
```
conda activate 3dasm
jupyter notebook
```
4. Open notebook of this Lecture.

## **OPTION 2**. Use **Google's Colab** (no installation required, but times out if idle):

1. go to https://colab.research.google.com
2. login
3. File > Open notebook
4. click on Github (no need to login or authorize anything)
5. paste the git link: https://github.com/bessagroup/3dasm_course
6. click search and then click on the notebook for this Lecture.

```python
# Basic plotting tools needed in Python.

import matplotlib.pyplot as plt # import plotting tools to create figures
import numpy as np # import numpy to handle a lot of things!
from IPython.display import display, Math # to print with Latex math

%config InlineBackend.figure_format = "retina" # render higher resolution images in the notebook
plt.style.use("seaborn") # style for plotting that comes from seaborn
plt.rcParams["figure.figsize"] = (8,4) # rescale figure size appropriately for slides
```

## Outline for today

* Bayesian inference for one hidden rv
    - Prior
    - Likelihood
    - Marginal likelihood
    - Posterior
    - Gaussian pdf's product

**Reading material**: This notebook + Chapter 3

### Recall the "slightly more complicated" car stopping distance problem (with two rv's)

We defined the governing model with two rv's $z_1$ and $z_2$ as:

$\require{color}{\color{red}y} = {\color{blue}z_1}\cdot x + {\color{magenta}z_2}\cdot x^2$

- ${\color{red}y}$ is the **output**: the car stopping distance (in meters)
- ${\color{blue}z_1}$ is an rv representing the driver's reaction time (in seconds)
- ${\color{magenta}z_2}$ is another rv that depends on the coefficient of friction, the inclination of the road, the weather, etc. (in m$^{-1}$s$^{-2}$).
- $x$ is the **input**: constant car velocity (in m/s).

where we knew the "true" distributions of the rv's: $z_1 \sim \mathcal{N}(\mu_{z_1}=1.5,\sigma_{z_1}^2=0.5^2)$, and $z_2 \sim \mathcal{N}(\mu_{z_2}=0.1,\sigma_{z_2}^2=0.01^2)$.

```python
# This cell is hidden during presentation. It's just to define a function to plot the governing model of
# the car stopping distance problem. Defining a function that creates a plot allows to repeatedly run
# this function on cells used in this notebook.
def car_fig_2rvs(ax):
    x = np.linspace(3, 83, 1000)
    mu_z1 = 1.5; sigma_z1 = 0.5;  # parameters of the "true" p(z_1)
    mu_z2 = 0.1; sigma_z2 = 0.01; # parameters of the "true" p(z_2)
    mu_y = mu_z1*x + mu_z2*x**2 # From Homework of Lecture 4
    sigma_y = np.sqrt( (x*sigma_z1)**2 + (x**2*sigma_z2)**2 ) # From Homework of Lecture 4
    ax.set_xlabel("x (m/s)", fontsize=20) # create x-axis label with font size 20
    ax.set_ylabel("y (m)", fontsize=20) # create y-axis label with font size 20
    ax.set_title("Car stopping distance problem with two rv's", fontsize=20); # create title with font size 20
    ax.plot(x, mu_y, 'k:', label="Governing model $\mu_y$")
    ax.fill_between(x, mu_y - 1.9600 * sigma_y,
                    mu_y + 1.9600 * sigma_y,
                    color='k', alpha=0.2,
                    label='95% confidence interval ($\mu_y \pm 1.96\sigma_y$)') # plot 95% credence interval
    ax.legend(fontsize=15)
```

```python
# This cell is also hidden during presentation.
from scipy.stats import norm # import the normal dist, as we learned before!
def samples_y_with_2rvs(N_samples,x): # observations/measurements/samples for car stop. dist. prob. with 2 rv's
    mu_z1 = 1.5; sigma_z1 = 0.5;
    mu_z2 = 0.1; sigma_z2 = 0.01;
    samples_z1 = norm.rvs(mu_z1, sigma_z1, size=N_samples) # randomly draw samples from the normal dist.
    samples_z2 = norm.rvs(mu_z2, sigma_z2, size=N_samples) # randomly draw samples from the normal dist.
    samples_y = samples_z1*x + samples_z2*x**2 # compute the stopping distance for samples of z_1 and z_2
    return samples_y # return samples of y
```

```python
# vvvvvvvvvvv this is just a trick so that we can run this cell multiple times vvvvvvvvvvv
fig_car_new, ax_car_new = plt.subplots(1,2); plt.close() # create figure and close it
if fig_car_new.get_axes():
    del ax_car_new; del fig_car_new # delete figure and axes if they exist
    fig_car_new, ax_car_new = plt.subplots(1,2) # create them again
# ^^^^^^^^^^^ end of the trick ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
N_samples = 3 # CHANGE THIS NUMBER AND RE-RUN THE CELL
x = 75; empirical_y = samples_y_with_2rvs(N_samples, x); # Empirical measurements of N_samples at x=75
empirical_mu_y = np.mean(empirical_y); empirical_sigma_y = np.std(empirical_y); # empirical mean and std
car_fig_2rvs(ax_car_new[0]) # a function I created to include the background plot of the governing model
for i in range(2): # create two plots (one is zooming in on the error bar)
    ax_car_new[i].errorbar(x , empirical_mu_y,yerr=1.96*empirical_sigma_y, fmt='m*', markersize=15);
    ax_car_new[i].scatter(x*np.ones_like(empirical_y),empirical_y, s=40,
                          facecolors='none', edgecolors='k', linewidths=2.0)
print("Empirical mean[y] is",empirical_mu_y, "(real mean[y]=675)")
print("Empirical std[y] is",empirical_sigma_y,"(real std[y]=67.6)")
fig_car_new.set_size_inches(25, 5) # scale figure to be wider (since there are 2 subplots)
```

### Note: comparison of car stopping distance problem of Lecture 3  & Lecture 4

* The car stopping distance problem in Lecture 3 only had one rv.
* While this car stopping distance problem (introduced in Lecture 4) **has 2 rv's**.
    - Observing the governing model when knowing the "true" distributions of $z_1$ and $z_2$ we see that:
        * The expected value (mean) of $y$ is the same for both problems because $\mu_{z_2}=0.1$.
        * The variance of $y$ is higher for this problem because of the additional randomness introduced by $z_2$.
            - For example, for $x=75$ m/s the $\text{std}[y] \approx 67.6$ m while it was 37.5 when only $z_1$ was an rv.

### Car stopping distance problem with 2 rv's but only 1 rv being unknown

Today we will finally do some predictions!

Recall the Homework of Lecture 4, and consider the car stopping distance problem for constant velocity $x=75$ m/s and for which **it is known** that $z_2 \sim \mathcal{N}(z_2|\mu_{z_2}=0.1,\sigma_{z_2}^2=0.01^2)$.

The only information that we do not know is the driver's reaction time $z$ (here we call it $z$, instead of $z_1$ as in Lecture 4, because this is the only hidden variable so we can **simplify the notation**).

* Can we predict $p(y)$ without knowing $p(z)$?

Yes!! If we use Bayes' rule!

### Recall the Homework of Lecture 4

From last lecture's Homework, you demonstrated that the conditional pdf of the stopping distance given the reaction time $z$ (for convenience we write here $z$ instead of $z_1$) is

$$
p(y|z) = \mathcal{N}\left(y | \mu_{y|z}=w z+b, \sigma_{y|z}^2=s^2\right)
$$

where $w$, $b$ and $s$ are all constants that you determined to be:

$w=x=75$

$b=x^2\mu_{z_2}=75^2\cdot0.1=562.5$

$s^2=(x^2 \sigma_{z_2})^2=(75^2\cdot0.01)^2=56.25^2$

because we are considering that the car is going at constant velocity $x=75$ m/s and that we know $z_2= \mathcal{N}(z_2|\mu_{z_2}=0.1,\sigma_{z_2}^2=0.01^2)$.

### Solution to Homework of Lecture 4

What we know:

$\require{color}{\color{red}y} = {\color{blue}z_1}\cdot 75 + {\color{magenta}z_2}\cdot 75^2 = 75 {\color{blue}z_1} + 5625 {\color{magenta}z_2}$

where $z_1 \sim \mathcal{N}(\mu_{z_1}=1.5,\sigma_{z_1}^2=0.5^2)$, and $z_2 \sim \mathcal{N}(\mu_{z_2}=0.1,\sigma_{z_2}^2=0.01^2)$.

1. To calculate the conditional pdf $p(y|z_1)$, i.e. the observation distribution, we note that when given $z_1$ we just have $y$ as a function of $z_2$:

$$
y \equiv f(z_2) = x z_1 + x^2 z_2 \Rightarrow z_2 = \frac{y}{x^2}-\frac{z_1}{x} \equiv g(y)
$$

From the change of variables formula (Lecture 3),

$$\begin{align}
p_{y|z_1}(y) &= p_{z_2}\left( g(y) \right) \left| \frac{d}{dy}g(y)\right| \\
&=  \mathcal{N}\left( \frac{y}{x^2}-\frac{z_1}{x}\left| \mu_{z_2}, \sigma_{z_2}^2\right.\right) \left|\frac{1}{x^2}\right| \\
&= \frac{1}{\sqrt{2\pi \sigma_{z_2}^2}} \exp\left[ -\frac{1}{2\sigma_{z_2}^2}\left( \frac{y}{x^2}-\frac{z_1}{x}-\mu_{z_2} \right)^2 \right] \left|\frac{1}{x^2}\right|\\
&= \frac{1}{\sqrt{2\pi \left(x^2\sigma_{z_2}\right)^2}} \exp\left[ -\frac{1}{2\left(x^2\sigma_{z_2}\right)^2}\left( y-x z_1-x^2\mu_{z_2} \right)^2 \right]
\end{align}
$$

So, the conditional pdf $p(y|z_1)$ is also a Gaussian:

$$
p(y|z_1) = \mathcal{N}\left( y| \mu_{y|z_1}=x^2\mu_{z_2}+x z_1, \sigma_{y|z_1}=\left( x^2 \sigma_{z_2}\right)^2 \right)
$$

(Alternative way to answer Question 1 without using the change of variables formula)

There is a different way to derive the same result without even using the change of variables formula. We can obtain the same result as above by calculating the joint distribution $p(y,z_1,z_2)$:
$$\begin{align}
p(y,z_1,z_2) &= p(y|z_1, z_2) p(z_1, z_2) \\
&= p(y|z_1,z_2) p(z_1) p(z_2) \\
&= \delta(y-x z_1 - x^2 z_2) p(z_1) p(z_2) \\
&= \frac{1}{|x^2|} \delta(\frac{y-x z_1}{x^2} - z_2) p(z_1) p(z_2) \\
&= \frac{1}{|x^2|} p_{z_2}(\frac{y-x z_1}{x^2}) p(z_1) \\
&= \mathcal{N}\left( y| x^2\mu_{z_2}+x z_1, \left( x^2 \sigma_{z_2}\right)^2 \right) p(z_1)
\end{align}
$$

2. The joint distribution is simply $p(y, z_1)$:

$$
p(y, z_1) = p(y|z_1)p(z_1) = \mathcal{N}\left( y| \mu_{y|z_1}=x^2\mu_{z_2}+x z_1, \sigma_{y|z_1}=\left( x^2 \sigma_{z_2}\right)^2 \right) \mathcal{N}\left( z_1| \mu_{z_1}=1.5, \sigma_{z_1}=0.5^2 \right)
$$

which we will learn how to calculate in this lecture (spoiler alert: it's another gaussian 😆)

3. The covariance matrix is calculated as:

$
\mathbb{E}[z_1] = \mu_{z_1} \, , \quad \mathbb{V}[z_1] = \sigma_{z_1}^2
$

$
\mathbb{E}[y] = \mathbb{E}[z_1 x + x^2 z_2]=\mathbb{E}[z_1]x + x^2 \mathbb{E}[z_2] = x\mu_{z_1}+x^2\mu_{z_2}
$

$\begin{align}
\mathbb{E}[y^2] &= \mathbb{E}\left[ (z_1 x + x^2 z_2)(z_1 x + x^2 z_2) \right]\\
&= \mathbb{E}\left[ z_1^2 x^2+2x^3z_1 z_2 + x^4 z_2^2 \right] \\
&= x^2 \mathbb{E}[z_1^2] + 2x^3 \mathbb{E}[z_1 z_2] + x^4 \mathbb{E}[z_2^2] \\
&= x^2\left( \sigma_{z_1}^2 + \mu_{z_1}^2\right) + 2x^3 \mu_{z_1}\mu_{z_2} + x^4\left( \sigma_{z_2}^2 + \mu_{z_2}^2\right)
\end{align}
$

$\begin{align}
\mathbb{V}[y] &= \mathbb{E}[y^2]-\mathbb{E}[y]^2\\
&= x^2\left( \sigma_{z_1}^2 + \mu_{z_1}^2\right) + 2x^3 \mu_{z_1}\mu_{z_2} + x^4\left( \sigma_{z_2}^2 + \mu_{z_2}^2\right) - \left( x\mu_{z_1}+x^2\mu_{z_2}\right)^2 \\
&= \left( x\sigma_{z_1}\right)^2+\left(x^2\sigma_{z_2}\right)^2 \\
\end{align}
$

$\begin{align}
\text{Cov}[y, z_1] &= \mathbb{E}[y z_1] - \mathbb{E}[y] \mathbb{E}[z_1] \\
&= \mathbb{E}[z_1^2 x + x^2 z_2 z_1] - \left(x\mu_{z_1}+x^2\mu_{z_2}\right) \mu_{z_1} \\
&= x\mathbb{E}[z_1^2]+x^2\mathbb{E}[z_1 z_2]  - \left(x\mu_{z_1}+x^2\mu_{z_2}\right) \mu_{z_1} \\
&= x\left( \sigma_{z_1}^2 + \mu_{z_1}^2\right)+x^2\mu_{z_1}\mu_{z_2}  - x\mu_{z_1}-x^2x\mu_{z_1}\mu_{z_2} \\
&= x\sigma_{z_1}^2
\end{align}
$

From where we can finally calculate the Covariance matrix:

$$
\begin{align}
\boldsymbol{\Sigma} &= \text{Cov}\begin{bmatrix}y\\ z_1\end{bmatrix}
 = \begin{bmatrix}
\mathbb{V}[y] & \text{Cov}[y,z_1] \\
\text{Cov}[z_1,y] & \mathbb{V}[z_1]
\end{bmatrix}\\
&= \begin{bmatrix}
\left( x\sigma_{z_1}\right)^2+\left(x^2\sigma_{z_2}\right)^2 & x\sigma_{z_1}^2 \\
x\sigma_{z_1}^2 & \sigma_{z_1}^2
\end{bmatrix}
\end{align}
$$

### Understanding the Bayes' rule

$\require{color}$
$$
{\color{green}p(z|y)} = \frac{ {\color{blue}p(y|z)}{\color{red}p(z)} } {p(y)}
$$

* ${\color{red}p(z)}$ is the **prior distribution**
* ${\color{blue}p(y|z)}$ is the **observation distribution** (conditional pdf)
* $p(y)$ is the **marginal distribution**
* ${\color{green}p(z|y)}$ is the **posterior distribution**

### A note about the term "distribution"

The term distribution can mean two things:
1. For **continuous** rv's, the term *distribution* means *probability density function* (pdf).

2. For **discrete** rv's the term *distribution* means *probability mass function* (pmf), as we will see later in the course.

We won't talk about categorical distributions or pmf's for a while. So, for now, when you see the term *distribution* it is the same as saying pdf.

### Understanding the Bayes' rule

Let's start by understanding the usefulness of Bayes' rule by calculating the posterior $p(z|y)$ for the car stopping distance problem (Homework of Lecture 4).

As we mentioned, for our problem we know the **observation distribution**:

$p(y|z) = \mathcal{N}\left(y | \mu_{y|z}=w z+b, \sigma_{y|z}^2\right)$

where $\sigma_{y|z} = \text{const}$, as well as $w$ and $b$.

but we **don't know** the prior $p(z)$.

### Prior: our beliefs about the problem

If we have absolutely no clue about what the distribution of the hidden rv $z$ is, then we can use a **Uniform distribution** (a.k.a. uninformative prior).

This distribution assigns equal probability to any value of $z$ within an interval $z \in (z_{min}, z_{max})$.

$$
p(z) = \frac{1}{C_z}
$$

where $C_z = z_{max}-z_{min}$ is the **normalization constant** of the Uniform pdf, i.e. the value that guarantees that $p(z)$ integrates to one.

For the time being, we will not assume any particular values for $z_{max}$ and $z_{min}$. So, we will consider the completely uninformative prior: $z_{max}\rightarrow \infty$ and $z_{min}\rightarrow -\infty$. If we had some information, we could consider some values for these bounds (e.g. $z_{min} = 0$ seconds would be the limit of the fastest reaction time that is humanly possible, and $z_{max} = 3$ seconds would be the slowest reaction time of a human being).

### Summary of our Model

1. The **observation distribution**:

$$\begin{align}
p(y|z) &= \mathcal{N}\left(y | \mu_{y|z}=w z+b, \sigma_{y|z}^2\right) \\
&= \frac{1}{C_{y|z}} \exp\left[ -\frac{1}{2\sigma_{y|z}^2}(y-\mu_{y|z})^2\right]
\end{align}
$$

where $C_{y|z} = \sqrt{2\pi \sigma_{y|z}^2}$ is the **normalization constant** of the Gaussian pdf, and where $\mu_{y|z}=w z+b$, with $w$, $b$ and $\sigma_{y|z}^2$ being constants, as previously mentioned.

2. and the **prior distribution**: $p(z) = \frac{1}{C_z}$

where $C_z = z_{max}-z_{min}$ is the **normalization constant** of the Uniform pdf, i.e. the value that guarantees that $p(z)$ integrates to one.

### Posterior from Bayes' rule

Since we have defined the **observation distribution** and the **prior distribution**, we can now compute the posterior distribution from Bayes' rule.

But this requires a bit of algebra... Let's do it!

First, in order to apply Bayes' rule $p(z|y) = \frac{ p(y|z)p(z)}{p(y)}$ we need to calculate $p(y)$.

$p(y)$ is obtained by marginalizing the joint distribution wrt $z$:

$
p(y) = \int p(y|z)p(z) dz
$

which implies an integration over $z$. So, let's rewrite $p(y|z)$ so that the integration becomes easier.

$$\begin{align}
p(y|z) &= \mathcal{N}\left(y | \mu_{y|z}=w z+b, \sigma_{y|z}^2\right) \\
&= \frac{1}{C_{y|z}} \exp\left[ -\frac{1}{2\sigma_{y|z}^2}(y-(w z+b))^2\right] \\
&= \frac{1}{C_{y|z}} \exp\left\{ -\frac{1}{2\left(\frac{\sigma_{y|z}}{w}\right)^2}\left[z-\left(\frac{y-b}{w}\right)\right]^2\right\} \\
&= \frac{1}{|w|}\frac{1}{\sqrt{2\pi \left(\frac{\sigma_{y|z}}{w}\right)^2}} \exp\left\{ -\frac{1}{2\left(\frac{\sigma_{y|z}}{w}\right)^2}\left[z-\left(\frac{y-b}{w}\right)\right]^2\right\}
\end{align}
$$

Note: This Gaussian pdf $\mathcal{N}\left(z | \frac{y-b}{w}, \left(\frac{\sigma_{y|z}}{w}\right)^2\right)$ is unnormalized when written wrt $z$ (due to $\frac{1}{|w|}$).

We can now calculate the marginal distribution $p(y)$:

$$
\begin{align}
p(y) &= \int p(y|z)p(z) dz \\
&= \int \frac{1}{|w|}\frac{1}{\sqrt{2\pi \left(\frac{\sigma_{y|z}}{w}\right)^2}} \exp\left\{ -\frac{1}{2\left(\frac{\sigma_{y|z}}{w}\right)^2}\left[z-\left(\frac{y-b}{w}\right)\right]^2\right\} \frac{1}{C_z} dz
\end{align}
$$

We can rewrite this expression as,

$$\require{color}
\begin{align}
p(y) &= \frac{1}{|w|\cdot C_z} {\color{blue}\int \frac{1}{\sqrt{2\pi \left(\frac{\sigma_{y|z}}{w}\right)^2}} \exp\left\{ -\frac{1}{2\left(\frac{\sigma_{y|z}}{w}\right)^2}\left[z-\left(\frac{y-b}{w}\right)\right]^2\right\} dz} \\
\end{align}
$$

What is the result for the blue term?

From where we conclude that the marginal distribution is:

$$\require{color}
p(y) = \frac{1}{|w| C_z }
$$

So, now we can determine the posterior:

$$\require{color}
\begin{align}
p(z|y) &= \frac{ p(y|z)p(z)}{p(y)} \\
&= |w| C_z \cdot \frac{1}{|w|}\frac{1}{\sqrt{2\pi \left(\frac{\sigma_{y|z}}{w}\right)^2}} \exp\left\{ -\frac{1}{2\left(\frac{\sigma_{y|z}}{w}\right)^2}\left[z-\left(\frac{y-b}{w}\right)\right]^2\right\} \cdot \frac{1}{C_z}\\
&= \frac{1}{\sqrt{2\pi \left(\frac{\sigma_{y|z}}{w}\right)^2}}\exp\left\{ -\frac{1}{2\left(\frac{\sigma_{y|z}}{w}\right)^2}\left[z-\left(\frac{y-b}{w}\right)\right]^2\right\}
\end{align}
$$

which is a **normalized** Gaussian pdf in $z$: $\mathcal{N}\left(z | \frac{y-b}{w}, \left(\frac{\sigma_{y|z}}{w}\right)^2\right)$

* **This is what the Bayes' rule does!** Computes the posterior $p(z|y)$ from $p(y|z)$ and $p(z)$.

## Why should we care about the Bayes' rule?

There are a few reasons:

1. As we will see, models are usually (always?) wrong.

2. But our beliefs may be a bit closer to reality! Bayes' rule enables us to get better models if our beliefs are reasonable!

3. We don't observe distributions. We observe **DATA**. Bayes' rule is a very powerful way to predict the distribution of our quantity of interest (here: $y$) from data!

## Bayes' rule applied to observed data

Previously, we already introduced Bayes' rule when applied to observed data $\mathcal{D}_y$.

$\require{color}$
$$
{\color{green}p(z|y=\mathcal{D}_y)} = \frac{ {\color{blue}p(y=\mathcal{D}_y|z)}{\color{red}p(z)} } {p(y=\mathcal{D}_y)} = \frac{ {\color{magenta}p(y=\mathcal{D}_y, z)} } {p(y=\mathcal{D}_y)}
$$

* ${\color{red}p(z)}$ is the **prior** distribution
* ${\color{blue}p(y=\mathcal{D}_y|z)}$ is the **likelihood** function
* ${\color{magenta}p(y=\mathcal{D}_y, z)}$ is the **joint likelihood** (product of likelihood function with prior distribution)
* $p(y=\mathcal{D}_y)$ is the **marginal likelihood**
* ${\color{green}p(z|y=\mathcal{D}_y)}$ is the **posterior**

We can write Bayes' rule as posterior $\propto$ likelihood $\times$ prior , where we are ignoring the denominator $p(y=\mathcal{D}_y)$ because it is just a **constant** independent of the hidden variable $z$.

## Bayes' rule applied to observed data

But remember that Bayes' rule is just a way to calculate the posterior:

$$
p(z|y=\mathcal{D}_y) = \frac{ p(y=\mathcal{D}_y|z)p(z) } {p(y=\mathcal{D}_y)}
$$

Usually, what we really want is to be able to predict the distribution of the quantity of interest (here: $y$) after observing some data $\mathcal{D}_y$:

$$\require{color}
{\color{orange}p(y|y=\mathcal{D}_y)} = \int p(y|z) p(z|y=\mathcal{D}_y) dz
$$

which is often written in simpler notation: $p(y|\mathcal{D}_y) = \int p(y|z) p(z|\mathcal{D}_y) dz$

### Bayesian inference for car stopping distance problem

Now we will solve the first Bayesian ML problem from some given data $y=\mathcal{D}_y$:

| $y_i$ (m) |
| ---- |
| 601.5 |
| 705.9 |
| 693.8 |
| ...   |
| 711.3 |

where the data $\mathcal{D}_y$ could be a Pandas dataframe with $N$ data points ($N$ rows).

* **Very Important Question (VIQ)**: Can we calculate the **likelihood** function from this data?

### Likelihood for car stopping distance problem

Of course! As we saw a few cells ago, the **likelihood** is obtained by evaluating the **observation distribution** at the data $\mathcal{D}_y$.

Noting that each observation in $\mathcal{D}_y$ is independent of each other, then:

$$
p(y=\mathcal{D}_y | z) = \prod_{i=1}^{N} p(y=y_i|z) = p(y=y_1|z)p(y=y_2|z) \cdots p(y=y_N|z)
$$

which gives the **probability density** of observing that data if using our observation distribution (part of our model!).

#### Calculating the likelihood

Let's calculate it:

$$
\begin{align}
p(y=\mathcal{D}_y | z) &= \prod_{i=1}^{N} p(y=y_i|z) \\
&= \prod_{i=1}^{N} \frac{1}{C_{y|z}} \exp\left\{ -\frac{1}{2\left(\frac{\sigma_{y|z}}{w}\right)^2}\left[z-\left(\frac{y_i-b}{w}\right)\right]^2\right\}
\end{align}
$$

This seems a bit daunting... I know. Do not dispair yet!

##### Product of Gaussian pdf's of the same rv $z$

It can be shown that the product of $N$ univariate Gaussian pdf's of the same rv $z$ is:

$$
\prod_{i=1}^{N} \mathcal{N}(z|\mu_i, \sigma_i^2) = C \cdot \mathcal{N}(z|\mu, \sigma^2)
$$

with mean: $\mu = \sigma^2 \left( \sum_{i=1}^{N} \frac{\mu_i}{\sigma_i^2}\right)$

variance: $\sigma^2= \frac{1}{\sum_{i=1}^{N} \frac{1}{\sigma_i^2}}$

and normalization constant: $C = \frac{1}{\left(2\pi\right)^{(N-1)/2}}\sqrt{\frac{\sigma^2}{\prod_{i=1}^N \sigma_i^2}} \exp\left[-\frac{1}{2}\left(\sum_{i=1}^{N} \frac{\mu_i^2}{\sigma_i^2} - \frac{\mu^2}{\sigma^2}\right)\right]$

Curiosity: the normalization constant $C$ is itself a Gaussian! You can see it more clearly if you consider $N=2$

Note that the normalization constant shown in the previous cell can also be written as:

$$
C = \frac{1}{\left(2\pi\right)^{(N-1)/2}}\sqrt{\frac{\sigma^2}{\prod_{i=1}^N \sigma_i^2}} \exp\left[-\frac{1}{2}\left(\sum_{i=1}^{N-1}\sum_{j=i+1}^{N} \frac{(\mu_i-\mu_j)^2}{\sigma_i^2 \sigma_j^2}\sigma^2\right)\right]
$$

# HOMEWORK

Show that the product of two Gaussian pdf's for the same rv $z$ is:

$\mathcal{N}(z|\mu_1, \sigma_1^2)\cdot \mathcal{N}(z|\mu_2, \sigma_2^2)= C \cdot \mathcal{N}(z | \mu, \sigma^2)$

$$
\begin{align}
\sigma^2&=\frac{1}{\frac{1}{\sigma_1^2}+\frac{1}{\sigma_2^2}}\\
\mu&=\sigma^2\left(\frac{\mu_1}{\sigma_1^2} +  \frac{\mu_2}{\sigma_2^2}\right)\\
C &= \frac{1}{\sqrt{2\pi(\sigma_1^2+\sigma_2^2)}} \exp\left[-\frac{1}{2(\sigma_1^2+\sigma_2^2)}(\mu_1-\mu_2)^2\right]
\end{align}
$$

#### Side note

It's interesting to note that the product of MVN's for the same rv's $\mathbf{z}$ is also a Gaussian!

To keep things simple, here's the result for the product of 2 Gaussian pdf's:

$\mathcal{N}(\mathbf{z}|\boldsymbol{\mu}_1, \boldsymbol{\Sigma}_1)\cdot \mathcal{N}(\mathbf{z}|\boldsymbol{\mu}_2, \boldsymbol{\Sigma}_2)= C \cdot \mathcal{N}(\mathbf{z} | \boldsymbol{\mu}, \boldsymbol{\Sigma})$

where

$\boldsymbol{\mu} = \boldsymbol{\Sigma}\left(\boldsymbol{\Sigma}_1^{-1}\boldsymbol{\mu}_1 + \boldsymbol{\Sigma}_2^{-1}\boldsymbol{\mu}_2 \right)$

$\boldsymbol{\Sigma} = \left( \boldsymbol{\Sigma}_1^{-1}+\boldsymbol{\Sigma}_2^{-1}\right)^{-1}$

$
\begin{align}
C &= \mathcal{N}_{\boldsymbol{\mu}_1}\left(\boldsymbol{\mu}_2, \left( \boldsymbol{\Sigma}_1+\boldsymbol{\Sigma}_2\right)\right)\\
&= \frac{1}{\sqrt{\det[2\pi \left( \boldsymbol{\Sigma}_1+\boldsymbol{\Sigma}_2 \right)]}} \exp\left[-\frac{1}{2} \left( \boldsymbol{\mu}_1-\boldsymbol{\mu}_2\right)^T\cdot\left( \boldsymbol{\Sigma}_1+\boldsymbol{\Sigma}_2 \right)^{-1}\left( \boldsymbol{\mu}_1-\boldsymbol{\mu}_2 \right) \right]\\
\end{align}
$

#### Back to calculating the likelihood

$$
\begin{align}
p(y=\mathcal{D}_y | z) &= \prod_{i=1}^{N} p(y=y_i|z) \\
&= \prod_{i=1}^{N} \frac{1}{|w|} \frac{1}{\sqrt{2\pi \left(\frac{\sigma_{y|z}}{w}\right)^2}} \exp\left\{ -\frac{1}{2\left(\frac{\sigma_{y|z}}{w}\right)^2}\left[z-\left(\frac{y_i-b}{w}\right)\right]^2\right\} \\
&= \frac{1}{|w|^N} \prod_{i=1}^{N} \frac{1}{\sqrt{2\pi \left(\frac{\sigma_{y|z}}{w}\right)^2}} \exp\left\{ -\frac{1}{2\left(\frac{\sigma_{y|z}}{w}\right)^2}\left[z-\left(\frac{y_i-b}{w}\right)\right]^2\right\}
\end{align}
$$

So, using the result of a product of $N$ Gaussian pdf's to calculate the likelihood, and noting that $\sigma_i = \frac{\sigma_{y|z}}{w}$ and $\mu_i = \frac{y_i - b}{w}$ we get:

$$
p(y=\mathcal{D}_y | z) = \frac{1}{|w|^N} \cdot C \cdot \frac{1}{\sqrt{2\pi \sigma^2}} \exp\left[ -\frac{1}{2\sigma^2}(z-\mu)^2\right]
$$

where

$\mu = \frac{\sigma^2}{\sigma_i^2} \sum_{i=1}^N \mu_i = \frac{w^2\sigma^2}{\sigma_{y|z}^2} \sum_{i=1}^N \mu_i$

$\sigma^2 = \frac{1}{ \sum_{i=1}^N \frac{1}{\sigma_i^2} } = \frac{1}{ \sum_{i=1}^N \frac{w^2}{\sigma_{y|z}^2} } = \frac{\sigma_{y|z}^2}{w^2 N}$

$
C = \frac{1}{\left(2\pi\right)^{(N-1)/2}} \sqrt{\frac{\sigma^2}{\left( \frac{\sigma_{y|z}^2}{w^2}\right)^N}} \exp\left[-\frac{1}{2}\left(\frac{w^2}{\sigma_{y|z}^2}\sum_{i=1}^N \mu_i - \frac{\mu^2}{\sigma^2}\right) \right] = \frac{1}{\left(2\pi\right)^{(N-1)/2}} \sqrt{\frac{\sigma^2}{\left( \frac{\sigma_{y|z}^2}{w^2}\right)^N}}
$

#### Calculating the marginal likelihood

$$\begin{align}
p(y=\mathcal{D}_y) &= \int p(y=\mathcal{D}_y | z) p(z) dz \\
&= \int \frac{1}{|w|^N} C \cdot \mathcal{N}(z|\mu, \sigma^2)\cdot \frac{1}{C_z} dz\\
&= \frac{C}{|w|^N C_z} \int \mathcal{N}(z|\mu, \sigma^2)dz = \frac{C}{|w|^N C_z} \\
\end{align}
$$

We can now calculate the posterior:

$$\begin{align}
p(z|y=\mathcal{D}_y) &= \frac{ p(y=\mathcal{D}_y|z)p(z) } {p(y=\mathcal{D}_y)} \\
&= \frac{1}{p(y=\mathcal{D}_y)} \cdot \frac{1}{|w|^N} C \cdot \mathcal{N}(z|\mu,\sigma^2) \cdot \frac{1}{C_z} \\
&= \mathcal{N}(z|\mu, \sigma^2)
\end{align}
$$

#### Calculating the Posterior Predictive Distribution (PPD)

Having found the posterior, we can determine the PPD:

$$
p(y|\mathcal{D}_y) = \int p(y| z) p(z|\mathcal{D}_y) dz
$$

To calculate this, we will have to use the identity for a product of two Gaussians.

$$
\begin{align}
p(y|\mathcal{D}_y) &= \int \frac{1}{|w|} \mathcal{N}\left(z|\frac{y-b}{w}, \left(\frac{\sigma_{y|z}}{w}\right)^2\right) \mathcal{N}(z|\mu, \sigma^2) dz \\
&= \int \frac{1}{|w|} C^* \mathcal{N}\left(z|\mu^*, \left(\sigma^*\right)^2\right) dz \\
\end{align}
$$

#### Calculating the Posterior Predictive Distribution (PPD)

We can find these parameters from the identity for a product of two Gaussians.

$$
p(y|\mathcal{D}_y) = \int \frac{1}{|w|} C^* \mathcal{N}\left(z|\mu^*, \left(\sigma^*\right)^2\right) dz
$$

where

$\mu^* = \left(\sigma^* \right)^2 \left( \frac{\mu}{\sigma^2} + \frac{(y-b)/w}{\left(\frac{\sigma_{y|z}}{w}\right)^2} \right) = \left(\sigma^* \right)^2 \left( \frac{\mu}{\sigma^2} + \frac{(y-b)\cdot w}{\sigma_{y|z}^2} \right)$

$\left( \sigma^* \right)^2 = \frac{1}{\frac{1}{\sigma^2}+\frac{1}{\left( \frac{\sigma_{y|z}}{w}\right)^2}}= \frac{1}{\frac{1}{\sigma^2}+\frac{w^2}{\sigma_{y|z}^2}}$

$C^* = \frac{1}{\sqrt{2\pi \left( \sigma^2 + \frac{\sigma_{y|z}^2}{w^2} \right)}}\exp\left[ - \frac{\left(\mu - \frac{y-b}{w}\right)^2}{2\left( \sigma^2+\frac{\sigma_{y|z}^2}{w^2}\right)}\right]$

## Next class

In the next class we will finish this example, by solving this integral to determine the PPD $p(y|\mathcal{D}_y)$.

### See you next class

Have fun!


```python
%run ../../common/import_all.py

from common.setup_notebook import set_css_style, setup_matplotlib, config_ipython
config_ipython()
setup_matplotlib()
set_css_style()
```

# The $\chi^2$ test

## What is

The $\chi^2$ test is a statistical hypothesis test in which the distribution of the test statistic calculated on the data is a [$\chi^2$ distribution](../distributions-measures/famous-distributions.ipynb#Chi-squared,-$\chi^2$) under null hypothesis. The assumption is that  data is normally distributed and independent so the $\chi^2$ test can also be used to reject the  hypothesis that data are independent.

It is used with categorical data to see if the number of individuals in each category is consistent with the expected values. In practice, the test is used to determine if there is a significant difference between the expected frequencies and the observed frequencies of the outcomes of an experiment in one or more categories, that is, if the observed differences are due to chance. The idea is: is the number of individuals falling into each category significantly different from the number you would expect under the null hypothesis? Is this difference between expected and observed  data due to sampling or is it real?

The $\chi^2$ is defined as

$$
\chi^2 = \sum_i \frac{(o_i - h_i)^2}{h_i} \ ,
$$

where $o_i$ is the observed value and $h_i$ the null hypothesis value.

The computed $\chi^2$ has to be compared to [table values](http://sites.stat.psu.edu/~mga/401/tables/Chi-square-table.pdf) for the $\chi^2$ distribution at the chosen level of significance and for given degrees of  freedom one has in order to decide if the null hypothesis can be rejected or not, using the [$p$-value](../concepts/p-value-confidence-level.ipynb).

## An example

This simple example has been adapted from [[here]](#example-dice).

Let's say that we have a (6-faces) dice and we want to know if it is fair, that is, if each of the faces is equiprobable or if there is any bias towards a face. We throw the dice 60 times: in the case of a fair dice we would have each face appearing 10 times (60/6 where 6 is the number of possible results). This will be the null hypothesis.

We build a table containing the actual counts we get for each face, and said null hypothesis:

|  |  |  |  |  |  |  |
| :--- |:---:|:---:|:---:|:---:|:---:|:---:|
|  | **Face 1** | **Face 2** | **Face 3** | **Face 4** | **Face 5** | **Face 6** |
| **Observations** | 5 | 8 | 9 | 8 | 10 | 20 |
| **Null Hypothesis** | 10 | 10 | 10 | 10 | 10 | 10 |

The $\chi^2$ gets calculated as

\begin{align}
\chi^2 = \frac{(5-10)^2}{10} + \frac{(8-10)^2}{10} + \frac{(9-10)^2}{10} &+& \\
\frac{(8-10)^2}{10} + \frac{(10-10)^2}{10} + \frac{(20-10)^2}{10} = 13.4
\end{align}

The number of degrees of freedom is the number of terms minus 1 , so $6-1=5$. Looking up for the values of the $\chi^2$ distribution at this number of degrees of freedom and for a confidence level of $95\%$ we get a value of $11.070$. Because our calculated $\chi^2$ exceeds the table value, this means that the $p$-value associated to it is smaller than $0.05$, so we can discard the null hypothesis at that significance level.

Nevertheless, note that if we choose a confidence level of $99\%$ instead, so want to be safer, we cannot discard the null hypothesis as the table value for the $\chi^2$ at that level is $15.086$, bigger than our calculated one, hence the $p$-value does pass the required threshold of $0.01$.

## A typical use case: the goodness of a distribution fit

The $\chi^2$ test is widely used to determine how good a fit is, that is, how well a statistical model describes (fits) the observational points. The number of degrees of freedom to be used to retrieve the comparison with table values is the total number of observations minus the number of fit parameters.

### Fitting a uniform distribution

Let's say we have $n$ data points and we bin them into $b$ bins, the expected occurrence frequency of each bin (the number per bin expected) would be, given that the distribution is uniform,

$$
h_i = \frac{n}{b} \ \ \forall i \ ,
$$

$i$ being the index of the bins.

Our $\chi^2$ test statistic is

$$
\chi^2 = \sum_{i=1}^b \frac{(o_i - h_i)^2}{h_i}
$$

where $o_i$ is the observed number of data points in the bin.

### Fitting a non-uniform distribution

In that case the hypothesis values have to be computed from the hypothesis distribution.

## References

1.  [The original source for the dice example](http://ccnmtl.columbia.edu/projects/qmss/the_chisquare_test/about_the_chisquare_test.html)

```python

```