Probabilistic Programming
=====
and Bayesian Methods for Hackers
========

#####Version 0.1
Welcome to *Bayesian Methods for Hackers*. The full Github repository, and additional chapters, is available at [github/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers](https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers). We hope you enjoy the book, and we encourage any contributions!

Chapter 1
======
***

The Philosophy of Bayesian Inference
------

> You are a skilled programmer, but bugs still slip into your code. After a particularly difficult implementation of an algorithm, you decide to test your code on a trivial example. It passes. You test the code on a harder problem. It passes once again. And it passes the next, *even more difficult*, test too! You are starting to believe that there may be no bugs in this code...

If you think this way, then congratulations, you already are a Bayesian practitioner! Bayesian inference is simply updating your beliefs after considering new evidence. A Bayesian can rarely be certain about a result, but he or she can be very confident. Just like in the example above, we can never be 100% sure that our code is bug-free unless we test it on every possible problem; something rarely possible in practice. Instead, we can test it on a large number of problems, and if it succeeds we can feel more *confident* about our code.  Bayesian inference works identically: we update our beliefs about an outcome; rarely can we be absolutely sure unless we rule out all other alternatives.

###The Bayesian state of mind

Bayesian inference differs from more traditional statistical inference by preserving *uncertainty* about our beliefs. At first, this sounds like a bad statistical technique. Isn't statistics all about deriving *certainty* from randomness? To reconcile this, we need to start thinking like Bayesians.

The Bayesian world-view interprets probability as measure of *believability in an event*, that is, how confident we are in an event occurring. In fact, we will see in a moment that this is the natural interpretation of probability.

For this to be clearer, we consider an alternative interpretation of probability: *Frequentist* methods assume that probability is the long-run frequency of events (hence the bestowed title). For example, the *probability of plane accidents* under a frequentist philosophy is interpreted as the *long-term frequency of plane accidents*. This makes logical sense for many probabilities of events, but becomes more difficult to understand when events have no long-term frequency of occurrences. Consider: we often assign probabilities to outcomes of presidential elections, but the election itself only happens once! Frequentists get around this by invoking alternative realities and saying across all these universes, the frequency of occurrences defines the probability.

Bayesians, on the other hand, have a more intuitive approach. Bayesians interpret a probability as measure of *belief*, or confidence, of an event occurring. An individual who assigns a belief of 0 to an event has no confidence that the event will occur; conversely, assigning a belief of 1 implies that the individual is absolutely certain of an event occurring. Beliefs between 0 and 1 allow for weightings of other outcomes. This definition agrees with the probability of a plane accident example, for having observed the frequency of plane accidents, an individual's belief should be equal to that frequency, excluding any outside information. Similarly, under this definition of probability being equal to beliefs, it is clear how we can speak about probabilities (beliefs) of presidential election outcomes: how confident are you candidate A will win?

Notice in the paragraph above, I assigned the belief (probability) measure to an *individual*, not to Nature. This is very interesting, as this definition leaves room for conflicting beliefs between individuals. Again, this is appropriate for what naturally occurs: different individuals have different beliefs of events occurring, because they possess different *information* about the world. Consider the following examples demonstrating the relationship between individual beliefs and probabilities:

- I flip a coin, and we both guess the result. We would both agree, assuming the coin is fair, that the probability of heads is 1/2. Assume, then, that I peek at the coin. Now I know for certain what the result is: I assign probability 1.0 to either heads or tails. Now what is *your* belief that the coin is heads? My knowledge of the outcome has not changed the coin's results. Thus we assign different probabilities to the result.

-  Your code either has a bug in it or not, but we do not know for certain which is true. Though we have a belief about the presence or absence of a bug.

-  A medical patient is exhibiting symptoms $x$, $y$ and $z$. There are a number of diseases that could be causing all of them, but only a single disease is present. A doctor has beliefs about which disease.

This philosophy of treating beliefs as probability is natural to humans. We employ it constantly as we interact with the world and only see partial evidence. Alternatively, you have to be *trained* to think like a frequentist.

To align ourselves with traditional probability notation, we denote our belief about event $A$ as $P(A)$. We call this quantity the *prior probability*.

John Maynard Keynes, a great economist and thinker, said "When the facts change, I change my mind. What do you do, sir?" This quote reflects the way a Bayesian updates his or her beliefs after seeing evidence. Even — especially — if the evidence is counter to what was initially believed, the evidence cannot be ignored. We denote our updated belief as $P(A |X )$, interpreted as the probability of $A$ given the evidence $X$. We call the updated belief the *posterior probability* so as to contrast it with the prior probability. For example, consider the posterior probabilities (read: posterior beliefs) of the above examples, after observing some evidence $X$.:

1\. $P(A): \;\;$ the coin has a 50 percent chance of being heads. $P(A | X):\;\;$ You look at the coin, observe a heads has landed, denote this information $X$, and trivially assign probability 1.0 to heads and 0.0 to tails.

2\.   $P(A): \;\;$  This big, complex code likely has a bug in it. $P(A | X): \;\;$ The code passed all $X$ tests; there still might be a bug, but its presence is less likely now.

3\.  $P(A):\;\;$ The patient could have any number of diseases. $P(A | X):\;\;$ Performing a blood test generated evidence $X$, ruling out some of the possible diseases from consideration.

It's clear that in each example we did not completely discard the prior belief after seeing new evidence $X$, but we *re-weighted the prior* to incorporate the new evidence (i.e. we put more weight, or confidence, on some beliefs versus others).

By introducing prior uncertainty about events, we are already admitting that any guess we make is potentially very wrong. After observing data, evidence, or other information, we update our beliefs, and our guess becomes *less wrong*. This is the alternative side of the prediction coin, where typically we try to be *more right*.

###Bayesian Inference in Practice

 If frequentist and Bayesian inference were programming functions, with inputs being statistical problems, then the two would be different in what they return to the user. The frequentist inference function would return a number, whereas the Bayesian function would return *probabilities*.

For example, in our debugging problem above, calling the frequentist function with the argument "My code passed all $X$ tests; is my code bug-free?" would return a *YES*. On the other hand, asking our Bayesian function "Often my code has bugs. My code passed all $X$ tests; is my code bug-free?" would return something very different: a probabilities of *YES* and *NO*. The function might return:

>    *YES*, with probability 0.8; *NO*, with probability 0.2

This is very different from the answer the frequentist function returned. Notice that the Bayesian function accepted an additional argument:  *"Often my code has bugs"*. This parameter is the *prior*. By including the prior parameter, we are telling the Bayesian function to include our belief about the situation. Technically this parameter in the Bayesian function is optional, but we will see excluding it has its own consequences.

####Incorporating evidence

As we acquire more and more instances of evidence, our prior belief is *washed out* by the new evidence. This is to be expected. For example, if your prior belief is something ridiculous, like "I expect the sun to explode today", and each day you are proved wrong, you would hope that any inference would correct you, or at least align your beliefs better. Bayesian inference will correct this belief.

Denote $N$ as the number of instances of evidence we possess. As we gather an *infinite* amount of evidence, say as $N \rightarrow \infty$, our Bayesian results align with frequentist results. Hence for large $N$, statistical inference is more or less objective. On the other hand, for small $N$, inference is much more *unstable*: frequentist estimates have more variance and larger confidence intervals. This is where Bayesian analysis excels. By introducing a prior, and returning probabilities (instead of a scalar estimate), we *preserve the uncertainty* that reflects the instability of stastical inference of a small $N$ dataset.

One may think that for large $N$, one can be indifferent between the two techniques since they offer similar inference, and might lean towards the computational-simpler, frequentist methods. An individual in this position should consider the following quote by Andrew Gelman (2005)[1], before making such a decision:

> Sample sizes are never large. If $N$ is too small to get a sufficiently-precise estimate, you need to get more data (or make more assumptions). But once $N$ is "large enough," you can start subdividing the data to learn more (for example, in a public opinion poll, once you have a good estimate for the entire country, you can estimate among men and women, northerners and southerners, different age groups, etc.). $N$ is never enough because if it were "enough" you'd already be on to the next problem for which you need more data.

### Are frequentist methods incorrect then?

**No.**

Frequentist methods are still useful or state-of-the-art in many areas. Tools like Least Squares linear regression, LASSO regression, EM algorithm etc. are all very powerful and incredibly fast. Bayesian methods are a compliment to solve the problems these solutions cannot or to gain further insight into the underlying system by offering more flexibility in modeling.

#### A note on *Big Data*
Paradoxically, big data's predictive analytic problems are actually solved by relatively simple algorithms [2][4]. Thus we can argue that big data's prediction difficulty does not lie in the algorithm used, but instead on the computational difficulties of storage and execution on big data. (One should also consider Gelman's quote from above and ask "Do I really have big data?" )

The much more difficult analytic problems involve *medium data* and, especially troublesome, *really small data*. Using a similar argument as  Gelman's above, if big data problems are *big enough* to be readily solved, then we should be more interested in the *not-quite-big enough* datasets.

### Our Bayesian framework

We are interested in beliefs, which can be interpreted as probabilities by thinking Bayesian. We have a *prior* belief in event $A$, beliefs formed by previous information, e.g., our prior belief about bugs being in our code before performing tests.

Secondly, we observe our evidence. To continue our buggy-code example: if our code passes $X$ tests, we want to update our belief to incorporate this. We call this new belief the *posterior* probability. Updating our belief is done via the following equation, known as Bayes' Theorem, after it's discoverer Thomas Bayes:

\begin{align}
 P( A | X ) = & \frac{ P(X | A) P(A) } {P(X) } \\\\[5pt]
& \propto P(X | A) P(A)\;\; (\propto \text{is proportional to } )
\end{align}

The above formula is not unique to Bayesian inference: it is a mathematical fact with uses outside Bayesian inference. Bayesian inference merely uses it to connect prior probabilities $P(A)$ with an updated posterior probabilities $P(A | X )$.

##### Example: Mandatory coin-flip example

Every statistics text must contain a coin-flipping example, I'll use it here to get it out of the way. Suppose, naively, that you are unsure about the probability of heads in a coin flip (spoiler alert: it's 50%). You believe there is some true underlying ratio, call it $p$, but have no prior opinion on what $p$ might be.

We begin to flip a coin, and record the observations: either $H$ or $T$. This is our observed data. An interesting question to ask is how our inference changes as we observe more and more data? More specifically, what do our posterior probabilities look like when we have little data, versus when we have lots of data.

Below we plot a sequence of updating posterior probabilities as we observe increasing amounts of data (coin flips).

```
"""
The book uses a custom matplotlibrc file, which provides the unique styles for matplotlib plots.
If executing this book, and you wish to use the book's styling, provided are two options:
    1. Overwrite your own matplotlibrc file with the rc-file provided in the book's styles/ dir.
       See http://matplotlib.org/users/customizing.html
    2. Also in the styles is  bmh_matplotlibrc.json file. This can be used to update the styles
       in only this notebook. Try running the following code:

        import json
        s = json.load( open("../styles/bmh_matplotlibrc.json") )
        matplotlib.rcParams.update(s)

"""

#the code below can be passed over, as it is currently not important.
%pylab inline
figsize( 11, 9)

import scipy.stats as stats

dist = stats.beta
n_trials = [0,1,2,3,4,5,8,15, 50, 500]
data = stats.bernoulli.rvs(0.5, size = n_trials[-1] )
x = np.linspace(0,1,100)

for k, N in enumerate(n_trials):
    sx = subplot( len(n_trials)/2, 2, k+1)

    plt.setp(sx.get_yticklabels(), visible=False)
    heads = data[:N].sum()
    y = dist.pdf(x, 1 + heads, 1 + N - heads )
    plt.plot( x, y, label= "observe %d tosses,\n %d heads"%(N,heads) )
    plt.fill_between( x, 0, y, color="#348ABD", alpha = 0.4 )
    plt.vlines( 0.5, 0, 4, color = "k", linestyles = "--", lw=1 )

    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight = True)

plt.suptitle( "Bayesian updating of posterior probabilities",
              y = 1.02,
              fontsize = 14);

plt.tight_layout()
```

The posterior probabilities are represented by the curves, and our confidence is proportional to the height of the curve. As the plot above shows, as we start to observe data our posterior probabilities start to shift and move around. Eventually, as we observe more and more data (coin-flips), our probabilities will lump closer and closer around the true value of $p=0.5$ (marked by a dashed line).

Notice that the plots are not always *peaked* at 0.5. There is no reason it should be: recall we assumed we did not have a prior opinion of what $p$ is. In fact, if we observe quite extreme data, say 8 flips and only 1 observed heads, our distribution would look very biased *away* from lumping around 0.5. As more data accumulates, we would see more and more probability being assigned at $p=0.5$.

The next example is a simple demonstration of the mathematics of Bayesian inference.

#####Example: Bug, or just sweet, unintended feature?

Let $A$ denote the event that our code has **no bugs** in it. Let $X$ denote the event that the code passes all debugging tests. For now, we will leave the prior probability of no bugs as a variable, i.e. $P(A) = p$.

We are interested in $P(A|X)$, i.e. the probability of no bugs, given our debugging tests $X$. To use the formula above, we need to compute some quantities.

What is $P(X | A)$, i.e., the probability that the code passes $X$ tests *given* there are no bugs? Well, it is equal to 1, for a code with no bugs will pass all tests.

$P(X)$ is a little bit trickier: The event $X$ can be divided into two possibilities, event $X$ occurring even though our code *indeed has* bugs (denoted $\sim A\;$, spoken *not $A$*), or event $X$ without bugs ($A$). $P(X)$ can be represented as:

\begin{align}
P(X ) & = P(X \text{ and } A) + P(X \text{ and } \sim A) \\\\[5pt]
 & = P(X|A)P(A) + P(X | \sim A)P(\sim A)\\\\[5pt]
& = P(X|A)p + P(X | \sim A)(1-p)
\end{align}

We have already computed $P(X|A)$ above. On the other hand, $P(X | \sim A)$ is subjective: our code can pass tests but still have a bug in it, though the probability there is a bug present is reduced. Note this is dependent on the number of tests performed, the degree of complication in the tests, etc. Let's be conservative and assign $P(X|\sim A) = 0.5$. Then

\begin{align}
P(A | X) & = \frac{1\cdot p}{ 1\cdot p +0.5 (1-p) } \\\\
& = \frac{ 2 p}{1+p}
\end{align}
This is the posterior probability. What does it look like as a function of our prior, $p \in [0,1]$?

```
figsize(12.5,4)
p = np.linspace( 0,1, 50)
plt.plot( p, 2*p/(1+p), color = "#348ABD", lw = 3 )
#plt.fill_between( p, 2*p/(1+p), alpha = .5, facecolor = ["#A60628"])
plt.scatter( 0.2, 2*(0.2)/1.2, s = 140, c ="#348ABD"  )
plt.xlim( 0, 1)
plt.ylim( 0, 1)
plt.xlabel( "Prior, $P(A) = p$")
plt.ylabel("Posterior, $P(A|X)$, with $P(A) = p$")
plt.title( "Are there bugs in my code?");
```

We can see the biggest gains if we observe the $X$ tests passed when the prior probability, $p$, is low. Let's settle on a specific value for the prior. I'm a strong programmer (I think), so I'm going to give myself a realistic prior of 0.20, that is, there is a 20% chance that I write code bug-free. To be more realistic, this prior should be a function of how complicated and large the code is, but let's pin it at 0.20. Then my updated belief that my code is bug-free is 0.33.

Recall that the prior is a probability: $p$ is the prior probability that there *are no bugs*, so $1-p$ is the prior probability that there *are bugs*.

Similarly, our posterior is also a probability, with $P(A | X)$ the probability there is no bug *given we saw all tests pass*, hence $1-P(A|X)$ is the probability there is a bug *given all tests passed*. What does our posterior probability look like? Below is a graph of both the prior and the posterior probabilities.

```
figsize( 12.5, 4 )
colours = ["#348ABD", "#A60628"]

prior = [0.20, 0.80]
posterior =  [1./3, 2./3]
plt.bar( [0,.7], prior ,alpha = 0.70, width = 0.25, \
            color = colours[0], label = "prior distribution",
             lw = "3", edgecolor = colours[0])

plt.bar( [0+0.25,.7+0.25], posterior ,alpha = 0.7, \
          width = 0.25, color =  colours[1],
          label = "posterior distribution",
          lw = "3", edgecolor = colours[1])

plt.xticks( [0.20,.95], ["Bugs Absent", "Bugs Present"] )
plt.title("Prior and Posterior probability of bugs present, prior = 0.2")
plt.ylabel("Probability")
plt.legend(loc="upper left");
```

Notice that after we observed $X$ occur, the probability of bugs being absent increased. By increasing the number of tests, we can approach confidence (probability 1) that there are no bugs present.

This was a very simple example of Bayesian inference and Bayes rule. Unfortunately, the mathematics necessary to perform more complicated Bayesian inference only becomes more difficult, except for artificially constructed cases. We will later see that this type of mathematical analysis is actually unnecessary. First we must broaden our modeling tools. The next section deals with *probability distributions*. If you are already familiar, feel free to skip (or at least skim), but for the less familiar the next section is essential.

_______

##Probability Distributions

**Let's quickly recall what a probability distribution is:** Let $Z$ be some random variable. Then associated with $Z$ is a *probability distribution function* that assigns probabilities to the different outcomes $Z$ can take. Graphically, a probability distribution is a curve where the probability of an outcome is proportional to the height of the curve. You can see examples in the first figure of this chapter.

We can divide random variables into three classifications:

-   **$Z$ is discrete**: Discrete random variables may only assume values on a specified list. Things like populations, movie ratings, and number of votes are all discrete random variables. Discrete random variables become more clear when we contrast them with...

-   **$Z$ is continuous**: Continuous random variable can take on arbitrarily exact values. For example, temperature, speed, time, color are all modeled as continuous variables because you can progressively make the values more and more precise.

- **$Z$ is mixed**: Mixed random variables assign probabilities to both discrete and continuous random variables, i.e. it is a combination of the above two categories.

###Discrete Case
If $Z$ is discrete, then its distribution is called a *probability mass function*, which measures the probability $Z$ takes on the value $k$, denoted $P(Z=k)$. Note that the probability mass function completely describes the random variable $Z$, that is, if we know the mass function, we know how $Z$ should behave. There are popular probability mass functions that consistently appear: we will introduce them as needed, but let's introduce the first very useful probability mass function. We say $Z$ is *Poisson*-distributed if:

$$P(Z = k) =\frac{ \lambda^k e^{-\lambda} }{k!}, \; \; k=0,1,2, \dots $$

What is $\lambda$? It is called the parameter, and it describes the shape of the distribution. For the Poisson random variable, $\lambda$ can be any positive number. By increasing $\lambda$, we add more probability to larger values, and conversely by decreasing $\lambda$ we add more probability to smaller values. Unlike $\lambda$, which can be any positive number, $k$ must be a non-negative integer, i.e., $k$ must take on values 0,1,2, and so on. This is very important, because if you wanted to model a population you could not make sense of populations with 4.25 or 5.612 members.

If a random variable $Z$ has a Poisson mass distribution, we denote this by writing

$$Z \sim \text{Poi}(\lambda) $$

One very useful property of the Poisson random variable, given we know $\lambda$, is that its expected value is equal to the parameter, ie.:

$$E\large[ \;Z\; | \; \lambda \;\large] = \lambda $$

We will use this property often, so it's something useful to remember. Below we plot the probability mass distribution for different $\lambda$ values. The first thing to notice is that by increasing $\lambda$ we add more probability to larger values occurring. Secondly, notice that although the graph ends at 15, the distributions do not. They assign positive probability to every non-negative integer.

```
figsize( 12.5, 4)

import scipy.stats as stats
a = np.arange( 16 )
poi = stats.poisson
lambda_ = [1.5, 4.25 ]

plt.bar( a, poi.pmf( a, lambda_[0]), color=colours[0],
        label = "$\lambda = %.1f$"%lambda_[0], alpha = 0.60,
        edgecolor = colours[0], lw = "3")

plt.bar( a, poi.pmf( a, lambda_[1]), color=colours[1],
         label = "$\lambda = %.1f$"%lambda_[1], alpha = 0.60,
          edgecolor = colours[1], lw = "3")

plt.xticks( a + 0.4, a )
plt.legend()
plt.ylabel("probability of $k$")
plt.xlabel("$k$")
plt.title("Probability mass function of a Poisson random variable; differing \
$\lambda$ values");
```

###Continuous Case
Instead of a probability mass function, a continuous random variable has a *probability density function*. This might seem like unnecessary nomenclature, but the density function and the mass function are very different creatures. An example of continuous random variable is a random variable with a *exponential density*. The density function for an exponential random variable looks like:

$$f_Z(z | \lambda) = \lambda e^{-\lambda z }, \;\; z\ge 0$$

Like the Poisson random variable, an exponential random variable can only take on non-negative values. But unlike a Poisson random variable, the exponential can take on *any* non-negative values, like 4.25 or 5.612401. This makes it a poor choice for count data, which must be integers, but a great choice for time data, or temperature data (measured in Kelvins, of course), or any other precise *and positive* variable. Below are two probability density functions with different $\lambda$ value.

When a random variable $Z$ has an exponential distribution with parameter $\lambda$, we say *$Z$ is exponential* and write

$$Z \sim \text{Exp}(\lambda)$$

Given a specific $\lambda$, the expected value of an exponential random variable is equal to the inverse of $\lambda$, that is:

$$E[\; Z \;|\; \lambda \;] = \frac{1}{\lambda}$$

```
a = np.linspace(0,4, 100)
expo = stats.expon
lambda_ = [0.5, 1]

for l,c in zip(lambda_,colours):
    plt.plot( a, expo.pdf( a, scale=1./l), lw=3,
                color=c, label = "$\lambda = %.1f$"%l)
    plt.fill_between( a, expo.pdf( a, scale=1./l), color=c, alpha = .33)

plt.legend()
plt.ylabel("PDF at $z$")
plt.xlabel("$z$")
plt.title("Probability density function of an Exponential random variable;\
 differing $\lambda$");
```

###But what is $\lambda \;$?

**This question is what motivates statistics**. In the real world, $\lambda$ is hidden from us. We only see $Z$, and must go backwards to try and determine $\lambda$. The problem is so difficult because there is not a one-to-one mapping from $Z$ to $\lambda$. Many different methods have been created to solve the problem of estimating $\lambda$, but since $\lambda$ is never actually observed, no one can say for certain which method is better!

Bayesian inference is concerned with *beliefs* about what $\lambda$ is. Rather than try to guess $\lambda$ exactly, we can only talk about what $\lambda$ is likely to be by assigning a probability distribution to $\lambda$.

This might seem odd at first: after all, $\lambda$ is fixed, it is not (necessarily) random! How can we assign probabilities to a non-random event. Ah, we have fallen for the frequentist interpretation. Recall, under our Bayesian philosophy, we *can* assign probabilities if we interpret them as beliefs. And it is entirely acceptable to have *beliefs* about the parameter $\lambda$.

##### Example: Inferring behaviour from text-message data

Let's try to model a more interesting example, concerning text-message rates:

>  You are given a series of text-message counts from a user of your system. The data, plotted over time, appears in the graph below. You are curious if the user's text-messaging habits changed over time, either gradually or suddenly. How can you model this? (This is in fact my own text-message data. Judge my popularity as you wish.)

```
figsize( 12.5, 3.5 )
count_data = np.loadtxt("data/txtdata.csv")
n_count_data = len(count_data)
plt.bar( np.arange( n_count_data ), count_data, color ="#348ABD"  )
plt.xlabel( "Time (days)")
plt.ylabel("count of text-msgs received")
plt.title("Did the user's texting habits change over time?")
plt.xlim( 0, n_count_data );
```

Before we begin, with respect to the plot above, would you say there was a change in behaviour
during the time period?

How can we start to model this? Well, as I conveniently already introduced, a Poisson random variable would be a very appropriate model for this *count* data. Denoting day $i$'s text-message count by $C_i$,

$$ C_i \sim \text{Poisson}(\lambda)  $$

We are not sure about what the $\lambda$ parameter is though. Looking at the chart above, it appears that the rate might become higher at some later date, which is equivalently saying the parameter $\lambda$ increases at some later date (recall a higher $\lambda$ means more probability on larger outcomes, that is, higher probability of many texts.).

How can we mathematically represent this? We can think, that at some later date (call it $\tau$), the parameter $\lambda$ suddenly jumps to a higher value. So we create two $\lambda$ parameters, one for behaviour before the $\tau$, and one for behaviour after. In literature, a sudden transition like this would be called a *switchpoint*:

$$
\lambda =
\begin{cases}
\lambda_1  & \text{if } t \lt \tau \cr
\lambda_2 & \text{if } t \ge \tau
\end{cases}
$$

 If, in reality, no sudden change occurred and indeed $\lambda_1 = \lambda_2$, the $\lambda$'s posterior distributions should look about equal.

We are interested in inferring the unknown $\lambda$s. To use Bayesian inference, we need to assign prior probabilities to the different possible values of $\lambda$. What would be good prior probability distributions for $\lambda_1$ and $\lambda_2$? Recall that $\lambda_i, \; i=1,2,$ can be any positive number. The *exponential* random variable has a density function for any positive number. This would be a good choice to model $\lambda_i$. But, we need a parameter for this exponential distribution: call it $\alpha$.

\begin{align}
&\lambda_1 \sim \text{Exp}( \alpha ) \\\
&\lambda_2 \sim \text{Exp}( \alpha )
\end{align}

$\alpha$ is called a *hyper-parameter*, or a *parent-variable*, literally a parameter that influences other parameters. The influence is not too strong, so we can choose $\alpha$ liberally.  A good rule of thumb is to set the exponential parameter equal to the inverse of the average of the count data, since we're modeling $\\lambda$ using an Exponential distribution we can use the expected value identity shown earlier to get:

$$\frac{1}{N}\sum_{i=0}^N \;C_i \approx E[\; \lambda \; |\; \alpha ] = \frac{1}{\alpha}$$

Alternatively, and something I encourage the reader to try, is to have two priors: one for each $\lambda_i$; creating two exponential distributions with different $\alpha$ values reflects a prior belief that the rate changed after some period.

What about $\tau$? Well, due to the randomness, it is too difficult to pick out when $\tau$ might have occurred. Instead, we can assign an *uniform prior belief* to every possible day. This is equivalent to saying

\begin{align}
& \tau \sim \text{DiscreteUniform(1,70) }\\\\
& \Rightarrow P( \tau = k ) = \frac{1}{70}
\end{align}

So after all this, what does our overall prior for the unknown variables look like? Frankly, *it doesn't matter*. What we should understand is that it would be an ugly, complicated, mess involving symbols only a mathematician would love. And things would only get uglier the more complicated our models become. Regardless, all we really care about is the posterior distribution. We next turn to PyMC, a Python library for performing Bayesian analysis, that is agnostic to the mathematical monster we have created.

Introducing our first hammer: PyMC
-----

PyMC is a Python library for programming Bayesian analysis [3]. It is a fast, well-maintained library. The only unfortunate part is that documentation can be lacking in areas, especially the bridge between beginner to hacker. One of this book's main goals is to solve that problem, and also to demonstrate why PyMC is so cool.

We will model the above problem using the PyMC library. This type of programming is called *probabilistic programming*, an unfortunate misnomer that invokes ideas of randomly-generated code and has likely confused and frightened users away from this field. The code is not random. The title is given because we create probability models using programming variables as the model's components, that is, model components are first-class primitives in this framework.

B. Cronin [5] has a very motivating description of probabilistic programming:

>   Another way of thinking about this: unlike a traditional program, which only runs in the forward directions, a probabilistic program is run in both the forward and backward direction. It runs forward to compute the consequences of the assumptions it contains about the world (i.e., the model space it represents), but it also runs backward from the data to constrain the possible explanations. In practice, many probabilistic programming systems will cleverly interleave these forward and backward operations to efficiently home in on the best explanations.

Due to its poorly understood title, I'll refrain from using the name *probabilistic programming*. Instead, I'll simply use *programming*, as that is what it really is.

The PyMC code is easy to follow along: the only novel thing should be the syntax, and I will interrupt the code to explain sections. Simply remember we are representing the model's components ($\tau, \lambda_1, \lambda_2$ ) as variables:

```
import pymc as mc

n = count_data.shape[0]

alpha = 1.0/count_data.mean() #recall count_data is
                              #the variable that holds our txt counts

lambda_1 = mc.Exponential( "lambda_1",  alpha )
lambda_2 = mc.Exponential( "lambda_2", alpha )

tau = mc.DiscreteUniform( "tau", lower = 0, upper = n )
```

In the above code, we create the PyMC variables corresponding to $\lambda_1, \; \lambda_2$. We assign them to PyMC's *stochastic variables*, called stochastic variables because they are treated by the backend as random number generators. We can test this by calling their built-in `random()` method.

```
print "Random output:", tau.random(),tau.random(), tau.random()
```

    Random output: 13 12 54

```
@mc.deterministic
def lambda_( tau = tau, lambda_1 = lambda_1, lambda_2 = lambda_2 ):
    out = np.zeros( n )
    out[:tau] = lambda_1 #lambda before tau is lambda1
    out[tau:] = lambda_2 #lambda after tau is lambda2
    return out
```

This code is creating a new function `lambda_`, but really we think of it as a random variable: the random variable $\lambda$ from above. Note that because `lambda_1`, `lambda_2` and `tau` are random, `lambda_` will be random. We are **not** fixing any variables yet. The `@mc.deterministic` is a decorator to tell PyMC that this is a deterministic function, i.e., if the arguments were deterministic (which they are not), the output would be deterministic as well.

```
observation = mc.Poisson( "obs", lambda_, value = count_data, observed = True)

model = mc.Model( [observation, lambda_1, lambda_2, tau] )
```

The variable `observation` combines our data, `count_data`, with our proposed data-generation scheme, given by the variable `lambda_`, through the `value` keyword. We also set `observed = True` to tell PyMC that this should stay fixed in our analysis. Finally, PyMC wants us to collect all the variables of interest and create a `Model` instance out of them. This makes our life easier when we try to retrieve the results.

The below code will be explained in the Chapter 3, but this is where our results come from. One can think of it as a *learning* step. The machinery being employed is called *Monte Carlo Markov Chains* (which I delay explaining until Chapter 3). It returns thousands of random variables from the posterior distributions of $\lambda_1, \lambda_2$ and $\tau$. We can plot a histogram of the random variables to see what the posterior distribution looks like. Below, we collect the samples (called *traces* in MCMC literature) in histograms.

```
### Mysterious code to be explained in Chapter 3.
mcmc = mc.MCMC(model)
mcmc.sample( 35000, 5000, 1 )
```

    [****************100%******************]  35000 of 35000 complete

```
lambda_1_samples = mcmc.trace( 'lambda_1' )[:]
lambda_2_samples = mcmc.trace( 'lambda_2' )[:]
tau_samples = mcmc.trace( 'tau' )[:]
```

```
figsize(12.5, 10)
#histogram of the samples:

ax = plt.subplot(311)
ax.set_autoscaley_on(False)

plt.hist( lambda_1_samples, histtype='stepfilled', bins = 30, alpha = 0.85,
        label = "posterior of $\lambda_1$", color = "#A60628",normed = True )
plt.legend(loc = "upper left")
plt.title(r"Posterior distributions of the variables $\lambda_1,\;\lambda_2,\;\tau$")
plt.xlim([15,30])
plt.xlabel("$\lambda_2$ value")
plt.ylabel("probability")

ax = plt.subplot(312)
ax.set_autoscaley_on(False)

plt.hist( lambda_2_samples,histtype='stepfilled', bins = 30, alpha = 0.85,
          label = "posterior of $\lambda_2$",color="#7A68A6", normed = True )
plt.legend(loc = "upper left")
plt.xlim([15,30])
plt.xlabel("$\lambda_2$ value")
plt.ylabel("probability")

plt.subplot(313)

w = 1.0/ tau_samples.shape[0] * np.ones_like( tau_samples )
plt.hist( tau_samples, bins = n_count_data, alpha = 1,
         label = r"posterior of $\tau$",
         color="#467821", weights=w, rwidth =1. )

plt.legend(loc = "upper left");
plt.ylim([0,.75])
plt.xlim([35, len(count_data)-20])
plt.xlabel("days")
plt.ylabel("probability");
```

### Interpretation

Recall that the Bayesian methodology returns a *distribution*, hence we now have distributions to describe the unknown $\lambda$'s and $\tau$. What have we gained? Immediately we can see the uncertainty in our estimates: the more variance in the distribution, the less certain our posterior belief should be. We can also say what a plausible value for the parameters might be: $\lambda_1$ is around 18 and $\lambda_2$ is around 23. What other observations can you make? Look at the data again, do these seem reasonable? The distributions of the two $\\lambda$s are positioned very differently, indicating that it's likely there was a change in the user's text-message behaviour.

Also notice that the posterior distributions for the $\lambda$'s do not look like any exponential distributions, though we originally started modelling with exponential random variables. They are really not anything we recognize. But this is OK. This is one of the benefits of taking a computational point-of-view. If we had instead done this mathematically, we would have been stuck with a very analytically intractable (and messy) distribution. Via computations, we are agnostic to the tractability.

Our analysis also returned a distribution for what $\tau$ might be. Its posterior distribution looks a little different from the other two because it is a discrete random variable, hence it doesn't assign probabilities to intervals. We can see that near day 45, there was a 50% chance the users behaviour changed. Had no change occurred, or the change been gradual over time, the posterior distribution of $\tau$ would have been more spread out, reflecting that many values are likely candidates for $\tau$. On the contrary, it is very peaked.

###Why would I want samples from the posterior, anyways?

We will deal with this question for the remainder of the book, and it is an understatement to say we can perform amazingly useful things. For now, let's end this chapter with one more example. We'll use the posterior samples to answer the following question: what is the expected number of texts at day $t, \; 0 \le t \le70$? Recall that the expected value of a Poisson is equal to its parameter $\lambda$, then the question is equivalent to *what is the expected value of $\lambda$ at time $t$*?

In the code below, we are calculating the following: Let $i$ index samples from the posterior distributions. Given a day $t$, we average over all possible $\lambda_i$ for that day $t$, using $\lambda_i = \lambda_{1,i}$ if $t \lt \tau_i$ (that is, if the behaviour change hadn't occurred yet), else we use $\lambda_i = \lambda_{2,i}$.

___________________

```
figsize( 12.5, 4)
# tau_samples, lambda_1_samples, lambda_2_samples contain
# N samples from the corresponding posterior distribution
N = tau_samples.shape[0]
expected_texts_per_day = np.zeros(n_count_data)
for  day in range(0, n_count_data):
    # ix is a bool index of all tau samples corresponding to
    # the switchpoint occurring prior to value of 'day'
    ix = day < tau_samples
    # Each posterior sample corresponds to a value for tau.
    # for each day, that value of tau indicates whether we're "before" (in the lambda1 "regime")
    # or "after" (in the lambda2 "regime") the switchpoint.
    # by taking the posterior sample of lambda1/2 accordingly, we can average
    # over all samples to get an expected value for lambda on that day.
    # As explained, the "message count" random variable is Poisson distributed,
    # and therefore lambda (the poisson parameter) is the expected value of "message count"
    expected_texts_per_day[day] = (lambda_1_samples[ix].sum()
                                    + lambda_2_samples[~ix].sum() ) /N

plt.plot( range( n_count_data), expected_texts_per_day, lw =4, color = "#E24A33" )
plt.xlim( 0, n_count_data )
plt.xlabel( "Day" )
plt.ylabel( "Expected # text-messages" )
plt.title( "Expected number of text-messages received")
#plt.ylim( 0, 35 )
plt.bar( np.arange( len(count_data) ), count_data, color ="#348ABD", alpha = 0.5,
            label="observed texts per day")

plt.legend(loc="upper left");
```

Our analysis shows strong support for believing the user's behavior did change ($\lambda_1$ would have been close in value to $\lambda_2$ had this not been true), and the change was sudden rather then gradual (demonstrated by $\tau$'s strongly peaked posterior distribution). We can speculate what might have caused this: a cheaper text-message rate, a recent weather-2-text subscription, or a new relationship. (The 45th day corresponds to Christmas, and I moved away to Toronto the next month leaving a girlfriend behind.)

##### Exercises

1\.  Using `lambda_1_samples` and `lambda_2_samples`, what is the mean of the posterior distributions of $\lambda_1$ and $\lambda_2$?

```
#type your code here.
```

2\.  What is the expected percentage increase in text-message rates? `hint:` compute the mean of `lambda_1_samples/lambda_2_samples`. Note that quantity is very different from `lambda_1_samples.mean()/lambda_2_samples.mean()`.

```
#type your code here.
```

3\. Looking at the posterior distribution graph of $\tau$, why do you think there is a small number of posterior $\tau$ samples near 0? `hint:` Look at the data again.

4\. What is the mean of $\lambda_1$ **given** we know $\tau$ is less than 45.  That is, suppose we have new information as we know for certain that the change in behaviour occurred before day 45. What is the expected value of $\lambda_1$ now? (You do not need to redo the PyMC part, just consider all instances where `tau_samples<45`. )

```
#type your code here.
```

### References

-  [1] Gelman, Andrew. N.p.. Web. 22 Jan 2013. <http://andrewgelman.com/2005/07/n_is_never_larg/>.
-  [2] Norvig, Peter. 2009. [*The Unreasonable Effectiveness of Data*](http://www.csee.wvu.edu/~gidoretto/courses/2011-fall-cp/reading/TheUnreasonable EffectivenessofData_IEEE_IS2009.pdf).
- [3] Patil, A., D. Huard and C.J. Fonnesbeck. 2010.
PyMC: Bayesian Stochastic Modelling in Python. Journal of Statistical
Software, 35(4), pp. 1-81.
- [4] Jimmy Lin and Alek Kolcz. Large-Scale Machine Learning at Twitter. Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (SIGMOD 2012), pages 793-804, May 2012, Scottsdale, Arizona.
- [5] Cronin, Beau. "Why Probabilistic Programming Matters." 24 Mar 2013. Google, Online Posting to Google . Web. 24 Mar. 2013. <https://plus.google.com/u/0/107971134877020469960/posts/KpeRdJKR6Z1>.

```
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 120%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }

</style>

```

```


```python
# Mount Google Drive
from google.colab import drive # import drive from google colab

ROOT = "/content/drive"     # default location for the drive
print(ROOT)                 # print content of ROOT (Optional)

drive.mount(ROOT,force_remount=True)
```

    /content/drive
    Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly

    Enter your authorization code:
    ··········
    Mounted at /content/drive

#  Neuromatch Academy: Week 1, Day 1, Tutorial 3
# Model Types: "Why" models
__Content creators:__ Matt Laporte, Byron Galbraith, Konrad Kording

__Content reviewers:__ Dalin Guo, Aishwarya Balwani, Madineh Sarvestani, Maryam Vaziri-Pashkam, Michael Waskom

We would like to acknowledge [Steinmetz _et al._ (2019)](https://www.nature.com/articles/s41586-019-1787-x) for sharing their data, a subset of which is used here.

___
# Tutorial Objectives
This is tutorial 3 of a 3-part series on different flavors of models used to understand neural data. In parts 1 and 2 we explored mechanisms that would produce the data. In this tutorial we will explore models and techniques that can potentially explain *why* the spiking data we have observed is produced the way it is.

To understand why different spiking behaviors may be beneficial, we will learn about the concept of entropy. Specifically, we will:

- Write code to compute formula for entropy, a measure of information
- Compute the entropy of a number of toy distributions
- Compute the entropy of spiking activity from the Steinmetz dataset

```python
#@title Video 1: “Why” models
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='OOIDEr1e5Gg', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video

```

    Video available at https://youtube.com/watch?v=OOIDEr1e5Gg

# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
```

```python
#@title Figure Settings
import ipywidgets as widgets #interactive display

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")

```

```python
#@title Helper Functions

def plot_pmf(pmf,isi_range):
  """Plot the probability mass function."""
  ymax = max(0.2, 1.05 * np.max(pmf))
  pmf_ = np.insert(pmf, 0, pmf[0])
  plt.plot(bins, pmf_, drawstyle="steps")
  plt.fill_between(bins, pmf_, step="pre", alpha=0.4)
  plt.title(f"Neuron {neuron_idx}")
  plt.xlabel("Inter-spike interval (s)")
  plt.ylabel("Probability mass")
  plt.xlim(isi_range);
  plt.ylim([0, ymax])
```

```python
#@title Download Data
import io
import requests
r = requests.get('https://osf.io/sy5xt/download')
if r.status_code != 200:
  print('Could not download data')
else:
  steinmetz_spikes = np.load(io.BytesIO(r.content), allow_pickle=True)['spike_times']
```

# Section 1: Optimization and Information

Neurons can only fire so often in a fixed period of time, as the act of emitting a spike consumes energy that is depleted and must eventually be replenished. To communicate effectively for downstream computation, the neuron would need to make good use of its limited spiking capability. This becomes an optimization problem:

What is the optimal way for a neuron to fire in order to maximize its ability to communicate information?

In order to explore this question, we first need to have a quantifiable measure for information. Shannon introduced the concept of entropy to do just that, and defined it as

\begin{align}
  H_b(X) &= -\sum_{x\in X} p(x) \log_b p(x)
\end{align}

where $H$ is entropy measured in units of base $b$ and $p(x)$ is the probability of observing the event $x$ from the set of all possible events in $X$. See the Appendix for a more detailed look at how this equation was derived.

The most common base of measuring entropy is $b=2$, so we often talk about *bits* of information, though other bases are used as well e.g. when $b=e$ we call the units *nats*.

First, let's explore how entropy changes between some simple discrete probability distributions. In the rest of this tutorial we will refer to these as probability mass functions (PMF), where $p(x_i)$ equals the $i^{th}$ value in an array, and mass refers to how much of the distribution is contained at that value.

For our first PMF, we will choose one where all of the probability mass is located in the middle of the distribution.

```python
n_bins = 50  # number of points supporting the distribution
x_range = (0, 1)  # will be subdivided evenly into bins corresponding to points

bins = np.linspace(*x_range, n_bins + 1)  # bin edges

pmf = np.zeros(n_bins)
pmf[len(pmf) // 2] = 1.0  # middle point has all the mass

# Since we already have a PMF, rather than un-binned samples, `plt.hist` is not
# suitable. Instead, we directly plot the PMF as a step function to visualize
# the histogram:
pmf_ = np.insert(pmf, 0, pmf[0])  # this is necessary to align plot steps with bin edges
plt.plot(bins, pmf_, drawstyle="steps")
# `fill_between` provides area shading
plt.fill_between(bins, pmf_, step="pre", alpha=0.4)
plt.xlabel("x")
plt.ylabel("p(x)")
plt.xlim(x_range)
plt.ylim(0, 1);
```

If we were to draw a sample from this distribution, we know exactly what we would get every time. Distributions where all the mass is concentrated on a single event are known as *deterministic*.

How much entropy is contained in a deterministic distribution? 0

## Exercise 1: Computing Entropy

Your first exercise is to implement a method that computes the entropy of a discrete probability distribution, given its mass function. Remember that we are interested in entropy in units of _bits_, so be sure to use the correct log function.

Recall that $\log(0)$ is undefined. When evaluated at $0$, NumPy log functions (such as `np.log2`) return `np.nan` ("Not a Number"). By convention, these undefined terms— which correspond to points in the distribution with zero mass—are excluded from the sum that computes the entropy.

```python
def entropy(pmf):
  """Given a discrete distribution, return the Shannon entropy in bits.

  This is a measure of information in the distribution. For a totally
  deterministic distribution, where samples are always found in the same bin,
  then samples from the distribution give no more information and the entropy
  is 0.

  For now this assumes `pmf` arrives as a well-formed distribution (that is,
  `np.sum(pmf)==1` and `not np.any(pmf < 0)`)

  Args:
    pmf (np.ndarray): The probability mass function for a discrete distribution
      represented as an array of probabilities.
  Returns:
    h (number): The entropy of the distribution in `pmf`.

  """
  ############################################################################
  # Exercise for students: compute the entropy of the provided PMF
  #   1. Exclude the points in the distribution with no mass (where `pmf==0`).
  #      Hint: this is equivalent to including only the points with `pmf>0`.
  #   2. Implement the equation for Shannon entropy (in bits).
  #  When ready to test, comment or remove the next line
  #raise NotImplementedError("Excercise: implement the equation for entropy")
  ############################################################################

  # reduce to non-zero entries to avoid an error from log2(0)
  pmf = pmf[pmf>0]

  # implement the equation for Shannon entropy (in bits)
  h = -np.sum(np.multiply(pmf,np.log2(pmf)))

  # return the absolute value (avoids getting a -0 result)
  return np.abs(h)

# Uncomment to test your entropy function
print(f"{entropy(pmf):.2f} bits")
```

    0.00 bits

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D1_ModelTypes/solutions/W1D1_Tutorial3_Solution_55c07dc8.py)

We expect zero surprise from a deterministic distribution. If we had done this calculation by hand, it would simply be

$-1\log_2 1 = -0=0$

Note that changing the location of the peak (i.e. the point and bin on which all the mass rests) doesn't alter the entropy. The entropy is about how predictable a sample is with respect to a distribution. A single peak is deterministic regardless of which point it sits on.

```python
pmf = np.zeros(n_bins)
pmf[2] = 1.0  # arbitrary point has all the mass

pmf_ = np.insert(pmf, 0, pmf[0])
plt.plot(bins, pmf_, drawstyle="steps")
plt.fill_between(bins, pmf_, step="pre", alpha=0.4)
plt.xlabel("x")
plt.ylabel("p(x)")
plt.xlim(x_range)
plt.ylim(0, 1);
```

What about a distribution with mass split equally between two points?

```python
pmf = np.zeros(n_bins)
pmf[len(pmf) // 3] = 0.5
pmf[2 * len(pmf) // 3] = 0.5

pmf_ = np.insert(pmf, 0, pmf[0])
plt.plot(bins, pmf_, drawstyle="steps")
plt.fill_between(bins, pmf_, step="pre", alpha=0.4)
plt.xlabel("x")
plt.ylabel("p(x)")
plt.xlim(x_range)
plt.ylim(0, 1);
```

Here, the entropy calculation is

$-(0.5 \log_2 0.5 + 0.5\log_2 0.5)=1$

There is 1 bit of entropy. This means that before we take a random sample, there is 1 bit of uncertainty about which point in the distribution the sample will fall on: it will either be the first peak or the second one.

Likewise, if we make one of the peaks taller (i.e. its point holds more of the probability mass) and the other one shorter, the entropy will decrease because of the increased certainty that the sample will fall on one point and not the other:

$-(0.2 \log_2 0.2 + 0.8\log_2 0.8)\approx 0.72$

Try changing the definition of the number and weighting of peaks, and see how the entropy varies.

If we split the probability mass among even more points, the entropy continues to increase. Let's derive the general form for $N$ points of equal mass, where $p_i=p=1/N$:

\begin{align}
 -\sum_i p_i \log_b p_i&= -\sum_i^N \frac{1}{N} \log_b \frac{1}{N}\\
                       &= -\log_b \frac{1}{N} \\
                       &= \log_b N
\end{align}
$$$$

If we have $N$ discrete points, the _uniform distribution_ (where all points have equal mass) is the distribution with the highest entropy: $\log_b N$. This upper bound on entropy is useful when considering binning strategies, as any estimate of entropy over $N$ discrete points (or bins) must be in the interval $[0, \log_b N]$.

```python
pmf = np.ones(n_bins) / n_bins  # [1/N] * N

pmf_ = np.insert(pmf, 0, pmf[0])
plt.plot(bins, pmf_, drawstyle="steps")
plt.fill_between(bins, pmf_, step="pre", alpha=0.4)
plt.xlabel("x")
plt.ylabel("p(x)")
plt.xlim(x_range);
plt.ylim(0, 1);
```

Here, there are 50 points and the entropy of the uniform distribution is $\log_2 50\approx 5.64$. If we construct _any_ discrete distribution $X$ over 50 points (or bins) and calculate an entropy of $H_2(X)>\log_2 50$, something must be wrong with our implementation of the discrete entropy computation.

# Section 2: Information, neurons, and spikes

```python
#@title Video 2: Entropy of different distributions
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='o6nyrx3KH20', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=o6nyrx3KH20

Recall the discussion of spike times and inter-spike intervals (ISIs) from Tutorial 1. What does the information content (or distributional entropy) of these measures say about our theory of nervous systems?

We'll consider three hypothetical neurons that all have the same mean ISI, but with different distributions:

1. Deterministic
2. Uniform
3. Exponential

Fixing the mean of the ISI distribution is equivalent to fixing its inverse: the neuron's mean firing rate. If a neuron has a fixed energy budget and each of its spikes has the same energy cost, then by fixing the mean firing rate, we are normalizing for energy expenditure. This provides a basis for comparing the entropy of different ISI distributions. In other words: if our neuron has a fixed budget, what ISI distribution should it express (all else being equal) to maximize the information content of its outputs?

Let's construct our three distributions and see how their entropies differ.

```python
n_bins = 50
mean_isi = 0.025
isi_range = (0, 0.25)

bins = np.linspace(*isi_range, n_bins + 1)
mean_idx = np.searchsorted(bins, mean_isi)

# 1. all mass concentrated on the ISI mean
pmf_single = np.zeros(n_bins)
pmf_single[mean_idx] = 1.0

# 2. mass uniformly distributed about the ISI mean
pmf_uniform = np.zeros(n_bins)
pmf_uniform[0:2*mean_idx] = 1 / (2 * mean_idx)

# 3. mass exponentially distributed about the ISI mean
pmf_exp = stats.expon.pdf(bins[1:], scale=mean_isi)
pmf_exp /= np.sum(pmf_exp)
```

```python
#@title
#@markdown Run this cell to plot the three PMFs
fig, axes = plt.subplots(ncols=3, figsize=(18, 5))

dists =  [# (subplot title, pmf, ylim)
          ("(1) Deterministic", pmf_single, (0, 1.05)),
          ("(1) Uniform", pmf_uniform, (0, 1.05)),
          ("(1) Exponential", pmf_exp, (0, 1.05))]

for ax, (label, pmf_, ylim) in zip(axes, dists):
  pmf_ = np.insert(pmf_, 0, pmf_[0])
  ax.plot(bins, pmf_, drawstyle="steps")
  ax.fill_between(bins, pmf_, step="pre", alpha=0.4)
  ax.set_title(label)
  ax.set_xlabel("Inter-spike interval (s)")
  ax.set_ylabel("Probability mass")
  ax.set_xlim(isi_range);
  ax.set_ylim(ylim);
```

```python
print(
  f"Deterministic: {entropy(pmf_single):.2f} bits",
  f"Uniform: {entropy(pmf_uniform):.2f} bits",
  f"Exponential: {entropy(pmf_exp):.2f} bits",
  sep="\n",
)
```

    Deterministic: 0.00 bits
    Uniform: 3.32 bits
    Exponential: 3.77 bits

The entropy here can be greater than the uniform because the exponential input range is unbounded

```python
#@title Video 3: Probabilities from histogram
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='e2U_-07O9jo', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=e2U_-07O9jo

In the previous example we created the PMFs by hand to illustrate idealized scenarios. How would we compute them from data recorded from actual neurons?

One way is to convert the ISI histograms we've previously computed into discrete probability distributions using the following equation:

\begin{align}
p_i = \frac{n_i}{\sum\nolimits_{i}n_i}
\end{align}

where $p_i$ is the probability of an ISI falling within a particular interval $i$ and $n_i$ is the count of how many ISIs were observed in that interval.

### Exercise 2: Probabilty Mass Function

Your second exercise is to implement a method that will produce a probability mass function from an array of ISI bin counts.

To verify your solution, we will compute the probability distribution of ISIs from real neural data taken from the Steinmetz dataset.

```python
neuron_idx = 283

isi = np.diff(steinmetz_spikes[neuron_idx])
bins = np.linspace(*isi_range, n_bins + 1)
counts, _ = np.histogram(isi, bins)
```

```python
def pmf_from_counts(counts):
  """Given counts, normalize by the total to estimate probabilities."""
  ###########################################################################
  # Exercise: Compute the PMF. Remove the next line to test your function
  #raise NotImplementedError("Student excercise: compute the PMF from ISI counts")
  ###########################################################################

  pmf = counts/np.sum(counts)

  return pmf

# Uncomment when ready to test your function
pmf = pmf_from_counts(counts)
plot_pmf(pmf,isi_range)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D1_ModelTypes/solutions/W1D1_Tutorial3_Solution_49231923.py)

*Example output:*

# Section 3: Calculate entropy from a PMF

```python
#@title Video 4: Calculating entropy from pmf
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='Xjy-jj-6Oz0', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=Xjy-jj-6Oz0

Now that we have the probability distribution for the actual neuron spiking activity, we can calculate its entropy.

```python
print(f"Entropy for Neuron {neuron_idx}: {entropy(pmf):.2f} bits")
```

    Entropy for Neuron 283: 3.36 bits

## Interactive Demo: Entropy of neurons

We can combine the above distribution plot and entropy calculation with an interactive widget to explore how the different neurons in the dataset vary in spiking activity and relative information. Note that the mean firing rate across neurons is not fixed, so some neurons with a uniform ISI distribution may have higher entropy than neurons with a more exponential distribution.

```python
#@title
#@markdown **Run the cell** to enable the sliders.

def _pmf_from_counts(counts):
  """Given counts, normalize by the total to estimate probabilities."""
  pmf = counts / np.sum(counts)
  return pmf

def _entropy(pmf):
  """Given a discrete distribution, return the Shannon entropy in bits."""
  # remove non-zero entries to avoid an error from log2(0)
  pmf = pmf[pmf > 0]
  h = -np.sum(pmf * np.log2(pmf))
  # absolute value applied to avoid getting a -0 result
  return np.abs(h)

@widgets.interact(neuron=widgets.IntSlider(0, min=0, max=(len(steinmetz_spikes)-1)))
def steinmetz_pmf(neuron):
  """ Given a neuron from the Steinmetz data, compute its PMF and entropy """
  isi = np.diff(steinmetz_spikes[neuron])
  bins = np.linspace(*isi_range, n_bins + 1)
  counts, _ = np.histogram(isi, bins)
  pmf = _pmf_from_counts(counts)

  plot_pmf(pmf,isi_range)
  plt.title(f"Neuron {neuron}: H = {_entropy(pmf):.2f} bits")

```

    interactive(children=(IntSlider(value=0, description='neuron', max=733), Output()), _dom_classes=('widget-inte…

---
# Summary

```python
#@title Video 5: Summary of model types
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='X4K2RR5qBK8', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=X4K2RR5qBK8

Congratulations! You've finished your first NMA tutorial. In this 3 part tutorial series, we used different types of models to understand the spiking behavior of neurons recorded in the Steinmetz data set.

 - We used "what" models to discover that the ISI distribution of real neurons is closest to an exponential distribution
 - We used "how" models to discover that balanced excitatory and inbhitiory inputs, coupled with a leaky membrane, can give rise to neuronal spiking with exhibiting such an exponential ISI distribution
 - We used "why" models to discover that exponential ISI distributions contain the most information when the mean spiking is constrained

# Bonus

### The foundations for Entropy

In his foundational [1948 paper](https://en.wikipedia.org/wiki/A_Mathematical_Theory_of_Communication) on information theory, Claude Shannon began with three criteria for a function $H$ defining the entropy of a discrete distribution of probability masses $p_i\in p(X)$ over the points $x_i\in X$:
1. $H$ should be continuous in the $p_i$.
  - That is, $H$ should change smoothly in response to smooth changes to the mass $p_i$ on each point $x_i$.
2. If all the points have equal shares of the probability mass, $p_i=1/N$, $H$ should be a non-decreasing function of $N$.
  - That is, if $X_N$ is the support with $N$ discrete points and $p(x\in X_N)$ assigns constant mass to each point, then $H(X_1) < H(X_2) < H(X_3) < \dots$
3. $H$ should be preserved by (invariant to) the equivalent (de)composition of distributions.
  - For example (from Shannon's paper) if we have a discrete distribution over three points with masses $(\frac{1}{2},\frac{1}{3},\frac{1}{6})$, then their entropy can be represented in terms of a direct choice between the three and calculated $H(\frac{1}{2},\frac{1}{3},\frac{1}{6})$. However, it could also be represented in terms of a series of two choices:
    1. either we sample the point with mass $1/2$ or not (_not_ is the other $1/2$, whose subdivisions are not given in the first choice),
    2. if (with probability $1/2$) we _don't_ sample the first point, we sample one of the two remaining points, masses $1/3$ and $1/6$.

    Thus in this case we require that $H(\frac{1}{2},\frac{1}{3},\frac{1}{6})=H(\frac{1}{2},\frac{1}{2}) + \frac{1}{2}H(\frac{1}{3}, \frac{1}{6})$

There is a unique function (up to a linear scaling factor) which satisfies these 3 requirements:

\begin{align}
  H_b(X) &= -\sum_{x\in X} p(x) \log_b p(x)
\end{align}

Where the base of the logarithm $b>1$ controls the units of entropy. The two most common cases are $b=2$ for units of _bits_, and $b=e$ for _nats_.

We can view this function as the expectation of the self-information over a distribution:

$$H_b(X) = \mathbb{E}_{x\in X} \left[I_b(x)\right]$$

$$I_b(x)=-\log_b p(x)$$

Self-information is just the negative logarithm of probability, and is a measure of how surprising an event sampled from the distribution would be. Events with $p(x)=1$ are certain to occur, and their self-information is zero (as is the entropy of the distribution they compose) meaning they are totally unsurprising. The smaller the probability of an event, the higher its self-information, and the more surprising the event would be to observe.


# Ipython notebook

Ipython est initialement un projet de console python spécialisé pour les applications scientifiques.
Avec le temps, le projet s'est élargi et inclus désormais un ensemble d'utilitaires pour les applications scientifiques, dont le notebook.

Ipython notebook est une feuille de calul interactive, inspirée de Maple et de Mathematica, qui fonctionne en mode web service. Au démarrage d'un notebook, Ipython lance un serveur d'application dans lequel tourne un kernel (noyau d'application) Ipython et le navigateur par défaut de votre OS sur l'adresse localhost 127.0.0.1.

Ce mode de fonctionnement client/serveur permet notamment de lancer des calculs en se connectant à un supercalculateur ou encore à un data center de type Amazon Web Service, Microsoft Azure ou Google App Engine.

Le notebook est un environnement de calcul interactif complémentaire avec Spyder.
En tant qu'IDE, Spyder est plus adapté au développement de fonctions et d'applications.
Le notebook est plus adapté à la réalisation d'étude ou au traitement de donnée.

## Fonctionnement de l'Ipython Notebook

Dans les versions >2.x de Ipython, le notebook s'utilise en basculant constemment entre 2 modes :

* mode commande des cellules:
    * Accessible en appuyant sur ESC
    * Permet de créer (A et B), supprimer (X), copier (C) et coller (V) des cellules
    * Changer de type de cellule : code (Y) ou markdown (M)

* mode édition
    * Accessible en appuyant sur ENTER
    * Executer le code de la cellule (SHIFT ENTER) ou afficher le rendu d'une cellule mardown

L'ensemble des commandes est accessible dans la barre d'outil (edit). L'ensemble des raccourcis est disponible dans le menu help de l'interface (help -> keyboard shortcut), ainsi qu'une visite guidée de l'interface (help -> user interface tour)

Pour plus d'éléments, se rérérer à la documentation officielle :

http://ipython.org/ipython-doc/2/notebook/notebook.html#structure-of-a-notebook-document

Une galerie de notebook est disponible en ligne via l'application web nbviewer qui permet la visualisation de fichiers .ipynb hébergés sur Github :
http://nbviewer.ipython.org/

# Numpy, Scipy et Matplotlib

Numpy, Scipy et Matplotlib sont les bibliothèques de calcul numérique les plus importantes de Python.

## Numpy

Inclus :

* l'ensemble des fonctions mathématiques
* les manipulations sur les vecteurs et les matrices
http://wiki.scipy.org/NumPy_for_Matlab_Users

Il comprend approximativement l'ensemble des fonctionalités de base de Matlab

```python
import numpy as np # np est une convention d'import
print np.array([0,1,2])
```

    [0 1 2]

Le numpy array est l'équivalent du vecteur matlab :

```python
x = np.arange(0,10)
print x
```

    [0 1 2 3 4 5 6 7 8 9]

```python
print x**2 # mise à la puissance 2. Toute les opérations sont par défaut élément par élément (elemnt wise)
print x+x
print x.dot(x) # produit scalaire
```

    [ 0  1  4  9 16 25 36 49 64 81]
    [ 0  2  4  6  8 10 12 14 16 18]
    285

```python
x2 = np.array([[0,1],[1,0]]) # matrice
print x2
```

    [[0 1]
     [1 0]]

```python
x3 = np.random.randint(low=0,high=10,size=[3,3])
u3 = np.array([2,3,4])
print x3
print x3.T # transposée
print x3*u3# multiplication de chaque ligne par le vecteur u3
print x3.dot(u3) # produit matricielle
```

    [[1 6 5]
     [7 3 8]
     [8 7 2]]
    [[1 7 8]
     [6 3 7]
     [5 8 2]]
    [[ 2 18 20]
     [14  9 32]
     [16 21  8]]
    [40 55 45]

De manière générale, les manipulations purement matricielles sont souvent plus lourdes en termes de syntaxe que Matlab®  (MATrix LABoratory) qui est spécialisé sur ce type d'objet. Cependant, les performances sont similaires.

Un benchmark sur ce type de manipulation réalisé par la NASA entre Matlab, Python, Java et Fortran est disponible en ligne : https://modelingguru.nasa.gov/docs/DOC-1762 .

En substance matlab et numpy ont des performances similaires, Fortran (avec compilateur intel) est toujours meilleur.

## Matplotlib

La bibliothèque graphique la plus populaire pour les graphes 2D (Mayavi pour la 3D).
La galerie en ligne est munie de nombreux exemple de codes :

http://matplotlib.org/gallery.html

Un tutorial de matplotlib : http://nbviewer.ipython.org/github/cmiller8/PythonforBuildingAnalysts/blob/master/0_PythonBaseLibraries/3_MatplotlibLibrary.ipynb

```python
import matplotlib.pylab as plt
import seaborn as sns
%matplotlib inline
```

La ligne précédente est une commande spécifique à Ipython. Elle ne fait pas partie du langage Python.
Les commandes commançant par **%** ou **%%** sont des **magic** qui configurent le comportement du notebook.
**%matplotlib inline**  indique que les graphes seront inclus en ligne dans le notebook (et sauvegardés avec lui).
Les graphes interactifs s'obtiennent avec la commande **%matplotlib qt **

```python
plt.plot([0,1,2,3],[0,2,1,2],'s-')
```

Création de 100 points de 0 à $2\pi$ dans le vecteur $x_4$, puis calcul de
$$y_4 = \sin(x4)$$

```python
x4 = np.linspace(0,5*np.pi,100)
y4 = np.sin(x4)

fig = plt.figure() # création d'une fenetre de graph
ax1 = fig.add_subplot(121) # graphe de gauche sur 2 graphes répartis horizontalement
ax2 = fig.add_subplot(122) # graphe de droite sur 2 graphes répartis horizontalement
ax1.plot(x4,y4)
ax2.plot(y4,x4)
ax1.set_title(r'$sin(x)$')
fig.savefig('test.png',dpi=150)
```

Le code suivant génère un échantillon de taille 1000 tiré dans la loi normal multivariée :

$$x_5 \sim  \mathcal{N}(\mu,\Sigma)$$
Avec $$\mu= \left(\begin{array}{cccc} 20 \\ 30 \\ 100\end{array} \right) $$

$$\Sigma= \left(\begin{array}{cccc} \\10&-5&0.3\\-5&10&0.2\\0.3&0.2&1000\end{array} \right)$$

Puis on trace uniquement les 2 premières composantes du vecteur dans un graphe de dispersion (*scatter*). On utilise la dernière composante pour la taille et la couleur des points.

```python
x5 = np.random.multivariate_normal(mean=[20,30,100],cov=[[10,-5,0.3],[-5,10,0.2],[0.3,0.2,1000]],size=1000)
plt.scatter(x5[:,0],x5[:,1],s=x5[:,2],c=x5[:,2],edgecolor='none',alpha=0.15,cmap='hot')
```

```python
res = plt.hist(x5.flatten(),bins=50)  # que fait la méthode flatten ?
```

## Scipy

Comprend l'équivalent des principales toolbox de Matlab.

```python
from scipy import linalg, optimize, sparse # par exemple : algèbre linéaire, optimisation et traitement de matrice crzeuse
from copy import copy
```

# TP : manipulation matricielle

http://en.wikibooks.org/wiki/LaTeX/Mathematics

### Exercice :
* En utilisant la fonction diag de numpy, écrire la fonction permettant de construire la matrice tridiagonale de taille $N_x$ suivante:

$$ A = \left( \begin{array}{cccc}
1+2Fo & -Fo & & 0\\
-Fo & 1+2Fo & \ddots & \\
\\
& \ddots & 1+2Fo & -Fo \\
0 & & -Fo & 1+2Fo \end{array} \right)$$

Où

$$Fo = \frac{\alpha \Delta t}{\Delta x^2}$$

Et le vecteur :

$$b^{i-1} = \left(\begin{array}{cccc} \theta^{i-1}_1+Fo\theta^{i-1}_{0} \\ \vdots \\ \theta^{i-1}_k\\ \vdots \\ \theta^{i-1}_{N_x}+Fo\theta^{i-1}_{N_x+1} \end{array} \right) $$

* En utilisant la fonction solve du package scipy.linalg, trouvez $T^i$ tel que :

$$ AT^i = b^{i-1} + \frac{\Delta t}{\rho C_p}S^{i-1}$$

Où $S$ est un terme source en $W.m^{-3}.$

** NB : Il s'agit de la résolution de l'équation de la chaleur en 1D en instationnaire sur un schema centré implicite.**

**$\theta^{i-1}_{0}$ et $\theta^{i-1}_{n+1}$ sont les conditions limites à gauche et à droites**

**Fo est le nombre de Fourier de maille. Sur ce schéma numérique implicite, la solution est inconditionnellement stable, mais la précision est dégradée pour Fo > 1.**

**Exercice : **

* Caluler l'évolution de la température pour $N_t$ itérations temporelles et $N_x$ mailles spatiales.
* Stocker le résultat dans une matrice, soit en affectant les valeurs dans une matrice définie à l'avance, soit en concatenant les vecteurs avec la fonction np.hstack
* Utilisez la commande **%%timeit** pour vérifier la performance de votre code

**Définition des paramètres physiques du système**

```python
alpha = 0.54e-6  # m2.s-1
rho   = 2.4e3    # kg.m-3
Cp    = 0.88e3   #J.kg-1.K-1

dt = 10*60. # s
dx = 0.01 # 1 m

L = 0.5 # 2 m
duration = 3*24*3600. # 3 jours

Nx = int(L/dx)
Nt = int(duration/dt)

Fo = alpha*dt/(dx**2) # Fourier de maille. Peremt d'apprecier la précision et la stabilité du schéma numérique
```

**Définition de la matrice $A$**

```python
A = np.diag(-Fo*np.ones(Nx-1),-1)+\
    np.diag(1+2*Fo*np.ones(Nx))+\
    np.diag(-Fo*np.ones(Nx-1),1)
```

```python
print A
```

    [[ 7.48 -3.24  0.   ...,  0.    0.    0.  ]
     [-3.24  7.48 -3.24 ...,  0.    0.    0.  ]
     [ 0.   -3.24  7.48 ...,  0.    0.    0.  ]
     ...,
     [ 0.    0.    0.   ...,  7.48 -3.24  0.  ]
     [ 0.    0.    0.   ..., -3.24  7.48 -3.24]
     [ 0.    0.    0.   ...,  0.   -3.24  7.48]]

**Définition du vecteur $b$**

```python
def rhs(Ti,Tg,Td,Fo):
    """
    rhs : right hand side of the equation

    Calcul le vecteur b tel que Ax = b

    Ti vecteur de solution à l'instant i
    Tg condition limite à gauche en température (scalaire)
    Td condition limite à droite en température (scalaire)
    Fo : Fourier de maille
    """
    b = copy(Ti)
    b[0]  += Fo*Tg
    b[-1] += Fo*Td
    return b
```

**Résolution du système pour $N_t$ pas de temps**

```python
#%%timeit
Ti = 15*np.ones([Nx,1])          # Condition initiale à 15°C
t = np.linspace(0,duration,Nt)   # vecteur temps
Tg = 10 + 2*np.sin(t/(24*3600)*2*np.pi)  # Condition limité à gauche sinusoidale centré autour de 10°C
TT = []                         # liste vide dans laquelle on va ajouter les profiles de température à chaque pas de temps
TT.append([Ti])

for t in range(Nt):
    b = rhs(Ti,Tg[t],19,Fo)
    Ti = np.linalg.solve(A,b)    # Résolution d'un système d'équation linéaire AX=b
    TT.append([Ti])

T =  np.vstack(TT)
T = np.reshape(T,[Nt+1,Nx])
```

### Exercice :

* Tracez de manière superposée le profil de température $\theta$ pour les 10 premiers pas de temps.

* Représentez les courbes d'isotempératures annotée en noire sur le champs de température représenté par la colormap "hot"

```python
with sns.color_palette("Blues_r",12): # définition du cycle des couleurs des courbes
    ax = plt.plot(T[0:10,:].T)
```

```python
X,Y  = np.meshgrid(np.arange(Nx),np.arange(Nt+1)) # Construction des matrices X et Y pour la fonction countour
```

```python
print X
print
print Y
```

    [[ 0  1  2 ..., 47 48 49]
     [ 0  1  2 ..., 47 48 49]
     [ 0  1  2 ..., 47 48 49]
     ...,
     [ 0  1  2 ..., 47 48 49]
     [ 0  1  2 ..., 47 48 49]
     [ 0  1  2 ..., 47 48 49]]

    [[  0   0   0 ...,   0   0   0]
     [  1   1   1 ...,   1   1   1]
     [  2   2   2 ...,   2   2   2]
     ...,
     [430 430 430 ..., 430 430 430]
     [431 431 431 ..., 431 431 431]
     [432 432 432 ..., 432 432 432]]

```python
ax = plt.imshow(T.T,cmap='RdBu_r',aspect='auto',interpolation='nearest') # création de la carte des températures

# Cration des contours
CS = plt.contour(Y,X,T,10,
                 colors='k',linewidth=1)   # negative contours will be dashed by default

plt.clabel(CS, fontsize=10, inline=0.5)    # tracer des labels

f = ax.get_figure()                        # récupération de l'objet figure qui "encapsule" l'objet ax

f.savefig('Chronogram_Temperature_1D.png',dpi=150) # Sauvegarde du graphe
```

### Sauvegarder vos résultats :

Sauvez la matrice T au format csv avec la fonction savetxt de numpy

```python
np.savetxt('temperature_1D_implicit.csv',T,delimiter=';')
```

## Bonus : Calcul symbolique avec Sympy

```python
from sympy import *
init_printing() # formatte les sorties avec Latex
```

```python
X = Symbol('X')
x, y, z, t, i = symbols('x y z t i')
k, m, n = symbols('k m n', integer=True)
f, g, h = symbols('f g h', cls=Function)
```

```python
expand((X+1)*(X+2))
```

```python
factor(x**2+3*x+2)
```

```python
(1/cos(x)).series(x, 0, 10) # développement en série limitée autour de 0 à l'ordre 10
```

```python
summation(1/2**i, (i, 0, oo)) # somme infinie de 1/2^n
```

```python
integrate(exp(-y**2)*erf(y), y) # intégration
```

```python
f = Function('f')
f(x).diff(x, x) + f(x)
```

```python
dsolve(f(x).diff(x, x) + f(x), f(x))   # résolution d'équations différentielles
```

```python
solve([x + 5*y - 2, -3*x + 6*y - 15], [x, y]) # résolution d'équation linéaire
```

    C:\Anaconda\lib\site-packages\IPython\core\formatters.py:239: FormatterWarning: Exception in image/png formatter:
    \begin{Bmatrix}x : -3, & y : 1\end{Bmatrix}
    ^
    Unknown symbol: \begin (at char 0), (line:1, col:1)
      FormatterWarning,

$$\begin{Bmatrix}x : -3, & y : 1\end{Bmatrix}$$

```python
import IPython
import sympy
from statsmodels import version as sm_version
print "Sympy version   \t:\t %s"%sympy.__version__
print "IPython version \t:\t %s"%IPython.__version__
print "numpy version   \t:\t %s"%np.__version__
print "statsmodels version \t:\t %s"%sm_version.full_version
print "matplotlib version \t:\t %s"%plt.__version__
```

    Sympy version   	:	 0.7.5
    IPython version 	:	 2.4.1
    numpy version   	:	 1.9.2
    statsmodels version 	:	 0.5.0
    matplotlib version 	:	 1.9.2


```python
%%HTML
<style>
.rendered_html table, .rendered_html th, .rendered_html tr, .rendered_html td {
     font-size: 100%;
}
</style>
```

<style>
.rendered_html table, .rendered_html th, .rendered_html tr, .rendered_html td {
     font-size: 100%;
}
</style>

# Metody Numeryczne

## Całkowanie numeryczne

### dr hab. inż. Jerzy Baranowski, Prof.AGH

## Całka oznaczona

$$
\int_a^b f(x) \mathrm{d}x
$$

Formalnie interesuje nas całka Riemanna, czyli granica ciągu sum Riemanna, niezależna od ciągu podziałów przedziału.

W praktyce możemy myśleć o całce względem jednej zmiennej jako polu pod krzywą.

Z tego powodu wzory na wyliczanie całek nazywamy kwadraturami.

## Zastosowania całkowania numerycznego

- Całkowanie w czasie rzeczywistym.
- Całkowanie funkcji jednej zmiennej.
- Całkowanie funkcji wielu zmiennych.

## Całkowanie w czasie rzeczywistym

Podstawowe zastosowanie całkowania w automatyce - regulacja PI, PID

Wraz z pojawieniem się nowego pomiaru wartość całki musi zostać zaktualizowana.

## Jak praktycznie to liczyć?

W zasadzie mamy dostępne trzy wzory - wzory prostokątów i wzór trapezów.

Dla ustalenia uwagi interesuje nas całka
$$
\int_{t_{k-1}}^{t_k} f(t) \mathrm{d}t
$$

## Wzór prostokątów w przód
Inaczej metoda Eulera w przód

$$
\int_{t_{k-1}}^{t_k} f(t) \mathrm{d}t\approx f(t_{k-1})(t_k-t_{k-1})
$$

dla pomiarów równoodległych (stały krok) $t_k=kh$
$$
\int_{t_{k-1}}^{t_k} f(t) \mathrm{d}t\approx f((k-1)h)h
$$

Co odpowiada transmitancji
$$
\frac{1}{s}\approx \frac{h}{z-1}
$$

## Wzór prostokątów w tył
Inaczej metoda różnic wstecznych

$$
\int_{t_{k-1}}^{t_k} f(t) \mathrm{d}t\approx f(t_{k})(t_k-t_{k-1})
$$

dla pomiarów równoodległych (stały krok) $t_k=kh$
$$
\int_{t_{k-1}}^{t_k} f(t) \mathrm{d}t\approx f((kh)h
$$

Co odpowiada transmitancji
$$
\frac{1}{s}\approx \frac{hz}{z-1}
$$

## Wzór trapezów
$$
\int_{t_{k-1}}^{t_k} f(t) \mathrm{d}t\approx \left(\frac{f(t_{k})+f(t_{k-1})}{2}\right)(t_k-t_{k-1})
$$

dla pomiarów równoodległych (stały krok) $t_k=kh$
$$
\int_{t_{k-1}}^{t_k} f(t) \mathrm{d}t\approx \frac{h}{2}(f(kh)+f((k-1)h))
$$

Co odpowiada transmitancji (tzw. aproksymacja Tustina)
$$
\frac{1}{s}\approx \frac{h}{2}\frac{z+1}{z-1}
$$

## Różnice
Metoda prostokątów w tył i trapezów są generalnie dobre. Jedna i druga metoda jest stabilna (o tym na następnym wykładzie). Metoda trapezów jest bardziej dokładna jej błąd:

$$
\text{error} \leq \frac{(h)^3}{12} M_2
$$
gdzie $M_2$ to maksimum modułu drugiej pochodnej $f$ na przedziale $(kh, (k-1)h)$, zaś błąd metody prostokątów w tył

$$
\text{error} \leq -\frac{(h)^2}{2} M_1
$$
gdzie $M_1$ to maksimum modułu pierwszej pochodnej $f$ na przedziale $(kh, (k-1)h)$.

## Liczenie całki z funkcji
Podobnie jak w przypadku wszystkich innych metod konieczne jest stworzenie modelu (np. wielomianu), z którego jest łatwiej policzyć całkę.

## Kwadratury interpolacyjne
Kwadratury interpolacyjne bazują na przybliżeniu funkcji wielomianem interpolacyjnym.

Wyróżniamy:

- Kwadratury Newtona-Cotesa (na węzłach równoodległych) (do 4 stopnia mają nazwy - wzór trapezów, wzór Simpsona, wzór Boole'a)
- Kwadratura Clenshawa-Curtisa (na węzłach Czebyszewa)
- Kwadratura Gaussa-Legendre'a (na węzłach Legendre'a)

## Postać kwadratury
W praktyce postać kwadratury to
$$
\int_{-1}^{1}f(x)\mathrm{d} x\approx\sum_{i=0}^n w_i f(x_i)
$$

gdzie $x_i$ to węzły kwadratury a $w_i$ to tzw. wagi kwadratury.

## Ocena jakości kwadratury

Dla kwadratury o $n$ węzłach mówimy, że jest rzędu wielomianowego $N$ jesżeli podaje dokładną wartość całki z wielomianów stopnia $N$ i mniejszych.

## Twierdzenie o rzędzie wielomianowym  kwadratur

Dla każdego $n\geq0$ kwadratura interpolacyjna stopnia $n+1$ jest dokładna dla wielomianów stopnia $n$, przy czym kwadratura Gaussa-Legendre'a jest dokładna dla wielomianów stopnia $2n+1$

**Dowód**

Dokładność dla wielomianów stopnia $n$ jest oczywista.
Dla ustalenia uwagi wielomiany Legendre'a to wielomiany ortogonalne z iloczynem skalnarnym
$$
f\circ g=\int_{-1}^{1}f(x)g(x)\mathrm{d} x
$$
węzły Legendre'a to pierwiastki tych wielomianów.

## Dowód cd..
Niech funkcja $f$ będzie wielomianem stopnia $2n+1$. Taką funkcję możńa zapisać w formie
$$f(x)=P_{n+1}(x)q_n(x)+r_n(x)$$ gdzie $P_{n+1}$ to wielomian Legendre'a stopnia $n+1$ a $q_n$ i $r_n$ są wielomianami stopnia $n$.

$$
I=\int_{-1}^{1}f(x)\mathrm{d} x=\int_{-1}^{1}P_{n+1}(x)q_n(x)\mathrm{d} x+\int_{-1}^{1}r_n(x)\mathrm{d} x
$$

Ponieważ wielomiany Legendre'a są ortogonalne, to pierwsza całka jest równa 0. więc
$$
I=\int_{-1}^{1}r_n(x)\mathrm{d} x
$$
Ponieważ węzły kwadratury są miejscami zerowymi $P_{n+1}$ to mamy, że $f(x_k)=r(x_k)$. Stąd wartość kwadratury interpolacyjnej dla $f$ i dla $r_n$ będą sobie równe. A ponieważ $r_n$ jest stopnia $n$ to wartość tej kwadratury jest dokłądnie równa wartości całki.

## Dodatkowe uwagi
- Kwadratura Gaussa-Legendra jest najdokładniejsza dla wielomianów, jednak w praktycznym całkowaniu jej przewaga nad kwadraturą Clenshawa-Curtisa (na węzłach Czebyszewa) nie jest taka duża.

- Dokładny wynik jest zbyt skomplikowany aby go tu omawiać, ale dla małych $n$ kwadratury zachowują się w praktyce tak samo, dopiero dla bardzo dużych $n$ kwadratura GL staje się dwa razy szybsza. Dla nieregularnych funkcji (nieanalitycznych) różnica ta może być niezauważalna w zakresie dokładności maszynowej.

## Kwadratura Clenshawa-Curtisa

Implementuje się ją przez FFT - kod w Matlabie:

```function I = clenshaw_curtis(f,n)
% (n+1)-pt C-C quadrature of f x = cos(pi*(0:n)'/n);
% Chebyshev points
fx = feval(f,x)/(2*n); % f evaluated at these points
g = real(fft(fx([1:n+1 n:-1:2]))); % fast Fourier transform
a = [g(1); g(2:n)+g(2*n:-1:n+2); g(n+1)]; % Chebyshev coeffs w = 0*a'; w(1:2:end) = 2./(1-(0:2:n).^2); % weight vector
I = w*a; % the integral```

W  Pythonie jest zaimplementowana jako ```scipy.integrate.quad```

## Kwadratura Gaussa-Legendra
Implementacja przez algorytm Goluba-Wlescha:
Pierwiastki wielomianu Legendre'a spełniają równanie
$$ J\tilde{P} = x\tilde{P} - p_n(x) \times \mathbf{e}_n$$
gdzie $\tilde{P} = \begin{bmatrix} p_0(x) & p_1(x) & \ldots & p_{n-1}(x) \end{bmatrix}^\mathsf{T}$, $\mathbf{e}_n = \begin{bmatrix} 0 & \ldots & 0  & 1 \end{bmatrix}^\mathsf{T}$ a $J$ jest macierzą Jacobiego
$$
\mathbf{J}=\begin{pmatrix}
     a_0 &      1 &      0 &  \ldots &  \ldots &  \ldots \\
     b_1 &    a_1 &      1 &       0 &  \ldots &  \ldots \\
       0 &    b_2 &    a_2 &       1 &       0 &  \ldots \\
       0 & \ldots & \ldots &  \ldots &  \ldots &       0 \\
  \ldots & \ldots &      0 & b_{n-2} & a_{n-2} &       1 \\
  \ldots & \ldots & \ldots &       0 & b_{n-1} & a_{n-1}
\end{pmatrix}
$$
Wartości własne $J$ są pierwiastkami wielomianów Legendre'a

## Wyliczanie wag kwadratur
Korzystamy z lekko zmodyfikowanej macierzy Jacobiego
$$ \begin{align}
                                            \mathcal{J}_{i,i} = J_{i,i} &= a_{i-1}        && i=1,\ldots,n \\
  \mathcal{J}_{i-1,i} = \mathcal{J}_{i,i-1} = \sqrt{J_{i,i-1}J_{i-1,i}} &= \sqrt{b_{i-1}} && i=2,\ldots,n.
\end{align}
$$
wtedy wagi kwadratury wynikają wprost z wektorów własnych macierzy.

## Kod w Matlabie
```
function I = gauss(f,n) % (n+1)-pt Gauss quadrature of f
beta = .5./sqrt(1-(2*(1:n)).^(-2)); % 3-term recurrence coeffs
T = diag(beta ,1) + diag(beta ,-1); % Jacobi matrix
[V,D] = eig(T); % eigenvalue decomposition
x = diag(D);
[x,i] = sort(x); % nodes (= Legendre points)
w = 2*V(1,i).^2; % weights
I = w*feval(f,x); % the integral
```

## Inne kwadratury
Generalnie kwadratura Gaussa-Legendrea jest najdokładniejsza.

Problem polega na tym, że jeżeli nie jesteśmy pewni jakiego rzędu użyć nie możemy wykorzystywać tych samych wartości wielokrotnie.

Potencjalnie problemem jest także to, że węzły Legendre'a nie zaczynają się na krańcach przedziału, więc przy podziale całkowania na podprzedziały nie ma zachowania ciągłości.

## Kwadratura Lobatto

- Modyfikacja kwadratury Gaussa-Legendre'a tak,  aby  pierwszy i ostatni węzeł były na brzegach przedziału.

- Dla $n$ węzłów jest dokładna dla wielomianów stopnia $2n-3$

## Kwadratura Gaussa-Konroda

- Opiera się na koncepcji tzw. zagnieżdżania.

- Pomysł jest taki, aby licząc kwadraturę Gaussa-Legendra wyliczyć dodatkowo wartość funkcji w pewnej liczbie dodatkowych węzłów przedziału. Tak aby za pomocą wszystkich wartości wyliczyć przybliżenie. W ten sposób dostajemy jednocześnie dwa przybliżenia i możemy oszacować błąd.

- Dla kwadratury GL rzędu $n$ (rząd przybliżenia $2n-1$) wylicza się dodatkowe $n+1$ węzłów uzyskując przybliżenie rzędu $3n +1$

## Kwadratury adaptacyjne

- W praktycznych obliczeniach, gdy nie wiemy jaki rząd kwadratury dobrać stosujemy kwadratury adaptacyjne.
- Jednocześnie na tym samym przedziale liczymy kwadratury dwóch różnych rzędów.
- Jeżeli różnica między wartościami jest mała to wynik wyższego rzędu uznajemy za rozwiązanie.
- Jeżeli różnica przekracza ustalony próg dzielimy przedział całkowania na podprzedziały (między policzonymi już węzłami) i w każdym z nich stosujemy te same dwie kwadratury.
- Proces powtarzamy do osiągnięcia tolerancji w każdym z podprzedziałów

## Implementacja kwadratur adaptacyjnych

- W Matlabie zaimplementowane są kwadratury oparte o kwadratury Lobatto niskich rzędów opracowane przez W. Gautshiego (funkcja ```quad```)
- W Pythonie wykorzystywane są kwadratury Gaussa-Konroda pochodzące z pakietu Fortrana QUADPACK (```scipy.integrate.quadrature```)

```python

```

Metody Numeryczne/Lecture 5 (integrals)/Lecture.ipynb