Probabilistic Programming and Bayesian Methods for Hackers
========

Welcome to *Bayesian Methods for Hackers*. The full Github repository is available at [github/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers](https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers). The other chapters can be found on the project's [homepage](https://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/). We hope you enjoy the book, and we encourage any contributions!

#### Looking for a printed version of Bayesian Methods for Hackers?

_Bayesian Methods for Hackers_ is now a published book by Addison-Wesley, available on [Amazon](http://www.amazon.com/Bayesian-Methods-Hackers-Probabilistic-Addison-Wesley/dp/0133902838)!

Chapter 1
======
***

The Philosophy of Bayesian Inference
------

> You are a skilled programmer, but bugs still slip into your code. After a particularly difficult implementation of an algorithm, you decide to test your code on a trivial example. It passes. You test the code on a harder problem. It passes once again. And it passes the next, *even more difficult*, test too! You are starting to believe that there may be no bugs in this code...

If you think this way, then congratulations, you already are thinking Bayesian! Bayesian inference is simply updating your beliefs after considering new evidence. A Bayesian can rarely be certain about a result, but he or she can be very confident. Just like in the example above, we can never be 100% sure that our code is bug-free unless we test it on every possible problem; something rarely possible in practice. Instead, we can test it on a large number of problems, and if it succeeds we can feel more *confident* about our code, but still not certain.  Bayesian inference works identically: we update our beliefs about an outcome; rarely can we be absolutely sure unless we rule out all other alternatives.

### The Bayesian state of mind

Bayesian inference differs from more traditional statistical inference by preserving *uncertainty*. At first, this sounds like a bad statistical technique. Isn't statistics all about deriving *certainty* from randomness? To reconcile this, we need to start thinking like Bayesians.

The Bayesian world-view interprets probability as measure of *believability in an event*, that is, how confident we are in an event occurring. In fact, we will see in a moment that this is the natural interpretation of probability.

For this to be clearer, we consider an alternative interpretation of probability: *Frequentist*, known as the more *classical* version of statistics, assumes that probability is the long-run frequency of events (hence the bestowed title). For example, the *probability of plane accidents* under a frequentist philosophy is interpreted as the *long-term frequency of plane accidents*. This makes logical sense for many probabilities of events, but becomes more difficult to understand when events have no long-term frequency of occurrences. Consider: we often assign probabilities to outcomes of presidential elections, but the election itself only happens once! Frequentists get around this by invoking alternative realities and saying across all these realities, the frequency of occurrences defines the probability.

Bayesians, on the other hand, have a more intuitive approach. Bayesians interpret a probability as measure of *belief*, or confidence, of an event occurring. Simply, a probability is a summary of an opinion. An individual who assigns a belief of 0 to an event has no confidence that the event will occur; conversely, assigning a belief of 1 implies that the individual is absolutely certain of an event occurring. Beliefs between 0 and 1 allow for weightings of other outcomes. This definition agrees with the probability of a plane accident example, for having observed the frequency of plane accidents, an individual's belief should be equal to that frequency, excluding any outside information. Similarly, under this definition of probability being equal to beliefs, it is meaningful to speak about probabilities (beliefs) of presidential election outcomes: how confident are you candidate *A* will win?

Notice in the paragraph above, I assigned the belief (probability) measure to an *individual*, not to Nature. This is very interesting, as this definition leaves room for conflicting beliefs between individuals. Again, this is appropriate for what naturally occurs: different individuals have different beliefs of events occurring, because they possess different *information* about the world. The existence of different beliefs does not imply that anyone is wrong. Consider the following examples demonstrating the relationship between individual beliefs and probabilities:

- I flip a coin, and we both guess the result. We would both agree, assuming the coin is fair, that the probability of Heads is 1/2. Assume, then, that I peek at the coin. Now I know for certain what the result is: I assign probability 1.0 to either Heads or Tails (whichever it is). Now what is *your* belief that the coin is Heads? My knowledge of the outcome has not changed the coin's results. Thus we assign different probabilities to the result.

-  Your code either has a bug in it or not, but we do not know for certain which is true, though we have a belief about the presence or absence of a bug.

-  A medical patient is exhibiting symptoms $x$, $y$ and $z$. There are a number of diseases that could be causing all of them, but only a single disease is present. A doctor has beliefs about which disease, but a second doctor may have slightly different beliefs.

This philosophy of treating beliefs as probability is natural to humans. We employ it constantly as we interact with the world and only see partial truths, but gather evidence to form beliefs. Alternatively, you have to be *trained* to think like a frequentist.

To align ourselves with traditional probability notation, we denote our belief about event $A$ as $P(A)$. We call this quantity the *prior probability*.

John Maynard Keynes, a great economist and thinker, said "When the facts change, I change my mind. What do you do, sir?" This quote reflects the way a Bayesian updates his or her beliefs after seeing evidence. Even — especially — if the evidence is counter to what was initially believed, the evidence cannot be ignored. We denote our updated belief as $P(A |X )$, interpreted as the probability of $A$ given the evidence $X$. We call the updated belief the *posterior probability* so as to contrast it with the prior probability. For example, consider the posterior probabilities (read: posterior beliefs) of the above examples, after observing some evidence $X$:

1\. $P(A): \;\;$ the coin has a 50 percent chance of being Heads. $P(A | X):\;\;$ You look at the coin, observe a Heads has landed, denote this information $X$, and trivially assign probability 1.0 to Heads and 0.0 to Tails.

2\.   $P(A): \;\;$  This big, complex code likely has a bug in it. $P(A | X): \;\;$ The code passed all $X$ tests; there still might be a bug, but its presence is less likely now.

3\.  $P(A):\;\;$ The patient could have any number of diseases. $P(A | X):\;\;$ Performing a blood test generated evidence $X$, ruling out some of the possible diseases from consideration.

It's clear that in each example we did not completely discard the prior belief after seeing new evidence $X$, but we *re-weighted the prior* to incorporate the new evidence (i.e. we put more weight, or confidence, on some beliefs versus others).

By introducing prior uncertainty about events, we are already admitting that any guess we make is potentially very wrong. After observing data, evidence, or other information, we update our beliefs, and our guess becomes *less wrong*. This is the alternative side of the prediction coin, where typically we try to be *more right*.

### Bayesian Inference in Practice

 If frequentist and Bayesian inference were programming functions, with inputs being statistical problems, then the two would be different in what they return to the user. The frequentist inference function would return a number, representing an estimate (typically a summary statistic like the sample average etc.), whereas the Bayesian function would return *probabilities*.

For example, in our debugging problem above, calling the frequentist function with the argument "My code passed all $X$ tests; is my code bug-free?" would return a *YES*. On the other hand, asking our Bayesian function "Often my code has bugs. My code passed all $X$ tests; is my code bug-free?" would return something very different: probabilities of *YES* and *NO*. The function might return:

>    *YES*, with probability 0.8; *NO*, with probability 0.2

This is very different from the answer the frequentist function returned. Notice that the Bayesian function accepted an additional argument:  *"Often my code has bugs"*. This parameter is the *prior*. By including the prior parameter, we are telling the Bayesian function to include our belief about the situation. Technically this parameter in the Bayesian function is optional, but we will see excluding it has its own consequences.

#### Incorporating evidence

As we acquire more and more instances of evidence, our prior belief is *washed out* by the new evidence. This is to be expected. For example, if your prior belief is something ridiculous, like "I expect the sun to explode today", and each day you are proved wrong, you would hope that any inference would correct you, or at least align your beliefs better. Bayesian inference will correct this belief.

Denote $N$ as the number of instances of evidence we possess. As we gather an *infinite* amount of evidence, say as $N \rightarrow \infty$, our Bayesian results (often) align with frequentist results. Hence for large $N$, statistical inference is more or less objective. On the other hand, for small $N$, inference is much more *unstable*: frequentist estimates have more variance and larger confidence intervals. This is where Bayesian analysis excels. By introducing a prior, and returning probabilities (instead of a scalar estimate), we *preserve the uncertainty* that reflects the instability of statistical inference of a small $N$ dataset.

One may think that for large $N$, one can be indifferent between the two techniques since they offer similar inference, and might lean towards the computationally-simpler, frequentist methods. An individual in this position should consider the following quote by Andrew Gelman (2005)[1], before making such a decision:

> Sample sizes are never large. If $N$ is too small to get a sufficiently-precise estimate, you need to get more data (or make more assumptions). But once $N$ is "large enough," you can start subdividing the data to learn more (for example, in a public opinion poll, once you have a good estimate for the entire country, you can estimate among men and women, northerners and southerners, different age groups, etc.). $N$ is never enough because if it were "enough" you'd already be on to the next problem for which you need more data.

### Are frequentist methods incorrect then?

**No.**

Frequentist methods are still useful or state-of-the-art in many areas. Tools such as least squares linear regression, LASSO regression, and expectation-maximization algorithms are all powerful and fast. Bayesian methods complement these techniques by solving problems that these approaches cannot, or by illuminating the underlying system with more flexible modeling.

#### A note on *Big Data*
Paradoxically, big data's predictive analytic problems are actually solved by relatively simple algorithms [2][4]. Thus we can argue that big data's prediction difficulty does not lie in the algorithm used, but instead on the computational difficulties of storage and execution on big data. (One should also consider Gelman's quote from above and ask "Do I really have big data?" )

The much more difficult analytic problems involve *medium data* and, especially troublesome, *really small data*. Using a similar argument as  Gelman's above, if big data problems are *big enough* to be readily solved, then we should be more interested in the *not-quite-big enough* datasets.

### Our Bayesian framework

We are interested in beliefs, which can be interpreted as probabilities by thinking Bayesian. We have a *prior* belief in event $A$, beliefs formed by previous information, e.g., our prior belief about bugs being in our code before performing tests.

Secondly, we observe our evidence. To continue our buggy-code example: if our code passes $X$ tests, we want to update our belief to incorporate this. We call this new belief the *posterior* probability. Updating our belief is done via the following equation, known as Bayes' Theorem, after its discoverer Thomas Bayes:

\begin{align}
 P( A | X ) = & \frac{ P(X | A) P(A) } {P(X) } \\\\[5pt]
& \propto P(X | A) P(A)\;\; (\propto \text{is proportional to } )
\end{align}

The above formula is not unique to Bayesian inference: it is a mathematical fact with uses outside Bayesian inference. Bayesian inference merely uses it to connect prior probabilities $P(A)$ with an updated posterior probabilities $P(A | X )$.

##### Example: Mandatory coin-flip example

Every statistics text must contain a coin-flipping example, I'll use it here to get it out of the way. Suppose, naively, that you are unsure about the probability of heads in a coin flip (spoiler alert: it's 50%). You believe there is some true underlying ratio, call it $p$, but have no prior opinion on what $p$ might be.

We begin to flip a coin, and record the observations: either $H$ or $T$. This is our observed data. An interesting question to ask is how our inference changes as we observe more and more data? More specifically, what do our posterior probabilities look like when we have little data, versus when we have lots of data.

Below we plot a sequence of updating posterior probabilities as we observe increasing amounts of data (coin flips).

```python
"""
The book uses a custom matplotlibrc file, which provides the unique styles for
matplotlib plots. If executing this book, and you wish to use the book's
styling, provided are two options:
    1. Overwrite your own matplotlibrc file with the rc-file provided in the
       book's styles/ dir. See http://matplotlib.org/users/customizing.html
    2. Also in the styles is  bmh_matplotlibrc.json file. This can be used to
       update the styles in only this notebook. Try running the following code:

        import json, matplotlib
        s = json.load( open("../styles/bmh_matplotlibrc.json") )
        matplotlib.rcParams.update(s)

"""

# The code below can be passed over, as it is currently not important, plus it
# uses advanced topics we have not covered yet. LOOK AT PICTURE, MICHAEL!
%matplotlib inline
from IPython.core.pylabtools import figsize
import numpy as np
from matplotlib import pyplot as plt
figsize(11, 9)

import scipy.stats as stats

dist = stats.beta
n_trials = [0, 1, 2, 3, 4, 5, 8, 15, 50, 1000]
data = stats.bernoulli.rvs(0.5, size=n_trials[-1])
x = np.linspace(0, 1, 100)

# For the already prepared, I'm using Binomial's conj. prior.
for k, N in enumerate(n_trials):
    sx = plt.subplot(len(n_trials) / 2, 2, k + 1)
    plt.xlabel("$p$, probability of heads") \
        if k in [0, len(n_trials) - 1] else None
    plt.setp(sx.get_yticklabels(), visible=False)
    heads = data[:N].sum()
    y = dist.pdf(x, 1 + heads, 1 + N - heads)
    plt.plot(x, y, label="observe %d tosses,\n %d heads" % (N, heads))
    plt.fill_between(x, 0, y, color="#348ABD", alpha=0.4)
    plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)

    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight=True)

plt.suptitle("Bayesian updating of posterior probabilities",
             y=1.02,
             fontsize=14)

plt.tight_layout()
```

The posterior probabilities are represented by the curves, and our uncertainty is proportional to the width of the curve. As the plot above shows, as we start to observe data our posterior probabilities start to shift and move around. Eventually, as we observe more and more data (coin-flips), our probabilities will tighten closer and closer around the true value of $p=0.5$ (marked by a dashed line).

Notice that the plots are not always *peaked* at 0.5. There is no reason it should be: recall we assumed we did not have a prior opinion of what $p$ is. In fact, if we observe quite extreme data, say 8 flips and only 1 observed heads, our distribution would look very biased *away* from lumping around 0.5 (with no prior opinion, how confident would you feel betting on a fair coin after observing 8 tails and 1 head). As more data accumulates, we would see more and more probability being assigned at $p=0.5$, though never all of it.

The next example is a simple demonstration of the mathematics of Bayesian inference.

##### Example: Bug, or just sweet, unintended feature?

Let $A$ denote the event that our code has **no bugs** in it. Let $X$ denote the event that the code passes all debugging tests. For now, we will leave the prior probability of no bugs as a variable, i.e. $P(A) = p$.

We are interested in $P(A|X)$, i.e. the probability of no bugs, given our debugging tests $X$ pass. To use the formula above, we need to compute some quantities.

What is $P(X | A)$, i.e., the probability that the code passes $X$ tests *given* there are no bugs? Well, it is equal to 1, for code with no bugs will pass all tests.

$P(X)$ is a little bit trickier: The event $X$ can be divided into two possibilities, event $X$ occurring even though our code *indeed has* bugs (denoted $\sim A\;$, spoken *not $A$*), or event $X$ without bugs ($A$). $P(X)$ can be represented as:

\begin{align}
P(X ) & = P(X \text{ and } A) + P(X \text{ and } \sim A) \\\\[5pt]
 & = P(X|A)P(A) + P(X | \sim A)P(\sim A)\\\\[5pt]
& = P(X|A)p + P(X | \sim A)(1-p)
\end{align}

We have already computed $P(X|A)$ above. On the other hand, $P(X | \sim A)$ is subjective: our code can pass tests but still have a bug in it, though the probability there is a bug present is reduced. Note this is dependent on the number of tests performed, the degree of complication in the tests, etc. Let's be conservative and assign $P(X|\sim A) = 0.5$. Then

\begin{align}
P(A | X) & = \frac{1\cdot p}{ 1\cdot p +0.5 (1-p) } \\\\
& = \frac{ 2 p}{1+p}
\end{align}
This is the posterior probability. What does it look like as a function of our prior, $p \in [0,1]$?

```python
figsize(12.5, 4)
p = np.linspace(0, 1, 50)
plt.plot(p, 2 * p / (1 + p), color="#348ABD", lw=3)
# plt.fill_between(p, 2*p/(1+p), alpha=.5, facecolor=["#A60628"])
plt.scatter(0.2, 2 * (0.2) / 1.2, s=140, c="#348ABD")
plt.xlim(0, 1)
plt.ylim(0, 1)
plt.xlabel("Prior, $P(A) = p$")
plt.ylabel("Posterior, $P(A|X)$, with $P(A) = p$")
plt.title("Is my code bug-free?")
```

We can see the biggest gains if we observe the $X$ tests passed when the prior probability, $p$, is low. Let's settle on a specific value for the prior. I'm a strong programmer (I think), so I'm going to give myself a realistic prior of 0.20, that is, there is a 20% chance that I write code bug-free. To be more realistic, this prior should be a function of how complicated and large the code is, but let's pin it at 0.20. Then my updated belief that my code is bug-free is 0.33.

Recall that the prior is a probability: $p$ is the prior probability that there *are no bugs*, so $1-p$ is the prior probability that there *are bugs*.

Similarly, our posterior is also a probability, with $P(A | X)$ the probability there is no bug *given we saw all tests pass*, hence $1-P(A|X)$ is the probability there is a bug *given all tests passed*. What does our posterior probability look like? Below is a chart of both the prior and the posterior probabilities.

```python
figsize(12.5, 4)
colours = ["#348ABD", "#A60628"]

prior = [0.20, 0.80]
posterior = [1. / 3, 2. / 3]
plt.bar([0, .7], prior, alpha=0.70, width=0.25,
        color=colours[0], label="prior distribution",
        lw="3", edgecolor=colours[0])

plt.bar([0 + 0.25, .7 + 0.25], posterior, alpha=0.7,
        width=0.25, color=colours[1],
        label="posterior distribution",
        lw="3", edgecolor=colours[1])

plt.ylim(0,1)
plt.xticks([0.20, .95], ["Bugs Absent", "Bugs Present"])
plt.title("Prior and Posterior probability of bugs present")
plt.ylabel("Probability")
plt.legend(loc="upper left");
```

Notice that after we observed $X$ occur, the probability of bugs being absent increased. By increasing the number of tests, we can approach confidence (probability 1) that there are no bugs present.

This was a very simple example of Bayesian inference and Bayes rule. Unfortunately, the mathematics necessary to perform more complicated Bayesian inference only becomes more difficult, except for artificially constructed cases. We will later see that this type of mathematical analysis is actually unnecessary. First we must broaden our modeling tools. The next section deals with *probability distributions*. If you are already familiar, feel free to skip (or at least skim), but for the less familiar the next section is essential.

_______

## Probability Distributions

**Let's quickly recall what a probability distribution is:** Let $Z$ be some random variable. Then associated with $Z$ is a *probability distribution function* that assigns probabilities to the different outcomes $Z$ can take. Graphically, a probability distribution is a curve where the probability of an outcome is proportional to the height of the curve. You can see examples in the first figure of this chapter.

We can divide random variables into three classifications:

-   **$Z$ is discrete**: Discrete random variables may only assume values on a specified list. Things like populations, movie ratings, and number of votes are all discrete random variables. Discrete random variables become more clear when we contrast them with...

-   **$Z$ is continuous**: Continuous random variable can take on arbitrarily exact values. For example, temperature, speed, time, color are all modeled as continuous variables because you can progressively make the values more and more precise.

- **$Z$ is mixed**: Mixed random variables assign probabilities to both discrete and continuous random variables, i.e. it is a combination of the above two categories.

#### Expected Value
Expected value (EV) is one of the most important concepts in probability. The EV for a given probability distribution can be described as "the mean value in the long run for many repeated samples from that distribution." To borrow a metaphor from physics, a distribution's EV acts like its "center of mass." Imagine repeating the same experiment many times over, and taking the average over each outcome. The more you repeat the experiment, the closer this average will become to the distributions EV. (side note: as the number of repeated experiments goes to infinity, the difference between the average outcome and the EV becomes arbitrarily small.)

### Discrete Case
If $Z$ is discrete, then its distribution is called a *probability mass function*, which measures the probability $Z$ takes on the value $k$, denoted $P(Z=k)$. Note that the probability mass function completely describes the random variable $Z$, that is, if we know the mass function, we know how $Z$ should behave. There are popular probability mass functions that consistently appear: we will introduce them as needed, but let's introduce the first very useful probability mass function. We say $Z$ is *Poisson*-distributed if:

$$P(Z = k) =\frac{ \lambda^k e^{-\lambda} }{k!}, \; \; k=0,1,2, \dots, \; \; \lambda \in \mathbb{R}_{>0} $$

$\lambda$ is called a parameter of the distribution, and it controls the distribution's shape. For the Poisson distribution, $\lambda$ can be any positive number. By increasing $\lambda$, we add more probability to larger values, and conversely by decreasing $\lambda$ we add more probability to smaller values. One can describe $\lambda$ as the *intensity* of the Poisson distribution.

Unlike $\lambda$, which can be any positive number, the value $k$ in the above formula must be a non-negative integer, i.e., $k$ must take on values 0,1,2, and so on. This is very important, because if you wanted to model a population you could not make sense of populations with 4.25 or 5.612 members.

If a random variable $Z$ has a Poisson mass distribution, we denote this by writing

$$Z \sim \text{Poi}(\lambda) $$

One useful property of the Poisson distribution is that its expected value is equal to its parameter, i.e.:

$$E\large[ \;Z\; | \; \lambda \;\large] = \lambda $$

We will use this property often, so it's useful to remember. Below, we plot the probability mass distribution for different $\lambda$ values. The first thing to notice is that by increasing $\lambda$, we add more probability of larger values occurring. Second, notice that although the graph ends at 15, the distributions do not. They assign positive probability to every non-negative integer.

```python
figsize(12.5, 4)

import scipy.stats as stats
a = np.arange(16)
poi = stats.poisson
lambda_ = [0.1, 3, 6, 7]
colours = ["#348ABD", "#A60628", "#A01212", "#B15555"]
for i,j in zip(lambda_, colours):
        plt.bar(a, poi.pmf(a, i), color=j,
                label="$\lambda = %.1f$" % i, alpha=0.60,
                edgecolor=j, lw="3")

plt.xticks(a + 0.4, a)
plt.legend()
plt.ylabel("probability of $k$")
plt.xlabel("$k$")
plt.title("Probability mass function of a Poisson random variable; differing \
$\lambda$ values")
```

### Continuous Case
Instead of a probability mass function, a continuous random variable has a *probability density function*. This might seem like unnecessary nomenclature, but the density function and the mass function are very different creatures. An example of continuous random variable is a random variable with *exponential density*. The density function for an exponential random variable looks like this:

$$f_Z(z | \lambda) = \lambda e^{-\lambda z }, \;\; z\ge 0$$

Like a Poisson random variable, an exponential random variable can take on only non-negative values. But unlike a Poisson variable, the exponential can take on *any* non-negative values, including non-integral values such as 4.25 or 5.612401. This property makes it a poor choice for count data, which must be an integer, but a great choice for time data, temperature data (measured in Kelvins, of course), or any other precise *and positive* variable. The graph below shows two probability density functions with different $\lambda$ values.

When a random variable $Z$ has an exponential distribution with parameter $\lambda$, we say *$Z$ is exponential* and write

$$Z \sim \text{Exp}(\lambda)$$

Given a specific $\lambda$, the expected value of an exponential random variable is equal to the inverse of $\lambda$, that is:

$$E[\; Z \;|\; \lambda \;] = \frac{1}{\lambda}$$

```python
a = np.linspace(0, 4, 100)
expo = stats.expon
lambda_ = [0.5, 1, 3]

for l, c in zip(lambda_, colours):
    plt.plot(a, expo.pdf(a, scale=1. / l), lw=3,
             color=c, label="$\lambda = %.1f$" % l)
    plt.fill_between(a, expo.pdf(a, scale=1. / l), color=c, alpha=.33)

plt.legend()
plt.ylabel("PDF at $z$")
plt.xlabel("$z$")
plt.ylim(0, 1.2)
plt.title("Probability density function of an Exponential random variable;\
 differing $\lambda$");
```

### But what is $\lambda \;$?

**This question is what motivates statistics**. In the real world, $\lambda$ is hidden from us. We see only $Z$, and must go backwards to try and determine $\lambda$. The problem is difficult because there is no one-to-one mapping from $Z$ to $\lambda$. Many different methods have been created to solve the problem of estimating $\lambda$, but since $\lambda$ is never actually observed, no one can say for certain which method is best!

Bayesian inference is concerned with *beliefs* about what $\lambda$ might be. Rather than try to guess $\lambda$ exactly, we can only talk about what $\lambda$ is likely to be by assigning a probability distribution to $\lambda$.

This might seem odd at first. After all, $\lambda$ is fixed; it is not (necessarily) random! How can we assign probabilities to values of a non-random variable? Ah, we have fallen for our old, frequentist way of thinking. Recall that under Bayesian philosophy, we *can* assign probabilities if we interpret them as beliefs. And it is entirely acceptable to have *beliefs* about the parameter $\lambda$.

##### Example: Inferring behaviour from text-message data

Let's try to model a more interesting example, one that concerns the rate at which a user sends and receives text messages:

>  You are given a series of daily text-message counts from a user of your system. The data, plotted over time, appears in the chart below. You are curious to know if the user's text-messaging habits have changed over time, either gradually or suddenly. How can you model this? (This is in fact my own text-message data. Judge my popularity as you wish.)

```python
figsize(12.5, 3.5)
count_data = np.loadtxt("data/txtdata.csv")
n_count_data = len(count_data)
plt.bar(np.arange(n_count_data), count_data, color="#348ABD")
plt.xlabel("Time (days)")
plt.ylabel("count of text-msgs received")
plt.title("Did the user's texting habits change over time?")
plt.xlim(0, n_count_data);
```

Before we start modeling, see what you can figure out just by looking at the chart above. Would you say there was a change in behaviour during this time period?

How can we start to model this? Well, as we have conveniently already seen, a Poisson random variable is a very appropriate model for this type of *count* data. Denoting day $i$'s text-message count by $C_i$,

$$ C_i \sim \text{Poisson}(\lambda)  $$

We are not sure what the value of the $\lambda$ parameter really is, however. Looking at the chart above, it appears that the rate might become higher late in the observation period, which is equivalent to saying that $\lambda$ increases at some point during the observations. (Recall that a higher value of $\lambda$ assigns more probability to larger outcomes. That is, there is a higher probability of many text messages having been sent on a given day.)

How can we represent this observation mathematically? Let's assume that on some day during the observation period (call it $\tau$), the parameter $\lambda$ suddenly jumps to a higher value. So we really have two $\lambda$ parameters: one for the period before $\tau$, and one for the rest of the observation period. In the literature, a sudden transition like this would be called a *switchpoint*:

$$
\lambda =
\begin{cases}
\lambda_1  & \text{if } t \lt \tau \cr
\lambda_2 & \text{if } t \ge \tau
\end{cases}
$$

If, in reality, no sudden change occurred and indeed $\lambda_1 = \lambda_2$, then the $\lambda$s posterior distributions should look about equal.

We are interested in inferring the unknown $\lambda$s. To use Bayesian inference, we need to assign prior probabilities to the different possible values of $\lambda$. What would be good prior probability distributions for $\lambda_1$ and $\lambda_2$? Recall that $\lambda$ can be any positive number. As we saw earlier, the *exponential* distribution provides a continuous density function for positive numbers, so it might be a good choice for modeling $\lambda_i$. But recall that the exponential distribution takes a parameter of its own, so we'll need to include that parameter in our model. Let's call that parameter $\alpha$.

\begin{align}
&\lambda_1 \sim \text{Exp}( \alpha ) \\\
&\lambda_2 \sim \text{Exp}( \alpha )
\end{align}

$\alpha$ is called a *hyper-parameter* or *parent variable*. In literal terms, it is a parameter that influences other parameters. Our initial guess at $\alpha$ does not influence the model too strongly, so we have some flexibility in our choice.  A good rule of thumb is to set the exponential parameter equal to the inverse of the average of the count data. Since we're modeling $\lambda$ using an exponential distribution, we can use the expected value identity shown earlier to get:

$$\frac{1}{N}\sum_{i=0}^N \;C_i \approx E[\; \lambda \; |\; \alpha ] = \frac{1}{\alpha}$$

An alternative, and something I encourage the reader to try, would be to have two priors: one for each $\lambda_i$. Creating two exponential distributions with different $\alpha$ values reflects our prior belief that the rate changed at some point during the observations.

What about $\tau$? Because of the noisiness of the data, it's difficult to pick out a priori when $\tau$ might have occurred. Instead, we can assign a *uniform prior belief* to every possible day. This is equivalent to saying

\begin{align}
& \tau \sim \text{DiscreteUniform(1,70) }\\\\
& \Rightarrow P( \tau = k ) = \frac{1}{70}
\end{align}

So after all this, what does our overall prior distribution for the unknown variables look like? Frankly, *it doesn't matter*. What we should understand is that it's an ugly, complicated mess involving symbols only a mathematician could love. And things will only get uglier the more complicated our models become. Regardless, all we really care about is the posterior distribution.

We next turn to PyMC, a Python library for performing Bayesian analysis that is undaunted by the mathematical monster we have created.

Introducing our first hammer: PyMC
-----

PyMC is a Python library for programming Bayesian analysis [3]. It is a fast, well-maintained library. The only unfortunate part is that its documentation is lacking in certain areas, especially those that bridge the gap between beginner and hacker. One of this book's main goals is to solve that problem, and also to demonstrate why PyMC is so cool.

We will model the problem above using PyMC. This type of programming is called *probabilistic programming*, an unfortunate misnomer that invokes ideas of randomly-generated code and has likely confused and frightened users away from this field. The code is not random; it is probabilistic in the sense that we create probability models using programming variables as the model's components. Model components are first-class primitives within the PyMC framework.

B. Cronin [5] has a very motivating description of probabilistic programming:

>   Another way of thinking about this: unlike a traditional program, which only runs in the forward directions, a probabilistic program is run in both the forward and backward direction. It runs forward to compute the consequences of the assumptions it contains about the world (i.e., the model space it represents), but it also runs backward from the data to constrain the possible explanations. In practice, many probabilistic programming systems will cleverly interleave these forward and backward operations to efficiently home in on the best explanations.

Because of the confusion engendered by the term *probabilistic programming*, I'll refrain from using it. Instead, I'll simply say *programming*, since that's what it really is.

PyMC code is easy to read. The only novel thing should be the syntax, and I will interrupt the code to explain individual sections. Simply remember that we are representing the model's components ($\tau, \lambda_1, \lambda_2$ ) as variables:

```python
import pymc as pm

alpha = 1.0 / count_data.mean()  # Recall count_data is the
                               # variable that holds our txt counts
lambda_1 = pm.Exponential("lambda_1", alpha)
lambda_2 = pm.Exponential("lambda_2", alpha)

tau = pm.DiscreteUniform("tau", lower=0, upper=n_count_data)
```

In the code above, we create the PyMC variables corresponding to $\lambda_1$ and $\lambda_2$. We assign them to PyMC's *stochastic variables*, so-called because they are treated by the back end as random number generators. We can demonstrate this fact by calling their built-in `random()` methods.

```python
print("Random output:", tau.random(), tau.random(), tau.random())
```

    Random output: 52 12 49

```python
@pm.deterministic
def lambda_(tau=tau, lambda_1=lambda_1, lambda_2=lambda_2):
    out = np.zeros(n_count_data)
    out[:tau] = lambda_1  # lambda before tau is lambda1
    out[tau:] = lambda_2  # lambda after (and including) tau is lambda2
    return out
```

This code creates a new function `lambda_`, but really we can think of it as a random variable: the random variable $\lambda$ from above. Note that because `lambda_1`, `lambda_2` and `tau` are random, `lambda_` will be random. We are **not** fixing any variables yet.

`@pm.deterministic` is a decorator that tells PyMC this is a deterministic function. That is, if the arguments were deterministic (which they are not), the output would be deterministic as well. Deterministic functions will be covered in Chapter 2.

```python
observation = pm.Poisson("obs", lambda_, value=count_data, observed=True)

model = pm.Model([observation, lambda_1, lambda_2, tau])
```

The variable `observation` combines our data, `count_data`, with our proposed data-generation scheme, given by the variable `lambda_`, through the `value` keyword. We also set `observed = True` to tell PyMC that this should stay fixed in our analysis. Finally, PyMC wants us to collect all the variables of interest and create a `Model` instance out of them. This makes our life easier when we retrieve the results.

The code below will be explained in Chapter 3, but I show it here so you can see where our results come from. One can think of it as a *learning* step. The machinery being employed is called *Markov Chain Monte Carlo* (MCMC), which I also delay explaining until Chapter 3. This technique returns thousands of random variables from the posterior distributions of $\lambda_1, \lambda_2$ and $\tau$. We can plot a histogram of the random variables to see what the posterior distributions look like. Below, we collect the samples (called *traces* in the MCMC literature) into histograms.

```python
# Mysterious code to be explained in Chapter 3.
mcmc = pm.MCMC(model)
mcmc.sample(40000, 10000, 1)
```

     [-----------------100%-----------------] 40000 of 40000 complete in 4.7 sec

```python
lambda_1_samples = mcmc.trace('lambda_1')[:]
lambda_2_samples = mcmc.trace('lambda_2')[:]
tau_samples = mcmc.trace('tau')[:]
```

```python
figsize(12.5, 10)
# histogram of the samples:

ax = plt.subplot(311)
ax.set_autoscaley_on(False)

plt.hist(lambda_1_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label="posterior of $\lambda_1$", color="#A60628", density=True)
plt.legend(loc="upper left")
plt.title(r"""Posterior distributions of the variables
    $\lambda_1,\;\lambda_2,\;\tau$""")
plt.xlim([15, 30])
plt.xlabel("$\lambda_1$ value")

ax = plt.subplot(312)
ax.set_autoscaley_on(False)
plt.hist(lambda_2_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label="posterior of $\lambda_2$", color="#7A68A6", density=True)
plt.legend(loc="upper left")
plt.xlim([15, 30])
plt.xlabel("$\lambda_2$ value")

plt.subplot(313)
w = 1.0 / tau_samples.shape[0] * np.ones_like(tau_samples)
plt.hist(tau_samples, bins=n_count_data, alpha=1,
         label=r"posterior of $\tau$",
         color="#467821", weights=w, rwidth=2.)
plt.xticks(np.arange(n_count_data))

plt.legend(loc="upper left")
plt.ylim([0, .75])
plt.xlim([35, len(count_data) - 20])
plt.xlabel(r"$\tau$ (in days)")
plt.ylabel("probability")
```

### Interpretation

Recall that Bayesian methodology returns a *distribution*. Hence we now have distributions to describe the unknown $\lambda$s and $\tau$. What have we gained? Immediately, we can see the uncertainty in our estimates: the wider the distribution, the less certain our posterior belief should be. We can also see what the plausible values for the parameters are: $\lambda_1$ is around 18 and $\lambda_2$ is around 23. The posterior distributions of the two $\lambda$s are clearly distinct, indicating that it is indeed likely that there was a change in the user's text-message behaviour.

What other observations can you make? If you look at the original data again, do these results seem reasonable?

Notice also that the posterior distributions for the $\lambda$s do not look like exponential distributions, even though our priors for these variables were exponential. In fact, the posterior distributions are not really of any form that we recognize from the original model. But that's OK! This is one of the benefits of taking a computational point of view. If we had instead done this analysis using mathematical approaches, we would have been stuck with an analytically intractable (and messy) distribution. Our use of a computational approach makes us indifferent to mathematical tractability.

Our analysis also returned a distribution for $\tau$. Its posterior distribution looks a little different from the other two because it is a discrete random variable, so it doesn't assign probabilities to intervals. We can see that near day 45, there was a 50% chance that the user's behaviour changed. Had no change occurred, or had the change been gradual over time, the posterior distribution of $\tau$ would have been more spread out, reflecting that many days were plausible candidates for $\tau$. By contrast, in the actual results we see that only three or four days make any sense as potential transition points.

### Why would I want samples from the posterior, anyways?

We will deal with this question for the remainder of the book, and it is an understatement to say that it will lead us to some amazing results. For now, let's end this chapter with one more example.

We'll use the posterior samples to answer the following question: what is the expected number of texts at day $t, \; 0 \le t \le 70$ ? Recall that the expected value of a Poisson variable is equal to its parameter $\lambda$. Therefore, the question is equivalent to *what is the expected value of $\lambda$ at time $t$*?

In the code below, let $i$ index samples from the posterior distributions. Given a day $t$, we average over all possible $\lambda_i$ for that day $t$, using $\lambda_i = \lambda_{1,i}$ if $t \lt \tau_i$ (that is, if the behaviour change has not yet occurred), else we use $\lambda_i = \lambda_{2,i}$.

```python
figsize(12.5, 5)
# tau_samples, lambda_1_samples, lambda_2_samples contain
# N samples from the corresponding posterior distribution
N = tau_samples.shape[0]
expected_texts_per_day = np.zeros(n_count_data)
for day in range(0, n_count_data):
    # ix is a bool index of all tau samples corresponding to
    # the switchpoint occurring prior to value of 'day'
    ix = day < tau_samples
    # Each posterior sample corresponds to a value for tau.
    # for each day, that value of tau indicates whether we're "before"
    # (in the lambda1 "regime") or
    #  "after" (in the lambda2 "regime") the switchpoint.
    # by taking the posterior sample of lambda1/2 accordingly, we can average
    # over all samples to get an expected value for lambda on that day.
    # As explained, the "message count" random variable is Poisson distributed,
    # and therefore lambda (the poisson parameter) is the expected value of
    # "message count".
    expected_texts_per_day[day] = (lambda_1_samples[ix].sum()
                                   + lambda_2_samples[~ix].sum()) / N

plt.plot(range(n_count_data), expected_texts_per_day, lw=4, color="#E24A33",
         label="expected number of text-messages received")
plt.xlim(0, n_count_data)
plt.xlabel("Day")
plt.ylabel("Expected # text-messages")
plt.title("Expected number of text-messages received")
plt.ylim(0, 60)
plt.bar(np.arange(len(count_data)), count_data, color="#348ABD", alpha=0.65,
        label="observed texts per day")

plt.legend(loc="upper left");
```

Our analysis shows strong support for believing the user's behavior did change ($\lambda_1$ would have been close in value to $\lambda_2$ had this not been true), and that the change was sudden rather than gradual (as demonstrated by $\tau$'s strongly peaked posterior distribution). We can speculate what might have caused this: a cheaper text-message rate, a recent weather-to-text subscription, or perhaps a new relationship. (In fact, the 45th day corresponds to Christmas, and I moved away to Toronto the next month, leaving a girlfriend behind.)

##### Exercises

1\.  Using `lambda_1_samples` and `lambda_2_samples`, what is the mean of the posterior distributions of $\lambda_1$ and $\lambda_2$?

```python
# type your code here.
print("{} {} ".format(lambda_1_samples.mean(), lambda_2_samples.mean()))
```

    17.750529345875638 22.709824130540305

2\.  What is the expected percentage increase in text-message rates? `hint:` compute the mean of `lambda_1_samples/lambda_2_samples`. Note that this quantity is very different from `lambda_1_samples.mean()/lambda_2_samples.mean()`.

```python
# type your code here.
(1 - np.mean(lambda_1_samples/lambda_2_samples) ) * 100
```

    21.71608702288387

3\. What is the mean of $\lambda_1$ **given** that we know $\tau$ is less than 45.  That is, suppose we have been given new information that the change in behaviour occurred prior to day 45. What is the expected value of $\lambda_1$ now? (You do not need to redo the PyMC part. Just consider all instances where `tau_samples < 45`.)

```python
# type your code here.
```

### References

-  [1] Gelman, Andrew. N.p.. Web. 22 Jan 2013. [N is never large enough](http://andrewgelman.com/2005/07/31/n_is_never_larg/).
-  [2] Norvig, Peter. 2009. [The Unreasonable Effectiveness of Data](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf).
- [3] Patil, A., D. Huard and C.J. Fonnesbeck. 2010.
PyMC: Bayesian Stochastic Modelling in Python. Journal of Statistical
Software, 35(4), pp. 1-81.
- [4] Jimmy Lin and Alek Kolcz. Large-Scale Machine Learning at Twitter. Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (SIGMOD 2012), pages 793-804, May 2012, Scottsdale, Arizona.
- [5] Cronin, Beau. "Why Probabilistic Programming Matters." 24 Mar 2013. Google, Online Posting to Google . Web. 24 Mar. 2013. <https://plus.google.com/u/0/107971134877020469960/posts/KpeRdJKR6Z1>.

```python
from IPython.core.display import HTML

def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunss.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsx.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsi.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunso.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 130%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 22pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```python

```


 **Chapter 3:  [Imaging](Ch3-Imaging.ipynb)**

# Energy-Loss Near-Edge Structure

part of

 **[Analysis of Transmission Electron Microscope Data](_Analysis_of_Transmission_Electron_Microscope_Data.ipynb)**

by Gerd Duscher, 2019

Microscopy Facilities

Joint Institute of Advanced Materials

The University of Tennessee, Knoxville

Model based analysis and quantification of data acquired with transmission electron microscopes

## Content

- Retrieving and Plotting of reference EELS spectra from the [EELS database](https://eelsdb.eu/spectra/)
- Discussion of the energy-loss near-edge structure (ELNES) of specific edges.

## Load important packages

### Check Installed Packages

```python
import sys
from pkg_resources import get_distribution, DistributionNotFound

def test_package(package_name):
    """Test if package exists and returns version or -1"""
    try:
        version = get_distribution(package_name).version
    except (DistributionNotFound, ImportError) as err:
        version = '-1'
    return version

# Colab setup ------------------
if 'google.colab' in sys.modules:
    !pip install pyTEMlib -q
# pyTEMlib setup ------------------
else:
    if test_package('sidpy') < '0.0.5':
        print('installing sidpy')
        !{sys.executable} -m pip install  --upgrade pyTEMlib -q
    if test_package('pyTEMlib') < '0.2021.4.20':
        print('installing pyTEMlib')
        !{sys.executable} -m pip install  --upgrade pyTEMlib -q
# ------------------------------
print('done')
```

    done

### Import all relevant libraries

Please note that the EELS_tools package from pyTEMlib is essential.

```python
import sys
if 'google.colab' in sys.modules:
    %pylab --no-import-all inline
else:
    %pylab --no-import-all notebook
    %gui qt

import warnings
warnings.filterwarnings('ignore')

from scipy.ndimage.filters import gaussian_filter

## import the configuration files of pyTEMlib (we need access to the data folder)
import pyTEMlib
import pyTEMlib.file_tools as ft
import pyTEMlib.eels_tools as eels

# For archiving reasons it is a good idea to print the version numbers out at this point
print('pyTEM version: ',pyTEMlib.__version__)

```

    Populating the interactive namespace from numpy and matplotlib
    pyTEM version:  0.2021.04.20

## Chemical Shift

The chemical shift is the first feature that we discuss in respect of the shape or appearance of the ionization edges: the energy-loss near edge structure (ELNES).
This section and the following one explain how to do simple analysis of near--edge features.

The chemical shift refers to small changes (up to a few eV) of the edge onset, and how this shift depends on the bonding of an element in a solid.
Going back to figure in the [Introduction to Core-Loss Spectra](CH4_07-Introduction_Core_Loss.ipynb), we see that such a change can be caused by a change of the band-gap, in which case the final states are moving or by a movement of the core-levels (initial states).

Please note, that this explanation above is a simplification; what we measure is the energy difference of an excited atom to one in ground state. In the excited atom all states react to the new electronic configuration and not only the final and initial states. In fact, to calculate the energy-difference, one cannot use the difference between core-levels and bottom of the conduction band.

However, we want to know which of the above effects (band gap changes or core-level shift) is the major one, so that we can conclude back on bonding of the element in question.

As an example of chemical shift we look at reference data of the silicon L$_{2,3}$ edge.

### Load reference data

```python
Si_L_reference_spectra = eels.get_spectrum_eels_db(element='Si',edge='L2,3')
```

    a_SiO2_Si_L3_S_Schamm_63
    a_SiO2_Si_L3_S_Schamm_64
    Si_Si_L3_S_Schamm_58
    Si3N4_Si_L3_Lingyang_Li_166
    Si3N4_(alpha)_Si_L3_S_Schamm_61
    Si3N4_(alpha)_Si_L3_S_Schamm_62
    SiC(6H)_Si_L3_S_Schamm_65
    SiC(6H)_Si_L3_S_Schamm_66
    found 8 spectra in EELS database)

### Plot silicon spectra

```python
plt.figure()
for name, spectrum  in Si_L_reference_spectra.items():
    if 'Core' in  spectrum['TITLE'] or 'L3' in  spectrum['TITLE']:
        #plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = spectrum['TITLE'])
        pass

for name, spectrum  in Si_L_reference_spectra.items():
    if 'a_SiO2_Si_L3_S_Schamm_63' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'Si-L$_{2,3}$: a-SiO$_2$')
    if 'Si_Si_L3_S_Schamm_58' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'Si-L$_{2,3}$: Si')
    if 'Si3N4_(alpha)_Si_L3_S_Schamm_62' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'Si-L$_{2,3}$: Si$_3$N$_4$')
    if 'SiC(6H)_Si_L3_S_Schamm_66'in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'Si-L$_{2,3}$: SiC')

plt.legend();

```

    <IPython.core.display.Javascript object>

The shift of the edges as above can be caused by the intial and/or the final states.

### Band gap
The band gap changes are treated in the solid state theory of band structure and are, therefore, well covered in other textbooks. The trend is that with increased oxidation (or more general increased electronegativity of the reaction partner as in the series: B, C, N, O), the band gap opens and the edge shifts to higher energies.
This is seen in the figure above,  where the onset of the Si-L$_{2,3}$ edge shifts to higher energies with Pauli electron--negativity of the reaction partner.

In fact, one can monitor band gap changes with stoichiometry at interfaces by the shift of the edge.
Please be aware that we see only the shift of the conduction band bottom and not the whole band gap change. This effect of the band gap is obvious between Si and SiO$_2$, where the edge shifts by about 6.5 eV.

### Core-level shift
The initial state, the ``core-level``,  can also shift, for example after oxidation. Some electrons will transfer to an anion (for example oxygen) and less electrons are available to fill the band structure. This is shown below for the case of Cu and its two oxides Cu$_2$O and CuO.

The more electrons transfer to oxygen for the ionic bonding of these materials, the more the edges shift to lower energies, even though a band gap opens up. The  opening up of the band gap will cause a shift too higher energies and counteracts the effect of ionization. Due to lower electron densities, at the Cu atoms in the oxides, the core levels are assumed to shift to higher energies (see below) and compensate a little for the effect.

>
>
>

The core-level shift is generally a small effect. This core states react to increase of electron density at the atom site with a decrease and vice versa. Simplified , we can think of the core level electrons getting repulsed from an increased electron density (through Coulomb interaction) and pushed closer (lower in energy) to the core.

```python
Cu_L_reference_spectra = eels.get_spectrum_eels_db(element='Cu',edge='L2,3')
```

    Cu_Cu_L3_Y_Kihn_124
    Cu2O_Cu_L3_Y_Kihn_129
    Profile Of Cu4O3-Cu L23
    CuO_Cu_L3_Y_Kihn_127
    found 4 spectra in EELS database)

```python
plt.figure()

for name, spectrum  in Cu_L_reference_spectra.items():
    if 'Cu_Cu_L3_Y_Kihn_124' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'Cu-L$_{2,3}$: Cu')
    if 'CuO_Cu_L3_Y_Kihn_127' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'Cu-L$_{2,3}$: CuO')
    if 'Cu2O_Cu_L3_Y_Kihn_129' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'Cu-L$_{2,3}$: Cu$_2$O')

plt.legend();
plt.xlim(910, 980)

```

    <IPython.core.display.Javascript object>

    (910.0, 980.0)

The case of the oxidized Cu where the slightly oxidized Cu$_2$O does shift to slighly lower energies compared to pure Cu,
the shift to lower energies due to ionic bonding is a little larger than the opening of the band gap and the core level shift to higher energies (further away from the nucleus) because of less Coulomb repulsion from the valence electrons.

This effect is even more pronounced for CuO with an larger band gap than Cu$_2$O.

In the figure below we see that the Si-L$_{3,2}$ edge shifts to higher energies with increased Ge content, when the spectra are taken from different locations at the diffuse Si - Ge interface. Intuitively, we would expect a shift to lower energies, because the band gap of SiGe alloys and of Ge is smaller than Si.

*We see that as we acquire spectra at the diffuse interface between Si and Ge, that the Si-L${_3,2}$ edge shifts to higher energies. This is surprising as SiGe and Ge posses a smaller band gap than Si and one would expect the opposite.*

This shift can be explained by a shift of core levels. An all electron calculations can determine the 2p$_{3/2}$ core levels of an atom in a compound which is shown for Si-Ge and Si-GaAs alloys in the figure below. The calculations show that there is a core level shift to lower energies with increased Ge and GaAs content. Ge and GaAs will add additional electrons to the electronic structure and the Coulomb repulsion between core level electrons and valence electrons will increase, pushing the core levels to lower energies.

*All electron ab initio calculations of the core level states for Si-Ge and Si-GaAs alloys with different composition. The calculations show an 2p$_{3/2}$ core level shift to lower energies with deviation of composition from pure Si (on the left).*

The shift of the core--levels to lower energies will increase the distance between core--level and conduction band bottom, which results in a simple picture in a shift to higher energies. We see that for pure Si, the 2p$_{3/2}$ core level is at about 89 eV but the edge is at 99.8 eV. The difference in energy is caused by relaxation of valence and core electrons. Effectively, we measure with the EELS edge onset  the energy difference between an excited atom and an atom in its ground state.

All electrons will relax according to the overall electron density at the atom sites and the calculated core--level shifts can not be used for predicting the edge shifts.  However, these calculations can explain the physical origin of the edge shift.

### Conclusion
In summation, we can say that the following effects (strongest first, weakest last) cause a chemical shift:

- band gap opening
- ionic bonding
- core level shift

All of these effects can be present at once, but usually  only one dominates the chemical shift.

## White Line

In this section, we try to analyze a distinct feature of the transition metal elements. The d-states of transition metal elements form a very flat band in the band structure. This flat band, creates a strong peak in the density of states. This analysis is based on the following simplification:

In the figure below, we see a s or p free electron like band in the general shape of a parabola, This parabola gives rise to a saw tooth like feature in the density of states (DOS) because flat bands have a higher density of states than steep ones. The DOS of the conduction band (above the Fermi level) is closely related to our EELS spectra. A flat d-band will cause a very prominent peak, a so called white line (in the age of photographic recording, these peaks appeared as white lines).

 *A schematic of the relationship of density of states (DOS on the left) and band structure of a transition metal element on the right). The s and p free electron like bands (parabola) give rise to saw tooth like feature in the DOS and the flat d bands (red) cause a sharp peak in the DOS.*

Since these d-bands are so prominent we can easily separate them from the rest. In figure \ref{fig:Ti-whiteline} we use the calculated cross--section as a model of the s and p free electron like states. After a subtraction, we get the pure d-band contribution.

*We use the cross section of  the Ti-L$_{2,3}$ edge (green) as a model for the free electron gas and subtract it from the experimental Ti-L$_{2,3}$ edge (red). The residual peaks (blue) can be analyzed as pure d-states. The two double peaks of  the Ti-L$_{2,3}$ edge indicate that there is some structure to the d-bands (here crystal field splitting).*

A simple analysis of the white line ratios of Ti-L$_3$ to Ti-L$_2$ of SrTiO$_3$ yields a intensity ratio of 242 / 314 = 0.8. However, just considering the initial states (and assuming the transition probability ( or more accurately the transition matrix elements) are the same for both edges) with 4 electrons in p$_{3/2}$ and 2 electrons in p$_{1/2}$ would let us expect a ration of 2 to 1.

>Please, note that both the Ti-L$_3$ and Ti-L$_2$ edge are split in two. We will discuss this crystal field splitting in chapter \ref{sec:Titanium} as an ELNES feature. Here we just consider the sum over the the whole Ti-L$_3$ or/and Ti-L$_2$ and ignore this splitting.

The deviation from the 2 : 1 white line ratio is assumed to be caused by J-J coupling, and is, therefore, symmetry dependent. The anomalous white line ratios have been used to determine valency of transition elements in compounds. Physically this approach is on shaky ground, because we do not know all the reasons for the change in the ratios, it has, however, been shown to be reliable for binary metallic alloys.

Fortunately, there is a easier method (from the physical point of view). We compare the total amount of white line intensities (which corresponds to the empty d-states) and normalize them by the free-electron gas like intensity beyond the white lines.

We use the method of Okamoto et al. \cite{Okamoto-Disko1992}.

The energy-window for the free-electron like part of the edge can be chosen arbitrarily and consistently.
Following Okamoto et al., a  50 eV  integration window should be used 50 eV beyond the edge onset. These will allow to compare the results to values in the literature.
The edge has to be taken in very thin areas and if possible corrected for single scattering distribution, because otherwise the free electron like part contains plasmon contributions, which change the analysis.

For the above spectrum in the figure  above, we get for the white line / free electron gas ratio (50 eV - 50 eV beyond edge onset) =  556 / 974 = 0.57. Ti in SrTiO$_3$ can be considered as Ti$^{4+}$ with no  electrons in the d-bands, but using this ratio in the paper of Okamoto et al.  would yield a d band occupancy of 4 as in a metal. The difference may lay in the usage of a Hatree-Slater X-section for the analysis while Okamoto et al. use a hydrogenic one. Also, the SrTiO$_3$ spectrum was presumably taken under completely different acquisition conditions than Okamoto's spectra.
\
For example, the SrTiO$_3$ spectrum was not corrected for convergence angle, even though it was acquired in Z-contrast mode. Another source of error, is of course the background fit, which could change especially the free electron integration result. The fact that the SrTiO$_3$ spectrum was not corrected for single scattering distribution may also overestimate the free electron gas contribution, even though the spectrum was taken in a every thin area.

For TiO$_2$ spectrum of the core-loss atlas I get for the white line / free electron gas 256 / 494 = 0.52. TiO$_2$ contains also only Ti$^{4+}$. This is the level of agreement we can expect, if we use two spectra with completely different acquisition parameters.

In the plotof the Cu-L edges above, we can see that Cu has no empty d-states but with oxidation the d-bands get unoccupied and white lines appear. The more electrons get transferred to the oxygen neighbors, the more empty d-states and the more prominent the white lines will appear.

This analysis of the occupancy of d-states is extremely important for magnetic materials where the strength depends on the unpaired (d-band or f-band) electrons.

The same analysis can be done for the empty f bands of M--edges, which are also rather flat. Usually, the M$_{4,5}$ and the M$_{2,3}$ edges form doublets of white lines.

## ELNES

So far, we have only interpreted distinct features of the shape of the ionization edges. A general approach is to look at the shape of the edges in total and use this shape as a kind of fingerprint for the interpretation. Another one is to try to understand the different features as means of electronic structure calculations of various sophistication.

In order to understand the different approaches ( and their level of confidence in the results), we will discuss the most important edges one by one.

The shape of the ELNES is closely related to the density of states of the conduction band. The next chapters discuss the basics for an electronic structure interpretation of ELNES

### Transition matrix and electronic structure
The single scattering intensity of an ionization edge $J_k^1(E)$ is related to the band structure through Fermi's Golden Rule: The transition rate is proportional to the density of final states $N(E)$ multiplied with the square of an atomic transition matrix $M(E)$

 \begin{equation} \Large
 J_k^1(E) \propto |M(E)|^2 N(E)
 \end{equation}

  The transition matrix describes the transition probability between the core states and the final states (given by $N(E)$). Because the transition probability generally decreases with higher energies (above the edge threshold, the transition matrix gives the overall shape of the edge (sawtooth) and can be determined by atomic physics.

  The density of final states (conduction band) ($N(E)$) expresses the chemical environment and its symmetry.
  Because the core--levels are highly localized the final states $N(E)$ present the local density of states. This localization causes a different shape for different elements in the same compound, even if they are nearest neighbors (with a distance of  only a few Angstrom). The $N(E)$ will of course be different for elements in materials with different (local) symmetry, coordination or chemical composition.

### Life-time broadening
  For arbitrary excitations, the $N(E)$ is the joint density of states, which means a convolution of the initial and the final states. The density of final states $N(E)$ is broadened in the spectrum by the energy resolution of the experimental  setup $\delta E$ and the width of the initial state $\Gamma_i$. $\Gamma_i$ can be approximated with the uncertainty principle:

  \begin{equation} \Large
  \Gamma_i \pi_h \approx \hbar
  \end{equation}

   The lifetime of the core -hole $\pi_h$ is determined how fast the core--hole is filled and the additional energy is dissipated through emission of Auger electrons (for light elements) or X-ray photons (heavier atoms). The value of $\Gamma_i$ depends on the threshold energy of the edge and is calculated to be around 0.1 and 2 eV for K-edges of the first 40 elements.

 Further broadening of the $N(E)$ is induced by the lifetime of the final states $\pi_f$. The inelastic mean free path of the ejected electron is only a few nm (assuming a kinetic energy of less than 50eV).
  Using the free electron approximation ($E_{kin} = m_0 v^2 / 2$), we get for the energy broadening  of the final states:
  \begin{equation} \Large
  \Gamma_f \approx \frac {\hbar}{\pi_f} = \frac{\hbar v}{\lambda_i } = \frac{\hbar}{\lambda_i} \sqrt{\frac{2E_{kin}}{m_0}}
  \end{equation}

  Since the inelastic mean free path $\lambda_i$ varies inversely with kinetic energy $E_{kin}$ below 50 eV (and raises only slightly above 50 eV), the observed density of state structure is more and more broadened the further the distance from the onset.

  The next two chapters discuss the density of final states $N(E)$ and the transition matrix $M(E)$ in detail.

### Dipole-selection rule

Assuming a single electron approximation (and almost no electronic theory theory solves the many-particle problem fully) for the excitation, we can replace the many electron transition matrix elements with single electron matrix elements $M(\vec{q},E)$.

\begin{equation} \Large
M(\vec{q},E) =  <f|\exp(i\vec{q}\vec{r})|i>
 \end{equation}

 with the initial wave function $|i> = \phi_i$ and the complex conjugated final wave function $<f|=\phi_f$.
 The integration is over the volume $\tau$ surrounding the initial state (and we get a high localization again).

 We now expand the operator $\exp(i\vec{q}\vec{r})$ as :

\begin{equation} \Large
\exp(i\vec{q}\vec{r}) = 1 + i\vec{q}\vec{r} + \mbox{higher--order terms}
\end{equation}

Because the final and initial wave functions must be orthogonal to each other, the first term gives zero.
The higher--order terms are assumed to be negligible.

The second term has odd symmetry and determines the dipole selection rule:
- if the initial and final wave functions are both even or odd, then the result is zero.
- only if one of the  initial and final wave are odd and one of them is even the result is not zero.

 There is no transition probability for s to s transitions or p-p transitions or d-d or s-d transitions.
 However, transitions are possible from s to p states and from p to s and d states.

 The (optical) dipole selection rule says the angular quantum number $l$ of the initial and final state must differ by 1: $\delta l = \pm 1$.

 For the dipole selection rule to be valid, obviously the higher order terms must be very small, otherwise a third integral (of the dipole--forbidden reflections) will modify the observed ELNES.

 A rough estimates leads to the requirement $\vec{v}\vec{r} <<1$ for all $r$, which is equivalent to $ q<<q_d = 1/r_c$ where $r_c$ is the radius of the core state. This radius $r_c$ denotes the volume where the transition occurs.
 And we get for the carbon K-shells ($Z^* = Z- 0.3$) and 100kV acceleration energy that the dipole selection rule should be applicable for $\theta << \theta_d = 63 mrad$. For the Si L$_{3,2}$ edge the dipole selection rule should prevail up to about 12 mrad.

 Therefore, we must use rather small aperture ( $ < 10 - 20$ mrad) centered at the zero beam to insure the dipole selection rule to be valid.

### Core--hole effects and excitons

 So far, we can say that our ELNES can be calculated by the site and angular momentum projected density of states. This is true for almost all metals, but the core-hole effects (through excitonic interactions of the core--hole with the excited electron and valence electrons)  further modifies the final density of states $N(E)$.

The missing electron in the core hole and the extra electron in the conduction band causes changes in the band structure (in the core, valence and conduction states regions).
These relaxation processes of all the electrons involved can be rather complicated and we have to know when these complications have to be applied.

*A method to determine the edge onset from XPS data and the band gap, if core--hole effects are  negligible. This is only true for metals with an almost complete screening of the core--hole.*

As illustrated in the figure above (Pantelides, PRL 1975), if there is no influence of the core-electron, the edge onset of the ionization edge should be determined by:
\begin{eqnarray}
E_{\mbox{onset}} &=& E_{\mbox{core-state}} - E_{\mbox{CB}}\\
 &=& (E_{\mbox{core-state}} - E_{\mbox{vacuum}} ) + (E_{\mbox{vacuum}} - E_{\mbox{CB}} )\\
 &=& (E_{\mbox{core-state}} - E_{\mbox{vacuum}} ) + (E_{\mbox{vacuum}} - E_{\mbox{VB}} ) +(E_{\mbox{VB}} - E_{\mbox{CB}} )
\end{eqnarray}

The distances between core--states and vacuum and valence band (VB) and vacuum are directly accessible with XPS, while the band gap ($E_{\mbox{VB}} - E_{\mbox{CB}} $) is available from optical spectroscopy.

Therefore the calculated density of states not including core holes should start at that energy, If the agreement is not In all other cases, we must assume that the (excitonic) core-hole effects modify the $N(E)$.

This modification of $N(E)$ can be compared to doping effects in semiconductors as shown in the figure below, where the whole band structure shifts due to the extra electron. This shifting of bands (up and down) is due to the different screened potential of an excited atom compared to an atom in ground state.

 *Comparison of doping and core--hole effects on the electronic structure. Both effects modify the band structure.*

The modifications of the core hole effect not be predicted without extensive calculations.
Two methods have profen to be successful for ELNES calculations:
- explicit core hole in an all electron calculation (like FLAPW)
- Z+1 approximation in pseudo potential calculations (like VASP)

In the explicit core-hole approach, we just take one electron from a core level of one atom in the supercell and place it (energetically) on top of the other electrons of that (now excited) atom.
Now, the electronic structure calculations give the ELNES as the local and angular projected density of states of the excited atom.

In the Z+1 calculation, the atom with the explicit core-hole is replaced by an atom with an atomic number increased by one. The core hole is simulated by an nuclear charge that is increased by one and the excess electron is there naturally. This Z+1 approximation can be used in combination with the much faster pseudo potential methods. The results of this approximation are very close to the explicit core hole simulations.

## ELNES Examples

### Carbon

Carbon has 4 valence electrons.\\
As a free atom has two electron in 2s and two electron in 2p states.\\
In a compound, it is easier to understand the bonding assuming that there is only one electron in the 2s state and three electrons in the 2p state.

Carbon can bond in three modes and as elemental material stays in two modification:
 - Diamond:  sp$_3$ hybridization
 - lonsdaleite: sp$_3$ hybridization
 - Graphite:  sp$_2$ hybridization

The energy levels of the 2s and 2p levels are so close together that they can hybridize (as many elements in the first row of the periodic table). Hybridization means the states are indistinguishable (in energy). Carbon is unique in a sense as it can hybridize so that 2, 3 or 4 electrons hybridize. The 1s (occupied by one electron) and two or all three p states hybridize into states with p character, which means they are directional and form covalent bonds.

In the case of the sp$_2$ bonds, one of the p electrons does not contribute to bonding and is de-localized. This de-localized electron is the reason for graphite to be conducting, while in diamond all electrons are involved in covalent bonding and consequently diamond is an insulator. This extra (non-hybridized) 2p electron does the double bonding in molecules and polymers.

*Comparison between  $sp_2$ and $sp_3$ hybridization. The  $sp_2$ hybridization causes a layered structure and a de--localized electron. The  $sp_3$hybridization causes a bond angles of 109.5${\circ}$.*

Graphite has three covalent bonds and forms a layered structure with three nearest neighbors within the layers \ref{fig:sp2}. Within the layer, the $sp_2$ hybridization does the covalent bonding. The layers are bonded together with the weak Van-der-Waals force. Diamond has 4 covalent bonds and forms with its 4 neighbors a tetrahedron. The bond angle in diamond are 109.5 and fill all three dimensions.

```python
C_reference_spectra = eels.get_spectrum_eels_db(element='C',edge='K')
```

    8-pm
    8-pm
    B4C_B_K_Otmane_Abidate_41
    B4C_C_K_Otmane_Abidate_42
    C_C_K_V_Serin_17
    C_C_K_M__Jaouen,_G__Tourillon_20xray
    C_C_K_M__Jaouen,_G__Tourillon_21xray
    C_C_K_Otmane_Abidate_24
    C_C_K_Odile_Stephan_226
    C_C_K__243
    C_(graphite)_C_K_David_Muller_(at_Cornell_University)_1995_23
    7-pm
    E113 250-2-cal-kgd-pm-dm
    e113 k-pm
    255-1-pmmod
    C3N6H6_C_K_Susana_Trasobares_114
    C3N6H6_C_K_Susana_Trasobares_115
    CN_C_K_Susana_Trasobares_122
    Cr3C2_C_K_R__Leapman_11
    Cr7C3_C_K_R__Leapman_13
    CrC_C_K_Christina_Scheu_176
    FeCO3_O_K_A__Gloter,_A__Chen_206
    MgCO3_O_K_Irmgard_Rom_43
    MgCO3_C_K_Irmgard_Rom_44
    SiC_beta_Si_K_Flank_30xray
    found 24 spectra in EELS database)

```python
plt.figure()
for name, spectrum  in C_reference_spectra.items():
    if 'Core' in  spectrum['TITLE'] or 'L3' in  spectrum['TITLE']:
        #plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = spectrum['TITLE'])
        pass

for name, spectrum  in C_reference_spectra.items():
    if 'C_(graphite)_C_K_David_Muller_(at_Cornell_University)_1995_23' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'C-K: graphite')
    if 'C_C_K_M__Jaouen,_G__Tourillon_21xray' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'C-K: diamond')
    if 'C_C_K__243' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'C-K: ')
    if 'CN_C_K_Susana_Trasobares_122' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'C-K: CN')
plt.legend()
plt.xlim(275, 310)
```

    <IPython.core.display.Javascript object>

    (275.0, 310.0)

Looking at the bonding of carbon in molecular orbital theory or (its predecessor) Ligand--Field theory the  non--hybridized p electron in graphite will form an occupied $\pi$ bond and an unoccupied $\pi^*$ bond. In figure \ref{fig:C-K} we see that the unoccupied $\pi^*$ state is visible in the graphite spectrum.
In diamond, there is no molecule like p electron and consequently there is no $\pi$ or $\pi^*$ bond.
The appearance of the $\pi^*$ state in a carbon spectrum is used as a fingerprint for $sp_2$ hybridization. In the case of so called diamond like carbon, an amorphous carbon with $sp_2$ and $sp_3$ bonds, the quality (amount of $sp_3$ bonds) of the diamond like carbon can be assessed by the  intensity of the  $\pi^*$ peak (or rather the lack of it).

Both spectra have a $\sigma_1^*$ and $\sigma_2^*$ peak which are associated with the molecule like s states.
The C-K edge should show only the p density of states due to the dipole selection rule. The $\sigma^*$ states show up in the C-K edge because of these states are already related to the s-p like free electron gas density of states (s-p hybridization) above the edge. The $\sigma^*$ states are the non-bonding states of the ( $sp_2$ or $sp_3$) hybridized states and are, therefore, present in any carbon compound.

The quantification of $sp_2$ versus $sp_3$ hybridization is also important in polymers (where the non hybridized p electron in a $sp_2$ configuration forms the  conducting double bonds. In Buckminster fullerens (bucky balls) and carbon nanotubes the $sp_3$ hybridization is always associated with a defect (dislocation like), where a carbon atom has now 4 nearest neighbors.

### Silicon

The calculation of the transition matrix $M(E)$ for
the Si-L$_{3,2}$ edge shows that the intensity of the ELNES consists almost exclusively of d-states. Less than 5\% of the intensity is from the also dipole--allowed s-DOS.

We can, therefore, assume that only d-states form the Si-L$_{3,2}$ ELNES.
The spin orbit splitting of the initial p states is 0.7 eV, and this means that the L$_3$ and the L$_2$ are separated by 0.7 eV, which cannot be resolved with most instrumental setups.
To the calculated (local) d-DOS the same DOS has  to be added (with a ratio of about 2:1)  but shifted by 0.7 eV.

*Comparison of experimental and theoretical data. While an effective mass exciton would explain the sharp raise, the effect is to small, the electronic structure  calculation without core hole effect, placed at the correct onset, does not agree with the experiment.*

The edge onset of the Si-L$_{3,2}$ of pure Si should be at 100 eV without core-hole effects.  A d--Dos calculated without influence of a core--hole is shown in figure \ref{fig:Si-L-pure} beginning at this value. We can clearly see that this DOS of state cannot reproduce the experimental ELNES. From this disagreement between experiment and DOS without core-hole effect, we conclude that the
	core-hole effects must be included.

The main feature of the Si-L$_{3,2}$ of pure Si is the extreme sharp raise of the edge at the onset.
This feature cannot be explained by the d-DOS calculated without core--hole effects, which raises not as steeply as the experimental ELNES.

This steep raise is another indication of the core--hole and must have its origin in an excitonic effect (an interaction of the excess electron in the conduction band an the hole in the core state).
In the figure above, the calculations of an effective mass electron (due to an state that is created just below the conduction band) is compared to the experimental ELNES. Such an effective mass electron must be considered delocalized. We see that the raise is steep enough to explain the experimental raise, we also see that the effect (intensity) is not change the ELNES.

Only the explicit inclusion of the core--hole or the Z+1 calculations in figure \ref{fig:Si-L-pure2} can explain this steep onset. We can, therefore, conclude on localized excitonic enhancement of the states at the bottom of conduction band. This is a rather localized excitonic effect.

We can also see in the comparison of the explicit inclusion of the core-hole and the Z+1 approximation that both simulations lead to the same ELNES, however only the explicit core-hole calculation can predict the exact intensity (cross section) of the Si-L$_{3,2}$ edge.

The same calculations are also successful for SiO$_2$ (quartz) as can be seen in figure \ref{fig:Si-L-sio2}. The experimental data show the spin--orbit splitting in the first peak, all other features are to smeared out to show a clear distinction between transitions originating from $2p_{3/2}$ and 2p$_1/2$.
Due to the simple addition of the shifted spectra, the splitting in the first peak is reproduced rather easily and cannot be used for further analysis. Again, this edge is completely dominated by the local d-DOS.

### Oxygen and Nitrogen

Oxygen and nitrogen edges are usually very similar. Here we will discuss mostly the oxygen edge, but this can be easily transferred to nitrogen.

The Si-SiO$_2$ interface shows oxygen deficiency in the oxide at the interface. In the following, I will show that the oxygen K edge ELNES cannot be used to probe the oxygen deficiency. Experimentally, the oxygen K edge has a chemical shift by about 1 eV. The structure of the edge is washed out at the interface as shown in figure . Higher spatial resolution experiments by Muller (nature 2003) show a completely structureless O-K edge. Simulation of the O-K edge show that this shift and the featureless structure is due to the dimer like structure (Si-O-Si) which is not avoidable at any Si-SiO$_2$ interface.

Another approach is the so called "finger-print" method. In this method, one compares edges from different but known materials and hopes that similar features are conclusive for different coordinations within the unknown material. This approach can be improved by using simulations of the ELNES of the edges as seen in the plot below.

```python
O_reference_spectra = eels.get_spectrum_eels_db(element='O',edge='K')
```

    (La0_7,Sr0_3)MnO3_O_K_imhoff_229
    8-pm
    Al2O3_Al_K_Flank_1xray
    Al2O3_(alpha)_Al_K_Stefan_Nufer_53
    AlPO4_Al_K_Flank_2xray
    B2O3_B_K_Karine_Varlot_8xray
    B2O3_B_K_Karine_Varlot_9xray
    7-pm
    E113 250-2-cal-kgd-pm-dm
    e113 k-pm
    255-1-pmmod
    CaCoO NT-2
    CaTiO3_Ti_L3_Saso_Sturm_133
    CoO_O_K_dominique_IMHOFF_235
    Cr2O3_(alpha)_O_K_Rik_Brydson_158
    Cu2O_O_K_Y_Kihn_130
    Profile Of Cu4O3-O-K
    CuO_O_K_Y_Kihn_126
    Fe2O3_(alpha)_O_K_A__Gloter,_A__Chen_201
    Fe2TiO5_Ti_L3_E_Fries_P_Perriat_36
    Fe3O4_O_K_A__Gloter,_A__Chen_205
    FeCO3_O_K_A__Gloter,_A__Chen_206
    FeOOH_O_K_A__Gloter,_A__Chen_204
    FeOOH__(alpha)_O_K_A__Gloter,_A__Chen_203
    FeOOH__(beta)_O_K_A__Gloter,_A__Chen_202
    lsmo2
    APL_article_LSMO_oxygen2
    APL_article_interface_oxygen2
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    EELS Spectrum Image (high-loss)model from decomposition with 6 components
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    EELS Spectrum Image (high-loss)model from decomposition with 6 components
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    EELS Spectrum Image (high-loss)model from decomposition with 6 components
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    EELS Spectrum Image (high-loss)model from decomposition with 6 components
    LaCoO3_O_K_Irmgrad_Rom,_Ferdinand_Hofer_162
    MgCO3_O_K_Irmgard_Rom_43
    MgCO3_C_K_Irmgard_Rom_44
    MgO_O_K_Giovanni_Bertoni_112
    MgO_Mg_K_Giovanni_Bertoni_113
    MgO_O_K_dominique_IMHOFF_233
    MgO_Mg_K_dominique_IMHOFF_234
    Na2O_Na_K_Flank_34xray
    Nb2O5_Nb_M_David_Bach,_Wilfried_Sigle_215
    NbO2_Nb_M_David_Bach,_Wilfried_Sigle_217
    SiO2_Si_K_Flank_27xray
    SiO2_Si_K_Flank_28xray
    SiO2_Si_K_Flank_29xray
    Spectre centre EELSDB
    SrTiO3_O_K_imhoff_232
    srtio32
    APL_article_STO_oxygen2
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    EELS Spectrum Image (high-loss)model from decomposition with 6 components
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    EELS Spectrum Image (high-loss)model from decomposition with 6 components
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    EELS Spectrum Image (high-loss)model from decomposition with 6 components
    TiO2_(amorphous)_O_K_Giovanni_Bertoni_106
    TiO2_(anatase)_O_K_Giovanni_Bertoni_103
    Y3Al5O12_O_K_Saso_Sturm_182
    Y3Al5O12_Al_K_Saso_Sturm_213
    Y4Al2O9_O_K_Saso_Sturm_178
    Y4Al2O9_Al_K_Saso_Sturm_179
    YAlO3_O_K_Saso_Sturm_186
    YAlO3_Al_K_Saso_Sturm_187
    ZnO_O_K_Wilfried_Sigle_169
    found 55 spectra in EELS database)

```python
O_reference_titles = ['SrTiO3_O_K_imhoff_232',
                      'MgO_O_K_Giovanni_Bertoni_112',
                      'ZnO_O_K_Wilfried_Sigle_169',
                      'Cr2O3_(alpha)_O_K_Rik_Brydson_158'
                     ]
O_reference_materials = ['SrTiO$_3$', 'MgO', 'ZnO', 'Cr$_2$O$_3$']
plt.figure()
i = 0
for name, spectrum  in O_reference_spectra.items():
    if spectrum['TITLE'] in O_reference_titles:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'O-K:'+O_reference_materials[i])
        i+=1

plt.legend();
plt.xlim(525,570)

```

    <IPython.core.display.Javascript object>

    (525.0, 570.0)

A comparison hows that the cubic oxides MgO and MgAl$_2$O$_4$ (spinel structure) have a similar ELNES which differs from the ELNES from the rhombohedral oxide Al$_2$O$_3$.

Correlation effects of valence electrons cause the so called Hubbard band. These bands dominate the electronic structure in high T$_c$ semiconductors, giant magnet resistors and other materials with high correlations.

In the figure below, we see that such a correlation effect takes place in the giant magneto-resistor LaMnO$_3$ but not in the perovskite LaAlO$_3$, which does not have this splitting of the d bands (Al does not have d electrons but Mn has), a precondition for the formation of a Hubbard band.

*The O-K edge in LaMnO$_3$ has a pre-peak that is interpreted as Hubbard band and is not present in LaAlO$_3$ even so it has the same structure.*

The same Hubbard band is visible in the high T$_c$ superconductor YBa$_2$Cu$_3$O$_{7-\delta}$.
In figure \ref{fig:O-K-YBCO-DL}, we see the probing of this Hubbard band at a dislocation in YBa$_2$Cu$_3$O$_{7-\delta}$.

*The O-K edge at a dislocation in YBa$_2$Cu$_3$O$_{7-\delta}$ has less of the signature of a Hubbard band than far away in the bulk. This lack of holes in the dislocation  disturbs the superconductivity and is utilized in grain boundaries for Josephson junctions but is detrimental in polycrystalline high T$_c$ wires.*

The prepeak that is caused by the Hubbard band is reduced in the vicinity of the dislocation and vanishes completely within the dislocation core. This lack of holes in the dislocation  disturbs the superconductivity and is utilized in grain boundaries for Josephson junctions but is detrimental in polycrystalline high T$_c$ wires.

## Spatial Resolution in EELS
% images/spat-dif-resolution.jpg}

### Methods to achieve spatial resolution
- Spot Mode
- Area Mode = Spatial Difference
- Lines Scan Mode
    - Each Spot of Line one Spectrum
    - Each Segement of Line one Spectrum
    - Each Area of Line one Spectrum
- Spectrum Imaging

### Spot mode
% {images/spat-dif-spot.jpg}

### Spatial difference
% images/spat-dif-spatdif1.jpg
% images/spat-dif-spatdif2.jpg
% images/spat-dif-spatdif3.jpg
*EELS at Bi  doped  Cu grain boundary*
% images/spat-dif-spatdif4.jpg
% images/spat-dif-ls1.jpg
*As segregation at Si/SiO$_2$ interface*
% images/spat-dif-ls2.jpg
*As segregation at Si/Sio$_2$ interface*

### Energy Filtered Imaging (EFTEM)
% images/spat-dif-eftem1.jpg}
% images/spat-dif-eftem2.jpg}
% images/spat-dif-eftem3.jpg}

## Summary

The core--loss part of the electron energy--loss spectrum allows us to determine:
- chemical composition
- bonding
- magnetic moment through ionic charge

>
>	with high spatial resolution!!!
>

## Navigation
-   **Up Chapter 4: [Imaging](CH4_00-Spectroscopy.ipynb)**
-   **Back: [Analysis of Core-Loss](CH4_09-Analysis_Core_Loss.ipynb)**
-   **List of Content: [Front](../_MSE672_Intro_TEM.ipynb)**

```python

```