*This notebook contains course material from [CBE40455](https://jckantor.github.io/CBE40455) by
Jeffrey Kantor (jeff at nd.edu); the content is available [on Github](https://github.com/jckantor/CBE40455.git).
The text is released under the [CC-BY-NC-ND-4.0 license](https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode),
and code is released under the [MIT license](https://opensource.org/licenses/MIT).*

< [Risk and Diversification](http://nbviewer.jupyter.org/github/jckantor/CBE40455/blob/master/notebooks/07.00-Risk-and-Diversification.ipynb) | [Contents](toc.ipynb) | [Geometric Brownian Motion](http://nbviewer.jupyter.org/github/jckantor/CBE40455/blob/master/notebooks/07.02-Geometric-Brownian-Motion.ipynb) >

# Measuring Return

How much does one earn relative to the amount invested?

This is the basic concept of return, and one of the fundamental measurements of financial performance. This notebook examines the different ways in which return can be measured.

## Pandas-datareader

As will be shown below, [pandas-datareader](https://github.com/pydata/pandas-datareader) provides a convenient means access and manipulate financial data using the Pandas library. The pandas-datareader is normally imported separately from pandas. Typical installation is

    pip install pandas-datareader

from a terminal window, or executing

    !pip install pandas-datareader

in a Jupyter notebook cell. Google Colab environment now includes pandas-datareader, so separate installation is required.

## Imports

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns

import datetime

import pandas as pd
import pandas_datareader as pdr
```

## Where to get Price Data

This notebook uses the price of stocks and various commodity goods for the purpose of demonstrating returns. Price data is available from a number of sources. Here we demonstrate the process of obtaining price data on financial goods from [Yahoo Finance](http://finance.yahoo.com/) and downloading price data sets from [Quandl](http://www.quandl.com/). (UPDATE: [Look here for an alternative descripton of how to get live market data from Yahoo Finance](https://towardsdatascience.com/python-how-to-get-live-market-data-less-than-0-1-second-lag-c85ee280ed93).)

The most comprehensive repositories of financial data are commercial enterprises. Some provide a free tier of service for limited use, typically 50 inquires a day or several hundred a month. Some require registration to access the free tier. These details are a constantly changing. A listing of free  services is available from [awesome-quant](https://github.com/wilsonfreitas/awesome-quant#data-sources), but please note that details change quickly. [Another useful collection of stock price data using Python](https://towardsdatascience.com/how-to-get-stock-data-using-python-c0de1df17e75).

### Stock Symbols

Stock price data is usually indexed and accessed by stock symbols. Stock symbols are unique identifiers for a stock, commodity, or other financial good on a specific exchanges. For example, [this is a list of symbols for the New York Stock Exchange (NYSE)](http://www.eoddata.com/symbols.aspx?AspxAutoDetectCookieSupport=1) The following function looks up details of stock symbol on yahoo finance..

```python
# python libraray for accessing internet resources
import requests

def lookup_yahoo(symbol):
    """Return a list of all matches for a symbol on Yahoo Finance."""
    url = f"http://d.yimg.com/autoc.finance.yahoo.com/autoc?query={symbol}&region=1&lang=en"
    return requests.get(url).json()["ResultSet"]["Result"]

lookup_yahoo("XOM")
```

    [{'exch': 'NYQ',
      'exchDisp': 'NYSE',
      'name': 'Exxon Mobil Corporation',
      'symbol': 'XOM',
      'type': 'S',
      'typeDisp': 'Equity'},
     {'exch': 'NMS',
      'exchDisp': 'NASDAQ',
      'name': 'XOMA Corporation',
      'symbol': 'XOMA',
      'type': 'S',
      'typeDisp': 'Equity'},
     {'exch': 'YHD',
      'exchDisp': 'Industry',
      'name': 'Exxon Mobil Corporation',
      'symbol': 'XOM.BA',
      'type': 'S',
      'typeDisp': 'Equity'},
     {'exch': 'YHD',
      'exchDisp': 'Industry',
      'name': 'Exxon Mobil Corporation',
      'symbol': 'XOM.MX',
      'type': 'S',
      'typeDisp': 'Equity'},
     {'exch': 'DUS',
      'exchDisp': 'Dusseldorf Stock Exchange',
      'name': 'XOMA CORP.  DL -,0005',
      'symbol': 'X0M1.DU',
      'type': 'S',
      'typeDisp': 'Equity'},
     {'exch': 'STU',
      'exchDisp': 'Stuttgart',
      'name': 'XOMA Corp. Registered Shares DL',
      'symbol': 'X0M1.SG',
      'type': 'S',
      'typeDisp': 'Equity'},
     {'exch': 'TLO',
      'exchDisp': 'TLX Exchange',
      'name': 'Exxon Mobil Corporation',
      'symbol': 'XOM-U.TI',
      'type': 'S',
      'typeDisp': 'Equity'},
     {'exch': 'VIE',
      'exchDisp': 'Vienna',
      'name': 'Exxon Mobil Corporation',
      'symbol': 'XOM.VI',
      'type': 'S',
      'typeDisp': 'Equity'},
     {'exch': 'BUE',
      'exchDisp': 'Buenos Aires',
      'name': 'EXXON MOBIL CORP',
      'symbol': 'XOMD.BA',
      'type': 'S',
      'typeDisp': 'Equity'}]

```python
def get_symbol(symbol):
    """Return exact match for a symbol."""
    result = [r for r in lookup_yahoo(symbol) if symbol == r['symbol']]
    return result[0] if len(result) > 0 else None

get_symbol('TSLA')
```

    {'exch': 'NMS',
     'exchDisp': 'NASDAQ',
     'name': 'Tesla, Inc.',
     'symbol': 'TSLA',
     'type': 'S',
     'typeDisp': 'Equity'}

### Yahoo Finance

[Yahoo Finance](http://finance.yahoo.com/) provides historical Open, High, Low, Close, and Volume date for quotes on traded securities. In addition, Yahoo Finance provides historical [Adjusted Close](http://marubozu.blogspot.com/2006/09/how-yahoo-calculates-adjusted-closing.html) price data that corrects for splits and dividend distributions. Adjusted Close is a useful tool for computing the return on long-term investments.

The following cell demonstrates how to download historical Adjusted Close price for a selected security into a pandas DataFrame.

```python
symbol = 'TSLA'

# get symbol data
symbol_data = get_symbol(symbol)
assert symbol_data, f"Symbol {symbol} wasn't found."

# start and end of a three year interval that ends today
end = datetime.datetime.today().date()
start = end - datetime.timedelta(3*365)

# get stock price data
S = pdr.data.DataReader(symbol, "yahoo", start, end)['Adj Close']

# plot data
plt.figure(figsize=(10,4))
title = f"{symbol_data['name']} ({symbol_data['exchDisp']} {symbol_data['typeDisp']} {symbol_data['symbol']})"
S.plot(title=title)
plt.ylabel('Adjusted Close')
plt.grid()
```

Note that `S` is an example of a Pandas time series.

```python
S
```

    Date
    2017-11-06     60.556000
    2017-11-07     61.209999
    2017-11-08     60.877998
    2017-11-09     60.598000
    2017-11-10     60.598000
                     ...
    2020-10-28    406.019989
    2020-10-29    410.829987
    2020-10-30    388.040009
    2020-11-02    400.510010
    2020-11-03    424.140015
    Name: Adj Close, Length: 754, dtype: float64

Pandas time series are indexed by datetime entries. There is a large collection of functions in Pandas for manipulating time series data.

```python
S["2018"].plot()
```

### Quandl

[Quandl](http://www.quandl.com/) is a searchable source of time-series data on a wide range of commodities, financials, and many other economic and social indicators. Data from Quandl can be downloaded as files in various formats, or accessed directly using the [Quandl API](http://www.quandl.com/help/api) or software-specific package. Here we use demonstrate use of the [Quandl Python package](http://www.quandl.com/help/packages#Python).

The first step is execute a system command to check that the Quandl package has been installed.

Here are examples of energy datasets. These were found by searching Quandl, then identifying the Quandl code used for accessing the dataset, a description, the name of the field containing the desired price information.

```python
%%capture
capture = !pip install quandl
```

```python
code = 'CHRIS/MCX_CL1'
description = 'NYMEX Crude Oil Futures, Continuous Contract #1 (CL1) (Front Month)'
field = 'Close'
```

```python
import quandl

end = datetime.datetime.today().date()
start = end - datetime.timedelta(5*365)

try:
    S = quandl.get(code, collapse='daily', trim_start=start.isoformat(), trim_end=end.isoformat())[field]

    plt.figure(figsize=(10,4))
    S.plot()
    plt.title(description)
    plt.ylabel('Price $/bbl')
    plt.grid()
except:
    pass
```

## Returns

The statistical properties of financial series are usually studied in terms of the change in prices. There are several reasons for this, key among them is that the changes can often be closely approximated as stationary random variables whereas prices are generally non-stationary sequences.

A common model is

$$S_{t} = R_{t} S_{t-1}$$

so, recursively,

$$S_{t} = R_{t} R_{t-1} \cdots R_{0} S_{0}$$

The gross return $R_t$ is simply the ratio of the current price to the previous, i.e.,

$$R_t = \frac{S_t}{S_{t-1}}$$

$R_t$ will typically be a number close to one in value. The return is greater than one for an appreciating asset, or less than one for a declining asset.

The Pandas timeseries `shift()` function is used compute the ratio $\frac{S_t}{S_{t-1}}$. Shifting a timeseries 1 day forward, i.e, `shift(1)`, shifts $S_{t-1}$ to time $t$. That's why

    R = S/S.shift(1)

provides the correct calculation for the quantities $R_t$.

```python
print([S, S.shift(1)])
```

    [Date
    2017-11-06     60.556000
    2017-11-07     61.209999
    2017-11-08     60.877998
    2017-11-09     60.598000
    2017-11-10     60.598000
                     ...
    2020-10-28    406.019989
    2020-10-29    410.829987
    2020-10-30    388.040009
    2020-11-02    400.510010
    2020-11-03    424.160004
    Name: Adj Close, Length: 754, dtype: float64, Date
    2017-11-06           NaN
    2017-11-07     60.556000
    2017-11-08     61.209999
    2017-11-09     60.877998
    2017-11-10     60.598000
                     ...
    2020-10-28    424.679993
    2020-10-29    406.019989
    2020-10-30    410.829987
    2020-11-02    388.040009
    2020-11-03    400.510010
    Name: Adj Close, Length: 754, dtype: float64]

```python
symbol = 'TSLA'

end = datetime.datetime.today().date()
start = end - datetime.timedelta(3*365)

# get stock price data
S = pdr.data.DataReader(symbol, "yahoo", start, end)['Adj Close']
R = S/S.shift(1)

# plot data
plt.figure(figsize=(10, 5))
plt.subplot(2, 1, 1)
S.plot(title=symbol)
plt.ylabel('Adjusted Close')
plt.grid()

plt.subplot(2, 1,  2)
R.plot()
plt.ylabel('Returns')
plt.grid()
plt.tight_layout()
```

### Linear fractional or Arithmetic Returns

Perhaps the most common way of reporting returns is simply the fractional increase in value of an asset over a period, i.e.,

$$r^{lin}_t = \frac{S_t - S_{t-1}}{S_{t-1}} = \frac{S_t}{S_{t-1}} - 1 $$

Obviously

$$r^{lin}_t = R_t  - 1$$

```python
symbol = 'TSLA'

end = datetime.datetime.today().date()
start = end - datetime.timedelta(3*365)

# get stock price data
S = pdr.data.DataReader(symbol, "yahoo", start, end)['Adj Close']
rlin = S/S.shift(1) - 1

# plot data
plt.figure(figsize=(10,5))
plt.subplot(2,1,1)
S.plot(title=symbol)
plt.ylabel('Adjusted Close')
plt.grid()

plt.subplot(2,1,2)
rlin.plot()
plt.title('Linear Returns (daily)')
plt.grid()
plt.tight_layout()
```

### Linear returns don't tell the whole story.

Suppose you put money in an asset that returns 10% interest in even numbered years, but loses 10% in odd numbered years. Is this a good investment for the long-haul?

If we look at mean linear return

\begin{align}
\bar{r}^{lin} & = \frac{1}{T}\sum_{t=1}{T} r^{lin}_t  \\
& = \frac{1}{T} (0.1 - 0.1 + 0.1 - 0.1 + \cdots) \\
& = 0
\end{align}

we would conclude this asset, on average, offers zero return. What does a simulation show?

```python
S = 100
log = [[0,S]]
r = 0.10

for k in range(1,101):
    S = S + r*S
    r = -r
    log.append([k,S])

df = pd.DataFrame(log,columns = ['k','S'])
plt.plot(df['k'],df['S'])
plt.xlabel('Year')
plt.ylabel('Value')
```

Despite an average linear return of zero, what we observe over time is an asset declining in price.  The reason is pretty obvious --- on average, the years in which the asset loses money have higher balances than years where the asset gains value.  Consequently, the losses are somewhat greater than the gains which, over time, leads to a loss of value.

Here's a real-world example of this phenomenon. For a three year period ending October 24, 2017, United States Steel (stock symbol 'X') offers an annualized linear return of 15.9%. Seems like a terrific investment opportunity, doesn't it?  Would you be surprised to learn that the actual value of the stock fell 18.3% over that three-year period period?

What we can conclude from these examples is that average linear return, by itself, does not provide us with the information needed for long-term investing.

```python
symbol = 'X'

end = datetime.datetime(2017, 10, 24)
start = end-datetime.timedelta(3*365)

# get stock price data
S = pdr.data.DataReader(symbol, "yahoo", start, end)['Adj Close']
rlin = S/S.shift(1) - 1
rlog = np.log(S/S.shift(1))

print('Three year return :', 100*(S[-1]-S[0])/S[0], '%')

# plot data
plt.figure(figsize=(10,5))
plt.subplot(2,1,1)
S.plot(title=symbol)
plt.ylabel('Adjusted Close')
plt.grid()

plt.subplot(2,1,2)
rlog.plot()
plt.title('Mean Log Returns (annualized) = {0:.2f}%'.format(100*252*rlog.mean()))
plt.grid()
plt.tight_layout()
```

### Compounded Log Returns

Compounded, or log returns, are defined as

$$r^{log}_{t} = \log R_t = \log \frac{S_{t}}{S_{t-1}}$$

The log returns have a very useful compounding property for aggregating price changes across time

$$ \log \frac{S_{t+k}}{S_{t}} = r^{log}_{t+1} + r^{log}_{t+2} + \cdots + r^{log}_{t+k}$$

If the compounded returns are statistically independent and identically distributed, then this property provides a means to aggregate returns and develop statistical price projections.

```python
symbol = 'TSLA'

end = datetime.datetime.today().date()
start = end - datetime.timedelta(3*365)

# get stock price data
S = pdr.data.DataReader(symbol, "yahoo", start, end)['Adj Close']
rlog = np.log(S/S.shift(1))

# plot data
plt.figure(figsize=(10,5))
plt.subplot(2,1,1)
S.plot(title=symbol)
plt.ylabel('Adjusted Close')
plt.grid()

plt.subplot(2,1,2)
rlin.plot()
plt.title('Log Returns (daily)')
plt.grid()
plt.tight_layout()
```

### Volatility Drag and the Relationship between Linear and Log Returns

For long-term financial decision making, it's important to understand the relationship between $r_t^{log}$ and $r_t^{lin}$. Algebraically, the relationships are simple.

$$r^{log}_t = \log \left(1+r^{lin}_t\right)$$

$$r^{lin}_t = e^{r^{log}_t} - 1$$

The linear return $r_t^{lin}$ is the fraction of value that is earned from an asset in a single period. It is a direct measure of earnings. The average value $\bar{r}^{lin}$ over many periods this gives the average fractional earnings per period. If you care about consuming the earnings from an asset and not about growth in value, then $\bar{r}^{lin}$ is the quantity of interest to you.

Log return $r_t^{log}$ is the rate of growth in value of an asset over a single period. When averaged over many periods, $\bar{r}^{log}$ measures the compounded rate of growth of value. If you care about the growth in value of an asset, then $\bar{r}^{log}$ is the quantity of interest to you.

The compounded rate of growth $r_t^{log}$ is generally smaller than average linear return $\bar{r}^{lin}$ due to the effects of volatility. To see this, consider an asset that has a linear return of -50% in period 1, and +100% in period 2. The average linear return is would be +25%, but the compounded growth in value would be 0%.

A general formula for the relationship between $\bar{r}^{log}$ and $\bar{r}^{lin}$ is derived as follows:

$$\begin{align*}
\bar{r}^{log} & = \frac{1}{T}\sum_{t=1}^{T} r_t^{log} \\
& = \frac{1}{T}\sum_{t=1}^{T} \log\left(1+r_t^{lin}\right) \\
& = \frac{1}{T}\sum_{t=1}^{T} \left(\log(1) + r_t^{lin} - \frac{1}{2} (r_t^{lin})^2 + \cdots
\right) \\
& = \frac{1}{T}\sum_{t=1}^{T} r_t^{lin} - \frac{1}{2}\frac{1}{T}\sum_{t=1}^{T} (r_t^{lin})^2 + \cdots \\
& = \bar{r}^{lin} - \frac{1}{2}\left(\frac{1}{T}\sum_{t=1}^{T} (r_t^{lin})^2\right) + \cdots \\
& = \bar{r}^{lin} - \frac{1}{2}\left((\bar{r}^{lin})^2 + \frac{1}{T}\sum_{t=1}^{T} (r_t^{lin}-\bar{r}^{lin})^2\right) + \cdots
\end{align*}$$

For typical values $\bar{r}^{lin}$ of and long horizons $T$, this results in a formula

$$\begin{align*}
\bar{r}^{log} & \approx \bar{r}^{lin} - \frac{1}{2} \left(\sigma^{lin}\right)^2
\end{align*}$$

where $\sigma^{lin}$ is the standard deviation of linear returns, more commonly called the volatility.

The difference $- \frac{1}{2} \left(\sigma^{lin}\right)^2$ is the _volatility drag_ imposed on the compounded growth in value of an asset due to volatility in linear returns. This can be significant and a source of confusion for many investors.

It's indeed possible to have a positive average linear return, but negative compounded growth.  To see this, consider a \$100 investment which earns 20% on even-numbered years, and loses 18% on odd-numbered years. The average linear return is 1%, and the average log return is -0.81%.

```python
symbol = 'TSLA'

end = datetime.datetime.today().date()
start = end - datetime.timedelta(3*365)

# get stock price data
S = pdr.data.DataReader(symbol, "yahoo", start,  end)['Adj Close']
rlin = (S - S.shift(1))/S.shift(1)
rlog = np.log(S/S.shift(1))

# plot data
plt.figure(figsize=(10,6))
plt.subplot(3,1,1)
S.plot(title=symbol)
plt.ylabel('Adjusted Close')
plt.grid()

plt.subplot(3,1,2)
rlin.plot()
plt.title('Linear Returns (daily)')
plt.grid()
plt.tight_layout()

plt.subplot(3,1,3)
rlog.plot()
plt.title('Log Returns (daily)')
plt.grid()
plt.tight_layout()
```

```python
print("Mean Linear Return (rlin) = {0:.7f}".format(rlin.mean()))
print("Linear Volatility (sigma) = {0:.7f}".format(rlin.std()))
print("Volatility Drag -0.5*sigma**2 = {0:.7f}".format(-0.5*rlin.std()**2))
print("rlin - 0.5*vol = {0:.7f}\n".format(rlin.mean() - 0.5*rlin.std()**2))

print("Mean Log Return = {0:.7f}".format(rlog.mean()))
```

    Mean Linear Return (rlin) = 0.0034779
    Linear Volatility (sigma) = 0.0422834
    Volatility Drag -0.5*sigma**2 = -0.0008939
    rlin - 0.5*vol = 0.0025840

    Mean Log Return = 0.0025842

```python
symbols = ['AAPL','MSFT','F','XOM','GE','X','TSLA','NIO']

end = datetime.datetime.today().date()
start = end -  datetime.timedelta(3*365)

rlin = []
rlog = []
sigma = []

for symbol in symbols:

    # get stock price data
    S = pdr.data.DataReader(symbol, "yahoo", start,  end)['Adj Close']
    r = (S - S.shift(1))/S.shift(1)
    rlin.append(r.mean())
    rlog.append((np.log(S/S.shift(1))).mean())
    sigma.append(r.std())

```

```python
import seaborn as sns
N = len(symbols)
idx = np.arange(N)
width = 0.2

plt.figure(figsize=(12, 6))

p0 = plt.bar(2*idx - 1.25*width, rlin, width)
p1 = plt.bar(2*idx, -0.5*np.array(sigma)**2, width, bottom=rlin)
p2 = plt.bar(2*idx + 1.25*width, rlog, width)

for k in range(0,N):
    plt.plot([2*k - 1.75*width, 2*k + 0.5*width], [rlin[k], rlin[k]], 'k', lw=1)
    plt.plot([2*k - 0.5*width, 2*k + 1.75*width], [rlog[k], rlog[k]], 'k', lw=1)

plt.xticks(2*idx, symbols)
plt.legend((p0[0], p1[0], p2[0]), ('rlin', '0.5*sigma**2', 'rlog'))
plt.title('Components of Linear Return')
plt.ylim(1.1*np.array(plt.ylim()))
plt.grid()
```

```python

```

< [Risk and Diversification](http://nbviewer.jupyter.org/github/jckantor/CBE40455/blob/master/notebooks/07.00-Risk-and-Diversification.ipynb) | [Contents](toc.ipynb) | [Geometric Brownian Motion](http://nbviewer.jupyter.org/github/jckantor/CBE40455/blob/master/notebooks/07.02-Geometric-Brownian-Motion.ipynb) >


```python
%%HTML
<!-- Mejorar visualización en proyector -->
<style>
.rendered_html {font-size: 1.2em; line-height: 150%;}
div.prompt {min-width: 0ex; padding: 0px;}
.container {width:95% !important;}
</style>
```

<!-- Mejorar visualización en proyector -->
<style>
.rendered_html {font-size: 1.2em; line-height: 150%;}
div.prompt {min-width: 0ex; padding: 0px;}
.container {width:95% !important;}
</style>

```python
%autosave 0
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt
from IPython.display import display
import ipywidgets as widgets
from matplotlib import animation
from functools import partial
slider_layout = widgets.Layout(width='600px', height='20px')
slider_style = {'description_width': 'initial'}
IntSlider_nice = partial(widgets.IntSlider, style=slider_style, layout=slider_layout, continuous_update=False)
FloatSlider_nice = partial(widgets.FloatSlider, style=slider_style, layout=slider_layout, continuous_update=False)
SelSlider_nice = partial(widgets.SelectionSlider, style=slider_style, layout=slider_layout, continuous_update=False)
```

    Autosave disabled

# Aprendizaje Supervisado

Esquema donde se busca aprender un mapeo o función

$$
f_{\theta}: \mathcal{X} \rightarrow \mathcal{Y},
$$

donde $\mathcal{X}$ es el dominio de nuestros datos (entrada) e $\mathcal{Y}$ es un objetivo (salida)

Entrenamos nuestro modelo a partir de un conjunto de $N$ ejemplos:

$$
\{(x_1, y_1), (x_2, y_2), \ldots, (x_N, y_N)\},
$$

donde cada ejemplo es una tupla formada de datos $x_i \in \mathcal{X}$ y objetivo $y_i \in \mathcal{Y}$

Si la variable objetivo es
- continua: hablamos de un problema de regresión o aproximación de funciones
- categórica: hablamos de un problema de clasificación

La naturaleza de los datos  depende del problema

Lo más común es que los datos $x_i$ se estructuren como arreglos de $M$ componentes

A los componentes los llamamos atributos o *features*

### Aprendizaje

El vector $\theta$ corresponde a los **parámetros** del modelo

> Aprender o ajustar el modelo corresponde a encontrar el valor "óptimo" de $\theta$

Usamos una **función de pérdida/costo** $L(\theta)$ para medir el error de nuestro modelo

> **Minimizamos** la función de costo para encontrar el mejor $\theta$

Cuando hablamos de $\theta$ "óptimo" lo decimos en el sentido de una función de costo particular

### Optimización

Queremos resolver el siguiente problema
$$
\min_\theta L(\theta)
$$

Una opción es evaluar $L()$ en todo el espacio de posibles $\theta$: Fuerza Bruta

Pero en general esto no es computacionalmente posible

Podemos usar **técnicas de optimización** para encontrar el mejor $\theta$

Si $L(\theta)$ es continua y derivable podemos escribir

$$
\nabla_\theta L(\theta) = \vec 0
$$

e intentar despejar $\theta~$

## Regresión lineal

Modelo para aprender una mapeo entre una o más variables continuas (atributos) hacia una variable continua (objetivo)

En un esquema con $M$ atributos y $N$ ejemplos tenemos

$$
y_i = f_\theta(\vec x_i) = \vec w^T \vec x_i + b = \sum_{j=1}^M w_j x_{ij} + b,
$$

donde $f_\theta$ es un modelo parámetrico (hiperplano) con $M+1$ parámetros

$$
\theta= \begin{pmatrix} b \\ w_1 \\ w_2 \\ \vdots \\ w_M \end{pmatrix}
$$

También podemos escribir el sistema matricialmente como

$$
Y = X \theta
$$

donde $Y= \begin{pmatrix} y_1 \\ y_2 \\ \vdots \\y_N\end{pmatrix} \in \mathbb{R}^N$, $X = \begin{pmatrix} 1 & x_{11} & x_{12}& \ldots& x_{1M} \\ \vdots & \vdots & \vdots& \ddots& \vdots \\ 1 & x_{N1} & x_{N2}& \ldots& x_{NM} \\ \end{pmatrix} \in \mathbb{R}^{N\times M}$ y $\theta \in \mathbb{R}^M$

Una función de costo razonable es

$$
L(\theta) = \frac{1}{2} (Y - X\theta)^T (Y - X\theta)
$$

que corresponde al cuadrado de los errores

Luego si derivamos e igualamos a cero obtenemos

$$
\nabla_\theta L(\theta) = -X^T(Y-X\theta) = 0,
$$

y despejando

$$
\hat \theta = (X^T X)^{-1} X^T Y
$$

siempre y cuando podamos invertir $X^T X$

> Esto se conoce como solución de mínimos cuadrados

#### Funciones base

Podemos generalizar el regresor lineal aplicando transformaciones a $X$

Por ejemplo una regresión polinomial de grado $M$ sería

$$
y_i = f_\theta(x_i) = \sum_{j=1}^M w_j x_{i}^j + b,
$$

y su solución sería

$$
\hat \theta = (\Phi^T \Phi)^{-1} \Phi^T Y,
$$

donde $\Phi = \begin{pmatrix} 1 & x_1 & x_1^2& \ldots& x_1^M \\ \vdots & \vdots & \vdots& \ddots& \vdots \\ 1 & x_N & x_N^2& \ldots& x_N^M \\ \end{pmatrix}$

```python
plt.close('all'); fig, ax = plt.subplots(figsize=(6, 4), tight_layout=True)
poly_basis = lambda x,N : np.vstack([x**k for k in range(N)]).T
theta = [10, -2, -0.3, 0.1]
x = np.linspace(-5, 6, num=21);
X = poly_basis(x, len(theta))
y = np.dot(X, theta)

rseed, sigma = 0, 1.
np.random.seed(rseed);
Y = y + sigma*np.random.randn(len(x))
P = np.random.permutation(len(x))
train_idx, valid_idx = P[:len(x)//2], P[len(x)//2:]

def update_plot(ax, N):
    ax.cla();
    Phi = poly_basis(x, N)
    theta_hat = np.linalg.lstsq(Phi[train_idx, :], Y[train_idx], rcond=None)[0]
    ax.plot(x, y, 'g-', linewidth=2, label='Implícito', alpha=0.6, zorder=-100)
    ax.scatter(x[train_idx], Y[train_idx], s=50, label='Entrenamiento')
    ax.scatter(x[valid_idx], Y[valid_idx], s=50, label='Validación')
    ax.vlines(x[train_idx], np.dot(Phi[train_idx, :], theta_hat), Y[train_idx])
    ax.vlines(x[valid_idx], np.dot(Phi[valid_idx, :], theta_hat), Y[valid_idx])
    x_plot = np.linspace(-5, 6, num=100);
    ax.plot(x_plot, np.dot(poly_basis(x_plot, N), theta_hat), 'k-', linewidth=2, label='Modelo')
    ax.set_ylim([-5, 15]); plt.legend()

widgets.interact(update_plot, ax=widgets.fixed(ax), N=IntSlider_nice(min=1, max=11));
```

    <IPython.core.display.Javascript object>

    interactive(children=(IntSlider(value=1, continuous_update=False, description='N', layout=Layout(height='20px'…

# Complejidad y Sobreajuste

En el ejemplo anterior vimos que se puede obtener modelos más flexibles si aumentamos el grado del polinomio

> Aumentar la cantidad de parámetros (grados de libertad) hace al modelo más flexible y más complejo

Si la flexibilidad es excesiva aproximamos los datos con cero error

Esto no es bueno ya que estamos aprendiendo "de memoria" los datos y ajustandonos al ruido

> Sobreajuste: Aprender perfectamente los datos usados para entrenar

El modelo sobreajustado predice muy mal los datos "que no ha visto"

> El sobreajuste es inversamente proporcional a la capacidad de generalización

Por esta razón usamos conjuntos de validación

Figura: https://www.d2l.ai/chapter_multilayer-perceptrons/underfit-overfit.html

# Representatividad y Validación

El primer paso para entrenar nuestro modelo es obtener datos (duh)

Es crítico que los datos que utilizemos **representen** adecuadamente el problema que queremos resolver

Sea un espacio de datos (circulo negro) y muestras (puntos azules), ¿que puede decir de los siguientes casos?

```python
from matplotlib.patches import Circle
fig, ax = plt.subplots(1, 2, figsize=(7, 3))
np.random.seed(19)
for ax_ in ax:
    ax_.axis('off')
    ax_.set_xlim([-3, 3])
    ax_.set_ylim([-3, 3])
r = 2.5*np.random.rand(100); t = np.random.rand(100);
ax[0].scatter(r*np.cos(2.0*np.pi*t), r*np.sin(2.0*np.pi*t))
p = Circle((0, 0), 3, fill=False, ec='k')
ax[0].add_artist(p)
r = 2.5*np.random.rand(100); t = np.random.rand(100);
ax[1].scatter(r*np.cos(t), r*np.sin(t))
p = Circle((0, 0), 3, fill=False, ec='k')
ax[1].add_artist(p);
```

    <IPython.core.display.Javascript object>

Siempre que podamos controlar el proceso de muestreo debemos poner atención a evitar **sesgos**

Asumiendo que nuestro dataset es representativo el siguiente paso es **entrenar**

Para combatir el sobreajuste podemos usar **estrategias de validación**

Consisten en separar el conjunto en dos o más subconjuntos
- Holdout: Entrenamiento/Validación/Prueba
- K-fold cross-validation y Leave one-out (N-fold) cross-validation
- Versiones estratificadas/balanceadas de las anteriores

Para que nuestro conjuntos de entrenamiento y validación sigan siendo representativos del total los **seleccionamos aleatoriamente**

Medimos $L(\theta)$ en entrenamiento y validación

> Optimizamos nuestro modelo minimizando el error de entrenamiento

> Seleccionamos los parámetros e hiper-parámetros que dan mínimo error de validación

> Comparamos distintas familias de modelos  con el error de prueba

En el ejemplo anterior:

```python
fig, ax = plt.subplots(figsize=(6, 4), tight_layout=True)
N_values = np.arange(1, 11)
mse = np.zeros(shape=(len(N_values), 2))
for i, N in enumerate(N_values):
    Phi = poly_basis(x, N)
    theta_hat = np.linalg.lstsq(Phi[train_idx, :], Y[train_idx], rcond=None)[0]
    mse[i, 0] = np.mean(np.power(Y[train_idx] - np.dot(Phi[train_idx, :], theta_hat), 2))
    mse[i, 1] = np.mean(np.power(Y[valid_idx] - np.dot(Phi[valid_idx, :], theta_hat), 2))
ax.plot(N_values, mse[:, 0], label='Entrenamiento')
ax.plot(N_values, mse[:, 1], label='Validación')
idx_best = np.argmin(mse[:, 1])
ax.scatter(N_values[idx_best], mse[idx_best, 1], c='k', s=100)
plt.legend()
ax.set_ylim([1e-4, 1e+4])
ax.set_yscale('log')
ax.set_xlabel('Grado del polinomio')
ax.set_ylabel('Loss');
```

    <IPython.core.display.Javascript object>

En resumen

- Bajo error de entrenamiento y de validación: **Ideal**

- Bajo error de entrenamiento y alto error de validación: **Modelo sobreajustado**

- Alto error de entrenamiento y de validación: Considera otro modelo y/o revisa tu código

## Neurona artificial o regresor logístico

Modelo para aprender una mapeo entre una o más variables continuas (atributos) hacia una variable binaria (objetivo)

$$
y_i \leftarrow f_\theta(\vec x_i) = \mathcal{S} \left(\theta_0 + \sum_{j=1}^M \theta_j x_{ij}\right)
$$

donde $\mathcal{S}(z) = \frac{1}{1+\exp(-z)} \in [0, 1]$ se conoce como función logística o sigmoide

> Modelo de clasificación binaria (dos clases)

Podemos interpretar la salida del clasificador como una probabilidad

```python

```

```python
data = np.concatenate((np.random.randn(50, 2), 1.5 + np.random.randn(50, 2)), axis=0)
label = np.array([0]*50 + [1]*50)
fig, ax = plt.subplots(1, figsize=(9, 4))
from matplotlib import cm
#fig.colorbar(cm.ScalarMappable(cmap=plt.cm.RdBu_r), ax=ax)
x_min, x_max = data[:, 0].min() - 0.5, data[:, 0].max() + 0.5
y_min, y_max = data[:, 1].min() - 0.5, data[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.05), np.arange(y_min, y_max, 0.05))

def sigmoid(X, w, b):
    Z = np.dot(X, w) + b
    return 1./(1 + np.exp(-Z))

def update_plot(w1, w2, b):
    ax.cla()
    ax.scatter(data[:50, 0], data[:50, 1], c='k', s=20)
    ax.scatter(data[50:, 0], data[50:, 1], c='k', s=20, marker='x')
    ax.contourf(xx, yy, sigmoid(np.c_[xx.ravel(), yy.ravel()], np.array([w1, w2]), b).reshape(xx.shape),
                cmap=plt.cm.RdBu_r, alpha=0.75)

widgets.interact(update_plot,
                 w1=FloatSlider_nice(min=-10, max=10),
                 w2=FloatSlider_nice(min=-10, max=10),
                 b=FloatSlider_nice(min=-10, max=10));
```

    <IPython.core.display.Javascript object>

    interactive(children=(FloatSlider(value=0.0, continuous_update=False, description='w1', layout=Layout(height='…

¿Qué función de costo es apropiada en este caso?

Tipicamente se usa la **Entropía Cruzada Binaria**

$$
L(\theta) = \sum_{i=1}^N  -y_i \log( f_\theta(\vec x_i) ) - (1-y_i) \log(1 - f_\theta(\vec x_i))
$$

¿Por qué?

Calculemos su gradiente
$$
\begin{align}
\frac{d}{d \theta_j} L(\theta) &= \sum_{i=1}^N  \left(-\frac{y_i}{f_\theta(\vec x_i)} + \frac{1-y_i}{1 - f_\theta(\vec x_i)}\right) \frac{f_\theta(\vec x_i)}{d\theta_j} \nonumber \\
&= (y_i - f_\theta(\vec x_i)) x_{ij}
\end{align}
$$

Por la no linealidad $f_\theta(\vec x_i)$ ya no podemos despejar analiticamente $\theta$

> Podemos usar métodos de optimización iterativos

## Optimización: Método de Newton

Sea el valor actual del vector de parámetros $\theta_t$

Queremos encontrar el mejor "próximo valor" según nuestra función objetivo
$$
\theta_{t+1} = \theta_t + \Delta \theta
$$
Consideremos la aproximación de Taylor de segundo orden de $f$
$$
f(\theta_{t} + \Delta \theta) \approx f(\theta_t) + \nabla f (\theta_t) \Delta \theta + \frac{1}{2} \Delta \theta^T H_f (\theta_t) \Delta \theta
$$
Derivando en función de $\Delta \theta$ e igualando a cero tenemos
$$
\begin{align}
\nabla f (\theta_t)  +  H_f (\theta_t) \Delta \theta &= 0 \nonumber \\
\Delta \theta &= - [H_f (\theta_t)]^{-1}\nabla f (\theta_t)  \nonumber \\
\theta_{t+1} &= \theta_{t} - [H_f (\theta_t)]^{-1}\nabla f (\theta_t)  \nonumber \\
\end{align}
$$

- Se obtiene una regla iterativa en función del **Gradiente** y del **Hessiano**
- La solución depende de $\theta_0$
- "Asumimos" que la aproximación de segundo orden es "buena"
- Si nuestro modelo tiene $N$ parámetros el Hessiano es de $N\times N$, ¿Qué pasa si $N$ es grande?

## Optimización: Gradiente descendente

Si el Hessiano es prohibitivo podemos usar una aproximación de primer orden

El método más clásico es el **gradiente descendente**
$$
\theta_{t+1} = \theta_{t} - \eta \nabla f (\theta_t)
$$

donde hemos reemplazado el Hessiano por una constante $\eta$ llamado "paso" o "tasa de aprendizaje"

- ¿Cómo cambia la optimización con distintos $\eta$?
- ¿Qué ocurre cuando la superficie de error tiene mínimos locales?

```python
plt.close('all'); fig, ax = plt.subplots(2, figsize=(7, 4), tight_layout=True, sharex=True)
x = np.linspace(-4, 6, num=100)
f = lambda theta : 5+ (theta-1.)**2 #+ 10*np.sin(theta)
df = lambda theta : 2*(theta -1.) #+ 10*np.cos(theta)
df2 = lambda theta : 2 #- 10*np.cos(theta)

t = 10*np.random.rand(10) - 4.
ax[0].plot(x, f(x))
sc = ax[0].scatter(t, f(t), s=100)

ax[1].set_xlabel(r'$\theta$')
ax[0].set_ylabel(r'$f(\theta)$')
ax[1].plot(x, -df(x))
ax[1].set_ylabel(r'$-\nabla f(\theta)$')
eta = 0.01

def update(n):
    t = sc.get_offsets()[:, 0]
    t -= eta*df(t)
    #t -= df(t)/(df2(t)+10)
    sc.set_offsets(np.c_[t, f(t)])

anim = animation.FuncAnimation(fig, update, frames=100, interval=200, repeat=False, blit=True)
```

    <IPython.core.display.Javascript object>

# Neurona artificial en [PyTorch](https://pytorch.org/)

```python
import torch
from torch.utils.data import DataLoader, TensorDataset, Subset

torch_set = TensorDataset(torch.from_numpy(data.astype('float32')),
                          torch.from_numpy(label.astype('float32')))

import sklearn.model_selection
train_idx, test_idx = next(sklearn.model_selection.ShuffleSplit(train_size=0.6).split(data, label))
torch_train_loader = DataLoader(Subset(torch_set, train_idx), shuffle=True, batch_size=16)
torch_valid_loader = DataLoader(Subset(torch_set, test_idx), shuffle=False, batch_size=256)
```

    /home/hackerter/.local/lib/python3.6/site-packages/sklearn/model_selection/_split.py:1788: FutureWarning: From version 0.21, test_size will always complement train_size unless both are specified.
      FutureWarning)

```python
class Neurona(torch.nn.Module):

    def __init__(self):

        super(Neurona, self).__init__()

        self.fc = torch.nn.Linear(in_features=2,  out_features=1, bias=True)
        self.activation = torch.nn.Sigmoid()

    def forward(self, x):
        return self.activation(self.fc(x))
```

```python
fig, ax = plt.subplots(1, 2, figsize=(8, 3.5), tight_layout=True)

net = Neurona()
n_epochs = 1000
optimizer = torch.optim.SGD(net.parameters(), lr=1e-2)
criterion = torch.nn.BCELoss(reduction='sum')
running_loss = np.zeros(shape=(n_epochs, 2))

def train_one_epoch(net):
    train_loss, valid_loss = 0.0, 0.0
    for sample_data, sample_label in torch_train_loader:
        output = net(sample_data)
        optimizer.zero_grad()
        loss = criterion(output, sample_label)
        train_loss += loss.item()
        loss.backward()
        optimizer.step()
    for sample_data, sample_label in torch_valid_loader:
        output = net(sample_data)
        loss = criterion(output, sample_label)
        valid_loss += loss.item()
    return train_loss/torch_train_loader.dataset.__len__(), valid_loss/torch_valid_loader.dataset.__len__()

def update_plot(k):
    global net, running_loss
    [ax_.cla() for ax_ in ax]
    running_loss[k, 0], running_loss[k, 1] = train_one_epoch(net)
    Z = net.forward(torch.from_numpy(np.c_[xx.ravel(), yy.ravel()].astype('float32')))
    Z = Z.detach().numpy().reshape(xx.shape)
    ax[0].contourf(xx, yy, Z, cmap=plt.cm.RdBu_r, alpha=1., vmin=0, vmax=1)
    for i, (marker, name) in enumerate(zip(['o', 'x'], ['Train', 'Test'])):
        ax[0].scatter(data[label==i, 0], data[label==i, 1], color='k', s=10, marker=marker, alpha=0.5)
        ax[1].plot(np.arange(0, k+1, step=1), running_loss[:k+1, i], '-', label=name+" cost")
    plt.legend(); ax[1].grid()

#update_plot(0)
anim = animation.FuncAnimation(fig, update_plot, frames=n_epochs,
                               interval=10, repeat=False, blit=False)
```

    <IPython.core.display.Javascript object>

# Métricas: Evaluando un clasificador binario

La salida de este clasificador es un valor en el rango $[0, 1]$

Para tomar un decisión binaria se debe seleccionar un umbral $\mathcal{T}$ tal que

$$
d_i =
\begin{cases}
0, & \text{si } f_\theta(\vec x_i)  < \mathcal{T} \\
1, & \text{si } f_\theta(\vec x_i) \geq \mathcal{T}
\end{cases}
$$

Una vez seleccionado el umbral se puede contar la cantidad de
- **True positives** (TP): Era clase (1) y lo clasifico como (1)
- **True negative** (TN): Era clase (0) y lo clasifico como (0)
- **False positives** (FP): Era clase (0) y lo clasifico como (1): Error tipo I
- **False negative** (FN): Era clase (1) y lo clasifico como (0): Error tipo II

A partir de estas métricas se construye la **tabla de confusión** del clasificador

|Clasificado como/En realidad era|Positivo|Negativo|
|---|---|---|
|Positivo:|TP | FP |
|Negativo:| FN | TN |

En base a estas métricas se construyen otras
$$
\text{Recall} = \frac{TP}{TP + FN}
$$
también conocida como la **Tasa de verdaderos positivos** (TPR) o sensitividad

> TPR: La proporción de positivos correctamente clasificados respecto al total de positivos

$$
\text{FPR} = \frac{FP}{TN + FP} = 1 - \frac{TN}{TN + FP}
$$

la **tasa de falsos positivos** (FPR) también representada como "1 - especificidad"

> FPR: La proporción de negativos incorrectamente clasificados respecto al total de negativos

$$
\text{Precision} = \frac{TP}{TP + FP}
$$

también conocido como pureza

> Precision: La proporción de positivos correctamente clasificados respecto a todos los ejemplos clasificados como positivo

$$
\text{Accuracy} = \frac{TP+TN}{TP + FP + FN+ TN}
$$

> Accuracy: La proporción de ejemplos correctamente clasificados

$$
\text{f1-score} = \frac{2*\text{Recall}*\text{Precision}}{\text{Recall} + \text{Precision}}
$$

> f1-score: Media armónica entre Recall y Precision asumiendo igual ponderación

Si las clases son desbalanceadas entonces f1-score es más aconsejable que accuracy

```python
net = Neurona()
probability = net(torch_set.tensors[0]).detach().numpy()
import sklearn.metrics

print("Matriz de confusión:")
print(sklearn.metrics.confusion_matrix(y_true=torch_set.tensors[1].numpy().astype(int),
                                       y_pred=probability[:, 0] > 0.5))

print(sklearn.metrics.classification_report(y_true=torch_set.tensors[1].numpy().astype(int),
                                      y_pred=probability[:, 0] > 0.5))
```

    Matriz de confusión:
    [[40 10]
     [ 9 41]]
                  precision    recall  f1-score   support

               0       0.82      0.80      0.81        50
               1       0.80      0.82      0.81        50

       micro avg       0.81      0.81      0.81       100
       macro avg       0.81      0.81      0.81       100
    weighted avg       0.81      0.81      0.81       100

Notar que a distintos umbrales $\mathcal{T}$ se obtienen distintas tablas de confusión

Se midemos estas métricas usando distintos umbrales podemos construir una curva de desempeño

Tipicamente se usan
- Curva ROC: TPR vs FPR
- Curva Precision vs Recall

```python
fpr, tpr, th = sklearn.metrics.roc_curve(y_true=torch_set.tensors[1].numpy().astype(int),
                          y_score=probability[:, 0])

fig, ax = plt.subplots(figsize=(7, 4))
ax.plot(fpr, tpr);
ax.set_xlabel('Tasa de Falsos positivos')
ax.set_ylabel('Tasa de Verdaderos Positivos (Recall)')
ax.set_title('AUC: %f' %sklearn.metrics.auc(fpr, tpr))
```

    <IPython.core.display.Javascript object>

    Text(0.5, 1.0, 'AUC: 0.866000')

```python
prec, rec, th = sklearn.metrics.precision_recall_curve(y_true=torch_set.tensors[1].numpy().astype(int),
                                                       probas_pred=probability[:, 0])

fig, ax = plt.subplots(figsize=(7, 4))
ax.plot(rec, prec, '-');
ax.set_xlabel('Recall')
ax.set_ylabel('Precision')
```

    <IPython.core.display.Javascript object>

    Text(0, 0.5, 'Precision')

# Ojo con:

#### La distribución de los datos donde se aplicará el clasificador es distinta a la que usaste para entrenar/validar

Actualiza tus conjuntos de datos para que sean representativos!

#### Usa los subconjuntos adecuadamente

Ajusta los parámetros con el set de validación

Compara distintas familias de modelos con el set de prueba

#### La métrica que usas no es la adecuada para el problema

Si el problema tiene clases desbalanceadas el *accuracy* puede ser muy alto, contrasta usando métricas sencibles al desbalance (e.g. *f1-score*)

#### Mi modelo se sobreajusta de inmediato

Prueba disminuyendo la complejidad/arquitectura del modelo o añadiendo **regularización**

Esto también puede ser señal de que necesitas más ejemplos para entrenar

Se pueden usar **Técnicas de aumentación** de datos

#### Mi modelo no está aprendiendo

Si estas seguro que no hay bugs prueba aumentando la complejidad del modelo

#### Estudia los errores de tu modelo para mejorarlo

Analiza los datos mal clasificados y busca patrones

Revisa que las etiquetas estén correctas

Revisa que los atributos estén adecuadamente calculados

Propon nuevos atributos que ayuden a clasificador los ejemplos difíciles

```python

```


```python
"""Tutorial: Introduction to the Spin-Orbital Formulation of Post-HF Methods"""

__author__    = "Adam S. Abbott"
__credit__    = ["Adam S. Abbott", "Justin M. Turney"]

__copyright__ = "(c) 2014-2017, The Psi4NumPy Developers"
__license__   = "BSD-3-Clause"
__date__      = "2017-05-23"
```

# Introduction to the Spin Orbital Formulation of Post-HF Methods
## Notation

Post-HF methods such as MPn, coupled cluster theory, and configuration interaction improve the accuracy of our Hartree-Fock wavefunction by including terms corresponding to excitations of electrons from occupied (i, j, k..) to virtual (a, b, c...) orbitals. This recovers some of the dynamic electron correlation previously neglected by Hartree-Fock.

It is convenient to introduce new notation to succinctly express the complex mathematical expressions encountered in these methods. This tutorial will cover this notation and apply it to a spin orbital formulation of conventional MP2. This code will also serve as a starting template for other tutorials which use a spin-orbital formulation, such as CEPA0, CCD, CIS, and OMP2.

### I. Physicist's Notation for Two-Electron Integrals
Recall from previous tutorials the form for the two-electron integrals over spin orbitals ($\chi$) and spatial orbitals ($\phi$):
\begin{equation}
 [pq|rs] = [\chi_p\chi_q|\chi_r\chi_s] = \int dx_{1}dx_2 \space \chi^*_p(x_1)\chi_q(x_1)\frac{1}{r_{12}}\chi^*_r(x_2)\chi_s(x_2) \\
(pq|rs) = (\phi_p\phi_q|\phi_r\phi_s) = \int dx_{1}dx_2 \space \phi^*_p(x_1)\phi_q(x_1)\frac{1}{r_{12}}\phi^*_r(x_2)\phi_s(x_2)
\end{equation}

Another form of the spin orbital two electron integrals is known as physicist's notation. By grouping the complex conjugates on the left side, we may express them in Dirac ("bra-ket") notation:
\begin{equation}
\langle pq \mid rs \rangle = \langle \chi_p \chi_q \mid \chi_r \chi_s \rangle = \int dx_{1}dx_2 \space \chi^*_p(x_1)\chi^*_q(x_2)\frac{1} {r_{12}}\chi_r(x_1)\chi_s(x_2)
\end{equation}

The antisymmetric form of the two-electron integrals in physcist's notation is given by

\begin{equation}
\langle pq \mid\mid rs \rangle = \langle pq \mid rs \rangle - \langle pq \mid sr \rangle
\end{equation}

### II. Kutzelnigg-Mukherjee Tensor Notation and the Einstein Summation Convention

Kutzelnigg-Mukherjee (KM) notation provides an easy way to express and manipulate the tensors (two-electron integrals, $t$-amplitudes, CI coefficients, etc.) encountered in post-HF methods. Indices which appear in the bra are expressed as subscripts, and indices which appear in the ket are expressed as superscripts:
\begin{equation}
g_{pq}^{rs} = \langle pq \mid rs \rangle \quad \quad \quad \overline{g}_{pq}^{rs} = \langle pq \mid\mid rs \rangle
\end{equation}

The upper and lower indices allow the use of the Einstein Summation convention. Under this convention, whenever an indice appears in both the upper and lower position in a product, that indice is implicitly summed over. As an example, consider the MP2 energy expression:

\begin{equation}
E_{MP2} = \frac{1}{4} \sum_{i a j b} \frac{ [ia \mid\mid jb] [ia \mid\mid jb]} {\epsilon_i - \epsilon_a + \epsilon_j - \epsilon_b}
\end{equation}
Converting to physicist's notation:

\begin{equation}
E_{MP2} = \frac{1}{4} \sum_{i j a b} \frac{ \langle ij \mid\mid ab \rangle \langle ij \mid \mid ab \rangle} {\epsilon_i - \epsilon_a + \epsilon_j - \epsilon_b}
\end{equation}
KM Notation, taking advantage of the permutational symmetry of $g$:
\begin{equation}
E_{MP2} = \frac{1}{4} \overline{g}_{ab}^{ij} \overline{g}_{ij}^{ab} (\mathcal{E}_{ab}^{ij})^{-1}
\end{equation}

where $\mathcal{E}_{ab}^{ij}$ is the sum of orbital energies $\epsilon_i - \epsilon_a + \epsilon_j - \epsilon_b$. Upon collecting every possible orbital energy sum into a 4-dimensional tensor, this equation can be solved with a simple tensor-contraction, as done in our MP2 tutorial.

The notation simplication here is minor, but the value of this notation becomes obvious with more complicated expressions encountered in later tutorials such as CCD. It is also worth noting that KM notation is deeply intertwined with the second quantization and diagrammatic expressions of methods in advanced electronic structure theory. For our purposes, we will shy away from the details and simply use the notation to write out readily-programmable expressions.

### III. Coding Spin Orbital Methods Example: MP2

In the MP2 tutorial, we used spatial orbitals in our two-electron integral tensor, and this appreciably decreased the computational cost. However, this code will only work when using an RHF reference wavefunction. We may generalize our MP2 code (and other post-HF methods) to work with any reference by expressing our integrals, MO coefficients, and orbital energies obtained from Hartree-Fock in a spin orbital formulation. As an example, we will code spin orbital MP2, and this will serve as a foundation for later tutorials.

### Implementation of Spin Orbital MP2
As usual, we import Psi4 and NumPy, and set the appropriate options. However, in this code, we will be free to choose open-shell molecules which require UHF or ROHF references. We will stick to RHF and water for now.

```python
# ==> Import statements & Global Options <==
import psi4
import numpy as np

psi4.set_memory(int(2e9))
numpy_memory = 2
psi4.core.set_output_file('output.dat', False)
```

```python
# ==> Molecule & Psi4 Options Definitions <==
mol = psi4.geometry("""
0 1
O
H 1 1.1
H 1 1.1 2 104
symmetry c1
""")

psi4.set_options({'basis':        '6-31g',
                  'scf_type':     'pk',
                  'reference':    'rhf',
                  'mp2_type':     'conv',
                  'e_convergence': 1e-8,
                  'd_convergence': 1e-8})
```

For convenience, we let Psi4 take care of the Hartree-Fock procedure, and return the wavefunction object.

```python
# Get the SCF wavefunction & energies
scf_e, scf_wfn = psi4.energy('scf', return_wfn=True)
```

We also need information about the basis set and orbitals, such as the number of basis functions, number of spin orbitals, number of alpha and beta electrons, the number of occupied spin orbitals, and the number of virtual spin orbitals. These can be obtained with MintsHelper and from the wavefunction.

```python
mints = psi4.core.MintsHelper(scf_wfn.basisset())
nbf = mints.nbf()
nso = 2 * nbf
nalpha = scf_wfn.nalpha()
nbeta = scf_wfn.nbeta()
nocc = nalpha + nbeta
nvirt = 2 * nbf - nocc
```

For MP2, we need the MO coefficients, the two-electron integral tensor, and the orbital energies. But, since we are using spin orbitals, we have to manipulate this data accordingly. Let's get our MO coefficients in the proper form first. Recall in restricted Hartree-Fock, we obtain one MO coefficient matrix **C**, whose columns are the molecular orbital coefficients, and each row corresponds to a different atomic orbital basis function. But, in unrestricted Hartree-Fock, we obtain separate matrices for the alpha and beta spins, **Ca** and **Cb**. We need a general way to build one **C** matrix regardless of our Hartree-Fock reference. The solution is to put alpha and beta MO coefficients into a block diagonal form:

```python
Ca = np.asarray(scf_wfn.Ca())
Cb = np.asarray(scf_wfn.Cb())
C = np.block([
             [      Ca           ,   np.zeros_like(Cb) ],
             [np.zeros_like(Ca)  ,          Cb         ]
            ])

# Result: | Ca  0 |
#         | 0   Cb|

```

It's worth noting that for RHF and ROHF, the Ca and Cb given by Psi4 are the same.

Now, for this version of MP2, we also need the MO-transformed two-electron integral tensor in physicist's notation. However, Psi4's default two-electron integral tensor is in the AO-basis, is not "spin-blocked" (like **C**, above!), and is in chemist's notation, so we have a bit of work to do.

First, we will spin-block the two electron integral tensor in the same way that we spin-blocked our MO coefficients above. Unfortunately, this transformation is impossible to visualize for a 4-dimensional array.

Nevertheless, the math generalizes and can easily be achieved with NumPy's kronecker product function `np.kron`. Here, we take the 2x2 identity, and place the two electron integral array into the space of the 1's along the diagonal. Then, we transpose the result and do the same. The result doubles the size of each dimension, and we obtain a "spin-blocked" two electron integral array.

```python
# Get the two electron integrals using MintsHelper
I = np.asarray(mints.ao_eri())

def spin_block_tei(I):
    """
    Function that spin blocks two-electron integrals
    Using np.kron, we project I into the space of the 2x2 identity, tranpose the result
    and project into the space of the 2x2 identity again. This doubles the size of each axis.
    The result is our two electron integral tensor in the spin orbital form.
    """
    identity = np.eye(2)
    I = np.kron(identity, I)
    return np.kron(identity, I.T)

# Spin-block the two electron integral array
I_spinblock = spin_block_tei(I)

```

From here, converting to antisymmetrized physicists notation is simply:

```python
# Converts chemist's notation to physicist's notation, and antisymmetrize
# (pq | rs) ---> <pr | qs>
# Physicist's notation
tmp = I_spinblock.transpose(0, 2, 1, 3)
# Antisymmetrize:
# <pr||qs> = <pr | qs> - <pr | sq>
gao = tmp - tmp.transpose(0, 1, 3, 2)

```

We also need the orbital energies, and just as with the MO coefficients, we combine alpha and beta together. We also want to ensure that the columns of **C** are sorted in the same order as the corresponding orbital energies.

```python
# Get orbital energies
eps_a = np.asarray(scf_wfn.epsilon_a())
eps_b = np.asarray(scf_wfn.epsilon_b())
eps = np.append(eps_a, eps_b)

# Before sorting the orbital energies, we can use their current arrangement to sort the columns
# of C. Currently, each element i of eps corresponds to the column i of C, but we want both
# eps and columns of C to be in increasing order of orbital energies

# Sort the columns of C according to the order of increasing orbital energies
C = C[:, eps.argsort()]

# Sort orbital energies in increasing order
eps = np.sort(eps)

```

Finally, we transform our two-electron integrals to the MO basis. For the sake of generalizing for other methods, instead of just transforming the MP2 relevant subsection as before:
~~~python
tmp = np.einsum('pi,pqrs->iqrs', Cocc, I)
tmp = np.einsum('qa,iqrs->iars', Cvirt, tmp)
tmp = np.einsum('iars,rj->iajs', tmp, Cocc)
I_mo = np.einsum('iajs,sb->iajb', tmp, Cvirt)
~~~

we instead transform the full array so it can be used for terms from methods other than MP2. The nested `einsum`'s work the same way as the method above. Here, we denote the integrals as `gmo` to differentiate from the chemist's notation integrals `I_mo`.

```python
# Transform gao, which is the spin-blocked 4d array of physicist's notation,
# antisymmetric two-electron integrals, into the MO basis using MO coefficients
gmo = np.einsum('pQRS, pP -> PQRS',
      np.einsum('pqRS, qQ -> pQRS',
      np.einsum('pqrS, rR -> pqRS',
      np.einsum('pqrs, sS -> pqrS', gao, C), C), C), C)

```

And just as before, construct the 4-dimensional array of orbital energy denominators. An alternative to the old method:
~~~python
e_ij = eps[:nocc]
e_ab = eps[nocc:]
e_denom = 1 / (e_ij.reshape(-1, 1, 1, 1) - e_ab.reshape(-1, 1, 1) + e_ij.reshape(-1, 1) - e_ab)
~~~
is the following:

```python
# Define slices, create 4 dimensional orbital energy denominator tensor
n = np.newaxis
o = slice(None, nocc)
v = slice(nocc, None)
e_abij = 1 / (-eps[v, n, n, n] - eps[n, v, n, n] + eps[n, n, o, n] + eps[n, n, n, o])
```

These slices will also be used to define the occupied and virtual space of our two electron integrals.

For example, $\bar{g}_{ab}^{ij}$ can be accessed with `gmo[v, v, o, o]`

We now have all the pieces we need to compute the MP2 correlation energy. Our energy expression in KM notation is

\begin{equation}
E_{MP2} = \frac{1}{4} \bar{g}_{ab}^{ij} \bar{g}_{ij}^{ab} (\mathcal{E}_{ab}^{ij})^{-1}
\end{equation}

which may be easily read-off as an einsum in NumPy. Here, for clarity, we choose to read the tensors from left to right (bra to ket). We also are sure to take the appropriate slice of the two-electron integral array:

```python
# Compute MP2 Correlation Energy
E_MP2_corr = (1 / 4) * np.einsum('abij, ijab, abij ->', gmo[v, v, o, o], gmo[o, o, v, v], e_abij)

E_MP2 = E_MP2_corr + scf_e

print('MP2 correlation energy: ', E_MP2_corr)
print('MP2 total energy: ', E_MP2)
```

    MP2 correlation energy:  -0.142119840297
    MP2 total energy:  -76.094648885

Finally, compare our answer with Psi4:

```python
# ==> Compare to Psi4 <==
psi4.driver.p4util.compare_values(psi4.energy('mp2'), E_MP2, 6, 'MP2 Energy')
```

    	MP2 Energy........................................................PASSED

    True

## References

1. Notation and Symmetry of Integrals:
    > C. David Sherill, "Permutational Symmetries of One- and Two-Electron Integrals" Accessed with http://vergil.chemistry.gatech.edu/notes/permsymm/permsymm.pdf
2. Useful Notes on Kutzelnigg-Mukherjee Notation:
    > A. V. Copan, "Kutzelnigg-Mukherjee Tensor Notation" Accessed with https://github.com/CCQC/chem-8950/tree/master/2017

3. Original paper on MP2: "Note on an Approximation Treatment for Many-Electron Systems"
	> [[Moller:1934:618](https://journals.aps.org/pr/abstract/10.1103/PhysRev.46.618)] C. Møller and M. S. Plesset, *Phys. Rev.* **46**, 618 (1934)

```python

```