Métodos Numéricos

Capítulo 4: Interpolación Numérica

2021/02

MEDELLÍN - COLOMBIA

 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license.(c) Carlos Alberto Alvarez Henao

***

***Docente:*** Carlos Alberto Álvarez Henao, I.C. D.Sc.

***e-mail:*** carlosalvarezh@gmail.com

***skype:*** carlos.alberto.alvarez.henao

***Linkedin:*** https://www.linkedin.com/in/carlosalvarez5/

***github:*** https://github.com/carlosalvarezh/Metodos_Numericos

***Herramienta:*** [Jupyter](http://jupyter.org/)

***Kernel:*** Python 3.8

***

Tabla de Contenidos

1  Introducción

1.1  Propósitos de la interpolación

1.2  Diferencias entre Interpolación, Aproximación y Ajuste de curvas

1.3  Interpolación vs Aproximacion

1.4  Interpolación vs Ajuste de curvas

1.5  Elección de la función de interpolación

2  Interpolación Polinomial

2.1  Introducción

2.2  Interpolación Lineal

2.2.1  Ejemplo interpolación lineal

2.2.2  Visualización computacional

2.2.3  Mejoras al esquema de interpolación lineal

2.3  Interpolación cuadrática

2.3.1  Ejemplo interpolación cuadrática

2.3.2  Implementación computacional

2.4  Polinomio de diferencias divididas de Newton

2.4.1  Implementación computacional

2.5  Análisis de Error para la interpolación polinomial tipo Newton

2.6  Polinomios de Interpolación de Lagrange

2.6.1  Introducción

2.6.2  Polinomio de Interpolación de Lagrange de primer grado

2.6.3  Polinomio de Interpolación de Lagrange de segundo grado

2.6.4  Ejemplo polinomio de interpolación de Lagrange

2.6.5  Implementación computacional

2.6.6  Funciones de base

2.6.7  Dificultades de los polinomios de Lagrange

3  Interpolación mediante trazadores

3.1  Introducción

3.2  Trazadores lineales

3.3  Trazadores cuadráticos

3.4  Trazadores cúbicos

3.4.1  Alternativa I:

3.4.2  Alternativa II:

3.4.3  Alternativa III:

3.4.4  Alternativa IV:

3.5  Ejemplo aplicación

3.5.1  Trazadores cuadráticos

3.5.2  Trazadores cúbicos

## Introducción

La información (datos) resultante de la medición de un evento ya sea natural o social viene dada en forma discreta o tabular, es decir, se expresa como un conjunto de pares ordenados $(x_i,y_i)$. Por ejemplo, los datos obtenidos de los censos poblacionales realizados en Colombia desde 1985 según el [DANE](https://www.dane.gov.co/) son:

|Año|Población*|
|:----:|:----:|
|1985|30802|
|1990|34130|
|1995|37472|
|2000|40296|
|2005|42889|
|2010|45510|
|2015|48203|

(\* en miles de habitantes)

***Nota:*** Ejemplo tomado de las notas de clase del curso Simulación Computacional de la Universidad EAFIT y es autoría  del profesor Nicolás Guarín.

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.interpolate
from scipy.optimize import curve_fit

```

```python
start = 1985.
stop = 2015.
step = int((stop - start) / 5) + 1

x = np.linspace(start, stop, step)
y = [30802, 34130, 37472, 40296, 42889, 45510, 48203]

plt.plot(x,y, 'o')
plt.grid(True)
```

Si quisieramos responder a la pregunta: ¿Cuál era la población de Colombia en el 2012?

```python
fig, ax = plt.subplots()

ax.plot(x, y, 'o')
ax.vlines(x=2012, ymin=30000.0, ymax=48203.0, color='r')
ax.hlines(y=46000.0, xmin=1985.0, xmax=2012.0, color='y')
ax.hlines(y=47000.0, xmin=1985.0, xmax=2012.0, color='b')
ax.hlines(y=48000.0, xmin=1985.0, xmax=2012.0, color='g')
plt.grid(True)
plt.show()
```

¿cuál de los valores mostrados arriba es el más correcto? Podemos plantear varias ideas para determinar dicho valor:

- Considerando que la función es constante entre los valores

```python
fig, ax = plt.subplots()

ax.plot(x, y, 'o')

ax.hlines(y=y[0], xmin=x[0], xmax=x[0] + 2, color='b')
ax.vlines(x=x[0] + 2, ymin=y[0], ymax=y[1], color='b')

for i in range(1, len(x)-1):
    ax.hlines(y=y[i], xmin=x[i]-3, xmax= x[i] + 2, color='b')
    ax.vlines(x=x[i]+2, ymin=y[i], ymax=y[i+1], color='b')

ax.hlines(y=y[-1], xmin=x[-1]-3, xmax=x[-1]+2, color='b')

ax.vlines(x = 2012, ymin = 30000.0, ymax = y[-1], color = 'r')
ax.hlines(y = y[-1], xmin = 1985.0, xmax = 2012.0, color = 'r')

plt.grid(True)
plt.show()
```

- Asumiendo que la función es lineal entre valores

```python
fig, ax = plt.subplots()

ax.plot(x, y, 'o--')
ax.vlines(x = 2012, ymin = 30000.0, ymax = 46700, color = 'r')
ax.hlines(y = 46700, xmin = 1985.0, xmax = 2012.0, color = 'r')

plt.grid(True)
plt.show()
```

- Determinando un polinomio que pase por cada uno de los puntos.

```python
fig, ax = plt.subplots()

ax.plot(x, y, 'o')

t = np.linspace(0, 1, len(x)) # parameter t to parametrize x and y
pxLagrange = scipy.interpolate.lagrange(t, x) # X(T)
pyLagrange = scipy.interpolate.lagrange(t, y) # Y(T)
n = 100
ts = np.linspace(t[0],t[-1],n)
xLagrange = pxLagrange(ts) # lagrange x coordinates
yLagrange = pyLagrange(ts) # lagrange y coordinates
ax.plot(xLagrange, yLagrange,'b-')

ax.vlines(x=2012, ymin=30000.0, ymax=46700, color='r')
ax.hlines(y=46700, xmin=1985.0, xmax=2012.0, color='r')

ax.grid(True)
plt.show()
```

- ajustando la curva que mejor se aproxime a cada uno de los datos. En este ejemplo haremos un ajuste lineal, pero no es la única forma de hacerlo.

```python
# define the true objective function
def objective(x, a, b):
    return a * x + b

# curve fit
popt, _ = curve_fit(objective, x, y)

# summarize the parameter values
a, b = popt
print('y = %.5f * x + %.5f' % (a, b))

fig, ax = plt.subplots()

# plot input vs output
ax.scatter(x, y)

# define a sequence of inputs between the smallest and largest known inputs
x_line = np.arange(min(x), max(x), 1)

# calculate the output for the range
y_line = objective(x_line, a, b)

# create a line plot for the mapping function
ax.plot(x_line, y_line, '--', color='b')

yfit = a * 2012 + b

ax.vlines(x=2012, ymin=30000.0, ymax=yfit, color='r')
ax.hlines(y=yfit, xmin=1985.0, xmax=2012.0, color='r')

plt.grid(True)
plt.show()
```

En este curso nos centraremos en los esquemas de interpolación, por lo que los esquemas de ajuste no se tratarán.

[Volver a la Tabla de Contenido](#TOC)

### Propósitos de la interpolación

Los problemas de interpolación surgen de muchas fuentes diferentes y pueden tener muchos propósitos diferentes. Algunos de estos incluyen:

- Trazar una curva suave a través de puntos de datos discretos

- Evaluación rápida y sencilla de una función matemática

- Reemplazar una función difícil por una fácil

- Diferenciar o integrar datos tabulares

[Volver a la Tabla de Contenido](#TOC)

### Diferencias entre Interpolación, Aproximación y Ajuste de curvas

Las técnicas para resolver el problema de determinar un valor intermedio entre dos valores conocidos se pueden enmarcar en:

- [Interpolación](https://en.wikipedia.org/wiki/Interpolation)

- [Aproximación](https://en.wikipedia.org/wiki/Approximation_theory)

- [Ajuste de curvas](https://en.wikipedia.org/wiki/Curve_fitting)

A continuación vamos a describir brevemente las diferencias entre ellas.

[Volver a la Tabla de Contenido](#TOC)

### Interpolación vs Aproximacion

En interpolación, se ajustan todos los puntos de datos de forma exacta mientras que la aproximación, como su nombre indica, solo se aproxima.

Cuando se trata de la idoneidad, la interpolación es apropiada para suavizar esos datos ruidosos y no es apropiada cuando los puntos de datos están sujetos a errores experimentales u otras fuentes de error significativo. Tener un gran conjunto de puntos de datos también puede sobrecargar la interpolación. Por otro lado, la aproximación es principalmente apropiada para el diseño de rutinas de biblioteca para calcular funciones especiales. Esto se debe a la naturaleza de estas funciones: se considera que los valores exactos no son esenciales y, hasta cierto punto, ineficaces cuando los valores aproximados funcionan.

[Volver a la Tabla de Contenido](#TOC)

### Interpolación vs Ajuste de curvas

En el ajuste de curvas, no ajustamos todos nuestros puntos de datos. Por eso tenemos el concepto de residuos. En la interpolación, se obliga a la función a ajustarse a todos los puntos de datos. Ver la referencia del [Cuarteto de ascombe](https://es.wikipedia.org/wiki/Cuarteto_de_Anscombe) como un contra ejemplo de los inconvenientes en los ajustes de curvas.

Ahora que sabemos de qué categoría estamos hablando, reduzcamos a las familias de funciones utilizadas para la interpolación.

[Volver a la Tabla de Contenido](#TOC)

### Elección de la función de interpolación

Es importante darse cuenta de que existe cierta arbitrariedad en la mayoría de los problemas de interpolación. Hay arbitrariamente muchas funciones que interpolan un conjunto dado de datos. Simplemente requiriendo que alguna función matemática se ajuste a los puntos de datos deje exactamente abiertos tales
preguntas como:

- ¿Qué forma debe tener la función? Puede haber consideraciones matemáticas o físicas relevantes que sugieran una forma particular de interpolante.

- ¿Cómo debería comportarse la función entre puntos de datos?

- ¿Debería la función heredar propiedades de los datos, como monotonicidad, convexidad o periodicidad?

- Si se grafican la función y los datos, ¿los resultados deberían ser agradables a la vista?

- ¿Estamos interesados principalmente en los valores de los parámetros que definen la función de interpolación, o simplemente en evaluar la función en varios puntos para graficar u otros propósitos?

La elección de la función de interpolación depende de las respuestas a estas preguntas, así como de los datos que se deben ajustar y generalmente se basa en:

- Qué tan fácil es trabajar con la función (determinar sus parámetros a partir de los datos, evaluar la función en un punto dado, diferenciar o integrar la función, etc.)

- Qué tan bien las propiedades de la función coinciden con las propiedades de los datos a ser t (suavidad, monotonicidad, convexidad, periodicidad, etc.)

Algunas familias de funciones que se utilizan comúnmente para la interpolación incluyen:

- [Polinomios](https://en.wikipedia.org/wiki/Polynomial_interpolation)

- [Interpolación a trazos](https://en.wikipedia.org/wiki/Spline_interpolation)

- Funciones trigonométricas

- Exponenciales

- Funciones racionales

En este capítulo nos centraremos en la interpolación por polinomios e interpolación a trazos.

[Volver a la Tabla de Contenido](#TOC)

## Interpolación Polinomial

### Introducción

La interpolación polinomial es el tipo de interpolación más simple y común. Una de sus características es que siempre hay un polinomio único de grado como máximo $n-1$ que pasa por $n$ puntos de datos.
Hay muchas formas de calcular o representar un polinomio, pero se reducen a la misma función matemática. Algunos de los métodos son la base monomial, la base de [Lagrange](https://en.wikipedia.org/wiki/Lagrange_polynomial) y la base de [Newton](https://en.wikipedia.org/wiki/Newton_polynomial). Como puede observar, reciben el nombre de su base.

***Inconvenientes:***

- ***Polinomio de alto grado:*** una elección adecuada de funciones de base y puntos de interpolación puede mitigar algunas de las dificultades asociadas con polinomio de alto grado.

- ***Sobreajuste ([overfitting](https://en.wikipedia.org/wiki/Overfitting)):*** ajuste de un solo polinomio a una gran cantidad de puntos de datos, lo que probablemente produciría un comportamiento de oscilación insatisfactorio en el interpolante.

La fórmula general de un polinomio de $n$-ésimo orden es:

\begin{equation*}
f_n(x) = a_0 + a_1x + a_2x^2 +…+  a_nx^n
\label{eq:Ec4_1} \tag{4.1}
\end{equation*}

El polinomio de interpolación dado por la ecuación $\eqref{eq:Ec4_1}$ consiste en determinar el único polinomio de n-ésimo orden que se ajusta a los $n+1$ puntos dados. Este polinomio proporciona una fórmula para calcular los valores intermedios.

[Volver a la Tabla de Contenido](#TOC)

### Interpolación Lineal

El método más simple de interpolación es conectar dos puntos mediante una línea recta.

 Fuente: Chapra, S., Canale, R. Métodos Numéricos para ingenieros, 5a Ed. Mc. Graw Hill. 2007

De la figura se tiene:

\begin{equation*}
\frac{f_1(x)-f(x_0)}{x-x_0}=\frac{f(x_1)-f(x_0)}{x_1-x_0}
\label{eq:Ec4_2} \tag{4.2}
\end{equation*}

reordenando,

\begin{equation*}
f_1(x)=f(x_0)+\frac{f(x_1)-f(x_0)}{x_1-x_0}(x-x_0)
\label{eq:Ec4_3} \tag{4.3}
\end{equation*}

[Volver a la Tabla de Contenido](#TOC)

#### Ejemplo interpolación lineal

- Estime el valor de $Ln(2)$ empleando la interpolación lineal, entre $x_0=0$ y $x_1=6$

Evaluando el valor del logaritmo en cada uno de los dos puntos, $Ln(1)=0$ y $Ln(6)=1.791759$

$$f_1(2)=0+\frac{1.791759-0}{6-1}(2-1)=0.3583519$$

El valor exacto es $Ln(2)=0.693147$, que representa un error relativo porcentual de

$$Er(\%)=\frac{|0.693147-0.3583519|}{0.693147}=48.3\%$$

Si se disminuye el valor del intervalo a evaluar, por ejemplo en $x_1=4$, se llega a

$$f_1(2)=0+\frac{1.386294-0}{4-1}(2-1)=0.462098$$

obteniendo un error relativo porcentual del $33.3\%$.

[Volver a la Tabla de Contenido](#TOC)

#### Visualización computacional

```python
import numpy as np
import matplotlib.pyplot as plt

n = 20

fig, ax = plt.subplots()

x = np.linspace(1, 6, n)
y = np.log(x)

x1 = [x[0], x[-1]]
y1 = [np.log(x[0]), np.log(x[-1])]

x2 = [x[0], 4]
y2 = [np.log(x[0]), np.log(4)]

ax.plot(x,y, '-', x1, y1, 'o-', x2,y2, 'o-')
ax.vlines(x = 2, ymin = 0.0, ymax = np.log(2), color = 'r', linestyles='dashed')

plt.grid(True)

```

El error en la interpolación lineal resulta de aproximar una curva con una línea recta.

[Volver a la Tabla de Contenido](#TOC)

#### Mejoras al esquema de interpolación lineal

- Disminuir el tamaño del intervalo.

- Introducir alguna curvatura en la línea que conecta los puntos.

[Volver a la Tabla de Contenido](#TOC)

### Interpolación cuadrática

Si tres (3) puntos de los datos están disponibles, esto puede realizarse con un polinomio de segundo grado (parábola).

 Fuente: Chapra, S., Canale, R. Métodos Numéricos para ingenieros, 5a Ed. Mc. Graw Hill. 2007

La forma general de una ecuación cuadrática puede expresarse de la siguiente forma:

\begin{equation*}
f_2(x) = b_0 + b_1(x–x_0) + b_2(x–x_0)(x–x_1)
\label{eq:Ec4_4} \tag{4.4}
\end{equation*}

Debemos determinar los valores de los coeficientes $b_i$.

- Para $b_0$, en la ecuación $\eqref{eq:Ec4_4}$, con $x = x_0$:

\begin{equation*}
b_0 = f(x_0)
\label{eq:Ec4_5} \tag{4.5}
\end{equation*}

- Para $b_1$, sustituyendo la ecuación $\eqref{eq:Ec4_5}$ en la la ecuación $\eqref{eq:Ec4_4}$, y evaluando en $x = x_1$:

\begin{equation*}
b_1 = \frac{f(x_1)-f(x_0)}{(x_1 - x_0)}
\label{eq:Ec4_6} \tag{4.6}
\end{equation*}

- Para $b_2$, las ecuaciones $\eqref{eq:Ec4_5}$ y $\eqref{eq:Ec4_6}$ pueden sustituirse en la ecuación $\eqref{eq:Ec4_4}$, evaluada en $x_2$

\begin{equation*}
b_2=\frac{\frac{f(x_2)-f(x_1)}{(x_2-x_1)}-\frac{f(x_1)-f(x_0)}{(x_1-x_0)}}{(x_2 - x_0)}
\label{eq:Ec4_7} \tag{4.7}
\end{equation*}

[Volver a la Tabla de Contenido](#TOC)

#### Ejemplo interpolación cuadrática

Continuando con el ejemplo anterior, se van a considerar los siguientes puntos:

$$x_0=1 \hspace{1cm} f(x_0)=0.000000$$
$$x_1=4 \hspace{1cm} f(x_1)=1.386294$$
$$x_2=6 \hspace{1cm} f(x_2)=1.791759$$

De las ecuaciones anteriores,

$$b_0=0$$

$$b_1=\frac{1.386294-0}{4-1}=0.4620981$$

$$b_2=\frac{\frac{1.791759-1.386294}{6-4}-0.4620981}{6-1}=-0.0518731$$

Sustituyendo estos valores en la ecuación cuadrática inicial, se llega a:

$$f_2(x)=0+0.4620981(x-1)-0.0518731(x-1)(x-4)$$

y evaluando en $x=2$, se llega a

$$f_2(2)=0.565844$$

que representa un error relativo porcentual del $18.4\%$

[Volver a la Tabla de Contenido](#TOC)

#### Implementación computacional

```python
def difdiv2o(x, y, xm):
    b0 = y[0]
    b1 = (y[1] - b0) / (x[1] - x[0])
    b2 = ((y[2] - y[1]) / (x[2] - x[1]) - b1) / (x[2] - x[0])

    return b0 + b1 * (xm - x[0]) + b2 * (xm - x[0]) * (xm - x[1])
```

```python
import numpy as np
import matplotlib.pyplot as plt

n = 20

fig, ax = plt.subplots()

x = np.linspace(1, 6, n)
y = np.log(x)

x1 = [x[0], 4, x[-1]]
y1 = [np.log(x[0]), np.log(x1[1]), np.log(x[-1])]
ym = difdiv2o(x1,y1,2)

x1.insert(1, 2)
y1.insert(1,ym)

ax.plot(x, y, '-', x1, y1, 'o-')
ax.vlines(x = 2, ymin = 0.0, ymax = np.log(2), color = 'r', linestyles='dashed')

plt.grid(True)
```

### Polinomio de diferencias divididas de Newton

Lo anterior puede ser generalizado para ajustar un polinomio de $n$-ésimo orden a $n+1$ datos:

 Fuente: medium.com

\begin{equation*}
f_n(x) = b_0+b_1(x–x_0)+\ldots+b_n(x–x_0)(x–x_1)\ldots(x – x_{n-1})
\label{eq:Ec4_8} \tag{4.8}
\end{equation*}

De igual manera que para las interpolaciones lineal y cuadrática, se llega a:

\begin{equation*}
f_n(x)=f(x_0)+(x–x_0)f[x_1,x_0]+(x–x_0)(x–x_1)f[x_2,x-,x_0]+\ldots+(x–x_0)(x–x_1)\ldots(x–x_{n-1})f[x_n, x_{n-1},\ldots,x_2,x_1,x_0]
\label{eq:Ec4_9} \tag{4.9}
\end{equation*}

Conocido como *Polinomio de interpolación por [diferencias divididas de Newton](https://en.wikipedia.org/wiki/Divided_differences)*. Las evaluaciones de las funciones puestas entre paréntesis son diferencias divididas finitas.

- ***Primera diferencia dividida:***

\begin{equation*}
f[x_i, x_j]=\frac{f(x_i)-f(x_j)}{(x_i-x_j)}
\label{eq:Ec4_10} \tag{4.10}
\end{equation*}

- ***Segunda diferencia dividida:*** representa la diferencia de las dos primeras diferencias divididas

\begin{equation*}
f[x_i, x_j,x_k]=\frac{f[x_i,x_j]-f[x_j,x_k]}{(x_i-x_k)}
\label{eq:Ec4_11} \tag{4.11}
\end{equation*}

$$\vdots$$

- ***$n$-ésima diferencia dividida:*** representa la diferencia de las dos primeras diferencias divididas
\begin{equation*}
f[x_n, x_{n-1},\ldots, x_1,x_0]=\frac{f[x_n,x_{n-1},\ldots,x_1]-f[x_{n-1},x_{n-2},\ldots,x_0]}{(x_n-x_0)}
\label{eq:Ec4_12} \tag{4.12}
\end{equation*}

Este proceso recursivo lo podemos visualizar de la siguiente manera:

 Fuente: Wikimedia.org

[Volver a la Tabla de Contenido](#TOC)

#### Implementación computacional

En la página 513 del libro de Chapra y Canale, Figura 18.7, se tiene un algoritmo en Fortran para la implementación del código de Diferencias Divididas tipo Newton. Se invita al estudiante a que lo estudie y codifique en el lenguaje de preferencia.

```python
# Escriba aquí su código
```

[Volver a la Tabla de Contenido](#TOC)

### Análisis de Error para la interpolación polinomial tipo Newton

La ecuación $\eqref{eq:Ec4_9}$ es similar a la *serie de expansión de Taylor*. Se agregan términos en forma secuencial para capturar el comportamiento de alto orden de la función a analizar. Estos términos son diferencias divididas finitas y, así, representan aproximaciones de derivadas de orden mayor.

El error de truncamiento se expresa entonces como:

\begin{equation*}
R_n=\frac{f^{(n+1)}(\xi)}{(n+1)!} \left ( x_{i+1}-x_i\right )^{n+1}
\label{eq:Ec4_13} \tag{4.13}
\end{equation*}

Para una interpolación de n-ésimo orden, una relación análoga para el error es

\begin{equation*}
R_n=\frac{f^{(n+1)}(\xi)}{(n+1)!}(x-x_0)(x-x_1) \ldots (x-x_n)
\label{eq:Ec4_14} \tag{4.14}
\end{equation*}

Observe que en la ecuación [(4.14)](#Ec4_14), la función debe conocerse. Para resolver esta situación, una formulación alternativa es el uso de la diferencia dividida para aproximar la derivada $(n+1)$–ésima y que no requiere el conocimiento previo de la función.

\begin{equation*}
R_n=f_n[x_n,x_{n-1},x_{n-2},\ldots,x_2,x_1,x_0](x-x_0)(x-x_1) \ldots (x-x_n)
\label{eq:Ec4_15} \tag{4.15}
\end{equation*}

Debido a que la ecuación [(4.15)](#Ec4_15) contiene el término $f_n(x)$ no puede resolverse para estimar el error, pero, si se dispone de un dato adicional, la ecuación [(4.15)](#Ec4_15) puede usarse:

\begin{equation*}
R_n=f_n[x_{n+1}, x_n,x_{n-1},x_{n-2},\ldots,x_2,x_1,x_0](x-x_0)(x-x_1) \ldots (x-x_n)
\label{eq:Ec4_16} \tag{4.16}
\end{equation*}

[Volver a la Tabla de Contenido](#TOC)

### Polinomios de Interpolación de Lagrange

#### Introducción

El [polinomio de interpolación de Lagrange](https://en.wikipedia.org/wiki/Lagrange_polynomial) evita el cálculo de las diferencias divididas en el esquema de Newton. De una forma general, se representa como la [combinación lineal](https://en.wikipedia.org/wiki/Linear_combination):

\begin{equation*}
f_n(x)=\sum \limits_{i=0}^n L_i(x)f(x_i)
\label{eq:Ec4_17} \tag{4.17}
\end{equation*}

donde $L_i$ son las bases polinómicas de *[Lagrange](https://es.wikipedia.org/wiki/Joseph-Louis_Lagrange)* dadas por:

\begin{equation*}
L_i(x)=\prod_{\substack{j=0\\ j \ne i}}^n \frac{x-x_j}{x_i-x_j}
\label{eq:Ec4_18} \tag{4.18}
\end{equation*}

de $\eqref{eq:Ec4_18}$ se observa que todas las funciones $L_i$ son polinomios de grado $n$ que tienen la propiedad

\begin{equation*}
L_i(x_j)=\delta_{ij}, \quad \delta_{ij} = \left \{
\begin{aligned}
1, \quad i=j,\\
0, \quad i \ne j
\end{aligned}
\right.
\label{eq:Ec4_19} \tag{4.19}
\end{equation*}

donde $\delta_{is}$ es el [delta de Kronecker](https://en.wikipedia.org/wiki/Kronecker_delta).

[Volver a la Tabla de Contenido](#TOC)

#### Polinomio de Interpolación de Lagrange de primer grado

Tomando $n=1$ (lineal), se tiene

\begin{equation*}
f_1(x)=\frac{(x-x_1)}{(x_0-x_1)}f(x_0)+\frac{(x-x_0)}{(x_1-x_0)}f(x_1)
\label{eq:Ec4_19a} \tag{4.19a}
\end{equation*}

[Volver a la Tabla de Contenido](#TOC)

#### Polinomio de Interpolación de Lagrange de segundo grado

Tomando $n=2$ (cuadrático), se tiene

\begin{equation*}
f_2(x)=\frac{(x-x_1)(x-x_2)}{(x_0-x_1)(x_0-x_2)}f(x_0) + \frac{(x-x_0)(x-x_2)}{(x_1-x_0)(x_1-x_2)}f(x_1) + \frac{(x-x_0)(x-x_1)}{(x_2-x_0)(x_2-x_1)}f(x_2)
\label{eq:Ec4_20} \tag{4.20}
\end{equation*}

[Volver a la Tabla de Contenido](#TOC)

#### Ejemplo polinomio de interpolación de Lagrange

retomando el ejemplo que se ha venido trabajando anteriormente,

$$x_0=1 \hspace{1cm} f(x_0)=0.000000$$
$$x_1=4 \hspace{1cm} f(x_1)=1.386294$$
$$x_2=6 \hspace{1cm} f(x_2)=1.791759$$

- ***Polinomio de primer grado:***

$$f_1(2)=\frac{2-4}{1-4}(0)+\frac{2-1}{4-1}(1.386294) = 0.462098$$

- ***Polinomio de segundo grado:***

$$f_2(2)=\frac{(2-4)(2-6)}{(1-4)(1-6)}(0)+\frac{(2-1)(2-6)}{(4-1)(4-6)}(1.386294)+\frac{(2-1)(2-4)}{(6-1)(6-4)}(1.791759) = 0.565844$$

- ***Nota:*** Realice una comparación con los resultados obtenidos con los correspondientes esquemas lineal y cuadrático en el esquema de [diferencias divididas de Newton](#DDN).

#### Implementación computacional

```python
def lagrange(x ,i , xm ):
    n = len(xm) - 1
    y = 1.0
    for j in range(n + 1):
        if i != j:
            y *= (x - xm[j]) / (xm[i] - xm[j])
    return y
```

```python
def interpolation(x, xm, ym):
    n = len(xm) - 1
    lagrpoly = np.array([lagrange(x, i, xm) for i in range(n + 1)])
    y = np.dot(ym, lagrpoly)
    return y
```

```python
import numpy as np
import matplotlib.pyplot as plt

xm = np.array([1,4,6])
ym = np.log(xm)
#xm = np.array([1, 2, 3, 4, 5, 6])
#ym = np.array([-3, 0, -1, 2, 1, 4])
#ym = np.sin(xm)

xplot = np.linspace(-1., 6.0, 100)
yplot = interpolation(xplot, xm, ym)
plt.plot(xm, ym, '--', xplot, yplot, '-')
plt.grid('True')
```

[Volver a la Tabla de Contenido](#TOC)

#### Funciones de base

Para entender un poco más cómo es que trabaja la interpolación entre cada uno de los puntos, recordemos que dichos polinomios interpolantes de Lagrange deben cumplir con la propiedad del delta de Kronecker. Para visualizar esto, se implementará la descripción propuesta en el dcumento [Interpolación de Lagrange 1D](https://github.com/AppliedMechanics-EAFIT/modelacion_computacional/blob/master/notebooks/02a_interpolacion.ipynb) realizado por los profesores *Juan David Gómez Cataño* y *Nicolás Guarín Zapata* para el curso de Modelación Computacional en el programa de Ingeniería Civil de la Universidad EAFIT. Todo el crédito para ellos.

```python
# llamado a las biblitecas numéricas, de visualización y simbólicas
import numpy as np
import matplotlib.pyplot as plt
from scipy import interpolate
import sympy as sym
sym.init_printing()
```

```python
def lagrange_poly(x, order, i, xi=None):
    if xi == None:
        xi = sym.symbols('x:%d'%(order+1))
    index = list(range(order+1))
    index.pop(i)
    return sym.prod([(x - xi[j])/(xi[i] - xi[j]) for j in index])
```

```python
fun = lambda x: x**3 + 4.0*x**2 - 10.0
```

```python
npts = 200
x_pts = np.linspace(-1, 1, npts)
```

```python
pts = np.array([-1, 1, 0])
fd = fun(pts)
```

```python
plt.figure()
y_pts = fun(x_pts)
plt.plot(x_pts , y_pts)
plt.plot(pts, fd, 'ko')
```

```python
x = sym.symbols('x')
pol = []
pol.append(sym.simplify(lagrange_poly(x, 2, 0, [-1,1,0])))
pol.append(sym.simplify(lagrange_poly(x, 2, 1, [-1,1,0])))
pol.append(sym.simplify(lagrange_poly(x, 2, 2, [-1,1,0])))
pol
```

```python
plt.figure()
print(npts)
for k in range(3):

    for i in range(npts):
        yy[i] = pol[k].subs([(x, x_pts[i])])
        print(yy[i])
    plt.plot(x_pts, yy)
```

[Volver a la Tabla de Contenido](#TOC)

#### Dificultades de los polinomios de Lagrange

Los polinomios de interpolación de Lagrange presentan dificultades cuando se tienen polinomios de orden muy alto, agravado cuando se tienen puntos equidistantes o se presentan saltos (discontinuidades) en la solución.

A esta situación se le conoce como *[fenómeno de Runge](https://en.wikipedia.org/wiki/Runge%27s_phenomenon)*. Veamos el siguiente ejemplo:

- Dada la ecuación

$$f(x)=\frac{1}{1+25x^2}$$

si se interpola esta función utilizando nodos equidistantes $x_i \in [-1, 1]$ tal que

$$x_i=-1+(i-1)\frac{2}{n} \quad i \in \{1, 2, 3, \ldots, n, n+1\}$$

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
n = 200
x = [-1 + 2 * (i - 1) / n for i in range(1, n + 2)]
y = [1 / (1+ 25 * i **2) for i in x]

plt.plot(x,y)
plt.grid(True)
```

Ahora vamos a emplear, como ejemplo, una serie de polinomios interpolantes para evaluar su comportamiento, desde orden 1 (2 puntos) hasta de orden 9 (10 puntos).

```python
# Polinomios interpolantes de orden 1-9 (2-10 puntos)

data = [2, 3, 4, 6, 10]

#fig = plt.figure(figsize=(16, 6), dpi= 50, facecolor='w', edgecolor='k')
#ax = fig.add_subplot(111)

plt.plot(x, y, '-', label = 'Exacta')
plt.title('Fenómeno de Runge')
xplot = np.linspace(-1., 1.0, 100)

for i in data:
    xRi = np.linspace(-1, 1, i)
    yRi = [1 / (1+ 25 * j **2) for j in xRi]

    yploti = interpolation(xplot, xRi, yRi)
    string = "P_orden " + str(i-1)
    plt.plot(xplot, yploti, '-', label = string)

#    idx = np.argwhere(np.diff(np.sign(f - g))).flatten()
#    plt.plot(x[idx], f[idx], 'ro')

    plt.legend()

plt.plot(x, y, '-', label = 'Exacta')
plt.title('Fenómeno de Runge')
plt.grid('True')

```

Se observa que a medida que se aumenta el orden del polinomio, intentando obtener un mejor ajuste, se presentan oscilaciones en los puntos extremos.

***Como actividad complementaria, se invita al estudiante reproducir el fenómeno de Runge que se representa en la gráfica de la Figura 18.14 (p. 526) que se encuentra en el libro de Chapra y Canale, 5a Ed.***

[Volver a la Tabla de Contenido](#TOC)

## Interpolación mediante trazadores

### Introducción

En las secciones anteriores, se usaron polinomios de $n$-ésimo grado para interpolar entre $n+1$ puntos que se tenían como datos, por ejemplo, para $10$ puntos se puede obtener un polinomio exacto de noveno grado. Esta curva podría agrupar todas las curvas al menos hasta, e incluso, la novena derivada. No obstante, hay casos, como el que se acaba de observar, donde estas funciones llevan a resultados erróneos a causa de los errores de redondeo y los puntos lejanos (*fenómeno de Runge*).

Como alternativa para intentar mitigar esta situación se pueden implementar polinomios de menor grado en subconjuntos de los datos. Tales polinomios conectores se denominan trazadores o [splines](https://en.wikipedia.org/wiki/Spline_(mathematics)).

Supongamos que empleamos curvas de tercer grado para unir dos conjuntos de datos, cada una de esas funciones se pueden construir de tal forma que las conexiones entre ecuaciones cúbicas adyacentes resulten visualmente suaves. Podría parecer que la aproximación de tercer grado de los trazadores sería inferior a la expresión de noveno grado, entonces, por qué un trazador resulta preferible?

El concepto de trazador se originó en la técnica de dibujo que usa una cinta delgada y flexible (llamada spline, en inglés), para dibujar curvas suaves a través de un conjunto de puntos. Se coloca un papel sobre una mesa y alfileres en el papel en la ubicación de los datos. Una curva cúbica suave resulta al entrelazar la cinta entre los alfileres. De aquí que se haya adoptado el nombre de Trazador Cúbico para los polinomios de este tipo.

 Fuente: Chapra, S., Canale, R. Métodos Numéricos para ingenieros, 5a Ed. Mc. Graw Hill. 2007

[Volver a la Tabla de Contenido](#TOC)

### Trazadores lineales

La unión más simple entre dos puntos es una línea recta. Los trazadores de primer grado para un grupo de datos ordenados pueden definirse como un conjunto de funciones lineales:

\begin{equation*}
\begin{split}
f(x) & = f(x_0) + m_0(x - x_0), \quad x_0 \le x \le x_1 \\
f(x) & = f(x_1) + m_1(x - x_1), \quad x_1 \le x \le x_2 \\
f(x) & = f(x_2) + m_2(x - x_2), \quad x_2 \le x \le x_3 \\
&\vdots \\
f(x) & = f(x_{n-1}) + m_{n-1}(x - x_{n-1}), \quad x_{n-1} \le x \le x_n
\end{split}
\label{eq:Ec4_21} \tag{4.21}
\end{equation*}

donde $m_i=\frac{f(x_{i+1})-f(x_i)}{(x_{i+1}-x_i)}$ es la pendiente de la línea recta que une los puntos. La principal desventaja de los trazadores de primer grado es que no son suaves. En los puntos donde se encuentran dos trazadores, la pendiente cambia de forma abrupta. La primer derivada de la función es discontinua en esos puntos.

[Volver a la Tabla de Contenido](#TOC)

### Trazadores cuadráticos

Para asegurar que las derivadas $m$-ésimas sean continuas en los nodos, se debe emplear un trazador de un grado de, al menos, $m+1$. El objetivo de los trazadores cuadráticos es obtener un polinomio de segundo grado para cada intervalo entre los datos. De manera general, el polinomio en cada intervalo se representa como:

\begin{equation*}
\begin{split}
f(x_i) = a_ix^2+b_ix+c_i
\end{split}
\label{eq:Ec4_22} \tag{4.22}
\end{equation*}

Para $n+1$ datos ($i=0, 1, 2,\ldots, n$) existen $n$ intervalos y, en consecuencia, $3n$ constantes desconocidas ($a$, $b$ y $c$) por evaluar. Por lo tanto, se requieren $3n$ ecuaciones o condiciones para evaluar las incógnitas. Éstas son:

1. Los valores de la función de polinomios adyacentes deben ser iguales en los nodos interiores. Esta condición se representa como:

\begin{equation*}
\begin{split}
a_{i−1}x_{i−1}^2+b_{i−1}x_{i−1}+c_{i−1}&=f(x_{i−1}) \\
a_i x_{i−1}^2+b_i x_{i−1}+c_i&=f(x_{i−1})
\end{split}
\label{eq:Ec4_23} \tag{4.23}
\end{equation*}

   para $i=2$ a $n$. Como sólo se emplean nodos interiores, las ecuaciones anteriores proporcionan, cada una, $n–1$ condiciones; en total, $2n–2$ condiciones.

2. La primera y la última función deben pasar a través de los puntos extremos. Esto agrega dos ecuaciones más:

\begin{equation*}
\begin{split}
a_{1}x_{0}^2+b_{1}x_{0}+c_{1}&=f(x_{0}) \\
a_n x_{n}^2+b_n x_{n}+c_n&=f(x_{n})
\end{split}
\label{eq:Ec4_24} \tag{4.24}
\end{equation*}

   En total se tienen $2n−2+2=2n$ condiciones

3. La primera derivada de la ecuación: $f_i(x)=a_ix^2+b_i x+c$ es: $f'(x_𝑖)=2a_ix+b$. De manera general, esta condición se representa como:

\begin{equation*}
\begin{split}
2a_{i−1} x_{i−1}+b_{i−1}=2a_i x_{i−1}+b_i
\end{split}
\label{eq:Ec4_25} \tag{4.25}
\end{equation*}

   para $i = 2an$. Esto proporciona otras $n–1$ condiciones, llegando a un total de $2n+n−1=3n−1$. Como se tienen $3n$ incógnitas, falta una condición más.

4. Suponga que en el primer punto la segunda derivada es cero. Esto es: $f''_𝑖(x)=2a_i$, que se puede expresar matemáticamente como:

\begin{equation*}
\begin{split}
a_1=0
\end{split}
\label{eq:Ec4_26} \tag{4.26}
\end{equation*}

 Fuente: Chapra, S., Canale, R. Métodos Numéricos para ingenieros, 5a Ed. Mc. Graw Hill. 2007

[Volver a la Tabla de Contenido](#TOC)

### Trazadores cúbicos

El objetivo es obtener polinomios de tercer grado para cada intervalo entre los nodos:

\begin{equation*}
\begin{split}
f_i(x)=a_i x^3+b_i x^2+c_i x+d_i
\end{split}
\label{eq:Ec4_27} \tag{4.27}
\end{equation*}

Así para los $n+1$ datos ($i=0,1,2,\ldots,n$) existen $n$ intervalos y, por lo tanto, se requerirán de $4n$ condiciones para evaluar las incógnitas. Estas son:

1. Los valores de la función deben ser iguales en los nodos interiores: $2n−2$ condiciones.

2. La primera y la última función deben pasar a través de los puntos extremos: $2$ condiciones.

3. Las primeras derivadas en los nodos interiores deben ser iguales: $n−1$ condiciones.

4. Las segundas derivadas en los nodos interiores deben ser iguales: $n−1$.

5. Las segundas derivadas en los nodos extremos son cero: $2$ condiciones.

Suponga que se tienen $n+1$ puntos, $P_k(x_k,y_k)$, donde los $y_k=f(x_k)$, $k=0m,1,2,\ldots,n$, en los cuales se requiere interpolar una función $f$. Las abcisas, $x_k$, no se requiere que sean equidistantes, pero sí estén ordenadas, es decir: $x_0<x_1<x_2<\ldots<x_n$.

Se definirá una función $s(x)$ cúbica a trazos en $[x_0, x_n]$, construida a partir de una serie de polinómios c´bicos $q_0(x), q_1(x), q_2(x), \ldots, q_{n-1}(x)$ tales que $s(x)=q_k(x)$ en $[x_k,x_{k+1}]$, para $k=0, 1, 2, \ldots, n-1$.

para que $s(x)$ interpole en los puntos $P_0, P_1, P_2, \ldots, P_n$, los polinomios $q_k(x)$ deben cumplir:

\begin{equation*}
\begin{split}
q_k(x_k)&=y_k \\
q_k(x_{k+1})&=y_{k+1} \quad k=0,1,2 \ldots, n-1
\end{split}
\label{eq:Ec4_28} \tag{4.28}
\end{equation*}

De la ecuación [(4.28)](#Ec4_28) se obtienen $2n$ condiciones. También, los polinomios $q_k(x)$ del interpolador cúbico $s(x)$ deben tener la misma pendiente y concavidad, es decir

\begin{equation*}
\begin{split}
q'_{k-1}(x_k)&=q'_k(x_k) \\
q''_{k-1}(x_{k})&=q''_{k}(x_k) \quad k=1,2 \ldots, n-1
\end{split}
\label{eq:Ec4_29} \tag{4.29}
\end{equation*}

de la ecuación [(4.29)](#Ec4_29) se obtienen otras $2(n-1)$ condiciones a ser satisfechas. Las ecuaciones [(4.28)](#Ec4_28) y [(4.29)](#Ec4_29) son condiciones de continuidad mediante la primera y segunda derivada.

Si $s(x)$ es cúbica a trozos en el intervalo $[x_0, x_n]$, su derivada segunda $s''(x)$ es lineal en el mismo intervalo e interpola en los puntos $(x_k, s''(x_k))$ y $(x_{k+1}, s''(x_{k+1}))$ en $[x_k, x_{k+1}]$. Por tanto, $q_k(x)$ es un polinomio de grado uno que interpola en los puntos $(x_k, s''(x_k))$ y $(x_{k+1}, s''(x_{k+1}))$

\begin{equation*}
\begin{split}
q''_k(x)=s''(x_k) \frac{x-x_{k+1}}{x_k-x_{k+1}}+s''(x_{k+1}) \frac{x-x_{k}}{x_{k+1}-x_{k}}, \quad k=0,1,2,\ldots,n-1
\end{split}
\end{equation*}

sean,

\begin{equation*}
\begin{split}
h_k&=x_{k+1}-x_k, \quad k=0,1,2,\ldots,n-1 \\
\sigma_k&=s''(x_k), \quad k=0,1,2,\ldots,n
\end{split}
\end{equation*}

reemplazando,

\begin{equation*}
\begin{split}
q''_k(x)=\frac{\sigma_k}{h_k}(x_{k+1}-x)+\frac{\sigma_{k+1}}{h_k}(x-x_k), \quad k=0,1,2,\ldots,n
\end{split}
\label{eq:Ec4_30} \tag{4.30}
\end{equation*}

donde $h_k$ y $\sigma_k$ son constantes, con $\sigma_k$ aún por determinar. Para ello, integrando dos veces se tiene

\begin{equation*}
\begin{split}
q_k(x)=\frac{\sigma_k}{h_k}\frac{(x_{k+1}-x)^3}{6}+\frac{\sigma_{k+1}}{h_k}\frac{(x-x_k)}{6}+C_k +D_kx
\end{split}
\label{eq:Ec4_31} \tag{4.31}
\end{equation*}

El término lineal $C_k+D_kx$, se puede reescribir como:

\begin{equation*}
\begin{split}
C_k+D_kx=A_k(x-x_k)+B_k(x_{k+1}-x)
\end{split}
\end{equation*}

donde $A_k$ y $B_k$ son constantes arbitrarias. La ecuación [(4.31)](#Ec4_31) queda

\begin{equation*}
\begin{split}
q_k(x)=\frac{\sigma_k}{h_k}\frac{(x_{k+1}-x)^3}{6}+\frac{\sigma_{k+1}}{h_k}\frac{(x-x_k)}{6}+A_k(x-x_k)+B_k(x_{k+1}-x)
\end{split}
\label{eq:Ec4_32} \tag{4.32}
\end{equation*}

Ahora aplicando las condiciones dadas en las ecuaciones [(4.28)](#Ec4_28) a esta ecuación,

\begin{equation*}
\begin{split}
y_k&=\frac{\sigma_k}{h_k}\frac{h_k^3}{6}+\frac{\sigma_{k+1}}{h_k}\times 0+A_k \times 0 + B_kh_k=\frac{\sigma_k}{6}h_k^2+B_kh_k \\
y_{k+1}&=\frac{\sigma_{k+1}}{h_k}h_k^3+A_kh_k=\frac{\sigma_{k+1}}{6}h_k^2+A_kh_k
\end{split}
\label{eq:Ec4_33} \tag{4.33}
\end{equation*}

De estas dos ecuaciones con dos incógnitas, se despejan $A_k$ y $B_k$, y se sustituyen en la ecuación [(4.32)](#Ec4_32), resultando:

\begin{equation*}
\begin{split}
q_k(x)&=\frac{\sigma_k}{6} \left[ \frac{(x_{k+1}-x)^3}{h_k}-h_k(x_{k+1}-x) \right] \\
&+\frac{\sigma_{k+1}}{6} \left[ \frac{(x-x_k)^3}{h_k}-h_k(x-x_k) \right] \\
&+y_k \left[ \frac{(x_{k+1}-x)}{h_k} \right] +y_{k+1} \left[ \frac{(x-x_k)}{h_k} \right], \quad k=0,1,2,\ldots,n-1
\end{split}
\label{eq:Ec4_34} \tag{4.34}
\end{equation*}

Que corresponde a la ecuación para el spline $q_k(x)$. Falta conocer los valores $\sigma_k$, con $k=0,1,2,\ldots, n$, que proporcionan otras $n+1$ incógnitas. Para esto, empleamos las condiciones dadas en la ecuación [(4.29)](#Ec4_29) y derivando la ecuación [(4.34)](#Ec4_34), se tiene:

\begin{equation*}
\begin{split}
q'_k(x)&=\frac{\sigma_k}{6} \left[ \frac{-3(x_{k+1}-x)^2}{h_k}+h_k \right]+\frac{\sigma_{k+1}}{6} \left[ \frac{3(x_k-x)^2}{h_k}-h_k \right]+\frac{y_{k+1}-y_k}{h_k}
\end{split}
\label{eq:Ec4_35} \tag{4.35}
\end{equation*}

entonces,

\begin{equation*}
\begin{split}
q'_k(x)&=\frac{\sigma_k}{6}(-2h_k)+\frac{\sigma_{k+1}}{6}(-h_k)+\frac{y_{k+1}-y_k}{h_k}
\end{split}
\label{eq:Ec4_36} \tag{4.36}
\end{equation*}

y

\begin{equation*}
\begin{split}
q'_k(x_{k+1})&=\frac{\sigma_k}{6}(h_k)+\frac{\sigma_{k+1}}{6}(2h_k)+\frac{y_{k+1}-y_k}{h_k}
\end{split}
\label{eq:Ec4_37} \tag{4.37}
\end{equation*}

Reemplazando $k$ por $k-1$ en la ecuación [(4.37)](#Ec4_37) para obtener $q'_{k-1}(x_k)$ e igualando en la ecuación [(4.36)](#Ec4_36) se llega a:

\begin{equation*}
\begin{split}
h_{k-1}\sigma_{k-1}+2(h_{k-1}+h_k)\sigma_k+h_k\sigma_{k+1}=6 \left( \frac{y_{k+1}-y_k}{h_k} - \frac{y_k-y_{k-1}}{h_{k-1}} \right), \quad k=1,2,3,\ldots, n-1
\end{split}
\label{eq:Ec4_38} \tag{4.38}
\end{equation*}

obsérvese que el término entre paréntesis se puede representar como diferenciales $(\Delta y_k)$ o incluso como diferencias divididas de Newton, $f[x_k,x_{k+1}]$ vistas al comienzo del capítulo.

Hay que tener en cuenta que el índice $k$ varía de $1$ a $n-1$, produciéndose $n-1$ ecuaciones lineales con $n+1$ incógnitas. Esto genera un sistema indeterminado, con infinitas soluciones. Existen varias formas de determinar $\sigma_0$ y $\sigma_n$ de la primera y $n-1$-ésima ecuación, llegándose a un sistema tridiagonal de orden $n-1$ con las variables $\sigma_k$, $k=1, 2, 3, \ldots, n-1$.

[Volver a la Tabla de Contenido](#TOC)

#### Alternativa I:

Especificando el valor de la segunda derivada, $s''(x)$ en los puntos extremos: $\sigma_0=s''(x_0)$ y $\sigma_n=s''(x_n)$. Si se supone $\sigma_0=0$ y $\sigma_n=0$ se denomina [spline cúbico natural](https://towardsdatascience.com/numerical-interpolation-natural-cubic-spline-52c1157b98ac)

[Volver a la Tabla de Contenido](#TOC)

#### Alternativa II:

Suponer que $s''(x)$ es constante en los extremos, es decir, $\sigma_0=\sigma_1$ y $\sigma_n=\sigma_{n-1}$

[Volver a la Tabla de Contenido](#TOC)

#### Alternativa III:

Suponer que $s''(x)$ es lineal cerca de los extremos, es decir,

$$\sigma_0=\frac{1}{h_1}((h_0+h_1)\sigma_1 - h_0\sigma_2)$$

y

$$\sigma_n=\frac{1}{h_{n-2}}((h_{n-2}-h_{n-1})\sigma_{n-2}+ (h_{n-2}+h_{n-1})\sigma_{n-1})$$

[Volver a la Tabla de Contenido](#TOC)

#### Alternativa IV:

Especificar el valor de $s'(x)$ en los puntos extremos:

$$\sigma_0=\frac{3}{h_0}[\Delta y_0-s'(x_0)]-\frac{1}{2}\sigma_1$$

y

$$\sigma_n=\frac{3}{h_{n-1}}[s'(x_n)-\Delta y_{n-1}]-\frac{1}{2}\sigma_{n-1}$$

[Volver a la Tabla de Contenido](#TOC)

### Ejemplo aplicación

Dado el siguiente conjunto de datos:

|x|f(x)|
|:--:|:--:|
|3.0|2.5|
|4.5|1.0|
|7.0|2.5|
|9.0|0.5|

Evalúe el valor en $x=5.0$ empleando trazadores cuadráticos y cúbicos

[Volver a la Tabla de Contenido](#TOC)

#### Trazadores cuadráticos

De la tabla se tienen cuatro puntos y $n=3$ intervalos, por lo que se deben determinar $3n=3\times3=9$ incógnitas.

1. De las ecuaciones [(4.23)](#Ec4_23) se determinan $2\times3-2=4$ condiciones así,

\begin{equation*}
\begin{split}
4.5^{2}a_{1}+4.5b_{1}+c_{1}&=1.0 \\
4.5^{2}a_{2}+4.5b_{2}+c_{2}&=1.0 \\
7.0^{2}a_{2}+7.0b_{2}+c_{2}&=2.5 \\
7.0^{2}a_{3}+7.0b_{3}+c_{3}&=2.5
\end{split}
\end{equation*}

2. La primera y última función pasan por los puntos extremos, agregando 2 ecuaciones más, ecuación [(4.24)](#Ec4_24):

\begin{equation*}
\begin{split}
3.0^2a_{1}+3.0b_{1}+c_{1}&=2.5 \\
9.0^2a_{3}+9.0b_{3}+c_{3}&=0.5
\end{split}
\end{equation*}

3. La continuidad de las derivadas crean adicionalmente $3-1=2$ condiciones, ecuación [(4.25)](#Ec4_25):

\begin{equation*}
\begin{split}
9.0a_{1}+b_{1}=9.0a_{2}+b_{2} \\
14.0a_{2}+b_{2}=14.0a_{3}+b_{3}
\end{split}
\end{equation*}

4. De la ecuación [(4.26)](#Ec4_26) se obtiene la ecuación faltante, es decir,

\begin{equation*}
\begin{split}
a_{1}=0
\end{split}
\end{equation*}

Esta última ecuación especifica de forma exacta el valor de una de las $9$ condiciones requridas, por lo que el problema se reduce a encontrar las restantes $8$ ecuaciones. Estas condiciones se expresan de forma matricial de la siguiente manera:

\begin{align*}
\left[\begin{array}{cccc}
  4.5 & 1 & 0     & 0   & 0 & 0 & 0 & 0 \\
  0   & 0 & 20.25 & 4.5 & 1 & 0 & 0 & 0 \\
  0   & 0 & 49    & 7   & 1 & 0 & 0 & 0 \\
  0   & 0 & 0     & 0   & 0 & 49 & 7 & 1 \\
  3   & 1 & 0     & 0   & 0 & 0  & 0 & 0 \\
  0   & 0 & 0     & 0   & 0 & 81 & 9 & 1 \\
  1   & 0 & -9    & -1  & 0 & 0 & 0 & 0 \\
  0   & 0 & 14    & 1   & 0 & -14 & -1 & 0 \\
\end{array}\right]
\begin{Bmatrix}
  b_{1}  \\
  c_{1}  \\
  a_{2}  \\
  b_{2}  \\
  c_{2}  \\
  a_{3}  \\
  b_{3}  \\
  c_{3}  \\
\end{Bmatrix}
= \begin{Bmatrix}
  1.0  \\
  1.0  \\
  2.5  \\
  2.5  \\
  2.5  \\
  0.5  \\
  0.0  \\
  0.0
\end{Bmatrix}
\end{align*}

Empleando una de las técnicas de resolución de sistemas de ecuaciones lineales vistas en el capítulo anterior, se llega a las respuetas:

\begin{array}{crl}
a_1&=0.0, &b_1&=-1, &c_1&=5.5 \\
a_2&=0.64, &b_2&=-6.76, &c_2&=18.46 \\
a_3&=-1.6, &b_3&=24.6, &c_3&=-91.3
\end{array}

Sustituyendo estos valores en las ecuaciones cuadráticas originales,

\begin{array}{crl}
f_1(x)=-x+5.5, &3.0\le x \le 4.5 \\
f_2(x)=0.64x^2-6.76x+18.46, &4.5 \le x \le 7.0 \\
f_3(x)=-1.6x^2+24.6x-91.3, &7.0\le x \le 9.0 \\
\end{array}

Por último, se emplea la ecuación $f_2$, para predecir el valor en $x=5$, que es $f_2(5)=0.66$

[Volver a la Tabla de Contenido](#TOC)

#### Trazadores cúbicos

[Volver a la Tabla de Contenido](#TOC)

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('./nb_style.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>


```python
from IPython.core.display import HTML
css_file = '../style.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Philosopher', sans-serif;
    font-weight: 400;
    font-size: 2.2em;
    line-height: 100%;
    color: rgb(0, 80, 120);
    margin-bottom: 0.1em;
    margin-top: 0.1em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Philosopher', serif;
    font-weight: 400;
    font-size: 1.9em;
    line-height: 100%;
    color: rgb(245,179,64);
    margin-bottom: 0.1em;
    margin-top: 0.1em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Philosopher', serif;
    margin-top:12px;
    margin-bottom: 3px;
    font-style: italic;
    color: rgb(94,127,192);
}

.text_cell_render h4 {
    font-family: 'Philosopher', serif;
}

.text_cell_render h5 {
    font-family: 'Alegreya Sans', sans-serif;
    font-weight: 300;
    font-size: 16pt;
    color: grey;
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.1em;
    display: block;
}

.text_cell_render h6 {
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 10pt;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 100%;
}

</style>

# Introduction to symbolic python

## Preamble

This first notebook introduces `python` and `sympy`, the computer algebra library.  As with most of the notebooks, we begin by importing the `sympy` language into `python`.  We also initialize the notebook to be able to rpint our results using Latex.

```python
from sympy import init_printing
init_printing()
```

## Creating mathematical variables

The equal symbol, `=`, is an assignment operator in Python.  It assigns what is on the right-hand side (RHS) to what is on the left hand side (LHS).  The LHS is is a name called a computer variable and we are free to choose this name, within certain limits.  The name is basically a reference to a part of computer memory that stores what is on the RHS.  The RHS is then an object, which can be of various types

The most common convention in Python is to use a descriptive name that may consist of more than one word, i.e. a phrase.  The first letter of each word is in lowercase and the words are concatenated by an underscore.  This convention is referred to as _snake case_.  For instance, if I want to create a computer variable to hold a number, I might call it `my_number`.  The name makes sense in as much as if I viewed my code months down the line or if I gve it to someone else, we should all be able to figure out what is was meant to hold.

Python contains many reserved words that make up the syntax of the language.  It is not advised to use these as you computer variable names.  Also steer clear from symbols and place numbers at the end of names.

You can use your favorite search engine and look up naming conventions.  Find the one that suits you.

Let's then create a computer variable named `my_number` and assign the valye $4$ to it.

```python
my_number = 4  # Creating a computer variable and assigning a value to it.
```

A small part of the computer memory is now _called_ `my_number` and it contains the value $4$.  We can access this value by simply calling the name of the computer variable.

```python
my_number
```

The value $4$ is an instance of an object and as an object, is of a certain type.  We can check up the object type with the `type()` function (a keyowrd from the Python syntax that actually perform a, well, _function_.).  All functions in Python have a set of parenthesis at the end.  Inside of these parenthesis we pass the information that the function required to do its job.  These pieces of information are called _arguments_.  We will pass `my_number` as argument.

```python
type(my_number)
```

    int

We note that it of type `int`, which is short for, _integer_, a whole number.  We can reassign the computer variable to hold another value, which may be an instanceof another type of object.

```python
my_number = 4.0  # The .0 indicates that this is a decimal value (we can also just type 4.)
type(my_number)
```

    float

Our computer variable is now an instance of a floating point (`float`) object.  We use the term _instance_, because in each case we create a single example of an object.

Computer variables and the assigment operator do not behave in the same way as their mathematical namesakes.  Look at the code below.

```python
my_number = my_number + 1
my_number
```

Initially, this makes no sense.  Just to be sure that we are all on the same page, let's create another computer variable called `x` and assign the integer value $7$ to it.

```python
x = 7  # Creating a computer variable and assigning the integer value 7 to it
x  # Caling the computer variable to access the value that it holds
```

Now we'll repeat the ` + 1` we did above.

```python
x = x + 1
x
```

Algebraically this makes no sense, until we remember that the `=` symbol is not an equal sign, but an assignment operator, assigning what is on the RHS to what is on the LHS.  On the RHS then we have `x + 1`.  At the moment `x` holds the value `7`.  We then add $1$ to it to make it $7+1=8$.  This new value, $8$, is then assigned to the LHS, which is the computer variable `x`.  The value $8$ then overwrites the previous value held in `x`.

In mathematics, then, we are used to mathematical variables, such as $x$ and $y$ and not computer variables, `x` and `y`.  We have to use a package such as `sympy` to help Python deal with this.  More specifically, we import just the `symbols()` function from the `sympy` package.  Note the syntaxh for doing so.

```python
from sympy import symbols
```

We now have the `symbols()` function added to the many built-in Python functions.  We will use it to reassign our computer variable `x` and turn it into a mathematical variable.

```python
x = symbols('x')  # Setting x as a mathematical symbol
type(x)  # Looking at the type of x
```

    sympy.core.symbol.Symbol

Now `x` is a mathematical symbol (in Python a `sympy.core.symbol.Symbol` object) and we can write an expression such as $x+1$.

```python
x + 1
```

You might wonder what type `x+1` is.  Certainly, `1` is a Python `int`.

```python
type(1)  # Checking the type of the mumber 1
```

    int

```python
type(x + 1)  # Checking the type of x + 1
```

    sympy.core.add.Add

We note that its is a `sympy.core.add.Add` type.  This allows Python to do the actual mathematical addition.

Let's add `y` as a mathematical symbol and try to create $\frac{x}{y}$.  Here we are not using an assignment operator, we simply write the expression $\frac{x}{y}$.

```python
y = symbols('y')
```

Just so that you know, the forward slash, `/`, symbol is used for division.

```python
x / y  # Stating a calculation without an assigment operator is an expression
```

The `sympy` package is great at creating mathematical typesetting.  Let's create an expression with a power.  In Python, two asterisks, `**`, are used to indicated powers.  Below we create the expression $x^2 -x$.

```python
x**2 - x
```

## Transformation of expressions

Now we will get right into things and examine the power of `sympy`.  One of the common tasks in algebra is factorization, as in $x^2 -x = x \left( x - 1 \right)$.

Python functions that act on objects (or expressions) that we create are termed _methods_.  The `.factor()` method is a function that will factor our expression.

```python
(x**2 - x).factor()
```

We can also expand an expression using the `expand()` method.  By the way, we use the single asterisk, `*` symbol for multiplication.

```python
(x * (x - 1)).expand()
```

Just to be clear, we can import the actual functions, `factor()` and `expand()`, from `sympy`.

```python
from sympy import factor, expand
```

Now we can use them as functions (instead of the method syntax we used above).

```python
factor(x**2 - x)
```

```python
expand(x * (x - 1))
```

Lastly, we are still using Python, so we can assign any `sympy` expression to a computer variable and use all of the functions and methods we have learned about.

```python
my_expression = x * (x - 1)  # Creating a computer variable
my_expression
```

```python
expand(my_expression)
```

## Common mathematical functions

The `sympy` package is really great for symbolic (mathematical) computation.  In this section we higlight the difference between numerical (Python) and symbolic (`sympy`) computation.

As an example of numerical computation, let's calculate $\frac{5}{3}$.

```python
5 / 3
```

We get an approximation, i.e. the reciprocal $6$ is terminated by rouding to a $7$.  The real solution is obviously $1.\dot{6}$.  Better yet, the exact solution is $\frac{5}{3}$..

There is package called `math` that expands Python's ability to do numerical computations.  Let's import it and calculate the square root of eight.

```python
import math
```

Because we did not import any specific `math` functions, we have to refer to them by dot notation, i.e. `math.sqrt()` for the square root function in the `math` function.

```python
math.sqrt(8)  # An approxximation of the square root of 8
```

The `math` package contains numerical approximations of constants such as $\pi$.

```python
math.pi
```

The `exp()` function contains an approximation of Euler's number.  We can get this approximation by passing the argument `1`, as in $e^1 = e$.

```python
math.exp(1)
```

Since we are dealing with approximations when doing numerical calculations, we have to deal with a bit of rounding.  Here is $\sin \left( \frac{\pi}{6} \right) = 0.5 $.

```python
math.sin(math.pi/6)
```

The `log()` function in the `math` package calculates the natural logarith.  The naural logarithm of Euler's number is $1$.

```python
math.log(math.exp(1))
```

Now, let's change to symbolic computation and import some useful functions from `sympy`.

```python
from sympy import Rational, sqrt, log, exp, sin, pi, I
```

The `Rational()` function calculates an exact solution for fraction.  Here is $\frac{5}{3}$ again.

```python
# Because we imported the function directly, we don't have to use the dot notation
Rational(5, 3)
```

The `evalf()` method still allows us to get a numerical approximation.

```python
(Rational(5, 3)).evalf()
```

We can even specify the number of significant digits we require by passing it as an argument.

```python
(Rational(5, 3)).evalf(3)  # Significant digits
```

Now for the square root of $8$.

```python
sqrt(8)
```

That's beautiful!  For even more beauty, here's $\pi$ and then a numerical approximation with $40$ significant digits.

```python
pi
```

```python
pi.evalf(40)
```

Euler's number is just as spectacular.

```python
exp(1)
```

```python
exp(1).evalf(40)  # Foury significant digits
```

The trigonometric and logarithmic expressions from above will also now give us an exact solution.

```python
sin(pi / 6)
```

```python
log(exp(1))
```

## Substitutions

While we create symbolic expressions, `sympy` does not exclude use from substituting actual numerical values into our mathematical variables.  The `.subs()` method does the job.  Below we create an expression and assign it to the computer variable `expr`.

```python
expr = x + 4  # x is a mathematical symbol
expr
```

We have to specify the mathematical variable we want substituted and then the value we want to substitute with.

```python
expr.subs(x, 3)
```

If we have more than one mathematical symbol, we can substitute all of them using the syntax below.

```python
expr = x + y  # Overwriting the expr computer variable
expr.subs([(x, 2), (y, 5)])
```

The square bracket, `[]` notation creates a list object.  An alternative syntax uses dictionary objects with curly braces, `{}`.  We will learn more about these at a later stage.

```python
expr.evalf(subs = {x:2, y:5})
```

## Equality

Substitution provides a great segway into Boolean logic.  The double equal symbol, `==` evaluates both sides of an equation to see of they are indeed equal.

```python
expr.subs(x, 3) == 7
```

    False

Indeed $7 = 7$, the LHS equals the RHS.

The `sympy` adheres to the principle of exact structural equality.  Let's look at the expression $\left( x+1  \right)^2$ and its expansion $x^2 + 2x +1$.

```python
(x + 1)**2
```

```python
((x + 1)**2).expand()
```

We know that these two expression are equal to each other.  Let's test this assumption.

```python
(x + 1)**2 == x**2 + 2 * x + 1
```

    False

We see a `False`.  This is what we mean by adherence to the principle of exact structural equality. If we expand the LHS or factor the RHS, we will get equality.

```python
((x + 1)**2).expand() == x**2 + 2 * x + 1
```

    True

```python
((x + 1)**2) == (x**2 + 2 * x + 1).factor()
```

    True

## Roundtripping to numerical packages

While `sympy` is great at symbolic computations, we often need numerical evaluations.  We did this using `.subs()` and `.evalf()` methods.  They do single substitutions, but what if want computations on many values.  To do this, we rondtrip to some of the Python packages that are designed for numerical computations such as `numpy` (numerical Python) and `scipy` (scientific Python).

Fortunately, `sympy` provides the `lambdify()` function.  Imagine then that we want to calculate the $\sin \left( x \right)$ for the integere values in the closed domain $\left[ -3, 3 \right]$.  The `arange()` function in the `numpy` package will let us create this list of values.  We will ause three argument.  The first is the start value, the second is the end value (`numpy` excludes this from the final list, so we will use `4`), and the step size, which is $1$.

```python
from numpy import arange
```

```python
my_domain = arange(-3, 4, 1)  # Creating the list of integers in our chosen domain
my_domain
```

    array([-3, -2, -1,  0,  1,  2,  3])

The `arange()` function creates a `numpy` object called an `array`.  We note the seven integers in our array object, `my_domain`.  Now we overwrite the `expr` computer variable to hold the sine function.

```python
expr = sin(x)
expr
```

```python
from sympy import lambdify  # Importing the required function
```

The `lambdify()` function as used below takes three arguments.  First is the mathematical variable of interest.  Then follows teh actual expression, and finally the package we want to use for the calculation.  We assign this to the computer variable `f`.

```python
f = lambdify(x, expr, 'numpy')
```

It is now a simple matter of providing the seven numbered array to the computer variable.  All seven values in the array will be passed to the expression.

```python
f(my_domain)
```

    array([-0.14112001, -0.90929743, -0.84147098,  0.        ,  0.84147098,
            0.90929743,  0.14112001])

```python

```

```python

```

```python

```

```python

```

Python/3. Computational Sciences and Mathematics/Linear Algebra/0.0 Start Here/0.1 Getting_started.ipynb


```python
# This mounts your Google Drive to the Colab VM.
from google.colab import drive
drive.mount('/content/drive')

# TODO: Enter the foldername in your Drive where you have saved the unzipped
# assignment folder, e.g. 'cs231n/assignments/assignment1/'
FOLDERNAME = 'cs231n/assignment2'
assert FOLDERNAME is not None, "[!] Enter the foldername."

# Now that we've mounted your Drive, this ensures that
# the Python interpreter of the Colab VM can load
# python files from within it.
import sys
sys.path.append('/content/drive/My Drive/{}'.format(FOLDERNAME))

# This downloads the CIFAR-10 dataset to your Drive
# if it doesn't already exist.
%cd /content/drive/My\ Drive/$FOLDERNAME/cs231n/datasets/
!bash get_datasets.sh
%cd /content/drive/My\ Drive/$FOLDERNAME
```

    Mounted at /content/drive
    /content/drive/My Drive/cs231n/assignment2/cs231n/datasets
    /content/drive/My Drive/cs231n/assignment2

# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train. One idea along these lines is batch normalization, proposed by [1] in 2015.

To understand the goal of batch normalization, it is important to first recognize that machine learning methods tend to perform better with input data consisting of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features. This will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance, since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, they propose to insert into the network layers that normalize batches. At training time, such a layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```python
# Setup cell.
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams["figure.figsize"] = (10.0, 8.0)  # Set default size of plots.
plt.rcParams["image.interpolation"] = "nearest"
plt.rcParams["image.cmap"] = "gray"

%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """Returns relative error."""
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print(f"  means: {x.mean(axis=axis)}")
    print(f"  stds:  {x.std(axis=axis)}\n")
```

    =========== You can safely ignore the message below if you are NOT working on ConvolutionalNetworks.ipynb ===========
    	You will need to compile a Cython extension for a portion of this assignment.
    	The instructions to do this will be given in a section of the notebook below.

```python
# Load the (preprocessed) CIFAR-10 data.
data = get_CIFAR10_data()
for k, v in list(data.items()):
    print(f"{k}: {v.shape}")
```

    X_train: (49000, 3, 32, 32)
    y_train: (49000,)
    X_val: (1000, 3, 32, 32)
    y_val: (1000,)
    X_test: (1000, 3, 32, 32)
    y_test: (1000,)

# Batch Normalization: Forward Pass
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network.
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))

# Means should be close to zero and stds close to one.
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])

# Now means should be close to beta and stds close to gamma.
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

    Before batch normalization:
      means: [ -2.3814598  -13.18038246   1.91780462]
      stds:  [27.18502186 34.21455511 37.68611762]

    After batch normalization (gamma=1, beta=0)
      means: [5.99520433e-17 6.93889390e-17 8.32667268e-19]
      stds:  [0.99999999 1.         1.        ]

    After batch normalization (gamma= [1. 2. 3.] , beta= [11. 12. 13.] )
      means: [11. 12. 13.]
      stds:  [0.99999999 1.99999999 2.99999999]

```python
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

    After batch normalization (test-time):
      means: [-0.03927354 -0.04349152 -0.10452688]
      stds:  [1.01531428 1.01238373 0.97819988]

# Batch Normalization: Backward Pass
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```python
# Gradient check batchnorm backward pass.
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)

# You should expect to see relative errors between 1e-13 and 1e-8.
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.6674604875341426e-09
    dgamma error:  7.417225040694815e-13
    dbeta error:  2.379446949959628e-12

# Batch Normalization: Alternative Backward Pass
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\...\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
& \mu=\frac{1}{N}\sum_{k=1}^N x_k  &  v=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2 \\
& \sigma=\sqrt{v+\epsilon}         &  y_i=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hard part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```python
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

    dx difference:  9.890497291190823e-13
    dgamma difference:  0.0
    dbeta difference:  0.0
    speedup: 1.62x

# Fully Connected Networks with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

**Hint:** You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`.

```python
np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.
for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            normalization='batchnorm')

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
  if reg == 0: print()
```

    Running check with reg =  0
    Initial loss:  2.2611955101340957
    W1 relative error: 1.10e-04
    W2 relative error: 5.65e-06
    W3 relative error: 4.14e-10
    b1 relative error: 2.78e-09
    b2 relative error: 2.22e-08
    b3 relative error: 1.02e-10
    beta1 relative error: 6.94e-09
    beta2 relative error: 1.89e-09
    gamma1 relative error: 7.47e-09
    gamma2 relative error: 3.35e-09

    Running check with reg =  3.14
    Initial loss:  6.996533220108303
    W1 relative error: 1.98e-06
    W2 relative error: 2.28e-06
    W3 relative error: 1.11e-08
    b1 relative error: 5.55e-09
    b2 relative error: 2.22e-08
    b3 relative error: 2.10e-10
    beta1 relative error: 6.32e-09
    beta2 relative error: 3.39e-09
    gamma1 relative error: 6.27e-09
    gamma2 relative error: 4.14e-09

# Batch Normalization for Deep Networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```python
np.random.seed(231)

# Try training a very deep net with batchnorm.
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

    Solver with batch norm:
    (Iteration 1 / 200) loss: 2.340974
    (Epoch 0 / 10) train acc: 0.107000; val_acc: 0.115000
    (Epoch 1 / 10) train acc: 0.314000; val_acc: 0.266000
    (Iteration 21 / 200) loss: 2.039365
    (Epoch 2 / 10) train acc: 0.385000; val_acc: 0.279000
    (Iteration 41 / 200) loss: 2.041102
    (Epoch 3 / 10) train acc: 0.494000; val_acc: 0.308000
    (Iteration 61 / 200) loss: 1.753902
    (Epoch 4 / 10) train acc: 0.531000; val_acc: 0.307000
    (Iteration 81 / 200) loss: 1.246584
    (Epoch 5 / 10) train acc: 0.574000; val_acc: 0.313000
    (Iteration 101 / 200) loss: 1.320589
    (Epoch 6 / 10) train acc: 0.634000; val_acc: 0.338000
    (Iteration 121 / 200) loss: 1.157328
    (Epoch 7 / 10) train acc: 0.683000; val_acc: 0.323000
    (Iteration 141 / 200) loss: 1.135180
    (Epoch 8 / 10) train acc: 0.770000; val_acc: 0.328000
    (Iteration 161 / 200) loss: 0.677145
    (Epoch 9 / 10) train acc: 0.783000; val_acc: 0.343000
    (Iteration 181 / 200) loss: 0.935965
    (Epoch 10 / 10) train acc: 0.803000; val_acc: 0.339000

    Solver without batch norm:
    (Iteration 1 / 200) loss: 2.302332
    (Epoch 0 / 10) train acc: 0.129000; val_acc: 0.131000
    (Epoch 1 / 10) train acc: 0.283000; val_acc: 0.250000
    (Iteration 21 / 200) loss: 2.041970
    (Epoch 2 / 10) train acc: 0.316000; val_acc: 0.277000
    (Iteration 41 / 200) loss: 1.900473
    (Epoch 3 / 10) train acc: 0.373000; val_acc: 0.282000
    (Iteration 61 / 200) loss: 1.713156
    (Epoch 4 / 10) train acc: 0.390000; val_acc: 0.310000
    (Iteration 81 / 200) loss: 1.662209
    (Epoch 5 / 10) train acc: 0.434000; val_acc: 0.300000
    (Iteration 101 / 200) loss: 1.696059
    (Epoch 6 / 10) train acc: 0.535000; val_acc: 0.345000
    (Iteration 121 / 200) loss: 1.557987
    (Epoch 7 / 10) train acc: 0.530000; val_acc: 0.304000
    (Iteration 141 / 200) loss: 1.432189
    (Epoch 8 / 10) train acc: 0.628000; val_acc: 0.339000
    (Iteration 161 / 200) loss: 1.033931
    (Epoch 9 / 10) train acc: 0.661000; val_acc: 0.340000
    (Iteration 181 / 200) loss: 0.901034
    (Epoch 10 / 10) train acc: 0.726000; val_acc: 0.318000

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```python
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch Normalization and Initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train eight-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```python
np.random.seed(231)

# Try training a very deep net with batchnorm.
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
    print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
    bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

    bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    bn_solver.train()
    bn_solvers_ws[weight_scale] = bn_solver

    solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    solver.train()
    solvers_ws[weight_scale] = solver
```

    Running weight scale 1 / 20
    Running weight scale 2 / 20
    Running weight scale 3 / 20
    Running weight scale 4 / 20
    Running weight scale 5 / 20
    Running weight scale 6 / 20
    Running weight scale 7 / 20
    Running weight scale 8 / 20
    Running weight scale 9 / 20
    Running weight scale 10 / 20
    Running weight scale 11 / 20
    Running weight scale 12 / 20
    Running weight scale 13 / 20
    Running weight scale 14 / 20
    Running weight scale 15 / 20
    Running weight scale 16 / 20

    /content/drive/My Drive/cs231n/assignment2/cs231n/layers.py:149: RuntimeWarning: divide by zero encountered in log
      loss = np.sum(-np.log(correct_score/score_sum))

    Running weight scale 17 / 20
    Running weight scale 18 / 20
    Running weight scale 19 / 20
    Running weight scale 20 / 20

```python
# Plot results of weight scale experiment.
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers_ws[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

  best_val_accs.append(max(solvers_ws[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs. weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs. weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs. weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the weight initialization scale affect models with/without batch normalization differently, and why?

## Answer:
The weight scaling affect largely on without normalized models as seen above. However, with batch normalized model, it deosn't affect much. Because batch normalization improves gradient flow and speed up the gradient descent process.

# Batch Normalization and Batch Size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```python
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)

    # Try training a very deep net with batchnorm.
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

    No normalization: batch size =  5
    Normalization: batch size =  5
    Normalization: batch size =  10
    Normalization: batch size =  50

```python
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:
During training, the accuracy increase with increase in batch size because the accuracy is calculated based on statistics of that particular batch and with small batch the statistics may not generalize properly. However, during test time, the accuracy is calculated with running statistics (mean and variance) so the accuracy don't get much affected by batch size.

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:
[FILL THIS IN]

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_forward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization.

# Simulate the forward pass for a two-layer network.
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(D3)
beta = np.zeros(D3)

# Means should be close to zero and stds close to one.
print('After layer normalization (gamma=1, beta=0)')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0])

# Now means should be close to beta and stds close to gamma.
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

    Before layer normalization:
      means: [-59.06673243 -47.60782686 -43.31137368 -26.40991744]
      stds:  [10.07429373 28.39478981 35.28360729  4.01831507]

    After layer normalization (gamma=1, beta=0)
      means: [-2.22044605e-16 -7.40148683e-17 -7.40148683e-17 -5.92118946e-16]
      stds:  [0.99999995 0.99999999 1.         0.99999969]

    After layer normalization (gamma= [3. 3. 3.] , beta= [5. 5. 5.] )
      means: [5. 5. 5. 5.]
      stds:  [2.99999985 2.99999998 2.99999999 2.99999907]

```python
# Gradient check batchnorm backward pass.
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

# You should expect to see relative errors between 1e-12 and 1e-8.
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.2514082056813097e-09
    dgamma error:  1.980045566295477e-12
    dbeta error:  2.5842537629899423e-12

# Layer Normalization and Batch Size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```python
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:
2 will cause layer normalization to be affected, mainly because layer normalization statistic on all neurons in each layer. If the dimension of the neuron is too small, the statistic difference may be too large and the performance of the model fluctuates too much.