# Bevezetés

### Python programozási nyelv

A Python egy open-source (OS), interpretált, általános célú programozási nyelv (vagy script-nyelv).

**Tulajdonságai:**
- Objektum orientált
- Interpretált
    - Nem szükséges fordítani (mint a pl a *C++*-t), elég csak beírni a parancsot, és már futtatható is a kód
    - Alkalmas teszi számítások gyors-prototipizálására
    - Cserébe lassú
- Open-source:
    - Ingyenes
    - Folyamatosan karban tartott
    - Széles körben felhasznált iparban és akadémiában is
    - Nagy "Community", sok segédlettel, fórummal (pl.: [stackoverflow](https://stackoverflow.com/questions/tagged/python))
- Moduláris:
    - Rengetek feladatra létezik "*package*" (pl.: numerikus számításokra *numpy*/*scipy*, szimbolikus számításokra *sympy*, táblázatfájl-kezelésre *CSV*)
    - Csak azt kell behívni, amire szükségünk van
    - Ismerni kell a *package* ekoszisztémát, mik léteznek, mi mire jó, stb...
- Sok IDE (*Integrated Development Environment*) létezik:
    - Alapvetően shell (terminál) alapú
    - Notebook: **_jupyter notebook_**, *jupyter lab*
    - Szövegszerkesztő: *Spyder*, *VS Code* (ingyenes/open source - ezek tartalmaznak *Debugger*-t is)
    - Fizetős szövegszerkeszők (lista nem teljes): *Visual Studio*, *PyCharm*, stb...

### Jupyter notebook működése (+ Python kernel):

Legfontosabb tudnivalók:

- Csak egy *front-end*, ami kommunikál egy *kernel*-lel (ez a kernel menüben választható).
- Két mód létezik:
    - Command mode (cellaműveleteket lehet végezni)
    - Edit mode (szövegbevitel cellába)
- Command mode (`ESC` billentyű lenyomásával érhető el, kék csík a cella kijelölése esetén):
    - Notebook mentése: `s`
    - Cellák hozzáadása: `b` az aktuális cella alá, `a` az aktuális cella fölé
    - Cella törlése: kétszer egymás után a `d` billentyű lenyomása
    - Cella törlésének visszavonása: `z`
    - Cella másolás: `c`, kivágás: `x`, beillesztés az aktuális cella alá: `v`
    - Számozás bekapcsolása a cella soraira: `l` (kis L), vagy `Shift + l` az összes cellára
    - Cellamódok: futtatandó kód: `y`, nyers kód (nem futtatható): `r`, markdown (formázott szöveg): `m`
- Edit mode (`Enter` billenytű lenyomásával érhető el, zöld szín):
    - Sor "kikommentelése"/"visszaállítása": `Ctrl + /`
    - Több kurzor lehelyezése: `Ctrl + Bal egérgomb`
    - Téglalap-szerű kijelölés (rectangular selection): `Alt + Bal egérgomb` "húzása" (dragging)
- Közös
    - Cella futtatása, majd cellaléptetés: `Shift + Enter` (ez létrehoz egy új cellát, ha nincs hova lépnie)
    - Cella futtatása cellaléptetés nélkül: `Ctrl + Enter`

**Jupyter notebook help-jének előhozása**: *Edit mode*-ban `h` lenyomásával
**Python help**: Kurzorral a függvény nevén állva `Shift + Tab` vagy egy cellába `?"fv_név"` beírása és futtatása

### A bevezető felépítése:
- [Alapműveletek](#Alapmuveletek)
- [Összetett függvények](#Osszetettfuggvenyek)
- [Saját függvények](#Sajatfuggvenyek)
- [Osztályok](#Osztalyok)
- [Vezérlési szerkezetek](#Vezerlesiszerkezetek)
- [Külső függvénykönyvtárak](#Kulsofuggvenykonyvtarak)
    - [Szimbolikus matematikai műveletek](#Szimbolikus)
    - [Deriválás/Integrálás](#DerivalIntegral)
    - [Vektor és mátrixszámítások Sympy-ban](#SzimVektorMatrix)
    - [Vektor és mátrixszámítások Scipy-ban](#NumVektorMatrix)
    - [Egyenletek megoldása Scipy-ban](#Egyenletek)
    - [Szimbolikus függvényekből numerikus függvények](#SymToNum)
- [Egyszerű ábrák készítése](#Egyszeruabrak)

# Python bevezető

## Alapműveletek (Shift/Ctrl + Enter-rel futtassuk)

```python
17 + 7 #Összeadás
```

    24

```python
333 - 7 #Kivonás
```

    326

```python
11 * 22 #Szorzás
```

    242

```python
7/9 #Osztás (ez nem egész (int) lesz: külön típus float)
```

    0.7777777777777778

```python
0.3-0.1-0.2 # float: számábrázolási hiba lehet!!
```

    -2.7755575615628914e-17

```python
2**3 # Hatványozás (** és NEM ^!)
```

    8

```python
2**(0.5) # Gyökvönás hatványozás segítségével
```

    1.4142135623730951

```python
5e-3 #normálalak e segítségével (vagy 5E-3)
```

    0.005

Néhány alapművelet működik szövegre is

```python
'str1_' + 'str2_' #Összeadás
```

    'str1_str2_'

```python
2 * 'str2_' #Szorzás
```

    'str2_str2_'

## Összetettebb függvények

```python
sin(2) #szinusz
```

Összetettebb függvények már nincsenek a python alapnyelvben - ilyenkor szükséges behívni külső csomagokat, pl a **math** csomagot

```python
import math
```

```python
sin(2) # ez így továbbra sem létezik
```

```python
math.sin(2)
```

    0.9092974268256817

```python
# Több parancs együttes beírásakor nem látszik, csak az utolsó sor kimenete: print függvény alkalmazása!
print(math.sqrt(2))
print(math.tan(2))
print(math.atan(2))
```

    1.4142135623730951
    -2.185039863261519
    1.1071487177940904

```python
# Kimenet el is rejthető a ; segítségével ("suppress output")
1+1;
```

Amennyiben szükséges, definiálhatunk mi is saját változókat az `=` jellel.
Megjegyzés: a `=` értékadó függvénynek nincs kimenete

```python
a=2
b=3
c=4.0 # automatikus típusadás
```

```python
(a+b*c)**a # a legáltalánosabb típus lesz a kimenet (int < float)
```

    196.0

```python
# Fontos, hogy igyekezzük kerülni védett változó neveket! ILYET NE!
math.sqrt = 1
math.sqrt(2)

# KERNEL RESTART SZÜKSÉGES
```

**Ha véletlenül ilyet teszünk, akkor érdemes újraindítani a *kernel* a fent látható körkörös nyíllal, vagy a *Kernel* $\rightarrow$ *Restart* segítségével**

## Saját függvények

Szerkezet:
```python
def function(*arguments):
    instruction1
    instruction2
    ...
    return result
```

A függvény alá tartozó utasításokat tabulátoros behúzással (indent) kell beírni (nincs `{}` zárójel, vagy `end`). A függvény neve után jönnek az argumentumok majd kettősponttal `:` jelezzük, hogy hol kezdődik a függvény.

```python
def foo(x):
    return 3*x

def bar(x,y):
    a = x+y**2
    return 2*a + 4
```

```python
print(foo(3))
print(foo(3.))
print(foo('szöveg_'))

print(bar(3,4.))
```

    9
    9.0
    szöveg_szöveg_szöveg_
    42.0

Lehetséges úgynevezett anonim függvényeket (*anonymous function* vagy *lambda function*) is létrehozni, amely gyors módja az egyszerű, egysoros függvények létrehozására:

```python
lambda arguments: instruction
```

Ez akár egy változóhoz is hozzárendelhető, mint egy szám vagy string.

```python
double = lambda x : x*2
multiply = lambda x,y : x*y
```

```python
print(double(3))
print(multiply(10,3))
```

    6
    30

## Osztályok

```python
def foo(x):
    return x**2

class MyClass:
    def __init__(self,x,y,z):
        self.square = foo(x)-z
        self.cubic = y**3+foo(y)

    @classmethod
    def createfrom_x(cls,x):
        return MyClass(x,x,x)

    def return_stuff(self):
        return self.square+3*self.cubic
```

```python
mcl=MyClass.createfrom_x(2)
```

```python
mcl.return_stuff()
```

    38

```python
def foo(x):
    return x**2

class MyClass:
    def __init__(self,x,y,z):
        self.square = foo(x)-z
        self.cubic = y**3+foo(y)

    @classmethod
    def createfrom_x(cls,x):
        return MyClass(x,x,x)

    def return_stuff(self):
        return self.square+3*self.cubic
```

```python
mcl=MyClass.createfrom_x(2)
```

```python
mcl.return_stuff()
```

    38

## Vezérlési szerkezetek (Control Flow) - csak a legfontosabbak

### Listák

```python
lista = [1,2,3,4,"valami",[1.0,4]]
```

```python
print(lista[0]) # lista 1. eleme
print(lista[3]) # lista 4. eleme
print(lista[-1]) # negatív számokkal hátulról indexeljük a listát, és (-1)-től indul
print(lista[-2]) # lista utolsó előtti eleme
```

    1
    4
    [1.0, 4]
    valami

```python
print(lista[1:-1]) # egyszerre több elem [inkluzív:exklúzív módon]
print(lista[1:2]) # egyszerre több elem [inkluzív:exklúzív módon]
print(lista[2:]) # lista utolsó elemét is figyelembe vesszük
```

    [2, 3, 4, 'valami']
    [2]
    [3, 4, 'valami', [1.0, 4]]

```python
lista = [2,3,64,89,1,4,9,0,1]

lista.sort()
lista
```

    [0, 1, 1, 2, 3, 4, 9, 64, 89]

### if-then-else

```python
if condition:
    instruction1
elif condition2:
    instruction2
else:
    intsturction3
```

```python
a=4
if a<=3:
    print('"a" nem nagyobb, mint 3')
elif a>=10:
    print('"a" nem kisebb, mint 10')
else:
    print('"a" nagyobb mint 3, de kisebb mint 10')
```

    "a" nagyobb mint 3, de kisebb mint 10

### for ciklus (for loop)
```python
for i in array:
    instruction
```

```python
for i in range(3):
    print(i)

print()

for (i,elem) in enumerate(lista):
    print('lista ',i,'. eleme: ',elem,sep='') # több elem printelése egyszerr, szeparátor = ''

```

    0
    1
    2

    lista 0. eleme: 0
    lista 1. eleme: 1
    lista 2. eleme: 1
    lista 3. eleme: 2
    lista 4. eleme: 3
    lista 5. eleme: 4
    lista 6. eleme: 9
    lista 7. eleme: 64
    lista 8. eleme: 89

## Listák gyors létrehozása (List comprehension)

```python
lista2 = [3*i**2 for i in range(2,5)] # range: 2,3,4
lista2
```

    [12, 27, 48]

```python
lista3 = list(range(10))
lista3
```

    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

```python
myfun = lambda x: 3*x**2

lista4 = [myfun(i) for i in range(2,10) if i%3 != 0] # ha i nem osztható 3-al
lista4
```

    [12, 48, 75, 147, 192]

# Külső függvénykönyvtárak:

A mérnöki gyakorlatban előforduló alapszámításokhoz 2 fő csomag importálása ajánlott:
- `sympy`: szimbolikus számításokhoz
- `scipy`/`numpy`: numerikus számításokhoz (pl mátrix algebra)
- `matplotlib.pyplot`: Függvények ábrázolása
A `module`-ok hívása az alábbi módokon történhet
- `import` *`modulename`* : importálja az adott `module`-t. Ekkor a `module`-ban lévő függvények hívása a `module.functionname(arguments)` segítségével történik (Az *1_Alapok.ipynb*-ben láthattunk erre példát a `math` `module` esetében).
- `import` *`modulename`* `as` *`alias`* : hasonló az előzőhöz, de megválasztjuk, hogy milyen *alias*-ként hivatkozunk a `module`-unkra
- `from` *`modulename`* ` import` *`function1, function2, ...`* : csak bizonyos függvények importálása (nem szükséges a `module`-ra hivatkozni a függvények hívása során)
- `from` *`modulename`* ` import *` : a `module` összes függvényének importálása (nem szükséges a `module`-ra hivatkozni a függvények hívása során)

## Szimbolikus matematikai műveletek

```python
import math
import sympy as sp
import scipy as sc
sp.init_printing()
```

```python
F, m, a, b, c, x = sp.symbols("F m a b c x")
```

```python
F=m*a
```

```python
F.subs([(a,7)])
```

```python
F.subs([(a,7),(m,1.1)])
```

```python
((a+b)**3).expand()
```

```python
((a+b)**7 - (b+2*a)**3).expand()
```

```python
(a**2+b**2+2*a*b).factor()
```

```python
sp.factor(a**2+b**2+2*a*b)
```

```python
sp.factor(b**3 + 3*a*b**2 + 3*a**2*b + a**3)
```

```python
a/b+c/b+7/b
```

```python
sp.ratsimp(a/b+c/b+7/b)
```

```python
(a/b+c/b+7/b).ratsimp()
```

```python
(sp.sin(x)**2 + sp.cos(x)**2).simplify()
```

```python
(sp.cos(2*x)).expand()
```

```python
sp.expand_trig(sp.cos(2*x))
```

```python
import scipy.constants
```

```python
sc.constants.golden
```

```python
math.sqrt(-1+0j)
```

```python
sc.sqrt(-1+0j)
```

```python
sp.limit(sp.sin(x)/x,x,0)
```

Taylor-sor megadása. Első paraméter a függvény, második a változó, harmadik az érték ami körül akarjuk a sort kifejteni, negyedik pedig a fokszám:

$$f\left(x\right) \approx \sum\limits_{i=0}^{N} \dfrac{\left(x - x_0\right)^i}{i!} \left.\dfrac{\mathrm{d}^i f}{\mathrm{d} x^i}\right|_{x = x_0}$$

```python
sp.series(sp.sin(x),x,0,20)
```

### Deriválás/Integrálás

```python
a,Δt,x = sp.symbols('a,Δt,x')
```

Deriválás

```python
sp.diff(sp.sin(x**3),x)
```

Többszöri deriválás

```python
sp.diff(sp.sin(x**3),x,3)
```

Integrálás

```python
sp.integrate(1/(1+x),x)
```

Határozott integrál

```python
sp.integrate(1/(1+x),(x,1,2))
```

```python
a = sp.Symbol('a')
```

```python
sp.integrate(1/(x**2 + a),x)
```

Láncszabályt is ismeri a szoftver

```python
y = sp.Symbol('y')
def f(x):
    return x**2
def g(y):
    return sp.sin(y)
```

```python
f(g(y))
```

```python
sp.diff(f(g(y)),y)
```

```python
sp.diff(g(f(x)),x)
```

Sok esetben nem létezik zárt alakú kifejezés a határozatlan integrálhoz. Ebben az esetben használhatjuk a [numerikus integrálást](https://en.wikipedia.org/wiki/Numerical_integration) a határozott integrál számításához:

```python
sp.integrate(sp.sin(sp.cos(x)),x)
```

[numerikus intágráls SymPy-vel](https://docs.sympy.org/latest/modules/integrals/integrals.html#numeric-integrals)

nem triviális

## Vektor- és mátrixszámítások *sympy*-ban
Kisebb méretek esetén, amelyek akár szimbolikus számokat is tartalmaznak

```python
v1= sp.Matrix([2.,3.,4.]) # oszlopvektor
v2= sp.Matrix([[3.,-2.,-7.]]) # sorvektor (még 1 szögletes zárójel)
mx1 = sp.Matrix([[1.,2.,3.],[2.,0.,4.],[3.,4.,1.]])
mx2 = sp.Matrix([[1.,2.,3.],[4.,5.,6.],[7.,8.,9.]])
EM = sp.eye(3) # egységmátrix
```

```python
v1
```

```python
v2
```

```python
v2.multiply(v1)
```

```python
mx2.multiply(v1)
```

```python
v2.multiply(mx2)
```

```python
EM
```

```python
mx1.eigenvals() # sajátértékek és multiplicitásuk (racionális számokkal)
```

```python
mx1.eigenvals(rational=False) # sajátértékek numerikusan
```

```python
mx1.eigenvects() # sajátvektorok numerikusan
```

```python
mx1.det() # mx1 determinánsa
```

```python
Ix,Iy,Ixy = sp.symbols('Ix,Iy,Ixy')
mxSP=sp.Matrix([[Ix,-Ixy],[-Ixy,Iy]])
display(mxSP)

print('\n Sajátértékek, vektorok: \n')
mxSP.eigenvects()
```

```python
mxSP=sp.Matrix([[Ix,0],[0,Iy]])
display(mxSP)

print('\n Sajátértékek, vektorok: \n')
mxSP.eigenvects()
```

## Vektor- és mátrixszámítások *scipy*-ban
Nagy mátrixok és vektorok esetén érdemes ezt használni, vagy ha sok numerikus adattal dolgozik az ember. Továbbá a sajátérték-sajátvektor számítások is jobban megoldott ebben a csomagban

```python
import sympy as sp
import scipy as sc
import scipy.linalg
```

```python
v1= sc.array([2.,3.,4.])
v2= sc.array([3.,-2.,-7.])
mx1 = sc.array([[1.,2.,3.],[2.,0.,4.],[3.,4.,1.]])
mx2 = sc.array([[1.,2.,3.],[4.,5.,6.],[7.,8.,9.]])
```

```python
print( sc.dot(mx2,v1) )  # skalár szorat mx2*v1
print( sc.dot(v1,mx2) )  # skalár szorat transpose(v1)*mx2
print( sc.cross(v1,v2) ) # keresztszorzat v1×v2
```

    [20. 47. 74.]
    [42. 51. 60.]
    [-13.  26. -13.]

```python
(λ,V) = sc.linalg.eig(mx1) # Mátrix sajátértékei, sajátvektorai a λ és a V változókban

# kiíratás 2 értékes tizedes jeggyel és for ciklusra példa
for (i,v) in enumerate(V):
    print(i+1, '. sajátérték és -vektor:',sep='')
    print('λ = ', sp.N(λ[i],3), '; v = ', [sp.N(e,3) for e in v], sep='', end='\n\n')
```

    1. sajátérték és -vektor:
    λ = 6.76; v = [0.529, 0.834, -0.157]

    2. sajátérték és -vektor:
    λ = -1.15; v = [0.543, -0.475, -0.693]

    3. sajátérték és -vektor:
    λ = -3.61; v = [0.653, -0.281, 0.704]

Figyeljük meg, hogy míg a sympy a sajátvektor utolsó koordinátáját 1-nek választja, addig a scipy a sajátvektorokat 1-re normálja!

## Egyenletek megoldása

```python
x, y, z = sp.symbols("x y z")
```

```python
egy1=77*x+6 - 160
```

```python
sp.solve(egy1,x)
```

```python
sp.solve(x**2 - 5, x)
```

```python
x
```

```python
e1 = 3*z + 2*y + 1*x - 7
e2 = 4*z + 0*y + 2*x - 8
e3 = 1*z + 4*y + 3*x - 9
```

```python
sp.solve([e1,e2,e3],[x,y,z])
```

```python
def f(x):
    return x**2 - 4
```

```python
# Numerikus gyökkereséssel
import scipy.optimize

gyok = scipy.optimize.brentq(f,0,10)
print(gyok)
```

    2.0

### Szimbolikus függvényekből numerikus függvények

Ezt a legegyszerűbben [*lambda function*](#Sajatfuggvenyek)-ök segítségével tehetjük meg

```python
a,b,c,x = sp.symbols('a,b,c,x')
adat = [(a, 3.0), (b,-2.), (c,1.0)]

masodfoku_x = a*x**2 + b*x + c

masodfoku_x
```

```python
masodfoku_num = sp.lambdify(x,masodfoku_x.subs(adat))
masodfoku_num(1.)
```

# Egyszerű ábrák készítése

```python
import scipy as sc # Ez egyszerűsíti a sok függvénnyel való munkát
import matplotlib.pyplot as plt # matplotlib csomag python rajzolójának behívása
```

## Függvények ábrázolása egyszerű diagramokon

```python
def f(x):
    return sc.sin(x)*(1-4*sc.exp(0.7*x)/(x**3))

x1 = sc.linspace(2,10,30)
y1 = f(x1)
```

```python
plt.figure(figsize=(20/2.54,12/2.54)) #inch-ben lehet megadni az ábra méretét - nem kötelező de érdemes, ha dokumentációba szánjuk

#plot objektum létrehozása
plt.plot(x1,y1)

# tengelyfeliratok
plt.xlabel('x',fontsize=12)
plt.ylabel('f(x)',fontsize=12)

# rácsozás
plt.grid()

# ábra mentés (még azelőtt kell, mielőtt kirajzolunk)
# plt.savefig("abra1.png", bbox_inches="tight")
# plt.savefig("abra1.pdf", bbox_inches="tight")

# plot kirajzolása
plt.show()

# plot mentése
plt.savefig('abra1.png')  # lehet .pdf, .svg formátumokban is menteni
```

```python
plt.plot(x1,y1,linewidth=3,color='k',linestyle='-.') # vonalvastagság, -szín, és -stílus módosítása
plt.xlabel('x')
plt.ylabel('f(x)')
# megjelenítési határ
plt.ylim(-0.5,0.5) #-0.5...0.5
plt.xlim(None,8) # xmin...8
plt.grid()

plt.show()
```

```python
def g(x):
    return sc.sqrt(x)
def h(x):
    return x**2

x2 = sc.linspace(0.,1.,100)

plt.plot(x2,g(x2))
plt.plot(x2,h(x2))

plt.xlabel('x',fontsize=12)
# felirat
plt.legend((r'$\sqrt{x}$',r'$x^2$'), fontsize=12, loc = 5) # loc: 1:jobbfenn, 2: balfent, 3:ballent,...10
plt.grid()

plt.show()
```

```python
x3 = sc.arange(1,14+1)
y3 = sc.exp(x3)/(1.4e4)
plt.plot(x3, y3, linestyle = '', marker = 'o')
plt.xlabel('szorgalmi hét sorszáma')
plt.ylabel('hallgatók terheltsége [%]-ban')
plt.grid()
plt.show()
```

```python
plt.plot(x3, y3, linestyle = '-', marker = 'o')
plt.yscale('log')
plt.xlabel('szorgalmi hét sorszáma')
plt.ylabel(r'hallgatók terheltsége [%]-ban')
plt.grid()
plt.show()
```

## Parametrikus függvényábrázolás:

```python
t = sc.linspace(0,2*sc.pi,100)
x = sc.sin(t)
y = sc.cos(t)

plt.plot(x, y, linestyle = '--')
plt.grid()
plt.axis('equal')
plt.show()
```

## 3D függvényábrázolás

```python
from mpl_toolkits.mplot3d import Axes3D
```

```python
def f(x, y):
    return sc.sin(sc.exp( - x ** 2 - y ** 2))

x = sc.linspace(-2, 2, 30)
y = sc.linspace(-2, 2, 30)

X, Y = sc.meshgrid(x, y)
Z = f(X, Y)
```

```python
fig = plt.figure(figsize=(16/2.54,10/2.54)) # itt manuálisan létre kell hozni a plot (figure) objektumot
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(X, Y, Z, cmap='viridis')
plt.show()
```


```python
%matplotlib inline
```

```python
import sympy
import math
import numpy as np
import matplotlib.pyplot as plt
```

# High-School Maths Exercise
## Getting to Know Jupyter Notebook. Python Libraries and Best Practices. Basic Workflow

### Problem 1. Markdown
Jupyter Notebook is a very light, beautiful and convenient way to organize your research and display your results. Let's play with it for a while.

First, you can double-click each cell and edit its content. If you want to run a cell (that is, execute the code inside it), use Cell > Run Cells in the top menu or press Ctrl + Enter.

Second, each cell has a type. There are two main types: Markdown (which is for any kind of free text, explanations, formulas, results... you get the idea), and code (which is, well... for code :D).

Let me give you a...
#### Quick Introduction to Markdown
##### Text and Paragraphs
There are several things that you can do. As you already saw, you can write paragraph text just by typing it. In order to create a new paragraph, just leave a blank line. See how this works below:
```
This is some text.
This text is on a new line, but it will continue the same paragraph (so you can make your paragraphs more easily readable by just continuing on a new line, or just go on and on like this one line is ever continuing).

This text is displayed in a new paragraph.

And this is yet another paragraph.
```
**Result:**

This is some text.
This text is on a new line, but it will continue the same paragraph (so you can make your paragraphs more easily readable by just continuing on a new line, or just go on and on like this one line is ever continuing).

This text is displayed in a new paragraph.

And this is yet another paragraph.

##### Headings
There are six levels of headings. Level one is the highest (largest and most important), and level 6 is the smallest. You can create headings of several types by prefixing the header line with one to six "#" symbols (this is called a pound sign if you are ancient, or a sharp sign if you're a musician... or a hashtag if you're too young :D). Have a look:
```
# Heading 1
## Heading 2
### Heading 3
#### Heading 4
##### Heading 5
###### Heading 6
```

**Result:**

# Heading 1
## Heading 2
### Heading 3
#### Heading 4
##### Heading 5
###### Heading 6

It is recommended that you have **only one** H1 heading - this should be the header of your notebook (or scientific paper). Below that, you can add your name or just jump to the explanations directly.

##### Emphasis
You can create emphasized (stronger) text by using a **bold** or _italic_ font. You can do this in several ways (using asterisks (\*) or underscores (\_)). In order to "escape" a symbol, prefix it with a backslash (\). You can also strike through your text in order to signify a correction.
```
**bold** __bold__
*italic* _italic_

This is \*\*not \*\* bold.

I ~~didn't make~~ a mistake.
```

**Result:**

**bold** __bold__
*italic* _italic_

This is \*\*not\*\* bold.

I ~~didn't make~~ a mistake.

##### Lists
You can add two types of lists: ordered and unordered. Lists can also be nested inside one another. To do this, press Tab once (it will be converted to 4 spaces).

To create an ordered list, just type the numbers. Don't worry if your numbers are wrong - Jupyter Notebook will create them properly for you. Well, it's better to have them properly numbered anyway...
```
1. This is
2. A list
10. With many
9. Items
    1. Some of which
    2. Can
        3. Be nested
42. You can also
    * Mix
    * list
    * types
```

**Result:**
1. This is
2. A list
10. With many
9. Items
    1. Some of which
    2. Can
        3. Be nested
42. You can also
    * Mix
    * list
    * types

To create an unordered list, type an asterisk, plus or minus at the beginning:
```
* This is
* An
    + Unordered
    - list
```

**Result:**
* This is
* An
    + Unordered
        - list

##### Links
There are many ways to create links but we mostly use one of them: we present links with some explanatory text. See how it works:
```
This is [a link](http://google.com) to Google.
```

**Result:**

This is [a link](http://google.com) to Google.

##### Images
They are very similar to links. Just prefix the image with an exclamation mark. The alt(ernative) text will be displayed if the image is not available. Have a look (hover over the image to see the title text):
```
 Do you know that "taco cat" is a palindrome? Thanks to The Oatmeal :)
```

**Result:**

 Do you know that "taco cat" is a palindrome? Thanks to The Oatmeal :)

If you want to resize images or do some more advanced stuff, just use HTML.

Did I mention these cells support HTML, CSS and JavaScript? Now I did.

##### Tables
These are a pain because they need to be formatted (somewhat) properly. Here's a good [table generator](http://www.tablesgenerator.com/markdown_tables). Just select File > Paste table data... and provide a tab-separated list of values. It will generate a good-looking ASCII-art table for you.
```
| Cell1 | Cell2 | Cell3 |
|-------|-------|-------|
| 1.1   | 1.2   | 1.3   |
| 2.1   | 2.2   | 2.3   |
| 3.1   | 3.2   | 3.3   |
```

**Result:**

| Cell1 | Cell2 | Cell3 |
|-------|-------|-------|
| 1.1   | 1.2   | 1.3   |
| 2.1   | 2.2   | 2.3   |
| 3.1   | 3.2   | 3.3   |

##### Code
Just use triple backtick symbols. If you provide a language, it will be syntax-highlighted. You can also use inline code with single backticks.

```python
def square(x):
    return x ** 2
```
This is `inline` code. No syntax highlighting here.

**Result:**
```python
def square(x):
    return x ** 2
```
This is `inline` code. No syntax highlighting here.

**Now it's your turn to have some Markdown fun.** In the next cell, try out some of the commands. You can just throw in some things, or do something more structured (like a small notebook).

##### Code
Just use triple backtick symbols. If you provide a language, it will be syntax-highlighted. You can also use inline code with single backticks.

```python
def square(x):
    return x ** 2
```
This is `inline` code. No syntax highlighting here.

**Result:**
```C#
private double square(x)
{
    return Math.Pow(x,2);
}
```
This is `inline C#` code. No syntax highlighting here.

### Problem 2. Formulas and LaTeX
Writing math formulas has always been hard. But scientists don't like difficulties and prefer standards. So, thanks to Donald Knuth (a very popular computer scientist, who also invented a lot of algorithms), we have a nice typesetting system, called LaTeX (pronounced _lah_-tek). We'll be using it mostly for math formulas, but it has a lot of other things to offer.

There are two main ways to write formulas. You could enclose them in single `$` signs like this: `$ ax + b $`, which will create an **inline formula**: $ ax + b $. You can also enclose them in double `$` signs `$$ ax + b $$` to produce $$ ax + b $$.

Most commands start with a backslash and accept parameters either in square brackets `[]` or in curly braces `{}`. For example, to make a fraction, you typically would write `$$ \frac{a}{b} $$`: $$ \frac{a}{b} $$.

[Here's a resource](http://www.stat.pitt.edu/stoffer/freetex/latex%20basics.pdf) where you can look up the basics of the math syntax. You can also search StackOverflow - there are all sorts of solutions there.

You're on your own now. Research and recreate all formulas shown in the next cell. Try to make your cell look exactly the same as mine. It's an image, so don't try to cheat by copy/pasting :D.

Note that you **do not** need to understand the formulas, what's written there or what it means. We'll have fun with these later in the course.

$$ x=ax_{1}^{2}+ 2ax_{2}-\sin120^{\circ} + ( \,\frac{2ab}{y} )\, $$
$$ $M = \left( \begin{matrix} a&b\\ c&d \\ x&y \end{matrix} \right)$ $$

### Problem 3. Solving with Python
Let's first do some symbolic computation. We need to import `sympy` first.

**Should your imports be in a single cell at the top or should they appear as they are used?** There's not a single valid best practice. Most people seem to prefer imports at the top of the file though. **Note: If you write new code in a cell, you have to re-execute it!**

Let's use `sympy` to give us a quick symbolic solution to our equation. First import `sympy` (you can use the second cell in this notebook):
```python
import sympy
```

Next, create symbols for all variables and parameters. You may prefer to do this in one pass or separately:
```python
x = sympy.symbols('x')
a, b, c = sympy.symbols('a b c')
```

Now solve:
```python
sympy.solve(a * x**2 + b * x + c)
```

Hmmmm... we didn't expect that :(. We got an expression for $a$ because the library tried to solve for the first symbol it saw. This is an equation and we have to solve for $x$. We can provide it as a second paramter:
```python
sympy.solve(a * x**2 + b * x + c, x)
```

Finally, if we use `sympy.init_printing()`, we'll get a LaTeX-formatted result instead of a typed one. This is very useful because it produces better-looking formulas.

```python
sympy.init_printing()
```

```python
sympy.symbols("x,y,z")
```

How about a function that takes $a, b, c$ (assume they are real numbers, you don't need to do additional checks on them) and returns the **real** roots of the quadratic equation?

Remember that in order to calculate the roots, we first need to see whether the expression under the square root sign is non-negative.

If $b^2 - 4ac > 0$, the equation has two real roots: $x_1, x_2$

If $b^2 - 4ac = 0$, the equation has one real root: $x_1 = x_2$

If $b^2 - 4ac < 0$, the equation has zero real roots

Write a function which returns the roots. In the first case, return a list of 2 numbers: `[2, 3]`. In the second case, return a list of only one number: `[2]`. In the third case, return an empty list: `[]`.

```python

def solve_quadratic_equation(a, b, c):
    """
    Returns the real solutions of the quadratic equation ax^2 + bx + c = 0
    """
    D= b**2 - 4*a*c

    if D > 0:
        x1=(-b -math.sqrt(D))/(2*a)
        x2=(-b +math.sqrt(D))/(2*a)
        return [x1,x2]
    elif D==0:
        x=-b /(2*a)
        return [x]
    elif D<0:
        return []

```

```python
# Testing: Execute this cell. The outputs should match the expected outputs. Feel free to write more tests
print(solve_quadratic_equation(1, -1, -2)) # [-1.0, 2.0]
print(solve_quadratic_equation(1, -8, 16)) # [4.0]
print(solve_quadratic_equation(1, 1, 1)) # []
```

    [-1.0, 2.0]
    [4.0]
    []

**Bonus:** Last time we saw how to solve a linear equation. Remember that linear equations are just like quadratic equations with $a = 0$. In this case, however, division by 0 will throw an error. Extend your function above to support solving linear equations (in the same way we did it last time).

### Problem 4. Equation of a Line
Let's go back to our linear equations and systems. There are many ways to define what "linear" means, but they all boil down to the same thing.

The equation $ax + b = 0$ is called *linear* because the function $f(x) = ax+b$ is a linear function. We know that there are several ways to know what one particular function means. One of them is to just write the expression for it, as we did above. Another way is to **plot** it. This is one of the most exciting parts of maths and science - when we have to fiddle around with beautiful plots (although not so beautiful in this case).

The function produces a straight line and we can see it.

How do we plot functions in general? Ww know that functions take many (possibly infinitely many) inputs. We can't draw all of them. We could, however, evaluate the function at some points and connect them with tiny straight lines. If the points are too many, we won't notice - the plot will look smooth.

Now, let's take a function, e.g. $y = 2x + 3$ and plot it. For this, we're going to use `numpy` arrays. This is a special type of array which has two characteristics:
* All elements in it must be of the same type
* All operations are **broadcast**: if `x = [1, 2, 3, 10]` and we write `2 * x`, we'll get `[2, 4, 6, 20]`. That is, all operations are performed at all indices. This is very powerful, easy to use and saves us A LOT of looping.

There's one more thing: it's blazingly fast because all computations are done in C, instead of Python.

First let's import `numpy`. Since the name is a bit long, a common convention is to give it an **alias**:
```python
import numpy as np
```

Import that at the top cell and don't forget to re-run it.

Next, let's create a range of values, e.g. $[-3, 5]$. There are two ways to do this. `np.arange(start, stop, step)` will give us evenly spaced numbers with a given step, while `np.linspace(start, stop, num)` will give us `num` samples. You see, one uses a fixed step, the other uses a number of points to return. When plotting functions, we usually use the latter. Let's generate, say, 1000 points (we know a straight line only needs two but we're generalizing the concept of plotting here :)).
```python
x = np.linspace(-3, 5, 1000)
```
Now, let's generate our function variable
```python
y = 2 * x + 3
```

We can print the values if we like but we're more interested in plotting them. To do this, first let's import a plotting library. `matplotlib` is the most commnly used one and we usually give it an alias as well.
```python
import matplotlib.pyplot as plt
```

Now, let's plot the values. To do this, we just call the `plot()` function. Notice that the top-most part of this notebook contains a "magic string": `%matplotlib inline`. This hints Jupyter to display all plots inside the notebook. However, it's a good practice to call `show()` after our plot is ready.
```python
plt.plot(x, y)
plt.show()
```

```python
x = np.linspace(-6, 6, 1000)
y = 2 * x + 3

ax = plt.gca()
ax.spines['top'].set_color('none')
ax.spines['bottom'].set_position('zero')
ax.spines['left'].set_position('zero')
ax.spines['right'].set_color('none')
plt.plot(x,y,'-r', label=r'$y = {2x}+3$')
plt.xlabel('x')
plt.ylabel('y')
plt.title('Exponential Curve')
plt.legend(loc='upper left')
plt.show()
```

It doesn't look too bad bit we can do much better. See how the axes don't look like they should? Let's move them to zeto. This can be done using the "spines" of the plot (i.e. the borders).

All `matplotlib` figures can have many plots (subfigures) inside them. That's why when performing an operation, we have to specify a target figure. There is a default one and we can get it by using `plt.gca()`. We usually call it `ax` for "axis".
Let's save it in a variable (in order to prevent multiple calculations and to make code prettier). Let's now move the bottom and left spines to the origin $(0, 0)$ and hide the top and right one.
```python
ax = plt.gca()
ax.spines["bottom"].set_position("zero")
ax.spines["left"].set_position("zero")
ax.spines["top"].set_visible(False)
ax.spines["right"].set_visible(False)
```

**Note:** All plot manipulations HAVE TO be done before calling `show()`. It's up to you whether they should be before or after the function you're plotting.

This should look better now. We can, of course, do much better (e.g. remove the double 0 at the origin and replace it with a single one), but this is left as an exercise for the reader :).

```python
# Copy and edit your code here
```

### * Problem 5. Linearizing Functions
Why is the line equation so useful? The main reason is because it's so easy to work with. Scientists actually try their best to linearize functions, that is, to make linear functions from non-linear ones. There are several ways of doing this. One of them involves derivatives and we'll talk about it later in the course.

A commonly used method for linearizing functions is through algebraic transformations. Try to linearize
$$ y = ae^{bx} $$

Hint: The inverse operation of $e^{x}$ is $\ln(x)$. Start by taking $\ln$ of both sides and see what you can do. Your goal is to transform the function into another, linear function. You can look up more hints on the Internet :).

```python
a = 5
b = 0.9
c = 1
x = np.linspace(0, 10, 256, endpoint = True)
y = (a * np.exp(-b*x))

plt.plot(x, y, '-r', label=r'$y = 5e^{2x}$')

axes = plt.gca()
axes.set_xlim([x.min(), x.max()])
axes.set_ylim([y.min(), y.max()])

plt.xlabel('x')
plt.ylabel('y')
plt.title('Exponential Curve')
plt.legend(loc='upper left')

plt.show()
```

### * Problem 6. Generalizing the Plotting Function
Let's now use the power of Python to generalize the code we created to plot. In Python, you can pass functions as parameters to other functions. We'll utilize this to pass the math function that we're going to plot.

Note: We can also pass *lambda expressions* (anonymous functions) like this:
```python
lambda x: x + 2```
This is a shorter way to write
```python
def some_anonymous_function(x):
    return x + 2
```

We'll also need a range of x values. We may also provide other optional parameters which will help set up our plot. These may include titles, legends, colors, fonts, etc. Let's stick to the basics now.

Write a Python function which takes another function, x range and number of points, and plots the function graph by evaluating it at every point.

**BIG hint:** If you want to use not only `numpy` functions for `f` but any one function, a very useful (and easy) thing to do, is to vectorize the function `f` (e.g. to allow it to be used with `numpy` broadcasting):
```python
f_vectorized = np.vectorize(f)
y = f_vectorized(x)
```

```python
def f(x):
    return 2.718281828459045**(1+x)

def tailor(x, eps):
    x = 1+x
    sum = 1+x
    term = x;
    n = 2;
    while term*term > eps*eps:
        term *= x/n
        n += 1
        sum += term
    return sum

a=3.0
b=4.0
krok=(b-a)/10

while a<=b:
    print(round(a,2), end=' ')
    print(round(f(a),5),end=' ')
    print(round(tailor(a,1e-6),5))
    ax = plt.gca()
    ax.spines['top'].set_color('none')
    ax.spines['bottom'].set_position('zero')
    ax.spines['left'].set_position('zero')
    ax.spines['right'].set_color('none')
    plt.plot(m,'-r', label=r'$y = {2x}+3$')
    plt.xlabel('x')
    plt.ylabel('y')
    plt.title('Exponential Curve')
    plt.legend(loc='upper left')
    plt.show()
    a+=krok
```

```python
def plot_math_function(f, min_x, max_x, num_points):
    # Write your code here
    pass
```

```python
plot_math_function(lambda x: 2 * x + 3, -3, 5, 1000)
plot_math_function(lambda x: -x + 8, -1, 10, 1000)
plot_math_function(lambda x: x**2 - x - 2, -3, 4, 1000)
plot_math_function(lambda x: np.sin(x), -np.pi, np.pi, 1000)
plot_math_function(lambda x: np.sin(x) / x, -4 * np.pi, 4 * np.pi, 1000)
```

### * Problem 7. Solving Equations Graphically
Now that we have a general plotting function, we can use it for more interesting things. Sometimes we don't need to know what the exact solution is, just to see where it lies. We can do this by plotting the two functions around the "=" sign ans seeing where they intersect. Take, for example, the equation $2x + 3 = 0$. The two functions are $f(x) = 2x + 3$ and $g(x) = 0$. Since they should be equal, the point of their intersection is the solution of the given equation. We don't need to bother marking the point of intersection right now, just showing the functions.

To do this, we'll need to improve our plotting function yet once. This time we'll need to take multiple functions and plot them all on the same graph. Note that we still need to provide the $[x_{min}; x_{max}]$ range and it's going to be the same for all functions.

```python
vectorized_fs = [np.vectorize(f) for f in functions]
ys = [vectorized_f(x) for vectorized_f in vectorized_fs]
```

```python
def plot_math_functions(functions, min_x, max_x, num_points):
    # Write your code here
    pass
```

```python
plot_math_functions([lambda x: 2 * x + 3, lambda x: 0], -3, 5, 1000)
plot_math_functions([lambda x: 3 * x**2 - 2 * x + 5, lambda x: 3 * x + 7], -2, 3, 1000)
```

This is also a way to plot the solutions of systems of equation, like the one we solved last time. Let's actually try it.

```python
plot_math_functions([lambda x: (-4 * x + 7) / 3, lambda x: (-3 * x + 8) / 5, lambda x: (-x - 1) / -2], -1, 4, 1000)
```

### Problem 8. Trigonometric Functions
We already saw the graph of the function $y = \sin(x)$. But, how do we define the trigonometric functions once again? Let's quickly review that.

The two basic trigonometric functions are defined as the ratio of two sides:
$$ \sin(x) = \frac{\text{opposite}}{\text{hypotenuse}} $$
$$ \cos(x) = \frac{\text{adjacent}}{\text{hypotenuse}} $$

And also:
$$ \tan(x) = \frac{\text{opposite}}{\text{adjacent}} = \frac{\sin(x)}{\cos(x)} $$
$$ \cot(x) = \frac{\text{adjacent}}{\text{opposite}} = \frac{\cos(x)}{\sin(x)} $$

This is fine, but using this, "right-triangle" definition, we're able to calculate the trigonometric functions of angles up to $90^\circ$. But we can do better. Let's now imagine a circle centered at the origin of the coordinate system, with radius $r = 1$. This is called a "unit circle".

We can now see exactly the same picture. The $x$-coordinate of the point in the circle corresponds to $\cos(\alpha)$ and the $y$-coordinate - to $\sin(\alpha)$. What did we get? We're now able to define the trigonometric functions for all degrees up to $360^\circ$. After that, the same values repeat: these functions are **periodic**:
$$ \sin(k.360^\circ + \alpha) = \sin(\alpha), k = 0, 1, 2, \dots $$
$$ \cos(k.360^\circ + \alpha) = \cos(\alpha), k = 0, 1, 2, \dots $$

We can, of course, use this picture to derive other identities, such as:
$$ \sin(90^\circ + \alpha) = \cos(\alpha) $$

A very important property of the sine and cosine is that they accept values in the range $(-\infty; \infty)$ and produce values in the range $[-1; 1]$. The two other functions take values in the range $(-\infty; \infty)$ **except when their denominators are zero** and produce values in the same range.

#### Radians
A degree is a geometric object, $1/360$th of a full circle. This is quite inconvenient when we work with angles. There is another, natural and intrinsic measure of angles. It's called the **radian** and can be written as $\text{rad}$ or without any designation, so $\sin(2)$ means "sine of two radians".

It's defined as *the central angle of an arc with length equal to the circle's radius* and $1\text{rad} \approx 57.296^\circ$.

We know that the circle circumference is $C = 2\pi r$, therefore we can fit exactly $2\pi$ arcs with length $r$ in $C$. The angle corresponding to this is $360^\circ$ or $2\pi\ \text{rad}$. Also, $\pi rad = 180^\circ$.

(Some people prefer using $\tau = 2\pi$ to avoid confusion with always multiplying by 2 or 0.5 but we'll use the standard notation here.)

**NOTE:** All trigonometric functions in `math` and `numpy` accept radians as arguments. In order to convert between radians and degrees, you can use the relations $\text{[deg]} = 180/\pi.\text{[rad]}, \text{[rad]} =  \pi/180.\text{[deg]}$. This can be done using `np.deg2rad()` and `np.rad2deg()` respectively.

#### Inverse trigonometric functions
All trigonometric functions have their inverses. If you plug in, say $\pi/4$ in the $\sin(x)$ function, you get $\sqrt{2}/2$. The inverse functions (also called, arc-functions) take arguments in the interval $[-1; 1]$ and return the angle that they correspond to. Take arcsine for example:
$$ \arcsin(y) = x: sin(y) = x $$
$$ \arcsin\left(\frac{\sqrt{2}}{2}\right) = \frac{\pi}{4} $$

Please note that this is NOT entirely correct. From the relations we found:
$$\sin(x) = sin(2k\pi + x), k = 0, 1, 2, \dots $$

it follows that $\arcsin(x)$ has infinitely many values, separated by $2k\pi$ radians each:
$$ \arcsin\left(\frac{\sqrt{2}}{2}\right) = \frac{\pi}{4} + 2k\pi, k = 0, 1, 2, \dots $$

In most cases, however, we're interested in the first value (when $k = 0$). It's called the **principal value**.

Note 1: There are inverse functions for all four basic trigonometric functions: $\arcsin$, $\arccos$, $\arctan$, $\text{arccot}$. These are sometimes written as $\sin^{-1}(x)$, $\cos^{-1}(x)$, etc. These definitions are completely equivalent.

Just notice the difference between $\sin^{-1}(x) := \arcsin(x)$ and $\sin(x^{-1}) = \sin(1/x)$.

#### Exercise
Use the plotting function you wrote above to plot the inverse trigonometric functions. Use `numpy` (look up how to use inverse trigonometric functions).

```python
# Write your code here
```

### ** Problem 9. Perlin Noise
This algorithm has many applications in computer graphics and can serve to demonstrate several things... and help us learn about math, algorithms and Python :).
#### Noise
Noise is just random values. We can generate noise by just calling a random generator. Note that these are actually called *pseudorandom generators*. We'll talk about this later in this course.
We can generate noise in however many dimensions we want. For example, if we want to generate a single dimension, we just pick N random values and call it a day. If we want to generate a 2D noise space, we can take an approach which is similar to what we already did with `np.meshgrid()`.

$$ \text{noise}(x, y) = N, N \in [n_{min}, n_{max}] $$

This function takes two coordinates and returns a single number N between $n_{min}$ and $n_{max}$. (This is what we call a "scalar field").

Random variables are always connected to **distributions**. We'll talk about these a great deal but now let's just say that these define what our noise will look like. In the most basic case, we can have "uniform noise" - that is, each point in our little noise space $[n_{min}, n_{max}]$ will have an equal chance (probability) of being selected.

#### Perlin noise
There are many more distributions but right now we'll want to have a look at a particular one. **Perlin noise** is a kind of noise which looks smooth. It looks cool, especially if it's colored. The output may be tweaked to look like clouds, fire, etc. 3D Perlin noise is most widely used to generate random terrain.

#### Algorithm
... Now you're on your own :). Research how the algorithm is implemented (note that this will require that you understand some other basic concepts like vectors and gradients).

#### Your task
1. Research about the problem. See what articles, papers, Python notebooks, demos, etc. other people have created
2. Create a new notebook and document your findings. Include any assumptions, models, formulas, etc. that you're using
3. Implement the algorithm. Try not to copy others' work, rather try to do it on your own using the model you've created
4. Test and improve the algorithm
5. (Optional) Create a cool demo :), e.g. using Perlin noise to simulate clouds. You can even do an animation (hint: you'll need gradients not only in space but also in time)
6. Communicate the results (e.g. in the Softuni forum)

Hint: [This](http://flafla2.github.io/2014/08/09/perlinnoise.html) is a very good resource. It can show you both how to organize your notebook (which is important) and how to implement the algorithm.

01.Math_Concept_For_Devs/02. High-School-Maths-Exercise/High-School Maths Exercise.ipynb


```
# this mounts your Google Drive to the Colab VM.
from google.colab import drive
drive.mount('/content/drive', force_remount=True)

# enter the foldername in your Drive where you have saved the unzipped
# assignment folder, e.g. 'cs231n/assignments/assignment3/'
FOLDERNAME = 'cs231n/assignments/assignment2/'
assert FOLDERNAME is not None, "[!] Enter the foldername."

# now that we've mounted your Drive, this ensures that
# the Python interpreter of the Colab VM can load
# python files from within it.
import sys
sys.path.append('/content/drive/My Drive/{}'.format(FOLDERNAME))

# this downloads the CIFAR-10 dataset to your Drive
# if it doesn't already exist.
%cd drive/My\ Drive/$FOLDERNAME/cs231n/datasets/
!bash get_datasets.sh
%cd /content
```

    Mounted at /content/drive
    /content/drive/My Drive/cs231n/assignments/assignment2/cs231n/datasets
    /content

# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train.
One idea along these lines is batch normalization which was proposed by [1] in 2015.

The idea is relatively straightforward. Machine learning methods tend to work better when their input data consists of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features; this will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, [1] proposes to insert batch normalization layers into the network. At training time, a batch normalization layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```
# As usual, a bit of setup
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """ returns relative error """
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print('  means: ', x.mean(axis=axis))
    print('  stds:  ', x.std(axis=axis))
    print()
```

    =========== You can safely ignore the message below if you are NOT working on ConvolutionalNetworks.ipynb ===========
    	You will need to compile a Cython extension for a portion of this assignment.
    	The instructions to do this will be given in a section of the notebook below.
    	There will be an option for Colab users and another for Jupyter (local) users.

```
# Load the (preprocessed) CIFAR10 data.
data = get_CIFAR10_data()
for k, v in data.items():
  print('%s: ' % k, v.shape)
```

    X_train:  (49000, 3, 32, 32)
    y_train:  (49000,)
    X_val:  (1000, 3, 32, 32)
    y_val:  (1000,)
    X_test:  (1000, 3, 32, 32)
    y_test:  (1000,)

## Batch normalization: forward
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))
# Means should be close to zero and stds close to one
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])
# Now means should be close to beta and stds close to gamma
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

    Before batch normalization:
      means:  [ -2.3814598  -13.18038246   1.91780462]
      stds:   [27.18502186 34.21455511 37.68611762]

    After batch normalization (gamma=1, beta=0)
      means:  [5.32907052e-17 7.04991621e-17 1.85962357e-17]
      stds:   [0.99999999 1.         1.        ]

    After batch normalization (gamma= [1. 2. 3.] , beta= [11. 12. 13.] )
      means:  [11. 12. 13.]
      stds:   [0.99999999 1.99999999 2.99999999]

```
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

    After batch normalization (test-time):
      means:  [-0.03927354 -0.04349152 -0.10452688]
      stds:   [1.01531428 1.01238373 0.97819988]

## Batch normalization: backward
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)
#You should expect to see relative errors between 1e-13 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.702926400705332e-09
    dgamma error:  7.420414216247087e-13
    dbeta error:  2.8795057655839487e-12

## Batch normalization: alternative backward
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\...\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
& \mu=\frac{1}{N}\sum_{k=1}^N x_k  &  v=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2 \\
& \sigma=\sqrt{v+\epsilon}         &  y_i=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hart part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

    dx difference:  9.431854837645182e-13
    dgamma difference:  0.0
    dbeta difference:  0.0
    speedup: 1.79x

## Fully Connected Nets with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

HINT: You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`. If you decide to do so, do it in the file `cs231n/classifiers/fc_net.py`.

```
np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.
for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            normalization='batchnorm')

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
  if reg == 0: print()
```

    Running check with reg =  0
    Initial loss:  2.2611955101340957
    W1 relative error: 1.10e-04
    W2 relative error: 2.85e-06
    W3 relative error: 4.05e-10
    b1 relative error: 4.44e-08
    b2 relative error: 2.22e-08
    b3 relative error: 1.01e-10
    beta1 relative error: 7.33e-09
    beta2 relative error: 1.89e-09
    gamma1 relative error: 6.96e-09
    gamma2 relative error: 1.96e-09

    Running check with reg =  3.14
    Initial loss:  6.996533220108303
    W1 relative error: 1.98e-06
    W2 relative error: 2.28e-06
    W3 relative error: 1.11e-08
    b1 relative error: 5.55e-09
    b2 relative error: 2.22e-08
    b3 relative error: 1.42e-10
    beta1 relative error: 6.65e-09
    beta2 relative error: 3.48e-09
    gamma1 relative error: 8.80e-09
    gamma2 relative error: 5.28e-09

# Batchnorm for deep networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

    Solver with batch norm:
    (Iteration 1 / 200) loss: 2.340974
    (Epoch 0 / 10) train acc: 0.107000; val_acc: 0.115000
    (Epoch 1 / 10) train acc: 0.315000; val_acc: 0.266000
    (Iteration 21 / 200) loss: 2.039365
    (Epoch 2 / 10) train acc: 0.384000; val_acc: 0.279000
    (Iteration 41 / 200) loss: 2.041102
    (Epoch 3 / 10) train acc: 0.494000; val_acc: 0.309000
    (Iteration 61 / 200) loss: 1.753902
    (Epoch 4 / 10) train acc: 0.531000; val_acc: 0.307000
    (Iteration 81 / 200) loss: 1.246584
    (Epoch 5 / 10) train acc: 0.573000; val_acc: 0.314000
    (Iteration 101 / 200) loss: 1.320590
    (Epoch 6 / 10) train acc: 0.632000; val_acc: 0.338000
    (Iteration 121 / 200) loss: 1.159472
    (Epoch 7 / 10) train acc: 0.686000; val_acc: 0.325000
    (Iteration 141 / 200) loss: 1.156477
    (Epoch 8 / 10) train acc: 0.767000; val_acc: 0.336000
    (Iteration 161 / 200) loss: 0.630541
    (Epoch 9 / 10) train acc: 0.802000; val_acc: 0.345000
    (Iteration 181 / 200) loss: 0.862820
    (Epoch 10 / 10) train acc: 0.788000; val_acc: 0.332000

    Solver without batch norm:
    (Iteration 1 / 200) loss: 2.302332
    (Epoch 0 / 10) train acc: 0.129000; val_acc: 0.131000
    (Epoch 1 / 10) train acc: 0.283000; val_acc: 0.250000
    (Iteration 21 / 200) loss: 2.041970
    (Epoch 2 / 10) train acc: 0.316000; val_acc: 0.277000
    (Iteration 41 / 200) loss: 1.900473
    (Epoch 3 / 10) train acc: 0.373000; val_acc: 0.282000
    (Iteration 61 / 200) loss: 1.713156
    (Epoch 4 / 10) train acc: 0.390000; val_acc: 0.310000
    (Iteration 81 / 200) loss: 1.662209
    (Epoch 5 / 10) train acc: 0.434000; val_acc: 0.300000
    (Iteration 101 / 200) loss: 1.696062
    (Epoch 6 / 10) train acc: 0.536000; val_acc: 0.346000
    (Iteration 121 / 200) loss: 1.550785
    (Epoch 7 / 10) train acc: 0.530000; val_acc: 0.310000
    (Iteration 141 / 200) loss: 1.436316
    (Epoch 8 / 10) train acc: 0.622000; val_acc: 0.342000
    (Iteration 161 / 200) loss: 1.001618
    (Epoch 9 / 10) train acc: 0.663000; val_acc: 0.332000
    (Iteration 181 / 200) loss: 0.922066
    (Epoch 10 / 10) train acc: 0.720000; val_acc: 0.338000

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch normalization and initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train 8-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
    print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
    bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

    bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    bn_solver.train()
    bn_solvers_ws[weight_scale] = bn_solver

    solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    solver.train()
    solvers_ws[weight_scale] = solver
```

    Running weight scale 1 / 20
    Running weight scale 2 / 20
    Running weight scale 3 / 20
    Running weight scale 4 / 20
    Running weight scale 5 / 20
    Running weight scale 6 / 20
    Running weight scale 7 / 20
    Running weight scale 8 / 20
    Running weight scale 9 / 20
    Running weight scale 10 / 20
    Running weight scale 11 / 20
    Running weight scale 12 / 20
    Running weight scale 13 / 20
    Running weight scale 14 / 20
    Running weight scale 15 / 20
    Running weight scale 16 / 20
    Running weight scale 17 / 20
    Running weight scale 18 / 20
    Running weight scale 19 / 20
    Running weight scale 20 / 20

```
# Plot results of weight scale experiment
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers_ws[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

  best_val_accs.append(max(solvers_ws[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the scale of weight initialization affect models with/without batch normalization differently, and why?

## Answer:
[FILL THIS IN]

# Batch normalization and batch size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)
    # Try training a very deep net with batchnorm
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

    No normalization: batch size =  5
    Normalization: batch size =  5
    Normalization: batch size =  10
    Normalization: batch size =  50

```
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:
[FILL THIS IN]

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:
[FILL THIS IN]

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_forward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(D3)
beta = np.zeros(D3)
# Means should be close to zero and stds close to one
print('After layer normalization (gamma=1, beta=0)')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0])
# Now means should be close to beta and stds close to gamma
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

    Before layer normalization:
      means:  [-59.06673243 -47.60782686 -43.31137368 -26.40991744]
      stds:   [10.07429373 28.39478981 35.28360729  4.01831507]

    After layer normalization (gamma=1, beta=0)
      means:  [ 4.81096644e-16 -7.40148683e-17  2.22044605e-16 -5.92118946e-16]
      stds:   [0.99999995 0.99999999 1.         0.99999969]

    After layer normalization (gamma= [3. 3. 3.] , beta= [5. 5. 5.] )
      means:  [5. 5. 5. 5.]
      stds:   [2.99999985 2.99999998 2.99999999 2.99999907]

```
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

#You should expect to see relative errors between 1e-12 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.4336158494902849e-09
    dgamma error:  4.519489546032799e-12
    dbeta error:  2.276445013433725e-12

# Layer Normalization and batch size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:
[FILL THIS IN]


+ This notebook is part of lecture 14 *Orthogonal vectors and subspaces* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
```

```python
# css_file = 'style.css'
# HTML(open(css_file, 'r').read())
```

```python
from sympy import init_printing, symbols, Matrix
from warnings import filterwarnings
```

```python
init_printing(use_latex='mathjax')
filterwarnings('ignore')
```

# Orthogonal vectors and subspaces
# Rowspace orthogonal to nullspace and columnspace to nullspace of AT
# N(ATA) = N(A)

## Orthogonal vectors

* Two vectors are orthogonal if their dot product is zero
* If they are written as column vectors **x** and **y**, their dot product is **x**T**y**
    * For orthogonal (perpendicular) vectors **x**T**y** = 0
* From the Pythagorean theorem they are orthogonal if
$$ { \left\| \overline { x }  \right\|  }^{ 2 }+{ \left\| \overline { y }  \right\|  }^{ 2 }={ \left\| \overline { x } +\overline { y }  \right\|  }^{ 2 }\\ { \left\| \overline { x }  \right\|  }=\sqrt { { x }_{ 1 }^{ 2 }+{ x }_{ 2 }^{ 2 }+\dots +{ x }_{ b }^{ 2 } }  $$

* The length squared of a (column) vector **x** can be calculated by **x**T**x**
* This achieves exactly the same as the sum of the squares of each element in the vector
$$ { x }_{ 1 }^{ 2 }+{ x }_{ 2 }^{ 2 }+\dots +{ x }_{ n }^{ 2 }$$

* Following from the Pythagorean theorem we have
$$ { \left\| \overline { x }  \right\|  }^{ 2 }+{ \left\| \overline { y }  \right\|  }^{ 2 }={ \left\| \overline { x } +\overline { y }  \right\|  }^{ 2 }\\ { \underline { x }  }^{ T }\underline { x } +{ \underline { y }  }^{ T }\underline { y } ={ \left( \underline { x } +\underline { y }  \right)  }^{ T }\left( \underline { x } +\underline { y }  \right) \\ { \underline { x }  }^{ T }\underline { x } +{ \underline { y }  }^{ T }\underline { y } ={ \underline { x }  }^{ T }\underline { x } +{ \underline { x }  }^{ T }\underline { y } +{ \underline { y }  }^{ T }\underline { x } +{ \underline { y }  }^{ T }\underline { y } \\ \because \quad { \underline { x }  }^{ T }\underline { y } ={ \underline { y }  }^{ T }\underline { x } \\ { \underline { x }  }^{ T }\underline { x } +{ \underline { y }  }^{ T }\underline { y } ={ \underline { x }  }^{ T }\underline { x } +2{ \underline { x }  }^{ T }\underline { y } +{ \underline { y }  }^{ T }\underline { y } \\ 2{ \underline { x }  }^{ T }\underline { y } =0\\ { \underline { x }  }^{ T }\underline { y } =0 $$
* This states that the dot product of orthogonal vectors equal zero

* The zero vector is orthogonal to all other similar dimensional vectors

## Orthogonality of subspaces

* Consider two subspaces *S* and *T*
* To be orthogonal every vector in *S* must be orthogonal to any vector in *T*

* Consider the *XY* and *YZ* planes in 3-space
* They are not orthogonal, since many combinations of vectors (one in each plane) are not orthogonal
* Vectors in the intersection, even though, one each from each plane can indeed be the same vector
* We can say that any planes that intersect cannot be orthogonal to each other

## Orthogonality of the rowspace and the nullspace

* The nullspace contains vectors **x** such that A**x** = **0**
* Now remembering that **x**T**y** = 0 for orthogonal column vectors and considering each row in A as a transposed column vector and **x** (indeed a column vector) and their product being zero meaning that they are orthogonal, we have:
$$ \begin{bmatrix} { { a }_{ 11 } } & { a }_{ 12 } & \dots  & { a }_{ 1n } \\ { a }_{ 21 } & { a }_{ 22 } & \dots  & { a }_{ 2n } \\ \vdots  & \vdots  & \vdots  & \vdots  \\ { a }_{ m1 } & { a }_{ m2 } & \dots  & { a }_{ mn } \end{bmatrix}\begin{bmatrix} { x }_{ 1 } \\ { x }_{ 2 } \\ \vdots  \\ { x }_{ n } \end{bmatrix}=\begin{bmatrix} 0 \\ 0 \\ \vdots  \\ 0 \end{bmatrix}\\ \begin{bmatrix} { a }_{ 11 } & { a }_{ 12 } & \dots  & { a }_{ 1n } \end{bmatrix}\begin{bmatrix} { x }_{ 1 } \\ { x }_{ 2 } \\ \vdots  \\ { x }_{ n } \end{bmatrix}=0\\ \dots  $$

* The rows (row vectors) in A are NOT the only vectors in the rowspace, since we also need to show that ALL linear combinations of them are also orthogonal to **x**
* This is easy to see by the structure above

## Orthogonality of the columnspace and the nullspace of AT

* The proof is the same as above

* The orthogonality of the rowspace and the nullspace is creating two orthogonal subspaces in ℝn
* The orthogonality of the columnspace and the nullspace of AT is creating two orthogonal subspaces in ℝm

* Note how the dimension add up to the degree of the space ℝ
    * The rowspace (a fundamental subspace in ℝn) is of dimension *r*
    * The dimension of the nullspace (a fundamental subspace in ℝn) is of dimension *n* - *r*
    * Addition of these dimensions gives us the dimension of the total space *n* as in ℝn
    * AND
    * The columnspace is of dimension *r* and the nullspace of AT is of dimension *m* - *r*, which adds to *m* as in ℝm

* This means that two lines that may be orthogonal in ℝ3 cannot be two orthogonal subspaces of ℝ3 since the addition of the dimensions of these two subspaces (lines) is not 3 (as in ℝ3)

* We call this complementarity, i.e. the nullspace and rowspace are orthogonal *complements* in ℝn

## ATA

* We know that
    * The result is square
    * The result is symmetric, i.e. (*n*×*m*)(*m*×*n*)=*n*×*n*
    * (ATA)T = ATATT = ATA

* When A**x** = **b** is not solvable we use ATA**x** = AT**b**
* **x** in the first instance did not have a solution, but after multiplying both side with AT, we hope that the second **x** has an solution, now called
$$ {A}^{T}{A}\hat{x} = {A}^{T}{b} $$

* Consider the matrix below with *m* = 4 equation in *n* = 2 unknowns
* The only **b** solutions must be linear combinations of the columnspace of A

```python
A = Matrix([[1, 1], [1, 2], [1, 5]])
A
```

$$ {x}_{1} \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} + {x}_{2} \begin{bmatrix} 1 \\ 2 \\ 5 \end{bmatrix} = \begin{bmatrix} {b}_{1} \\ {b}_{2} \\ {b}_{3} \end{bmatrix} $$

```python
A.transpose() * A
```

* Note how the nullspace of ATA is equal to the nullspace of A

```python
(A.transpose() * A).nullspace() == A.nullspace()
```

* The same goes for the rank

```python
A.rref(), (A.transpose() * A).rref()
```

* ATA is not always invertible
* In fact it is only invertible if the nullspace of A only contains the zero vector (has independent columns)

```python

```