##**Chapter 10 – Introduction to Neural Networks with Keras**

 Photo Credits: Galaxy's Edge by Rod Long licensed under the Unsplash License

> *The defnition of AI is a highly contested concept. It often refers to technologies that demonstrate levels of independent intelligence from humans. By its very
defnition, it is an intelligence that is differentiated from natural intelligence; it is
a constructed, artificial, or machine intelligence.*

$\quad$Ryan, M. (2020). In AI we trust: ethics, artificial intelligence, and reliability. Science and Engineering Ethics, 26(5), 2749-2767.

This notebook will be used in the lab session for week 5 of the course, covers Chapters 10 of Géron, and builds on the [notebooks made available on _Github_](https://github.com/ageron/handson-ml2).

Need a reminder of last week's labs? Click [_here_](https://colab.research.google.com/github/tbeucler/2022_ML_Earth_Env_Sci/blob/main/Lab_Notebooks/Week_4_Dimensionality_Reduction_Clustering.ipynb) to go to notebook for week 4 of the course.

## Notebook Setup

First, let's import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20 and TensorFlow ≥2.0.

```python
# Python ≥3.5 is required
import sys
assert sys.version_info >= (3, 5)

# Scikit-Learn ≥0.20 is required
import sklearn
assert sklearn.__version__ >= "0.20"

try:
    # %tensorflow_version only exists in Colab.
    %tensorflow_version 2.x
except Exception:
    pass

# TensorFlow ≥2.0 is required
import tensorflow as tf
assert tf.__version__ >= "2.0"

# Common imports
import numpy as np
import os

# to make this notebook's output stable across runs
rnd_seed = 42
rnd_gen = np.random.default_rng(rnd_seed)

# To plot pretty figures
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
mpl.rc('axes', labelsize=14)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)

# Where to save the figures
PROJECT_ROOT_DIR = "."
CHAPTER_ID = "ann"
IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, "images", CHAPTER_ID)
os.makedirs(IMAGES_PATH, exist_ok=True)

def save_fig(fig_id, tight_layout=True, fig_extension="png", resolution=300):
    path = os.path.join(IMAGES_PATH, fig_id + "." + fig_extension)
    print("Saving figure", fig_id)
    if tight_layout:
        plt.tight_layout()
    plt.savefig(path, format=fig_extension, dpi=resolution)

# Initialize the run_index
run_index = None

# Loading Tensorboard
%load_ext tensorboard
```

## Data Setup

Today, we'll once again be working on the MNIST handwritten digit database - we're becoming experts in typography! ✍

Let's begin by importing the dataset from the keras dataset library.

### **Q1) Load the MNIST dataset from Keras. Divide it into a training, validation, and test dataset**

*Hint 1: To access the Keras library, you can either reimport keras (e.g., `import tensorflow.keras as keras`), or you can access it from the instance of tensorflow we imported during setup (i.e., using `tf.keras`)*

*Hint 2: [Here is the documentation](https://keras.io/api/datasets/mnist/) for the Keras implementation of the MNIST dataset*

*Hint 3: If you use the `mnist.load_data()` method, what will be returned will be a set of tuples: (training_data, testing_data), where training_data and testing_data are tuples of inputs and labels (X, y)*

*Hint 4: You can break down the training dataset from the `.load()` method into a training and validation dataset. Since the full training dataset includes 60 000 samples, try using 50 000 samples as training data and 10 000 samples as validation data.*

```python
# Load the keras dataset data
( (X_train_full, y_train_full) , (X_testing, y_testing) ) = tf.keras.datasets.mnist.load_data()
```

    Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
    11493376/11490434 [==============================] - 0s 0us/step
    11501568/11490434 [==============================] - 0s 0us/step

```python
# Split the data
from sklearn.model_selection import train_test_split

(X_train,X_valid,y_train,y_valid) = train_test_split(X_train_full,y_train_full, test_size=10000, train_size=50000,random_state=42)
```

What does our data look like? Let's get an idea of the values and figure out what kind of preprocessing we should do before training our neural network.

### **Q2) Print the shape of the training, validation, and test sets. Then, print the maximum and minimum input values.**

*Hint 1: You loaded the data as numpy arrays. Thus, you can rely on the built-in methods for finding the shape and min/max values.*

*Hint 2: Click for the documentation on [`ndarray.max()`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.max.html), [`ndarray.min()`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.min.html), and [`ndarray.shape`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.shape.html)*

```python
#Write your code here
print('Shape of training inputs:',X_train.shape)
print('Shape of training outputs:',y_train.shape)
print('Shape of testinf inputs:',X_testing.shape)
print('Shape of testing outputs:',y_testing.shape)
print('Shape of validation inputs:',X_valid.shape)
print('Shape of validation outputs:',y_valid.shape)
print('Min of X values from ',X_train_full.min(),' to ',X_train_full.max())
print('Min of y values from ',y_train_full.min(),' to ',y_train_full.max())
```

    Shape of training inputs: (50000, 28, 28)
    Shape of training outputs: (50000,)
    Shape of testinf inputs: (10000, 28, 28)
    Shape of testing outputs: (10000,)
    Shape of validation inputs: (10000, 28, 28)
    Shape of validation outputs: (10000,)
    Min of X values from  0  to  255
    Min of y values from  0  to  9

If you used the same train/validation split as we did, you should have 50k samples in the training set, 10k in the validation set, and 10k in the test set.

Since the data represents grayscale image values, data values should vary between 0 and 255; Normalize the data by dividing it by 255.
### **Q3) Normalize the input data for the training, validation, and testing sets**

*Hint 1: The datasets are stored as simple numpy arrays, so you can perform arithmetic operations on them!*

```python
X_train = X_train / 255
X_testing = X_testing/255
X_valid = X_valid/255
```

We now have the normalized training, validation, and testing data that we'll use to train our neural network. Before moving on, it might be worth it to make a small visualiation of samples in our data to ensure that everything worked out correctly.

### **Q4) Write a function that:

1) Takes in an input dataset and its labels, a number of rows, and a number of columns
 2) Prints out a random n_rows by n_columns sample of images with their labels
**

*Hint 1: You can use the `rnd_seed.integers()` generator to generate a set of integers between 0 and the number of samples, with a size of (rows,columns). [Here is some documentation that can help](https://numpy.org/doc/stable/reference/random/generator.html#simple-random-data). It's best practice to take in the random generator as an argument for your function.*

*Hint 2: You can use matplotlib's `fig, axes = plt.subplots()` to make a grid of axes and call the `imshow()` method on each ax in order to plot the digit. It is recommended that you use the `cmap='binary'` argument in imshow to print the digits in black and white*. Click on the links for the documentation to [`plt.sublopts()`](https://matplotlib.org/3.5.0/api/_as_gen/matplotlib.pyplot.subplots.html), [`plt.imshow()`](https://matplotlib.org/3.5.0/api/_as_gen/matplotlib.pyplot.imshow.html), and [the colormaps (i.e., cmap values)](https://matplotlib.org/stable/gallery/color/colormap_reference.html) available in matplotlib.

*Hint 3: You can iterate using numpy `ndenumerate()` method, which will return the n-dimensional index of the array and the element located there. This will be useful when iterating through the indices you generated and plotting the corresponding digit and label*

*Hint 4: Feeling stuck? [Here is sample code for a way this function can be implemented.](https://unils-my.sharepoint.com/:t:/g/personal/tom_beucler_unil_ch/ERT6Sl_NHp5Nt2YjzkDPzLwB0RQ7_rVAl3RDx4BfHI047g?download=1)*

```python
def sample_plotter(X, y, n_rows, n_columns, rnd_gen):
    assert type(X) == type(np.empty(0))
    indices = rnd_gen.integers(0,X.shape[0], size=(n_rows, n_columns))

    fig, axes = plt.subplots(n_rows, n_columns, figsize=(8,6))

    for idx, element in np.ndenumerate(indices):
        axes[idx].imshow(X[element], cmap='binary')
        axes[idx].axis('off')
        axes[idx].title.set_text(y[element])
    return
```

Now that our function is defined, let's go ahead and print out a 4 row by 8 column sample from each dataset.

### **Q5) Grab a 4x8 sample of digits from each dataset and print out the image and labels**

```python
#Write your code here!

sample_plotter(X_train,y_train,4,8,rnd_gen)
```

We're now ready to start developing our neural network. The first thing that we want to do is figure out an appropriate learning rate for our model - after all, we want to choose one that converges to a solution *and* is the least computationally expensive possible.

Let's start by setting up a keras *callback* [(click here for the documentation)](https://keras.io/api/callbacks/), a type of object that will allow us to change the learning rate after every iteration (i.e., after every batch of data). We will set up what is called an exponential learning rate (that is, the learning will increase by a factor of $k$ after each iteration). Expressed mathematically,
\begin{align}
\eta_{\scriptsize{t}} = \eta_{\scriptsize{0}} \, \cdot \, k^{\scriptsize{t}}
\end{align}
where $t$ is the current iteration.

As a reminder, an epoch is an iteration through the entire training dataset, while a batch is an iteration through a predefined subset of . It's important to make this distinction, as ML algorithms are often trained in batches when dealing with large datasets, and we *normally* do not want to change the learning rate in between batches during model training. However, we will do so during this evaluation phase in order to determine an adequate learning rate.

We will therefore set a callback that will do two things after the end of each batch:

> 1) Keep a track of the losses
 2) Adjust the learning rate by multiplying it by a predefined factor

### **Q5) Set up an *Exponential_Learning_Rate* callback that, after each batch, logs the value of the loss function and learning rate, and then multiplies the learning rate by a factor of $k$**

*Hint 1: Multiple backend options are available with Keras. We will be using tensorflow, but the code is thought to be written in such a way that a different backend **could** be used. `tf.keras.backend` has a `.backend()` method that allows you to check what backend is being used.*

*Hint 2: You should extend the `tf.keras.callbacks.Callback` class. (Confused about extending classes? [Here is a question on stack overflow](https://stackoverflow.com/questions/15526858/how-to-extend-a-class-in-python) that could provide some context) *

*Hint 3: The ExponentialLearningRate callback we will implement will need to take in the $k$ factor durint its initialization ([here's a quick overview](https://stackoverflow.com/questions/625083/what-do-init-and-self-do-in-python) on the __init__ contructor method and **self** arguments in classes, with a focus on python.). You will also need to save an empty list as an attribute for both the losses and the learning rates*

*Hint 4: Keras model optimizers have an attribute where the learning rate is stored: `model.optimizer.learning_rate`. In order to read the value, you will have to use the keras backend's `.get_value()` method with the model's learning rate as an argument*

*Hint 5: the on_train_batch_end method pass the `logs` argument into the function. You can access the loss function by using `logs['loss']`*

*Hint 6: In order to set the learning rate to a different value, you will have to depend on the keras backend's `.set_value()` method. This method takes in two arguments: the first is the value that will be set (e.g., the learning rate in the model's optimizer) and the value that it will be set to (e.g., the learning rate multiplied by the k factor).*

*Hint 7: Unlike in other documentations we've seen, `backend.get_value()` and `backend.set_value()` don't yet have their own page. However, [here is the link](https://www.tensorflow.org/guide/keras/custom_callback#learning_rate_scheduling) to an example where both methods are used in a learning rate scheduler.*

```python
# We'll start by making it easier to access the keras backend. See hint #1 for
# more details
K = tf.keras.backend

# Use the .backend() method to determine what backend we're running
K.backend()
```

    'tensorflow'

```python
# Remember that you can access the keras.backend using K, which we defined in
# the code cell above!

class ExponentialLearningRate(tf.keras.callbacks.Callback): #define the ExponentialLearningRate class
    # Start
    def __init__(self, factor):
        self.factor = factor # set the factor
        self.losses = [] # initialize the losses list
        self.Lrate = [] # initialize the learning rates list

    def on_batch_end(self, batch, logs):
        # Add the value of the learning rate to the list
        self.Lrate.append(K.get_value(self.model.optimizer.learning_rate))

        # Add the value of the loss
        self.losses.append(logs['loss'])

        # Set the value of the learning rate
        K.set_value(self.model.optimizer.learning_rate, self.model.optimizer.learning_rate * self.factor)
```

Now that we've defined out callback, we can go ahead and start thinking about our neural network. For consistency's sake, let's start by clearing the Keras backend and setting our random state.

```python
# Run this cell
K.clear_session()
np.random.seed(rnd_seed)
tf.random.set_seed(rnd_seed)
```

Let's make a simple neural network model using Keras. For this, we will rely on a [*Sequential model*](https://keras.io/guides/sequential_model/), since we will want all of the inputs of one layer to be fed into the next layer. We recommend using the architecture described in the diagram below, but feel free to define your own architecture!

### **Q6) Write a sequential Keras model that will predict the digit class.**

*Hint 1: You can add the layers in the sequential model when initializing the model. It expects the layers in a list. Alternatively, you can add them one by one using the model's `.add()` method. [Check out the documentation here](https://keras.io/guides/sequential_model/#creating-a-sequential-model).*

*Hint 2: The input images should be flattened before feeding them into any densely connected layers. [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Flatten) for the flatten layer.*

*Hint 3: You want to use simple, densely connected layers for this exercise. [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense) for the dense layer.*

*Hint 4: Using a dense layer with the number of units set to the number of classes (e.g., the number of different digits in the MNIST dataset: 10) using a softmax activation unit can be interpreted as a probability of the input belonging to a given class. [Here is the documentation](https://keras.io/api/layers/activations/#softmax-function) for the softmax activation function in Keras*

```python
# Create your model! Feel free to use our outline, or make your own from scratch

model = tf.keras.models.Sequential([  # call the keras sequential model class
                            tf.keras.layers.Flatten(),  # 1st Layer
                            tf.keras.layers.Dense(300, activation = 'relu'),  # 2nd Layer
                            tf.keras.layers.Dense(100, activation = 'relu'),  # 3rd Layer
                            tf.keras.layers.Dense(10, activation= 'softmax')]) # 4th Layer
```

Now that we have a model defined, we need to run its `.compile()' method, in which we will give the model the following hyper-parameters:
> 1) Loss function will be set to sparse categorical cross entropy
 2) The optimizer will be set to Stochastic Gradient Descent with a learning rate of 1e-3
 3) The model metrics will include the accuracy score

### **Q7) Compile the model with the given hyperparameters (i.e., loss function, optimizer, and metrics) and instantiate the callback we defined previously using a $k$ factor of 1.005 (i.e., a 0.5% increase in learning rate per batch)**

*Hint 1: [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/sparse_categorical_crossentropy) for the sparse categorical cross entropy loss function in keras. You can simply reference the function using `loss='sparse_categorical_cross_entropy'` when compiling.*

*Hint 2: [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/SGD) for the Stochastic Gradient Descent optimizer in keras*

*Hint 3: [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/Accuracy) for the accuracy score implementation in keras. Like with the sparse_categorical_cross_entropy loss, you can reference the accuracy score in the metrics list, e.g. by setting `metrics=['accuracy']` when compiling.*

```python
model.compile(loss='sparse_categorical_crossentropy', # Set the loss function
              optimizer=tf.keras.optimizers.SGD(learning_rate=1e-3), # Set the optimizer and learning rate
              metrics=['accuracy']) # Set the metrics
```

```python
exponential_lr_callback = ExponentialLearningRate(factor=1.005)
```

Let's go ahead and train the compiled model for a single epoch.

### **Q8) Fit the model for a single epoch, using the exponential learning rate callback we defined in the previous code cell. Then, plot the Loss vs Learning rate.**

*Hint 1: Just like in scikit-learn, the keras model includes a `.fit()` method to train the algorithm! [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit).*

*Hint 2: After training, you can access the recorded losses and corresponding learning rates using the attributes we defined when we defined the class in Q5!*

```python
history = model.fit(X_train, # set the training inputs
                    y_train, # set the training labels
                    epochs=1, # set the number of epochs
                    validation_data=(X_valid, y_valid), # set validation input/labels
                    callbacks=[exponential_lr_callback]) # Set the callback
```

    1563/1563 [==============================] - 7s 4ms/step - loss: nan - accuracy: 0.6380 - val_loss: nan - val_accuracy: 0.0984

```python
# Plotting
fig, ax = plt.subplots()

ax.plot(exponential_lr_callback.Lrate, # learning rates
        exponential_lr_callback.losses) # losses

# Define a tuple with (min_learning_rate, max_learn_rate)
x_limits = ( min(exponential_lr_callback.Lrate), 10 )

# Set the xscale to logarithmic
ax.set_xscale('log')

# Draw a horizontal line at the minimum loss value
ax.hlines(min(exponential_lr_callback.losses), #Find the minimum loss value to draw a horizontal line
          *x_limits, # the star unpacks x_limits to the expected num of args
          'g')

# Set the limits for drawing the curves
ax.set_xlim(x_limits)
ax.set_ylim(0, exponential_lr_callback.losses[0]) # use the initial loss as the top y boundary

# Display gridlines to see better
ax.grid(which='both')

ax.set_xlabel("Learning rate")
ax.set_ylabel("Loss")
```

If you used the architecture we defined above with the learning rate we defined above, you should produce a graph that looks like this:

In this graph, you can see that the loss reaches a minimum at around 6e-1 and then begins to shoot up violently. Let's avoid that by using half that value (e.g., 3e-1).

If you have a different curve, try setting your learning rate to half of the learning rate with the minimum loss! 😃

Now that we have an idea of what the learning rate should be, let's go ahead and start from scratch once more.

```python
# Run this cell - let's go back to a clean slate!
K.clear_session()
np.random.seed(rnd_seed)
tf.random.set_seed(rnd_seed)
```

We also want to instantiate the model again - the weights in our current model are quite bad and if we use it as is it won't be able to learn since the weights are too far away from the solution. There are other ways to do this, but since our model is quite simple it's worth it to just redefine and recompile it.

### **Q9) Redefine and re-compile the model with the learning rate you found in Q8.**

```python
# redefine the model
model = tf.keras.models.Sequential([ # call the sequential model class
    tf.keras.layers.Flatten(), # flatten the data
    tf.keras.layers.Dense(300, activation = 'relu'), # densely connected ReLU layer, 300 units
    tf.keras.layers.Dense(100, activation = 'relu'), # densely connected ReLU layer, 100 units
    tf.keras.layers.Dense(10, activation = 'softmax')]) # densely connected Softmax layer, 10 units
```

```python
model.compile(loss='sparse_categorical_crossentropy', # Set the loss function
              optimizer=tf.keras.optimizers.SGD(learning_rate=3e-1), # Set the optimizer and learning rate
              metrics=['accuracy']) # Set the metrics
```

We're now going to set up a saving directory in case you want to try running the model with different learning rates or other hyper-parameters!

```python
#Change this number and rerun this cell whenever you want to change runs
run_index = 1

run_logdir = os.path.join(os.curdir, "my_mnist_logs", "run_{:03d}".format(run_index))

print(run_logdir)
```

    ./my_mnist_logs/run_001

We'll also set up some additional callbacks.
> 1) An early stopping callback ([documentation here](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping)). This callback will stop the training if no improvement is found after a `patience` number of epochs.
 2) A model checkpoint callback ([documentation here](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint)). This callback will ensure that only the best version of the model is kept (in case your model's performance reaches a maximum and then deteriorates after a certain number of epochs)
 3) A tensorboard callback ([documentation here](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/TensorBoard)). This callback will enable using Tensorboard to visualize learning curves, metrics, etc. Handy 🙌!

```python
early_stopping_cb = tf.keras.callbacks.EarlyStopping(patience=20)
checkpoint_cb = tf.keras.callbacks.ModelCheckpoint("my_mnist_model.h5", save_best_only=True)
tensorboard_cb = tf.keras.callbacks.TensorBoard(run_logdir)
```

Let's go ahead and fit the model again!

### **Q10) Fit the updated model for 100 epochs**

```python
history = model.fit(X_train, # inputs
                    y_train, # labels
                    epochs=100, #epochs
                    validation_data=(X_valid, y_valid),
                    callbacks=[checkpoint_cb, early_stopping_cb, tensorboard_cb])
```

    Epoch 1/100
    1563/1563 [==============================] - 6s 3ms/step - loss: 0.2415 - accuracy: 0.9513 - val_loss: 0.1419 - val_accuracy: 0.9578
    Epoch 2/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0872 - accuracy: 0.9719 - val_loss: 0.1025 - val_accuracy: 0.9711
    Epoch 3/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0625 - accuracy: 0.9799 - val_loss: 0.1586 - val_accuracy: 0.9527
    Epoch 4/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0452 - accuracy: 0.9854 - val_loss: 0.0963 - val_accuracy: 0.9722
    Epoch 5/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0365 - accuracy: 0.9880 - val_loss: 0.0913 - val_accuracy: 0.9770
    Epoch 6/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0261 - accuracy: 0.9916 - val_loss: 0.0913 - val_accuracy: 0.9793
    Epoch 7/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0209 - accuracy: 0.9929 - val_loss: 0.0980 - val_accuracy: 0.9789
    Epoch 8/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0194 - accuracy: 0.9940 - val_loss: 0.0929 - val_accuracy: 0.9795
    Epoch 9/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0126 - accuracy: 0.9960 - val_loss: 0.1096 - val_accuracy: 0.9783
    Epoch 10/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0157 - accuracy: 0.9944 - val_loss: 0.1120 - val_accuracy: 0.9772
    Epoch 11/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0172 - accuracy: 0.9943 - val_loss: 0.1989 - val_accuracy: 0.9528
    Epoch 12/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0136 - accuracy: 0.9955 - val_loss: 0.1749 - val_accuracy: 0.9655
    Epoch 13/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0126 - accuracy: 0.9964 - val_loss: 0.1339 - val_accuracy: 0.9757
    Epoch 14/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0111 - accuracy: 0.9967 - val_loss: 0.1036 - val_accuracy: 0.9801
    Epoch 15/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0080 - accuracy: 0.9973 - val_loss: 0.1105 - val_accuracy: 0.9806
    Epoch 16/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0045 - accuracy: 0.9985 - val_loss: 0.1032 - val_accuracy: 0.9823
    Epoch 17/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0043 - accuracy: 0.9989 - val_loss: 0.1141 - val_accuracy: 0.9819
    Epoch 18/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0039 - accuracy: 0.9988 - val_loss: 0.1135 - val_accuracy: 0.9803
    Epoch 19/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0055 - accuracy: 0.9983 - val_loss: 0.1536 - val_accuracy: 0.9745
    Epoch 20/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0154 - accuracy: 0.9955 - val_loss: 0.1281 - val_accuracy: 0.9798
    Epoch 21/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0073 - accuracy: 0.9977 - val_loss: 0.1290 - val_accuracy: 0.9796
    Epoch 22/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0083 - accuracy: 0.9975 - val_loss: 0.1104 - val_accuracy: 0.9823
    Epoch 23/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0071 - accuracy: 0.9978 - val_loss: 0.1405 - val_accuracy: 0.9781
    Epoch 24/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0066 - accuracy: 0.9981 - val_loss: 0.1156 - val_accuracy: 0.9829
    Epoch 25/100
    1563/1563 [==============================] - 5s 3ms/step - loss: 0.0075 - accuracy: 0.9976 - val_loss: 0.1398 - val_accuracy: 0.9787

Finally, we need to evaluate the performance of our model. Go ahead and try it out on the test set!

### **Q11) Evaluate the model on the test set.**

*Hint 1: Keras models include an `evaluate()` method that takes in the test set inputs/labels. [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/Model#evaluate).*

```python
# Rollback to best model, which was saved by the callback
model = tf.keras.models.load_model("my_mnist_model.h5") # rollback to best model

# Evaluate the model
model.evaluate(X_testing, y_testing)
```

    313/313 [==============================] - 1s 2ms/step - loss: 0.0862 - accuracy: 0.9768

    [0.08622584491968155, 0.9768000245094299]

Finally, we can use tensorboard to check out our model's performance! Note that the tensorboard extension was loaded in the notebook setup cell.

```python
%tensorboard --logdir=./my_mnist_logs --port=6008
```

    <IPython.core.display.Javascript object>

An enthusiastic (albeit somewhat sick 😷) TA noted that during the development of the notebook the accuracy reached on the test dataset was 97.84%. Additionally, the tensorboard curves from the test run is given below:


# メモ

matplotlib tutorials を利用した matplotlib の実験学習

https://matplotlib.org/stable/tutorials/index.html

# 日本語が使えるようにする

```
# 現在20210325 import japanize_matplotlib のために、下記が必要
%%capture
!pip install japanize_matplotlib
```

```
import matplotlib.pyplot as plt
import japanize_matplotlib
plt.text(0.5, 0.5, 'matplotlibで\n日本語が\n使える!!!!'
    , fontsize=40
    , horizontalalignment='center'
    , verticalalignment='center')
plt.show()
```

# 環境 スタイル
matplotlib.style.use で スタイルが選べる。

```
import matplotlib.pyplot as plt
import numpy as np
plt.style.use('ggplot') # ここでスタイル名を指定する
data = np.random.randn(50)
plt.plot(data) # x,y形式でなくてもplotできる
plt.show()
```

```
# print(plt.style.available) とすればstyleのリストが得られる
```

    ['Solarize_Light2', '_classic_test_patch', 'bmh', 'classic', 'dark_background', 'fast', 'fivethirtyeight', 'ggplot', 'grayscale', 'seaborn', 'seaborn-bright', 'seaborn-colorblind', 'seaborn-dark', 'seaborn-dark-palette', 'seaborn-darkgrid', 'seaborn-deep', 'seaborn-muted', 'seaborn-notebook', 'seaborn-paper', 'seaborn-pastel', 'seaborn-poster', 'seaborn-talk', 'seaborn-ticks', 'seaborn-white', 'seaborn-whitegrid', 'tableau-colorblind10']

```
# styleによってなにをしているかが違うので事前に使ったstyleの影響を受ける。使い物にならないのではないか
# それぞれのstyleの内容を知りたい。まあいいか
import matplotlib.pyplot as plt
import numpy as np
plt.style.use('dark_background')
plt.style.use('fivethirtyeight')
plt.figure(figsize=(4,4),facecolor='pink')
plt.style.use('Solarize_Light2')
plt.style.use('classic')
plt.style.use('seaborn')
plt.plot(np.sin(np.linspace(0, 2 * np.pi)), 'r-o')
plt.show()
```

plt.plot の中で 'r-o' とか指定できる。

plt.rcParamsで細かく指定できる。

rcParamsで指定したもののセットが style なのだろう。

plt.figure(figsize=(4,4),facecolor='pink')

とかで指定すると、rcParamsにかかわらず優先されるので、plt.plotを使う限りではrcParamsをあまり細かく学ぶ必要はないのかもしれない。 古い仕様なのかもしれない。

sympyのplotを使う場合、rcParamsの環境だけを使うようなので、知っておく必要はある。

```
import matplotlib.pyplot as plt
import numpy as np
data = np.random.randn(50)
plt.rcParams['lines.linewidth'] = 2
plt.rcParams['lines.linestyle'] = '--'
plt.plot(data)
plt.show()
# from sympy import *
# x =symbols('x')
# plot(x**2,(x,-0.5,0.5))
```

グラフの色は勝手に選ばれるのを変えるには axes.prop_cycle を変えるとあるが、次のプログラムはエラーになった。

colorについて調べる

cyclerについて調べる

```
# import matplotlib as mpl
# mpl.rcParams['axes.prop_cycle'] = cycler(color=['r', 'g', 'b', 'y']
# plt.plot(data)  # first color is red
# plt.show()
```

グラフのスタイルを設定するのに `style.use`, `rcParams` 以外に `rc` でキーワード引数をつかってまとめて設定する方法がある。

```
import matplotlib.pyplot as plt
import numpy as np
data = np.random.randn(50)
# plt.rcParams['lines.linewidth'] = 8
plt.rc('lines', linewidth=4, linestyle='-.')
plt.plot(data)
```

```
import matplotlib.pyplot as plt
import numpy as np
data = np.random.randn(50)
plt.rcdefaults() # defaultはcolabのdefaultではない
plt.rcParams['figure.figsize']=[4.2,2.8] # 反映する
# plt.figure(figsize=(4,4),facecolor='pink') # 反映する。優先される
plt.rcParams['figure.facecolor']='yellow' # 反映するが、弱い
plt.rcParams['lines.color']='black' # 反映しない
# plt.plot(data, 'k--') # 反映する
plt.plot(data)
plt.show()
```

```
# rcParamsの内容を知る方法
import matplotlib.pyplot as plt
# print(plt.rcParams) # lines.color:C0 , figure.figsize: [6.4, 4.8], C0 は cycler の 0番目の意味か?
```

`matplotlib.rcdefaults`  Matplotlib のデフォルトに戻る。

colab のデフォルトはランタイムを中断すればわかる => 違いを比較してみよう。 => 違いはほとんどなかった

rcParams の設定の validation については `matplotlib.rcsetup` にかかれているとか。

=> 例 >>> c = cycler(color=['red', 'green', 'blue']) とか。 わからない。

matplotlibの設定は `matplotlibrc` というファイルにあります、とか。

```
import matplotlib.pyplot as plt
# print(plt.rcParams)
```

# Usage Guide

matplotlibはfigureというエリアに設定されたaxという座標にデータを描画する。

axを持ったfigureを作る最も簡単な方法は

fig, ax = plt.subplots()

である。

```
import matplotlib.pyplot as plt
fig, ax = plt.subplots()  # 1つの座標axを持つfigureを作る
ax.plot([1, 2, 3, 4], [1, 4, 2, 3])  # axにplotする
```

MATLABを含め他のグラフ作成ライブラリーは座標を明示的に作る必要がない。

matplotlibでも、axを作る手間を省いて、次のようにすると現在(current)の座標にグラフを書くことができる。

```
import matplotlib.pyplot as plt
plt.plot([1, 2, 3, 4], [1, 4, 2, 3])  # Matplotlib plot.
```

```
import matplotlib
# help(matplotlib.figure.Figure)
# dir(matplotlib.figure.Figure)
dir(matplotlib.axes.Axes)
# help(matplotlib.axes.Axes)
```

plotで使うのは `numpy.array` なので、次のように変換する必要があるかもしれない。

pandas.Dataframe の場合:

    a = pandas.DataFrame(np.random.rand(4, 5), columns = list('abcde'))
    a_asarray = a.values

numpy.matrix の場合:

    b = np.matrix([[1, 2], [3, 4]])
    b_asarray = np.asarray(b)

```
import matplotlib.pyplot as plt
import numpy as np
x = np.linspace(0, 2, 100) # 0,2の間を100に刻む
fig, ax = plt.subplots()  # figure と ax を作る
ax.plot(x, x, label='linear')  # プロットする
ax.plot(x, x**2, label='quadratic')  # y=x^2のグラフ
ax.plot(x, x**3, label='cubic')  # y=x^3のグラフ
ax.set_xlabel('x label')  # x軸のラベル
ax.set_ylabel('y label')  # y軸のラベル
ax.set_title("Simple Plot")  # グラフにタイトルをつける
ax.legend()  # レジェンドを加える
```

or (pyplot-style)

```
# fgとaxを作らない場合の例
import matplotlib.pyplot as plt
import numpy as np
x = np.linspace(0, 2, 100)
plt.plot(x, x, label='linear')  # 明示されていない座標にプロットする
plt.plot(x, x**2, label='quadratic')
plt.plot(x, x**3, label='cubic')
plt.xlabel('x label')
plt.ylabel('y label')
plt.title("Simple Plot")
plt.legend()
plt.show()
```

fig,axを作ってオブジェクトに関数を使っていくいわゆるOO型のプログラミングのメリットは、違うデータセットで同じグラフを繰り返し書く際に現れる。

そのような場合の関数の書き方の例を次に示す。

```
def my_plotter(ax, data1, data2, param_dict):
    """
    A helper function to make a graph

    Parameters
    ----------
    ax : Axes
        The axes to draw to

    data1 : array
       The x data

    data2 : array
       The y data

    param_dict : dict
       Dictionary of kwargs to pass to ax.plot

    Returns
    -------
    out : list
        list of artists added
    """
    out = ax.plot(data1, data2, **param_dict)
    return out
```

この関数は次のように使う

```
import matplotlib.pyplot as plt
import numpy as np
data1, data2, data3, data4 = np.random.randn(4, 100)
fig, ax = plt.subplots(1, 1)
my_plotter(ax, data1, data2, {'marker': 'x'})
```

違うデータで同じ関数を使って2枚のグラフを描くことができる。

```
fig, (ax1, ax2) = plt.subplots(1, 2)
my_plotter(ax1, data1, data2, {'marker': 'x'})
my_plotter(ax2, data3, data4, {'marker': 'o'})
```

メモ randomについて

table | 意味
--- | ---
rand()    | 0.0以上、1.0未満の乱数を1個生成
rand(3)   | 0.0以上、1.0未満の乱数を3個生成
rand(2,3) | 0.0以上、1.0未満の乱数で 2x3 の行列を生成
(b-a) * np.random.rand() + a |（[a, b): a以上、b未満）の乱数
randn()         | 標準正規分布 (平均0, 標準偏差1)
randn(10)       | 標準正規分布を10個生成
randn(10,10)    | 標準正規分布による 10x10 の行列
normal(50,10)   | 平均50、標準偏差10の正規分布
binomial(n=100, p=0.5) | 二項分布
poisson(lam=10)     | λ=10 のポアソン分布
beta(a=3, b=5)      | a=3, b=5 のベータ分布
randint(100)            |  0〜99 の整数を1個生成
randint(30,70)          | 30〜69 の整数を1個生成
randint(0,100,20)       |  0〜99 の整数を20個生成
randint(0,100,(5,5))    |  0〜99 の整数で5x5の行列を生成
random_integers(100)    | 1〜100 の整数を1個生成
random_integers(30,70)  | 30〜70 の整数を1個生成
random.choice(city)                 | 1個をランダム抽出
random.choice(city,10)              | 10個をランダム抽出（重
random.choice(city,5,replace=False) | 5個をランダム抽出（重複なし)
random.choice(city, p=weight)       | 指定した確率で1個を抽出
seed(100)       | 数値はなんでもいい
numpy.random.random_sample((2,3)) | 0.0以上、1.0未満の乱数で 2x3 の行列を生成
 | np.random.random、np.random.ranf、
 | np.random.sampleはぜんぶ同じ
numpy.random.gamma(5,1,10) | 形状母数shape, 尺度母数scale, size
numpy.random.chisquare() | カイ二乗分布 自由度df, size

# バックエンド backend

バックエンドとはなにか。

グラフを出力する時、colab (jupyter) のように inline でプロットする場合だけでなく、さまざまな状況で matplotlib は使われる。 そのような様々な出力に対応する部分をバックエンドと呼ぶ。

例えば、次のようなユースケースがある。
- Pythonシェルからインタラクティブにmatplotlibを使用し、コマンドを入力するとプロットウィンドウがポップアップする
- wxpythonやpygtkなどのグラフィカルユーザーインターフェイスに埋め込んで、アプリケーションを構築する
- バッチスクリプトで数値シミュレーションからポストスクリプト画像を生成する
- Webアプリケーションサーバーを実行してグラフを動的に提供する

バックエンドには2つのタイプがある。
1. pygtk、wxpython、tkinter、qt4、macosx で使用するためのユーザーインターフェイスバックエンド。ユーザーインターフェイスバックエンドはインタラクティブバックエンド、対話型バックエンドとも呼ばれる
2. PNG、SVG、PDF、PS などの画像ファイルを作成するためのハードコピーバックエンド。非対話型バックエンドとも呼ばれる

バックエンドの設定は3通りの方法がある。
1. `matplotlibrc` ファイルで パラメーター `backend` で指定する
2. 環境変数 envvar:`MPLBACKEND` を使う
3. 関数 `matplotlib.use` を使う

# pyplot

pyplot は matplotlib で画像を描くための関数の集合である。

画像を描く座標のことを ax, axes というが、ここでいう ax, axes  は厳密な数学用語ではなく、matplotlib.pyplot の用語と思ってもらいたい。 pyplot が作用するのはこの ax, axes に対してである。

```
import matplotlib.pyplot as plt
plt.plot([1, 2, 3, 4])
plt.ylabel('some numbers')
plt.show()
```

上のグラフでなぜ x軸が 0-3 で、y軸が 1-4 なのか。

なにも指定しないで 1つのリストを plot に与えると、pyplot はそれを y の値とみなして、対応する x を自動生成する。 自動生成される x のリストが 0 ベースなので、この場合 [0,1,2,3] になる。

plot に 2 つのリストを与えると次のようになる。

```
import matplotlib.pyplot as plt
plt.plot([1, 2, 3, 4], [1, 4, 9, 16])
```

pyplot.plt はリストだけではなく、numpy.array を処理する。   list は numpy array に変換されて処理されている。

次の例では plot におけるさまざまなフォーマットを示す。

```
import numpy as np

# evenly sampled time at 200ms intervals
t = np.arange(0., 5., 0.2)

# red dashes, blue squares and green triangles
plt.plot(t, t, 'r--', t, t**2, 'bs', t, t**3, 'g^')
plt.show()
```

numpy.recarray と pandas.DataFrame については文字列で変数にアクセスすることができる。

そのようなオブジェクトは `data`キーワードを用いると、変数に対応するグラフを描くことができる。

```
import matplotlib.pyplot as plt
import numpy as np

data = {'a': np.arange(50),
        'c': np.random.randint(0, 50, 50),
        'd': np.random.randn(50)}
data['b'] = data['a'] + 10 * np.random.randn(50)
data['d'] = np.abs(data['d']) * 100

plt.scatter('a', 'b', c='c', s='d', data=data)
plt.xlabel('entry a')
plt.ylabel('entry b')
plt.show()
```

カテゴリー変数 categorical variables

カテゴリー変数で直接プロットすることができる。

```
names = ['group_a', 'group_b', 'group_c']
values = [1, 10, 100]

plt.figure(figsize=(9, 3))

plt.subplot(131)
plt.bar(names, values)
plt.subplot(132)
plt.scatter(names, values)
plt.subplot(133)
plt.plot(names, values)
plt.suptitle('Categorical Plotting')
plt.show()
```

```
ラインプロパティの制御

キーワードargsを使用する

```

```
x = np.arange(10) # [0,1,2,3,4,5,6,7,8,9] <= 10個
y = np.random.rand(10)*10 # [0..10) のランダムな実数
plt.plot(x, y, linewidth=10.0)
```

```
line, = plt.plot(x, y, '-')
line.set_antialiased(False) # turn off antialiasing
```

```
x1 = np.random.rand(10)
x2 = np.random.rand(10)
y1 = np.random.rand(10)
y2 = np.random.rand(10)
# lines = plt.plot(x1, y1, x2, y2)
line1 = plt.plot(x1, y1)
line2 = plt.plot(x2, y2)
# use keyword args
plt.setp(line1, color='k', linewidth=10.0)
# or MATLAB style string value pairs
plt.setp(line2, 'color', 'g', 'linewidth', 10.0)
plt.show()
```

```
# line2D のプロパティのリストは line を引数に plt.setp でも得られる

import matplotlib.pyplot as plt
import numpy as np
line = plt.plot(np.random.rand(7),np.random.rand(7))
plt.setp(line, linewidth=15, color='red')
# plt.setp(line)
```

```
複数のグラフを扱う方法
```

```
plt.gca は current axes (a matplotlib.axes.Axes instance) を返す

plt.gcf は current figure (a matplotlib.figure.Figure instance) を返す
```

```
def f(t):
    return np.exp(-t) * np.cos(2*np.pi*t)

t1 = np.arange(0.0, 5.0, 0.1)
t2 = np.arange(0.0, 5.0, 0.02)

plt.figure()
plt.subplot(211)
plt.plot(t1, f(t1), 'bo', t2, f(t2), 'k')

plt.subplot(212)
plt.plot(t2, np.cos(2*np.pi*t2), 'r--')
plt.show()
```

plt.figure は省略可能。

座標軸が1つだけのときは subplot（111）も省略可能。

plt.subplot の引数は 3桁の数字で 行数、列数、プロット番号の順。

subplot（211）はsubplot（2、1、1）と同じ。

格子状でなく自由な位置に座標を作成するには plt.axes([left, bottom, width, height]) を使う。 数字は 0から1の有理数 fractional である。

https://matplotlib.org/3.3.3/gallery/subplots_axes_and_figures/axes_demo.html

https://matplotlib.org/3.3.3/gallery/subplots_axes_and_figures/subplot_demo

を参照。

```
# 上記のサイトの例
import numpy as np
import matplotlib.pyplot as plt

# Data for plotting
x1 = np.linspace(0.0, 5.0)
x2 = np.linspace(0.0, 2.0)
y1 = np.cos(2 * np.pi * x1) * np.exp(-x1)
y2 = np.cos(2 * np.pi * x2)

# Create two subplots sharing y axis
fig, (ax1, ax2) = plt.subplots(2, sharey=True)

ax1.plot(x1, y1, 'ko-')
ax1.set(title='A tale of 2 subplots', ylabel='Damped oscillation')

ax2.plot(x2, y2, 'r.-')
ax2.set(xlabel='time (s)', ylabel='Undamped')

plt.show()
```

```
# 上記のサイトの例
import matplotlib.pyplot as plt
import numpy as np

np.random.seed(19680801)  # Fixing random state for reproducibility.

# create some data to use for the plot
dt = 0.001
t = np.arange(0.0, 10.0, dt)
r = np.exp(-t[:1000] / 0.05)  # impulse response
x = np.random.randn(len(t))
s = np.convolve(x, r)[:len(x)] * dt  # colored noise

fig, main_ax = plt.subplots()
main_ax.plot(t, s)
main_ax.set_xlim(0, 1)
main_ax.set_ylim(1.1 * np.min(s), 2 * np.max(s))
main_ax.set_xlabel('time (s)')
main_ax.set_ylabel('current (nA)')
main_ax.set_title('Gaussian colored noise')

# this is an inset axes over the main axes
right_inset_ax = fig.add_axes([.65, .6, .2, .2], facecolor='k')
right_inset_ax.hist(s, 400, density=True)
right_inset_ax.set(title='Probability', xticks=[], yticks=[])

# this is another inset axes over the main axes
left_inset_ax = fig.add_axes([.2, .6, .2, .2], facecolor='k')
left_inset_ax.plot(t[:len(r)], r)
left_inset_ax.set(title='Impulse response', xlim=(0, .2), xticks=[], yticks=[])

plt.show()
```

フィギュア番号を増やしながら複数の
plt.figure を使うことにより
複数のフィギュアを作成できる。

```
# 下の例は現在 20210329 警告 warning が出る。 将来は OO-style で書く方法のみになる
import matplotlib.pyplot as plt
plt.figure(1)                # the first figure
plt.subplot(211)             # the first subplot in the first figure
plt.plot([1, 2, 3])
plt.subplot(212)             # the second subplot in the first figure
plt.plot([4, 5, 6])

plt.figure(2)                # a second figure
plt.plot([4, 5, 6])          # creates a subplot(111) by default

plt.figure(1)                # figure 1 current; subplot(212) still current
plt.subplot(211)             # make subplot(211) in figure1 current
plt.title('Easy as 1, 2, 3') # subplot 211 title

plt.show()
```

plt.clf, plt.cla, plt.close について

figure や axes を plt.clf, plt.cla でクリアできる。

plt.close でメモリーを解放する。

とのこと。

テキスト処理

plt.textを使用して任意の場所にテキストを追加できる。

plt.xlabel、plt.ylabel、plt.titleを使用して、指定された場所にテキストを追加できる。

より詳細な例は

https://matplotlib.org/stable/tutorials/text/text_intro.html

```
import matplotlib
import matplotlib.pyplot as plt

fig = plt.figure()
ax = fig.add_subplot()
fig.subplots_adjust(top=0.85)

# Set titles for the figure and the subplot respectively
fig.suptitle('bold figure suptitle', fontsize=14, fontweight='bold')
ax.set_title('axes title')

ax.set_xlabel('xlabel')
ax.set_ylabel('ylabel')

# Set both x- and y-axis limits to [0, 10] instead of default [0, 1]
ax.axis([0, 10, 0, 10])

ax.text(3, 8, 'boxed italics text in data coords', style='italic',
        bbox={'facecolor': 'red', 'alpha': 0.5, 'pad': 10})

ax.text(2, 6, r'an equation: $E=mc^2$', fontsize=15)

ax.text(3, 2, 'unicode: Institut für Festkörperphysik')

ax.text(0.95, 0.01, 'colored text in axes coords',
        verticalalignment='bottom', horizontalalignment='right',
        transform=ax.transAxes,
        color='green', fontsize=15)

ax.plot([2], [1], 'o')
ax.annotate('annotate', xy=(2, 1), xytext=(3, 4),
            arrowprops=dict(facecolor='black', shrink=0.05))

plt.show()
```

```
mu, sigma = 100, 15
x = mu + sigma * np.random.randn(10000)

# the histogram of the data
n, bins, patches = plt.hist(x, 50, density=1, facecolor='g', alpha=0.75)

plt.xlabel('Smarts')
plt.ylabel('Probability')
plt.title('Histogram of IQ')
plt.text(60, .025, r'$\mu=100,\ \sigma=15$')
plt.axis([40, 160, 0, 0.03])
plt.grid(True)
plt.show()
```

すべてのplt.text関数は、matplotlib.text.Textインスタンスを返します。 キーワード引数をテキスト関数に渡すか、plt.setp を使用して、プロパティをカスタマイズできる。

```
import matplotlib.pyplot as plt
fig,ax=plt.subplots(figsize=(6,3))
t = plt.xlabel('my data', fontsize=14, color='red')
plt.setp(t, color='blue')
plt.show()
```

# テキストで数式を使用する

次の例でテキストの前の `r` は `raw` の意味で省略してはいけない。 なぜなら `r` のついたテキストの中では python のテキストのエスケープを扱わないことになっているから。

```
import matplotlib.pyplot as plt
fig,ax=plt.subplots(figsize=(6,3))
ax.set_title(r'$\sigma_i=15$', color='red')
plt.text(0.5, 0.5, 'No Japanese\nIn Matplotlib!'
    , fontsize=40
    , horizontalalignment='center'
    , verticalalignment='center')
plt.show()
```

```
plt.text?
```

# 注釈 Annotating text

上記の plt.text 関数で好きな位置に、テキストを配置できる。

plt.annotateメソッドは、注釈を簡単にするためのヘルパー機能を提供する。

引数 xyで表される注釈を付ける場所と、xytext で表されるテキストの場所を
両方とも（xy）タプルで指定する。

```
import matplotlib.pyplot as plt
import numpy as np
ax = plt.subplot(111)

t = np.arange(0.0, 5.0, 0.01)
s = np.cos(2*np.pi*t)
line, = plt.plot(t, s, lw=2)

plt.annotate('local max', xy=(2, 1), xytext=(3, 1.5),
             arrowprops=dict(facecolor='black', shrink=0.05),
             )

plt.ylim(-2, 2)
plt.show()
```

上の例では xy が矢印の先端の座標で、xytext がテキストの場所の座標である。

その他の例は

https://matplotlib.org/stable/gallery/text_labels_and_annotations/annotation_demo.html

を参照する。

# 対数グラフ Logarithmic and other nonlinear axes

対数スケールのグラフにするのは簡単で

plt.xscale('log')

とするだけ。

以下に例を示す。

```
import matplotlib.pyplot as plt
import numpy as np

# Fixing random state for reproducibility
np.random.seed(19680801)

# make up some data in the open interval (0, 1)
y = np.random.normal(loc=0.5, scale=0.4, size=1000)
y = y[(y > 0) & (y < 1)]
y.sort()
x = np.arange(len(y))

# plot with various axes scales
plt.figure()

# linear
plt.subplot(221)
plt.plot(x, y)
plt.yscale('linear')
plt.title('linear')
plt.grid(True)

# log
plt.subplot(222)
plt.plot(x, y)
plt.yscale('log')
plt.title('log')
plt.grid(True)

# symmetric log
plt.subplot(223)
plt.plot(x, y - y.mean())
plt.yscale('symlog', linthresh=0.01) # linthresh は deprecated という warning が出る
plt.title('symlog')
plt.grid(True)

# logit
plt.subplot(224)
plt.plot(x, y)
plt.yscale('logit')
plt.title('logit')
plt.grid(True)
# Adjust the subplot layout, because the logit one may take more space
# than usual, due to y-tick labels like "1 - 10^{-3}"
plt.subplots_adjust(top=0.92, bottom=0.08, left=0.10, right=0.95, hspace=0.25,
                    wspace=0.35)

plt.show()
```

自分で作ったスケールを加えることもできる。 詳細は adding-new-scales を参照。

# サンプルグラフについて

折れ線グラフ

複数のグラフを一度に扱う

画像

2次元グラフのカラー処理

ヒストグラム

自由曲線 path patch

3次元グラフ

流線グラフ、流線描画 streamplot

楕円 ellipse

棒グラフ bar chart

円グラフ pie chart

表 table

散布図 scatter plot

GUI部品 スライダー、ラジオボタンなど

塗りつぶされた曲線、ポリゴン

日付処理

対数グラフ

極座標

凡例 legend

数式処理 (内部プログラム、外部プログラム)

外部ツールキットへの出力  Qt, GTK, Tk, or wxWidgets, EEG viewer pbrain

スケッチ風グラフ XKCD-style sketch plot

複数の処理の組み合わせ subplot example

```
import matplotlib.pyplot as plt
import numpy as np

np.random.seed(19680801)
data = np.random.randn(2, 100)

fig, axs = plt.subplots(2, 2, figsize=(5, 5))
axs[0, 0].hist(data[0])
axs[1, 0].scatter(data[0], data[1])
axs[0, 1].plot(data[0], data[1])
axs[1, 1].hist2d(data[0], data[1])

plt.show()
```

```
import matplotlib.pyplot as plt
import numpy as np

data = np.random.randn(2, 3)
print(type(data))
print(data)
```

    <class 'numpy.ndarray'>
    [[-1.60179006  0.432704   -0.16409604]
     [ 0.81880165  0.95616235 -0.10459382]]

# 画像処理

Colab の環境ではチュートリアルの例が使えないので、この章は飛ばす。

表題のみ。

画像処理モジュールの import
```
import matplotlib.image as mpimg
```

画像データを numpy array に import する
matplotlib は画像データをロードするために pillow ライブラリーを使っている。
```
img = mpimg.imread('../../doc/_static/stinkbug.png')
print(img)
```

numpy array を画像グラフとして plot する
```
imgplot = plt.imshow(img)
```

画像グラフにカラースキーム pseudocolor schemes を適用する

```
# lum_img = img[:, :, 0]

# This is array slicing.  You can read more in the `Numpy tutorial
# <https://docs.scipy.org/doc/numpy/user/quickstart.html>`_.

# plt.imshow(lum_img)
```

```
# plt.imshow(lum_img, cmap="hot")
```

```
# imgplot = plt.imshow(lum_img)
# imgplot.set_cmap('nipy_spectral')
```

```
# カラーバーを表示する
# imgplot = plt.imshow(lum_img)
# plt.colorbar()
```

```
# 色データの範囲をヒストグラムにする
# plt.hist(lum_img.ravel(), bins=256, range=(0.0, 1.0), fc='k', ec='k')
```

```
# ピーク周辺にズームインする
# imgplot = plt.imshow(lum_img, clim=(0.0, 0.7))
```

```
# 返り値をつかって clim を指定する方法

# fig = plt.figure()
# ax = fig.add_subplot(1, 2, 1)
# imgplot = plt.imshow(lum_img)
# ax.set_title('Before')
# plt.colorbar(ticks=[0.1, 0.3, 0.5, 0.7], orientation='horizontal')
# ax = fig.add_subplot(1, 2, 2)
# imgplot = plt.imshow(lum_img)
# imgplot.set_clim(0.0, 0.7)
# ax.set_title('After')
# plt.colorbar(ticks=[0.1, 0.3, 0.5, 0.7], orientation='horizontal')
```

```
# 色補完スキーム Array Interpolation scheme
# 画像を縮小すると失われる情報がある。 拡大する際には補間する必要がある。
# 画像をロードしりサイズするのに Pillow ライブラリーを使う
# from PIL import Image

# img = Image.open('../../doc/_static/stinkbug.png')
# img.thumbnail((64, 64), Image.ANTIALIAS)  # resizes image in-place
# imgplot = plt.imshow(img)
```

上のコードではデフォルトの補間スキーム bilinear が使われている。

次のコードでは "nearest" を指定しているが、これは補間を行わない。

```
# imgplot = plt.imshow(img, interpolation="nearest")
```

次のコードでは "bicubic" を使っている。

"bicubic" は写真の拡大にしばしば使われる。 ピクセルが見えるよりもボケて見える方が好まれるからである。

```
# imgplot = plt.imshow(img, interpolation="bicubic")
```

# グラフ描画のライフサイクル

次にグラフを描いて、加工して、保存する手順の中で、参考になるベストプラスティスを示したい。

この章は
 <http://pbpython.com/effective-matplotlib.html>`_
    by Chris Moffitt
をベースに作成した。

## Our data

We'll use the data from the post from which this tutorial was derived.
It contains sales information for a number of companies.

```
# 以下のサンプルで使われるデータは元のサイトで使われていたもの
# 会社名をキーにした売上金額の辞書の形式になっている

import numpy as np
import matplotlib.pyplot as plt

data = {'Barton LLC': 109438.50,
        'Frami, Hills and Schmidt': 103569.59,
        'Fritsch, Russel and Anderson': 112214.71,
        'Jerde-Hilpert': 112591.43,
        'Keeling LLC': 100934.30,
        'Koepp Ltd': 103660.54,
        'Kulas Inc': 137351.96,
        'Trantow-Barrows': 123381.38,
        'White-Trantow': 135841.99,
        'Will LLC': 104437.60}
group_data = list(data.values())
group_names = list(data.keys())
group_mean = np.mean(group_data)
```

```
print(type(data))
```

    <class 'dict'>

# いまここ

## Getting started

This data is naturally visualized as a barplot, with one bar per
group. To do this with the object-oriented approach, we first generate
an instance of :class:`figure.Figure` and
:class:`axes.Axes`. The Figure is like a canvas, and the Axes
is a part of that canvas on which we will make a particular visualization.

Note

Figures can have multiple axes on them. For information on how to do this,
   see the :doc:`Tight Layout tutorial
   </tutorials/intermediate/tight_layout_guide>`.

```
fig, ax = plt.subplots()
```

Now that we have an Axes instance, we can plot on top of it.

```
fig, ax = plt.subplots()
ax.barh(group_names, group_data)
```

## Controlling the style

There are many styles available in Matplotlib in order to let you tailor
your visualization to your needs. To see a list of styles, we can use
:mod:`.style`.

```
print(plt.style.available)
```

You can activate a style with the following:

```
plt.style.use('fivethirtyeight')
```

Now let's remake the above plot to see how it looks:

```
fig, ax = plt.subplots()
ax.barh(group_names, group_data)
```

The style controls many things, such as color, linewidths, backgrounds,
etc.

## Customizing the plot

Now we've got a plot with the general look that we want, so let's fine-tune
it so that it's ready for print. First let's rotate the labels on the x-axis
so that they show up more clearly. We can gain access to these labels
with the :meth:`axes.Axes.get_xticklabels` method:

```
fig, ax = plt.subplots()
ax.barh(group_names, group_data)
labels = ax.get_xticklabels()
```

If we'd like to set the property of many items at once, it's useful to use
the :func:`pyplot.setp` function. This will take a list (or many lists) of
Matplotlib objects, and attempt to set some style element of each one.

```
fig, ax = plt.subplots()
ax.barh(group_names, group_data)
labels = ax.get_xticklabels()
plt.setp(labels, rotation=45, horizontalalignment='right')
```

It looks like this cut off some of the labels on the bottom. We can
tell Matplotlib to automatically make room for elements in the figures
that we create. To do this we set the ``autolayout`` value of our
rcParams. For more information on controlling the style, layout, and
other features of plots with rcParams, see
:doc:`/tutorials/introductory/customizing`.

```
plt.rcParams.update({'figure.autolayout': True})

fig, ax = plt.subplots()
ax.barh(group_names, group_data)
labels = ax.get_xticklabels()
plt.setp(labels, rotation=45, horizontalalignment='right')
```

Next, we add labels to the plot. To do this with the OO interface,
we can use the :meth:`.Artist.set` method to set properties of this
Axes object.

```
fig, ax = plt.subplots()
ax.barh(group_names, group_data)
labels = ax.get_xticklabels()
plt.setp(labels, rotation=45, horizontalalignment='right')
ax.set(xlim=[-10000, 140000], xlabel='Total Revenue', ylabel='Company',
       title='Company Revenue')
```

We can also adjust the size of this plot using the :func:`pyplot.subplots`
function. We can do this with the ``figsize`` kwarg.

Note

While indexing in NumPy follows the form (row, column), the figsize
   kwarg follows the form (width, height). This follows conventions in
   visualization, which unfortunately are different from those of linear
   algebra.

```
fig, ax = plt.subplots(figsize=(8, 4))
ax.barh(group_names, group_data)
labels = ax.get_xticklabels()
plt.setp(labels, rotation=45, horizontalalignment='right')
ax.set(xlim=[-10000, 140000], xlabel='Total Revenue', ylabel='Company',
       title='Company Revenue')
```

For labels, we can specify custom formatting guidelines in the form of
functions. Below we define a function that takes an integer as input, and
returns a string as an output. When used with `.Axis.set_major_formatter` or
`.Axis.set_minor_formatter`, they will automatically create and use a
:class:`ticker.FuncFormatter` class.

For this function, the ``x`` argument is the original tick label and ``pos``
is the tick position.  We will only use ``x`` here but both arguments are
needed.

```
def currency(x, pos):
    """The two args are the value and tick position"""
    if x >= 1e6:
        s = '${:1.1f}M'.format(x*1e-6)
    else:
        s = '${:1.0f}K'.format(x*1e-3)
    return s
```

We can then apply this function to the labels on our plot. To do this,
we use the ``xaxis`` attribute of our axes. This lets you perform
actions on a specific axis on our plot.

```
fig, ax = plt.subplots(figsize=(6, 8))
ax.barh(group_names, group_data)
labels = ax.get_xticklabels()
plt.setp(labels, rotation=45, horizontalalignment='right')

ax.set(xlim=[-10000, 140000], xlabel='Total Revenue', ylabel='Company',
       title='Company Revenue')
ax.xaxis.set_major_formatter(currency)
```

## Combining multiple visualizations

It is possible to draw multiple plot elements on the same instance of
:class:`axes.Axes`. To do this we simply need to call another one of
the plot methods on that axes object.

```
fig, ax = plt.subplots(figsize=(8, 8))
ax.barh(group_names, group_data)
labels = ax.get_xticklabels()
plt.setp(labels, rotation=45, horizontalalignment='right')

# Add a vertical line, here we set the style in the function call
ax.axvline(group_mean, ls='--', color='r')

# Annotate new companies
for group in [3, 5, 8]:
    ax.text(145000, group, "New Company", fontsize=10,
            verticalalignment="center")

# Now we move our title up since it's getting a little cramped
ax.title.set(y=1.05)

ax.set(xlim=[-10000, 140000], xlabel='Total Revenue', ylabel='Company',
       title='Company Revenue')
ax.xaxis.set_major_formatter(currency)
ax.set_xticks([0, 25e3, 50e3, 75e3, 100e3, 125e3])
fig.subplots_adjust(right=.1)

plt.show()
```

## Saving our plot

Now that we're happy with the outcome of our plot, we want to save it to
disk. There are many file formats we can save to in Matplotlib. To see
a list of available options, use:

```
print(fig.canvas.get_supported_filetypes())
```

We can then use the :meth:`figure.Figure.savefig` in order to save the figure
to disk. Note that there are several useful flags we show below:

* ``transparent=True`` makes the background of the saved figure transparent
  if the format supports it.
* ``dpi=80`` controls the resolution (dots per square inch) of the output.
* ``bbox_inches="tight"`` fits the bounds of the figure to our plot.

```
# Uncomment this line to save the figure.
# fig.savefig('sales.png', transparent=False, dpi=80, bbox_inches="tight")
```

# いまここ


```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib
%matplotlib inline
```

```python
matplotlib.rcParams['figure.figsize'] = (10.0, 8.0)
```

# Calculus Review: Introduction

[Integrals](http://en.wikipedia.org/wiki/Integral) and [derivatives](http://en.wikipedia.org/wiki/Derivative) are widely used in
scientific modeling and simulations.  Integration, along with its inverse operation
differentiation, are the two main operations of calculus.

In this lecture notebook we will review some of the basic concepts from Calculus, including going all the way back
to discussing functions and their graphs, and the concept of the slope of a line; looking at limits and
derivatives, and how the slop of a function relates to the derivative; and then at the concept of integration
and integrals, and specifying and solving differential equations.

This notebook is only meant to be a quick review.  It is hoped and expected that at some point you have studied at least
the basics of these topics in secondary education and during your undergraduate degree.  Most students should be
able to follow these materials and understand the important concepts who have at least a good understanding of
algebra and geometry, especially equations of lines and functions, as a starting point to understanding
the calculus mathematics.

The following suggested materials have more information and activities that you may use to supplement this
review.  Some of the examples and materials used in this review come from examples in these links:

- The [Kahn Academy courses on  Differential and Integral calculus](https://www.khanacademy.org/math/calculus-1?t=classes)
  as well as [Integral Calculus](https://www.khanacademy.org/math/integral-calculus) would be good
  reviews for weeks 7 and 8 materials.
- There are multiple good online textbooks for studying or reviewing differential and integral Calculus.
  There are several listed [here](https://www.math.ucdavis.edu/resources/learning/online-textbooks/)
  (I may be assigning some readings from here when we review Probability and Linear Algebra as well).
  I also like the [MIT Online Calculus book](https://ocw.mit.edu/resources/res-18-001-calculus-online-textbook-spring-2005/textbook/).
- [Basic Calculus Refresher (UW-Madison)](http://pages.stat.wisc.edu/~ifischer/calculus.pdf) - a lot of the content
   of this review came from this 20 page calculus refresher.
- Kahn Academy [Introduction to Integral Calculus](https://www.khanacademy.org/math/ap-calculus-ab/ab-integration-new/ab-6-1/v/introduction-to-integral-calculus)
- [Understanding Calculus in 10 min](https://www.youtube.com/watch?v=WjJ-kpgps1c)

# Functions and Graphs

We have been using functions in both the mathematical sense and the programming sense extensively in this
class.  In the mathematical sense, the simplest function is a function of a single independent variable
$x$.  We often also use a dependent variable $y$ and say that "y is a **function** of x", written this
way

\begin{equation}
y = f(x)
\end{equation}

The simplest function we can imagine is a constant function, for example,

\begin{equation}
y = 3
\end{equation}

Here you may ask how is this a function of the independent variable $x$?  In this case it is not, $y$ does
not depend on $x$, although we can also think of it like this:

\begin{equation}
y = 0 \cdot x + 3
\end{equation}

We could also graph this constant function.  It is a pretty uninteresting plot, since the value of $y$ is
always 3 no matter what the value of $x$ is:

```python
# Example of constant function, it doesn't matter what the
# independent variable x value is, y is always a constant 3
# y = 3
def f(x):
    return 0*x + 3

x = np.linspace(-3, 3)
plt.plot(x, f(x));
plt.ylim([-1, 4])
plt.grid();
```

## Slope of a Line and the Slope-Intercept Equation

The slope of a line is simply the ratio of the change in $y$, written as $\Delta y$ to the change in $x$, $\Delta x$

\begin{equation}
m = \frac{\Delta y}{\Delta x}
\end{equation}

(m is often used to represent the slop of a line).  For our previous equation $y = 0x + 3$ you
should see that the slope of the line is 0, because $y$ never changes
thus the ratio is 0.  We say that the line has a slope of 0 in this case.

Lets try a different function, say

\begin{equation}
y = 2x
\end{equation}

Here is the graph of this line.

```python
def f(x):
    return 2*x

x = np.linspace(-3, 3)
plt.plot(x, f(x));
plt.xlim([-6, 6])
plt.grid();
```

Here the slope $m = 2$ because for every change of $x$ $\Delta x = 1$, $y$ will change by 2 $\Delta y = 2$
and thus the ratio is $m = \Delta y / \Delta x = 2 / 1 = 2$

If we add another term to the equation

\begin{equation}
y = 2x + 3
\end{equation}

The graph of the equation now looks like this:

```python
def f(x):
    return 2*x + 3

x = np.linspace(-3, 3)
plt.plot(x, f(x));
plt.xlim([-6, 6])
plt.grid();
```

Here the slope of the line is still the same $m = 2$.  But when $x = 0$ $y$ has a value of $3$.
Whenever you rearrange a linear function into the form $y = mx + b$, this is known as the slope-intercept
form of the equation of the line.  $m$ will be the slope, which gives the ratio of how $y$ changes to the amount
that $x$ changes.  The constant $b$ will be the intercept of the line.  When $x = 0$ this will be the value of $y$
or in other words this will be the intercept location, the value of $y$ when x changes from negative to positive (at
the point $x = 0$ ).

At this point you should understand the following about the slope and the equation of a line

- A line always has a constant slope.  At all places on the line, the slope is always the same $m$.  The
  slope never changes.
- The slope-intercept equation of a line allows you to directly see the slope of the line and the intercept
  location where the line crosses the axis at $x = 0$.
- A positive slope, like $+2$ above will describe a line that goes up, as $x$ gets bigger $y$ gets bigger.  You can of
  course have a negative slope, in which case the line goes down, as $x$ gets bigger $y$ decreases.
- A slope of 0 is a constant function, and defines a flat line as seen on its graph.

## Other Functions of a Single Variable

A line is a fairly simple function of a single varible.  We can of course have much more complex functions, even
if we still restrict ourself to functions of a single variable $f(x)$. Here are a few examples

\begin{equation}
y = x^2 + 2x + 1 \\
y = \frac{2}{3}x^3 - 5x^2 + 5 \\
y = x^{-1} \\
y = 2^x
\end{equation}

The first 2 equations are examples of polynomial functions.  Anytime $y$ is a functions of a  power of $x$
like $x^1, x^2, x^3  \cdots$ is an example of a polynomial function.  The first function with a power of $x^2$
is known as a quadratic function and the second with $x^3$ is a cubic function.  The last equation is an
example of an exponential function (the $x$ appears in the exponent of one of the terms of the function).

We can of course graph these equations in a similar way as we did for our linear equations.  They define
a unique mapping from any value of $x$ to a unique single value of $y$, so by mapping many values of $x$ to $y$ over some range
we can see the shape of the equation in that range.  Here are the quadratic and cubic equations from above plotted separately.

```python
def quadratic_function(x):
    return x**2

def cubic_function(x):
    return 2/3 * x**3 - 5 * x**2 + 5

x = np.linspace(-5, 5)

plt.subplots(1,2)
plt.subplot(121)
plt.plot(x, quadratic_function(x))
plt.grid();

plt.subplot(122)
plt.plot(x, cubic_function(x))
plt.grid();

```

# Limits and Derivatives

The higher order polynomial functions, and in fact most any more complex function of the variable $x$
no longer have a constant slope.  The rate of change of the line at any point $x$ of the function will
depend on the function properties at that point.

Lets say we want to know what the slope is of the basic quadratic function $y = x^2$ is at the point where $x = 2$.
We could use the equation for the slope of a line, and calculate the ratio of the change of $y$ to $x$
like this:

```python
def f(x):
    return x**2

plt.figure(figsize=(10,10))
x = np.linspace(0, 5)
plt.plot(x, f(x));

# slope when x changes from 2 to 5 is
deltay = f(5.0) - f(2.0)
deltax = 5.0 - 2.0
m = deltay / deltax
print("slope when x changes from 2 to 5: ", m)
plt.plot([2.0, 5.0], [f(2.0), f(5.0)], 'k--')

# slope when x changes from 2 to 4 is
deltay = f(4.0) - f(2.0)
deltax = 4.0 - 2.0
m = deltay / deltax
print("slope when x changes from 2 to 4: ", m)
plt.plot([2.0, 4.0], [f(2.0), f(4.0)], 'k--')

# slope when x changes from 2 to 3 is
deltay = f(3.0) - f(2.0)
deltax = 3.0 - 2.0
m = deltay / deltax
print("slope when x changes from 2 to 3: ", m)
plt.plot([2.0, 3.0], [f(2.0), f(3.0)], 'k--')

# slope when x changes from 2 to 3 is
deltay = f(2.1) - f(2.0)
deltax = 2.1 - 2.0
m = deltay / deltax
print("slope when x changes from 2 to 2.5: ", m)
plt.plot([2.0, 2.1], [f(2.0), f(2.1)], 'k--')

# plot the tangent line, instantaneous slope is 4
# instantaneous slope (derivative) is 4, so line is y = 4x + b that goes through point (2,4), so y = 4x -4
# plot tangent at x=1.5 to x=3.5
x1 = 1.0
y1 = 4 * x1 - 4
x2 = 3.5
y2 = 4 * x2 - 4
plt.plot([x1, x2], [y1, y2], 'r-')
plt.plot(2, 4, 'ro');
plt.grid();

# example of slope as delta x decreases from 1 in powers of 10
x = 2
deltax = 10

for i in range(8):
    deltax = deltax / 10
    deltay = f(x + deltax) - f(x)
    m = deltay / deltax
    print("slope at x=%f for deltax=%0.16f is:%0.16f" % (x, deltax, m))
```

You should intuitively see that as we make the change in $x$ smaller ($\Delta x$) the lines they define that we
measure the slope of are approaching the red line.  The red line represents the tangent to the curve right at the point
we want to measure the slope $x = 2$.  For a quadratic curve like this one, the slope is not constant, it is constantly
changing.  Each location on the equation will have a slightly different slope.  But we can measure the
"instantaneous slope", also called the "instantaneous rate of change" at a given point.  This is simply the slope
of the red line shown.  More formally, in the limit as $\Delta x$ gets smaller and smaller and eventually reaches
0, the lines we measure the slope of get closer and closer to the red line, the tangent line which is the true
instantaneous slope of the equation at the point $x = 2$.  Formally we can use the concept of a limit to specify
this idea:

\begin{equation}
m_{\text{tan}} = \lim\limits_{\Delta x \to 0} \frac{\Delta y}{\Delta x} = \lim\limits_{\Delta x \to 0} \frac{f(x + \Delta x) - f(x)}{\Delta x}
\end{equation}

This idea, denoted by the notation $\frac{dy}{dx}$ is called the **derivative** of the function
$y = f(x)$ measured instantaneously at a given point $x$.  The derivative is another name for this notion
of the measure of the instantaneous rate of change, or the slope of the function at a particular point on
the function.

There are many different notations to represent this concept of the derivative:
$\frac{d f(x)}{dx}$, $\frac{d}{dx} f(x)$, $f'(x)$. (The last one is sometimes referred to as "prime notation.")

It is possible to come up with a formula that can be used to calculate the derivitaive of a function at
any point.  For example, lets look quickly at the function $y = f(x) = x^2$ above. For any value of $x$,
using the definition above, we have the following expression

\begin{equation}
m = \frac{\Delta y}{\Delta x}
  =  \frac{(x + \Delta x)^2 - x^2}{\Delta x}
  = \frac{x^2 + 2x \Delta x + (\Delta x)^2 - x^2}{\Delta x}
  = \frac{\Delta x (2x + \Delta x)}{\Delta x}
  = 2x + \Delta x
\end{equation}

The previous sequence may look complicated, but we are using simple algebra to expand and cancel out terms, just
with the maybe unfamiliar term $\Delta x$ in the expressions.  The result of this rearrangement though shows
something interesting.  This says simply that the slope of any line at any point of the function
is $2x + \Delta x$ for a given $\Delta$.  But also it says, in the limit, when $\Delta x = 0$, the
instantaneous slope will be simply $2x$.  Or in other words we have shown that for our function $f(x) = x^2$

\begin{equation}
\frac{d f(x)}{dx} = 2x
\end{equation}

This means that the derivative (instantaneous rate of change) of the function is $2x$ at any point $x$
of this function.  We can use this expression for the derivative of our function $y = x^2$
to determine the instantaneous rate of change at any point of our function.  For example:

```python
def f(x):
    """The simple quadratic function we are using y = x^2
    """
    return x**2

def df(x):
    """The derivative of our equation, can be used to determine the instantaneous
    rate of change, or in other words the slope, of the equation at any particular
    point x.
    """
    return 2 * x;

def plot_tangent_line(x):
    """A little utility function, this will plot a tangent line
    on the current figure at the location x, making use of the
    slope of the function f(x) using its derivative df(x), and
    plotting a line that goes through x with that slope
    """
    # determine value of y at the point x, and the slope of the line
    # at point x
    y = f(x)
    m = df(x)

    # y = mx + b
    # we have y and m for our needed line, need to determine
    # correct value of be, so rearrange
    # b = y - mx
    # and determine the correct intercept of the tangent line
    b = y - m * x

    # we will plot a line from -0.5 to +0.5 of x
    deltax = 0.75
    x0 = x - deltax
    x1 = x + deltax

    # we have the equation of the line, and we have two point x0, x1, determine
    # the points y0, y1 on the line
    y0 = m * x0 + b
    y1 = m * x1 + b

    # now we can plot the tangent line
    plt.plot([x0, x1], [y0, y1], 'r--')

    # plot a marker at the tangent point
    plt.plot(x, y, 'ro')

    # annotate the plot
    plt.text(x+0.05, y, 'x = %f\nm = %f' % (x, m))

# plot the original function
x = np.linspace(-5.0, 5.0)
plt.plot(x, f(x))
plt.grid();

# plot some tangent lines at different points, using the derivative (slope) of
# the function to do it.  We use a small utility function to plot several
# example tangent lines
plot_tangent_line(-4)
plot_tangent_line(-2)
plot_tangent_line(0)
plot_tangent_line(2)
plot_tangent_line(4)
```

# General Rules of Derivation

Using similar methods to this, we can also analyze functions like $y = x^3$ or $y = x^4$.  In general if you do
this you can easily see that a general rule exists for any power function $x^p$.  Namely if $y = f(x) = x^p$
then

**Power Rule**

\begin{equation}
\frac{d}{dx} (x^p) = p x^{p-1}
\end{equation}

This means that for example for the function $y = x^5$ the derivative of this function can be calculated
as

\begin{equation}
\frac{dy}{dx} = f'(x) = 5 x^{4}
\end{equation}

Likewise through similar analysis it is possible to derive other general rules of derivation.  Here are a few
that are useful:

**Exponential Rule**

If $y = f(x) = b^x$ then

\begin{equation}
\frac{d}{dx} (b^x) = b^x \ln(b)
\end{equation}

If $y = f(x) = e^{ax}$ then

\begin{equation}
\frac{d}{dx} (e^{ax}) = a e^{ax}
\end{equation}

**Logarathim Rule**

if $y = f(x) = \log_b(x)$ then

\begin{equation}
\frac{d}{dx} (\log_b(x)) = \frac{1}{x} \frac{1}{\ln(b)}
\end{equation}

as a special case, when $(b = e)$ then

\begin{equation}
\frac{d}{dx} (\ln(x)) = \frac{1}{x}
\end{equation}

## Properties of Derivatives

Given some general rules like this, we can define several properties of derivatives, that allow us to
determine exact equations for the derivatives of many different functions and combinations of functions.

1. Derivative of a **constant**
2. **Sum and Difference Rules**
3. **Product Rule**
4. **Quotient Rule**
5. **Chain Rule**

In a introductory calculus class you often spend a lot of time deriving the derivative
of mathematical expressions and learning to apply these rules to determine the symbolic
derivative for many types of functions.  In this class we do not really calculate
symbolic solutions for the derivatives of functions, because we are learning about
computational (approximate) methods to calculate things like the derivative of
functions.  So if you are rusty or haven't learned or practiced how to determine symbolically
the expression for the derivative of a function, it is not really that important for this
class, as long as you understand the basic concept of a derivative and what it is
and what it means.  With a solid understanding of the concept of a derivative, we can
then better understand the uses we might put this concept to to solve other types of
problems, including its relationship to the concept of an integral.

# Applications of Derivatives: Roots and Minima / Maxima

## Finding roots of an Equation

It is often useful to find what are known as the **roots** of an equation, i.e. the location where a function
$f(x) = 0$, or in other words the values where the graph of $f(x)$ intersects the x-axis.  Algebraically this can
be extremely tedious or even impossible, so we often use numerical techniques.  We will later look at a few such
root finding techniques.  The most basic technique known as the Newton method, or the Netwon-Raphson method, starts
with an initial guess $x_0$ and then produces a sequences of values (i.e.
it defines a series) that converges to a numerical solution
of a root.  By iterating the expression

\begin{equation}
x_i = x_{i-1} - \frac{f(x_{i-1})}{f'(x_{i-1})}
\end{equation}

Take for example the function

\begin{equation}
f(x) = x^3 - 21 x^2 + 135x - 220
\end{equation}

If we explore the graph of this funtion, it appears to have a root somewhere between 2.4 and 2.6

```python
def f(x):
    return x**3 - 21 * x**2 + 135 * x - 220

x = np.linspace(2.0, 3.0)
plt.plot(x, f(x))
plt.grid();
```

Starting with an initial value of $x_0 = 2$ we can iterate using the previous described Newton's method
like this:

```python
def f(x):
    return x**3 - 21 * x**2 + 135 * x - 220

def df(x):
    """Derivative of f(x)
    """
    return 3 * x**2 - 42 * x + 135

NUM_ITER = 10
x = np.zeros(NUM_ITER)
x[0] = 2.0
print(0, x[0])
for i in range(1, NUM_ITER):
    x[i] = x[i-1] - f(x[i-1]) / df(x[i-1])
    print(i, x[i])
```

    0 2.0
    1 2.4126984126984126
    2 2.461290404582591
    3 2.461940602324453
    4 2.4619407179495005
    5 2.461940717949505
    6 2.4619407179495045
    7 2.461940717949506
    8 2.4619407179495045
    9 2.461940717949506

We can confirm that this final value in the sequence is indeed a root of the equation
by plugging it back into the function.

```python
print(x[9], f(x[9]))
```

    2.461940717949506 5.684341886080802e-14

Since this is a numerical method, the result won't be exactly 0, but the function
should essentially be very close to 0 at the point we numerically determined is a
root.  You can also check the graph of the function below and see that the function
crosses the x axis at this value of x.

To understand why this works, look at the expression for the series.  We are
subtracting the expression

\begin{equation}
\frac{f(x)}{f'(x)}
\end{equation}

each time to come up with the next value in the series.  Remember we are trying to find
the root of the equation $y = f(x)$.  So when $f(x)$ is 0 or close to 0, this
expression is going to be 0 or close to 0, and we won't be subtracting much from the
current value in the series.

Also we need to combine information about whether $f(x)$ is positive or negative
along with information about whether the derivative (slope) is positive or negative
to determine whether we want to make $x$ bigger or smaller to move in a
direction closer to a root.  Look at the figure for our example function.  At the
starting point of $x_0 = 2$, the function $f(x) < 0$  thus $f(x)$ is negative.
The root in our vicinity is to the right (we need $x$ to increase).  The fact that
the derivative (slope) is positive at the point $x_0 = 2$ means that if we increase
$x$ we should be moving up the curve to a location closer to where $f(x) = 0$.  Thus
by combining information about the sign of $f(x)$ and the sign of the slope
$f'(x)$ this lets us determine if we need to increase or decrease our next value
of $x$.  In this particular case, since $f(x)$ is negative and the derivative $f'(x)$
is positive, the expression ends up being negative, and since we are subtracting a negative
we end up increasing $x$ in the next step.  If instead we had started at
$x_0 = 3$, the function $f(x)$ would be positive at this location, and the derivative
would also be positive, which would indicate we need to subtract some amount from
$x$ to go towards the root.  The same argument/intuition can be made for when the
derivative is negative, in which case which direction you need to go to approach the
root will be reversed from our example here, but again the general Newton's method
expression will work to move you closer to the root of the equation.

## Minimum and Maximum of Equations

Often it is useful to be able to "minimize" or "maximize" a function.  This means
that we want to find the location where the function is at its largest or smallest
value (often restricted to some range).

A function will be at a local maximum or minimum when its derivative is 0.

\begin{equation}
f'(x) = 0
\end{equation}

Take for example again the same function we have been using

\begin{equation}
f(x) = x^3 - 21 x^2 + 135x - 220
\end{equation}

```python
def f(x):
    return x**3 - 21 * x**2 + 135 * x - 220

x = np.linspace(2, 12, 1000)
#x = np.linspace(2.45, 2.47)
plt.plot(x, f(x))
plt.grid();
```

If we take the derivative of the function and set it to 0 to find
the locations of the minima and maxima we have

\begin{equation}
f'(x) = 3x^2 - 42x + 135 \\
3x^2 - 42x + 135 = 0 \\
3(x - 5) (x - 9) = 0
\end{equation}

We could have used the quadratic equation to determine the solution to setting our
derivative to 0, but in this case we can also relatively easily factor the quadratic
expression to easily see where the roots must be located.
Thus wee see that when $x = 5$ or $x = 9$ these will be solutions to this equation,
and from the graph we see that these are indeed the locations of the minimum and the
maximum of the function.

# Integrals of Functions

A definite integral of a continuous function $f(x)$ is a measure of the area under the graph of $f$ in
some interval $[a, b]$.  The integral of a function is related to the derivative of the function.  It is not
too difficult to derive this association.  Lets say you have a continuous function $f(x)$ and you want to
find the area under the curve of this graph in the interval $[a, x]$ from some *fixed* lower value $a$ to
any *variable* upper value $x$.

```python
def f(x):
    """An example of a specific function we want to find the area under the
    graph of.
    """
    return x * (x - 3) * (x - 6) + 20

plt.figure(figsize=(16, 12))
x = np.linspace(0, 6.25)
plt.plot(x, f(x), 'k-', linewidth=3)
plt.grid();

# a = 0, x = 4, deltax = 2
# represent F(x), function to calculate area from fixed a to x
a = 0
b = 4
deltax = 2
z = 5.35
x = np.linspace(a, b)
plt.fill_between(x, f(x), color='lightblue')

# represent F(x + deltax) - F(x)
deltax = 2
x = np.linspace(b, b+deltax)
plt.fill_between(x, f(x), color='lightgreen');

# label the graph
# first remove the specific tics and set general/example tic labels
plt.yticks([0])
plt.xticks([a, b, z, b+deltax], ['$a$', '$x$', '$z$', '$x + \Delta x$'], fontsize=20);

# label the functions
plt.xlim([0, 7.6])
plt.text(2.75, 25, '$y = f(x)$', fontsize=20)
plt.text(1.5, 12, '$F(x)$', fontsize=20)
plt.text(6.25, 15, '$f(z)$', fontsize=20)
plt.arrow(6.25, 15, z-6.05, f(z)-14.4, head_width=0.1, head_length=0.5, color='black')
plt.text(6.25, 7, '$F(x + \Delta x) - F(x)$', fontsize=20)
plt.arrow(6.25, 7, -0.5, 0, head_width=0.2, head_length=0.1, color='black')

# add lines for width and height of area
plt.plot([z, z], [0, f(z)], 'k--')
plt.plot([b, b+deltax], [f(z), f(z)], 'k--');
```

Let $F(x)$ = Area under the graph of $f$ in the interval $[a, x]$.

then $F(x + \Delta x)$ = Area under the graph of $f$ in the interval $[a, x + \Delta x]$

if we take the difference of these two areas we have

\begin{equation}
F(x + \Delta x) - F(x) = \text{Area under the graph of} f \; \text{in the interval} \; [x, x + \Delta x]
\end{equation}

The area of this difference (the green bit of the figure) has to be equal to the area of a rectangle with height $f(z)$,
where $z$ is some value in the interval $[x, x + \Delta x]$, and with width of $\Delta x$,
thus the area of the previous difference can be written as:

\begin{equation}
F(x + \Delta x) - F(x) = f(z) \Delta x
\end{equation}

rewriting we have

\begin{equation}
\frac{F(x + \Delta x) - F(x)}{\Delta x} = f(z)
\end{equation}

This should look a little familiar.  This is the finite difference equation we used (but with the function $F$) in
developing the concept of the derivative.  If we take this expression in the limit again as $\Delta x \to 0$
we see that the right hand side $f(z)$ becomes $f(x)$ (because z is some value in the interval from
$[x, x + \Delta x]$, and as $\Delta x \to 0$, $z$ simply becomes $x$).  Thus we have

\begin{equation}
\frac{d}{dx} F(x) = f(x)
\end{equation}

$F$ is called the **antiderivative** of $f$.  In other words, $F$ is the function such
that, if you take the derivative of it $F'(x)$ you will end up the the function
$f(x)$.  In essence, if you do the inverse of the process of determining
the derivative of a function, we can use this antiderivative to calculate values of the function $F$ in order to
exactly measure the area under the curve of functions $f$.

We formally express this concept of calculating the area under the graph of a function $f$ from its
antiderivative using the symbol for the definite integral of a function like this:

\begin{equation}
F(x) = \int_a^b f(x) \; dx
\end{equation}

## Properties of Integrals

Integrals possess the analogues of the properties of derivatives.  For example the antiderivative
of a constant and of the sum and difference of functions work in the same way as for derivatives.
The Power Rule, Logarithm Rule and Exponential Rule work basically in an inverted manner to calculate
the antiderivative of a function.

## Calculating Areas using Definite Integrals

If you can calculate the antiderivative of a function $f$, you can use this to calculate the area under
the graph of the curve of the function $f$ on any finite interval $[a, b]$.

For example, take the function

\begin{equation}
f(x) = x^3 (1 - x^4)^2
\end{equation}

The graph of this function in the interval $[0, 1]$ looks like this.

```python
def f(x):
    return x**3 * (1 - x**4)**2

a = 0
b = 1.0
x = np.linspace(a, b, 1000)
plt.plot(x, f(x), 'k-', linewidth=3)
plt.fill_between(x, f(x))
plt.grid();
```

As a rough estimate, since the maximum height of the function is 0.2, we know that the area of the box is 0.2.
The area under the graph looks like it is probably less than $1/2$ of the total area of this bounding box, so
I would expect an area of a bit under 0.1 for this function.

We can calculate the antiderivative of the function.  This function is a polynomial, so it is probably easiest
to simply exapand the terms and then calculate the antiderivative of the sum/difference of the terms, like this:

\begin{equation}
\int_0^1 x^3 (1 - x^4)^2 \; dx \\
\int_0^1 x^3 (1 - 2x^4 + x^8) \; dx \\
\int_0^1 x^3 - 2x^7 + x^{11} \; dx \\
\end{equation}

The antiderivative of simple terms raised to a power will use the reverse of the Power Rule, ths we have the
antiderivative

\begin{equation}
F(X) = \Big[ \frac{x^4}{4} - \frac{2 x^8}{8} + \frac{x^{12}}{12} \Big]_0^1
\end{equation}

(We leave it as an exercise for the student to prove to yourself that if you
take the derivative $\frac{d}{dx} F(X)$ you get the original function $f(x)$).)

Notice the notation of the 0 and 1 on the expression for the antiderivative here.  To use this
antiderivative to calculate the area under the curve exactly, you take

\begin{equation}
F(b) - F(a)
\end{equation}

thus we can calculate the area of this particular function in the interval $[a=0, b=1]$ as

\begin{equation}
\Big[ \frac{1^4}{4} - \frac{2 \cdot 1^8}{8} + \frac{1^{12}}{12} \Big] -
\Big[ \frac{0^4}{4} - \frac{2 \cdot 0^8}{8} + \frac{0^{12}}{12} \Big] =
\frac{1}{12} - 0 = \frac{1}{12} = 0.083333
\end{equation}

This is the general procedure for calculating a definite integral symbolically
(an exact solution).  You first need to derive the expression
for the antiderivative, using the inverse of the derivative rules, if you can.  If such an antiderivate
expression is possible, then the area under the curve for any definite interval can be calculated by taking
the difference of the antiderivative on the two ends of the interval.

# Acknowledgements and Versions

```python
%load_ext version_information

%version_information numpy, scipy, matplotlib
```

<table><tr><th>Software</th><th>Version</th></tr><tr><td>Python</td><td>3.7.3 64bit [GCC 7.3.0]</td></tr><tr><td>IPython</td><td>7.6.1</td></tr><tr><td>OS</td><td>Linux 5.0.0 29 generic x86_64 with debian buster sid</td></tr><tr><td>numpy</td><td>1.16.4</td></tr><tr><td>scipy</td><td>1.3.0</td></tr><tr><td>matplotlib</td><td>3.1.0</td></tr><tr><td colspan='2'>Fri Oct 04 09:59:24 2019 CDT</td></tr></table>


# Topic 1: Probability

## Associated Reading: Bishop 1.1, 1.2, 2.1

## 1. A definition of probability
We live in an uncertain world.  Sometimes, as computer scientists, we're sheltered from that because we think about things like deterministic algorithms for which there are certain guarantees about, for example, the amount of time it will take to run, or perhaps even more basic, that the procedure will yield a correct result every time.  For a given input, the path to a given output is well-defined (suspending consideration of the very interesting class of randomized algorithms).  However, when we do Machine Learning, we're doing something a bit different: we don't know how to write the program, so we're going to try and reconstruct it by observing the world$^1$.  In so doing, we have to come to terms with the notion that those observations are going to be imperfect in a variety of ways: imperfection in our means of observing, systematic errors in the way that we represent such information, wrong assumptions about how we allow our learning machines to behave, and old-fashioned, honest-to-goodness randomness.  Ignoring these sources of imperfection does not work: it is far better to be conscious of our uncertainties than overconfident in a notion that doesn't actually capture the truth.  The practical implications of this is that we are going to need to become comfortable expressing the lessons that we learn from data, and the predictions that we make based on what we've learned, in a language that's suited towards expressing these uncertainties.  That language is probability.

The stuff that we need to start working with the language of probability in a practical way is less extensive than you are probably expecting.  All that we need to take us quite a ways is a definition and a few rules for manipulation.  Let's start with a definition, and there's actually some competition between a few different ideas here.  You would think that with the ubiquity of statistics in modern society, that this would have already been settled 200 years ago.  But no, the debate continues to this day.  The two candidate definitions are:
- (Frequentist) $P(X=x)$ is the number of times we observe that a random variable $X=x$ for an infinite number of trials, divided by the total number of trials.  For example, in the limit as $n\rightarrow \infty$, the probability of a fair coin landing on heads is 1/2, so $P(coin=heads)=0.5$.
$$ \frac{N_{X=x}}{N_{trials}}, N_{trials}\rightarrow \infty $$
- (Bayesian) $P(X=x)$ is the degree to which we believe that for a given trial, we will observe that $X=x$.  So for $P(X=x)=1$, we're certain that $X=x$, and for $P(X=x)=0$, we're certain that this is not the case, and $P(X=x) = 0.5$, it's fifty-fifty.

The difference probably seems totally pedantic to you right now, but it actually isn't, especially in the context of machine learning or inverse problems in general.  In the case of the coin flip example, there's really not that much difference, because we can perform many repeats of the same experiment, such that we can approximate the limit as the number of trials go to infinity, and use those trials to update our probabilistic model of the coin.  However, there's a meaningful distinction when we start to reason about things that we can't conduct experiments on.  My research application area is in predicting change in the polar ice caps, and it provides a nice example: let's say that $\mathcal{H}$ is the specific hypothesis that 200 years from now, the Greenland Ice Sheet will have completely melted.  Intuitively, it's reasonable to ask what's $P(\mathcal{H})$?  What's the probability that this hypothesis is true?

If we plug this specific case into the frequentist definition, we see the problem: $P(H)$ is the number of times that we observe the Greenland Ice Sheet to melt out of an infinite number of trials?  We don't have an infinite number of trials to work with: we only have the one.  In contrast, there's no particular problem with the Bayesian definition: it's simply a statement that $P(H)$ encodes our current state of information about Greenland's future, or more generally $P(H)$ is a formal quantification of the information that we have about a preposition, when we're not yet exactly sure whether the preposition is true or false.  In fact, this idea is just a generalization of something that you already know about, which is prepositional logic: probability is the extension of logic to situations where prepositions aren't either one or zero, but something in the middle.  If it's not already clear, this second so-called Bayesian definition of probability is the one that we're going to concern ourselves with.  For practical purposes, we need to know just a few identities that largely mirror prepositional logic: the product rule, which is akin to the AND operation, and the sum rule, which is akin to OR.

$^1$. (This analogy is pretty loose, because we're actually going to exert quite a bit of control on exactly what types of programs we "learn" by specifying a mathematical model, although some cutting edge work towards general AI is indeed oriented towards trying to synthesize syntactically valid programs from data).

## 2. Joint probability rules

These two rules are best understood in the context of joint probability distributions, or the probability of two events occurring at the same time.  For example, $P(X=x,Y=y)$ quantifies our belief that $X=x$ and $Y=y$$^2$.

This is most easily understood by drawing a grid in two dimensions.  On the horizontal axis is the random variable X, on the vertical axis is the random variable Y.  Then we divide the axes into three subdivisions each, and call each subdivision 1,2, or 3.

$^2$ (If you haven't seen this upper case-lower case notation before, here's how it works.  Upper case letters are random variables: it's just the name of a thing that could take different values.  It's been declared, and it has a type, but we don't know the exact value.  Lower case letters represent actual specific values.  For example if $X$ is a random variable representing the outcome of a coin flip, then the *support* of $X$ is the set $\{heads,tails\}$.  $x$ in this case is a specific outcome from that set, i.e. for a fair coin $P(X=heads) = 50$%.)

### 2.1 Sum rule
So imagine, that this thing is a set of boxes, and we're randomly throwing balls into the boxes.  The probability of the ball landing in any particular box is given by the joint probability $P(X,Y)$.  In our case, we have 9 cells, so $P(X=2,Y=2) = 1/9$.

**Now, what is $P(X=2)$?**

It is, of course, $1/3$.  This is pretty obvious, but it's instructive to write it as
$$P(X=2) = P(X=2,Y=1) + P(X=2,Y=2) + P(X=2,Y=3) = \sum_{i=1}^3 P(X=2,Y=y_i) = 1/3.$$
$P(X=2)$ is known as the *marginal probability*, and this procedure is known as the sum rule, or generally:
$$
P(X=x_j) = \sum_{i=1}^m P(X=x_j,Y=y_i).
$$

### 2.2 Product rule
Now let's consider cases where we're given the value of one of the random variables.  This is called *conditional probability*, annotated $P(Y=y_j|X=x_i)$.

**If we know that $X=2$, what is the probability that $Y=2$?**  This is pretty obvious graphically, but worth writing down the actual mathematical relationship between the joint distribution and the conditional, which is
$$P(X=x,Y=y) = P(X=x|Y=y) P(Y=y)$$.  We are after the first term on the right side, so if we divide the left hand term (which is 1/9) by the rightmost (which is 1/3), we get, as expected, 1/3.

### 2.3 Exchangability
Nothing changes if we swap the axis-labels, so $P(X=x,Y=y) = P(Y=y,X=x)$.

### 2f.4 Conditional probability examples
Equipped with these probability rules, now let's use them to answer some simple questions.  To begin, Let's say that we've got two two bowls, green and blue.  Inside green, we have 1 kiwis and 3 oranges.  Inside blue, we have 3 kiwis and 1 orange.  Now what if I asked the question, whats the probability of drawing an orange if I sample from both bowls with equal likelihood?  Then we have that
\begin{align}
P(Orange) & = P(Orange,Blue) + P(Orange,Green) \\
          & = P(Orange|Blue)P(Blue) + P(Orange|Green)P(Green) \nonumber \\
        & = (1/4)(1/2) + (3/4)(1/2) = 1/2. \nonumber
\end{align}
*Note that I've simplified notation a little bit: Instead of writing P(Fruit=Orange,Bowl=Blue), I'm just writing P(Orange,Blue)*.  What if I move these three kiwis out of blue, and into green.  So now there's 1 orange in green, and 4 kiwis and three oranges in blue.  What's the probability of an orange?
\begin{align}
P(Orange) & = P(Orange,Blue) + P(Orange,Green) = P(Orange|Blue)P(Blue) \nonumber \\
          & + P(Orange|Green)P(Green) \nonumber \\
          & = (3/7)(1/2) + (1)(1/2) = 3/14 + 7/14 = 5/7
\end{align}
We are now more likely to pick a bowl with more oranges, so the overall likelihood of picking an orange goes up.  Now, what if I ask, under the same rules as above, "I picked an orange.  What is the probability that this orange came from the green bowl?".  This requires a bit more thought.  Before, we had a scenario where our conditional probabilities were determined entirely by things that we knew already.  Given a bowl color, I can tell you directly what the probability of a fruit is.  But in this case, oranges could come from either bowl, so having an orange doesn't directly determine the probability of a bowl.  Instead we need to come up with a rule for inverting probability in some way.  And this isn't very difficult using the tools we have above.  Let's start with the identity:
$$P(X=x,Y=y) = P(Y=y,X=x),$$
now apply the product rule and divide:
$$P(X=x|Y=y) = \frac{P(Y=y|X=x)P(X=x)}{P(Y=y)}$$
Now we can use this result directly:
$$P(Green|Orange) = \frac{P(Orange|Green)P(Green)}{P(Orange)} = (1)(1/2)(7/5) = 7/10$$
Because green was a sure bet for orange, and the probability of choosing green was equal to that of blue, we find that it's quite a bit more likely that the orange came from the green bowl.

### 2.5 Bayes rule
The formula that we used to compute this turns out to be so important that it has its own special proper-noun name: Bayes' Theorem.  The name is stupid by the way, it was in fact Pierre-Simon Laplace who first understood it in its modern usage and also developed some of its philosophical implications.  And once again, you're probably thinking "so what?", how does this help me do machine learning?  And the answer becomes a little bit more plain if we apply Bayes' theorem to the case where we have to pick between competing hypotheses that describe the world, which we'll call models $M$ given some real data about the world, which we'll call $D$.  Bayes' theorem then says
$$P(M=m|D=d) = \frac{P(D=d|M=m)P(M=m)}{P(D=d)},$$
which is to say that it provides us a statistical formula for selecting between models of reality given observations of said reality.  Let's take a look at the anatomy of this equation:  The thing on the left is called the *posterior probability* or the probability that a given hypothesis is true after considering the data.  On the right hand side, $P(M=m)$ is called the *prior probability*, which is the probability of a particular hypothesis being the correct one *prior* to considering the data.  The prior probability is updated according to the first term in the numerator which is called the *likelihood*, which answers the question, assuming that a particular model $M=m$, what is the probability that we will observe the data $y$?  Finally, the denominator is often called the *evidence*, and it's the probability of observing the data under all possible hypotheses.

This formula shapes your life and your thinking in ways that you have never imagined.  Least squares linear regression (which we'll get to)? Bayes rule.  Want to go bet on a horse race and be sure that you're not being scammed?  Bayes' rule.  *The fundamental human process of updating actions based on sensory input*?  Empirically, also Bayes' rule.  Everything (and I mean everything) that we do in this course from here on will be some application of Bayes rule, sometimes explicitly, sometimes implicitly.

# Lab 09/02
*From your group, choose a scribe.  Then compose a document responding to the following.  When complete, please turn in the assignment to the appropriate link in this week's Moodle page.*

A classic example of Bayes' theorem in action comes from the world of disease testing. This is a really good example because it shows the importance of the prior distribution. So here's the problem: imagine that you're concerned that you have COVID-19, something that currently affects roughly 7 in 1000 people in the US, and you go down to the county health services to take a COVID test. The doctor tells you that the sensitivity of the test is 70%, which is to say that if you have the disease the test will read positive 70% of the time. We also know that the test has a specificity of 98%, which implies that 98% of the time, if you don't have the disease, the test will read negative. Unfortunately, your test comes back positive.

**Compute the probability that you actually have the disease.**

**Using python (matplotlib), generate a plot showing the probability above as a function of the specificity.**

**Devise a simple strategy for reducing the false positive rate**

```python

```