# (Bonus) Tutorial 2: Facial recognition using modern convnets

**Week 2, Day 2: Modern Convnets**

**By Neuromatch Academy**

__Content creators:__ Laura Pede, Richard Vogg, Marissa Weis, Timo Lüddecke, Alexander Ecker

__Content reviewers:__ Arush Tagade, Polina Turishcheva, Yu-Fang Yang, Bettina Hein, Melvin Selim Atay, Kelson Shilling-Scrivo

__Content editors:__ Roberto Guidotti, Spiros Chavlis

__Production editors:__ Anoop Kulkarni, Roberto Guidotti, Cary Murray, Spiros Chavlis

*Notebook is based on an initial version by Ben Heil*

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
# Tutorial Objectives

In this tutorial you will learn about:

1. An application of modern CNNs in facial recognition.
2. Ethical aspects of facial recognition.

```python
# @title Tutorial slides

# @markdown These are the slides for the videos in this tutorial

# @markdown If you want to download locally the slides, click [here](https://osf.io/4r2dp/download)
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/4r2dp/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

---
# Setup

```python
# @title Install dependencies
# @markdown Install `facenet` - a model used to do facial recognition
!pip install facenet-pytorch --quiet
!pip install Pillow --quiet
```

```python
# Imports
import glob
import torch

import numpy as np
import sklearn.decomposition
import matplotlib.pyplot as plt

from PIL import Image

from torchvision import transforms
from torchvision.utils import make_grid
from torchvision.datasets import ImageFolder

from facenet_pytorch import MTCNN, InceptionResnetV1
```

```python
# @title Set random seed

# @markdown Executing `set_seed(seed=seed)` you are setting the seed

# for DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html

# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch

def set_seed(seed=None, seed_torch=True):
  if seed is None:
    seed = np.random.choice(2 ** 32)
  random.seed(seed)
  np.random.seed(seed)
  if seed_torch:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

  print(f'Random seed {seed} has been set.')

# In case that `DataLoader` is used
def seed_worker(worker_id):
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)
```

```python
# @title Set device (GPU or CPU). Execute `set_device()`
# especially if torch modules used.

# inform the user if the notebook uses GPU or CPU.

def set_device():
  device = "cuda" if torch.cuda.is_available() else "cpu"
  if device != "cuda":
    print("WARNING: For this notebook to perform best, "
        "if possible, in the menu under `Runtime` -> "
        "`Change runtime type.`  select `GPU` ")
  else:
    print("GPU is enabled in this notebook.")

  return device
```

```python
SEED = 2021
set_seed(seed=SEED)
DEVICE = set_device()
```

    Random seed 2021 has been set.
    GPU is enabled in this notebook.

---
# Section 1: Face Recognition

*Time estimate: ~12mins*

## Section 1.1: Download and prepare the data

```python
# @title Download Data
import requests, zipfile, io, os

# original link: https://github.com/ben-heil/cis_522_data.git
url = 'https://osf.io/2kyfb/download'

fname = 'faces'

if not os.path.exists(fname+'zip'):
  print("Data is being downloaded...")
  r = requests.get(url, stream=True)
  z = zipfile.ZipFile(io.BytesIO(r.content))
  z.extractall()
  print("The download has been completed.")
else:
  print("Data has already been downloaded.")
```

    Data is being downloaded...
    The download has been completed.

```python
# @title Video 1: Face Recognition using CNNs
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV17B4y1K7WV", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"jJqEv8hpRa4", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

One application of large CNNs is **facial recognition**. The problem formulation in facial recognition is a little different from the image classification we've seen so far. In facial recognition, we don't want to have a fixed number of individuals that the model can learn. If that were the case then to learn a new person it would be necessary to modify the output portion of the architecture and retrain to account for the new person.

Instead, we train a model to learn an **embedding** where images from the same individual are close to each other in an embedded space, and images corresponding to different people are far apart. When the model is trained, it takes as input an image and outputs an embedding vector corresponding to the image.

To achieve this, facial recognitions typically use a **triplet loss** that compares two images from the same individual (i.e., "anchor" and "positive" images) and a negative image from a different individual (i.e., "negative" image). The loss requires the distance between the anchor and negative points to be greater than a margin $\alpha$ + the distance between the anchor and positive points.

## Section 1.2: View and transform the data

A well-trained facial recognition system should be able to map different images of the same individual relatively close together. We will load 15 images of three individuals (maybe you know them - then you can see that your brain is quite well in facial recognition).

After viewing the images, we will transform them: MTCNN ([github repo](https://github.com/ipazc/mtcnn)) detects the face and crops the image around the face. Then we stack all the images together in a tensor.

```python
# @title Display Images
# @markdown Here are the source images of Bruce Lee, Neil Patrick Harris, and Pam Grier
train_transform = transforms.Compose((transforms.Resize((256, 256)),
                                      transforms.ToTensor()))

face_dataset = ImageFolder('faces', transform=train_transform)

image_count = len(face_dataset)

face_loader = torch.utils.data.DataLoader(face_dataset,
                                          batch_size=45,
                                          shuffle=False)

dataiter = iter(face_loader)
images, labels = dataiter.next()

# show images
plt.figure(figsize=(15, 15))
plt.imshow(make_grid(images, nrow=15).permute(1, 2, 0))
plt.axis('off')
plt.show()
```

```python
# @title Image Preprocessing Function
def process_images(image_dir: str, size=256):
  """
  This function returns two tensors for the given image dir: one usable for inputting into the
  facenet model, and one that is [0,1] scaled for visualizing

  Parameters:
    image_dir: The glob corresponding to images in a directory

  Returns:
    model_tensor: A image_count x channels x height x width tensor scaled to between -1 and 1,
                  with the faces detected and cropped to the center using mtcnn
    display_tensor: A transformed version of the model tensor scaled to between 0 and 1
  """
  mtcnn = MTCNN(image_size=size, margin=32)
  images = []
  for img_path in glob.glob(image_dir):
    img = Image.open(img_path)
    # Normalize and crop image
    img_cropped = mtcnn(img)
    images.append(img_cropped)

  model_tensor = torch.stack(images)
  display_tensor = model_tensor / (model_tensor.max() * 2)
  display_tensor += .5

  return model_tensor, display_tensor
```

Now that we have our images loaded, we need to preprocess them. To make the images easier for the network to learn, we crop them to include just faces.

```python
bruce_tensor, bruce_display = process_images('faces/bruce/*.jpg')
neil_tensor, neil_display = process_images('faces/neil/*.jpg')
pam_tensor, pam_display = process_images('faces/pam/*.jpg')

tensor_to_display = torch.cat((bruce_display, neil_display, pam_display))

plt.figure(figsize=(15, 15))
plt.imshow(make_grid(tensor_to_display, nrow=15).permute(1, 2, 0))
plt.axis('off')
plt.show()
```

## Section 1.3: Embedding with a pretrained network

We load a pretrained facial recognition model called [FaceNet](https://github.com/timesler/facenet-pytorch). It was trained on the [VGGFace2](https://github.com/ox-vgg/vgg_face2) dataset which contains 3.31 million images of 9131 individuals.

We use the pretrained model to calculate embeddings for all of our input images.

```python
resnet = InceptionResnetV1(pretrained='vggface2').eval().to(DEVICE)
```

```python
# Calculate embedding
resnet.classify = False
bruce_embeddings = resnet(bruce_tensor.to(DEVICE))
neil_embeddings = resnet(neil_tensor.to(DEVICE))
pam_embeddings = resnet(pam_tensor.to(DEVICE))
```

### Think! 1.3: Embedding vectors

We want to understand what happens when the model receives an image and returns the corresponding embedding vector.

- What are the height, width and number of channels of one input image?
- What are the dimensions of one stack of images (e.g. bruce_tensor)?
- What are the dimensions of the corresponding embedding (e.g. bruce_embeddings)?
- What would be the dimensions of the embedding of one input image?

**Hints:**
- You can double click on a variable name and hover over it to see the dimensions of tensors.
- You do not have to answer the questions in the order they are asked.

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W2D2_ModernConvnets/solutions/W2D2_Tutorial2_Solution_22e742e7.py)

We cannot show 512-dimensional vectors visually, but using **Principal Component Analysis (PCA)** we can project the 512 dimensions onto a 2-dimensional space while preserving the maximum amount of data variation possible. This is just a visual aid for us to understand the concept. Note that if you would like to do any calculation, like distances between two images, this would be done with the whole 512-dimensional embedding vectors.

```python
embedding_tensor = torch.cat((bruce_embeddings,
                              neil_embeddings,
                              pam_embeddings)).to(device='cpu')

pca = sklearn.decomposition.PCA(n_components=2)
pca_tensor = pca.fit_transform(embedding_tensor.detach().cpu().numpy())
```

```python
num = 15
categs = 3
colors = ['blue', 'orange', 'magenta']
labels = ['Bruce Lee', 'Neil Patrick Harris', 'Pam Grier']
markers = ['o', 'x', 's']
plt.figure(figsize=(8, 8))
for i in range(categs):
   plt.scatter(pca_tensor[i*num:(i+1)*num, 0],
               pca_tensor[i*num:(i+1)*num, 1],
               c=colors[i],
               marker=markers[i], label=labels[i])
plt.legend()
plt.title('PCA Representation of the Image Embeddings')
plt.xlabel('PC 1')
plt.ylabel('PC 2')
plt.show()
```

Great! The images corresponding to each individual are separated from each other in the embedding space!

If Neil Patrick Harris wants to unlock his phone with facial recognition, the phone takes the image from the camera, calculates the embedding and checks if it is close to the registered embeddings corresponding to Neil Patrick Harris.

---
# Section 2: Ethics – bias/discrimination due to pre-training datasets

*Time estimate: ~19mins*

Popular facial recognition datasets like VGGFace2 and CASIA-WebFace consist primarily of caucasian faces.
As a result, even state of the art facial recognition models [substantially underperform](https://openaccess.thecvf.com/content_ICCV_2019/papers/Wang_Racial_Faces_in_the_Wild_Reducing_Racial_Bias_by_Information_ICCV_2019_paper.pdf) when attempting to recognize faces of other races.

Given the implications that poor model performance can have in fields like security and criminal justice, it's very important to be aware of these limitations if you're going to be building facial recognition systems.

In this example we will work with a small subset from the [UTKFace](https://susanqq.github.io/UTKFace/) dataset with 49 pictures of black women and 49 picture of white women. We will use the same pretrained model as in Section 8 of Tutorial 1, see and discuss the consequences of the model being trained on an imbalanced dataset.

```python
# @title Video 2: Ethical aspects
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Jo4y1Q7K3", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"vYilJV3PqUM", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

## Section 2.1: Download the Data

```python
# @title Run this cell to get the data

# original link: https://github.com/richardvogg/face_sample.git
url = 'https://osf.io/36wyh/download'
fname = 'face_sample2'

if not os.path.exists(fname+'zip'):
  print("Data is being downloaded...")
  r = requests.get(url, stream=True)
  z = zipfile.ZipFile(io.BytesIO(r.content))
  z.extractall()
  print("The download has been completed.")
else:
  print("Data has already been downloaded.")
```

## Section 2.2: Load, view and transform the data

```python
black_female_tensor, black_female_display = process_images('face_sample2/??_1_1_*.jpg', size=150)
white_female_tensor, white_female_display = process_images('face_sample2/??_1_0_*.jpg', size=150)
```

We can check the dimensions of these tensors and see that for each group we have images of size $150 \times 150$ and three channels (RGB) of 49 individuals.

**Note:** Originally, the size of images was $200 \times 200$, but due to RAM resources, we have reduced it. You can change it back, i.e., `size=200`.

```python
print(white_female_tensor.shape)
print(black_female_tensor.shape)
```

```python
# @title Visualize some example faces
tensor_to_display = torch.cat((white_female_display[:15],
                               black_female_display[:15]))

plt.figure(figsize=(12, 12))
plt.imshow(make_grid(tensor_to_display, nrow = 15).permute(1, 2, 0))
plt.axis('off')
plt.show()
```

## Section 2.3: Calculate embeddings

We use the same pretrained facial recognition network as in section 8 to calculate embeddings. If you have memory issues running this part, go to `Edit > Notebook settings` and check if GPU is selected as `Hardware accelerator`. If this does not help you can restart the notebook, go to `Runtime -> Restart runtime`.

```python
resnet.classify = False
black_female_embeddings = resnet(black_female_tensor.to(DEVICE))
white_female_embeddings = resnet(white_female_tensor.to(DEVICE))
```

We will use the embeddings to show that the model was trained on an imbalanced dataset. For this, we are going to calculate a distance matrix of all combinations of images, like in this small example with $n=3$ (in our case $n=98$).

Calculate the distance between each pair of image embeddings in our tensor and visualize all the distances. Remember that two embeddings are vectors and the distance between two vectors is the Euclidean distance.

```python
# @title Function to calculate pairwise distances

# @markdown [`torch.cdist`](https://pytorch.org/docs/stable/generated/torch.cdist.html) is used

def calculate_pairwise_distances(embedding_tensor):
  """
  This function calculates the distance between each pair of image embeddings
  in a tensor using the `torch.cdist`.

  Parameters:
    embedding_tensor : torch.Tensor
      A num_images x embedding_dimension tensor

  Returns:
    distances : torch.Tensor
      A num_images x num_images tensor containing the pairwise distances between
      each to image embedding
  """

  distances = torch.cdist(embedding_tensor, embedding_tensor)

  return distances
```

```python
# @title Visualize the distances

embedding_tensor = torch.cat((black_female_embeddings,
                              white_female_embeddings)).to(device='cpu')

distances = calculate_pairwise_distances(embedding_tensor)

plt.figure(figsize=(8, 8))
plt.imshow(distances.detach().cpu().numpy())
plt.annotate('Black female', (2, -0.5), fontsize=20, va='bottom')
plt.annotate('White female', (52, -0.5), fontsize=20, va='bottom')
plt.annotate('Black female', (-0.5, 45), fontsize=20, rotation=90, ha='right')
plt.annotate('White female', (-0.5, 90), fontsize=20, rotation=90, ha='right')
cbar = plt.colorbar()
cbar.set_label('Distance', fontsize=16)
plt.axis('off')
plt.show()
```

## Exercise 2.1

What do you observe? The faces of which group are more similar to each other for the Face Detection algorithm?

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W2D2_ModernConvnets/solutions/W2D2_Tutorial2_Solution_2309aa23.py)

## Exercise 2.2
- What does it mean in real life applications that the distance is smaller between the embeddings of one group?
- Can you come up with example situations/applications where this has a negative impact?
- What could you do to avoid these problems?

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W2D2_ModernConvnets/solutions/W2D2_Tutorial2_Solution_9a053438.py)

Lastly, to show the importance of the dataset which you use to pretrain your model, look at how much space white men and women take in different embeddings. *FairFace* is a dataset which is specifically created with completely balanced classes. The blue dots in all visualizations are white male and white female.

Adopted from [Kärkkäinen and Joo, 2019, arXiv](https://arxiv.org/abs/1908.04913)

---
# Section 3: Within Sum of Squares

*Time estimate: ~10mins*

We can try to put this observation in numbers. For this we work with the embeddings.
We want to calculate the centroid of each group, which is the average of the 49 embeddings of the group. As each embedding vector has a dimension of 512, the centroid will also have this dimension.

Now we can calculate how far away the observations $x$ of each group $S_i$ are from the centroid $\mu_i$. This concept is known as Within Sum of Squares (WSS) from cluster analysis.

\begin{equation}
\text{WSS} = \sum_{x\in S_i} ||x - \mu_i||^2
\end{equation}

where $|| \cdot ||$ is the Euclidean norm.

The Within Sum of Squares (WSS) is a number which measures this variability of a group in the embedding space. If all embeddings of one group were very close to each other, the WSS would be very small. In our case we see that the WSS for the black females is much smaller than for the white females. This means that it is much harder for the model to distinguish two black females than to distinguish two white females. The WSS complements the observation from the distance matrix, where we observed overall smaller pairwise distances between black females.

```python
# @title Function to calculate WSS

def wss(group):
  """
  This function returns the sum of squared distances of the N vectors of a
  group tensor (N x K) to its centroid (1 x K).

  Args:
    group: A image_count x embedding_size tensor

  Returns:
    sum_sq: A 1x1 tensor with the sum of squared distances.

  Hints:
    - to calculate the centroid, torch.mean() will be of use.
    - We need the mean of the N=49 observations. If our input tensor is of size
      N x K, we expect the centroid to be of dimensions 1 x K.
      Use the axis argument within torch.mean
    """

  centroid = torch.mean(group, axis=0)
  distance = torch.linalg.norm(group - centroid.view(1, -1), axis=1)
  sum_sq = torch.sum(distance**2)
  return sum_sq
```

```python
# @markdown Let's calculate the WSS for the two groups of our example.

print(f"Black female embedding WSS: {np.round(wss(black_female_embeddings).item(), 2)}")
print(f"White female embedding WSS: {np.round(wss(white_female_embeddings).item(), 2)}")
```

---
# Summary

In this tutorial we have learned how to apply a modern convnet in real application such as facial recognition. However, as the state-of-the-art tools for facial recognition are trained mostly with caucasian faces, they fail or they perform much worst when they have to deal with faces from other races.


```python
%matplotlib notebook
%matplotlib inline
import math
import matplotlib.pyplot as plt

```

# Nuclear Fuel Resources, Mining, and Milling

**Mining:** Nuclear fuel starts as natural uranium or thorium deposits in the earth. This material can be mined from the ground or even extracted from the sea.

**Milling:** and other chemical and physical processes then follow in order to produce a uniform, concentrated material. In the case of Uranium, the product is called 'yellowcake'.

## Learning Objectives

This lesson should equip you to:

- Describe the extent, geography, and characteristics of uranium and thorium resources.
- Understand various methods of prospecting.
- Differentiate various methods for uranium extraction.
- Model the economic relationship between uranium resources and extraction price.
- Apply decay energy calculations to the Radon decay chain.
- Evaluate the radiation danger associated with uranium mining.
- Calculate radition exposure for mine workers.
- Understand the process in milling operations.
- Recognize the chemical form of yellowcake and its role in the fuel cycle.

# Uranium Resources

There are important distinctions between production, supply, known reserves, and estimated resources. The book makes the distinction, further, among  "probable," "possible," and "speculative" resources.

## Geography of Uranium Resources

| Mine | Country | Main owner | Type | Production (tU) | % of world |
| ------------ |:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|
| McArthur River | Canada | Cameco | underground | 7356 | 13 |
| Tortkuduk & Moinkum | Kazakhstan | Katco JV/Areva, Kazatomprom | ISL | 4322 | 8 |
| Olympic Dam | Australia | BHP Billiton | by-product/underground | 3351 | 6 |
|  SOMAIR | Niger | Areva | open pit | 2331 | 5 |
| Budenovskoye 2 | Kazakhstan | Karatau JV/Kazatomprom, Uranium One | ISL | 2084 | 4 |
| South Inkai | Kazakhstan | Betpak Dala JV/Uranium One, Kazatomprom | ISL | 2002 | 4 |
| Priagunsky | Russia | ARMZ | underground | 1970 | 4 |
| Langer Heinrich | Namibia | Paladin | open pit | 1947 | 4 |
| Inkai | Kazakhstan | Inkai JV/Cameco, Kazatomprom | ISL | 1922 | 3 |
| Central Mynkuduk | Kazakhstan | JSC Ken Dala, Kazatomprom | ISL | 1790 | 3 |
| Top 10 total |   | | | 29,075 | 54% |
[Source: WNA](http://www.world-nuclear.org/information-library/nuclear-fuel-cycle/mining-of-uranium/uranium-mining-overview.aspx)

## Geology of Uranium Resources

### Grades of Uranium Ore
Uranium deposits are found in different "grades" of ore. The "grade" is the mass percentage of the rock that is made up of elemental uranium.

| Ore Grade        | PPM  |
| ------------- |:-------------:|
| Very high-grade ore (Canada) – 20% U	| 200,000 ppm U
| High-grade ore – 2% U,	| 20,000 ppm U |
| Low-grade ore – 0.1% U, |	1,000 ppm U |
| Very low-grade ore* (Namibia) – 0.01% U	| 100 ppm U |
| Granite	| 3-5 ppm U |
| Sedimentary rock	| 2-3 ppm U |
| Earth's continental crust (av)	| 2.8 ppm U |
| Seawater |	0.003 ppm U |
[Source: WNA](http://www.world-nuclear.org/information-library/nuclear-fuel-cycle/uranium-resources/supply-of-uranium.aspx)

```python
def U(U_0, z):
    """Uranium concentration as a function of depth.
    :param U_0: concentration at surface in ppm
    :param z: depth
    :return c: concentration at depth z"""
    c = U_0*math.exp(-z/6300)
    return c
```

```python
b = range(1, 1000, 5)
to_plot = [U(2.8, z) for z in b]
```

```python
plt.plot(to_plot)
plt.title('Uranium Concentration as a Function of Depth')
plt.ylabel('Uranium Concentration (ppm)')
plt.xlabel('Depth [m]')
```

## Extent of Uranium Resources

Uranium production was at one point far ahead of demand.
Recently, mining follows demand a bit more closely.

And, we know that there is a lot more out there.

|   Nation  | tonnes U | percentage of world |
| ------------ |:-------------:|:-------------:|
|  Australia | 1,706,100 | 29% |
|  Kazakhstan | 679,300 | 12% |
|  Russia | 505,900 | 9% |
|  Canada | 493,900 | 8% |
|  Niger | 404,900 | 7% |
|  Namibia | 382,800 | 6% |
|  South Africa | 338,100 | 6% |
|  Brazil | 276,100 | 5% |
|  USA | 207,400 | 4% |
|  China | 199,100 | 4% |
|  Mongolia | 141,500 | 2% |
|  Ukraine | 117,700 | 2% |
|  Uzbekistan | 91,300 | 2% |
|  Botswana | 68,800 | 1% |
|  Tanzania | 58,500 | 1% |
|  Jordan | 40,000 | 1% |
|  Other | 191,500 | 3% |
|  World total | 5,902,900 |  100% |
[Reasonably Assured Resources, Source: WNA](http://www.world-nuclear.org/information-library/nuclear-fuel-cycle/mining-of-uranium/world-uranium-mining-production.aspx)

## Prospecting

- Geologic Study

- Botanical Study

- Airborne Study

- Surface Study

- Well Logging

### Gamma Decay Review

Radiometric measurements are typically taken from the air by a gamma ray detector.

### Discussion: What naturally occurring elements have reasonably detectable gamma decays?

## Open Pit Mining

## Underground Mining

Mi Vida uranium mine in Moab, UT

### Radon

### Discussion: If the daughters of radon are solids, how do they get into your lungs?

### Discussion: Given this, what methods, in addition to moving the air around, might mitigate recieved dose?

### Working Level

The working level is a unit of measurement for radon exposure. Alpha particles from radon and its progeny (daughters) are ionizing radiation and damage the DNA of the cells of your lungs, sometimes leading to cancer.

The radon progeny in equilibrium with $100 \frac{pCi}{L}$ radon gas at a mine would typically release approximately 130,000 MeV alpha energy in decay, so the WL was defined as:

\begin{align}
1 WL &= 130,000\frac{MeV}{L} \mbox{ alpha energy from radon + daughters}\\
     &= 20.8 \frac{\mu J}{m^3} \mbox{ alpha energy from radon + daughters}\\
\end{align}

The nominal working hours per month assume 40 hours per week and 4.25 weeks per month. Accordingly, the working level month (WLM) was introduced:

\begin{align}
WLM &= \mbox{Working Level Month}\\
    &= \mbox{1 WL exposure for 170 hours}\\
1 WLM &= 170 x 20.8 = 3.54 \frac{\mu J h}{m^3}\\
\end{align}

Note that for conversion's sake:

\begin{align}
17,000\frac{pCi h}{L} &= 1 WLM\\
1\frac{pCi h}{L} &= 59 \mu WLM
\end{align}

```python
r = 2.6e5 # energy from alphas emitted by radon + progeny [MeV/L]
h = 60*12 # working hours
wl = 1.3e5 # worker level in [MeV/L]

print((r/wl), " working level ")
print((r/wl)*(h/170), " working level months")
```

    2.0  working level
    8.470588235294118  working level months

## Energy vs. Rate of decays

Radiation energy is measured in **electronvolts (eV), megaelectronvolts (MeV)** for convenience, and **Joules (J)**:

\begin{align}
1 J &= 6.242\times10^{12} MeV\\
1 \frac{J}{s} &= 1 W\\
\end{align}

**Becquerels (Bq)** or **Curies (Ci)** measure the rate of decays from a source :
\begin{align}
1 Bq &= \frac{\mbox{emission}}{s}\\
1 Ci &= 37 GBq\\
     &= 37000 MBq\\
1 Bq &= 27 pCi\\
     &= 27\times10^{-12}Ci\\
\end{align}

### Discussion: What is another name for this rate?

To convert between these, one must consider the energy per emission.

### Calculation of Alpha Decay Energy

Calculate the energy Q, in MeV, released during the alpha
decay $^{ZZZ}I_i \stackrel{\alpha}{\longrightarrow} {^{YYY}}I_d$:

\begin{align}
                        m_i &= \mbox{initial mass }[u]\nonumber\\
                        m_f &= \mbox{final mass }[u]\nonumber\\
                            &= \mbox{mass of daughter + mass of alpha }[u]\nonumber\\
                            &= m_d + m_{\alpha}\\
                        m_d &= \mbox{mass of daughter}[u]\nonumber\\
                        m_{\alpha} &= \mbox{mass of alpha }[u]\nonumber\\
                        Q\left[\frac{u\cdot m^2}{s^2}\right] &= m_ic^2 - m_fc^2\nonumber\\
\end{align}

This equation will be used for all alpha decay calculations. These
will only vary in the masses. Masses, however, are typically expressed in units of amu (or, u).

\begin{align*}
                        m_{\alpha} &= m_{\alpha} [u]\\
                        m_i &= m_{I_i} [u]\\
                        m_d &= m_{I_f} [u]\\
                 Q[MeV] &= (m_i - m_f)c^2\left(931.494 \frac{MeV}{c^2u}\right)\\
                        &= (m_i - m_d - m_{\alpha})\left(931.494\right)\\
                 Q[MeV] &= 931.494\left(m_i - m_d - m_{\alpha}\right)\\
\end{align*}

The energy released in alpha decay is converted into kinetic energy of the two final particles (daughter and alpha). Recall that kinetic energy is $\frac{1}{2}mv^2$, so if the energy is split, the speed of the alpha particle will be significantly higher than the speed of the (much larger, typically) daughter particle.

### A Note on Radiation Weighting Factors

The conversion from energy to biological effect is complex. First, one must calculate absorbed dose $D_T$, the amount of energy from ionizing radiation actually deposited in tissue.

\begin{align}
D_T = \frac{\Delta \epsilon}{\Delta m}\\
\Delta \epsilon = \mbox{ energy deposited }\\
\Delta m = \mbox{ mass of tissue }
\end{align}

```python
def absorbed_dose(del_eps, del_m):
    return del_eps/del_m
```

The biological effect is related to the dose and depends on the nature of the
radiation.

The ICRP has suggested a quantifying factor that translates from energy to dose.

- (1991). "1990 Recommendations of the International Commission on Radiological Protection". Annals of the ICRP 21 (1-3). Retrieved on 17 May 2012.
- (2007). "The 2007 Recommendations of the International Commission on Radiological Protection". Annals of the ICRP 37 (2-4). Retrieved on 17 May 2012.

| Radiation | Radiation Energy Weighting Factor WR (formerly Q)            |
|:-----------:|:-----------------------------------------------------------:|
| x-rays, gammas, betas, muons | $1$ |
| neutrons (< 1 MeV)                  | $2.5 + 18.2\cdot e-\frac{[ln(E)]^2}{6}$ |
| neutrons (1 - 50 MeV) | $5.0 + 17.0\cdot e-\frac{[ln(2\cdot E)]^2}{6}$ |
| neutrons (> 50 MeV)      | $2.5 + 3.25\cdot e-\frac{[ln(0.04\cdot E)]^2}{6}$  |
| protons, charged pions | 2 |
| alphas, fission products, heavy nuclei | 20 |

Epidemiological studies vary on the factors for both dose conversion and risk from dose from various sources. There is a nice recent study on what we know. [http://www.ncbi.nlm.nih.gov/pubmed/27334644](http://www.ncbi.nlm.nih.gov/pubmed/27334644)

### Discussion: What are the key takeaways from this review of Underground Mining ?

- .Radon gas is a big issue, so you have to be careful if you're a miner.
- .Have a very dense lung...
- .wear a mask
- .keep in mind of the countries with thier underground mining and resources.
- WL & WLM
- you can calculate the energy of the alphas

## In Situ Leaching

￼

From [WNA](http://www.world-nuclear.org/information-library/nuclear-fuel-cycle/mining-of-uranium/in-situ-leach-mining-of-uranium.aspx):

> In 2013, 47% of world uranium mined was from ISL operations. Most uranium mining in the USA, Kazakhstan and Uzbekistan is now by in situ leach methods, also known as in situ recovery (ISR).

### Discussion: What are some benefits and drawbacks of in situ leaching?

### Uranium in Seawater

Great presentation by expert professor Erich Schneider. [http://www-pub.iaea.org/iaeameetings/cn216pn/Thursday/Session13/180-Schneider.pdf](http://www-pub.iaea.org/iaeameetings/cn216pn/Thursday/Session13/180-Schneider.pdf) .

---

---

---

---

\begin{align}
y &= \frac{\beta_{max} t}{K_D + t}\\
y &= \mbox{uranium uptake} \left[\frac{g_U}{kg_{ads}}\right]\\
\beta_{max} &= \mbox{saturation capacity } \left[\frac{g_U}{kg_{ads}}\right]\\
K_D &= \mbox{half saturation time }[d]\\
t &= \mbox{exposure time }[d]\\
\end{align}

```python
def uptake(beta_max, k_d, t):
    return beta_max*t/(k_d+t)

beta_max = 48.9
k_d = 28.0

import numpy as np
y = np.arange(0.0,100.0)

for t in range(0, 100):
    y[t] = uptake(beta_max, k_d, t)

plt.plot(y)
plt.ylabel(r'Uranium uptake $\left[\frac{g_U}{kg_{ads}}\right]$', fontsize=20)
plt.xlabel(r'Time $\left[d\right]$', fontsize=20)

```

## Thorium Resources

[Source: http://minerals.usgs.gov/minerals/pubs/commodity/thorium/690397.pdf](http://minerals.usgs.gov/minerals/pubs/commodity/thorium/690397.pdf): Where's Turkey?

This reading from the black monazite rich beach in Brazil, is in units of $\frac{\mu Sv}{h}$. Typical background radiation is around $0.3\frac{\mu Sv}{h}$.

## Milling

Milling is the process of concentrating uranium ore into a substance known as yellow cake, concentrated $U_3O_8$.

For open pit or underground mining, the first step is to crush the material. The second step is to dissolve out the uranium. This is not a pretty industrial process.

However, for ISL, the material is already in liquid form (because the leaching chemistry already occurred underground.)

ISL Ion Exchange uses resin beads to separate the $U_3O_8$.

As an aside, united states mining & milling has declined.

## Wrap-up

- There is a great deal of both uranium and thorium available, depending on the price.
- Certain nations have a great deal more resources than other nations (see: Canada, Khazakstan, Australia)
- Prospecting uses biology, geology, radiology, and electromagnetism to identify uranium and thorium resources
- Biological prospecting relies on indicator plants which seek out areas rich in indicative minerals
- Aerial prospecting relies on gamma spectrometers and magnetometers
- Surface prostpecting relies on handheld detectors
- Well logging maps the subsurface vertically with detection in wells
- Uranium can be extracted from the ground with open pit mines, underground mines, in situ leaching, and even seawater extraction.
- Prospecting for mineral resources and the ability to extract those resources respond to market price.
- The Radon decay chain is responsible for radiation dangers to mine workers and is measured in Working Level or Working Level Months.
- The energy of radon daughter alphas can be calculated for each decay.
- Radiation effects are expressed in many units and the weighting factors related to biological impacts are approximate and controversial.
- Milling requires separation of the uranium via ion exchange with carbonates or acids.
- The milling process associated with in situ leaching conducts chemistry underground and accordingly avoids tailings piles.
- Concentrate, yellow, $U_3O_8$ is the results of milling operations and is known as 'yellowcake'.

## References

This section was developed to complement Chapter 2 of [1]. You can find a video at [2].

[1] N. Tsoulfanidis, The Nuclear Fuel Cycle. La Grange Park, Illinois, USA: American Nuclear Society, 2013.

[2] The Heritage Foundation. Powering America: Uranium Mining and Milling. https://www.youtube.com/watch?v=oT2LHGG-9Ko

```python

```


# PHY321: Classical Mechanics 1

**Solution Homework 7, due to March 22**

Date: **Mar 24, 2021**

### Introduction to homework 7

In this week's homework we will apply our insights about harmonic
oscillations and link this with our activity from the lecture on
Friday March 12. The relevant material to survey is chapter 5 of Taylor.

We have also added an exercise (exercise 2) related to our discussion of two-body problems.
The relevant reading background for exercise 2 is sections 8.1-8.2 of Taylor.

### Exercise 1 (80 pt), particle/object in a harmonic oscillator  potential

In the midterm and in exercise 4 of homework 6, we looked at an
object/particle moving in a potential which resulted in harmonic
oscillations.  The aim here is to summarize in more detail the
material from harmonic oscillations. See also the bonus exercise below
here (from the discussions of Friday March 12).

Relevant reading here is Taylor chapter 5 and the lecture notes on oscillations.

We will consider a particle of mass $m$ moving in a one-dimensional potential,

$$
V(x)=k\frac{x^2}{2},
$$

where $k$ is a parameter.

We will limit ourselves to a one-dimensional system. You will need to select values for the initial conditions and the various parameters $k$, $m$, $b$, $\omega$ and $F_0$ discussed here.

* 1a (20pt)  Assume no driving force first and add a drag force $-bv$, where $v$ is the velocity. Find the forces acting on the object. Find the analytical solutions to the equations of motion. Discuss the three cases: **underdamping**, **critical damping** and **overdamping**.

The text here is taken from the lecture notes of week 9. We have made this text more extensive than needed. This is done for the sake of completeness.

We consider only the case where the damping force is proportional to
the velocity. This is counter to dragging friction, where the force is
proportional in strength to the normal force and independent of
velocity, and is also inconsistent with wind resistance, where the
magnitude of the drag force is proportional the square of the
velocity. Rolling resistance does seem to be mainly proportional to
the velocity. However, the main motivation for considering damping
forces proportional to the velocity is that the math is more
friendly. This is because the differential equation is linear,
i.e. each term is of order $x$, $\dot{x}$, $\ddot{x}\cdots$, or even
terms with no mention of $x$, and there are no terms such as $x^2$ or
$x\ddot{x}$. The equations of motion for a spring with damping force
$-b\dot{x}$ are

$$
\begin{equation}
m\ddot{x}+b\dot{x}+kx=0.
\label{_auto1} \tag{1}
\end{equation}
$$

Just to make the solution a bit less messy, we rewrite this equation as

$$
\begin{equation}
\label{eq:dampeddiffyq} \tag{2}
\ddot{x}+2\beta\dot{x}+\omega_0^2x=0,~~~~\beta\equiv b/2m,~\omega_0\equiv\sqrt{k/m}.
\end{equation}
$$

Both $\beta$ and $\omega$ have dimensions of inverse time. To find solutions (see appendix C in the text) you must make an educated guess at the form of the solution. To do this, first realize that the solution will need an arbitrary normalization $A$ because the equation is linear. Secondly, realize that if the form is

$$
\begin{equation}
x=Ae^{rt}
\label{_auto2} \tag{3}
\end{equation}
$$

that each derivative simply brings out an extra power of $r$. This
means that the $Ae^{rt}$ factors out and one can simply solve for an
equation for $r$. Plugging this form into Eq. ([2](#eq:dampeddiffyq)),

$$
\begin{equation}
r^2+2\beta r+\omega_0^2=0.
\label{_auto3} \tag{4}
\end{equation}
$$

Because this is a quadratic equation there will be two solutions,

$$
\begin{equation}
r=-\beta\pm\sqrt{\beta^2-\omega_0^2}.
\label{_auto4} \tag{5}
\end{equation}
$$

We refer to the two solutions as $r_1$ and $r_2$ corresponding to the
$+$ and $-$ roots. As expected, there should be two arbitrary
constants involved in the solution,

$$
\begin{equation}
x=A_1e^{r_1t}+A_2e^{r_2t},
\label{_auto5} \tag{6}
\end{equation}
$$

where the coefficients $A_1$ and $A_2$ are determined by initial
conditions.

The roots listed above, $\sqrt{\omega_0^2-\beta_0^2}$, will be
imaginary if the damping is small and $\beta<\omega_0$. In that case,
$r$ is complex and the factor $\exp{(rt)}$ will have some oscillatory
behavior. If the roots are real, there will only be exponentially
decaying solutions. There are three cases:

### Underdamped: $\beta<\omega_0$

$$
\begin{eqnarray}
x&=&A_1e^{-\beta t}e^{i\omega't}+A_2e^{-\beta t}e^{-i\omega't},~~\omega'\equiv\sqrt{\omega_0^2-\beta^2}\\
\nonumber
&=&(A_1+A_2)e^{-\beta t}\cos\omega't+i(A_1-A_2)e^{-\beta t}\sin\omega't.
\end{eqnarray}
$$

Here we have made use of the identity
$e^{i\omega't}=\cos\omega't+i\sin\omega't$. Because the constants are
arbitrary, and because the real and imaginary parts are both solutions
individually, we can simply consider the real part of the solution
alone:

$$
\begin{eqnarray}
\label{eq:homogsolution} \tag{7}
x&=&B_1e^{-\beta t}\cos\omega't+B_2e^{-\beta t}\sin\omega't,\\
\nonumber
\omega'&\equiv&\sqrt{\omega_0^2-\beta^2}.
\end{eqnarray}
$$

### Critical dampling: $\beta=\omega_0$

In this case the two terms involving $r_1$ and $r_2$ are identical
because $\omega'=0$. Because we need to arbitrary constants, there
needs to be another solution. This is found by simply guessing, or by
taking the limit of $\omega'\rightarrow 0$ from the underdamped
solution. The solution is then

$$
\begin{equation}
\label{eq:criticallydamped} \tag{8}
x=Ae^{-\beta t}+Bte^{-\beta t}.
\end{equation}
$$

The critically damped solution is interesting because the solution
approaches zero quickly, but does not oscillate. For a problem with
zero initial velocity, the solution never crosses zero. This is a good
choice for designing shock absorbers or swinging doors.

### Overdamped: $\beta>\omega_0$

$$
\begin{eqnarray}
x&=&A_1\exp{-(\beta+\sqrt{\beta^2-\omega_0^2})t}+A_2\exp{-(\beta-\sqrt{\beta^2-\omega_0^2})t}
\end{eqnarray}
$$

This solution will also never pass the origin more than once, and then
only if the initial velocity is strong and initially toward zero.

Given $b$, $m$ and $\omega_0$, find $x(t)$ for a particle whose
initial position is $x=0$ and has initial velocity $v_0$ (assuming an
underdamped solution).

The solution is of the form,

$$
\begin{eqnarray*}
x&=&e^{-\beta t}\left[A_1\cos(\omega' t)+A_2\sin\omega't\right],\\
\dot{x}&=&-\beta x+\omega'e^{-\beta t}\left[-A_1\sin\omega't+A_2\cos\omega't\right].\\
\omega'&\equiv&\sqrt{\omega_0^2-\beta^2},~~~\beta\equiv b/2m.
\end{eqnarray*}
$$

From the initial conditions, $A_1=0$ because $x(0)=0$ and $\omega'A_2=v_0$. So

$$
x=\frac{v_0}{\omega'}e^{-\beta t}\sin\omega't.
$$

* 1b (5pt) Scale your equations of motion in terms of a dimensionless time $\tau = \omega_0 t$, where $t$ is time and $\omega_0^2=k/m$ is the so-called natural frequency.

To scale the equations we start again with the full equation

$$
\begin{equation}
m\ddot{x}+b\dot{x}+kx=0.
\label{_auto6} \tag{9}
\end{equation}
$$

We divide by $m$ and get

$$
\begin{equation}
\ddot{x}+\frac{b}{m}\dot{x}+\frac{k}{m}x=0.
\label{_auto7} \tag{10}
\end{equation}
$$

Defining the natural frequency $\omega_0^2=k/m$   we introduce a dimensionless time $\tau = \omega_0 t$ and replace $t$ with $\tau$.
This leads to us rewriting

$$
\begin{equation}
\frac{d^2 x}{dt^2}+\frac{b}{m}\frac{dx}{dt}+\frac{k}{m}x=0,
\label{_auto8} \tag{11}
\end{equation}
$$

as

$$
\begin{equation}
\omega_0^2\frac{d^2 x}{d\tau^2}+\frac{\omega_0b}{m}\frac{dx}{d\tau}+\omega_0^2x=0,
\label{_auto9} \tag{12}
\end{equation}
$$

and dividing by $\omega_0^2$ and defining $\gamma = b/(2m\omega_0)$ we have the final scaled equation

$$
\begin{equation}
\frac{d^2 x}{d\tau^2}+2\gamma\frac{dx}{d\tau}+x=0.
\label{_auto10} \tag{13}
\end{equation}
$$

This equation has dimension length only and time $\tau$ is dimensionless.  It means also that our solutions become now

In this case the variable $r$ becomes

$$
\begin{equation}
r=-\gamma\pm\sqrt{\gamma^2-1}.
\label{_auto11} \tag{14}
\end{equation}
$$

We refer to the two solutions as $r_1$ and $r_2$ corresponding to the
$+$ and $-$ roots. As expected, there should be two arbitrary
constants involved in the solution,

$$
\begin{equation}
x=A_1e^{r_1\tau}+A_2e^{r_2\tau},
\label{_auto12} \tag{15}
\end{equation}
$$

where the coefficients $A_1$ and $A_2$ are determined by initial
conditions.

For the roots listed above, $\sqrt{\gamma^2-1}$, will be
imaginary if the damping is small and $\gamma < 1$. In that case,
$r$ is complex and the factor $\exp{(rt)}$ will have some oscillatory
behavior. If the roots are real, there will only be exponentially
decaying solutions. For $\gamma =1$, we have what we labeled as critical damping while for $\gamma > 1$, we have over-critical damping.

In the codes below, we have implemented the dimensionless  equations.

* 1c (25pt) You can use your codes from either the first midterm or from homeworks 5 and/or 6.  Study numerically the three cases from 1a, that is the underdamped motion, the critically damped one and finally the overdamped one. Compare your numerical results with the analytical ones from 1a. Discuss your results. You can use the Euler-Cromer method, or the Velocity-Verlet method or the Runge-Kutta methods discussed during the lectures, see for example <https://mhjensen.github.io/Physics321/doc/pub/week10/html/week10.html>. Alternatively, you could use the **odeint** solvers included functionality in Python, see <https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.odeint.html>. Give a short argument about the numerical algorithm you ended up using.

We have chosen to implement the Runge-Kutta4 method since this has a truncation error in terms of the step size $\Delta t$ to the power of five. The codes are included after part 1d.

* 1d (30pt) We add now a driving force $F=F_0\cos{(\omega t}$. Find the particular solution and its analytical solution. Include this force in your code (remember to scale the equations) and compare your numerical results with the analytical results. Discuss your results. How does the system evolve over time with a given frequency $\omega$ for the driving force?   Is energy conserved? If not, why?

To find a particular solution, one first guesses at the form,

$$
\begin{equation}
\label{eq:partform} \tag{16}
x_p(t)=D\cos(\omega t-\delta),
\end{equation}
$$

and rewrite the differential equation as

$$
\begin{equation}
D\left\{-\omega^2\cos(\omega t-\delta)-2\beta\omega\sin(\omega t-\delta)+\omega_0^2\cos(\omega t-\delta)\right\}=\frac{F_0}{m}\cos(\omega t).
\label{_auto13} \tag{17}
\end{equation}
$$

One can now use angle addition formulas to get

$$
\begin{eqnarray}
D\left\{(-\omega^2\cos\delta+2\beta\omega\sin\delta+\omega_0^2\cos\delta)\cos(\omega t)\right.&&\\
\nonumber
\left.+(-\omega^2\sin\delta-2\beta\omega\cos\delta+\omega_0^2\sin\delta)\sin(\omega t)\right\}
&=&\frac{F_0}{m}\cos(\omega t).
\end{eqnarray}
$$

Both the $\cos$ and $\sin$ terms need to equate if the expression is to hold at all times. Thus, this becomes two equations

$$
\begin{eqnarray}
D\left\{-\omega^2\cos\delta+2\beta\omega\sin\delta+\omega_0^2\cos\delta\right\}&=&\frac{F_0}{m}\\
\nonumber
-\omega^2\sin\delta-2\beta\omega\cos\delta+\omega_0^2\sin\delta&=&0.
\end{eqnarray}
$$

After dividing by $\cos\delta$, the lower expression leads to

$$
\begin{equation}
\tan\delta=\frac{2\beta\omega}{\omega_0^2-\omega^2}.
\label{_auto14} \tag{18}
\end{equation}
$$

Using the identities $\tan^2+1=\csc^2$ and $\sin^2+\cos^2=1$, one can also express $\sin\delta$ and $\cos\delta$,

$$
\begin{eqnarray}
\sin\delta&=&\frac{2\beta\omega}{\sqrt{(\omega_0^2-\omega^2)^2+4\omega^2\beta^2}},\\
\nonumber
\cos\delta&=&\frac{(\omega_0^2-\omega^2)}{\sqrt{(\omega_0^2-\omega^2)^2+4\omega^2\beta^2}}
\end{eqnarray}
$$

Inserting the expressions for $\cos\delta$ and $\sin\delta$ into the expression for $D$,

$$
\begin{equation}
\label{eq:Ddrive} \tag{19}
D=\frac{F_0/m}{\sqrt{(\omega_0^2-\omega^2)^2+4\omega^2\beta^2}}.
\end{equation}
$$

For a given initial condition, e.g. initial displacement and velocity,
one must add the homogenous solution then solve for the two arbitrary
constants. However, because the homogenous solutions decay with time
as $e^{-\beta t}$, the particular solution is all that remains at
large times, and is therefore the steady state solution. Because the
arbitrary constants are all in the homogenous solution, all memory of
the initial conditions are lost at large times, $t>>1/\beta$.

The amplitude of the motion, $D$, is linearly proportional to the
driving force ($F_0/m$), but also depends on the driving frequency
$\omega$. For small $\beta$ the maximum will occur at
$\omega=\omega_0$. This is referred to as a resonance. In the limit
$\beta\rightarrow 0$ the amplitude at resonance approaches infinity.

Let us now for simplicty assume that our external force is given by

$$
F_{\mathrm{ext}}(t) = F_0\cos{(\omega t)},
$$

where $F_0$ is a constant (what is its dimension?) and $\omega$ is the frequency of the applied external driving force.
**Small question:** would you expect energy to be conserved now?

Introducing the external force into our lovely differential equation
and dividing by $m$ and introducing $\omega_0^2=\sqrt{k/m}$ we have

$$
\frac{d^2x}{dt^2} + \frac{b}{m}\frac{dx}{dt}+\omega_0^2x(t) =\frac{F_0}{m}\cos{(\omega t)},
$$

Thereafter we introduce a dimensionless time $\tau = t\omega_0$
and a dimensionless frequency $\tilde{\omega}=\omega/\omega_0$. We have then

$$
\frac{d^2x}{d\tau^2} + \frac{b}{m\omega_0}\frac{dx}{d\tau}+x(\tau) =\frac{F_0}{m\omega_0^2}\cos{(\tilde{\omega}\tau)},
$$

Introducing a new amplitude $\tilde{F} =F_0/(m\omega_0^2)$ (check dimensionality again) we have

$$
\frac{d^2x}{d\tau^2} + \frac{b}{m\omega_0}\frac{dx}{d\tau}+x(\tau) =\tilde{F}\cos{(\tilde{\omega}\tau)}.
$$

Our final step, as we did in the case of various types of damping, is
to define $\gamma = b/(2m\omega_0)$ and rewrite our equations as

$$
\frac{d^2x}{d\tau^2} + 2\gamma\frac{dx}{d\tau}+x(\tau) =\tilde{F}\cos{(\tilde{\omega}\tau)}.
$$

Eenergy is not conserved since we have a time and velocity dependent total net force acting on the system.

Note that here the forward Euler method does not know the specific force function to be called.
It receives just an input the name. We can easily change the force by adding another function.

```python
def ForwardEuler(v,x,t,n,Force):
    for i in range(n-1):
        v[i+1] = v[i] + DeltaT*Force(v[i],x[i],t[i])
        x[i+1] = x[i] + DeltaT*v[i]
        t[i+1] = t[i] + DeltaT
```

```python
def SpringForce(v,x,t):
#   note here that we have divided by mass and we return the acceleration
    return  -2*gamma*v-x+Ftilde*cos(t*Omegatilde)
```

It is easy to add a new method like the Euler-Cromer

```python
def ForwardEulerCromer(v,x,t,n,Force):
    for i in range(n-1):
        a = Force(v[i],x[i],t[i])
        v[i+1] = v[i] + DeltaT*a
        x[i+1] = x[i] + DeltaT*v[i+1]
        t[i+1] = t[i] + DeltaT
```

and the Velocity Verlet method (be careful with time-dependence here, it is not an ideal method for non-conservative forces))

```python
def VelocityVerlet(v,x,t,n,Force):
    for i in range(n-1):
        a = Force(v[i],x[i],t[i])
        x[i+1] = x[i] + DeltaT*v[i]+0.5*a*DeltaT*DeltaT
        anew = Force(v[i],x[i+1],t[i+1])
        v[i+1] = v[i] + 0.5*DeltaT*(a+anew)
        t[i+1] = t[i] + DeltaT
```

Finally, we can now add the Runge-Kutta2 method via a new function

```python
def RK2(v,x,t,n,Force):
    for i in range(n-1):
# Setting up k1
        k1x = DeltaT*v[i]
        k1v = DeltaT*Force(v[i],x[i],t[i])
# Setting up k2
        vv = v[i]+k1v*0.5
        xx = x[i]+k1x*0.5
        k2x = DeltaT*vv
        k2v = DeltaT*Force(vv,xx,t[i]+DeltaT*0.5)
# Final result
        x[i+1] = x[i]+k2x
        v[i+1] = v[i]+k2v
	    t[i+1] = t[i]+DeltaT
```

Finally, we can now add the Runge-Kutta2 method via a new function

```python
def RK4(v,x,t,n,Force):
    for i in range(n-1):
# Setting up k1
        k1x = DeltaT*v[i]
        k1v = DeltaT*Force(v[i],x[i],t[i])
# Setting up k2
        vv = v[i]+k1v*0.5
        xx = x[i]+k1x*0.5
        k2x = DeltaT*vv
        k2v = DeltaT*Force(vv,xx,t[i]+DeltaT*0.5)
# Setting up k3
        vv = v[i]+k2v*0.5
        xx = x[i]+k2x*0.5
        k3x = DeltaT*vv
        k3v = DeltaT*Force(vv,xx,t[i]+DeltaT*0.5)
# Setting up k4
        vv = v[i]+k3v
        xx = x[i]+k3x
        k4x = DeltaT*vv
        k4v = DeltaT*Force(vv,xx,t[i]+DeltaT)
# Final result
        x[i+1] = x[i]+(k1x+2*k2x+2*k3x+k4x)/6.
        v[i+1] = v[i]+(k1v+2*k2v+2*k3v+k4v)/6.
        t[i+1] = t[i] + DeltaT
```

The code below uses the Runge-Kutta4 methods.

```python
%matplotlib inline

# Common imports
import numpy as np
import pandas as pd
from math import *
import matplotlib.pyplot as plt
import os

# Where to save the figures and data files
PROJECT_ROOT_DIR = "Results"
FIGURE_ID = "Results/FigureFiles"
DATA_ID = "DataFiles/"

if not os.path.exists(PROJECT_ROOT_DIR):
    os.mkdir(PROJECT_ROOT_DIR)

if not os.path.exists(FIGURE_ID):
    os.makedirs(FIGURE_ID)

if not os.path.exists(DATA_ID):
    os.makedirs(DATA_ID)

def image_path(fig_id):
    return os.path.join(FIGURE_ID, fig_id)

def data_path(dat_id):
    return os.path.join(DATA_ID, dat_id)

def save_fig(fig_id):
    plt.savefig(image_path(fig_id) + ".png", format='png')

DeltaT = 0.001
#set up arrays
tfinal = 20 # in dimensionless time
n = ceil(tfinal/DeltaT)
# set up arrays for t, v, and x
t = np.zeros(n)
v = np.zeros(n)
x = np.zeros(n)
# Initial conditions (can change to more than one dim)
x0 =  1.0
v0 = 0.0
x[0] = x0
v[0] = v0
gamma = 0.2
Omegatilde = 0.5
Ftilde = 1.0
# Start integrating using Euler's method
# Note that we define the force function as a SpringForce
RK4(v,x,t,n,SpringForce)

# Plot position as function of time
fig, ax = plt.subplots()
ax.set_ylabel('x[m]')
ax.set_xlabel('t[s]')
ax.plot(t, x)
fig.tight_layout()
save_fig("ForcedBlockRK4")
plt.show()
```

### Exercise 2 (20pt), Center-of-Mass and Relative Coordinates and Reference Frames

We define the two-body center-of-mass coordinate and relative coordinate by expressing the trajectories for
$\boldsymbol{r}_1$ and $\boldsymbol{r}_2$ into the center-of-mass coordinate
$\boldsymbol{R}_{\rm cm}$

$$
\boldsymbol{R}_{\rm cm}\equiv\frac{m_1\boldsymbol{r}_1+m_2\boldsymbol{r}_2}{m_1+m_2},
$$

and the relative coordinate

$$
\boldsymbol{r}\equiv\boldsymbol{r}_1-\boldsymbol{r_2}.
$$

Here, we assume the two particles interact only with one another, so $\boldsymbol{F}_{12}=-\boldsymbol{F}_{21}$ (where $\boldsymbol{F}_{ij}$ is the force on $i$ due to $j$.

* 2a (5pt) Show that the equations of motion then become $\ddot{\boldsymbol{R}}_{\rm cm}=0$ and $\mu\ddot{\boldsymbol{r}}=\boldsymbol{F}_{12}$, with the reduced mass $\mu=m_1m_2/(m_1+m_2)$.

The first expression simply states that the center of mass coordinate $\boldsymbol{R}_{\rm cm}$ moves at a fixed velocity. The second expression can be rewritten in terms of the reduced mass $\mu$.

* 2b (5pt) Show that the linear momenta for the center-of-mass $\boldsymbol{P}$ motion and the relative motion $\boldsymbol{q}$ are given by $\boldsymbol{P}=M\dot{\boldsymbol{R}}_{\rm cm}$ with $M=m_1+m_2$ and $\boldsymbol{q}=\mu\dot{\boldsymbol{r}}$.  The linear momentum of the relative motion is defined $\boldsymbol{q} = (m_2\boldsymbol{p}_1-m_1\boldsymbol{p}_2)/(m_1+m_2)$.

* 2c (5pt) Show then that the  kinetic energy for two objects can then be written as

$$
K=\frac{P^2}{2M}+\frac{q^2}{2\mu}.
$$

* 2d (5pt) Show that the total angular momentum for two-particles in the center-of-mass frame $\boldsymbol{R}=0$, is given by

$$
\boldsymbol{L}=\boldsymbol{r}\times \mu\dot{\boldsymbol{r}}.
$$


```python
from IPython.display import Image
Image('../../Python_probability_statistics_machine_learning_2E.png',width=200)
```

# Conditional Expectation as Projection

Now that we understand projection
methods geometrically, we can apply
them to conditional probability. This is the
*key* concept that ties
probability to geometry, optimization, and linear
algebra.

### Inner Product for Random Variables

 From our previous work on
projection for vectors in
$\mathbb{R}^n$, we have a good geometric grasp on how
projection is related to
Minimum Mean Squared Error (MMSE). By one abstract
step, we can carry
all of our geometric interpretations to the space of random
variables.
For example, we previously noted that at the point of projection, we
had the
following orthogonal (i.e.,  perpendicular vectors) condition,

$$
( \mathbf{y} - \mathbf{v}_{opt} )^T \mathbf{v} = 0
$$

 which by noting the inner product slightly more abstractly as
$\langle\mathbf{x},\mathbf{y} \rangle = \mathbf{x}^T \mathbf{y}$, we can
express
as

$$
\langle \mathbf{y} - \mathbf{v}_{opt},\mathbf{v} \rangle = 0
$$

 and by defining the inner product for the random variables
$X$ and $Y$ as

$$
\langle X,Y \rangle = \mathbb{E}(X Y)
$$

 we have the same relationship:

$$
\langle X-h_{opt}(Y),Y \rangle = 0
$$

 which holds not for vectors in $\mathbb{R}^n$, but for random
variables $X$ and
$Y$ and functions of those random variables. Exactly why this
is true is
technical, but it turns out that one can build up the *entire theory
of
probability* this way [[edward1987radically]](#edward1987radically), by using
the expectation as
an inner product.

Furthermore, by abstracting out the inner
product concept, we have connected
minimum-mean-squared-error (MMSE)
optimization problems, geometry, and random
variables.  That's  a lot of mileage
to get a out of an abstraction and it
enables us to shift between these
interpretations to address real problems.
Soon, we'll do this with some
examples, but first we collect the most important
result that flows naturally
from this abstraction.

### Conditional Expectation as Projection

The
conditional expectation is the minimum mean squared error (MMSE) solution
to the
following problem [^proof]:

$$ \min_h \int_{\mathbb{R}^2} (x - h(y) )^2 f_{X,Y}(x,y) dx dy $$

with the minimizing $h_{opt}(Y) $ as

$$
h_{opt}(Y) = \mathbb{E}(X|Y)
$$

[^proof]: See appendix for proof using the Cauchy-Schwarz inequality.

 which is
another way of saying that among all possible functions
$h(Y)$, the one that
minimizes the MSE is $ \mathbb{E}(X|Y)$. From our previous discussion on
projection, we noted that
these MMSE solutions can be thought of as projections
onto a subspace that
characterizes $Y$. For example, we previously noted that at
the point of
projection, we have perpendicular terms,

$$
\begin{equation}
\langle X-h_{opt}(Y),Y \rangle = 0
\end{equation}
\label{eq:ortho} \tag{1}
$$

 but since we know that the MMSE solution

$$
h_{opt}(Y) = \mathbb{E}(X|Y)
$$

 we have by direct substitution,

$$
\begin{equation}
\mathbb{E}(X-\mathbb{E}(X|Y),Y) = 0
\end{equation}
\label{eq:ortho_001} \tag{2}
$$

 That last step seems pretty innocuous, but it ties MMSE to
conditional
expectation to the inner project abstraction, and in so doing,
reveals the
conditional expectation to be a projection operator for random
variables. Before
we develop this further, let's grab some quick dividends.
From the previous
equation, by linearity of the expectation, we obtain,

$$
\mathbb{E}(X Y) =  \mathbb{E}(Y \mathbb{E}(X|Y))
$$

 which is the so-called *tower property* of the expectation. Note that
we could
have found this by using the formal definition of conditional
expectation,

$$
\mathbb{E}(X|Y) = \int_{\mathbb{R}^2} x \frac{f_{X,Y}(x,y)}{f_Y(y)} dx dy
$$

 and brute-force direct integration,

$$
\begin{align*}
\mathbb{E}(Y \mathbb{E}(X|Y)) &= \int_{\mathbb{R}} y
\int_{\mathbb{R}} x \frac{f_{X,Y}(x,y)}{f_Y(y)}  f_Y(y) dx dy \\\
&=\int_{\mathbb{R}^2} x y f_{X,Y}(x,y) dx dy \\\
&=\mathbb{E}( X Y)
\end{align*}
$$

 which is not very geometrically intuitive. This lack of geometric
intuition
makes it hard to apply these concepts and keep track of these
relationships.
We can keep pursuing this analogy and obtain the length of the error term
from
the orthogonality property of the MMSE solution as,

$$
\langle X-h_{opt}(Y),X-h_{opt}(Y)\rangle = \langle X,X  \rangle - \langle
h_{opt}(Y),h_{opt}(Y)  \rangle
$$

 and then by substituting all the notation we obtain

$$
\mathbb{E}(X-  \mathbb{E}(X|Y))^2 = \mathbb{E}(X)^2 -
\mathbb{E}(\mathbb{E}(X|Y) )^2
$$

 which would be tough to compute by direct integration.

To formally establish
that $\mathbb{E}(X|Y)$ *is* in fact *a projection operator* we
need to show
idempotency.  Recall that idempotency means that once we project
something onto
a subspace, further projections do nothing. In the space of
random variables,
$\mathbb{E}(X|\cdot$) is the idempotent projection as we can
show by noting that

$$
h_{opt} = \mathbb{E}(X|Y)
$$

 is purely a function of $Y$, so that

$$
\mathbb{E}(h_{opt}(Y)|Y) = h_{opt}(Y)
$$

 because $Y$ is fixed, this verifies idempotency. Thus, conditional
expectation
is the corresponding projection operator for random variables. We
can continue
to carry over our geometric interpretations of projections for
vectors
($\mathbf{v}$) into random variables ($X$).  With this important
result, let's
consider some examples of conditional expectations obtained by
using brute force
to find the optimal MMSE function $h_{opt}$ as well as by
using our new
perspective on conditional expectation.

**Example.** Suppose we have a random
variable, $X$, then what constant is closest to $X$ in
the sense of the mean-
squared-error (MSE)? In other words, which $c \in
\mathbb{R}$ minimizes the
following mean squared error:

$$
\mbox{MSE} = \mathbb{E}( X - c )^2
$$

 we can work this out many ways. First, using calculus-based optimization,

$$
\mathbb{E}(X-c)^2=\mathbb{E}(c^2-2 c X + X^2)=c^2-2 c \mathbb{E}(X) +
\mathbb{E}(X^2)
$$

 and then take the first derivative with respect to $c$ and solve:

$$
c_{opt}=\mathbb{E}(X)
$$

 Remember that $X$ may potentially take on many values, but this says
that the
closest number to $X$ in the MSE sense is $\mathbb{E}(X)$.  This is
intuitively
pleasing.  Coming at this same problem using our inner product,
from Equation
[2](#eq:ortho_001) we know that at the point of projection

$$
\mathbb{E}((X-c_{opt}) 1) = 0
$$

  where the $1$ represents the space of constants
we are projecting onto. By
linearity of the expectation, gives

$$
c_{opt}=\mathbb{E}(X)
$$

 Using the projection approach, because $\mathbb{E}(X|Y)$ is
the projection
operator, with $Y=\Omega$ (the entire underlying
probability space), we have,
using the definition of conditional
expectation:

$$
\mathbb{E}(X|Y=\Omega) = \mathbb{E}(X)
$$

 This is because of the subtle fact that a random variable over the entire
$\Omega$ space can only be a constant.  Thus, we just worked the same problem
three ways (optimization, orthogonal inner products, projection).

**Example.**
Let's consider the following example with probability density
$f_{X,Y}= x + y $
where $(x,y) \in [0,1]^2$ and compute the conditional
expectation straight from
the definition:

$$
\mathbb{ E}(X|Y) = \int_0^1 x \frac{f_{X,Y}(x,y)}{f_Y(y)} dx=  \int_0^1 x
\frac{x+y}{y+1/2} dx =\frac{3 y + 2}{6 y + 3}
$$

 That was pretty easy because the density function was so simple. Now,
let's do
it the hard way by going directly for the MMSE solution $h(Y)$. Then,

$$
\begin{align*}
\mbox{ MSE } &= \underset{h}\min \int_0^1\int_0^1 (x - h(y)
)^2 f_{X,Y}(x,y)dx dy \\\
             &= \underset{h}\min \int_0^1 y h^2 {\left
(y \right )} - y h{\left (y \right )} + \frac{1}{3} y + \frac{1}{2} h^{2}{\left
(y \right )} - \frac{2}{3} h{\left (y \right )} + \frac{1}{4} dy
\end{align*}
$$

 Now we have to find a function $h$ that is going to minimize this.
Solving for
a function, as opposed to solving for a number, is generally very,
very hard,
but because we are integrating over a finite interval, we can use
the Euler-
Lagrange method from variational calculus to take the derivative of
the
integrand with respect to the function $h(y)$ and set it to zero. Using
Euler-
Lagrange methods, we obtain the following result,

$$
2 y h{\left (y \right )} - y + h{\left (y \right )} - \frac{2}{3} =0
$$

 Solving this gives

$$
h_{opt}(y)= \frac{3 y + 2}{6 y + 3}
$$

  which is what we obtained before. Finally, we can solve this
using our inner
product in Equation [1](#eq:ortho) as

$$
\mathbb{E}((X-h(Y)) Y)=0
$$

  Writing this out gives,

$$
\int_0^1\int_0^1 (x-h(y))y(x+y) dx dy = \int_0^1\frac{1}{6}y(-3(2 y+1) h(y)+3
y+2) dy=0
$$

 and the integrand must be zero,

$$
2 y + 3 y^2 - 3 y h(y) - 6 y^2 h(y)=0
$$

 and solving this for $h(y)$ gives the same solution:

$$
h_{opt}(y)= \frac{3 y + 2}{6 y + 3}
$$

 Thus, doing it by the brute force integration from the definition,
optimization, or inner product gives us the same answer; but, in general, no
method is necessarily easiest because they both involve potentially difficult
or
impossible integration, optimization, or functional equation solving.  The
point
is that now that we have a deep toolbox, we can pick and choose which
tools we
want to apply for different problems.

Before we leave this example, let's use
Sympy to verify the length of the error
function we found earlier for this
example:

$$
\mathbb{E}(X-\mathbb{E}(X|Y))^2=\mathbb{E}(X)^2-\mathbb{E}(\mathbb{E}(X|Y))^2
$$

 that is based on the Pythagorean theorem. First, we
need to compute the
marginal densities,

```python
from sympy.abc import y,x
from sympy import integrate, simplify
fxy = x + y                 # joint density
fy = integrate(fxy,(x,0,1)) # marginal density
fx = integrate(fxy,(y,0,1)) # marginal density
```

Then, we need to write out the conditional expectation,

```python
EXY = (3*y+2)/(6*y+3) # conditional expectation
```

Next, we can compute the left side, $\mathbb{E}(X-\mathbb{E}(X|Y))^2$,
as the
following,

```python
# from the definition
LHS=integrate((x-EXY)**2*fxy,(x,0,1),(y,0,1))
LHS # left-hand-side
```

$\displaystyle \frac{1}{12} - \frac{\log{\left(3 \right)}}{144}$

We can similarly compute the right side,
$\mathbb{E}(X)^2-\mathbb{E}(\mathbb{E}(X|Y))^2$,
as the following,

```python
# using Pythagorean theorem
RHS=integrate((x)**2*fx,(x,0,1))-integrate((EXY)**2*fy,(y,0,1))
RHS # right-hand-side
```

$\displaystyle \frac{1}{12} - \frac{\log{\left(3 \right)}}{144}$

Finally, we can verify that the left and right sides match,

```python
print(simplify(LHS-RHS)==0)
```

    True

In this section, we have pulled together all the projection and least-squares
optimization ideas from the previous sections to connect geometric notions of
projection from vectors in $\mathbb{R}^n$ to random variables. This resulted in
the remarkable realization that the conditional expectation is in fact a
projection operator for random variables.  Knowing this allows to approach
difficult problems in multiple ways, depending on which way is more intuitive
or
tractable in a particular situation. Indeed, finding the right problem to
solve
is the hardest part, so having many ways of looking at the same concepts
is
crucial.

For much more detailed development, the book by Mikosch
[[mikosch1998elementary]](#mikosch1998elementary) has some excellent sections
covering much of this
material with a similar geometric interpretation.
Kobayashi
[[kobayashi2011probability]](#kobayashi2011probability) does too.
Nelson [[edward1987radically]](#edward1987radically) also
has a similar
presentation based on hyper-real numbers.

## Appendix

We want to prove that we
the conditional expectation is the
minimum mean squared error minimizer of the
following:

$$
J= \min_h \int_{ \mathbb{R}^2 } \lvert X - h(Y) \rvert^2 f_{X,Y}(x,y) dx dy
$$

 We can expand this as follows,

$$
\begin{multline*}
J=\min_h \int_{ \mathbb{R}^2 } \lvert X \rvert^2
f_{X,Y}(x,y) dx dy + \int_{ \mathbb{R}^2 } \lvert h(Y) \rvert^2 f_{X,Y}(x,y) dx
dy \\\
- \int_{ \mathbb{R}^2 } 2 X h(Y) f_{X,Y}(x,y) dx dy
\end{multline*}
$$

 To minimize this, we have to maximize the following:

$$
A=\max_h \int_{ \mathbb{R}^2 }  X h(Y) f_{X,Y}(x,y) dx dy
$$

 Breaking up the integral using the definition of conditional expectation

$$
\begin{equation}
A =\max_h \int_\mathbb{R} \left(\int_\mathbb{R} X  f_{X|Y}(x|y)
dx \right)h(Y) f_Y(y) dy
\label{_auto1} \tag{3}
\end{equation}
$$

$$
\begin{equation} \
=\max_h \int_\mathbb{R} \mathbb{E}(X|Y) h(Y)f_Y(Y) dy
\label{_auto2} \tag{4}
\end{equation}
$$

 From properties of the Cauchy-Schwarz inequality, we know that the
maximum
happens when $h_{opt}(Y) = \mathbb{E}(X|Y)$, so we have found the
optimal $h(Y)$
function as:

$$
h_{opt}(Y) = \mathbb{E}(X|Y)
$$

 which shows that the optimal function is the conditional expectation.