\documentclass[11pt]{article}

\input{preamble}

%\usepackage{tikz}
%\usepackage{tikz-qtree}
\usepackage{qtree}
\usepackage{xfrac}

\title{week 3: ROC curves, Normalization and PCA}
\author{\url{http://mlvu.github.io}}
\begin{document}

\maketitle
\section{Exam questions}

At this point in the course, it's a good idea to have a look at the practice exam. The homework prepares you for the exam, but the format of the homework questions is different from that of the exam questions. Here are some questions you should be able to answer based on the material covered so far. You can find two complete practice exams on Canvas.

\begin{enumerate}
\item What is the advantage of gradient descent over random search?
\choicesv{In gradient descent, parallel searches are allowed to communicate.}{Gradient descent is less likely to get stuck in local minima.}{Gradient descent computes the direction of steepest descent, random search approximates it.\an}{Gradient descent is easier to parallelise.}

% 12 R
\item Why is accuracy a bad loss function to use in gradient descent?
\choicesv{It is expensive to compute.}{It makes the gradient zero almost everywhere. \an}{It is unreliable in situations with high class imbalance.}{The confidence interval is high on small test sets.}

% 11 R
\item I'm performing spam classification. I represent each email by three numbers: how often the word \emph{pill} occurs, how often the word \emph{hello} occurs and how often the word \emph{congratulations} occurs. What are these three attributes called?
\choicesv{The instances}{The classes}{The features \an}{The principal components}
\end{enumerate}

\noindent Here we see the derivation of the gradient of the squared-error loss for linear regression.

\begin{align}
\frac{\kp \frac{1}{2}\sum_i \left(f(x_i) - y_i\right)^2}{\kp b}	&= \frac{1}{2}\frac{\kp \sum_i \left(x_iw + b - y_i\right)^2}{\kp b}  \label{line:zero}\\
&= \frac{1}{2}\sum_i \frac{\kp \left(x_iw + b - y_i\right)^2}{\kp b} \label{line:sum}\\
&= \frac{1}{2}\sum_i \frac{\kp \left(x_iw + b - y_i\right)^ 2}{\kp \left(x_iw + b - y_i\right)} \frac{\kp \left(x_iw + b - y_i\right)}{\kp b} \label{line:chain}\\
&= \sum_i \left(x_iw + b - y_i\right) \frac{\kp \left(x_iw + b - y_i\right)}{\kp b} \label{line:exp}\\
&=  \ans{\sum_i\left ( x_iw + b - y_i\right)}{...} \label{line:final}
\end{align}

\begin{enumerate}[resume]

% 12 A
\item To get from line (\ref{line:zero}) to line (\ref{line:sum}), we use the
\choices{Chain rule}{Product rule}{Exponent rule}{Sum rule \an}

%12 A
\item To get from line (\ref{line:sum}) to line (\ref{line:chain}), we use the
\choices{Chain rule \an}{Product rule}{Exponent rule}{Sum rule}

\item In line \ref{line:final}, the correct result is
\choicesv{$\sum_i \left({x_i}^2w + b - y_i\right)$}{$\sum_i x_i \left(x_iw + b - y_i\right)$}{$\sum_i \left( x_iw + b - y_i\right)$ \an}{$\sum_i \left(x_iw + b - y_i\right)^2$}

\end{enumerate}

\section{ROC Curves and confusion matrices}

In this section, we'll look at evaluation metrics. This section covers the lecture ``Methodology 1'' and section 2.2 of Peter Flach's \emph{Machine Learning} (see the PDF on Canvas).

As a running example, we will use the following classification dataset:

\begin{center}
	\begin{tabular}{c c c c}
		& ${x_1}$ & ${x_2}$ & label\\
		\hline
		\kc{a} & 1 & 0 & \rc{Neg} \\
		\kc{b} & 2 & 0 & \bc{Pos} \\
		\kc{c} & 3 & 0 & \bc{Pos} \\
		\kc{d} & $\sfrac{1}{2}$ & 1 & \bc{Pos} \\
		\kc{e} & 2 & 2 & \bc{Pos} \\
		\kc{f} & 0 & 3 & \rc{Neg} \\
		\kc{g} & 1 & 3 & \rc{Neg} \\
		\kc{h} & 2 & 3 & \rc{Neg} \\
		\hline
	\end{tabular}
\end{center}
If you have trouble with the exercises below, we recommend drawing the data in its feature space, and drawing the decision boundary of each classifier.

\subsection{A linear classifier}

\qu As a first classifier, $c_\text{\oc{lin}}$, we will use a diagonal line crossing all the points where $x_1 = x_2$. Points above this line will be classified as \rc{Neg}, points below or on the line as \bc{Pos}. How do we describe this classifier mathematically? Fill in the blanks:
	\[
	c_\text{\oc{lin}}(x_1, x_2) = \begin{cases} \text{\bc{Pos}} & \text{ if } \ans{x_1 \geq x_2}{\ldots} \\ \text{\rc{Neg}} & \text{otherwise}\end{cases}\p
	\]
\ans{The diagonal that the question asks for (the decision boundary) is the set of points where $x_1 = x_2$. If $x_1 > x_2$, we're above this line, so we should classify the point as \bc{Pos}.}{}

\noindent $c_\text{lin}$ is a simple classifier. Let's try a more complex one before we move on.

\qu Let $c_\oc{2}$ be the classifier whose decision boundary crosses the points $(0, 1)$ and $(1, 1.5)$, with the class \emph{Neg} above the line. Draw this line first. How do we define this classifier?
	\[
	c_\oc{2}(x_1, x_2) = \begin{cases} \text{\bc{Pos}} & \text{ if } \ans{- 0.5 x_1 + x_2 - 1 < 0}{\ldots} \\ \text{\rc{Neg}} & \text{otherwise}\end{cases}
	\]
\ans{

We could fill the given values in to the equation $ax_1 + bx_2 + c = 0$ and solve for zero. But that would give us two problems: it's a lot of work, and we have two equations for three variables. There are infinitely many pairs $a$ and $b$ that would give us this decision boundary. Let's make things easier by assuming that $b$ is 1, and rewriting to $x_2 = -ax_1 - c$. This is just the familiar function for a line in the plane: $-a$ is the slope, and $-c$ is the bias.

The given points tell us that the line crosses the $x_2$ axis at 1, and the slope should be $0.5$. Thus, $-a = 0.5$ and $-c = 1$. A quick check tells us that for a point below the line, $(0,0)$, the quantity $- 0.5 x_1 + x_2 - 1$ is negative, so such points should be classified as \bc{Pos}.

Here is a diagram of the data and the boundaries for the two classifiers:

{\centering
\includegraphics[width=0.7\linewidth]{{w3.linear}.pdf}
}

}{}

\qu We will compare the linear classifier $c_\text{\oc{lin}}$ with a \emph{decision tree} classifier, $c_\text{\oc{tree}}$. Here is the decision tree:

\Tree[.{$x_2 > 1.5$?} [.{$x_1 > 0.5$?} [.{\ans{\bc{Pos}}{\oc{A}}} ][.{\ans{\bc{Pos}}{\oc{B}}} ] ][.{$x_1 > 1.5$?} [.{\ans{\rc{Neg}}{\oc{C}}} ][.{\ans{\bc{Pos}}{\oc{D}}} ] ] ]

\noindent If the inequality on the node is true, move right, otherwise move left.

This tree divides the feature space into four \textbf{segments} (\oc{A}, \oc{B}, \oc{C} and \oc{D}). To each segment we assign the majority class in that segment (using \bc{Pos} for a draw). Label the leaves \oc{A}, \oc{B}, \oc{C} and \oc{D} with classes.

\ans{Here is a diagram of the data and the boundaries for the tree classifier:

{\centering
\includegraphics[width=0.7\linewidth]{{w3.tree}.pdf}
}

}{}

\qu In this exercise we don't do any training (we just evaluate the given classifiers), so we'll evaluate all metrics on a single given dataset.

If you do train, you would \emph{split} your dataset. You would compute your evaluation metrics on the \ans{validation data}{\ldots} during hyperparameter selection and on the \ans{test data}{\ldots} during final testing.

Evaluating on the training data is the worst sin you can commit in Machine Learning. Can you think of a situation when you would still want to \emph{compute} the training loss?

\ans{Getting good training loss, but poor validation loss is a sure sign of overfitting. If overfitting is likely (as it is in decision trees and neural networks), you may want to compare the training loss to the validation loss to see if overfitting explains the poor performance.}{}

\qu Give the \emph{confusion matrices} for classifiers $c_\text{\oc{lin}}$ and $c_\text{\oc{tree}}$.
\ans{

\begin{center}
\begin{tabular}{c c | c c}
$c_\text{\oc{lin}}$  & & predicted & \\
 & & Pos & Neg \\
\hline
true & Pos & 3 & 1 \\
     & Neg & 1 & 3 \\
\end{tabular}
\end{center}

\begin{center}
\begin{tabular}{c c | c c}
$c_\text{\oc{tree}}$  & & predicted & \\
 & & Pos & Neg \\
\hline
true & Pos & 4 & 0 \\
     & Neg & 2 & 2 \\
\end{tabular}
\end{center}
}{}

\qu From the confusion matrices, we can compute several metrics. Compute the accuracy, precision, recall, true positive rate and false positive rate

\ans{
We compute these metrics from the four values of the confusion matrix: the number of true positives (TP), true negatives (TN), false positives (FP) and false negatives(FN):

\begin{center}
\begin{tabular}{c c | c c}
  & & predicted & \\
 & & \bc{Pos} & \rc{Neg} \\
\hline
true & \bc{Pos} & T\bc{P} & F\rc{N} \\
     & \rc{Neg} & F\bc{P} & T\rc{N} \\
\end{tabular}
\end{center}

\begin{center}
\bgroup
\def\arraystretch{2}
\begin{tabular}{r c c c}
& & $c_\text{\oc{lin}}$ & $c_\text{\oc{tree}}$	 \\
\hline
accuracy & $\frac{T\bc{P} + T\rc{N}}{\text{total}}$ & $\sfrac{3}{4}$ & $\sfrac{3}{4}$ \\
precision & $\frac{T\bc{P}}{T\bc{P} + F\bc{P}}$ & $\sfrac{3}{4}$ & $\sfrac{2}{3}$\\
recall & $\frac{T\bc{P}}{T\bc{P} + F\rc{N}}$ & $\sfrac{3}{4}$ & $1$ \\
true positive rate & $\frac{T\bc{P}}{T\bc{P} + F\rc{N}}$  & $\sfrac{3}{4}$ & $1$ \\
false positive rate & $\frac{F\bc{P}}{T\rc{N} + F\bc{P}}$  & $\sfrac{1}{4}$ & $\sfrac{1}{2}$\\
\end{tabular}
\egroup
\end{center}

Most of these are pretty intuitive, if you work out the meaning of the various sums. For instance, the false positive rate is the proportion of actual negatives, that were predicted to be positive.

}{}

\noindent To plot a ROC and precision/recall curves, and to compute AUC metrics, we need to turn our simple binary classifier into a \emph{ranking} classifier. That is, we don't just want classifications, we want it to rank the data from most \rc{Neg} to most \bc{Pos}. How we do this depends on the classifier.

\qu How do we turn a linear classifier into a ranking classifier? Give the ranking that $c_\text{lin}$ assigns to the training data.

\ans{We compute the distance to the classification boundary (the negative distance on one side, and the positive on the other), and rank points by this distance. For this course (the homework and the exam), you don't actually need to compute the distance, it's fine to draw the dataset and ``measure'' the distances. We'll make sure that all examples allow this approach.

Here's the sort of diagram you should end up with:

{\centering
\includegraphics[width=0.9\linewidth]{{w3.ranking}.pdf}
}

So, for $c_\text{\oc{lin}}$ the ranking from negative to positive is
\[
\bf{\rc{f}~\rc{g}~\rc{h}~\bc{d}~\bc{e}~\rc{a}~\bc{b}~\bc{c}}~\text{.}
\]}{}

\qu
How do we turn a decision tree classifier into a ranking classifier? Give the ranking that $c_\text{tree}$
 assigns to the training data.

\ans{
We group the points by the way the decision tree segments the feature space. We then assign each segment a class probability based on the relative frequencies of \emph{training} data points. We sort the groups by class probability. Points within the same group are ranked equal.

For $c_\text{\oc{tree}}$ the ranking from negative to positive is
\[
(\rc{\bf{f}}~\rc{\bf{g}})~(\rc{\bf{h}}~\bc{\bf{e}})~(\rc{\bf{a}}~\bc{\bf{b}}~\bc{\bf{c}})~(\bc{\bf{d}}) \text{.}
\]
Brackets indicate equal rank.
}{}

We can't tell whether our ranking is perfect, because the training data doesn't give us a correct \emph{ranking}, only a correct \emph{labeling}. But for some pairs of instances we \emph{do} know how they should be ranked: all pairs of positive and negative examples.

We can visualize this in a matrix with \rc{negative} instances on the horizontal axis, and \bc{positive} instances on the vertical, both sorted with the most positive (according to the classifier's ranking) in the bottom left corner. The matrix is usually colored \gc{green} for pairs that are ranked correctly, \rc{red} for pairs ranked incorrectly, and \yc{yellow} for pairs that are ranked equal.

\qu

\begin{enumerate}
\item Draw the coverage matrices of $c_\text{\oc{lin}}$ and $c_\text{\oc{tree}}$.

\ans{

\begin{center}
\begin{tabular}{c | c c c c}
	\bc{d} & \cellcolor{DarkRed}   & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} \\
	\bc{e} & \cellcolor{DarkRed}   & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} \\
	\bc{b} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} \\
	\bc{c} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} \\
	\hline
	$c_\text{\oc{lin}}$ & \rc{a} & \rc{h} & \rc{g} & \rc{f} \\
\end{tabular}
\end{center}

\begin{center}
\begin{tabular}{c | c c c c}
	\bc{e} & \cellcolor{DarkRed}   & \cellcolor{DarkOrange} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} \\
	\bc{c} & \cellcolor{DarkOrange}   & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} \\
	\bc{b} & \cellcolor{DarkOrange} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} \\
	\bc{d} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} \\
	\hline
	$c_\text{\oc{tree}}$ & \rc{a} & \rc{h} & \rc{f} & \rc{g} \\
\end{tabular}
\end{center}

}{}

\item How many ranking errors does each classifier make on the training data?
\ans{$c_\text{lin}$ makes 2 ranking errors. $c_\text{tree}$ makes 2.5, counting each tie as half a ranking error.}{}

\item If we have class imbalance (one class has more instances than the other), how can we tell by the coverage matrix?

\ans{It will be non-square. The more stretched out, the higher the class imbalance.}{}

\end{enumerate}

\qu How do we convert a coverage matrix to an \textbf{ROC plot}? How does the green area in a coverage matrix relate to the ROC plot? How do they differ?

\ans{We can convert the coverage matrix to ROC space by normalizing the axes. That is, if we have a green cell at $(2, 3)$, counting from the bottom left, we can draw a point in ROC space at $(\sfrac{2}{4}, \sfrac{3}{4})$. That is, we know we have a classifier that can achieve a true positive rate of $\sfrac{3}{4}$ and a false negative rate of $\sfrac{2}{4}$.

{\centering
\includegraphics[width=0.9\linewidth]{{w3.roc}.pdf}
}

The green area gives us an estimate of the probability of a ranking error. The same is true of the \emph{Area Under the Curve}. For the latter, we draw the convex hull of all achievable classifiers, and compute the area.

An important difference is that the green area in a coverage matrix may be \emph{concave}. For instance:

\begin{center}
\begin{tabular}{c | c c c c}
	e & \cellcolor{DarkRed}   & \cellcolor{DarkRed} & \cellcolor{DarkRed} & \cellcolor{DarkGreen} \\
	c & \cellcolor{DarkRed}   & \cellcolor{DarkRed} & \cellcolor{DarkRed} & \cellcolor{DarkGreen} \\
	b & \cellcolor{DarkRed} & \cellcolor{DarkRed} &
	    \cellcolor{DarkRed} & \cellcolor{DarkGreen} \\
	d & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} & \cellcolor{DarkGreen} \\
	\hline
	$c_\text{poor}$ & a & h & f & g \\
\end{tabular}
\end{center}

The corresponding ROC curve is convex (it covers the bottom right half of the ROC space).

Since a classifier with AUC below 0.5 is pretty useless, we don't usually worry about this. The differences between the (normalized) coverage matrix and the ROC space are not important for reasonable classifiers and large datasets.

}{}

\section{Whitening}

\paragraph{Basis systems} The \emph{standard basis system} (in 2D) is just the vectors $b_1 =\begin{pmatrix}1 \\0\end{pmatrix}$ and $b_2 = \begin{pmatrix}0 \\1\end{pmatrix}$. When a point has coordinates $(x, y)$, that just means that we can get to the point by adding $x$ times the $b_1$ vector to $y$ times the $b_2$ vector.

For the standard basis, we just end up at the point $\begin{pmatrix}x \\y\end{pmatrix}$. But we can also use a nonstandard basis. If a point $p$ has coordinates $(x, y)$ in the coordinate system with basis vectors $b_1$, $b_2$, it means that $p$ has coordinates $xb_1 + yb_2$ in the standard basis system.

\qu Convert the following points to a representation in the standard basis system.

\begin{enumerate}
	\item The point $\begin{pmatrix}1 \\1\end{pmatrix}$ expressed in the coordinate system with basis vectors $b_1 = \begin{pmatrix}2 \\0\end{pmatrix}$, $b_2 = \begin{pmatrix}0 \\2\end{pmatrix}$. \ans{$x = 1, y=1$, so we get to the standard basis point by computing $1 \cdot b_1 + 1 \cdot b_1 = \begin{pmatrix}2 \\2\end{pmatrix}$}{}
	\item The point $\begin{pmatrix}5 \\3 \end{pmatrix}$ expressed in the coordinate system with basis vectors $b_1 = \begin{pmatrix}0 \\1\end{pmatrix}$, $b_2 = \begin{pmatrix}1 \\0\end{pmatrix}$. \ans{$\begin{pmatrix}3 \\5 \end{pmatrix}$}{}
	\item The point $\begin{pmatrix}1 \\2 \end{pmatrix}$ expressed in the coordinate system with basis vectors $b_1 = \begin{pmatrix}1 \\2\end{pmatrix}$, $b_2 = \begin{pmatrix}3 \\4\end{pmatrix}$. \ans{$\begin{pmatrix} 7\\ 10\end{pmatrix}$}{}
\end{enumerate}

\qu A basis system is \textbf{orthonormal} if two conditions hold:
\begin{enumerate}
	\item \ans{The basis vectors all have norm (i.e. length) 1. (normality)}{\ldots}
	\item \ans{The angles between any two basis vectors are $90\deg$. (orthogonality) }{\ldots}
\end{enumerate}
Consider the following three bases:
\[
	\A     = \begin{pmatrix}0 & 1\\1 & 1\end{pmatrix} \;\;
	\B     = \begin{pmatrix}1 & -1\\1 & 1\end{pmatrix} \;\;
	\bm{C} = \sqrt{\sfrac{1}{2}}\begin{pmatrix}1 & -1\\1 & 1\end{pmatrix}
\]
The columns of each matrix are the basis vectors (it may help to draw them in standard coordinates). Which of these have orthogonal basis vectors? Which one is orthonormal?
\ans{The columns of $\A$ are neither normal nor orthogonal. Matrix $\B$ has orthogonal columns, but they are not unit vectors (vectors of length one). Matrix $\C$ represents an orthonormal basis.}{}

\qu Why is it useful to have an orthonormal basis?
\ans{Matrices representing an orthonormal basis are \emph{orthogonal}. This means that $\C^{-1} = \C^T$. This means that we can easily transform from and to the basis without having to compute an expensive matrix inverse.}

\paragraph{Covariance and normal distributions}

PCA is closely related to the normal distribution. We can think of PCA as a basis transformation that makes it look like the data was sampled from a standard multivariate normal distribution: it makes the variance 1 in every dimension, and the covariance 0 for every pair of dimensions.

\qu Here is some data.

\begin{tabular}{c c}
$x_1$ & $x_2$ \\
\hline
0 & 2 \\
2 & 0 \\
0 & 2 \\
2 & 0 \\
\hline
\end{tabular}

\noindent Compute the sample covariance.

\ans{
We first compute the sample mean: $\bm{m} = \begin{pmatrix} 1 \\ 1\end{pmatrix}$.

Subtracting the sample mean, and arranging the data in a matrix (with instances as columns), we get
\[
\X =  \begin{pmatrix} -1 & 1 & -1 & 1 \\  1 & -1 & 1 & -1 \end{pmatrix} \p
\]

The sample covariance is
\begin{align*}
\bm{S} = \frac{1}{n-1} \X\X^T = \frac{1}{3}\begin{pmatrix} 4 & -4  \\  -4 & 4 \end{pmatrix}
\end{align*}
}{}

\noindent Let $\y = \A\x + \bt$ be an \emph{affine} transformation (a linear transformation $\A$ followed by a translation $\bt$). We draw $\x$ from a standard normal distribution
\[
N(\mathbf{0}, \mathbf{I}) = N\left( \begin{pmatrix}0 \\ 0\end{pmatrix}, \begin{pmatrix}1 & 0 \\ 0 & 1\end{pmatrix} \right)
\]
and compute $\y$. This is equivalent to sampling from a normal distribution with mean $\bm{\mu} = \bt$ and covariance $\bm{\Sigma} = \A\A^T $.

\qu Let
\[
\mathbf{A} = \begin{pmatrix} 1 & 0 \\ 0 & 2\end{pmatrix}, \mathbf{t} = \begin{pmatrix} 1 \\ 1\end{pmatrix}
\]
Sampling $\x$ from $N(\bm{0}, \bm{I})$ is easy: we just sample $x_1$ and $x_2$ independently from a \emph{uni}variate standard normal distributions. Here are some \href{https://goo.gl/Jw7Hjm}{standard normally distributed numbers}:
\[
0.5, 1.1, 0.1, 0.9, -0.2, 1.3
\]

\noindent Turn these into three samples from $N(\bm{0}, \bm{I})$ and transform them to samples from $N(\bm{\mu},\bm{\Sigma})$.

\ans{
Using samples
\[
\x^1 = \ma{0.5, & 1.1}^T, \;\; \x^2 = \ma{0.1, & 0.9}^T \;\; \x^3 = \ma{-0.2, & 1.3}^T
\]
and computing $\y^1 = \A\x^1 + \bt$, we get:
\[
\y^1 = \ma{1.5, & 3.2}^T, \;\; \y^2= \ma{1.1, & 2.8}^T \;\; \y^3 = \ma{0.8, & 3.6}^T\p
\]
}{}

\paragraph{Bonus question:} Compute the sample covariance of the resulting data, and the actual covariance $\Sigma$.

\ans{For the sample covariance, follow the same procedure as in the previous question. After a bit of calculating, we get:
\[
\bf{S} \approx \ma{0.123 & 0.06\\0.06 & 0.16}
\]

The true covariance we get by computing $\bf{\Sigma} =\A\A^T = \ma{1 & 0 \\ 0 & 4}$. They're quite different at the moment. If we increase the size of the sample, $\bf{S}$ will converge to $\bf{\Sigma}$.
}{}
%
%\paragraph{Singular value decomposition}
%\emph{These questions may be a little challenging if you're new to Linear Algebra. If you're struggling, have another look at the slides from the \emph{Methodology 2} lecture. If you can't work it out, read the answer, and see if you can repeat the process on another matrix.}
%
%\begin{Exercise}
%Let $\A$ be a linear transformation. This transformation has two eigenvectors, which are aligned with the standard basis vectors. The eigenvalue along the horizontal axis is $3$, and along the vertical axis is $2$. What is $\A$?
%
%\ans{$\A$ is a \emph{scaling matrix}. That means it's a diagonal matrix, with the eigenvalues on the diagonal:
%\[
%\begin{pmatrix} 3 & 0 \\ 0  & 2 \end{pmatrix}
%\]
%}{}
%\end{Exercise}
%
%Let $\B$ be a linear transformation. $\B$ has two eigenvectors, one points in the direction of the horizontal axis rotated by 45 degrees, and the other in the direction of the vertical axis rotated by 45 degrees.
%The eigenvalue for the first is 5, and for the second is 2. What is $\B$?
%\ans{
%If the eigenvectors were axis-aligned (as with $\A$), $\B$ would be
%\[
%\bm{Z} = \begin{pmatrix} 5 & 0 \\ 0  & 2 \end{pmatrix}
%\]
%We can use this by decomposing $\B$ in three steps: rotating by -45 degrees to align the axes to the eigenvectors, applying $\bm{Z}$ and rotating back. The rotation matrix for a 45 degree rotation is
%\[
%\U = \sqrt{\sfrac{1}{2}}\begin{pmatrix} 1 & -1 \\ 1 & 1\end{pmatrix} \p
%\]
%Since $\U$ is orthogonal (its columns form an orthonormal basis), we can take the transpose $\U^T$ to rotate back. Putting all this together, we get
%\[
%\B\x = \U\Z\U^T\x \p
%\]
%
%This is the eigenvalue decomposition of $\B$. This gives us
%
%\begin{align*}
%\B = \U\Z\U^T &= \sqrt{\sfrac{1}{2}}\begin{pmatrix}1 & -1\\1 & 1\end{pmatrix}\begin{pmatrix} 5 & 0 \\ 0 & 2\end{pmatrix}\sqrt{\sfrac{1}{2}}\begin{pmatrix}1 & 1\\-1 & 1\end{pmatrix}\\
%&= \frac{1}{2}\begin{pmatrix}1 & -1\\1 & 1\end{pmatrix}\begin{pmatrix} 5 & 0 \\ 0 & 2\end{pmatrix}\begin{pmatrix}1 & 1\\-1 & 1\end{pmatrix}\\
%&= \frac{1}{2}\begin{pmatrix}5 & -2\\5 & 2\end{pmatrix}\begin{pmatrix}1 & 1\\-1 & 1\end{pmatrix} \\
%&= \frac{1}{2}\begin{pmatrix}7 & 3\\3 & 7\end{pmatrix} = \begin{pmatrix}3.5 & 1.5\\1.5 & 3.5\end{pmatrix}
%\end{align*}
%
%%As usual, we can check with \href{https://goo.gl/v8d4Yg}{Wolfram Alpha} to see we haven't made any mistakes. You may get $-\U$ instead of $\U$. The negatives cancel out, so the SVD can work out both ways.
%}{}

\end{document}


This is implementation of efficient and simple model for simulation of
High-resolution atomic force microscopy (AFM), scanning probe microscopy (STM)
and inelastic tunneling microscopy (IETS) images using classical forcefileds.

There are two versions of the code:

\begin{enumerate}
   \item currently developed Python/C++ version in PyProbe\_nonOrtho (branch master
    );
        to get quick esence of this model you can also try web interface hostet here: http://nanosurf.fzu.cz/ppr/
        for more details see wikipage:
        https://github.com/ProkopHapala/ProbeParticleModel/wiki
   \item Legacy fortran version in SHTM\_springTip2 (branch fortran );
        more detailed description o the fortran version is here:
        http://nanosurf.fzu.cz/wiki/doku.php?id=probe\_particle\_model
\end{enumerate}
\cite{phapalamechhighresol}
\cite{phapalaoriginhighresol}

\section{C++ \& Python version}

\subsection{Examples of results}

\subsection{How it works}

\begin{itemize}

    \item  Pauli-repulsion ( currently approximated by repulsive part of
    Lenard-Jones potential $r^{-12}$ )
    \item van der Waals attraction ( currently approximated by attractive part
    of Lenard-Jones potentia $r^{-6}$ )
    \item electrostatic ( currently can be computed as coulomb pairwise
    interaction between PP and point charges in centers of atoms of sample, or
    by reading electrostatic force-field obtained by derivative of sample
    Hartree potential as described in supplementary of this paper
    \cite{phapalaoriginhighresol}.

\end{itemize}

The computation of images is divided into two parts:
\begin{enumerate}
    \item Precompute vector Forcefield ( Fx(x,y,z), Fy(x,y,z), Fz(x,y,z) ) over
    sample and store it on a 3D-grid. ( see getLenardJonesFF and getCoulombFF
    functions for more details ). After individial components of forcefiled are
    sampled ( i.e. Lenard-Jones and electrostatic ) they are summed up to form
    one single total effective forcefield in which Probe-Particle moves.

    \item relax Probe-Particle attached to the tip under influence of the total
    effective forcefield. The relaxation of Probe-Particle is done using "Fast
    Inertial Realxation Engine" (FIRE) algorithm
    \cite{ebitzekstructrelaxmadesimple}. implemented in
    FIRE:move() function. In each step of relaxation the forcefield on the grid
    is interpolated in interpolate3DvecWrap function, considering periodic
    boundary condition. From python the relaxation is called as relaxTipStroke
    function providing 1D-array of tip positions, and obtaining back 1D-array of
    Probe-Particle position after the relaxation and of force between tip
    ProbePartcile and sample at that position. The lateral scan in (x,y) to
    obtained stack of images is done by calling relaxTipStroke at different
    (x,y) position of tip, where each call of relaxTipStroke does one approach
    along z-direction.

\end{enumerate}

\subsection{Why it is splitted like this ?}

This splitting of computation ( first sampling of forcefiled on the grid, and
than relaxation using interpolation) has several advantages over strightforward
computation of interaction on the fly during relaxation process.

\begin{itemize}

    \item \textbf{It is faster} - if there is ~100 atoms of sample, summing over
    each pairwise Lennard-Jones interactions, is much slower, than just
    interpolating the forcefield from the grid. Because force evaluation is done
    several-times for each voxel of the 3D scanning grid, it is more efficient
    to precompute it just once for each voxel, and then interpolate.

    \item \textbf{It is more flexible and general} - Decoupling of relaxation
    process and computation of forcefiled allows us to plug-in any forcefield.
    The original motivation was to used electrostatic forcefield obtained from
    Hartree potential from DFT calculation. However, it is not limited to that.
    We can plug in e.g. a derivative of potential energy of probe peraticle
    (i.e. Xe or CO ) obtained by scanning it in DFT code, in similar way as Guo
    did in \cite{chshguohighresolmodel} . The only limitation here is computational cost of obtaining
    such potential from ab-initio calculation.
\end{itemize}

\subsection{Code structure}
The code is divided into Python and C++, where performance intensive
computations are done in C++ and python is used as convenient scripting
interface for doing tasks like file I/O, plotting, memory management. For
binding of Python and C++ is used python ctypes library.

\subsubsection{Dependencies:}
\begin{itemize}
    \item \textbf{C++} : g++ ( tested with g++ (Ubuntu 4.8.1-2ubuntu1~12.04) 4.8.1 )
    \item \textbf{Python} : python ( 2.7.3 ), numpy (1.9.2), matplotlib ( 1.4.3 ), ctypes
    ( 1.1.0 )
\end{itemize}

\subsubsection{C++ source:}
\begin{itemize}
    \item  ProbeParticle.cpp - implementation of all performance intensive
    ProbeParticle model as dynamic library which can be called dynamically from
    python.
    \item Vec3.cpp,Mat3.cpp math subroutines for operations with 3D vectors and
    metrices
\end{itemize}

\subsubsection{Python source:}
\begin{itemize}
    \item ProbeParticle.py - Interface between C++ core and python ising
    C-types. Here are also defined some python rutines like:
    \begin{itemize}
        \item  conversion from Force to frequency shift ( Fz2df ),
        \item  evaluation of Lenard-Jones coeffitints ( getAtomsLJ )
        \item  copy sample geometry to simulate periodic boundary condition (
        PBCAtoms )
        \item automatic setup of imagining area acroding to geometry of
        nonperiodic sample ( autoGeom )
        \item default parameters of simulation ( params ) with subroutine to
        read this parameters from a file ( loadParams )
    \end{itemize}

    \item  test2.py, testServer2.py - two examples of python scripts with run
    actual computations using ProbeParticle library. The idea is that this files
    can be modified by user to match particular task
    \item basUtils.py - routines for loading of molecule geometry from
    xyz-format, ( loadAtoms ), finding bonds and other.
    \item Element.py and elements.py contains just parameters of atoms form
    periodic table ( like effective raidus, color for visualization etc. ). It
    is used by basUtils.py

\end{itemize}


% !TEX root = collhdl_ICRA_17.tex

\section{Evaluation and Case Studies}
\label{sec:simu}

\subsection{General Simulation Setup}
\label{sec:setup}
The simulative setup is as follows.
We apply our collision detection, isolation and identification algorithm in simulation to an \emph{Atlas} robot consisting of cylinders and cuboids as collision geometry from \cite{DRCSim}.
We assume perfect measurements of the generalized joint forces and the force/torque sensors unless stated differently, meaning all sensors measure the true value without noise or delay.
Force/torque sensors are placed at the end-effectors (feet and hands), before the first arm and leg joints and at the upper torso after the three back joints (see Fig.~\ref{fig:coll_singlecontact_overview}).
Furthermore, all acting external forces comply with the assumption of a pushing force and the line of action of the forces is chosen such that it does not intersect with any other collision body before the contact point.
The robot pose (see Fig.~\ref{fig:coll_singlecontact_overview}) has been chosen in such a way that the whole body Jacobian has full rank.
The robot is modeled in free space, meaning there are no other external forces acting on the robot than the collision forces.
The simulation is done on a standard Core i7-3770 PC and the algorithm is run as compiled Matlab functions.
With the current implementation, the majority of the runtime for all contact cases is spent on the not yet optimized intersection algorithm.

\subsection{Ideal Analysis of Single Contacts}
\label{sec:eval_single_static}
We compare the ideal performance of the isolation based on estimated generalized external forces $\hat{\bm{\tau}}_\epsilon$ only (Sec.~\ref{sec:isolation}) to the isolation based only on force/torque sensor measurements $\hat{\bm{\mathcal{F}}}_{\mathrm{ext},S}$ (from (\ref{eqn:cmp})) for a single contact.
Dynamic effects due to the observer and load compensation are neglected for sake of clarity.
The contact points are distributed randomly and equally over the robot surface.
Figure~\ref{fig:coll_singlecontact_overview} depicts the results of the isolation procedures with some representative samples of the 3000 tested points.
With both methods, the contact points are isolated correctly up to rounding errors, if possible.
The contacts at the pelvis cannot be seen by any force/torque sensor and can therefore only be isolated with the generalized external forces acting on the base.
The maximum and mean execution times for the isolation of the one contact in this experiment were 0.67~ms and 0.25~ms using force/torque sensors as well as 0.57~ms and 0.17~ms using joint torque measurements only.
\begin{figure}
\begin{center}
\begin{overpic}{figures/collest_single_allresults_nosim/coll_est_sc_all_nosim}
\put(82,54){\line(-1,-2){7}}
\put(56,11){\line(1,2){11}}
\put(67,35){F/T sensor}
\end{overpic}
\end{center}
\caption{Distribution of true ($\bm{r}_\mathrm{C}$) and estimated contact locations using two different methods.
The points $\hat{\bm{r}}_C(\hat{\bm{\tau}}_\epsilon)$ are isolated using only generalized external forces and the points $\hat{\bm{r}}_C(\hat{\bm{\mathcal{F}}}_{\mathrm{ext,S}})$ are isolated using the force/torque sensor measurements only.
For the pelvis (base link) and first torso links, no force measurement is available. Therefore contact points at these links cannot be found with the latter method.
All overlying points are identical up to rounding errors.}
\vspace*{-0.8cm}
\label{fig:coll_singlecontact_overview}
\end{figure}

\subsection{Ideal Analysis of Multiple Contacts}
\label{sec:eval_multi_static}
In the following, we analyze multiple contacts of the type specified in Sec.~\ref{sec:eval_single_static}.

\subsubsection{Two Contacts at different links}

The considered contact points are located at different links, since two contacts at the same link cannot be located in our setup. In order to be able to do so, one would need two force/torque sensors in the same link.
5000 random combinations were examined.

The algorithm using generalized forces is able to isolate all two contact point coordinates up to rounding errors if the rank of the Jacobian in (\ref{eqn:force_identification}) is sufficiently high, e.g. $\mathrm{rank}(\bm{J})=12$ for $N_\mathrm{C}=2$ contact points.
Figure~\ref{fig:coll_multicontact_rankJ} shows the success of the isolation for different contact links, which is equivalent to the plot of the rank deficit
\begin{equation}
RD=6N_\mathrm{C}-\mathrm{rank}(\bm{J}).
\label{eqn:rank_deficit}
\end{equation}
For example, a contact at the pelvis (torso chain link 1) and another contact at the left lower leg (left leg chain link 4) does only produce 10 nonzero columns in the stacked Jacobian and the isolation fails (see corresponding entry in Fig.~\ref{fig:coll_multicontact_rankJ}).
If instead the foot on the left leg chain (link 6) is in contact, then 12 nonzero columns exist and the isolation succeeds.

For a rank deficit, the isolation method minimizes the Cartesian error of the stacked identified wrenches and finds contact points on the entire following kinematic chain.
The distribution of errors for different ranks of the Jacobian is shown in Fig.~\ref{fig:hist_rank_err_2coll}.
An error $\leq$ 25~cm occurs in about 80\% of the cases with the joint configuration from Fig.~\ref{fig:coll_singlecontact_overview}.
Please note that in case of a rank deficit, the algorithm may accidentally estimate an additional contact point, which would be located close to the base.
The error is then calculated between the real contact points and the ones estimated in the same kinematic chain.
The maximum and mean execution times for the isolation in this experiment were 0.75~ms and 0.28~ms using force/torque sensors as well as 31.8~ms and 3.6~ms using joint torque measurements only.

\begin{figure}
\begin{center}
\includegraphics[width=\columnwidth]{figures/collest_multi_stat_2coll_rankmat/rankmat_fig_2coll_Modus2}
\end{center}\vspace*{-0.3cm}
\caption{Overview of success and failure for the isolation of two simultaneous contacts.
The $x$-axis gives the first contact link and the $y$-axis the second one.
The second and the last body of the torso chain have no collision body in the model, the according columns are therefore left empty.
A green upward triangle marks the successful isolation of both contact points for the given combination.
A red downward triangle marks a combination for which it was impossible to isolate the contacts correctly due to rank loss of the combined Jacobian in (\ref{eqn:multi_iso}).
In some fields, green and red triangles are plotted meaning that for this combination, the isolation was only partly correct.
For the combination of the fifth link in the right leg and the first link of the right arm, this is due to the fact, that for some contacts, the joint torque threshold for the last joint is not exceeded and the algorithm is therefore started with a loss of rank in the Jacobian.
However, when reducing the joint torque threshold or starting the algorithm with the correct contact links, this could not be observed anymore.
In the other cases, the contact point was estimated correctly by accident with a rank deficient Jacobian.
Overall 5000 combinations were tested.}\vspace*{-0.5cm}
\label{fig:coll_multicontact_rankJ}
\end{figure}

\begin{figure}
\vspace*{-0cm}
\begin{center}
% Um die Prozentangaben für Overpic zu generieren: Skript histogram_error_rank.m im Bildordner ausführen und aus Textausgaben entnehmen.
\begin{overpic}{figures/collest_multi_stat_2coll_error_rank/hist_rank_err_2coll}
\put(50,78){\line(2,0){10}}
\put(63,76){$100\%$}

\put(49,47){\line(2,3){8}}
\put(56,61){$34\%$}

\put(43,35){\line(1,1){15}}
\put(55,52){$21\%$}

\put(55,36){\line(2,1){7}}
\put(55,40){$11\%$}

\put(68,17){\line(1,1){15}}
\put(80,34){$2\%$}
\end{overpic}
\end{center}
\caption{Histogram of the maximum point estimation error for two contacts with different random contact configurations. Read: ``34\% of the results for $\mathrm{rank}(\bm{J})=11$ lie between 15 and 25~cm.''}
\label{fig:hist_rank_err_2coll}
\vspace*{-0.3cm}
\end{figure}

\subsubsection{Contacts at four end-effectors}

For the considered 30~DoF humanoid, it is possible to detect and isolate up to $N_\mathrm{C}=5$ contacts, if the rank deficit $RD$ is zero.
This is the case for four contacts at the end-effectors of every kinematic chain (feet, hands) and an additional contact located at the torso or first shoulder link, which is the typical manipulation scenario from Fig~\ref{fig:manip}.
The distribution of position errors for this case are depicted in Fig.~\ref{fig:error_rank_manipcoll} with markers for different rank deficits $RD$.

Without rank deficit, the isolation is successful in all but 5 of the 5000 examined points, where there was no rank deficit.
The errors occur in situations, where the joint torque threshold for the last joint in at least one chain is not exceeded.
This leads to a Jacobian with rank-deficit for the first iteration step of the algorithm, making it in a few cases impossible to find the correct contact link.
However, if the threshold is reduced to $10^{-5}$~N or Nm (for all  $\lVert\bm{f}_{\mathrm{ext},i}\rVert<1$), or the algorithm is started with perfect information about the contact links, no errors are observed.
The maximum and mean execution times for the isolation in this experiment were 0.92~ms and 0.52~ms using force/torque sensors as well as 43.2~ms and 11.0~ms using joint torque measurements only.

\begin{figure*}
\begin{center}
\includegraphics{figures/collest_multi_stat_manipcoll_error/error_rank_manipcoll_hist_breit}
\end{center}\vspace*{-0.4cm}
\caption{Distribution of the isolation position error based on joint torques only with contacts at all end-effectors for different rank deficits $RD$ of the Jacobian.
The position error distributions are drawn over the location for a fifth contact.
The fifth contact location can be none (only four contacts) or any of the links, which are not an end-effector link.
Therefore the four columns of the end-effectors are left empty.
The columns of the second and the last torso chain link are left empty, because their model does not have any contact bodies, making it impossible to place contacts at these links.
Markers in the lines denote the mean errors of all contact points including this link.
The distribution lines are normalized to the highest corresponding value of the distribution.}\vspace*{-0.7cm}
\label{fig:error_rank_manipcoll}
\end{figure*}

\subsubsection{Benefits of additional force/torque sensors}

The results show the benefits of supplementary force/torque sensors, see Table~\ref{tab:sensor_benefits}.
Without force/torque sensors, up to five contacts can be detected, isolated and identified correctly under certain conditions.
With force/torque sensors in the distal links, at least the generalized external manipulation and locomotion forces can be found and further contacts can be isolated by additional force/torque sensors (max. one per sensor) or the first method (up to 5 in theory).
%
\begin{table}
\caption{Possibilities of collision detection, isolation and identification with different numbers of force/torque sensors in the kinematic tree. The sensors are meant to be added up down the lines of the table.}\label{tab:sensor_benefits}
\begin{tabular}{|p{2.8cm}|p{5cm}|}
\hline
sensors & possibilities\\\hline
only joint torque and base movement sensors & identify ground contact and manipulation contacts, detect single collision, isolate and identify single collision under certain conditions (see Fig.~\ref{fig:error_rank_manipcoll})\\\hline
distal force/torque sensors & full elimination of ground contact and manipulation forces, detection isolation and identification of single collisions, multiple collisions can be detected, isolated and identified in many cases\\\hline
additional force/torque sensors in the kinematic chains & detect isolate and identify one additional contact wrench per additional sensor\\\hline
\end{tabular}\vspace*{-0.3cm}
\end{table}

\subsection{Time Series Analysis of a Single Contact}
The influence of the observer dynamics on the isolation and identification is investigated by using generalized external forces including observer dynamics.
The external force acts over a sinus half wave with an amplitude of one and a cycle time of 40~ms.
The observer is run at a sample frequency of 1000~Hz and $K_O=500$ as suggested in \cite{Haddadin2014}.
In this simulation, the observer error and the error in acceleration estimation for the force/torque sensor compensation are assumed to follow the dynamics presented in (\ref{eqn:obs_dynamic}) and (\ref{eqn:qDD_dyn}) respectively.
%
\begin{figure}
\begin{center}
\includegraphics{figures/colltest_single_filter/colltest_single_filter_f}
\end{center}\vspace*{-0.4cm}
\caption{The $x$-component of the external force $\bm{f}_\mathrm{ext}$, the external force estimated with force/torque sensors  $\hat{\bm{f}}_{\mathrm{ext}}(\hat{\bm{\mathcal{F}}}_{\mathrm{ext,S}})$ according to (\ref{eqn:cmp}) and (\ref{eqn:cmp_ext_vorher}) as well as the external force estimated with the observed generalized forces $\hat{\bm{f}}_{\mathrm{ext}}(\hat{\tau}_\epsilon)$ according to (\ref{eqn:force_identification}) is depicted over time for a collision of  20~ms.
When using the generalized forces for identification, the estimated external force is a first order filtered version of the real force with time constant $K_O^{-1}$.
Using compensated force/torque sensors, the estimated force follows the real force without any delay.}\vspace*{-0.6cm}
\label{fig:coll_singlecontact_filter}
\end{figure}
%
With both methods described before, the contact point is isolated correctly up to numerical errors.
If the observed generalized joint forces are used for isolation, there is a delay of one time step, as the filtered generalized forces are still zero for the first time step of the collision.

Fig.~\ref{fig:coll_singlecontact_filter} shows the external forces over the collision time and the forces identified with and without the use of force/torque sensors.
It can be seen that if no force/torque sensors are used, the contact force is estimated with a delay of approximately 3~ms.
It has to be noted, that the collision will be seen about 10~ms longer with the observer dynamics taken into account.
This timespan depends on the thresholds for the generalized external forces, which was here chosen to be 0.001~N or 0.001~Nm, respectively and the filter frequency $1/K_O$.
Furthermore, when using force/torque sensors, smaller delay but larger error in the estimated external force can be observed.
As the error of $\hat{\ddot{\bm{q}}}$ is driven by $\bm{\tau}_\mathrm{ext}$ and $\bm{M}(\bm{q})$ is nearly constant for the robot at standstill, the contact point can still be estimated correctly with the compensated force/torque sensor.


\documentclass{article}

%----------------------------------------------------------------------------------------

\usepackage{listings} % Required for inserting code snippets
\usepackage{geometry}
\geometry{margin=0.7in}
\usepackage[usenames,dvipsnames]{color} % Required for specifying custom colors and referring to colors by name
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{tikz}
\usepackage{enumerate}
\usepackage[T1]{fontenc}

\delimitershortfall-1sp
\newcommand\abs[1]{\left|#1\right|}

\definecolor{DarkGreen}{rgb}{0.0,0.4,0.0} % Comment color
\definecolor{highlight}{RGB}{255,251,204} % Code highlight color

%----------------------------------------------------------------------------------------

\begin{document}

%----------------------------------------------------------------------------------------

\section*{Problem 1}
\begin{flushleft}
\textit{Several useful inequalities involving absolute values can be derived from the triangle inequality by a judicious choice for a and b.}
\begin{enumerate}[(a)]
For any of these examples, we can redefine $x$ and $y$ to be $a$ and $b$, which will help us reduce to the following variants of the triangle inequalities. For each of the following inequalities, I use the following format $\abs{a} + \abs{b} \leq \abs{a+b}$.

\item $\abs{x} - \abs{y} \leq \abs{x-y}$ \\
\vspace{.3cm}
Here we will let $a=(x-y)$ and $b=y$. Then, we see that \\
\begin{center}
$\abs{x-y+y} \leq \abs{x-y} + \abs{y} \implies \abs{x} \leq \abs{x-y} + \abs{y} \implies \abs{x} - \abs{y} \leq \abs{x-y}$
\end{center}

\item $\abs{x} - \abs{y} \leq \abs{x+y}$ \\
\vspace{.3cm}
Here we will let $a=(x+y)$ and $b=-y$. Then, we see that \\
\begin{center}
$\abs{x+y+(-y)} \leq \abs{x+y} + \abs{-y} \implies \abs{x} \leq \abs{x+y} + \abs{y} \implies \abs{x} - \abs{y} \leq \abs{x+y}$
\end{center}

\item $\abs{x} + \abs{y} \geq \abs{x-y}$ \\
\vspace{.3cm}
Here we will let $a=x$ and $b=-y$. Then, we see that \\
\begin{center}
$\abs{x+(-y)} \leq \abs{x} + \abs{-y} \implies \abs{x-y} \leq \abs{x} + \abs{y}$
\end{center}

\item $\abs{\abs{x} - \abs{y}} \leq \abs{x-y}$ \\
\vspace{.5cm}
To prove this, we will set up 2 different equations. \\
\begin{center}
\begin{tabular}{c c}
(1) $\abs{y-x+x} \leq  \abs{y-x} + \abs{x}$ &
(2) $\abs{x-y+y} \leq \abs{x-y} + \abs{y}$ \\
\end{tabular}
\end{center}
(1) We can move the $\abs{x}$ from the left side to the right side and take negative one out of the left hand side, \\
which gives us $-(\abs{x}-\abs{y}) \leq \abs{y-x} $. \\
(2) We can move the $\abs{y}$ from the left side to the right side, which gives us $\abs{x}-\abs{y} \leq \abs{x-y}$. \\
\vspace{0.5cm}
From the properties of absolute values, we know $\abs{x-y} = \abs{y-x}$ and if $x \geq a$ and $x \geq -a$ then $x \geq \abs{a}$. \\

From the combination of these two facts, we now have $\abs{\abs{x}-\abs{y}} \leq \abs{x-y}$.
\end{enumerate}
\end{flushleft}

\section*{Problem 2}
\begin{flushleft}
Recall the definition of what it means for $S \subset \mathbb{R}$ to be \textit{bounded above, bounded below} and just \textit{bounded}. Prove that $S$ is bounded if and ony if there is a real number $M > 0$ such that
\begin{center}
$\forall s \in S, \abs{s} \leq M$
\end{center}

\textbf{Proof} \\
\vspace{.3cm}

Let's assume $S \subset \mathbb{R}$, then $\abs{s} \leq M$ means that $-M < s < M$, $\forall s \in S$. $S$ is bounded above if there exists some $X \in \mathbb{R}$ such that $\forall s \in S$, $s < X$. Likewise, $S$ is bounded below if there exists some $x \in \mathbb{R}$ such that $\forall s \in S$, $x < s$.  Thus, we see that $M$ is in fact an upper bound for $S$ because $s < M$, $\forall s \in S$ and that $-M$ is in fact a lower bound for $S$ because $-M < s$, $\forall s \in S$. Therefore, since $S$ is bounded above and below, $S$ is, by definition, bounded.
\end{flushleft}

\section*{Problem 3}
\begin{flushleft}
We say that a function f is \textit{invertible} if $f^{-1} = \{ (b,a) : (a,b) \in f \}$ is also a function, in which case we call it the inverse function to $f$. Notice that \\
\begin{center}
$f^{-1}(b) = (a) \leftrightarrow b = f(a)$,
\end{center}
assuming that $f^{-1}$ is a function.
\begin{enumerate}[(a)]
\item If $f$ is invertible, what are the domain and range of $f^{-1}$? \\
\item Which of the following functions are invertible? For those that are invertible, give the inverse. \\
\end{enumerate}
\begin{center}
\begin{tabular}{l l}
$l = \{(x,y) \in \mathbb{R}^2 : y = 2x + 1\}$ & Inverse: $y=\frac{x-1}{2}$, Domain: $\mathbb{R}$, Range: $\mathbb{R}$ \\
\vspace{.2cm}
$c = \{(x,y) \in \mathbb{R}^2 : y \geq 0, x^2 + y^2 = 1\}$ & Not invertible \\
\vspace{.2cm}
$s = \{(x,y) \in \mathbb{R}^2 : y = x^2\}$ & Not invertible\\
\vspace{.2cm}
$ \sqrt = (x,y) \in \mathbb{R}^2 : y \geq 0, y^2 = x\}$ & Inverse: $y=x^2$, Domain: $\mathbb{R}$, Range: $[0, \infty)$ \\
\vspace{.2cm}
$sin = \{(x,y) \in \mathbb{R}^2 : y = sin(x)\}$ & Not invertible \\
\end{tabular}
\end{center}
\end{flushleft}

\section*{Problem 4}
\begin{flushleft}
Assume that $g$ is bounded function, that is, $\abs{g(x)} < B$ for all $x \in \mathbb{R}$. Prove that
$\displaystyle \lim_{x \to 0}  x \cdot g(x) = 0$. \\

\vspace{.5cm}

\begin{center}
$c=0$, $L=0$, $f(x)=x \cdot g(x)$
\end{center}

$\abs{f(x) - L} < \epsilon \implies \abs{x \cdot g(x) - 0} < \epsilon \implies \abs{x \cdot g(x)} < \epsilon$. \\
\vspace{.2cm}
At its max, $g(x)$ can be no greater than $B$. Thus, we know $B$ is an upper bound for $f(x)$. We can rewrite $f(x)$ as $\abs{x \cdot B}$. Thus, $\abs{x \cdot B} < \epsilon \implies -\epsilon < x \cdot B < \epsilon$. By dividing by $B$ on both sides, we have $\frac{-\epsilon}{B} < x < \frac{\epsilon}{B}$. \\

\subsection*{Proof}
Let $\delta_\epsilon = \frac{\epsilon}{B}$. Then $\abs{x - 0} < \delta_\epsilon \implies \delta_\epsilon < x - 0 < \delta_\epsilon$. \\
\begin{center}
$\frac{-\epsilon}{B} < x < \frac{\epsilon}{B}$ \\
\vspace{.2cm}
$-\epsilon < x \cdot B < \epsilon$ \\
\vspace{.2cm}
$\abs{x \cdot B} < \epsilon$
\end{center}

Since we said $g(x) < B$, $\forall x \in \mathbb{R}$, we know the following inequality to be true: $x \cdot g(x) < \abs{x \cdot B} < \epsilon$. This completes the proof.
\end{flushleft}

\section*{Problem 5}
\begin{flushleft}
Using the \textquotedbl$\epsilon - \delta$\textquotedbl \hspace{.01cm} definition to prove that \\
\begin{enumerate}[(a)]
\item $\displaystyle \lim_{x \to 2} (4x+1) = 9$

\begin{center}
$\abs{f(x) - L} < \epsilon \implies \abs{(4x+1) - 9} < \epsilon \implies \abs{4x-8} < \epsilon \implies  \abs{4(x-2)} \implies \abs{x-2} < \frac{\epsilon}{4}$ \\
\end{center}

\subsection*{Proof}

Let $\delta_\epsilon \implies \frac{\epsilon}{4}$. Then, $0 < \abs{x-2} < \frac{\epsilon}{4} \implies -\frac{\epsilon}{4} < x-2 < \frac{\epsilon}{4}$. \\
\begin{center}
\vspace{.2cm}
$\frac{\epsilon}{4} < x-2 < \frac{\epsilon}{4}$ \\
\vspace{.2cm}
$-\epsilon < 4(x-2) < \epsilon$ \\
\vspace{.2cm}
$-\epsilon < 4x-8 < \epsilon$ \\
\vspace{.2cm}
$-\epsilon < (4x+1)-9 < \epsilon$ \\
\vspace{.2cm}
$\abs{(4x+1)-9} < \epsilon$ \\
\end{center}

This completes the proof because we showed $\abs{f(x) - L} < \epsilon$ for a specific $\delta$. \\

\item $\displaystyle \lim_{x \to 5} \sqrt{x + 4} = 3$

\begin{center}
$\abs{f(x) - L} < \epsilon \implies \abs{\sqrt{x + 4} - 3} < \epsilon$ implies \\
\vspace{.3cm}
$-\epsilon < \sqrt{x + 4} - 3 < \epsilon$ \\
\vspace{.2cm}
$-\epsilon  + 3< \sqrt{x + 4} < \epsilon + 3$ \\
\vspace{.2cm}
$(-\epsilon  + 3)^2 < x + 4 < (\epsilon + 3)^2$ \\
\vspace{.2cm}
$(-\epsilon  + 3)^2 - 9 < x - 5 < (\epsilon + 3)^2 - 9$ \\
\end{center}

You can see that we subtracted 9 to both sides at the end. That is because our $\abs{x-p}$ value is $\abs{x-5}$. \\

\subsection*{Proof}
Let $\delta_\epsilon = (\epsilon + 3)^2 - 9$. Then, $0 < \abs{x-5} < (\epsilon + 3)^2 + 9$ implies \\
\begin{center}
\vspace{.2cm}
$(-\epsilon  + 3)^2 - 9 < x - 5 < (\epsilon + 3)^2 - 9$ \\
\vspace{.2cm}
$(-\epsilon  + 3)^2 < x + 4 < (\epsilon + 3)^2$ \\
\vspace{.2cm}
$-\epsilon  + 3< \sqrt{x + 4} < \epsilon + 3$ \\
\vspace{.2cm}
$-\epsilon < \sqrt{x + 4} - 3 < \epsilon$ \\
\end{center}

This completes the proof because we showed $\abs{f(x) - L} < \epsilon$ for a specific $\delta$. \\

\item $\displaystyle \lim_{x \to 3} \frac{1}{8 - 4x} = \frac{1}{4}$

\begin{center}
$\abs{f(x) - L} < \epsilon \implies \abs{\frac{1}{8 - 4x} - \frac{1}{4}} < \epsilon$ implies \\
\vspace{.3cm}
$-\epsilon < \frac{1}{8 - 4x} - \frac{1}{4} < \epsilon$ \\
\vspace{.2cm}
$-\epsilon + \frac{1}{4} < \frac{1}{-4(x-2)} < \epsilon + \frac{1}{4}$ \\
\vspace{.2cm}
$4\epsilon - 1 > \frac{1}{x-2} > -4\epsilon - 1$ \\
\vspace{.2cm}
$\frac{1}{4\epsilon - 1} > x-2 > \frac{1}{-4\epsilon - 1}$ \\
\vspace{.2cm}
$\frac{1}{4\epsilon - 1} - 1> x-3 > \frac{1}{-4\epsilon - 1} - 1$ \\
\end{center}

You can see that we subtracted 1 to on sides at the end. That is because our $\abs{x-p}$ value is $\abs{x-3}$. \\

\subsection*{Proof}

Let $\delta_\epsilon = \frac{1}{-4\epsilon - 1} - 1$. Then, $0 < \abs{x-3} < \frac{1}{-4\epsilon - 1} - 1$ implies \\
\begin{center}
\vspace{.2cm}
$\frac{1}{4\epsilon - 1} - 1 > x - 3 > \frac{1}{-4\epsilon - 1} - 1$ \\
\vspace{.2cm}
$\frac{1}{4\epsilon- 1} > x - 2 > \frac{1}{-4\epsilon - 1}$ \\
\vspace{.2cm}
$4\epsilon - 1 > \frac{1}{x - 2} > -4\epsilon - 1$ \\
\vspace{.2cm}
$-\epsilon + \frac{1}{4} < \frac{1}{-4(x-2)} < \epsilon + \frac{1}{4}$ \\
\vspace{.2cm}
$-\epsilon < \frac{1}{-4(x-2)} - \frac{1}{4} < \epsilon$ \\
\vspace{.2cm}
$-\epsilon < \frac{1}{8-4x} - \frac{1}{4} < \epsilon$ \\
\end{center}

This completes the proof because we showed $\abs{f(x) - L} < \epsilon$ for a specific $\delta$. \\

\end{enumerate}
\end{flushleft}

%----------------------------------------------------------------------------------------

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                                                                 %
%  GEANT manual in LaTeX form                              %
%                                                                 %
%  Michel Goossens (for translation into LaTeX)                   %
%  Version 1.00                                                   %
%  Last Mod. Jan 24 1991  1300   MG + IB                          %
%                                                                 %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Origin{R.Brun,F.Bruyant,A.McPherson}
\Submitted{01.11.83}                     \Revised{18.11.93}
\Version{Geant 3.16}\Routid{GEOM199}
\Makehead{The volume data structure -- JVOLUM}

\section{The data structure {\tt JVOLUM} and {\tt JGPAR}}

The meaning of the variables in the data structure {\tt JVOLUM} shown
in fig. \ref{fg:geom199-1}
is the following:

\begin{DLtt}{MMMMMMMM}
\item[ISEARC] search flag {\tt [GEOM400], [GEOM410]}:
\begin{DLtt}{MMMM}
\item[$=$0] volume positioning order or ordering by \Rind{GSNEXT}/\Rind{GSNEAR};
\item[$<$0] binary search as defined by \Rind{GSORD};
\item[$>$0] user ordering \Rind{GSUNEA}/\Rind{GUNEAR};
\end{DLtt}
\item[ISHAPE] system shape number (see {\tt [GEOM050]};
\item[NIN] number of volumes contained in the mother volume --
if negative the volume is divided;
\item[NMED] medium number for the volume;
\item[NPAR] number of shape parameters;
\item[NATT] number of attributes;
\item[PAR] array of shape parameters;
\item[IAT] array of attributes;
\item[IAXIS] direction of the division (see {\tt [GEOM130]});
\item[IVO] system volume number;
\item[NDIV] number of divisions ({\tt -NDVMX}, if computed dynamically,
see {\tt [GEOM140]});
\item[C0] coordinate value at which the division starts;
\item[STEP] coordinate step for the division;
\item[NR] user copy number;
\item[IROT] rotation matrix number defining the orientation of the volume
with respect to the mother reference system;
\item[X,Y,Z] position of the volume with respect to the mother reference
system;
\item[KONLY] {\tt ONLY/MANY} flag, see {\tt [GEOM110]} for more information;
\end{DLtt}

\begin{figure}[hbt]
     \centering
     \epsfig{file=eps/geom199-1.eps,width=14cm}
     \caption{Example of geometrical tree structure}
     \label{fg:geom199-1}
\end{figure}


\section{Operational Semantics}

\subsection{Notation}

The state of the evaluator, $\sigma$, is the tuple consisting of \\
packet, $\pi$\\
action\_list, $\alpha$ \\
stored\_action\_list, $\Lambda$ \\
context, $\Gamma$, which is the tuple: \\
$\langle ingress\_port, \ egress\_port, \ physical\_port, \ key, \ meta, \ header\_offset, \ table \rangle$ \\ \\
Compactly, $\sigma = \langle \pi, \alpha, \Lambda, \Gamma \rangle$
\\ \\
We can examine the status of $\pi$ or $\Gamma$ by using the index operator, []. The index itself is context-sensitive: $\pi$ is indexed by some \textit{field}, $\Gamma$ is indexed by one of its substructures. We will use the assignment operator, $\leftarrow$, within the index operator to assign a bitfield to a value or the value of another bitfield.
A field in $\pi$ can be set to a literal value or copied from a bitfield or register. For example:
\begin{equation}
  \pi[\textit{field} \leftarrow \textit{value}]
\end{equation}
\begin{equation}
  \pi[\textit{field} \leftarrow \mathct{r}]
\end{equation}
\begin{equation}
  \pi[\textit{field} \leftarrow \textit{bitfield}]
\end{equation}
A variable in $\Gamma$ is accessed similarly, but uses the dot operator, ., to access its substructures within the index operator. Example:
\begin{equation}
  \Gamma[\Gamma.\mathct{key} \leftarrow \textit{bitfield}]
\end{equation}
One should not confuse the range operator, [), with the index operator. To avoid ambiguity, the range operator \textit{always} appears inside
of the index operator, []. Example:
\begin{equation}
  \pi[field_1 \leftarrow field_2[0, 10)]
\end{equation}

Comments:
\begin{enumerate}
\item Field \textit{field} with a packet, $\pi$, is set to some literal value, \textit{value}.
\item The value of register $\mathct{r}$ is copied into \textit{field}.
\item The value of some bitfield, \textit{bitfield} is copied into \textit{field}.
\item The value of the $\mathct{key}$ register in $\Gamma$ is set to the value of \textit{bitfield}
\item The bits of $field_1$ in $\pi$ are set to the bits 0, up to but not including 10, in $field_2$.
\end{enumerate}

\subsection{Operational Semantics 5-Tuple}
Turbak and Gifford define operational semantics for a language as the 5-Tuple, $\langle CF, \Rightarrow, FC, IF, OF \rangle$. CF is the set of all possible configurations that an abstract machine may be in. $\Rightarrow$ is the transition relation, a function that transitions from one configuration to another. FC is the set of all final configurations, the possible ending configurations of the machine. IF is the input function, a function that translates an input into the beginning state of the machine. Finally, OF is the output function, or the transition of the final state into some useful value or domain \cite{Turbak2008}. In Pip, we define the 5-tuple as follows:\\
pip = $\langle IF, \ CF, \ \Rightarrow, \ FC, \ OF \rangle$\\
IF = $\textrm{program} \times \pi \rightarrow CF$ \\
CF = $\alpha \times \Gamma \times \pi \times Lambda$ \\
$\Rightarrow$ = Defined in section ``Transition Relation''. \\
FC = $(\Lambda = \varnothing) \times (\alpha = \varnothing) \times \pi \times \Gamma[egress\_port \neq 0]$ \\
OF = $FC \rightarrow$ \texttt{serialization}

\texttt{serialization} is the output state of Pip: a textual logging of all states that the machine entered.
\subsection{Transition Relation}
\setlength{\mathindent}{0pt}

\begin{mdframed}
\begin{gather*}
  (\mathct{advance}~n \cdot \bar\alpha), \langle \pi, \Lambda, \Gamma \rangle
  \Rightarrow
  (\bar\alpha), \langle \pi, \Lambda, \Gamma[\mathct{header}
  \leftarrow \mathct{header} + n] \rangle
\\
  (\mathct{clear} \cdot \bar\alpha), \langle \pi, \Lambda, \Gamma \rangle
  \Rightarrow
  (\bar\alpha), \langle \pi, \varnothing, \Gamma \rangle
\\
  (\mathct{drop} \cdot \bar\alpha), \langle \pi, \Lambda, \Gamma \rangle
  \Rightarrow
  \varnothing, \langle \pi, \varnothing, \Gamma [\mathct{out} \leftarrow 0] \rangle
\\
  (\mathct{write} \ \textit{action} \cdot \bar\alpha), \langle \pi, \Lambda,
  \Gamma \rangle
  \Rightarrow
  (\bar\alpha), \langle \pi, \Lambda \cdot \textit{action}, \Gamma \rangle
\\
  (\mathct{goto} \ T \cdot \bar\alpha),
  \langle \pi, \Lambda, \Gamma \rangle
  \Rightarrow
  (\bar\alpha), \langle \pi, \Lambda,
  \Gamma[\mathct{current\_table} \leftarrow T] \rangle
\\
  (\mathct{set} \ \textit{field} \ \textit{value} \cdot \bar\alpha), \langle \pi,
  \Lambda, \Gamma \rangle
  \Rightarrow
  (\bar\alpha), \langle \pi', \Lambda, \Gamma' \rangle
\\
  (\mathct{output} \ \mathct{port} \ n \cdot \bar \alpha), \langle \pi, \Lambda,
  \Gamma \rangle
  \Rightarrow
  (\bar\alpha), \langle \pi, \Lambda, \Gamma[\mathct{egress\_port}
  \leftarrow \mathct{port} \ n] \rangle
\\
  (\mathct{copy} \ as_1 \ src \ as_2 \ dst \cdot \bar\alpha), \langle \pi,
  \Lambda, \Gamma \rangle
  \Rightarrow
  (\bar\alpha), \langle \pi', \Lambda, \Gamma' \rangle
\end{gather*}
\end{mdframed}

The $\mathct{advance}$ command advances the header offset by \textit{n} bits.
This can be used by a programmer to adjust the decoding offset relative to a
packet header.
The $\mathct{clear}$ command clears the stored action list. This can be used
by a program reset any accumulated actions if certain conditions are detected
during packet analysis (e.g., filtering flows based on UDP packet content).
The $\mathct{drop}$ command truncates both the action list and stored action list,
and sets the output port to 0 (the null port).
The effect of this to cause the program to terminate such that the execution
environment will not forward the packet.
The \mathct{write} command appends an action to the stored action list. This is
the only way to write to the stored action list, allowing for some action to be
taken upon egress, providing the packet is not dropped.
The \mathct{goto} command jumps from the current match table to a new table. The
key register is cleared and rewritten based on the new table's rules. This can
be used as a rudimentary control structure.
The \mathct{set} command sets a bitfield, \textit{field} to some value \textit{value}.
This is used to set fields to a literal value.
The \mathct{output} action outputs to the specified \mathct{port}, \textit{n}, beginning
egress processing.
The \mathct{copy} action copies from one bitfield, \textit{src} to another bitfield, \textit{dst}.
A bitfield can be copied from any address space to any other address space, except for key registers,
which cannot be used as a source.

Definition of copy function:
\begin{mdframed}
\begin{gather*}
  (\mathct{copy}  \ meta \ src \ packet \ dst \ n)
  \rightarrow
  \langle \pi[dst \leftarrow \Gamma.\mathct{meta}[dst, dst + n)],
  \Lambda, \Gamma \rangle
\\
  (\mathct{copy} \ meta \ src \ header \ dst \ n)
  \rightarrow
  \langle \pi[dst + \mathct{hoff} \leftarrow \Gamma.meta[dst, dst + n)],
  \Lambda, \Gamma \rangle
\\
  (\mathct{copy} \ meta \ src \ key \ dst \ n)
  \rightarrow
  \langle \pi, \Lambda, \Gamma[\Gamma.\mathct{key} \leftarrow
  \Gamma.\mathct{meta}[meta, meta + n)] \rangle
\\
  (\mathct{copy} \ packet \ src \ packet \ dst \ n)
  \rightarrow
  \langle \pi[dst \leftarrow \pi[src, src + n)], \Lambda, \Gamma \rangle
\\
  (\mathct{copy} \ packet \ src \ header \ dst \ n)
  \rightarrow
  \langle \pi[dst + \mathct{hoff} \leftarrow
  \pi[src, src + n)],
  \Lambda, \Gamma \rangle
\\
  (\mathct{copy} \ packet \ src \ key \ dst \ n)
  \rightarrow
  \langle \pi, \Lambda, \Gamma[\Gamma.\mathct{key} \leftarrow \pi[src, src + n)]
  \rangle
\\
  (\mathct{copy} \ header \ src \ header \ dst \ n)
  \rightarrow
  \langle \pi[dst + \mathct{hoff} \leftarrow
  \pi[src + \mathct{hoff}, src + \mathct{hoff} + n)],
  \Lambda, \Gamma \rangle
\\
  (\mathct{copy} \ header \ src \ packet \ dst \ n)
  \rightarrow
  \langle \pi[dst \leftarrow
  \pi[src + \mathct{hoff}, src +
  \mathct{hoff} + n)], \Lambda, \Gamma \rangle
\\
  (\mathct{copy} \ header \ src \ key \ dst \ n)
  \rightarrow
  \langle \pi, \Lambda, \Gamma[\Gamma.\mathct{key}[dst] \leftarrow
  \pi[src +
  \mathct{hoff}, src + \mathct{hoff} + n)] \rangle
\\
  (\mathct{copy} \ header \ src \ meta \ dst \ n)
  \rightarrow
  \langle \pi, \Lambda, \Gamma[\Gamma.\mathct{meta}[dst] \leftarrow
  \pi[src +
  \mathct{hoff}, src + \mathct{hoff} + n)] \rangle
\end{gather*}
\end{mdframed}

Defintion of set function:
\begin{mdframed}
\begin{gather*}
  (\mathct{set} \ (meta \ field \ n) \ value)
  \rightarrow
  \langle \pi, \Lambda, \Gamma[\Gamma.\mathct{meta}[field, field + n) \leftarrow value] \rangle
  \\
  (\mathct{set} \ (key \ field \ n) \ value)
  \rightarrow
  \langle \pi, \Lambda, \Gamma[\Gamma.\mathct{key}[field, field + n) \leftarrow value] \rangle
  \\
  (\mathct{set} \ (packet \ field \ n) \ value)
  \rightarrow
  \langle \pi, \Lambda, \Gamma[\Gamma.\mathct{packet}[field, field + n) \leftarrow value] \rangle
  \\
  (\mathct{set} \ (header \ field \ n) \ value)
  \rightarrow
  \langle \pi, \Lambda, \Gamma[\Gamma.\mathct{packet}[field + \mathct{hoff},
      field + \mathct{hoff} + n) \leftarrow value] \rangle
  \\
\end{gather*}
\end{mdframed}
\textit{Note:} the context variable $\mathct{header\_offset}$ is abbreviated to $\mathct{hoff}$ for brevity.


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{tikz}
\usepackage{tkz-graph}
\usepackage{float}
\usepackage{amsmath}
\GraphInit[vstyle = Shade]

\title{Assignment 2}
\author{Jason Fantl, Jacob Tuttle}
\date{November 2019}

\begin{document}

\maketitle

\section{Theory vs. Practice}

3 reasons why asymptotic analysis may be misleading with respect to actual performance in practice:
\begin{itemize}
  \item Because asymptotic analysis ignores constants, it can lead to misrepresenting which algorithm is more efficient for specific situations. For example, if one algorithm runs in O(n) time and another runs in O($n^2$) time, you would assume the O(n) algorithm is more efficient. But if you also know that constants are included, and that the linear time algorithm would run in $10000n$ time, and the other in $n^2$ time, the more efficient algorithm depends on your input size, not just asymptotic time complexity.
  \item In practice there are always restrictions on both memory and time, and asymptotic analysis only considers how well an algorithm scales, not how well it will fit within the given conditions. This could lead to using an algorithm that scales very well but fails to complete the task at all within the available constraints. For example, if an algorithm were to rely on a large volume of memory access, when run on a computer with very efficient memory access techniques, these actions would have little effect on the runtime; however, if it were run on a device with much less efficient memory access, the execution time would balloon as a result that has nothing to do with the $O$ complexity.
  \item While asymptotic analysis will provide information about how the algorithm will behave in the abstract sense, it says nothing about how the code will run on a specific machine. Even if one were to run the algorithm on one machine, and use this data to extrapolate the runtimes for various input sizes, that information would mean nothing if the algorithm were run on another machine. The asymptotic runtime of an algorithm says very little about the true execution time for an algorithm, and it provides even less information when considering the differences of running on different machines.
\end{itemize}

Given a binary search tree with $1,000$ elements and a measured time of $5$ seconds to access one of those elements, we could use asymptotic analysis to predict how long it would take to find in element in a binary search tree with $10,000$ elements.
We know the time complexity of a searching in a binary search tree is $O(log(n))$. We can find the ratio of time it should take between different trees using this fact. The ratio of time between a tree with $1,000$ elements and a tree with $10,000$ should be $log_2(1,000)/log_2(10,000) = 0.75$. Using this calculation and the fact that the $1,000$ element tree took $5$ seconds, we predict it should take $\frac{5}{0.75} = 6.67$ seconds to find an element.

If the measured time of the two binary search trees doesn't match our prediction, here are three reasons why that might have happened:
\begin{itemize}
  \item Since $O$ only describes the highest order term, disregarding constants and lower order terms, one of these discarded terms may have been much larger than was initially considered. If one of these constants or lower order terms were to be quite large, increasing the input size could change the run time more significantly than calculations indicate.
  \item Depending on what form the input binary tree takes, the worst case time complexity of searching can be $O(n)$. It's possible the first execution, the $5$ second one, ran on an average or best case tree and the second execution ran on a worst case input. This would result in a much larger execution time since the worst case takes asymptotically more time than the average case.
  \item If the algorithm were run on different computers, then even if one were to run the same algorithm on the same tree, different run times could be produced. When run on a slower computer, this difference would be even more pronounced and could result in the difference between calculated and observed run times discussed in this problem.
\end{itemize}

\section{Graph Properties}

Definitions:
\begin{itemize}
\item Isomorphic: G1 = (V1, E1) is isomorphic to G2 = (V2, E2) if there is a
one-to-one and onto function $f : V1 \rightarrow V2$ such that
$(u, v) \in E1 \iff (f(u), f(v)) \in E2$
\item Completely connected: A graph is completely connected if there exists an edge between every pair of vertices.
\item One-to-One function: A function f is one-to-one if for all x and y f(x) = f(y) then x = y.
\item Onto function: A function f is onto if for all y in the range of f there exists an x such that f(x) = y
\end{itemize}

%Theorem: If two graphs A and B have the same number of nodes and are completely connected, they are isomorphic.

%Proof:
%We have a completely connected graph G1 = (V1, E1), and a completely connected graph G2 = (V2, E2) both with x vertices. We will label each vertex of graph G1 from V1 to Vx since it has x vertices. Then for each vertex, it has x edges since a completely connected graph connects every pair of vertices, and there are x vertices. For each edge edge coming out of vertex Vn, connecting to Vm, we will label that edge as En,m. Since G2 has the same number of vertices and is also completely connected, we will label all of its vertices and edges in the same exact way. Now there exists a simple function f which takes an edge or vertex and returns the same vertex or edge, that is f(u) = u and f(v) = v. This function is one-to-one since for all Pn and Qm where Pn and Qm are arbitrary vertices or edges, if f(Pn) = f(Qm) then Pn = Qm by the definition of f. f is also onto since for any value I in the range of f we pick a new value J = I, then f(J) = I by the definition of f.

%To be continued \\

Theorem: If two graphs A and B have the same number of nodes and are completely connected, they are isomorphic. \\

Proof: Let $A$ and $B$ be two graphs that are completely connected and have the same number of nodes. By the definition of two completely connected graphs, we can define $A$ as a set of vertices $V_A$ labeled $a_1, a_2, \hdots a_n$ and a set of edges $E_A : (a_1, a_2), (a_1, a_3), \hdots (a_1, a_n), (a_2, a_3), (a_2, a_4), \hdots (a_2, a_n), \hdots (a_{n-1}, a_n)$. Likewise, $B$ is defined in a similar fashion with vertices $V_B : b_1, b_2, \hdots b_n$ and edges $E_B : (b_1, b_2), (b_1, b_3), \hdots (b_1, b_n), (b_2, b_3), (b_2, b_4), \hdots (b_2, b_n), \hdots (b_{n-1}, b_n)$. For these two graphs to be isomorphic, there must be a one-to-one and onto function $f: V1 \rightarrow V2$ such that $(u, v) \in E_A \iff (f(u), f(v)) \in E_B$ \\

If we can find this bijection, $f$, such that $(u, v) \in E_A \iff (f(u), f(v)) \in E_B$ then it will be shown that $A$ and $B$ are isomorphic. Let $f$ be defined as the function such that:
\begin{align*}
    f(a_1) &= b_1 \\
    f(a_2) &= b_2 \\
    &\vdots \\
    f(a_n) &= b_n
\end{align*}

$f$ is onto and one-to-one as we have defined it because every node in $V_A$ can be mapped by $f$ to a single corresponding node in $V_B$.

Using this function, it can be shown that every edge in $A$ corresponds to the same edge $(f(a_i), f(a_j)) = (b_i, b_j)$ for all $i, j \leq n$. Therefore, $A$ and $B$ are isomorphic. \\

% Old proof that is added
%Theorem: If two graphs A and B are isomorphic they do not need to be completely connected.

%Proof: Let there be graphs G1 and G2 such that both have 3 vertices, called A1, B1, and C1 for G1, and A2, B2, and C2 for G2. Let G1 only have edges from A1 to B1 called AB1 and B1 to C1 called BC1, and let G2 only have edges from A2 to B2 called AB2 and B2 to C2 called BC2. Neither of these graphs are fully connected since there isn't an edge from A1 to C1 and there isn't an edge from A2 to C2. These graphs are isomorphic since the edge AB1 can be mapped to AB2 and BC1 mapped to BC2, and the edge and AB2 can be mapped to AB1 and BC2 mapped to BC1. This exhaustively finds a mapping between the sets of edges. By this example, we have found two graphs which are isomorphic and not fully connected, proving the theorem.*/

Theorem: If two graphs $A$ and $B$ are isomorphic, they do not need to be completely connected. \\

Proof: Let $A$ and $B$ be the graphs by Figure 1 and Figure 2 respectively and be defined as:

\begin{align*}
    A &= (V_A, E_A) \\
    V_A &= \{a, b, c\} \\
    E_A &= \{(a, c), (c, b)\} \\
    B &= (V_B, E_B) \\
    V_B &= \{x, y, z\} \\
    E_B &= \{(x, y), (y, z)\}
\end{align*}

\begin{figure}[H]
    \centering
    \begin{tikzpicture}[shorten >=1pt, auto, node distance=3cm]

    % draw nodes
    \foreach [count=\i] \x/\y/\t in {0/0/c, 0/3/a, 2/2/b}
        \node [circle, draw]
            (v\t) at (\x, \y) {\t};

    % draw edges
    \foreach \i/\j in {a/c, c/b}
        \draw (v\i) edge[post] (v\j);

    \end{tikzpicture}
    \caption{Graph $A$}
    \label{fig:graph_a}
\end{figure}
\begin{figure}[H]
    \centering
    \begin{tikzpicture}[shorten >=1pt, auto, node distance=3cm]

    % draw nodes
    \foreach [count=\i] \x/\y/\t in {0/0/x, 3/0/y, 6/0/z}
        \node [circle, draw]
            (v\t) at (\x, \y) {\t};

    % draw edges
    \foreach \i/\j in {x/y, y/z}
        \draw (v\i) edge[post] (v\j);

    \end{tikzpicture}
    \caption{Graph $B$}
    \label{fig:graph_b}
\end{figure}
Let $f$ be a one-to-one and onto bijection from $A \rightarrow B$ such that:
\begin{align*}
    f(a) &= x \\
    f(b) &= z \\
    f(c) &= y
\end{align*}
Using the definitions above, it can be seen that all edges are mapped from $E_A$ to $E_B$ by $f$ and $f$ is one-to-one and onto from $A \rightarrow B$. Furthermore, in graph $A$ nodes $a$ and $b$ are not connected and in graph $B$ nodes $x$ and $z$ are not connected; therefore, the graphs are not completely connected. These graphs satisfy the theorem as two graphs that are isomorphic and not completely connected; therefore two graphs do not need to be completely connected to be isomorphic.

\section{Ford-Fulkerson --- Augmenting Paths}
Code solutions to this problem are provided in: \\ \texttt{Ford-Fulkerson\_Augmenting\_Paths.js}

\section{Ford-Fulkerson Algorithm }

Residual graph, iteration 1:

\begin{tikzpicture}[shorten >=1pt, auto, node distance=3cm]

   \foreach [count=\i] \x/\y/\t in {0/0/S, 0/4/B, 3/4/C, 3/2/D, 10/0/E, 7/4/F, 10/4/T}
     \node [circle,draw]
        (v\t) at (\x,\y) {\t};

   \foreach \i/\j/\t in {S/B/8, S/D/2, S/E/3, B/C/1, D/C/3, D/F/9, D/E/4, E/F/2, E/T/1, C/F/5, F/T/7}
    \draw (v\i) edge[post] node{\t} (v\j);
\end{tikzpicture}

Residual graph, iteration 2:

\begin{tikzpicture}[shorten >=1pt, auto, node distance=3cm]

   \foreach [count=\i] \x/\y/\t in {0/0/S, 0/4/B, 3/4/C, 3/2/D, 10/0/E, 7/4/F, 10/4/T}
     \node [circle,draw]
        (v\t) at (\x,\y) {\t};

    \foreach \i/\j/\t in {S/B/8, S/D/2, B/C/1, D/C/3, D/F/9, D/E/4, E/F/2, T/E/1, C/F/5, F/T/7}
        \draw (v\i) edge[post] node{\t} (v\j);
    \foreach \i/\j/\t in {S/E/2, E/S/1}
        \draw (v\i) edge[post, bend right=10] node{\t} (v\j);

\end{tikzpicture}

Residual graph, iteration 3:

\begin{tikzpicture}[shorten >=1pt, auto, node distance=3cm]

   \foreach [count=\i] \x/\y/\t in {0/0/S, 0/4/B, 3/4/C, 3/2/D, 10/0/E, 7/4/F, 10/4/T}
     \node [circle,draw]
        (v\t) at (\x,\y) {\t};

    \foreach \i/\j/\t in {S/B/8, S/D/2, B/C/1, D/C/3, D/F/9, D/E/4, F/E/2, T/E/1, C/F/5, E/S/3}
        \draw (v\i) edge[post] node{\t} (v\j);
    \foreach \i/\j/\t in {F/T/5, T/F/2}
        \draw (v\i) edge[post, bend left=10] node{\t} (v\j);

\end{tikzpicture}

Residual graph, iteration 4:

\begin{tikzpicture}[shorten >=1pt, auto, node distance=3cm]

   \foreach [count=\i] \x/\y/\t in {0/0/S, 0/4/B, 3/4/C, 3/2/D, 10/0/E, 7/4/F, 10/4/T}
     \node [circle,draw]
        (v\t) at (\x,\y) {\t};

    \foreach \i/\j/\t in {S/B/8, D/S/2, B/C/1, D/C/3, D/E/4, F/E/2, T/E/1, C/F/5, E/S/3}
        \draw (v\i) edge[post] node{\t} (v\j);
    \foreach \i/\j/\t in {F/T/3, T/F/4, D/F/7, F/D/2}
        \draw (v\i) edge[post, bend left=10] node{\t} (v\j);

\end{tikzpicture}

Residual graph, iteration 5:

\begin{tikzpicture}[shorten >=1pt, auto, node distance=3cm]

   \foreach [count=\i] \x/\y/\t in {0/0/S, 0/4/B, 3/4/C, 3/2/D, 10/0/E, 7/4/F, 10/4/T}
     \node [circle,draw]
        (v\t) at (\x,\y) {\t};

    \foreach \i/\j/\t in {D/S/2, C/B/1, D/C/3, D/E/4, F/E/2, T/E/1, E/S/3}
        \draw (v\i) edge[post] node{\t} (v\j);
    \foreach \i/\j/\t in {S/B/7, B/S/1, C/F/4, F/C/1, F/T/2, T/F/5, D/F/7, F/D/2}
        \draw (v\i) edge[post, bend left=10] node{\t} (v\j);

\end{tikzpicture}

Original graph: Max flow of 6

\begin{tikzpicture}[shorten >=1pt, auto, node distance=3cm]

   \foreach [count=\i] \x/\y/\t in {0/0/S, 0/4/B, 3/4/C, 3/2/D, 10/0/E, 7/4/F, 10/4/T}
     \node [circle,draw]
        (v\t) at (\x,\y) {\t};

    \foreach \i/\j/\f/\t in {S/B/1/8, S/D/2/2, S/E/3/3, B/C/1/1, D/C/0/3, D/F/2/9, D/E/0/4, E/F/2/2, E/T/1/1, C/F/1/5, F/T/5/7}
        \draw (v\i) edge[post] node{\f/\t} (v\j);

\end{tikzpicture}

\section{Kruskal’s Algorithm }
Minimum spanning tree, iteration 1:

\begin{tikzpicture}[shorten >=1pt, auto, node distance=3cm]

   \foreach [count=\i] \x/\y/\t in {0/0/A, 0/4/B}
     \node [circle,draw]
        (v\t) at (\x,\y) {\t};

    \foreach \i/\j/\t in {A/B/1}
        \draw (v\i) edge node{\t} (v\j);

\end{tikzpicture}

Minimum spanning tree, iteration 2:

\begin{tikzpicture}[shorten >=1pt, auto, node distance=3cm]

   \foreach [count=\i] \x/\y/\t in {0/0/A, 0/4/B, 10/0/E, 7/4/F}
     \node [circle,draw]
        (v\t) at (\x,\y) {\t};

    \foreach \i/\j/\t in {A/B/1, E/F/2}
        \draw (v\i) edge node{\t} (v\j);

\end{tikzpicture}

Minimum spanning tree, iteration 3:

\begin{tikzpicture}[shorten >=1pt, auto, node distance=3cm]

   \foreach [count=\i] \x/\y/\t in {0/0/A, 0/4/B, 3/4/C, 3/2/D, 10/0/E, 7/4/F}
     \node [circle,draw]
        (v\t) at (\x,\y) {\t};

    \foreach \i/\j/\t in {A/B/1, E/F/2, A/E/3, D/C/3}
        \draw (v\i) edge node{\t} (v\j);

\end{tikzpicture}

Minimum spanning tree, iteration 4:

\begin{tikzpicture}[shorten >=1pt, auto, node distance=3cm]

   \foreach [count=\i] \x/\y/\t in {0/0/A, 0/4/B, 3/4/C, 3/2/D, 10/0/E, 7/4/F, 10/4/G}
     \node [circle,draw]
        (v\t) at (\x,\y) {\t};

    \foreach \i/\j/\t in {A/B/1, E/F/2, A/E/3, D/C/3, E/G/4, D/E/4}
        \draw (v\i) edge node{\t} (v\j);

\end{tikzpicture}

Minimum spanning tree, iteration 5:

\begin{tikzpicture}[shorten >=1pt, auto, node distance=3cm]

   \foreach [count=\i] \x/\y/\t in {0/0/A, 0/4/B, 3/4/C, 3/2/D, 10/0/E, 7/4/F, 10/4/G, 7/2/H}
     \node [circle,draw]
        (v\t) at (\x,\y) {\t};

    \foreach \i/\j/\t in {A/B/1, E/F/2, A/E/3, D/C/3, E/G/4, D/E/4, D/H/9}
        \draw (v\i) edge node{\t} (v\j);

\end{tikzpicture}
\end{document}