\documentclass[journal]{IEEEtran}

\usepackage[style=ieee]{biblatex}
\usepackage{amsmath}
\usepackage{url}
\usepackage{graphicx}
\usepackage{float}

\usepackage{blindtext, subfig}
\usepackage{dblfloatfix}

% \usepackage[caption=false]{subfig}
\usepackage{todonotes}

\bibliography{ParticleSizeDistribution.bib}

\begin{document}

\title{Investigating Cross-Sectional Size Distribution of Randomly Distributed
       3D Spheres}

\author{Sai Pandian, ID:\@ 29899923}%

% The report headers
\markboth{PHYS6017 Computer Techniques in Physics Report 2, May 2020}
{Shell \MakeLowercase{\textit{et al.}}: Bare Demo of IEEEtran.cls for IEEE Journals}

\maketitle

\begin{abstract}
  In this paper we present a 3-dimensional numerical simulation demonstrating
  that it is possible to reproduce the known planar cross-section distributions
  for Wicksell's Corpuscle Problem, thereby recovering 3D particle size
  distributions from 2D cross-sections. We investigate the effects of different
  particle sizes on the experimental data, and show this method is accurate for
  2 different starting 3D size distributions. We also present our findings on
  how sensitive this method is to experimental data and discuss its efficacy in
  a real world application such as identifying a material using a planar
  cross-section image from a microscope.
\end{abstract}

\section{Introduction}

\IEEEPARstart{I}{n} the world of Material Science and Crystallography, it is
often the case that a material is identified and classified using the
distribution of its constituent particle sizes. However, direct observation of
the particles is often impossible. Using a modern microscope or laser imaging
techniques, it is possible to observe particles in a cross-section of
material. But the observed radii of particles in this cross-section do not
necessarily correspond directly to the radii of the particles in the material,
since the plane is not oriented such that it passes through the centre of all
observed particles. Thus, it becomes difficult to relate the observed
distribution of particle sizes in the plane to the real distribution of particle
sizes in the material.

Mathematically, determining the distribution of spherical particles from planar
sections is a well known problem called Wicksell's Corpuscle Problem
\cite{DWicksell}, named after S. D. Wicksell, who solved the problem in
1925. With this solution, it is possible to determine the distribution of
particle sizes in a solid from its observed cross-section, allowing us to
identify and classify materials better \cite{Cuzzi2016}.

In this paper we aim to show that it is possible to reproduce theoretical
results for cross-sectional distributions using numerical simulation, and it is
thus possible to solve Wicksell's Corpuscle Problem in this manner.

\section{Method}

The numerical simulation of the 3D particle distribution was done using
Python. It looked at many spheres of different radii, distributed randomly in
space with a given "box of length 20. The spheres were not allowed to overlap,
which we ensured by placing the spheres such that if one sphere has vector
position $\overrightarrow{\textbf{a}}$ and radius $r_{1}$ and the other has
vector position $\overrightarrow{\textbf{b}}$ and radius $r_{2}$, the spheres
must have positions such that:
\begin{equation*}
|\overrightarrow{\textbf{a}} - \overrightarrow{\textbf{b}}| > (r_{1} + r_{2})
\end{equation*}
Random coordinates for a sphere were drawn from a uniform distribution and
checked against every existing sphere until a set of coordinates was drawn that
ensures no overlap, at which point the sphere was placed in space and the process
was repeated until all the necessary spheres have been placed.

\begin{figure}%[H]%[!ht]
\begin{center}
\includegraphics[width=0.45\textwidth]{images/box3d.pdf}
\caption{$750$ spheres of radius 0.7 distributed randomly in a cube of length 20,
  such that no sphere is touching or overlapping another sphere. A horizontal
  plane, in blue, passes through the box and ``cuts'' through the
  spheres.}\label{fig:3dplot_plane}
\end{center}
\end{figure}

We then placed a plane into the cube, with a random orientation, which
``cuts'' through many of the spheres in the cube. This is shown in
Figure~\ref{fig:3dplot_plane}, where we see a plane (in blue) intersecting 750
spheres of constant radius, distributed at random in a box.

In order to obtain the planar cross-section, we employed the spherical cap
method \cite{Sykes1922}. If we consider a sphere with centre
$\overrightarrow{\textbf{c}_0} = (x_0, y_0, z_0)$ and radius $R$, and a plane
with equation $Ax + By + Cz = D$ such that $\overrightarrow{\textbf{n}} = (A, B,
C)$ is the normal vector to the plane, then the distance from the centre of the
circle to the closest point on the plane $\overrightarrow{\textbf{p}_0}$ is given by $\rho$:
\begin{equation*}
\rho = \frac{(\overrightarrow{\textbf{c}_0} - \overrightarrow{\textbf{p}_0}) \cdot{}
  \overrightarrow{\textbf{n}}}{|\overrightarrow{\textbf{n}}|} = \frac{Ax_0 +
  By_0 + Cz_0 - D}{\sqrt{A^2 + B^2 + C^2}}
\end{equation*}
If $|\rho| < R$, then there is an intersection of the plane and the sphere. The
radius of the circle $r$ in the planar cross-section is given by:
\begin{equation*}
r = \sqrt{R^2 - \rho^2}
\end{equation*}
And we then binned this radius to produce a histogram of the distribution of
planar circle radii. It is also possible to recover the centre
$\overrightarrow{\textbf{c}}$ of the circle visible on the plane using:
\begin{equation*}
\overrightarrow{\textbf{c}} = \overrightarrow{\textbf{c}_0} +
\rho\frac{\overrightarrow{\textbf{n}}}{|\overrightarrow{\textbf{c}}|} = (x_0,
y_0, z_0) + \rho\frac{(A, B, C)}{\sqrt{A^2 + B^2 + C^2}}
\end{equation*}

\begin{figure}%[H]%[!ht]
\begin{center}
\includegraphics[width=0.45\textwidth]{images/circles.pdf}
\caption{Planar cross-section constructed using mock data. Spheres cut by the
  plane appear as circles on the plane, with radius dependent on the relative
  position of the spheres to the plane. This distribution of the radii of the
  circles can be modelled.}\label{fig:circles}
\end{center}
\end{figure}

Figure~\ref{fig:circles} presents a planar cross-section generated using this
method and mock data. As we see, there are a range of circular radii observed
since even though the spheres all had the same radius, they were at different
distances from the plane, and were thus cut at different points. The many
observed circular radii are binned to produce a distribution.

If $f(R)dR$ is the number of spheres per unit volume with radii in interval $[R,
R + dR]$, then we can define a distribution $\phi(x)dx$ which represents the
number of circles per unit area on the plane with radii in interval $[x,
x+dx]$ \cite{Kiderlen2011}. This distribution is given by:
\begin{equation}
\phi(x) = \int_{x}^{\infty}\frac{xf(R)}{\sqrt{R^2 - x^2}}
\label{eq:phi_OG}
\end{equation}
We can compute this integral to obtain a theoretical planar distribution for
different starting spherical radius distributions.

We first considered the case in which all the spheres had the same radius
$R_0$. In this case, the spherical radius distribution is simply a Dirac delta
function centred on the radius $R_0$.

If we compute the integral in Equation~\ref{eq:phi_OG}, replacing $f(R)$ with
$\delta(R-R_0)$ we find:
\begin{equation}
\phi(x) = \int_{x}^{\infty}\frac{x\delta(R-R_0)}{\sqrt{R^2 - x^2}} =
\frac{xH(R_0-x)}{\sqrt{R_0^2-x^2}}
\label{eq:phi_constant}
\end{equation}
where $H(R_0 - x)$ is the Heaviside function, which ``turns on'' the
distribution only for $x < R_0$ as it is not possible to have a circular radius
$x$ larger than the spherical radius. So we aim to see our experimental data
reproduce this distribution \cite{Kong2005}. We also investigated how changing
the size of the spheres used in the simulation affects the experimental data.

We next considered the case in which each sphere has a radius $R$ uniformly
distributed in a range $R_1 < R < R_2$. In this case, we modify $f(R)$ in
Equation~\ref{eq:phi_OG} to be:
\begin{equation*}
  f(R) =
  \begin{cases}
    1,& \text{if } R_1 < R < R_2\\
    0,& \text{} Otherwise
  \end{cases}
\end{equation*}
In this case the integral in Equation~\ref{eq:phi_OG} does not need to be
computed with the limits shown, but instead with limits between $R_1$ and $R_2$
since $f(R)$ is 0 otherwise:

\begin{equation}
\begin{split}
  \phi(x) & = \int_{R_1}^{R_2}\frac{xf(R)}{\sqrt{R^2 - x^2}} \\
  = x\ln\left(\sqrt{R_2^2 - x^2}+R_2\right) & - x\ln\left(\sqrt{R_1^2 - x^2}+R_1\right)
\end{split}
\end{equation}
Since this distribution produces complex numbers for $ x < R_1$, we only
consider the real part of this distribution, which appears to produce a peak at
$x = R_1$.

\section{Results and Discussion}

We found that for both the case of the spheres of constant radius, and the case
of the spheres of varying radius, the experimental data reproduced a
distribution in the same shape as the theoretical distribution.

We first investigated the case where we placed $750$ spheres of radius $0.1$
each in a cube of length $20$. In this case we found that a plane in a random
orientation intersects very few spheres, since the spheres are very small. The
planar distribution is presented in Figure~\ref{fig:size1}.

\begin{figure}%[H]%[!ht]
\begin{center}
\includegraphics[width=0.45\textwidth]{images/750_01.pdf}
\caption{Distribution of circle radii on plane for $750$ spheres of radius $0.1$
  each. There are relatively few points since the spheres are small, so there is
  relatively small chance that they will be cut by the plane. So only a few
  circles are visible on the plane. The theoretical distribution is shown, with
  the experimental data matching this distribution, showing a slow increase in
  number or circles as radius increases.}\label{fig:size1}
\end{center}
\end{figure}

We observe relatively few experimental data points, as there were relatively few
intersections with the plane. This was likely because the spheres' radii were
very small compared to box length and there were relatively few spheres. While
the theoretical distribution shows a strong increase near $0.1$, this is not
evident from the experimental data. We also see that there is a lot of variance
between the points so the slow rise evident in the theoretical distribution is
not clear in the experimental data. It is thus necessary that we have more
spheres that intersect with the plane, in order to obtain a larger data-set.

We found that increasing the spheres' size or placing more spheres in the box,
or making the box length smaller, are all good methods of increasing the number
of intersections we observe, as they all increase the likelihood of intersection
by increasing the ratio of sphere occupied space to total box space. We thus
next used $750$ spheres of radius $0.3$ each. The planar cross-sectional
distribution found is presented in Figure~\ref{fig:size3}.

\begin{figure}%[H]%[!ht]
\begin{center}
\includegraphics[width=0.45\textwidth]{images/750_03.pdf}
\caption{Distribution of circle radii on plane for $750$ spheres of radius $0.3$
  each. Compared to Figure~\ref{fig:size1}, there are many more experimental data
  points, since it is more likely that the plane will intersect these larger
  spheres. The experimental data more clearly resembles the theoretical
  distribution, showing a strong increase in number of circles as radius
  increases towards the maximum of 0.3.}\label{fig:size3}
\end{center}
\end{figure}

In Figure~\ref{fig:size3} we see that there are clearly more experimental data
points. This is because more spheres had intersections with the plane. Compared
to Figure~\ref{fig:1noise}, the slow rise in the theoretical distribution is
more closely followed by the experimental data. However, we still encounter
significant variance and fluctuation in lower radius bins, even though the
theoretical data is a smooth gradual increase. This is likely due to the
relatively few number of circles populating these bins, causing greater relative
difference between neighbouring bins. One method to solve this would be to use
fewer bins, which would have the effect of ``summing and averaging''
neighbouring bins so the experimental data has a smoother trend. However, this
would also reduce the number of data points so it was not implemented in our
investigation.

The sharp rise in the theoretical distribution close to the maximum radius is
also clear in the experimental data. It can be understood intuitively that we
expect to see more cross-sections with radius close to radius of the spheres
since there is a greater probability of intersection with the centre of a sphere
than any other point on the sphere.

\begin{figure}%[H]%[!ht]
\begin{center}
\includegraphics[width=0.45\textwidth]{images/750_07.pdf}
\caption{Distribution of circle radii on plane for $750$ spheres of radius $0.7$
  each. We do not observe much difference in how well the experimental data
  resembles the theoretical distribution between this figure and
  Figure~\ref{fig:size3}. With larger sized spheres, the simulation becomes harder
  to run as it is more difficult to randomly place spheres in a finite space
  without overlap.}\label{fig:size7}
\end{center}
\end{figure}

\begin{figure}[H]%[!ht]
\begin{center}
\includegraphics[width=0.45\textwidth]{images/750_07_random.pdf}
\caption{Distribution of circle radii on plane for $750$ spheres of radius $R$
  where $0.65 < R < 0.75$. The theoretical distribution is now different,
  although the experimental data still resembles this new distribution, showing
  a maximum around $R = 0.7$ and a sharp decrease after this. This is likely
  because there are more spheres with radii such that an overlap between $0$ and
  $0.7$ is possible, and relatively few spheres with radii such that a larger
  overlap is possible.}
\label{fig:random}
\end{center}
\end{figure}

\begin{figure*}[ht!]
  \centering
  \subfloat[Noise 2 orders of magnitude smaller than radius]
  {\includegraphics[width=0.45\textwidth]{images/750_07_Noise_100.pdf}
  \label{fig:2noise}}
  \centering
  \subfloat[Noise 1 order of magnitude smaller than radius]
  {\includegraphics[width=0.45\textwidth]{images/750_07_Noise_10.pdf}
  \label{fig:1noise}}
  \caption{Distribution of circle radii on plane for $750$ spheres of radius $0.7$
    each. Each recorded radius has a random noise added to it in order to
    determine how sensitive the data is to experimental
    error. Figure~\ref{fig:2noise} shows the distribution for an added noise 2
    orders of magnitude smaller than the recorded radius, while
    Figure~\ref{fig:1noise} is the same for noise 1 order of magnitude smaller. As
    we can see, the experimental data very resembles the theoretical
    distribution when noise of 2 orders of magnitude smaller are added. But this
    is not true when noise of 1 order of magnitude smaller is added, implying
    the data is very sensitive to experimental data.}
  \label{fig:noiseplots}
\end{figure*}

Increasing the size of the spheres further to a radius of $0.7$, we might
initially expect the experimental data to resemble the theoretical distribution
even more closely, but this is not what we see. Figure~\ref{fig:size7} presents
the planar distribution for $750$ spheres of radius $0.7$ each.

In Figure~\ref{fig:size7}, we see a similar experimental distribution to that of
Figure~\ref{fig:size3}, as opposed to a distribution that more closely resembles
the theoretical distribution. While initially this may seem contrary to the
notion that increasing size of spheres would increase the number of
intersections and thus give a better distribution, this may simply be due to the
fact that increasing the size of the spheres beyond a certain radius does not
provide continually better results. We observe a similar number of intersections
as the case with spheres of radius $0.3$, and thus it is possible to conclude
that the best experimental distribution is found for a spherical size that
ensures a significant number of intersections. Though continually increasing
spherical size provides diminishing returns.

Indeed, we found that making the sphere sizes much larger made the placing of
the spheres in a finite space more difficult, and exponentially increased the
time complexity of the placing algorithm. This is because if the placed spheres
are too large, while it is initially easy to place spheres, as more are placed,
more coordinates must be randomly drawn before one if found for the next sphere
to be placed into. It is also possible that the algorithm never finds a possible
coordinate for later spheres. One possible solution to this may be to use a
better placing algorithm. Choosing a tight packing algorithm \cite{Chen2001} would
maximise the number of spheres of a given radius that can be placed. However,
this would likely result in a less than random distribution of spheres if the
spheres are all of identical sizes. This would likely cause some systematic
error in the data, and so it was not investigated here. If we randomised the
sizes of the spheres as we do in the next part of the investigation, the problem
become significantly more complicated. Tight packing algorithms for irregular
shaped objects are very difficult to create and computationally expensive
\cite{Ma2018}.

Figure~\ref{fig:random} presents the experimental data for 750 spheres with each
sphere having a radius $0.65 < R < 0.75$. As we can see, the theoretical
distribution has a very different shape to the case of constant sphere
radius. We see that the theoretical distribution has a sharp peak centred around
(but not precisely at) $0.65$, the lower limit of the uniform range the radii
were drawn from, and a sharp decrease after this point. This is likely due to
the higher number of spheres available in the box which could be intersected so
as to show a planar cross-section with radius $\approx 0.65$. In a range of
radii between $0.65 < R < 0.75$, all the spheres in the box can be intersected
so as to produce a circle of radius $0.65$, but there will be fewer spheres
which can be intersected to produce a circle of radius greater than this. So a
peak at this point and then a sharp decrease is expected.

The experimental data resembles the theoretical distribution, clearly showing a
peak and sharp decrease after. However, we also see that there is significant
variance between bins, and there is a large scatter around the theoretical
data. As explained for Figure~\ref{fig:size3}, it would be possible to reduce
this fluctuation by using fewer bins, at the cost of having fewer data
points. With a more powerful computer, or if the simulation is optimised and
written with a compiled language, there is also scope for significantly
increasing the size and number of spheres used, which would provide better
experimental data.

While there is significant fluctuation and scatter in all of the simulations
presented so far, the shapes of the distributions were mostly clear, and they
resembled the theoretical distribution. However, if the system is very sensitive
to experimental error, it may be difficult to identify distribution shapes, and
thus prevent us from properly identifying the starting distribution of spherical
radii. We therefore introduced some additional random error into the data. After
the radii of the circles on the plane are calculated, a small random number is
added to each radius. The random number is drawn from a uniform distribution in
the range $[\frac{-r}{n}, \frac{r}{n}]$, where r is the circle's radius and n is
the number of orders of magnitude we wish to choose the errors. We tested the
system with errors 2 orders of magnitude smaller than the radii and 1 order of
magnitude smaller than the radii. This is presented in
Figure~\ref{fig:noiseplots}.

Figure~\ref{fig:2noise} presents the distribution for $750$ spheres or radius
$0.7$ each, with an added ``noise'' that is 2 orders of magnitude smaller than
each measured radius. We can see that the experimental data looks very similar
to the data presented in Figure~\ref{fig:size7}. The shape of the distribution
is also very clear and resembles the theoretical distribution.

Figure~\ref{fig:1noise} presents the same simulation but with an added noise
that is 1 order of magnitude smaller than each measured radius. Here we can see
that the experimental data is unrecognisable, and the shape of the distribution
is no longer clear. We thus conclude that the system is very sensitive to noise
greater than 2 orders of magnitude compared to the measured data.

However, it would be more prudent to use a Gaussian distribution to draw the
values of the noise. This is because it is likely that the many sources of error
possible in an experimental measurement of particle cross-sections in a material
will be convoluted together into a Gaussian distribution of errors due to the
central limit theorem \cite{Islam2018}. Thus, it would be necessary for any
experimentalist attempting to carry out a similar investigation with real
materials, perhaps to determine porosity \cite{Lekakh}, to carefully consider
all errors in measurements.

\section{Conclusions}
In this investigation we showed that it is possible to reproduce expected planar
distributions for the Corpuscle Problem with a 3-dimensional numerical
simulation of spheres distributed randomly in space, where a plane oriented
randomly intersects the spheres. We investigated how effective this method of
simulation is for different particle sizes, and showed that while initially
larger particles provide better data since they allow for more intersections,
continually increasing particle size becomes ineffective and computationally
expensive.

We showed it was possible to reproduce the expected planar distributions for two
starting spherical radii distributions: a Dirac delta distribution, in which all
the spheres have the same radius, and a bounded uniform distribution, in which
all the spheres' radii are drawn randomly from a uniform distribution. Both
produce different shaped planar distributions but both provide data that
resembles the theoretical distributions.

We also investigated how sensitive this method is to experimental error. We
introduced some random noise into the system from a random uniform distribution
and demonstrated that the system is very sensitive to noise greater than two
orders of magnitude relative to measured data. Thus it is important for any
experimentalist making measurements of particle cross-sections in a material to
be very careful with error handling, as any error larger than 2 orders of
magnitude compared to measured data will result in great difficulty in
reproducing the spherical radii distributions.

We therefore showed in this investigation that it is possible to use the known
solutions to Wicksell's Corpuscle Problem to better identify and classify
materials by using planar measurements to gain information about 3D particle
distributions.

\printbibliography

\end{document}


\documentclass[12pt]{article}
\usepackage[usenames]{color} %used for font color
\usepackage{amsmath, amssymb, amsthm}
\usepackage{wasysym}
\usepackage[utf8]{inputenc} %useful to type directly diacritic characters
\usepackage{graphicx}
\usepackage [english]{babel}
\usepackage [autostyle, english = american]{csquotes}
\MakeOuterQuote{"}
\graphicspath{ {./} }
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\N}{\mathbb{N}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\Q}{\mathbb{Q}}
\newcommand{\prob}{\mathbb{P}}
\newcommand{\degrees}{^{\circ}}

\author{Tianshuang (Ethan) Qiu}
\begin{document}
\title{Math 74, Week 4}
\maketitle

\section{Lec Mon, 1c}
\subsection{a}
Since each term is the product of $x^a, y^b, z^c$, and $a+b+c = 2020$, we can simplify this problem into dogs and biscuits with 2020 biscuits and 3 dogs: $\binom{2020+3-1}{3-1} = \binom{2022}{2}$

\subsection{b}
Before combining, we expand each term by picking one variable from each of the 2020 $(x+y+z)$ multiplied together. So we have $3^{2020}$.

\subsection{c}
This is a "no dogs go hungry" problem. We can just simply pick $x,y,z$ from the first three brackets, feeding each dog a biscuit. Now we have 2017 biscuits with 3 dogs $\binom{2017+3-1}{3-1} = \binom{2019}{2}$.

\subsubsection{Alternate solution}
We can reach the same result by subtracting the amount where there is only $x$, or only $y$, or only $z$, or $xy$, $xz$, $yz$.
\newline
For the first three, there is only 1 way for that to happen since that variable has to be raised to 2020. For $xz$, we have $a+b = 2020$, feeding 2018 biscuits to 2 dogs. We need to subtract 2 since we have already counted having only one term. Therefore $\binom{2018+2-1}{2-1} = 2019$.
\newline
Adding them together we have $1 \times 3 + 2019 \times 3 = 6060$. Now we subtract it from what we had in part (a): $\binom{2022}{2}-6060$ which is the same as $\binom{2019}{2}$
\newpage

\section{Dis Mon, 1a}
LHS is the amount of ways to choose a team with $k$ people and a captain from a group with $n$ people. It chooses the team first: $\binom{n}{k}$. Then from that team we choose a captain with $k$ ways to do it.
\newline
RHS calculates the amount of ways to choose a captain first: $n$, then the rest of the team: $\binom{n-1}{k-1}$. Both sides calculate the same thing. Therefore LHS = RHS.
\newline
Q.E.D.

\section{Dis Mon, 4}
$$x+\frac{1}{x}=7$$
$$(x+\frac{1}{x})^2=49$$
$$x^2+\frac{1}{x^2}+2=49$$
$$x^2+\frac{1}{x^2}=47$$
\newpage

\section{Lec Wed, 1a}

\begin{figure}[h]
    \includegraphics{GRAPH1}
\end{figure}
The sum of these angles is 90$\degrees$.
\newline
Proof: We attempt to move all three angles to $\angle HBC$, since a square has 4 right angles, we can see that $\angle HBC = 90\degrees$.
\newline
We construct a row of 3 identical squares with $A'M' = AM, M'H' = MH, etc$, and connect $DH', H'B$
\newline
Consider $\triangle DAM$, it is a right isosceles triangle, since it is in a square ($DA = AM, DA \perp AM$). Now consider $\triangle DA'H', \triangle H'B'B$. Since all the squares have the same side length, we have $DA' = H'B', A'H'=B'B$. Furthermore, since these are all squares, we have $\angle DA'H' = \angle H'B'B = 90\degrees$.
\newline
Therefore $\triangle DA'H' \cong \triangle H'B'B$ by SAS property. $DH' = H'B, \angle H'B'B = \angle DH'A$. Since the sum of the three angles in a triangle is $180\degrees$ and $\angle H'B'B = DA'H' = 90\degrees$, we can see that $\angle B'H'B + \angle AH'D = 90\degrees$. Finally, since M'H'B' forms a line $\angle M'H'B' = 180 \degrees$, we have $\angle DH'B = \angle DAM = 90 \degrees$.
\newline
Now we have $\triangle DH'B \sim \triangle DAM$ by RAR property for similarity, therefore $\angle AMD = \angle H'BD$.
\newline
By essentially the same logic as $\triangle DA'H' \cong \triangle H'B'B$, we can prove that $\triangle DAH \cong \triangle H'B'B$, so $\angle DHA = \angle B'BH$.
\newline
We have proven that $\angle AMD = \angle H'BD$, $\angle DHA = \angle B'BH$. Since the sum these two and $\angle DBA$ is a right angle, we have proven our claim.
\newpage

\section{Lec Wed, 1b}
\begin{figure}[h]
    \includegraphics[width = 100mm]{GRAPH2.png}
\end{figure}
We reflect F over $AD$ to $F'$, and connect $F'C$. Now we first show that B is the optimal point.
\newline
By the definiton of reflection, we can see that if we were to reflect $F'$ back over $AD$ it will overlap with our original point. This "overlap" causes all three vertices of $\triangle ABF$ to overlap with those of $\triangle ABF'$. By Euclid's definition they are congruent.
\newline
Consider a point on $AD$ that is not $B: B'$. By similar reasoning we can also show that $\triangle AB'F \cong \triangle AB'F'$.
\newline
From these congruencies we see that $B'F = B'F', BF = BF'$. The farmers route can then be converted into $F'B+B'C$ and $F'B + BC$. Furthermore, since F'B'C forms a triangle, we have $F'B+B'C > F'B + BC$ by the triangle inequality, thus proving that B gives the shortest route to the cow.
\newline
Since $FF' \perp AD$ and $CD \perp AD$, we have $FF' \parallel CD$. By the property of alternate interior angles, $\angle FF'C = \angle F'CD$.
\newline
$\angle ABF' + \angle ABC = 180 \degrees$ since F'C is a line. By the same logic, $\angle CBD + \angle ABC  = 180 \degrees$. Therefore $\angle ABF' = DBC$. We have now shown $\triangle ABF' \sim \triangle CBD$ by AAA property.
\newline
$AF = AF' = 2$. Let $AB = x$, $BD$ would then equal $4-x$. By property of similarity we have $\frac{CD}{F'A} = \frac{DB}{AB}$
$$\frac{6}{2} = \frac{4-x}{x}$$
Solving the above equation yields $x=1$.
\newpage

\section{Lec Wed, 2b}
\begin{figure}[h]
    \includegraphics[width = 100mm]{GRAPH3.png}
\end{figure}
Let $AD = CD, CE = EB$, the two segments intersect at O. Connect $BD$, $AE$, $DE$, $CO$, and extend $CO$ to intersect $AB$ at $F$.
\newline
This assumes that 2a ($DE \parallel AB$, $DE = 0.5AB$) has already been proven.
\newline
Since $DE \parallel AB$, the alternate interior angles are equal, $\angle EDB = \angle DBA$, $\angle DEA = EAB$. Since the sum of all the angles of a triangle is $180 \degrees$, the third angle must also be the same. Therefore $\triangle DEO \sim \triangle BAO$ by AAA similarity.
\newline
Because $DE = 0.5AB$, $DO = 0.5BO, EO = 0.5 AO$ by the principles of similarity.
\newline
Using $DE \parallel AB$ again, we can see that $\angle EGF = \angle AFG$. By the same reasoning as above we can see that $\triangle GOE \sim \triangle FOA$ by AAA similarity. Therefore $$\frac{GE}{AF} = \frac{EO}{OA} = \frac{1}{2}$$
Since $DE \parallel AB$, we can show that $\angle CGE = \angle CFB$, $\angle CEG = \angle CBF$ due to the corresponding angles being equal. $\triangle CGE \sim \triangle CFB$ by AAA similarity. Since E is the midpoint of BC, $CE = 0.5 CB$, then by the principle of similarity, $GE = 0.5FB$
\newline
Since $GE = \frac{1}{2}FB = \frac{1}{2}AF$, $AF = FB$. Therefore all three medians intersect at point $O$.
\newpage

\section{Dis Wed, 1a}
\begin{figure}[h]
    \includegraphics[width = 100mm]{GRAPH7.png}
\end{figure}
Since these are all squares we have $DA = AM = BC = CJ$. And since $DM$ connects the diagonal of a square, $\angle DMA = \angle ADM = 45 \degrees$.
\newline
Let $DA$ have length $x$, then $DM$ would have length $\sqrt 2 x$. Since $\angle DMA = 45 \degrees$, its compliment $\angle DMH = 135 \degrees$. $MB$ has length $2x$, and $DM$ has length $x$.
\newline
$\frac{MH}{DM} = \frac{DM}{MB} = \frac{1}{\sqrt2}$, and since $\triangle DMH, \triangle BMD$ both have angle DMH, $\triangle DMH \sim \triangle BMD$ (RAR similarity), and $\angle DHM = \angle BDM$.
\newline
Now we prove a congruency between two right triangles. From the squares we have $AD = CB = x$,  $CD = AB = 3x$, and $\angle DAB = \angle BCD = 90 \degrees$, so $\triangle DAB \cong \triangle BCD$ by SAS congruency. Therefore $\angle ABD = \angle BDC$
\newline
Thus we have moved all three angles into the right angle $\angle ADC$, proving that the sum of these three angles is $90 \degrees$. Q.E.D.
\newpage

\section{Dis Wed, 3b}
\begin{figure}[h]
    \includegraphics[width = 100mm]{GRAPH6.png}
\end{figure}
Let $\triangle ABC$ be an arbitrary triangle, extend AC to E, construct $a \parallel AB$.
\newline
For this proof I need the axiom that a straight line is $180 \degrees$, and that the corresponding angles and alternate interior angles between two parallel lines are equal.
\newline
Since $a \parallel AB$, we have $\angle BAC = \angle DCE$, and $\angle ABC = \angle BCD$. Since the three angles form $\angle ACE$, which is a straight line, the 3 angles add up to $180 \degrees$.
\newpage

\section{Lec Fri, 3}
If the 5th postulate was indeed redundant (provable from others), then the 4 postulates alone must always define the Euclidean space, and that there should be no other spaces that could follow the first 4 postulates but not the fifth.
\newline
Hyperbolic geometry satisfies the first 4 with its own definitions of lines, circles, and right angles. The resulting space does not follow Euclid's 5th postulate: two lines whose sum of the inner angles on one side is less than $180 \degrees$ can still curve away from each other and never intersect.
\newline
This shows that we need Euclid's 5th postulate to properly define a Euclidean space.
\newpage

\section{Lec Fri, 4b}
Let us label "There is at most 1 parallel line to a given line $l$ through a given point $P$" "Statement A", and "two lines that are parallel to  the same line are also parallel to eachother" "Statement B".
\newline
For this problem we need to prove that $A \iff B$. We will begin by proving $A \implies B$.
\newline
\begin{figure}[h]
    \includegraphics[width = 100mm]{GRAPH5.png}
\end{figure}
\paragraph{1}
Assume that statement B is false, so the two lines are not parallel to each other. Then let $a \parallel c, b \parallel c$, and $a, c$ intersect at point $P$. Then at $P$, there are two different lines that are both parallel to $c$. \lightning
\newline
Therefore our initial assumption is incorrect, statement B is true.
\paragraph{2}
Now we try to prove that $B \implies A$. Assume that statement A is false, so there are at least 2 different lines that go through that point. Then let $a \parallel c, b \parallel c$, $a,b$ go through the same point $P$.
Then by statement B, $a \parallel b$, but $a, b$ both contain $P$, so they have to be the same line. \lightning
\newline
Therefore there exists at most one line through $P$ parallel to $c$.

\end{document}


\section{What is data compression}

\singlespacing
In the context of data files, data compression is the process of encoding information using a smaller representation than the original file \cite{WikiDataCompression}. For data files, the size of the representation can be defined in bits or other derived units such as bytes.

\doublespacing
\singlespacing
Benefits of data compression occur from reducing the resources required to store and transmit data \cite{WikiDataCompression}. However, in the process of compressing and decompressing data computational resources are consumed which may impact benefits gained by compression.

\doublespacing
\singlespacing
The ratio of compression can be defined as:

\begin{equation*}
  C_{r} = \dfrac{d_{1}}{d_{2}}
\end{equation*}

\doublespacing
\singlespacing
The compressed ratio (C$_{r}$) is equal to the compressed file size ($d_{1}$) over the orginal file size ($d_{2}$) \cite{MasseyStudyGuide}.

\doublespacing
\singlespacing
\section{Lossless Compression}

\singlespacing

Lossless compression is a method of encoding a file with the goal of representing the file using a smaller amount of bits but not losing any information from the decoding process. The opposite of lossless compression is lossy compression where this is not the case. Huffman coding, the subject of this report, is used to implement lossless compression \cite{WikiHuffman}.

\doublespacing
\singlespacing
Lossless compression works because most real-world data contains statistical redundancy \cite{WikiDataCompression}. If a file contains symbols that occur more frequently than others we can use smaller representations to represent the frequent symbols and larger representations to represent the less frequent symbols. If a symbol doesn't occur at all in a file then no representation for that symbol is needed.

\doublespacing
\singlespacing
The two most common ways of constructing statistical models for the purposes of data compression are static and adaptive models. Static models build a model after reading all the data and store the model representation in the encoded file. Adaptive models update a model as the file is compressed \cite{WikiLossless}. The implementation of Huffman coding presented in this report is an example of a static model.

\doublespacing
\singlespacing
\section{Lossy Compression}

\singlespacing

Lossy compression works by removing less important information from a data file. Audio and Image data are examples of where lossy compression is applied. Human eyes are more sensitive to variations in luminance than they are to variations in color \cite{WikiDataCompression}. Because of this fact image compression can remove information from files that humans can not perceive thereby improving compression ratio for a file by reducing the number of bits needed to represent the said file. Similarly human limitations for hearing various audio signals present opportunities to decrease the required bit representation of audio files.

\doublespacing
\singlespacing
\section{Use cases}

\singlespacing

Various uses for data compression can be defined. General purpose data compression algorithms generally refer to compression algorithms that apply to standard text or binary files. Of these algorithms, some may be optimized for a particular type of input file such as text compression. As mentioned in previous sections specialized audio and image compression algorithms exist. Indeed for any particular domain, there may be a case for developing a special purpose compression algorithm, such is the case of HAPZIPPER a compression application for the purpose of compressing genetic data \cite{WikiLossless}.

\doublespacing
\singlespacing

Many of the lossless compression algorithms in use today combine Huffman Coding and other compression algorithms. DEFLATE combines Lempel-Ziv with Huffman coding and is used in zip files, gzip files and PNG images.

\doublespacing
\singlespacing
\section{Entropy}

\singlespacing

Information theory underpins the theoretical background of data compression. Claude Shannon published pioneering papers on the topic in the 1940s and 1950s \cite{WikiDataCompression}. Shannon came up with the idea of information entropy which can be defined by the following equation in regards to binary data \cite{MasseyStudyGuide}:

\begin{equation*}
  H = -\left(\sum\limits_{i=0}^n p_{i} log_{2}(p_{i})\right)
\end{equation*}

\doublespacing
\singlespacing
The negative of the sum of the probability of the symbol ($p_{i}$) occurring in a file times log base 2 of the probability of the symbol occurring in a file.

\doublespacing
\singlespacing
The implications of information entropy impose a limit on the potential compression ratio. If the likelihood of any symbol is the same entropy will equal the number of bits it takes to represent each symbol in the original file. A lower entropy allows more frequent symbols to be represented in a smaller number of bits.

\doublespacing
\singlespacing
There is no single lossless data compression algorithm that can compress any and all data. In fact, it is provably impossible to create such an algorithm \cite{WikiLossless}.

\section{Huffman Coding}

\doublespacing
\singlespacing
Huffman coding is an algorithm developed by David Huffman in 1951. He came up with his method as a student studying information theory at the Massachusetts Institute of Technology \cite{WikiHuffman}.

\doublespacing
\singlespacing
A Huffman code refers to a prefix code, an encoded representation of a symbol whereas Huffman coding is the process of creating and decoding Huffman codes. Huffman coding is a lossless data compression algorithm. There are two major tasks for the algorithm to perform. Encoding a data file and decoding an encoded file with the goal being achieving an optimal compression ratio.

\doublespacing
\singlespacing
Encoding a file involves building a Huffman tree, a type of top-down binary tree used to create prefix codes for symbols. Symbols that occur more frequently are closer to the root of the tree and ones that occur less frequently are found at lower levels of the tree.

\doublespacing
\singlespacing
Decoding a file involves using a built Huffman tree to decode a sequence of encoded symbols into their original form. A Huffman tree for any given file is not necessarily unique so a Huffman tree must also be extracted in the decoding process.


\lab{Pandas 1: Introduction}{Pandas 1: Introduction}
\objective{Though NumPy and SciPy are powerful tools for numerical computing, they lack some of the high-level functionality necessary for many data science applications.
Python's \emph{pandas} library, built on NumPy, is designed specifically for data management and analysis.
In this lab, we introduce pandas data structures, syntax, and explore its capabilities for quickly analyzing and presenting data.
}
\label{lab:pandas1}

% Just as NumPy is built on the \li{ndarray} data structure suited for efficient
% scientific and numerical computation, pandas is centered around a handful of
% core data structures custom built for data analysis: the \li{Series}, \li{DataFrame}, and \li{Panel}, which roughly correspond to one, two, and three-dimensional arrays.
% The \li{Panel} data structure is not used nearly

\section*{Series} % ===========================================================

A pandas \li{Series} is generalization of a one-dimensional NumPy array.
Like a NumPy array, every \li{Series} has a data type (\li{dtype}), and the entries of the \li{Series} are all of that type.
Unlike a NumPy array, every \li{Series} has an \emph{index} that labels each entry, and a \li{Series} object can also be given a name to label the entire data set.

\begin{lstlisting}
>>> import numpy as np
>>> import pandas as pd

# Initialize a Series of random entries with an index of letters.
>>> pd.Series(np.random.random(4), index=['a', 'b', 'c', 'd'])
<<a    0.474170
b    0.106878
c    0.420631
d    0.279713
dtype: float64>>

# The default index is integers from 0 to the length of the data.
>>> pd.Series(np.random.random(4), name="uniform draws")
<<0    0.767501
1    0.614208
2    0.470877
3    0.335885
Name: uniform draws, dtype: float64>>
\end{lstlisting}

The index in a \li{Series} is a pandas object of type \li{Index} and is stored as the \li{index} attribute of the \li{Series}.
The plain entries in the \li{Series} are stored as a NumPy array and can be accessed as such via the \li{values} attribute.

\begin{lstlisting}
>>> s1 = pd.Series([1, 2, 3, 4], index=['a', 'b', 'c', 'd'], name="some ints")

>>> s1.values                           # Get the entries as a NumPy array.
<<array([1, 2, 3, 4])>>

>>> print(s1.name, s1.dtype, sep=", ")  # Get the name and dtype.
<<some ints, int64>>

>>> s1.index                            # Get the pd.Index object.
<<Index(['a', 'b', 'c', 'd'], dtype='object')>>
\end{lstlisting}

The elements of a \li{Series} can be accessed by either the regular position-based integer index, or by the corresponding label in the index.
New entries can be added dynamically as long as a valid index label is provided, similar to adding a new key-value pair to a dictionary.
A \li{Series} can also be initialize from a dictionary: the keys become the index labels, and the values become the entries.

\begin{lstlisting}
>>> s2 = pd.Series([10, 20, 30], index=["apple", "banana", "carrot"])
>>> s2
<<apple     10
banana    20
carrot    30
dtype: int64>>

# s2[0] and s2["apple"] refer to the same entry.
>>> print(s2[0], s2["apple"], s2["carrot"])
<<10 10 30>>

>>> s2[0] += 5              # Change the value of the first entry.
>>> s2["dewberry"] = 0      # Add a new value with label 'dewberry'.
>>> s2
<<apple       15
banana      20
carrot      30
dewberry     0
dtype: int64>>

# Initialize a Series from a dictionary.
>>> pd.Series({"eggplant":3, "fig":5, "grape":7}, name="more foods")
<<eggplant    3
fig         5
grape       7
Name: more foods, dtype: int64>>
\end{lstlisting}

Slicing and fancy indexing also work the same way in \li{Series} as in NumPy arrays.
In addition, multiple entries of a \li{Series} can be selected by indexing a list of labels in the index.

\begin{lstlisting}
>>> s3 = pd.Series({"lions":2, "tigers":1, "bears":3}, name="oh my")
>>> s3
<<bears     3
lions     2
tigers    1
Name: oh my, dtype: int64>>

# Get a subset of the data by regular slicing.
>>> s3[1:]
<<lions     2
tigers    1
Name: oh my, dtype: int64>>

# Get a subset of the data with fancy indexing.
>>> s3[np.array([len(i) == 5 for i in s3.index])]
<<bears    3
lions    2
Name: oh my, dtype: int64>>

# Get a subset of the data by providing several index labels.
>>> s3[ ["tigers", "bears"] ]
tigers    1                     # Note that the entries are reordered,
bears     3                     # and the name stays the same.
<<Name: oh my, dtype: int64>>
\end{lstlisting}

\begin{comment} % Unnecessary, already getting too long.
We can create a \li{Series} having constant values in the following manner:
\begin{lstlisting}
>>> val = 4     #desired constant value of Series
>>> n = 6       #desired length of Series
>>> s3 = pd.Series(val, index=np.arange(n))
>>> s3
0    4
1    4
2    4
3    4
4    4
5    4
dtype: int64
\end{lstlisting}
\end{comment}

\begin{problem}
Create a pandas \li{Series} where the index labels are the even integers $0,2,\ldots,50$, and the entries are $n^2 - 1$, where $n$ is the entry's label.
Set all of the entries equal to zero whose labels are divisible by $3$.
\end{problem}

\subsection*{Operations with Series} % ----------------------------------------

A \li{Series} object has all of the advantages of a NumPy array, including entry-wise arithmetic, plus a few additional features (see Table \ref{table:pandas-numerical-methods}).
Operations between a \li{Series} $S_1$ with index $I_1$ and a \li{Series} $S_2$ with index $I_2$ results in a new \li{Series} with index $I_1\cup I_2$.
In other words, the index dictates how two \li{Series} can interact with each other.

\begin{lstlisting}
>>> s4 = pd.Series([1, 2, 4], index=['a', 'c', 'd'])
>>> s5 = pd.Series([10, 20, 40], index=['a', 'b', 'd'])
>>> 2*s4 + s5
<<a    12.0>>
<<b     NaN>>                           # s4 doesn't have an entry for b, and
<<c     NaN>>                           # s5 doesn't have an entry for c, so
<<d    48.0>>                           # the combination is Nan (np.nan / None).
<<dtype: float64>>
\end{lstlisting}

\begin{table}[H]
\begin{tabular}{r|l}
Method & Returns \\ \hline
\li{<<abs>>()}     & Object with absolute values taken (of numerical data) \\
\li{argmax()}  & The index label of the maximum value \\
\li{argmin()}  & The index label of the minimum value \\
\li{count()}   & The number of non-null entries \\
\li{cumprod()} & The cumulative product over an axis \\
\li{cumsum()}  & The cumulative sum over an axis \\
\li{<<max>>()}     & The maximum of the entries \\
\li{mean()}    & The average of the entries \\
\li{median()}  & The median of the entries \\
\li{<<min>>()}     & The minimum of the entries \\
\li{mode()}    & The most common element(s) \\
\li{prod()}    & The product of the elements \\
\li{<<sum>>()}     & The sum of the elements \\
\li{var()}     & The variance of the elements \\
\end{tabular}
\caption{Numerical methods of the \li{Series} and \li{DataFrame} pandas classes.
% Methods marked with a * are not methods of NumPy's \li{ndarray} class.
}
\label{table:pandas-numerical-methods}
\end{table}

Many \li{Series} are more useful than NumPy arrays primarily because of their index.
For example, a \li{Series} can be indexed by time with a pandas \li{DatetimeIndex}, an index with date and/or time values.
The usual way to create this kind of index is with \li{pd.date_range()}.

\begin{lstlisting}
# Make an index of the first three days in July 2000.
>>> pd.date_range("7/1/2000", "7/3/2000", freq='D')
<<DatetimeIndex(['2000-07-01', '2000-07-02', '2000-07-03'],
                dtype='datetime64[ns]', freq='D')>>
\end{lstlisting}

\begin{problem}
Suppose you make an investment of $d$ dollars in a particularly volatile stock.
Every day the value of your stock goes up by \$$1$ with probability $p$, or down by \$$1$ with probability $1-p$ (this is an example of a \emph{random walk}).

Write a function that accepts a probability parameter $p$ and an initial amount of money $d$, defaulting to $100$.
Use \li{pd.date_range()} to create an index of the days from 1 January 2000 to 31 December 2000.
Simulate the daily change of the stock by making one draw from a Bernoulli distribution with parameter $p$ (a binomial distribution with one draw) for each day.
Store the draws in a pandas \li{Series} with the date index and set the first draw to the initial amount $d$.
Sum the entries cumulatively to get the stock value by day.
Set any negative values to $0$, then plot the series using the \li{plot()} method of the \li{Series} object.

Call your function with a few different values of $p$ and $d$ to observe the different possible kinds of behavior.
\label{prob:pandas-random-walk}
\end{problem}

\begin{info}
The \li{Series} in Problem \ref{prob:pandas-random-walk} is an example of a \emph{time series}, since it is indexed by time.
Time series show up often in data science; we will explore them in more depth in another lab.
\end{info}

\begin{table}[H]
\begin{tabular}{r|l}
Method & Description \\ \hline
\li{append()} & Concatenate two or more \li{Series}. \\
\li{drop()} & Remove the entries with the specified label or labels \\
\li{drop_duplicates()} & Remove duplicate values \\
\li{dropna()} & Drop null entries \\
\li{fillna()} & Replace null entries with a specified value or strategy \\
\li{reindex()} & Replace the index \\
\li{sample()} & Draw a random entry \\
\li{shift()} & Shift the index \\
% \li{truncate()} & \\
\li{unique()} & Return unique values \\
% \li{where()} & Search for entires that satisfy some criteria \\
\end{tabular}
\caption{Methods for managing or modifying data in a pandas \li{Series} or \li{DataFrame}.}
\label{table:pandas-manage-data}
\end{table}

\section*{Data Frames} % ======================================================

A \li{DataFrame} is a collection of \li{Series} that share the same index, and is therefore a two-dimensional generalization of a NumPy array.
The row labels are collectively called the \emph{index}, and the column labels are collectively called the \emph{columns}.
An individual column in a \li{DataFrame} object is one \li{Series}.

There are many ways to initialize a \li{DataFrame}.
In the following code, we build a \li{DataFrame} out of a dictionary of \li{Series}.

\begin{lstlisting}
>>> x = pd.Series(np.random.randn(4), ['a', 'b', 'c', 'd'])
>>> y = pd.Series(np.random.randn(5), ['a', 'b', 'd', 'e', 'f'])
>>> df1 = pd.DataFrame({"series 1": x, "series 2": y})
>>> df1
<<   series 1  series 2
a -0.365542  1.227960
b  0.080133  0.683523
c  0.737970       NaN
d  0.097878 -1.102835
e       NaN  1.345004
f       NaN  0.217523>>
\end{lstlisting}

Note that the index of this \li{DataFrame} is the union of the index of \li{Series} \li{x} and that of \li{Series} \li{y}.
The columns are given by the keys of the dictionary \li{d}.
Since \li{x} doesn't have a label \li{e}, the
value in row \li{e}, column \li{1} is \li{NaN}.
This same reasoning explains the other missing values as well.
Note that if we take the first column of the \li{DataFrame} and drop the missing values, we recover the \li{Series} \li{x}:

\begin{lstlisting}
>>> df1["series1"].dropna()
<<a   -0.365542
b    0.080133
c    0.737970
d    0.097878
Name: series 1, dtype: float64>>
\end{lstlisting}

\begin{warn}
A pandas \li{DataFrame} cannot be sliced in exactly the same way as a NumPy array.
Notice how we just used \li{df1["series 1"]} to access a \emph{column} of the
the \li{DataFrame} \li{df1}.
We will discuss this in more detail later on.
\end{warn}

We can also initialize a \li{DataFrame} using a NumPy array, creating custom
row and column labels.
\begin{lstlisting}
>>> data = np.random.random((3, 4))
>>> pd.DataFrame(data, index=['A', 'B', 'C'], columns=np.arange(1, 5))

            1	        2	        3	        4
A	 0.065646	 0.968593	 0.593394	 0.750110
B	 0.803829	 0.662237	 0.200592	 0.137713
C	 0.288801	 0.956662	 0.817915	 0.951016
3 rows     4 columns
\end{lstlisting}

As with Series, if we don't specify the index or columns, the default is
\li{np.arange(n)}, where \li{n} is either the number of rows or columns.

\begin{comment} % Not necessary here. Too much info.
It is also possible to create multi-indexed arrays, for example:
\begin{lstlisting}
>>> grade=['eighth', 'ninth', 'tenth']
>>> subject=['math', 'science', 'english']
>>> myindex = pd.MultiIndex.from_product([grade, subject], names=['grade', 'subject'])
>>> myseries = pd.Series(np.random.randn(9), index=myindex)
>>> myseries
grade   subject
eighth  math       1.706644
        science   -0.899587
        english   -1.009832
ninth   math       2.096838
        science    1.884932
        english    0.413266
tenth   math      -0.924962
        science   -0.851689
        english    1.053329
dtype: float64
\end{lstlisting}

Multi-indexing is visually convenient, and will be explored further in Lab
\ref{lab:pandas3}, where we will discuss pandas pivot tables.
\end{comment}

\section*{Viewing and Accessing Data}

In this section we will explore some elementary accessing and querying
techniques that enable us to maneuver through and gain insight into our data.
Try using the \li{describe()} and \li{head()} methods for quick data summaries.

\subsection*{Basic Data Access}

We can slice the rows of a \li{DataFrame} much as with a NumPy array.

\begin{lstlisting}
>>> df = pd.DataFrame(np.random.randn(4, 2), index=['a', 'b', 'c', 'd'],
                      columns = ['I', 'II'])
>>> df[:2]

          I        II
a  0.758867  1.231330
b  0.402484 -0.955039

[2 rows x 2 columns]
\end{lstlisting}

More generally, we can select subsets of the data using the \li{iloc} and
\li{loc} indexers.
The \li{loc} index selects rows and columns based on
their \emph{labels}, while the \li{iloc} method selects them based on their
integer \emph{position}.
Accessing \li{Series} and \li{DataFrame} objects
using these indexing operations is more efficient than using bracket indexing, because the bracket indexing has to check many cases before it can determine how to slice the data structure.
Using \li{loc}/\li{iloc} explicitly, bypasses the extra checks.

\begin{lstlisting}
>>> # select rows a and c, column II
>>> df.loc[['a','c'], 'II']

a    1.231330
c    0.556121
Name: II, dtype: float64

>>> # select last two rows, first column
>>> df.iloc[-2:, 0]

c   -0.475952
d   -0.518989
Name: I, dtype: float64
\end{lstlisting}

Finally, a column of a \li{DataFrame} may be accessed using simple square
brackets and the name of the column, or alternatively by treating the label
as an object:
\begin{lstlisting}
>>> # get second column of df
>>> df['II']        # or, equivalently, df.II

a    1.231330
b   -0.955039
c    0.556121
d    0.173165
Name: II, dtype: float64
\end{lstlisting}

All of these techniques for getting subsets of the data may also be used to set
subsets of the data:

\begin{lstlisting}
>>> # set second columns to zeros
>>> df['II'] = 0
>>> df['II']
a    0
b    0
c    0
d    0
Name: II, dtype: int64

>>> # add additional column of ones
>>> df['III'] = 1
>>> df
            I  II  III
a   -0.460457   0    1
b    0.973422   0    1
c   -0.475952   0    1
d   -0.518989   0    1
\end{lstlisting}

% I FEEL LIKE THERE SHOULD BE A PROBLEM HERE!!

\begin{comment} % Moved random walk to Series section.
\subsection*{Plotting}

Plotting is often a much more effective way to view and gain understanding of a dataset than simply viewing the raw numbers.
Fortunately, pandas interfaces well with matplotlib, allowing relatively
painless data visualization.

Most of the functionality of plotting using pandas will be discussed in Lab \ref{lab:pandas2}.
 In this lab, we will simply show how to plot a \li{Series}.
Doing so is easy, as the \li{Series} object is equipped with its own plot
function.
% Change this example to use the titanic data, which will be cleaned up.

Let's start with visualizing a simple random walk.
By way of background, a \emph{random walk} is a
stochastic process used to model a non-deterministic path through some space.
It is a useful construct in
many fields and can be used to explain things like the motion of a molecule as it travels through a liquid to modeling the fluctuations
of stock prices.
Here we will simulate a one-dimensional symmetric random walk on the integers, which can be
described as follows.
\begin{enumerate}
  \item Start at 0.
  \item Flip a fair coin.
  \item If heads, move one unit to the right.
Otherwise, move one unit to the left.
  \item Go to Step 2.
\end{enumerate}
How can we simulate this random walk efficiently? Note that the walk is really characterized by the outcomes
of the coin flip.
If we represent heads by the number $1$ and tails by $-1$, then our position at a given moment
is just the cumulative sum of all previous outcomes.
Below, we simulate a sequence of coin flips, build the
resulting random walk, and plot the outcome.
\begin{lstlisting}
>>> import matplotlib.pyplot as plt
>>> N = 1000        # length of random walk
>>> s = np.zeros(N)
>>> s[1:] = np.random.binomial(1, .5, size=(N-1,))*2-1 #coin flips
>>> s = pd.Series(s)
>>> s = s.cumsum()  # random walk
>>> s.plot()
>>> plt.ylim([-50, 50])
>>> plt.show()
>>> plt.close()
\end{lstlisting}

The random walk is shown in Figure \ref{fig:PandasRandomWalk}.

\begin{figure}
\centering
\includegraphics[width=.7 \textwidth]{randomWalk.pdf}
\caption{Random walk of length 1000.}
\label{fig:PandasRandomWalk}
\end{figure}

\begin{problem}
Create five random walks of length 100, and plot them together in the same plot.

Next, create a ``biased'' random walk by changing the coin flip probability of head from 0.5 to 0.51.
Plot this biased walk with lengths 100, 10000, and then 100000.
Notice the definite trend that emerges.
Your results should be comparable to those in Figure \ref{pandas:biasedRandomWalk}.
\end{problem}

\begin{figure}
\centering
\includegraphics[width=.7 \textwidth]{biasedRandomWalk.pdf}
\caption{Biased random walk of length 100 (above) and 10000 (below).}
\label{pandas:biasedRandomWalk}
\end{figure}
\end{comment}

\section*{SQL Operations in pandas} % =========================================

The \li{DataFrame}, being a tabular data structure, bears an obvious resemblance to a typical relational
database table.
SQL is the standard for working with relational databases, and in this section we will
explore how pandas accomplishes some of the same tasks as SQL.
The SQL-like functionality of pandas is
one of its biggest advantages, since it can eliminate the need to switch between programming languages
for different tasks.
Within pandas we can handle both the querying \emph{and} data analysis.

For the following examples, we will use the following data:
\begin{lstlisting}
>>> #build toy data for SQL operations
>>> name = ['Mylan', 'Regan', 'Justin', 'Jess', 'Jason', 'Remi', 'Matt', 'Alexander', 'JeanMarie']
>>> sex = ['M', 'F', 'M', 'F', 'M', 'F', 'M', 'M', 'F']
>>> age = [20, 21, 18, 22, 19, 20, 20, 19, 20]
>>> rank = ['Sp', 'Se', 'Fr', 'Se', 'Sp', 'J', 'J', 'J', 'Se']
>>> ID = range(9)
>>> aid = ['y', 'n', 'n', 'y', 'n', 'n', 'n', 'y', 'n']
>>> GPA = [3.8, 3.5, 3.0, 3.9, 2.8, 2.9, 3.8, 3.4, 3.7]
>>> mathID = [0, 1, 5, 6, 3]
>>> mathGd = [4.0, 3.0, 3.5, 3.0, 4.0]
>>> major = ['y', 'n', 'y', 'n', 'n']
>>> studentInfo = pd.DataFrame({'ID': ID, 'Name': name, 'Sex': sex, 'Age': age, 'Class': rank})
>>> otherInfo = pd.DataFrame({'ID': ID, 'GPA': GPA, 'Financial_Aid': aid})
>>> mathInfo = pd.DataFrame({'ID': mathID, 'Grade': mathGd, 'Math_Major': major})
\end{lstlisting}

Before querying our data, it is important to know some of its basic properties,
such as number of columns, number of rows, and the datatypes of the columns.
This can be done by simply calling the \li{info()} method on the desired
\li{DataFrame}:

\begin{lstlisting}
>>> mathInfo.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 5 entries, 0 to 4
Data columns (total 3 columns):
Grade         5 non-null float64
ID            5 non-null int64
Math_Major    5 non-null object
dtypes: float64(1), int64(1), object(1)
\end{lstlisting}

We can also get some basic information about the structure of the \li{DataFrame}
using the \li{head()} or \li{tail()} methods.

\begin{lstlisting}
>>> mathInfo.head()
   Grade  ID Math_Major  ID  Age  GPA
0    4.0   0          y   0   20  3.8
1    3.0   1          n   2   18  3.0
2    3.5   5          y   4   19  2.8
3    3.0   6          n   6   20  3.8
4    4.0   3          n   7   19  3.4
\end{lstlisting}

The method \li{isin()} is a useful way to find certain values in a
\li{DataFrame}.
 It compares the input (a list, dictionary, or \li{Series})
to the \li{DataFrame} and returns a boolean \li{Series} showing whether or
not the values match.
You can then use this boolean array to select appropriate
locations.
Now let's look at the pandas equivalent of some SQL SELECT statements.
\begin{lstlisting}
>>> # SELECT ID, Age FROM studentInfo
>>> studentInfo[['ID', 'Age']]

>>> # SELECT ID, GPA FROM otherInfo WHERE Financial_Aid = 'y'
>>> otherInfo[otherInfo['Financial_Aid']=='y'][['ID', 'GPA']]

>>> # SELECT Name FROM studentInfo WHERE Class = 'J' OR Class = 'Sp'
>>>  studentInfo[studentInfo['Class'].isin(['J','Sp'])]['Name']
\end{lstlisting}

\begin{problem}
The example above shows how to implement a simple WHERE condition, and it is easy
to have a more complex expression.
Simply enclose each condition by parentheses,
and use the standard boolean operators \li{\&} (AND), \li{\|} (OR), and \li{\~} (NOT) to
connect the conditions appropriately.
Use pandas to execute the following query:
\begin{lstlisting}[language=SQL]
SELECT ID, Name from studentInfo WHERE Age > 19 AND Sex = 'M'
\end{lstlisting}
\end{problem}

Next, let's look at JOIN statements.
In pandas, this is done with the \li{merge} function,
which takes as arguments the two \li{DataFrame} objects to join, as well as keyword arguments specifying
the column on which to join, along with the type (left, right, inner, outer).

\begin{lstlisting}
>>> # SELECT * FROM studentInfo INNER JOIN mathInfo ON studentInfo.ID = mathInfo.ID
>>> pd.merge(studentInfo, mathInfo, on='ID') # INNER JOIN is the default
   Age Class  ID    Name Sex  Grade Math_Major
0   20    Sp   0   Mylan   M    4.0          y
1   21    Se   1   Regan   F    3.0          n
2   22    Se   3    Jess   F    4.0          n
3   20     J   5    Remi   F    3.5          y
4   20     J   6    Matt   M    3.0          n
[5 rows x 7 columns]

>>> # SELECT GPA, Grade FROM otherInfo FULL OUTER JOIN mathInfo ON otherInfo.ID = mathInfo.ID
>>> pd.merge(otherInfo, mathInfo, on='ID', how='outer')[['GPA', 'Grade']]
   GPA  Grade
0  3.8    4.0
1  3.5    3.0
2  3.0    NaN
3  3.9    4.0
4  2.8    NaN
5  2.9    3.5
6  3.8    3.0
7  3.4    NaN
8  3.7    NaN
[9 rows x 2 columns]
\end{lstlisting}

\begin{problem}
Using a join operation, create a \li{DataFrame} containing the ID, age, and GPA of all male individuals.
You ought to be able to accomplish this in one line of code.
\end{problem}

\begin{comment}
This may be going into too much detail for an introductory lab...
It is sometimes desirable to join to DataFrames on their indexes rather than on a particular column.
The \li{join} method makes this operation convenient.
\begin{lstlisting}
>>> #create new DataFrame
>>> sibs = [0, 1, 0, 5, 2, 9]
>>> sibInfo = pd.DataFrame({'Siblings':sibs})
>>> sibInfo
   Siblings
0         0
1         1
2         0
3         5
4         2
5         9
[6 rows x 1 columns]

>>> #now join studentInfo with sibInfo
>>> studentInfo.join(sibInfo)
   Age Class  ID    Name Sex  Siblings
0   20    Sp   0    Mylan   M         0
1   21    Se   1   Regan   F         1
2   18    Fr   2     Justin   M         0
3   22    Se   3   Jess   F         5
4   19    Sp   4     Jason   M         2
5   20     J   5  Remi   F         9
6   20     J   6   Matt   M       NaN
7   19     J   7  Alexander   M       NaN
8   20    Se   8     JeanMarie   F       NaN
[9 rows x 6 columns]
\end{lstlisting}
The default is a left join, but this can be altered.
When attempting to join to DataFrames that share common columns on their indexes,
we must rename the columns to avoid contradictions.
We do this by simply appending
characters to the existing column names, specified by the \li{lsuffix} and \li{rsuffix}
parameters (for the left and right DataFrames, respectively).
\begin{lstlisting}
>>> #join studentInfo and mathInfo on indexes
>>> studentInfo.join(mathInfo, lsuffix='_left', rsuffix='_right', how='inner')
   Age Class  ID_left   Name Sex  Grade  ID_right Math_Major
0   20    Sp        0   Mylan   M    4.0         0          y
1   21    Se        1  Regan   F    3.0         1          n
2   18    Fr        2    Justin   M    3.5         5          y
3   22    Se        3  Jess   F    3.0         6          n
4   19    Sp        4    Jason   M    4.0         3          n
[5 rows x 8 columns]
\end{lstlisting}

The final SQL-like operation we will discuss is the UNION, or concatenation of DataFrames.
In the simplest setting, this operation is useful when we have two DataFrames that have the
same columns but possibly different rows.
\begin{lstlisting}
>>> #create another df holding more math info
>>> mathID2 = [0, 5, 2, 4]
>>> mathGd2 = [4.0, 3.5, 2.0, 3.0]
>>> major2 = ['y', 'y', 'n', 'y']
>>> mathInfo2 = pd.DataFrame({'Grade':mathGd2, 'ID':mathID2, 'Math_Major':major2})
>>> pd.concat([mathInfo, mathInfo2], ignore_index=True).drop_duplicates()
   Grade  ID Math_Major
0    4.0   0          y
1    3.0   1          n
2    3.5   5          y
3    3.0   6          n
4    4.0   3          n
7    2.0   2          n
8    3.0   4          y
[7 rows x 3 columns]
\end{lstlisting}
The \li{ignore_index=True} argument means we discard the indexes of the two input DataFrames and create
a new one for the concatenated DataFrame.
The \li{drop_duplicates} method simply drops duplicate rows.
\end{comment}

Be aware that other types of SQL-like operations are also possible, such as UNION.
When you find yourself unsure of how to carry out a more involved SQL-like operation, the online
pandas documentation will be of great service.

\section*{Analyzing Data}
Although pandas does not provide built-in support for heavy-duty statistical analysis of data, there
are nevertheless many features and functions that facilitate basic data manipulation and computation,
even when the data is in a somewhat messy state.
We will now explore some of these features.

\subsection*{Basic Data Manipulation}
Because the primary pandas data structures are subclasses of the \li{ndarray}, they are valid input
to most NumPy functions, and can often be treated simply as NumPy arrays.
For example, basic
vectorized operations work just fine:
\begin{lstlisting}
>>> x = pd.Series(np.random.randn(4), index=['a', 'b', 'c', 'd'])
>>> y = pd.Series(np.random.randn(5), index=['a', 'b', 'd', 'e', 'f'])
>>> x**2
a    1.710289
b    0.157482
c    0.540136
d    0.202580
dtype: float64
>>> z = x + y
>>> z
a    0.123877
b    0.278435
c         NaN
d   -1.318713
e         NaN
f         NaN
dtype: float64
>>> np.log(z)
a   -2.088469
b   -1.278570
c         NaN
d         NaN
e         NaN
f         NaN
dtype: float64
\end{lstlisting}
Notice that pandas automatically aligns the indexes when adding two \li{Series} (or \li{DataFrames}),
so that the the index of the output is simply the union of the indexes of the two inputs.
The default
missing value \li{NaN} is given for labels that are not shared by both inputs.

It may also be useful to transpose \li{DataFrames}, re-order the columns or rows, or sort according to a
given column.
Here we demonstrate these capabilities:
\begin{lstlisting}
>>> df = pd.DataFrame(np.random.randn(4,2), index=['a', 'b', 'c', 'd'], columns=['I', 'II'])
>>> df
          I        II
a -0.154878 -1.097156
b -0.948226  0.585780
c  0.433197 -0.493048
d -0.168612  0.999194

[4 rows x 2 columns]

>>> df.transpose()
           a         b         c         d
I  -0.154878 -0.948226  0.433197 -0.168612
II -1.097156  0.585780 -0.493048  0.999194

[2 rows x 4 columns]

>>> # switch order of columns, keep only rows 'a' and 'c'
>>> df.reindex(index=['a', 'c'], columns=['II', 'I'])
         II         I
a -1.097156 -0.154878
c -0.493048  0.433197

[2 rows x 2 columns]

>>> # sort descending according to column 'II'
>>> df.sort_values('II', ascending=False)
          I        II
d -0.168612  0.999194
b -0.948226  0.585780
c  0.433197 -0.493048
a -0.154878 -1.097156

[4 rows x 2 columns]
\end{lstlisting}

\begin{comment}
\subsection*{Basic Statistical Functions}

The pandas library allows us to easily calculate basic summary statistics of our data,
useful when we want a quick description of the data.
The \li{describe()} function
outputs several such summary statistics for each column in a \li{DataFrame}:
\begin{lstlisting}
>>> df.describe()
              I        II
count  4.000000  4.000000
mean  -0.209630 -0.001308
std    0.566696  0.964083
min   -0.948226 -1.097156
25%   -0.363516 -0.644075
50%   -0.161745  0.046366
75%   -0.007859  0.689133
max    0.433197  0.999194

[8 rows x 2 columns]
\end{lstlisting}

Functions for calculating means and variances, the covariance and correlation matrices, and other
basic statistics are also available.
Below, we calculate the means of each row, as well as the
covariance matrix:
\begin{lstlisting}
>>> df.mean(axis=1)
a   -0.626017
b   -0.181223
c   -0.029925
d    0.415291
dtype: float64

>>> df.cov()
           I        II
I   0.321144 -0.256229
II -0.256229  0.929456

[2 rows x 2 columns]
\end{lstlisting}
\end{comment}

\subsection*{Dealing with Missing Data} % -------------------------------------

Missing data is a ubiquitous problem in data science.
Fortunately, pandas is particularly well-suited to
handling missing and anomalous data.
As we have already seen, the pandas default for a missing value is \li{NaN}.
In basic arithmetic operations, if one of the operands is \li{NaN}, then the output is also \li{NaN}.
The following example illustrates this concept:
\begin{lstlisting}
>>> x = pd.Series(np.arange(5))
>>> y = pd.Series(np.random.randn(5))
>>> x.iloc[3] = np.nan
>>> x + y
0    0.731521
1    0.623651
2    2.396344
3         NaN
4    3.351182
dtype: float64
\end{lstlisting}

If we are not interested in the missing values, we can simply drop them from the data altogether:

\begin{lstlisting}
>>> (x + y).dropna()
0    0.731521
1    0.623651
2    2.396344
4    3.351182
dtype: float64
\end{lstlisting}

This is not always the desired behavior, however.
It may well be the case that missing data actually corresponds to
some default value, such as zero.
In this case, we can replace all instances of \li{NaN} with a specified value:
\begin{lstlisting}
>>> # fill missing data with 0, add
>>> x.fillna(0) + y
0    0.731521
1    0.623651
2    2.396344
3    1.829400
4    3.351182
dtype: float64
\end{lstlisting}

Other functions, such as \li{<<sum()>>} and \li{mean()} ignore NaN values in the computation.
When dealing with missing data, make sure you are aware of the behavior of the pandas functions you are using.

\section*{Data I/O} % =========================================================

Being able to import and export data is a fundamental skill in data science.
Unfortunately, with the multitude of data formats and conventions, importing data can often be a painful task.
The pandas library seeks to reduce some of the difficulty by providing file readers for various types of formats,
including CSV, Excel, HDF5, SQL, JSON, HTML, and pickle files.

\begin{table}[H]
\begin{tabular}{r|l}
Method & Description \\ \hline
\li{describe()}  & Return a \li{Series} describing the data structure \\
\li{head()}      & Return the first $n$ rows, defaulting to 5 \\
\li{tail()}      & Return the last $n$ rows, defaulting to 5 \\
\li{to_csv()}    & Write the index and entries to a CSV file \\
\li{to_json()}   & Convert the object to a JSON string \\
\li{to_pickle()} & Serialize the object and store it in an external file \\
\li{to_sql()}    & Write the object data to an open SQL database \\
\end{tabular}
\caption{Methods for viewing or exporting data in a pandas \li{Series} or \li{DataFrame}.}
\label{table:pandas-view-or-export}
\end{table}

The CSV (comma separated values) format is a simple way of storing tabular data
in plain text.
Because CSV files are one of the most popular file formats for
exchanging data, we will explore the \li{read_csv()} function in more detail.
To learn to read other types of file formats, see the online pandas
documentation.
To read a CSV data file into a \li{DataFrame}, call the
\li{read_csv()} function with the path to the CSV file, along with the
appropriate keyword arguments.
Below we list some of the most important keyword
arguments:
\begin{itemize}
\item \li{delimiter}:
This argument specifies the character that separates data fields, often a
comma or a whitespace character.

\item \li{header}:
The row number (starting at 0) in the CSV file that contains the column names.

\item \li{index_col}:
If you want to use one of the columns in the CSV file as the index for the
\li{DataFrame}, set this argument to the desired column number.

\item \li{skiprows}:
If an integer $n$, skip the first $n$ rows of the file, and then start reading
in the data.
If a list of integers, skip the specified rows.

\item \li{names}:
If the CSV file does not contain the column names, or you wish to use other
column names, specify them in a list assigned to this argument.

\end{itemize}
There are several other keyword arguments, but this should be enough to get you
started.

When you need to save your data, pandas allows you to write to several different
file formats.
A typical example is the \li{to_csv()} function method attached to
\li{Series} and \li{DataFrame} objects, which writes the data to a CSV file.
Keyword arguments allow you to specify the separator character, omit writing the
columns names or index, and specify many other options.
The code below
demonstrates its typical usage:
\begin{lstlisting}
>>> df.to_csv("my_df.csv")
\end{lstlisting}

\begin{problem} % Crime data.
The file \texttt{crime\_data.csv} contains data on types of crimes committed in the United States from 1960 to 2016.
\begin{itemize}
\item Load the data into a pandas \li{DataFrame}, using the column names in the file and the column titled
``Year'' as the index.
Make sure to skip lines that don't contain data.

\item Insert a new column into the data frame that contains the crime rate by year (the ratio of ``Total'' column
to the ``Population'' column).

\item Plot the crime rate as a function of the year.

\item List the 5 years with the highest crime rate in descending order.

\item Calculate the average number of total crimes as well as burglary crimes between 1960 and 2012.

\item Find the years for which the total number of crimes was below average, but the number of burglaries
was above average.

\item Plot the number of murders as a function of the population.

\item Select the Population, Violent, and Robbery columns for all years in the 1980s, and save
this smaller data frame to a CSV file \texttt{crime\_subset.csv}.
\end{itemize}
\end{problem}

\begin{problem}
In 1912 the RMS \emph{Titanic} sank after colliding with an iceberg.
The file \texttt{titanic.csv} contains data on the incident.
Each row represents a different passenger, and the columns describe various features of the passengers (age, sex, whether or not they survived, etc.)

Start by cleaning the data.
\begin{itemize}
    \item Read the data into a \li{DataFrame}.
    Use the first row of the file as the column labels, but do not use any of the columns as the index.
    \item Drop the columns \li{"Sibsp"}, \li{"Parch"}, \li{"Cabin"}, \li{"Boat"}, \li{"Body"}, and \li{"home.dest"}.
    \item Drop any entries without data in the \li{"Survived"} column, then change the remaining entries to \li{True} or \li{False} (they start as 1 or 0).
    \item Replace null entries in the \li{"Age"} column with the average age.
    \item Save the new \li{DataFrame} as \texttt{titanic\_clean.csv}.
\end{itemize}
Next, answer the following questions.
\begin{itemize}
    \item How many people survived? What percentage of passengers survived?
    \item What was the average price of a ticket? How much did the most expensive ticket cost?
    \item How old was the oldest survivor? How young was the youngest survivor? What about non-survivors?
\end{itemize}

\end{problem}