# PHY321: Work and Energy  and conservation theorems
**[Morten Hjorth-Jensen](http://mhjgit.github.io/info/doc/web/)**, Department of Physics and Astronomy and Facility for Rare Ion Beams (FRIB), Michigan State University, USA

Date: **Feb 4, 2022**

Copyright 1999-2022, [Morten Hjorth-Jensen](http://mhjgit.github.io/info/doc/web/). Released under CC Attribution-NonCommercial 4.0 license

## Plans for the week January 31-February 4

### Monday

Short repetition from last week on the work-energy theorem with examples. Discussion of momentum and angular momentum.
Reading suggestion: Taylor sections 3.1-3.4 and 4.1-4.3

### Wednesday

Examples and conservation of  angular momentum, Taylor sections 3.4 and 3.5.
Discussion of exercises 5 and 6. This is also the assignment for Friday's session.

### Friday

Solution of exercises and discussion of homework 3. Focus is exercises 5 and 6, see end of these slides.

If you wish to read more about conservative forces or not, Feyman's lectures from 1963 are quite interesting.
He states for example that **All fundamental forces in nature appear to be conservative**.
This statement was made while developing his argument that *there are no nonconservative forces*.
You may enjoy the link to [Feynman's lecture](http://www.feynmanlectures.caltech.edu/I_14.html).

## Work, Energy, Momentum and Conservation laws

The systems we studied the first three weeks have shown us how to use Newton’s laws of
motion to determine the motion of an object based on the forces acting
on it. For some of the cases there is an underlying assumption that we can find an analytical solution to a continuous problem.
With a continuous problem we mean a problem where the various variables can take any value within a finite or infinite interval.

Unfortunately, in many cases we
cannot find an exact solution to the equations of motion we get from
Newton’s second law. The numerical approach, where we discretize the continuous problem, allows us however to study a much richer set of problems.
For problems involving Newton's laws and the various equations of motion we encounter, solving the equations numerically, is the standard approach.

It allows us to focus on the underlying forces. Often we end up using the same numerical algorithm for different problems.

Here we introduce a commonly used technique that allows us to find the
velocity as a function of position without finding the position as a
function of time—an alternate form of Newton’s second law. The method
is based on a simple principle: Instead of solving the equations of
motion directly, we integrate the equations of motion. Such a method
is called an integration method.

This allows us also to introduce the **work-energy** theorem. This
theorem allows us to find the velocity as a function of position for
an object even in cases when we cannot solve the equations of
motion. This introduces us to the concept of work and kinetic energy,
an energy related to the motion of an object.

And finally, we will link the work-energy theorem with the principle of conservation of energy.

## The Work-Energy Theorem

Let us define the kinetic energy $K$ with a given velocity $\boldsymbol{v}$

$$
K=\frac{1}{2}mv^2,
$$

where $m$ is the mass of the object we are considering.
We assume also that there is a force $\boldsymbol{F}$ acting on the given object

$$
\boldsymbol{F}=\boldsymbol{F}(\boldsymbol{r},\boldsymbol{v},t),
$$

with $\boldsymbol{r}$ the position and $t$ the time.
In general we assume the force is a function of all these variables.
Many of the more central forces in Nature however, depende only on the
position. Examples are the gravitational force and the force derived
from the Coulomb potential in electromagnetism.

## Rewriting the Kinetic Energy

Let us study the derivative of the kinetic energy with respect to time $t$. Its continuous form is

$$
\frac{dK}{dt}=\frac{1}{2}m\frac{d\boldsymbol{v}\cdot\boldsymbol{v}}{dt}.
$$

Using our results from exercise 3 of homework 1, we can write the derivative of a vector dot product as

$$
\frac{dK}{dt}=\frac{1}{2}m\frac{d\boldsymbol{v}\cdot\boldsymbol{v}}{dt}= \frac{1}{2}m\left(\frac{d\boldsymbol{v}}{dt}\cdot\boldsymbol{v}+\boldsymbol{v}\cdot\frac{d\boldsymbol{v}}{dt}\right)=m\frac{d\boldsymbol{v}}{dt}\cdot\boldsymbol{v}.
$$

We know also that the acceleration is defined as

$$
\boldsymbol{a}=\frac{\boldsymbol{F}}{m}=\frac{d\boldsymbol{v}}{dt}.
$$

We can then rewrite the equation for the derivative of the kinetic energy as

$$
\frac{dK}{dt}=m\frac{d\boldsymbol{v}}{dt}\boldsymbol{v}=\boldsymbol{F}\frac{d\boldsymbol{r}}{dt},
$$

where we defined the velocity as the derivative of the position with respect to time.

## Discretizing

Let us now discretize the above equation by letting the instantaneous terms be replaced by a discrete quantity, that is
we let $dK\rightarrow \Delta K$, $dt\rightarrow \Delta t$, $d\boldsymbol{r}\rightarrow \Delta \boldsymbol{r}$ and $d\boldsymbol{v}\rightarrow \Delta \boldsymbol{v}$.

We have then

$$
\frac{\Delta K}{\Delta t}=m\frac{\Delta \boldsymbol{v}}{\Delta t}\boldsymbol{v}=\boldsymbol{F}\frac{\Delta \boldsymbol{r}}{\Delta t},
$$

or by multiplying out $\Delta t$ we have

$$
\Delta K=\boldsymbol{F}\Delta \boldsymbol{r}.
$$

We define this quantity as the **work** done by the force $\boldsymbol{F}$
during the displacement $\Delta \boldsymbol{r}$. If we study the dimensionality
of this problem we have mass times length squared divided by time
squared, or just dimension energy.

## Difference in kinetic energy

If we now define a series of such displacements $\Delta\boldsymbol{r}$ we have a difference in kinetic energy at a final position $\boldsymbol{r}_n$ and an
initial position $\boldsymbol{r}_0$ given by

$$
\Delta K=\frac{1}{2}mv_n^2-\frac{1}{2}mv_0^2=\sum_{i=0}^n\boldsymbol{F}_i\Delta \boldsymbol{r},
$$

where $\boldsymbol{F}_i$ are the forces acting at every position $\boldsymbol{r}_i$.

The work done by acting with a force on a set of displacements can
then be as expressed as the difference between the initial and final
kinetic energies.

This defines the **work-energy** theorem.

## From the discrete version to the continuous version

If we take the limit $\Delta \boldsymbol{r}\rightarrow 0$, we can rewrite the sum over the various displacements in terms of an integral, that is

$$
\Delta K=\frac{1}{2}mv_n^2-\frac{1}{2}mv_0^2=\sum_{i=0}^n\boldsymbol{F}_i\Delta \boldsymbol{r}\rightarrow \int_{\boldsymbol{r}_0}^{\boldsymbol{r}_n}\boldsymbol{F}(\boldsymbol{r},\boldsymbol{v},t)d\boldsymbol{r}.
$$

This integral defines a path integral since it will depend on the given path we take between the two end points. We will replace the limits with the symbol $c$ in order to indicate that we take a specific countour in space when the force acts on the system. That is the work $W_{n0}$ between two points $\boldsymbol{r}_n$ and $\boldsymbol{r}_0$ is labeled as

$$
W_{n0}=\frac{1}{2}mv_n^2-\frac{1}{2}mv_0^2=\int_{c}\boldsymbol{F}(\boldsymbol{r},\boldsymbol{v},t)d\boldsymbol{r}.
$$

Note that if the force is perpendicular to the displacement, then the force does not affect the kinetic energy.

Let us now study some examples of forces and how to find the velocity from the integration over a given path.

Thereafter we study how to  evaluate an integral numerically.

## Studying the Work-energy Theorem numerically

In order to study the work- energy, we will normally need to perform
a numerical integration, unless we can integrate analytically. Here we
present some of the simpler  methods such as the **rectangle** rule, the **trapezoidal** rule and higher-order methods like the Simpson family of methods.

## Example of an Electron moving along a Surface

As an example, let us consider the following case.
We have  classical electron which moves in the $x$-direction along a surface. The force from the surface is

$$
\boldsymbol{F}(x)=-F_0\sin{(\frac{2\pi x}{b})}\boldsymbol{e}_1.
$$

The constant $b$ represents the distance between atoms at the surface of the material, $F_0$ is a constant and $x$ is the position of the electron.

Using the work-energy theorem we can find the work $W$ done when moving an electron from a position $x_0$ to a final position $x$ through the
 integral

$$
W=\int_{x_0}^x \boldsymbol{F}(x')dx' =  -\int_{x_0}^x F_0\sin{(\frac{2\pi x'}{b})} dx',
$$

which results in

$$
W=\frac{F_0b}{2\pi}\left[\cos{(\frac{2\pi x}{b})}-\cos{(\frac{2\pi x_0}{b})}\right].
$$

## Finding the Velocity

If we now use the work-energy theorem we can find the the velocity at a final position $x$ by setting up
the differences in kinetic energies between the final position and the initial position $x_0$.

We have that the work done by the force is given by the difference in kinetic energies as

$$
W=\frac{1}{2}m\left(v^2(x)-v^2(x_0)\right)=\frac{F_0b}{2\pi}\left[\cos{(\frac{2\pi x}{b})}-\cos{(\frac{2\pi x_0}{b})}\right],
$$

and labeling $v(x_0)=v_0$ (and assuming we know the initial velocity) we have

$$
v(x)=\pm \sqrt{v_0^2+\frac{F_0b}{m\pi}\left[\cos{(\frac{2\pi x}{b})}-\cos{(\frac{2\pi x_0}{b})}\right]},
$$

Choosing $x_0=0$m and $v_0=0$m/s we can simplify the above equation to

$$
v(x)=\pm \sqrt{\frac{F_0b}{m\pi}\left[\cos{(\frac{2\pi x}{b})}-1\right]},
$$

## Harmonic Oscillations

Another well-known force (and we will derive when we come to Harmonic
Oscillations) is the case of a sliding block attached to a wall
through a spring. The block is attached to a spring with spring
constant $k$. The other end of the spring is attached to the wall at
the origin $x=0$.  We assume the spring has an equilibrium length
$L_0$.

The force $F_x$ from the spring on the block is then

$$
F_x=-k(x-L_0).
$$

The position $x$ where the spring force is zero is called the equilibrium position. In our case this is
$x=L_0$.

We can now compute the work done by this force when we move our block from an initial position $x_0$ to a final position $x$

$$
W=\int_{x_0}^{x}F_xdx'=-k\int_{x_0}^{x}(x'-L_0)dx'=\frac{1}{2}k(x_0-L_0)^2-\frac{1}{2}k(x-L_0)^2.
$$

If we now bring back the definition of the work-energy theorem in terms of the kinetic energy we have

$$
W=\frac{1}{2}mv^2(x)-\frac{1}{2}mv_0^2=\frac{1}{2}k(x_0-L_0)^2-\frac{1}{2}k(x-L_0)^2,
$$

which we rewrite as

$$
\frac{1}{2}mv^2(x)+\frac{1}{2}k(x-L_0)^2=\frac{1}{2}mv_0^2+\frac{1}{2}k(x_0-L_0)^2.
$$

What does this mean? The total energy, which is the sum of potential and kinetic energy, is conserved.
Wow, this sounds interesting. We will analyze this next week in more detail when we study energy, momentum and angular momentum conservation.

## Work-Energy Theorem and Energy Conservation

We have made the observation that energy was conserved for a force which
depends only on the position.
In particular we considered a force acting on a block
attached to a spring with a so-called spring
constant $k$. The other end of the spring was attached to the wall.

The force $F_x$ from the spring on the block was defined as

$$
F_x=-kx.
$$

The work done on the block due to a displacement from a position $x_0$ to $x$

$$
W=\int_{x_0}^{x}F_xdx'=\frac{1}{2}kx_0^2-\frac{1}{2}kx^2.
$$

## Conservation of energy
With the definition of the work-energy theorem in terms of the kinetic energy we obtained

$$
W=\frac{1}{2}mv^2(x)-\frac{1}{2}mv_0^2=\frac{1}{2}kx_0^2-\frac{1}{2}kx^2,
$$

which we rewrote as

$$
\frac{1}{2}mv^2(x)+\frac{1}{2}kx^2=\frac{1}{2}mv_0^2+\frac{1}{2}kx_0^2.
$$

The total energy, which is the sum of potential and kinetic energy, is conserved.
We will analyze this interesting result now in more detail when we study energy, momentum and angular momentum conservation.

But before we start with energy conservation, conservative forces and potential energies, we need to revisit our definitions of momentum and angular momentum.

## What is a Conservative Force?

A conservative force is a force whose property is that the total work
done in moving an object between two points is independent of the
taken path. This means that the work on an object under the influence
of a conservative force, is independent on the path of the object. It
depends only on the spatial degrees of freedom and it is possible to
assign a numerical value for the potential at any point. It leads to
conservation of energy. The gravitational force is an example of a
conservative force.

## Two important conditions

First, a conservative force depends only on the spatial degrees of freedom. This is a necessary condition for obtaining a path integral which is independent of path.
The  important condition for the final work to be independent of the path is that the **curl** of the force is zero, that

$$
\boldsymbol{\nabla} \times \boldsymbol{F}=0
$$

## Work-energy theorem to show that energy is conserved with a conservative force

The work-energy theorem states that the work done $W$ by a force $\boldsymbol{F}$ that moves an object from a position $\boldsymbol{r}_0$ to a new position $\boldsymbol{r}_1$

$$
W=\int_{\boldsymbol{r}_0}^{\boldsymbol{r}_1}\boldsymbol{F}\boldsymbol{dr}=\frac{1}{2}mv_1^2-\frac{1}{2}mv_0^2,
$$

where $v_1^2$ is the velocity squared at a time $t_1$ and $v_0^2$ the corresponding quantity at a time $t_0$.
The work done is thus the difference in kinetic energies. We can rewrite the above equation as

$$
\frac{1}{2}mv_1^2=\int_{\boldsymbol{r}_0}^{\boldsymbol{r}_1}\boldsymbol{F}\boldsymbol{dr}+\frac{1}{2}mv_0^2,
$$

that is the final kinetic energy is equal to the initial kinetic energy plus the work done by the force over a given path from a  position $\boldsymbol{r}_0$ at time $t_0$ to a final position position $\boldsymbol{r}_1$ at a later time $t_1$.

## Conservation of Momentum

Before we move on however, we need to remind ourselves about important aspects like the linear momentum and angular momentum. After these considerations, we move back to more details about conservatives forces.

Assume we have $N$ objects, each with velocity $\boldsymbol{v}_i$ with
$i=1,2,\dots,N$ and mass $m_i$. The momentum of each object is
$\boldsymbol{p}_i=m\boldsymbol{v}_i$ and the total linear (or mechanical) momentum is
defined as

$$
\boldsymbol{P}=\sum_{i=1}^N\boldsymbol{p}_i=\sum_{i=1}^Nm_i\boldsymbol{v}_i,
$$

## Two objects first

Let us assume we have two objects only that interact with each other and are influenced by an external force.

We define also the total net force acting on object 1 as

$$
\boldsymbol{F}_1^{\mathrm{net}}=\boldsymbol{F}_1^{\mathrm{ext}}+\boldsymbol{F}_{12},
$$

where $\boldsymbol{F}_1^{\mathrm{ext}}$ is the external force
(for example the force due to an electron moving in an electromagnetic field) and $\boldsymbol{F}_{12}$ is the
force between object one and two. Similarly for object 2 we have

$$
\boldsymbol{F}_2^{\mathrm{net}}=\boldsymbol{F}_2^{\mathrm{ext}}+\boldsymbol{F}_{21}.
$$

## Newton's Third Law

Newton's third law which we met earlier states that **for every action there is an equal and opposite reaction**.
It is more accurately stated as

**if two bodies exert forces on each other, these forces are equal in magnitude and opposite in direction**.

This means that for two bodies $i$ and $j$, if the force on $i$ due to $j$ is called $\boldsymbol{F}_{ij}$, then

$$
\begin{equation}
\boldsymbol{F}_{ij}=-\boldsymbol{F}_{ji}.
\label{_auto1} \tag{1}
\end{equation}
$$

For the abovementioned two objects we have thus $\boldsymbol{F}_{12}=-\boldsymbol{F}_{21}$.

## Newton's Second Law and Momentum

With the net forces acting on each object we can now related the momentum to the forces via

$$
\boldsymbol{F}_1^{\mathrm{net}}=m_1\boldsymbol{a}_i=m_1\frac{d\boldsymbol{v}_1}{dt}=\boldsymbol{F}_1^{\mathrm{ext}}+\boldsymbol{F}_{12},
$$

and

$$
\boldsymbol{F}_2^{\mathrm{net}}=m_2\boldsymbol{a}_2=m_2\frac{d\boldsymbol{v}_2}{dt}=\boldsymbol{F}_2^{\mathrm{ext}}+\boldsymbol{F}_{21}.
$$

Recalling our definition for the linear momentum we have then

$$
\frac{d\boldsymbol{p}_1}{dt}=\boldsymbol{F}_1^{\mathrm{ext}}+\boldsymbol{F}_{12},
$$

and

$$
\frac{d\boldsymbol{p}_2}{dt}=\boldsymbol{F}_2^{\mathrm{ext}}+\boldsymbol{F}_{21}.
$$

## The total Momentum

The total momentum $\boldsymbol{P}$ is defined as the sum of the individual momenta, meaning that we can rewrite

$$
\boldsymbol{F}_1^{\mathrm{net}}+\boldsymbol{F}_2^{\mathrm{net}}=\frac{d\boldsymbol{p}_1}{dt}+\frac{d\boldsymbol{p}_2}{dt}=\frac{d\boldsymbol{P}}{dt},
$$

that is the derivate with respect to time of the total momentum. If we now
write the net forces as sums of the external plus internal forces
between the objects we have

$$
\frac{d\boldsymbol{P}}{dt}=\boldsymbol{F}_1^{\mathrm{ext}}+\boldsymbol{F}_{12}+\boldsymbol{F}_2^{\mathrm{ext}}+\boldsymbol{F}_{21}=\boldsymbol{F}_1^{\mathrm{ext}}+\boldsymbol{F}_2^{\mathrm{ext}}.
$$

The derivative of the total momentum is just **the sum of the external
forces**. If we assume that the external forces are zero and that only
internal (here two-body forces) are at play, we obtain the important
result that the derivative of the total momentum is zero. This means
again that the total momentum is a constant of the motion and
conserved quantity. This is a very important result that we will use
in many applications to come.

## Newton's Second Law

Let us now general to several objects $N$ and let us also assume that there are no external forces. We will label such a system as **an isolated system**.

Newton's second law, $\boldsymbol{F}=m\boldsymbol{a}$, can be written for a particle $i$ as

$$
\begin{equation}
\boldsymbol{F}_i=\sum_{j\ne i}^N \boldsymbol{F}_{ij}=m_i\boldsymbol{a}_i,
\label{_auto2} \tag{2}
\end{equation}
$$

where $\boldsymbol{F}_i$ (a single subscript) denotes the net force acting on $i$ from the other objects/particles.
Because the mass of $i$ is fixed and we assume it does not change with time, one can see that

$$
\begin{equation}
\boldsymbol{F}_i=\frac{d}{dt}m_i\boldsymbol{v}_i=\sum_{j\ne i}^N\boldsymbol{F}_{ij}.
\label{_auto3} \tag{3}
\end{equation}
$$

## Summing over all Objects/Particles

Now, one can sum over all the objects/particles and obtain

$$
\frac{d}{dt}\sum_i m_iv_i=\sum_{ij, i\ne j}^N\boldsymbol{F}_{ij}=0.
$$

How did we arrive at the last step?  We rewrote the double sum as

$$
\sum_{ij, i\ne j}^N\boldsymbol{F}_{ij}=\sum_i^N\sum_{j>i}\left(\boldsymbol{F}_{ij}+\boldsymbol{F}_{ji}\right),
$$

and using Newton's third law which states that
$\boldsymbol{F}_{ij}=-\boldsymbol{F}_{ji}$, we obtain that the net sum over all  the two-particle
forces is zero when we only consider so-called **internal forces**.
Stated differently, the last step made use of the fact that for every
term $ij$, there is an equivalent term $ji$ with opposite
force. Because the momentum is defined as $m\boldsymbol{v}$, for a system of
particles, we have thus

$$
\begin{equation}
\frac{d}{dt}\sum_im_i\boldsymbol{v}_i=0,~~{\rm for~isolated~particles}.
\label{_auto4} \tag{4}
\end{equation}
$$

## Conservation of total Momentum

By "isolated" one means that the only force acting on any particle $i$
are those originating from other particles in the sum, i.e. "no
external" forces. Thus, Newton's third law leads to the conservation
of total momentum,

$$
\boldsymbol{P}=\sum_i m_i\boldsymbol{v}_i,
$$

and we have

$$
\frac{d}{dt}\boldsymbol{P}=0.
$$

## Example: Rocket Science

Consider a rocket of mass $M$ moving with velocity $v$. After a
brief instant, the velocity of the rocket is $v+\Delta v$ and the mass
is $M-\Delta M$. Momentum conservation gives

$$
\begin{eqnarray*}
Mv&=&(M-\Delta M)(v+\Delta v)+\Delta M(v-v_e)\\
0&=&-\Delta Mv+M\Delta v+\Delta M(v-v_e),\\
0&=&M\Delta v-\Delta Mv_e.
\end{eqnarray*}
$$

In the second step we ignored the term $\Delta M\Delta v$ because it is doubly small. The last equation gives

$$
\begin{eqnarray}
\Delta v&=&\frac{v_e}{M}\Delta M,\\
\nonumber
\frac{dv}{dt}&=&\frac{v_e}{M}\frac{dM}{dt}.
\end{eqnarray}
$$

## Integrating the Equations

Integrating the expression with lower limits $v_0=0$ and $M_0$, one finds

$$
\begin{eqnarray*}
v&=&v_e\int_{M_0}^M \frac{dM'}{M'}\\
v&=&v_e\ln(M/M_0)\\
&=&v_e\ln[(M_0-\alpha t)/M_0].
\end{eqnarray*}
$$

Because the total momentum of an isolated system is constant, one can
also quickly see that the center of mass of an isolated system is also
constant. The center of mass is the average position of a set of
masses weighted by the mass,

$$
\begin{equation}
\bar{x}=\frac{\sum_im_ix_i}{\sum_i m_i}.
\label{_auto5} \tag{5}
\end{equation}
$$

## Rate of Change

The rate of change of $\bar{x}$ is

$$
\dot{\bar{x}}=\frac{1}{M}\sum_i m_i\dot{x}_i=\frac{1}{M}P_x.
$$

Thus if the total momentum is constant the center of mass moves at a
constant velocity, and if the total momentum is zero the center of
mass is fixed.

## Conservation of Angular Momentum

The angular momentum is defined as

$$
\begin{equation}
\boldsymbol{L}=\boldsymbol{r}\times\boldsymbol{p}=m\boldsymbol{r}\times\boldsymbol{v}.
\label{_auto6} \tag{6}
\end{equation}
$$

It means that the angular momentum is perpendicular to the plane defined by position $\boldsymbol{r}$ and the momentum $\boldsymbol{p}$ via $\boldsymbol{r}\times \boldsymbol{p}$.

## Rate of Change of Angular Momentum

The rate of change of the angular momentum is

$$
\frac{d\boldsymbol{L}}{dt}=m\boldsymbol{v}\times\boldsymbol{v}+m\boldsymbol{r}\times\dot{\boldsymbol{v}}=\boldsymbol{r}\times{\boldsymbol{F}}
$$

The first term is zero because $\boldsymbol{v}$ is parallel to itself, and the
second term defines the so-called torque. If $\boldsymbol{F}$ is parallel to $\boldsymbol{r}$ then the torque is zero and we say that angular momentum is conserved.

If the force is not radial, $\boldsymbol{r}\times\boldsymbol{F}\ne 0$ as above, and angular momentum is no longer conserved,

$$
\begin{equation}
\frac{d\boldsymbol{L}}{dt}=\boldsymbol{r}\times\boldsymbol{F}\equiv\boldsymbol{\tau},
\label{_auto7} \tag{7}
\end{equation}
$$

where $\boldsymbol{\tau}$ is the torque.

## The Torque, Example 1 (hw 4, exercise 4)

Let us assume we have an initial position $\boldsymbol{r}_0=x_0\boldsymbol{e}_1+y_0\boldsymbol{e}_2$ at a time $t_0=0$.
We add now a force in the positive $x$-direction

$$
\boldsymbol{F}=F_x\boldsymbol{e}_1=\frac{d\boldsymbol{p}}{dt},
$$

where we used the force as defined by the time derivative of the momentum.

We can use this force (and its pertinent acceleration) to find the velocity via the relation

$$
\boldsymbol{v}(t)=\boldsymbol{v}_0+\int_{t_0}^t\boldsymbol{a}dt',
$$

and with $\boldsymbol{v}_0=0$ we have

$$
\boldsymbol{v}(t)=\int_{t_0}^t\frac{\boldsymbol{F}}{m}dt',
$$

where $m$ is the mass of the object.

## The Torque, Example 1 (hw 4, exercise 4)

Since the force acts only in the $x$-direction, we have after integration

$$
\boldsymbol{v}(t)=\frac{\boldsymbol{F}}{m}t=\frac{F_x}{m}t\boldsymbol{e}_1=v_x(t)\boldsymbol{e}_1.
$$

The momentum is in turn given by $\boldsymbol{p}=p_x\boldsymbol{e}_1=mv_x\boldsymbol{e}_1=F_xt\boldsymbol{e}_1$.

Integrating over time again we find the final position as (note the force depends only on the $x$-direction)

$$
\boldsymbol{r}(t)=(x_0+\frac{1}{2}\frac{F_x}{m}t^2) \boldsymbol{e}_1+y_0\boldsymbol{e}_2.
$$

There is no change in the position in the $y$-direction since the force acts only in the $x$-direction.

## The Torque, Example 1 (hw 4, exercise 4)

We can now compute the angular momentum given by

$$
\boldsymbol{l}=\boldsymbol{r}\times\boldsymbol{p}=\left[(x_0+\frac{1}{2}\frac{F_x}{m}t^2) \boldsymbol{e}_1+y_0\boldsymbol{e}_2\right]\times F_xt\boldsymbol{e}_1.
$$

Computing the cross product we find

$$
\boldsymbol{l}=-y_0F_xt\boldsymbol{e}_3=-y_0F_xt\boldsymbol{e}_z.
$$

The torque is the time derivative of the angular momentum and we have

$$
\boldsymbol{\tau}=-y_0F_x\boldsymbol{e}_3=-y_0F_x\boldsymbol{e}_z.
$$

The torque is non-zero and angular momentum is not conserved.

## The Torque, Example 2

One can write the torque about a given axis, which we will denote as $\hat{z}$, in polar coordinates, where

$$
\begin{eqnarray}
x&=&r\sin\theta\cos\phi,~~y=r\sin\theta\cos\phi,~~z=r\cos\theta,
\end{eqnarray}
$$

to find the $z$ component of the torque,

$$
\begin{eqnarray}
\tau_z&=&xF_y-yF_x\\
\nonumber
&=&-r\sin\theta\left\{\cos\phi \partial_y-\sin\phi \partial_x\right\}V(x,y,z).
\end{eqnarray}
$$

## Chain Rule and Partial Derivatives

One can use the chain rule to write the partial derivative w.r.t. $\phi$ (keeping $r$ and $\theta$ fixed),

$$
\begin{eqnarray}
\partial_\phi&=&\frac{\partial x}{\partial\phi}\partial_x+\frac{\partial_y}{\partial\phi}\partial_y
+\frac{\partial z}{\partial\phi}\partial_z\\
\nonumber
&=&-r\sin\theta\sin\phi\partial_x+\sin\theta\cos\phi\partial_y.
\end{eqnarray}
$$

Combining the two equations,

$$
\begin{eqnarray}
\tau_z&=&-\partial_\phi V(r,\theta,\phi).
\end{eqnarray}
$$

Thus, if the potential is independent of the azimuthal angle $\phi$,
there is no torque about the $z$ axis and $L_z$ is conserved.

## System of Isolated Particles

For a system of isolated particles, one can write

$$
\begin{eqnarray}
\frac{d}{dt}\sum_i\boldsymbol{L}_i&=&\sum_{i\ne j}\boldsymbol{r}_i\times \boldsymbol{F}_{ij}\\
\nonumber
&=&\frac{1}{2}\sum_{ij, i\ne j} \boldsymbol{r}_i\times \boldsymbol{F}_{ij}+\boldsymbol{r}_j\times\boldsymbol{F}_{ji}\\
\nonumber
&=&\frac{1}{2}\sum_{ij, i\ne j} (\boldsymbol{r}_i-\boldsymbol{r}_j)\times\boldsymbol{F}_{ij}=0,
\end{eqnarray}
$$

where the last step used Newton's third law,
$\boldsymbol{F}_{ij}=-\boldsymbol{F}_{ji}$. If the forces between the particles are
radial, i.e. $\boldsymbol{F}_{ij} ~||~ (\boldsymbol{r}_i-\boldsymbol{r}_j)$, then each term in
the sum is zero and the net angular momentum is fixed. Otherwise, you
could imagine an isolated system that would start spinning
spontaneously.

## Homework 3, exercises 5 and 6, numerical solution

```python
%matplotlib inline

# let's start by importing useful packages we are familiar with
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

We will choose the following values
1. mass $m=0,2$ kg

2. accelleration (gravity) $g=9.81$ m/s$^{2}$.

3. initial position is the height $h=2$ m

4. initial velocities $v_{x,0}=v_{y,0}=10$ m/s

You need also to define an initial time and
the step size $\Delta t$. We can define the step size $\Delta t$ as the difference between any
two neighboring values in time (time steps) that we analyze within
some range. It can be determined by dividing the interval we are
analyzing, which in our case is time $t_{\mathrm{final}}-t_0$, by the number of steps we
are taking $(N)$. This gives us a step size $\Delta t = \dfrac{t_{\mathrm{final}}-t_0}{N}$.

With these preliminaries we are now ready to plot our results from exercise 5.

In setting up our code we need to

1. Define and obtain all initial values, constants, and time to be analyzed with step sizes as done above (you can use the same values)

2. Calculate the velocity using $v_{i+1} = v_{i} + (\Delta t)*a_{i}$

3. Calculate the position using $pos_{i+1} = r_{i} + (\Delta t)*v_{i}$

4. Calculate the new acceleration $a_{i+1}$.

5. Repeat steps 2-4 for all time steps within a loop.

```python
# Exercise 6, hw3, brute force way with declaration of vx, vy, x and y
# Common imports
import numpy as np
import pandas as pd
from math import *
import matplotlib.pyplot as plt
import os

# Where to save the figures and data files
PROJECT_ROOT_DIR = "Results"
FIGURE_ID = "Results/FigureFiles"
DATA_ID = "DataFiles/"

if not os.path.exists(PROJECT_ROOT_DIR):
    os.mkdir(PROJECT_ROOT_DIR)

if not os.path.exists(FIGURE_ID):
    os.makedirs(FIGURE_ID)

if not os.path.exists(DATA_ID):
    os.makedirs(DATA_ID)

def image_path(fig_id):
    return os.path.join(FIGURE_ID, fig_id)

def data_path(dat_id):
    return os.path.join(DATA_ID, dat_id)

def save_fig(fig_id):
    plt.savefig(image_path(fig_id) + ".png", format='png')

# Output file
outfile = open(data_path("Eulerresults.dat"),'w')

from pylab import plt, mpl
plt.style.use('seaborn')
mpl.rcParams['font.family'] = 'serif'

g = 9.80655 #m/s^2
# The mass and the drag constant D
D = 0.00245 #mass/length   kg/m
m = 0.2 #kg, mass of falling object
DeltaT = 0.001
#set up arrays
tfinal = 1.4
# set up number of points for all variables
n = ceil(tfinal/DeltaT)
# define scaling constant vT used in analytical solution
vT = sqrt(m*g/D)
# set up arrays for t, a, v, and y and arrays for analytical results
#brute force setting up of arrays for x and y, vx, vy, ax and ay
t = np.zeros(n)
vy = np.zeros(n)
y = np.zeros(n)
vx = np.zeros(n)
x = np.zeros(n)
yanalytic = np.zeros(n)
# Initial conditions, note that these correspond to an object falling in the y-direction only.
vx[0] = 0.0 #m/s
vy[0] = 0.0  #m/s
y[0] = 10.0 #m
x[0] = 0.0 #m
yanalytic[0] = y[0]
# Start integrating using Euler's method
for i in range(n-1):
    # expression for acceleration, note the absolute value and division by mass
# Note: you need to think of the sign for the drag force as function of the velocity vector
    ax = -D*vx[i]*sqrt(vx[i]**2+vy[i]**2)/m
    ay = -g - D*vy[i]*sqrt(vx[i]**2+vy[i]**2)/m
    # update velocity and position
    vx[i+1] = vx[i] + DeltaT*ax
    x[i+1] = x[i] + DeltaT*vx[i]
    vy[i+1] = vy[i] + DeltaT*ay
    y[i+1] = y[i] + DeltaT*vy[i]
    # update time to next time step and compute analytical answer
    t[i+1] = t[i] + DeltaT
    yanalytic[i+1] = y[0]-(vT*vT/g)*log(cosh(g*t[i+1]/vT))+vy[0]*t[i+1]
    if ( y[i+1] < 0.0):
        break
data = {'t[s]': t,
        'Relative error in y': abs((y-yanalytic)/yanalytic),
        'vy[m/s]': vy,
        'x[m]': x,
        'vx[m/s]': vx
}
NewData = pd.DataFrame(data)
display(NewData)
# save to file
NewData.to_csv(outfile, index=False)
#then plot
fig, axs = plt.subplots(4, 1)
axs[0].plot(t, y)
axs[0].set_xlim(0, tfinal)
axs[0].set_ylabel('y')
axs[1].plot(t, vy)
axs[1].set_ylabel('vy[m/s]')
axs[1].set_xlabel('time[s]')
axs[2].plot(t, x)
axs[2].set_xlim(0, tfinal)
axs[2].set_ylabel('x')
axs[3].plot(t, vx)
axs[3].set_ylabel('vx[m/s]')
axs[3].set_xlabel('time[s]')
fig.tight_layout()
save_fig("EulerIntegration")
plt.show()
```

## A more compact version of the code

```python
# Exercise 6, hw3, smarter way with declaration of vx, vy, x and y
# Common imports
import numpy as np
import pandas as pd
from math import *
import matplotlib.pyplot as plt
import os

# Where to save the figures and data files
PROJECT_ROOT_DIR = "Results"
FIGURE_ID = "Results/FigureFiles"
DATA_ID = "DataFiles/"

if not os.path.exists(PROJECT_ROOT_DIR):
    os.mkdir(PROJECT_ROOT_DIR)

if not os.path.exists(FIGURE_ID):
    os.makedirs(FIGURE_ID)

if not os.path.exists(DATA_ID):
    os.makedirs(DATA_ID)

def image_path(fig_id):
    return os.path.join(FIGURE_ID, fig_id)

def data_path(dat_id):
    return os.path.join(DATA_ID, dat_id)

def save_fig(fig_id):
    plt.savefig(image_path(fig_id) + ".png", format='png')

from pylab import plt, mpl
plt.style.use('seaborn')
mpl.rcParams['font.family'] = 'serif'

g = 9.80655 #m/s^2 g to 6 leading digits after decimal point
D = 0.00245 #m/s
m = 0.2 # kg
# Define Gravitational force as a vector in x and y. It is a constant
G = -m*g*np.array([0.0,1])
DeltaT = 0.01
#set up arrays
tfinal = 1.3
n = ceil(tfinal/DeltaT)
# set up arrays for t, a, v, and x
t = np.zeros(n)
v = np.zeros((n,2))
r = np.zeros((n,2))
# Initial conditions as compact 2-dimensional arrays
r0 = np.array([0.0,10.0])
v0 = np.array([10.0,0.0])
r[0] = r0
v[0] = v0
# Start integrating using Euler's method
for i in range(n-1):
    # Set up forces, air resistance FD, note now that we need the norm of the vector
    # Here you could have defined your own function for this
    vabs = sqrt(sum(v[i]*v[i]))
# Note: you need to think of the sign for the drag force
    FD = -D*v[i]*vabs
    # Final net forces acting on falling object
    Fnet = FD+G
    # The accelration at a given time t_i
    a = Fnet/m
    # update velocity, time and position using Euler's method
    v[i+1] = v[i] + DeltaT*a
    r[i+1] = r[i] + DeltaT*v[i]
    t[i+1] = t[i] + DeltaT

fig, axs = plt.subplots(4, 1)
axs[0].plot(t, r[:,1])
axs[0].set_xlim(0, tfinal)
axs[0].set_ylabel('y')
axs[1].plot(t, v[:,1])
axs[1].set_ylabel('vy[m/s]')
axs[1].set_xlabel('time[s]')
axs[2].plot(t, r[:,0])
axs[2].set_xlim(0, tfinal)
axs[2].set_ylabel('x')
axs[3].plot(t, v[:,0])
axs[3].set_ylabel('vx[m/s]')
axs[3].set_xlabel('time[s]')

fig.tight_layout()
save_fig("EulerIntegration")
plt.show()
```


```python
# %load ../../preconfig.py
%matplotlib inline
import matplotlib.pyplot as plt
plt.rcParams['axes.grid'] = False
import seaborn as sns
sns.set(color_codes=True)

import numpy as np
import pandas as pd
#import itertools

import sklearn

import logging
logger = logging.getLogger()
```

Exercises
=========

### Ex 2.1
##### Question
Suppose each of K-classes has an associated target $t_k$, which is a vector of all zeros, except a one in the $k$th position. Show that classifying to the largest element of $\hat{y}$ amounts to choosing the closest target, $\min_k \|tk − \hat{y}\|$, if the elements of $\hat{y}$ sum to one.

##### Solution
Given:
$t_k = e_k$;
$\sum_i \hat{y}_i = 1$.

need to proof: $\text{arg max}_i \hat{y}_i = \text{arg min}_k \|t_k - \hat{y}\|$.

Proof:

\begin{align}
\text{arg min}_k \|t_k - \hat{y}\| &= \text{arg min}_k \left ( \displaystyle \sum_{j \neq k} \hat{y}_j + | \hat{y}_k - 1 | \right )\\
        &= \text{arg min}_k \left ( \displaystyle \sum_{j \neq k} \hat{y}_j + 1 - \hat{y}_k  \right ) \\
        &= \text{arg min}_k \left ( 1 - \hat{y}_k + 1 - \hat{y}_k \right ) \\
        &= \text{arg min}_k 2(1 - \hat{y}_k) \\
        &= \text{arg max}_k \hat{y}_k
\end{align}

### Ex 2.2
##### Question
Show how to compute the Bayes decision boundary for the simula- tion example in Figure 2.5.

##### Solution
[ref: Elements of Statistical Learning - Andrew Tulloch](https://github.com/ajtulloch/Elements-of-Statistical-Learning/blob/master/ESL-Solutions.pdf)

As Eq.(2.22) of textbook:
\begin{align}
    \hat{G}(x) &= \text{arg min}_{g \in \mathcal{G}} [ 1 - P(g | X = x) ] \\
               &= \text{arg max}_{g \in \mathcal{G}} P(g | X = x)
\end{align}

The optimal Bayes decision boundary is where:
\begin{align}
    P(\text{orange} | X = x) &= P(\text{blue} | X = x) \\
    \frac{P( X = x | orange ) P( orange )}{P(X = x)} &= \frac{P( X = x | blue ) P( blue )}{P(X = x)} \\
    P( X = x | orange ) P( orange ) &= P( X = x | blue ) P( blue )
\end{align}

As descriped in Sec 2.3.3, $P(orange)$ is same as $P(blue)$, and $P(X = x | orange)$ and $P(X = x | blue)$ are generated as bivariate Gassuian distribution. Hence we can work out the optimal Bayes decision boundary exactly.

### Ex 2.3
##### Question
Derive equation (2.24).

Suppose there are $N$ points, uniformly distributed in the unit sphere in $\mathbb{R}^p$. What is the median distance from the origin to the closest data point?

##### Solution
[ref: Example Sheet 1: Solutions](http://www2.stat.duke.edu/~banks/cam-lectures.dir/Examples1-solns.pdf)

(1)
Suppose $r$ is the median distance from the origin to the closest data point.

Let $r_{\text{closest}}$ are all possible closetst points.
Because $r$ is the median case, $\forall j$
$$P(r_{\text{closest}}^j \geq r) = \frac{1}{2}$$

Because $r_{\text{closest}}^j$ is the closest point, so all $N$ points have distance $\geq r_{\text{closest}}^j \geq r$.

together, we get:
$$P(\text{all N points have distance } \geq r) = \frac{1}{2}$$

(2)
First, all points are uniformly distributed in the unit sphere in $\mathbb{R}^p$.

Second, [the p-dimensional volume of a Euclidean ball of radius R in p-dimensional Euclidean space is](https://en.wikipedia.org/wiki/Volume_of_an_n-ball):
$$V_p(R) = \frac{\pi^{p/2}}{\Gamma(\frac{p}{2} + 1)}R^p$$

together, for any point $x$,
\begin{align}
P(x \text{ has distance } \geq r) &= 1 - P(x \text{ has distance } < r) \\
    &= 1 - \frac{\pi^{p/2}}{\Gamma(\frac{p}{2} + 1)}r^p \Big{/} \frac{\pi^{p/2}}{\Gamma(\frac{p}{2} + 1)}1^p
    &= 1 - r^p
\end{align}

Then:
\begin{align}
    P(\text{all N points have distance } \geq r) &= P^N (x \text{ has distance } \geq r) \\
        &= (1 - r^p)^N
\end{align}

(3)
In all,
$$\frac{1}{2} = P(\text{all N points have distance } \geq r) = (1 - r^p)^N$$

we get the solution:
$$r = (1 - (\frac{1}{2})^{1/N})^{1/p}$$

### Ex 2.4
##### Question
The edge effect problem discussed on page 23 is not peculiar to uniform sampling from bounded domains.

Consider inputs drawn from a spherical multinormal distribution $X \sim N(0,I_p)$. The squared distance from any sample point to the origin has a $\mathcal{X}^2_p$ distribution with mean $p$. Consider a prediction point $x_0$ drawn from this distribution, and let $a = x_0 \big{/} \| x0 \|$ be an associated unit vector. Let $z_i = a^T x_i$ be the projection of each of the training points on this direction.

Show that the $z_i$ are distributed $N(0,1)$ with expected squared distance from the origin 1, while the target point has expected squared distance $p$ from the origin.

##### Solution
$z_i = \alpha^T x_i$, which is a linear combination.  Moreover, $x_i \sim N(0, I_p)$ means that its elements are all independant.

As [the variance of a linear combination](https://en.wikipedia.org/wiki/Variance#Sum_of_uncorrelated_variables_.28Bienaym.C3.A9_formula.29) is:
$$\operatorname{Var}\left( \sum_{i=1}^N a_iX_i\right) = \sum_{i=1}^N a_i^2\operatorname{Var}(X_i)$$

We get:
\begin{align}
E(z_i) &= \alpha^T E(x_i) \\
       & = 0
\end{align}
and
\begin{align}
\operatorname{Var} (z_i) &= \sum \alpha_j^2 \operatorname{Var}(x_j^i) \\
       & = \sum \alpha_j^2 \\
       & = \| \alpha \|^2_2 \\
       & = 1
\end{align}

Thus, $z_i \sim N(0,1)$.

The target point has expected suqared distace $\| x_i \|_2^2 = p$.

### Ex 2.5
#### (a)
Derive equation (2.27). The last line makes use of (3.8) through a conditioning argument.

#### Solution
First, we give:

1. for $y_0 = x_0^T \beta + \epsilon; \ \epsilon \sim N(0, \sigma^2)$:
   + $E_{y_0 | x_0}(y_0) = E(y_0 | x_0) = E(x_0^T \beta + \epsilon) = x_0^T \beta$
   + $\operatorname{Var}_{y_0 | x_0}(y_0) = \operatorname{Var}(y_0 | x_0) = \operatorname{Var}(x_0^T \beta + \epsilon) = \operatorname{Var}(\epsilon) = \sigma^2$

2. for $\hat{y}_0 = x_0^T \hat{\beta} = x_0^T \beta + x_0 (X^T X)^{-1} x_0 \epsilon$:
   + expected value:
     \begin{equation}
        E_{\tau}(\hat{y_0}) = E(y_0 | x_0) = x_0^T \beta \quad \text{unbiased}
     \end{equation}
   + variance:
     \begin{align}
        \operatorname{Var}_{\tau}(\hat{y_0}) &= \operatorname{Var}_{\tau}(x_0^T \hat{\beta}) \\
            &= x_0^T \operatorname{Var}_{\tau}(\hat{\beta}) x_0 \\
            &= x_0^T E_{\tau}((X^T X)^{-1} \sigma^2) x_0 \quad \text{see Eq 3.8} \\
            &= E_{\tau} x_0^T (X^T X)^{-1} x_0 \sigma^2
     \end{align}

3. [Proof of variance and bias relationship](https://en.wikipedia.org/wiki/Mean_squared_error):
\begin{align}
    E( (\hat{\theta} - \theta)^2 ) &= E( \left (\hat{\theta} - E(\hat{\theta}) \right )^2 ) + (E(\hat{\theta}) - \theta)^ 2 \\
        &= \operatorname{Var}(\hat{\theta}) + \operatorname{Bias}^2 (\hat{\theta}, \theta)
\end{align}

Thus,
\begin{align}
    \operatorname{EPE}(x_0) &= E_{y_0 | x_0} E_{\tau} (y_0 - \hat{y}_0)^2 \\
        &= E_{\tau} E_{\color{blue}{y_0 | x_0}} (\color{blue}{y_0} - \hat{y}_0)^2 \\
        &= E_{\tau} \left ( \operatorname{Var}(y_0 | x_0)  + (E_{y_0 | x_0}(y_0) - \hat{y}_0)^2 \right ) \\
        &= \operatorname{Var}(y_0 | x_0) + E_{\color{blue}{\tau}} \left( E(y_0 | x_0) - \color{blue}{\hat{y}_0} \right )^2 \\
        &= \operatorname{Var}(y_0 | x_0) + \operatorname{Var}_{\tau}(\hat{y}_0) + \left ( E_{\tau}(\hat{y}_0) - E(y_0 | x_0) \right)^2 \\
        &= \operatorname{Var}(y_0 | x_0) + \operatorname{Var}_{\tau}(\hat{y}_0) + \left ( E_{\tau}(\hat{y}_0) -  x_0^T \beta \right)^2 \\
        &= \sigma^2 + E_{\tau} x_0^T (X^T X)^{-1} x_0 \sigma^2 + 0^2
\end{align}

#### (b)
Derive equation (2.28), making use of the cyclic property of the trace operator [trace(AB) = trace(BA)], and its linearity (which allows us to interchange the order of trace and expectation).

#### Solution
[ref: A Solution Manual and Notes for: The Elements of Statistical Learning by Jerome Friedman, Trevor Hastie, and Robert Tibshirani](https://www.google.com.sg/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwiF6sf2hsfLAhWSco4KHfJQCCwQFggbMAA&url=http%3A%2F%2Fwaxworksmath.com%2FAuthors%2FG_M%2FHastie%2FWriteUp%2Fweatherwax_epstein_hastie_solutions_manual.pdf&usg=AFQjCNH3VN6HgCDHtXNIbJtAjEEQNZFINA&sig2=b_zFhNYsupRwqtY62dGnwA)

1. $x_0$ is $p \times 1$ vector, and $\mathbf{X}$ is $N \times p$ matrix, hence $x_0^T (\mathbf{X}^T \mathbf{X})^{-1} x_0 = C_{1 \times 1} = \operatorname{trance}(C_{1 \times 1})$

2. [properity of Covariance matrix](https://en.wikipedia.org/wiki/Covariance_matrix)
\begin{align}
    \operatorname{Cov}(x_0) &= E(x_0 x_0^T) - E(x_0) E(x_0)^T \\
        &= E(x_0 x_0^T) \quad \text{as } E(\mathbf{X}) = 0 \text{ and $x_0$ is picked randomly}
\end{align}

Thus,
\begin{align}
    E_{x_0} \operatorname{EPE}(x_0) &= E_{x_0} x_0^T (\mathbf{X}^T \mathbf{X})^{-1} x_0 \sigma^2 + \sigma^2 \\
        &= E_{x_0} \operatorname{trance} \left ( x_0^T (\mathbf{X}^T \mathbf{X})^{-1} x_0 \right ) \sigma^2 + \sigma^2 \\
        &= E_{x_0} \operatorname{trance} \left ( (\mathbf{X}^T \mathbf{X})^{-1} x_0 x_0^T \right ) \sigma^2 + \sigma^2 \quad \text{cyclic property} \\
        &\approx E_{x_0} \operatorname{trance} \left ( \operatorname{Cov}^{-1}(\mathbf{X}) x_0 x_0^T \right ) \frac{\sigma^2}{N} + \sigma^2  \quad \text{as } \mathbf{X}^T \mathbf{X} \to N \operatorname{Cov}(\mathbf{X}) \\
        &= \operatorname{trance} \left ( \operatorname{Cov}^{-1}(\mathbf{X}) \, E_{x_0}(x_0 x_0^T) \right ) \frac{\sigma^2}{N} + \sigma^2  \quad \text{linearity, interchange}\\
        &= \operatorname{trance} \left ( \operatorname{Cov}^{-1}(\mathbf{X}) \, \operatorname{Cov}(x_0) \right ) \frac{\sigma^2}{N} + \sigma^2  \quad \text{see 2. above}\\
        &= \operatorname{trance} (I_p) \frac{\sigma^2}{N} + \sigma^2 \quad \text{as } \operatorname{Cov}(x_0) \to \operatorname{Cov}(\mathbf{X}) \\
        &= p \frac{\sigma^2}{N} + \sigma^2
\end{align}

### Ex 2.6
#### Question
Consider a regression problem with inputs $x_i$ and outputs $y_i$, and a parameterized model $f_{\theta}(x)$ to be fit by least squares. Show that if there are observations with tied or identical values of $x$, then the fit can be obtained from a reduced weighted least squares problem.

#### Solution
[ref: A Solution Manual and Notes for: The Elements of Statistical Learning by Jerome Friedman, Trevor Hastie, and Robert Tibshirani](https://www.google.com.sg/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwiF6sf2hsfLAhWSco4KHfJQCCwQFggbMAA&url=http%3A%2F%2Fwaxworksmath.com%2FAuthors%2FG_M%2FHastie%2FWriteUp%2Fweatherwax_epstein_hastie_solutions_manual.pdf&usg=AFQjCNH3VN6HgCDHtXNIbJtAjEEQNZFINA&sig2=b_zFhNYsupRwqtY62dGnwA)

Assume we have $N$ train samples, and let $N_u$ be the number of *unique* inputs $x$. And for $i$th unique $x_i$, the $y_i = \{ y_{i,1}, y_{i,2}, \dotsc, y_{i, n_i} \} $, namely, consists of $n_i$ observation.

\begin{align}
    \displaystyle \operatorname{argmin}_{\theta} \sum_{k=1}^{N} (y_k - f_{\theta}(x_k))^2 &= \operatorname{argmin}_{\theta} \sum_{i=1}^{N_u} \sum_{j=1}^{n_i} (y_{ij} - f_{\theta}(x_i))^2 \\
    &= \operatorname{argmin}_{\theta} \sum_{i=1}^{N_u} \sum_{j=1}^{n_i} y_{ij}^2 - 2 f_{\theta}(x_i) y_{ij} + f_{\theta}(x_i)^2 \\
    &= \operatorname{argmin}_{\theta} \sum_{i=1}^{N_u} n_i \left ( \color{blue}{\frac{1}{n_i} \sum_{j=1}^{n_i} y_{ij}^2} - 2 f_{\theta}(x_i) \frac{1}{n_i} \sum_{j=1}^{n_i} y_{ij} + f_{\theta}(x_i)^2 \right ) \\
    &= \operatorname{argmin}_{\theta} \sum_{i=1}^{N_u} n_i \left ( \color{red}{(\frac{1}{n_i} \sum_{j=1}^{n_i} y_{ij})^2}  - 2 f_{\theta}(x_i) \frac{1}{n_i} \sum_{j=1}^{n_i} y_{ij} + f_{\theta}(x_i)^2  - \color{red}{(\frac{1}{n_i} \sum_{j=1}^{n_i} y_{ij})^2} + \color{blue}{\frac{1}{n_i} \sum_{j=1}^{n_i} y_{ij}^2} \right ) \\
    &= \operatorname{argmin}_{\theta} \sum_{i=1}^{N_u} n_i \left ( \color{red}{\bar{y}_i^2}  - 2 f_{\theta}(x_i) \bar{y}_i^2  + f_{\theta}(x_i)^2  - \color{red}{\bar{y}_i^2} + \frac{1}{n_i} \sum_{j=1}^{n_i} y_{ij}^2 \right )  \quad \text{def: } \bar{y}_i = \frac{1}{n_i} \sum_{j=1}^{n_i} y_{ij}\\
    &= \operatorname{argmin}_{\theta} \sum_{i=1}^{N_u} n_i \left ( \left ( \bar{y}_i^2 - f_{\theta}(x_i) \right )^2  - \bar{y}_i^2 + \frac{1}{n_i} \sum_{j=1}^{n_i} y_{ij}^2 \right ) \\
    &= \operatorname{argmin}_{\theta} \left ( \sum_{i=1}^{N_u} n_i \left ( \bar{y}_i^2 - f_{\theta}(x_i) \right )^2  - \sum_{i=1}^{N_u} n_i \bar{y}_i^2 + \sum_{i=1}^{N_u} \sum_{j=1}^{n_i} y_{ij}^2  \right ) \\
    &= \operatorname{argmin}_{\theta} \left ( \sum_{i=1}^{N_u} n_i \left ( \bar{y}_i^2 - f_{\theta}(x_i) \right )^2 + \mathcal{C} \right ) \quad \text{as $y_{ij}$ is fixed} \\
    &= \operatorname{argmin}_{\theta} \sum_{i=1}^{N_u} n_i \left ( \bar{y}_i^2 - f_{\theta}(x_i) \right )^2
\end{align}

Thus, it's a *weighted* least squares as every $\bar{y_i}$ is weighted by $n_i$. And also it is a *reduced* problem since $N_u < N$.

### Ex 2.7
Suppose we have a sample of $N$ pairs $x_i, y_i$ draw i.i.d from the distribution characterized as follows:
\begin{align}
    &x_i \sim h(x), &\text{the ddesign density} \\
    &y_i = f(x_i) + \epsilon_i, &\text{$f$ is the regression function} \\
    &\epsilon_i \sim (0, \sigma^2), &\text{mean zero, variance $\sigma^2$}
\end{align}

We construct an estimator for $f$ *linear* in the $y_i$,
$$\hat{f}(x_0) = \displaystyle \sum_{i=1}^N \ell_i(x_0; \mathcal{X}) y_i,$$
where the weights $\ell_i(x_0; \mathcal{X}$ do not depend on the $y_i$, but do depend on the entire training sequence of $x_i$, denoted here by $\mathcal{X}$.

#### (a)
Show that linear regression and $k$-nearest-neighbor regression are mem- bers of this class of estimators. Describe explicitly the weights $\ell_i(x_0; \mathcal{X}$ in each of these cases.

#### solution
1. for linear regression
   $\hat{\beta} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}$

   so:
   \begin{align}
       \hat{f}(x_0) &= x_0^T \hat{\beta} \\
                    &= x_0^T (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y} \\
                    &= \displaystyle \sum_{i=1}^N \left [ x_0^T (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \right ]_i \ y_i
   \end{align}

2. for neighbor model
   \begin{align}
       \hat{f}(x_0) &= \frac{1}{k} \displaystyle \sum_{x_i \in N_k(x_0)} y_i \\
                    &= \displaystyle \sum_{i=1}^N \frac{1}{k} \ I(x_i \in N_k(x_0)) \ y_i
   \end{align}

#### (b)
Decompose the conditional mean-squared error
$$E_{\mathcal{Y} | \mathcal{X}} ( f(x_0) - \hat{f}(x_0) )^2$$
into a conditional squared bias and a conditional variance component. Like $\mathcal{X}$, $\mathcal{Y}$ represents the entire training sequence of $y_i$.

#### solution
[ref:](https://en.wikipedia.org/wiki/Mean_squared_error)
Here $\mathcal{X}$ is fixed, and $\mathcal{Y}$ varies. Also $x_0$ and $f(x_0)$ are fixed.
so:
\begin{align}
E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0)) &= E_{\mathcal{Y} | \mathcal{X}} \displaystyle \sum_{i=1}^N \ell_i(x_0; \mathcal{X}) y_i \\
        &= \displaystyle \sum_{i=1}^N \ell_i(x_0; \mathcal{X}) \, E_{\mathcal{Y} | \mathcal{X}} ( y_i ) \\
        &= \displaystyle \sum_{i=1}^N \ell_i(x_0; \mathcal{X}) \, E_{\mathcal{Y} | \mathcal{X}} ( f(x_i) + \epsilon_i ) \\
        &= \displaystyle \sum_{i=1}^N \ell_i(x_0; \mathcal{X}) \, \left ( E_{\mathcal{Y} | \mathcal{X}} ( f(x_i) ) + E_{\mathcal{Y} | \mathcal{X}} ( \epsilon_i ) \right ) \\
        &= \displaystyle \sum_{i=1}^N \ell_i(x_0; \mathcal{X}) \, E_{\mathcal{Y} | \mathcal{X}} ( f(x_i) ) \\
        &= \displaystyle \sum_{i=1}^N \ell_i(x_0; \mathcal{X}) \, f(x_i)  \\
        &= C \quad \text{constant when $\mathcal{X}$ is fixed}
\end{align}

Thus:

\begin{align}
    {} & E_{\mathcal{Y} | \mathcal{X}} ( f(x_0) - \hat{f}(x_0) )^2 \\
    = & E_{\mathcal{Y} | \mathcal{X}} ( \hat{f}(x_0) - f(x_0) )^2 \\
    = & E_{\mathcal{Y} | \mathcal{X}} \left ( \hat{f}(x_0) - E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0)  + E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0)  - f(x_0) \right )^2 \\
    = & E_{\mathcal{Y} | \mathcal{X}} \left ( \left ( \hat{f}(x_0) - E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0) \right )^2  + 2 \left ( \hat{f}(x_0) - E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0) \right ) \left ( E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0)  - f(x_0) \right ) + \left ( E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0)  - f(x_0) \right )^2 \right ) \\
    = & E_{\mathcal{Y} | \mathcal{X}} \left ( \hat{f}(x_0) - E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0) \right )^2  + E_{\mathcal{Y} | \mathcal{X}} 2 \left ( \hat{f}(x_0) - E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0) \right ) \left ( E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0)  - f(x_0) \right ) + E_{\mathcal{Y} | \mathcal{X}} \left ( E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0)  - f(x_0) \right )^2 \\
    = & \operatorname{Var}(\hat{f}(x_0) + E_{\mathcal{Y} | \mathcal{X}} 2 \left ( \hat{f}(x_0) - E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0) \right ) \left ( \underbrace{E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0)  - f(x_0)}_\text{constant} \right ) + \color{blue}{E_{\mathcal{Y} | \mathcal{X}}} \left ( \underbrace{E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0)  - f(x_0)}_\text{constant} \right )^2 \\
    = & \operatorname{Var}(\hat{f}(x_0)  +  2 \left ( f(x_0) - C \right ) \, E_{\mathcal{Y} | \mathcal{X}} \left ( \hat{f}(x_0) - E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0) \right )  + \operatorname{Bias}^2 (\hat{f}(x_0), f(x_0)) \\
    = & \operatorname{Var}(\hat{f}(x_0)  +  2 \left ( f(x_0) - C \right ) \, \left ( E_{\mathcal{Y} | \mathcal{X}} \hat{f}(x_0) - E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0) \right )  + \operatorname{Bias}^2 (\hat{f}(x_0), f(x_0)) \\
    = & \operatorname{Var}(\hat{f}(x_0)  +  \operatorname{Bias}^2 (\hat{f}(x_0), f(x_0)) \\
\end{align}

#### (c)
Decompose the (unconditional) mean-squared error
$$E_{\mathcal{Y}, \mathcal{X}} ( f(x_0) - \hat{f}(x_0) )^2$$
into a squared bias and a variance component.

#### solution
\begin{align}
    {} & E_{\mathcal{Y}, \mathcal{X}} (f(x_0) - \hat{f}(x_0))^2 \\
    = & E_{\mathcal{X}} E_{\mathcal{Y} | \mathcal{X}} (f(x_0) - \hat{f}(x_0))^2 \\
\end{align}
use similar method as above.

#### (d)
Establish a relationship between the squared biases and variances in the above two cases.

#### solution
##### 1. for variance,
As we known in (b),
$$E_{\mathcal{Y} | \mathcal{X}}(\hat{f}(x_0)) = \displaystyle \sum_{i=1}^N \ell_i(x_0; \mathcal{X}) \, f(x_i)$$

and also:
$$ \hat{f}(x_0) = \displaystyle \sum_{i=1}^N \ell_i(x_0; \mathcal{X}) y_i $$
$$ y_i = f(x_i) + \epsilon_i $$

Thus,
\begin{align}
    \operatorname{Var}(\hat{f}(x_0)) &= E_{\mathcal{Y} | \mathcal{X}} \left ( \hat{f}(x_0) - E_{\mathcal{Y} | \mathcal{X}} \left ( \hat{f}(x_0) \right ) \right )^2 \\
            &= E_{\mathcal{Y} | \mathcal{X}} \left ( \displaystyle \sum_{i=1}^N \ell_i(x_0; \mathcal{X}) y_i - \displaystyle \sum_{i=1}^N \ell_i(x_0; \mathcal{X}) \, f(x_i) \right )^2 \\
            &= E_{\mathcal{Y} | \mathcal{X}} \left ( \displaystyle \sum_{i=1}^N \ell_i(x_0; \mathcal{X}) \left ( y_i - f(x_i) \right ) \right )^2 \\
            &= E_{\mathcal{Y} | \mathcal{X}} \left ( \displaystyle \sum_{i=1}^N \ell_i(x_0; \mathcal{X}) \epsilon_i \right )^2 \\
\end{align}

##### 2. for squared bias,
\begin{align}
    \operatorname{Bias}^2 (\hat{f}(x_0), f(x_0)) &= \left ( E_{\mathcal{Y} | \mathcal{X}} \hat{f}(x_0) - f(x_0) \right )^2 \\
            &= \left( \displaystyle \sum_{i=1}^N \ell_i(x_0; \mathcal{X}) \, f(x_i) - f(x_0) \right )^2 \\
\end{align}

##### 3. guess
variance is only affected by $\epsilon$, while squared bias is only affected by $f(x)$. Because $\epsilon$ is independent with $f(x)$, it might not possible that there is a relation between variance and squared bias.

### Ex 2.8
Compare the classification performance of linear regression and $k$-nearest neighbor classification on the zipcode data. In particular, consider only the 2’s and 3’s, and $k$ = 1, 3, 5, 7 and 15. Show both the training and test error for each choice. The zipcode data are available from the book website www-stat.stanford.edu/ElemStatLearn.

```python
train_dat = pd.read_csv('./res/zip.train', header=None, sep=' ')
train_dat.rename(columns={0: 'digital'}, inplace=True)
train_dat = train_dat.query('digital == 2 or digital == 3')
train_dat.dropna(axis=1, inplace=True)
```

```python
train_dat.set_index('digital', inplace=True)
```

```python
train_dat.head(3)
```

<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>4</th>
      <th>5</th>
      <th>6</th>
      <th>7</th>
      <th>8</th>
      <th>9</th>
      <th>10</th>
      <th>...</th>
      <th>247</th>
      <th>248</th>
      <th>249</th>
      <th>250</th>
      <th>251</th>
      <th>252</th>
      <th>253</th>
      <th>254</th>
      <th>255</th>
      <th>256</th>
    </tr>
    <tr>
      <th>digital</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>3</th>
      <td>-1</td>
      <td>-1</td>
      <td>-1</td>
      <td>-1.00</td>
      <td>-1.000</td>
      <td>-0.928</td>
      <td>-0.204</td>
      <td>0.751</td>
      <td>0.466</td>
      <td>0.234</td>
      <td>...</td>
      <td>0.466</td>
      <td>0.639</td>
      <td>1.000</td>
      <td>1.000</td>
      <td>0.791</td>
      <td>0.439</td>
      <td>-0.199</td>
      <td>-0.883</td>
      <td>-1</td>
      <td>-1</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-1</td>
      <td>-1</td>
      <td>-1</td>
      <td>-0.83</td>
      <td>0.442</td>
      <td>1.000</td>
      <td>1.000</td>
      <td>0.479</td>
      <td>-0.328</td>
      <td>-0.947</td>
      <td>...</td>
      <td>1.000</td>
      <td>0.671</td>
      <td>0.345</td>
      <td>-0.507</td>
      <td>-1.000</td>
      <td>-1.000</td>
      <td>-1.000</td>
      <td>-1.000</td>
      <td>-1</td>
      <td>-1</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-1</td>
      <td>-1</td>
      <td>-1</td>
      <td>-1.00</td>
      <td>-1.000</td>
      <td>-0.104</td>
      <td>0.549</td>
      <td>0.579</td>
      <td>0.579</td>
      <td>0.857</td>
      <td>...</td>
      <td>0.388</td>
      <td>0.579</td>
      <td>0.811</td>
      <td>1.000</td>
      <td>1.000</td>
      <td>0.715</td>
      <td>0.107</td>
      <td>-0.526</td>
      <td>-1</td>
      <td>-1</td>
    </tr>
  </tbody>
</table>
<p>3 rows × 256 columns</p>
</div>

```python
plt.imshow(np.reshape(train_dat.iloc[0].values, (16, 16)))
```

```python
test_dat = pd.read_csv('./res/zip.test', header=None, sep=' ')
test_dat.rename(columns={0: 'digital'}, inplace=True)
test_dat = test_dat.query('digital == 2 or digital == 3')
test_dat.dropna(axis=1, inplace=True)
```

```python
test_dat.set_index('digital', inplace=True)
```

```python
test_dat.head(3)
```

<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>4</th>
      <th>5</th>
      <th>6</th>
      <th>7</th>
      <th>8</th>
      <th>9</th>
      <th>10</th>
      <th>...</th>
      <th>247</th>
      <th>248</th>
      <th>249</th>
      <th>250</th>
      <th>251</th>
      <th>252</th>
      <th>253</th>
      <th>254</th>
      <th>255</th>
      <th>256</th>
    </tr>
    <tr>
      <th>digital</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>3</th>
      <td>-1.000</td>
      <td>-1.000</td>
      <td>-1.000</td>
      <td>-0.593</td>
      <td>0.700</td>
      <td>1.000</td>
      <td>1.000</td>
      <td>1.000</td>
      <td>1.000</td>
      <td>0.853</td>
      <td>...</td>
      <td>1.000</td>
      <td>0.717</td>
      <td>0.333</td>
      <td>0.162</td>
      <td>-0.393</td>
      <td>-1.000</td>
      <td>-1</td>
      <td>-1</td>
      <td>-1</td>
      <td>-1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>-0.996</td>
      <td>0.572</td>
      <td>0.396</td>
      <td>0.063</td>
      <td>-0.506</td>
      <td>-0.847</td>
      <td>-1.000</td>
      <td>-1.000</td>
      <td>-1.000</td>
      <td>-1.000</td>
      <td>...</td>
      <td>-0.668</td>
      <td>-1.000</td>
      <td>-1.000</td>
      <td>-1.000</td>
      <td>-1.000</td>
      <td>-1.000</td>
      <td>-1</td>
      <td>-1</td>
      <td>-1</td>
      <td>-1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>-1.000</td>
      <td>-1.000</td>
      <td>0.469</td>
      <td>0.413</td>
      <td>1.000</td>
      <td>1.000</td>
      <td>0.462</td>
      <td>-0.116</td>
      <td>-0.937</td>
      <td>-1.000</td>
      <td>...</td>
      <td>1.000</td>
      <td>1.000</td>
      <td>1.000</td>
      <td>0.270</td>
      <td>-0.280</td>
      <td>-0.855</td>
      <td>-1</td>
      <td>-1</td>
      <td>-1</td>
      <td>-1</td>
    </tr>
  </tbody>
</table>
<p>3 rows × 256 columns</p>
</div>

```python
def eva(conf, train_dat, test_dat):
    x_train = train_dat.values
    y_train = train_dat.index.values

    conf['cls'].fit(x_train, y_train)

    x_test = test_dat.values
    y_test = test_dat.index.values

    y_pred = conf['cls'].predict(x_test)

    accuracy = sklearn.metrics.accuracy_score(y_test, y_pred)
    print('{}, parameter: {}, accuracy: {:.4}'.format(conf['name'], conf['parameter'], accuracy))
```

```python
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier

configuration = [
    {'cls': LogisticRegression(), 'name': 'LogisticRegression', 'parameter': None},
    {'cls': KNeighborsClassifier(n_neighbors=1), 'name': 'KNeighborsClassifier', 'parameter': 'N=1'},
    {'cls': KNeighborsClassifier(n_neighbors=3), 'name': 'KNeighborsClassifier', 'parameter': 'N=3'},
    {'cls': KNeighborsClassifier(n_neighbors=5), 'name': 'KNeighborsClassifier', 'parameter': 'N=5'},
    {'cls': KNeighborsClassifier(n_neighbors=7), 'name': 'KNeighborsClassifier', 'parameter': 'N=7'},
    {'cls': KNeighborsClassifier(n_neighbors=15), 'name': 'KNeighborsClassifier', 'parameter': 'N=15'},
]

for conf in configuration:
    eva(conf, train_dat, test_dat)
```

    LogisticRegression, parameter: None, accuracy: 0.9643
    KNeighborsClassifier, parameter: N=1, accuracy: 0.9753
    KNeighborsClassifier, parameter: N=3, accuracy: 0.9698
    KNeighborsClassifier, parameter: N=5, accuracy: 0.9698
    KNeighborsClassifier, parameter: N=7, accuracy: 0.967
    KNeighborsClassifier, parameter: N=15, accuracy: 0.9615

### 2.9
Consider a linear regression model with $p$ parameters, fit by least squares to a set of training data $(x_1,y_1), \dotsc,(x_N,y_N)$ drawn at random from a population. Let $\hat{\beta}$  be the least squares estimate. Suppose we have some test data $(\hat{x}_1, \hat{y}_1), \dotsc,(\hat{x}_N, \hat{y}_N)$ drawn at random from the same propulation as the training data.

If $R_{tr}(\beta) = \frac{1}{N} \sum_1^N (y_i - \beta^T x_i)^2$, and $R_{te}(\beta) = \frac{1}{M} \sum_1^M (\hat{y}_i - \beta^T \hat{x}_i)^2$, prove that
$$E[R_{tr}(\hat{\beta})] \leq E[R_{te}(\hat{\beta})],$$
where the expectation are over all that is random in each expression.

#### solution
Ref: Hint from [Homework 2 - Hector Corrada Bravo](http://www.cbcb.umd.edu/~hcorrada/PracticalML/assignments/hw2.pdf)

define: $\beta^{\ast}$ is the least squares estimate in **test data**.

1.
As both training data and test data are picked randomly, so obviously:
$$E[R_{tr}(\hat{\beta})] = E[R_{te}(\beta^{\ast})]$$

2.
On the other hand, because $\beta^{\ast}$ is the least squares estimate in **test data**, namely,
$$\beta^\ast = \operatorname{argmin}_\beta \frac{1}{M} \sum_1^M (\hat{y}_i - \beta^T \hat{x}_i)^2 $$
so obviously:
$$R_{te}(\beta^\ast) = \frac{1}{M} \sum_1^M (\hat{y}_i - \left (\color{blue}{\beta^{\ast}} \right )^T \hat{x}_i)^2 \leq  \frac{1}{M} \sum_1^M (\hat{y}_i - \color{blue}{\hat{\beta}}^T \hat{x}_i)^2 = R_{te}(\hat{\beta}) $$
Thus:
$$E[R_{te}(\beta^\ast)] \leq E[R_{te}(\hat{\beta})]$$

3.
Final, we get:
$$E[R_{tr}(\hat{\beta})] = E[R_{te}(\beta^{\ast})] \leq E[R_{te}(\hat{\beta})]$$
namely,
$$E[R_{tr}(\hat{\beta})] \leq E[R_{te}(\hat{\beta})]$$


# PHY321: Conservative Forces, Examples and start Harmonic Oscillations

**[Morten Hjorth-Jensen](http://mhjgit.github.io/info/doc/web/)**, Department of Physics and Astronomy and Facility for Rare Ion Beams (FRIB), Michigan State University, USA and Department of Physics, University of Oslo, Norway

Date: **Feb 14, 2021**

Copyright 1999-2021, [Morten Hjorth-Jensen](http://mhjgit.github.io/info/doc/web/). Released under CC Attribution-NonCommercial 4.0 license

## Aims and Overarching Motivation

### Monday

Short repetition from last week about conservative forces. Discussion
of conditions for conservative forces and the Earth-Sun gravitional
force example.  **Reading suggestion**: Taylor sections 4.3, 4.4 and 4.8.

### Wednesday

Potential curves and discussion of the Earth-Sun example, analytical and numerical considerations.
**Reading suggestions**: Taylor section 4.6, 4.8 and 4.9.

### Friday

Earth-Sun,  conservative forces and potential energy.
**Reading suggestion**: Taylor sections 4.8 and 4.9.

If we get time, we start with harmonic oscillations and Hooke's law. **Reading suggestion**: Taylor section 5.1.

## One Figure to Rule All Forces (thx to Julie)

Figure 1:

## Repetition from last week: Work, Energy, Momentum and Conservation laws

Energy conservation is most convenient as a strategy for addressing
problems where time does not appear. For example, a particle goes
from position $x_0$ with speed $v_0$, to position $x_f$; what is its
new speed? However, it can also be applied to problems where time
does appear, such as in solving for the trajectory $x(t)$, or
equivalently $t(x)$.

## Energy Conservation
Energy is conserved in the case where the potential energy, $V(\boldsymbol{r})$, depends only on position, and not on time. The force is determined by $V$,

$$
\begin{equation}
\boldsymbol{F}(\boldsymbol{r})=-\boldsymbol{\nabla} V(\boldsymbol{r}).
\label{_auto1} \tag{1}
\end{equation}
$$

## Conservative forces

We say a force is conservative if it satisfies the following conditions:
1. The force $\boldsymbol{F}$ acting on an object only depends on the position $\boldsymbol{r}$, that is $\boldsymbol{F}=\boldsymbol{F}(\boldsymbol{r})$.

2. For any two points $\boldsymbol{r}_1$ and $\boldsymbol{r}_2$, the work done by the force $\boldsymbol{F}$ on the displacement between these two points is independent of the path taken.

3. Finally, the **curl** of the force is zero $\boldsymbol{\nabla}\times\boldsymbol{F}=0$.

## Forces and Potentials

The energy $E$ of a given system is defined as the sum of kinetic and potential energies,

$$
E=K+V(\boldsymbol{r}).
$$

We define the potential energy at a point $\boldsymbol{r}$ as the negative work done from a starting point $\boldsymbol{r}_0$ to a final point $\boldsymbol{r}$

$$
V(\boldsymbol{r})=-W(\boldsymbol{r}_0\rightarrow\boldsymbol{r})= -\int_{\boldsymbol{r}_0}^{\boldsymbol{r}}d\boldsymbol{r}'\boldsymbol{F}(\boldsymbol{r}').
$$

If the potential depends on the path taken between these two points there is no unique potential.

## Example (relevant for homework 5)

We study a classical electron which moves in the $x$-direction along a surface. The force from the surface is

$$
\boldsymbol{F}(x)=-F_0\sin{(\frac{2\pi x}{b})}\boldsymbol{e}_1.
$$

The constant $b$ represents the distance between atoms at the surface of the material, $F_0$ is a constant and $x$ is the position of the electron.

This is indeed a conservative force since it depends only on position
and its **curl** is zero, that is $-\boldsymbol{\nabla}\times \boldsymbol{F}=0$. This means that energy is conserved and the
integral over the work done by the force is independent of the path
taken. We will come back to this in more detail next week.

## Example Continues

Using the work-energy theorem we can find the work $W$ done when
moving an electron from a position $x_0$ to a final position $x$
through the integral

$$
W=-\int_{x_0}^x \boldsymbol{F}(x')dx' =  \int_{x_0}^x F_0\sin{(\frac{2\pi x'}{b})} dx',
$$

which results in

$$
W=\frac{F_0b}{2\pi}\left[\cos{(\frac{2\pi x}{b})}-\cos{(\frac{2\pi x_0}{b})}\right].
$$

Since this is related to the change in kinetic energy we have, with $v_0$ being the initial velocity at a  time $t_0$,

$$
v  = \pm\sqrt{\frac{2}{m}\frac{F_0b}{2\pi}\left[\cos{(\frac{2\pi x}{b})}-\cos{(\frac{2\pi x_0}{b})}\right]+v_0^2}.
$$

## The potential energy from this example

The potential energy, due to energy conservation is

$$
V(x)=V(x_0)+\frac{1}{2}mv_0^2-\frac{1}{2}mv^2,
$$

with $v$ given by the velocity from above.

We can now, in order to find a more explicit expression for the
potential energy at a given value $x$, define a zero level value for
the potential. The potential is defined, using the work-energy
theorem, as

$$
V(x)=V(x_0)+\int_{x_0}^x (-F(x'))dx',
$$

and if you recall the definition of the indefinite integral, we can rewrite this as

$$
V(x)=\int (-F(x'))dx'+C,
$$

where $C$ is an undefined constant. The force is defined as the
gradient of the potential, and in that case the undefined constant
vanishes. The constant does not affect the force we derive from the
potential.

We have then

$$
V(x)=V(x_0)-\int_{x_0}^x \boldsymbol{F}(x')dx',
$$

which results in

$$
V(x)=\frac{F_0b}{2\pi}\left[\cos{(\frac{2\pi x}{b})}-\cos{(\frac{2\pi x_0}{b})}\right]+V(x_0).
$$

We can now define

$$
\frac{F_0b}{2\pi}\cos{(\frac{2\pi x_0}{b})}=V(x_0),
$$

which gives

$$
V(x)=\frac{F_0b}{2\pi}\left[\cos{(\frac{2\pi x}{b})}\right].
$$

## Force and Potential

We have defined work as the energy resulting from a net force acting
on an object (or sseveral objects), that is

$$
W(\boldsymbol{r}\rightarrow \boldsymbol{r}+d\boldsymbol{r})= \boldsymbol{F}(\boldsymbol{r})d\boldsymbol{r}.
$$

If we write out this for each component we have

$$
W(\boldsymbol{r}\rightarrow \boldsymbol{r}+d\boldsymbol{r})=\boldsymbol{F}(\boldsymbol{r})d\boldsymbol{r}=F_xdx+F_ydy+F_zdz.
$$

The work done from an initial position to a final one defines also the difference in potential energies

$$
W(\boldsymbol{r}\rightarrow \boldsymbol{r}+d\boldsymbol{r})=-\left[V(\boldsymbol{r}+d\boldsymbol{r})-V(\boldsymbol{r})\right].
$$

## Getting to $\boldsymbol{F}(\boldsymbol{r})=-\boldsymbol{\nabla} V(\boldsymbol{r})$

We can write out the differences in potential energies as

$$
V(\boldsymbol{r}+d\boldsymbol{r})-V(\boldsymbol{r})=V(x+dx,y+dy,z+dz)-V(x,y,z)=dV,
$$

and using the expression the differential of a multi-variable function $f(x,y,z)$

$$
df=\frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz,
$$

we can write the expression for the work done as

$$
W(\boldsymbol{r}\rightarrow \boldsymbol{r}+d\boldsymbol{r})=-dV=-\left[\frac{\partial V}{\partial x}dx+\frac{\partial V}{\partial y}dy+\frac{\partial V}{\partial z}dz \right].
$$

## Final expression

Comparing the last equation with

$$
W(\boldsymbol{r}\rightarrow \boldsymbol{r}+d\boldsymbol{r})=F_xdx+F_ydy+F_zdz,
$$

we have

$$
F_xdx+F_ydy+F_zdz=-\left[\frac{\partial V}{\partial x}dx+\frac{\partial V}{\partial y}dy+\frac{\partial V}{\partial z}dz \right],
$$

leading to

$$
F_x=-\frac{\partial V}{\partial x},
$$

and

$$
F_y=-\frac{\partial V}{\partial y},
$$

and

$$
F_z=-\frac{\partial V}{\partial z},
$$

or just

$$
\boldsymbol{F}=-\frac{\partial V}{\partial x}\boldsymbol{e}_1-\frac{\partial V}{\partial y}\boldsymbol{e}_2-\frac{\partial V}{\partial z}\boldsymbol{e}_3=-\boldsymbol{\nabla}V(\boldsymbol{r}).
$$

And this connection is the one we wanted to show.

## Net Energy

The net energy, $E=V+K$ where $K$ is the kinetic energy, is then conserved,

$$
\begin{eqnarray}
\frac{d}{dt}(K+V)&=&\frac{d}{dt}\left(\frac{m}{2}(v_x^2+v_y^2+v_z^2)+V(\boldsymbol{r})\right)\\
\nonumber
&=&m\left(v_x\frac{dv_x}{dt}+v_y\frac{dv_y}{dt}+v_z\frac{dv_z}{dt}\right)
+\partial_xV\frac{dx}{dt}+\partial_yV\frac{dy}{dt}+\partial_zV\frac{dz}{dt}\\
\nonumber
&=&v_xF_x+v_yF_y+v_zF_z-F_xv_x-F_yv_y-F_zv_z=0.
\end{eqnarray}
$$

## In Vector Notation

The same proof can be written more compactly with vector notation,

$$
\begin{eqnarray}
\frac{d}{dt}\left(\frac{m}{2}v^2+V(\boldsymbol{r})\right)
&=&m\boldsymbol{v}\cdot\dot{\boldsymbol{v}}+\boldsymbol{\nabla} V(\boldsymbol{r})\cdot\dot{\boldsymbol{r}}\\
\nonumber
&=&\boldsymbol{v}\cdot\boldsymbol{F}-\boldsymbol{F}\cdot\boldsymbol{v}=0.
\end{eqnarray}
$$

Inverting the expression for kinetic energy,

$$
\begin{equation}
v=\sqrt{2K/m}=\sqrt{2(E-V)/m},
\label{_auto2} \tag{2}
\end{equation}
$$

allows one to solve for the one-dimensional trajectory $x(t)$, by finding $t(x)$,

$$
\begin{equation}
t=\int_{x_0}^x \frac{dx'}{v(x')}=\int_{x_0}^x\frac{dx'}{\sqrt{2(E-V(x'))/m}}.
\label{_auto3} \tag{3}
\end{equation}
$$

Note this would be much more difficult in higher dimensions, because
you would have to determine which points, $x,y,z$, the particles might
reach in the trajectory, whereas in one dimension you can typically
tell by simply seeing whether the kinetic energy is positive at every
point between the old position and the new position.

## Harmonic Oscillator Potential

Consider a simple harmonic oscillator potential, $V(x)=kx^2/2$, with a particle emitted from $x=0$ with velocity $v_0$. Solve for the trajectory $t(x)$,

$$
\begin{eqnarray}
t&=&\int_{0}^x \frac{dx'}{\sqrt{2(E-kx^2/2)/m}}\\
\nonumber
&=&\sqrt{m/k}\int_0^x~\frac{dx'}{\sqrt{x_{\rm max}^2-x^{\prime 2}}},~~~x_{\rm max}^2=2E/k.
\end{eqnarray}
$$

Here $E=mv_0^2/2$ and $x_{\rm max}$ is defined as the maximum
displacement before the particle turns around. This integral is done
by the substitution $\sin\theta=x/x_{\rm max}$.

$$
\begin{eqnarray}
(k/m)^{1/2}t&=&\sin^{-1}(x/x_{\rm max}),\\
\nonumber
x&=&x_{\rm max}\sin\omega t,~~~\omega=\sqrt{k/m}.
\end{eqnarray}
$$

## The Earth-Sun system

We will now venture into a study of a system which is energy
conserving. The aim is to see if we (since it is not possible to solve
the general equations analytically) we can develop stable numerical
algorithms whose results we can trust!

We solve the equations of motion numerically. We will also compute
quantities like the energy numerically.

We start with a simpler case first, the Earth-Sun system  in two dimensions only.  The gravitational force $F_G$ on the earth from the sun is

$$
\boldsymbol{F}_G=-\frac{GM_{\odot}M_E}{r^3}\boldsymbol{r},
$$

where $G$ is the gravitational constant,

$$
M_E=6\times 10^{24}\mathrm{Kg},
$$

the mass of Earth,

$$
M_{\odot}=2\times 10^{30}\mathrm{Kg},
$$

the mass of the Sun and

$$
r=1.5\times 10^{11}\mathrm{m},
$$

is the distance between Earth and the Sun. The latter defines what we call an astronomical unit **AU**.

## The Earth-Sun system, Newton's Laws

From Newton's second law we have then for the $x$ direction

$$
\frac{d^2x}{dt^2}=-\frac{F_{x}}{M_E},
$$

and

$$
\frac{d^2y}{dt^2}=-\frac{F_{y}}{M_E},
$$

for the $y$ direction.

Here we will use  that  $x=r\cos{(\theta)}$, $y=r\sin{(\theta)}$ and

$$
r = \sqrt{x^2+y^2}.
$$

We can rewrite

$$
F_{x}=-\frac{GM_{\odot}M_E}{r^2}\cos{(\theta)}=-\frac{GM_{\odot}M_E}{r^3}x,
$$

and

$$
F_{y}=-\frac{GM_{\odot}M_E}{r^2}\sin{(\theta)}=-\frac{GM_{\odot}M_E}{r^3}y,
$$

for the $y$ direction.

## The Earth-Sun system, rewriting the Equations

We can rewrite these two equations

$$
F_{x}=-\frac{GM_{\odot}M_E}{r^2}\cos{(\theta)}=-\frac{GM_{\odot}M_E}{r^3}x,
$$

and

$$
F_{y}=-\frac{GM_{\odot}M_E}{r^2}\sin{(\theta)}=-\frac{GM_{\odot}M_E}{r^3}y,
$$

as four first-order coupled differential equations

4
3

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

4
4

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

4
5

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
\frac{dy}{dt}=v_y.
$$

## Building a code for the solar system, final coupled equations

The four coupled differential equations

4
7

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

4
8

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

4
9

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
\frac{dy}{dt}=v_y,
$$

can be turned into dimensionless equations or we can introduce astronomical units with $1$ AU = $1.5\times 10^{11}$.

Using the equations from circular motion (with $r =1\mathrm{AU}$)

$$
\frac{M_E v^2}{r} = F = \frac{GM_{\odot}M_E}{r^2},
$$

we have

$$
GM_{\odot}=v^2r,
$$

and using that the velocity of Earth (assuming circular motion) is
$v = 2\pi r/\mathrm{yr}=2\pi\mathrm{AU}/\mathrm{yr}$, we have

$$
GM_{\odot}= v^2r = 4\pi^2 \frac{(\mathrm{AU})^3}{\mathrm{yr}^2}.
$$

## Building a code for the solar system, discretized equations

The four coupled differential equations can then be discretized using Euler's method as (with step length $h$)

5
4

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

5
5

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

5
6

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
y_{i+1}=y_i+hv_{y,i},
$$

## Code Example with Euler's Method

The code here implements Euler's method for the Earth-Sun system using a more compact way of representing the vectors. Alternatively, you could have spelled out all the variables $v_x$, $v_y$, $x$ and $y$ as one-dimensional arrays.

```python
%matplotlib inline

# Common imports
import numpy as np
import pandas as pd
from math import *
import matplotlib.pyplot as plt
import os

# Where to save the figures and data files
PROJECT_ROOT_DIR = "Results"
FIGURE_ID = "Results/FigureFiles"
DATA_ID = "DataFiles/"

if not os.path.exists(PROJECT_ROOT_DIR):
    os.mkdir(PROJECT_ROOT_DIR)

if not os.path.exists(FIGURE_ID):
    os.makedirs(FIGURE_ID)

if not os.path.exists(DATA_ID):
    os.makedirs(DATA_ID)

def image_path(fig_id):
    return os.path.join(FIGURE_ID, fig_id)

def data_path(dat_id):
    return os.path.join(DATA_ID, dat_id)

def save_fig(fig_id):
    plt.savefig(image_path(fig_id) + ".png", format='png')

DeltaT = 0.001
#set up arrays
tfinal = 10 # in years
n = ceil(tfinal/DeltaT)
# set up arrays for t, a, v, and x
t = np.zeros(n)
v = np.zeros((n,2))
r = np.zeros((n,2))
# Initial conditions as compact 2-dimensional arrays
r0 = np.array([1.0,0.0])
v0 = np.array([0.0,2*pi])
r[0] = r0
v[0] = v0
Fourpi2 = 4*pi*pi
# Start integrating using Euler's method
for i in range(n-1):
    # Set up the acceleration
    # Here you could have defined your own function for this
    rabs = sqrt(sum(r[i]*r[i]))
    a =  -Fourpi2*r[i]/(rabs**3)
    # update velocity, time and position using Euler's forward method
    v[i+1] = v[i] + DeltaT*a
    r[i+1] = r[i] + DeltaT*v[i]
    t[i+1] = t[i] + DeltaT
# Plot position as function of time
fig, ax = plt.subplots()
#ax.set_xlim(0, tfinal)
ax.set_ylabel('x[m]')
ax.set_xlabel('y[m]')
ax.plot(r[:,0], r[:,1])
fig.tight_layout()
save_fig("EarthSunEuler")
plt.show()
```

## Problems with Euler's Method

We notice here that Euler's method doesn't give a stable orbit. It
means that we cannot trust Euler's method. In a deeper way, as we will
see in homework 5, Euler's method does not conserve energy. It is an
example of an integrator which is not
[symplectic](https://en.wikipedia.org/wiki/Symplectic_integrator).

Here we present thus two methods, which with simple changes allow us to avoid these pitfalls. The simplest possible extension is the so-called Euler-Cromer method.
The changes we need to make to our code are indeed marginal here.
We need simply to replace

```python
    r[i+1] = r[i] + DeltaT*v[i]
```

in the above code with the velocity at the new time $t_{i+1}$

```python
    r[i+1] = r[i] + DeltaT*v[i+1]
```

By this simple caveat we get stable orbits.
Below we derive the Euler-Cromer method as well as one of the most utlized algorithms for sovling the above type of problems, the so-called Velocity-Verlet method.

## Deriving the Euler-Cromer Method

Let us repeat Euler's method.
We have a differential equation

$$
\begin{equation}
y'(t_i)=f(t_i,y_i)
\label{_auto4} \tag{4}
\end{equation}
$$

and if we truncate at the first derivative, we have from the Taylor expansion

$$
\begin{equation}
y_{i+1}=y(t_i) + (\Delta t) f(t_i,y_i) + O(\Delta t^2), \label{eq:euler} \tag{5}
\end{equation}
$$

which when complemented with $t_{i+1}=t_i+\Delta t$ forms
the algorithm for the well-known Euler method.
Note that at every step we make an approximation error
of the order of $O(\Delta t^2)$, however the total error is the sum over all
steps $N=(b-a)/(\Delta t)$ for $t\in [a,b]$, yielding thus a global error which goes like
$NO(\Delta t^2)\approx O(\Delta t)$.

To make Euler's method more precise we can obviously
decrease $\Delta t$ (increase $N$), but this can lead to loss of numerical precision.
Euler's method is not recommended for precision calculation,
although it is handy to use in order to get a first
view on how a solution may look like.

Euler's method is asymmetric in time, since it uses information about the derivative at the beginning
of the time interval. This means that we evaluate the position at $y_1$ using the velocity
at $v_0$. A simple variation is to determine $x_{n+1}$ using the velocity at
$v_{n+1}$, that is (in a slightly more generalized form)

$$
\begin{equation}
y_{n+1}=y_{n}+ v_{n+1}+O(\Delta t^2)
\label{_auto5} \tag{6}
\end{equation}
$$

and

$$
\begin{equation}
v_{n+1}=v_{n}+(\Delta t) a_{n}+O(\Delta t^2).
\label{_auto6} \tag{7}
\end{equation}
$$

The acceleration $a_n$ is a function of $a_n(y_n, v_n, t_n)$ and needs to be evaluated
as well. This is the Euler-Cromer method.

**Exercise**: go back to the above code with Euler's method and add the Euler-Cromer method.

## Deriving the Velocity-Verlet Method

Let us stay with $x$ (position) and $v$ (velocity) as the quantities we are interested in.

We have the Taylor expansion for the position given by

$$
x_{i+1} = x_i+(\Delta t)v_i+\frac{(\Delta t)^2}{2}a_i+O((\Delta t)^3).
$$

The corresponding expansion for the velocity is

$$
v_{i+1} = v_i+(\Delta t)a_i+\frac{(\Delta t)^2}{2}v^{(2)}_i+O((\Delta t)^3).
$$

Via Newton's second law we have normally an analytical expression for the derivative of the velocity, namely

$$
a_i= \frac{d^2 x}{dt^2}\vert_{i}=\frac{d v}{dt}\vert_{i}= \frac{F(x_i,v_i,t_i)}{m}.
$$

If we add to this the corresponding expansion for the derivative of the velocity

$$
v^{(1)}_{i+1} = a_{i+1}= a_i+(\Delta t)v^{(2)}_i+O((\Delta t)^2)=a_i+(\Delta t)v^{(2)}_i+O((\Delta t)^2),
$$

and retain only terms up to the second derivative of the velocity since our error goes as $O(h^3)$, we have

$$
(\Delta t)v^{(2)}_i\approx a_{i+1}-a_i.
$$

We can then rewrite the Taylor expansion for the velocity as

$$
v_{i+1} = v_i+\frac{(\Delta t)}{2}\left( a_{i+1}+a_{i}\right)+O((\Delta t)^3).
$$

## The velocity Verlet method

Our final equations for the position and the velocity become then

$$
x_{i+1} = x_i+(\Delta t)v_i+\frac{(\Delta t)^2}{2}a_{i}+O((\Delta t)^3),
$$

and

$$
v_{i+1} = v_i+\frac{(\Delta t)}{2}\left(a_{i+1}+a_{i}\right)+O((\Delta t)^3).
$$

Note well that the term $a_{i+1}$ depends on the position at $x_{i+1}$. This means that you need to calculate
the position at the updated time $t_{i+1}$ before the computing the next velocity.  Note also that the derivative of the velocity at the time
$t_i$ used in the updating of the position can be reused in the calculation of the velocity update as well.

## Adding the Velocity-Verlet Method

We can now easily add the Verlet method to our original code as

```python
DeltaT = 0.01
#set up arrays
tfinal = 10
n = ceil(tfinal/DeltaT)
# set up arrays for t, a, v, and x
t = np.zeros(n)
v = np.zeros((n,2))
r = np.zeros((n,2))
# Initial conditions as compact 2-dimensional arrays
r0 = np.array([1.0,0.0])
v0 = np.array([0.0,2*pi])
r[0] = r0
v[0] = v0
Fourpi2 = 4*pi*pi
# Start integrating using the Velocity-Verlet  method
for i in range(n-1):
    # Set up forces, air resistance FD, note now that we need the norm of the vecto
    # Here you could have defined your own function for this
    rabs = sqrt(sum(r[i]*r[i]))
    a =  -Fourpi2*r[i]/(rabs**3)
    # update velocity, time and position using the Velocity-Verlet method
    r[i+1] = r[i] + DeltaT*v[i]+0.5*(DeltaT**2)*a
    rabs = sqrt(sum(r[i+1]*r[i+1]))
    anew = -4*(pi**2)*r[i+1]/(rabs**3)
    v[i+1] = v[i] + 0.5*DeltaT*(a+anew)
    t[i+1] = t[i] + DeltaT
# Plot position as function of time
fig, ax = plt.subplots()
ax.set_ylabel('x[m]')
ax.set_xlabel('y[m]')
ax.plot(r[:,0], r[:,1])
fig.tight_layout()
save_fig("EarthSunVV")
plt.show()
```

You can easily generalize the calculation of the forces by defining a function
which takes in as input the various variables. We leave this as a challenge to you.

## Harmonic Oscillator

The harmonic oscillator is omnipresent in physics. Although you may think
of this as being related to springs, it, or an equivalent
mathematical representation, appears in just about any problem where a
mode is sitting near its potential energy minimum. At that point,
$\partial_x V(x)=0$, and the first non-zero term (aside from a
constant) in the potential energy is that of a harmonic oscillator. In
a solid, sound modes (phonons) are built on a picture of coupled
harmonic oscillators, and in relativistic field theory the fundamental
interactions are also built on coupled oscillators positioned
infinitesimally close to one another in space. The phenomena of a
resonance of an oscillator driven at a fixed frequency plays out
repeatedly in atomic, nuclear and high-energy physics, when quantum
mechanically the evolution of a state oscillates according to
$e^{-iEt}$ and exciting discrete quantum states has very similar
mathematics as exciting discrete states of an oscillator.

## Harmonic Oscillator, deriving the Equations
The potential energy for a single particle as a function of its position $x$ can be written as a Taylor expansion about some point $x_0$

$$
\begin{equation}
V(x)=V(x_0)+(x-x_0)\left.\partial_xV(x)\right|_{x_0}+\frac{1}{2}(x-x_0)^2\left.\partial_x^2V(x)\right|_{x_0}
+\frac{1}{3!}\left.\partial_x^3V(x)\right|_{x_0}+\cdots
\label{_auto7} \tag{8}
\end{equation}
$$

If the position $x_0$ is at the minimum of the resonance, the first two non-zero terms of the potential are

$$
\begin{eqnarray}
V(x)&\approx& V(x_0)+\frac{1}{2}(x-x_0)^2\left.\partial_x^2V(x)\right|_{x_0},\\
\nonumber
&=&V(x_0)+\frac{1}{2}k(x-x_0)^2,~~~~k\equiv \left.\partial_x^2V(x)\right|_{x_0},\\
\nonumber
F&=&-\partial_xV(x)=-k(x-x_0).
\end{eqnarray}
$$

Put into Newton's 2nd law (assuming $x_0=0$),

$$
\begin{eqnarray}
m\ddot{x}&=&-kx,\\
x&=&A\cos(\omega_0 t-\phi),~~~\omega_0=\sqrt{k/m}.
\end{eqnarray}
$$

## Harmonic Oscillator, Technicalities

Here $A$ and $\phi$ are arbitrary. Equivalently, one could have
written this as $A\cos(\omega_0 t)+B\sin(\omega_0 t)$, or as the real
part of $Ae^{i\omega_0 t}$. In this last case $A$ could be an
arbitrary complex constant. Thus, there are 2 arbitrary constants
(either $A$ and $B$ or $A$ and $\phi$, or the real and imaginary part
of one complex constant. This is the expectation for a second order
differential equation, and also agrees with the physical expectation
that if you know a particle's initial velocity and position you should
be able to define its future motion, and that those two arbitrary
conditions should translate to two arbitrary constants.

A key feature of harmonic motion is that the system repeats itself
after a time $T=1/f$, where $f$ is the frequency, and $\omega=2\pi f$
is the angular frequency. The period of the motion is independent of
the amplitude. However, this independence is only exact when one can
neglect higher terms of the potential, $x^3, x^4\cdots$. Once can
neglect these terms for sufficiently small amplitudes, and for larger
amplitudes the motion is no longer purely sinusoidal, and even though
the motion repeats itself, the time for repeating the motion is no
longer independent of the amplitude.

One can also calculate the velocity and the kinetic energy as a function of time,

$$
\begin{eqnarray}
\dot{x}&=&-\omega_0A\sin(\omega_0 t-\phi),\\
\nonumber
K&=&\frac{1}{2}m\dot{x}^2=\frac{m\omega_0^2A^2}{2}\sin^2(\omega_0t-\phi),\\
\nonumber
&=&\frac{k}{2}A^2\sin^2(\omega_0t-\phi).
\end{eqnarray}
$$

## Harmonic Oscillator, Total Energy

The total energy is then

$$
\begin{equation}
E=K+V=\frac{1}{2}m\dot{x}^2+\frac{1}{2}kx^2=\frac{1}{2}kA^2.
\label{_auto8} \tag{9}
\end{equation}
$$

The total energy then goes as the square of the amplitude.

A pendulum is an example of a harmonic oscillator. By expanding the
kinetic and potential energies for small angles find the frequency for
a pendulum of length $L$ with all the mass $m$ centered at the end by
writing the eq.s of motion in the form of a harmonic oscillator.

The potential energy and kinetic energies are (for $x$ being the displacement)

$$
\begin{eqnarray*}
V&=&mgL(1-\cos\theta)\approx mgL\frac{x^2}{2L^2},\\
K&=&\frac{1}{2}mL^2\dot{\theta}^2\approx \frac{m}{2}\dot{x}^2.
\end{eqnarray*}
$$

For small $x$ Newton's 2nd law becomes

$$
m\ddot{x}=-\frac{mg}{L}x,
$$

and the spring constant would appear to be $k=mg/L$, which makes the
frequency equal to $\omega_0=\sqrt{g/L}$. Note that the frequency is
independent of the mass.