\title{\bf Distance Ladder}

\section{Basics \& Nomenclature}

Determination of distance is a fundamental problem in astronomy, to
which is owed numerous breakthroughs. The distance to nearby galaxies
has been established through a calibrated {\it distance ladder},
refined over the past century.

The distance ladder begins with the measurement of {\it parallax},
which is the apparent motion of sufficiently nearby stars due to the
motion of the Earth around the Sun. Precise measurements of stars over
time in a fixed reference frame yield {\it proper motions} and
parallax-based distances. Most recently, the Gaia satellite has
provided a large sample of stars with measured parallaxes reaching
across our Galaxy.

To reach distances outside our galaxy, we must rely on less direct
measurements of distance. Usually this involves finding objects or
phenomena that have pairs of observables with a known relation but
with different dependences on distance.

Many of these techniques fall into the {\it standard candle} category:
a luminosity is estimated by some means, and a flux is measured, and
the distance is inferred from the inverse-square law $f=L/4\pi D^2$.
The most prominent standard candles are:
\begin{itemize}
\item {\it RR Lyrae} variable stars. These stars are relatively low mass
  horizontal branch stars. They vary due to the $\kappa$ mechanism,
  which is an instability associated with the reaction of the
  atmospheric opacity to changes in pressure and density that causes
  oscillations in stellar size and luminosity.  They have periods of
  order a day. They are prominent in globular clusters but can also be
  found throughout the Milky Way. Historically they have been most
  useful for determining distances within the Galaxy. However, the
  advent of Gaia and the James Webb Space Telescope opens up the
  possibility of observing these stars in nearby galaxies.
\item {\it Cepheid} variable stars. These stars are high mass and
  luminous.  They also vary due to the $\kappa$ mechanism but have
  longer periods. Historically, they have provided one of the most
  direct links between local distance scales and distant galaxies.
\item {\it Tip of the Red Giant Branch} stars. The red giant branch is
  commences when core hydrogen burning ends; at this point, shell
  hydrogen burning commences and the star becomes rapidly brighter and
  much larger. At a certain point the helium ash at the core ignites,
  moving the star to its next phase, the smaller and less luminous
  horizontal branch. The core ignites rapidly at a specific stage in
  evolution, so that at a given metallicity red giant stars all end
  this stage at the same luminosity. This leads to a rapid fall-off in
  the luminosity distribution of RGB stars and a clear tip of the red
  giant branch that can be used as a standard candle.
\item {\it Type Ia supernovae}. These supernovae result from the
  nuclear detonation of white dwarfs, probably due to accretion from
  or collision with a binary companion. The Phillips relation connects
  the time scale of the supernova light curve to the luminosity of the
  supernova. These sources were used to provide the first definitive
  evidence for the existence of cosmic acceleration.
\end{itemize}
Historically, a number of other distance measurements have also been
used that could be categorized as standard candle distances. Some
methods have relied on other supernova types. Other methods have
generally capitalized on scaling relations of galaxies, such as the
Tully-Fisher relation for spirals relating luminosity and circular
velocity, and the Fundamental Plane for ellipticals relating
luminosity, size, and velocity dispersion. The difficulty in
implementing these methods in a way that is free of systematic errors
generally has led to the use of supernovae for the most recent
measurements of the distance ladder.

Several standard candle techniques prove useful, or may soon do so. I
will discuss eclipsing binaries, masers, and gravitational waves:
\begin{itemize}
\item Some binary stars, like Algol, eclipse each other during their
orbit. Their light curve alone yields a period $P$ and their radii
relative to the orbit size, $r_1/a$ and $r_2/a$ for any given
eccentricity and phase of eclipse. If both their spectra can also be
measured, this can yield velocities, which yields enough information
to determine their total mass $M$, eccentricity $e$, and the
line-of-sight semimajor axis of the orbit $a \sin i$. The mass and
semimajor axis are related by:
\begin{equation}
M = \frac{4\pi^2 a^3}{GP^2}.
\end{equation}
For eclipsing binaries $i\approx 90^\circ$.  Thus, from gravitational
and geometric considerations, you can determine the sizes of the
stars. Spectra can yield constraints on their surface
temperatures. The sizes and temperatures determines the luminosities,
which combined with fluxes determines the distance.
\item Under certain conditions the supermassive
black holes at the centers of galaxies may contain H$_2$O
masers. Under even more special conditions, these systems can be
sufficiently clean that one may measure velocities and accelerations
over time of clouds in circular orbits, which yields a physical
distance. Combined , this combination yields
the mass of the black hole and the distance. Only one clean enough
system is known, about 8 Mpc away in NGC 4258.
\item Colliding compact objects whose gravitational waves are detected can
yield constraints on distance from the gravitational waveform, because
the total mass of the system is related to both the waveform frequency
and the total energy. If the redshift is known from an electromagnetic
counterpart, then the rest-frame waveform frequency can be determined,
and the object can be placed on the redshift-distance relation.
\end{itemize}

A major application of these methods is to compare the redshifts and
distances of local galaxies. This comparison led Hubble to the Hubble
Law:
\begin{equation}
v = H_0 d
\end{equation}
where $v$ is velocity, $H_0$ is the Hubble constant, and $d$ is the
distance. Gravitational attraction causes galaxies to move toward each
other with respect to this flow, with {\it peculiar velocities} of
100s of km s$^{-1}$. For example, M31 is moving toward us at 400 km
s$^{-1}$, and large clusters, such as Virgo and Coma, have internal
motions and thus peculiar velocities of 1000s of km s$^{-1}$. Thus,
determinations of the Hubble constant need to use galaxies
sufficiently far to reduce this effect. Alternatively, they need to
estimate and correct for these peculiar velocities, using the density
field of galaxies itself to estimate the magnitudes and directions of
the expected velocities.

The Hubble Law is a fundamental measurement of the nature of the

universe we are in, but it also is a tool to map the universe on
larger scales, since measuring Doppler shifts is relatively easy
compared to galactic distances.

\section{Commentary}

Briefly, the most precise current estimates of the local Hubble
constant come from the \citet{riess16a} group. These use a number of
Milky Way Cepheids with parallax distances, along with eclipsing
binaries and NGC 4258, to anchor extragalactic Cepheids in galaxies
with SNe Ia, and use those to anchor a much larger SN Ia data set. As
of 2018, the results for $H_0$ from these analyses tend to be
considerably higher (several sigma) than those inferred from cosmic
microwave background and large scale structure measurements, using
standard cosmological models to extrapolate to $z=0$. The cause of
this discrepancy is not known.

The current ladder to a precise $H_0$ is strongly dependent on the
Cepheid step. There are several important issues with the
interpretation of the Cepheid distance scale, the most prominent of
which is that the period-luminosity relation is metallicity
dependent. The LMC has a much lower metallicity than the other
galaxies in the analysis, and so if improperly modelled the
metallicity dependence can cause a bias.

\citet{beaton16a} present an alternative path using different
indicators (RR Lyrae and TRGB) to calibrate the SN Ia, both of which
tend to trace an older, more metal poor population even within metal
rich galaxies. This path may soon become competitive. This second
indepedent measurement will be critical in assessing the existing
discrepancy.

The detection of much more distant masing systems suitable for an
analysis similar to that of NGC 4258, or a sufficient number of
gravitational wave sources, could provide a ``one step'' alternative
to the distance ladder for the purpose of determining the cosmological
constant.

\section{Key References}

\begin{itemize}
  \item
    {\it A 2.4\% Determination of the Local Value of the Hubble
    Constant}, \citet{riess16a}
  \item
    {\it The Carnegie-Chicago Hubble Program. I. An Independent
    Approach to the Extragalactic Distance Scale Using Only Population
    II Distance Indicators}, \citet{beaton16a}
\end{itemize}

\section{Order-of-magnitude Exercises}

\begin{enumerate}
\item Estimate age of the universe from the Hubble constant assuming
    constant expansion.

\begin{answer}[Author: Jiarong Zhu]
Hubble's law states $v = H_0 d$, where $H_0\approx 70$ km s$^{-1}$
Mpc$^{-1}$. If we assume constant expansion (no acceleration or
deceleration), the age of universe will be:
\begin{equation}
T = \frac{d}{v}=\frac{1}{H_0}\approx
\frac{{\rm Mpc}}{70 {\rm ~km} {\rm ~s}^{-1}} =
\frac{10^6*3.086*10^{13} {\rm ~km}}{70 {\rm ~km}} {\rm s}\approx
4.4*10^{17} {\rm ~s}\approx 13.9 {\rm ~Gyr}.
\end{equation}
\end{answer}

\item What is the maximum parallax at 1 pc, 10 pc, 1 kpc, and 1 Mpc?
    Assuming you can get good positions to with a few hundredths of a
    FWHM, what size telescopes do you need so that you can measure
    parallax at these distances?

\begin{answer}[Author: Jiarong Zhu]
The definition of the maximum parallax p is:
\begin{equation}
    p = \frac{1 {\rm ~AU}}{d} = \frac{1''}{d / 1 {\rm ~pc}}.
\end{equation}

Thus if $d$ equals 1 pc, 10 pc, 1 kpc, 1 Mpc, the parallax will be
$1''$, $0.1''$, $10^{-3}\,''$, $10^{-6}\,''$, respectively.

If we can get good positions to $\sim 10^{-2}$ FWHM, suppose the
telescope has size $D$ and measures light at a wavelength $\lambda$,
we find we can measure parallaxes of:
\begin{equation}
    p \sim \frac{\lambda}{D} \times 10^{-2}
\end{equation}
Recalling that $1'' = \pi / 180 / 3600 {\rm ~rad} \approx 4.84*10^{-6}
{\rm ~rad}$, then:
\begin{equation}
    D \sim 10^{-2} \times \frac{\lambda}{p}\approx \frac{\lambda (d /
    1 {\rm ~pc})}{484\times 10^{-6}} \approx
    \left(\frac{\lambda}{484 {\rm ~nm}}\right)
    \left(\frac{d}{1 {\rm ~pc}}\right) 10^{-3} {\rm ~m}
\end{equation}

For instance, if the telescope is optical telescope, $\lambda \sim
500$ nm, the size $D$ of the telescope would need to be $\sim 1$ km in
order to have the resolution to measure parallax.

This diameter requirement only accounts for the diffraction limit and
not other considerations. For example, although the diffraction limit
of a $\sim$ mm-sized telescope is in principle sufficient to measure
parallax at 1 pc, obviously that is impractical because you cannot
gather enough photons in a reasonable time to perform the measurement!
Conversely, although a baseline $D\sim 1$ km is necessary based on the
diffraction limit to measure parallax at 1 Mpc, you may not need a
km-diameter worth of light-gathering power for that measurement.
E.g. using an interferometer fed by two large but possible-to-build
optical telescopes, separated by a km, would be enough (though also
challenging and expensive to do!!).
\end{answer}

\item If the typical peculiar velocity of a galaxy is $\sim 300$ km
    s$^{-1}$, at what redshift does its contribution to the distance
   error drop below 5\%?
\item Based on their luminosities, what are the relative distances to
   which you should be able to determine distances to RR Lyrae,
   TRGB stars, Cepheids, and SN Type Ia?

\begin{answer}[Author: Nanoom Lee]
Let us assume there is a minimum flux required for the object to be
detected and measured at some given precision. For this fixed minimum
flux, the distance to some object is proportional to $L^{1/2}$, where
$L$ is the luminosity of the object. Roughly, the typical luminosity
of RR Lyrae stars is $10^2 L_\text{sun}$, of TRGB stars is $10^4
L_\text{sun}$, of Cepheids is $10^2$--$10^4 L_\text{sun}$, and of SN
Type Ia is $10^{10}L_\text{sun}$. Therefore the distances (relative to
RR Lyrae) to which we should be able to determine distance with these
standard candles is 1, 10, 1--10, and $10^4$ for RR Lyrae, TRGB stars,
Cepheids, and SN Type Ia, respectively. This explains why RR Lyrae are
the most difficult to use for extragalactic distances, why SN Type Ia
are necessary in order to reach distances which can measure the Hubble
flow well, and why the Cepheid and TRGB methods are in between.
\end{answer}

\end{enumerate}

\section{Numerics and Data Exercises}

\begin{enumerate}
\item Allen Sandage used to use brightest galaxies in clusters as a
   standard
   candle. Download \href{http://risa.stanford.edu/redmapper/}{redmapper},
   a modern catalog of galaxy clusters from SDSS. Select the brightest
   cluster galaxies for massive clusters. Based on looking at these
   galaxies within a narrow range of redshift, are they reasonable
   standard candles? Show a Hubble diagram assuming that they are and
   compare to a standard Planck cosmology.
\item \citet{clementini18a} publish a catalog of RR Lyrae variables
   from Gaia DR2. \citet{muraveva18a} publish a calibration of the RR
   Lyrae period-luminosity-metallicity (PLZ) relation. Use the RR
   Lyrae stars in the catalog and the published PLZ relation to
   estimate the ratio of the distance to the Sculptor dwarf galaxy to
   the distance to the Messier 62 globular cluuster.
\end{enumerate}

\bibliographystyle{apj}
\bibliography{exex}


\documentclass[12pt]{article}
\usepackage[T1]{fontenc}

% Document Details
\newcommand{\CLASS}{AMATH 586}
\newcommand{\assigmentnum}{Numerical SDE Solvers}

\usepackage[margin = 1.1in, top = 1.25in, bottom = 1.in]{geometry}
\input{../../TeX_headers/title.tex} % Title Styling
\input{../../TeX_headers/styling.tex} % General Styling
\input{../../TeX_headers/code.tex} % Code Display Setup
\input{../../TeX_headers/math.tex} % Math shortcuts

%\usepackage{algorithm}
%\usepackage[noend]{algpseudocode}

\usepackage{amsthm}
\newtheorem*{lemma}{Lemma}
\newtheorem*{theorem}{Theorem}
\newtheorem*{definition}{Definition}
\newtheorem*{prop}{Proposition}
\renewenvironment{proof}{{\bfseries Proof.}}{\qed}

\hypersetup{
   colorlinks=true,       % false: boxed links; true: colored links
   linkcolor=violet,          % color of internal links (change box color with linkbordercolor)
   citecolor=blue,        % color of links to bibliography
   filecolor=magenta,      % color of file links
   urlcolor=cyan           % color of external links
}

\begin{document}
\maketitle

\section*{Abstract}
This report is intended as a starting reference for numerical methods to SDEs and contains content similar to \cite{holmes,sdes}. The standard text on this topic is \cite{kloeden}, which provides an in-depth approach to stochastic calculus, and numerical methods for SDEs. As a starting point to stochastic calculus in general we refer readers to \cite{lorig}.

\section{Introduction}

A Stochastic Differential Equation (SDE) is an equation of the form,
\begin{align}
    \d X_t =  \mu(X_t,t)\d t + \sigma(X_t,t)\d W_t \label{diffform}
\end{align}
where \( W_t \) denotes a standard Brownian motion \cite{lorig}. We can write this in integral form as,
\begin{align}
    X_{t} - X_{0} = \int_{0}^{t} \mu(s,X_s)\d s + \int_{0}^{t} \sigma(s,X_s) \d W_s \label{intform}
\end{align}

In general we would like to write our solution \( X_T \) as an explicit function of a Brownian motion trajectory. For some processes this is possible, however in general it is not. As a result, numerical methods are required for many SDEs.

SDEs are commonly used to model processes such as molecular dynamics, neurodynamics, financial quantities (stocks, asset prices, interest rates, etc.) SDEs can also be used to efficiently solve high dimensional PDEs numerically, (standard mesh based methods scale exponentially with the dimension of the system).

In this report we first introduce Brownian motion and stochastic calculus. We then discuss how Brownian motion can be approximated numerically and discuss what it means for a numerical SDE method to converge. Next we present simple numerical method and two higher order methods which are obtained using stochastic Taylor expansions. Finally, we conduct some experiments to corroborate the theoretical claims we present earlier in the paper.

\section{Brownian motion}
There are many equivalent characterizations of Brownian motion. Perhaps the most standard is the following definition.
\begin{definition}
A Brownian motion is a stochastic process \( W = (W_t)_{t\geq0} \) defined on some probability space \( (\Omega,\cF,\bP) \) satisfying,
\begin{enumerate}[nolistsep]
    \item \( W_0 = 0 \)
    \item \( (W_d-W_c) \indep (W_b-W_a) \) for all \( 0\leq a\leq b\leq c\leq d \))
    \item \( (W_t - W_s) \sim \cN(0,t-s) \) for all \( 0\leq s\leq t \)
    \item the map \( t \to W_t(\omega) \) is continuous for almost all \( \omega\in\Omega \)
\end{enumerate}
\end{definition}

We attempt to parse this definition and notation. The notation \( W_t \) denotes a random variable which depends on the time \( t \). That is, \( W_t \) is a function from \( \Omega \to \bR \). For a fixed \( \omega \in \Omega \), \( W_t(\omega) \) can be seen as a function of \( t \), and is referred to as a path or a trajectory of the Brownian motion.

Condition 1 means that all paths of the Brownian motion starts start at the origin. Condition 2 means that the random variables \( W_d-W_c \) and \( W_b - W_a \) are independent of one another. That is, knowing how much the a path of the Brownian motion changes over the time interval \( [a,b] \) does not give any information about how much the path changes over the interval \( [c,d] \). Condition 3 means that over any time interval \( [s,t] \) how much the Brownian motion changes is a normal random variable with mean zero and variance \( t-s \). Finally, Condition 4 means that almost all paths of the Brownian motion are continuous.

Brownian motion can be constructed as a scaled random walk. Such a construction can be readily obtained on the internet or in \cite{lorig}. We will simply note that it may be useful to view Brownian motion as the continuous analog to a discrete random walk.

\section{Stochastic Calculus}
In this section we aim to provide enough background to allow all statements in the main portion of the paper to be understood. That said, this done in the admittedly unsatisfying way of black box lemma's and heuristics which we do not explain or prove. For a more complete overview of Stochastic Calculus we refer readers to \cite{kloeden, lorig}.

We first introduce Riemann--Stieltjes integrals.

\begin{definition}
For real valued functions \( f \) and \( g \) the Riemann--Stieltjes integral is defined as,
\begin{align*}
    \int_{a}^{b} f(x) \d g(x) := \lim\limits_{\norm{\Pi}\to 0} \sum_{i=0}^{n-1} f(c_i)(g(x_{i+1}) - g(x_i))
\end{align*}
where \( \Pi = \{a=x_0<x_1 < \cdots < x_n = b\} \) is a partition of \( [a,b] \), \( \norm{\Pi} \) is the length of the largest subinterval, and \( c_i \) is any point in \( [x_i,x_{i+1}] \).
\end{definition}

We note that if \( g(x) = x \) the Riemann--Stieltjes integral is the standard Riemann integral and that if \( g \) is continuously differentiable,
\begin{align*}
    f(g(t)) - f(g(0)) = \int_0^t \d f(g(s)) = \int_0^t f'(g(s))g'(s) \d s
\end{align*}

Brownian motion and many processes involving Brownian motion are not differentiable. It\^o's Lemma gives us a way to compute the analogous result for a class of stochastic processes called It\^o (drift-diffussion) processes. For our purposes we can think of It\^o processes as processes with an integral with respect to \( t \) and an integral with respect to \( W_t \).

\begin{lemma}[It\^o] For \( f:\bR\to\bR \) sufficiently differentiable and It\^o process \( X_t \),
\begin{align}
    \d f(X_t) &=  f'(X_t)\d X_t + \dfrac{1}{2}f''(X_t)\d[X,X]_t %\label{ito}
\end{align}
\end{lemma}
Here we have used the notation \( [X,X]_t \) to denote the quadratic variation of the process \( X_t \). Heuristically we can compute \( \d[X,X]_s \) by expanding  \( (\d X_s)(\d X_s) \) and simplifying using the identities,
\begin{align}
    \d t \d t = 0, && \d t \d W_t = 0, && \d W_t \d W_t = \d t \label{heuristics}
\end{align}

It\^o's Lemma can be generalized to functions and processes of higher dimension.
\begin{lemma}[It\^o] For \( f:\bR^n\to\bR^n \) sufficiently differentiable and It\^o process \( X_t = [X_t^1, X_t^2, \ldots X_t^n]^T \),
\begin{align}
    \d f(X_t) = \sum_{i=1}^{n} \left[\pp{}{x_i}f(X_t)\right] \d X_t^i + \dfrac{1}{2} \sum_{i=1}^{n} \sum_{j=1}^{n}\left[ \pp{}{x_i} \pp{}{x_j} f(X_t) \right] \d [X^i,X^j]_t \label{itond}
\end{align}
\end{lemma}

Similar to before, we compute \( \d[X^i,X^j]_t \) by expanding \( (\d X^i)(\d X^j) \) and using the heuristics in (\ref{heuristics}).

Consider the special case when \( n=2 \), \( X_t^1 = t\), and \( X_t^2 = X_t \). Using our heuristics in (\ref{heuristics}) we have \( \d[X^1,X^2]_t = (\d t)(\d X_t) = 0 \) and \( \d[X^1,X^1] = (\d t)(\d t) = 0 \). Therefore, by (\ref{itond}),
\begin{align}
    \d f(t,X_t) &= \pp{}{t} f(t,X_t) + \pp{}{x} f(t,X_t) \d X_t + \dfrac{1}{2} \pp[2]{}{x}f(t,X_t) \d [X,X]_t
    \\ &= \left( \pp{}{t} + \dfrac{1}{2}\pp[2]{}{x} \right)f(t,X_t)\d t + \pp{}{x} f(t,X_t) \d X_t \label{2dito}
\end{align}

\section{Example Processes}
In this section we provide some results about a few important stochastic processes which we will use in our numerical tests. Both the SDEs we introduce can be solved analytically and the mean and variance can be computed

For constants \( \mu \) and \( \sigma \) a Geometric Brownian motion satisfies,
\begin{align}
    \d X_t = \mu X_t \d t + \sigma X_t \d W_t, && t\in[0,T] \label{GBMeqn}
\end{align}

We will solve this equation to build some familiarity with stochastic calculus.
To this end, note that \( \d X_t = \mu X_t \d t + \sigma X_t \d W_t \) so using the heuristics (\ref{heuristics}) we have \( \d[X,X]_t = \sigma^2 \d t \). We note apply It\^o's Lemma to \( \ln(x) \) to obtain,
\begin{align*}
    \d \ln (X_t) &= \dfrac{1}{X_t} \d X_t - \dfrac{1}{2} \dfrac{1}{X_t^2} \d[X,X]_t
    = \left( \mu - \dfrac{\sigma^2}{2} \right) \d t + \sigma \d W_t
\end{align*}

Integrating both sides from \( T \) to \( t \) we obtain,
\begin{align*}
    \ln (X_t) = \int_0^t \d \ln(X_s) = \int_0^t \left( \mu - \dfrac{\sigma^2}{2} \right)\d s + \int_0^t \sigma \d W_s = \left( \mu - \dfrac{\sigma^2}{2} \right) t + \sigma W_t
\end{align*}

Therefore, the solution to (\ref{GBMeqn}) is,
\begin{align*}
    X_t = X_0 \exp\left( \left( \mu - \dfrac{\sigma^2}{2} \right)t + \sigma W_t \right)
\end{align*}

For constants \( \theta, \mu, \sigma \), an Ornstein--Uhlenbeck (OU) process satisfies,
\begin{align}
    \d X_t = \theta(\mu-X_t) \d t + \sigma \d W_t, && \theta > 0 \label{OUeqn}
\end{align}

We note that if \( X_t \) is away from \( \mu \) it will tend towards this value in expectation. Figure~\ref{OU_em} shows Euler--Maruyama method applied to (\ref{OUeqn}). As expected, the trajectories all end up ``centered'' about \( \mu \). More precisely, (\ref{OUeqn}) has solution,
\begin{align*}
    X_t = X_0 \exp(-\theta t) + \mu(1-\exp(-\theta t)) + \sigma \int_0^t \exp(-\theta(t-s))\d W_s
\end{align*}

The mean of \( X_t \) is,
\begin{align*}
    \bE[X_t] = (X_0 - \mu)\exp \left( -\theta t \right)+ \mu
\end{align*}
Likewise, the variance of \( X_t \) is,
\begin{align*}
    \bE\left[(X_t-\bE[X_t])^2\right] = \dfrac{\sigma^2}{2\theta}(1-\exp(-2\theta t))
\end{align*}

\section{Generating Brownian motion Numerically} \label{brownian_motion}

Given the four properties of Brownian motion, how can we numerically generate trajectories which are representative of a Brownian motion? The most standard way to do this is to first pick a time mesh on which we will have the values of the Brownian motion. Let \( t = [t_0, t_1, \ldots, T] \) where \( 0 = t_0 < t_1 < \cdots < t_N = T \) be such a mesh. We will generate a numerical realization \( \hat{W} = [0, \hat{W}_1, \ldots, \hat{W}_T] \) of a Brownian motion trajectory. To do this we generate the incriments. That is, we start with \( \hat{W}_0 = 0 \) and recursively define,
\begin{align*}
    \hat{W}_{t_{k+1}} = \hat{W}_{t_k} + \Delta \hat{W}_{t_k}
\end{align*}
where, using property (3) of Brownian motion, we must sample \( \Delta \hat{W}_{t_k} \) from a normal distribution with mean 0 and variance \( t_{k+1} -t_{k} \).

Since we only sample at discrete points, we have not generated a ``true'' trajectory of the Brownian motion. In fact, refining the mesh and generating the interior points is not trivial. However, it is clear that the property 2 is satisfied since we sample or increments independently. Similarly, property 3 is satisfied since the sum of iid normal random variables is a iid random variable with mean equal to the sum of the means, and variance equal to the sum of the variances. Therefore this method gives us trajectories which satisfy the first three properties of Brownian motion.

We now make a few practical observations about implementing this procedure on a computer. First, while we have explicitly defined the vector \( \hat{W} \), it may be more efficient to save the vector \( \Delta \hat{W} \) since the numerical methods we present involve \( \Delta W_t \). Second, if it is slow to sample normal random variables, other simpler distributions coud be used. For instance, we could pick \( \Delta W_t \in\{-\sqrt{\d t}, \sqrt{\d t}\} \) with equal probabilities for each choice. In this case the mean and variance will still match Brownian motion, but the higher moments may not (i.e. the increments will no longer be normally distributed). Whether or not this is important is dependent on the application.

\section{Convergence of Numerical Methods}
A SDE solution may have many distinct trajectories as the solution itself is a random process. Our standard definitions of convergence for IVPs are no longer sufficient.

For some applications we would like to be able to compute trajectories numerically which are representative of the actual trajectories of the solution to the SDE. This gives rise to the following definition convergence.
\begin{definition}
A discrete time approximation \( \hat{X}^{(k)} \) is said to converge strongly to the solution \( X_t \) at time \( T \) if,
\begin{align*}
    \lim_{k\to 0}\bE\left[\left| X_T - \hat{X}_T^{(k)} \right|\right] = 0
\end{align*}
and is said to converge strongly with order \( m \) if for sufficiently small \( k \),
\begin{align*}
    \bE\left[\left| X_T - \hat{X}_T^{(k)} \right|\right] = \cO(k^m)
\end{align*}
\end{definition}

Since strong convergence looks at the absolute difference of \( \hat{X}_T^{(k)} \) and \( X_T \) it tells us about how individual trajectories vary. That is, to compute this expectation we first find the absolute difference of our solution and the exact solution for every trajectory of Brownian motion, and then take the mean.

For other applications it may be sufficient to generate trajectories whose moments (mean, variance, etc.) are the same as the trajectories of the solution to the SDE, even if individual trajectories don't necessarily converge to trajectories of the solution. This gives rise to the notion of weak convergence.
\begin{definition}
A discrete time approximation \( \hat{X}^{(k)} \) is said to converge weakly to the solution \( X_t \) at time \( T \) if for every polynomial \( p \),
\begin{align*}
    \lim_{k\to 0} \left| \bE\left[ p(X_T) - p\left(\hat{X}_T^{(k)}\right) \right] \right| = 0
\end{align*}
and is said to converge weakly with order \( m \) if for sufficiently small \( k \) and every polynomial \( p \),
\begin{align*}
    \left|\bE\left[ p(X_T) - p\left(\hat{X}_T^{(k)}\right) \right] \right| = \cO(k^m)
\end{align*}
\end{definition}

Since weak convergence looks at the signed difference of \( \hat{X}_T^{(k)} \) and \( X_T \) the errors can ``cancel'' with one another.

Note that the term corresponding to \( \d t \) is deterministic. This means that if \( \sigma \equiv 0 \), Equation (\ref{intform}) becomes the initial value problem \( \d X_t/\d t = \mu(X_t,t) \) with initial condition at time \( t \) given by the value of \( X_t \). In this case both definitions of convergence agree with the standard definition of convergence for IVPs provided in \cite{leveque}.

We now prove the following (unsurprising) theorem following closely to the proof provided in \cite{holmes}.
\begin{theorem}
The weak order of a SDE method is always less than or equal to the strong order of the method.
\end{theorem}
\begin{proof}
Suppose a method \( \hat{X}^{(k)} \) is strongly convergent. Then the set \( S:=\{\hat{X}_T^{(k)}\}_k\cup\{X_T\} \) is bounded.
Therefore, there exists \( K > 0 \) such that \( K > |p'(x)| \) for all \( x \) in some finite interval containing \( S \).

Since \( |\bE Z| \leq \bE|Z| \) for any random variable \( Z \), and by the Mean Value Theorem,
\begin{align*}
    \left|\bE\left[ p(X_T) - p\left(\hat{X}_T^{(k)}\right) \right] \right|
    \leq \bE\left[\left| p(X_T)  -  p\left(\hat{X}_T^{(k)}\right) \right| \right]
    \leq K \bE\left[\left| X_T - \hat{X}_T^{(k)} \right| \right]
\end{align*}
This proves the result.
\end{proof}

\section{A simple Numerical Method}
All the numerical methods in this report find an approximate solution to a given SDE on some time grid. More specifically, the methods take in a series of time points and the value of a Brownian motion trajectory at these time points and output the approximate solution.

We first introduce the natural generalization of the Forward Euler method. The so called Euler--Maruyama method \cite{sdes} is given by the relation,
\begin{align*}
        \hat{X}_{t+\Delta t} &= \hat{X}_t + \mu(t,\hat{X}_{t}) \Delta t + \sigma(t,\hat{X}_{t}) \Delta W_{t}
\end{align*}
where \( \Delta W_t:=W_{t+\Delta t}-W_t \).

On SDEs where \( \sigma \) and \( \mu \) satisfy the appropriate conditions (these condtions are generally satisifed), the Euler-Maruyama method has strong and weak orders of 1/2 and 1 respectively. A commonly cited proof of this result can be found in \cite{kloeden}. However we note that on some SDEs strong convergence of order 1 is attained.

We test the Euler--Maruyama method on the Ornstein--Uhlenbeck equation and Geometric Brownian motion equation. Figure~\ref{EM_tests} shows the result of two tests. More specifically, Figure~\ref{OU_em} shows 10 trajectories of the OU process computed using the Euler-Maruyama method. As expected these trajectories all seem to be heading towards a mean of 1 as time progresses. Figure~\ref{GBM_em} shows the result of the Euler-Maruyama a large step size in comparison to the exact solution.

\begin{figure}[ht!]\centering
\begin{subfigure}{.48\textwidth}\centering
    \includegraphics[width=\textwidth]{img/OU_em_10.pdf}
    \caption{10 trajectories of the OU equation with \( \theta = 10 \), \( \mu = 1 \), \( \sigma = 1/2 \) computed using the Euler--Maruyama method with a step size of \( \Delta t = 0.004 \).}
    \label{OU_em}
\end{subfigure}\hfill
\begin{subfigure}{.48\textwidth}\centering
    \includegraphics[width=\textwidth]{img/GBM_true_vs_10.pdf}
    \caption{Exact trajectory of Geometric Brownian motion with \( \mu = 1 \), \( \sigma = 1/2 \), and solution computed using the Euler--Maruyama method with a step size of \( \Delta t = 1/10 \).}
    \label{GBM_em}
\end{subfigure}
\caption{Basic numerical tests of Euler--Maruyama method}
\label{EM_tests}
\end{figure}

\section{Higher Order Methods}
The Euler--Maruyama method is commonly used due to its simplicity and ease of implementation. However, the strong convergence of this method is quite slow. As a result, higher order methods have been developed and studied. We introduce two methods in the hopes of illustrating how such methods can be derived. Both methods have strong and weak order one. A similar derivation can be found in \cite{holmes}.

By It\^o's Lemma (\ref{2dito}), if \( X_T \) is the solution to (\ref{intform}), for any sufficiently differentiable function \( f \) we have,
\begin{align}
    f(T,X_T) &= f(t,X_t) + \int_0^t \cM f(s,X_s)\d s + \int_0^t \cS f(s,X_s) \d W_s \label{ito1}
\end{align}
where,
\begin{align*}
    \cM := \pp{}{s} +  \mu(s,X_s)\pp{}{x} + \frac{1}{2}\sigma^2(s,X_s) \pp[2]{}{x}, && \cS := \sigma(s,X_s) \pp{}{x}
\end{align*}

Applying the relation in (\ref{ito1}) with \( f = \mu \) and \( f=\sigma \) gives,
\begin{align*}
    \mu(s,X_s) &= \mu(t,X_t) + \int_0^s \cM \mu(u,X_u) \d u + \int_0^s \cS \mu(u,X_u)\d W_u \\
    \sigma(s,X_s) &= \sigma(t,X_t) + \int_0^s \cM \sigma(u,X_u) \d u + \int_0^s \cS \sigma(u,X_u)\d W_u
\end{align*}

Therefore, inserting these expressions into (\ref{intform}) gives the relations,
\begin{align}
    X_t %&= X_t + \int_t^T \mu(t,X_t) \d s + \int_t^T \sigma(t,X_t) \d W_s + R \\
    &= X_0 + \mu(t,X_t) \int_0^t \d s +  \sigma(t,X_t)\int_0^t \d W_s + R \label{emexpr}
\end{align}
where
\begin{align}
    R &=  \int_0^t \int_0^s \cM \mu(u,X_u) \d u \d s + \int_0^t \int_0^s \cS \mu(u,X_u)\d W_u  \d s \tag*{}
    \\ &\hspace{4em} + \int_0^t \int_0^s \cM \sigma(u,X_u)\d u \d W_s + \int_0^t \int_0^s \cS \sigma(u,X_u) \d W_u \d W_s
\end{align}

Note the Euler--Maruyama method arises from (\ref{emexpr}) after dropping all the terms in \( R \) and setting \( T = t+\Delta t \).

Using the hueristics (\ref{heuristics}) the \( \d W_s \d W_t \) term is the highest order term of \( R \), so a higher order method must keep more of this term. We therefore apply the relation in (\ref{ito1}) with \( f = \cS \sigma \) to obtain,
\begin{align*}
    \cS \sigma(u,X_u) = \cS \sigma(t,X_t) + \int_{t}^{u} \cM \cM \sigma(v,X_v) \d v + \int_t^u \cS \cM \sigma(v,X_v) \d W_v
\end{align*}

Therefore,
\begin{align}
    X_t = X_0 + \int_0^t \mu(t,X_t) \d s + \int_0^t \sigma(t,X_t) \d W_s + \int_0^t \int_0^s \cS \sigma(t,X_t)  \d W_u \d W_v + R \label{milexpr}
\end{align}
where \( R \) contains higher order terms (and is different from the \( R \) in (\ref{emexpr})).

Applying It\^o's Lemma to \( f(x) = x^2 \) we obtained the well known result,
\begin{align*}
    \int_0^t \int_t^s \d W_u \d W_s
    = \int_0^t W_{s} \d W_s
    = \dfrac{1}{2} \left( (W_{t})^2 - t \right)
\end{align*}

Dropping all the terms in the new \( R \) and setting \( T = t+\Delta t \), (\ref{milexpr}) gives rise to the  Milstein method,
\begin{align*}
    \hat{X}_{t+\Delta t} &= \hat{X}_t + \mu(t,\hat{X}_{t}) \Delta t + \sigma(t,\hat{X}_{t}) \Delta W_{t} + \sigma(t,\hat{X}_t) \partial_x \sigma(t,\hat{X}_t) \left( (\Delta W_t)^2 - \Delta t \right)/2
\end{align*}

Both the Milstein and Forward Euler methods can be seen as Taylor series methods. Using similar approximations to what we have done here arbitrary order methods can be obtained. However, deriving and implementing such methods can easily become unwieldy. Therefore, as with deterministic IVP methods, the order of accuracy of a method must be balanced with the complexity of the method when deciding which method to apply.

As with deterministic IVP Taylor methods, knowing the derivative to a function is necessary. It is sometimes preferable to have methods which do not rely on knowing the derivative of any functions ahead of time.

We can approximate the \( x \)-derivative of \( \sigma \) by,
\begin{align}
    \pp{}{x}\sigma(\hat{X}_t,t) = \dfrac{\sigma(t,\hat{X}_t + \sigma(\hat{X})\sqrt{\Delta t}) - \sigma(t,\hat{X}_t)}{\sigma(\hat{X},t)\sqrt{\Delta t}} + \cO\left(\sigma(t,\hat{X}_t)\sqrt{\Delta t}\right) \label{derivapprox}
\end{align}

Using our heuristics (\ref{heuristics}) we expect substituting the right hand side to give a first order method which is the case (although not proved here).

Using the approximation in (\ref{derivapprox}) yields the strong first order Runge-Kutta method,
\begin{align*}
    \hat{X}_{t+\Delta t} &= \hat{X}_t + \mu(t,\hat{X}_{t}) \Delta t + \sigma(t,\hat{X}_{t}) \Delta W_{t} + \sigma(t,\hat{X}_t-X^*) \left( (\Delta W_t)^2 - \Delta t \right) / (2 \sqrt{\Delta t}) \\
    X^* &=  \hat{X}_t  + \sigma(t,\hat{X}_t)\sqrt{\Delta t}
\end{align*}

\section{Numerical Convergence Experiments}
In this section we conduct a few numerical experiments to see if they agree with the theoretical results presented above.

We implement the Euler-Maruyama and Runge-Kutta methods described above in Python using Numpy. These functions take in the Brownian motion in increment form as suggested previously. Note the notation change where we have denoted \( \mu \) and \( \sigma \) by \( a \) and \( b \).
\lstinputlisting[linerange=\#<start:em>-\#<end:em>]{solvers.py}
\lstinputlisting[linerange=\#<start:rk>-\#<end:rk>]{solvers.py}

Figure~\ref{weak_order_test} shows the absolute difference between the mean and variance of the actual solution and the mean and variance of various numerical methods. All three methods tested match the expected orders of weak convergence for the mean and variance of the process.

\begin{figure}[b!]\centering
\begin{subfigure}{.48\textwidth}\centering
    \includegraphics[width=\textwidth]{img/weak_order_2000.pdf}
    \caption{\( \dfrac{\big|\bE[X_T] - \bE[\hat{X}_T]\big|}{\big|\bE[X_T]\big|} \)}
    \label{mean}
\end{subfigure}
\begin{subfigure}{.48\textwidth}\centering
    \includegraphics[width=\textwidth]{img/weak_order_2_mom_2000.pdf}
    \caption{\( \dfrac{\big|\bE[(X_T-\bE[X_T])^2] - \bE[(\hat{X}_T-\bE[X_T])^2]\big|}{\big|\bE[(X_T-\bE[X_T])^2]\big|} \) }
    \label{var}
\end{subfigure}
    \caption{Statistics based on 2000 trajectories, where \( T = 1 \) and \( X_t \) is a OU process with \( \theta = 1 \), \( \mu = 1 \), and \( \sigma = 1 \) for various \( \hat{X}_t \).  Circles correspond to the Euler--Maruyama method at various mesh sizes, triangles correspond to the Runge-Kutta method at various mesh sizes, and squares correspond to Euler--Maruyama method with \( \Delta W_t \in \{-\Delta t, \Delta t\} \). }
\label{weak_order_test}
\end{figure}

Note that the best fit lines in Figure~\ref{mean} have slopes of \protect\input{img/weak_order_em_2000.txt}\unskip, \protect\input{img/weak_order_rk_2000.txt}\unskip, and \protect\input{img/weak_order_em2_2000.txt}respectively. This is the expected weak convergence of order one.

Interestingly, the best fit lines in Figure~\ref{var} have slopes of \protect\input{img/weak_order_em_2_mom_2000.txt}\unskip, \protect\input{img/weak_order_rk_2_mom_2000.txt}\unskip, and \protect\input{img/weak_order_em2_2_mom_2000.txt}respectively. This seems to happen when adjusting the parameters of the processes as well as the number of trajectories used, etc. While it still satisfies the definition of weak convergence of order 1 it was unexpected.

We can now use our numerical solutions to generate trajectories of a Geometric Brownian motion. In particular, if for each trajectory we output the values of \( \Delta W_t \) at each time point, the Brownian motion \( W_t \) can be computer. This means we can compare our computed results at time \( T \) to the exact solution at time \( T \).

Figure~\ref{strong_order_test} shows the results of this for the Euler-Maruayma and Runge-Kutat methods. Note that the best fit lines have slopes of \protect\input{img/strong_order_em_1000.txt}and \protect\input{img/strong_order_rk_1000.txt}respectively. These are the order of strong convergence expected for both methods.

\begin{figure}[t!]\centering
    \includegraphics[width=.5\textwidth]{img/strong_order_1000.pdf}
    \caption{Values of \( \bE|X_T - \hat{X}_T|/\bE|X_T| \) approximated using 1000 trajectories, where \( T = 8 \) and \( X_t \) is a Geometric Brownian motion with \( \mu = 1 \) and \( \sigma = 1/2 \) for various \( \hat{X}_t \).  Circles correspond to the Euler--Maruyama method at various mesh sizes, and triangles correspond to the Runge-Kutta method at various mesh sizes.}
\label{strong_order_test}
\end{figure}

\section{Conclusion}
We introduce stochastic differential equations. In order to solve these numerically we introduced methods of generating Brownian motion on a discrete time mesh. We then introduced numerical SDE methods by approximating the solution using a (stochastic) Taylor expansion. Finally some numerical results were tested. For the most part the figures matched our expectations. However, the plot of Figure~\ref{var} shows that \( \big|\bE[(X_T-\bE[X_T])^2] - \bE[(\hat{X}_T-\bE[X_T])^2]\big|/\big|\bE[(X_T-\bE[X_T])^2]\big| \) goes to zero like \( k^2 \) rather than \( k \). At the moment it is not clear what affect the polynomial we chose, the processes, and the method have on this result. A more detailed look at this would be warranted.

\bibliography{project}{}
\bibliographystyle{ieeetr}

\pagebreak
\section{Code for Numerical Tests}
\lstinputlisting[basicstyle=\tiny\tt]{solvers.py}

\end{document}


\documentclass[12pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amsfonts,amsthm}
\usepackage{parskip}
\addtolength{\jot}{1em}
\title{Binomial Options Pricing Model}
\date{October 2018}

\begin{document}
\begin{titlepage}
\maketitle
\end{titlepage}

\section{One-Step Binomial Model}
\paragraph{No-Arbitrage Assumption} \hspace{0pt}

Consider a stock currently priced at $S_{0}=\text{\$100}$, and at the end of three months ($t_{T}$), the stock price will be either $S_{T}=\text{\$110}$ or $S_{T}=\text{\$90}$. Assume the scenario where we are valuing a European call option to purchase the stock for \$105 at $t_{T}$. There are two possible outcomes at $t_{T}$: if $S_{T}=\text{\$110}$, the value of the call option will be \$5; if $S_{T}=\text{\$90}$, the value of the call option will be 0.

Assume that arbitrage opportunities do not exist. The scenario is also set up in such a way that there is no uncertainty about the value of the stock at the end of three months. In the absence of arbitrage opportunities and uncertainties, the return it earns must then equal the risk-free rate of interest ($R_{f}$). It is always possible to set up a riskless portfolio because there are two securities (stock and call option) and only two possible outcomes.

Consider a portfolio consisting of a long position in $\Delta$ shares of the stock and a short position in one call option. Then, calculate the value of $\Delta$ that makes the portfolio riskless. If the $S$ moves up from $S_{0}=100$ to $S_{T}=110$, the value of the shares is $110\Delta$ and the value of the option is 1; the portfolio value is $110\Delta-1$. If the $S$ moves down from $S_{0}=100$ to $S_{T}=90$, the value of the shares is $90\Delta$ and the value of the option is 0; the portfolio value is $90\Delta$. To set up a riskless portfolio, the portfolio value at $t_{T}$ must be the same for either outcome.
\begin{align*}
110\Delta-1&=90\Delta\\
\Delta&=0.05
\end{align*}
The portfolio is riskless if it consists of a long position in 0.05 shares and a short position in one call option. In either outcome, the value of the portfolio is always 4.5 at $t_{T}$. $\Delta$ is the number of shares necessary to hedge a short position in one call option.

In the absence of arbitrage opportunities, a riskles portfolio must earn the risk-free rate of interest, $R_{f}$. Assuming that $R_{f}=0.05$ per annum, the value of the portfolio today must be:
\begin{align*}
4.5e^{-0.05\cdot3/12}&=4.4441
\end{align*}
The stock price today at $t_{0}$ is known to be \$100. Let $f=\text{option price today}$. The value of the portfolio today must be:
\begin{align*}
100\cdot0.05-f&=4.4441\\
5-f&=4.4441\\
f&=0.5559
\end{align*}
In the absence of arbitrage opportunities, the value of the option at $t_{0}$ must be $f=0.5559$. If $f>0.5559$, the portfolio would cost less than 4.4441 to set up and would earn more than $R_{f}$. Conversely, if $f<0.5559$, shorting the stock and buying the option would allow for borrowing at less than $R_{f}$.

\paragraph{Generalization} \hspace{0pt}

Let the current stock price $=S_{0}$ and the stock option price $=f$. Suppose that during the life of the option (from $t_{0}$ to $t_{T}$), the stock price can either move up from $S_{0}$ to $S_{0}u$ or move down from $S_{0}$ to $S_{0}d$; where $(u>1; d<1)$. The proportional increase in stock price in an up movement is $u-1$; the proportional decrease in stock price in a down movement is $1-d$. Suppose that for $S_{0}u$, the option payoff is $f_{u}$ and for $S_{0}d$, the option payoff is $f_{d}$.

Consider a portfolio consisting of a long position in $\Delta$ shares and a short position in one stock option. In an up movement in the stock price, the portfolio value at $t_{T}$ is:
\begin{align*}
S_{0}u\Delta-f_{u}
\end{align*}
In a down movement in the stock price, the portfolio value at $t_{T}$ is:
\begin{align*}
S_{0}d\Delta-f_{d}
\end{align*}
For a riskless portfolio, the two must equal:
\begin{align*}
S_{0}u\Delta-f_{u}&=S_{0}d\Delta-f_{d}\\
\Delta&=\frac{f_{u}-f_{d}}{S_{0}u-S_{0}d}
\end{align*}
$\Delta$ is the ratio of the change in option price to the change in stock price as $t_{0}$ approaches $t_{T}$. Let the risk-free rate of interest $=r$. The present value of the portfolio is:
\begin{align*}
(S_{0}u\Delta-f_{u})e^{-rT}
\end{align*}
The portfolio set up cost (at $t_{0}$) is:
\begin{align*}
S_{0}\Delta-f
\end{align*}
Then,
\begin{align*}
S_{0}\Delta-f&=(S_{0}u\Delta-f_{u})e^{-rT}\\
f&=S_{0}\Delta(1-ue^{-rT})+f_{u}e^{-rT}
\end{align*}
Recall that
\begin{align*}
\Delta&=\frac{f_{u}-f_{d}}{S_{0}u-S_{0}d}
\end{align*}
Substituting $\Delta$,
\begin{align*}
f&=S_{0}\left(\frac{f_{u}-f_{d}}{S_{0}u-S_{0}d}\right)(1-ue^{-rT})+f_{u}e^{-rT}\\
&=S_{0}\left(\frac{f_{u}-f_{u}ue^{-rT}-f_{d}+f_{d}ue^{-rT}}{S_{0}u-S_{0}d}\right)+f_{u}e^{-rT}\\
&=\frac{S_{0}f_{u}-S_{0}f_{u}ue^{-rT}-S_{0}f_{d}+S_{0}f_{d}ue^{-rT}+S_{0}f_{u}ue^{-rT}-S_{0}f_{u}de^{-rT}}{S_{0}u-S_{0}d}\\
&=\frac{S_{0}f_{u}-S_{0}f_{d}+S_{0}f_{d}ue^{-rT}-S_{0}f_{u}de^{-rT}}{S_{0}u-S_{0}d}\\
&=\frac{f_{u}(1-de^{-rT})+f_{d}(ue^{-rT}-1)}{u-d}
\end{align*}
Let
\begin{align*}
fe^{rT}&=\frac{f_{u}(e^{rT}-e^{rT}de^{-rT})+f_{d}(e^{rT}ue^{-rT}-e^{rT})}{u-d}\\
&=\frac{f_{u}(e^{rT}-d)+f_{d}(u-e^{rT})}{u-d}\\
&=f_{u}\left(\frac{e^{rT}-d}{u-d}\right)+f_{d}\left(\frac{u-e^{rT}}{u-d}\right)
\end{align*}
Where
\begin{align*}
p=\frac{e^{rT}-d}{u-d} \text{ ; }
1-p=\frac{u-e^{rT}}{u-d}
\end{align*}
Then
\begin{align*}
f=[f_{u}(p)+f_{d}(1-p)]e^{-rT}\\
\end{align*}
Consider the same scenario, where $r=0.05$, $T=0.25$, $f_{u}=1$, $f_{d}=0$, $u=1.10$, and $d=0.90$:\\
\begin{align*}
p&=\frac{e^{0.05\cdot0.25}-0.9}{1.1-0.9}=0.5629\\
f&=e^{-0.05\cdot0.25}(0.5629\cdot1+0.4371\cdot0)=0.5559
\end{align*}

\newpage
\begin{thebibliography}{1}
\bibitem{} Hull, John C. {\em Fundamentals of Futures and Options Markets.} 2017.
\end{thebibliography}

\end{document}


\documentclass[letterpaper,12pt,addpoints,answers]{exam}
%\usepackage[utf8]{inputenc}
\usepackage[english]{babel}

\usepackage[top=1in, bottom=1in, left=0.75in, right=0.75in]{geometry}
\usepackage{amsmath,amssymb,graphicx,xcolor,asymptote}
\usepackage{embedall}

\newcommand{\university}{SANTA MONICA COLLEGE}
\newcommand{\faculty}{Department of Mathematics}
\newcommand{\class}{Math 10}
\newcommand{\examnum}{Midterm Exam}
\newcommand{\examdate}{April 8, 2022}
\newcommand{\timelimit}{24 hours}

\newcommand{\tf}{$\quad\quad T\quad F$}
\newcommand{\tft}{$\quad\quad \boxed T\quad F$}
\newcommand{\tff}{$\quad\quad T\quad \boxed F$}
\newcommand{\tfnt}{$\quad\quad \boxed T\quad F\quad$neither}
\newcommand{\tfnf}{$\quad\quad T\quad \boxed F\quad$neither}
\newcommand{\tfnn}{$\quad\quad T\quad F\quad \boxed{\text{ neither}}$}
\newcommand{\tfn}{$\quad\quad T\quad F\quad$neither}

\pagestyle{headandfoot}
\firstpageheader{}{}{}
\firstpagefooter{}{Page \thepage\ of \numpages}{}
\runningheader{\class}{\examnum}{\examdate}
\runningheadrule
\runningfooter{}{Page \thepage\ of \numpages}{}

\begin{document}

\title{\Large \textbf{\university\\ \faculty\\
        \bigskip
        \class -- \examnum }}
\author{Instructor: Kevin Arlin}
\date{\examdate}
\maketitle
\noindent \rule{\textwidth}{1pt}

Submit your work as you would any homework assignment: direct submission of a .pdf is ideal.
The exam is open book and open notes. \textcolor{red}{It is not open Internet.}  I take
academic honesty during COVID even more seriously than usual, and have been known to
be rather vindictive when I see solutions that were garnered online. Nobody wants that.
Don't do it. Not even if you're having lots of trouble with the test. It is fine if you only get
a 65\%; you might even still get an 'A' that way. I am generous with curves as appropriate,
and may both award bonus points to the toughest problems here and also curve grades toward the
end of the course.

Show your work. Partial credit will be given when earned. Explanations are
required unless explicitly not requested.

\emph{Enter your name below to signal your assent to the following honor statement:}

I, \fillin[\emph{Elias Schablowski}][2.5in] affirm that I have not discussed the contents of this exam with anyone, whether in person or online, except perhaps Kevin Arlin, and that I will not do so before Tuesday, April 12.

\begin{center}
    \textbf{Distribution of Points}\\
    \medskip
    \multicolumngradetable{2}[questions]
\end{center}

\clearpage

This page is intentionally left blank to accommodate work that wouldn't fit elsewhere and/or scratch work.
\clearpage
\emph{Note: this problem serves as your entire quiz 6, but it'll be graded with partial credit.}

\begin{questions}
    \section*{Quiz 6}

    \question[4] Prove that, if $n>0$ is any positive integer, then $\sum_{j=1}^n 4j\cdot 3^j=(6n-3)\cdot 3^n+3.$

    \begin{solutionorbox}[7in]
        \paragraph{Definitions:}
        Let $f(n)=\sum_{j=1}^n 4j\cdot 3^j$.
        $$f(1) = 12$$
        $$f(n) = f(n-1) + 4n \cdot 3^n$$

        \paragraph{Base Step:} $f(1) = 12 = (6(1) - 3) \cdot 3^{(1)} + 3 = (3) \cdot 3 + 3 = 9 + 3 = 12$
        \paragraph{Inductive Step:} Assume
        $f(k) = (6k-3)\cdot 3^k+3$ when $k \geq 1$.
        Then $f(k+1) = f(k) + 4(k+1) \cdot 3^{(k+1)}$.
        Substituting $f(k)$ we get $f(k+1) = (6k-3)\cdot 3^k+3 + 4(k+1) \cdot 3^{(k+1)}$.
        Exapnding $f(k+1) = 6k \cdot 3^k + 4k \cdot 3^{(k+1)} + 4 \cdot 3^{(k+1)} - 3 \cdot 3^k + 3$.
        Substitute $3^k = \frac{3^{k+1}}{3}$ we get $f(k+1) = 2k \cdot 3^{k+1} + 4k \cdot 3^{(k+1)} + 4 \cdot 3^{(k+1)} - 3^{k+1} + 3$.
        Simplify $f(k+1) = 6k \cdot 3^{(k+1)} + 3 \cdot 3^{(k+1)} + 3$.
        Simplifying further we get $f(k+1) = (6k + 3) \cdot 3^{(k+1)} + 3 = (6(k + 1) - 3) \cdot 3^{(k+1)} + 3$.
    \end{solutionorbox}

    \clearpage

    \section*{Midterm}

    \question[10] The $abc$ conjecture states that there is a certain positive real number $\varepsilon$ such that
    whenever $a,b,c$ are natural numbers with $c=a+b,$ all bigger than a certain $k\in \mathbb N,$ then $c$ is no bigger than the $\mathrm{rad}(abc)^{1+\varepsilon}.$ Here $\mathrm{rad}:\mathbb N\to \mathbb N$ is a certain function
    whose definition you don't have to worry about. A student attempts to formalize this conjecture in predicate
    logic as follows:
    \[\exists\varepsilon\in (0,\infty)\forall (a,b,c)\in \mathbb N^3\exists k\in \mathbb N (a>k\wedge b>k\wedge c>k\to c\le \mathrm{rad}(abc)^{1+\varepsilon}.) \]
    There are exactly two errors in the student's formalization. Explain what they are and how to fix them.

    \begin{solutionorlines}[1in]
        The students logic equates the sentence "There exists a positive real number $\varepsilon$ such that for all $a$, $b$, and $c$ in $\mathbb{N}$, there exists a $k$ in $N$ such that $a$, $b$, and $c$ are larger than $k$ then $c\le \mathrm{rad}(abc)^{1+\varepsilon}$."\\
        This has two problems,
        primarily it does not say anything about $c=a+b$ (which can be fixed by either adding that $c=a+b$ to his logic or by substituting $c$ for $a+b$ which would also reduce the variables),
        and furthermore the student switched the relationship between $k$ and $a$, $b$, and $c$, which can be fixed by moving the $k$ before $a$, $b$, and $c$.

        \[\exists\varepsilon\in (0,\infty)\exists k\in \mathbb N\forall (a,b)\in \{(x,y) | (x, y) \in \mathbb N^2 \land x > k \land y > k\} \exists c \in \{a + b\}(c\le \mathrm{rad}(abc)^{1+\varepsilon}) \]
    \end{solutionorlines}
    \question
    \begin{parts}
        \part[3] How many elements are in $\mathcal P(\{\mathrm{purdy},\{\mathrm{purdy}\}\})\cap \{\mathrm{purdy},\{\mathrm{purdy}\}\},$ where purdy is my anxious tabby cat? Explain why.

        \begin{solutionorlines}[1in]
            $$\mathcal P(\{\mathrm{purdy},\{\mathrm{purdy}\}\}) = \{\emptyset, \{\mathrm{purdy}\}, \{\{\mathrm{purdy}\}\}, \{\mathrm{purdy}, \{\mathrm{purdy}\}\}$$

            The only common element between them is $\{\mathrm{purdy}\}$ so \linebreak $\mathcal P(\{\mathrm{purdy},\{\mathrm{purdy}\}\})\cap \{\mathrm{purdy},\{\mathrm{purdy}\}\} = \{\{\mathrm{purdy}\}\}$, which is one element.
        \end{solutionorlines}

        \part[7] Suppose $A$ and $B$ are finite sets with cardinalities $|A|=a$ and $|B|=b,$ with $b\ge a.$ In terms of $a$ and
        $b,$ what are all the possibilities for $|A\cup B|?$ Show that all the possibilities you claim are possible,
        and that no other possibilities are possible.

        \begin{solutionorlines}[2in]
            \paragraph{Case $A \subseteq B$:} This means that $\forall x \in A(x \in B)$. Therefore $A \cup B = B$ and therefore $|A\cup B| = |B| = b$.
            \paragraph{Case $A \subset B$:} Let $C = A \cap B$ and $c = |C|$. Then $|A \cup B| = a + b - c$ where $c < a \land c < b$ since all of $B$'s and some of $A$'s elements are counted. Therefore $b < |A \cup B| < a + b$.
            \paragraph{Case $A \nsubseteq B$:} Then $|A \cup B| = a + b$ as all elements from both $A$ and $B$ are counted.

            \vspace{0.5cm}

            Thus we have proben: $b \leq |A\cup B| \leq a + b$.
        \end{solutionorlines}

    \end{parts}

    \question Say that a function $f:A\to B$ has the epi property if the following holds:
    \[\forall X\in\mathrm{Set}\forall g,h:B\to X (g\circ f=h\circ f\to g=h.)\]
    Here $\mathrm{Set}$ is the collection of all possible sets, so the first quantifier is over all sets
    $X$ and the second, over all functions $B\to X.$
    \begin{parts}
        \part[2] Show that $f:\{a,b,c\}\to \{d,e\}$ given by $f(a)=d,f(b)=d,f(c)=e$ has the epi property.

        \begin{solutionorlines}[3in]
            Let $z_1, z_2, z_3, z_4 \in X$.
            Let $g(d) = z_1$, $g(e) = z_2$, $h(d) = z_3$ and $h(e) = z_4$.
            Then $g(f(a)) = g(f(b)) = z_1$, $h(f(a)) = h(f(b)) = z_3$, and $g(f(a)) = g(f(b)) = h(f(a)) = h(f(b))$, therefore $z_1 = z_3$.
            Also $g(f(c)) = g(f(c)) = z_1$, $h(f(c)) = z_4$, and $g(f(c)) = h(f(c))$, therefore $z_2 = z_4$.
            Therefore $g = h$ and $f$ has the epi property.
        \end{solutionorlines}

        \part[2] Show that $f:\{1,2,3\}\to \{4,5\}$ given by $f(1)=f(2)=f(3)=4$ does not have the epi property.
        \begin{solutionorlines}[1.5in]
            Let $z_1, z_2 \in X$.
            Let $g(e) = z_1$, $h(e) = z_2$.
            Since $g(e)$ and $h(e)$ do not exist in $g \circ f$ and $h \circ f$, $z_1$ does not have to equal $z_2$ and therefore $g$ does not have to equal $h$.
            Thus $f$ does not have the epi property.
        \end{solutionorlines}
        \part[6] Conjecture and prove a more familiar way of describing a function with the epi property.
        \begin{solutionorlines}[3in]
            If and only if $\forall b \in B \exists a \in A(f(a) = b)$ ($f$ is \textbf{subjunctive}), then $f$ has the epi property.

            $g = h \leftrightarrow \forall b \in B(g(b) = h(b))$, therefore if $f$ is subjunctive and $g\circ f=h\circ f$, then $g = h$ as $\forall b \in B \exists a \in A(g(f(a)) = g(f(a)))$.

            Furthermore if $f$ isn't subjective, $\exists b \in B \forall a \in A(f(a) \notin B \to g(b) \neq h(b))$
        \end{solutionorlines}
    \end{parts}

    \question For any $n\in \mathbb Z_{>0}$ and $k\in \mathbb Z_n,$ let $\mu_k:\mathbb Z_n\to \mathbb Z_n$ be the multiplication-by-$k$ function, so $\mu_k(m)=k\times_n m.$
    \begin{parts}
        \part[2] Give an example of an $n$ and a $k\ne 0$ for which $\mu_k$ is not injective. (Explain why not.)
        \begin{solutionorlines}[0.5in]
            Let $n = 10$, $k = 5$.

            Since $\mu_5(5) = 25 \mod 10 = 5 = \mu_k(1)$
        \end{solutionorlines}
        \part[2] Give an example of an $n$ and a $k\ne 1$ for which $\mu_k$ is injective. (Prove it.)
        \begin{solutionorlines}[1in]
            Let $n = 3$ and $k = 2$.
            \paragraph{0} $\mu_2(0) = 0 \mod 3 = 0$
            \paragraph{1} $\mu_2(1) = 2 \mod 3 = 2$
            \paragraph{2} $\mu_2(2) = 4 \mod 3 = 1$
        \end{solutionorlines}
        \part[6] Do your best to state and prove necessary and sufficient conditions on $n$ and $k$ which guarantee $\mu_k$ is injective.
        \begin{solutionorlines}[3in]
            $$\forall x \in Z_n \forall y \in Z_n(\mu_k(x) = \mu_k(y) \to x = y)$$
            Therefore $kx = ky (\mod{n}) \to x = y$.

            Assume $k = n - 1 \lor k = 1$ is the sufficient and necessary condition for $\mu_k$ to be injective.

            Let $f(a) = (na - a) \mod n$
            $$f(a) = ((na \mod n) - (a \mod n)) \mod n$$
            $$ = ((((n \mod n)(a \mod n)) \mod n) - (a \mod n)) \mod n$$
            $$ = ((((0)(a \mod n)) \mod n) - (a \mod n)) \mod n$$
            $$ = (((0) \mod n) - (a \mod n)) \mod n$$
            $$ = ((0 - (a \mod n)) \mod n$$
            $$ = ((-(a \mod n))$$
            $$ = -(a \mod n)$$

            \paragraph{$k = n - 1$:}
            $$(n - 1)x = (n - 1)y (\mod{n}) \to x = y$$
            $$nx - x = ny - y (\mod{n}) \to x = y$$
            $$(nx - x) \mod{n} = (ny - y) \mod n \to x = y$$
            $$f(x) = f(y) \to x = y$$
            $$-x = -y \to x = y$$
            \paragraph{$k = 1$:}
            $$(1)x = (1)y \to x = y$$
            $$x = y \to x = y$$
        \end{solutionorlines}
    \end{parts}

    \question Let $C$ be the set of real numbers defined recursively as follows:
    \begin{itemize}
        \item (Base case) $\mathbb Q\subseteq C$
        \item (Recursive step) If $x,y\in C$ then $x+y,x-y,xy,x/y$ are in $C,$ and if $x\ge 0$ then so is $\sqrt x.$
    \end{itemize}
    \begin{parts}
        \part[2] Give three examples of elements of $C$ that are significantly different-looking from each other.
        \answerline[$34$, $\frac{321}{456}$, $\sqrt{345}$, $\frac{85\sqrt{432}}{453}$]
        \part[1] Give an example of a real number which is not in $C.$
        \answerline[$e$, $\pi$, $\tau$, etc.]
        \part[7] Suppose we want to define a function $d:C\to \mathbb N$ such that $d(x)$ is the number of nested square
        roots needed to express $x.$ What is wrong with this definition, and how could you improve it?
        \begin{itemize}
            \item $d(x)=0,$ if $x\in \mathbb Q$
            \item $d(x+y)=d(x-y)=d(xy)=d(x/y)=\max(d(x),d(y)),$ and $d(\sqrt x)=d(x)+1,$ when $x>0.$
        \end{itemize}
        \begin{solutionorlines}[1in]
            The problem is that it is undefined what happens if $x < 0$,
            one way to mitigate this is to add another case that $d(xy) = d(y)$ when $x < 0$.
        \end{solutionorlines}
    \end{parts}

    \bonusquestion (Bonus:) Consider an alphabet
    $\Sigma.$ In principle, $\Sigma$ can be any
    set, but you should think $\Sigma=\{0,1\}$
    or $\Sigma = \{a,b,\ldots,z\}.$

    The set $\mathcal R_\Sigma$
    of \emph{regular languages} over $\Sigma$ is defined
    as follows. Every element of $\mathcal R_\Sigma$ is
    in particular a set of strings over $\Sigma;$ that is, $\mathcal R_\Sigma\subseteq \mathcal P(\Sigma^*).$
    \begin{itemize}
        \item $\emptyset\in \mathcal R_\Sigma$
        \item $\forall s\in \Sigma, \{s\}\in \mathcal R_\Sigma$
        \item $\forall S \in \mathcal R_\Sigma, S^*\in \mathcal R_\Sigma,$ where $S^*$ is the smallest set
              containing $S$ and closed under concatenation. (So it contains all concatenations of $n$ strings from $S,$ for every $n.$)
        \item The union $S\cup T$ and concatenation $S\bullet T=\{st\mid s\in S,t\in T\}$ are in $\mathcal R_\Sigma$ whenever $S$ and $T$ are.
    \end{itemize}

    Prove that, with $\Sigma=\{0,1\},$ the set $S$ of
    bit strings that flip bits at most once (so
    $S$ contains $000011111$ and $11111$ but not $010$) is
    a regular language.
    \begin{solutionorlines}[7in]
        Let $S_k$ be the set of bit strings of $k$ bits that flip at most once.
        Let $T_1 = \{1\}$.
        Let $T_k$ be the set of $k$ $1$s.
        Let $F_1 = \{0\}$.
        Let $F_k$ be the set of $k$ $0$s.

        \paragraph{Base Step:} Since $\forall s \in \Sigma, \{s\} \in \mathcal R_\Sigma$, $T_0 = \{1\}$ is a regular language.
        \paragraph{Inductive Step:}
        Assume that $T_k \in \mathcal R_\Sigma$.
        Since $S\bullet T=\{st\mid s\in S,t\in T\}$ are in $\mathcal R_\Sigma$,
        if we substitute $T_0$ for $S$ and $T_k$ for $T$,
        we get $T_{k+1} = T_0 \bullet T_k = {t_0t_k | t_0 \in T_0, t_k \in T_k}$
        which is all in $\mathcal R_\Sigma$.

        \paragraph{Base Step:} Since $\forall s \in \Sigma, \{s\} \in \mathcal R_\Sigma$, $F_0 = \{0\}$ is a regular language.
        \paragraph{Inductive Step:}
        Assume that $F_k \in \mathcal R_\Sigma$.
        Since $S\bullet F=\{st\mid s\in S,t\in F\}$ are in $\mathcal R_\Sigma$,
        if we substitute $F_0$ for $S$ and $F_k$ for $F$,
        we get $F_{k+1} = F_0 \bullet F_k = {f_0f_k | f_0 \in F_0, f_k \in F_k}$
        which is all in $\mathcal R_\Sigma$.

        \paragraph{Base Step:}
        Since $S \in \mathcal R_\Sigma \land T \mathcal R_\Sigma \implies S \cup T \in \mathcal R_\Sigma$,
        and $S_0 = T_0 \cup F_0$ and $T_0, F_0  \in \mathcal R_\Sigma$, $S_0  \in \mathcal R_\Sigma$.

        \paragraph{Inductive Step:}
        Let $a, b \in \mathbb N(a + b = k)$.
        Since $S\bullet F=\{st\mid s\in S,t\in F\}$ and $S\cup T$ are in $\mathcal R_\Sigma$ if $S$ and $T$ are,
        if we substitute $T_a \cup F_a$ for $S$ and $T_b \cup F_b$ for $F$,
        we get $S_k = (T_a \cup F_a) \bullet (T_b \cup F_b)$ which is in $\mathcal R_\Sigma$.

        Since $S = \bigcup_i^\infty S_i$, $S \in \mathcal R_\Sigma$ as $S_k in \mathcal R_\Sigma$ and $S,T \in \mathcal R_\Sigma \to S \cup T$
    \end{solutionorlines}
\end{questions}
\end{document}


\section{Introduction} Instrumental variables estimation has gained considerable traction in recent decades as a tool for causal inference, particularly amongst empirical researchers. However, this has also highlighted the importance of taking a deeper look at the theoretical properties underlying one's estimator of choice, whether it is instrumental variable estimator or any other estimator. A case-in-point is the well-known 1991 quarter-of-birth study on returns to schooling estimates by \cite{angrist1991does} and the equally famous rebuttal of their results published by \cite{bound1995problems}.

\par The \cite{bound1995problems} critique of the quarter-of-birth study in essence formed the starting point of the weak instruments literature. They provided simulation evidence that the instruments used by \cite{angrist1991does} were very weak and this in turn led to misleading results. Interestingly, their study was not the first time that the poor finite-sample behavior of the instrumental variables estimator (two-stage least squares estimator in particular) was discussed. Previously, numerous authors had presented results on the magnitude of the finite-sample bias of the 2SLS estimator towards OLS, such as \cite{nagar1959bias}, \cite{basmann1960asymptotic}, \cite{richardson1968exact} and \cite{sawa1969exact}.
However, their critique was the first to suggest some practices for the detection of possible presence of weak instruments (reporting first-stage F-statistic and the $R^2$ of the first stage regression), which have since become standard practice for using instrumental variables in empirical studies.

\par The literature on weak instruments have since developed considerably and continues to do so. \cite{staiger1997stock} showed that conventional asymptotic approaches fail to provide good approximations for weak instruments, and formalized the problem of weak instruments by introducing `weak-instrument asymptotics' which mimics the situation of weak instruments better. \cite{stock2002survey} provide a comprehensive survey of the weak instruments literature, wherein they emphasize that the problem of bias of the two-stage least-squares estimator is not
solely a small sample problem. It affects estimates carried out on large sample sizes as well, as is the case with the quarter-of-birth study which has a dataset of around 300,000 observations.

\par Further, given the poor finite-sample properties of the two-stage least squares estimator, numerous alternate estimators have also been proposed. These include the USSIV estimator suggested by \cite{angrist1995split}, Fuller-k estimator by \cite{fuller1977some}, bias-adjusted 2SLS estimator by \cite{donald2001choosing}, JIVE estimators by \cite{angrist1999jackknife} and \cite{blomquist1999small}. Another estimator, the LIML was formalized by \cite{anderson1949estimation}. In our paper, we take a deeper look at two of these estimators - JIVE and LIML estimators.

\par We introduce the two conditions for instrumental variables estimation, the first of which will be a recurring theme in our paper, since it connects to the problem of weak instruments. Consider the standard population regression model:
\begin{align*}
Y_i &= \beta_0 + \beta_1X_i + \varepsilon_i, \: \: \: \: i = 1,2,...n
\end{align*}
where $\varepsilon_i$ is the error term representing omitted factors that determine $Y_i$. Variables correlated with the error term are termed endogenous variables and those uncorrelated with the error term are exogenous. A valid instrument $Z_i$ must satisfy two conditions:
\begin{itemize}
    \item Instrument Relevance: corr ($Z_i$, $X_i$) $\neq 0.$\\
    It is the implications of this condition, specifically the strength (or weakness) of this correlation between the instrument and the endogenous regressor, that we are interested in. Instruments that explain little of the variation in X are called weak instruments. A precise definition is introduced in section 3.1.
    \item Instrument Exogeneity: corr ($Z_i$, $\varepsilon_i$) = 0\\
    Note that this condition cannot be statistically tested, since it involves the covariance between $Z_i$ and the unobserved $\varepsilon_i$.
\end{itemize}
\par If number of instruments (K) equals the number
of endogenous regressors (L) we have the just or exactly identified case.  If the number of instruments exceeds the number of endogenous regressors, K $\geq$ L, then we have the over-identified case.

\par The rest of the paper is sectioned as follows. Chapter 2 presents a derivation of the 2SLS estimator and its limitations in the weak instruments case. Chapter 3 discusses methods to test for weak instruments. Chapter 4 presents the alternative estimators (JIVE and LIML). Chapter 5 presents the simulation study. Chapter 6 presents the application to returns to schooling. Chapter 7 concludes.


\documentclass[12pt,pdf,singlespace]{article}

\usepackage{times}

\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{listings}
\usepackage{verbatim}

\title{Using geodesic Levenberg-Marquardt}

\author{Colin Clement \& Mark K.~Transtrum}

\begin{document}
{
\maketitle
}

\section{General strategy:}

The first thing to do is examine the document string in the \texttt{geodesiclm} subroutine found in \texttt{geodesiclm.f90}, and make sure you provide at least the function which computes the residuals (\texttt{func}). Be careful to use the form specified in the code's documentation.

The strategy for use we suggest is to start by turning everything fancy off, so that this algorithm functions as a standard Levenberg-Marquardt (LM) algorithm. I.e. start with \texttt{iaccel}=0, \texttt{ibold}=0 and \texttt{ibroyden}=0. When you do this, you should pay most attention to \texttt{factoraccept} and \texttt{factorreject}. I recommend starting with 5 and 2, respectively. Standard LM algorithms set both of these parameters to 10. The problem with making them equal is that the optimizer can get stuck in cycles. For more nuanced reasons (we call it `delayed gratification' \cite{transtrum2012improvements}) we also recommend that the accept factor be larger than the reject factor. Play around with these until you are satisfied with the convergence rate.

You might find at first that the default convergence criterion (tolerances and goals, see the list below) are causing the algorithm to give up before true convergence. The procedure for diagnosing and ameliorating this is as follows: After the algorithm gives up, read the convergence message. The integer provided will represent a specific convergence criterion being hit (the list in is \texttt{geodesiclm.f90}), and you should decrease the offending condition to force the algorithm to work harder. Do so until the algorithm hits a different convergence criterion, and repeat until either you are satisfied with the minimum, or the speed of convergence becomes your principal concern. If you are faced with the latter problem, read on for useful options for speeding convergence.

Once you get things working reasonably well with standard LM settings, try turning on geodesic acceleration by setting \texttt{iaccel}=1. If you are not providing a routine to calculate the second directional derivative (see \cite{transtrum2012improvements,transtrum2012geodesic, PRE83} for details), then you need to set \texttt{analytic\_avv} = \texttt{.false.} so that finite differences are used to estimate it. You will need to pay attention to the finite difference step size \texttt{h2} to make sure your calculation is stable. We suggest using \texttt{h2}=0.1. Since the acceleration should be a correction to standard LM, the parameter \texttt{avmax} limits the length of the acceleration vector in proportion to the proposed LM step size. As such, \texttt{avmax}$<1.$ are recommended.

If you are satisfied with the behavior at this point, experiment with allowing uphill steps by setting \texttt{ibold} from 1 (least `bold'  uphill allowances) to 4 (most `bold' uphill allowances). You can also play with setting \texttt{ibroyden}=1, which will employ an estimate of the Jacobian to speed up some iterations, sacrificing some accuracy.

For more details on why least-squares fitting is so challenging, and for intuition on how various parts of this algorithm work, see \cite{PRL104, PRE83}. See \cite{transtrum2012improvements} for details specific to implementation and benchmarked performance of the algorithm.

\section{Parameter defaults and details:}

These parameters are ordered by their relative importance. A lot of this information is a repeat of what can be found in the documentation to the source code in \texttt{geodesiclm.f90}.
~\\

\noindent {\bf Format:}
\begin{itemize}
	\item \texttt{parameter\_name} = $<$suggested default$>$ : Some words you might find useful.
\end{itemize}

\noindent {\bf Basic Levenberg-Marquardt (LM) Parameters:}
\begin{itemize}
	\item \texttt{factoraccept} = 5 : This is the factor by which the LM parameter (often denoted $\lambda$) is {\bf decreased after a step is accepted.}
	\item \texttt{factorreject} = 2 : This is the factor by which the LM parameter is {\bf increased after a step is rejected.}
	\item \texttt{maxlam} = 1.E7 : The routine will never let $\lambda$ become larger than this.
	\item \texttt{imethod} = 0 : This is an integer which specifies the exact way $\lambda$ is updated. See \texttt{geodesiclm.f90} and \cite{transtrum2012improvements} for more details.
	\item \texttt{initialfactor} = 1 : The initial $\lambda$.
	\item \texttt{mode} = 0 : 0 or 1. Chooses between the `Levenberg' and `Marquardt' style of matrix added to $J^T J$ \texttt{geodesiclm.f90}.
\end{itemize}
\noindent {\bf Basic convergence criterion:}

\begin{itemize}
	\item \texttt{maxiter} = 500 : The total number of steps the algorithm will attempt. This should depend on how long a function and Jacobian evaluation takes, and how long you are willing to wait.
	\item \texttt{maxfev} = 0 : The maximum number of allowed residual function (\texttt{func}) evaluations. 0 indicates no limit.
	\item \texttt{Cgoal} = 1 : The goal for the minimum squared error.
	\item \texttt{maxjev} = 0 : The maximum number of allowed Jacobian (\texttt{jacobian}) evaluations. 0 indicates no limit.
\end{itemize}

\noindent {\bf Geodesic acceleration parameters:}

\begin{itemize}
	\item \texttt{iaccel} = 1 : 1 or 0 for geodesic acceleration on or off.
	\item \texttt{h2} = 0.1 :  Finite-difference step size for second-directional derivative estimation.
	\item \texttt{avmax} = 0.75 : The `acceleration' should be a correction to the `velocity' and this enforces that.
	\item \texttt{maxaev} = 0 : Maximum number of analytic second directional derivative (\texttt{avv}) evaluations. 0 indicates no limit.
\end{itemize}

\noindent {\bf Extra flags:}

\begin{itemize}

\item \texttt{print\_level} = 5 : 0-1, how much information to print to \texttt{print\_unit}.
\item \texttt{print\_unit} = 6 : The integer representing the place to print messages. Either open a file or set to 6 for standard out.

\item \texttt{analytic\_jac} = \texttt{.true.} : True if you supply a routine to compute the Jacobian (\texttt{jacobian}).
\item \texttt{analytic\_avv} = \texttt{.false.} : True if you supply a routine to compute the second directional-derivative (\texttt{avv}).
\item \texttt{center\_diff} = \texttt{.true.} : True if you want to use centered-difference. More accurate but slower than forward-difference.

\end{itemize}

\noindent {\bf Advanced Jacobian update and convergence criterion:}

\begin{itemize}

\item \texttt{ibold} = 0 : (0-4) If nonzero this allows some uphill steps which can speed convergence.
\item \texttt{ibroyden} = 0 : If nonzero this employs Broyden approximate Jacobian updates. Can speed up algorithm with cost of accuracy.
\item \texttt{eps} = 1.5E-6 : Precision of evaluation of residuals in \texttt{func}.

\item \texttt{h1} = 1.0E-5 : Step size for finite-difference Jacobian estimate.
\item \texttt{artol} = 1.E-3 : See \texttt{geodesiclm.f90} for details on these convergence tolerances.
\item \texttt{gtol} = 1.5E-8
\item \texttt{xtol} = 1.E-10
\item \texttt{xrtol} = 1.5E-8
\item \texttt{frtol} = 1.5E-8

\end{itemize}

\input{Bibliography}

\end{document}


\documentclass[12pt]{article}
\usepackage[utf8]{inputenc}
\usepackage{float}
\usepackage{amsmath}

\usepackage[hmargin=3cm,vmargin=6.0cm]{geometry}
%\topmargin=0cm
\topmargin=-2cm
\addtolength{\textheight}{6.5cm}
\addtolength{\textwidth}{2.0cm}
%\setlength{\leftmargin}{-5cm}
\setlength{\oddsidemargin}{0.0cm}
\setlength{\evensidemargin}{0.0cm}

%misc libraries goes here
\usepackage{amssymb}

\begin{document}

\section*{Student Information }
%Write your full name and id number between the colon and newline
%Put one empty space character after colon and before newline
Full Name : Yavuz Selim YEŞİLYURT \\
Id Number : 2259166 \\

% Write your answers below the section tags
\section*{Answer 1}
\hspace{5mm} a) Using membership table to prove $A \cap B \subseteq (A \cup \overline B) \cap (\overline A \cup B)$:
\begin{table}[H]
\small
\centering
\begin{tabular}{|c|c|c|c|c|c|c|c|c}
\hline
$A$ & $B$ & $ \overline A $ & $ \overline B$ & $A \cap B$ & $A \cup \overline B$ & $\overline A \cup B$ & $ (A \cup \overline B) \cap (\overline A \cup B)$\\
\hline
1 & 1 & 0 & 0 & 1 & 1 & 1 & 1\\
1 & 0 & 0 & 1 & 0 & 1 & 0 & 0\\
0 & 1 & 1 & 0 & 0 & 0 & 1 & 0\\
0 & 0 & 1 & 1 & 0 & 1 & 1 & 1\\
\hline
\end{tabular}
\end{table}
b) Using membership table to prove $\overline A \cap \overline B \subseteq (A \cup \overline B) \cap (\overline A \cup B)$:
\begin{table}[H]
\small
\centering
\begin{tabular}{|c|c|c|c|c|c|c|c|c}
\hline
$A$ & $B$ & $ \overline A $ & $ \overline B$ & $\overline A \cap \overline B $ & $A \cup \overline B$ & $\overline A \cup B$ & $ (A \cup \overline B) \cap (\overline A \cup B)$\\
\hline
1 & 1 & 0 & 0 & 0 & 1 & 1 & 1\\
1 & 0 & 0 & 1 & 0 & 1 & 0 & 0\\
0 & 1 & 1 & 0 & 0 & 0 & 1 & 0\\
0 & 0 & 1 & 1 & 1 & 1 & 1 & 1\\
\hline
\end{tabular}
\end{table}

\section*{Answer 2}
\begin{align*}
f^{-1}((A \cap B) \times C) &= f^{-1}(A \times B) \cap f^{-1}(B \times C) \\
f^{-1}((A \times C) \cap (B \times C)) &= f^{-1}(A \times C) \cap f^{-1}(B \times C) \\
\end{align*}

\hspace*{5mm}\textbf{Part 1}, show that $f^{-1}((A \times C) \cap (B \times C)) \subseteq  f^{-1}(A \times C) \cap f^{-1}(B \times C)$;\\
Assume that $x \in f^{-1}((A \times C) \cap (B \times C))$, then $f(x) \in ((A \times C) \cap (B \times C))$,\\ so $f(x) \in (A \times C)$ and $f(x) \in (B \times C)$, therefore $x \in f^{-1}(A \times C)$ and $x \in  f^{-1}(B \times C)$,\\ so $ x \in ( f^{-1}(A \times C) \cap  f^{-1}(B \times C))$\\\\

\hspace*{5mm}\textbf{Part 2}, show that $ f^{-1}(A \times C) \cap f^{-1}(B \times C) \subseteq f^{-1}((A \times C) \cap (B \times C))$;\\
Assume that $x \in f^{-1}(A \times C) \cap f^{-1}(B \times C)$, then $x \in  f^{-1}(A \times C)$ and $x \in  f^{-1}(B \times C)$,\\ so $f(x) \in (A \times C)$ and $f(x) \in (B \times C)$, which implies $ f(x) \in ((A \times C) \cap (B \times C))$, \\ therefore $x \in  f^{-1}((A \times C) \cap (B \times C))$\\\\

\hspace*{5mm}Hence by part1 and part2 $f^{-1}((A \times C) \cap (B \times C)) =  f^{-1}(A \times C) \cap f^{-1}(B \times C)$\\
\hspace*{5mm}\quad \ Therefore $f^{-1}((A \cap B) \times C) = f^{-1}(A \times B) \cap f^{-1}(B \times C)$

\section*{Answer 3}
\hspace*{5mm}a)	 From the definition of one to one functions, if a function is one to one then for all x and for all y; if $f(x) = f(y)$ then $x = y$. We can clearly see this is not the case for this function. Since the function has $x^2$ term in its interior, it maps the negative items from $R$ to their positive equivalents in $R$ (i.e $(-x)^2 = x^2$ while $x \neq -x$ , even function). So we see that there is at least one situation which contradicts with our assumption, therefore $f(x)$ can not be one-to-one. We know that $f : A \rightarrow B$ is onto if $f(A) = B$. Which means if a function f is onto then every element in its codomain is being mapped. This function is defined from $R$ to $R$ so if $f(x)$ is onto then every item on $R$ must be mapped by $f(x)$. But negative values on $R$ will not be mapped by $f(x)$ . Therefore $f(x)$ can not be onto.\\\\
\hspace*{5mm}b)	From the definitions of one-to-one functions and onto functions on part a, $f(x)$ is one-to-one, since the function maps every distinct item from $R$ to a distinct item in $R$. But $f(x)$ is not onto, since the $0$ in $R$ will remain unmapped from $f(x)$. Therefore $f(x)$ can not be onto.

\section*{Answer 4}

\hspace{6mm}a)
Let A and/or B countably infinite sets such that $A=\{a_1,a_2...\}$ and
$B=\{b_1,b_2...\}$ we can write the elements of $A\times B$ as;\\\\
\begin{tabular}{ c| c c c c c}
 $A \setminus B$  &$ b_1$ & $b_2$ & $b_3$ & ...  \\
   \hline
 $ a_1$ & $(a_1,b_1)$ & $(a_1,b_2)$ & $(a_1,b_3)$ &...\\
 $ a_2$ & $(a_2,b_1)$ & $(a_2,b_2)$ & $(a_2,b_3)$ &...\\
 $ a_3$ & $(a_3,b_1)$ & $(a_3,b_2)$ & $(a_3,b_3)$ &...\\
 ... & ... & ... & ... & ...
\end{tabular}
\\\\
In table, first row, above the horizontal line contains the countably infinite elements of set $B$, first column, left of vertical line contains the countably infinite elements of set $A$. We can write $A \times B$ on this table and we can count the elements of $A \times B$ diagonally (i.e. 1st element: $(a_1,b_1)$, 2nd element: $(a_2,b_1)$, 3rd element $(a_1,b_2)$, 4th element: $(a_1,b_3)$, 5th element: $(a_2,b_2)$, 6th element: $(a_3,b_1)$ and so on. ). With this way of counting we will eventually get all elements in $A \times B$, then we can index these elements with unique numbers from $\mathbb{N}$, therefore we can get the one-to-one correspondence between $A \times B$ and  $\mathbb{N}$. Therefore we can infer that $A \times B$ is countable.\\

b)	Prove by contradiction, suppose that $B$ were countable, say with elements $b1, b2 , ...$ .Then since $A \subseteq B$ , we can list the elements of $A$ using the order in which they appear in this listing of $B$. Therefore $A$ is countable. But this contradicts with the hypothesis ($A$ is uncountable). Thus $B$ is not countable.\\

c)	If $B$ is countable with the elements $b1, b2 , ...$ and $A \subseteq B$, then we can list the elements of $A$ using the order in which they appear in this listing of $B$. Therefore $A$ is countable.

\newpage

\section*{Answer 5}
\hspace{6mm}a)	Let $f_1(x) = 7x^2$ and $f_2(x)=x^3$ ($f_1(x)$ is $\mathcal{O}(f_2(x)$) and they are increasing functions). \\
Prove by the Big-Oh definition,  $ln|f_1(x)|$ is $\mathcal{O}(ln|f_2(x)|)$ if $ln|f_1(x)|\leq  c.(ln|f_2(x)|)$ for some $x>k$. Let us check this condition: If  $ln|7x^2|$ is $\mathcal{O}(ln|x^3|)$ then $ln|7x^2|\leq c.ln|x^3|$ . Exponential both sides  $e^{ln|7x^2|} \leq e^{c.ln|x^3|}$, eliminate the terms  $7x^2 \leq e^c.x^3$, note that when $x\geq k=7$ and $c \geq 1$, we have $7x^2 \leq e^c.x^3$, consequently, we can take $c=1$ and $k=7$ as witnesses to establish the relationship $ln|7x^2|$ is $\mathcal{O}(ln|x^3|)$. Therefore $ln|f_1(x)|$ is $\mathcal{O}(ln|f_2(x)|)$.\\

b)	Let $f_1(x) = 7x^2$ and $f_2(x)=x^3$ ($f_1(x)$ is $\mathcal{O}(f_2(x)$) and they are increasing functions). \\
Prove by the Big-Oh definition, $3^{f_1(x)}$ is $\mathcal{O}(3^{f_2(x)})$ if  $3^{f_1(x)} \leq c.3^{f_2(x)}$ for some $x>k$. Let us check this condition: If $3^{7x^2}$ is $\mathcal{O}(3^{x^3})$ then $3^{7x^2} \leq c.3^{x^3}$ . Take the log of both sides $7x^2.log3 \leq x^3.log3+logc$, divide log3 both sides $7x^2 \leq x^3 + log(c-3)$, note that when $x\geq k=7$ and $c \geq 4$, we have $7x^2 \leq x^3 + log(c-3)$, consequently, we can take $c=4$ and $k=7$ as witnesses to establish the relationship $3^{7x^2}$ is $\mathcal{O}(3^{x^3})$. Therefore $3^{f_1(x)}$ is $\mathcal{O}(3^{f_2(x)})$.\\

\section*{Answer 6}
\hspace{6mm}a)
\begin{equation*}
\begin{split}
(3^x -1 )\ mod(3^y -1) &= 3^{(x \ mod \ y)} -1 \\
3^x -1		 &= k.(3^y -1) + 3^{(x \ mod \ y)}-1 \ \ \ \ \ \ \ where \  k \in \mathbb{Z} \\
3^x 			 &= k.(3^y -1) + 3^{(x \ mod \ y)} \\
log_3(3^x)		 &= log_3(k.(3^y -1) + 3^{(x \ mod \ y)}) \\
x			 &= log_3(k.(3^y -1) + 3^{(x \ mod \ y)})  \\
\end{split}
\end{equation*} \\
\quad Now substitute $x$ in original equation:\\
\begin{equation*}
\begin{split}
(3^{log_3(k.(3^y -1) + 3^{(x \ mod \ y)})}-1)\ mod(3^y -1) &= 3^{(x \ mod \ y)} -1 \\
k.(3^y -1) + 3^{(x \ mod \ y)}-1		 				&= k.(3^y -1) + 3^{(x \ mod \ y)}-1  \\
0											&= 0 \ \ \ \ \ \ \ (1) \\
\end{split}
\end{equation*} \\
\quad Since we get (1) in the last step, $(3^x -1 )\ mod(3^y -1) = 3^{(x \ mod \ y)} -1$ is true.\\

b)
\begin{equation*}
\begin{split}
gcd(123,277) &= gcd(277,123 \ mod\ 277) \\
			 &= gcd(277,123) \\
			 &= gcd(123,277 \ mod\ 123) \\
			 &= gcd(123,31) \\
			 &= gcd(31,123 \ mod\ 31)  \\
			 &= gcd(31,30)  \\
			 &= gcd(30,31 \ mod\ 30)   \\
		 	 &= gcd(30,1)   \\
		 	 &= gcd(1,30 \ mod\ 1)    \\
			 &= gcd(1,0)    \\
			 &= 1           \\
\end{split}
\end{equation*} \\ \\

\end{document}