\chapter{Graphing}
\section{Objectives}

\begin{enumerate}
    \item To learn to plot graphs using (i) a pencil and graph paper, (ii) spreadsheets, and (iii) Python.
\end{enumerate}

\section{Introduction}

Before you begin, read the \href{https://www.physics.mun.ca/~cdeacon/publications/graphs_tpt.pdf}{article on graphing by Christopher Deacon}. You don't need to grasp all the points in it right way, but as you grow in experience you will appreciate them.

While we will show you how to use Python, the purpose of this lab is not to familiarise you to coding -- there is another course that does that. Everything here can alternatively be done using MS Excel.

\begin{imp}
As with the other Physics courses, we will be using \textbf{\href{https://jupyter.org}{Jupyter Notebook}} for coding in Python.  Go to the link and follow the instruction procedure to install \href{https://www.anaconda.com/downloads}{Anaconda}, a Python distribution that comes with Jupyter pre-installed. This installation takes a significant amount of time. As a result, you should get it done \textbf{before this session begins}.
\end{imp}

\section{Graphing using a pencil and graph paper}
Graphing, like many other scientific activities, begins with a set of questions: we wish to find something, in this case a pattern, and we would like to plot our graph in such as way as to see and show that pattern most effectively. Some of these patterns may already be obvious in the data, e.g. you must have noticed that the time period $T$ of the pendulum does not seem to depend on the mass $m$ of the bob. When you compare the columns for $T$ and the length $l$ of the pendulum, you see that $T$ increases with $l$, but not in the most obvious way. Finally, when you look at how (and whether) $T$ changes with the amplitude $\theta$ of the oscillation, you will see no obvious pattern, even though there is one hidden in the data.

\subsection{Plotting: Stage 1}
Let us begin by exploring the variation of $T$ with respect to $m$ and with respect to $l$. We will consider the variation of $T$ with $\theta_\text{o}$ later.
\vspace{-0.5cm}
\begin{enumerate}
\itemsep0em
    \item On a graph sheet, draw the $x$ and $y$ axes. The $x$ axis is normally used for the parameter that you control, sometimes called the independent parameter, and $y$ axis for the parameter that you observe, sometimes called the dependent parameter. For example, when you plot $T$ vs $m$, $T$ is along $y$ and $m$ along $x$.

    \item Along each axis, choose a range that (i) covers the values you use and observe, and (ii) uses a scale that assigns a reasonable value to each division of the grid. So far as possible use units that make the numbers for the variables (i) small and (ii) similar along both axes. (If you write the mass in $kg$ and the time in $s$, you will understand what \textit{not} to do.)

    \item Plot the observed point, and then circle each point so that it visible.

    \item If the pattern appears to be straight line, use a ruler to draw the line that seems intuitively the most appropriate. (We will see later what this means.) If the pattern seems non-linear, sketch free-hand the curve that seems, once again, most appropriate.

\end{enumerate}

\subsection{Plotting: Stage 2}

You will have noticed that the way $T$ changes with $l$ is not linear. In fact, if you look carefully at the values, you will see that when $l$ is doubled, $T$ changes by about a factor of $1.4$. Since $1.4$ is close to $\sqrt{2}$, this suggests that it is $T^2$ that doubles when $l$ is doubled. In other words, we expect a linear dependence of $T^2$ on $l$. To see if this true, plot $T^2$ against $l$.

\subsection{Plotting: Stage 3}

If $T^2$ depends linearly on $l$, $T$ must depend linearly on $\sqrt{l}$. This implies that $\log T$ depends linearly on $\log l$. Make columns for $\log T$ and $\log l$ in your table, and make the corresponding plot.

\begin{question}
\paragraph{Question:} Compare the slopes of the $T^2 - l$ and $\log T - \log l$ graphs.

\paragraph{Question:} What are the advantages of plotting the log of a quantity?

\paragraph{Question:} Suppose that you are examining the variation of $y$ with respect to $x$, it is sometimes useful to plot $\log y$ vs $x$ (not $\log x$). What kind of dependence would this indicate?

\end{question}

\section{Graphing using your computer}

When you were drawing the $T^2 - l$ and $\log T - \log l$ graphs you were essentially using your artistic sense, which allowed you to judge what the \textit{best} straight line is, i.e. the line that seems to best fit the points. Notice that the confidence with which you arrive, artistically, at the the best line depends on the scatter of the points about it. Thus, the greater the scatter, the less confident you are (or ought to be) of your artistic fit. It becomes imperative, therefore, to find an unambiguous, quantitative way of arriving at the best fit.

To do this we begin by understanding that a straight line is determined \textit{completely} by just two points. In other words, when we have more than two points, there is no such thing as \textit{the} line that fits the points. In mathematical language, the problem is \textit{over-determined}. However, we can certainly ask what is the \textit{best} line, i.e. the one that is most representative of all the points, or, in some sense (yet to be defined) closest to them.

To find the best line, we go back to the remark that a line is determined completely by two points. Another way of saying this is that a line is determined uniquely by two \textit{parameters} -- the slope $m$ and the $y$-intercept $c$. So when we have many points, we can think of a host of lines with different with a range of different $m$s and $c$s. The question is: Which line is the best? To answer this question mathematically (rather than artistically) we need a quantitative measure of the \textbf{goodness of fit} of a line given more than two points. This measure of the goodness of fit must depend on $m$ and $c$, and the process of finding the best line is thus the process of \textit{optimisation} with respect to $m$ and $c$. The mathematics of this is a little advanced, but by the end of this semester you will be able to work it out.

For the moment, you will allow the computer to use standard algorithms to find the best straight line for us. To do that, you have to learn how to get our computer to import the data we want to plot and use it in such a way as to find the best straight line. For that purpose we will first use Microsoft Excel and then Python.

\begin{question}
\paragraph{Question:} Show that if you have just two points, then $m$ and $c$ are uniquely determined.
\end{question}

\subsection{Graphing with Microsoft Excel}

Excel is a \textit{spreadsheet}. Computer spreadsheets are powerful tools to analyse data and are used in a variety of different disciplines as they allow one to represent and analyse data easily and efficiently.

\begin{tip}
We will be describing using Microsoft Excel in this section. However other spreadsheet programs are very similar. For example, LibreOffice Calc and Google Spreadsheets are two other programs that can be used and possess more or less the same functionality.
\end{tip}

\subsubsection{Tabulating Data}

Let's begin with a sample data table verifying Ohm's law, shown in Table (\ref{table:ohm}).

\begin{table}[!htb]
\centering
\begin{tabular}{|C{4cm}|C{4cm}|}
\hline
\rowcolor{orange!25}
\hline
\multicolumn{2}{|l|}{Experiment $n$: }\\
\rowcolor{orange!25}
\multicolumn{2}{|l|}{\textbf{Verification of Ohm's Law for a large resistance}}\\
\hline
\multicolumn{2}{|l|}{Data collected by: Philip Cherian,}\\
\multicolumn{2}{|l|}{Expt. performed : 29/01/2019 to 05/01/2019}\\
\hline
\multicolumn{2}{|l|}{Resistance $R1$; Multimeter \textit{Victor} VC97}\\
\hline
\rowcolor{orange!50}
\textbf{Voltage} {\color{black}(V)} & \textbf{Current} {\color{black}($\mu A$)}\\
\hline
1&3\\
2&5\\
3&6\\
4&7\\
5&10\\
6&13\\
7&15\\
8&16\\
9&19\\
10&20\\
11&22\\
12&24\\
13&26\\
14&29\\
15&31\\
16&33\\
17&35\\
18&36\\
19&39\\
20&41\\
 \hline
\end{tabular}
\caption{Sample data for verification of Ohm’s law for a large resistance.}
\label{table:ohm}
\end{table}

Reproduce the above data table in MS Excel (be sure you learn how to include the $\mu$ symbol!\footnote{You will often need to insert Greek characters ($\lambda$, $\theta$, $\sigma\hdots$); add these using the \texttt{Insert} $\rightarrow$ \texttt{Symbol} option in MS Excel. Google Sheets does not have this functionality, but you can paste special characters from Google Docs.}).

\begin{tip}
Leave a few rows at the top of the spreadsheet for information about the data collected. In the above case we have included the information about who took the data and when, and also information about the equipment.
\end{tip}

\subsubsection{Plotting and Formatting your Graph}

You are now ready to begin plotting this data. Begin by selecting both columns and \texttt{Insert} a \texttt{Chart}. When you are asked to select a \texttt{Chart Type}, select the \texttt{XY (Scatter)} option with points \textbf{only}. This is important: you might be tempted to ``join the dots'', but that would be misleading: especially if the graph is not linear! When you are done with this, label your axes (\textbf{including units!}): most spreadsheet software take the first column to be the $x$-values and the second the $y$-values, but this can be changed. Your graph should look something like Figure (\ref{fig:samplegraph1}).

\begin{figure}[!htb]
    \centering
    \includegraphics[scale=0.75]{figs/samplegraph1.png}
    \caption{Sample graph of the data in Table (\ref{table:sampledata}) with much to be improved.}
    \label{fig:samplegraph1}
\end{figure}

Excel's biggest user base is business, so default graph formats are mostly setup for that purpose. As a result, changes need to be made to bring it to a ``scientific'' format: \begin{enumerate}
    \item Remove any unnecessary white space. You can do this by adjusting the minimum and maximum values of the axes. Do this by right clicking the axes and selecting \texttt{Format Axes}.

    \item Remove the legend on the right when possible: any information regarding the graph should be added in the caption. You can select it and press \texttt{Delete}.

    \item Change the marking icon to a more scientific size and shape. You can right click the graph and select \texttt{Format Data Series} and edit the icon size and shape there.

    \item Remove the grid-lines in the plot area.
\end{enumerate}

You should now have a graph like the one shown in Figure (\ref{fig:samplegraph2}).

\begin{figure}[!htb]
    \centering
    \includegraphics[scale=0.8]{figs/samplegraph2.png}
    \caption{Same graph as in Figure (\ref{fig:samplegraph1}), with some modifications made.}
    \label{fig:samplegraph2}
\end{figure}

\subsubsection{Adding a Trendline}

It should appear to you now that this trend is quite linear. We can now use the computer's algorithm to get the ``best'' straight line between these points. Don't worry right now about how this is done, you will learn about this in some detail at a later point.

Right-clicking the data-points, select \texttt{Add Trendline}. In the dialog box that opens, choose a \texttt{Linear} regression type and select the checkbox that says \texttt{Show Equation}. You can also make the line dotted, and change its colour.

If your trendline equation looks something like this:
\begin{center}
    \texttt{f(x) = 2.02556390977444 x + 0.231578947368423}
\end{center}

then edit it to round it off to a more reasonable number of digits. You will understand how many digits is reasonable in the next chapter on error analysis, for right now, let us choose to keep three decimal places. You should now have a graph that looks like Figure (\ref{fig:samplegraph3}).

\begin{figure}[!htb]
    \centering
    \includegraphics[scale=0.8]{figs/samplegraph3.png}
    \caption{Same graph as in Figure (\ref{fig:samplegraph1}), with a formatted trendline.}
    \label{fig:samplegraph3}
\end{figure}

\begin{tip}
In Microsoft Excel, you can make the current graph style a ``template'', so that you don't have to keep making most of these changes every time you plot new data. To do this, right-click the chart, and select \texttt{Save as Template}. Choose an appropriate name for the chart template and save it. The chart template automatically appears in the \texttt{Templates} folder for charts. You can access it on the \texttt{All Charts} tab in the \texttt{Insert Chart} or \texttt{Change Chart Type} dialog box, where you can apply a chart template like any other chart type.
\end{tip}

%\todo[inline,color=cyan]{Include corrected JupterNotebooks.}

\newpage


\documentclass[../main.tex]{subfiles}
\begin{document}
\chapter{Iteration and Recursion}
\label{chapter_iteration_recursion}
\begin{chapquote}
{Niklaus Wirth, \textit{Algorithms + Data Structures = Programs, 1976}}
``The power of recursion evidently lies in the possibility of defining an infinite set of objects by a finite statement. In the same manner, an infinite number of computations can be described by a finite recursive program, even if this program contains no explicit repetitions.''
\end{chapquote}

\section{Introduction}
\label{iteration_recursion_introduction}
\begin{figure}[h]
    \centering
    \includegraphics[width = 0.7\columnwidth]{fig/iteration_recursion.png}
    \caption{Iteration vs recursion: in recursion, the line denotes the top-down process and the dashed line is the bottom-up process. }
    \label{fig:iteration_vs_recursion}
\end{figure}
In computer science, software programs can be categorized as either \textit{iteration} or \textit{recursion}, thus making iteration and recursion as the topmost level of concepts in software development and the very first base for us to study computer science techniques. Iteration refers to a \textit{looping} process which repeats some part of the code until a certain condition is met. Recursion, similarly, needs to stop at a certain condition, but it replaces the loop with recursive function calls; meaning  a function calls itself from within its own code. The process is shown in Fig.~\ref{fig:iteration_vs_recursion}.

Do you still have the feeling that you seemingly already understand the iteration even without code, but what is recursion exactly? Recursion can be a bit of challenging for beginners, it differs from our normal way of thinking. It is a bit of similar to the vision of being in the restroom which has two mirrors abrest on each side and facing each other, we see multiple images of the things in front of each mirror, and these images usually appear from large to small. This is similar to recursion. The relation between these recurred images can be called \textit{recurrence relation}.

Understanding recursion and learning basic rules to solve recurrence relation are two of the most purposes in this chapter. Thus, we organize the content of this chapter following this trend:
\begin{enumerate}
    \item Section.~\ref{sec_iteration_recursion} will first address our question by analyzing the recursion mechanism within the computer program, and we further understand the different between by seeing example of factorial series and examines the pros and cons of each.
    \item Section.~\ref{sec_recursion} advances our knowledge about recursion by studying the recurrence relation, including its definition, categorization and addressing how to solve recurrence relation.
    \item Section.~\ref{sec_iter_recur_examples} gives us two examples to see how iteration and recursion works in real practice.
\end{enumerate}
\begin{importantnote}
Deduce(find) the recurrence relation and sometimes solves it is a key step in algorithm design and problem solving,  solving the recurrence time relation is important to  algorithm analysis.
\end{importantnote}
%we will first andwer the question: recursion handles function calls in two passes -- \textit{top-down} and \textit{bottom-up}. In the top-down process, recursion starts from the entrance of the program, call itself   it is composed of two passes:Thus, in this chapter, we   The purpose of this chapter is to understand how iteration and recursion works in algorithms and software developing. Especially, to understand how recursion works, and we also include the most common usage of being recursion.

% \section{Iteration and Recursion}
% \label{sec_iteration_recursion}
In this section, we first learn iteration and Python Syntax that can be used to implement. We then examine a classic and elementary example--Factorial sequence to catch a glimpse of how iteration and recursion can be applied to solve this problem. Then, we discuss more details about recursion. We end this section by comparing iteration and recursion; their pros and cons and their relation between.

\section{Iteration}
In simple terms, an iterative function is one that loops to repeat some part of the code. In Python, the loops can be expressed with \texttt{for} and \texttt{while} loop.

Enumerating the number from $1$ to $10$ is a simple iteration. Implementation wise:
\begin{itemize}
    \item \texttt{for} usually is used together with function \texttt{range(start, stop, step)} which creates a sequence of numbers from \texttt{start} to \texttt{stop} in range $[start, end)$, and increments by $step$ (1 by default). Thus, we need to set \texttt{start} as 1, and \texttt{end} as 11 to get numbers from 1 to 10.
\begin{lstlisting}[language=Python]
# enumerate 1 to 10 with for loop
for i in range(1, 11):
  print(i, end=' ')
\end{lstlisting}
\item \texttt{while} is used with syntax
\begin{lstlisting}[numbers=none]
while expression
    statement
\end{lstlisting}
In our case, we need to set start condition which is $i=1$, and the expression will be limiting $i <= 10$. In the statement, we need to manually increment the variable \texttt{i} so that we wont not end up with infinite loop.
\begin{lstlisting}[language=Python]
i = 1
while i <= 10:
  print(i, end = ' ')
  i += 1
\end{lstlisting}
\end{itemize}

\section{Factorial Sequence}  The factorial of a positive integer $n$, denoted by $n!$, is the product of all positive integers less than or equal to $n$:
\begin{lstlisting}[numbers=none]
For example:
5! = 5 \times 4 \times 3 \times 2 \times 1 = 120.
0! = 1
\end{lstlisting}
% In this section, we are going to approach the problem using both iteration and recursion, conceptually and implementally.
To compute the factorial sequence at $n$, we need to know the factorial sequence at $n-1$, which can be expressed as a \textit{recurrence relation}, that $n!= n \times (n-1)!$).
\begin{itemize}
    \item Solving with iteration: we use a \texttt{for} loop starts at 1 up till $n$ so that we eventually build up our answer at $n$. We use a variable \texttt{ans} to save the factorial result for each number, and once the program stops, \texttt{ans} gives the result of our factorial for $n$.
\begin{lstlisting}[language=Python]
def factorial_iterative(n):
  ans = 1
  for i in range(1, n+1):
    ans = ans * i
  return ans
\end{lstlisting}
\item Solving with recursion: we start to call a recursive function at $n$, within this function, we can itself but instead with $n-1$ just as shown in the recurrence relation. We then multiply this recursive call with $n$. We need to define a bottom, which is the end condition for the recursive function calls to avoid infinite loop. In this case, it bottoms out at $n=1$, which we can know its answer would be $1$, thus we return 1 to stop further function calls and recursively return to its upmost level.
\begin{lstlisting}[language=Python]
def factorial_recursive(n):
  if n == 1:
    return 1
  return n * factorial_recursive(n-1)
\end{lstlisting}
\end{itemize}

\section{Recursion}
In this section, we reveal how the recursion mechanism works: function calls and stack, two passes.
\label{sec_recursion}
\begin{figure}[!ht]
    \centering
    \includegraphics[width=0.6\columnwidth]{fig/activation_records.png}
    \caption{Call stack of recursion function}
    \label{fig:call_stack_recursion_function}
\end{figure}
\paragraph{Two Elements}
When a routine calls itself either directly or indirectly, it is said to be making a recursive function call. The basic idea behind solving problems via recursion is to break the instance of the problem into smaller and smaller instances until the instances are so small they can be solved trivially. We can view a recursive routine as consisting of two parts.
\begin{itemize}
    \item Recursive Calls: As in the factorial sequence, when the instance of the problem is still too large to solve directly, we recursive call this function itself to solve problems of smaller size. Then the result returned from the recursive calls are used to build upon the result of the upper level using \textit{recurrence relation}. For example, If we use $f(n)$ to denote the factorial at $n$, the recurrence relation would be $f(n)=n\times f(n-1), n>0$.
    \item End/Basis  Cases: The above resursive call needs to bottom-out; stop when the instance is so small to be solved directly. This stop condition is called end/basic case. Without this case, the recursion will continue to dive infinitely deep and eventually we run of memory and get a crash. A recursive function can have one or more base cases. In the example of factorial, the base case is when $n=0$, by definition $0!=1$.
\end{itemize}

\paragraph{Recursive Calls and Stacks}
The recursive function calls of the recursive factorial we implemented in the last section can be demonstrated as Fig.~\ref{fig:call_stack_recursion_function}.

The execution of recursive function $f(n)$ will pay two visits to each resursive function $f(i), i \in [1, n]$ through two passes: \textit{top-down} and \textit{bottom-up} as we have illustrated in Fig.~\ref{fig:iteration_vs_recursion}. The recursive function handles this process via a \textit{stack} data structure which follows a Last In First Out (LIFO) principle to record all function calls.

\begin{itemize}
    \item In the top-down pass, each recursive function's execution context is ``pushed'' into the stack in the order of $f(n)$, $f(n-1)$, ..., $f(1)$. The process ends till it hits to the end case $f(0)$, which will not be ``pushed'' into the stack but execution some code and \texttt{returns} value(s). The end case marks as the start of the bottom-up process.

\item In the bottom-up pass, the recursive function's execution context in the stack is ``poped'' off the stack in a reversed order: $f(1)$, ..., $f(n-1)$, $f(n)$. And $f(1)$ takes the returned value from the base case to construct its value using the recurrence relation. Then it returns its value up to the next recursive function $f(2)$. This whole process ends at $f(n)$ which returns its value.
\end{itemize}

\paragraph{How Import Recursion Is?}
Recursion is a very powerful and fundamental technique, and it is basis for several other design principles, such as:
\begin{itemize}
    \item Divide and Conquer (Chapter.~\ref{chapter_divide_conquer}).
    \item Recursive Search, such as Tree Traversal and graph search.
    \item Dynamic Programming (Chapter.~\ref{chapter_dynamic-programming}.
    \item Combinatorics such as enumeration (permutation and combination) and branch and bound etc.
    \item Some classes of greedy algorithms.
\end{itemize}
It also supports the proof of correctness of algorithms via mathematical induction, and consistently arise in the algorithm complexity analysis. We shall see through out this book and will end up drawing this conclusion ourselves.

\paragraph{Practical Guideline}
In real algorithmic problem solving, different process normally has different usage.

In top-down process we do:
\begin{enumerate}
    \item Break problems into smaller problems, there are different ways of ``breaking'' and depends on which, they can either be \textit{divide and conquer} or \textit{decrease and conquer} which we will further expand in Chapter.~\ref{} and \ref{}. Divide and conquer will divide the problems into disjoint subproblems, whereas in decrease and conquer, the problems
    \item Searching: visit nodes in non-linear data structures (graph/tree), visit nodes in linear data structures. Also, at the same time, we can use \textbf{pass by reference} to track the state change such as the traveled path in the path related graph algorithms.
\end{enumerate}

In bottom-up process, we can either return \texttt{None} or \texttt{variables}. Assume if we already used \textbf{pass by reference} to tack the change of state, then it is not necessarily to return variables. In some scenario, tracking states with by passing by reference can be more easier and more intuitive. For example, in the graph algorithm, we mostly like to use this method.

\paragraph{Tail Recursion}
This is also called \textit{tail recursion}  where the function calls itself at the end (``tail'') of the function in which no computation is done after the return of recursive call. Many compilers optimize to change a recursive call to a tail recursive or an iterative call.

%Returning variables
% \begin{enumerate}
%     \item Using rreturn None: Simply return to the upper level if we already used \textbf{pass by reference} to tack the change of state.
%     \item return variables: We get the result of the next recursive function call and using the recurrence relation to construct the result of current function. if we have return result, we do process of these results with current state and return to the upper level. In divide and conquer, we mostly likely need to merge its results. For iteration, this process gives the iteration process the backward traveling process.
% \end{enumerate}

% \paragraph{Examples of Applications}
% For iteration, the top-down process is visiting nodes in 'forwarding' direction, and the bottom-up process on the other hand functions as a reverse visiting process. This makes a linear data structures function as a doubly linked list, and make a one direction tree structure function as one with parent. Here we list some examples that used recursive so that we can go backward:
% \begin{enumerate}
%     \item 2. Add Two Numbers
% \end{enumerate}

\section{Iteration VS Recursion}

\paragraph{Stack Overflow Problem}
In our example, if we call function \texttt{factorial\_recursive()} with $n=1000$, Python would have complain an error as:
\begin{lstlisting}
RecursionError: maximum recursion depth exceeded in comparison
\end{lstlisting}
which is a \textit{stack overflow} problem.
A stack overflow is when we run out of memory to hold items in the stack. These situations can incur the stack overflow problem:
\begin{enumerate}
    \item No base case is defined.
    \item The recursion is too deep which is out of the assigned memory limit of the executing machine.
\end{enumerate}
\paragraph{Stack Overflow for Recursive Function and Iterative Implementation} According to Wikipedia, in software, a stack overflow occurs if the call stack pointer exceeds the stack bound. The call stack may consist of a limited amount of address space, often determined at the start of the program depending on many factors, including the programming language, machine architecture, multi-threading, and amount of available memory. When a program attemps to use more space than is available on the call stack, the stack is said to \textit{overflow}, typically resulting in a program crash. The very deep recursive function is faced with the threat of stack overflow. And the only way we can fix it is by transforming the recursion into a loop and storing the function arguments in an explicit stack data structure, this is often called the iterative implementation which corresponds to the recursive implementation.

We need to follow these points:
\begin{enumerate}
    \item End condition, Base Cases and Return Values: either return an answer for base cases or None, and used to end the recursive calls.
    \item Parameters: parameters include: data needed to implement the function, current paths, the global answers and so on.
    \item Variables: What the \textbf{local} and {global} variables. In Python any pointer type of data can be used as global variable global result putting in the parameters.
    \item Construct current result: when to collect the results from subtree and combine to get the result for current node.
    \item Check the depth: if the program will lead to the heap stack overflow.
\end{enumerate}

\paragraph{Conversion} For a given problem, conversion between iteration and recursion is possible, but the difficulty of the conversion is highly dependable on specific problem context. For example, the iteration of a range of numbers can be represented with recurrence relation $T(n)=T(n-1)+1$. On the side of implementation, some recursion and iteration can be easily converted between such as linear search; in some other cases, it takes more tricks and requires more sophisticated data structures to assist the conversion, such as in the iterative implementation of the recursive depth-first-search, it uses stack. Do not worry about these concepts here, as you flip more pages in the book, you will know and start to think better.

\paragraph{Tail recursion and Optimization}

In a typical recursive function, we usually make the recursive calls first, and then take the return value of the recursive call to calculate the result. Therefore, we only get the final result after all the recursive calls have returned some value. But in a tail recursive function, the various calculations and statements are performed first and the recursive call to the function is made after that. By doing this, we pass the results of the current step to the next recursive call to the function. Hence, the last statement in a Tail recursive function is the recursive call to the function.
This means that when we perform the next recursive call to the function, the current stack frame (occupied by the current function call) is not needed anymore. This allows us to optimize the code. We Simply reuse the current stack frame for the next recursive step and repeat this process for all the other function calls.

Using regular recursion, each recursive call pushes another entry onto the call stack. When the functions return, they are popped from the stack. In the case of tail recursion, we can optimize it so that only one stack entry is used for all the recursive calls of the function. This means that even on large inputs, there can be no stack overflow. This is called Tail recursion optimization.

Languages such as lisp and c/c++ have this sort of optimization. But, the Python interpreter doesn’t perform tail recursion optimization. Due to this, the recursion limit of python is usually set to a small value (approx, $10^4$). This means that when you provide a large input to the recursive function, you will get an error. This is done to avoid a stack overflow. The Python interpreter limits the recursion limit so that infinite recursions are avoided.

\paragraph{Handling recursion limit}
The ``sys module'' in Python provides a function called \texttt{setrecursionlimit()} to modify the recursion limit in Python. It takes one parameter, the value of the new recursion limit. By default, this value is usually $10^4$. If you are dealing with large inputs, you can set it to, $10^6$ so that large inputs can be handled without any errors.

% \section{Hands-on Examples}
% \label{sec_iter_recur_examples}

\section{Exercises}
\begin{enumerate}
    \item Compute factorial sequence using \texttt{while} loop.
\end{enumerate}

\section{Summary}
If a cursive algorithm can be further optimized, the optimization method can either be divide and conquer or decrease and conquer. We have put much effort into solving recurrence relation of both: the linear recurrence relation for decrease and conquer, the divide and conquer recurrence relation for divide and conquer.  Right now, do not struggle and eager to know what is divide or decrease and conquer, it will be explained in the next two chapters.
    % A conditional statement decides the termination of recursion while a control variable’s value decide the termination of the iteration statement (except in the case of a while loop).
    % Infinite recursion can lead to system crash whereas, infinite iteration consumes CPU cycles.
    % Recursion repeatedly invokes the mechanism, and consequently the overhead, of method calls. This can be expensive in both processor time and memory space while iteration doesn’t.
    % Recursion makes code smaller while iteration makes it longer.
\end{document}


\chapter{Building a Decision Tree \label{chapter:decisiontrees}}

{\bf Decision trees} were developed as an alternative to neural networks in the 1970s. We already saw them in Chapters~\ref{chapter:classification} and \ref{chapter:regression} as examples of supervised learning algorithms. Now we're going to get into a bit more detail about how these trees are learned from data.

Decision trees can be adapted to solve supervised learning problems with different types of outcomes. We will focus on \textbf{classification trees} in this chapter, but the same tree building principles can be applied to solve regression problems (see Chapter~\ref{chapter:regression}). Similar modifications can also allow us to construct \textbf{survival trees}, which model survival outcomes (represented as Kaplan-Meier curves; see Chapter~\ref{chapter:km}). Decision trees can also handle count outcomes, modeling them using Poisson distributions (see Chapter~\ref{chapter:probabilitydistributions}).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Example: The Wisconsin Breast Cancer Dataset \label{section:wisconsin}}

The Wisconsin breast cancer dataset\footnote{You can download the dataset from the UCI Machine Learning Repository here: \texttt{https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)}.} contains information about imaging features of a fine needle aspirate (FNA) of a breast mass from $569$ different subjects. Ten real-valued features are computed for each cell nucleus in the image:
{\small
\begin{enumerate}[label=(\alph*)]
\item radius (mean of distances from center to points on the perimeter)
\item texture (standard deviation of gray-scale values)
\item perimeter
\item area
\item smoothness (local variation in radius lengths)
\item compactness (perimeter$^2$ / area - 1.0)
\item concavity (severity of concave portions of the contour)
\item concave points (number of concave portions of the contour)
\item symmetry
\item fractal dimension, a statistical index of complexity quantifying how detail in a pattern changes with the scale on which it is measured
\end{enumerate}
}

The mean, standard deviation, and worst value of each feature are then computed, creating a total of $30$ features. Each image is also labeled by its true diagnosis: $B$ (for benign) or $M$ (for malignant).

Here is a decision tree for this dataset, built using the \texttt{rpart} package in R with default parameters:
\begin{center}
\includegraphics[width=0.7\textwidth]{img/wisconsin-decision-tree.png}
\end{center}

\begin{question}{}
What is the most important feature, as identified by the decision tree learning algorithm, for determining whether a breast mass is benign or malignant? What two other features are considered important by the tree? Which features are ignored completely?
\end{question}

\begin{question}{}
Looking at the decision tree for the Wisconsin Breast Cancer dataset, what do you think the advantages of a decision tree are for this problem over other classification methods, such as logistic regression and KNN?
\end{question}

Although there are several tree building algorithms, all of them are conceptually similar. They all try to reduce ``impurity'', or ``uncertainty'', in the outcome variable by intelligently splitting on the predictors. Trees are built recursively from root to leaves. Each internal node of the tree ``contains'' a subset of the overall dataset (this is why tree building is often called \textbf{recursive partitioning}, and why the R package is named \texttt{rpart}). At each stage, the algorithm will consider each of the existing leaf nodes and choose the split variable that maximally reduces uncertainty in the outcome. To understand how trees are built computationally, all we need to do is look at the math behind this idea.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The ID3 Algorithm}

One of the earliest approaches to building decision trees was the \textbf{ID3 algorithm}\footnote{Quinlan, J.R. ``Induction of decision trees'', Machine Learning, num. 1, pp. 81-
106, 1986.}. The ID3 Algorithm uses the concepts of entropy and information gain to build trees.

\subsection{Entropy}

\textbf{Entropy}, usually abbreviated $H$, is a measure of the uncertainty in the value of a random variable. It is the number of bits (on average) required to describe the outcome of the random variable. Here is the formula for the entropy of the discrete probability distribution governing the outcome of a random variable, $X$:
$$ H(X) = - \sum_{x} P(X = x) \log_2\left(P(X = x)\right) $$
For a Bernoulli random variable (see Chapter~\ref{chapter:probabilitydistributions}, Section~\ref{sect:bernoulli}), there are only two possible outcomes: 0 and 1. The entropy of this random variable is given by:
$$ H_\text{Bernoulli} = -\mu \log_2(\mu) - (1 - \mu) \log_2 (1 - \mu) $$
where $\mu$ is the sole parameter of the Bernoulli distribution: the probability of a positive outcome. Here is what the entropy of a Bernoulli distribution looks like as a function of $\mu$:
\begin{center}
\includegraphics[width=0.7\textwidth]{img/entropy-bernoulli.png}
\end{center}

\begin{question}{}
At which value(s) of $\mu$ are we maximally uncertain about the outcome? At which value(s) of $\mu$ are we completely certain about the outcome? This should make intuitive sense if you consider the meaning of $\mu$.
\end{question}

\subsection{Information Gain}

The goal of a decision tree is to reduce uncertainty about the outcome with every split. Let $Y$ be the outcome variable of a training set. Let $X$ be one of the predictors. \textbf{Information gain} is defined as:
\begin{align*} \text{Gain}(Y, X) &= H(Y) - \sum_{x \in \text{Values(X)}} \frac{|Y(X=x)|}{|Y|}~ H(Y(X=x)) \\
&= H(Y) - H(Y|X) \end{align*}
Since entropy is a measure of uncertainty, or impurity, in the outcome, information gain is the reduction in that uncertainty achieved by conditioning on the predictor, $X$. The tree will choose split variables for which the information gain is maximized.
\vspace{5mm}

\begin{question}{}
Say you have a dataset where the outcome is $Y = [0, 1, 0, 1, 0, 1]$ and there are two predictors: $X = [0, 0, 1, 1, 2, 2]$, and $Z = [1, 2, 1, 2, 1, 2]$. Intuitively, which predictor would make the better splitting variable and why? Calculate $Gain(Y, X)$ and $Gain(Y, Z)$. Which value is higher?
\end{question}

\subsection{Using Entropy to Build a Decision Tree}

By understanding the concepts of entropy and information gain, we arrive naturally at the ID3 Algorithm:
\begin{enumerate}
\item Start with a single node: the root of the tree.
\item At each current leaf node:
  \begin{enumerate}
  \item Compute the information gain for each feature.
  \item Split on the one with the highest gain.
  \end{enumerate}
\item Return to Step 2. Stop when either the class distributions at all leaf nodes are entirely pure or there are no more variables left to split on.
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Example: Happiness Dataset \label{section:happiness}}

Let's build a decision tree for a simple example, using the ID3 algorithm. Assume we surveyed 10 people and asked them whether they were happy or unhappy. We also asked whether they had friends (yes/no), money (poor/enough/rich), and free time (none/some). The data look like this:

{\small
\begin{center}
\begin{tabular}{ccccc}
\toprule
Datapoint ID & friends $(X_1)$ & money $(X_2)$ & free time $(X_3)$ & happy $(Y)$ \\
\midrule
1 & 1 & 1 & 0 & 0 \\
2 & 1 & 1 & 1 & 0 \\
3 & 0 & 1 & 1 & 0 \\
4 & 0 & 0 & 0 & 0 \\
5 & 1 & 0 & 0 & 0 \\
6 & 0 & 0 & 0 & 0 \\
\midrule
7 & 1 & 2 & 1 & 1 \\
8 & 1 & 0 & 1 & 1 \\
9 & 0 & 0 & 1 & 1 \\
10 & 1 & 0 & 0 & 1 \\
\bottomrule
\end{tabular}

\begin{align*}
	x_1 = \left\{ \begin{array}{cc} 0 & \text{no friends} \\ 1 & \text{friends} \end{array} \right.
	\qquad
	x_2 = \left\{ \begin{array}{cc} 0 & \text{poor} \\ 1 & \text{enough money} \\ 2 & \text{rich} \end{array} \right.
	\qquad
	x_3 = \left\{ \begin{array}{cc} 0 & \text{no free time} \\ 1 & \text{some free time} \end{array} \right.
\end{align*}
\end{center}
}

\begin{question}{}
Build a decision tree for this dataset using the ID3 algorithm. To get started, you need to know the entropy of the overall outcome distribution. It is:
$$ H(Y) = - \frac{4}{10} \log_2 \frac{4}{10} - \frac{6}{10} \log_2 \frac{6}{10} = \textbf{0.971} $$
We will go through the calculations below. As we go, you can start to draw the tree on another page.
\begin{enumerate}
\item[(a)] Perform the initial split at the tree root to determine which variable to split on first. Update the tree with this information.

\begin{align*}
\frac{|Y(X_1 = 0)|}{|Y|} H[Y(X_1 = 0)] &= \hspace{0.5\textwidth}\\[2mm]
\frac{|Y(X_1 = 1)|}{|Y|} H[Y(X_1 = 1)] &= \\[2mm]
\text{Gain}(Y, X_1) &= \\[2mm]
\frac{|Y(X_2 = 0)|}{|Y|} H[Y(X_2 = 0)] &= \\[2mm]
\frac{|Y(X_2 = 1)|}{|Y|} H[Y(X_2 = 1)] &= \\[2mm]
\frac{|Y(X_2 = 2)|}{|Y|} H[Y(X_2 = 2)] &= \\[2mm]
\text{Gain}(Y, X_2) &= \\[2mm]
\frac{|Y(X_3 = 0)|}{|Y|} H[Y(X_3 = 0)] &= \\[2mm]
\frac{|Y(X_3 = 1)|}{|Y|} H[Y(X_3 = 1)] &= \\[2mm]
\text{Gain}(Y, X_3) &=
\end{align*}

\item[(b)] We see that two of the leaves of our tree are ``pure'', meaning that all of the training examples that arrive there are of one outcome class. For those two leaves, we're done. However, for the third ($X_2 = 0$, or poor), we need to perform another split. Perform the split at the $X_2=0$ node to find which variable to split on next and update the tree with this information.

\begin{align*}
H[Y(X_2 = 0)] &= \\[2mm]
\frac{|Y(X_2 = 0, X_1 = 0)|}{|Y(X_2=0)|} H[Y(X_2 = 0, X_1 = 0)] &= \hspace{0.5\textwidth}\\[2mm]
\frac{|Y(X_2 = 0, X_1 = 1)|}{|Y(X_2=0)|} H[Y(X_2 = 0, X_1 = 1)] &= \hspace{0.5\textwidth}\\[2mm]
\text{Gain}(Y(X_2=0), X_1) &= \\[2mm]
\frac{|Y(X_2 = 0, X_3 = 0)|}{|Y(X_2=0)|} H[Y(X_2 = 0, X_3 = 0)] &= \hspace{0.5\textwidth}\\[2mm]
\frac{|Y(X_2 = 0, X_3 = 1)|}{|Y(X_2=0)|} H[Y(X_2 = 0, X_3 = 1)] &= \hspace{0.5\textwidth}\\[2mm]
\text{Gain}(Y(X_2=0), X_3) &= \\[2mm]
\end{align*}

\item[(c)] We need to do one more split on the $X_2 = 0, X_3 = 0$ node. The only variable left to split on is $X_1$ (friends). Perform this split and add this information to the tree.
\end{enumerate}
\end{question}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Alternative Splitting Criteria}

Information gain is not the only criterion that is used in decision tree algorithms. In fact, the tree built for the Wisconsin Breast Cancer dataset in Section~\ref{section:wisconsin} used a different criterion, the mean decrease in Gini impurity, or \textbf{Gini index}, because it is the default in R's \texttt{rpart} package.

The Gini impurity measures how often a randomly chosen element from a set would be incorrectly labeled if it were randomly labeled using the distribution of labels in the subset. It sums up the probability of an item with label $i$ being chosen ($p_i$) multiplied by the probability $\sum_{j \neq i} p_j = 1 - p_i$ that a mistake is made in classifying it. The formula is:
$$ G(p) = \sum_{i} p_i (1 - p_i) = 1 - \sum_i p_i^2 $$
which for a Bernoulli distribution is
$$ G_\text{Bernoulli} = 1 - \mu^2 - (1-\mu)^2. $$
The Gini impurity plays a role analogous to entropy. The Gini index is a weighted sum of the Gini impurities within the children of a particular split; if it is low, it means the split was successful in reducing impurity.

\vspace{3mm}

\begin{question}{}
Plots of the Gini impurity vs. entropy for a Bernoulli distribution are shown below. What do you notice about the value(s) of $\mu$ for which each is maximized or minimized?
\begin{center}
\includegraphics[width=0.8\textwidth]{img/comparison-entropy-gini-bernoulli.png}
\end{center}
\end{question}

Entropy and Gini impurity are similar and usually, though not always, yield similar trees. The Gini impurity is computationally faster because it does not make use of logarithms as the entropy does. This can make a difference as the number of features grows.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Decision Tree Regression}

So far we've assumed that our outcome is discrete. But what happens if it's numeric? (That is, what if we want to perform regression instead of classification?)

In that case, we use \textbf{standard deviation reduction} instead of information gain to decide which variables to split on. The sample standard deviation of an outcome, $y$, is defined as:
$$ S(Y) = \sqrt{\frac{\sum_i(y^{(i)} - \overline{y})}{n-1}} $$
The procedure is identical to the ID3 algorithm except that it uses conditional standard deviation instead of information gain to decide on features. We define
$$ S(Y, X) = \sum_{x} P(X = x)~ S(Y(X=x)) $$
and at each current leaf node, we split on the variable where the reduction in standard deviation, $S(Y) - S(Y,X)$, is the highest.
\vspace{2mm}

\begin{question}{}
Imagine you have a dataset with two predictors, $x_1$ and $x_2$, each of which is binary (can only be 0 or 1). Here are the distributions of outcome values associated with $x_1$ and $x_2$:
\begin{center}
\includegraphics[width=0.7\textwidth]{img/esl-reg-decision-tree-varsplit.png}
\end{center}
Based on the idea of standard deviation reduction, which of these two variables, $x_1$ or $x_2$, would make the most sense for a decision tree to split on? What would such a split look like and what would the output value of the tree (the predicted value of $y$) be for each side of the split?
\end{question}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Numeric Predictors}

Although the Happiness Dataset contained only discrete predictors, decision trees can also handle numeric predictors. We've seen this already with the Wisconsin Breast Cancer dataset.

There are different strategies for deciding on an optimal split for a predictor. The most common approach is to consider all possible splits. So for example, if a predictor has values $10, 11, 16, 18, 20$, and $35$, the tree building algorithm would consider all $N-1 = 5$ possible split points. If a dataset has a large number of numeric features or features with lots of different possible values, therefore, it can really slow down the construction of the tree.
\vspace{5mm}

\begin{question}{}
Here are histograms of $10$ of the predictors in the Wisconsin Breast Cancer dataset. Only the ``worst'' variable version of each predictor is shown for clarity. Which variable, and which threshold, appears to show the clearest division of samples into $B$ and $M$ groups? Compare your choice to the first split of the tree in Section~\ref{section:wisconsin}.
\begin{center}
\includegraphics[width=\textwidth]{img/wisconsin-variable-plots.png}
\end{center}
\end{question}


%%%%%%%%%%%%%%%%%%%%%%%%% preamble %%%%%%%%%%%%%%%%%%
\documentclass{article}
\usepackage[utf8]{inputenc}  % input encoding
\usepackage{graphicx}  % to insert figures
\usepackage{amsmath,amsfonts}  %to insert complex equations

\usepackage{natbib}  % bibliography package
\bibliographystyle{dinat}  % bibliography style

\usepackage{hyperref}  %

\usepackage[english]{babel}  % to insert text in English
\usepackage{blindtext}  % to insert sample text

\title{Intro to \LaTeX}
\author{Dr Patricia Ternes\thanks{p.ternesdallagnollo@leeds.ac.uk}}
\date{\today}
%%%%%%%%%%%%%%%%%%%%%%%%% preamble %%%%%%%%%%%%%%%%%%

\begin{document}
\maketitle
\begin{abstract}
    \blindtext[1]
\end{abstract}

\tableofcontents
\listoftables
\listoffigures

%%%%%%%%%%%%%%%%%%%%%%%%% sessions and subsections! %%%%%%%%%%%%%%%%%%
\section{Introduction}
\blindtext[1]
\subsection{My first subsection}
\blindtext[1]
\subsection{Another subsection}
\blindtext[1]
\subsection{Another subsection}
\blindtext[1]
\subsubsection{Sub sub session}
\blindtext[1]
\paragraph{Is it works?}
\blindtext[1]

%%%%%%%%%%%%%%%%%%%%%%%%% Lists! %%%%%%%%%%%%%%%%%%
\section{Lists}
\subsection{Unordered List}
\begin{itemize}
    \item red
    \item blue
    \item green
\end{itemize}

\blindtext[1]

\subsection{Description List}
\begin{description}
    \item[apple] red
    \item[sky] blue
    \item[leaf] green
    \item yellow
\end{description}

\blindtext[1]

\subsection{Ordered List}
\begin{enumerate}
    \item green
    \item red
    \item yellow
    \item blue
\end{enumerate}

\blindtext[1]

\subsection{Nested list}
\begin{enumerate}
    \item green
    \begin{itemize}
        \item leaf
    \end{itemize}
    \item red
    \begin{itemize}
        \item apple
    \end{itemize}
    \item yellow
    \item blue
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%% Figures! %%%%%%%%%%%%%%%%%%
\section{Figure}\label{sec:fig}
Figure \ref{fig:sample-fig} shows a sample figure from the graficx package.
\blindtext[1]
\begin{figure}[h]
    \centering
    \includegraphics[width=0.4\textwidth]{example-grid-100x100bp}
    \caption{Sample figure from graphicx package.}
    \label{fig:sample-fig}  % used to reference the figure in the text
\end{figure}

Figure \ref{fig:uni-pic} shows a picture of University of Leeds. This picture is part of the Session \ref{sec:fig}.
\blindtext[1]
\begin{figure}[h]  % [h] = try to put float "here"
    \centering
    \includegraphics[width=0.4\textwidth]{uni.jpg}
    \caption{University of Leeds picture.}
    \label{fig:uni-pic}  % used to reference the figure in the text
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%% Table! %%%%%%%%%%%%%%%%%%
\section{Table}\label{sec:tab}
\blindtext[1] See Table \ref{tab:col_fruit} in Session \ref{sec:tab} to see a fruit of each colour.
\begin{table}[h]
    \centering
    \begin{tabular}{|c||c|}\hline  % "\hline" = drawn a horizontal line
        Colour & Fruit     \\ \hline \hline % "&" = change column, "\\" = line break
        Red    & Apple     \\
        Yellow & Banana    \\
        Green  & Grape     \\
        Blue   & blueberry \\ \hline
    \end{tabular}
    \caption{Table of colours and fruits.}
    \label{tab:col_fruit}
\end{table}

\blindtext[1]

%%%%%%%%%%%%%%%%%%%%%%%%% Math! %%%%%%%%%%%%%%%%%%
\section{Math}\label{sec:math}
You can insert math inline by using \$ delimiters, like: $y = \alpha + \beta x$.

You can add the equation in a new paragraph:
\[
\text{May the } \frac{d p}{d t} \text{ be with you!}
\]

Or you can add the equation in an ordered math environment:

According \cite{newton1850}, the force is defined as:
\begin{equation}
    F = ma = m\frac{dv}{dt} = \frac{dp}{dt},
\end{equation}
where $p$ is the momentum.

It is also possible combine the equation environment with an array environment:
\begin{equation}
    A = \left(
        \begin{array}{ccc}
            a_{11} & a_{12} & a_{13} \\
            a_{21} & a_{22} & a_{23} \\
            a_{31} & a_{32} & a_{33} \\
        \end{array}
    \right)
\end{equation}

\bibliography{sample}

\end{document}


\documentclass[12pt]{article}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Margins and Heading and Such
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\usepackage{amsmath}
\usepackage{amssymb}

\setlength{\oddsidemargin}{.25in}
\setlength{\evensidemargin}{.25in}
\setlength{\textwidth}{6.25in}
\setlength{\topmargin}{-0.4in}
\setlength{\textheight}{8.5in}

\newcommand{\heading}[5]{
   \renewcommand{\thepage}{#1-\arabic{page}}
   \noindent
   \begin{center}
   \framebox{
      \vbox{
    \hbox to 5.78in { {\bf 6.875/18.425 Cryptography and Cryptanalysis}
         \hfill #2 }
       \vspace{4mm}
       \hbox to 5.78in { {\Large \hfill #5  \hfill} }
       \vspace{2mm}
       \hbox to 5.78in { {\it #3 \hfill #4} }
      }
   }
   \end{center}
   \vspace*{4mm}
}

\setlength{\parindent}{0in}
\setlength{\parskip}{0.1in}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% MODIFY THE FOLLOWING LINE  WITH YOUR NAME

\newcommand{\handout}[3]{\heading{#1}{#2}{Your Name Goes Here}{}{#3}}

\begin{document}
\handout{5}{February 16, 2005}{Problem Set 1}

\textbf{Collaborators:} I collaborated with the following students on
this assignment: <Insert Names Here>

\subsection*{Problem 1}

Suppose $p$ is a prime and $g$ and $h$ are both generators of
$Z_p^*$. Prove or disprove the following statements about equality
of probability distributions:

\begin{eqnarray*}
\textbf{A:} & \ \ \ \ \ &
\{x \leftarrow Z_p^*: g^x \mod p\} = \{x \leftarrow Z_p^*; y \leftarrow Z_p^*: g^{xy} \mod p\} \\
\textbf{B:} & \ \ \ \ \ &
\{x \leftarrow Z_p^*: g^x \mod p\} = \{x \leftarrow Z_p^*: h^{x} \mod p\} \\
\textbf{C:} & \ \ \ \ \ &
\{x \leftarrow Z_p^*: g^x \mod p\} = \{x \leftarrow Z_p^*: x^g \mod p\} \\
\textbf{D:} & \ \ \ \ \ &
\{x \leftarrow Z_p^*: x^g \mod p\} = \{x \leftarrow Z_p^*: x^{gh} \mod p\} \\
\end{eqnarray*}

\textbf{Solution: } <Your Solution Goes Here>

\newpage

\subsection*{Problem 2}
Suppose that the Prime Discrete Logarithm Problem is easy. That is,
suppose that there exists a probabilistic, polynomial time algorithm
$A$ that, on inputs $p$, $g$ and $g^x \mod p$,  outputs $x$ if $p$
is a prime, $g$ is a generator of $Z_p^*$ and $g^x \mod p$ is prime.
Show that there exists a probabilistic polynomial-time algorithm,
$B$, that solves the Discrete Logarithm Problem.

\textbf{Solution: } <Your Solution Goes Here>

\newpage

\subsection*{Problem 3 }

We define the Lily problem as: given two integers $n$ and $S$
determine whether $S$ is relatively prime to $\phi(n)$. Prove that
if it is hard to determine on inputs two integers $n$ and $e$
whether $e$ is relatively prime with $\phi(n)$, then  the RSA
function is hard to invert.

\textbf{Solution: } <Your Solution Goes Here>

\newpage

\subsection*{Problem 4: Factoring }

Let $O_n$ be an oracle that on input $x$ returns a square root of $x
\mod n$, if one exists, and $\bot$ otherwise. Prove that there
exists a probabilistic polynomial-time algorithm that on input an
integer $n$ and access to  $O_n$ outputs $n$'s factorization.

\textbf{Solution: } <Your Solution Goes Here>

\newpage

\subsection*{Problem 5: Factoring and OWF (OPTIONAL)}

Prove that if factoring is hard, then one-way
  functions (as defined in class) exist.

\textbf{Solution: } <Your Solution Goes Here>

\end{document}


%!TEX root = ../main.tex

\chapter{Discussion}
\label{ch:disc}

\lhead{Discussion}

Over the course of this thesis, we have described a collection of tools aimed
at facilitating the computational analysis of RNA. The work in \Chref{rfinder}
aims to address the open problem of determining full \rb sequences, along
with their `on' and `off' conformational states in an efficient and generalized
manner. \Rbs are an imporant regulatory motif that can influence
transcription by the introduction of an intrinsic terminator, an extended stem-loop
followed by a series of uracil residues which cause RNA polymerase to prematurely
terminate transcription \citep{gusarov:1999uu,yarnell:1999wt}. Alternately,
\rbs can regulate translation, generally by sequestration of the ribosome
binding site in response to the presence or absence of a corresponding ligand
\citep{barrick:2007gw}. This degree of control makes \rbs well suited for
the regulation of metabolic pathways, and are generally found within bacterial
genomes. While the aptamer portion of the \rb necessarily exhibits high
sequence and structural conservation---necessary to retain a high binding affinity
for the corresponding ligand---the downstream expression platform is much more
variable, and difficult to detect using covariance models such as \infernal
\citep{infernal}. For this reason, databases such as Rfam \citep{nawrocki:2014uy}
only contain alignments for the conserved aptamer domain; rarely enough to
understand the structural characteristics of the \rb `states' in the greater
context of the mRNA. \rfinder aims to provide a holistic description of
\rbs by coupling powerful aptamer-detection tools (such as \infernal) with software
designed to detect intrinsic terminators (such as \tthp), and use
experimentally-derived structural information from the literature in conjunction
with constrained folding methods to predict full gene `on' and `off' conformations
in a general manner.

The remainder of the thesis follows a natural progression. In \Chref{fftbor} we
present the algorithm \fftbor, an efficient approach to compute the
{\em parameterized} Boltzmann probabilities $\pk = \frac{\bfZ{k}{}}{\fullZ}$ of
those sequences having \bpd $k$ from an arbitrary input structure
\strSt. By using the \fft along with \nRoUs, \fftbor operates in \On{4} speed,
an order of magnitude faster than its predecessor. This approach of applying the
\fft to compute parameterized partition functions was put to use in
\citep{ding:2014ex} as well, where we develop programs \ms{FFThairpin}
[resp. \msresp{FFTmultiloop}] to describe the probability distribution of structures
having exactly $k$ hairpins [resp. multiloops]. Though not discussed here, by
training a support vector machine (SVM) on the probability distributions output
by these
programs we were able to classify Rfam families using five-fold cross validation
with excellent `area under curve' (AUC) values.

\Chref{ffttwo} uses the \fft to compute the \twoD probability distribution, where
given two compatible input secondary structures \strA and \strB, position
$(x,y)$ on the discrete energy landscape corresponds to the Boltzmann
probability for those structures \str which have \bpd $x$ [resp. $y$]
from \strA [resp. \strBresp]. Compared against its closest relative \rnatwofold,
\ffttwo is two orders of magnitude faster, operating in \On{5} speed and only
\On{2} space. This opens the door for using \ffttwo in a high-throughput fashion,
where the probability landscapes can be used as a starting point for kinetic
analysis.

With \hermes in \Chref{hermes}, we do just that; representing the output of
\ffttwo as a graph, we can create a corresponding Markov chain [resp. Markov
process] to estimate \mfpt [resp. equilibrium time] with correlation to the
mathematically-exact values as shown in Table \ref{table:correlationHermes}.
The speed of these approximate tools within \hermes---namely \fftmfpt and
\ffteq---present what we believe to be the first suite of kinetic analysis
tools for RNA sequences that are suitable for high throughput usage, something
we believe to be of interest in the field of synthetic design.

Finally, though not discussed in the body of this thesis, we have also
developed a collection of packages for working with command-line tools for
computational RNA analysis in a more streamlined fashion. Two of these programs,
\href{https://github.com/evansenter/wrnap}{\wrnap} and
\href{https://github.com/evansenter/rbfam}{\rbfam} are among the first of their
type for the Ruby programming langage, a popular interpreted language that
enjoys wide adoption in the web development community. In an era of increasingly
cloud-focused software, we believe that researchers will need a robust set of
tools for working with their data in a web-oriented environment, something
\wrnap and \rbfam aim to provide.


\documentclass[10pt,letterpaper]{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage[margin=1in]{geometry}
\usepackage{graphicx}
\usepackage{sidecap}
\usepackage{float}
\title{\vspace{-4ex}Regression Variations\vspace{-3.5ex}}
\begin{document}
\newgeometry{top=0.75in,left=1in,right=1in,bottom=1.25in}
\maketitle
\vspace{-0.5em}
\begin{abstract}
A key feature of most machine learning models is their ability to learn and predict effectively; i.e. based on some given data points $X, Y$ that were generated non-randomly (with error given by a distribution), a well-formed model should be able to predict some reasonable pattern. To do this, we will often need to estimate some parameter $\theta$ that assigns weights to terms in our predicted function. The goal, then, is to choose the $\theta$ that leads to the minimum value of some error or loss function, and in this way minimize the error. This can be done in several ways, and a more formal and thorough treatment of this topic will be the subject of this paper.\\
\end{abstract}

\section{Gradient Descent}
First, we discuss the use of a tool that we will use often in calculating the minimum - gradient descent. We investigate several variations on its implementation, including analytic gradient descent, finite difference gradient descent, and Matlab's \texttt{fminunc}, which includes some variations such as adaptive step size, using a quadratic approximation instead of linear approximation, and using a 2-dimensional subspace to reduce computational complexity. First, we'll examine the analytic gradient descent, which relies on having an analytic expression for the gradient.

We'll refer to three example functions: the $n$-dimensional quadratic ``bowl'', $Q_n$, the $n$-dimensional inverted Gaussian (centered at $\mu = 0$, with $\Sigma = \mathbf I_n$), $N_n$, and the $n$-dimensional sum of sines, $S_n$. These are defined as follows (leaving off the normalization constant on the inverted Gaussian for simplicity):

\[ Q_n = \left\| \mathbf x \right\| ^2 \]

\[ N_n = -\exp{\left(-\dfrac{1}{2} \left\| \mathbf x \right\| ^2 \right)} \]

\[ S_n = \sum_{i=1}^n \sin(x_n) \]
Next, we benchmark the gradient descent variations on these two functions (seeding with an initial guess of $(1,1,\ldots)$):

\begin{table}[h]
\centering
\caption{Iterations (step = .1, threshold = .001)}
\begin{tabular}{r|ccc|ccc|ccc}
   & \multicolumn{3}{|c}{Analytic Gradient} & \multicolumn{3}{|c}{Finite Differences} & \multicolumn{3}{|c}{\texttt{fminunc}} \\
$n$& $Q_n$         & $N_n$        & $S_n$   & $Q_n$              & $N_n$   & $S_n$           & $Q_n$         & $N_n$        & $S_n$ \\\hline
2  & 16            & 33           & 46      & 16                 & 33      & 46              & 6             & 15           & 18    \\
5  & 18            & 60           & 50      & 18                 & 60      & 50              & 12            & 48           & 36    \\
20 & 21            &  1           & 57      & 21                 &  1      & 57              & 42            & 21           & 126   \\
100& 25            &  1           & 64      & 25                 &  1      & 64              & 202           & 101          & 606
\end{tabular}
\end{table}

Focusing on just $S_n$, and leaving the step size, starting point, and threshold the same unless otherwise noted, we may also examine the effects of each variable (defaulting to $n=5$) in table 2.

\begin{table}[h!]
\centering
\caption{Examining the effect of other variables}
\begin{tabular}{cc|cc|cc}
\multicolumn{2}{c}{Step Size}& \multicolumn{2}{|c}{Starting Point} & \multicolumn{2}{|c}{Threshold} \\
$\Delta$  & Iterations        & $x_0$   & Iterations   & $\delta$  & Iterations     \\\hline
2.0       & 177               & 0.001   & 49           & 0.01      & 39     \\
1.0       & 5                 & 0.01    & 49           & 0.001     & 50     \\
0.1       & 61                & 0.1     & 50           & 0.0001    & 61     \\
0.01      & 503               & 1.0     & 61           & 0.00001   & 72
\end{tabular}
\end{table}

We notice that the finite differences and analytic methods yield the exact same number of iterations for all functions. This can be explained by examining the diagonal of the Hessian matrix for each of the functions - in all cases, the values are small, indicating low curvature, and thus that the function can be well-approximated by the linear finite difference approximation. Therefore, when examining the effects of other variables, we only present the numbers using the finite-difference gradient (the analytic gradient produces identical numbers).

We may draw a few conclusions by examining the effects of other variables. As we decrease the step size, it takes longer to converge. However, increasing the step size beyond 1 causes the iteration to overshoot, resulting in an overall higher number of required iterations. Moving the starting point further from the minimum increases the number of iterations, but only slightly, because the gradient increases, which increases the step size proportionally. Finally, requiring a stricter convergence criteria via a smaller threshold increases the number of iterations, but only by a constant amount, because the gradient again ensures the step size is in proportion to the distance from the minimum.

By plotting the difference between the finite differences approximation, we can see the accuracy achieved. Here, we again use $\delta = .01$. Figures \ref{fig:gradDifQ} through \ref{fig:gradDifS} are plots of $(\nabla f(\mathbf{x}) - \tilde{\nabla} f(\mathbf{x}))/f(\mathbf{x})$ for $x=(0,1)$.

\begin{figure}[!htb]
\minipage{0.32\textwidth}
  \includegraphics[width=\linewidth]{figures/gradDifQ.png}
  \caption{$Q_1$, scale of $10^{-12}$}\label{fig:gradDifQ}
\endminipage\hfill
\minipage{0.32\textwidth}
  \includegraphics[width=\linewidth]{figures/gradDifN.png}
  \caption{$N_1$, scale of $10^{-5}$}\label{fig:gradDifN}
\endminipage\hfill
\minipage{0.32\textwidth}
  \includegraphics[width=\linewidth]{figures/gradDifS.png}
  \caption{$S_1$, scale of $10^{-6}$}\label{fig:gradDifS}
\endminipage
\end{figure}

Given the scales of the plots, it is clear that the finite differences method is quite accurate for these functions over these distance scales. As noted, this is generally true when the diagonal entries of the Hessian matrix are small.

\section{Some Simple Regression}
Now that we've established gradient descent as a reliable tool to help us calculate minima, we can benchmark it by applying it directly through performing basis function regression on some given sample data points. Given, say, $10$ data points $X = x^{(1:10)}$ that are generated from a noisy distribution around $y = \sin(2\pi x)$, we'll try to estimate the generating function based on (a) the values at the data points $Y = y^{(1:10)}$ and (b) a basis set of functions.

Our ultimate goal is the minimize the sum of the squared errors of each of our proposed predictions, or $\text{SSE}(\theta)$, which is equal to
\[ \text{SSE}(\theta) = \sum_{i=1}^n (\theta^\intercal\phi(x^{(i)}) - y^{(i)})^2 \]

We can begin by considering a simple polynomial basis; i.e. $\phi(x) = [1, x, x^2, \ldots , x^M]$ where $M$ is the degree of the polynomial we are considering. We wish to minimize our parameter $\theta$, which has a well-known analytic solution:
\[ \theta = (\Phi^\intercal\Phi)^{-1}\Phi^\intercal y \]
given that $\Phi = \phi(x)$ generally, and specifically
\[ \Phi = \left( \begin{array}{cccc}
 1 &  x^{(1)} & \ldots & (x^{(1)})^M \\
 \vdots & \vdots & & \vdots \\
 1 &  x^{(n)} & \ldots & (x^{(n)})^M \end{array} \right) \]
for this polynomial basis. We can easily test this by plotting our data $(X, Y)$ with our learned model $\theta^\intercal\phi(X)$ for varying values of $M$, shown in figures \ref{fig:basism1} through \ref{fig:basism9}.

\begin{figure}[h]
\minipage{0.33\textwidth}
  \includegraphics[width=\linewidth]{figures/basism1.png}
  \caption{$M = 1$}\label{fig:basism1}
\endminipage\hfill
\minipage{0.33\textwidth}
  \includegraphics[width=\linewidth]{figures/basism3.png}
  \caption{$M = 3$}\label{fig:basism3}
\endminipage\hfill
\minipage{0.33\textwidth}
  \includegraphics[width=\linewidth]{figures/basism9.png}
  \caption{$M = 9$}\label{fig:basism9}
\endminipage
\end{figure}
We note that we can't have $M > 9$, or else we have too many parameters to adjust for the data points that we have. The reason the formula breaks down is that $\Phi^\intercal\Phi$ is no longer full rank, which means we can no longer simply take the inverse to find $\theta$. In other words, there are now many (in fact, infinitely many) sets of parameters that would perfectly fit the criteria. We can adjust for this using the \textit{Moore-Penrose inverse} which pulls the $\theta$ from the list with the smallest norm. We will not address this in detail here.

From figure \ref{fig:basism9}, it's also clear that when $M$ is too high, we begin to overfit our data, even though the training error here is $0$. We will come back to this problem later, when we deal with regularization.

At the moment, however, we can generalize our basis functions to more than simply polynomials in $x$. In this way, we can introduce trigonometric bases, exponential bases, etc. to give us more flexibility in choosing the parameter $\theta$.
We can minimize this by using gradient descent, which we discussed earlier. We can then confirm our calculations analytically by computing the derivative of $\text{SSE}(\theta)$:
\[ \frac{\partial}{\partial\theta}\Big(\sum_{i=1}^n (\theta^\intercal\phi(x^{(i)}) - y^{(i)})^2\Big) = 2\sum_{i=1}^n \phi(x^{(i)})(\theta^\intercal\phi(x^{(i)}) - y^{(i)}) \]
Plugging in our optimal $\theta$ into this formula and calculating the numerical gradient with a $\delta$ of 0.00001, we obtain the following results for our derivative, confirming the optimality of $\theta$:
\begin{table}[h]
\centering
\caption{Examining the the numerical vs. analytic derivatives}
\begin{tabular}{c|c|c}
$M$& $\lVert\nabla\text{SSE}(\theta)\rVert$ (numerical) & $d\text{SSE} (\theta)/d\theta$ (analytic)\\\hline
0       &  $2.1067\cdot10^{-12}$  & $1.6653\cdot10^{-16}$   \\
1       & $2.1165\cdot10^{-12}$  & $1.0335\cdot10^{-14}$ \\
3       & $1.7099\cdot10^{-11}$ & $7.9814\cdot10^{-10}$  \\
9      &   $7.9814\cdot 10^{-10}$    & $-1.6414\cdot 10^{-07}$
\end{tabular}
\end{table}

Optimizing the SSE via gradient descent reveals a few interesting characteristics of gradient descent, especially as compared to more sophisticated optimizers such as Matlab's \texttt{fminunc}. Firstly, if our initial guess is worse, we will require more iterations to reach our final optimization. This is mitigated slightly by scaling the step size by the gradient, but this only really works when the gradient is well-behaved. Similarly, tightening the convergence threshold increases the number of required iterations. This is also discussed further at the end of Section 1, which discusses the effect of the Hessian matrix on the required number of iterations.

In addition, it is tempting to fit this data set by using $\phi_1(x) = \sin(2\pi x), \ldots , \phi_M(2\pi Mx)$ as our basis set of functions. If we use these, our plots in figures \ref{fig:sin1} to \ref{fig:sin6} look much closer to the original function. However, if we didn't already know anything about the data, guessing a basis of only sinusoidal functions comes with a serious disadvantage. It guarantees that all our generated $\theta^\intercal\phi(x)$ will be periodic in nature, when the actual data/observations really aren't. This can cause unnecessarily varying, and potentially completely incorrect, data to be predicted.\\
\begin{figure}[h!]
\minipage{0.33\textwidth}
  \includegraphics[width=\linewidth]{figures/sin1.png}
  \caption{$M = 1$}\label{fig:sin1}
\endminipage\hfill
\minipage{0.33\textwidth}
  \includegraphics[width=\linewidth]{figures/sin3.png}
  \caption{$M = 3$}\label{fig:sin3}
\endminipage\hfill
\minipage{0.33\textwidth}
  \includegraphics[width=\linewidth]{figures/sin6.png}
  \caption{$M = 6$}\label{fig:sin6}
\endminipage\hfill
\end{figure}

\section{Ridge Regression}

Recall that earlier, we had the problem that having too high of an $M$ could cause overfitting to the training data. Now we will examine ridge regression, which is a form of regression that attempts to minimize the coefficients of $\Theta$; this will reduce the magnitudes of the coefficients of a high-order polynomial to more reasonable sizes. It does this by imposing a penalty, $\lambda$, on the Euclidean (squared) norm of the coefficients of $\Theta$:
$$\theta_{\text{ridge}} = \text{argmin}_\theta\sum_{i=0}^n(y^{(i)} - \theta^\intercal\phi(x^{(i)}))^2 + \lambda\lVert\theta\rVert^2$$
The fact that we impose a penalty on the Euclidean norm is a direct result of considering the noise as a Gaussian. Considering the noise as a Laplacian distribution leads to imposing a penalty on the absolute value norm, while other noise distributions lead to yet more functions of $\theta$ to apply the penalty to. The first term is simply $\text{SSE}(\theta)$, so it isn't surprising that the analytic solution to this minimization is similar to that of just $\text{SSE}(\theta)$; it's given by
\[ \theta = (\Phi^\intercal \Phi + \lambda \mathbf{I})^{-1}\Phi^\intercal y \]
where $\Phi = \phi(X)$. We then have the predictor function given by $\phi$ weighted by $\theta$, eg, we would predict $y_{predicted} = \theta^\intercal \phi(x')$.
\begin{figure}[!htb]
\minipage{0.49\textwidth}
  \includegraphics[width=\linewidth]{figures/M3.png}
  \caption{$M=3$}\label{fig:M3}
\endminipage\hfill
\minipage{0.49\textwidth}
  \includegraphics[width=\linewidth]{figures/M7.png}
  \caption{$M=7$}\label{fig:M7}
\endminipage
\end{figure}
We can see the effect $\lambda$ has on the result by examining a few values of $\lambda$ for the same dataset used previously (generated from $\sin(2\pi x)$).

The effect of $\lambda$ is clear in Figures \ref{fig:M3} and \ref{fig:M7}, where higher values of $\lambda$ lead to flatter, smoother approximations. As $M$ increases (especially relative to $n$), the values in $\theta$ begin rising enormously, but adding the $\lambda$ factor helps to keep them under control. For example, in this example with $M=7$, $\max \theta_i = 1595.9$ for $\lambda = 0$, but for $\lambda = .01$, $\max \theta_i = 1.7781$, a much more reasonable value, and far more likely to approximate the truth accurately. In particular, Figure 1.8 in Bishop nicely summarizes the assistance $\lambda$ gives in generalizing $\theta$.

\begin{figure}[!h]
\begin{center}
\includegraphics[width=4in]{figures/blogData.png}
\caption{Effect of $\lambda$ on training and test errors.}\label{fig:blogData}
\end{center}
\end{figure}
Applying ridge regression to a real-world data set, we can also observe the effects of $\lambda$. Here, we apply ridge regression to a dataset regarding blogs, where the $x$ vector contains 280 metrics for each blog post, and $y$ is the number of comments in the first 24 hours. We can see the effect $\lambda$ has on the training error and testing error in figure \ref{fig:blogData}. From this, we conclude that $\lambda=10^8$ is the approximately optimal value of $\lambda$ for this problem.\\

\section{LAD}
All this time, in applying various regression techniques, we have been assuming the use of our convenient $\text{SSE}(\theta)$ function to minimize our $\theta$ (with the possible addition of a regularization term, e.g. $\lambda\lVert\theta\rVert^2$ in ridge regression). We see that a squared regularization term comes naturally in the context of Gaussians; if instead we used a linear loss based on the non-square norm (using $\lambda\lVert\theta\rVert$), we are pulling from a Laplace distribution - this regularization is commonly termed LASSO.

However, in our case, we'll leave the regularization term alone and instead investigate changes in our $\text{SSE}(\theta)$ term. Here we can use a linear model instead of a quadratic one, so that we sum our absolute errors to obtain our residual error. This method is called ``LAD'' (least absolute deviations):

$$\theta_{\text{LAD}} = \text{argmin}_\theta\sum_{i=0}^n|y^{(i)} - \theta^\intercal\phi(x^{(i)})| + \lambda\lVert\theta\rVert^2$$

Note that we are no longer minimizing anything similar to $\text{SSE}(\theta)$. As this problem has no simple analytical solution, we can again make use of gradient descent, this time to solve for our parameter $\theta$. After attempting various values for $\lambda$ and $M$, we arrive at $\lambda = 1$, $M=1$ as our optimal solution for the \texttt{regress} data set, as seen in Figure \ref{fig:line}.

\begin{figure}[!htb]
\begin{center}
\includegraphics[width=3in]{figures/line.png}
\caption{Our predictor at the optimal value of $\theta$, calculated at $\lambda=1$, $M=1$, on the \texttt{regress} training data set.}\label{fig:line}
\end{center}
\end{figure}

\begin{figure}[!htb]
\begin{center}
\includegraphics[width=3in]{figures/lambdaline.png}
\caption{Effect of $\lambda$, where the x-axis is $\log \lambda$ while the y-axis is $\log E_{test}$}\label{fig:lambdaline}
\end{center}
\end{figure}

Furthermore, we can see the effect of $\lambda$ by examining Figure \ref{fig:lambdaline}. This is how we observe that $\lambda=0$ leads to the lowest testing error.

The results are expected; in general, we find that least absolute deviations is more robust than least squares regression, as it is less susceptible to outliers in the dataset. This is an especially important point - if we don't wish for our model to give heavier weight to outliers, then LAD is the better choice.

Ridge regression, however, produces results that are more stable, with the derivative falling off more slowly and the solution being unique. It has the advantage of having a simple, standard and well-known analytic solution that is more easily implemented. In addition, if we want outliers to have more effect on our model, then we should choose ridge regression. The choice of regression function is entirely up to the characteristics of the data we would like to model.

\end{document}


\documentclass{tufte-handout}
\usepackage{amsmath}
\usepackage[utf8]{inputenc}
\usepackage{mathpazo}
\usepackage{graphicx}
\usepackage{listings}

\usepackage{tikz}
\usetikzlibrary{chains}
\usetikzlibrary{decorations}
\input{vc.tex}

\title{Four-Sum}
\author{Thore Husfeldt and Riko Jacob}
\date{\GITAuthorDate, rev. \GITAbrHash}
\begin{document}
\maketitle
\begin{abstract}
  Implement a very simple algorithm in two different languages, run some experiments.
  Improve the algorithm.
\end{abstract}

 \section{Description}

 Given a list of $N$ (long) integers, detect if there are four of them that sum to $0$.

\begin{description}
  \item[Exhaustive search.]
The simplest solution is to use exhaustive search using four nested for loops.
Go ahead and implement that in the language that you’re comfortable in, Python or Java.
There is a code skeleton called \texttt{sol/Simple.java} or \texttt{sol/simple.py} that you must use.
The program expects a list of $N$ integers on standard input, following the number $N$ itself.
The directory \texttt{data} contains several of such lists for various $N$.

The program must write \texttt{True} to standard output if it finds a solution, and \texttt{False} otherwise;
your program may write other output to standard error (\texttt{stderr}).
This is the basis of the tests on codeJudge, and all your programs must at least pass all of \texttt{TestsTiny} and \texttt{TestsSmall}.

Make it work; and determine the running time on your machine on the input files.
There are a handful of inputs for each size.
Report the fastest observed time, the slowest, and the average.
(Try to be half-way serious about this--- \emph{e.g.}, restart the computer before the experiment, so as to make sure no spurious processes are running in the background, don’t have a World of Warcraft-server or Bitcoin mining programme running at the same time.)
You probably won’t be able to handle the larger inputs---don’t report experiments that take more than a few minutes.
Plot the result using both a standard and a log-log plot.

\item[Exhaustive search in a different language.]
  Do the same as above, but in the other language.
  This may involve installing compiler or a run-time environment, figuring out where the semicolons or brackets go (or don’t go) and other annoyances.
  Compare the running times of the two implementations of exhaustive search in the different languages.
  This is the only task you are to implement in both languages.

\item[Faster.]
  Use the idea of the fast three-sum algorithm in [SW, Chapter 1.4] \texttt{
  ThreeSumFast} to write a faster algorithm for the four-sum problem.
  Your running time must be close to cubic time, proportional to $N^3\operatorname{lg} N$.
  Call it \texttt{Faster.java}  or \texttt{faster.py}.
  Redo the experiments and report them.

  \emph{Optional} There is a cute, simple, and very satisfying way to solve the problem in time proportional to $N^2\operatorname{lg} N$ instead.
If you can figure that trick out, implement it and hand in that code instead.
\end{description}

\section{Tips and comments}

\begin{description}
  \item[Java types.] The integers in the input files are very large, and will not fit into the Java datatypes \texttt{int} or \texttt{Integer}.
    Use \texttt{Long} instead.
    Do not worry about overflow.
  \item[Python dialect.] You can use Python 2 (which may be already installed on your computer) or Python 3 (remember that the library of the course is written in python3).
  \item[Course libraries.]
    In this exercise (and this exercise \emph{only}) you are \emph{not} required to use the course libraries, say for input/output.
    This is because it is too much overhead to ask students in language $x$ to install the libraries for language $y$.
    Therefore, this particular exercise is very bare-bones.
    You are welcome to use the course library if you want to.
  \item[Measuring time.]
    On a unix system, the \texttt{time} command gives sufficiently good performance estimates, and is better than sitting with a stopwatch.
    Both Java and Python come with better solutions; the course libraries offer a \texttt{Stopwatch} class, and Python has a handy \texttt{time} module. You are free to use either.
    On some modern Java systems, the compiler performs very funky optimisations behind your back, giving wildy divergent measurements for the same code on the same machine.
    You can handle that by running time code a few times before measuring it.
    It’s a black art.
  \item[Experiment design.]
    If you like doing repetitive work, you  can run your experiments individually by starting dozens of processes from the command line (or, worse, an IDE) and writing the results into a spreadsheet to compute averages.
    However, such an approach quickly becomes both tedius and error-prone.

    Instead, construct another program that performs all the experiments for you, and writes the results, including a time stamp of the experiment, into another file.
    There are many schools of thought about how to write that program; you can make it part of your original solution, or you can use a separate programme in a dedicated scripting language.
    One useful program (or \texttt{builtin} in \texttt{bash}) is \texttt{time}, a simple usage would be
    \begin{lstlisting}
      time java Simple < ../data/ints-200-2.txt
    \end{lstlisting}
    Because of the setup on codeJudge, our programs have debugging output on \texttt{stderr}, and there is the need to ignore this when collecting experimental results into a file.
    The following is an example how this could look like (note that the trailing \verb+\+ continues the command on the next line):
    \lstinputlisting{../src/simpleExp.sh}

    More advanced solutions (maybe in other languages) could compute averages, or even draw the plots for you.
    One example (that turned out to be somewhat more complicated) can be found in \texttt{../src/experiment.py}.
\end{description}

\section{Learning outcomes}

Besides some important secondary skills (input/output, exposing yourself to an unfamiliar programming language, basic experimentation, drawing graphs, computing averages), the main points of this exercise are
\begin{itemize}
  \item It does \emph{not} make a big difference which language you use.
  \item It \emph{does} make a big difference which algorithm you use.
  \item The slowest observed performance is well characterised by the predicted worst-case running time.
  \item The fastest observed performance contains very little information about general behaviour.
\end{itemize}

\end{document}


\documentclass[11pt]{article}
\usepackage{enumitem}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{float}

\graphicspath{{images/}}

\begin{document}
\title{MAT1830 --- Assignment 7}
\author{Dylan Pinn --- 24160547}
\maketitle

\section*{Question 1}

Let $R$ be the relation on $\mathbb{Z} \times \mathbb{Z}$ defined by $(w, x)R(y,
z)$ if any only if $w + x \leq y + z$.

\noindent
Let $S$ be the relation on $\mathbb{Z} \times \mathbb{Z}$ defined by $(w, x)S(y,
z)$ if and only if $w \leq y$ and $x \leq z$

\begin{enumerate}[label= (\alph*)]
  \item Is $R$ antisymmetric?

  Yes. Because the relation is defined across integers, if we make $w + x$ some
    integer $a$ and $y + z$ soem integer $b$. Then we can show that for any $a,b
    \in \mathbb{Z}$ we have that $a \leq b$ or $b \leq a \Rightarrow a = b$.

  \item Is $S$ antisymmetric?

    No. Because we can show that for $(1, 4)\not S(2, 2)$ and $(2,2) \not
    S(1,4)$

\end{enumerate}
\break{}

\section*{Question 2}

You know the following facts about a relation $T$:

\begin{itemize}
  \item $T$ is defined on the set $\{a, b, c, d, e \}$,
  \item $aTc, bTc, cTd, eTb$ and $e\not{T}a$,
  \item $T$ is a partial order relation.
\end{itemize}

Draw a Hasse diagram for every possible relation that $T$ could be and, for
each, write down whether it is a total order relation.

\begin{center}
    \includegraphics{A7}
\end{center}

No, not total order relation.

\break{}
\section*{Question 3}

Find expressions for each of the following

\begin{enumerate}[label= (\alph*)]
  \item The number of strings of 7 lower case letters (a-z) that do not contain
    any letter twice or more.

    As we need to do unordered selections without repetition it will be:
  \[
    = \binom{n}{r} \\
    = \binom{26}{7}
  \]

  \item The number of binary strings of length 50 that contain at most three 1s.

    As we want to get all of these that contain at most three it will be:

    \[
      =\binom{50}{0} + \binom{50}{1} + \binom{50}{2} + \binom{50}{3}
    \]

  \item The number of ternary strings of length 10 (strings of 0s, 1s and 2s)
    containing exactly one 1 and exactly four 2s.

    We want to find the strings such that 1222200000. As order is not important
    it will be and we don't want repetition. It is:

    \[
      =\binom{10}{5}
    \]

\end{enumerate}

\break{}
\section*{Question 4}

A small school with 14 students has a basketball team with 5 players, a chess
team with 4 players, a netball team with 7 players, and a soccer team with 11
players.

\begin{itemize}
  \item Each student is on at least one of these teams and no student is on all
    four teams.
  \item Only one student, Hawa, is on three of the teams; she plays chess,
    netball and soccer.
  \item There is exactly 1 student who plays basketball and chess.
  \item There is exactly 1 student who plays basketball and netball.
  \item Hawa is the only student who plays chess and netball.
  \item There are exactly 3 students (including Hawa) who play chess and soccer.
  \item There are exactly 5 students (including Hawa) who play netball and
    soccer.
\end{itemize}

How many students play basketball and soccer?

\begin{align*}
  |B \cup C \cup N \cup S| &= |B| + |C| + |N| + |S| \\
  &- |B \cap C| - |B \cap N| - |B \cap S| - |C \cap N| - |C \cap S| - |N \cap
  S| \\
  &+|B \cap C \cap N| + |B \cap C \cap S| + |B \cap N \cap S| + |C \cap N \cap
  S| \\
  &- |B \cap C \cap N \cap S| \\
  14 &= 5 + 4 + 7 + 11 - 1 - 1 - x - 1 - 3 - 5 + 1 + 0 + 0 + 0 + 0 - 0 \\
  14 &= 17 - x \\
  -3 &= -x \\
  x &= 3
\end{align*}

Therefore there are three students how play soccer and basketball.

\end{document}


\documentclass{beamer}

\mode<presentation>
{
  \usetheme{Hawke}
  \setbeamercovered{transparent}
}

\usepackage[english]{babel}
\usepackage[latin1]{inputenc}
\usepackage{times}
\usepackage[T1]{fontenc}
\usepackage{multimedia}

%%%%%%
% My Commands
%%%%%%

\newcommand{\bb}{{\boldsymbol{b}}}
\newcommand{\bx}{{\boldsymbol{x}}}
\newcommand{\by}{{\boldsymbol{y}}}
\newcommand{\bfm}[1]{{\boldsymbol{#1}}}
\newcommand{\pda}[2]{\frac{\partial{#1}}{\partial{#2}}}
\newcommand{\pdb}[2]{\frac{\partial^2{#1}}{\partial{#2}^2}}
\newcommand{\pdc}[3]{\frac{\partial^2{#1}}{\partial{#2}\partial{#3}}}

%%%%

\title[Lecture 28] % (optional, use only with long paper titles)
{Lecture 28 - Von Neumann stability 2}

\author[I. Hawke] % (optional, use only with lots of authors)
{I.~Hawke}
\institute[University of Southampton] % (optional, but mostly needed)
{
  School of Mathematics, \\
  University of Southampton, UK
}

\date[Semester 1] % (optional, should be abbreviation of conference name)
{MATH3018/6141, Semester 1}

\subject{Numerical methods}

\pgfdeclareimage[height=0.5cm]{university-logo}{mathematics_7469}
\logo{\pgfuseimage{university-logo}}

\AtBeginSection[]
{
  \begin{frame}<beamer>
    \frametitle{Outline}
    \tableofcontents[currentsection]
  \end{frame}
}

\begin{document}

\begin{frame}
  \titlepage
\end{frame}

\section{Stability}

\subsection{Evidence}

\begin{frame}
  \frametitle{Partial differential equations}

  Partial differential equations (PDEs) involve derivatives of
  functions of more than one variable, say $u(x, y)$ or $y(t,
  x)$. Hence more complex behaviour and more interesting
  physics. \pause

  \vspace{1ex}

  Only look at linear problems.  Also only consider finite difference
  methods: simple to analyse but not always competitive. \pause

  \vspace{1ex}

  Previously looked at simple finite difference algorithms applied to
  heat and advection equations. At present have no explanation for
  which worked and which did not; for that we need stability theory.

\end{frame}

\begin{frame}
  \frametitle{The results}

  \begin{center}
    \begin{tabular}{c|c c}
      Algorithm & Heat equation & Advection equation \\ \hline
      FTCS & Stable $s < 1/2$ & Unstable \\
      BTCS & Stable $\forall s$ & ?? \\
      FTBS & ?? & Stable $c < 1$
    \end{tabular}
  \end{center}

  Here $s = k \delta / h^2$ for the heat equation
  \begin{equation*}
    \partial_t y - k \partial_{x x} y = 0,
  \end{equation*}
  and $c = v \delta / h$ for the advection equation
  \begin{equation*}
    \partial_t y + v \partial_x y = 0.
  \end{equation*}

\end{frame}

\section{Von Neumann stability theory}

\subsection{Von Neumann stability theory}

\begin{frame}
  \frametitle{Determining stability}

  Von Neumann stability method for linear algorithms and equations:
  \begin{enumerate}
  \item<2-> Make ansatz that solutions of the finite difference
    equation have form
    \begin{equation*}
      y_{\ell}^k =  e^{\alpha j \ell h} q^k, \qquad (\, j = \sqrt{-1}, \alpha \in {\mathbb R}, q \in {\mathbb C} ).
    \end{equation*}
  \item<3-> Substitute this solution into the finite difference
    equation.
  \item<4-> Determine $q$ as a function of $\alpha$, the grid spacings
    $h,\delta$, and known numbers (such as the advection velocity
    $v$).
  \item<5-> Determine under what conditions we have stability, i.e.\ when
    \begin{equation*}
      |q| < 1 \quad \forall \alpha.
    \end{equation*}
  \end{enumerate}

\end{frame}

\section{Von Neumann stability in practise}

\subsection{Von Neumann stability in practise}

\begin{frame}
  \frametitle{BTCS for the heat equation}

  We look at BTCS in the form
  \begin{equation*}
    (1 + 2 s) y_i^{n+1} = y_i^n + s \left( y_{i+1}^{n+1} +
      y_{i-1}^{n+1} \right).
  \end{equation*} \pause
  Use ansatz $y_{\ell}^k =  \exp (\alpha j \ell h) q^k$
  and removing common factors:
  \begin{equation*}
    (1 + 2 s) q = 1 + s q \left( e^{\alpha j h} +  e^{-\alpha j h}
    \right).
  \end{equation*} \pause
  Rewrite this as
  \begin{align*}
    && q & = 1 + 2 s \left( e^{\alpha j h} +  e^{-\alpha j h} - 2 \right)
    q \\
    && & = 1 - 4 s q \sin^2 \left( \tfrac{\alpha h}{2} \right) \\
    & \implies & q & = \left( 1 + 4 s \sin^2 \left( \tfrac{\alpha
          h}{2} \right) \right)^{-1},
  \end{align*} \pause
  which is \emph{always} less than 1. Hence BTCS is unconditionally
  stable.

\end{frame}

\begin{frame}
  \frametitle{FTCS for the advection equation}

  Ignore difference in \emph{differential} equation; only need
  analysis at level of finite difference equation:
  \begin{equation*}
    y_i^{n+1} = y_i^n - \frac{c}{2} \left( y_{i+1}^n - y_{i-1}^n
    \right).
  \end{equation*}
  Use ansatz $y_{\ell}^k = \exp (\alpha j \ell h) q^k$ and remove
  common factors:
  \begin{align*}
    && q & = 1 - \frac{c}{2} \left( e^{\alpha j h} - e^{-\alpha j h}
    \right) \\
    & \implies & q & = 1 - j c \sin ( \alpha h ).
  \end{align*} \pause
  As this is complex the modulus is
  \begin{equation*}
    |q|^2 = 1 + c^2 \sin^2 ( \alpha h )
  \end{equation*}
  which is always greater than 1: method is unconditionally unstable.

\end{frame}

\begin{frame}
  \frametitle{FTBS for the advection equation}

  Finally for the upwind method, FTBS, we had the difference equation
  \begin{equation*}
    y_i^{n+1} = y_i^n - c \left( y_{i}^n - y_{i-1}^n \right).
  \end{equation*}
  Use ansatz $y_{\ell}^k =  \exp (\alpha j \ell h) q^k$
  and remove common factors:
  \begin{equation*}
    q = 1 - c \left( 1 - e^{-\alpha j h} \right).
  \end{equation*} \pause

  Describes circle radius $c$ in the complex plane, centre at $1 -
  c$. \pause If $c > 1$ there are finite values of $\alpha$ for which
  $|q| > 1$, leading to instability; if $c < 1$ then $q$ is at most 1,
  leading to stability.

\end{frame}

\section{Summary}

\subsection{Summary}

\begin{frame}
  \frametitle{Summary}

  \begin{itemize}
  \item Von Neumann stability analysis is the key tool for
    understanding the results seen ``experimentally'' in the last
    lecture.
  \item By looking at the growth or decay of individual frequency
    modes we can check the stability of the method; Lax's theorem then
    proves convergence.
  \item For complex equations and / or algorithms, it is often very
    difficult or impossible to apply even Von Neumann stability
    analysis.
  \item If an algorithm does not work in the linear case then it is
    highly unlikely to work for more complex problems!
  \end{itemize}

\end{frame}

\end{document}


\lab{Python}{F2PY}{F2PY}
\label{lab:f2py}
\objective{Learn how to use F2PY, especially for wrapping Fortran code involving arrays.}

Here we will explore another simple way to access external code from Python.
Various utilities exist to do this, and many are specialized for specific tasks.
F2PY is a utility that can be used to automatically wrap Fortran subroutines so that they are callable from Python.
It is included with NumPy and in most cases does not usually require heavy configuration for use.
The primary requirement for using F2PY is a working Fortran compiler.
If you have access to GFortran, the Fortran equivalent of gcc, it should work well.

\section*{Examples}

F2PY is very simple to use.
Many simple Fortran files can be compiled \emph{automatically} without any hand-written interfaces between languages.
Windows 64 bit users, if you encounter errors, see the Windows 64 bit install guide.
We will start by wrapping the Fortran function we wrapped as a part of Lab \ref{lab:cythonwrap}.
Here we will wrap the subroutine in unmodified form.
In Lab \ref{lab:cythonwrap} we modified the Fortran file to make it callable from C.
F2PY will take care of this for us so that we can wrap the subroutine directly.
Here is the file in its unmodified form:
\lstinputlisting[style=fromfile, language=Fortran]{ftridiag.f90}
We can wrap this file by running the following in our terminal:
\begin{lstlisting}[style=ShellInput]
python f2py.py -c ftridiag.f90 -m ftridiag
\end{lstlisting}
We can also tell F2PY to pass an optimization flag to the compiler using the \li{--opt} flag, as in:
\begin{lstlisting}[style=ShellInput]
python f2py.py -c ftridiag.f90 -m ftridiag --opt="-O3"
\end{lstlisting}
Depending on your system configuration, you may be able to call F2PY without expressly calling python, or without including its file extension (or both).

Running these commands will create a file called \li{ftridiag.pyd} on a Windows computer or \li{ftridiag.so} on a Linux or Mac based computer.
This file is a Python module that is linked against the compiled Fortran code.
All the details about array type, etc. have been infered by F2PY.
We can test the module the same way we tested the module we built using Cython.
\lstinputlisting[style=fromfile]{ftridiag_test.py}
For our example here we compiled the subroutine by itself instead of compiling an entire module.
If the function were a part of a Fortran module \li{tridiag} F2PY would have made it available to us as \li{ftridiag.tridiag.ftridiag}.

% Show example of wrapping FORTRAN using Cython.

% Introduce f2py, teach how to wrap FORTRAN automatically.

% Wrap a C function using f2py to teach about signature files.

% Wrap a LAPACK function (dgtsv?), have them do it too.

% Discuss how memory layout affects performance.

% Discuss programmer time value.


\section{Parton string model algorithm.}

\hspace{1.0em}
The parton string model algorithm can be considered as a set of
steps should be performed:
\begin{enumerate}
\item Create two vectors (projectile and target) of particles: assign initial
projectile and target particle types, their coordinates and momenta.
In the case of hadron-nucleus (or nucleus-nucleus) interaction
one should perform
target nucleus (or projectile and target nuclei) initial state simulation
 and sample impact parameter;
\item Sample
collision participants and separated them into diffractive and non-diffractive.
Store the total interaction four momentum of participants;
\item For non-diffractive inelastic collisions sample the number
of the soft longitudinal and hard kinky string can be produced.
\item Excite and reexcite colliding particles in the case of diffractive collisions and
create diffractive longitudinal strings, if particles are not participate in
further soft or hard collisions;
\item Perform longitudinal and kinky string excitations;
\item Perform string decay simulation;
\item Correct energies and momenta of produced particles, if it is needed.
\end{enumerate}