 \documentclass [12pt]{article}

\usepackage {amsmath}
\usepackage {amsthm}
\usepackage {amssymb}
\usepackage {graphicx}
\usepackage {float}
\usepackage {multirow}
\usepackage {xcolor}
\usepackage {algorithmic}
\usepackage [ruled,vlined,commentsnumbered,titlenotnumbered]{algorithm2e} \usepackage {array}
\usepackage {booktabs}
\usepackage {url}
\usepackage {parskip}
\usepackage [margin=1in]{geometry}
\usepackage [T1]{fontenc}
\usepackage {cmbright}
\usepackage [many]{tcolorbox}
\usepackage [colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage {enumitem}
\usepackage {xparse}
\usepackage {verbatim}
\usepackage{listings}
\usepackage{xcolor}
\lstset { %
    language=C++,
    backgroundcolor=\color{black!5}, % set backgroundcolor
    basicstyle=\footnotesize,% basic font setting
}
\newtheorem{theorem}{Theorem}
\newtheorem{remark}{Remark}

\DeclareTColorBox {Solution}{}{breakable, title={Solution}} \DeclareTColorBox {Solution*}{}{breakable, title={Solution (provided)}} \DeclareTColorBox {Instruction}{}{boxrule=0pt, boxsep=0pt, left=0.5em, right=0.5em, top=0.5em, bottom=0.5em, arc=0pt, toprule=1pt, bottomrule=1pt} \DeclareDocumentCommand {\Expecting }{+m}{\textbf {[We are expecting:} #1\textbf {]}} \DeclareDocumentCommand {\Points }{m}{\textbf {(#1 pt.)}}

\begin {document}

\vspace {1em}
\begin {Instruction}
Adapted From Virginia Williams' lecture notes.
\end {Instruction}

{\LARGE \textbf {COMP 285 (NC A\&T, Spr `22)}\hfill \textbf {Lecture 12} }

\begin{centering}
\section*{Data Structures: Heaps and Red-Black Trees}
\end{centering}

\section{Data Structures Review}

Thus far in this course we have mainly discussed algorithm design, and have specified algorithms at a relatively high level. For example, in describing sorting algorithms, we often assumed that we could insert numbers into a list in constant time; we didn’t worry too much about the actual implementation of this, and took it as an assumption that this could actually be implemented (e.g. via a linked-list).

We will wrap-up this lecture by talking about the actual design and implementation of some useful datastructures. The purpose is mainly to give you a taste of data-structure design. We won’t discuss this area too much more in the course, and will switch back to discussing higher-level algorithms.

To motivate the data structures that we will discuss in this course, consider the following table that lists a bunch of basic operations that we would like to perform on a set/list of numbers, together with the worst-case runtime of performing those operation for two of the data structures that you are (hopefully) familiar with: linked lists of n (unsorted) numbers, and an array of n sorted numbers. In this course, we will assume that all the numbers we store are unique.

\begin{center}
\begin{tabular}{|c|p{2.4cm}|c|}
\hline \\
Operation & Runtime with Unsorted Linked Lists & Runtime with Sorted Array \\
\hline \\
Search (ie, find 17) & $\Theta(n)$ & $\Theta(\log n)$(via binary search) \\
Select (ie, find 12 smallest element) & $\Theta(n)$ & $\Theta(1)$ (just look at the 12th element) \\
Rank (ie, \# elt. less than 17) & $\Theta(n)$ & $\Theta(\log n)$\\
Predecessor/Successor & $\Theta(n)$ & $\Theta(1)$ \\
Insert element & $\Theta(1)$ & $\Theta(n)$ \\
Delete element & $\Theta(1)$ & $\Theta(n)$ \\
\hline
\end{tabular}
\end{center}

The above table makes Sorted Arrays look like a pretty decent data structure for \textit{static} data. In many applications, however, the data changes often, in which case the $\Theta(n)$ time it takes to insert or delete an element is prohibitive. This prompts the question: \textit{Is it possible to have one data structure that is the best of both worlds (logarithmic or constant time for all these operations)?} The answer is, essentially, ``Yes'', provided we don’t mind if some of the $\Theta(1)$’s turn into $\Theta(log n)$.

We will sketch out the Binary Search Tree. Before talking about binary search trees, we sketch another data structure called Heap. Heaps are worse than both unsorted linked lists and sorted arrays in just about any parameter, except that they can allow for efficient ($O(\log n)$ time) insertions - still worse than linked lists. But there is one operation that Heaps do exceptionally well (again, $\Theta(\log n)$): \texttt{extract-min}, which outputs the minimum element, and then deletes it from the heap. Heaps (in their simple form that we consider here) still don't beat the asymptotic runtime of binary search trees for any operation, but they have two other advantages: they are much simpler, and if you only want to run insert and \texttt{extract-min} they are also much faster in practice.

\section{Heaps} \textbf{Definition} (Complete binary tree). A complete binary tree is a rooted binary tree where each level is full except maybe the last level, and all nodes on the last level are as far left as they can be.

\textbf{Definition} (Binary min-heap).

A \textit{Binary min-heap} is a data structure that stores elements that have keys from a totally ordered universe (say, the integers). A binary min-heap supports the following operations:

\begin{itemize}
  \item \texttt{insert}(i): Inserts an element with key $i$ into the data structure
  \item \texttt{extract-min}: Returns the element with the minimum key and deletes it from the data structure.
\end{itemize}

A binary heap stores elements in a complete binary tree with a root $r$. Each node $x$ has key($x$) (the key of the element stored in $x$), p($x$) (the parent of $x$, where p($r$) = NIL), left($x$) (the left child of $x$), and right($x$) (the right child of $x$). The children of $x$ are either other nodes or NIL.

A binary heap should satisfy two key properties: The first key property is that for every node $x$, the keys of all nodes under $x$ are greater than key($x$). The second key property of a heap is that its tree is complete (see 1).

As explained above, binary min-heaps (which we call heaps for short) support two operations3 : \texttt{insert}(i) and \texttt{extract-min}. These are particularly useful if you want to have a priority queue where elements (e.g. jobs) arrive in an arbitrary order, but always leave in order of their key/priority (e.g. when the highest priority job is executed first).

\subsection{Basic Operations on Heaps}
In the heap data structure we always maintain a pointer to the last node in the heap (the right most node in the last level), as well as a pointer to the next node to be created (typically the one to the right of the last node; except when the last level is full, in which case this is the left-most node in the next level). Maintaining those is easy when we think of the
representation of a heap as an array. But to make things simpler, we will abstract these details and simply assume access to these pointers. (See CLRS for details.)

\subsubsection{\texttt{insert}(i)}
To insert an element, we create a new node with key $i$ at the bottom of the heap. Then recursively “propagate it up the heap” until the heap property is restored: That is compare i to the key of the new element's parent; if $i$ is smaller than the key of its parent, replace the keys, and continue to compare with the parent's parent, etc. Once the algorithm reaches a parent whose key is smaller, the insertion is complete.

\begin{algorithm}
\caption{Heap \texttt{insert}(i)}\label{alg:heap_insert}
\begin{algorithmic}
\STATE $x \gets $ first node without two children
\STATE $y \gets$ new node with key$(y) \gets i$, left$(y) \gets $ NIL, right$(y) \gets $ NIL, $p(y)\gets x$
\IF {$left(x) == NIL$}
  \STATE left$(x) \gets y$
\ELSE
  \STATE right$(x) \gets y$
\ENDIF
\WHILE {$p(y) \neq \text{NIL} \& \text{key}(p(y)) > \text{key}(y)$}
  \STATE $v \gets \text{key}(y)$
  \STATE $\text{key}(y) \gets \text{key}(p(y)) $
  \STATE $\text{key}(p(y)) \gets v$
  \STATE $y \gets p(y)$
\ENDWHILE
\end{algorithmic}
\end{algorithm}

\subsubsection{\texttt{extract-min}}
To extract-min, we save the key of the root, replace it with the key of the last node, and delete the last node. Then we recursively ``propagate the key copied from the last node down the tree''. That is compare to the children of the root; if the key of the root is larger than the keys of one of the children, swap the smaller key up, and recurse on the corresponding sub-tree. Once the algorithm reaches children whose keys are larger, the heap property is restored. Then we can return the value we saved at the beginning.

\begin{algorithm}
\caption{Heap \texttt{extract-min}(i)}
\label{alg:heap_extract}
\begin{algorithmic}
\STATE $x \gets $ last node
\STATE $r \gets$ root, $m \gets$ key($r$)
\STATE key($r) \gets $ key($x$)
\IF {right($p(x)$) == NIL}
  \STATE left($p(x)) \gets$ NIL
\ELSE
  \STATE right($p(x)) \gets$ NIL
\ENDIF
delete node($x$)
\WHILE {True}
  \IF {left($r$) == NIL}
    \RETURN m
  \ELSIF{key(left($r$)) $<$ key($r$) \& right($r$) == NIL}
    \STATE $v \gets \text{key}(r)$
    \STATE $\text{key}(r) \gets \text{key}(\text{left}(r))$
    \STATE $\text{key}(\text{left}(r)) \gets v$
    \STATE $r \gets \text{left}(r)$
  \ELSIF{key(left($r$)) $<$ key($r$), key(righ($r$))}
    \STATE $v \gets \text{key}(r)$
    \STATE $\text{key}(r) \gets \text{key}(\text{left}(r))$
    \STATE $\text{key}(\text{left}(r)) \gets v$
    \STATE $r \gets \text{left}(r)$
  \ELSIF{key(right($r$)) $<$ key($r$), key(left($r$))}
    \STATE $v \gets \text{key}(r)$
    \STATE $\text{key}(r) \gets \text{key}(\text{right}(r))$
    \STATE $\text{key}(\text{right}(r)) \gets v$
    \STATE $r \gets \text{right}(r)$
  \ELSE
    \RETURN m
  \ENDIF
\ENDWHILE
\RETURN m

\end{algorithmic}
\end{algorithm}

\section{Binary Search Trees}
\textbf{Definition} (Binary search tree).

A binary search tree (BST) is a data structure that stores elements that have keys from a totally ordered universe (say, the integers). In this lecture, we will assume that each element has a unique key. A BST supports the following operations:

\begin{itemize}
  \item search(i): Returns an element in the data structure associated with key i
  \item insert(i): Inserts an element with key i into the data structure
  \item delete(i): Deletes an element with key i from the data structure, if such an element exists
\end{itemize}

A BST stores the elements in a binary tree with a root $r$. Each node $x$ has key($x$) (the key of the element stored in $x$), $p(x)$ (the parent of $x$, where $p(r ) = NIL$), $left(x$) (the left child of $x$), and right($x$) (the right child of $x$). The children of $x$ are either other nodes or NIL.

The key BST property is that for every node $x$, the keys of all nodes under left($x$) are less than key($x$) and the keys of all nodes under right($x$) are greater than key($x$).

\textit{Example 4}. In the following example, the root node $r$ stores $3$ in it (key($r) = 3$), its left child left($x$) stores 1, its right child right($x$) stores 7, and all leaf nodes (storing 1, 4, and 8, respectively) have NIL as their two children.

\begin{center}
\begin{tikzpicture}[scale=0.15]
\tikzstyle{every node}+=[inner sep=0pt]
\draw [black] (25.7,-16.1) circle (3);
\draw (25.7,-16.1) node {$3$};
\draw [black] (25.7,-29.4) circle (3);
\draw (25.7,-29.4) node {$4$};
\draw [black] (31.4,-22.2) circle (3);
\draw (31.4,-22.2) node {$7$};
\draw [black] (19.9,-22.2) circle (3);
\draw (19.9,-22.2) node {$1$};
\draw [black] (36.7,-29.4) circle (3);
\draw (36.7,-29.4) node {$8$};
\draw [black] (23.63,-18.27) -- (21.97,-20.03);
\fill [black] (21.97,-20.03) -- (22.88,-19.79) -- (22.16,-19.1);
\draw [black] (27.75,-18.29) -- (29.35,-20.01);
\fill [black] (29.35,-20.01) -- (29.17,-19.08) -- (28.44,-19.76);
\draw [black] (29.3,-24.4) -- (27.45,-26.97);
\fill [black] (27.45,-26.97) -- (28.33,-26.61) -- (27.51,-26.02);
\draw [black] (33.18,-24.62) -- (34.92,-26.98);
\fill [black] (34.92,-26.98) -- (34.85,-26.04) -- (34.04,-26.64);
\end{tikzpicture}
\end{center}

Example 5. The following binary tree is not a BST since 2 > 3 and 2 is a child of 7, which is the right child of 3:

\begin{center}
\begin{tikzpicture}[scale=0.15]
\tikzstyle{every node}+=[inner sep=0pt]
\draw [black] (25.7,-16.1) circle (3);
\draw (25.7,-16.1) node {$3$};
\draw [black] (25.7,-29.4) circle (3);
\draw (25.7,-29.4) node {$2$};
\draw [black] (31.4,-22.2) circle (3);
\draw (31.4,-22.2) node {$7$};
\draw [black] (19.9,-22.2) circle (3);
\draw (19.9,-22.2) node {$1$};
\draw [black] (36.7,-29.4) circle (3);
\draw (36.7,-29.4) node {$8$};
\draw [black] (23.63,-18.27) -- (21.97,-20.03);
\fill [black] (21.97,-20.03) -- (22.88,-19.79) -- (22.16,-19.1);
\draw [black] (27.75,-18.29) -- (29.35,-20.01);
\fill [black] (29.35,-20.01) -- (29.17,-19.08) -- (28.44,-19.76);
\draw [black] (29.3,-24.4) -- (27.45,-26.97);
\fill [black] (27.45,-26.97) -- (28.33,-26.61) -- (27.51,-26.02);
\draw [black] (33.18,-24.62) -- (34.92,-26.98);
\fill [black] (34.92,-26.98) -- (34.85,-26.04) -- (34.04,-26.64);
\end{tikzpicture}
\end{center}

\textbf{Some properties.} \textit{Relationship to Quicksort: We can think of each node $x$ as a pivot for quicksort for the keys in its subtree. The left subtree contains $A_<$ for key$(x)$ and the right subtree contains $A_>$ for key($x$).}

\textit{Sorting the keys}: We can do an \textit{inorder} traversal of the tree to recover the nodes in sorted order from left to right (the smallest element is in the leftmost node and the largest element is in the rightmost node). The Inorder procedure takes a node $x$ and returns the keys in the subtree under $x$ in sorted order. We can recursively define Inorder($x$) as: (1) If left$(x) \neq$  NIL, then run Inorder(left$(x)$), then: (2) Output key$(x)$ and then: (3) If right$(x) \neq$ NIL, run Inorder(right$(x)$). With this approach, for every $x$, all keys in its left subtree will be output
before $x$, then $x$ will be output and then every element in its right subtree.

\textit{Subtree property}: If we have a subtree where $x$ has $y$ as a left child, $y$ has $z$ as a right child, and $z$ is the root for the subtree $T_z$ , then our BST property implies that all keys in $T_z$ are $> y $ and $< x$. Similarly, if we have a subtree where $x$ has $y$ as a right child, $y$ has $z$ as a left child, and $z$ is the root of the subtree $T_z$ , then our BST property implies that all keys in $T_z$ are between $x$ and $y$.

\subsection{Basic Operations of BSTs}
The three core operations on a BST are search, insert, and delete. For this lecture, we will assume that the BST stores distinct numbers, i.e. we will identify the objects with their names and we will have each name be represented by a number.

\subsubsection{search}
To search for an element, we start at the root and compare the key of the node we are looking at to the element we are searching for. If the node's key matches, then we are done. If not, we recursively search in the left or right subtree of our node depending on whether this node was too large or too small, respectively. If we ever reach NIL, we know the element does not exist in our BST. In the following algorithm in this case, we simply return the node that would be the parent of this node if we inserted it into our tree.

\begin{algorithm}
\caption{\texttt{search}(i)}\label{alg:bst_search}
\begin{algorithmic}
\RETURN search(root, i)
\end{algorithmic}
\end{algorithm}

\begin{algorithm}
\caption{\texttt{search}(x, i)}\label{alg:bst_search_substree}
\begin{algorithmic}
\IF {key(x) == i}
  \RETURN $x$
\ELSIF {key(x) < i}
  \IF {right(x) == NIL}
    \RETURN $x$
  \ENDIF
  \RETURN search(right(x), i)
\ELSIF {key(x) > i}
  \IF {left(x) == NIL}
    \RETURN $x$
  \ENDIF
  \RETURN search(left(x), i)
\ENDIF
\end{algorithmic}
\end{algorithm}

\textit{Example 6}. If we call \texttt{search}(5.5) on the following BST, we will return the node storing 5 (by taking the path $4 \to 6 \to 5$). This also corresponds to the path taken when calling \texttt{search}(4.5) or
\texttt{search}(5).

\begin{center}
\begin{tikzpicture}[scale=0.15]
\tikzstyle{every node}+=[inner sep=0pt]
\draw [black] (25.2,-16) circle (3);
\draw (25.2,-16) node {$4$};
\draw [black] (17.6,-25.2) circle (3);
\draw (17.6,-25.2) node {$3$};
\draw [black] (12.4,-34.4) circle (3);
\draw (12.4,-34.4) node {$1$};
\draw [black] (20.6,-34.4) circle (3);
\draw (20.6,-34.4) node {$2$};
\draw [black] (32.9,-25.2) circle (3);
\draw (32.9,-25.2) node {$6$};
\draw [black] (38.2,-34.4) circle (3);
\draw (38.2,-34.4) node {$8$};
\draw [black] (29.1,-34.4) circle (3);
\draw (29.1,-34.4) node {$5$};
\draw [black] (42.5,-43.4) circle (3);
\draw (42.5,-43.4) node {$7$};
\draw [black] (23.29,-18.31) -- (19.51,-22.89);
\fill [black] (19.51,-22.89) -- (20.41,-22.59) -- (19.63,-21.95);
\draw [black] (27.13,-18.3) -- (30.97,-22.9);
\fill [black] (30.97,-22.9) -- (30.84,-21.97) -- (30.08,-22.61);
\draw [black] (16.12,-27.81) -- (13.88,-31.79);
\fill [black] (13.88,-31.79) -- (14.71,-31.34) -- (13.83,-30.85);
\draw [black] (18.53,-28.05) -- (19.67,-31.55);
\fill [black] (19.67,-31.55) -- (19.9,-30.63) -- (18.95,-30.94);
\draw [black] (31.75,-27.97) -- (30.25,-31.63);
\fill [black] (30.25,-31.63) -- (31.01,-31.08) -- (30.09,-30.7);
\draw [black] (34.4,-27.8) -- (36.7,-31.8);
\fill [black] (36.7,-31.8) -- (36.74,-30.86) -- (35.87,-31.36);
\draw [black] (39.49,-37.11) -- (41.21,-40.69);
\fill [black] (41.21,-40.69) -- (41.31,-39.76) -- (40.41,-40.19);
\end{tikzpicture}
\end{center}

\textbf{Claim}. \texttt{search}(i) returns a node containing $i$ if $i \in$ BST, otherwise a node $x$ such that either key($x$) is the smallest in BST > $i$ or the largest in BST < $i$, where node $x$ happens to be the parent of the new node if it were to be inserted into BST. This follows directly from
the BST property. Try to formally prove this claim as an exercise.

\begin{center}
\begin{tikzpicture}[scale=0.15]
\tikzstyle{every node}+=[inner sep=0pt]
\draw [black] (25.2,-16) circle (3);
\draw (25.2,-16) node {$5$};
\draw [black] (18.5,-24.1) circle (3);
\draw (18.5,-24.1) node {$2$};
\draw [black] (23.29,-18.31) -- (20.41,-21.79);
\fill [black] (20.41,-21.79) -- (21.31,-21.49) -- (20.54,-20.85);
\end{tikzpicture}
\end{center}

\textit{Remark 8}. \texttt{search}(i) does not necessarily return the node in BST that is closest in value. Consider the tree above. If we \texttt{search}for an element with a key of 4, the node with key 2 is returned, whereas the element with key 5 is the closest element by value

\subsubsection{insert}
As before, we will assume that all keys are distinct. We will search(i) for a node $x$ to be the parent and create a new node y , placing it as a child of $x$ where it would logically go according to the BST property.

\begin{algorithm}
\caption{\texttt{insert}(i)}\label{alg:bst_inert}
\begin{algorithmic}
\STATE $x \gets $ search(i)
\STATE $y \gets$ new node with key$(y) \gets i$, left$(y) \gets$ NIL, right($y) \gets$ NIL, $p(y) \gets x$
\IF {i < key(x)}
  \STATE left$(x) \gets y$
\ELSE
  \STATE right$(x) \gets y$
\ENDIF
\end{algorithmic}
\end{algorithm}

\textit{Remark 9}. Notice that $x$ needed to have NIL as a child where we want to put $y$ by the properties of our search algorithm

\subsubsection{delete}
Deletion is a bit more complicated. To delete a node $x$ that exists in our tree, we consider several cases:
\begin{enumerate}
  \item If $x$ has no children, we simply remove it by modifying its parent to replace $x$ with NIL.
  \item If $x$ has only one child c, either left or right, then we elevate c to take $x$'s position in the tree by modifying the appropriate pointer of $x$'s parent to replace $x$ with c, and also fixing c's parent pointer to be $x$'s parent.
  \item  If $x$ has two children, a left child $c_1$ and right child $c_2$, then we find $x$'s immediate successor $z$ and have $z$ take $x$'s position in the tree. Notice that $z$ is in the subtree under $x$'s right child $c_2$ and we can find it by running $z \gets \texttt{search}(c_2, key(x))$. Note that since $z$ is $x$'s successor, it doesn't have a left child, but it might have a right child. If $z$ has a right child, then we make $z$'s parent point to that child instead of z (also fixing the child's parent pointer). Then we replace $x$ with z, fixing up all relevant pointers: the rest of $x$'s original right subtree becomes z's new right subtree, and $x$'s left subtree becomes z's new left subtree. (Note that alternatively, we could have used $x$'s immediate predecessor y and followed the same analysis in a mirrored fashion.)

\end{enumerate}
In the following algorithm, if $p$ is the parent of $x$, $child(p)$ refers to $left(p)$ if $x$ was the left child of $p$ and to $right(p)$ otherwise.

\begin{algorithm}
\caption{\texttt{delete}(i)}\label{alg:bst_delete}
\begin{algorithmic}
\STATE $x \gets $ search(i)
\IF {key(x) $\neq i$}
  \RETURN
\ENDIF
\IF {NIL = left(x) and NIL = right(x)}
  \STATE child(p(x)) $\gets$ NIL
  \STATE delete-node(x)
\ENDIF
\IF {NIL = left(x)}
  \STATE $y \gets $ right(x)
  \STATE $p(y) \gets p(x)$
  \STATE $child(p(y)) \gets y$
  \STATE delete-node(x)
\ELSIF {NIL = right(x)}
  \STATE $y \gets $ right(x)
  \STATE $p(y) \gets p(x)$
  \STATE $child(p(y)) \gets y$
  \STATE delete-node(x)
\ELSE
  \STATE $z \gets $ search(right(x), key(x))
  \STATE $z' \gets$ right(z)
  \STATE left($p(z)) \gets z'$
  \STATE $p(z') \gets p(z)$
  \STATE replace $x$ with $z$
  \STATE delete-node(x)
\ENDIF
\end{algorithmic}
\end{algorithm}

\subsubsection{Runtimes}
The worst-case runtime for search is $O(\text{height of tree})$. As both insert and delete call search a constant number of times (once or twice) and otherwise perform $O(1)$ work on top of that, their runtimes are also $O(\text{height of tree})$. In the best case, the height of the tree is $O(\log n)$, e.g., when the tree is completely balanced. However, in the worst case it can be $O(n)$ (a long rightward path, for example). This could happen because insert can increase the height of the tree by 1 every time it is called. Currently our operations do not guarantee logarithmic runtimes. To get $O(\log n)$ height we would need to rebalance our tree. There are many examples of self-balancing BSTs, including AVL trees, red-black trees, splay trees (somewhat different but super cool!), etc. Today, we will talk about red-black trees.

\section{Red-Black Trees}
One of the most popular balanced BST is the red-black tree developed by Guibas and Sedgewick in 1978. In a red-black tree, all leaves are assumed to have NILs as children.

\textbf{Definition}. A red-black tree is a BST with the following additional properties:

\begin{enumerate}
  \item Every node is red or black
  \item The root is black
  \item NILs are black
  \item The children of a red node are black
  \item For every node $x$, all $x$ to NIL paths have the same number of black nodes on them
\end{enumerate}

\textit{Example}  (B means a node is black, R means a node is red.) The following tree is not a red-black tree since property 5 is not satisfied:

\begin{center}
\begin{tikzpicture}[scale=0.1]
\tikzstyle{every node}+=[inner sep=0pt]
\draw [black] (25.2,-16) circle (3);
\draw (25.2,-16) node {$B$};
\draw [black] (18.5,-24.1) circle (3);
\draw (18.5,-24.1) node {$B$};
\draw [black] (30.5,-24.1) circle (3);
\draw (30.5,-24.1) node {$B$};
\draw [black] (25.2,-33.5) circle (3);
\draw (25.2,-33.5) node {$B$};
\draw [black] (23.29,-18.31) -- (20.41,-21.79);
\fill [black] (20.41,-21.79) -- (21.31,-21.49) -- (20.54,-20.85);
\draw [black] (26.84,-18.51) -- (28.86,-21.59);
\fill [black] (28.86,-21.59) -- (28.84,-20.65) -- (28,-21.19);
\draw [black] (29.03,-26.71) -- (26.67,-30.89);
\fill [black] (26.67,-30.89) -- (27.5,-30.44) -- (26.63,-29.94);
\end{tikzpicture}
\end{center}

\textit{Example}. (B means a node is black, R means a node is red.) The following tree is a
red-black tree since all the properties are satisfied:

\begin{center}
\begin{tikzpicture}[scale=0.1]
\tikzstyle{every node}+=[inner sep=0pt]
\draw [black] (25.2,-16) circle (3);
\draw (25.2,-16) node {$B$};
\draw [black] (18.5,-24.1) circle (3);
\draw (18.5,-24.1) node {$B$};
\draw [black] (30.5,-24.1) circle (3);
\draw (30.5,-24.1) node {$B$};
\draw [black] (25.2,-33.5) circle (3);
\draw (25.2,-33.5) node {$R$};
\draw [black] (23.29,-18.31) -- (20.41,-21.79);
\fill [black] (20.41,-21.79) -- (21.31,-21.49) -- (20.54,-20.85);
\draw [black] (26.84,-18.51) -- (28.86,-21.59);
\fill [black] (28.86,-21.59) -- (28.84,-20.65) -- (28,-21.19);
\draw [black] (29.03,-26.71) -- (26.67,-30.89);
\fill [black] (26.67,-30.89) -- (27.5,-30.44) -- (26.63,-29.94);
\end{tikzpicture}
\end{center}

\textit{Example}. (B means a node is black, R means a node is red.) The following tree is not a
red-black tree since property 4 is not satisfied:

\begin{center}
\begin{tikzpicture}[scale=0.1]
\tikzstyle{every node}+=[inner sep=0pt]
\draw [black] (28.3,-15.5) circle (3);
\draw (28.3,-15.5) node {$B$};
\draw [black] (19.8,-23) circle (3);
\draw (19.8,-23) node {$B$};
\draw [black] (37.7,-23) circle (3);
\draw (37.7,-23) node {$R$};
\draw [black] (33,-30.6) circle (3);
\draw (33,-30.6) node {$R$};
\draw [black] (28.3,-38.2) circle (3);
\draw (28.3,-38.2) node {$R$};
\draw [black] (26.05,-17.48) -- (22.05,-21.02);
\fill [black] (22.05,-21.02) -- (22.98,-20.86) -- (22.32,-20.11);
\draw [black] (30.65,-17.37) -- (35.35,-21.13);
\fill [black] (35.35,-21.13) -- (35.04,-20.24) -- (34.42,-21.02);
\draw [black] (36.12,-25.55) -- (34.58,-28.05);
\fill [black] (34.58,-28.05) -- (35.42,-27.63) -- (34.57,-27.11);
\draw [black] (31.42,-33.15) -- (29.88,-35.65);
\fill [black] (29.88,-35.65) -- (30.72,-35.23) -- (29.87,-34.71);
\end{tikzpicture}
\end{center}

\textit{Example}. (B means a node is black, R means a node is red.) The following tree is a
red-black tree:

\begin{center}
\begin{tikzpicture}[scale=0.1]
\tikzstyle{every node}+=[inner sep=0pt]
\draw [black] (28.3,-15.5) circle (3);
\draw (28.3,-15.5) node {$B$};
\draw [black] (19.8,-23) circle (3);
\draw (19.8,-23) node {$R$};
\draw [black] (37.7,-23) circle (3);
\draw (37.7,-23) node {$R$};
\draw [black] (33,-30.6) circle (3);
\draw (33,-30.6) node {$B$};
\draw [black] (11.6,-30.2) circle (3);
\draw (11.6,-30.2) node {$B$};
\draw [black] (23.9,-30.2) circle (3);
\draw (23.9,-30.2) node {$B$};
\draw [black] (43.4,-30.1) circle (3);
\draw (43.4,-30.1) node {$B$};
\draw [black] (48.5,-37.4) circle (3);
\draw (48.5,-37.4) node {$R$};
\draw [black] (26.05,-17.48) -- (22.05,-21.02);
\fill [black] (22.05,-21.02) -- (22.98,-20.86) -- (22.32,-20.11);
\draw [black] (30.65,-17.37) -- (35.35,-21.13);
\fill [black] (35.35,-21.13) -- (35.04,-20.24) -- (34.42,-21.02);
\draw [black] (36.12,-25.55) -- (34.58,-28.05);
\fill [black] (34.58,-28.05) -- (35.42,-27.63) -- (34.57,-27.11);
\draw [black] (17.55,-24.98) -- (13.85,-28.22);
\fill [black] (13.85,-28.22) -- (14.79,-28.07) -- (14.13,-27.32);
\draw [black] (21.28,-25.61) -- (22.42,-27.59);
\fill [black] (22.42,-27.59) -- (22.45,-26.65) -- (21.59,-27.15);
\draw [black] (39.58,-25.34) -- (41.52,-27.76);
\fill [black] (41.52,-27.76) -- (41.41,-26.82) -- (40.63,-27.45);
\draw [black] (45.12,-32.56) -- (46.78,-34.94);
\fill [black] (46.78,-34.94) -- (46.73,-34) -- (45.91,-34.57);
\end{tikzpicture}
\end{center}

\textbf{Claim} \textit{Any valid red-black tree on $n$ nodes (non-NIL) has height $\leq 2 \log_2(n + 1) = O(\log n)$.}

\textit{Proof} For some node $x$, let $b(x)$ be the ``black height'' of $x$, which is the number of black nodes on a $x \to$ NIL path excluding $x$. We first show that the number of non-NIL descendants of $x$ is at least $2^
{b(x)} - 1$ (including $x$) via induction on the height of $x$.

Base case: NIL node has $b(x) = 0$ and $2^{0}- 1 = 0$ non-NIL descendants. \checkmark

For our inductive step, let $d(x)$ be the number of non-NIL descendants of $x$. Then
\begin{align*}
  d(x) &= 1 + d(\text{left}(x)) + d(\text{right}(x)) \\
  &\geq 1 + (2^{b(x) - 1} - 1) + (2^{b(x) - 1} - 1) \tag{by induction} \\
  &= 2^{b(x)} - 1 \checkmark
\end{align*}

Notice that $b(x) \geq \frac{h(x)}{2}$ (where $h(x)$ is the height of $x$) since on any root to NIL path there are no two consecutive red nodes, so the number of black nodes is at least the number of red nodes, and hence the black height is at least half of the height. We apply this and the above inequality to the root $r$ (letting $h = h(r )$) to obtain $n \geq 2^{b(r)} - 1 \geq 2^{\frac{h}{2}} - 1$, and hence $h \leq 2 log(n + 1)$. Here is some intuition on why the tree is roughly balanced.

\textit{Intuition}: By Property (5) of a red-black tree, all $r \to$ NIL paths have $b(r)$ black nodes (excluding the root). Therefore, all these paths have length $\geq b(r)$. However, they also have length $\leq 2 \cdot b(r)$: by Property (4), the number of red nodes is limited to half of the path, since every red node must be followed by a black node, and hence the number of black nodes is at least half of the length of the path. Hence, the lengths of all paths from $r$ to a NIL are within a factor of $2$ of each other, and the tree must be reasonably balanced. Today we will take a brief look at how the red-black tree properties are maintained. Our coverage here is detailed, but not comprehensive, and meant as a case study. For complete coverage, please refer to Chapter 13 of CLRS.

\subsection{Rotations} Red-black trees, as do other balanced BSTs, use a concept called rotation. A tree rotation restructures the tree shape locally, usually for the purpose of balancing the tree better. A rotation preserves the BST property (as shown in the following two diagrams). Notably, tree rotations can be performed in $O(1)$ time.

\begin{center}
\begin{tikzpicture}[scale=0.15]
\tikzstyle{every node}+=[inner sep=0pt]
\draw [black] (28.3,-15.5) circle (3);
\draw (28.3,-15.5) node {$x$};
\draw [black] (19.8,-23) circle (3);
\draw (19.8,-23) node {$y$};
\draw [black] (37.7,-23) circle (3);
\draw (37.7,-23) node {$\gamma$};
\draw [black] (11.6,-30.2) circle (3);
\draw (11.6,-30.2) node {$\alpha$};
\draw [black] (23.9,-30.2) circle (3);
\draw (23.9,-30.2) node {$\beta$};
\draw [black] (26.05,-17.48) -- (22.05,-21.02);
\fill [black] (22.05,-21.02) -- (22.98,-20.86) -- (22.32,-20.11);
\draw [black] (30.65,-17.37) -- (35.35,-21.13);
\fill [black] (35.35,-21.13) -- (35.04,-20.24) -- (34.42,-21.02);
\draw [black] (17.55,-24.98) -- (13.85,-28.22);
\fill [black] (13.85,-28.22) -- (14.79,-28.07) -- (14.13,-27.32);
\draw [black] (21.28,-25.61) -- (22.42,-27.59);
\fill [black] (22.42,-27.59) -- (22.45,-26.65) -- (21.59,-27.15);
\end{tikzpicture}
\end{center}

\begin{center}
\begin{tikzpicture}[scale=0.15]
\tikzstyle{every node}+=[inner sep=0pt]
\draw [black] (28.3,-15.5) circle (3);
\draw (28.3,-15.5) node {$y$};
\draw [black] (19.8,-23) circle (3);
\draw (19.8,-23) node {$\alpha$};
\draw [black] (37.7,-23) circle (3);
\draw (37.7,-23) node {$x$};
\draw [black] (31.6,-31.9) circle (3);
\draw (31.6,-31.9) node {$\beta$};
\draw [black] (43.6,-31.9) circle (3);
\draw (43.6,-31.9) node {$\gamma$};
\draw [black] (26.05,-17.48) -- (22.05,-21.02);
\fill [black] (22.05,-21.02) -- (22.98,-20.86) -- (22.32,-20.11);
\draw [black] (30.65,-17.37) -- (35.35,-21.13);
\fill [black] (35.35,-21.13) -- (35.04,-20.24) -- (34.42,-21.02);
\draw [black] (36,-25.47) -- (33.3,-29.43);
\fill [black] (33.3,-29.43) -- (34.16,-29.05) -- (33.34,-28.48);
\draw [black] (39.36,-25.5) -- (41.94,-29.4);
\fill [black] (41.94,-29.4) -- (41.92,-28.46) -- (41.08,-29.01);
\end{tikzpicture}
\end{center}

Moving from the first tree to the second is known as a right rotation of $x$. The other direction (from the second tree to the first) is a left rotation of $y$. Notice that we only move the $\beta$ subtree, which is why we preserve the BST property.

\subsection{Insertion in a Red-Black Tree}
Let's see how we can perform Insert(i) on a red-black tree while still maintaining all of its properties. The process for inserting a new node is initially similar to that of insertion into any BST.

\begin{algorithm}
\caption{\texttt{insert\_rb}(i)}\label{alg:insert_rb}
\begin{algorithmic}
\STATE $x \gets $ search(i)
\STATE $y \gets$ new node with key$(y) \gets i$, left$(y) \gets$ NIL, right($y) \gets$ NIL, $p(y) \gets x$
\IF {i < key(x)}
  \STATE left$(x) \gets y$
\ELSE
  \STATE right$(x) \gets y$
\ENDIF
\STATE color(x) $\gets $ red
\STATE recolor if needed
\end{algorithmic}
\end{algorithm}

Note that when $x$ is inserted as a red node: Property (1) is satisfied, as we colored the new node red; Property (2) is satisfied, as we did not touch the root; Property (3) is satisfied, as we can color the new NILs black; and Property (5) is satisfied, as we did not change the number of black nodes in the tree. Thus, the only invariant we have to worry about is Property (4), that red nodes have black children.

The recoloring step is broken down into multiple cases. We consider each of them:

Case 1: p is black. In this case, Property (4) is also maintained. So, we simply add $x$ as a new red child of p, and the red-black tree properties are maintained.

\begin{center}
\begin{tikzpicture}[scale=0.2]
\tikzstyle{every node}+=[inner sep=0pt]
\draw [black] (28.3,-15.5) circle (3);
\draw (28.3,-15.5) node {$p\mbox{ }(B)$};
\draw [black] (19.7,-23) circle (3);
\draw (19.7,-23) node {$x\mbox{ }(R)$};
\draw [black] (26.04,-17.47) -- (21.96,-21.03);
\fill [black] (21.96,-21.03) -- (22.89,-20.88) -- (22.24,-20.13);
\end{tikzpicture}
\end{center}

Case 2: p is red, and $x$'s uncle $u$ is red. In this case, we insert a red $x$, change p and $u$ to black, and change p' to red. Because we switched the colors of two nodes on each of these paths (one red $\to$ black and one black $\to$ red), the number of black nodes on each path is unchanged, so Property (5) remains unchanged. If the parent of p' , p'', is black, then Property (4) is maintained. Otherwise, if p'' is red and breaks Property (4) by introducing a ``double-red'' pair of nodes (p' and p''), then we have to recolor recursively starting at p'.

\begin{center}
\begin{tikzpicture}[scale=0.2]
\tikzstyle{every node}+=[inner sep=0pt]
\draw [black] (28.3,-15.5) circle (3);
\draw (28.3,-15.5) node {$p'\mbox{ }(B)$};
\draw [black] (19.7,-23) circle (3);
\draw (19.7,-23) node {$p\mbox{ }(R)$};
\draw [black] (16,-31.6) circle (3);
\draw (16,-31.6) node {$x\mbox{ }(R)$};
\draw [black] (24.6,-31.2) circle (3);
\draw (24.6,-31.2) node {$...$};
\draw [black] (36.5,-22.3) circle (3);
\draw (36.5,-22.3) node {$u\mbox{ }(R)$};
\draw [black] (26.04,-17.47) -- (21.96,-21.03);
\fill [black] (21.96,-21.03) -- (22.89,-20.88) -- (22.24,-20.13);
\draw [black] (18.51,-25.76) -- (17.19,-28.84);
\fill [black] (17.19,-28.84) -- (17.96,-28.31) -- (17.04,-27.91);
\draw [black] (21.24,-25.58) -- (23.06,-28.62);
\fill [black] (23.06,-28.62) -- (23.08,-27.68) -- (22.22,-28.19);
\draw [black] (30.61,-17.42) -- (34.19,-20.38);
\fill [black] (34.19,-20.38) -- (33.89,-19.49) -- (33.26,-20.26);
\end{tikzpicture}
\end{center} becomes \begin{center}
\begin{tikzpicture}[scale=0.2]
\tikzstyle{every node}+=[inner sep=0pt]
\draw [black] (28.3,-15.5) circle (3);
\draw (28.3,-15.5) node {$p'\mbox{ }(R)$};
\draw [black] (19.7,-23) circle (3);
\draw (19.7,-23) node {$p\mbox{ }(B)$};
\draw [black] (16,-31.6) circle (3);
\draw (16,-31.6) node {$x\mbox{ }(R)$};
\draw [black] (24.6,-31.2) circle (3);
\draw (24.6,-31.2) node {$...$};
\draw [black] (36.5,-22.3) circle (3);
\draw (36.5,-22.3) node {$u\mbox{ }(B)$};
\draw [black] (26.04,-17.47) -- (21.96,-21.03);
\fill [black] (21.96,-21.03) -- (22.89,-20.88) -- (22.24,-20.13);
\draw [black] (18.51,-25.76) -- (17.19,-28.84);
\fill [black] (17.19,-28.84) -- (17.96,-28.31) -- (17.04,-27.91);
\draw [black] (21.24,-25.58) -- (23.06,-28.62);
\fill [black] (23.06,-28.62) -- (23.08,-27.68) -- (22.22,-28.19);
\draw [black] (30.61,-17.42) -- (34.19,-20.38);
\fill [black] (34.19,-20.38) -- (33.89,-19.49) -- (33.26,-20.26);
\end{tikzpicture}
\end{center}

Case 3: p is red, and $u$ is black. There are two possibilities here:

\begin{enumerate}
  \item We are inserting $x$ as a leaf node. Then, $u$ must be NIL for the red-black tree to have been valid before inserting $x$. We insert a red $x$.
  \item We are not inserting $x$; rather, we are recoloring the tree at $x$ from the recursive call in Case 2. We aim to recolor the tree and maintain the number of black nodes on each path from the root to NIL. Note that in this case, $x$ actually has nodes under it.
\end{enumerate}

In both cases, $x$ is red, so we make p black, make p' red, and do a right rotation at p' . We can see that this also maintains the same number of black nodes on each path from the root to NIL, and satisfies Property (4) below p because the original tree was a red-black tree.

\begin{center}
\begin{tikzpicture}[scale=0.2]
\tikzstyle{every node}+=[inner sep=0pt]
\draw [black] (28.3,-15.5) circle (3);
\draw (28.3,-15.5) node {$p'\mbox{ }(B)$};
\draw [black] (19.7,-23) circle (3);
\draw (19.7,-23) node {$p\mbox{ }(R)$};
\draw [black] (16,-31.6) circle (3);
\draw (16,-31.6) node {$x\mbox{ }(R)$};
\draw [black] (24.6,-31.2) circle (3);
\draw (24.6,-31.2) node {$...$};
\draw [black] (36.5,-22.3) circle (3);
\draw (36.5,-22.3) node {$u\mbox{ }(B)$};
\draw [black] (26.04,-17.47) -- (21.96,-21.03);
\fill [black] (21.96,-21.03) -- (22.89,-20.88) -- (22.24,-20.13);
\draw [black] (18.51,-25.76) -- (17.19,-28.84);
\fill [black] (17.19,-28.84) -- (17.96,-28.31) -- (17.04,-27.91);
\draw [black] (21.24,-25.58) -- (23.06,-28.62);
\fill [black] (23.06,-28.62) -- (23.08,-27.68) -- (22.22,-28.19);
\draw [black] (30.61,-17.42) -- (34.19,-20.38);
\fill [black] (34.19,-20.38) -- (33.89,-19.49) -- (33.26,-20.26);
\end{tikzpicture}
\end{center}

becomes

\begin{center}
\begin{tikzpicture}[scale=0.2]
\tikzstyle{every node}+=[inner sep=0pt]
\draw [black] (28.3,-15.5) circle (3);
\draw (28.3,-15.5) node {$p\mbox{ }(B)$};
\draw [black] (19.7,-23) circle (3);
\draw (19.7,-23) node {$x\mbox{ }(R)$};
\draw [black] (36.5,-22.3) circle (3);
\draw (36.5,-22.3) node {$p'(R)$};
\draw [black] (30.8,-29.4) circle (3);
\draw (30.8,-29.4) node {$...$};
\draw [black] (42.4,-29.4) circle (3);
\draw (42.4,-29.4) node {$u\mbox{ }(B)$};
\draw [black] (26.04,-17.47) -- (21.96,-21.03);
\fill [black] (21.96,-21.03) -- (22.89,-20.88) -- (22.24,-20.13);
\draw [black] (30.61,-17.42) -- (34.19,-20.38);
\fill [black] (34.19,-20.38) -- (33.89,-19.49) -- (33.26,-20.26);
\draw [black] (34.62,-24.64) -- (32.68,-27.06);
\fill [black] (32.68,-27.06) -- (33.57,-26.75) -- (32.79,-26.12);
\draw [black] (38.42,-24.61) -- (40.48,-27.09);
\fill [black] (40.48,-27.09) -- (40.36,-26.16) -- (39.59,-26.8);
\end{tikzpicture}
\end{center}

If we end up in Case 2 and recursively call recolor, then in the worst case the recursion will bottom out when we hit the root, with a constant number of relabelings and rotations ateach level. So, it will be an $O(h)$ operation overall, where $h$ is the height of the tree.

In the analysis above, we considered the cases where $x$ is a left child of $p$ and $u$ is a rightchild of its parent $p'$. These cases are representative, showing most of the machinery that we’ll need to insert an arbitrary element into an arbitrary red-black tree. (Within Case 2 and Case 3, there are actually a total of four cases each, where p's tree and p`'s children could each be swapped, but the recoloring procedure is similar. You are encouraged to read the text for details.)

To summarize, the following is the algorithm for recoloring, in the case where $x$ is a left childand $u$ is a right child.

\begin{algorithm}
\caption{\texttt{recolor}(x)}\label{alg:recolor_rb}
\begin{algorithmic}
\STATE $p \gets $ parent(x)
\IF {black = color(p))}
  \RETURN
\ENDIF

\STATE $p' \gets \text{parent}(p)$
\STATE $u \gets \text{right}(p')$
\IF {red = color(u) }
  \STATE color(p) $\gets $ black
  \STATE color(u) $\gets $ black
  \STATE color(p') $\gets $ red
  \STATE recolor(p')
\ELSIF {black = color(u)}
  \STATE color(p) $\gets $ black
  \STATE color(p') $\gets $ red
  \STATE \texttt{right\_rotate}(p')
\ENDIF
\end{algorithmic}
\end{algorithm}

Based on our analysis above, we can update our red-black trees in $O(h)$ time upon insertion, where $h$ is the height of the tree. The other operations are similar, and also give the guarantee of worst-case performance of $O(h)$ search, insertion, and deletion. Together with our claim that $h = O(\log n)$, we get:

\textbf{Claim} \textit{Red-Black trees support \texttt{insert}, \texttt{delete}, and \texttt{search} in $O(\log n)$ time}

As we have seen, BSTs are very nice – they allow us to maintain a set and report membership, insert, and delete in $O(\log n)$ time. In addition to these basic underlying operations, we can also support other types of queries efficiently. Because the elements are stored maintaining the binary search tree property, we can search for the next largest element or the elements on a range very efficiently. But what if we don’t care about these properties? What if we only need to support membership queries? Can we improve our performance of $O(\log n)$ time to nearly constant time? This question motivates our discussion of hash tables, which we will cover in the next lecture.

\end{document}


\section{Partial Redundancy Elimination}
\begin{flushright}
\textit{Notes by Akshin Singh}
\end{flushright}

A particularly good place to optimize is in loops. Since loops can run for a million iterations, moving any computation from
inside the loop to outside while maintaing correctness is a desirable optimization. Such kind of optimization is called loop invariant
code motion.

\begin{figure}[h]
\centering
\includegraphics[scale = 0.5]{images/mod_105_fig1.png}
\caption{Loop invariant Code Motion Example}
\label {fig:mod_105_01}
\end{figure}

Let us explain with the help of an example. See the original code present in Figure \ref{fig:mod_105_01}. Since i is the loop variable,
that gets incremented every iteration, both \textbf{A[i]} and \textbf{A[i+1]} are dependent on the loop's progress. Hence these variables
are not invariant. However, the expression \textbf{m+n} and \textbf{A[n-1]} are both loop invariant. This is because both n and m hold their
values across iterations. We can therefore transform the code as shown.

Notice that since we have removed an addition as well as an address calculation (which is a multiply and add instruction), we have reduced
the number of instructions in the loop by two. Now if the loop runs a million times, we have saved a lot of work (typically on the order of
two to three hunderd cycles).

What happens if the loop does not run even once? Our transformed code will then have two extra instructions and we have inadvertently slowed
ourseleves by optimizing. This is the tradeoff that an optimizing compiler decides to make or not.

\subsection{Partial Redundancy Elimination}

Let us take a deeper look into why we wanted to transform code in \ref{fig:mod_105_01}. Let us try to follow the control of the program.
We enter the program and then enter the loop. Inside the loop we calculate \textbf{m+n} and \textbf{A[n-1]} in the first iteration.
Then, before the values of m and n are changed we subsequently calculated \textbf{m+n} and \textbf{A[n-1]} again in the next iteration.

Keeping this in mind, let us define the problem of Partial Redundancy. If an expression is redundant (we will
define redundancy in a while) on any path leading to it, it is said to be partially redundant. This is not the case in Common subexpression
elimination, where we can remove only those expressions which are completely redundant, i.e., redundant on all paths.

\begin{figure}[h]
\centering
\includegraphics[scale = 0.5]{images/mod_105_fig2.png}
\caption{Partial Redundancy Elimination Example}
\label {fig:mod_105_02}
\end{figure}

A clearer example can be seen in Figure \ref{fig:mod_105_02}. Here \textbf{b+c} is redundant on the path from \textbf{a=b+c} and not on the
other path. The transformed code has no redundancy. Partial Redundancy Elimination (or PRE for short) is the problem of placing computations (like \textbf{a=b+c}) such that no path recomputes
the same expression.

\begin {itemize}
\item Since PRE is a general optimization that does not need to work on only loops, PRE subsumes Loop invariant code motion. This is because
any computation that is loop invariant is redundant on the path that goes from the loop body back to the loop body.

\begin{figure}[h]
\centering
\includegraphics[scale = 0.3]{images/mod_105_fig3.png}
\caption{Redundancy Example}
\label {fig:mod_105_03}
\end{figure}

\item PRE may require addition of basic blocks. To understand this we need to define redundancy. Let us first take an example. Figure
\ref{fig:mod_105_03}(a) has a redundancy on the green path. If the computation is moved as shown in \ref{fig:mod_105_03}(b), we have
a problem on the red path. The problem is that \textbf{a+b} is not needed anywhere on that path, and yet it is being computed. Naturally we
only want those computations to be present on a path that will be used eventually on that path. This leads us to define redundancy as
any computation on a path that is either happening more than once if needed or at least once if not needed. To solve the problem, we need
to add a basic block as shown in \ref{fig:mod_105_03}(c). Only then can we remove all partial redundancies.

Formally, the problem arised from the fact that we are hoisting computation across a \textbf{critical edge}. A critical edge is an edge
in the CFG where the source block has multiple successors and the desitination block has multiple predecessors. Hoisting across a critical
edge requires a new basic block since we may add redundancy on other paths. The pruple edge in \ref{fig:mod_105_03}(c) is the critical edge.

\item Even after adding basic blocks, there can be cases where we cannot elimate PRE until we decide to duplicate the code. See the
Figure \ref{fig:mod_105_04}(a).

\begin{figure}[h]
\centering
\includegraphics[scale = 0.3]{images/mod_105_fig4.png}
\caption{PRE Duplication Example}
\label {fig:mod_105_04}
\end{figure}

Here the expression \textbf{b+c} is redundant on the red path as shown. Here we need to duplicate the code as shown in
Figure \ref{fig:mod_105_04}(b). Code duplication has caching effects. A larger code-size means a jump can be a jump to
a far instruction which may incur a cache-miss which is not desirable, especially in today's out-of-order 6-8 wide issue processors.
The penalty for i-cache miss can be very high.
\end{itemize}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsfonts, amsmath, enumitem, float, graphicx, hyperref}

\title{Artificial Intelligence - Notes}
\author{Matteo Alberici}
\date{January 2022}

\begin{document}

\maketitle
\newpage
\tableofcontents
\newpage

% ------------------------------------- %
% Section 1: Search Problems Definition %
% ------------------------------------- %
\section{Search Problems}
In order to define a \textbf{search problem}, we need to introduce the following terms:
\begin{itemize}
    \item \textbf{States}: the set of possible configurations
    \item \textbf{Initial state}: the state where the search starts
    \item \textbf{Actions}: the set of possible actions
    \item \textbf{Goal}: the destination of the search
    \item \textbf{Cost function}: gives a cost to the solution path
\end{itemize}
\subsection{Graph Definition}
A search problem is modelled as a \textbf{graph} having the following finite sets:
\begin{center}
    \textbf{nodes}: $(n_1, n_2, ..., n_n)$ \\
    \vspace{0.2cm}
    \textbf{edges}: $((n_i, n_j), ..., (n_k, n_n))$
\end{center}
Given a direct edge $(n_i, n_j)$, we say that $n_j$ is a \textbf{child} of $N_i$. If a node has no children, then it is a \textbf{leaf}. \\
A \textbf{path} $(n_i, n_{i+1}, ..., n_n)$ is a sequence with each couple $n_i, n_{i+1}$ being an edge. If a path contains the same node more times, then it is a \textbf{cycle}. \\
A graph is \textbf{rooted} when a node $n_i$ is the origin of all the paths. Moreover, two nodes are \textbf{connected} if there exists a path from one to the other.
\subsection{Search Algorithms}
\textbf{Search algorithms} take as input a state space and a starting state, then tries to compute a path producing a search tree. The strategy consists of searching which node to add to the \textbf{fringe} and \textbf{expand}, considering all nodes reachable in one action from the current one.
\subsection{Search Tree}
A node in the \textbf{search tree} has five components:
\begin{itemize}
    \item The state
    \item The node that generated it
    \item The action that generates it
    \item The depth of the tree
    \item The cost of the path from the root
\end{itemize}
\subsection{General Search Algorithm}
The following is the procedure of a \textbf{general search algorithm} which takes as input a problem and a strategy and returns a sequence of operators that brings to the goal:
\begin{enumerate}
    \item Initialize the search tree
    \item Push the root node to the fringe
    \item Start the loop
    \item If the fringe is empty, then return failure
    \item Choose the next node according to a strategy
    \item If the node is the goal, then return the solution
    \item Expand the node
    \item Add the children to the fringe
\end{enumerate}
\subsection{Evaluation of Search Strategies}
The strategy determines the order of expansion and is evaluated according to the following criteria:
\begin{itemize}
    \item \textbf{Completeness}: does it always find a solution if it exists?
    \item \textbf{Space Complexity}: how much memory does it require?
    \item \textbf{Time Complexity}: how long does it take to find a solution?
    \item \textbf{Optimality}: does it guarantee the least-cost solution?
\end{itemize}
Time and space complexities are measured in terms of the maximum branching factor $b$, the depth of the least-cost solution $d$, and the maximum depth of the tree $m$. \\
\textbf{Polynomial-time algorithms} solve problems in a time growing polynomially with the input size. If such an algorithm does not solve the problem, then the problem is a \textbf{nondeterministic-polynomial time problem}.

\newpage

% ---------------------------------- %
% Section 2: Blind Search Algorithms %
% ---------------------------------- %
\section{Blind Search Algorithms}
\textbf{Blind search algorithms} use non informed search strategies.
% Algorithm 1: Breadth-first Search %
\subsection{Breadth-first Search Algorithm}
The \textbf{breadth-first search algorithm} expands the shallowest unexpanded node using a \textbf{FIFO} queue: nodes at distance $d$ are expanded before nodes at distance $d + 1$. \\
The procedure of the algorithm is the following:
\begin{enumerate}
    \item Push the root node to the fringe
    \item If the fringe is empty, then return failure
    \item Remove the first node $n$ from the fringe
    \item If $n$ is the goal, then return the solution
    \item If $n$ is in the explored list, then repeat from step 2
    \item Expand $n$
    \item Add the children at the end of the fringe
    \item Add $n$ to the explored list
    \item Repeat from step 2
\end{enumerate}
\vspace{0.2cm}
The algorithm is evaluated as follows:
\begin{itemize}
    \item Completeness: yes
    \item Space Complexity: $O(b^d)$ - all nodes until goal depth are created
    \item Time Complexity: $O(b^d)$ - all nodes until goal depth are created
    \item Optimality: yes, if all steps have the same cost $g(n) = k \cdot depth(n)$
\end{itemize}
\newpage
% Algorithm 2: Uniform Cost Search %
\subsection{Uniform Cost Search Algorithm}
The \textbf{uniform cost search algorithm} associates a cost $g(n)$ to each node and expands the one with the lowest $g(n)$ in the fringe, which is kept sorted in increasing cost order. \\
The procedure of the algorithm is the following:
\begin{enumerate}
    \item Push the root node to the fringe
    \item If the fringe is empty, then return failure
    \item Remove the first node $n$ from the fringe
    \item If $n$ is the goal, then return the solution
    \item If $n$ is in the explored list, then repeat from step 2
    \item Expand $n$
    \item Add the children not in the explored list to the fringe
    \item Sort the fringe by the path cost
    \item Add $n$ to the explored list
    \item Repeat from step 2
\end{enumerate}
If $g(n)$ is equal to the depth of node $n$, then the procedure is the same of the breadth-first search algorithm. \\
The algorithm is evaluated as follows:
\begin{itemize}
    \item Completeness: yes, if the step cost $\geq \epsilon > 0$
    \item Space Complexity: $O(b^d)$, number of nodes with $g(n) \leq g(\text{solution})$
    \item Time Complexity: $O(b^d)$, number of nodes with $g(n) \leq g(\text{solution})$
    \item Optimality: yes, if all costs are non-negative
\end{itemize}
\newpage
% Algorithm 3: Depth-first Search %
\subsection{Depth-first Search Algorithm}
The \textbf{depth-first search algorithm} expands a node generated in the previous step using a \textbf{LIFO} queue. \\
The procedure of the algorithm is the following:
\begin{enumerate}
    \item Push the root node to the fringe
    \item If the fringe is empty, then return failure
    \item Remove the first node $n$ from the fringe
    \item If $n$ is the goal, then return the solution
    \item If $n$ is in the explored list, then repeat from step 2
    \item Expand $n$
    \item Add the children not in the explored list to the beginning of the fringe
    \item Add $n$ to the explored list
    \item Repeat from step 2
\end{enumerate}
The algorithm is evaluated as follows:
\begin{itemize}
    \item Completeness: no, it could proceed to infinity
    \item Space Complexity: $O(b \cdot m)$
    \item Time Complexity: $O(b^m)$
    \item Optimality: no, cannot be sure the first goal is the optimal one
\end{itemize}
\newpage
% Algorithm 4: Depth-limited Search %
\subsection{Depth-limited Search Algorithm}
The \textbf{depth-limited search algorithm} offers a solution to the completeness problem of the depth-first search algorithm by taking a parameter $deep$ in input. \\
The procedure of the algorithm is the following:
\begin{enumerate}
    \item Push the root node to the fringe
    \item If the fringe is empty, then return failure
    \item Remove the first node $n$ from the fringe
    \item If $n$ is the goal, then return the solution
    \item If $n$ not\_in(closed) and $d(n) < deep$, go ahead; otherwise repeat from step $2$
    \item Expand $n$
    \item Add the children not in the explored list to the beginning of the fringe
    \item Add $n$ to the explored list
    \item Repeat from step 2
\end{enumerate}
We can only reach solutions that are at distance $deep$ and we need knowledge to set a fixed $deep$. \\
The algorithm is evaluated as follows:
\begin{itemize}
    \item Completeness: yes, if we know a bound to the solution depth
    \item Space Complexity: $O(b \cdot deep)$
    \item Time Complexity: $O(b^{deep})$
    \item Optimality: no
\end{itemize}
% Algorithm 5: Iterative-deepening Search %
\subsection{Iterative-deepening Search Algorithm}
The \textbf{iterative-deepening search algorithm} is a depth-limited search where the depth is incremented step by step. \\
The procedure of the algorithm is the following:
\begin{enumerate}
    \item Loop for depth $= 0$ to $\infty$
    \item If Depth-Limited-Search(root, deep) $==$ solution, then return the solution
    \item Return failure
\end{enumerate}
It is the best compromise between breadth-first search and depth-first search. \\
The algorithm is evaluated as follows:
\begin{itemize}
    \item Completeness: yes, if we know a bound to the solution depth
    \item Space Complexity: $O(b \cdot d)$
    \item Time Complexity: $O(b^d)$
    \item Optimality: yes, if all costs are non-negative
\end{itemize}
\newpage
% Algorithm 6: Bidirectional Search %
\subsection{Bidirectional Search}
A problem related to search is its direction: in forward search, we start from the initial state and proceed to the goal, while in backward search, we start from the goal and go back to the initial state. The \textbf{bidirectional search algorithm} proceeds in both directions until the queues contain a common node. \\
The procedure of the algorithm is the following:
\begin{enumerate}
    \item Initialize two queues, one for forward and one for backward search
    \item Remove the first path from both the queues
    \item Create new paths to the children
    \item Reject a path if it contains a cycle
    \item Add the new paths to the queues
    \item Repeat from step 2 until the queues are not empty and do not share a node
    \item If the queues do not share a node, then return the solution
    \item Return Failure
\end{enumerate}
The algorithm is evaluated as follows:
\begin{itemize}
    \item Completeness: yes
    \item Space Complexity: $O(b^{d/2})$
    \item Time Complexity: $O(b^{d/2})$
    \item Optimality: yes
\end{itemize}

\newpage

% -------------------------------------- %
% Section 3: Heuristic Search Algorithms %
% -------------------------------------- %
\section{Heuristic Search Algorithms}
Exhaustive blind search is not applicable for problems with exponential complexity. \textbf{Heuristic search algorithms} use problem specific strategies with some knowledge and experience to identify the most promising paths. These algorithms do not guarantee completeness and optimality, but find a good solution in a short time.
\subsection{Heuristic Evaluation}
The \textbf{heuristic evaluation} estimates the distance from a state to the goal:
\begin{center}
    $h : s \rightarrow R \ \ $ such that $\ \ h(goal) = 0$
\end{center}
\subsubsection{Admissible Heuristic}
In \textbf{admissible heuristic}, the cost estimated to reach the goal is not higher than the lowest possible cost from the current node. Given a node $n$, a cost $h(n)$ from $n$ to the goal, an optimal cost $h*(n)$ from $n$ to the goal, then we have the following relation:
\begin{center}
    $\forall n \ | \ h(n) \leq h*(n)$
\end{center}
\subsubsection{Monotonic Heuristic}
In \textbf{monotone heuristic}, for each node $n$ and its successor $n'$, the estimated cost of reaching the goal from a node $n$ is no longer greater than the step cost of getting its successor $n'$ plus the estimated cost of reaching the goal from $n'$. \\
Let $f(n)$ be the cost from the root to node $n$ plus the cost estimation from $n$ to the goal, then we have the following relation:
\begin{center}
    $f(n) \leq f(n')$
\end{center}
All monotonic heuristics guarantee local optimality and are admissible, but not all admissible heuristics are monotonic.
\newpage
% Algorithm 7: Best-first Search %
\subsection{Best-first Search Algorithm}
The \textbf{best-first search algorithm} evaluates each node estimating its distance to the goal, then sorts the fringe according to the estimated distances: the node with the best heuristic value is expanded. \\
The procedure of the algorithm is the following:
\begin{enumerate}
    \item Push the root node to the fringe
    \item If the fringe is empty, then return failure
    \item Remove the first node $n$ from the fringe
    \item If $n$ is the goal, then return the solution
    \item If $n$ is in the explored list, then repeat from step $2$
    \item Expand $n$
    \item Add the children not in the explored list to the fringe
    \item Add $n$ to the explored list
    \item Sort the fringe based on the heuristic values
    \item Repeat from step 2
\end{enumerate}
All nodes on the border are kept in memory. An alternative is represented by the beam-search algorithm that keeps in memory only the $k$ states with the best evaluation, where $k$ is the search radius. \\
The algorithm is evaluated as follows:
\begin{itemize}
    \item Completeness: no
    \item Space Complexity: $O(b^{m})$
    \item Time Complexity: $O(b^{m})$
    \item Optimality: no
\end{itemize}
\newpage
% Algorithm 8: A* %
\subsection{A* Algorithm}
A algorithms are best-first search algorithms with an evaluation function:
\begin{center}
    $f(n) = g(n) + h(n)$, where $h(n) \geq 0$ and $h(goal) = 0$
\end{center}
There exist some particular cases:
\begin{itemize}
    \item if $f(n) = g(n)$, then it performs a uniform search
    \item if $f(n) = g(n) = depth(n)$, then it performs a breadth-first search
    \item if $f(n) = h(n)$, then it performs a greedy best first search
\end{itemize}
\textbf{A* algorithm} is an A algorithm with an admissible heuristic function:
\begin{center}
    $f*(n) = g*(n) + h*(n)$, where:
\end{center}
\begin{itemize}
    \item $f*(n) =$ estimated cost of the path from the root to the goal
    \item $g*(n) =$ cost of the path from the root to node $n$
    \item $h*(n) =$ estimated cost of the path from node $n$ to the goal
\end{itemize}
The algorithm expands the node with the smallest $f*(n)$. All nodes are kept in memory. \\
The algorithm is evaluated as follows:
\begin{itemize}
    \item Completeness: yes, unless there are infinite nodes with $f \leq f(G)$
    \item Space Complexity: $O(b^m)$
    \item Time Complexity: $O(b^m)$
    \item Optimality: given a goal $p$
        \begin{enumerate}
            \item Assume that a path $p'$ is the shortest path to the goal
            \item Consider that a partial path $p''$ of $p'$ is on the fringe
            \item Since $p$ is expanded before $p''$, then we have:
                \begin{center}
                    $f(p) \leq f(p'')$
                \end{center}
            \item Since $p$ is a goal, then:
                \begin{center}
                    $g(p) \leq g(p'') + h(p'') $
                \end{center}
            \item Since $h$ is admissible, then for any path $p'$ that extends $p''$:
                \begin{center}
                    $g(p'') + h(p'') \leq g(p')$
                \end{center}
            \item Finally, for any path $p'$ to a goal:
                \begin{center}
                    $g(p) \leq g(p')$,
                \end{center}
                which contradicts point $1$.
        \end{enumerate}
\end{itemize}

\newpage

% ----------------------------------------- %
% Section 4: Constructive Greedy Algorithms %
% ----------------------------------------- %
\section{Constructive Greedy Algorithms}
\textbf{Constructive greedy algorithms} take a data set and fastly produce a feasible solution through the following procedure:
\begin{enumerate}
    \item Start from a random node $n$
    \item Expand $n$
    \item Choose the next node according to a local strategy
    \item Extend the solution with the new node, which becomes the new $n$
    \item Repeat from step $2$ until a feasible solution is computed
\end{enumerate}
% Algorithm 9: Nearest Neighbors %
\subsection{Nearest Neighbors Algorithm}
\textbf{Nearest neighbors algorithm} is one of the most common algorithms used to solve TSP problems. \\
Given $n$ cities, the algorithm performs the following procedure,
\begin{enumerate}
    \item Consider a starting tour consisting of a random city $c_i$
    \item Add the closest new city $c_{i+1} $ to the tour
    \item Repeat step $2$ until no new cities are available
\end{enumerate}
The algorithm is not very efficient: the first edges are short while the final ones are long and the length of the tour with respect to the optimal tour length grows following a $log(n)$ formula. \\
The algorithm is evaluated as follows:
\begin{itemize}
    \item Completeness: no
    \item Space Complexity: $O(n^2)$
    \item Time Complexity: $O(n^2)$
    \item Optimality: no, cannot guarantee optimality
\end{itemize}
\newpage
% Algorithm 10: Multi Fragment %
\subsection{Multi Fragment Algorithm}
\textbf{Multi fragment algorithm} starts with the shortest edge and add the others in increasing order only if they do not create a $3-$degree city. \\
The procedure of the algorithm is the following:
\begin{enumerate}
    \item Sort the edge costs in ascending order
    \item Consider a tour with the the first edge
    \item Add a new edge in incremental order only if it does not create a cycle
    \item Repeat step $3$ until no edge is available
\end{enumerate}
The algorithm is evaluated as follows:
\begin{itemize}
    \item Space Complexity: $O(n^2log(n))$
    \item Time Complexity: $O(n^2log(n))$
    \item Optimality: no, cannot guarantee optimality
\end{itemize}
% Algorithm 11: Nearest Insertion %
\subsection{Nearest Insertion Algorithm}
Given $n$ cities, the \textbf{nearest insertion algorithm} performs the following procedure:
\begin{enumerate}
    \item Build an initial tour formed by the nearest cities
    \item Choose a new node such that the Euclidean distance between it and two other nodes in the tour is minimum
    \item Insert the new node between the two nodes of the tour
    \item Repeat from step $2$ until no new nodes are available
\end{enumerate}
The algorithm is evaluated as follows:
\begin{itemize}
    \item Space Complexity: $O(n^2)$
    \item Time Complexity: $O(n^2)$
    \item Optimality: no, cannot guarantee optimality
\end{itemize}
\newpage
% Algorithm 12: Farthest Insertion %
\subsection{Farthest Insertion Algorithm}
Given $n$ cities, the \textbf{farthest insertion algorithm} performs the following procedure:
\begin{enumerate}
    \item Build an initial tour formed by the farthest cities
    \item Choose a new node such that the Euclidean distance between it and two other nodes in the tour is maximum
    \item Insert the new node between the two nodes of the tour
    \item Repeat from step $2$ until no new nodes are available
\end{enumerate}
The algorithm is evaluated as follows:
\begin{itemize}
    \item Space Complexity: $O(n^2)$
    \item Time Complexity: $O(n^2)$
    \item Optimality: no, cannot guarantee optimality
\end{itemize}

\newpage

% ----------------------- %
% Section 5: Local Search %
% ----------------------- %
\section{Local Search Algorithms}
\textbf{Local search algorithms} optimize solutions through the following procedure:
\begin{enumerate}
    \item Start from a complete solution
    \item Compute the best edge exchange that improves the solution
    \item Execute the edge exchange
    \item Repeat from step $2$ until no improvement is possible
\end{enumerate}
% Algorithm 13: Hill Climbing %
\subsection{Hill Climbing Algorithm}
Given a set of solutions $A$ and the following objective function $f$:
\begin{center}
    $\{f(s) \ | \ s \in A \}$,
\end{center}
then, we define a neighborhood function $N$ that computes a subset of solutions $N(s)$ for each solution $s \in A$:
\begin{center}
    $A \ \rightarrow \ 2^A$,
\end{center}
The \textbf{hill climbing algorithm} explores the neighborhood searching for a better solution. \\
We can perform two different types of search: a greedy search and a simplified version. In the greedy search, if the best solution in $N(current)$ is better than the actual best, then we restart from current, otherwise we stop. In the simplified version, as soon as we find a better neighborhood we continue the search from the associated solution, avoiding to visit all the neighborhoods. \\
The procedure of the algorithm is the following:
\begin{verbatim}
    function Hill-Climbing(s, f, N):
        current = s
        while terminating condition is met:
            next = best solution in N(current)
            if f(next) < f(current):
                current = next
        return current
\end{verbatim}
\newpage
% Algorithm 14: 2-opt %
\subsection{2-opt Algorithm}
The \textbf{2-opt algorithm} removes crossings in the solution path by reversing the path between two nodes. The implementation of the \textbf{2-opt algorithm} is the following:
\begin{verbatim}
    function 2-Opt():
        while best_gain != 0:
            best_gain = 0
            for i = 1 to n:
                for j = 1 to n:
                    gain = Compute-Gain(i,j)
                    if (gain < best_gain):
                        best_gain = gain
                        best_i = i
                        best_j = j
                        if first_improvement == true:
                            break
                if (best_gain < 0 && first_improvement == true):
                    break
            Exchange(best_i, best_j)
\end{verbatim}
The algorithm is evaluated as follows:
\begin{itemize}
    \item Space Complexity: $O(n^2)$
    \item Time Complexity: $O(n^2)$
\end{itemize}
% Algorithm 15: 2.5-opt %
\subsection{2.5-opt Algorithm}
The \textbf{2.5-opt algorithm} performs the 2-opt algorithm procedure and check for two possible node shifts. A shift consists of moving a node to another position in the tour.
\begin{figure}[H]
    \centering
    \includegraphics[width=7cm]{Screen Shot 2022-01-28 at 18.11.07.png}
\end{figure}
The algorithm is evaluated as follows:
\begin{itemize}
    \item Space Complexity: $O(n^2)$
    \item Time Complexity: $O(n^2)$
\end{itemize}
\newpage
% Algorithm 16: 3-opt %
\subsection{3-opt Algorithm}
The \textbf{3-opt algorithm} removes three links form the tour, obtaining three segments with which we check for 2-opt swaps and node shifts.
\begin{figure}[H]
    \centering
    \includegraphics[width=7cm]{2.png}
\end{figure}
The algorithm is evaluated as follows:
\begin{itemize}
    \item Space Complexity: $O(n^3)$
    \item Time Complexity: $O(n^3)$
    \item Optimality: TODO
\end{itemize}

\newpage

% -------------------------------- %
% Section 6: Meta-heuristic Search %
% -------------------------------- %
\section{Meta-heuristic Search Algorithms}
Local search algorithms cannot escape from a local minimum and they are inefficient with large neighborhoods, thus either we accept solutions that are worst than the previous, or stochastically explore only a subset of a neighborhood through \textbf{meta-heuristic search algorithms}. \\
Meta-heuristic search algorithms have the following implementation:
\begin{verbatim}
    function Meta-Heuristic-Search(s, f, N):
        current = s
        while terminating condition is met:
            stochastically compute next in N(current)
            Decide whether to continue with current = next
        return current
\end{verbatim}
% Algorithm 17: Simulated Annealing %
\subsection{Simulated Annealing Algorithm}
The \textbf{simulated annealing algorithm} is inspired by the natural annealing of metals and has the following procedure:
\begin{enumerate}
    \item Start from an initial solution
    \item Choose a random new solution from the neighborhood of the previous
        \begin{itemize}
            \item If $f(next) < f(current)$, then $f(next)$ becomes the current node
            \item Otherwise, given a temperature $T$, we use the following probabilistic function:
                \begin{center}
                    $\Delta E = f(next) - f(current)$ \\
                    $e^{-\frac{\Delta E}{T}}$
                \end{center}
        \end{itemize}
    \item Repeat from step $2$ until the solution converges
\end{enumerate}
Given a value $\beta = [1, 100]$, then the steps $L$ from one temperature to the next are computed as follows:
\begin{center}
    $L=\beta \cdot NN_length$
\end{center}
Given the number of nodes $n$, then the temperature $T$ is initialized as follows:
\begin{center}
    $T = \displaystyle\frac{L}{\sqrt{n}}$
\end{center}
Given a value $\alpha \approx 0.95$, then $T$ decreases during the search as follows:
\begin{center}
    $T_{i + 1} = T \cdot \alpha$
\end{center}

\newpage

% ----------------------------- %
% Section 7: Genetic Algorithms %
% ----------------------------- %
\section{Genetic Algorithms}
\textbf{Genetic algorithms} are based on Darwin's evolution theory: individuals are described by chromosomes, which are sets of genes; populations are sets of individuals; generations are sets of populations. \\
Individuals are evaluated using a \textbf{fitness function} representing their adaptation to the environment. \\

Genetic algorithms have the following implementation:
\begin{verbatim}
    function Genetic-Algorithms():
        t = 0
        initialize P(t) with m individuals
        evaluate P(t)
        while terminating condition is not met:
            select parents from P(t)
            generate individuals with reproduction rules
            some individuals die in P(t)
            form population P(t + 1)
            evaluate population P(t + 1)
            t = t + 1
        return best individual
\end{verbatim}
% Algorithm 18: Ant Colony %
\subsection{Ant Colony Algorithm}
The basic principle of the \textbf{ant colony algorithm} is \textbf{stigmergy}, meaning a particular kind of indirect communication through which ants find the shortest paths in dynamic environments:
\begin{itemize}
    \item While moving between nest and food, ants mark their path by pheromone laying
    \item Step-by-step decisions are biased by stigmergy
\end{itemize}
Pheromone is the colony’s collective and distributed memory: it encodes the quality of local routing choices towards the destination target. \\
The implementation of the algorithm is the following:
\begin{verbatim}
    function Ant-Colony():
        while terminating condition is not met:
            randomly position m ants on n cities
            for city = 1 to n:
                for ant = 1 to m:
                    select next city through exploration and exploitation
                    apply local trail updating rule
                compute length l of tour generated by m
            apply global trail updating rule using best ant
\end{verbatim}
\end{document}


% Options for packages loaded elsewhere
\PassOptionsToPackage{unicode}{hyperref}
\PassOptionsToPackage{hyphens}{url}
\PassOptionsToPackage{dvipsnames,svgnames*,x11names*}{xcolor}
%
\documentclass[
]{article}
\usepackage{lmodern}
\usepackage{amssymb,amsmath}
\usepackage{ifxetex,ifluatex}
\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
  \usepackage[T1]{fontenc}
  \usepackage[utf8]{inputenc}
  \usepackage{textcomp} % provide euro and other symbols
\else % if luatex or xetex
  \usepackage{unicode-math}
  \defaultfontfeatures{Scale=MatchLowercase}
  \defaultfontfeatures[\rmfamily]{Ligatures=TeX,Scale=1}
\fi
% Use upquote if available, for straight quotes in verbatim environments
\IfFileExists{upquote.sty}{\usepackage{upquote}}{}
\IfFileExists{microtype.sty}{% use microtype if available
  \usepackage[]{microtype}
  \UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
}{}
\makeatletter
\@ifundefined{KOMAClassName}{% if non-KOMA class
  \IfFileExists{parskip.sty}{%
    \usepackage{parskip}
  }{% else
    \setlength{\parindent}{0pt}
    \setlength{\parskip}{6pt plus 2pt minus 1pt}}
}{% if KOMA class
  \KOMAoptions{parskip=half}}
\makeatother
\usepackage{xcolor}
\IfFileExists{xurl.sty}{\usepackage{xurl}}{} % add URL line breaks if available
\IfFileExists{bookmark.sty}{\usepackage{bookmark}}{\usepackage{hyperref}}
\hypersetup{
  colorlinks=true,
  linkcolor=blue,
  filecolor=Maroon,
  citecolor=Blue,
  urlcolor=Blue,
  pdfcreator={LaTeX via pandoc}}
\urlstyle{same} % disable monospaced font for URLs
\usepackage{longtable,booktabs}
% Correct order of tables after \paragraph or \subparagraph
\usepackage{etoolbox}
\makeatletter
\patchcmd\longtable{\par}{\if@noskipsec\mbox{}\fi\par}{}{}
\makeatother
% Allow footnotes in longtable head/foot
\IfFileExists{footnotehyper.sty}{\usepackage{footnotehyper}}{\usepackage{footnote}}
\makesavenoteenv{longtable}
\setlength{\emergencystretch}{3em} % prevent overfull lines
\providecommand{\tightlist}{%
  \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
\setcounter{secnumdepth}{-\maxdimen} % remove section numbering
\ifluatex
  \usepackage{selnolig}  % disable illegal ligatures
\fi

\author{}
\date{}

\begin{document}

\newcommand{\Ber}{\operatorname{Ber}}
\newcommand{\E}{\operatorname{E}}
\newcommand{\V}{\operatorname{Var}}
\newcommand{\diag}{\operatorname{diag}}

\newcommand{\bi}{\mathbf{i}}
\newcommand{\bj}{\mathbf{j}}
\newcommand{\bt}{\mathbf{t}}

\newcommand{\bw}{{\boldsymbol{w}}}
\newcommand{\bv}{{\boldsymbol{v}}}
\newcommand{\bx}{{\boldsymbol{x}}}
\newcommand{\by}{{\boldsymbol{y}}}
\newcommand{\bb}{{\boldsymbol{b}}}
\newcommand{\bz}{{\boldsymbol{z}}}
\newcommand{\bu}{{\boldsymbol{u}}}
\newcommand{\bX}{{\boldsymbol{X}}}
\newcommand{\bY}{{\boldsymbol{Y}}}
\newcommand{\bZ}{{\boldsymbol{Z}}}

\hypertarget{logistic-regression}{%
\section{Logistic Regression}\label{logistic-regression}}

\hypertarget{maximum-likelihood}{%
\subsection{Maximum Likelihood}\label{maximum-likelihood}}

Suppose that there is a coin which you think is biased. You flip the
coin 10 times and get 7 heads. Based on this \emph{training data} set,
what is the probability that the next flip will show heads? Can you
justify your answer? To provide a rigorous argument for the answer, we
can use \emph{maximum likelihood} estimation.

Let \(y\) be the (unknown) probability that the biased coin shows heads.
Since there are only two possible results, heads or tails, the
experiment of flipping the coin is a \emph{Bernoulli trial}. We
introduce a \emph{Bernoulli random variable} \(T\) such that
\[ T=\begin{cases} 1 & \text{ if the coin shows heads}, \\ 0 & \text{ otherwise}. \end{cases} \]
with probabilities
\[ P(T=1|y)=y \quad \text{ and } \quad P(T=0|y)=1-y, \quad \text{respectively}. \]\\
The probability mass function \(\operatorname{Ber}(t|y)\) of \(T\) can
be written in a compact way
\[ \operatorname{Ber}(t|y)= y^t (1-y)^{1-t}, \quad t=0,1. \]

Suppose that we have a data set \(\mathcal D = \{ t_1, \dots, t_N \}\)
of observed values of \(t \in \{ 0,1 \}\) from \(N\) coin flips. Let
\(p(\mathcal D|y)\) denote the probability of getting the set
\(\mathcal D\), given that \(y\) is the probability of the biased coin.
Since each flip is independent, the probability \(p(\mathcal D|y)\) is
equal to the product of the probabilities of individual flip results,
i.e.~it is equal to
\[ p(\mathcal D| y) = \prod_{n=1}^N \operatorname{Ber}(t_n|y) = \prod_{n=1}^N y^{t_n} (1-y)^{1-t_n}. \]

This is a function of \(y\); as \(y\) varies, the function
\(p(\mathcal D|y)\) produces probabilities of the set \(\mathcal D\)
determined by \(y\). When \(p(\mathcal D|y)\) is maximized at a value
\(y_{\mathrm {ML}}\), we can say that \(y_{\mathrm{ML}}\) has maximum
likelihood based on the data st \(\mathcal D\). This leads to

\[\text{ Task: Estimate a value $y_{\mathrm{ML}}$ that maximizes $p(\mathcal D|y)$}.\]

This is a Calculus problem. We will have to take the derivative of the
function. Before taking the derivative, we notice that the expression of
\(p(\mathcal D|y)\) is a product of exponentials. In such a situation,
logarithmic differentiation can be used to ease the process of
differentiation, and the function can be replace by
\[ \ln p(\mathcal D|y) = \sum_{n=1}^N \{ t_n \ln y + (1-t_n) \ln(1-y)\} .\]

Now we take the derivative of the function with respect to \(y\), set
the derivative equal to \(0\) to obtain a critical value, and check if
the critical value \(y_{\mathrm{ML}}\) indeed maximizes the function.
The result is \[ y_{\mathrm{ML}}= \frac 1 N \sum_{n=1}^N t_n ,\] which
is the \emph{sample mean}. This justifies our guess for a biased coin in
a rigorous way.

\textbf{Exercises}: Provide details to obtain \(y_{\mathrm{ML}}\).

\hypertarget{logistic-regression-1}{%
\subsection{Logistic Regression}\label{logistic-regression-1}}

Suppose that there is a certain test (similar to GRE) for undergraduate
students who apply for a graduate program. The test score is used for
admission decisions as one of the credentials that students provide. The
table below shows the test scores of 10 students and the results of
admission decisions:

\begin{longtable}[]{@{}llcccccccccc@{}}
\toprule
\(x\) & : & \(272\) & \(331\) & \(295\) & \(287\) & \(315\) & \(266\) &
\(303\) & \(294\) & \(317\) & \(309\)\tabularnewline
\midrule
\endhead
\(t\) & : & 0 & 1 & 1 & 0 & 1 & 0 & 0 & 0 & 1 & 1\tabularnewline
\bottomrule
\end{longtable}

\(x\) test score \(t=1\) accepted \(t=0\) rejected

There is a student whose test score is 299. Based on this data, what is
the probability that the student will be accepted? If the probability is
greater than 0.5, we can \emph{classify} the student as \emph{Likely
Accepted}; if the probability is less than 0.5, as \emph{Likely
Rejected}. This type of problem is called \emph{classification} problem
in machine learning.

An answer to this problem would have a procedure to produce probability
\(y\) from a test score \(x\). A typical way to obtain a number between
0 and 1 (that is to be considered as probability) from an arbitrary real
number is to use a function whose graph is an \(S\)-shaped curve, called
a \emph{sigmoid} function. One of the most common sigmoid functions is
the \emph{logistic} function
\[ \sigma(x) = \frac {e^x}{e^x+1}= \frac 1 {1+e^{-x}}.\]

\begin{center}
\includegraphics[scale=0.2]{sigmoid}

Graph of the Logistic Function

\end{center}

Clearly, the logistic function converts \((-\infty, \infty)\) to
\((0,1)\). In particular, the value \(0\) corresponds to \(0.5\). In
order to apply this function to\\
a data set \(\{ x_1, x_2, \dots , x_N \}\) of test scores, we need to
first adjust the values \(x_n\) in accordance with the logistic function
in such a way that the result will best fit for the given data. Here we
make an assumption that the adjustment is done by a linear function of
the form \(w_1 x_n+w_2\) so that the probability is given by
\begin{equation} \label{eqn-y-sigma} y_n=\sigma(w_1 x_n +w_2). \end{equation}
This is the same kind of assumption we made for linear regression.

Now the question is how to find the best values of \(w_0\) and \(w_1\).
Let us consider the likelihood of the given data set. Each score \(x\)
corresponds to probability \(y\) and a student with score \(x\) will be
accepted with probability \(y\) through a Bernoulli trial. It is like
flipping a biased coin with probability \(y\). Given a data set
\(\mathcal D=\{(x_n, t_n) \}\), \(t_n \in \{ 0,1 \}\),
\(n=1, \dots, N\), we compute the likelihood of this data set and obtain
\[ p(\mathcal D|{\boldsymbol{w}}) =  \prod_{n=1}^N \operatorname{Ber}(t_n|y_n) = \prod_{n=1}^N y_n^{t_n} (1-y_n)^{1-t_n}, \]
where \(y_n= \sigma(w_1 x_n+w_2)\) and \({\boldsymbol{w}}=(w_1, w_2)\).
The best choice of \({\boldsymbol{w}}=(w_1,w_2)\) would make this
likelihood the largest possible. Thus our task can be summarized as the
following.

\[\text{Task: Determine ${\boldsymbol{w}}$ that maximize the likelihood $p(\mathcal D|{\boldsymbol{w}})$.}\]

Test scores make only one feature in credentials for admission
decisions. Assume that there are \(k\) different features.\\
Then the \(n^\mathrm{th}\) student provides credentials
\(\{x_{n1}, x_{n2}, \dots , x_{nk} \}\), and as in linear regression,
the credentials of \(N\) student can be arranged into a matrix
\[  \begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1k} \\ x_{21} & x_{22} & \cdots & x_{2k} \\ \vdots & \vdots & & \vdots \\ x_{N1} & x_{N2} & \cdots & x_{Nk} \end{bmatrix} .\]

The function in \eqref{eqn-y-sigma} is now generalized to
\[ y_n=\sigma(w_1 x_{n1}+ w_2 x_{n2} + \cdots + w_k x_{nk}+w_{k+1} ), \]
which can be rewritten in vector notations
\[ y_n= \sigma( {\boldsymbol{x}}_n{\boldsymbol{w}}) ,\] where
\({\boldsymbol{w}}=(w_1, w_2, \dots , w_k, w_{k+1})^\top\) is a column
vector to be determined and
\({\boldsymbol{x}}_n=(x_{n1}, x_{n2}, \dots, x_{nk}, 1)\). With
\(y_n=\sigma({\boldsymbol{x}}_n {\boldsymbol{w}})\), the expression of
the likelihood \(p(\mathcal D| {\boldsymbol{w}})\) does not change.

To make differentiation easier, we take the logarithm of the likelihood
and want to determine \({\boldsymbol{w}}\) which maximizes
\[ \ln p(\mathcal D|{\boldsymbol{w}}) = \sum_{n=1}^N \{ t_n \ln y_n + (1-t_n) \ln (1-y_n)\} ,\]
or equivalently, \({\boldsymbol{w}}\) which minimizes
\[ E({\boldsymbol{w}}):=-\ln p(\mathcal D|{\boldsymbol{w}}) = -\sum_{n=1}^N \{ t_n \ln y_n + (1-t_n) \ln (1-y_n)\} .\]
This problem is called \emph{logistic regression}.

Unlike linear regression, it is difficult to find closed form formulas
for solutions to this problem. Instead, we will use an inductive process
to approximate the vector \({\boldsymbol{w}}\). We will study two
standard methods for this purpose, called \emph{Gradient Descent} and
\emph{Newton's Method}.

\hypertarget{applying-newtons-method-to-the-logistic-regression}{%
\subsection{Applying Newton's method to the logistic
regression:}\label{applying-newtons-method-to-the-logistic-regression}}

\[ E ({\boldsymbol{w}}) = - \ln p(t|{\boldsymbol{w}}) = - \sum_{n=1}^N \{ t_n \ln y_n + (1-t_n) \ln (1-y_n)\},  \]
where
\(y_n=\sigma(a_n)=\sigma({\boldsymbol{x}}_n^\top {\boldsymbol{w}})\).
Note that \(\sigma'(x) =\sigma(x)(1-\sigma(x))\).

\begin{itemize}
\item
  We obtain
  \[ \nabla E({\boldsymbol{w}})= \sum_{n=1}^N (y_n-t_n){\boldsymbol{x}}_n = X ({\boldsymbol{y}}-\mathbf{t}), \]
  where
  \[ X=\begin{bmatrix} 1 &1 & \cdots &1 \\ x_1 & x_2 & \cdots &x_N \\ x_1^2 & x_2^2 & \cdots &x_N^2 \\ \vdots & \vdots & & \vdots \\ x_1^D & x_2^D & \cdots &x_N^D \\\end{bmatrix}, \quad {\boldsymbol{y}}=\begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_N \end{bmatrix}, \quad \text{ and } \mathbf{t}= \begin{bmatrix} t_1 \\ t_2 \\ \vdots \\ t_N \end{bmatrix} . \]
\item
  We also get
  \[ \mathbf H = \sum_{n=1}^N y_n(1-y_n) {\boldsymbol{x}}_n {\boldsymbol{x}}_n^\top = X R X^\top, \]
  where \(R=\operatorname{diag}( y_n(1-y_n))\).
\item
  Then we have
  \[ \boxed{{\boldsymbol{w}}_{k+1} = {\boldsymbol{w}}_k - (X R X^\top)^{-1} X ({\boldsymbol{y}}-\mathbf{t}) },\]
  where \(R\) and \({\boldsymbol{y}}\) are determined by
  \({\boldsymbol{w}}_k\) in each step.
\end{itemize}

\hypertarget{example-test-scores-and-admission-to-a-graduate-school}{%
\subsubsection{Example: Test scores and admission to a graduate
school}\label{example-test-scores-and-admission-to-a-graduate-school}}

\begin{itemize}
\item
  Chose \(D=1\). Then \(a_n=w_0+w_1x_n\). \textbackslash{} From the
  data, we have \[X= \begin{bmatrix}  1 &1 &1 &1 &1 &1 &1 &1 &1 &1 \\
  272&331&295&287&315&266&303&294&317&309 \end{bmatrix}, \] and
  \[ \mathbf{t}=[0,1,1,0,1,0,0,0,1,1]^\top .\]
\item
  Choose \({\boldsymbol{w}}_0=(0,0)^\top\). Then \({\boldsymbol{w}}_k\)
  converges to \[ (-57.2937, 0.1910)^\top .\]
\item
  If \(x=299\), then the probability of admission is
  \[ y= \sigma(-57.2937+ 0.1910*299) \approx 0.454 .\]
\end{itemize}

\end{document}