% $Id: aboutcryptool.tex 3714 2016-04-08 18:34:16Z esslinger $
% ............................................................................
%                 TEXT OF THE 2nd PAGE (Overview)
%
% ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

\clearpage\phantomsection

\addcontentsline{toc}{chapter}{Overview about the Content of the CrypTool Book}
\chapter*{Overview about the Content of the CrypTool Book}

\parskip 4pt
%\vskip +12 pt
The rapid spread of the Internet has led to intensified research in the
technologies involved, especially within the area of cryptography where a good
deal of new knowledge has arisen.

In this {\em book accompanying the CrypTool programs} \index{CrypTool}
you will find predominantly mathematically oriented information on using
cryptographic procedures. Also included are many sample code pieces written in the
computer algebra system {\bf SageMath}\index{SageMath} (see appendix~\ref{s:appendix-using-sage}).
The main chapters have been written by various {\bf authors}
(see appendix~\ref{s:appendix-authors}) %\hyperlink{appendix-authors}{authors}
and are therefore independent from one another. At the end of most chapters
you will find references and web links.
The sections have been enriched with many {\em footnotes}. Within the footnotes
you can see where the described functions can be called in the different CrypTool
versions.

The \hyperlink{Kapitel_1}{first chapter} explains the principles of symmetric
and asymmetric {\bf encryption} and definitions for their resistibility.

Because of didactic reasons the \hyperlink{Kapitel_PaperandPencil}
{second chapter} gives an exhaustive overview
about {\bf paper and pencil encryption methods}.

Big parts of this book are dedicated to the fascinating topic of
{\bf prime numbers} (chap. \ref{Label_Kapitel_Primes}).
%\hyperlink{Kapitel_2}{{\bf prime numbers}}.
Using numerous examples, {\bf modular arithmetic} and
{\bf elementary number theory} (chap. \ref{Chapter_ElementaryNT})
are introduced. The features of the {\bf RSA procedure} are a key aspect.

By reading chapter \ref{Chapter_ModernCryptography}
you'll gain an insight into the mathematical ideas and concepts behind
{\bf modern cryptography}.

Chapter \ref{Chapter_Hashes-and-Digital-Signatures} gives
an overview about the status of attacks against modern {\bf hash algorithms}
and is then shortly devoted to {\bf digital signatures},
which are an essential component of e-business applications.

Chapter \ref{Chapter_EllipticCurves} describes {\bf elliptic curves}:
They could be used as an alternative to RSA and in addition are extremely
well suited for implementation on smartcards.

Chapter \ref{Chapter_BitCiphers} introduces {\bf Boolean algebra}.
Boolean algebra is the foundation for most modern, symmetric encryption algorithms
as these operate on bit streams and bit groups. Principal construction
methods are described and implemented in SageMath.

Chapter \ref{Chapter_HomomorphicCiphers} describes {\bf homomorphic crypto
functions}: They are a modern research topic which got especial attention
in the course of cloud computing.

The \hyperlink{Chapter_Crypto2020}{last chapter} {\bf Crypto2020}
discusses threats for currently used cryptographic methods and introduces
alternative research approaches to achieve long-term security
of cryptographic schemes.

Whereas the CrypTool \textit{e-learning programs}\index{e-learning} motivate
and teach you how to use cryptography in practice, the \textit{book} provides
those interested in the subject with a deeper understanding of the mathematical
algorithms used -- trying to do it in an instructive way.

Within the {\bf appendices}
\ref{s:appendix-menu-overview-CT1},
\ref{s:appendix-template-overview-CT2},
\ref{s:appendix-function-overview-JCT}, and
\ref{s:appendix-function-overview-CTO}
you can gain a fast overview about the functions delivered by the different
CrypTool variants\index{CrypTool 1}\index{CrypTool 2}\index{JCrypTool} via:
\begin{itemize}
  \item the function list and
        the \hyperlink{appendix-menu-overview-CT1}
                      {menu tree of CrypTool 1 (CT1)},
  \item the function list and
        the \hyperlink{appendix-template-overview-CT2}
                      {templates in CrypTool 2 (CT2)},
  \item the \hyperlink{appendix-function-overview-JCT}
                      {function list of JCrypTool (JCT)}, and
  \item the \hyperlink{appendix-function-overview-CTO}
                      {function list of CrypTool-Online (CTO)}.
\end{itemize}

% Bernhard Esslinger, Matthias B\"uger, Bartol Filipovic, Henrik Koy,
% Roger Oyono and J\"org Cornelius Schneider
The authors would like to take this opportunity to thank their colleagues
in the particular companies and at the universities of Bochum, Darmstadt,
Frankfurt, Gie\ss en, Karlsruhe and Siegen.

\enlargethispage{12pt}
As with the e-learning program CrypTool\index{CrypTool}, the quality of the
book is enhanced by your suggestions and ideas for improvement.
We look forward to your feedback.

% Local Variables:
% TeX-master: "../script-en.tex"
% End:


%\documentclass[12pt]{article}
\documentclass[12pt,landscape]{article}

\include{preamble}

\newcommand{\instr}{\small Your answer will consist of a lowercase string (e.g. \texttt{aebgd}) where the order of the letters does not matter. \normalsize}

\title{Math 368 / 650 Fall \the\year{} \\ Midterm Examination One}
\author{Professor Adam Kapelner}

\date{Thursday, October 7, \the\year{}}

\begin{document}
\maketitle

%\noindent Full Name \line(1,0){410}

\thispagestyle{empty}

\section*{Code of Academic Integrity}

\footnotesize
Since the college is an academic community, its fundamental purpose is the pursuit of knowledge. Essential to the success of this educational mission is a commitment to the principles of academic integrity. Every member of the college community is responsible for upholding the highest standards of honesty at all times. Students, as members of the community, are also responsible for adhering to the principles and spirit of the following Code of Academic Integrity.

Activities that have the effect or intention of interfering with education, pursuit of knowledge, or fair evaluation of a student's performance are prohibited. Examples of such activities include but are not limited to the following definitions:

\paragraph{Cheating} Using or attempting to use unauthorized assistance, material, or study aids in examinations or other academic work or preventing, or attempting to prevent, another from using authorized assistance, material, or study aids. Example: using an unauthorized cheat sheet in a quiz or exam, altering a graded exam and resubmitting it for a better grade, etc.
\\

\noindent By taking this exam, you acknowledge and agree to uphold this Code of Academic Integrity. \\

%\begin{center}
%\line(1,0){250} ~~~ \line(1,0){100}\\
%~~~~~~~~~~~~~~~~~~~~~signature~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ date
%\end{center}

\normalsize

\section*{Instructions}
This exam is 70 minutes (variable time per question) and closed-book. You are allowed \textbf{one} page (front and back) of a \qu{cheat sheet}, blank scrap paper and a graphing calculator. Please read the questions carefully. Within each problem, I recommend considering the questions that are easy first and then circling back to evaluate the harder ones. No food is allowed, only drinks. %If the question reads \qu{compute,} this means the solution will be a number otherwise you can leave the answer in \textit{any} widely accepted mathematical notation which could be resolved to an exact or approximate number with the use of a computer. I advise you to skip problems marked \qu{[Extra Credit]} until you have finished the other questions on the exam, then loop back and plug in all the holes. I also advise you to use pencil. The exam is 100 points total plus extra credit. Partial credit will be granted for incomplete answers on most of the questions. \fbox{Box} in your final answers. Good luck!

\pagebreak

\problem\timedsection{10} These are questions about indicator functions. Let $a,b \in \reals$ and $b > a$. And let $X$ be a discrete rv.

\vspace{-0.2cm}\benum\truefalsesubquestionwithpoints{18}

\begin{enumerate}[(a)]
\item $\sum_{x \in \reals} \indic{x \in \braces{a}} = a$
\item $\sum_{x \in \reals} \indic{x \in \braces{a}} = 1$
\item $\sum_{x \in \reals} a\indic{x \in \braces{a}} = a$
\item $\sum_{x \in \reals} a\indic{x \in \braces{a}} = 1$
\item $\prod_{x \in \reals} a\indic{x \in \braces{a}} = a$
\item $\prod_{x \in \reals} a\indic{x \in \braces{a}} = 1$

\item $\int_{\reals} a\indic{x \in [a,b]} = a$
\item $\int_{\reals} a\indic{x \in [a,b]} = b$
\item $\int_{\reals} a\indic{x \in [a,b]} = b-a$
\item $\int_0^1 \indic{x \in [a,b]} = b-a$

\item $p(x) = p^{old}(x)\indic{x \in \support{X}}$
\item $\sum_{x \in \reals} p^{old}(x) = 1$
\item $\sum_{x \in \support{X}} p^{old}(x) = 1$

\item $\sum_{x \in \naturals} p(x) = 1$
\item $\sum_{x \in \integers} p(x) = 1$

\item $\int_{\reals} p^{old}(x) = 1$
\item $\int_{\support{X}} p^{old}(x) = 1$

\item $\sum_{x \in \support{X}} p^{old}(x)^2 = 1$

\end{enumerate}
\eenum\instr\pagebreak

%%%%%%%%%%%%%%%%%%
\problem\timedsection{11} Let $X \sim U(\braces{1,2,3})$ and $Y \sim U(\braces{-1,-2,-3})$ and $T = X + Y$.

\vspace{-0.2cm}\benum\truefalsesubquestionwithpoints{19}

\begin{enumerate}[(a)]
\item $\sum_{x \in \reals} p_{X,Y}(x,y) = 1$
\item $p_{X,Y}$ has at most 9 $x,y$ input pairs that produce nonzero values
\item $p_X^{old}(x) = 1/3$
\item $p_Y^{old}(y) = 1/3$
\item $p_T^{old}(t) = 1/9$ if $X,Y$ are independent.
\item $p_T^{old}(2) = 1/9$ if $X,Y$ are independent.
\item $p_T(t) = p_{X,Y}(x,y) \star p_{X,Y}(x,y)$
\item $p_T(t) = \sum_{x \in \reals} \sum_{y \in \reals} p_{X,Y}(x,y)$
\item The expectation of $T$ is 0 if $X,Y$ are independent.
\item The expectation of $T$ is 0 regardless of the dependence relationship of $X, Y$.
\item $\support{T} = \braces{-2, -1, 0, 1, 2}$ if $X,Y$ are independent.
\item $\support{T} = \braces{-2, -1, 0, 1, 2}$ regardless of the dependence relationship of $X, Y$.
\item $T$ could be a degenerate rv.
\item You can compute $p_T(t)$ for all $t \in \reals$ if $X,Y$ are independent given the information provided.
\item You can compute $p_T(t)$ for all $t \in \reals$ if $X,Y$ are dependent given the information provided.
\item $\cov{X}{Y} = 0$ if $X,Y$ are independent.
\item $\cov{X}{T} = 0$ if $X,Y$ are independent.
\item $\cov{Y}{T} = \expe{YT} - \expe{Y}\expe{T}$ if $X,Y$ are independent.
\item $\cov{X}{Y} = \cov{Y}{X}$ if $X,Y$ are dependent.
\end{enumerate}
\eenum\instr\pagebreak

%%%%%%%%%%%%%%%%%%
\problem\timedsection{10} Let $X \sim \geometric{p_x}$ independent of $Y \sim \geometric{p_y}$ and $T = X + Y$.

\vspace{-0.2cm}\benum\truefalsesubquestionwithpoints{15}

\begin{enumerate}[(a)]
\item The PMF of $T$ can be derived using one of the discrete convolution formulas
\item $\support{X} = \support{T}$
\item If $p_x > p_y$ it is likely that $X > Y$
\item $T \sim \geometric{p_x+p_y}$
\item $T \sim \negbin{2}{p_x+p_y}$
\item If $p_x = p_y = 1$, $T$ is a degenerate rv
\item If $p_x = p_y = 0$, $T$ is a degenerate rv
\item If $p_x = p_y = \half$ then $T \sim \negbin{2}{\half}$
\item If $p_x = p_y = \half$ then $\prob{X = Y} > 0$
\item If $p_x = p_y = \half$ then $\prob{X = Y} = \frac{3}{4}$
\item If $p_x = p_y = \half$ then $\prob{X = Y} = \oneover{3}$
\item If $p_x = p_y = \half$ then $\prob{X = Y} = \half$
\item If $p_x = p_y = \half$ then $\prob{X = Y} = \oneover{4}$
\item If $p_x = p_y = \half$ then $\prob{X = Y} = \oneover{8}$
\item If $p_x = p_y = \half$ then $\prob{X = Y} = \oneover{16}$
\end{enumerate}
\eenum\instr\pagebreak

%%%%%%%%%%%%%%%%%%
\problem\timedsection{10} Let $X \sim \geometric{p}$ independent of $Y \sim \geometric{p}$ and $T = X + Y$.

\vspace{-0.2cm}\benum\truefalsesubquestionwithpoints{13}

\begin{enumerate}[(a)]
%\item $\prob{T = 6} = 7(1-p)^7 p^2$
%\item $\prob{T = 6} = 7(1-p)^6 p^2$
%\item $\prob{T = 6} = 6(1-p)^7 p^2$
%\item $\prob{T = 6} = 6(1-p)^6 p^2$
\item $p_{X\,|\,T}(x,t) = p_{X,T}(x,t)$
\item $p_{X\,|\,T}(x,t) = p_{X,Y}(x,t) / p_T(t)$
\item $p_{X\,|\,T}(x,t) = p_{X,Y}(x,t-x) / p_T(t)$
\item $p_{X\,|\,T}(x,t) = \displaystyle \frac{(1-p)^x (1-p)^{t-x}}{(t+1) (1-p)^t}$
\item $p_{X\,|\,T}(x,t) = \displaystyle \frac{(1-p)^x (1-p)^{t-x}}{(t+1) (1-p)^t p^2}$
\item $p_{X\,|\,T}(x,t) = \displaystyle \frac{(1-p)^x (1-p)^{t-x}}{t (1-p)^{t+1}}$
\item $p_{X\,|\,T}(x,t) = \displaystyle \frac{(1-p)^x (1-p)^{t-x}}{t (1-p)^{t+1} p^2}$
\item $p_{X\,|\,T}(x,t)$ is a geometric rv
\item $p_{X\,|\,T}(x,t)$ is a negative binomial rv
\item $p_{X\,|\,T}(x,t)$ is a binomial rv
\item $p_{X\,|\,T}(x,t)$ is a poisson rv
\item $p_{X\,|\,T}(x,t)$ is a uniform discrete rv
\item $p_{X\,|\,T}(x,t)$ is a degenerate rv
\end{enumerate}
\eenum\instr\pagebreak

%%%%%%%%%%%%%%%%%%
\problem\timedsection{7} Let $\X = \bracks{\Xoneton}^\top \sim p(\x)$ and $\expe{\X} = \zerovec_n$.

\vspace{-0.2cm}\benum\truefalsesubquestionwithpoints{10}

\begin{enumerate}[(a)]
\item If $\A$ be an $m \times n$ matrix of constants, then $\expe{\A\X} = \zerovec_m$
\item $\var{\X}$ is an $n \times n$ matrix with entries $\expe{X_i X_j}$ at row $i$ and column $j$
\item $\var{\X}$ can be written as a quadratic form
\item $\var{\X}$ can be written as the expectation of an inner product
\item $\var{\X}$ can be written as the expectation of an outer product
\item If $\Xoneton \inddist$ then $\var{\X} = \I_n$
\item There exists a $p(\x)$ where $\var{\X}$ is \emph{not} symmetric
\item There exists a $p(\x)$ where $\var{\onevec^\top \X} > \sum_{i=1}^n \var{X_i}$
\item There exists a $p(\x)$ where $\var{\onevec^\top \X} = \sum_{i=1}^n \var{X_i}$
\item There exists a $p(\x)$ where $\var{\onevec^\top \X} < \sum_{i=1}^n \var{X_i}$
\end{enumerate}
\eenum\instr\pagebreak

%%%%%%%%%%%%%%%%%%
\problem\timedsection{10} Let $X_1 \sim \binomial{n_1}{p_1}$ independent of $X_2 \sim \binomial{n_2}{p_2}$ and consider the difference $D = X_1 - X_2 \sim p_D(d)$. Let $Y = -X_2$.

\vspace{-0.2cm}\benum\truefalsesubquestionwithpoints{10}

\begin{enumerate}[(a)]
\item If $n_1$ was large and $p_1 \approx 0$, then the PMF of $X_1$ can be approximated with low error by $\poisson{n_1 p_1}$
\item $X_1 \sim \displaystyle \binom{n_1}{x} p_1^x (1-p_1)^{n_1 - x} \indic{x \in \braces{0, 1, \ldots, n_1}}$
\item $X_2 \sim \displaystyle \binom{n_2}{x} p_2^x (1-p_2)^{n_2 - x} \indic{x \in \braces{0, 1, \ldots, n_2}}$
\item $Y \sim \displaystyle \binom{n_2}{-y} p_2^{-y} (1-p_2)^{n_2 + y} \indic{y \in \braces{0, 1, \ldots, n_2}}$
\item $p_D(d) = \displaystyle\sum_{x \in \reals} p_{X_1}(x) p_{X_2}(d - x)$
\item $p_D(d) = \displaystyle\sum_{x \in \support{X_1}} p_{X_1}^{old}(x) p_{X_2}^{old}(d - x) \indic{d-x \in \braces{0, 1, \ldots, n_2}}$
\item $p_D(d) = \displaystyle\sum_{x = 0}^{n_1} \binom{n_1}{x} p_1^x (1-p_1)^{n_1 - x} \binom{n_2}{x-d} p_2^{x-d} (1-p_2)^{n_2 + d - x} \indic{x-d \in \braces{0, 1, \ldots, n_2}}$
\item $p_D(d) = \displaystyle\sum_{x = d}^{n_1} \binom{n_1}{x} p_1^x (1-p_1)^{n_1 - x} \binom{n_2}{x-d} p_2^{x-d} (1-p_2)^{n_2 + d - x}$
\item $D$ is a binomial rv
\item $D$ is a poisson rv
\end{enumerate}
\eenum\instr\pagebreak

%%%%%%%%%%%%%%%%%%
\problem\timedsection{12} A large factory produces marbles with the following color distribution:

\begin{table}[h]
\centering
\begin{tabular}{l|lllll}
Color 			& Blue & Red & Green & Yellow & Orange \\ \hline
Percentage	     & 20    & 20   & 30      & 5        & 25
\end{tabular}
\end{table}

\noindent You sample 100 marbles randomly from the assembly line. Let $X_b, X_r, X_g, X_y, X_o$ denote the rv's modeling the counts of Blue, Red, Green, Yellow and Orange marbles respectively in your sample and let $\X$ denote the column vector of those rv's stacked.

\vspace{-0.2cm}\benum\truefalsesubquestionwithpoints{15}

\begin{enumerate}[(a)]
\item $X_b \sim \binomial{100}{0.2}$ and $X_r \sim \binomial{100}{0.2}$
\item $X_b + X_r \sim \binomial{200}{0.2}$
\item $\expe{X_b + X_r} = 40$.
\item $\x \in \reals$
\item $\x^\top \onevec = 100$ for all $\x \in \support{\X}$
\item $|\corr{X_b}{X_r}| = |\cov{X_b}{X_r}|$
\item $|\corr{X_b}{X_r}| > |\corr{X_y}{X_g}|$
\item $p_{X_b\,|\,X_r}(x, y)$ is undefined for $y > 100$
\item Generally speaking, the more blue marbles in the sample, the less yellow marbles in the sample.
\item If it is known that there are 8 blue marbles in the sample, then the number of yellow marbles is expected to be lower as compared to if you have no information about the number of blue marbles in the sample.\\

For the remaining questions, assume that we are told there are 8 blue marbles in the sample.
\item The number of yellow marbles will be a drawn from a $\binomial{92}{0.05}$ rv.
\item The number of yellow marbles will be a drawn from a $\binomial{92}{0.0625}$ rv.
\item The number of yellow marbles will be a drawn from a $\binomial{75}{0.05}$ rv.
\item The number of yellow marbles will be a drawn from a $\binomial{75}{0.0625}$ rv.
\item $\bracks{X_r\, X_g\, X_y\, X_o}^\top \sim \multinomial{4}{92}{\bv{p}}$ where the vector $\bv{p}$ can be computed using information provided in this problem.
\end{enumerate}
\eenum\instr\pagebreak

\end{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\problem\timedsection{5} These are conceptual questions ...

\vspace{-0.2cm}\benum\truefalsesubquestionwithpoints{14}

\begin{enumerate}[(a)]
\item
\end{enumerate}
\eenum\instr\pagebreak


\section{Optimization Overview}

Most intermediate representations are organised as \textit{control flow graphs (CFG)} over \textit{basic blocks}.

\subsection{Basic Blocks}

A basic block is a single entry, single exit, straight line code segment. More formally, it is a maximal sequence of instructions with no labels (except at the first instruction) and no jumps (except at the last instruction).

In the following figure, (a) is a basic block. (b) is not a basic block because it has multiple exits; (c) is not a basic block because it has multiple entries; (d) is not a basic block because it does not represent a straight line code.

\begin{figure}[htp]
\centering
\includegraphics[height=6cm]{images/1.png}
\caption{Examples and Counterexamples for basic block}
\end{figure}

Consider the following example of a single basic block:

\begin{enumerate}
    \item L:
    \item \(t := 2 * x\)
    \item \(w := t + x\)
    \item if w \(>\) 3 goto L'
\end{enumerate}

\begin{itemize}
    \item Is it ok to change (3) to \(w := 3 * x\) ?
    It is ok. If addition of two numbers is more expensive as compared to multiplication of a number with a constant, the above change can be considered an optimization.
    \item Is it ok to change (4) to if x \(>\) 1 goto L' ? It is not ok. Because here x is a finite bounded integer and due to overflow, it is possible to have \(3 * x > 3\) and not x \(>\) 1.
    \item Is it ok to remove (2) ? Depends on whether variable t is used later in the code or not.
\end{itemize}

\subsection{Control Flow Graph}

A control flow graph is a directed graph with
\begin{itemize}
    \item basic blocks as nodes
    \item edge from block A to block B if the execution can pass from the last instruction in A to the first instruction in B, for example
    \begin{itemize}
        \item the last instruction in A is: jump Lb
        \item the last instruction in A is: if id1 = id2 then goto Lb
        \item execution can fall-through from block A to block B
    \end{itemize}
\end{itemize}

Consider the following example of a CFG:

\begin{figure}[htp]
\centering
\includegraphics[height=6cm]{images/CFG Example.png}
\caption{CFG Example}
\end{figure}

The body of a method (or procedure) can be represented as a control-flow graph. There is one initial node (entry node). All "return" nodes are terminal.

\subsection{Optimization}
The optimizations are performed on the control flow graph of the intermediate representation of the code to improve program's resource utilization:

\begin{itemize}
    \item Execution Time
    \item Code Size
    \item Memory Usage
    \item Frequency of Disk I/O operations (or network operations)
    \item Power Consumption (Not same as energy)
\end{itemize}

It is important to remember that optimization should not alter the meaning of the program. It should not alter what the program computes.

For example, the following optimization changes the meaning of the program and thus, it is not a valid optimization.

w := 3 * x
if w \(>\) 3 goto L

cannot be converted to: if x \(>\) 1 goto L

\subsection{Typical Granularity of Optimization}

\begin{enumerate}
    \item \textbf{Local Optimizations}
    \begin{itemize}
        \item applied to a basic block in isolation
        \item easiest to implement
    \end{itemize}

    \item \textbf{Global Optimizations}
    \begin{itemize}
        \item applied to a CFG (method body) in isolation while crossing the boundaries of basic blocks
    \end{itemize}

    \item \textbf{Inter-Procedural Optimizations}
    \begin{itemize}
        \item applied across method (CFG) boundaries
        \item difficult to implement but usually most effective
    \end{itemize}
\end{enumerate}

\subsection{Economics of the Optimization}

Optimizations are more of an art rather than science. The current state of the art methods are based on the concept of "Maximum benefit for minimum cost" where cost can denote the development and integration costs of the optimization.

\begin{itemize}
    \item Some optimizations are hard to implement
    \item Some optimizations require large compilation time
    \item Some optimizations have low payoff (the benefits) and it is often difficult to quantify payoff.
\end{itemize}


%% FINAL PROJECT FOR ANALYTICAL MODELS
%% last modifed 05/08/06
\documentclass[times,12pt,fullpage]{article}

\usepackage{hyperref}
\usepackage{times}
\usepackage{amsmath,amssymb,amsfonts} % Typical maths resource packages

\topmargin 0.0in \oddsidemargin -0.2in \evensidemargin 0.8in
\textwidth 6.80in \textheight 8.6in
\parskip 7.6pt
\parindent 0.25in

\makeindex
\usepackage{fancyhdr}
\thispagestyle{fancy} \pagestyle{fancy} \fancyhf{}
\lhead{Multiprocessor Modeling Guide} \chead{} \rhead{Bennett
\thepage} \lfoot{} \cfoot{} \rfoot{}

\title{Modeling Guide for Multiprocessing Systems}
\author{Matthew Bennett \\
{\small School of Computing, University of Southern Mississippi} \\
{\small {\em matthew.bennett@usm.edu} \ Typeset in \LaTeX on \
\today{} } }
\date{ }

\begin{document}
\maketitle
\begin{abstract}\noindent A tremendous variety of techniques exist for the modeling and
subsequent simulation of multiprocessor systems. This paper presents
several of the approaches of Marsan et al. \cite{marsan}, in a
condensed format suitable to practical systems engineer or computer
scientist. Most of these techniques are based upon Generalized
Stochastic Petri Nets. The guide covers Bucci's Preemptive Time Nets
\cite{bucci} to fill in gaps where continuous-time models fail.
\end{abstract}

\section*{Preliminaries}
A working knowledge of multiprocessor architecture, statistical
distributions, and the modeling technique of General Stochastic
Petri Nets is required. For the latter, Murata \cite{murata}
provides a succinct and quick introduction. Markov chains are
mentioned, but the results presented do not require thorough
understanding of the device, since the work has already been
completed and proven by others.

Many of the graphs or numerical results in Marsan \cite{marsan}, and
\cite{murata} are similar enough to be asymptotically bound for some
constant. The purpose of this guide is to provide a straightforward
and efficient means of guiding the systems architect through the
modeling process and on into either simulation or analysis. The
reader may check those sources cited for more in-depth coverage of
very detailed models, since only the simplest models are presented
within.

A few taxonomic niceties will save the designer some time. The
modeling phase has been split into five categories (as in
\cite{marsan}). After the model is developed, the developer may move
on to modeling fault tolerance (\ref{fault tolerance}) or
verification (\ref{analysis}).

\section{Bus-Contention Free Architectures}
The most trivial multiprocessor system to model is one that
experiences no contention whatsoever. The canonical example in
academia (as in \cite{marsan}) is a crossbar-connected switch
between $n$ processor elements and $m$ memory elements. An analog of
crossbar switch is the Plain Old Telephone System (POTS) exchange.
These are rarely seen in practice, since the number of switches and
interconnections increases by a factor of $\Theta(mn)$, making the
interconnection network extremely expensive for even small numbers
of processors and memories.

Marsan \cite{marsan} mentions that most time spent in any
multiprocessing time is idle time due to contention for a common
resource such as a bus. It is therefore no surprise that modeling
contention-free systems is strait-forward in terms of waiting. The
analysis instead concentrates on statistical modeling of random
activity within the system. This is easily accomplished using
queueing networks, where processors are nodes without queues, and
memory modules are nodes with finite queues \cite{marsan}. For a
crossbar architecture, there is an edge from every processor element
to every memory element, since any processor may read any memory
element regardless of other processors. Some simplification occurs
when two or more processors try to access the same memory node in a
given time step: only one is served. Simulation can be performed
using the Monte Carlo method, with memory accesses being produced
using a random variable (``Markovian process'' is exponential) on
all producers (processors) and consumers (memories) in the net.

As more becomes known about the system under analysis, more can be
included in the modeling queueing network. Marsan \cite{marsan}
provides a useful ontology by asking if the devices are synchronous,
or asynchronous, and also whether memory accesses are uniform or
non-uniform between processors.

Marsan gives a number of case studies. The simplest one is
attributed to Bhandarkar \cite{bhan}, who assumes that memory
accesses are uniform across processors, with lost requests in the
case of memory contention. Bandarkar creates a vector of states that
the queueing network can take on, and then further simplifies the
amount of information at hand by creating super-states, or {\em
equivalence classes} of mutually indistinguishable states (since
processors and memories are assumed identical). He derives a formula
(Marsan fig 6.2) which can be used to algorithmically calculate
super-state transition probabilities in a Markov process, and Marsan
(p. 124, 136) gives further approximation formulas from others' work
which can be calculated in better time on a computer. These
approximations are not as important as they once were, because
computers are substantially more capable today than they were in
1986, at the time of publication. They are all valid, as they are
equivalent to the exact result obtained by Bandarkar \cite{bhan}. A
good approximation closed-form for the number of busy memory modules
$ \beta $ is given by Rau \cite{rau} , using binomial approximations
for all events (equation \ref{raus}). In this formula, $m$ counts
memories, and $p$ counts processors.
\begin{equation}\label{raus}
\frac{\sum_{i=0}^{\text{min}(m,p)-1} 2^i \binom {m-1}{i} \binom
{p-1}{i}}
{\sum_{i=0}^{\text{min}(m,p)-1}\frac{2^i}{i+1}\binom{m-1}{i}\binom{p-1}{i}}
\end{equation}
Delay models, in which memory contention requests are queued instead
of dropped, were also considered my multiple authors. Many of the
results were similar, with memory utilization expectedly being
better with the addition of queues to memory \cite{marsan} (136).
Rau's formula given above is an excellent minimum bound for back of
the envelope calculations for networks without bus contention.

\section{Shared Memory Systems}

Most real systems are not free of bus contention, because of the
nonlinear overhead cost of building complete connection networks.
When bus contention is introduced, the location of the bus with
respect to the memory elements and processors plays a vital role in
the performance and contention of the system as a whole. In this
section, we investigate the subset of systems for which a number of
processors each have a private local memory, connected by a local
bus, and must contend for access to a shared memory, which requires
control of one of one or more global buses. This multiprocessor
architecture is common to many multiprocessor, multi-core
workstations as well as supercomputers such as the classic Cray
series. Classical multi-threading problems, including deadlock
prevention and avoidance, starvation, and mutual exclusion must all
be taken into account for any shared memory systems. Marsan
\cite{marsan} lays out a number of simplifying assumptions for
modeling shared memory systems. Namely, they are: no delay when
capturing the bus (other than normal bus contention), CSMA/CA bus
discipline, and immediate release of the bus upon completion (as
with capture). Marsan uses the term ``active state'' to mean that a
processor is not currently accessing memory, ``waiting state'' to
mean that a processor is waiting for the bus or memory, and ``access
state'' to mean that the processor is currently in memory.

\subsection{Single-Bus Shared Memory Systems}
The architecture is defined by Marsan to be a single global bus
connecting a number of local buses to the shared memory resource.
Each local bus serves a single processor and its private memory.

He chooses three values to take into consideration: access times,
active times, and

\subsubsection{Exponential Distribution}
The Exponential Distribution represents . In a generalized
stochastic petri net, a single exponentially distributed random
event can be represented by a place connected to a timed transition,
where the firing rate of that transition takes on the parameter
$\lambda$ of the exponential distribution, and determines the
likelihood of that even firing depending on a continuous time
variable \cite{marsan}. An example of an Exponentially distributed
random variable is likelihood of a webserver receiving a packet at a
particular time.

\subsubsection{Erlang-k Distribution}
An Erlang-K distribution models k Exponentially distributed random
events which depend upon one another. They must occur in {\em
serial}. In the Petri Net representation, Erlang-K events should be
represented by a k-chain of exponential General Stochastic
Continuous-Timed Petri events \cite{marsan}. An example of an Erlang
distributed random variable is the number of packets to a web server
at a particular instant of time.

\subsubsection{Hyper-Exponential Distribution}
A hyper-exponentially distributed random variable counts the number
of simultaneous occurrences of an exponentially distributed random
event, such as processors trying to access the same piece of memory
simultaneously.

\subsubsection{Queueing disciplines}

Marsan \cite{marsan} defines three queueing disciplines for dealing
with memory access contention, and mutual exclusion assurance. Fixed
priority means that the processors have some preordained priority
for which can preempt one another for memory access. Process sharing
is like round-robin in that processors take their turn accessing
memory in the order they attempt to get in, but they are guaranteed
access because of some global device, like in time sharing. First
come first serve simply operated like a FIFO queue, but provides no
guarantee that a process will give up the bus to allow others in.

All of the combinations of these ideas can be analyzed, and have
been in several different papers. Again, see Marsan p. 157 for a
full bibliography of works. A simplified queuing network model or
Markov chain was successfully employed to deal with simple cases in
which access times and/or active times were equally exponentially
distributed, but Petri nets were used by Marsan to get results for
the more complex models where distribution was hyper-exponential,
Erlang, or more complicated.

In all cases, the numerical results of Marsan seem to indicate that
modeling with a generalized stochastic Petri net can produce
approximate simulation results fairly close to the analytical
results for simpler single-bus shared memory systems. The numerical
results show that the queueing discipline is not an important
consideration when modeling small systems. Since Petri nets are more
graphical and more intuitive than analytical methods, and because
the mathematics become complicated for very complex systems, the
recommended method for modeling anything complex is Generalized
Stochastic Petri nets as described in the 2nd half of Marsan
\cite{marsan} (Chapter 7 and the beginning of Chapter 8).

\subsection{Multi-Bus Shared Memory Systems}
Multi-bus Shared Memory systems can be modeled with Petri nets using
the same techniques as single-bus shared memory systems. Generally,
the number of buses are represented as tokens in a resource pool,
which is a place. Depending upon the architecture of the system, and
because of Marsan's simplifying assumptions for shared memory
systems, immediate transitions usually follow the global bus
resource pool (capturing and releasing being instantaneous, but
active, access times being exponential).

Marsan gives a full analytical treatment to many complicated
multi-bus systems, but that is unnecessary since most systems can be
easily intuitively modeled with Petri nets. Marsan et al. first
obtains upper and lower bounds limiting the gain or loss of
processing power, so any numerical results given by Petri net
modeling should fall within the domain on \cite{marsan} (p.
186-189). This is generally a good methodology, since most
bottlenecks in this configuration are assumed to stem from global
bus contention.

\section{Distributed Memory Systems}
Shared memory systems are not the only type of distributed system. A
true distributed system uses a paradigm more like message passing.
This architecture consists of one or more global buses connecting
local buses. Each local bus contains a processor and at least one
memory element. Unlike previous shared memory systems, there is no
memory connected to the global bus(es). Instead, the memory at each
local bus is used by any processor in the system, and both local and
global buses must be secured at each access step. An example of this
type of system is a computer network utilizing file sharing. Because
of the high dynamism of such networks, analysis of any but the
simplest distributed memory systems is impossible with queueing
nets. Marsan \cite{marsan} proposes tens of architectures that are
both single global bus and multiple global bus and also incorporate
distributed memory, but does not give any indication of extending
this methodology to generalized distributed systems.

To model any distributed memory systems, a Petri net should be
employed, and numerical results obtained. The reachability tree of
the Petri net can then be used to discover some analytical artifacts
of the system, such as whether it is live, as described in
\cite{murata}.

\section{Extended Petri Nets}

Bucci \cite{bucci} describes an extended version of Petri nets where
the continuous time model is replaced with discrete time events. A
global clock of some sort . A ``firing event'' occurs whenever the
random variable distribution coincides {\em as well as} the time has
come for that transition to fire. The idea of this extension is that
a Petri net can operate within a discrete time domain, following a
global clock, while keeping the ability to induce transitions from
state-to-state concurrently (ie using threads or a similar
mechanism). This dramatically increases the simulation uses of GSPN
models, but does not really add much to their descriptive power.
Bucci's method should be taken into account any time that a
simulation must occur, especially in a simulation with lots of
simultaneity.

\section{Fault Tolerance Modeling}\label{fault tolerance}
If systems include repairable components, a GSPN event (place /
timed transition) can represent the break and rapair of any
component in the system, just as

\section{Analysis and Verification}\label{analysis}
Generally, any sort of Petri Net model is good for modeling a
specific system, but can only provide numerical results. In order to
assure that those results are statistically significant, many runs
(at least 30) should be performed in simulation, and every variable
must be checked for consistency with a real system. One technique
which is liberally applied by many \cite{murata} \cite{marsan} is to
use a verifiable Markovian process or queueing network to provide
upper and lower analytical bounds. At this point, the system can
simply be built and run. Analysis can be extremely difficult, so
most of the time the Petri net approach will be more useful for the
budding system developer, without rigorous mathematical background.

\begin{thebibliography}{5}

\bibitem{marsan} Marsan, N. Balbo, G. Conte, G. "Stochastic Petri Nets." Performance Models of Multiprocessor Systems. Cambridge: MIT Press. 1986. pp. 72 - 98.

\bibitem{bucci}  Bucci, G. Correctness Verification and Performance Analysis Using Stochastic Preemptive Time Petri Nets. IEEE Transactions on Software Engineering. Vol 23, No 11. November 2005. pp. 913 - 937.

\bibitem{murata} Murata, T. Petri Nets: Properties, Analysis, and Applications. Invited Paper. Proceedings of the IEEE. Vol 77. No 4. April 1989. pp. 541 - 579.

\bibitem{bhan} Bhandarkar, D. Analysis of Memory Interference in
Multiprocessors. IEEE Transactions on Computers Vol 24. No 9.
September 1975. 897-908.

\bibitem{rau} Rau, B. Interleaved Memory Bandwidth in a Model of a Multiprocessor Computer System. IEEE Transactions on Computers Vol 28. No 9.
September 1979. 678 - 681.

\end{thebibliography}

\end{document}


% Template for PLoS (Adapted for Frontiers, MdK))
% Version 1.0 January 2009
%
% To compile to pdf, run:
% latex plos.template
% bibtex plos.template
% latex plos.template
% latex plos.template
% dvipdf plos.template

\documentclass[12pt]{article}

% amsmath package, useful for mathematical formulas
\usepackage{amsmath}
% amssymb package, useful for mathematical symbols
\usepackage{amssymb}

% graphicx package, useful for including eps and pdf graphics
% include graphics with the command \includegraphics
\usepackage{graphicx}

% cite package, to clean up citations in the main text. Do not remove.
\usepackage{natbib}
\usepackage{cite}
\usepackage{lineno}
\usepackage{color}

% Use doublespacing - comment out for single spacing
%\usepackage{setspace}
%\doublespacing

% Text layout
\topmargin 0.0cm
\oddsidemargin 0.5cm
\evensidemargin 0.5cm
\textwidth 16cm
\textheight 21cm

% Bold the 'Figure #' in the caption and separate it with a period
% Captions will be left justified
\usepackage[labelfont=bf,labelsep=period,justification=raggedright]{caption}

% Use the Frontiers provided bibtex style

% Remove brackets from numbering in List of References
%\makeatletter
%\renewcommand{\@biblabel}[1]{\quad#1.}
%\makeatother

% Leave date blank
\date{}

\pagestyle{myheadings}
%% ** EDIT HERE **

% the subfigure package
\usepackage{caption}
\usepackage{subcaption}

\DeclareCaptionFormat{subfig}{\figurename~#1#2#3}
\DeclareCaptionSubType*{figure}
\captionsetup[subfigure]{format=subfig,labelsep=colon,labelformat=simple}

% the listings package
\usepackage{listings}
\lstset{
language=C++,
basicstyle=\footnotesize,
numbers=right,
numberstyle=\tiny,
frame=tb,
columns=fullflexible,
showstringspaces=false
}

%% ** EDIT HERE **
%% PLEASE INCLUDE ALL MACROS BELOW

%% END MACROS SECTION

\begin{document}

% Title must be 150 characters or less
\begin{center}
\thispagestyle{empty}
{\Large
\textbf{Parallel MIIND: a Parallel Framework for Populations of Spiking Neurons Using Population Density Techniques}
}
\vspace{1.0cm}
% Insert Author names, affiliations and corresponding author email.
\\
Marc de Kamps$^{1,\ast}$,
David Sichau$^{2}$
\vspace{0.5cm}
\\
\bf{1} School of Computing, University of Leeds, Leeds, West Yorkshire, UK
\\
\bf{2} LAV, Institute of Energy Technology, ETH Zurich, 8092 Zurich, Switzerland
\\
\vspace{0.5cm}
$\ast$ E-mail: m.dekamps@leeds.ac.uk
\end{center}

% Please keep the abstract between 250 and 300 words
\section*{Abstract}

%\section*{Author Summary}
\newpage
\setcounter{page}{1}
\linenumbers

\section{Introduction}
Multiple Interacting Instantiations of Neural Dynamics \citep{dekamps2008} (MIIND) is a simulation framework designed for modeling large networks of
neuronal populations. It is written in C++ and employs MPI \citep{gropp1998} for parallelisation. Python extensions can be provided on request.
Due to its design philosophy it can be easily applied to modeling
biological processes outside computational neuroscience. MIIND abstracts the interaction between neuronal populations into an interaction
between nodes on a directed graph. The neuronal processes describing the activity of the population live on these nodes as objects and evolve
the population’s state. In general we aim to support network processes that use CPU-intensive computations to model nodes, but
require little bandwidth in the internode communication. Neuronal networks where individual spikes are less important than the transmission
of population firing rates are well supported by this paradigm. As such, the processes that are supported by this framework are trivially
parallelizable. Unfortunately, the word trivial does not apply to the actual implementation: we found that in the original code base of MIIND there
were too many obstacles to parallelisation and we decided to start from a fresh code based.

As with any framework, MIIND comes with a number of predefined algorithms, some allowing sophisticated simulations of neuronal
dynamics. The simplest algorithms implement Wilson-Cowan dynamics \citep{wilson1972}, but, uniquely,
MIIND provides an implementation of population density techniques (e.g.) \citep{stein1965,knight1972,knight1996}. They provide efficient simulations
of large populations of neurons by means of a density function over the state variables of a given neuron. Such techniques have undergone rapid
development over the last decade \citep{omurtag2000}. In practice these techniques have been mainly used for leaky-integrate-and-fire (LIF) neurons,
in the so-called diffusion limit: small synaptic efficacies and high firing rate. In  this limit the distribution of membrane potentials of the
population can be modeled by a diffusion process, using Fokker-Planck equations.

Recent progress has extended the usefulness of these technique considerably.  First, the techniques are no longer restricted to the diffusion limit
\citep{omurtag2000,dekamps2003,dekamps2006}, indeed arbitrary large efficacies can be used.  Second, they are no longer restricted to LIF neurons:
a method to model any one-dimensional neuronal model was described recently \citep{dekamps2013}, and will soon be part of the code base.
Others have described density approaches based on 2D neuronal models, including synaptic kinetics, and have established that networks
of 2D populations are computationally still competitive, compared to direct simulation.

The methods described here are at least an order of magnitude more efficient than direct simulation (a real second for a population of LIF neurons
can be modeled in well under 0.5 s simulated time \citep{dekamps2006}). Nevertheless, the computational load is such that a network of hundreds of such
populations would lead to lengthy simulation runs on a serial machine. Therefore investigating the performance of such a network provides a good
benchmark for the scalability of the parallelisation approach.

% Results and Discussion can be combined.
\section{Results}
\subsection{The Framework}
In \citep{dekamps2008} we have outlined the general design ideas of the framework: populations can be modeled by nodes on a directed graph,
which interact via the edges. The evolution of a population is determined by an Algorithm object which lives on the node and which maintains an
AlgorithmGrid. A central simulation loop visits each node in term, collects the contributions from every other node connected to this node and calculates
its input contribution weighted by the link value of the connection. Thus an instantaneous weighted input contribution from the entire network to node will
be calculated, and this will be used by the algorithm to evolve its node state a small moment in time. This will result into an update of the AlgorithmGrid.
By repeating this process for all nodes in the network for a desired period of time, a simulation of the network process is implemented.
The central idea is represented in Figure 1. In the next section we will illustrate how network creation and configuring a simulation is done
from the user's perspective.
\subsection{Simulating Networks as a Use Case}
Here we present a small program that performs a simulation of two neuronal populations: one excitatory and one inhibitory, driven by a common external input.
In listing \ref{code:twopopnet} the shortened code of this simulation is provided.
To increase the readability parameters are not provided.
As a placeholder for the parameters the word PARAM is used.
Additional namespaces are ignored.
For the whole program the reader is refered to the source code where this example is provided as \texttt{two-population.cpp}.
In the next section the setup of an simulation is explained step by step with the two population network.
\begin{lstlisting}[caption=Simulation of a two neuronal population network.,label=code:twopopnet]
int main(int argc, char* argv[]) {
#ifdef ENABLE_MPI
	boost::mpi::environment env(argc, argv);
#endif
	try {
		NodeId id_cortical_background;
		NodeId id_excitatory_main;
		NodeId id_inhibitory_main;
		MPINetwork<OrnsteinUhlenbeckConnection, CircularDistribution> network;

		// Create cortical background, and add to network
		populist::RateFunctor<WeightValue> cortical_background(PARAM);
		id_cortical_background = network.addNode(cortical_background, EXCITATORY);
		// Create excitatory main population
		PopulationAlgorithm_<OrnsteinUhlenbeckConnection> algorithm_excitatory(PARAM);
		id_excitatory_main = network.addNode(algorithm_excitatory, EXCITATORY);
		// Create inhibitory main population
		PopulationAlgorithm_<OrnsteinUhlenbeckConnection>  algorithm_inhibitory(PARAM);
		id_inhibitory_main = network.addNode(algorithm_inhibitory, INHIBITORY);

		// Background and excitatory connection only differ in x, 1- x
		network.makeFirstInputOfSecond(id_cortical_background, id_excitatory_main, PARAM);
		// Excitatory connection to itself
		network.makeFirstInputOfSecond(id_excitatory_main, id_excitatory_main, PARAM);
		// Background connection to I
		network.makeFirstInputOfSecond(id_cortical_background, id_inhibitory_main, PARAM);
		// E to I
		network.makeFirstInputOfSecond(id_excitatory_main, id_inhibitory_main, PARAM);
		// I to E
		network.makeFirstInputOfSecond(id_inhibitory_main, id_excitatory_main, PARAM);
		// I to I
		network.makeFirstInputOfSecond(id_inhibitory_main, id_inhibitory_main, PARAM);

		network.configureSimulation(PARAM);
		network.evolve();
	} catch (std::exception&e) {
		std::cout << e.what() << std::endl;
		return 1;
	}
	return 0;
}
\end{lstlisting}

For parallel MIIND it is important to initialise the mpi environment (see line 3 in listing \ref{code:twopopnet}).
To allow that the code is compilable without an mpi compiler the preprocessor is used to remove the code if mpi is not enabled.
These 3 lines are the only one where actually the user is forced to handle the mpi environment direct.
All other mpi related code is hidden from the user.

After the mpi environment is initialised one can start with setting up an simulation.
The network is created by instantiating a network object (line 9).
The network gets two template parameters.
The first parameter defines the connection type of the network.
In the example above this is a Ornstein Uhlenbeck type of connection.
The second parameter is the distribution of the nodes.
These parameter controls the distribution of the nodes in an mpi environment.
More details will be provided in the next section.

So far only the type of the network is defined.
To define the structure of the network two methods are needed.

The first method \texttt{addNode} adds a new nodes to the network (see line 13, 16 and 19).
This method expects the algorithm executed on this node and the type of the node.
In the example above 3 nodes are generated.
The cortical background driving the simulation with a constant firing rate (line 13).
And an excitatory and inhibitory node where leaky-integrate-and-fire neurons are simulated (line 16 and 19). %what reference for that

The method \texttt{addNode} returns the node id of the generated node.
The node ids start from 0 and are incremented by one for each additional nodes.
The order of the node ids correspond to the chronological order of the calls to \texttt{addNode}.
These ids are needed to identify the individual nodes and are needed for the second method to generate the network structure.

After the creation of the nodes the interactions of the network needs to be created.
Therefore directed connections between two nodes are generated.
Directed Connections between two nodes are generated by calling the method \texttt{makeFirstInputOfSecond}.
The method expects as parameters the node id of the first and the second node in the connection and the parameters of the connection (see line 22 - 32).
In the case of the two population network six connections are generated.
The background node is connected to the exhibitory and the inhibitory node.
The exhibitory and inhibitory nodes are connected to itself and to each other.
With these two methods the structure of the network is constructed.
The structure of the generated network can be seen in figure \ref{fig:twoPopNetwork}.

With these two methods all kind of networks can be generated.
To execute a simulation on that network the simulation needs to be configured.
The configuration is done by providing the simulation parameters to the method \texttt{configureSimulation} (see line 34 in listing \ref{code:twopopnet}).
When the simulation is configured the main simulation loop can be executed with the method \texttt{evolve}.

With the example provided above one can see that the generation of a simulation is straightforward.
It does not need any knowledge about the underlying parallelisation. More details about the parallelisation is provided in the next section.

\subsection{The Simulation Engine: An MPI-enabled Simulation Loop}%TODO(MdK) it is not only the simulation loop which is parallised. Maybe another title would be more suitable
In this section we will describe the mpi-based infrastructure that forms the heart of the framework.

To start with an illustration will be used.
A person wants to solve a large equation consiting of lot of different smaller equations.
In the beginning the person solved the equation on its own.
However it took a large amount of time for one person to solve the equations.
Therefore the management decided to hire additonal people to speed up the process.
To solve the equations faster each person needs to know what equation it should solve and sometimes the extension of another person equation is needed.
To share results and work they need to communicate with each other.
When people communicate with each other they cannot solve equations on the same time.
Additional the they might need lot of extensions of other people so they need to wait until the other people are finished with their calculation.
These two problems show some of the largest issues with parallelisation.
One is the overhead for communication.
And the other the latency.
The illustration above allegorize the message passing interface.

The Message passing interface mpi is a standard to develop applications aimed at large scale parallel applications.
An mpi program resembles an ordinary program written for a serial processor.
This similarity is deceptive, however.
In reallity the code is executed in parallel on different mpi processes.
A mpi process can be either executed on a single core of a multi core processor or on a different CPU.
These execution in different processes results that each process has its own memory space and not a shared memory space like a serial program.
That increases the amount of work to share data between the mpi processes.
For sharing data messages are sent between the individual mpi processes.
To send messages conncetions between the individual processes are needed.
The connections between the individual processes can be either a internal bus or a simple network connection.
However mpi is normally executed on super clusters with a very fast network connection (e.g. InfiniBand).

To speed-up a serial program with mpi the serial program is decomposed in parts that can be executed in parallel.
A serial program that is not embarrasing parallel results in data that needs to be shared between the mpi processes.
Therefore the individual mpi processes needs to communicate with each other to share the data.
To share the data between the mpi processes messages are sent from one mpi process to another with the data.
Compared to computation a message is very slow.
To reduce the overhead of parallelisation it is very important to reduce the number of messages to the minimum.

For the parallelisation of MIIND the idea was to distribute the nodes to MPI processes.
Each node can locally envole one time step without any communication.
The evolve step is responsible for most of the run time of an simulation the run time should be reduced by parallising these method.
To distribute the nodes to the mpi processes a distibution function is used.
As a default distribution function the circular distribution (\texttt{CircularDistribution}) is provided.
The circular distribution assignes the nodes in a circular manar to the mpi processes by using the modulor operator.
For example the network \ref{fig-realNetwork} would be represented internaly like \ref{fig-internalNetwork} when 4 mpi processes are available.
As the default distribution is not always the best choice it can be easily exchanged.
A new distribution can be implemented by implementing the virtual methods of the base class \texttt{NodeDistributionInterface}.
The implementation of a new distribution might be useful if another distribution would decrease the number of messages across the MPI processes (the dotted arrows in figure \ref{fig-internalNetwork}).
To implement a new distribution one might make use of additional knowledge of the network to decrease the messages over MPI process boarders.

With the distribution function the nodes are assigned to distinct mpi processes.
As the nodes are distributed over several mpi processes major changes inside the library where needed.

The parallelisation starts during the creation of the network.
When a node is added to the network the node is only generated on its responsible mpi process.
The method \texttt{addNode} is responsible for creating nodes.

\begin{lstlisting}[caption=The \texttt{addNode} method of the network.,label=code:addNode]
addNode(const AlgorithmInterface<WeightValue>& alg, NodeType nodeType) {
	NodeId tempNodeId = getMaxNodeId();
	if (nodeDistribution.isLocalNode(tempNodeId)) {
		MPINode node = MPINode(...);
		localNodes.insert(std::make_pair(tempNodeId, node));
	}
	incrementMaxNodeId();
	utilities::MPIProxy().barrier();
	return tempNodeId;
}
\end{lstlisting}

In listing \ref{code:addNode} the simplified method is provided.
The method starts with getting the next node id (see line 2 in listing \ref{code:addNode}.
The node id needs to be synchonised between the mpi processes to make sure that each nodes gets an unique id.
The ids of the nodes starts with 0 and increasing numbers are assigned to each node depending on the time of their initialization.
At this moment all mpi processes have the same node id.
To generate a node only on the responsibel mpi process the check on line 3 is carried out.
The \texttt{nodeDistribution} is the choosen distribution class, which checks if the node should be assigned to this mpi process.
If the node should generate on this node the node is generated and stored in the map \texttt{localNodes}, where the nodes on this mpi process are stored with their id as an key.
After the node is generated, or not the node id needs to be increased by calling the method \texttt{incrementMaxNodeId()}.
To avoid side effects the \texttt{addNode} method is synchronised.
For synchronous code mpi barriers are used which make sure that the function is only finished if all processes have executed all code.
As the generation of nodes happen only during the initialization this synchronous code has no impact on the performance.

As the mpi processes have different memory spaces the connections between the nodes cannot be stored anymore by connecting the nodes direct (via pointers or references).
The connections are generated by calls to the \texttt{makeFirstInputOfSecond} method.

\begin{lstlisting}[caption=The simplified \texttt{makeFirstInputOfSecond} method of the network.,label=code:makeFirstInputOfSecond]
makeFirstInputOfSecond(NodeId first, NodeId second, const WeightValue& weight) {
	//Make sure that the node exists and then add the successor
	if (nodeDistribution.isLocalNode(first)) {
		localNodes.find(first)->second.addSuccessor(second);
	}
	// Make sure that the second node exist and then set the precursor
	if (nodeDistribution.isLocalNode(second)) {
		localNodes.find(second)->second.addPrecursor(first, weight, nodeIdsType_[second]);
	}
}
\end{lstlisting}

In listing \ref{code:makeFirstInputOfSecond} the simplified method is provided.
As the connections cannot be stored with pointers or references it is stored by saving the node ids of the precursors respective the successor of a node.
During the \texttt{makeFirstInputOfSecond} method on the local nodes either the successor (line 4) or precursor (line 8) of a node is stored.

After the network has been generated the simulation is initialized.
To avoid to much communication between the nodes each mpi process has its own log and output file.
In the output file the activities of the local nodes are stored.
The advantage of several output files is that the node activities had not to be collected on one mpi process, which would require lot of communication.
This performance gain is paid for by a more difficult post processing.
As the activites of the nodes are distributed into several files and are not stored all in one file.

When the simulation is configured the main simulation loop is started.
The main simulation loop iterates over the local nodes and envoles them one step.
To make sure that all node have recieved and sended their activity at the beginning of each \texttt{evolve} step an barrier is introduced.
At this barrier all nodes waits until all mpi messages are successfully finished.
This barrier increased the performance as the latency of the mpi messages was hidded.
The mpi messages where sent ascynchronus such that the mpi process could continue with other computation and does not need to wait for each to be finished.

The evolve of a node does require that all nodes have the activity of their precursors.
Each node has a temporary storage (\texttt{\_precursorActivity}) of the activity of their precursors.
During the \texttt{evolve} method each node calls his corresponding algorithm and passes the activity of the precursors the weights of the connection and the type of the precursor (see line 3 in listing \ref{code:evolve}).

\begin{lstlisting}[caption=The simplified \texttt{evolve} method of the nodes.,label=code:evolve]
evolve(Time time) {
	while (_pAlgorithm->getCurrentTime() < time) {
		_pAlgorithm->evolveNodeState(_precursorActivity, _weights, time,_precursorTypes);
	}
	this->setActivity(_pAlgorithm->getCurrentRate());
	sendOwnActivity();
	receiveData();
}
\end{lstlisting}

After the algorithm had evolved the state of the node the new rate of the node is stored (line 5).
When the new activity of the node is available the node sends its new activity to all its successors (line 6).
The details of the \texttt{sendOwnActivity} method is explained below.
Additonal the node prepares itself to receive data from his precursors (line 7).
The details of the method \texttt{receiveData} is explained below.
As the  \texttt{sendOwnActivity} and  \texttt{receiveData} are asynchronous they return immediately such that the next node can be envolved without waiting for the data transmission.

The activity of a node is sent to the successors with the help of the  \texttt{sendOwnActivity} method.
In listing \ref{code:sending} the simplified method is provided.
\begin{lstlisting}[caption=The simplified \texttt{sendOwnActivity} method of the nodes.,label=code:sending]
sendOwnActivity() {
  for (auto& it : _successors) {
	if (!_rNodeDistribution.isLocalNode(it)) {
	  MPIProxy().isend(_rNodeDistribution.getResponsibleProcessor(it), _nodeId, _activity);
	}
  }
}
\end{lstlisting}
The method  \texttt{sendOwnActivity}  iterates ofter the ids of the successors of a node (see line 2 in listing \ref{code:sending}.
For each successor it is checked if the successor is on the same mpi process or not (line 3).
If not the activity of the code needs to be send with mpi to the corresponding processor (line 4).
The \texttt{MPIProxy} class is a simple wrapper over the mpi environment to avoid any bare mpi code in library.
\texttt{MPIPoroxy} is explained later into more detail.
The method \texttt{isend} sends an asynchronous mpi message to a processor with some data.
The first parameter \texttt{isend} is the identifier of the mpi process which is responsible for the successor.
The second parameter is a message identifer.
Together with the mpi process these need to be unique.
The last parameter is the actual data that is send to the other node.

After the own data sending was started each node prepares itself to recieve the activities of the other nodes.
Therefore the \texttt{receiveData} method is used.

\begin{lstlisting}[caption=The simplified \texttt{receiveData} method of the nodes.,label=code:receive]
receiveData() {
  int i = 0;
  for (auto it = _precursors.begin(); it != _precursors.end(); it++, i++) {
	if (_rNodeDistribution.isLocalNode(*it)) {
  	  _precursorActivity[i] = _rLocalNodes.find(*it)->second.getActivity();
	} else {
	  MPIProxy().irecv(_rNodeDistribution.getResponsibleProcessor(*it), *it, _precursorActivity[i]);
	}
  }
}
\end{lstlisting}

The \texttt{receiveData} method iterates over all precursors of a node (see line 3 in listing \ref{code:receive}).
For each precursor it is checked if it is local to this mpi process (line 4).
If a node is local the activity is received without any communication with a lookup in the map of the local nodes (line 5).
When a node is not local to this mpi process the method opens an asynchronous receive (line 7).
The first parameter the method \texttt{irecv} is the identifier of the mpi process which is responsible for the precursor.
The second parmeter is the identfier of the message.
Each \texttt{isend} and \texttt{irecv} method must pair depending on responsible processors and the message identifier.
The last parameter is the storage location where the location where the received activity should be stored.
It is the temporary storage of the activities of the precursors which are used during the next \texttt{evolve} step.

These are where all the magic of the parallelisation of MIIND is done.
The algorithms of Parallel MIIND have not changed significantly compared to MIIND.
For an algorithm developer only the way the data is provided has changed.
Additional he does not need to care about concurrency issues as the parallelisation is encapsuled from the algorithm.
This simplifies the development of new algorithms significantly as developers does not need to gain internal knowledge of pmiind.

The most visible change of Parallel MIIND for an application developer is the output.
As explained above each mpi process writes to its own output file.
Instead of one root file with all results for each MPI process a root file with the corresponding local nodes is generated.
Therefore the activity of the nodes are distributed to several files and need to be collected from different files for postprocessing.

To reduce the impact of pmiind all mpi related code is encapsulated in the class \texttt{MPIProxy} and can be turned off.
To improve the readability of the code the mpi related code is wrapped by the class \texttt{MPIProxy}.
Therefore when MPI methods are used boost mpi is not called direct instead the class \texttt{MPIProxy} forwards the calls to boost mpi.
Additional this class stores all open mpi requests of the asynchronous calls.
Reducing the code to handle all open communications as all mpi request are stored in one place and not scattered around the code of the library.
As all mpi related code is encapsulated in one class it can be easily turned off.
When MPI is turned off parallel MIIND can be builded and executed without mpi available on the computer.
This reduces the impact of the development of new algorithms further as the development computer does not need to have mpi installed.
After the testing the parallelisation can be turned on and one can benefit of the speed ups achieved by parallel execution on clusters.

As a parallel environment is very hard to debug a new logging class had been developed.
The class \texttt{Log} provides methods to print logging and debugging messages.
Additional it provides additional information about the mpi environment allowing to identify the mpi process which might have an error.
The generation of a huge number of debug logs is very time consuming. Therefore the logging can be turned off completely for release builds.
Instead off a simple on or off the class provides several log levels allowing a fine granularity of log messages.
This allows the usage of this class not only for debugging purposed but additional for error messages or important information during a simulation run.

\subsection{A Scalable Model of Waves in a Large Network of Leaky-Integrate-and-Fire Neurons}
Consider the local circuit displayed in Figure \ref{fig-network} (A). It consists if an excitatory and inhibitory population, which are fully connected
and driven by a common input. It is possible to predict the firing rate of the populations in the circuit using a number of simple equations
cite{amit199a}, which can be solved with the algorithms provided in MIIND (see section \ref{sec-methods}). A population density-based simulation
shows that the network indeed converges to the predicted rates.

In Figure \ref{fig-network} (C), a hexagonal network is created, whose populations should converge to the same rates. To achieve this, the efficacy
of the local EE connections are halved and each excitatory population is connected laterally with an efficacy that makes up for the reduced
efficacy in the self-connection (section \ref{sec-methods}). Such a network can be easily expanded into an overall hexagonal structure that consists
of a number of rings, and the number of local circuits in the network scales quadratically with the number of rings. By simply expanding the number
of rings, we can investigate the scalability of the network. Finally, a burst from an external input is introduced to the central population.

When no delays are introduced in the lateral connections, the network quickly converges to the same firing rate as for the single local circuit. When
the burst occurs, it is transmitted without delays to the outer populations, which respond immediately. Although this may seem odd, large population
of neurons can respond considerably faster that the time membrane constant of the comprising neurons would suggest. Indeed, population density methods
that model infinitely large populations responds immediately.

When delays are introduced the dynamics of the network becomes very interesting: the network never quite settles in its equilibrium mode as
delayed contributions from further away keep  causing local disturbances (Figure \ref{fig-sim}. The burst is now clearly delayed and
ripples from inside to outside. The outside nodes, however functions as a secondary source, and a rebound on more central populations can
also be discerned (see Figure \ref{fig-sim}).

\section{Discussion}

\section*{Methods}
\label{sec-methods}
\subsection{Network Model}
\subsubsection{Finding Steady State Firing Rate}
\label{sec-steady}
The architecture of the network has been described in Figure \ref{fig-bb}. It consists of a local circuit comprised of an excitatory and an inhibitory
population that are fully connected. We will first describe how this local circuit the local circuit is simulated.
In section \ref{sec-hexagon} we will then describe how this circuit is embedded in an overall hexagonal structure.
Each population
is assumed to be comprised of a large number of spiking LIF neurons. It is assumed that both the populations input and output spike trains
are Poisson distributed and that populations are connected sparsely but by a large number of connections. Under these assumptions an expression
can be found that expresses the population's output firing rate as a function of the input contributions: the so-called gain functions.
Assume a population of LIF neurons with time constant $\tau$, threshold $V_{th}$, reset potential $V_{reset}$ and reversal potential $V_{rev}$
receives Poisson distributed spike trains from another external population, which has a population firing rate of $\nu_{ext}$.
Furthermore, assume that each such input spike raises the membrane potential
instantaneously by a magnitude $h$ (delta peak synapses), that $N_{input}\epsilon$ external neurons connect to a neuron in the population on average,
where $N_{input}$ is the average number of input neurons and $\epsilon$ is a sparseness parameter. The average contribution to the
membrane potential per time unit $\tau$ is then given by:
\begin{equation}
  \mu = \tau \epsilon \nu_{ext} J,
\end{equation}
the variability by:
\begin{equation}
  \sigma^2 = \tau \epsilon \nu_{ext} J^2,
\end{equation}

The population's own firing rate is then given by:
\begin{equation}
\nu = \phi( \mu, \sigma)
\label{eq-transduc}
\end{equation}
with $\phi$ given by:
\begin{equation}
\phi(\mu, \sigma) = \int^{bla}_{ble}du \exp(u^2)(1 + )
\end{equation}
If the population is large, Equation \ref{eq-transduc} is a very good approximation of the firing rate that would be found in a simulation
of LIF neurons indeed. In a network of LIF neuron populations this leads to self-consistency relationships in term of gain functions, that describe
the steady state firing rates that eventually may emerge \footnote{This is not necessarily the case. See section \ref{sec-Fokker-Planck}.} if external contributions to the network are kept constant. For each population $i$ one has:
\begin{equation}
  \nu_i = \phi_i( \mu_i, \sigma_i), i = 0 \cdots N-1
  \label{eq-steady}
\end{equation}
with:
\begin{equation}
  \mu_i = \tau_i (\sum^{N-1}_{j=0} J_{ij}\epsilon_{ij} N_j\nu_j + \sum^{N_{ext}-1}_{k=0} J_{ik} \epsilon_{ik}N_k \nu_k)
  \label{eq-mui}
\end{equation}
\begin{equation}
  \sigma^2_i = \tau (\sum^{N-1}_{j=0} J^2_{ij}\epsilon_{ij} N_j\nu_j + \sum^{N_{ext}-1}_{k=0} J_{ik} \epsilon_{ik}N_k \nu_k)
  \label{eq-sigi}
\end{equation}
The $J_{ij}$ are now the (average) efficacy of population $j$ onto $i$, which can be positive (excitatory) or negative (inhibitory)
with the understanding that  ... Dale's law.

Here $N$ is the number of populations in the network, and the existence of a number of $N_{k}$ external sources is assumed
whose activation drive the dynamics of the network. If a steady state exists where the external populations fire at a constant rate,
then also the network will settle into a steady state where each population $i$ fires with a constant rate. These rates can be found
with Equations \ref{eq-mui} and \ref{eq-sigi}. Although this is an elaborate process, computationally it is much faster than
simulating a network of large populations, waiting until the network relaxates and then finally collating population statistics.

One can find solutions to this set of equations by postulating a state $E_i$ for each node $i$, which evolves according to:
\begin{equation}
  \frac{dE_i}{dt} = -E_i + \phi_i (\mu_i, \sigma_i)
  \label{eq-wc}
\end{equation}
One can pick 'sensible' values  $E_i(0)$ and numerically solve Equation \ref{eq-wc}. After some time, the values $E_i$ will start to approximate
those of the $\nu_i$, given by Equation \ref{eq-steady}.

Although this seems a roundabout way of doing things, in practice it is convenient since the solution of sets of Equations such
as \ref{eq-wc} are precisely what MIIND is designed to do. There is an algorithm provided with MIIND that evolves a node's state
$E_i$ according to Equation \ref{eq-wc}. This allows one to play with the values of connectivities, synaptic efficacies, etc.
and observe their effect on the steady state firing rates of the network.

\subsubsection{Modeling the Network Dynamics}
\label{sec-Fokker-Planck}
The methods described in the last section are well established techniques to find the steady state firing rates
of networks of LIF neuron populations, but there are sitautions where Equation \ref{eq-wc} provides a poor
model for the transient dynamics of the network, or indeed the stability of the steady state solution. It may well be possible
that the network never settles into a steady state. A much better description of the firing rate of neurons is given
by population density methods. Population density methods do not resort to the simulation of large groups of neuronal models,
but instead they define a density function over the neuronal population. Assume the existence of some minimum potential, that
any neuron in the population is likely to cross with vanishing probaility.
One can then define a density function $\rho(V,t)$, defined as the fraction of neurons in the population that have
their membrane potential in interval $[V, v+ dv)$. Clearly:
\begin{equation}
  \int^{V_{th}}_{V_{min}} \rho(V,t) dv = 1
\end{equation}
It can be demonstrated that all relevant quantities that exist at the population level, such as its firing rate, can be calculated
from thsi function. Although mathematically more complex, this method is an order of magnitude more efficient than large
scale neuronal simulations.

Assume the same situation as in the previous section where we defined the gain function of a population, and consider another
population connecting onto this population with all parameters defined in the same way as before.

It can be demonstrated (see \citet{omurtag2000} for a lucid demonstration) that:
\begin{equation}
  \frac{\partial}{\partial t} \rho(V,t) + \frac{1}{\tau}\frac{\partial \rho(V,t)}{\partial V} = \nu_{ext}( \rho(V-h,t) - \rho(V,t)) +\nu(t) \delta(V -V_{reset})
  \label{eq-balance}
\end{equation}
where $\nu(t)$ is given by:
\begin{equation}
  \nu(t) = \int^{V_{th}}_{V_{th} - h} rho(v,t) dv)
\end{equation}
Moreover, one must impose:
\begin{equation}
  \rho(V_{th},t) = 0
\end{equation}
as a boundary condition. Note that in the limit where $h \rightarrow 0$, but $\nu_{ext} h $ is constant, one can Taylor expand the right hand side of
Equation \ref{eq-balance} to obtain the form:
\begin{equation}
\frac{\partial \rho(V,t)}{\partial t} + \frac{1}{\tau}\frac{\partial}{\partial}\left\{(V - \mu)\rho)\right\}  + \frac{sigma^2}{2 \tau}\frac{\partial^2}{\partial v^2} \rho(V,t) = \nu_{ext} \delta (V - V_{reset}),
\label{eq-fp}
\end{equation}
which is a Fokker-Planck equation.
Introducing $\mathcal{L}_{FP}(\mu, \sigma)$ as a shorthand for the second order differential operator defined by Equation \ref{eq-fp}.
The network can now be described by the following set of coupled Fokker-Planck equations:
\begin{equation}
  \mathcal{L}_{FP,i}(\mu_{i},\sigma_{i})\rho_i(V,t) = nu_{i} \delta(V - V_{reset}),
\end{equation}
with:
\begin{equation}
  \nu_i = \int^{V_{th}}_{V_{th}-h} \rho_i(V,t)
\end{equation}
and $\mu_i$, $\sigma_i$ given by Equations \ref{eq-mui} and \ref{eq-sigi}.

Although the state of each individual node is much more complex than for the simulation described in the previous section. The network structure
of the simulation is identical. Again, MIIND provides algorithms that can evolve the state $\rho_i$ of node $i$ numerically, and in this way
an accurate simulation of transient network dynamics and stability can be simulated without having to model thousands of nodes individually, and then
collating their population statistics. The gain in overall efficiency is still large enough to warrant the added complexity of the poulation
density approach.

\subsubsection{A Hexagonal Network of Populations}
\label{sec-hexagon}
Using parameters described as biologically plausible elsewhere \citet{amit1997}, and tinkering with them to obtain
steady state firing rates of 2 Hz for the excitatory population and 5 Hz for the inhibitory population, we arrive at the
connectivity parameters described in Table \ref{tab-con}
\begin{table}[h]
\label{tab-con}
For these parameters the network is stable, although not far away from instability: a mere change of $x$ in $J_{EE}$ brings the network
in a permanent oscillatory state. The convergence of the firing rates and the steady state density distributions are given in Fig \ref{fig-bingo}.
The network can maintain its low level firing rates due a balance of excitation and inhibition and individual neurons in this regime
would fire irregularly.

We now reduce take one such local circuit at the centre of a hexagon, and place six others at the cornes of the hexagon. We
reduce all intra-circuit parameters $J_{EE}$ by a factor half. We connect all excitatory populations laterally by with an efficacy
$\frac{1}{2N_{neigh}}$ where $N_{neigh}$ is the number of neighbours. For the centre node $N_{neigh}= 6$ for the outside nodes $N_{neigh} = 3$.
This ensures that when all excitatory nodes fire with a firing rate of 2 Hz, the neighbour nodes make up for the lost connectivity,
the previous self-stable. Clearly, hexagonal networks with more rings can be created by taking following this procedure, always taking
the correct number of neighbours of a node into account. In general the number of populations in the network $N_{pop}$ is given
by:
\begin{equation}
N_{pop} = 2 ...,
\end{equation}
where the factor of two accounts for there being two populations in the circuit. Finally, it is possible to provide a delay
\caption{Connectivity parameters for the local cortical circuit.}
\end{table}
\section*{Acknowledgments}
This parallelization of MIIND was carried out as a Google Summer of Code 2012 project. We gratefully acknowledge Google's support.

% The bibtex filename
\bibliographystyle{front}
\bibliography{pmiind}

\section*{Figure Legends}
\begin{figure}[tbp]
	\includegraphics[width=0.99\textwidth]{twoPopNetwork.eps}
    \label{fig:twoPopNetwork}
    \caption{The network of the two population network simulation.}
\end{figure}

\begin{figure}[tbp]
	\begin{subfigure}[b]{0.48\linewidth}
		\includegraphics[width=0.99\textwidth]{realNetwork.eps}
        \caption{An example network representation. The circles correspond to the neurons and the arrows between them to directed connection between them.}
		\label{fig-realNetwork}
	\end{subfigure}
	\quad
	\begin{subfigure}[b]{0.48\linewidth}
		\includegraphics[width=0.99\textwidth]{internalNetwork.eps}
        \subcaption{The internal representation of the network. The circles correspond to the neurons. The rounded boxes represent the individual MPI Nodes. For the distribution the default circular distribution was used. The dashed arrows correspond to asynchronous MPI messages. The normal arrows correspond to internal messages.}
		\label{fig-internalNetwork}
	\end{subfigure}
    \caption{The real and internal representation of a small network.}
\end{figure}

\end{document}


\section{Comparison to DNS - Application to Slope Stability Analysis}

To validate the up-scaling methodology used, a simple 2-D slope problem is presented and loaded from the top until failure using both DEM and the up-scaled CDM model. Here, the resultant stress distribution are compared just as failure occurs.

In the DEM model, failure can be assessed based on the unbalanced forces in the model. Since the joints in the model have a stiffness and cohesion, when the slope fails, the explicit quasi-static solution becomes dynamic because of a sudden release of elastic energy and the inability of the applied damping to suppress it all. At this point, the total unbalanced forces in the model increase and the slope can be said to have failed.

For the CDM model, failure can be assessed based on non-convergence of the model when run as an implicit static simulation, which does not converge when the slope fails. The load step in which the CDM model fails to converge because the slope fails dynamically is considered the point of failure.