\section{Introduction}\label{sec:intro}

Fish population (aka "stock")  assessment models determine the impact of past fishing on the historical and current abundance of the population, evaluate sustainable rates of removals (catch), and project future levels of catch reflecting one or more risk-averse catch rules.  These catch rules are codified in regional Fishery Management Plans according to requirements of the Sustainable Fisheries Act.  In the U.S., approximately 500 federally managed fish and shellfish populations are managed under approximately 50 Fishery Management Plans.  About 200 of these populations are assessed each year, based on a prioritized schedule for their current status. Despite this, many minor species have never been quantitatively assessed.  Although the pace is slower than that for weather forecasting, fish stock assessments are operational models for fisheries management.

Assessment models typically assimilate annual catches, data on fish abundance from diverse surveys and fishery sources, and biological information regarding fish body size and proportions at age.  A suite of models is available depending on the degree of data availability and unique characteristics of the fish population or its fishery.  Where feasible, environmental time series are used as indicators of changes in population or observation processes, especially to improve the accuracy of the projections of abundance and sustainable catch into the future.  Such linkages are based principally on correlations given the challenge of conducting field observations on an appropriate scale.  The frontier of model development is in the rapid estimation of parameters to include random temporal effects, in the simultaneous modeling of a suite of interacting species, and in the explicit treatment of the spatial distribution of the population.

Assessment models are loosely coupled to other models. For example, an ocean-temperature or circulation model or benthic-habitat map may be directly included in the pre-processing of the fish abundance survey.  A time series of a derived ocean factor, like the North Atlantic Oscillation, can be included as an indicator of a change in a population process.  Output of a multi-decadal time series of derived fish abundance can be an input to ecosystem and economic models to better understand cumulative impacts and benefits.

Stock Synthesis is an age- and size-structured assessment model in the class of models termed integrated analysis models. Stock Synthesis has evolved since its initial inception in order to model a wide range of fish populations and dynamics.  The most recent major revision to Stock Synthesis occurred in 2016, when version 3.30 was introduced. This new version of Stock Synthesis required major revisions to the input files relative to earlier versions (see the \hypertarget{ConvIssues}{Converting Files} section for more information). The acronym for Stock Synthesis has evolved over time with earlier versions being referred to as SS2 (Stock Synthesis v.2.xx) and older versions as SS3 (Stock Synthesis v.3.xx).

SS3 has a population sub-model that simulates a stock's growth, maturity, fecundity, recruitment, movement, and mortality processes, an observation sub-model estimates expected values for various types of data, a statistical sub-model characterizes the data’s goodness of fit and obtains best-fitting parameters with associated variance, and a forecast sub-model projects needed management quantities.  SS3 outputs the quantities, with confidence intervals, needed to implement risk-averse fishery control rules. The model is coded in C++ with parameter estimation enabled by automatic differentiation (\href{http://www.admb-project.org}{admb}).  Windows, Linux, and iOS versions are available.  Output processing and associated tools are in R, and a graphical interface is in QT.  SS3 is available from NOAA’s VLAB. The rich feature set in SS3 allows it to be configured for a wide range of situations.  SS3 has become the basis for a large fraction of U.S. assessments and many other assessments around the world.

This manual provides a guide for using SS3. The guide contains a description of the input and output files and usage instructions. An overview and technical description of the model itself is in \citet{methotstock2013}. However, SS3 has continued to evolve and grow since the publication in 2013, with this manual reflecting the most up to date information regarding SS3.  The model and a graphical user interface are available from the NOAA \href{https://vlab.noaa.gov/web/stock-synthesis}{VLAB}. The VLAB site also provides a user forum for posting questions and for accessing various additional materials.  An output processor package, r4ss, in R is available for download from CRAN or \href{https://github.com/r4ss/r4ss}{GitHub}.

Additional guidance for new users can be found in the SS3 \href{https://vlab.noaa.gov/web/stock-synthesis/document-library}{document library} within the NOAA VLAB website.  The "Begin Here - Introduction to Stock Synthesis" folder located in the Document Library contains step-by-step guidance for running Stock Synthesis.

  \subsection{How To Cite}

Please cite Stock Synthesis as:

Methot, R.D. and Wetzel, C.R. (2013). Stock Synthesis: A biological and statistical
framework for fish stock assessment and fishery management. Fisheries Research,
142: 86-99. \href{https://doi.org/10.1016/j.fishres.2012.10.012}{https://doi.org/10.1016/j.fishres.2012.10.012}

Please cite the Stock Synthesis User Manual as:

Methot, R. D., Jr., C. R. Wetzel, I. G. Taylor, and K. Doering. (2020). Stock Synthesis User Manual Version 3.30.15. U.S. Department of Commerce, NOAA Processed Report NMFS-NWFSC-PR-2020-05.\href{https://doi.org/10.25923/5wpn-qt71}{https://doi.org/10.25923/5wpn-qt71}

\pagebreak

\section{File Organization}\label{FileOrganization}
	\subsection{Input Files}
	\begin{enumerate}
		\item starter.ss: required file containing filenames of the data file and the control file plus other run controls (required).
		\item datafile: file containing model dimensions and the data (required)
		\item control file: file containing set-up for the parameters (required)
		\item forecast.ss: file containing specifications for reference points and forecasts (required)
		\item ss.par: previously created parameter file that can be read to overwrite the initial parameter values in the control file (optional)
		\item wtatage.ss: file containing empirical input of body weight by fleet and population and empirical fecundity-at-age (optional)
		\item runnumber.ss: file containing a single number used as run number in output to CumReport.sso and in the processing of profilevalues.ss (optional)
		\item profilevalues.ss: file contain special conditions for batch file processing (optional)
	\end{enumerate}

	\subsection{Output Files}
	\begin{enumerate}
		\item data\_echo.ss\_new: Contains the input data as read by the model. In model versions prior to 3.30.19 a single data.ss\_new file was created that included the echoed data, the expected data values (data\_expval.ss), and any bootstap data files seleected (data\_boot\_x.ss).
		\item data\_expval.ss: Contains the expected data values given the model fit. This file is only created if the value for "Number of datafiles to produce" in the starter file is set to 2 or greater.
		\item data\_boot\_x.ss: A new data file filled with bootstrap data based on the original input data and variances. This file is only created if the value in the "Number of datafiles to produce" in the starter file is set to 3 or greater. A separate bootstrap data file will be written for the number of bootstrap data file requests where x in the file name indicates the bootstrap simulation number (e.g., data\_boot\_001.ss, data\_boot\_002.ss,...).
		\item control.ss\textunderscore new: Updated version of the control file with final parameter values replacing the initial parameter values.
		\item starter.ss\textunderscore new: New version of the starter file with annotations.
		\item Forecast.ss\textunderscore new: New version of the forecast file with annotations.
		\item warning.sso: This file contains a list of warnings generated during program execution.
		\item echoinput.sso: This file is produced while reading the input files and includes an annotated echo of the input. The sole purpose of this output file is debugging input errors.
		\item Report.sso: This file is the primary report file.
		\item ss\_summary.sso: Output file that contains all the likelihood components, parameters, derived quantities, total biomass, summary biomass, and catch. This file offers an abridged version of the report file that is useful for quick model evaluation. This file is only available in SS3 v.3.30.08.03 and greater.
		\item CompReport.sso: Observed and expected composition data in a list-based format.
		\item Forecast-report.sso: Output of management quantities and for forecasts.
		\item CumReport.sso: This file contains a brief version of the run output, output is appended to current content of file so results of several runs can be collected together.  This is useful when a batch of runs is being processed.
		\item Covar.sso: This file replaces the standard ADMB ss.cor with an output of the parameter and derived quantity correlations in database format.
		\item ss.par: This file contains all estimated and fixed parameters from the model run.
		\item ss.std, ss.rep, ss.cor etc.:  Standard ADMB output files.
		\item checkup.sso: Contains details of selectivity parameters and resulting vectors.  This is written during the first call of the objective function.
		\item Gradient.dat: New for SS3 v.3.30, this file shows parameter gradients at the end of the run.
		\item rebuild.dat: Output formatted for direct input to Andre Punt's rebuilding analysis package.  Cumulative output is output to REBUILD.SS (useful when doing MCMC or profiles).
		\item SIS\_table.sso: Output formatted for reading into the NMFS Species Information System.
		\item Parmtrace.sso: Parameter values at each iteration.
		\item posteriors.sso, derived\_posteriors.sso, posterior\_vectors.sso: Files associated with MCMC.
	\end{enumerate}

\pagebreak

\section{Starting Stock Synthesis}
SS3 is typically run through the command line interface, although it can also be called from another program, R, the Stock Synthesis Interface, or a script file (such as a DOS batch file). SS3 is compiled for Windows, Mac, and Linux operating systems. The memory requirements depend on the complexity of the model you run, but in general, SS3 will run much slower on computers with inadequate memory. See the Running Stock Synthesis section on page \pageref{sec:RunningSS} for additional notes on methods of running SS3.

Communication with the program is through text files. When the program first starts, it reads the file starter.ss, which typically must be located in the same directory from which SS3 is being run. The file starter.ss contains required input information plus references to other required input files, as described in the File Organization section on page \pageref{FileOrganization}. The names of the control and data files must match the names specified in the starter.ss file. File names, including starter.ss, are case-sensitive on Linux and Mac systems but not on Windows.The echoinput.sso file outputs how the executable read each input file and can be used for troubleshooting when trying to setup a model correctly. Output from SS3 consists of text files containing specific keywords. Output processing programs, such as the SSI, Excel, or R can search for these keywords and parse the specific information located below that keyword in the text file.

\pagebreak


\chapter{Differentiation}
\section{Gradients}
\section{Differentiation from first principles}
\section{Differentiation of standard functions}
\section{Tangents, normals, stationary points, increasing and decreasing functions}
\section{Techniques of differentiation}
\section{Parametric and implicit differentiation}
\section{Constructing differential equations}


\documentclass[a4paper,10pt]{article}
\usepackage[utf8]{inputenc}
\usepackage{graphicx}
%
%opening
\title{}
\author{}

\begin{document}

\maketitle

%\begin{abstract}

%\end{abstract}

\section{Automatic generation of neural mass model kernels}
\subsection{High level description of a simulation of a network of neural masses}
Neural mass models (NMMs) are mathematical tools to describe ensabled behaviour of groups of neurons through time.
These models contain a set of internal states which describe the system and a set of coupled differential equations which define how the states of the system evolve.
An instance of these models is called a node.
The output of the model consists of a set of observables which identify states of interest for other nodes.
Nodes are linked to each other using a coupling function.
This coupling function is the equivalent of a synapse in a point neuron network.
Coupling defines how the input coming from other nodes should be treated.
Usually the coupling involves weighting the incoming signals by a factor and then passing this through a simple function.
The weights used for the coupling are of such level of detail that are matchable to connectivity maps which can be obtained from experimental measurements, e.g. tractography.

Certain observables of a neural mass can be post-processed to produce simulated BOLD, EEG and EMG signals among others.
These signals can afterwards be input to an analysis step where a measure of the ability of the model to reproduce given aspects of the signal is performed.
The amount of open degrees of freedom in the Neural Mass Models generates a vast parameter space.
The nature of the workflow described above allows the iterative modification and exploration of parameters in this admissible space.
Brute force of more intelligent approaches can be used to optimize the behaviour of the models with respect to fitness functions defined by experts and related to the essential characteristics of the higher level signals.

A general description of the simulation can be seen in Figure \ref{fig}
\begin{figure}
  \begin{center}
    \includegraphics[scale=0.4]{DiagramNMM.eps}
  \end{center}
  \caption{Diagram of the interaction between the different computation stages in a neural mass model simulation.}
  \label{fig}
\end{figure}

\subsection{High level description of neural mass models}
In order to ease the high level description of the computation models used in this workflow, we have implemented the BaseModel class.
This class defines a set of attributes:
\begin{enumerate}

\item State: Stores the internal states of the model.
\item Auxex: Auxiliary mathematical expressions which are used for internal calculations in the model.
\item Input: Stores the input comming from other neural masses into this neural mass.
\item Drift: Set of equations which transfer the model from a state at t-1 to another at time t.
\item Diffs: Differentiable variables in the system.
\item Observ: Observers are the data structures which contain inner states which must be available to the outside of the model. They represent the output of the model.
\item Const: Constant values specifically defined for a each model.
\item Param: Parameters provided to an specific model.
\item Limit: Defines the min and max limits within which the state values must be wrapped to ensure mathematical consistency.

\end{enumerate}

\subsection{Loopy as automatic code generation library}
Loopy \cite{loopy2014} is a python library intended to aid the automatic generation of parallel kernels for different target hardware platforms.
It includes targets for CUDA, OpenCL, KNL and CPUs.
Parallel code in Loopy is generated by enclosing a set of functions into an independent execution domains.
The dimensions of these domains are specified using inames.
These inames represent the number of parallel instances that one can process at the same time of the given set of functions.

Other feature of loopy is the integration of multiple kernels into a single one.
The set of independent kernels is combined using a data flow which defines how variables are sent from one kernel into the next as input.
This allows the creation of complex kernels where data dependencies are assured.

Loopy automatically analizes the data structures used within the domain and the access patterns to them.
The user can specify type and ranges for values and access limits to the data structures to control the way the data is handled.
Loopy assambles the computation within a loop-like environment, where each iteration is independent.
The code can then be specifically ported to a target platform using its specific language.

\subsection{Implementation of neural mass models with Loopy}
Our objective is to have a high level description of neural mass models which can then be used to automatically generate high performance parallel code for multiple platforms.
To achieve this goal with Loopy, we need to match the high level description language of the models to the specific API of Loopy.

The BaseModel class has functions which translate the information provided in the attributes of a model instance
Generate the data structures
Define the kernel domain
Define the types for the attributes of the model
Generate expressions for constant deffinition
We unpack the values for the input, state and parameters.
Generate expressions for the auxiliary expressions
Generate expressions to store observables
We use pybolic to build a set of symbolic expressions representing the drift, diffs and observables.
We wrap the output within certain limits to avoid numerical inaccuracies.

\bibliographystyle{abbrvnat}
\bibliography{references}

\end{document}


\chapter{\MSTlong{} Methodology}\label{chap:methodology}
An effective, automated \mstlong{} (\mst{}) methodology is key to \cplop{}'s success as a tool to aid fecal contamination investigators.
Until recently, \cp{} researchers performed most \mst{} by hand \cite{moritz2015application, shapiro2015source}.
The goal for any \mst{} method, when used to source fecal contamination, is to take fecal matter or a substance contaminated with fecal matter and determine, or classify, the \spec{} that provided the fecal matter; \libdep{} \mst{} methods leverage the known-\spec{} information stored in their library, usually digital representations of \fiblong{} (\fib{}) stored in a database.
\cplop{} is such a \libdep{} technique that aims to support \mst{} using \pyros{} of both \itsshort{} regions of the \fib{} \ecoli{}.
Given a \fib{} \isol{} from an unknown-\spec{}\footnote{often referred to as the ``the unknown'' or ``the unknown \isol{}''}, a \libdep{} \mst{} technique determines the \spec{} of the unknown using the information in the library.
Towards this end, we built and investigated two \mst{} techniques, one that approaches \spec{} classification from the perspective of strains in the database and another that directly uses \isols{} present in the database.
This Chapter outlines the abstract approaches we chose to take, while \autoref{chap:clustering} and \autoref{chap:krap} detail the specific techniques we used for the strain-based and \isol{}-based approaches respectively.

\section{Strain-Based}
Strain typing is central to \libdep{} \mst{} methods and building a \spec{} classification technique that uses strains directly is an intuitive approach to take.
If an unknown-\spec{} matches a strain in the library, then we can make a reasonable assertion as to its \spec{} if the strain has a dominant \spec{}.
Thus, given an unknown-\spec{} \isol{}:
\begin{enumerate}
    \item Incorporate the unknown-\spec{} \fib{} \isol{} into \cplop{}
    \item Build strains of \fib{} using the \isols{} in \cplop{}
    \item Classify the source \spec{} of the \isol{} as the dominant \spec{} of the strain it ended up in
\end{enumerate}
Strain construction can happen in many ways, but from a computational perspective, it is very amenable to clustering.
If one can imagine a coordinate space that encapsulates mathematical representations (vectors) of \fib{} \isols{}, then strains are the close groupings (clusters) of these \isol{} representations.
For \cplop{}, this means constructing strains from the \pyros{} of collected \ecoli{} \isols{}.

As explained in \autoref{chap:clustering}, we use \dbscan{}, a \dbased{} clustering algorithm that we introduce in \autoref{sec:background:dbscan}, which builds clusters using a similarity metric, allowing for the concept of noise --- datapoints that remain unclustered.
Dense groupings of similar \isols{} fits our intuition of bacterial \isol{} strains because closely related ``families'' of \isols{} will appear in the same cluster.
A primary limitation of \dbscan{} is that it may not cluster some \isols{}, which still aligns with our notion of strains; sometimes, \isols{} are not part of any strain present in \cplop{}.
As a result, we need a fallback \mst{} method that can work for every \isol{}.

\section{\Isol{}-Based}
Using \isols{} directly can give us a level of flexibility that strict strain typing may not allow.
An unknown-\spec{} may not fit within the sometimes strict definition of a strain that a particular library may have, but we may still be able to make a reasonable assertion as to its \spec{} based on the \isols{} that are most similar to the unknown.
Thus, given an unknown-\spec{} \isol{}:
\begin{enumerate}
    \item Find $k$ known-\spec{} \isols{} from \cplop{} most similar to it, called the \knnlong{}
    \item Classify the source \spec{} of the \isol{} as the dominant \spec{} of the \knnlong{}
\end{enumerate}

The described approach is the \kNNlong{} classification algorithm (\kNN{}) introduced in \autoref{sec:background:knn}, however, due to the multiple \compfuncs{} needed between \isols{}, \kNN{}, in its natural form, will not work with \cplop{}.
\autoref{sec:background:isolates} details why the comparison of two \isols{} to each other requires multiple \compfuncs{}.
\autoref{chap:krap} explains the \compfunc{} resolution strategy employed by the \kraplong{}, the \isol{}-based \mst{} method we built and investigated.


\documentclass[12pt]{article}
\setlength{\evensidemargin}{5pt}
\setlength{\marginparwidth}{0pt}
\setlength{\marginparsep}{0pt}
\setlength{\textheight}{9in}
\setlength{\tabbingsep}{0pt}
\setlength{\headsep}{30pt}
\setlength{\fboxsep}{40pt}
\setlength{\fboxrule}{4pt}
\setlength{\footnotesep}{20pt}
\setlength{\parindent}{0pt}
\setlength{\parskip}{8pt plus 5pt}
\setlength{\topmargin}{-0.5in}
\setlength{\oddsidemargin}{0in}
\setlength{\textwidth}{6.5in}

\usepackage{times}
\usepackage{url}
\usepackage{graphics}
\usepackage{graphicx}      % extended graphics package
\usepackage{epsfig}        % wrapper for graphicx package
\usepackage[noend]{algpseudocode}
\algrenewcommand\algorithmicprocedure{\textbf{ALGORITHM}}

\begin{document}

\title{An Empirical Study of the Brute-Force Closest Pairs Algorithm}

\author{Jim Teresco\\
jteresco@siena.edu\\
\and
My Partner\\
partner@fake.email}

\maketitle

\begin{abstract}
  In this study, which is intended as an example of how to put
  together the writeup for an empirical study of algorithm
  performance, we present and analyze timings and operation counts for
  a straightforward brute-force closest pairs implementation.  Points
  are generated randomly in a specified range, and the closest pair is
  computed and reported by a Java program.
\end{abstract}

\section{Introduction}

This empirical analysis study focuses on the brute-force closest pairs
algorithm in two dimensions.  We will consider a straightforward
implementation of the algorithm in Java, instrumented to count basic
operations and gather timings.

The remainder of this report is organized as follows.
Section~\ref{sec:environment} describes the computing environment
used.  Section~\ref{sec:bfcp} describes the algorithm, our
expectations based on the theoretical analysis, presents our results,
and relates them back to the theory.  We conclude with some additional
discussion in Section~\ref{sec:conclusions}.

\section{Computing Environment}
\label{sec:environment}

For this study, a Java program was developed that implements the
algorithm of interest.  Random inputs of varying sizes and other
characteristics are generated, and the solution is computed.

All runs are using the following Java version:

\begin{verbatim}
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
\end{verbatim}

The computing environment is a MacBook Pro (15-inch, 2016) with a 2.6
GHz Intel Core i7, 16 GB of 2133 MHz LPDDR3 memory, and running macOS
Sierra Version 10.12.6.  There are 4 cores, each with 256 KB L2 cache,
and these cores share a 6 MB L3 cache.

\section{Brute-Force Closest Pairs}
\label{sec:bfcp}

The brute-force closest pairs algorithm as shown in
Figure~\ref{fig:alg} was implemented in Java, and instrumented to
count the number of times the distance between pairs of points is
computed and to report the elapsed time for the main loops to execute.

\begin{figure}[htb]
  \centering
\begin{algorithmic}
  \Procedure{BruteForceClosestPoints}{$P$}
  \State //Input: a set of points $P[0..n-1]$
  \State $d_{min} \gets \infty$
  \For{$i \gets 0 .. n-2$}
  \For{$j \gets i+1 .. n-1$}
  %\State $d \gets distance(P[i], P[j])$
  \State $d \gets \sqrt{(P[i].x - P[j].x)^2 +(P[i].y - P[j].y)^2}$
  \If {$d < d_{min}$}
  \State $d_{min} \gets d$
  \State $index_1 \gets i$
  \State $index_2 \gets j$
  \EndIf
  \EndFor
  \EndFor
  \State \textbf{return} $(index_1, index_2)$
\EndProcedure
\end{algorithmic}
\caption{The brute-force closest pairs algorithm, as implemented for
  this study.}
\label{fig:alg}
\end{figure}

The number of points for the runs is all powers of 2 from
$2^{10}=1024$ to $2^{19}=524,288$.  Points for each run are generated
randomly, within a range specified as a parameter to the program.
Points are generated within the square with x- and y-coordinates no
further than that range from the origin.

Each combination of the number of points and range of point positions
is run a total of 5 times.  The number of distance calculations is
expected to be identical for each of the runs for a given number of
points.  Run times will vary, however, and the minimum time for a
given combination of the number of points and range of point positions
is chosen to minimize the effects from other computations that could
be in process on the computer during the study.

\subsection{Expectations}

The theoretical expectation is for both the number of distance
calculations and elapsed time to be $\Theta(n^2)$ in the best,
average, and worst cases.  There are no shortcuts out of the loops,
which execute about $\frac{n^2}{2}$ times.

\subsection{Results}

The raw results of the study are available in the file
\texttt{timings.dat} in the GitHub repository.  Each line, such as

{\small
\begin{verbatim}
BFCP 262144 64.0 92375.0 34359607296 (99936,203911) 4.069483000753879E-4
\end{verbatim}
}

has 7 space-separated fields.  The first field indicates the algorithm name,
and is followed by the number of points, the range of coordinate
values (each of x and y are within this distance of the origin), the
elapsed time in milliseconds, the number of distance calculations, the
indices of the two points that were found to be the closest pair, and
the distance between those points.  Of these, we are interested in the
number of points, the elapsed time, and distance calculation count.

\begin{figure}[htb]
  \centering
  \includegraphics{bfcp-times.pdf}
  \caption{Times in milliseconds for the brute-force closest pairs to
    complete its computation for problem sizes from $2^{10}$ to
    $2^{19}$ points.}
  \label{fig:timings}
\end{figure}

\begin{figure}[htb]
  \centering
  \includegraphics{bfcp-opcounts.pdf}
  \caption{Number of distances between pairs of points computed the
    brute-force closest pairs to complete its computation for problem
    sizes from $2^{10}$ to $2^{19}$ points.}
  \label{fig:opcounts}
\end{figure}

\begin{table}[htb]
  \centering
  \begin{tabular}{|c|c|c|}
    \hline
    Problem Size & Time (ms) & Distance Calculations \\ \hline \hline
1024 & 11.0 & 523776 \\ \hline
2048 & 15.0 & 2096128 \\ \hline
4096 & 35.0 & 8386560 \\ \hline
8192 & 81.0 & 33550336 \\ \hline
16384 & 298.0 & 134209536 \\ \hline
32768 & 1171.0 & 536854528 \\ \hline
65536 & 4607.0 & 2147450880 \\ \hline
131072 & 18496.0 & 8589869056 \\ \hline
262144 & 84123.0 & 34359607296 \\ \hline
524288 & 368682.0 & 137438691328 \\ \hline
  \end{tabular}
  \caption{Actual times and operation counts used to create the plots
    in Figure~\ref{fig:timings} and Figure~\ref{fig:opcounts}.}
  \label{tab:rawdata}
\end{table}

Figure~\ref{fig:timings} shows the times in milliseconds (minimum
across 5 runs of each) and Figure~\ref{fig:opcounts} shows the number
of computations of the distance between two points for this study.
Unsurprisingly, the timings are very similar and the operation counts
identical for the different ranges of point placements, so
Table~\ref{tab:rawdata} shows the data as plotted in the figures but
only for the range value of 1.0.

\subsection{Discussion}

Both the timings and distance calulation counts match well with the
expected $\Theta(n^2)$ behavior of this algorithm.  The graphs in
Figures~\ref{fig:timings} and~\ref{fig:opcounts} show the expected
parabolic shape.  The actual numbers in Table~\ref{tab:rawdata} align
well.  The number of distance calculations is exactly as predicted.
Any variation there would have indicated an error in the algorithm or
in the instrumentation that counted the operations.  For the timings,
small problem sizes are subject to errors due to the accuracy of the
millsecond timer over short timespans.  However, when we look at the
larger problem sizes, we see exactly what we would expect: when we
double the problem size, the time taken to compute the solution
increased by a factor of four.

\section{Conclusions}
\label{sec:conclusions}

In this simple study, we found that all of our results matched the
expected behavior from the theory.  In this particular algorithm,
since there are no asymptotic differences among the best-, average-,
and worst-case behavior, this should be unsurprising.  The small
variations in run time can perhaps be explained by how soon the
algorithm finds the closest pair (or at least a very close pair).  The
more times that a new closest pair is found, the more often the body
of the \texttt{if} statement following the distance calculation will
need to execute.  There is also the issue that the computer used for
the study is a laptop system that had many processes such as web
browsers with many tabs and terminal windows in regular interactive
use during the timing studies.  This is partially accounted for by
taking the minimum time across 5 runs, but as each run has its own set
of random points generated, the previously-mentioned effect from the
different numbers of times the body of the \texttt{if} statement is
executed will potentially skew the results more toward a best case
behavior.  Any such variations are likely minor, and do not decrease
the confidence in our conclusions that the timing results match
closely with theoretical expectations.

\end{document}