%% ------------------------------------------------------------------------- %%
\chapter{Introduction and Motivation} \label{Chap:Intro}
\lettrine[findent=2pt]{\textbf{T}}{}he parallel and distributed platforms available today become more and more \textit{heterogeneous}.
Such heterogeneous architectures, composed of several kinds of computing units, have had a growing impact on performance in high-performance computing.
Hardware accelerators, such as General Purpose Graphical Processing Units (in short GPUs), are often used in conjunction with multiple computing units (CPUs) on the same chip sharing the same common memory.
As an instance of this, the number of platforms of the ~\cite{TOP500} equipped with accelerators has significantly increased during the last years. In the future it is expected that the nodes of such platforms will be even more diverse than today: they will be composed of fast computing nodes, hybrid computing nodes mixing general purpose units with accelerators, I/O nodes, nodes specialized in data analytics, etc. In order to use all the computational power available, applications should be composed of multiple tasks that allow the usage of the available resources as efficiently as possible.

The Job Management System (JMS) is the middleware responsible for distributing computing power to applications. A JMS must allocate resources to tasks in order to optimize the use of the available resources while guaranteeing good performance for all applications running in parallel. A promising way to achieve this is using performance prediction. However, in a scenario with millions of processors and a large number of tasks it is very difficult to predict the performance of applications.

Job management systems require that users provide an upper bound of the execution times of their jobs (wall time). Usually, if the execution goes beyond this upper bound, the job is killed. According to~\cite{Gaj:Trystram:2002}, this leads to very bad estimations, with an obvious bias that tends to overestimate this duration. Moreover, those jobs are subject to uncertainties that depend on the context and other running jobs. The uncertain information is generated by communications and executions that depends on the specific context.

The GPUs were initially conceived with the purpose to accelerate the 2D and 3D graphics tasks. However, the evolution of the GPU architectures has gone from a single core to a set of dense arrays of highly parallel cores for more general purpose computation. Not surprisingly, the use of this hardware for parallel computing is a very attractive and it is used in several scientific areas. Moreover, nowadays GPUs can be found in a wide number of electronic devices, just to mention a few: desktops, laptops, smartphones, cars and supercomputers~\citep[chap. 10]{verber2014future}.

GPUs are integrated with multi-core CPUs with the goal to boost more computational processing power. In this way, the regular users have access to a massively parallel environment when acquiring any kind of these devices. Figure \ref{fig:gflops} shows the theoretical performance and bandwidth reached for those devices, this figure shows a comparison between NVIDIA GPUs and current x86 CPUs. The predominant manufacturer of CPUs are Intel and AMD, and of GPUs are NVIDIA and AMD.

\begin{figure}[htpb]
\centering
\includegraphics[width=\linewidth]{./images/Volta-GPU.png}
\caption{Theoretical performance and bandwidth reached for recent GPUs. Copied from~\cite{CUDAGuide}}
\label{fig:gflops}
\end{figure}

In GPU applications the computation is done with a large amount of data, which expose restrictions on the storage space available in the devices and the latency of data transfers~\citep{Gregg:2011}. GPUs are composed of hundreds and even thousands of simple cores. Figure~\ref{fig:gpuComp} shows a generic scheme of a heterogeneous system composed of a CPU and a GPU. In this figure we can see that CPUs have much more transistors for control and cache, on other hand GPUs have much more ALUs and few transistors for cache and control.

\begin{figure}[htpb]
\centering
\includegraphics[scale=.85]{./images/gpu-computing.png}
\caption{Multiples Cores CPU + Hundreds or Thousands Cores GPU. Copied from~\cite{CUDAGuide}}
\label{fig:gpuComp}
\end{figure}

Last decade with the development of application programming interfaces for GPUs emerged the concept of General Purpose Computing on GPU or GPGPU. GPGPU has evidenced outstanding results in many scientific areas, however, tools to further capitalize these parallel devices and more refined implementations are still emerging~\citep{CUDAGuide,bsgp}. Researchers in this area can create applications with a concept of parallelism that are able to run on both CPU and GPU architectures. Nevertheless, the implementation of an algorithm, for parallel execution on CPU or GPU, does not guarantee that will execute with the same efficiency in the two architectures. In particular, algorithms with parallelism in large blocks of data, i.e. applications with intensity in array arithmetic and linear data structures, can obtain great benefits when they are implemented for execution exclusively on a GPU~\citep{zhong2012data, Benedict:12}.

Nowadays, GPUs are general purpose parallel processing units with accessible programming interfaces, including standard languages such as C, Java, and Python. In particular, the Compute Unified Device Architecture (CUDA) is a parallel computing platform that facilitates the development of applications which execute in GPU~\citep{CUDAGuide}. CUDA was introduced by NVIDIA in 2006 for their GPU hardware line. CUDA is a SIMT programming model, it is an extension of SIMD according to Flynn architecture. MIMD is the model used to characterize a machine in an exchange of messages. CUDA is an extension that developers of different languages can use, there is a phase to preprocess the code and extract the CUDA functions. After, CUDA functions are translated into a pseudo-assembly language named PTX  (Parallel Threads eXecution), and the graphics driver contains a compiler which translates the PTX into a binary code which can be run on the processing cores.

CUDA applications are organized in kernels, which are functions executed in GPUs. Kernels are compounded of thread blocks of execution and each one of these blocks can have hundreds or even thousands of threads. Each kernel is executed asynchronously, and various kernels can execute concurrently. The programming model allows directive of synchronization in different levels of the memory hierarchy.

The software development process and performance modeling of GPU applications require a high degree of understanding and knowledge~\citep{Baghsorkhi:2010:APM}. Performance prediction is often just ignored in a development process, this is due to the in-depth analysis that must be done. Models with specific properties to these problems have been created ~\citep{Skillicorn:1998:MLP} to facilitate programming on parallel machines.  Properties such as synchronization and division of work between computing and communication improve the way developers can solve a problem and how the performance of applications running on parallel machines can be modeled.

The accuracy of a GPU performance model is subject to low-level elements such as instruction pipeline usage and cache hierarchy. GPU performance approaches its peak when the instruction pipeline is saturated but becomes unpredictable when the pipeline is under-utilized~\citep{Baghsorkhi:2010:APM}. Effects of cache hierarchy and memory-access divergence are also critical to GPU performance models. Some parallel programs can be efficiently executed on some architectures, but not on others.

Performance prediction over these devices is a great challenge because hardware characteristics can impact their performance in different ways. There are different approaches to do this, such as analytical modeling and machine learning techniques. Analytic predictive models are useful, but require the manual inclusion of interactions between architecture and software, and may not capture the complex interactions in GPU architectures. Machine learning techniques can learn to capture these interactions without manual intervention but can require large training sets.

There are different models of parallel machines that help to deal with these problems. Among these models, we have the Parallel Random Access Memory (PRAM), Bulk Synchronous Parallel (BSP) and Coarse Grained Multicomputer (CGM). The main objective of a parallel computing model is to provide a set of parameters that have an impact on the performance of applications in parallel architectures, and thus to be able to simulate the behavior of these applications in their respective architectures.

The Bulk Synchronous Parallel is a computing model for parallel computing introduced for~\cite{Valiant:1990}, BSP model provides a simple but accurate characterization of all parallel machines-past, present, and future. There are other parallel models parameterized, almost all of them using or extending the focus of the BSP model. \cite{Dehne:2002} studied the problem to design scalable parallel geometric algorithms for the Coarse Grained Multicomputer model which are optimal or at least efficient for a wide range of the ratio $\frac{n}{p}$, with $n$ the size of the problem and $p$ the number of processors.

The BSP model allows developers to design algorithms and software that can run on any standard parallel architecture with very high performance~\citep{goldchleger2005implementation}. This model may potentially serve as a unified programming model for both coarse-grained and fine-grained. The model BSP has been implemented as API libraries and programming languages \citep{BSPLib} and API libraries have been developed to use BSP on GPU architectures~\citep{bsgp}, these developments are to create scientific applications in massively parallel environments computing in an easy and better way. Recently, ~\cite{Valiant:2011} has implemented the BSP model over multicore architectures. The implementation of BSP model for multicore architecture incorporates the memory size shared as a further parameter in the computer. The BSP model is a good alternative to tackle parallel problems in massively parallel architectures.

%Data are produced either by the Job management systems, to effectively leverage these platforms, and by the user's applications that are executed. These applications could be in execution, awaiting execution, monitoring the machine, so forth. While this happens, a lot of information is registered in traces. There are several recent works on using this information~\citep{Tsafrir:Backfilling:2007,Lublin01theworkload,Ernemann:2003}, in other cases these data are ignored by the job management systems~\citep{andreetto2008glite}.

The developed methods in the area of \textbf{big data} (especially machine learning) can be used for the scheduling of jobs in new computing platforms to large scale. Information about profiling and traces of heterogeneous applications can be used to improve current JMS, which require a better knowledge of the applications~\citep{JMR:Ruiz:2014}. To predict execution or communication times in heterogeneous applications is a great challenge, for this reason, it always is meritorious to research more effective mechanisms to obtain a better performance prediction.

This thesis is related to the modeling and performance prediction of applications executed in GPUs using approaches of analytical model and machine learning techniques. Considering that the market has been taken by NVIDIA graphics cards, our models and experimentation are done on GPUs manufactured by NVIDIA. We evaluated our model using 9 developed kernels and other kernels from Rodinia Benchmark suite~\citep{Rodinia:Che:2009}, Rodinia has been used and accepted by the GPGPU community~\citep{che:2010:Rodinia, nagasaka:2010:Rodinia}, and is used in various GPU simulators~\citep{power:2014:gem5-gpu}.

These CUDA kernels were executed over GPUs with different architectures among these architectures are Kepler, Maxwell, and Pascal. We showed by using profile information for a single board, that the models are general enough to predict the execution time of an application with different input sizes and on different boards.

This thesis also presents the creation of a benchmark, which is freely available in Standard Workload Format (SWF), consisting of five applications of dense linear algebra from the Chameleon software~\citep{agullo2012morse} and a \emph{fork-join} application generated using GGen~\citep{GGen:simutools10}, this benchmark was used for ~\cite{mommessin:2017, mommesin:2017:CCPE} in the experimental evaluation of on-line and off-line scheduling algorithms in machines with heterogeneous resources.

This benchmark is available online in a web-based hosting service for version control using git, inside the repository, there are a set of application workloads with their respective execution times and dependencies, researchers can use it to test scheduler algorithms of task-based applications over CPU and GPUs.

%Our model predictions were within $XXXX$ to $XXXX$ times the measured execution times, which are reasonable for such a simple model. These results indicate that the model is good enough to generalize the predictions for different problem sizes and GPU configurations.

\section{Objectives and Thesis Contributions}
The main propose of this work is to present an analysis and design of techniques to predict the performance of applications executed on General-purpose Graphic Processing Units.

In this thesis, we first present an analysis and design of an easy and simple BSP-based model to predict the performance of GPU applications~\citep{amaris:2015:HiPC}. The model is based on the number of computations and memory accesses of the GPU, with additional information on cache usage obtained from profiling. Scalability, divergence, the effect of optimizations and differences of architectures are adjusted by a single parameter.

We also evaluate the use of different machine learning techniques to reach the same objective. We compare three different machine learning approaches: linear regression, support vector machines and random forests with our BSP-based analytical model~\citep{amaris:2016:NCA}. Our main contribution was showing that machine learning techniques provided acceptable predictions for all the applications over all the GPUs. Although the analytical model provided better predictions for some applications, it requires a deep analysis and knowledge of the applications and hardware structure. Consequently, machine learning techniques can be useful for deploying automated on-line performance prediction for scheduling applications on heterogeneous architectures with GPUs, whenever a large data set with information about similar applications is available.

Finally, we developed a benchmark of task-based applications executed on heterogeneous machines. These files are saved in format SWF (Standard Workload Format). This format was defined in order to use a pattern accepted and well used for the community in scheduling research. The benchmark is collected of six parallel applications. Five of them have been created from Chameleon, a dense linear algebra software which is part of the MORSE project~\citep{agullo2012morse}, while the sixth is a fork-join application generated automatically with GGen, a library for generating directed acyclic graphs~\citep{GGen:simutools10}. This benchmark has been tested for \citep{mommessin:2017, mommesin:2017:CCPE}.

This work was presented in the Doctoral Showcase of the International Conference for High Performance Computing, Networking, Storage and Analysis of 2017. In this conference, a summary of the thesis was presented in a short talk and poster.

We list the publications directly related to this thesis. %Each one of these papers can be found in the appendix~\ref{app:papers} in the same order as they are presented below.

\begin{itemize}

\bibitem[Amaris {\rm{\em et~al.}} (2015)Amaris, Cordeiro, Goldman, e
  Camargo]{amaris:2015:HiPC}
{\bf Marcos Amaris}, Daniel Cordeiro, Alfredo Goldman e Raphael~Y.de Camargo.
\newblock A simple bsp-based model to predict execution time in gpu
  applications.
\newblock In \emph{High Performance Computing (HiPC), 2015 IEEE 22nd
  International Conference on}, p\'{a}ginas 285--294.

\bibitem[Amaris {\rm{\em et~al.}} (2016)Amaris, de~Camargo, Dyab, Goldman, e
  Trystram]{Amaris:2016:NCA}
{\bf Marcos~Amaris}, Raphael~Y.de Camargo, Mohammed~Dyab, Alfredo~Goldman e Denis~Trystram.
\newblock A comparison of GPU execution time prediction using machine learning
  and analytical modeling.
\newblock In \emph{2016 IEEE 15th International Symposium on Network Computing
  and Applications (NCA)}, pages 326--333.

 \bibitem[Mommessin {\rm{\em et~al.}} (2017)Mommessin, Amaris, Lucarelli, e
  Trystram]{Mommessin:2017}
Clement~Mommessin, {\bf Marcos~Amaris}, Giorgio~Lucarelli and Denis~Trystram.
\newblock Generic algorithms for scheduling applications on hybrid multi-core
  machines.
\newblock In \emph{2017 Euro-Par: International Conference on Parallel and
  Distributed Computing}, pages 220--231.

  \bibitem[Mommessin {\rm{\em et~al.}} (2018)Mommessin, Amaris, Lucarelli, e
  Trystram]{mommesin:2017:CCPE}
Clement~Mommessin, {\bf Marcos~Amaris}, Giorgio~Lucarelli and Denis~Trystram.
\newblock Generic algorithms for scheduling applications on heterogeneous multi-core platforms
\newblock \emph{arXiv preprint arXiv:1711.06433} in revision. Submmited in
\emph{Concurrency and Computation: Practice and Experience}
\end{itemize}

There are other works which were published during the Doctorate of the candidate and are indirectly related to this thesis:
%\cite{bruel:2015:WSCAD}\cite{gonccalves:2016:HICSS}\cite{okada:2015:WSCAD}\cite{bruel:2017:CCPE}

\begin{itemize}
  \bibitem[Okada {\rm{\em et~al.}} (2015)Okada, Amar{\'\i}s, e vel
  Lejbman]{okada:2015:WSCAD}
Thiago~Kenji Okada, {\bf Marcos Amar{\'\i}s} e Alfredo~Goldman vel Lejbman.
\newblock Scheduling moldable bsp tasks on clouds.
\newblock In \emph{WSCAD'15 - XVI Simp{\'o}sio em Sistemas Computacionais de
  Alto Desempenho},

    \bibitem[Bruel {\rm{\em et~al.}} (2015)Bruel, Amar{\'\i}s, e
  Goldman]{bruel:2015:WSCAD}
Pedro Bruel, {\bf Marcos Amar{\'\i}s} and Alfredo Goldman.
\newblock Autotuning gpu compiler parameters using opentuner.
\newblock In \emph{WSCAD'15 - XVI Simp{\'o}sio em Sistemas Computacionais de
  Alto Desempenho}, pages\'{a}ginas 13--23.

  \bibitem[Gon{\c{c}}alves {\rm{\em et~al.}} (2016)Gon{\c{c}}alves, Amar{\'\i}s,
  Okada, Bruel, e Goldman]{gonccalves:2016:HICSS}
Rog{\'e}rio Gon{\c{c}}alves, {\bf Marcos Amar{\'\i}s}, Thiago Okada, Pedro Bruel and
  Alfredo Goldman.
\newblock Openmp is not as easy as it appears.
\newblock In \emph{49th Hawaii International Conference on System Sciences}, January-2016,
  pages 5742--5751.

\bibitem[Bruel {\rm{\em et~al.}} (2017)Bruel, Amar{\'\i}s, e
  Goldman]{bruel:2017:CCPE}
Pedro Bruel, {\bf Marcos Amar{\'\i}s} and Alfredo Goldman.
\newblock Autotuning cuda compiler parameters for heterogeneous applications
  using the opentuner framework.
\newblock \emph{Concurrency and Computation: Practice and Experience}
    29\penalty0 (22). 2017
\end{itemize}

\section{Thesis Outline}
This document is organized into 6 chapters. In Chapter~\ref{chap:characterization}, we show a detailed analysis and characterizations of GPU applications, Chapter~\ref{chap:BSPmodel} shows our proposed BSP-based model used to predict execution times of GPU applications. Chapter~\ref{Chap:ML} presents a machine learning approach for predicting execution times and compare the predictions with those of the analytical model. Chapter~\ref{Chap:Bench} describes the creation of a benchmark of task-based applications executed over heterogeneous resources, CPU and GPUs. Finally we present the conclusions and future works in the Chapter~\ref{chap:conclusion}


% !TeX spellcheck = en_GB
\section{Optimal Estimation Retrieval Algorithm} %Section - 1.2
\label{sec:retrieval}
The purpose of this study is to apply an optimal-estimation snowfall retrieval on ground based measurements to estimate the surface accumulation and vertical snow water content for an extreme event during Christmas 2016. These will later be used to compare to \SI{48}{\hour} MEPS model forecasts to see if the model was able to predict synoptical features and precipitation related to the extreme event 'Urd' in 2016.
% maybe start out by saying that you are trying to estimate atmospheric parameters  (SWC in our case) from a set of measurements.   The optimal-estimation approach provides a flexible retrieval framework that can incorporate multiple measurements/ pieces of information into a common retrieval framework.  In this way, we can use measurements from MRR, PIP, and MASC to estimate vertical profiles of snowfall.  You could then discuss the reindeer -> tracks thing to illustrate the general principle.
% literally taken from paper until Skofronick, Noh
The quantitative estimation of snowfall at the global scale from spaceborne measurements has been available only recently. Initial retrieval approaches were based on passive microwave measurements \citep{skofronick-jackson_physical_2004,noh_development_2006}. But since these passive measurements can only assess total integrated snow water path for a given column, such efforts were unable to provide much information on the profiles of snow water content.
The launch of the CloudSat \SI{94}{\giga\hertz} Cloud Profiling Radar (CPR) in 2006, however, provided the first opportunity to examine such vertical structure at a global scale.
%Several studies, such as \citet{matrosov_modeling_2007} and \citet{kulie_utilizing_2009}, have shown that the CPR can be used to estimate snofall rate but that estimated snowfall values depend heavily upon assumed snowflake microphysical properties.
Several studies have shown, that estimated snow rate depends upon retrieval assumptions such as snowflake habit and particle size distribution (PSD) and can give large differences for a given radar reflectivity.
\\
For the operational CloudSat snowfall retrieval scheme (2C-SNOW-PROFILE), \citet{wood_microphysical_2015} developed snowflake particle models based upon video snow disdrometer observations from the Canadian CloudSat-CALIPSO Validation Project \citep[C3VP,][]{hudak_canadian_2006}. Scattering properties for these snow particle models were based upon the Discrete Dipole Approximation (DDA) method.
%It was hoped that the use of realistic snow properties in the retrievals would lead to reasonable estimates of snowfall in the retrieval.
In addition, they derived an a priori relationship between particle size distribution and temperature that they could use as an additional constraint for the snowfall scheme. Use of the flexible optimal-estimation retrieval framework allowed to develop a best estimate of snow properties that are consistent with both the CPR reflectivities and the a priori constraint.
\\
They have also been used to estimate snowfall in remote locations such as the Antarctic and Arctic \citep{palerme_how_2014,kulie_shallow_2016} that in turn have been used to evaluate the representation of snowfall in climate models \citep{palerme_evaluation_2017,christensen_arctic_2016}.
These estimates have been used to assess the performance of ground-based radar schemes such as those based upon the operational weather radar system in Sweden \citep{norin_intercomparison_2015}. Despite such progress, however, the CloudSat scheme can still lead to uncertainties in the retrievals of up to \SIrange{140}{200}{\percent} \citep{wood_estimation_2011} for individual storms.
\\
Again, these uncertainties arise from the large variance in snowflake microphysical  properties as observed in nature. In response, \citet{cooper_variational_2017} explored the use of in-situ, event specific  observations of snowflake microphysical properties to improve radar-based retrievals of snowfall. This work was based upon observations from the Ka-band ARM (The Atmospheric Radiation Measurement) Zenith Radar (KAZR) and Multi-Angle Snow Camera (MASC) deployed at the ARM Climate Facility Site at Barrow, Alaska in Spring 2014. This ground-based \SI{35}{\giga\hertz} retrieval scheme was modified from the space-borne \SI{94}{\giga\hertz} CloudSat retrieval scheme developed by \citet{wood_estimation_2011}. But instead of using a temperature dependent a priori characterisation of PSD, \citeauthor{cooper_variational_2017} introduced the in-situ observations of particle size distribution through the a priori terms of the optimal-estimation framework.
\\
%
Preliminary analyses suggested good performance for this retrieval scheme at Barrow, Alaska. Estimates of snowfall from the \citet{cooper_variational_2017} approach differed by \SI{18}{\percent} relative to nearby National Weather Service snow gauge measurements for total accumulation over multiple snow events. However, given limited snowfall observed at Barrow during the deployment of the MASC, it was difficult to come to any definitive conclusions about retrieval performance. The NSF (National Science Foundation) funded field campaign with MRR, MASC, and PIP (Precipitation Imaging Package) deployment at Haukeliseter provides an ideal opportunity to further explore the \citet{cooper_variational_2017} retrieval approach. This thesis will continue to examine the sensitivity of retrieval surface snowfall rate to assumptions of habit, fall speed, and particle size distribution as in \citet{cooper_variational_2017}. In addition, this study here will will examine the vertical profiles of snowfall profiles in the atmospheric column.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%\newpage
%%%% OE scheme %%%%%%%
\subsection{Snowfall Retrieval Scheme}\label{sec:ret_scheme}
%%% image forward problem %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\input{./images_tex/retrieval_dragon_track}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The optimal-estimation snowfall retrieval scheme was modified for the Barrow instrumentation described in \citet{cooper_variational_2017} to the \SI{24}{\giga\hertz} MRR, MASC, and PIP located at Haukeliseter. This scheme was then used to derive surface snowfall rates and vertical profiles of snow water content using different combinations of retrieval assumptions based upon in-situ observations.
Here a discussion of the optimal-estimation framework is presented so that the reader can understand exactly how the different measurements in the retrieval scheme were incorporated. The differences in the forward and inverse problem are reviewed in the beginning. Both, forward and inverse problem underlie the basic methodologies of remote sensing.
\par\medskip\noindent
The general concepts of forward and inverse problems are illustrated in \Cref{fig:dragon}.  The forward problem describes the relationship between the physical parameters of interest and the measurements (\Cref{fig:dragon:forward}). In \Cref{fig:dragon:inverse}, the physical parameter is the reindeer and the measurements are its tracks. In the here presented thesis snowfall scheme, the physical parameters are the vertical profiles of snow water content (SWC) and the measurements are the MRR reflectivities.  The inverse problem represents the opposite goal.  The physical entity (reindeer and SWC) must be inferred from the measurements (reindeer tracks or MRR reflectivities).
\\
The optimal-estimation framework is used to solve the inverse problem for the thesis work. But it is more complex than the direct inversion as represented in \Cref{fig:dragon:inverse}. Instead of inverting directly for the parameter of interest (SWC) from the measurements [\SI{}{\decibel ze}], the scheme includes additional information based upon an 'a priori' understanding of the physics of the retrieval problem. Thus, it represents a weighted balance between what the data can tell about the state and what is already known about it. For the CloudSat retrieval scheme, this a priori information came from a parametrisation relating particle size distribution (PSD) to temperature. For the Barrow and Haukeliseter schemes, the a priori information could also come from the in-situ observations of snowflake microphysics. These constraints also add numerical stability to the inversion process when there are either calibration errors in the measurements or uncertainties in the forward model that relates the physical parameter to measurement space. Details of the radar forward model are discussed at the end of this section.
\\
The optimal estimation method is based on Gaussian statistics.  It solves for snowfall properties of interest or retrieval vector, $\mathbf{x}$, by minimizing the scalar cost function, $\Phi$ , as in \Cref{eq:scalar_cost_fct}.
\begin{equation}
\begin{split}
\Phi(\mathbf{x},y,a) = & (y- F(\mathbf{x}))^T \mathbf{S}_y^{-1} 			(y-F(\mathbf{x})) \\
&+(\mathbf{x}-a)^T \mathbf{S}_{a}^{-1} (\mathbf{x}-a)
\end{split} \label{eq:scalar_cost_fct}
\end{equation}
Specifically, or this thesis retrieval scheme, $\mathbf{x}$ represents the PSD parameters of slope and number intercept of an assumed exponential size distribution for each radar range bin as in \Cref{eq:num_dens}. $y$ is the vector of MRR reflectivites. The vector $a$ is the a priori guess for slope parameter and number in each range bin. $F(\mathbf{x})$ represents the forward model that translates snow properties into reflectivity space.  Minimizing the cost function therefore seeks to reduce the difference between the observations, $y$, and simulated observations, $F(x)$ and between the a priori guess ($a$) and the retrieval vector ($x$).
\begin{align}
	n(r) & = N_{0} \exp\left(-\lambda r\right) \qquad [ \SI{}{\per\cubic\metre\per\mm} ] \label{eq:num_dens}
\end{align}
The $\mathbf{S}_y$ and $\mathbf{S}_{a}$ terms, in \Cref{eq:scalar_cost_fct}, represent the forward model and measurement error covariance matrix and the a priori error covariance matrix, respectively. The relative differences between $\mathbf{S}_{a}$ and $\mathbf{S}_y$ weight the importance of the observations and the a priori considerations in determining our best estimate of PSD properties.
\\
Newtonian iteration is used until the value of the cost function converges and our best estimate of snowfall properties are found.  The optimal-estimation scheme also provides error diagnostics through the retrieval error covariance matrix, $\mathbf{S}_x$, as in \Cref{eq:Sx}.
\begin{align}
	\mathbf{S}_x & = \left( \mathbf{S}_a^{-1} + \mathbf{K}^T \mathbf{S}_y^{-1} \mathbf{K} \right)^{-1}\label{eq:Sx}
\end{align}
The Jacobian matrix, $\mathbf{K}$, represents the sensitivity matrix of the perturbed result of the forward model. The true state $\mathbf{x}$ is perturbed by \SI{0.2}{\percent} and thus $\mathbf{K}$ represents the relation between simulated values to the true state and how sensitive the simulated values are to small changes when starting a new retrieval cycle. The closer $\mathbf{K}$ is diagonal, the more is $\mathbf{x}$ determined by the real observed and a priori values. If the limit of the partial derivative is close to unity, the retrieved value $\mathbf{x}$ is its true state \citep{wood_estimation_2011}.
\\
In practical application for this multiple layer retrieval scheme, log-transformed particle size distribution parameters of slope and number intercept were used due to the large expected range of these variables. The state vector, $\mathbf{x}$, is defined in \Cref{eq:snow_prop}.
\begin{align}
	\mathbf{x} & = \begin{bmatrix}
		log(\lambda)_0 	\\
		\vdots 			\\
		log(\lambda)_{\text{nlayer}} 	\\
		log(N_0)_0		\\
		\vdots			\\
		log(N_0)_{\text{nlayer}}
	\end{bmatrix} \qquad \text{nlayer} = 14
	\label{eq:snow_prop}
\end{align}
The usage of a priori terms were explored both from in-situ microphysical observations and from the PSD-temperature relationship developed for the CloudSat scheme as in \Cref{eq:lambda,eq:N0}. Temperatures in \SI{}{\celsius} at Haukeliseter were taken from site measurements with an assumption of a moist adiabatic lapse rate for the observed snow events. The log transformed slope and number intercept values were taken from \citet{wood_estimation_2011}.
\begin{align}
	\log(\lambda) & = -0.03053 \cdot T_{ap} - 0.08258  \label{eq:lambda} \qquad [ \log(\SI{}{\per\mm}) ] \\
	\log(N_0) & = -0.07193 \cdot T_{ap} +2.665  \qquad [ \log(\SI{}{\per\cubic\metre\per\mm})]
	\label{eq:N0}
\end{align}
The log-transformed equations are useful, since the results from C3VP were similar to other observations. The study showed, that $N_0$ ranges over several order of magnitude as well as $\lambda$ was non-Gaussian for the snow events \citet{wood_estimation_2011}. The diagonal matrix elements in $\mathbf{S}_a$ (\Cref{eq:scalar_cost_fct,eq:Sx}) are equal to \numlist{0.133;0.95} for the particle slope parameter and the number intercept, respectively, as from Eq. 7.35 and 7.36 in \citet{wood_estimation_2011}. The diagonal matrix elements for $\mathbf{S}_y$ are $2.5^2$ in \Cref{eq:scalar_cost_fct,eq:Sx}.
\\
After the best estimate of PSD parameters are found, the snow water content in each layer is calculated using the snow particle mass-dimension relationships as in \Cref{app:scat_scheme}.
%
\begin{align}
	\text{SWC} & = \int_{r_{min}}^{r_{max}} m(r) n(r) dr \qquad [\SI{}{\gram\per\cubic\metre}] \label{eq:SWC}
\end{align}
$r$ is the particle maximum dimension and $m(r)$ the related mass.
\par\medskip\noindent
This thesis work considered the database of particle models developed for the CloudSat mission, e.g. different types of aggregates, sector plates, and columns. Scattering properties for these snowflakes were calculated for the \SI{24}{\giga\hertz} frequency using discrete dipole approximation (DDA). Observations from the MASC of snowflake habit were used to guide particle selection. Snow water content, in turn, was translated into a snowfall rate using fall speed observations (MASC, PIP, or MRR Doppler velocity) or climatological analyses ($V = \SI{0.85}{\mPs}$). Surface snowfall rate is estimated using the SWC from the lowest non-noise reflectivity and radar bin.
\\
The forward model that calculates simulated \SI{24}{\giga\hertz} MRR reflectivities from PSD parameters was modified from that used in the CloudSat \SI{94}{\giga\hertz} operational snowfall product (2C-SNOW-PROFILE). Backscatter from frozen hydrometeors in each radar bin is summed up as in \Cref{eq:backscatter} form which reflectivity factor, Z, can be found (\Cref{eq:singleZ}).
% singly-scattered non-attenuated reflectivity
\begin{align}
	\eta_{bk} & = \int_{r_{min}}^{r_{max}} n(r) \sigma_{bk} dr \qquad [\SI{}{\per\metre}] \label{eq:backscatter} \\
	Ze^{ss,na} & = \frac{\Lambda^4}{\left\| K_w \right\|^2 \pi^5} \eta_{bk} \qquad [\SI{}{\mm^6\metre^{-3}}] \label{eq:singleZ}
\end{align}
Where, $\Lambda$ is the wavelength of the radar; $\left\| K_w \right\|^2$ is the complex refractive index of water. Radar backscatter values were estimated using discrete dipole approximation for the CloudSat particle models at \SI{24}{\giga\hertz}. Unlike the \SI{94}{\giga\hertz} spaceborne CloudSat mission, multiple scattering and attenuation can be neglected for \SI{24}{\giga\hertz} MRR and the short path length retrievals as viewed from the ground perspective.

%\newpage
\subsection{Environmental Masks for the Optimal Estimation Retrieval}\label{sec:pre_snow}
Different steps and assumptions are done in the here presented snowfall retrieval, to achieve vertical profiles of snowfall from MRR. The snowfall rate at the surface can be estimated from one of the lower levels. The optimal estimation retrieval is only performed for profiles, which are likely to have observed snow.
%Relationships between reflectivity and snowfall have been developed in previous studies. Different crystal shapes led to different results, even if the PSD of ice particles is known. Snow densities vary significantly from storm to storm, where small particles are still Rayleigh scattered, and larger particles non-Rayleigh scattered \citep{gunn_microwave_1954}.
%%%% image surface temperature and MRR %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\input{./images_tex/retrieval_MRR_sfcT}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\noindent
\\
This value was  chosen as sensitivity studies \citep[e.g.][]{wood_level_2013} that a days worth of such reflectivities would produce only a trace of snow.  Such a value therefore guarantees that a significant snow event is not missed and that any storms with lower \SI{}{\decibel Z} values would not produce meaningful precipitation.
%A reflectivity threshold of \SI{- 15}{\decibel Z} was selected for the scheme. The threshold is similar to the one used in \citet{wood_level_2013}. Light liquid precipitation is related to \SI{- 10}{\decibel Z} \citep{stephens_properties_2007}. \citet{wood_estimation_2011} found, that the reflectivity above \SI{-15}{\dB Z} are not influenced by ground clutter, when the reflectivity in the lowest bin and adjacent bin is compared.
The Haukeliseter measurement site is equipped with a weather mast, measuring the air temperature every minute at two-meter height (compare \Cref{fig:MRR_sfcT}, upper panel).
Since the MRR measures above \SI{300}{\metre} and temperature measurements exists only at the surface, a priori temperature ($T_{ap}$) at the surface is assumed to be similar to the observed near-surface air temperature. The use of a moist adiabatic lapse rate of $dT/dz = \SI{5}{\kelvin\per\km}$ gives $T_{ap}$ in each layer.
Snow existence at temperature measurements up to a threshold of +\SI{2}{\celsius} are assumed. \citet{liu_g._deriving_2008} validated this threshold, by analysing present weather reports to find the distinction between liquid and solid precipitation.\\
The purple line in the lower panel of \Cref{fig:MRR_sfcT} represents the time frame during \SI{25}{\dec}, where the MRR reflectivity is less than \SI{- 15}{\decibel Z}, and a priori temperature passes the \SI{2}{\celsius} limit at the surface.


\subsection{Datasets}
\begin{frame}
  \frametitle{Modeling the University of Illinois}
  \begin{itemize}
    \item All data is from the University of Illinois
    \item UIUC is a good model for thinking about hybrid energy systems.
    \begin{itemize}
      \item Solar Power
      \item Wind Power
      \item Natural Gas
      \item District Heating
    \end{itemize}
  \end{itemize}
\end{frame}

\subsection{Echo State Networks}
\begin{frame}
  \frametitle{Echo State Networks}
  \begin{figure}
    \includegraphics[width=\textwidth]{reservoir_network.jpg}
    \caption{A conceptual diagram of an echo state network. The reservoir is a large sparse matrix with randomly assigned entries \cite{pathak_model-free_2018, lukosevicius_practical_2012}.}
    \label{fig:resnet}
  \end{figure}
\end{frame}

\begin{frame}
  \frametitle{Hyper-parameter Optimization}
  \begin{columns}
    \column[t]{5cm}
      \begin{figure}
        \includegraphics[width=\columnwidth]{solar-angle-gridsearchNP.png}
        \caption{A grid search for the set of noise and spectral radius, $\rho$, that minimizes the mean squared error.}
        \label{fig:gridsearch}
      \end{figure}
    \column[t]{5cm}
      \vspace{1cm}
      \begin{align}
        MSE = \frac{1}{N}\sum_i^N(\hat y - y_i)^2
      \end{align}

      \begin{itemize}
        \item The optimal set of parameters is ``reservoir specific''
        \item Several other parameters need to be optimized such as:
          \begin{itemize}
            \item Reservoir Size
            \item Sparsity
            \item Training Length
          \end{itemize}
      \end{itemize}
  \end{columns}
\end{frame}

\begin{frame}
  \frametitle{Uncertainty Analysis}
  \begin{figure}
    \centering
    \includegraphics[width=0.85\textwidth]{demsol-bars01.png}
    \caption{Total demand prediction with error bars. The 60\% confidence is $\pm 1\sigma$ and the 95\% confidence interval is $\pm 2\sigma$. Mean is generated from predictions made by several different reservoirs.}
    \label{fig:uncertainty}
  \end{figure}
\end{frame}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%              tW                     %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Single-top \texorpdfstring{$tW$}{tW} associated production}
\label{subsec:tW}

This section describes the MC samples used for the modelling of single-top $tW$ associated production.
Section~\ref{subsubsec:tW_PP8} describes the \POWPY[8] samples -- both for the diagram removal (DR) set-ups,
which are used for the nominal prediction as well as uncertainties due to additional radiation and PDFs,
and for the diagram subtraction (DS) set-ups, which are used for the uncertainty due to the treatment
of the overlap with \ttbar production.
Section~\ref{subsubsec:tW_PH7} describes the \POWHER[7] samples used for the uncertainty
due to parton showering and hadronisation modelling, and Section~\ref{subsubsec:tW_aMCP8} describes the
\MGNLOPY[8] samples used for the uncertainty due to the choice of matching scheme.

The reference cross-section values are extracted from Ref.~\cite{LHCTopWGsgtopXsec}.

\subsubsection[Powheg+Pythia8]{\POWPY[8]}
\label{subsubsec:tW_PP8}

\paragraph{Samples}
%\label{par:tW_PP8_samples}

Table~\ref{tab:tW_PP8} gives the DSIDs of the $tW$ \POWPY[8] samples, for both the DR and DS schemes.
Single-top and single-anti-top ($tW^-$ and $\bar{t}W^+$) events were generated in different samples.
The dileptonic samples overlap with the inclusive ones.
\begin{table}[htbp]
\begin{center}
\caption{Single-top $tW$ associated production samples produced with \POWPY[8].}
\label{tab:tW_PP8}
\begin{tabular}{ l | l }
\hline
DSID & Description \\
\hline
410646 & $tW^-$ (DR) inclusive \\
410647 & $\bar{t}W^+$ (DR) inclusive \\
410648 & $tW^-$ (DR) dileptonic \\
410649 & $\bar{t}W^+$ (DR) dileptonic \\
\hline
410654 & $tW^-$ (DS) inclusive \\
410655 & $\bar{t}W^+$ (DS) inclusive \\
410656 & $tW^-$ (DS) dileptonic \\
410657 & $\bar{t}W^+$ (DS) dileptonic \\
\hline
\end{tabular}
\end{center}
\end{table}

\paragraph{Short description:}

The associated production of top quarks with $W$ bosons ($tW$) was
modelled by the
\POWHEGBOX[v2]~\cite{Re:2010bp,Nason:2004rx,Frixione:2007vw,Alioli:2010xd}
generator at NLO in QCD using the five-flavour scheme and the
\NNPDF[3.0nlo] set of PDFs~\cite{Ball:2014uwa}.
The diagram removal scheme~\cite{Frixione:2008yi} was used to
remove interference and overlap with \ttbar production.
The related uncertainty was estimated by comparison with an alternative sample
generated using the diagram subtraction scheme~\cite{Frixione:2008yi,ATL-PHYS-PUB-2016-020}.\footnote{Analyses which do not use this approach
should obviously not use this sentence in their description.}
The events were interfaced to \PYTHIA[8.230]~\cite{Sjostrand:2014zea} using the A14
tune~\cite{ATL-PHYS-PUB-2014-021} and the \NNPDF[2.3lo] set of
PDFs~\cite{Ball:2012cx}.

\input{top_isrfsr_powpy_short}

\paragraph{Long description:}

Single-top $tW$ associated production was modelled using the
\POWHEGBOX[v2]~\cite{Re:2010bp,Nason:2004rx,Frixione:2007vw,Alioli:2010xd}
generator, which provided matrix elements at next-to-leading
order~(NLO) in the strong coupling constant \alphas\ in the five-flavour
scheme with the \NNPDF[3.0nlo]~\cite{Ball:2014uwa} parton
distribution function~(PDF) set.  The functional form of the
renormalisation and factorisation scales was set to the default scale,
which is equal to the top-quark mass ($\mtop = 172.5\,\GeV$).
The diagram removal scheme~\cite{Frixione:2008yi} was employed to handle the interference
with \ttbar production~\cite{ATL-PHYS-PUB-2016-020}.  The events were
interfaced with \PYTHIA[8.230]~\cite{Sjostrand:2014zea} using the A14
tune~\cite{ATL-PHYS-PUB-2014-021} and the \NNPDF[2.3lo] PDF set.  The
decays of bottom and charm hadrons were simulated using the
\EVTGEN[1.6.0] program~\cite{Lange:2001uf}.

The inclusive cross-section was corrected to the theory prediction
calculated at NLO in QCD with NNLL soft-gluon
corrections~\cite{Kidonakis:2010ux,Kidonakis:2013zqa}.  For proton--proton
collisions at a centre-of-mass energy of \rts~=~\SI{13}{\TeV}, this
cross-section corresponds to $\sigma(tW)_\text{NLO+NNLL}= 71.7\pm3.8$\,pb,
using a top-quark mass of $\mtop = 172.5\,\GeV$.  The uncertainty in
the cross-section due to the PDF was calculated using the \MSTW[nnlo] 90\%
CL~\cite{Martin:2009iq,Martin:2009bu} PDF set, and was added in
quadrature to the effect of the scale uncertainty.

\input{top_isrfsr_powpy_long}

The nominal \POWPY[8] sample was compared with an alternative
sample generated using the diagram subtraction
scheme~\cite{Frixione:2008yi,ATL-PHYS-PUB-2016-020} to estimate the
uncertainty arising from the interference with \ttbar production.

To evaluate the PDF uncertainties for the nominal PDF, the 100
variations for \NNPDF[3.0nlo] were taken into account.
In addition, the central value of this PDF was compared with the
central values of the \CT[14nnlo]~\cite{Dulat:2015mca} and
\MMHT[nnlo]~\cite{Harland-Lang:2014zoa} PDF sets.

\subsubsection[Powheg+Herwig7]{\POWHER[7]}
\label{subsubsec:tW_PH7}

\paragraph{Samples}
%\label{par:tW_PH7_samples}

Table~\ref{tab:tW_PH7} gives the DSIDs of the $tW$ \POWHER[7] DR samples.
Single-top and single-anti-top ($tW^-$ and $\bar{t}W^+$) events were generated in different samples.
The dileptonic samples overlap with the inclusive ones.
\begin{table}[htbp]
\begin{center}
\caption{Single-top $tW$ associated production samples produced with \POWHER[7].}
\label{tab:tW_PH7}
\begin{tabular}{ l | l }
\hline
DSID & Description \\
\hline
411036 & $tW^-$ (DR) inclusive \\
411037 & $\bar{t}W^+$ (DR) inclusive \\
411038 & $tW^-$ (DR) dileptonic \\
411039 & $\bar{t}W^+$ (DR) dileptonic \\
\hline
\end{tabular}
\end{center}
\end{table}

\paragraph{Short description:}

The uncertainty due to the parton shower and hadronisation model was
evaluated by comparing the nominal sample of events with a sample where
events generated with the
\POWHEGBOX[v2]~\cite{Re:2010bp,Nason:2004rx,Frixione:2007vw,Alioli:2010xd}
generator were interfaced to
\HERWIG[7.04]~\cite{Bahr:2008pv,Bellm:2015jjp}, using the H7UE set
of tuned parameters~\cite{Bellm:2015jjp} and the \MMHT[lo] PDF set~\cite{Harland-Lang:2014zoa}.

\paragraph{Long description:}

The impact of using a different parton shower and hadronisation model was evaluated
by comparing the nominal $tW$ sample with another sample produced with the
\POWHEGBOX[v2]~\cite{Re:2010bp,Nason:2004rx,Frixione:2007vw,Alioli:2010xd}
generator but interfaced with \HERWIG[7.04]~\cite{Bahr:2008pv,Bellm:2015jjp},
using the H7UE set of tuned parameters~\cite{Bellm:2015jjp} and the
\MMHT[lo] PDF set \cite{Harland-Lang:2014zoa}.
\POWHEGBOX provided matrix elements at next-to-leading order~(NLO) in the
strong coupling constant \alphas in the five-flavour scheme with the
\NNPDF[3.0nlo]~\cite{Ball:2014uwa} parton distribution function~(PDF).
The functional form of the renormalisation and factorisation scales was set to
the default scale, which is equal to the top-quark mass.  The diagram removal
scheme~\cite{Frixione:2008yi} was employed to handle the interference
with \ttbar production~\cite{ATL-PHYS-PUB-2016-020}.  The decays of bottom
and charm hadrons are simulated using the \EVTGEN[1.6.0]
program~\cite{Lange:2001uf}.

\subsubsection[MadGraph5\_aMC@NLO+Pythia8]{\MGNLOPY[8]}
\label{subsubsec:tW_aMCP8}

\paragraph{Samples}
%\label{par:tW_aMCP8_samples}

Table~\ref{tab:tW_aMCP8} gives the DSIDs of the $tW$ \MGNLOPY[8] samples.
The dileptonic sample overlaps with the inclusive one.
\begin{table}[htbp]
\begin{center}
\caption{Single-top $tW$ associated production samples produced with \MGNLOPY[8].}
\label{tab:tW_aMCP8}
\begin{tabular}{ l | l }
\hline
DSID & Description \\
\hline
412002 & $tW$ inclusive \\
412003 & $tW$ dileptonic \\
\hline
\end{tabular}
\end{center}
\end{table}

\paragraph{Short description:}

To assess the uncertainty in the matching of NLO matrix elements to the
parton shower, the nominal $tW$ sample was compared with a sample generated
with the \MGNLO[2.6.2] generator at NLO in QCD using the five-flavour
scheme and the \NNPDF[2.3nlo]~\cite{Ball:2014uwa} PDF set. The events were
interfaced with \PYTHIA[8.230]~\cite{Sjostrand:2014zea}, using the A14
set of tuned parameters~\cite{ATL-PHYS-PUB-2014-021} and the \NNPDF[2.3lo]
PDF.

\paragraph{Long description:}

To assess the uncertainty due to the choice of matching scheme, the nominal $tW$ sample was compared with a sample generated
with the \MGNLO[2.6.2] generator, which provided matrix elements at next-to-leading order~(NLO) in the strong coupling constant \alphas
in the five-flavour scheme, using the \NNPDF[2.3nlo]~\cite{Ball:2014uwa} PDF set.
The functional form of the renormalisation and factorisation scale was set to the default scale, which is equal to the top-quark mass.
The parton-shower starting scale had the functional form $\muQ = H_{\mathrm T}/2$~\cite{ATL-PHYS-PUB-2017-007},
where $H_{\mathrm T}$ is defined as the scalar sum of the \pT of all outgoing partons.
The diagram removal scheme~\cite{Frixione:2008yi} was employed to handle the interference with \ttbar production~\cite{ATL-PHYS-PUB-2016-020}.
The events were interfaced with \PYTHIA[8.230]~\cite{Sjostrand:2014zea}, using the A14 set of tuned parameters~\cite{ATL-PHYS-PUB-2014-021}
and the \NNPDF[2.3lo] PDF.
The decays of bottom and charm hadrons were simulated using the \EVTGEN[1.6.0] program~\cite{Lange:2001uf}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%              t-channel              %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Single-top \texorpdfstring{$t$}{t}-channel production}
\label{subsec:tchan}

This section describes the MC samples used for the modelling of single-top $t$-channel production.
Section~\ref{subsubsec:tchan_PP8} describes the \POWPY[8] samples used for the nominal prediction
and for the uncertainty from additional radiation and due to PDFs.
Section~\ref{subsubsec:tchan_PH7} describes the \POWHER[7] samples used for the uncertainty due to the choice of parton shower and hadronisation model,
and Section~\ref{subsubsec:tchan_aMCP8} describes the \MGNLOPY[8] samples used for the uncertainty due to the choice of matching scheme.

The reference cross-section values are extracted from Ref.~\cite{LHCTopWGsgtopXsec}.

\subsubsection[Powheg+Pythia8]{\POWPY[8]}
\label{subsubsec:tchan_PP8}

\paragraph{Samples}
%\label{par:tchan_PP8_samples}

Table~\ref{tab:tchan_PP8} gives the DSIDs of the $t$-channel \POWPY[8] samples.
Single-top and single-anti-top events were generated in distinct samples.
\begin{table}[!htbp]
\begin{center}
\caption{Single-top $t$-channel event samples produced with \POWPY[8].}
\label{tab:tchan_PP8}
\begin{tabular}{ l | l }
\hline
DSID & Description \\
\hline
410658 & $t$-channel $t$ leptonic \\
410659 & $t$-channel $\bar{t}$ leptonic \\
\hline
\end{tabular}
\end{center}
\end{table}

\paragraph{Short description:}

Single-top $t$-channel production was modelled using the
\POWHEGBOX[v2]~\cite{Frederix:2012dh,Nason:2004rx,Frixione:2007vw,Alioli:2010xd}
generator at NLO in QCD using the four-flavour scheme and the
corresponding \NNPDF[3.0nlo] set of PDFs~\cite{Ball:2014uwa}.  The events were
interfaced with \PYTHIA[8.230]~\cite{Sjostrand:2014zea} using the A14
tune~\cite{ATL-PHYS-PUB-2014-021} and the \NNPDF[2.3lo] set of
PDFs~\cite{Ball:2012cx}.

\input{top_isrfsr_powpy_short}

\paragraph{Long description:}

Single-top $t$-channel production was modelled using the
\POWHEGBOX[v2]~\cite{Frederix:2012dh,Nason:2004rx,Frixione:2007vw,Alioli:2010xd}
generator, which provided matrix elements at next-to-leading-order~(NLO)
accuracy in the strong coupling constant \alphas\ in the four-flavour
scheme with the corresponding \NNPDF[3.0nlo]~\cite{Ball:2014uwa} parton
distribution function~(PDF) set.  The functional form of the
renormalisation and factorisation scales was set to
$\sqrt{m_b^2 + p_{\text{T},b}^2}$ following the
recommendation of Ref.~\cite{Frederix:2012dh}. Top quarks were decayed at
LO using \MADSPIN~\cite{Frixione:2007zp,Artoisenet:2012st} to preserve
all spin correlations.  The events were interfaced with
\PYTHIA[8.230]~\cite{Sjostrand:2014zea} using the A14
tune~\cite{ATL-PHYS-PUB-2014-021} and the \NNPDF[2.3lo] PDF set.
The decays of bottom and charm hadrons were simulated using the
\EVTGEN[1.6.0] program~\cite{Lange:2001uf}.

The inclusive cross-section was corrected to the theory prediction calculated at NLO in QCD with
\HATHOR[2.1]~\cite{Aliev:2010zk,Kant:2014oha}.
For proton--proton collisions at a centre-of-mass energy of \rts~=~\SI{13}{\TeV}, this cross-section corresponds to
$\sigma(t,t\text{-chan})_\text{NLO}= 136.02^{+5.40}_{-4.57}$\,pb ($\sigma(\bar{t},t\text{-chan})_\text{NLO}=80.95^{+4.06}_{-3.61}$\,pb)
for single-top (single-anti-top) production, using a top-quark mass of $\mtop = 172.5\,\GeV$.
The uncertainties in the cross-section due to the PDF and \alphas were calculated using the \PDFforLHC prescription~\cite{Butterworth:2015oua}
with the \MSTW[nlo] 68\% CL~\cite{Martin:2009iq,Martin:2009bu}, \CT[10nlo]~\cite{Lai:2010vv}
and \NNPDF[2.3nlo]~\cite{Ball:2012cx} PDF sets,
and were added in quadrature to the effect of the scale uncertainty.

\input{top_isrfsr_powpy_long}

To evaluate the PDF uncertainties for the nominal PDF, the 100 variations for \NNPDF[3.0nlo] were taken into account.
In addition, the central value of this PDF was compared with the central values of the \CT[14nnlo]~\cite{Dulat:2015mca}
and \MMHT[nnlo]~\cite{Harland-Lang:2014zoa} PDF sets.

\subsubsection[Powheg+Herwig7]{\POWHER[7]}
\label{subsubsec:tchan_PH7}

\paragraph{Samples}
%\label{par:tchan_PH7_samples}

Table~\ref{tab:tchan_PH7} gives the DSIDs of the $t$-channel \POWHER[7] samples.
Single-top and single-anti-top events were generated in distinct samples.
\begin{table}[htbp]
\begin{center}
\caption{Single-top $t$-channel event samples produced with \POWHER[7].}
\label{tab:tchan_PH7}
\begin{tabular}{ l | l }
\hline
DSID & Description \\
\hline
411032 & $t$-channel $\bar t$ leptonic \\
411033 & $t$-channel $t$ leptonic \\
\hline
\end{tabular}
\end{center}
\end{table}

\paragraph{Short description:}

The uncertainty due to the parton shower and hadronisation model was
evaluated by comparing the nominal sample of events with a sample where
the events generated with the
\POWHEGBOX[v2]~\cite{Frederix:2012dh,Nason:2004rx,Frixione:2007vw,Alioli:2010xd}
generator were interfaced to
\HERWIG[7.04]~\cite{Bahr:2008pv,Bellm:2015jjp}, using the H7UE set
of tuned parameters~\cite{Bellm:2015jjp} and the \MMHT[lo] PDF set
\cite{Harland-Lang:2014zoa}.

\paragraph{Long description:}

The impact of using a different parton shower and hadronisation model was evaluated by comparing the nominal sample
with another sample produced with the \POWHEGBOX[v2]~\cite{Frederix:2012dh,Nason:2004rx,Frixione:2007vw,Alioli:2010xd}
generator but interfaced with \HERWIG[7.04]~\cite{Bahr:2008pv,Bellm:2015jjp}, using the H7UE set of
tuned parameters~\cite{Bellm:2015jjp} and the \MMHT[lo] PDF set \cite{Harland-Lang:2014zoa}.
\POWHEGBOX provided matrix elements at next-to-leading order~(NLO) in the strong coupling constant \alphas
in the four-flavour scheme with the corresponding \NNPDF[3.0nlo]~\cite{Ball:2014uwa} parton distribution function~(PDF).
The functional form of the renormalisation and factorisation scales was set to $\sqrt{m_b^2 + p_{\text{T},b}^2}$
following the recommendation of Ref.~\cite{Frederix:2012dh}.
Top quarks were decayed at LO using \MADSPIN~\cite{Frixione:2007zp,Artoisenet:2012st} to preserve all spin correlations.
The decays of bottom and charm hadrons were simulated using the \EVTGEN[1.6.0] program~\cite{Lange:2001uf}.

\subsubsection[MadGraph5\_aMC@NLO+Pythia8]{\MGNLOPY[8]}
\label{subsubsec:tchan_aMCP8}

\paragraph{Samples}
%\label{par:tchan_aMCP8_samples}

Table~\ref{tab:tchan_aMCP8} gives the DSIDs of the $t$-channel \MGNLOPY[8] samples.
\begin{table}[htbp]
\begin{center}
\caption{Single-top $t$-channel event samples produced with \MGNLOPY[8].}
\label{tab:tchan_aMCP8}
\begin{tabular}{ l | l }
\hline
DSID & Description \\
\hline
412004 & $t$-channel leptonic \\
\hline
\end{tabular}
\end{center}
\end{table}

\paragraph{Short description:}

To assess the uncertainty in the matching of NLO matrix elements to the
parton shower, the nominal sample was compared with a sample generated
with the \MGNLO[2.6.2] generator at NLO in QCD using the five-flavour
scheme and the \NNPDF[2.3nlo]~\cite{Ball:2014uwa} PDF set. The events were
interfaced with \PYTHIA[8.230]~\cite{Sjostrand:2014zea}, using the A14
set of tuned parameters~\cite{ATL-PHYS-PUB-2014-021} and the \NNPDF[2.3lo] PDF set.

\paragraph{Long description:}

To assess the uncertainty due to the choice of matching scheme, the nominal sample was compared with a sample generated
with the \MGNLO[2.6.2] generator, which provided matrix elements at next-to-leading order~(NLO) in the strong coupling constant \alphas
in the four-flavour scheme, using the corresponding \NNPDF[3.0nlo]~\cite{Ball:2014uwa} PDF set.
The functional form of the renormalisation and factorisation scales was set to $\sqrt{m_b^2 + p_{\text{T},b}^2}$
following the recommendation of Ref.~\cite{Frederix:2012dh}.
The parton-shower starting scale had the functional form $\muQ = H_\text{T}/2$~\cite{ATL-PHYS-PUB-2017-007},
where $H_\text{T}$ is defined as the scalar sum of the \pT of all outgoing partons.
Top quarks were decayed at LO using \MADSPIN~\cite{Frixione:2007zp,Artoisenet:2012st} to preserve all spin correlations.
The events were interfaced with \PYTHIA[8.230]~\cite{Sjostrand:2014zea}, using the A14 set of tuned parameters~\cite{ATL-PHYS-PUB-2014-021}
and the \NNPDF[2.3lo] PDF set.
The decays of bottom and charm hadrons were simulated using the \EVTGEN[1.6.0] program~\cite{Lange:2001uf}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%              s-channel              %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Single-top \texorpdfstring{$s$}{s}-channel production}
\label{subsec:schan}

This section describes the MC samples used for the modelling of single-top $s$-channel production.
Section~\ref{subsubsec:schan_PP8} describes the \POWPY[8] samples used for the nominal prediction
and for the uncertainty from additional radiation and due to PDFs.
Section~\ref{subsubsec:schan_PH7} describes the \POWHER[7] samples used for the uncertainty due to the parton shower and hadronisation model,
and Section~\ref{subsubsec:schan_aMCP8} describes the \MGNLOPY[8] samples used for the uncertainty due to the choice of matching scheme.

The reference cross-section values are extracted from Ref.~\cite{LHCTopWGsgtopXsec}.

\subsubsection[Powheg+Pythia8]{\POWPY[8]}
\label{subsubsec:schan_PP8}

\paragraph{Samples}
%\label{par:schan_PP8_samples}

Table~\ref{tab:schan_PP8} gives the DSIDs of the $s$-channel \POWPY[8] samples.
Single-top and single-anti-top events were generated in distinct samples.
\begin{table}[htbp]
\begin{center}
\caption{Single-top $s$-channel event samples produced with \POWPY[8].}
\label{tab:schan_PP8}
\begin{tabular}{ l | l }
\hline
DSID & Description \\
\hline
410644 & $s$-channel $t$ leptonic \\
410645 & $s$-channel $\bar{t}$ leptonic \\
\hline
\end{tabular}
\end{center}
\end{table}

\paragraph{Short description:}

Single-top $s$-channel production was modelled using the \POWHEGBOX[v2]~\cite{Alioli:2009je,Nason:2004rx,Frixione:2007vw,Alioli:2010xd}
generator at NLO in QCD in the five-flavour scheme with the \NNPDF[3.0nlo]~\cite{Ball:2014uwa} parton distribution function~(PDF) set.
The events were interfaced with \PYTHIA[8.230]~\cite{Sjostrand:2014zea} using the A14 tune~\cite{ATL-PHYS-PUB-2014-021} and the
\NNPDF[2.3lo] PDF set.

\input{top_isrfsr_powpy_short}

\paragraph{Long description:}

Single-top $s$-channel production was modelled using the \POWHEGBOX[v2]~\cite{Alioli:2009je,Nason:2004rx,Frixione:2007vw,Alioli:2010xd}
generator, which provided matrix elements at next-to-leading order~(NLO) in the strong coupling constant \alphas in the
five-flavour scheme with the \NNPDF[3.0nlo]~\cite{Ball:2014uwa} parton distribution function~(PDF) set.
The functional form of the renormalisation and factorisation scales was set to the default scale, which was equal to the top-quark mass.
The events were interfaced with \PYTHIA[8.230]~\cite{Sjostrand:2014zea} using the A14 tune~\cite{ATL-PHYS-PUB-2014-021} and the
\NNPDF[2.3lo] PDF set.
The decays of bottom and charm hadrons were simulated using the \EVTGEN[1.6.0] program~\cite{Lange:2001uf}.

The inclusive cross-section was corrected to the theory prediction calculated at NLO in QCD with
\HATHOR[2.1]~\cite{Aliev:2010zk,Kant:2014oha}.
For proton--proton collisions at a centre-of-mass energy of \rts~=~\SI{13}{\TeV}, this cross-section corresponds to
$\sigma(t,s\text{-chan})_\text{NLO}= 6.35^{+0.23}_{-0.20}$\,pb ($\sigma(\bar{t},s\text{-chan})_\text{NLO}=3.97^{+0.19}_{-0.17}$\,pb)
for single-top (single-anti-top) production, using a top-quark mass of $\mtop = 172.5\,\GeV$.
The uncertainties in the cross-section due to the PDF and \alphas were calculated using the \PDFforLHC prescription~\cite{Butterworth:2015oua}
with the \MSTW[nlo] 68\% CL~\cite{Martin:2009iq,Martin:2009bu}, \CT[10nlo]~\cite{Lai:2010vv} and \NNPDF[2.3nlo]~\cite{Ball:2012cx} PDF sets,
and were added in quadrature to the effect of the scale uncertainty.

\input{top_isrfsr_powpy_long}

To evaluate the PDF uncertainties for the nominal PDF, the 100 variations for \NNPDF[3.0nlo] were taken into account.
In addition, the central value of this PDF was compared with the central values of the
\CT[14nnlo]~\cite{Dulat:2015mca} and \MMHT[nnlo]~\cite{Harland-Lang:2014zoa} PDF sets.

\subsubsection[Powheg+Herwig7]{\POWHER[7]}
\label{subsubsec:schan_PH7}

\paragraph{Samples}
%\label{par:schan_PH7_samples}

Table~\ref{tab:schan_PH7} gives the DSIDs of the $s$-channel \POWHER[7] samples.
Single-top and single-anti-top events were generated in distinct samples.
\begin{table}[htbp]
\begin{center}
\caption{Single-top $s$-channel event samples produced with \POWHER[7].}
\label{tab:schan_PH7}
\begin{tabular}{ l | l }
\hline
DSID & Description \\
\hline
411034 & $s$-channel $t$ leptonic \\
411035 & $s$-channel $\bar{t}$ leptonic \\
\hline
\end{tabular}
\end{center}
\end{table}

\paragraph{Short description:}

The impact of using a different parton shower and hadronisation model was evaluated by comparing the nominal sample
with another sample produced with the \POWHEGBOX[v2]~\cite{Alioli:2009je,Nason:2004rx,Frixione:2007vw,Alioli:2010xd}
generator at NLO in the strong coupling constant \alphas in the five-flavour scheme using the
\NNPDF[3.0nlo]~\cite{Ball:2014uwa} parton distribution function~(PDF).
Events in the latter sample were interfaced with \HERWIG[7.04]~\cite{Bahr:2008pv,Bellm:2015jjp}, using the H7UE set of
tuned parameters~\cite{Bellm:2015jjp} and the \MMHT[lo] PDF set \cite{Harland-Lang:2014zoa}.

\paragraph{Long description:}

The impact of using a different parton shower and hadronisation model was evaluated by comparing the nominal sample
with another sample produced with the \POWHEGBOX[v2]~\cite{Alioli:2009je,Nason:2004rx,Frixione:2007vw,Alioli:2010xd}
generator but interfaced with \HERWIG[7.04]~\cite{Bahr:2008pv,Bellm:2015jjp}, using the H7UE set of
tuned parameters~\cite{Bellm:2015jjp} and the \MMHT[lo] PDF set \cite{Harland-Lang:2014zoa}.
\POWHEGBOX provided matrix elements at next-to-leading order~(NLO) in the strong coupling constant \alphas
in the five-flavour scheme with the \NNPDF[3.0nlo]~\cite{Ball:2014uwa} parton distribution function~(PDF).
The functional form of the renormalisation and factorisation scales was set to the default scale, which is equal to the top-quark mass.
The decays of bottom and charm hadrons were simulated using the \EVTGEN[1.6.0] program~\cite{Lange:2001uf}.

\subsubsection[MadGraph5\_aMC@NLO+Pythia8]{\MGNLOPY[8]}
\label{subsubsec:schan_aMCP8}

\paragraph{Samples}
%\label{par:schan_aMCP8_samples}

Table~\ref{tab:schan_aMCP8} gives the DSIDs of the $s$-channel \MGNLOPY[8] samples.
\begin{table}[htbp]
\begin{center}
\caption{Single-top $s$-channel event samples produced with \MGNLOPY[8].}
\label{tab:schan_aMCP8}
\begin{tabular}{ l | l }
\hline
DSID & Description \\
\hline
412005 & $s$-channel leptonic \\
\hline
\end{tabular}
\end{center}
\end{table}

\paragraph{Short description:}

%% the MG5_aMC@NLO version was doulble checked in the tag collector
To assess the uncertainty due to the choice of matching scheme, the nominal sample was compared with a sample generated
with the \MGNLO[2.6.2] generator at NLO in the strong coupling constant \alphas in the five-flavour scheme,
using the \NNPDF[3.0nlo]~\cite{Ball:2014uwa} PDF set.
The events were interfaced with \PYTHIA[8.230]~\cite{Sjostrand:2014zea}, using the A14 set of tuned parameters~\cite{ATL-PHYS-PUB-2014-021}
and the \NNPDF[2.3lo] PDF set.

\paragraph{Long description:}

To assess the uncertainty due to the choice of matching scheme, the nominal sample was compared with a sample generated
with the \MGNLO[2.6.2] generator, which provided matrix elements at next-to-leading order~(NLO) in the strong coupling constant \alphas
in the five-flavour scheme with the \NNPDF[3.0nlo]~\cite{Ball:2014uwa} parton distribution function~(PDF).
The functional form of the renormalisation and factorisation scales was set to the default scale, which is equal to the top-quark mass.
The parton-shower starting scale had the functional form $\muQ = H_{\mathrm T}/2$~\cite{ATL-PHYS-PUB-2017-007},
where $H_{\mathrm T}$ is defined as the scalar sum of the \pT of all outgoing partons.
Top quarks were decayed at LO using \MADSPIN~\cite{Frixione:2007zp,Artoisenet:2012st} to preserve all spin correlations.
The events were interfaced with \PYTHIA[8.230]~\cite{Sjostrand:2014zea}, using the A14 set of tuned parameters~\cite{ATL-PHYS-PUB-2014-021}
and the \NNPDF[2.3lo] PDF set.
The decays of bottom and charm hadrons were simulated using the \EVTGEN[1.6.0] program~\cite{Lange:2001uf}.


\begin{figure}
\begin{center}
    \includegraphics[width=0.9\textwidth]{figs/abc_slope_AV_starforming.pdf}
    \caption{\label{fig:slope}
    The attenuation-slope relation of star-forming galaxies ($\ssfr >
    10^{-11}yr^{-1}$), using the attenuation curves predicted by our
    \eda~prescription for the median posterior parameter values of SIMBA
    (left), TNG (center) and EAGLE (right).
    For comparison, we include the observed attenuation-slope relation
    from GSWLC2~\citep{salim2020}.
    We use $A_V$ and $S = A(1500\AA)/A_V$ as measurements of attenuation and
    slope, respectively.
    \emph{The \eda~does not predict $A_V < 0.3$ because star-forming galaxies in
    the simulations are intrinsically too lumnious and require significant attenuation to
    match observations.}
    Beyond $A_V > 0.3$, however, there is good agreement between the
    attenuation-slope relation predicted by the \eda~and observations.
    }
\end{center}
\end{figure}

\subsection{Comparison to Dust Observations} \label{sec:reproduce}
In addition to  reproducing observations, the \eda~assigns dust attenuation
curves to each simulated galaxy so we can compare the \eda~attenuation curves
to dust attenuation measured from observations.
We begin with the well-established attenuation-slope relation: star-forming
galaxies with higher dust attenuation have shallower attenuation curves.
This relation is a consequence of dust scattering dominating absorption at
low attenuation while dust absorption dominates at high
attenuation~\citep{gordon1994, witt2000, draine2003, chevallard2013}.
In Figure~\ref{fig:slope}, we present the attenuation-slope relation of
star-forming galaxies with $\ssfr > 10^{-11}yr^{-1}$ using the
dust attenuation curves predicted by the \eda~for the median posteriors of
SIMBA (left), TNG (center) and EAGLE (right).
For comparison, we include the observed attenuation-slope relations of
GSWLC2 galaxies~\citep[grey shaded;][]{salim2020}.
For attenuation, we use $A_V$; for slope, we use the UV-optical slope, $S
= A(1500\AA)/A_V$.
The contours mark the 68 and 95 percentiles.

Most noticably, we find that the \eda~does not predict $A_V < 0.3$ for
star-forming galaxies.
%\chedit{
%    This is not due to the selection function imposed by our forward model.
%    The attenuation-slope relation of star-forming galaxies in GSWLC2 does not
%    change significantly if we impose similar selection cuts as our
%    observational sample.
%    Instead, the lack of star-forming galaxies with $A_V < 0.3$ is a
This is a consequence of SIMBA, TNG, and EAGLE predicting star-forming galaxies
that are intrinsically more luminous than observations.
All of the simulations have star-forming galaxies with intrinsic $M_r <
-21$ and $\gr < 0.5$ (Figure~\ref{fig:obs}).
This is further corroborated by the $\sfr-M*$ relations in
Figure~\ref{fig:smf_msfr}, where the simulations all have star-forming
galaxies with $M_* > 10^{11}M_\odot$, not found in SDSS.
To reproduce the SDSS optical color-magnitude relation these galaxies would
need to be significantly reddened and attenuated so any dust prescription would
need to assign high $A_V$ to star-forming galaxies.
Nevertheless, for $A_V > 0.3$, we find good agreement between the
attenuation-slope relation predicted by the \eda~and observations.
We refrain from further scrutinizing on $A_V$ values, since they can vary {\em
significantly} between different observational measurements --- even for the
same galaxy.
SDSS star-forming galaxies, for instance, have significantly higher
$A_V > 0.3$ according to the \cite{brinchmann2004} measurements
(Appendix~\ref{sec:slab}).

%We note that the difference in the $A_V$ ranges is due to the $M_r$
%completeness limit imposed by our forward model (Section~\ref{sec:fm}).
%The GSWLC2 sample in \cite{salim2020} extends down to $M_* \sim
%10^{8.5}M_\odot$; however, the TNG and EAGLE samples do not extend below
%$M_* \sim10^{10}M_\odot$.
%\emph{The \eda~predicts attenuation-slope relations for TNG and EAGLE that
%are in excellent agreement with observations.}

\begin{figure}
\begin{center}
    \includegraphics[width=0.5\textwidth]{figs/abc_sf_attenuation.pdf}
    \caption{\label{fig:sfatten}
    The normalized attenuation curves of star-forming galaxies predicted by
    the \eda~for median posterior parameter values of SIMBA (orange), TNG
    (blue), and EAGLE (green).
    We classify galaxies with $\log \ssfr > -11~yr^{-1}$ as star-forming.
    The attenuation curves are normalized at $3000\AA$ and we mark the
    68 percentile of the attenuation curves with the shaded region.
    For comparison, we include $A(\lambda)/A(3000\AA)$ measurements from
    the~\cite{narayanan2018} radiative transfer simulation (dashed) and
    \cite{salim2018} observations (dotted).
    %The \cite{calzetti2000} and \cite{battisti2017} attenuation curves are shallower than the \eda~attenuation curves; however, they probe lower $M_*$ galaxies than our forward modeled TNG and EAGLE samples.  For attenuation curve from \cite{salim2018}, which probe a similar $M_*$ range, we find goood agreement.
    {\em The \eda~predict attenuation curves of star-forming galaxies
    are in good agreement with the attenuation curves measured from
    the simulation and observations in the literature.}
    %We also find good agreement with median attenuation curve of star-forming galaxies in the radiative transfer simulations of \cite{narayanan2018}.
    }
\end{center}
\end{figure}

In addition to the attenuation-slope relation, we can also directly compare
the attenuation curves predicted by the \eda~to measurements from
observations for star-forming galaxies.
In Figure~\ref{fig:sfatten}, we present the normalized attenuation curves
of star-forming galaxies predicted by the \eda~for the median posterior
parameter values of SIMBA(orange), TNG (blue), and EAGLE (green).
We again define galaxies with $\ssfr > 10^{-11}{yr}^{-1}$ as star-forming.
The attenuation curves are normalized at $3000\AA$ and we present the
variation in the attenuation curves in the shaded region, 68 percentile.
For comparison, we include $A(\lambda)/A(3000\AA)$ from the
\cite{narayanan2018} radiative transfer simulation (dashed) and
observations~\citep[][dotted]{salim2018}.
The attenuation curve from \cite{salim2018} corresponds to star-forming
galaxies with $M_* > 10^{10.5}M_\odot$, a similar $M_*$ range as our
forward modeled samples.
Since we do not vary the UV bump in our \eda~prescription, we ignore any
discrepancies in the amplitudes of the bump.
\emph{Overall, we find good agreement between the \eda~attenuation curves for
star-forming galaxies and the attenuation curves from observations and
simulations.}

%Again, the fact that we reproduce the detailed dust attenuation curves of star-forming galaxies in observations and simulations with the \eda~without fitting for them, highlights the advantages of a forward modeling approach.

%The \eda~attenuation curves are slightly steeper than the \cite{calzetti2000} and \cite{battisti2017} curves.
%These attenuation curves, however, are derived from $M_* < 10^{9.9}M_\odot$
%star-forming galaxies, which lie below the $M_*$ limit of our forward
%modeled TNG and EAGLE samples.
%Meanwhile, the TNG and EAGLE \eda~attenuation curves are in good agreement
%with the \cite{salim2018} attenuation curve for $M_* > 10^{10.5}M_\odot$ star-forming galaxies.
%They are also consistent with the median curve of \cite{narayanan2018}.

%The \eda~attenuation curves are noticeably steeper than the \cite{calzetti2000} and \cite{battisti2017} curves.  These attenuation curves, however, are derived from $M_* < 10^{9.9}M_\odot$ star-forming galaxies --- below our $M_*$ range.
%Since we find $\mdeltam < 0$ for both the TNG and EAGLE posteriors, the \eda~attenuation curves are consistent with \cite{calzetti2000} and \cite{battisti2017}.

%\chedit{
%    The \eda~predicts higher dust attenuation at lower wavelenghts for
%    star-forming galaxies.
%    Without dust attenuation, both TNG and EAGLE predict star-forming galaxies
%    that are bluer in the optical and UV than observations
%    (Figure~\ref{fig:obs}).
%    To reproduce the SDSS, the \eda~significantly reddens star-forming galaxies.
%}
%In Figure~\ref{fig:raw_atten}, we also find that more massive star-forming
%galaxies have higher attenuation. This is because the simulations overpredict
%luminous blue star-forming galaxies, which must be attenuated to reproduce
%observations.

%At low attenuation, dust scattering dominates absoprtion so the
%attenuation curve steepens because red light scatters isotropically while blue light
%scatters forward~\citep{gordon1994, witt2000, draine2003}. %, which causes more optical-to-IR light to escape the galaxy than UV light
%At high attenuation dust absorption is dominant and the attenuation curve is
%shallower~\citep{chevallard2013}. For the $A_V$ range probed by the DEM, the
%$A_V$--slope relation is in good agreement with GSWLC2 galaxies~\citep[black shaded][]{salim2020}.
%They are also consistent with \cite{leja2017}. We also compare our results to
%theoretical predictions from radiative transfer models, \cite{inoue2005}
%(dotted), the radiative transfer models considered in \cite{chevallard2013}
%(dot dashed), and \cite{trayford2020} (light shaded), which all predict shallower
%attenuation curves than observations. This is also the case for the
%\cite{narayanan2018} attenuation curves (not included).
%\emph{The attenuation curve slopes from the DEM for are in excellent
%agreement with observations and better reproduces the observed
%attenuation--slope relation than radiative transfer models.}


%-------------------------------------------------------------------------------
\section{Event reconstruction}
\label{sec:objects}
%-------------------------------------------------------------------------------

The event reconstruction is affected by multiple $pp$ collisions in a single bunch crossing and by collisions
in neighbouring bunch crossings, referred to as pile-up.
Interaction vertices from the $pp$ collisions are reconstructed from at least two tracks
with transverse momentum ($\pt$) larger than $400~\mev$ that are consistent with originating from the
beam collision region in the $x$--$y$ plane. If more than one primary vertex candidate is found, the
candidate whose associated tracks form the largest sum of squared $\pt$~\cite{ATL-PHYS-PUB-2015-026}
is selected as the hard-scatter primary vertex.

Electron candidates~\cite{ATLAS-CONF-2016-024,Aaboud:2018ugz} are reconstructed from energy
clusters in the EM calorimeter that are matched to reconstructed tracks in the inner detector;
electron candidates in the transition region between the EM barrel and endcap calorimeters
($1.37 < |\eta_{\textrm{cluster}}| < 1.52$) are excluded.
Electron candidates are required to have $|\eta_{\textrm{cluster}}| < 2.47$, and to satisfy tight likelihood-based identification
criteria~\cite{ATLAS-CONF-2016-024} based on calorimeter, tracking and combined variables that provide
separation between electrons and jets.

Muon candidates~\cite{Aad:2016jkr} are reconstructed by matching track segments in %the various
different layers of the muon spectrometer to tracks found in the inner detector;
the resulting muon candidates are re-fitted using the complete track information from both detector systems.
Muon candidates are required to satisfy medium identification criteria~\cite{Aad:2016jkr}.

Electron (muon) candidates are matched to the primary vertex by requiring that the significance of their transverse impact parameter, $d_0$,
satisfies $|d_0/\sigma(d_0)|<5\,(3)$, where $\sigma(d_0)$ is the measured uncertainty in $d_0$,
and by requiring that their longitudinal impact parameter, $z_0$, satisfies $|z_0 \sin\theta|<0.5$~mm. To reduce the impact of the trigger efficiency uncertainty around the threshold, the leptons are required to have \pt 1~GeV above the threshold.
To further reduce the background from non-prompt leptons, photon conversions and hadrons, lepton candidates are also required to be isolated
in the tracker and in the calorimeter.
A track-based lepton isolation criterion is defined by calculating the quantity $I_R = \sum \pt^{\textrm{trk}}$ and $E_R = \sum E^{\textrm{clst}}$, where
the scalar sum includes all tracks and energy deposit (excluding the lepton candidate itself) within the cone defined by $\Delta R<0.2 (0.3)$ around the %axis
direction of the electron (muon). The electron (muon) candidates are required satisfy both $I_R/\pt^\ell < 0.2~(0.3)$ and $E_R/\pt^\ell < 0.2~(0.3)$.

Tight isolation working points are also applied in some channels to reduce fake and non-prompt lepton contributions based a trained isolation boosted decision tree \texttt{PromptLeptonVeto}(PLIV), which is to identify non-prompt light leptons using lifetime information associated with a track jet that matches the selected light lepton~\cite{ATLAS-CONF-2019-045}. These additional reconstructed charged particle tracks inside the jet can be
used to increase the precision of identifying the displaced decay vertex of heavy flavor (b, c) hadrons that produced non-prompt leptons.
The tight working points are used for high $\pt$ lepton.
The scale factors for the efficiencies of the tight \texttt{PLIV} working points are measured using the tag and probe method
with $Z\rightarrow l^+l^-$ events. These scale factors are also checked applicable to the electron or muon from the tau decay
using $Z\rightarrow\tau\tau\rightarrow e\mu$ samples, which is consistent with 1 at 2\% level. To be conservative, we assign
additional $\pm 2\%$ systematic uncertainty for the PLIV efficiency for the tau-lepton in the lepton+$\thad$ channels.
%Additionally, the $\Htautau$ search requires leptons to
%satisfy a calorimeter-based isolation criterion; a requirement is made on the sum of the transverse energy of
%topological clusters within the cone defined by $\Delta R<0.2$ around the lepton, after subtracting the contribution
%from the energy deposit of the lepton itself and correcting for pile-up effects, to be less than a $\pt$-dependent
%percentage of the lepton energy.
%The combined efficiency of track-based and calorimeter-based isolation requirements made by the $\Htautau$
%search is 90\% (99\%) at $\pt=25\;(60)~\gev$.

%a Gradient isolation working points are used for both electrons and muons. The working points are based on cut maps derived from the $Z\to l^+l^-$ Monte Carlo samples. The efficiency maps are simply derived from the sum of transverse energy of the clusters deposited in the calorimeter in the given cone. The efficiency for the leptons with a certain $\pt$ is $(0.1143*\pt[\GeV]+92.14)\%*(0.1143*\pt[\GeV]+92.14)\%$.
%draft 1 version
%Candidate jets are reconstructed with the anti-$k_t$ algorithm~\cite{Cacciari:2008gp,Cacciari:2005hq,Cacciari:2011ma} with a
%radius parameter $R=0.4$, using topological clusters~\cite{Aad:2016upy}
%built from energy deposits in the calorimeters calibrated to the electromagnetic scale.
Candidate jets are reconstructed using the anti-$k_t$ algorithm~\cite{Cacciari:2008gp,Cacciari:2005hq} with a
radius parameter $R=0.4$ applied to topological energy clusters~\cite{Aad:2016upy} and charged-particle tracks, processed using
a particle-flow algorithm~\cite{Aad:2017epj77}. %newref Eur. Phys. J. C 77 (2017) 466
%as implemented in the \fastjet\ package~\cite{Cacciari:2011ma}.
%Jet reconstruction in the calorimeter starts from topological clustering~\cite{Aad:2016upy} of individual calorimeter cells calibrated to the electromagnetic energy scale.
The reconstructed jets are then calibrated to the particle level by the application of a jet energy scale
derived from simulation and in situ corrections based on $\sqrt{s}=13~\tev$ data~\cite{Aaboud:2017jcu}. %(TBD: check PFlow jets description)
The calibrated jets used are required to have $\pt > 25~\gev$ and $|\eta| < 2.5$.
Jet four-momenta are corrected for pile-up effects using the jet-area method~\cite{Cacciari:2008gn}.

Quality criteria are imposed to reject events that contain any jets arising from non-collision sources
or detector noise~\cite{ATLAS-CONF-2015-029}.  To reduce the contamination due to jets originating from pile-up interactions,
additional requirements are imposed on the jet vertex tagger (JVT)~\cite{Aad:2015ina} output for jets with $\pt<60~\gev$ and $|\eta| < 2.4$.

Jets containing $b$-hadrons are identified ($b$-tagged) via the DL1r tagger~\cite{Aad:2019epj79,ATL-PHYS-PUB-2017-013} %newref
%\cite{Aad:2015ydr,ATL-PHYS-PUB-2016-012}
that uses multivariate techniques to combine information about the impact parameters of displaced tracks and the  topological properties
of secondary and tertiary decay vertices reconstructed within the jet. For each jet, a value for the multivariate $b$-tagging discriminant is
calculated. A jet is considered $b$-tagged if this value is above the threshold corresponding to
an average 70\% efficiency to tag a $b$-quark jet, with a light-jet\footnote{Light-jet refers to a jet originating from the hadronisation of a light quark ($u$, $d$, $s$) or a gluon.} rejection factor of about 385 and a charm-jet rejection factor of about 12, as determined for jets with
$\pt >20~\gev$ and $|\eta|<2.5$ in simulated $\ttbar$ events.

Hadronically decaying $\tau$-lepton ($\had$) candidates are reconstructed from energy clusters in the calorimeters and
associated inner-detector tracks~\cite{ATL-PHYS-PUB-2019-033}. %newref
%~\cite{ATL-PHYS-PUB-2015-045}.
Candidates are required to have either one or three associated tracks,
with a total charge of $\pm 1$. Candidates are required to have \pt 5 GeV higher than the trigger threshold, and $|\eta|<2.5$,
excluding the EM calorimeter's transition region.
A Recurrent Neural Network (RNN)~\cite{Graves:2012SCI}
%~\cite{Breiman:1984jka,Friedman:2002we,Freund:1997xna}
%A. Graves, Supervised Sequence Labelling with Recurrent Neural Networks, Studies in Computational Intelligence 385, Springer, 2012.
using calorimeter- and tracking-based variables is used to identify $\had$ candidates and reject jet backgrounds.
Three working points labelled loose, medium and tight are defined, and correspond to different $\had$ identification efficiency values, with the efficiency designed to be independent of $\pt$. The $\Htautau$ search uses the medium
working point for the nominal selection.
%, while the loose working point is used for background estimation.
The medium working point has a combined reconstruction and identification efficiency of 75\% (60\%) for one-prong (three-prong) $\had$
decays, and an expected rejection factor against light-jets of 35 (240)~\cite{ATL-PHYS-PUB-2019-033}.
Electrons that are reconstructed as one-prong $\had$ candidates are removed via a BDT trained to reject electrons.
Events with$\had$ candidate that is $b$-tagged are rejected.
%(TBD: check RNN description and update then numbers)

%Overlaps between candidate objects are removed sequentially, following this order: muons, electrons,
%$\tauhad$ (only for the $\Htautau$ search), and jets. In the $\Hbb$ search, firstly, electron candidates that lie
%within $\Delta R = 0.01$ of a muon candidate are removed to suppress contributions from muon bremsstrahlung.
%Overlaps between electron and jet candidates are resolved next, and finally, overlaps between remaining jet candidates
%and muon candidates are removed. Clusters from identified electrons are not excluded during jet reconstruction.
%In order to avoid double-counting of electrons as jets, the closest jet whose axis is within ${\Delta}R = 0.2$ of an electron
%is discarded. If the electron is within ${\Delta}R = 0.4$ of the axis of any jet after this initial removal, the jet is retained and  the electron is removed.
%The overlap removal procedure between the remaining jet candidates and muon candidates is designed to remove those muons
%that are likely to have arisen in the decay chain of hadrons and to retain the overlapping jet instead.
%Jets and muons may also appear in close proximity when the jet results from high-$\pt$ muon bremsstrahlung,
%and in such cases the jet should be removed and the muon retained. Such jets are characterised by having very
%few matching inner-detector tracks. Selected muons that satisfy $\Delta R(\mu,{\textrm{jet}}) < 0.04+10~\gev/\pt^\mu$ are rejected
%if the jet has at least three tracks originating from the primary vertex; otherwise the jet is removed and the muon is kept.
%In the $\Htautau$ search, a fixed cone size of $\Delta R=0.2$ is used to determine the overlap between
%candidate objects, and only the highest-$\pt$ (leading) or the two leading $\tauhad$ candidates (depending on the
%analysis channel, see Section~\ref{sec:data_presel}) are considered to resolve their overlap with jets.

Overlaps between reconstructed objects are removed sequentially. Firstly, electron candidates that lie
within $\Delta R = 0.01$ of a muon candidate are removed to suppress contributions from muon bremsstrahlung.
Overlaps between electron and jet candidates are resolved next, and finally, overlaps between remaining jet candidates
and muon candidates are removed. Energy clusters from identified electrons are not excluded during jet reconstruction.
In order to avoid double-counting of electrons as jets, the closest jet whose axis is within ${\Delta}R = 0.2$ of an electron
is discarded if the jet is not $b$-tagged, otherwise the electron is removed.
If the electron is within ${\Delta}R = 0.4$ of the axis of any jet after this initial removal, the jet is retained and  the electron is removed.
The overlap removal procedure between the remaining jet candidates and muon candidates is designed to remove those muons
that are likely to have arisen in the decay of hadrons and to retain the overlapping jet instead.
Jets and muons may also appear in close proximity when the jet results from high-$\pt$ muon bremsstrahlung,
and in such cases the jet should be removed and the muon retained. Such jets are characterised by having very
few matching inner-detector tracks. Selected muons that satisfy $\Delta R(\mu,{\textrm{jet}}) < 0.2$ are rejected
if the jet is either $b$-tagged or has at least three tracks originating from the primary vertex; otherwise the jet is removed and the muon is kept.
The $\tauhad$ within a $\Delta R=0.2$ cone of an electron or muon are removed.
In order to avoid double-counting of $\tauhad$ as jets, the closest jet whose axis is
within ${\Delta}R = 0.2$ of a $\tauhad$ is discarded if the jet is not $b$-tagged, otherwise the $\tauhad$ is removed.
% TEXT FROM BOYANG. NEED TO RECHECK WHAT WE HAVE ABOVE IS ACCURATE
%In the $\Htautau$ search, the objects are removed with the following sequence: if two electrons have overlapping second-layer cluster, or shared tracks, the electron with lower $\pt$ is removed; $\tauhad$ within a $\Delta R=0.2$ cone of an electron or muon are removed; if a muon sharing an ID track with an electron and the muon is calo-tagged, the muon is removed, otherwise the electron is removed; jets within a $\Delta R=0.2$ cone of an electron are removed; electrons within a $\Delta R=0.4$ cone of a jet are removed; when a muon ID track is ghost associated to a jet or within a $\Delta R=0.2$ cone of a jet, the jet is removed if it has less than 3 tracks with $\pt>500$ MeV or has a relative small $\pt$ ($\pt^{\mu}>0.5\pt^{\text{jet}} \text{ and } \pt^{\mu}>0.7[\text{the scalar sum of the } \pt \text{'s of the jet tracks with } \pt>500$ MeV]); muons within a $\Delta R=0.4$ cone of a jet are removed; jets within a $\Delta R=0.2$ cone of the leading $\tauhad$ ($\lephad$), or with the two leading $\tauhad$'s ($\hadhad$), are removed.

%firstly, the electron within a ${\Delta}R = 0.2$ cone of muons are excluded, then the $\tau_{had}$'s or jets within a ${\Delta}R = 0.2$ cone of an electron or muon are excluded. Finally, the jets within a ${\Delta}R = 0.2$ cone of the leading $\had$ ($\lephad$), or with the two leading $\had$’s ($\hadhad$) are excluded.

The missing transverse momentum $\mpt$ (with magnitude $\met$) is defined as the negative vector sum of the
$\pt$ of all selected and calibrated objects in the event, including a term to account for momentum from soft particles
in the event which are not associated with any of the selected objects.
This soft term is calculated from inner-detector tracks matched to the selected primary vertex to make it more resilient to
contamination from pile-up interactions~\cite{Aaboud:2018tkc}.


\section{Hadron propagation.}

\hspace{1.0em}
So far the hadrons are transported along the straight line paths (cascade
approach), i. e. particle coordinates ${\bf r}$ for time "shift"
between $t_0$ and
$t_1$ are updated according to
\begin{equation}
\label{HP1} {\bf r}(t_1) = {\bf r}(t_0) + \frac{ {\bf p}(t_0)}{E(t_0)},
\end{equation}
where ${\bf p}$ and $E$ denote particles momenta and energies.
The intranuclear nucleons are assumed to be "frozen" before collision,
i. e.  they are allowed to move only with the c. m. nucleus velocity.
However, their Fermi momenta are taken into account to find interaction
cross sections or characteristics of secondary particles produced in the
course of an intranuclear collision.


\hypertarget{performance-metrics-for-parallel-systems}{%
\section{Performance Metrics for Parallel
Systems}\label{performance-metrics-for-parallel-systems}}

\hypertarget{analytical-modeling}{%
\subsection{Analytical Modeling}\label{analytical-modeling}}

\begin{itemize}
\tightlist
\item
  Sequential Runtime

  \begin{itemize}
  \tightlist
  \item
    A sequential algorithm is evaluated by its runtime
  \end{itemize}
\item
  Parallel Runtime

  \begin{itemize}
  \tightlist
  \item
    The parallel runtime of a program depends on

    \begin{itemize}
    \tightlist
    \item
      the input size n,
    \item
      the number of processors p,
    \item
      and the communication parameters of the machine.
    \end{itemize}
  \item
    An algorithm must therefore be analyzed in the context of the
    underlying platform.
  \end{itemize}
\item
  Parallel System

  \begin{itemize}
  \tightlist
  \item
    A parallel system is a combination of a parallel algorithm and an
    underlying parallel platform.
  \end{itemize}
\item
  Wall-clock time

  \begin{itemize}
  \tightlist
  \item
    the time from the start of the first processor to the end of the
    last processor in a parallel ensemble
  \item
    in other words: the operation time of the processor which needs the
    most time
  \end{itemize}
\end{itemize}

\hypertarget{metrics}{%
\subsubsection{Metrics}\label{metrics}}

\begin{itemize}
\tightlist
\item
  $T_s(n)$: Zeitkomplexität des besten sequentiellen Vergleichalgorithmus
\item
  $T_p(p,n)$: Zeitkomplexität des besten parallelen Algorithmus
\item
  n: eindimensionale Inputgrösse
\item
  $p$: Anzahl ``Prozessoren'' (evtl. auch Threads, Cores, etc.)
\item
  Total Parallel Overhead $T_O = pT_P - T_S$
\item
  Speedup $S = T_S / T_P$
\item
  Speedup per processor = Efficiency $E = S / p = T_S / pT_P$
\end{itemize}

\hypertarget{speedup}{%
\subsubsection{Speedup}\label{speedup}}

\begin{itemize}
\tightlist
\item
  The lower bound of the speedup can be 0 (for parallel programs that
  never terminate)
\item
  The upper bound of the speedup should in theory be p, but in practice
  it is often lower than p.
\end{itemize}

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{figures/metricExample.png}
\caption{Metric Example}
\end{figure}

\hypertarget{big-o-landau-notation}{%
\subsubsection{Big-O / Landau Notation}\label{big-o-landau-notation}}

\begin{itemize}
\tightlist
\item
  $\mathcal{O}$ (Big O): Obergrenze, maximal so viele Schritte
\item
  $\Omega$ (Omega): Untergrenze, mindestens so viele Schritte
\item
  $\Theta$ (Theta): Obergrenze = Untergrenze, genau so viel
\end{itemize}

\hypertarget{cost-of-a-parallel-system}{%
\subsection{Cost of a Parallel System}\label{cost-of-a-parallel-system}}

\begin{itemize}
\tightlist
\item
  Cost (amount of total work)

  \begin{itemize}
  \tightlist
  \item
    is the product of parallel runtime and the number of processing
    elements used
  \item
    Cost $= p * T_P$
  \end{itemize}
\item
  Cost-Optimal System

  \begin{itemize}
  \tightlist
  \item
    a parallel system is said to be cost-optimal if the cost of solving
    a problem on a parallel computer is asymptotically identical to
    serial cost since $E = T_S / pT_P$, for cost-optimal systems: $E = \mathcal{O}(1)$
  \end{itemize}
\end{itemize}

\hypertarget{impact-of-non-cost-optimality}{%
\subsection{Impact of
Non-Cost-Optimality}\label{impact-of-non-cost-optimality}}

\begin{itemize}
\tightlist
\item
  The efficiency is calculated from the speedup over the number of
  processors (S/P)
\item
  Increasing the processors doesn't always solve the problem. Even if
  the speedup is increasing, the efficiency can shrink
\item
  speedup goes down as the problem size n is increased for a given p
\item
  efficiency doesn't depend on p, but goes down as the problem size is
  increased
\end{itemize}

\hypertarget{effect-of-granularity-on-performance}{%
\subsection{Effect of Granularity on
Performance}\label{effect-of-granularity-on-performance}}

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{figures/granularityOnPerformance.png}
\caption{Effect of Granularity on Performance}
\end{figure}

\begin{itemize}
\tightlist
\item
  Scaling-Down a parallel system is using fewer than the maximum
  possible number of processing elements to execute a parallel algorithm
\item
  This often improves the efficiency
\item
  One way of scaling down could be

  \begin{itemize}
  \tightlist
  \item
    think of each processor in the original case as a virtual processor
    and to assign virtual processors equally to scaled down processors
  \item
    since the number of processing elements decreases by a factor of
    $n/p$, the computation at each processing element increases by a
    factor of $n/p$
  \end{itemize}
\end{itemize}

\clearpage