% JuliaCon proceedings template
\documentclass{juliacon}
\setcounter{page}{1}
\usepackage{graphicx}

\hypersetup{colorlinks=true}
\begin{document}

\input{header}

\maketitle

\begin{abstract}

Flux \cite{innes:2018} is a machine learning framework, written using the numerical computing
language Julia\cite{bezanson2017julia}. The framework makes writing layers as simple as
writing mathematical formulae, and it's advanced AD,
Zygote \cite{DBLP:journals/corr/abs-1810-07951} , applies automatic differentiation (AD) to
calculate derivatives and train the model. It makes heavy use of Julia's language and
compiler features to carry out code analysis and make optimisations.  For example, Julia's
GPU compilation support \cite{besard:2017} can be used to JIT-compile custom GPU kernels for
model layers \cite{CuArrays.jl}. Flux also supports a number of a hardware options, from CPUs,
GPUs and even TPUs via XLA.jl, that compiles Julia code to XLA: an advanced compiler for
linear algebra that is capable of greatly optimizing speed and memory usage in large
deep learning models.

ONNX.jl is an Open Neural Network Exchange backend for the Flux.jl deep learning framework.
ONNX.jl supports directly importing high quality ONNX standard models into Flux, thus saving
time and reducing the need for additional computation resources. This paper aims at introducing
ONNX.jl and explaining how it fits into the bigger picture: How we can use the Julia Language,
specifically Flux.jl and ONNX.jl as a starting for high quality transfer learning of large deep
learning models.

\end{abstract}

\section{Introduction}

The Julia language was introduced to solve the two language problem: In simple words, languages that
are simple to write (high-level) are very slow but those which are difficult to use (low-level) are
way faster. This is because most of the high-level languages weren't written to process a large
amount of data. Thus engineers, researchers and developers have a hard time developing a lot of
high performance languages. At the moment, the common protocol is to write the core of the software
in a low-level language (C/C++/Fortran) and wrap it in a high-level language (Python). This results
in optimized performance and ease of use. The Julia language aims to make best of both worlds. It
provides a high level syntax but manages to perform as fast as C (sometimes even faster).

Flux.jl is a library for implementing machine learning models, written completely in the Julia
programming language. At the heart of Flux.jl lies Zygote.jl: A source-to-source automatic
differentiation (AD) library that makes complete use of the Julia language compiler to generate
backward pass during training phase of a neural network, with complete support for control flow,
recursion, closures and data structures.
Implementing models in Flux.jl is as simple as writing regular Julia code. Implementing models
is as simple as writing the formulae for those, and Zygote.jl will compute the derivatives
seamlessly.

Flux.jl also provides support for other hardware options using external packages such as CuArrays.jl
and CLArrays.jl. CuArrays is written completely in Julia, making implementing GPU kernels very simple.
Making a model run on GPU can be done in a hassle-free manner: It is as simple as calling a few functions
to transfer data to GPU. Flux.jl also has support for running models on Google's Tensor Processing
Unit (TPU). TPUs help in very fast linear algebra computation. Running Flux models on TPUs is possible
through XLA.jl that compiles Julia code to XLA.

The FluxML ecosystem provides a number of supporting packages that provide additional functionalities , some of them being (apart from the aforementioned Flux.jl, Zygote.jl and XLA.jl);

\begin{itemize}
    \item ONNX.jl : Open Neural Network eXchange backend for Flux.jl
    \item Metalhead.jl \cite{Metalhead.jl}: Simple plug and play pretrained Flux.jl computer vision models.
    \item Torch.jl \cite{Torch.jl}: This package aims at exposing Torch.tensor types in Julia.
    \item IRTools.jl \cite{IRTools.jl} : Provides an IR format that is easy to manipulate.
    \item FluxJS.jl \cite{FluxJS.jl} : Runs Flux models in the browser, via tensorflow.js
    \item model-zoo \cite{model-zoo}: Collection of implementation of various Flux deep learning models.
\end{itemize}

\section{Open Neural Network eXchange (ONNX)}
Open Neural Network Exchange (ONNX) \cite{bai2019} is an open ecosystem that empowers AI developers to
choose the right tools as their project evolves. ONNX provides an open source format for AI
models, both deep learning and traditional machine learning. ONNX defines the computation graph for a deep learning model along with various operators used in the model.
It provides a set of specifications to convert a model to a basic ONNX format, and
another set of specifications to get the model back from this ONNX form.   At a high
level, ONNX is designed to allow framework inter-operability. There are many excellent
machine learning libraries in various languages : PyTorch \cite{bai2019} , TensorFlow \cite{abadi2016tensorflow} , MXNet \cite{chen2015mxnet} , and Caffe \cite{jia2014caffe} are
just a few that have become very popular in recent years, but there are many others as well.
Machine learning models can be converted to a serialized ONNX format which can then be run
on a number devices. ONNX Runtime is an inference engine written in C++ framework used to
deploy ONNX format models into production. It works on diverse hardware and support both
deep learning as well as traditional machine learning models.

\subsection{Where does ONNX come in?}
ONNX is a format for representing deep learning models, which can be further run on numerous devices
without worrying much about the implementation. This helps researchers, developers and engineer to
focus on the problem in hand without worrying much about the peripherals, such as the framework
to use, the ability to run a model trained using this particular framework on specialized hardware.
ONNX is is usable anywhere from small mobile devices to large server farms, across chipsets and vendors,
and with extensive runtimes and tools support. ONNX reduces the friction of moving trained AI
models among your favorite tools and frameworks and platforms. A simple example of how ONNX is ideal
for ML is the case when large deep learning models need to be deployed.     Consider
the simple case of deploying a Deep Learning model to an iOS application. This particular model can
be implemented in any framework : TensorFlow, PyTorch, MXNet just to name a few. However, iOS
applications expect to use CoreML inside the application. Up until now, developers have been porting
large models to different frameworks,
which is a waste of time and energy, better spent somewhere else. This is also retraining the entire model from scratch, which isn't efficient. This makes the entire process cumbersome and impractical. ONNX exists to solve this very problem : By connecting the common dots from
different frameworks, ONNX makes it possible to express a model of type A to type B, thus saving time and the need to train the model again.

\section{ONNX backend in Julia}

ONNX.jl is an ONNX backend written in Julia for the Flux.jl machine learning framework. It can be
used for high-quality inference of pretrained ONNX format machine learning models. At the heart of it, ONNX.jl solves a compiler problem by dealing with intermediate code representations to generate readable graphs. While doing this, ONNX operators are mapped to corresponding Flux layers, thus tracing out the model's computation graph at the end. This graph can then be travelled to generate the Julia code for the model.

The first step towards reading any ONNX model in Julia is to have access to all the data structures in the model,
that essentially hold all the required information to load the model. This information includes everything that
is used to completely define the model: Hyper-parameters and parameters in the model. So in the case of a simple
Convolutional Neural Network, this may contain information such as the number of layers, number of units in each
layer, strides, padding, kernel size, number of filters as hyper-parameters and the trained values associated
with each layer as parameters.

Since ONNX models as protobuf serialized, we need a way to read this serialized data into specific data
structures. ProtoBuf.jl is a Julia implementation of protocol buffers that solves this very issue (covered in
section 3.1). It is used to read the ONNX model into the generated Julia data structures. Once we have the
entire model present as a complex Julia structure, we need to read through this structure and map ONNX operators
to corresponding Flux layers/operations (covered in section 3.2). At the same time, model weights or
parameters are separately stored and saved externally as BSON serialized file. Once the model has been loaded,
we end up with two files: `model.jl`: The Julia code for the machine learning model and `weights.bson`: The
weights associated with the layers defined in the `model.jl` file (section 3.3). In the further sections we'll walk through the internals of these individual processes

\subsection{ProtoBuf.jl}

ProtoBuf.jl \cite{Protobuf.jl} is a Julia implementation of Protocol Buffers. It provides a way to read and write data to and
from Julia types from I/O streams. What ProtoBuf.jl gives from onnx.proto3 is the Julian definition of various
data structures that in themselves have all the required attributes to load any ONNX serialized model. As an
example, as simple message defined in onnx.proto3 as:

\begin{lstlisting}[language=julia]
message NodeProto {
  repeated string input = 1;    // namespace Value
  repeated string output = 2;   // namespace Value

  // An optional identifier for this node in a graph.
  // This field MAY be absent in ths version of the
  // IR.
  string name = 3;     // namespace Node

  // The symbolic identifier of the Operator to
  // execute.
  string op_type = 4;  // namespace Operator
  string domain = 7;   // namespace Domain

  // Additional named attributes.
  repeated AttributeProto attribute = 5;

  string doc_string = 6;
}
\end{lstlisting}{}
\textit{(Code snippet taken from onnx/onnx.proto3)}

Results in the corresponding Julia definition of the model as :

\begin{lstlisting}[language=julia]
mutable struct NodeProto <: ProtoType
    input::Vector{AbstractString}
    output::Vector{AbstractString}
    name::AbstractString
    op_type::AbstractString
    domain::AbstractString
    attribute::Vector{AttributeProto}
    doc_string::AbstractString
    NodeProto(; kwargs...) = (o=new(); fillunset(o); isempty(kwargs) || ProtoBuf._protobuild(o, kwargs); o)
end
\end{lstlisting}

Since ONNX tries to inherit properties from diverse frameworks, ONNX serialized models can be large and complicated. While there are a number of complex generated data structures, three of those are essential towards
understanding how data is stored internally:

\begin{itemize}
    \item ModelProto: A very high-level struct that holds all the information. ONNX models are read directly into
            this structure.
    \item GraphProto: This structure captures the entire computation graph of the model.
    \item NodeProto and TensorProto: Information regarding individual nodes in the graph (inputs, outputs and finer attributes) and weights associated with the nodes.
\end{itemize}

\subsection{ModelProto}

ModelProto structure is the structure that holds all the information needed to load a model. Internally, it holds
data such as the version information, model version, docstring, producer details and most importantly: the
computation graph. \newpage

\begin{lstlisting}[language=julia]
mutable struct ModelProto <: ProtoType
    ir_version::Int64
    opset_import::Vector{OperatorSetIdProto}
    producer_name::AbstractString
    producer_version::AbstractString
    domain::AbstractString
    model_version::Int64
    doc_string::AbstractString
    graph::GraphProto
    metadata_props::Vector{StringStringEntryProto}
end #mutable struct ModelProto
\end{lstlisting}

An ONNX model, once read using ProtoBuf.jl is loaded into this ModelProto object before extracting the graph
details. Naturally, at the heart of this is the \textit{graph::GraphProto} attribute that stores the computation
graph of the model.

\subsection{GraphProto}

The GraphProto structure stores information about particular nodes in the graph. This includes
the node metadata, name, input, output and the pre-trained parameters in the \textit{initializer} attribute.

\begin{lstlisting}[language=julia]
mutable struct GraphProto <: ProtoType
    node::Vector{NodeProto}
    name::AbstractString
    initializer::Vector{TensorProto}
    doc_string::AbstractString
    input::Vector{ValueInfoProto}
    output::Vector{ValueInfoProto}
    value_info::Vector{ValueInfoProto}
end #mutable struct GraphProto
\end{lstlisting}

\subsection{TensorProto}
This is the main structure that holds the raw model parameters. For example, in comvolutional layers, the weights
associated with the kernel as available as dense vectors in the \textit{raw\_data} attribute. During graph
traversal, these weights are extracted and reshaped according the shape that is available as a node attribute.

\begin{lstlisting}[language=julia]
mutable struct TensorProto <: ProtoType
    dims::Vector{Int64}
    data_type::Int32
    segment::TensorProto_Segment
    float_data::Vector{Float32}
    int32_data::Vector{Int32}
    string_data::Vector{Array{UInt8,1}}
    int64_data::Vector{Int64}
    name::AbstractString
    doc_string::AbstractString
    raw_data::Array{UInt8,1}
    double_data::Vector{Float64}
    uint64_data::Vector{UInt64}
end #mutable struct TensorProto
\end{lstlisting}

This is most of the information needed to build the model. In the next section, we discuss how we use DataFlow.jl to
travel this graph and extract model parameters and other relevant information.

\subsection{Graph operations via DataFlow.jl}

Once we have the entire model data present as a ModelProto object, the next step is to travel the computation
graph and capture all the operation being done in the graph while mapping those simultaneously to the
corresponding Flux operators.

DataFlow.jl \cite{DataFlow.jl} is a code intermediate representation format, representing Julia code as an expression graph. It provides functions for graph re-structuring , even on cyclic graphs. Graphs can then also be used to generate Julia expression. It can be efficiently used to traverse our \textit{ModelProto.graph} object. However,
during this traversal we want to map ONNX operators to Flux layers and functions. In DataFlow.jl, this becomes
equivalent to creating a new vertex for the required operator and calling in with appropriate Flux functions
, which are inferred from the ONNX operator itself. As an example, let's consider the simple case of the
\textit{BatchNorm} operator in ONNX. Relu is a commonly used activation function in neural networks that can be
expressed as :

\[ relu(x) = max(x, 0) \]

It basically just turns negative neurons off (sets them to 0) and bypasses positive neurons. The Relu operator
in ONNX performs the same operation on an entire vector elementwise. It takes in a single parameter: The input
vector and returns a single value: the result of applying relu on the input. Using DataFlow.jl, this
operator is mapped as :
\newline \newline
\begin{lstlisting}[language=julia]
ops[:Relu] = function (params, x)
  vcall(broadcast, :relu, x)
end
\end{lstlisting}

The definition of \textit{relu} here is defined in Flux: Once the model is written to an external model.jl file,
we can include the file directly after importing Flux and all definitions should be ready for use.

Other complex layers such as Convolution have more complicated implementations, but the essence remains the same,
to collect all inputs and call them with the corresponding Flux function. During this process, model weights
also computed and stored in a dictionary, mapping layer name to the parameters. At the end of the graph traversal,
we have both the required values: the DataFlow graph containing Flux layers and the model weights corresponding to
each of these layers. The DataFlow graph can converted to Julia code using \textit{DataFlow.syntax} that also
assigns variable names as and when needed. This Julia code is then written to an external \textit{model.jl} file.
For saving the weights, we use the BSON.jl package. BSON stands for Binary JSON, a binary encoded serialization
of JSON objects. BSON.jl \cite{BSON.jl} can be used to store and load such structures, our dictionary containing the model weights being one of them.
\subsection{Interface and Design:}
At a top level, ONNX.jl provides a minimal interface for the user; it is just a tool for loading ONNX format
models. Once the model and weight file has been successfully generated, ONNX.jl provides no further functionality.
Users can then treat the resultant model as any other Flux model.

\begin{lstlisting}[language=Julia]
using ONNX, Flux
ONNX.load_model("path_to_onnx_file")
weights = ONNX.load_weights("weights.bson")
model = include("model.jl")
\end{lstlisting}{}

\textit{ONNX.load\_model} here generates the required model and weights file. Internally, it carries out all the
above mentioned graph operations. \textit{model} above can be treated as any other Flux model.

The significant advantage the ONNX.jl provides is that is treats a compiler problem as a graph problem. It
generates the Flux code for the model, which makes it very easy and intuitive to use the same model for
further applications, such as fine-tuning or even replacing existing layers for some other use case. This
is ideal in the case of applications such as neural style transfer, where it is very common to use a pre-trained
network and modify it a bit as a starting point. The generated code can also be helpful for finer debugging of
the model.

Overall, the entire process from the ONNX serialized file to generation of model and weight file can be
summarized as:

\includegraphics[width=8.8cm]{onnx-3.png}

\begin{center}
    \textit{Fig. 1: Flow diagram of ONNX.jl}
\end{center}

Additionally, ONNX.jl also provides helper functions for inspecting the model before loading it. \textit{ONNX.layers} reads an ONNX file and returns a list of all the layers in the model. With the growing
interest around more complicated and deep models, it is possible that an ONNX model might have layers that Flux
itself doesn't support. For handling these, ONNX.jl leaves a \textit{hook} for the users to implement
additional functionality. A \textit{hook} is a function that doesn't have an existing implementation: one
would have to write an implementation for it themselves. However any operator that also has a corresponding
implementation in Flux is completely recognized by ONNX.jl at the moment.

\section{Usage Scenarios}
The ONNX format and ONNX.jl can be used for transfer learning in Flux, where we store knowledge while training a model and
use this knowledge for some other task. The idea is that rather than random initialization of parameters for training a neural network, it's better to take an already trained model, since
it leads to faster convergence. In transfer learning, we take a pretrained model and train it on another
dataset, which might also have a different class distribution. Fine tuning is an approach to transfer learning
where we train on a subset of training data with a smaller learning rate. Transfer learning learning has shown
tremendous results in image classification, object detection, simulations, sentiment and NLP based classification
in recent past.
This is also pretty common when talking about tasks such as neural style transfer where we want to change the
style of an image in accordance with the style of another image. Generative Adversarial Networks (GANs) have shown
to deliver high quality results when trained on top of a pre-trained model. StyleGAN \cite{DBLP:journals/corr/abs-1812-04948} , for example, can use a
pre-trained model to train a custom model to deliver high quality super-resolution results.

\section{Related Work}

In recent times several projects have come up that solve similar issue. One of the most notable project is
TensorFlow's mlir (Multi-Level Intermediate Representation) \cite{lattner2020mlir} . mlir is an evolution of LLVM
\cite{LLVM:CGO04} that defines a
common Intermediate Representation (IR) format, which can be used to represent any DataFlow graph. This common
format unifies machine learning models in TensorFlow or other frameworks.
Other noteworthy approaches in this direction are PFA and NNVM. PFA \cite{10.1145/2939672.2939731} or Portable Format for Analytics is a common
language that aims as easing the transition from development to production. It can be expressed within
the common JSON format and has functionalities such as control structures, loops and user-defined functions.

NNVM\cite{nnvm} is an end-to-end compiler for AI frameworks. It aims at solving the challenges posed by using different
diverse machine learning frameworks. It consist if two major components: NNVM (Neural Network Virtual Machine)
and TVM (Tensor Virtual Machine) \cite{article} . NNVM defines a common computational graph intermediate representation
format and TVM implements the operators used in these computation graphs while optimizing them for the backend hardware.

\section{Future Work}

As ONNX.jl becomes the beginning point for various researchers interested in using Julia for their research,
it is important to note that it also has certain shortcomings. The most significant is that a model can't
be completely loaded unless there's an equivalent implementation of the operator in Flux.jl. An example of this
is Grouped Convolutions. These variants of Convolutional layers were used in AlexNet \cite{NIPS2012_4824} and showed amazing
results. However, since Flux doesn't support these at the moment, the users will need to have an implementation
ready if they choose to import an ONNX model with this particular layer into Flux using ONNX.jl. On the plus side,
a lot of the most commonly used layers are available in Flux and can be readily used. Another thing to note is
that to run ONNX.jl generated code in some other hardware, one might need to do a little restructuring. The model
should work directly on the CPU.

Another challenge moving forward is that we need to constantly update ONNX.jl to support Flux's latest API
changes. This also applies for the other way round: As ONNX operators are updated, we would have to update these
corresponding implementations in ONNX.jl to support the newer models using these updated specifications. Over
the long run, we'd need to constantly keep an eye out for such changes and adapt ONNX.jl to those. Moreover,
there are subtle differences in the way Flux implements operators as compared to other frameworks. As an example,
consider the simple \textit{AveragePool} layer. The way Flux implements this is by padding the input tensor
appropriately and then performing the pooling operation. However, Keras-tensorflow does this by pooling and then
padding. This leads to subtle changes along the edges of the tensor in the output. Such differences occur due to
the way most frameworks deal with such layers, and the only way to avoid this is to check for such
discrepancies.

In recent past, DataFlow.jl has been superseded by another intermediate representation format tool: IRTools.jl.
It provides the ability to work with both lowered and typed Julia code and can be used together with
metaprogramming tools such as Cassette.jl.

There has also been some talk about splitting ONNX.jl into two packages: The first one would do the code
generation and DataFlow related functions while the other would be solely responsible for implementation of the
ONNX operators. This would be greater control and ease while implementing layers or debugging a loaded model. This
should also make implementation pretty straight-forward wherever missing.
For the moment, all this continues to be done by a single package.

\section{Conclusion}

Developing ONNX.jl has been tremendous learning experience. From studying about Intermediate Representation formats for deep
learning models with millions of parameters to loading them in just a couple of lines of code, ONNX.jl has
made it very easy and straight-forward to use a high quality trained model as a starting point for many projects.
Once such example I'd like to point out is DenseNet-121 model. This is deep convolutional network that has
multiple Convolutional, Dense and Pooling blocks. Naturally, implementing this in any framework is going to be
a challenging task. However, thanks to ONNX, we can now use an earlier implementation to import this model into
any other framework. Importing this model (train in Caffe2) via ONNX.jl into Flux can be done in ~3 lines of code.

I was also able to load several large computer vision models loaded from ONNX format at the time of actively
developing the package. Most of these have been added to FluxML/Metalhead.jl for a direct plug-and-play use. These
included:

\begin{itemize}
    \item SqueezeNet \cite{i2016squeezenet}
    \item DenseNet 121  \cite{huang2016densely}
    \item ResNet 50     \cite{he2015deep}
    \item GoogleNet     \cite{szegedy2014going}
\end{itemize}

ONNX.jl serves as a entry point for people looking to use Flux for their research, but want quick results. It
combines the power of the Julia language, the elegance of Flux and the availability of a vast number of pre-trained models. This enables researchers to spend time focusing on the real issues, rather than model portability. \newpage

\input{bib.tex}

\end{document}

% Inspired by the International Journal of Computer Applications template


% !TeX root = ./paper.tex
\documentclass[modern]{aastex62}

% Load the corTeX style definitions
\input{cortex}

% Bibliography stuff
\bibliographystyle{aasjournal}

% Begin!
\begin{document}

% Title
%\title{Inferring a time dependent map of Io's surface from occultations and phase curves.}
\title{Occultation mapping of Io's surface in the near-infrared II: Inferring time variable maps using latent Nonnegative Matrix Factorization}

% Author list
\author{Fran Bartoli\'c}
\email{fbartolic@flatironinstitute.org}
\affil{Center~for~Computational~Astrophysics, Flatiron~Institute, New~York, NY}
\affil{Centre for Exoplanet Science, SUPA, School of Physics and Astronomy, University of St. Andrews, St. Andrews, UK}
\author{Rodrigo Luger}
\author{Daniel Foreman-Mackey}
\affil{Center~for~Computational~Astrophysics, Flatiron~Institute, New~York, NY}
%

\begin{abstract}

\end{abstract}

%
\section{Introduction}
\section{Data}
\label{sec:data}

\begin{figure}[h!]
    \begin{centering}
    \includegraphics[width=0.5\linewidth]{figures/irtf_max_flux.pdf}
    \oscaption{irtf_dataset}{%
        Maximum flux for each curve taken using NASA's IRTF telescope during an
        occultation of Io by Jupiter.
       \label{fig:irtf_max_flux}
    }
    \end{centering}
\end{figure}

The dataset we use to infer maps of Io's surface consists of 112 observations of Io in the near infrared taken from 1996 until 2018 using various instruments at NASA's IRTF observatory.
Figure~\ref{fig:irtf_max_flux} shows the maximum flux in $\mathrm{GW}/\mu \mathrm{m}/\mathrm{sr}$ for each light curve plotted as a function of time which approximately corresponds to the disc integrated flux of Io at the beginning of ingress or the end of egress prior to it being occulted by Jupiter.
Two features of this plot are apparent.
First although the observations span decades the cadence is non-uniform with only a few observations taken between 2008 and 2016.
Second, the baseline brightness is varying stochastically by a large amount and there are several notable events of increased volcanic activity.
All of the observations were taken while Io was \emph{in eclipse}, meaning that it was in Jupiter's shadow and all of the observed emission from the surface is due to thermal radiation.
Observations of Io \emph{in sunlight} on the other hand probe both the thermal (volcanic) emission and the surface albedo variations in the near infrared.

\begin{figure}[h!]
    \begin{centering}
    \includegraphics[width=0.5\linewidth]{figures/irtf_sample_lightcurves.pdf}
    \oscaption{irtf_dataset}{%
        A selection of sample light curves taken during occultations of Io by Jupiter in our dataset.
        The step-like morphology of the light curves is due to bright volcanoes on Io's surface coming in or out of view during an occultation.
        These light curves thus visibly encode information about the features on the surface.
        \label{fig:irtf_sample_lightcurves}
    }
    \end{centering}
\end{figure}

In Figure~\ref{fig:irtf_sample_lightcurves} we display a random subset of all the light curves
in the dataset.
All occultations last for $\sim4$ min and the cadence for each light varies but it on the order of a second.
It follows that over the course of a single exposure Jupiter's limb move by about 15km which provides a lower bound for the size of the features on the surface that we can reliably estimate.
The shapes of light curves strongly deviate from the smooth variability one would expect assuming a homogeneous distribution of thermal emission on the surface.
Especially prominent are light curves with clear step-like features which are present when bright spots come in or out of view during the course of an occultation.
The fact that these features are so clearly visible means that even individual light curves encode a wealth of information about the surface.

The photometric quality varies from year to year because multiple instruments were used over the years.
An additional issue is that estimated errorbars aren't provided.
As with all ground based photometry, the observations are influenced by atmospheric variability which induces correlated noise in the light curves.
Because of this the flux isn't always monotonically increasing or decreasing as one would expect.
All of these issues need to be accounted for in the final model.

\section{Nonnegative Matrix Factorization}
\label{sec:nmf}
In the case when we cannot assume a static map, the situation is considerably more complicated.
In principle, the surface map is different for each \emph{data point}.
Of course, fitting one map per data point is intractable so we might instead fit a single map per light curve, generated by Eq.~\ref{eq:linear_model_pix}.
Even then, we would need to fit on the order of 1k parameters!
This is both becuse the number of spherical coefficients scales as $(l+1)^2$ and because when fitting in pixel space we need many more pixels than spherical harmonics to ensure positivity.
Although fitting a model of such high dimensionality for a single IRTF light curve of high signal to noise ratio is tractable thanks to exact gradients provided by autodifferentiation in \textsf{starry}, we don't want to do that for a hundred or so separate light curves.
Such a model would be very difficult to fit and it would require strong regularization between successive maps in time.
It also wouldn't directly provide much physical insight into the volcanic activity, we would have to conduct a separate analysis on the inferred maps to constrain the time variability of the volcanoes.
Instead, we need an approach which reduces the dimensionality of the problem.
One way of accomplishing that is to expand the spherical harmonic coefficients (or pixels) in a Taylor series in time about a certain point.
This is the approach \cite{luger_tess_2019} took to model a map of Earth's albedo in reflected light from stray Earthshine in the aperture of the TESS space telescope.
This issue with this approach and similar series expansions such as the Fourier series is that with the very long baseline of IRTF data and the observations being scattered throughout the years sporadically, it is unclear which point should be the origin of the expansion.
More importantly, there likely doesn't exist a well defined global time-averaged map of Io so requiring too much smoothness between successive maps wouldn't work.
Another way of reducing the dimensionality of the problem would be a parametric approach where where we place a few spots on the sphere and then fit only for the locations and intensities of each spot.
This is assuming that we know a priori that the surface features are spot like and roughly how many we should expect.
While such a strong assumption might work for studying individual spots in the context of Io, it certainly isn't justified for exoplanets.
In Section~\ref{sec:inverse_problem} we do assume stronger priors on the locations of and shapes of surface features to better constrain the variability of individual volcanoes, but we still fit for pixels, at least in principle allowing the data to override our assumptions.

The approach approach opted for is to treat the problem of inferring a single map per light curve as a probabilistic matrix factorization problem.
The idea is to assume that a map for any given light curve can be expressed as a linear combination of $K$ "basis maps".
Consider $L$ lightcurves, the model for the $l$-th light curve is
\begin{equation}
    \mathbf{f}_l=\mathbf{A}_l\,\mathbf{P}^\dagger\,\mathbf{p}_l'
\end{equation}
We can stack the column vectors $\mathbf{p}_l$ into a matrix $\mathbf{Y}$ of shape $(N_p, L)$ where $N_p$ is the number of pixels for each light curve.
We then assume that the $\mathbf{Y}$ can be decomposed into a product of two matrices $\mathbf{B}$ and $\mathbf{Q}$ as
\begin{equation}
    \mathbb{Y}=\mathbf{B}\,\mathbf{Q}
    \label{eq:nmf}
\end{equation}
where $\mathbf{B}$ has shape $(N_p, K)$ and $\mathbf{Q}$ has shape $(K, L)$.
The model for all light curves can be written as
\begin{equation}
    \mathbb{f}=\mathbb{A}\,\mathrm{vec}(\mathbf{\mathbb{Y}})
    \label{eq:model_all_lcs}
\end{equation}
Where $\mathbf{f}$ is a tall column vector consisting of predictions for all light curves stacked together, $\mathbb{A}$ is a block diagonal matrix with matrices $\mathbf{A}_l\,\mathbf{P}^\dagger$ on the diagonal for $l=1\dots, L$ and the $\mathrm{vec}$ operator stacks the columns of $\mathbb{Y}$ into a tall vector.
The model is \emph{bilinear} which means that is linear in the matrices $\mathbf{B}$ and $\mathbf{Q}$ separately.
The interpretation of Equation~\ref{eq:nmf} is simple, each of the $K$ columns of $\mathbf{B}$ represents pixels of a basis map and the columns of $\mathbf{Q}$ determine how those maps add together to produce a final map for the $l$-th light curve.
For this reason we call $\mathbf{B}$ the \emph{basis matrix} and the matrix $\mathbf{Q}$ the \emph{encoding matrix}.
Ideally, we want the basis maps to be physically meaningful and the coefficients encode the time variability.
A nice feature of this model is that there is no requirement that successive maps are smoothly varying between different light curves.

The matrix factorization problem as written in Equation~\ref{eq:nmf} is highly degenerate and practically intractable a probabilistic framework with a small dataset.
Since each of the columns of $\mathbf{B}$ are pixels representing emitted light or albedo, we can substantially reduce the ambiguity in the decomposition by requiring that both matrices are strictly positive.
This matrix factorization problem is known as \emph{Nonnegative Matrix Factorization} (NMF) \citep{paatero_positive_1994,lee_algorithms_2001} and it has a rich history across many different fields.
It is commonly used for decomposing physical signals in the spectral or the time domain.
Even though NMF is more tractable than unconstrained matrix factorization, it is still an NP hard problem \citep{vavasis_complexity_2009} and it requires additional constraints or priors to be tractable.
Simultaneously transforming $\mathbf{B}\leftarrow \mathbf{B}\,\mathbf{S}^{-1}$ and
$\mathbf{Q}\leftarrow \,\mathbf{S}\mathbf{Q}$ with a nonsingular matrix $\mathbf{S}$ doesn't change the value of the objective function.
The objective function is also invariant to a simultaneous permutation of the columns of $\mathbf{B}$ and rows of $\mathbf{Q}$ and rescaling either of the two matrices by a scalar.
Nevertheless, getting around these degeneracies in practice is possible, especially when we have a lot of prior knowledge on the problem.
For a recent review article with a detalied analysis of these degeneracies and common algorithms, see \cite{fu_nonnegative_2019}.

Broadly, there are two approaches to fitting NMF models, the constrained optimization approach and the probabilistic approach.
In the optimization approach we optimize $\mathbb{Y}$ in a least-squares sense under a set of constraints on either one or both of the matrices \citep[see][for recent examples from the astronomical literature]{acosta-pulido_new_2017, melchior_scarlet_2018,ren_non-negative_2018,ren_using_2020} with the goal of obtaining a point estimate for the two matrices.
The probabilistic approach introduces priors and the inference is usually done with variational inference in order to obtain an approximate posterior distribution over the matrices.
In this work we opt for the probabilistic approach because we care about the uncertainties on the map features a lot.

One difference between most applications of NMF in the literature and NMF within the context of inferring surface maps from 1D light curves is that
in the majority of the work in the literature \citep[except][]{kawahara_global_2020} the matrix $\mathbb{Y}$ is assumed to be directly observed  up to a simple noise term.
In our case the problem is considerably more challenging because we observe the light curves
$\mathbb{A}\,\mathrm{vec}(\mathbf{\mathbb{Y}})$ instead of $\mathbb{Y}$ directly, a process in which some information is lost because, depending on the geometry of the occultations, certain combinations of spherical harmonic coefficients will be in the nullspace.
\cite{kawahara_global_2020} are using NMF to model phase variations in directly imaged exoplanets with the goal of simultaneously inferring a surface map and a spectral decomposition of the map into several components, a case in which the
matrix which is to be decomposed is also unobserved.
Although we don't attempt to solve the problem of spectrally decomposing the maps, our approach is directly applicable to that problem as well. (TODO: should I add more details here?)
One could also imagine inferring a multi spectral component and time dependent map by solving two NMF problems simultaneously.
We leave this application for future work.

\subsection{The likelihood}
\label{ssec:likelihood}
Given Eq.~\ref{eq:nmf} we can write down the likelihood for our data and in Section~\ref{sec:inverse_problem} we dicuss in detail the priors which make the problems of inferring static and dynamic maps tractable.
Assuming a Gaussian noise process for the data with a dense covariance matrix, the (log) likelihood is
\begin{equation}
    \ln\mathcal{L}=-\frac{1}{2}\left[\mathbb{f}_\mathrm{obs}-(\mathbb{f} + \mathbb{b})\right]^{\top} \boldsymbol{\Sigma}^{-1}\left[\mathbb{f}_\mathrm{obs}-(\mathbb{f} + \mathbb{b})\right]
\end{equation}
where $\mathbb{f}_\mathrm{obs}$ is the vector of stacked observed light curves, $\boldsymbol{\Sigma}$ is the data covariance matrix and $\mathbb{b}$ is a fixed flux offset per light curve accounting for stray flux not attributed to Io (mostly due to Jupiter).
We model the data covariance with a Gaussian Process using the fast Celerite method\citep{foreman-mackey_fast_2017} as implemented in the \textsf{exoplanet} package (TODO: cite) to capute correlations imposed by the seeing plus an additional white noise term which is different for each light curve.
The IRTF data isn't provided with any uncertainties so we estimate the uncertainties by filtering the data with the Savitzky Golay filter as implemented in \textsf{SciPy} (TODO: CITE) and estimating the variance of the residuals.
We assume identical errors for all data points in a given light curve but allow for a rescaling factor when fitting the model.
In the next section, we add priors to this likelihood and fit the model on simulated data using optimization and Variational Inference (VI).

\section{The inverse problem}
\label{sec:inverse_problem}

\subsection{Fitting a dynamic map}
\label{ssec:dynamic_map}
% Priors for NMF. Optimization vs. bayesian approach to NMF. Results on simulated data. Comparison of results to matrix factorization without the positivity constraint and clever priors.

\section{Results}
\label{sec:results}

\subsection{Individual events}
\label{ssec:individual}
% Fits of individual events. In particular, those light curves that were analyzed in previous papers.

\subsection{The time-variable map}
\label{ssec:time_variable_map}
%

\subsection{Variability of known hotspots}
\label{ssec:variability_hotspots}
% Plot inferred intensities of known hotspots as function of time, reference previous work.

\section{Mapping volcanic exoplanets}
\label{sec:exoplanets}
% Application to exoplanets. Fitting mock JWST observations.

\section{Conclusions}
\label{sec:conclusions}

% Bibliography
\bibliography{bib}
\end{document}


\section{Background Modelling}
\label{sec:background_model}

\subsection{Background modeling strategy}
\label{ssec:bkg_strategy}

The strategy for the continuous background modelling follows what traditionally done in $H\to\gamma\gamma$ measurements, with the addition of some dedicated procedures to cope with specific issues related to insufficient MC statistics or very low-statistics categories. This section only contains a simplified description, more details could be found in HGam coupling analysis ~\ref{ANA-HIGG-2020-16-INT1}.

The steps comprising the background modelling strategy can be summarised as follows:
\begin{itemize}

  \item The composition of the continuous background in term of \yy, \yjet and \jetjet components is estimated with data driven technique for each category entering the measurement, as detailed in \Sect{\ref{ssec:bkg_composition}}.
  \item The background fractions are used to build \myy background templates for each category entering the measurement, where the \yy component is obtained from MC, while the \yjet and \jetjet components are measured from data driven control regions and used to reweight the \yy MC, as detailed in \Sect{\ref{ssec:bkg_templates}}.

  \item In order to improve the background modeling and reduce the impact of limited MC statistics, a smoothing procedure is being developed to be eventually applied to the background template, provided that the improvements on the results outweight the possible associated systematic uncertainties.

  \item The Spurious Signal approach is used in the smoothed \yy templates to select a functional form to describe each template, as well as the associated bias that will enter the measurement as a systematic uncertainties, as detailed in \Sect{\ref{ssec:spurious_signal}}. The Spurious Signal criteria are defined in order to cope with the limited MC statistics that in some case leads to non-physical large bias measurements.

  \item In very low-statistic categories, defined as those categories with \textcolor{red}{less than 10 events per bin in the \myy sidebands}, the smoothing procedure tends to introduce large non-physical structures in the templates, and it is therefore not suited to be used for them. For this reason, for this categories we initially select the statistically-justified function performing a Wald-test on the data sidebands using a family of function based on a exponential of increasing complexity. We accept the simplest function justifies by the Wald-test, and perform the Spurious Signal measurement with this function tolerating a bias that could be larger than the criteria used for the high-statistic categories, but would have a small impact on the overall sensitivity. This procedure is described in \Sect{\ref{sssec:bkg_functions}}.

\end{itemize}

\subsection{Background compositions}
\label{ssec:bkg_composition}

The dominant background entering the invariant mass spectrum comes from SM continuum diphoton production. A large number of photons are produced inside jets due to de decays of neutral mesons to photon pairs. Thus, photon-jet and di-jet events in which the jets are mis-identified as photons represent a non-negligible source of background. Photon-pairs can be also faked from Drell-Yan events in which both electrons are misidentified as photons. However, this background only contributes with a small fraction ($<1\%$).

The number of \yy, \yjet and \jetjet events entering each category after the final selection is estimated by means of a double two-dimensional sideband ABCD method~\cite{ATL-COM-PHYS-2012-592}. This data-driven method extrapolates the fraction of fake photons within the signal region from the composition of the side-band control regions, built by inverting photon identification and isolation requirements. The fractions are calculated individually in each categories and OO bins for the background template building. Table ~\ref{tab:yyfraction} lists the \yy fraction in all categories. More details are shown in Appendix ~\ref{appendix:2x2DSB}.

\begin{table}[htbp]
  \centering
  \begin{tabular}{c|cccc}
  \toprule
           &  TT  &  TL  &  LT  &  LL  \\
  \toprule
  OO bin1  & $0.76 \pm 0.16$ & $0.90 \pm 0.10$ & $0.93 \pm 0.03$ & $0.86 \pm 0.05$ \\ \hline
  OO bin2  & $0.89 \pm 0.10$ & $0.81 \pm 0.10$ & $0.81 \pm 0.06$ & $0.87 \pm 0.07$ \\ \hline
  OO bin3  & $0.79 \pm 0.15$ & $0.75 \pm 0.12$ & $0.88 \pm 0.03$ & $0.83 \pm 0.06$ \\ \hline
  OO bin4  & $0.92 \pm 0.08$ & $0.81 \pm 0.09$ & $0.84 \pm 0.05$ & $0.90 \pm 0.05$ \\ \hline
  OO bin5  & $0.78 \pm 0.14$ & $0.81 \pm 0.10$ & $0.86 \pm 0.05$ & $0.88 \pm 0.07$ \\ \hline
  OO bin6  & $0.88 \pm 0.10$ & $0.87 \pm 0.09$ & $0.88 \pm 0.05$ & $0.83 \pm 0.04$ \\
  \bottomrule
  \end{tabular}
  \caption{\yy fraction calculated with double two-dimensional sideband ABCD method. \yjet and \jetjet components are merged together in following analysis, so the fraction is $1-f_{\yy}$. }
  \label{tab:yyfraction}
\end{table}

\begin{figure}[tbp]
  \centering
  \includegraphics[width=0.45\linewidth]{figure/bkgtmpls/TT/yyfrac_TT.png}
  \includegraphics[width=0.45\linewidth]{figure/bkgtmpls/TL/yyfrac_TL.png} \\
  \includegraphics[width=0.45\linewidth]{figure/bkgtmpls/LT/yyfrac_LT.png}
  \includegraphics[width=0.45\linewidth]{figure/bkgtmpls/LL/yyfrac_LL.png}
  \caption{\yy, \yjet, \jetjet fraction in TT(upper left), TL(upper right), LT(bottom left), LL(bottom right) categories, as a function of OO. }
  \label{fig:yyfraction}
\end{figure}

\clearpage
\subsection{Background templates}
\label{ssec:bkg_templates}

Considering the low \jetjet fraction and similar performance of \yjet and \jetjet, those two components are merged together as \yjet+\jetjet for the background estimation. The shape is determined with a data control region, in TT, TL and LT categories it is defined as at least one of the two selected photons fail the tight identification and loose isolation requirement due to insufficient MC statistics, and in LL category it is defined as inverting the identification and isolation of exactly one of two photon candidates.
Contamination from \yy has been tested to be ignorable as shown in Figure ~\ref{fig:yjCR}, and a \textcolor{red}{linear} reweighting is derived to match the \yy and control region shapes. The bin width for ratio function smoothing varies from 5GeV, 2GeV, 1GeV and 0.5GeV respectively in TT, TL, LT, LL categories to match the MC statistics for an un-biased result. Uncertainty of this reweighted template contains two part: from the choice of smoothing function and from \yy fraction, while the latter is dominant.
These derived shapes are combined with the measured relative event fractions to obtain the total background shape. The \yy fraction is measured as mentioned in ~\ref{ssec:bkg_composition} \\

The obtained templates show a good agreement when comparing them with events in data passing the nominal selection (with tight photon identification and isolation cuts, TI) in the sidebands, as shown in \Fig{\ref{fig:BkgData-MC}}.\\

\clearpage
\begin{figure}[htbp]
  \centering
  \subfloat[TT category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/CR/b1.png} }
  \subfloat[TT category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/CR/b2.png} }
  \subfloat[TT category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/CR/b3.png} } \\
  \subfloat[TT category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/CR/b4.png} }
  \subfloat[TT category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/CR/b5.png} }
  \subfloat[TT category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/CR/b6.png} } \\
  \subfloat[TL category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/CR/b1.png} }
  \subfloat[TL category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/CR/b2.png} }
  \subfloat[TL category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/CR/b3.png} } \\
  \subfloat[TL category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/CR/b4.png} }
  \subfloat[TL category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/CR/b5.png} }
  \subfloat[TL category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/CR/b6.png} } \\
\end{figure}
\clearpage
\begin{figure}[htbp]
\centering
  \subfloat[LT category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/CR/b1.png} }
  \subfloat[LT category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/CR/b2.png} }
  \subfloat[LT category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/CR/b3.png} } \\
  \subfloat[LT category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/CR/b4.png} }
  \subfloat[LT category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/CR/b5.png} }
  \subfloat[LT category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/CR/b6.png} } \\
  \subfloat[LL category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/CR/b1.png} }
  \subfloat[LL category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/CR/b2.png} }
  \subfloat[LL category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/CR/b3.png} } \\
  \subfloat[LL category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/CR/b4.png} }
  \subfloat[LL category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/CR/b5.png} }
  \subfloat[LL category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/CR/b6.png} } \\
  \caption{Control region data in all categories and OO bins.}
  \label{fig:yjCR}
\end{figure}
\clearpage

\begin{figure}[htbp]
  \centering
  \subfloat[TT category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/reweight/b1.png} }
  \subfloat[TT category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/reweight/b2.png} }
  \subfloat[TT category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/reweight/b3.png} } \\
  \subfloat[TT category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/reweight/b4.png} }
  \subfloat[TT category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/reweight/b5.png} }
  \subfloat[TT category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/reweight/b6.png} } \\
  \subfloat[TL category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/reweight/b1.png} }
  \subfloat[TL category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/reweight/b2.png} }
  \subfloat[TL category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/reweight/b3.png} } \\
  \subfloat[TL category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/reweight/b4.png} }
  \subfloat[TL category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/reweight/b5.png} }
  \subfloat[TL category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/reweight/b6.png} } \\
\end{figure}
\clearpage
\begin{figure}[htbp]
\centering
  \subfloat[LT category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/reweight/b1.png} }
  \subfloat[LT category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/reweight/b2.png} }
  \subfloat[LT category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/reweight/b3.png} } \\
  \subfloat[LT category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/reweight/b4.png} }
  \subfloat[LT category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/reweight/b5.png} }
  \subfloat[LT category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/reweight/b6.png} } \\
  \subfloat[LL category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/reweight/b1.png} }
  \subfloat[LL category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/reweight/b2.png} }
  \subfloat[LL category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/reweight/b3.png} } \\
  \subfloat[LL category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/reweight/b4.png} }
  \subfloat[LL category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/reweight/b5.png} }
  \subfloat[LL category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/reweight/b6.png} } \\
  \caption{Reweight \yy MC with a linear function to match the \yjet+\jetjet shape in each bins and categories.}
  \label{fig:yyReweight}
\end{figure}
\clearpage

\begin{figure}[htbp]
  \centering
  \subfloat[TT category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/raw_tmpl/b1.png} }
  \subfloat[TT category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/raw_tmpl/b2.png} }
  \subfloat[TT category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/raw_tmpl/b3.png} } \\
  \subfloat[TT category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/raw_tmpl/b4.png} }
  \subfloat[TT category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/raw_tmpl/b5.png} }
  \subfloat[TT category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/raw_tmpl/b6.png} } \\
  \subfloat[TL category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/raw_tmpl/b1.png} }
  \subfloat[TL category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/raw_tmpl/b2.png} }
  \subfloat[TL category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/raw_tmpl/b3.png} } \\
  \subfloat[TL category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/raw_tmpl/b4.png} }
  \subfloat[TL category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/raw_tmpl/b5.png} }
  \subfloat[TL category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/raw_tmpl/b6.png} } \\
\end{figure}
\clearpage
\begin{figure}[htbp]
\centering
  \subfloat[LT category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/raw_tmpl/b1.png} }
  \subfloat[LT category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/raw_tmpl/b2.png} }
  \subfloat[LT category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/raw_tmpl/b3.png} } \\
  \subfloat[LT category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/raw_tmpl/b4.png} }
  \subfloat[LT category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/raw_tmpl/b5.png} }
  \subfloat[LT category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/raw_tmpl/b6.png} } \\
  \subfloat[LL category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/raw_tmpl/b1.png} }
  \subfloat[LL category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/raw_tmpl/b2.png} }
  \subfloat[LL category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/raw_tmpl/b3.png} } \\
  \subfloat[LL category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/raw_tmpl/b4.png} }
  \subfloat[LL category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/raw_tmpl/b5.png} }
  \subfloat[LL category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/raw_tmpl/b6.png} } \\
  \caption{Final background templates.}
  \label{fig:BkgData-MC}
\end{figure}
\clearpage

\subsubsection{Background smoothing}
\label{ssec:bkg_smoothing}
\textcolor{red}{Need Simplification} \\

Because of the limited statistics of the simulation samples used in the calculation, the value
of the spurious signal systematic uncertainty is subject to significant statistical fluctuations within
many of the analysis categories~\cite{Hyneman:2712576}. Statistical fluctuations in the background-only sample may cause
signal-like bumps, which are then fit as signal by the spurious signal procedure. These statistical fluctuations do
not capture the shape mismodeling from the analytic function, and they often drastically inflate the
value of the systematic.

Although simply producing additional simulation samples would alleviate the issue of statistical
fluctuations fitted as spurious signal, producing more simulated events is computationally
expensive. Additionally, producing events which fall into specific phase spaces is often highly inefficient.
Therefore, an alternative solution using the available simulation samples is preferred.

A Gaussian Process (GP) is defined as a set of random processes, where all finite subsets of these
processes have a multivariate normal distribution~\cite{ebden2015gaussian}.
Given a finite dataset – such as the bin contents of a smooth histogram – with corresponding
mean and covariance matrices, a Gaussian Process may be defined. The “correct” mean and the
covariance, however, are not necessarily well defined, as they encode specific assumptions about
the underlying dataset. In practice, the two quantities are fit to a finite dataset using a minimization
algorithm. In the case of a one-dimensional histogram with a finite number of bins, the mean can
be interpreted as a “rough” description of the underlying shape. The diagonal elements of the
covariance matrix represent the error of each bin while the off-diagonal elements specify
how “similar” the bin content of two different bins should be.

The covariance matrix can be simplified through the introduction of a kernel, which analytically
determines the level of correlation between two distinct points (i.e., the length scales in X at which points are expected to influence one another in Y). Two useful kernels are the Radial Basis Function (RBF) kernel and the Gibbs kernel~\cite{3569,Gibbs}.

The RBF kernel has one hyperparameter, the constant length scale $l$, and it is defined as

\begin{equation}
K_\text{RBF}(x,x') = exp\left(\frac{-(x-x')^2}{2l^2}\right)
\end{equation}

The RBF kernel is useful for mostly-flat functions. However, for smoothly-falling functions, it is likely that nearby points will be more correlated in some regions than in others, so a constant length scale is a suboptimal model. The Gibbs kernel allows the length scale $l(x)$ to vary linearly as a function of $x$, and thus has two hyperparameters: the initial length scale and the length scale slope. The Gibbs kernel function is:

\begin{equation}
K_\text{Gibbs}(x, x') = \frac{\sqrt{2l(x)l(x')}}{l(x)^2 + l(x')^2 } \cdot exp\left( \frac{-(x-x')^2}{l(x)^2 + l(x')^2} \right)
\end{equation}

The background templates used in the spurious signal test for the analysis categories are all smooth,
roughly exponentially falling distributions with statistical fluctuations. Fitting a background template
using Gaussian Process Regression (using the Gibbs kernel with the errors as determined by the initial templates)
offers a consistent method of estimating the underlying smooth shape of the template, without the problematic fluctuations.
Notably, the GP smoothing technique makes no assumption on the underlying distribution other than that it is smooth and falling, hence the choice of functional form from the spurious signal test will not be biased.

The hyperparameters (initial length scale and length scale slope) are allowed to vary over a range specified by the user; the optimal hyperparameters within this range are determined by the Gaussian Process fitting procedure.

A GP is fit to the original (noisy) background template in each category. The GP mean in the fits is defined as an exponential function,
the parameters of which are obtained by a fit to the original background template. The exponential
shape has been observed to be a sufficiently close guess for the categories used by the analysis.
However, in cases where the input template has very few statistics (less than about ten events per
bin on average), the resulting GP fit may be nearly identical to the mean exponential shape. This
issue occurs when the statistical uncertainties of the original template are so large that the template
is fully compatible with the preliminary exponential shape. Although the exponential shape
is technically an adequate descriptor of the template shape, the choice of the exponential mean
does bias the functional choice of the spurious signal test in this case. Therefore, a check has been
added to re-perform the GP fit using a flat mean in cases where the resulting GP shape and the
mean exponential shape disagree with a $\chi^2/DoF < 0.1$.

The resulting smoothed shape obtained from the GP fit is then saved as a new histogram. This
smoothed histogram is passed as the background template to the spurious signal test, which then
determines the background functional form and spurious signal systematic uncertainty as described
in \Sect{\ref{ssec:spurious_signal}}.

Note that based on previous study, the GPR method can only remain effectively unbiased for smoothly-falling templates containing more than an average of 20 Monte Carlo events/ bin ~\cite{ATLAS-CONF-2020-026}.
The TT and TL categories in this analysis even do not have sufficient statistics for GPR. Only LT and LL categories used this method, and the treatment of TT and TL is discussed in Section ~\ref{sssec:bkg_functions}.

Examples of the smoothed templates are presented in \Fig{\ref{fig:exampleGPR}} for a sample category with a high
level of statistics and for a category with a very low level of statistics.
The original templates are shown as well, for comparison. The data sidebands are also shown for
validation, although the GP smoothing technique does not take into account the data sidebands.

\begin{figure}[htbp]
  \centering
  \subfloat[LT category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/smooth_tmpl/b1.png} }
  \subfloat[LT category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/smooth_tmpl/b2.png} }
  \subfloat[LT category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/smooth_tmpl/b3.png} } \\
  \subfloat[LT category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/smooth_tmpl/b4.png} }
  \subfloat[LT category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/smooth_tmpl/b5.png} }
  \subfloat[LT category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LT/smooth_tmpl/b6.png} } \\
  \subfloat[LL category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/smooth_tmpl/b1.png} }
  \subfloat[LL category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/smooth_tmpl/b2.png} }
  \subfloat[LL category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/smooth_tmpl/b3.png} } \\
  \subfloat[LL category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/smooth_tmpl/b4.png} }
  \subfloat[LL category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/smooth_tmpl/b5.png} }
  \subfloat[LL category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/LL/smooth_tmpl/b6.png} } \\
  \caption{LT and LL category background templates after GPR smooth.}
  \label{fig:exampleGPR}
\end{figure}

%The smoothed background templates used in this analysis and the spurious signal values extracted from them can be found in the \App{\ref{sec:GPR_templates}}.

%Extensive validation tests were performed with the GP smoothing technique in order to ensure that the smoothing itself does not introduce a substantial bias. These tests primarily use “toy” templates – randomly-generated background templates constructed from either simulated diphoton events or from the probability distribution function of a known analytic function. We find that GPR remains effectively unbiased for smoothly-falling templates containing more than an average of 20 Monte Carlo events/ bin. These procedures to validate the GPR smoothing are reported in \App{\ref{sec:GPR_validation}}.

\subsection{Spurious signal test}
\label{ssec:spurious_signal}

The background \myy shape, for each analysis category, is described using an analytic function whose parameters and normalization are fitted to data. The choices for the analytic function that has been considered are:
\begin{itemize}
\item Exponential Function: $f(\myy) = e^{c\cdot \myy}$
\item Exponential Function of $2^{nd}$ Order Polynomial: $f(\myy) = e^{c_1\cdot m^2_{\gamma\gamma}+c_2\cdot \myy}$
\item Exponential Function of $3^{nd}$ Order Polynomial: $f(\myy) = e^{c_1\cdot m^3_{\gamma\gamma}+c_2\cdot m^2_{\gamma\gamma}+c_3\cdot \myy}$
\item Bernstein polynomial of order $N$: $B_{N}(\myy) = \sum_{i=0}^N c_i\cdot b_{i,N}$ with $b_{i,N} = \begin{pmatrix}N\\i\end{pmatrix}\myy^i (1-\myy)^{N-i}$, N=3, 4, 5
\item First-Order Power Law Function: $f(\myy) = \myy^c$
\end{itemize}
The method to select the functional form in each category is based on the spurious signal. The spurious signal test is described in more detail in \cite{Hasib:2238687}.

To perform the spurious signal test, the full analytic signal plus background model is fitted to a background-only template for each category separately. The fit is performed in the nominal diphoton mass range of $105 \leq \myy \leq \SI{160}{\GeV}$. The number of fitted signal events as a function of the Higgs mass is computed in intervals of \SI{1}{\GeV} within the diphoton mass range of $120 \leq \myy \leq \SI{130}{GeV}$. Negative fluctuations are taken into account. The number of spurious signal events $N_{sp}$ is then defined as the maximum of the absolute value of the fitted number of signal events within the signal diphoton mass sub-range. The statistical error is always taken into account.

The choice of analytic function is done such that the function satisfies at least one of the following criteria:
\begin{itemize}
\item $N_{sp}$ < 10\% $N_{s,exp}$, where $N_{s,exp}$ is the expected number of signal events within the SM in the category ($\mu_{sp} = N_{sp} / N_{s,exp}$)
\item $N_{sp}$ < 20\% $\sigma_{bkg}$, where $\sigma_{bkg}$ is the statistical uncertainty on the fitted number of signal events, when fitting the signal+background model to a background only Asimov dataset ($Z_{sp} = N_{sp} / \sigma_{bkg}$).
\end{itemize}

If multiple functions pass the test, then the one with the least number of parameters is chosen.

%The study including all the categories (also non-ttH) will be performed in the future in accordance with the method described in the previous iteration of the analysis. Due to the higher statistics, new functions (with more parameters) might be needed for the high statistics event categories, as for example the ggH categories. For the ttH only analysis, this does not need to be done.
%These should pass at least the
%Due to the limitation of MC statistic amount,
    A relaxed spurious signal criteria, that are made to accommodate $2\sigma$ local statistical fluctuations in the MC template, in case the MC statistics is small compared to the expected signal. This is done by defining a new variable:
\begin{align}
    \zeta_{sp} =
   \begin{cases}
N_{sp} + 2\Delta_{MC},~~~~ N_{sp} + 2\Delta_{MC}<0\\
N_{sp} - 2\Delta_{MC},~~~~ N_{sp} - 2\Delta_{MC}>0\\
0, ~~~~~~~~~~~~~~~~~~~~~~~\text{otherwise}\\
\end{cases}
 \label{eq:relSpuriousSignalMetric}
\end{align}
where $\Delta_{MC}$ is a local statistical fluctuation of the MC background template. This is illustrated in \Fig{\ref{fig:relaxed_ss_cartoon}}. This newly defined variable should then pass the criteria as $N_{sp}$ before.

For categories with at least 100 data events in the sideband regions defined as $105 < m_{\gamma\gamma} < 120$ GeV or $130 < m_{\gamma\gamma} < 160$ GeV, the background functions are selected to result in a $|S_{\text{spur}}|$ that is smaller than either 10\% of the total expected Higgs boson signal event ($S_{\text{exp}}$) or 20\% of the statistical uncertainty of the fitted signal yield ($\sigma_{\text{exp}}$). The fit of the analytic function to the background template is required to yield a $\chi^2$ probability of at least $1\%$~\footnote{The $\chi^2$ is computed with a background template uniformly binned over $105<m_{\gamma\gamma}<160$ GeV. The number of bins is 22, and the degrees of freedom used in the computation is 21 - $N_{\text{pars}}$, the number of free function parameters.}.

If multiple functions pass the requirements, the one with the smallest number of degrees of freedom is chosen. If no function passes, the spurious signal requirement is relaxed to
 $\zeta_{sp}$ < 10\% $N_{s,exp}$ or $\zeta_{sp}$ < 20\% $\sigma_{bkg}$.
For 28 out of the 88 categories, these relaxed conditions were used.
It's to be emphasized that even the relaxed condition is used, the background uncertainty is still $N_{sp}$ from the nominal approach of spurious signal study.

\subsubsection{Treatment of low-statistic cetegories}
\label{sssec:bkg_functions}

The determination of analytical functions is part of the result of spurious signal study, however, for low stastics categories with a background template suffering of very large statistical flucttuation we introduce a preliminary step. The straight use of the spurious signal procedure on these templates, in fact, would potentially lead to select a function with large number of degrees of freedom to accomodate these statistical fluctuation, and these function can prove to be dangerous when used on data. For this reason, for categories with less then 100 events in the \myy\ sidebads, we restrict the selection of the modeling function to exponential of first, second and third order, collectively described by:
\begin{equation}
  f_i{x} = e^{\sum_{i=0}^2 a_i x^i}
\end{equation}
where $x = \myy$.

We accept as modeling function the $f_i$ function with the lowest number of degrees of freedom for which the addition of more degrees of freedom does not lead to a significant improvement in the quality of the fit of the \myy\ sidebands.
To this purpose, we checked the likelihood ratio between the fits to the side-band data with two different background models:
\begin{equation}
\lambda_{(1,2)} = -2ln(L_{1}/L_{2})
\end{equation}
Where $L_{1}$ and $L_{2}$ are the likelihood values of the two fits.

The distributions of these test statistics are computed with \myy\
 side-band data, fitting it with two models and calculating the value of the test statistics.
 Once the distributions of the test-statistics
are computed, we can reject the simpler (less parameters) background
model by checking the P-value $P(\lambda >= \lambda_{\mathrm{data}})$ with the $\lambda$ measured in data ($\lambda_{\mathrm{data}}$).
This procedure to reject simpler background function is called Wald test.

Figure ~\ref{fig:Wtest} shows plots of the Wald-test on the low-statistic categories. All of the categories selected exponential function as the final decison.
Where the lowest P-value among all categories are around 3$\%$, corresponding to a significance of 2.2 $\sigma$ standard deviations.
The background uncertainty for these categories will be the spurious signal from exponential function.

As a cross-check, a F-test is also performed in Appendix ~\ref{append:ss}, and we draw the same conclusion that all of the categories can select exponential function.
The reason we use Wald test to decide the background function is the fits on side-band data are un-bin fit (so that it's consistent with the final result),
    there might be bias if we calculate the $\chi^2$ values in an artifical bin-size for the F-test. In this case the Wald test based on likelihood ratio is more reliable.

\begin{figure}[htbp]
  \centering
  \subfloat[TT category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/wald/b1.png} }
  \subfloat[TT category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/wald/b2.png} }
  \subfloat[TT category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/wald/b3.png} } \\
  \subfloat[TT category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/wald/b4.png} }
  \subfloat[TT category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/wald/b5.png} }
  \subfloat[TT category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TT/wald/b6.png} } \\
  \subfloat[TL category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/wald/b1.png} }
  \subfloat[TL category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/wald/b2.png} }
  \subfloat[TL category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/wald/b3.png} } \\
  \subfloat[TL category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/wald/b4.png} }
  \subfloat[TL category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/wald/b5.png} }
  \subfloat[TL category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgtmpls/TL/wald/b6.png} } \\
  \caption{Wald test in TT and TL categories. In all OO bins the first order exponential function shows the lower p value and is selected.}
  \label{fig:Wtest}
\end{figure}

\subsection{Background modelling results}
\label{ssec:bck_results}

The determined analytical functions modeling the baground in each category entering the measureument are found in Table~\ref{tab:spurious_sig} , as well as the corresponsing Spurious Signal bias.
The background only fit on the background templates based on MC samples is shown in Figure ~\ref{fig:bkgonly}

\begin{table}[htbp]
\centering
\small
\begin{tabular}{l|l|l|l|l|l|l|l|l}
\hline
           & \multicolumn{2}{c|}{TT} & \multicolumn{2}{c|}{TL} & \multicolumn{2}{c|}{LT} & \multicolumn{2}{c|}{LL} \\ \hline
           & Selected func. & maxS   & Selected func. & maxS   & Selected func. & maxS   & Selected func. & maxS   \\ \hline \hline
bin0       & Exp            & -0.29  & Exp            & -0.468 & Exp            & 0.87   & Exp            & 0.885  \\ \hline
bin1       & Exp            & 0.582  & Exp            & -0.882 & Exp            & -0.534 & Exp            & 1.54   \\ \hline
bin2       & Exp            & 0.235  & Exp            & -0.355 & Pow            & 0.437  & ExpPoly2       & -1.64  \\ \hline
bin3       & Exp            & 0.409  & Exp            & -0.304 & Pow            & 0.642  & Exp            & -0.718 \\ \hline
bin4       & Exp            & -0.404 & Exp            & 0.779  & Exp            & 0.775  & Exp            & -1.19  \\ \hline
bin5       & Exp            & 0.293  & Exp            & 0.408  & Exp            & 0.627  & Exp            & 0.52   \\ \hline
\end{tabular}
\caption{The final background modelling decision and the size of spurious signal uncertainties.}
\label{tab:spurious_sig}
\end{table}

\begin{figure}[htbp]
  \centering
  \subfloat[TT category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/TT/b1.pdf} }
  \subfloat[TT category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/TT/b2.pdf} }
  \subfloat[TT category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/TT/b3.pdf} } \\
  \subfloat[TT category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/TT/b4.pdf} }
  \subfloat[TT category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/TT/b5.pdf} }
  \subfloat[TT category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/TT/b6.pdf} } \\
  \subfloat[TL category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/TL/b1.pdf} }
  \subfloat[TL category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/TL/b2.pdf} }
  \subfloat[TL category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/TL/b3.pdf} } \\
  \subfloat[TL category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/TL/b4.pdf} }
  \subfloat[TL category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/TL/b5.pdf} }
  \subfloat[TL category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/TL/b6.pdf} } \\
\end{figure}
\clearpage
\begin{figure}[htbp]
\centering
  \subfloat[LT category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/LT/b1.pdf} }
  \subfloat[LT category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/LT/b2.pdf} }
  \subfloat[LT category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/LT/b3.pdf} } \\
  \subfloat[LT category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/LT/b4.pdf} }
  \subfloat[LT category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/LT/b5.pdf} }
  \subfloat[LT category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/LT/b6.pdf} } \\
  \subfloat[LL category OO bin1]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/LL/b1.pdf} }
  \subfloat[LL category OO bin2]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/LL/b2.pdf} }
  \subfloat[LL category OO bin3]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/LL/b3.pdf} } \\
  \subfloat[LL category OO bin4]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/LL/b4.pdf} }
  \subfloat[LL category OO bin5]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/LL/b5.pdf} }
  \subfloat[LL category OO bin6]{ \includegraphics[width= 0.30\textwidth]{figure/bkgfit/LL/b6.pdf} } \\
  \caption{Background only fit on the background templates based on MC samples. }
  \label{fig:bkgonly}
\end{figure}


\section{The Model}

Protocol participants are represented as parties---formally Interactive Turing Machine instances (ITIs)---in a multi-party computation. We assume a central adversary who  corrupts miners and  uses them to attack the protocol. The adversary is {\em adaptive}, i.e., can corrupt (additional) parties  at any point  and depending on his current view of the protocol execution. Our protocols are synchronous (G)UC protocols~\cite{C:BMTZ17,TCC:KMTZ13}:  parties have access to  a (global) clock setup, denoted by~$\gclock,$ and can communicate over a network of authenticated multicast channels. We assume instant and {\em fetch-based}  delivery channels~\cite{TCC:KMTZ13,AC:CGHZ16}. Such channels, whenever they receive a message from their sender, they record it and deliver it to the receiver upon his request with a ``fetch'' command. In fact, all functionalities we design in this work will have such  fetch-based delivery of their outputs. We remark that the instant-delivery assumption is without loss of generality as the channels are only used for communicating the timestamped object to the verifier which can anyway happen at any point after its creation. However, our treatment trivially applies also to the setting where parties communicate over  bounded-delay channels as in~\cite{C:BMTZ17}.

We adopt the {\em dynamic availability} model implicit in~\cite{C:BMTZ17} which was fleshed out in~\cite{CCS:BGKRZ18}. We next sketch its main components: All functionalities, protocols, and global setups have a dynamic party set. i.e., they all include special instructions allowing parties to register,  deregister, and allowing the adversary to learn the current set of registered parties. Additionally,  global setups allow any other setup (or functionality) to register and deregister with them, and they also allow other setups to learn their set of registered parties. For more details on the registration process we refer the reader to Appendix~\ref{se:registration}.
\paragraph{The Clock Functionality} $\gclock$ (cf. Fig.~\ref{fig:gclock}).
%As in~\cite{BMTZ17} we  assume a global UC-functionality $\gclock$  that allows the parties to ensure that the protocol they are running proceeds in synchronized rounds.
The \emph{clock functionality}  was initially proposed in~\cite{TCC:KMTZ13} to enable synchronous execution of UC protocols. Here we adopt its global-setup version, denoted by $\gclock$, which was proposed by~\cite{C:BMTZ17} and was used in the (G)UC proofs of the ledger's security.\footnote{As a global setup, $\gclock$ also exists in the ideal world and the ledger connects to it to keep track of rounds.}  $\gclock$  allows  parties (and functionalities) to ensure that the protocol they are running proceeds in synchronized rounds; it keeps track of  round variable whose value can be retrieved by parties (or by functionalities) via sending to it the pair: $\gread$. This value is increased when every honest party has sent to the clock a command $\gupdate$.
The parties use the clock as follows.  Each party starts every operation by reading the current round from $\gclock$ via the command $\gread$. Once any party  has executed all its instructions
for that round it instructs the clock to advance by sending a $\gupdate$ command, and gets in an idle mode where it simply reads the clock time in every activation until the round advances. %The clock advances to the next round when all the parties have requested to advance to the next round.
%In~\cite{KZZ16,BMTZ17}, this idea was transfered to the (G)UC setting, by assuming that the clock is a global setup. This allows  different protocols to use the same clock and we will also use the latter model in this paper. In particular, we model the \emph{time aspect} of all the functionalities that we propose, in terms of number of rounds. Therefore, when  $\gclock$ proceeds from one round to another, we say that $\gclock$ has \emph{ticked}.
%For completeness, we have included the global clock functionality in Appendix~\ref{app:funcs} We refer  readers to Fig.~\ref{fig:gclock} for a formal description of the functionality.
To keep more compact the description of our functionalities that rely on $\gclock$, we implicitly assume that
whenever an input is received the command $\gread$ is sent to $\gclock$ to retrieve the current round. Moreover, before giving the output, the functionalities request to advance the clock by sending $\gupdate$ to $\gclock$.

\subsection{Ledger Consensus: Model}\label{se:ledgermodel}

In this section, we define our notion of protocol execution following~\cite{EC:GarKiaLeo15,FOCS:Canetti01}.
The execution of a protocol $\Pi$ is driven by an
environment program $\env$ that may spawn multiple instances running the protocol $\Pi$. The programs
in question can be thought of as interactive Turing machines (ITM) that have communication,
input and output tapes. An instance of an ITM running a certain program will be referred to as
an interactive Turing machine instance or ITI. The spawning of new ITI's by an existing ITI as
well as the interaction between them is at the discretion of a control program which is also an ITM
and is denoted by $C$. The pair $(\env, C)$ is called a system of ITM's, cf.~\cite{FOCS:Canetti01}.
Specifically, the execution driven by $\env$ is defined with respect to a protocol $\Pi$, an adversary $\adv$
(also an ITM) and a set of parties $P_1,\dots,P_n$; these are hardcoded in the control program $C$.
Initially, the environment $\env$ is restricted by $C$ to spawn the adversary $\adv$. Each time the
adversary is activated, it may send one or more messages of the form $(\corrupt, P_i)$ to $C$. The control
program $C$ will register party $P_i$ as corrupted, only provided that the environment has previously
given an input of the form $(\corrupt, P_i)$ to $\adv$ and that the number of corrupted parties is less or
equal $\tcorrupt$, a bound that is also hardcoded in $C$.

We divide time into discrete units called \emph{time slots} or {\emph round}. Players are equipped with (roughly)
synchronized clocks $\gclock$ that indicate the current slot: we assume that any clock drift is subsumed in the slot
length.
%Each slot $\slr$ is indexed by an integer $r\in\{1,2,\dots\}$.

%The first ITI party to be spawned running protocol
%$\Pi$ is restricted by $C$ to be party $P_1$. After a party $P_i$ is activated, the environment is restricted to
%activate party $P_{i+1}$, except when $P_n$ is activated in which case the next ITI to be activated is always
%the adversary $\adv$. Note that when a corrupted party $Pi$ is activated the adversary $\adv$ is activated
%instead.

\subsubsection{Ledger Consensus.}
Ledger consensus (a.k.a. ``Nakamoto consensus'') is the problem where a set of nodes
(or parties) operate continuously accepting inputs
that are called transactions and incorporate them in a public data
structure called the {\em ledger}.
A ledger (denoted in calligraphic-face, e.g. $\ledger$) is a mechanism for maintaining a sequence of transactions, often
stored in the form of a blockchain. In this work, we denote with $\ledger$
the algorithms used to maintain the sequence, and with $\state$ all the views of the
participants of the state of these algorithms when being executed. For example, the (existing) ledger Bitcoin
consists of the set of all transactions that ever took place in the Bitcoin network, the current UTXO set, as
well as the local views of all the participants.
In contrast, we call a \emph{ledger state} a concrete sequence of transactions $\tx_1,\tx_2,\dots$ stored in the stable
part of a ledger state $\state$, typically as viewed by a particular party. Hence, in every blockchain-based ledger $\ledger$,
every fixed chain $\mathcal{C}$ defines a concrete ledger state by applying the interpretation rules given as a part of the
description of $\ledger$. In this work, we assume that the ledger state is obtained from the blockchain by dropping the last $k$ blocks
and serializing the transactions in the remaining blocks. We refer to $k$ as the \emph{common-prefix parameter}.
We denote by $\state^P[t]$ the ledger state of a ledger $\ledger$ as viewed by a party $P$ at the beginning of a time slot $t$
and by ${\check \state}^{P}[t]$ the complete state of the ledger (at time $t$) including all pending transactions that are not stable yet. $\state^P[t]$ can be obtained from  ${\check \state}^{P}[t]$ by dropping the last $k$ block.
%ledger states (or, more generally, any sequences), we denote by the prefix relation.
% and by $\hat P^\ledger[t]$ the complete state of the ledger (at time $t$) including all pending transactions that are not stable yet.

For two ledger states (or, more generally, any sequences), we denote by $\preceq$ the prefix relation.
Recall the definition of secure ledger protocol given in~\cite{RSA:GarKia20}.

\begin{definition}\label{de:ledger} A ledger protocol $\ledger$ is secure if it enjoys the following properties.

\ignore{
\begin{itemize}
\item {\em Consistency:} This property mandates that if a client queries
an honest node's ledger at round $t_1$
and receives the response $\mc{L}_1$,
then a client querying an honest node's ledger at round $t_2\geq t_1 $
 receives a response $\mc{L}_2$ that satisfies $\mc{L}_1 \preceq \mc{L}_2$,
where $\preceq$ denotes the standard prefix operation.

\item {\em Liveness:} If a transaction $tx$ is given as input to all honest
nodes continuously for a certain number of rounds denoted by $u$,
and a client queries any honest node's ledger
after these $u$ rounds, then the node responds with a value
$\mc{L}$ that includes $tx$.
%\end{itemize}
}

\begin{myitemize}
  \item[]{\bf Consistency.}
    For any two honest parties $\party_1,\party_2$ and two time slots $t_1\leq t_2$,
    it holds  $\LView{\state}{\party_1}{t_1} \preceq
    \LView{\check{\state}}{\party_2}{t_2}$.

  \item[]{\bf Liveness.}
    If all honest parties in the system attempt to include a  transaction $\tx$
    then, at any slot $t$ after $s$ slots (called the
    liveness parameter), any honest party $\party$, if queried,
    will report $\tx \in \LView{\state}{\party}{t}$.
\end{myitemize}

\end{definition}

In this work we also explicitly rely on the properties of \emph{Common Prefix (CP)}, \emph{Chain Growth (CG)}
and  \emph{Chain Quality (CQ)}.

\begin{myitemize}
  \item[] {\bf Common Prefix (CP); with parameters $k\in\mathbb{N}$}
states that for any pair of honest players  $P_1,P_2$ at rounds $r_1\leq r_2$ respectively, it holds that  ${ \state}^{P_1}[r_1] \preceq  {\check \state}^{P_2}[r_2] $.
  \item[] {\bf Chain Growth (CG); with parameters $\cg\in (0,1]$ and $s\in\mathbb{N}$}. Consider the chain $\mathcal{C}$
  adopted by an honest party at the onset of a slot and any portion of $\mathcal{C}$ spanning $s$ prior slots; then the number of
  blocks appearing in this portion of the chain is at least $\tau s$.

  \item[] {\bf Chain Quality (CQ) with parameters $\mu\in\mathbb{R}$
and $\ell\in\mathbb{N}$}. For any honest party $P$ with chain $\mathcal{C}$  it holds that for any $\ell$ consecutive blocks of $\mathcal{C}$ the ratio of honest blocks is at least $\mu$.
 \end{myitemize}

We consider a setting where a set of parties run a protocol maintaining a ledger $\ledger_1$. Following~\cite{SP:GazKiaZin19}, we denote by $\asmp_1$
the assumptions for $\ledger_1$.
 That is, if the assumption $\asmp_1$ holds, then ledger $\ledger_1$ is secure under the Definition~\ref{de:ledger}.
Formally, $\asmp_i$ for a ledger $\ledger_i$ is a sequence of events $\asmp_i[t]$ for each time slot $t$ that can assume value $1$,
if the assumption is satisfied, and $0$ otherwise.
For example, $\asmp_i$ may denote that
there has never been a majority of hashing power (or stake in a particular
asset, on this ledger or elsewhere) under the control of the adversary; that a
particular entity (in case of a centralized ledger) was not corrupted; and so
on.
Without loss of generality, we say that the assumption $\asmp_1$ for the ledger $\ledger_1$ holds if and only if the fraction of corrupted parties (the parties
that received the input $(\corrupt, \cdot)$) is below the threshold $\tcorrupt_1$ (where $\tcorrupt_1$ is part of the control function as described in the beginning
of this section).

\paragraph{Chain selection rule and block validation.} We sometimes assume that a ledger protocol describes a \emph{chain selection rule} that we denote with $\chainsel$.
That is, we assume that  each party in each round of the execution of the protocol collects all chains that come from the network and runs the algorithm $\chainsel$ to decide whether to keep his current local chain $\cloc$, or adopt one of the newly received chains.
Following~\cite{C:BMTZ17} we also assume
that before applying the chain-selection rule, any given chain is tested
 using the procedure $\isvalid$. $\isvalid$ checks filters the
 valid chains among all the chain received from the network and only
 the valid chain are used as input for $\chainsel$.
 $\chainsel$ in turns rely on the algorithm $\isvalidb$.
 $\isvalidb$ take as input a block $B$ of $\cloc$ and outputs $1$ if $B$ is a valid block (i.e., the structure of
 the block is correct) and  $0$ otherwise.

 We note that by assuming that a ledger protocol is always equipped with the algorithms $\chainsel$,
 $\isvalid$ and $\isvalidb$ make some of our results less general. However, we will show that it is possible to
obtain a better updatable ledger in the case when the two ledgers (the current ledger) and the new ledger have the same chain selection rule (among other similarities).

\subsection{Genesis Block Functionality}
The ledger protocols that we consider in this work are equipped with the description of an algorithm $\genesis$ that, on input a random value of appropriate length, outputs a valid genesis block (i.e., the first block of the chain). The security of most of the known ledger protocols holds
under the additional assumption that the genesis block is correct. That is, the genesis block has been generated
accordingly to $\genesis$ using appropriate randomness.
Multiple ways have been presented to generate a correct genesis block in the literature (i.e., by relying on a trusted authority, use
unpredictable information (like in bitcoin), run a multi-party computation (MPC) protocol~\cite{zcash},
rely on PoW~\cite{PKC:GKLP18} assumptions and so on and so forth).
In this work we abstract the generation of the genesis block by means of an ideal functionality.
The ideal functionality that one might expect, upon being activated from the adversary or from an honest party, should sample a random string and use it to run the algorithm $\genesis$. Unfortunately this simple functionality does not cover real world scenarios where an adversarial party might see the genesis block before the honest parties do. This, for example, can happen in the case when $\genesis$ is realized via an MPC protocol and a rushing adversary\footnote{A rushing adversary waits to receive the messages from all the honest parties and then computes its reply. Note that this means that, in general, the adversary is always able to see the output of the computation before the honest parties do.} could hold the genesis block (the output of the computation) for some bounded amount of time $\delay$ before the honest parties can see it.
We note that an adversary can use this strategy to take an advantage on the generation of the blocks that extend the genesis block.
 Therefore, the first modification that we consider for our ideal functionality is to allow the adversary to see the genesis block up to $\delay$ rounds earlier than the honest parties.
 The second relaxation allows the adversary to see up to
 $\maxblocks$ honestly generated genesis blocks and consequently decide which of these blocks will become the
 genesis block.
 We propose the formal description of our genesis functionality $\fgenesis$ in Fig.~\ref{fig:genesis}.
 We note that the case where $\delay=0$ and $\maxblocks=1$ corresponds to the case where there is only one candidate genesis block and all the parties can see it at the same round.

\begin{figure}[]
\small
\setlength{\fboxsep}{8pt}
\begin{center}
\begin{boxedminipage}{12cm}
{\bf Genesis Functionality for $\ledger$}

{\bf Parameters.} The functionality is parametrized by $\delay$, the maximum number of candidate genesis block $\maxblocks$, the genesis block $\bgen$ initialized with a default value $\bot$ and the procedure $\genesis()$.
We assume the functionality to be registered to $\gclock$ and that it maintains a set of registered parties $\parties$.
On any input $I$ the functionality queries $\gclock$, and we denote with  $\now$ be the response obtained by $\gclock$.

\begin{description}
\item[-] If $I=\gengenesis$ is received  from the adversary $\adv$ then  set $\tau:=\now$, generate $\maxblocks$
 genesis blocks (each block is generated by running the procedure $\genesis()$) $\GB:=\{\bgen_1,\dots, \bgen_\maxblocks\}$ for $\ledger$, and send $\GB$ to the adversary.

 \item[-] If $I=\getgenesis$ is received from an honest party $p_i\in\parties$ do the following
 \begin{itemize}
 	\item If $\bgen\neq\bot$ then return $\bgen$ to $p_i$.
	\item If $\bgen =\bot$ and $R-\tau>\delay$ then set generate a genesis block $\tilde \bgen$ by running
	 $\genesis$, set
	$\bgen\leftarrow \tilde \bgen$ and send $\bgen$ to $p_i$.
\end{itemize}
 \item[-] If $I=(\setgenesis,{\bgen}')$ is received from the adversary do the following
 \begin{itemize}
 	\item If $(R-\tau)\leq\delay$ and ${\bgen}'\in\GB$ then set $\bgen:={\bgen}'$.
	\item Else, return $\bot$ to the adversary.
 \end{itemize}

\end{description}

\end{boxedminipage}
\end{center}
\caption{The genesis functionality $\fgenesis$.}
\label{fig:genesis}
\end{figure}