\documentclass[11pt, oneside]{article}

% Packages
\usepackage{jmlr2e}
\usepackage{xcolor}
\usepackage[super]{nth}
\usepackage{mathtools, amsfonts}

% Package config
% \sectionfont{\normalfont\sffamily\bfseries}
% \subsectionfont{\normalfont\sffamily\bfseries}
\hypersetup{colorlinks,
    linkcolor={red!50!black},
    citecolor={blue!50!black},
    urlcolor={blue!80!black}
}

% Notation and macros
\input{notation.tex}

% Preamble

% Heading arguments are
% {volume}{year}{pages}{submitted}{published}{author-full-names}
% \jmlrheading{}{}{}{}{}{}

% Short headings should be running head and authors last names
\ShortHeadings{\revrand~Technical Report}{Steinberg, Tiao, Reid, McCalman and
    O'Callaghan}
\firstpageno{1}

\title{\revrand: Technical Report}
\author{\name Daniel Steinberg \email daniel.steinberg@data61.csiro.au \\
        \name Louis Tiao \email louis.tiao@data61.csiro.au \\
        \name Alistair Reid \email alistair.reid@data61.csiro.au \\
        \name Lachlan McCalman \email lachlan.mccalman@data61.csiro.au \\
        \name Simon O'Callaghan \email simon.ocallaghan@data61.csiro.au \\
        \addr DATA61, CSIRO \\
        Sydney, Australia}

\date{}

\begin{document}

\maketitle
% \vspace{-0.5cm}
% \noindent\makebox[\linewidth]{\rule{\linewidth}{0.8pt}}
% \vspace{0.3cm}

\begin{abstract}
    This is a technical report on the \revrand software library. This
    library implements Bayesian linear models (Bayesian linear regression),
    generalized linear models and approximate Gaussian processes. These
    algorithms have been implemented such that they can be used for large-scale
    learning by using stochastic variational inference. All of the algorithms
    in \revrand use a unified feature composition framework that allows
    for easy concatenation and selective application of regression basis
    functions.
\end{abstract}

\tableofcontents

\section{Core Algorithms}

Recent developments in stochastic gradient optimisation have simplified the
application of machine learning to massive datasets, in particular, modern
stochastic optimisation algorithms are far more robust to initial learning rate
settings. Furthermore, Bayesian machine learning algorithms have a number of
well defined methods for learning model parameters and \emph{hyperparameters}
from training data, that do not involve cross validation. When used in
combination, stochastically optimized Bayesian machine learning algorithms
allow practitioners to learn probabilistic predictors from large data sets with
minimal tuning and retraining.

We make use of some of these recent developments in stochastic gradient methods
and stochastic variational inference in \revrand for supervisd regression
tasks. We outline the core algorithms implemented in \revrand in this section.

\subsection{Stochastic Gradients and Variational Objective Functions}
\label{sub:stochvar}

When a machine learning objective function factorises over data,
\begin{equation}
    \ffunc{\ins, \param}{} = \sum_{\inss \in \ins} \ffunc{\inss, \param}{},
\end{equation}
a regular gradient descent would perform the following iterations to minimise
the function w.r.t.\ the parameters $\param$,
\begin{equation}
    \param_k := \param_{k-1} - \lrate_k \sum_{\inss \in \ins}
    \nabla_{\param} \ffunc{\inss, \param}{}\!|_{\param = \param_{k-1}},
\end{equation}
where $\lrate_k$ is the learning rate (step size) at iteration $k$. Stochastic
gradients proposes the following update,
\begin{equation}
    \param_k := \param_{k-1} - \lrate_k \sum_{\inss \in \minibatch}
    \nabla_{\param} \ffunc{\inss, \param}{}\!|_{\param = \param_{k-1}},
    \label{eq:sg}
\end{equation}
where $\minibatch \subset \ins$ is a mini-batch of the original dataset, where
$|\minibatch| \ll |\ins|$.
Frequently objective functions to not entirely decompose over the data, i.e.,
\begin{equation}
    \ffunc{\ins, \param}{} = \sum_{\inss \in \ins} \ffunc{\inss, \param}{}
    + \func{g}{\param}{}.
    \label{eq:objwconst}
\end{equation}
However, it is trivial to make these objectives work in a stochastic gradient
setting. Let $M = |\minibatch|$ and $N = |\ins|$, then we divide the
contribution of the constant term amongst the mini-batches in stochastic
gradients,
\begin{equation}
    \param_k := \param_{k-1} - \lrate_k \sum_{\inss \in \minibatch}
    \nabla_{\param} \ffunc{\inss, \param}{}\!|_{\param = \param_{k-1}}
    - \frac{M}{N} \lrate_k \nabla_{\param}
    \func{g}{\param}{}\!|_{\param = \param_{k-1}}.
    \label{eq:wsg}
\end{equation}
or, equivalently, boost the contribution of the mini-batch,
\begin{equation}
    \param_k := \param_{k-1} - \frac{N}{M} \lrate_k \sum_{\inss \in \minibatch}
    \nabla_{\param} \ffunc{\inss, \param}{}\!|_{\param = \param_{k-1}}
    - \lrate_k \nabla_{\param}
    \func{g}{\param}{}\!|_{\param = \param_{k-1}}.
    \label{eq:wsg2}
\end{equation}
This is particularly relevant for variational inference where the evidence
lower bound objective has a component independent of the data. For example,
lets consider the model,
\begin{align}
    \text{Likelihood:} \quad &\prod^N_{n=1} \probC{\targs_n}{\param}, \\
    \text{prior:} \quad &\probC{\param}{\hyper},
\end{align}
where we want to learn the values of the hyper-parameters, $\hyper$. Minimising
negative log-marginal likelihood is a good objective in this instance, since we
don't care about the value(s) of $\param$,
\begin{equation}
    \argmin_\hyper - \log \int \prod^N_{n=1} \probC{\targs_n}{\param}
    \probC{\param}{\hyper} d \param.
    \label{eq:lml}
\end{equation}
There are two problems with this objective however, (1) it may not factor over
data and (2) the integral may be intractable, for instance, if the prior and
likelihood are not conjugate. In variational inference we use Jensen's
inequality to lower-bound log-marginal likelihood with a tractable objective
function called the evidence lower bound (ELBO),
\begin{align}
    \log \probC{\targ}{\hyper} =& \log \int
        \prod^N_{n=1} \probC{\targs_n}{\param}
        \probC{\param}{\hyper} d \param \nonumber \\
        =& \log \int
        \frac{\prod_n \probC{\targs_n}{\param} \probC{\param}{\hyper}}
        {\qrob{\param}} \qrob{\param} d \param \nonumber \\
        \geq& \int \qrob{\param} \log \sbrac{%
            \frac{\prod_n \probC{\targs_n}{\param}
            \probC{\param}{\hyper}}{\qrob{\param}}}
        d \param
\end{align}
where $\qrob{\param}$ is an approximation of $\probC{\param}{\hyper}$ that
makes inference easier. This can be re-written as,
\begin{equation}
    \elbo = \sum^N_{n=1} \expec{q}{\log\probC{\targs_n}{\param}} -
    \KL{\qrob{\param}}{\probC{\param}{\hyper}},
    \label{eq:elbo}
\end{equation}
which takes the form of Equation~\eqref{eq:objwconst}, and so if we use
stochastic gradients optimisation we can weight the Kullback-Leibler term like
the constant term, $\func{g}{\cdot}{}$, from Equation~\eqref{eq:wsg}, or boost
the expected log likelihood term like in Equation~\eqref{eq:wsg2}.

\subsection{Bayesian Linear Regression -- \texttt{StandardLinearModel}}

The first machine learning algorithm in \revrand is a simple Bayesian linear
regressor of the following form,
\begin{align}
    \text{Likelihood:} \quad &\prod^N_{n=1}
    \gausC{\targs_n}{\feats_n\T\weights, \var}, \\
    \text{prior:} \quad &\gausC{\weights}{\mathbf{0}, \Reg},
\end{align}
where $\feats_n := \func{\featsym}{\inss_n, \param}{}$ is a feature, or basis,
function that maps $\featsym : \real{d} \to \real{D}$, and $\Reg \in
\real{D\times D}$ is a diagonal matrix (i.e.~it could be $\reg\ident{D}$) that
has the effect of regularising the magnitude of the weights. This is the same
algorithm described in~\citet[Chapter 2]{Rasmussen2006}. We then:
\begin{itemize}
    \item Optimise $\var, \Reg$ and $\param$ w.r.t.\ log-marginal likelihood,
        \begin{equation}
            \log \probC{\targ}{\var, \Reg, \param} =
            \log \gausC{\targ}{\mathbf{0}, \var\ident{N} + \feat\T\Reg\feat},
        \end{equation}
        where $\feat \in \real{N \times D}$ is the concatenation of all the
        features, $\feats_n$. Note this results in the covariance of the
        log-marginal likelihood being $N \times N$, though we can use the
        Woodbury identity to simplify the corresponding matrix inversion.
    \item Solve analytically for the posterior over weights, $\weights | \targ
        \sim \gaus{\pomean, \pocov}$ given the above hyperparameters, where,
        \begin{align*}
            \pocov &= \sbrac{\Reg\inv + \frac{1}{\var}
                \feat\T \feat}\inv, \\
            \pomean &= \frac{1}{\var} \pocov \feat\T \targ.
        \end{align*}
    \item Use the predictive distribution
        \begin{align}
            \probC{\targs\test}{\targ, \ins, \inss\test} &= \int
            \gausC{\targs\test}{\feats\testT\weights, \var}
            \gausC{\weights}{\pomean, \pocov} d\weights, \nonumber \\
            &= \gausC{\targs\test}{\feats\testT \pomean,
                \var + \feats\testT \pocov \feats\test}
        \end{align}
        for query inputs, $\inss\test$. This gives us the useful expectations,
        \begin{align}
            \expece{}{\targs\test} &= \feats\testT\pomean, \\
            \evar{}{\targs\test} &= \var + \feats\testT\pocov\feats\testT.
        \end{align}
\end{itemize}

It is actually easier to use the ELBO form with stochastic gradients for
learning the parameters of this algorithm, rather than log-marginal likelihood
recast using the Woodbury identity.  This is because it is plainly in the same
form as Equation \eqref{eq:objwconst}, though it would give the same result as
log-marginal likelihood because the ``approximate'' posterior is the same form
as the true posterior, i.e.\ $\qrob{\weights} = \gausC{\weights}{\pomean,
    \pocov}$. The ELBO for this model is,
\begin{equation}
    \elbo = \sum^N_{n=1}
    \expec{q}{\log\gausC{\targs_n}{\feats_n\T\weights, \var}}
    - \KL{\gausC{\weights}{\pomean, \pocov}}
        {\gausC{\weights}{\mathbf{0}, \Reg}}.
    \label{eq:slmobj}
\end{equation}
More specifically,
\begin{align*}
    \expec{q}{\log\gausC{\targs_n}{\feats_n\T\weights, \var}} &=
    \log \gausC{\targs_n}{\feats_n\T\pomean, \var}
    - \frac{1}{2 \var} \trace{\feats_n\T\feats_n\pocov}, \\
    \KL{\gausC{\weights}{\pomean, \pocov}}
        {\gausC{\weights}{\mathbf{0}, \Reg}} &=
        \frac{1}{2} \sbrac{\trace{\Reg\inv\pocov} + \pomean\T\Reg\inv\pomean
        -  \log\deter{\pocov} + \log\deter{\Reg} - D}
\end{align*}
We have not implemented a stochastic gradient version of this algorithm since
it still requires a matrix solve of a $D \times D$ matrix, and so is \bigo{D^3}
in complexity, per iteration. This is true even if we optimise the posterior
covariance directly (or a triangular parameterisation). The GLM presented in
the next section circumvents this issue, and is more suited to really large $N$
and $D$ problems.

\subsection{Bayesian Generalized Linear Models --
    \texttt{GeneralizedLinearModel}}

The algorithm of primary interest in \revrand is the Bayesian generalized
linear model. The general form of the model implemented by this algorithm is,
\begin{align}
    \text{Likelihood:} \quad &\prod^N_{n=1}
        \probC{\targs_n}{\activ{\feats_n\T\weights}, \lparam},
        \label{eq:glmlike} \\
    \text{prior:} \quad &\gausC{\weights}{\mathbf{0}, \Reg},
\end{align}
for an arbitrary univariate likelihood, $\prob{\cdot}$, with an appropriate
transformation (inverse link) function, $\activ{\cdot}$, and parameter(s),
$\lparam$.

Naturally, both calculating the exact posterior over the weights,
$\probC{\weights}{\targ, \ins}$, and the log-marginal likelihood,
$\prob{\targ}$, for hyperparameter learning are intractable since we may have a
non-conjugate relationship between the likelihood and prior. Therefore we must
resort to approximating the true posterior and the log-marginal likelihood.

Firstly, we approximate the true posterior over weights with a mixture of $K$
diagonal Gaussians,
\begin{align}
    \probC{\weights}{\targ, \ins} &\approx \qrob{\weights}, \nonumber \\
    &= \frac{1}{K} \sum^K_{k=1} \gausC{\weights}{\pomean_k, \dpocov_k},
\end{align}
where $\dpocov_k = \diag{[\dpocovs_{k,1}, \ldots, \dpocovs_{k, D}]\T}$, which
is inspired from similar approximations made in \citet{gershman2012,
    nguyen2014automated}. This is a very flexible form for the approximate
posterior, and has the nice property that our algorithm no longer has a
\bigo{D^3} cost associated with the number of features.

Then we approximate the log marginal likelihood using auto-encoding variational
Bayes \citep{kingma2014auto}. The exact lower bound on log marginal likelihood
is,
\begin{equation}
    \elbo = \sum^N_{n=1}
    \expec{q}{\log\probC{\targs_n}{\activ{\feats_n\T\weights}, \lparam}}
    - \KL{\textstyle\frac{1}{K}\textstyle\sum_{k}
            \gausC{\weights}{\pomean_k, \dpocov_k}}
        {\gausC{\weights}{\mathbf{0}, \Reg}}.
    \label{eq:glmobj}
\end{equation}
This can be expanded,
\begin{multline}
    \elbo = \frac{1}{K} \sum^K_{k=1} \sum^N_{n=1}
    \expec{q_k}{\log\probC{\targs_n}
        {\activ{\feats_n\T\weights}, \lparam}}
    + \frac{1}{K} \sum^K_{k=1}
        \expec{q_k}{\log\gausC{\weights}{\mathbf{0}, \Reg}} \\
    + \entropy{\textstyle\frac{1}{K}\textstyle\sum_{k}
            \gausC{\weights}{\pomean_k, \dpocov_k}},
    \label{eq:glmobj_exp}
\end{multline}
but unfortunately there are two intractable integrals here, the expected log
likelihood, and the entropy of the Gaussian mixture. We can use the lower bound
on the entropy term also used in \citet{gershman2012, nguyen2014automated},
\begin{equation}
    \entropy{\textstyle\frac{1}{K}\textstyle\sum_{k}
        \gausC{\weights}{\pomean_k, \dpocov_k}} \geq
    - \frac{1}{K} \sum_{k=1}^K \log \sum_{j=1}^K \frac{1}{K}
    \gausC{\pomean_k}{\pomean_j, \dpocov_k + \dpocov_j}.
\end{equation}
We can then use the reparameterisation trick in auto-encoding variational Bayes
\citep{kingma2014auto} to sample the expected log likelihood,
\begin{equation}
    \sum^N_{n=1}
    \expec{q_k}{\log\probC{\targs_n}
        {\activ{\feats_n\T\weights}, \lparam}} \approx
    \frac{1}{L} \sum^{L}_{l=1} \sum^N_{n=1} \log\probC{\targs_n}
    {\activ{\feats_n\T\hat{\weights}_k^{\brac{l}}}, \lparam}
\end{equation}
where,
\begin{equation}
    \hat{\weights}_k^{\brac{l}} =
    \func{f}{\pomean_k, \dpocov_k, \resamp^{(l)}}{k} =
    \pomean_k + \sqrt{\dpocov_k} \odot \resamp^{(l)},
    \qquad \resamp^{(l)} \sim \gaus{\mathbf{0}, \ident{D}}.
\end{equation}
Here $\odot$ is the element-wise product. We can also use this trick to compute
approximate derivatives, $\frac{\partial}{\partial \alpha}
\expec{q(\alpha)}{\log \probC{\targs}{\alpha}} \approx \frac{1}{L} \sum^L_{l=1}
\frac{\partial}{\partial\alpha} \log \probC{\targs}{\func{f}{\alpha,
        \resamp^{(l)}}{}}$, which simplifies the implementation greatly! The
final auto-encoding variational Bayes objective for our GLM is,
\begin{multline}
    \elbo \approx \frac{1}{K} \sum^K_{k=1} \Bigg[
    \frac{1}{L} \sum^L_{l=1} \sum^N_{n=1}
    \log\probC{\targs_n}
    {\activ{\feats_n\T \hat{\weights}_k^{\brac{l}}}, \lparam}
    + \log\gausC{\pomean_k}{\mathbf{0}, \Reg}
    - \frac{1}{2} \trace{\Reg\inv\dpocov_k} \\
    - \log \sum_{j=1}^K \frac{1}{K}
    \gausC{\pomean_k}{\pomean_j, \dpocov_k + \dpocov_j}
    \Bigg].
    \label{eq:glmobj_exp}
\end{multline}
We can straight forwardly use this objective within a stochastic gradients
setting using with the tactic in Equations \eqref{eq:wsg} or \eqref{eq:wsg2}.

The most simple and accurate method for approximating the predictive
distribution, $\probC{\targs\test}{\targ, \ins, \inss\test}$ is to Monte-Carlo
sample the integral,
\begin{equation}
    \probC{\targs\test}{\targ, \ins, \inss\test} \approx
    \int \probC{\targ}{\activ{\feats\testT\weights}, \lparam}
    \frac{1}{K} \sum^K_{k=1} \gausC{\weights}{\pomean_k, \dpocov_k} d \weights.
    \label{eq:glppred}
\end{equation}
However, this integral is not particularly useful unless we wish to evaluate
known $\targ\test$ under the model. For prediction, it is more useful to
compute (using Monte-Carlo integration) the predictive expectation,
\begin{align}
    \expece{}{\targs\test} \approx&
    \int \frac{1}{K} \sum^K_{k=1} \gausC{\weights}{\pomean_k, \dpocov_k}
    \int \targs\test \probC{\targs\test}{\activ{\feats\testT\weights}, \lparam}
    d \targs\test d \weights
    \nonumber\\
    =& \int \expece{}{\probC{\targs\test}
        {\activ{\feats\testT\weights}, \lparam}}
    \frac{1}{K} \sum^K_{k=1} \gausC{\weights}{\pomean_k, \dpocov_k}
    d \weights.
    \label{eq:glmexpec}
\end{align}
Often we find $\expece{}{\probC{\targs\test}{\activ{\feats\testT\weights},
        \lparam}} = \activ{\feats\testT\weights}$, however this is only true
with with right choice and usage of the activation function. Furthermore, it is
useful to compute quantiles of the predictive density in order to ascertain the
predictive uncertainty. We start by sampling the predictive cumulative density
function, $\Prob{\cdot}$,
\begin{align}
    &\ProbC{\targs\test \leq \cdfAlph}{\targ, \ins, \inss\test} \nonumber\\
    &\qquad\approx \int
    \frac{1}{K} \sum^K_{k=1} \gausC{\weights}{\pomean_k, \dpocov_k}
    \int^{\cdfAlph}_{-\infty}
    \probC{\targs\test}{\activ{\feats\testT\weights}, \lparam}
    d \targs\test d \weights \nonumber\\
    &\qquad= \int
    \ProbC{\targs\test \leq \cdfAlph}{\activ{\feats\testT\weights}, \lparam}
    \frac{1}{K} \sum^K_{k=1} \gausC{\weights}{\pomean_k, \dpocov_k}
    d \weights.
    \label{eq:lapexpec}
\end{align}
Once we have obtained sufficient samples from the (mixture) posterior we can
obtain quantiles, $\cdfAlph$, for some chosen level of probability, $p$, using
root finding techniques. Specifically, we use root finding techniques to solve
the following for $\cdfAlph$,
\begin{equation}
    \ProbC{\targs\test \leq \cdfAlph}{\targ, \ins, \inss\test} - p = 0.
\end{equation}

\subsection{Large Scale Gaussian Process Approximation}

In \revrand we approximate Gaussian Processes \citep{Rasmussen2006} with
our standard and generalized linear models by using random feature functions
such as those of \citeauthor{rahimi2007} \citeyearpar{rahimi2007,rahimi2008}
and \cite{le2013fastfood}. They use Bochner's theorem regarding the
relationship between a kernel and the Fourier transform of a non-negative
measure (e.g.~a probability measure) that establishes the duality of the
covariance function of a stationary process and its spectral density,
\begin{align}
	\kernl(\tfourier) &= \int \specfourier(\ffourier)
    e^{i \ffourier\T  \tfourier} d \ffourier,  \\
	\specfourier(\ffourier) &= \int \kernl(\tfourier)
    e^{- i \ffourier\T \tfourier}  d \tfourier.
	\label{eq:fourier}
\end{align}
where $\kernl(\cdot)$ is a kernel function, and $\specfourier(\cdot)$ its
spectral density. \citeauthor{rahimi2007}'s  main insight
\citeyearpar{rahimi2007} is that we can approximate the kernel by constructing
`suitable' random features and Monte Carlo averaging over samples from
$\specfourier(\ffourier)$ for \emph{shift invariant} kernels,
\begin{equation}
    \kernl(\inss - \inssprime) = \kernl(\tfourier)
    \approx \frac{1}{D} \sum_{i=1}^{D} \singlefeatfunc{\inss}{i}\T\!
	\singlefeatfunc{\inssprime}{i},
	\label{eq:mcapprox}
\end{equation}
$\singlefeatfunc{\inss}{i}$ corresponds to the $i$th sample from the feature
map. An example of a radial basis kernel feature vector construction using the
above approximation is,
\begin{align}
	\nonumber
    \sbrac{\singlefeatfunc{\inss}{i}, \singlefeatfunc{\inss}{D+i}}&=
    \frac{1}{\sqrt{D}} \sbrac{\cos(\ffourier_i^T \inss),
    \sin(\ffourier_i^T \inss)}, \qquad \\
    \text{with}~\ffourier_i & \sim
    \gausC{\ffourier_i}{\mathbf{0}, \lenscale\inv \ident{d}},
\end{align}
for $i=1, \ldots, D$,  which in fact is a mapping into a $2 D$-dimensional
feature space. See Table~\ref{tab:randommappings} for some of the random kernel
approximations we use in \revrand.

\begin{table}[htb]

    \centering

    \caption{Kernels and the corresponding Fourier weight ($\ffourier_i$)
        sampling distributions for the \citet{rahimi2007}-style random bases in
        \revrand. Here GAL refers to a multivariate Laplace
        distribution~\cite{kozubowski2013multivariate}.}

    \begin{tabular}{r|c c}
        \textbf{Kernel} & $\kernl\!\brac{\inss - \inssprime}$
        & $\specfourier(\ffourier)$ \\
        \hline
        RBF &
        $\exp\brac{-\frac{\lnorm{}{\inss - \inssprime}^2}{2\lenscale^2}}$ &
            $\ffourier_i \sim \gaus{\mathbf{0}, \lenscale\inv\ident{d}}$, \\
        Laplace & $\exp\brac{-\frac{\lnorm{}{\inss-\inssprime}}{\lenscale}}$ &
            $\ffourier_i \sim \prod_d \text{Cauchy}\!\brac{\lenscale\inv}$ \\
        Cauchy &
            $\frac{1}{1+\brac{\lnorm{}{\inss-\inssprime}/\lenscale}^2}$ &
            $\ffourier_i \sim \text{GAL}\brac{1, \mathbf{0},
                \lenscale\inv\ident{d}}$ \\
        Matern $3/2$ &
            $\brac{1 + \frac{\sqrt{3}\lnorm{}{\inss-\inssprime}}{\lenscale}}
            \exp\brac{-\frac{\sqrt{3}\lnorm{}{\inss-\inssprime}}{\lenscale}}$ &
            $\ffourier_i \sim
            \text{t}_{\nu=3}\!\brac{\mathbf{0}, \lenscale\inv\ident{d}}$ \\
        Matern $5/2$ &
            $\brac{1 + \frac{\sqrt{5}\lnorm{}{\inss-\inssprime}}{\lenscale} +
                \frac{5\lnorm{}{\inss - \inssprime}^2}{3\lenscale^2}}
            \exp\brac{-\frac{\sqrt{5}\lnorm{}{\inss-\inssprime}}{\lenscale}}$ &
            $\ffourier_i \sim
            \text{t}_{\nu=5}\!\brac{\mathbf{0}, \lenscale\inv\ident{d}}$ \\
        \hline
    \end{tabular}

    \label{tab:randommappings}

\end{table}

All of the kernel functions in Table \ref{tab:randommappings} have length scale
hyperparameters, and in \revrand these can be isotropic or anisotropic
length scales that are learned alongside the other hyperparameters. We have
also implemented a few other variants of these random basis functions, namely
the Fastfood \citep{le2013fastfood} and A la Carte \citep{yang2014} basis
functions that improve scalability and representational flexibility
respectively. We refer the reader to our demo notebooks for further discussion
and experiments comparing these approximate basis functions and the kernels
they approximate.

TODO: a few figures of the approximate kernels?

TODO: Basis CONCATENATION and how regularizer act as amplitude parameters

An important feature of \revrand is that we can combine these random features
(and non-random features) to build even more expressive features. In
particular, we support basis \emph{concatenation} while still allowing basis
function hyperparameter learning,
\begin{equation}
    \func{\featsym}{\ins, \boldsymbol\param}{} = \sbrac{
        \func{\featsym}{\ins, \param_1}{1},
        \func{\featsym}{\ins, \param_2}{2},
        \ldots,
        \func{\featsym}{\ins, \param_P}{P}
    }
\end{equation}
This is in fact equivalent to kernel addition,
\begin{equation}
    TODO
\end{equation}
We also support \emph{partial application} of basis functions to certain
dimensions of the inputs ($\ins$), while also allowing concatentation, e.g,
\begin{equation}
    \func{\featsym}{\ins, \boldsymbol\param}{} = \sbrac{
        \func{\featsym}{\ins_{1:10}, \param_1}{1},
        \func{\featsym}{\ins_{5:D}, \param_2}{2},
        \ldots,
        \func{\featsym}{\ins, \param_P}{P}
    }
\end{equation}
Where the subscript of $\ins$ denotes (arbitrary) column slices. See \revrand's
documentation for how to use this features.

\section{Experiments}

\subsection{Boston Housing Regression}

\begin{table}[htb]

    \centering
    \begin{tabular}{r|c c}
        \textbf{Algorithm} & \textbf{R-square} & \textbf{MSLL} \\
        \hline
        \emph{SLM} & 0.9018 (0.0134) & -1.504 (0.1191) \\
        % \emph{GLM} & 0.8340 (0.0491) & -0.8763 (0.1242) \\
        \emph{GLM} & 0.8411 (0.0491) & -0.9209 (0.1530) \\
        GP & \textbf{0.9027 (0.0137)} & \textbf{-1.1792 (0.1581)} \\
        RF & 0.8467 (0.0709) & N/A \\
        SVR & 0.6295 (0.0863) & N/A \\
        \hline
    \end{tabular}

\end{table}

\subsection{Handwritten Digits Classification}

\begin{table}[htb]

    \centering
    \begin{tabular}{r|c c}
        \textbf{Algorithm} & \textbf{Log-loss} & \textbf{Error (\%)} \\
        \hline
        \emph{GLM} & 0.1138 & \textbf{2.07} \\
        Logistic & 0.1734 & 3.62 \\
        SVC & \textbf{0.1003} & 6.99 \\
        GPC &  0.1405 & 2.59 \\
        RF & 0.1368 & 2.72 \\
        \hline
    \end{tabular}

\end{table}

\subsection{SARCOS Regression}

1 million iterations

\begin{table}[htb]

    \centering
    \begin{tabular}{r|c|c c}
        \textbf{Algorithm} & \textbf{Approximation size} & \textbf{SMSE} &
        \textbf{MSLL} \\
        \hline
        \emph{GLM} & 256 & 0.0358 & -1.6644 \\
        & 512 & 0.0252 & -1.8397 \\
        & 1024 & 0.0207 & -1.9341 \\
        & 2048 & 0.0171 & -2.0243 \\
        & 8192 & 0.0152 & -1.9804 \\
        \hline
    \end{tabular}

\end{table}

\bibliography{report}

\end{document}


% !TEX root=/home/tavant/these/manuscript/src/manuscript.tex

\section{Characteristics and performances of the LPPic simulation code }
\label{sec-lppic}

\LPPic is the electrostatic particle-in-cell simulation code developed at \ac{LPP} to study principally magnetized plasma thruster, such as the \ac{HET}.
Written in Fortran (2008), it uses the Message Passing Interface (MPI) to parallelize the computational load using a domain decomposition.
The principle of the domain decomposition is that each Central Processing Unit (CPU) knows a small chunk of the whole domain.
We use {\sc Hypre}'s multigrid solver to solve Poisson equation in \ac{2D} \citep{falgout2002}.

The main data input/output is done using the HDF5 format, which enables parallel access to a hierarchical file.
The output data does not yet use the open standard for particle-mesh data files (openPMD).
Thus, the data post processing is done using a python package.

The use of these efficient libraries allows us to obtain a good scalability.
\Cref{fig-strongperfo} shows the speed up measured for the Strong Scalability test.
The Strong scalability compares the performances of the execution of the task at fixed workload with varying the number of CPUs.
The theoretical speedup accessible is given by \citet{amdahl1967} as
\begin{equation} \label{eq-amdahl}
  S = \frac{1}{1 - \alpha + \frac{\alpha}{s}}
\end{equation}
with $\alpha$ the proportion of the task that is parallel, and $s$ the speedup of the parallelized part.
A perfectly parallel code has $\alpha=1$, hence a speedup $S=s$.

In \cref{fig-strongperfo} are shown the speedup for the overall simulation, as well as the main functions of the \ac{PIC} algorithm (Charge deposition to the mesh, Particle pusher, Monte Carlo Collision and the Poisson solver).
The scalability test is realized over two \ac{2D} radial-azimuthal cases.
The bigger case uses a mesh of $1000\times1024$ cells with $140$ particles per cell, which corresponds to a typical simulation case.
Unfortunately, this case cannot be run with less than 240 CPUs because of the maximum memory available per CPU.
Hence, we use a case four times smaller to test fewer number of CPU.
The simulation is run over 2000 time steps.
More information on the scalability is given in \cref{an-scalability}.
\begin{figure}[hbt]
  \centering
  \includegraphics[width=\textwidth]{StrongScalingOccigen_speedup.pdf}
  \caption{Strong scalability profile of \LPPic with a small and a large job on the Cluster Occigen (CINES). In addition to the overall performance, the profiles of the main sub-functions of the \acs{PIC} loops are given. }
  \label{fig-strongperfo}
\end{figure}

We can see that the overall performance of the code is very good, as it scales as if it is parallelized at 91\% from Amdahl's law.
Most of the sub-functions have higher speed-ups, but the Poisson solver does not scale at all.
It is due to the size of the chunk of domain of each CPU, which is too small.
A way to improve the performances of \LPPic would be to use an hybrid parallelization with the shared memory openMP library.
But the current performance is acceptable, as the typical simulation cases presented in this manuscript run in around one week (7 days) on 360 CPUs.

\vspace{1em}

The development of the  simulation code \LPPic started in 2015 by Vivien Croes.
I joined the development with Romain Lucken in 2016, and Thomas Charoy started in 2017.
Because of the number of contributors, and our different experiences in software development, we quickly set a Continuous Integration (CI) pipeline on a self hosted server using Teamcity.
This allows a systematic verification of the modifications by the mean of unit, mezzanine and integration tests \citep{turner2016}.

The validation of the code is done using the Benchmark of Turner et al.(2013) \citenum{turner2013}.
While it allows to validate a significant part of the \ac{PIC} code, it is not magnetized and only in \ac{1D}.
Unfortunately, this is the only Benchmark available.
Hence, the comparison of several codes on a \ac{2D} magnetized  has been conducted in 2019 by Thomas Charoy in order to create a new Benchmark to validate such simulation codes, in \citet{charoy2019}.

Since 2015, \LPPic has grown to 23000 lines of code, has been used on more than 5 different clusters of various architectures, and has been used to produce several publications \citep{croes2017a,croes2018,tavant2018,tavant2019,lucken2018,lucken2019}.


%%
%% Automatically generated file from DocOnce source
%% (https://github.com/hplgit/doconce/)
%%
%%
% #ifdef PTEX2TEX_EXPLANATION
%%
%% The file follows the ptex2tex extended LaTeX format, see
%% ptex2tex: http://code.google.com/p/ptex2tex/
%%
%% Run
%%      ptex2tex myfile
%% or
%%      doconce ptex2tex myfile
%%
%% to turn myfile.p.tex into an ordinary LaTeX file myfile.tex.
%% (The ptex2tex program: http://code.google.com/p/ptex2tex)
%% Many preprocess options can be added to ptex2tex or doconce ptex2tex
%%
%%      ptex2tex -DMINTED myfile
%%      doconce ptex2tex myfile envir=minted
%%
%% ptex2tex will typeset code environments according to a global or local
%% .ptex2tex.cfg configure file. doconce ptex2tex will typeset code
%% according to options on the command line (just type doconce ptex2tex to
%% see examples). If doconce ptex2tex has envir=minted, it enables the
%% minted style without needing -DMINTED.
% #endif

% #define PREAMBLE

% #ifdef PREAMBLE
%-------------------- begin preamble ----------------------

\documentclass[%
oneside,                 % oneside: electronic viewing, twoside: printing
final,                   % draft: marks overfull hboxes, figures with paths
10pt]{article}

\listfiles               %  print all files needed to compile this document

\usepackage{relsize,makeidx,color,setspace,amsmath,amsfonts,amssymb}
\usepackage[table]{xcolor}
\usepackage{bm,ltablex,microtype}

\usepackage[pdftex]{graphicx}

\usepackage{ptex2tex}
% #ifdef MINTED
\usepackage{minted}
\usemintedstyle{default}
% #endif

\usepackage[T1]{fontenc}
%\usepackage[latin1]{inputenc}
\usepackage{ucs}
\usepackage[utf8x]{inputenc}

\usepackage{lmodern}         % Latin Modern fonts derived from Computer Modern

% Hyperlinks in PDF:
\definecolor{linkcolor}{rgb}{0,0,0.4}
\usepackage{hyperref}
\hypersetup{
    breaklinks=true,
    colorlinks=true,
    linkcolor=linkcolor,
    urlcolor=linkcolor,
    citecolor=black,
    filecolor=black,
    %filecolor=blue,
    pdfmenubar=true,
    pdftoolbar=true,
    bookmarksdepth=3   % Uncomment (and tweak) for PDF bookmarks with more levels than the TOC
    }
%\hyperbaseurl{}   % hyperlinks are relative to this root

\setcounter{tocdepth}{2}  % levels in table of contents

% prevent orhpans and widows
\clubpenalty = 10000
\widowpenalty = 10000

% --- end of standard preamble for documents ---

% insert custom LaTeX commands...

\raggedbottom
\makeindex
\usepackage[totoc]{idxlayout}   % for index in the toc
\usepackage[nottoc]{tocbibind}  % for references/bibliography in the toc

%-------------------- end preamble ----------------------

\begin{document}

% matching end for #ifdef PREAMBLE
% #endif

\newcommand{\exercisesection}[1]{\subsection*{#1}}

% ------------------- main content ----------------------

% ----------------- title -------------------------

\thispagestyle{empty}

\begin{center}
{\LARGE\bf
\begin{spacing}{1.25}
Day 3: Exercise set 2
\end{spacing}
}
\end{center}

% ----------------- author(s) -------------------------

\begin{center}
{\bf Data Analysis and Machine Learning for Nuclear Physics${}^{}$} \\ [0mm]
\end{center}

\begin{center}
% List of all institutions:
\end{center}

% ----------------- end author(s) -------------------------

% --- begin date ---
\begin{center}
Jun 25, 2020
\end{center}
% --- end date ---

\vspace{1cm}

\subsection{Day three exercises}

\paragraph{Exercise 1.}
This exercise is a continuation of exercise 3 from exercise set 1. We will
use the same function to generate our data set, still staying with a
simple function $y(x)$ which we want to fit using linear regression,
but now extending the analysis to include the Ridge and the Lasso
regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods, see the \href{{https://compphysics.github.io/MachineLearning/doc/pub/Regression/html/Regression-bs.html}}{regression slides}).

We will thus again generate our own dataset for a function $y(x)$ where
$x \in [0,1]$ and defined by random numbers computed with the uniform
distribution. The function $y$ is a quadratic polynomial in $x$ with
added stochastic noise according to the normal distribution $\cal{N}(0,1)$.

The following simple Python instructions define our $x$ and $y$ values (with 100 data points).
\bpycod
x = np.random.rand(100,1)
y = 5*x*x+0.1*np.random.randn(100,1)
\epycod

\begin{enumerate}
\item (1a) Write your own code for the Ridge method (see chapter 3.4 of Hastie \emph{et al.}, equations (3.43) and (3.44)) and compute the parametrization for different values of $\lambda$. Compare and analyze your results with those from exercise 2. Study the dependence on $\lambda$ while also varying the strength of the noise in your expression for $y(x)$.

\item (1b) Repeat the above but using the functionality of \textbf{Scikit-Learn}. Compare your code with the results from \textbf{Scikit-Learn}. Remember to run with the same random numbers for generating $x$ and $y$.

\item (1c) Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parameterizing our function with a second-order polynomial). We will use standard linear regression and the Ridge regression.  You can now opt for either writing your own function or using \textbf{Scikit-Learn} to find the parameters $\beta$. From your results calculate the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})+\lambda\hat{I})^{-1}$). Discuss the results of these variances as functions of $\lambda$. In particular, try to link your discussion with the discussion in Hastie \emph{et al.} and their figures 3.10 and  3.11. \textbf{Scikit-Learn} may not provide the variance of the parameters $\beta$. This needs to be checked. With your own code you can however do so.

\item (1d) Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie \emph{et al.}. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of \textbf{scikit-learn}.  We recommend the latter since we have not yet discussed how to solve the Lasso equations numerically. Also, you do not need to compute the variance of the parameters $\beta$ but you can extract their values and study their behavior as functions of the regularization parameter $\lambda$.

\item (1e) Finally, using \textbf{Scikit-Learn} or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
\end{enumerate}

\noindent
\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
\]
and the $R^2$ score function.
If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as
\[
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
\]
where we have defined the mean value  of $\hat{y}$ as
\[
\bar{y} =  \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
\]
Discuss these quantities as functions of the variable $\lambda$ in the Ridge and Lasso regression methods.

\subsection{Exercise 2}

A much used approach before starting to train the data is  to preprocess our
data. Normally the data may need a rescaling and/or may be sensitive
to extreme values. Scaling the data renders our inputs much more
suitable for the algorithms we want to employ.

\textbf{Scikit-Learn} has several functions which allow us to rescale the
data, normally resulting in much better results in terms of various
accuracy scores.  The \textbf{StandardScaler} function in \textbf{Scikit-Learn}
ensures that for each feature/predictor we study the mean value is
zero and the variance is one (every column in the design/feature
matrix).  This scaling has the drawback that it does not ensure that
we have a particular maximum or minimum in our data set. Another
function included in \textbf{Scikit-Learn} is the \textbf{MinMaxScaler} which
ensures that all features are exactly between $0$ and $1$. The

The \textbf{Normalizer} scales each data
point such that the feature vector has a euclidean length of one. In other words, it
projects a data point on the circle (or sphere in the case of higher dimensions) with a
radius of 1. This means every data point is scaled by a different number (by the
inverse of it’s length).
This normalization is often used when only the direction (or angle) of the data matters,
not the length of the feature vector.

The \textbf{RobustScaler} works similarly to the StandardScaler in that it
ensures statistical properties for each feature that guarantee that
they are on the same scale. However, the RobustScaler uses the median
and quartiles, instead of mean and variance. This makes the
RobustScaler ignore data points that are very different from the rest
(like measurement errors). These odd data points are also called
outliers, and might often lead to trouble for other scaling
techniques.

It also common to split the data in a \textbf{training} set and a \textbf{testing} set. A typical split is to use $80\%$ of the data for training and the rest
for testing. This can be done as follows with our design matrix $\bm{X}$ and data $\bm{y}$ (remember to import \textbf{scikit-learn})
\bpycod
# split in training and test data
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2)
\epycod
Then we can use the stadndard scale to scale our data as
\bpycod
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
\epycod

In this exercise we want you to to compute the MSE for the training
data and the test data as function of the complexity of a polynomial,
that is the degree of a given polynomial. We want you also to compute the $R2$ score as function of the complexity of the model for both training data and test data.  You should also run the calculation with and without scaling.

One of
the aims is to reproduce Figure 2.11 of \href{{https://github.com/CompPhysics/MLErasmus/blob/master/doc/Textbooks/elementsstat.pdf}}{Hastie et al}.
We will also use Ridge and Lasso regression.

Our data is defined by $x\in [-3,3]$ with a total of for example $100$ data points.
\bpycod
np.random.seed()
n = 100
maxdegree = 14
# Make data set.
x = np.linspace(-3, 3, n).reshape(-1, 1)
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
\epycod
where $y$ is the function we want to fit with a given polynomial.

\begin{enumerate}
\item (2a) Write a first code which sets up a design matrix $X$ defined by a fifth-order polynomial.  Scale your data and split it in training and test data.

\item (2b) Perform an ordinary least squares and compute the means squared error and the $R2$ factor for the training data and the test data, with and without scaling.

\item (2c) Add now a model which allows you to make polynomials up to degree $15$.  Perform a standard OLS fitting of the training data and compute the MSE and $R2$ for the training and test data and plot both test and training data MSE and $R2$ as functions of the polynomial degree. Compare what you see with Figure 2.11 of Hastie et al.~Comment your results. For which polynomial degree do you find an optimal MSE (smallest value)?

\item (2d) Repeat part (2c) but now using Ridge regressions with various hyperparameters $\lambda$. Make the same plots for the optimal $\lambda$ value for each polynomial degree. Compare these results with those from the standard OLS approach.
\end{enumerate}

\noindent
\subsection{Example of how to solve the previous exercise}

\bpycod
import matplotlib.pyplot as plt
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error

n = 100
# Make data set.
x = np.linspace(-3, 3, n).reshape(-1, 1)
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Decide which values of lambda to use
nlambdas = 500
lambdas = np.logspace(-3, 5, nlambdas)

estimated_mse_sklearn = np.zeros(nlambdas)
i = 0
for lmb in lambdas:
    clf_ridge = Ridge(alpha=lmb).fit(X_train_scaled, y_train)
    yridge = clf_ridge.predict(X_test_scaled)
    estimated_mse_sklearn[i] = mean_squared_error(y_test, yridge)
    i += 1
plt.figure()
plt.plot(np.log10(lambdas), estimated_mse_sklearn, label = 'Ridge MSE')
plt.xlabel('log10(lambda)')
plt.ylabel('MSE')
plt.legend()
plt.show()

\epycod

\subsection{And now with OLS only and Bootstrap}

\bpycod

import matplotlib.pyplot as plt
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.utils import resample
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import PolynomialFeatures
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error

n = 100
n_boostraps = 100
maxdegree = 14

# Make data set.
x = np.linspace(-3, 3, n).reshape(-1, 1)
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
error = np.zeros(maxdegree)
bias = np.zeros(maxdegree)
variance = np.zeros(maxdegree)
polydegree = np.zeros(maxdegree)

# Make data set.
x = np.linspace(-3, 3, n).reshape(-1, 1)
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

for degree in range(maxdegree):
    model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
    y_pred = np.empty((y_test.shape[0], n_boostraps))
    for i in range(n_boostraps):
        x_, y_ = resample(X_train_scaled, y_train)
        y_pred[:, i] = model.fit(x_, y_).predict(X_test_scaled).ravel()

    polydegree[degree] = degree
    error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
    bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
    variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
    print('Polynomial degree:', degree)
    print('Error:', error[degree])
    print('Bias^2:', bias[degree])
    print('Var:', variance[degree])
    print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))

plt.plot(polydegree, error, label='Error')
plt.plot(polydegree, bias, label='bias')
plt.plot(polydegree, variance, label='Variance')
plt.legend()
plt.show()

\epycod

% ------------------- end of main content ---------------

% #ifdef PREAMBLE
\end{document}
% #endif


This chapter describes the proposed system for sense discovery problem. First, I would like to present the tools and operations in Section \ref{section:operations}, then clarify their different variations and combinations in Section \ref{section:variations}. The pipeline starts with calculating substitute distribution of a word in a given context explained in Section \ref{section:subs-dist}. Then, a list of substitutes are generated by using this substitute distribution and paired with context id. Context id (e.g., $car_1$) is basically a unique identifier that helps to distinguish each context from others. The euclidean embedding algorithm SCODE uses this co-occurrence data and outputs the embedding for each word. Section \ref{section:scode} clarifies co-occurrence modeling step and Section \ref{section:diff-approaches} defines three types of approaches: local approach, part-of-speech based approach, global approach. Using these word embeddings and the substitute distributions I obtained word representations and context representations which will be clarified in Section \ref{section:representations}. When each context is represented in spherical space, then $k$-means algorithm is employed detailed in Section \ref{section:clustering}.

\section{Tools and Operations}
\label{section:operations}

\subsection{Data Enrichment}
\label{section:enrichment}

Sense annotation is a hard task. For the lexical sample tasks, organizers provide a test data annotated by humans which is used for system evaluation. The systems should use a training data if provided, or corpora for sense induction. Typically test data does not contain enough instance for sense induction. Data enrichment aims to increase the number of instances of target words. Assuming that our target word is $book$ in noun form. I randomly fetch arbitrary number of additional contexts from ukWaC where our target word occurs with the same part-of-speech tag. This implies that I skip those sentences in which the word $book$ functions as a verb. \\

For all experiments in Section \ref{chapter:experiments}, the number of additional contexts is fixed and it is equal to $20000$, and uKWaC is used as a corpus for data enrichment.

\subsection{Substitute Word Distributions}
\label{section:subs-dist}

In this section, I will explain how substitute word distribution captures the paradigmatic relations of word token, and how it is computed. \cite{sahlgren2006word} investigates the syntagmatic and paradigmatic relations captured by the Vector Space Models. His paradigmatic representation models word types using co-occurrence counts. Following the methodology used in \citet{yatbaz2012learning} and \citet{yatbaz2012learning}, I represent the context of word tokens using probabilities of likely substitutes. The substitute word distribution is a probability distribution of observing a word in the context of a word token. Context of a target word in a sentence is defined as the sequence of words in the window of size $2n - 1$ centered at the position of the
target word token. The context excludes the target word token.

\begin{quote}
``\emph{Accordingly, the last chapter of this \textbf{book} is devoted to our reflections on the impact of channel integration on certain public policies issues.}''
\end{quote}

For instance, in the sentence above, the context of the word token `\emph{book}', when $n = 4$, is ``\emph{chapter of this --- is devoted to}'', where `\emph{---}' denotes the target word token. \\

Assuming the position of the target word token is $0$, the context
spans from positions $-n + 1$ to $n - 1$ and the probability of
observing each type $w$ in our vocabulary in the context of the target
word token is computed using the following equation:
\begin{eqnarray}
  P(w_{0} = w | c_{w_{0}}) & \propto & P(w_{-n + 1} \dots w_{0} \dots w_{n - 1}) \label{eq:ngram1} \\
  & = & P(w_{-n + 1}) \dots P(w_{0} | w_{-n + 1}^{-1}) \dots P(w_{n - 1} | w_{-n + 1}^{n - 2}) \label{eq:ngram2} \\
  & \approx & P(w_{-n + 1}) \dots P(w_{0} | w_{-n + 1}^{-1}) \dots P(w_{n - 1} | w_{0}^{n - 2}) \label{eq:ngram3} \\
  & \propto & P(w_{0} | w_{-n + 1}^{-1}) \dots P(w_{n - 1} | w_{0}^{n - 2}) \label{eq:ngram4}
\end{eqnarray}
Where $w_{i}^j$ is the word sequence from $w_i$ to $w_j$ (for $i < j$)
and $c_{w_{0}}$ is the context of the target word token at position
$0$ of length $2n - 1$, meaning $w_{-n+1}^{-1}$ to the left and
$w_{1}^{n-1}$ to the right of $w_{0}$. \\

In the Equation \ref{eq:ngram1}, the right-hand side is proportional
to the left-hand side because $P(c_{w_{0}})$ is independent of any
given word $w$ for $w_{0}$.  Using the chain rule,
Equation \ref{eq:ngram2} is obtained from the Equation
\ref{eq:ngram1}.  By applying $n^{th}$-order Markov assumption, only the
closest $n - 1$ words in each term of the Equation \ref{eq:ngram2} are
used and others excluded. Result is the Equation \ref{eq:ngram3}.  The Equation \ref{eq:ngram4} is proportional to the Equation \ref{eq:ngram3}
because terms which are do not depend on $w_{0}$ is fixed since the
context of the word at position $0$ is fixed.  Near the boundaries of
the sentence, specifically the first and last $n - 1$ words,
appropriate terms of the Equation \ref{eq:ngram4} are truncated or
dropped (e.g. if $0$ is the first word of a sentence, $P(w_{0} | w_{-n
  + 1}^{-1})$ becomes $P(w_{0})$). \\

The probabilities required to compute the Equation \ref{eq:ngram4} can
be obtained from an $n$-gram language model. For this step, I used SRILM (\cite{stolcke02srilm}) to build a 4-gram language model with interpolated Kneser-Ney discounting using the uKWaC, a web corpus with 2 billion tokens. Words that were observed less than 2 times in the language model training data were replaced by <unk> tags, which gave us a vocabulary size of 4,254,946. For computational efficiency only the top 100 substitutes and their probabilities were computed for each position in the context of a target word using the FASTSUBS algorithm (\cite{fastsubs}). These top 100 probabilities were normalized to add up to 1.0 giving us a legitimate probability distribution for a context.

As you seen in Table \ref{table:subs}, the high probability substitutes reflect both semantic and syntactic properties of the context of a target word in the sentence above.

\begin{table}
\begin{center}
    \begin{tabular}{ l | r }  \Xhline{2\arrayrulewidth}
     \bf Substitute Word & \bf Probability \\  \Xhline{2\arrayrulewidth}
     book & .638 \\
     section & .139 \\
     paper & .089 \\
     volume & .053 \\
     document & .014 \\
     report & .013 \\
     work & .012 \\
     tutorial & .005 \\
     edition & .003 \\
     biography & .001 \\
     \Xhline{2\arrayrulewidth}
    \end{tabular}
\end{center}
    \caption[Top 10 substitutes and their probabilities for a context]{\label{table:subs}{Top 10 substitutes and their probabilities for the context, ``\emph{chapter of this --- is devoted to}''.}}
\end{table}

\subsection{Discretization of Substitute Word Distribution}
\label{section:sampling}

Although the context of a target word was modeled as a substitute word distribution, the Euclidean co-occurrence modeling algorithm which will be introduced in the next section expects its input as discrete objects. I will list two methods to discretize substitute word distribution into discrete counterpart. \cite{sert2013word} proposed a method which creates proximity matrix of substitute word distributions, and  ids of $k$-nearest neighbors of each distribution. The second method introduced in \cite{yatbaz2012learning} draws $k$ word types with replacement from the substitute word distributions. This method is computationally cheaper than the former one, however, the number of samples should not be small and requires attention to pick. Too small $k$ values make this setting not to represent the distribution, however, too large $k$ values make the following steps computationally expensive. From my preliminary  experiments, 100 samples enough for each context. Thus, for each context, I sampled 100 word types from their correspondent substitute distributions.

\subsection{Co-occurrence Modeling with S-CODE algorithm}
\label{section:scode}

In this section, I give a brief introduction for the Symmetric Interaction Model of the Co-occurrence Data Embedding (CODE, \cite{globerson2004euclidean}) and its extension Spherical Co-Occurrence Data Embedding (S-CODE, \cite{Maron2010}). S-CODE randomly initializes co-occurrence data generated using substitute word distributions and discretization operation explained in the previous section into $d$ dimensional Euclidean sphere. After the algorithm converges, the contexts with similar substitute word distributions are close to each other. In Section \ref{section:clustering}, I will explain how S-CODE output used, and how it combined with substitute word distributions in order to obtain high quality context embeddings. \\

Let $X$ and $Y$ be two categorical variables with finite cardinality
$|X|$ and $|Y|$.  We observe a set of pairs $\{x_i, y_i\}_{i=1}^n$
drawn IID from the joint distribution of $X$ and $Y$.  These pairs are
summarized by the empirical distributions $\bar{p}(x,y)$, $\bar{p}(x)$
and $\bar{p}(y)$. The task is to find embeddings $\phi(x)$ and
$\psi(y)$ for each unique variable $x$ and $y$. These embeddings require to reflect the statistical relationship between the variables $x$ and $y$ such that pairs occur together frequently should close to each other. hence, the square of Euclidean distance, \mbox{$d^2_{x,y} = |\phi(x) - \psi(y)|^2$}, should be small in $d$ dimensional space. \\

Later, \citet{Maron2010} constrain the embedding to lie on a high dimensional
unit sphere so that enables to approximate the partition function which is a dynamic variable, $Z$, using a constant, $\bar{Z}$, which arises from a coarse approximation in which all pairs of embedded variables are distributed uniformly and independently on the sphere. This constraint allows to work on large datasets by avoiding to calculate $Z$ after each gradient step. Thus, in this thesis I used Spherical CODE:

\begin{equation} \label{eq:marginal-marginal-model}
  p(x,y) = \frac{1}{Z} \bar{p}(x) \bar{p}(y) e^{-d^2_{x,y}}
\end{equation}
where $Z = \sum_{x,y} \bar{p}(x) \bar{p}(y) e^{-d^2_{x,y}}$ is the
normalization term.  We can express the log-likelihood of the joint
distribution over all embeddings $\phi$ and $\psi$ as the following:

\begin{eqnarray} \label{eq:log-likelihood}
  \ell(\phi,\psi) & = & \sum_{x,y} \bar{p}(x,y) \log p(x,y) \nonumber \\
  & = & \sum_{x,y} \bar{p}(x,y) (-\log Z + \log \bar{p}(x) \bar{p}(y) - d^2_{x,y}) \nonumber \\
  & = & -\log Z + const - \sum_{x,y} \bar{p}(x,y) d^2_{x,y}
\end{eqnarray}

The gradient of the log-likelihood depends on the sum of embeddings
$\phi(x)$ and $\psi(y)$, for $x \in X$ and $y \in Y$, and to maximize
the log-likelihood, \cite{Maron2010} use a gradient-ascent
approach. The gradient is as follows:

\begin{equation} \label{eq:gradient-phi}
  \frac{\partial \ell(\phi,\psi)}{\partial \phi(x)} =%
  \sum_{y} 2 \bar{p}(x,y) {[}\psi(y) - \phi(x){]} +%
  \frac{1}{Z} \sum_{y} \bar{p}(x) \bar{p}(y) {[}\phi(x) - \psi(y){]} e^{-d^2_{x,y}}
\end{equation}

\begin{equation} \label{eq:gradient-psi}
  \frac{\partial \ell(\phi,\psi)}{\partial \psi(y)} =%
  \sum_{x} 2 \bar{p}(x,y) {[}\phi(x) - \psi(y){]} +%
  \frac{1}{Z} \sum_{x} \bar{p}(x) \bar{p}(y) {[}\psi(y) - \phi(x){]} e^{-d^2_{x,y}}
\end{equation}

The first sum, the gradient of the part with $d^2_{x,y}$ in
(\ref{eq:log-likelihood}), in (\ref{eq:gradient-phi})
{[}(\ref{eq:gradient-psi}){]} acts attraction force between the
$\phi(x)$ ($\psi(y)$) and all the embeddings $\psi$ ($\phi$) in
proportion to respective joint empirical probabilities $\bar{p}(x,y)$. The second sum, the gradient of $-\log Z$ in
(\ref{eq:log-likelihood}), in (\ref{eq:gradient-phi})
{[}(\ref{eq:gradient-psi}){]} acts a repulsion force between the
$\phi(x)$ ($\psi(y)$) and all the embeddings $\psi$ ($\phi$) in
proportion to respective marginal empirical probabilities $\bar{p}(x)$
and $\bar{p}(y)$. \\

In this work, I use S-CODE with sampling based stochastic gradient ascent, a constant approximation of $Z$ (which is 0.166) and randomly initialized $\phi$ and $\psi$ vectors in 100 dimensional sphere ($d = 100$). The algorithm stops if the difference between two consecutive likelihood values is less than 0.0001.

\subsection{Clustering}
\label{section:clustering}

In this thesis, I tested three different clustering algorithms: Weighted $k$-means which is $k$-means algorithm with (optional) instance weights with careful initialization (\cite{arthur2007k})\footnote{An efficient implementation can be found in https://github.com/ai-ku/wkmeans}. In literature, there are two common methods to specify $k$, required parameter for $k$-means algorithm: \emph{Gap Statistics} (\cite{tibshirani00estimating}) and Pham’s heuristic (\cite{pham2005selection}). \\

I also tried Spherical Clustering (\cite{shi2000normalized}) and DBSCAN (\cite{ester1996density}). The representations I used for clustering will be explain in the following section \ref{section:representations} and the performance of these algorithms will compare in Chapter \ref{chapter:experiments}.

\section{Variations and combinations}
\label{section:variations}

In the previous section I explained the pipeline of operations I used for Word Sense Induction problem. In this section, I will explain the combinations of operations and their variations. In Section \ref{section:scode} we successfully obtain embeddings that reflect the statistical relations by using S-CODE algorithm. The following section describes how can we use these embeddings to acquire context embeddings. After that, in Section \ref{section:diff-approaches} I will discuss local, POS-based, and global approach for the co-occurrence modeling step and the clustering step.

\subsection{Different Representations}
\label{section:representations}

\subsection{Different approaches}
\label{section:diff-approaches}

The following three approaches tested in the thesis:

\paragraph{Local approach:} In the local approach, clustering is employed for each target word, separately. In other words,