\chapter{Implementation of Non-dyson ADC}
In this chapter I describe the implementation of second order Non-dyson ADC of restricted Electron Affinity calculation in development version of \emph{adcman}.
\emph{Adcman} is a part of \emph{Q-Chem} package, which is a general-purpose electronic structure package based on C++ and Fortran.
It contains a variety of established and new methods implemented using innovative algorithms that enable fast calculations of large systems.
It's features include:
\begin{itemize}
	\item Variety of local, GGA, mGGA, hybrid, double-hybrid, dispersion-corrected, range separated functionals for DFT calculation.
	\item TDDFT and spin-flip-TDDFT
	\item Continuous Fast Multipole Method in HF and DFT
	\item Linear-scaling HF-exchange method
	\item Fourier transform Coulomb method
	\item Fast numerical integration of exchange-correlation with multiresolution exchange-correlation (mrXC)
	\item MP2
	\item SCS and SOS MP2
	\item CCD, QCISD, CCSD, OOCCD, VOOCCD
	\item EOM-XX-CCSD methods for open-shell and electronically excited species
	\item ADC methods
	\item CIS, TDDFT, CIS(D), and SOS-CIS(D) methods for excited states
\end{itemize}

\begin{figure}[h]
	\centering
	\includegraphics[width=\textwidth]{./figures/adcman.png}
	\caption{Overview of implementation of ADC methods in \emph{Q-Chem}}
	\label{codestucture}
\end{figure}

Then I will explain the overall code structure and then focus on my implementation part.
Finally I will present the results calculated by my implementation and compare them with CCSD.

\section{Code Overview and Structure}
The overview of implementation of ADC methods is in Fig. \ref{codestucture}.
All modules are written in the object-oriented C++ programming language.
The adcman module contains all top-level routines and methods necessary for an ADC calculation.
It coordinates the calculation process, sets up the solver and writes the results into an output file.
I implemented restricted calculation of EA part in adcman.

The next sub-level contains electronic structure related libraries and utilities.
\emph{liblegacy} is the gateway between the ADC code and the Q-Chem interface.
These routines help to import data from other Q-Chem modules, for example all SCF results and MOs, which are needed for the ADC calculation.
The \emph{libmo} module contains routines to set up the MO spaces, integrals and symmetry using the information imported from Q-Chem via the \emph{liblegacy} interface.
Hence, both \emph{libmo} and \emph{liblegacy} are important for ADC calculations, because all integrals and Fock-matrix elements are indexed and created corresponding to the additional restriction of the core space.
In the \emph{libadc} module, all ADC equations are implemented explicitly.
I also implemented restricted calculation of EA part in \emph{libadc}.

Furthermore, \emph{libmo} transforms the imported data in a format being compatible to the \emph{libtensor} \cite{libtensor} interface, which can be called the heart of the albegraic computation, since it contains all numerical routines to perform tensor algebra, which is the most time-consuming part all most of the quantum chemistry calculations.
\emph{libtensor} is specially designed for Post-Hartree-Fock electronic structure methods by including consideration of symmetries in the tensor structure.
These symmetries include Abelian point group symmetry, spin symmetry and other inner symmetry like that of two-electron integrals.
\begin{equation}
	\langle i j\| a b\rangle=-\langle j i\| a b\rangle=-\langle i j \| b a\rangle=\langle j i\| b a\rangle
\end{equation}
Thus, tensors of arbitrary order and size are stored in a blockwise manner to different kinds of symmetry to limit the amount of required memory and decrease the time needed.
The tensor operations in \emph{libtensor} are fully parallelized and efficiently implemented.

The second important numerical library is \emph{libsolve}, which contains all routines of
generic solvers, e.g the Davidson algorithm, which are called from the top-level adcman
routines. \emph{libsolve} is also based on the \emph{libtensor} interface. At last, there are low-level
routines, which are responsible for support and operations in the background. The libvmm
routines manage the virtual memory, while \emph{libctx} is a context manager based on key-value
mapping of data objects. All imported or generated objects, e.g. integrals, keywords,
tensors, vectors, etc., are stored in the context and can be easily accessed by other routines.
The libutil library contains further low-level machine-dependent routines.

The procedure of ADC calculation states as follows:
\begin{enumerate}
	\item A HF calculation is performed, and all information including one-electron integrals, two-electron integrals, MOs, eigenvalues, Overlap matrix, Fock matrix and point group symmetry are imported through \emph{libledacy} interface.
	\item All tensors calculated in the previous step are transformed from atomic orbital (AO) basis to MO basis by \emph{libmo} module.
	\item MP2 calculation is performed and results are printed.
	\item ADC secular matrix is constructed from the tensors in MO basis.
	\item Guess vectors which are required by Davidson algorithm are set up with correct symmetry.
	\item Davidson solver is started.
	\item When convergence requirements are met, ADC energies and transition amplitudes are printed.
	\item Other information required by adcman are printed.
\end{enumerate}

\section{Implementation of Non-dyson ADC}

\subsection{Implementation of ADC Secular Matrix}

We have illustrated the \emph{libtensor} library which can efficiently and conveniently realize tensor algebra.
From Eq \ref{KCeigen}, our final purpose is to calculate the eigenvalues and eigenvectors of ADC secular matrix.
However, usually we do not ask for all the eigenvalues and eigenvectors.
Instead, people are only interested in the few states with lowest energies.
Thus, full diagonalization will waste much effort on eigenstates that we are not interested in.
To solve this problem, we can use Davidson algorithm, which can determine eigenstates one by one with lower algorithmic complexity.
Since Davidson algorithm works by performing matrix vector product rather than direct access to matrix elements, we can rewrite the ADC secular matrix to matrix vector product form, i.e. calculate the result $W$ of matrix vector product when a vector $Y$ is given.

Before writing down the ADC secular matrix, we introduce some symbols:
\begin{equation}
	\begin{aligned}
		\langle pq \| rs \rangle&=V_{pq[rs]}
		\\
		\mathrm{t}_{\text { ijab }}&=\frac{\langle i j\| a b\rangle}{\varepsilon_{a}+\varepsilon_{\mathrm{b}}-\varepsilon_{i}-\varepsilon_{j}}
	\end{aligned}
\end{equation}

Up to second order, ADC secular matrix of ionization potential calculation is
\begin{equation}
	\mathrm{W}_{i}=\sum_{j} \mathrm{I}_{i j}^{(1)} \mathrm{Y}_{j}-\frac{1}{\sqrt{2}} \sum_{j k b}\langle j k\| b i\rangle Y_{j k b}
\end{equation}
for 1h part, where
\begin{equation}
\mathrm{I}_{i j}^{(1)}=-f_{i j}+\frac{1}{4}\left(1+\hat{\mathrm{P}}_{i j}\right) \sum_{a b k} t_{i k a b}\langle j k \| a b\rangle
\end{equation}
and
\begin{equation}
	\mathrm{W}_{i j a}=\sum_{\mathrm{b}} f_{a b} Y_{i j b}-\left(1-\hat{\mathrm{P}}_{i j}\right) \sum_{k} f_{i k} \mathrm{Y}_{k j a}+\frac{1}{\sqrt{2}} \sum_{k}\langle i j\| k a\rangle \mathrm{Y}_{k}
\end{equation}
for 1p-2h part.

Up to second order, ADC secular matrix of EA calculation is
\begin{equation}
	\mathrm{W}_{a}=\sum_{b} \mathrm{I}_{a b}^{(1)} \mathrm{Y}_{b}+\frac{1}{\sqrt{2}} \sum_{b c j}\langle b c\| j a\rangle Y_{b c j}
\end{equation}
for 1p part, where
\begin{equation}
\mathrm{I}_{a b}^{(1)}=-f_{a b}-\frac{1}{4}\left(1+\hat{\mathrm{P}}_{a b}\right) \sum_{i j c} t_{i j a c}\langle b c \| i j\rangle
\end{equation}
and
\begin{equation}
	\mathrm{W}_{a b i}=\sum_{\mathrm{b}} f_{i j} Y_{a b j}-\left(1-\hat{\mathrm{P}}_{a b}\right) \sum_{c} f_{a c} \mathrm{Y}_{c b i}-\frac{1}{\sqrt{2}} \sum_{c}\langle a b\| c i\rangle \mathrm{Y}_{c}
\end{equation}
for 1h-2p part.

The implementation of ADC secular matrix is by a class \emph{adc2\_matrix}:

\lstinputlisting[language=C++]{./code/adc2matrix.h}

\subsection{Implementation of Setting up Guess Vectors}

Before the Davidson algorithm starts, guess vectors need to be set up for the first matrix vector product.
Usually, the setting up of guess vectors should be subjected to some principles: computationally cheap and close to the final eigenvectors.
A simple idea is to use eigenvectors of some simple approximation of the ADC secular matrix.
In the implementation in Q-Chem, the approximation of second order ADC secular matrix is chosen to be the diagonal elements.
Since people usually care about the states with lowest enegies, the guess vectors should be chosen to be those with smallest diagonal elements.
Mathmatically, if $d_a=K_{aa}+C_{aa}$, and the $n$ smallest elements are $d_{k_1}<d_{k_2}< \dots <d_{k_n}$, then the $m$th guess vector should be
\begin{equation}
	v^m_p=\delta_{p, k_m}
\end{equation}

Of course, we will get correct results if we set up guess vectors in the above way.
However, since Hamiltonian is commutable with total spin operator $S^2$, it is often required the diagonalization of Hamiltonian also diagonalizes $S^2$.
Thus, it is required that the initial guess vectors should have correct spin symmetry.
In addition, the 1h-2p guess vectors themselves has permutation symmetry
\begin{equation}
	V_{abi}=-V_{bai}
\end{equation}
, which is easy to deal with. Thus we will focus on spin symmetry of guess vectors.

For restricted calculation, electrons in ground state are always paired, thus the EA does not depend on whether alpha or beta electron is added, thus we can always choose it to be alpha electron.
Thus, the final state will be restricted to $m_s=\frac{1}{2}$.

For 1p guess vectors, there is no choice for spin since the added electron must be spin spin, and thus
\begin{equation}
	v^m_{a s}=\delta_{a, k_m} \delta_{s, \alpha}
\end{equation}
where $s$ is the spin of orbital a.

For 1h-2p guess vectors, there are two options:
\begin{itemize}
	\item One particle is alpha, one particle is beta, and the hole is beta.
	\item All two particles and the hole are alpha.
\end{itemize}

Thus there are totally three configurations (two for the first case and one for the second case).
To correcctly set up spin symmetry of guess vectors, we need to form linear combinations of these three configurations that are diagonal under $S^2$.
In fact, the coefficients of the linear combinations are refered as Clebsch-Gordan coefficients.
By refering to table of Clebsch-Gordan coefficients, the three final spin states reads:
\begin{equation}
	\begin{array}{|l|l|}
		\hline
		S=\frac{1}{2} & \sqrt{\frac{1}{2}} \ket{\alpha_p \beta_p \beta_h} - \sqrt{\frac{1}{2}} \ket{\beta_p \alpha_p \beta_h}
		\\
		\hline
		S=\frac{1}{2} & \sqrt{\frac{2}{3}} \ket{\alpha_p \alpha_p \alpha_h} - \sqrt{\frac{1}{6}} \ket{\alpha_p \beta_p \beta_h} - \sqrt{\frac{1}{6}} \ket{\beta_p \alpha_p \beta_h}
		\\
		\hline
		S=\frac{3}{2} & \sqrt{\frac{1}{3}} \ket{\alpha_p \alpha_p \alpha_h} + \sqrt{\frac{1}{3}} \ket{\alpha_p \beta_p \beta_h} + \sqrt{\frac{1}{3}} \ket{\beta_p \alpha_p \beta_h}
		\\
		\hline
	\end{array}
\end{equation}

The above tables works when the spacial orbital of two particles are different, otherwise they must have different spins, which means the second case is not allowed.
In such situation, the two paricles are paired, and will not contribute to the total spin.
Thus it reduces to the 1p case.

To summarize, if $a \neq b$, the first guess vector for spin $\frac{1}{2}$ is
\begin{equation}
	\begin{array}{ll}
		V_{a \alpha, b \beta, i \beta}=\frac{1}{2} & V_{b \beta, a \alpha, i \beta}=-\frac{1}{2}
		\\
		V_{a \beta, b \alpha, i \beta}=-\frac{1}{2} & V_{b \alpha, a \beta, i \beta}=\frac{1}{2}
	\end{array}
\end{equation}
the second guess vector for spin $\frac{1}{2}$ is
\begin{equation}
	\begin{array}{ll}
		V_{a \alpha, b \alpha, i \alpha}=\sqrt{\frac{1}{3}} & V_{b \alpha, a \alpha, i \beta}=-\sqrt{\frac{1}{3}}
		\\
		V_{a \alpha, b \beta, i \beta}=-\sqrt{\frac{1}{12}} & V_{b \beta, a \alpha, i \beta}=\sqrt{\frac{1}{12}}
		\\
		V_{a \beta, b \alpha, i \beta}=-\sqrt{\frac{1}{12}} & V_{b \alpha, a \beta, i \beta}=\sqrt{\frac{1}{12}}
	\end{array}
\end{equation}
the guess vector for spin $\frac{3}{2}$ is
\begin{equation}
	\begin{array}{ll}
		V_{a \alpha, b \alpha, i \alpha}=\sqrt{\frac{1}{6}} & V_{b \alpha, a \alpha, i \beta}=-\sqrt{\frac{1}{6}}
		\\
		V_{a \alpha, b \beta, i \beta}=\sqrt{\frac{1}{6}} & V_{b \beta, a \alpha, i \beta}=-\sqrt{\frac{1}{6}}
		\\
		V_{a \beta, b \alpha, i \beta}=\sqrt{\frac{1}{6}} & V_{b \alpha, a \beta, i \beta}=-\sqrt{\frac{1}{6}}
	\end{array}
\end{equation}
if $a=b$, the first guess vector for spin $\frac{1}{2}$ is
\begin{equation}
	\begin{array}{ll}
		V_{a \alpha, b \beta, i \beta}=\frac{1}{2} & V_{b \beta, a \alpha, i \beta}=-\frac{1}{2}
		\\
		V_{a \beta, b \alpha, i \beta}=-\frac{1}{2} & V_{b \alpha, a \beta, i \beta}=\frac{1}{2}
	\end{array}
\end{equation}
the second guess vector for spin $\frac{1}{2}$ is
\begin{equation}
	\begin{array}{ll}
		V_{a \alpha, b \beta, i \beta}=\frac{1}{2} & V_{b \beta, a \alpha, i \beta}=\frac{1}{2}
		\\
		V_{a \beta, b \alpha, i \beta}=\frac{1}{2} & V_{b \alpha, a \beta, i \beta}=\frac{1}{2}
	\end{array}
\end{equation}

For unrestricted case, it is much more complicated.
Although I do not implement it, it does not hurt to discuss a little about this.

Consider the ground state of a system with $p$ net positive charge, in which $p_{\alpha}$ are spin alpha while $p_{\beta}$ are spin beta.
After affinity of an electron, which could be either spin alpha or spin beta.
Thus, the general question is the linear combination of $n_{\alpha}$ alpha spins and $n_{\beta}$ beta spins, where
\begin{equation}
	\begin{array}{lll}
		n_{\alpha}=p_{\alpha} + 1, & n_{\beta}=p_{\beta} & \text{if added alpha electron}
		\\
		n_{\alpha}=p_{\alpha}, & n_{\beta}=p_{\beta} + 1 & \text{if added beta electron}
	\end{array}
\end{equation}

The $n=n_{\alpha} + n_{\beta}$ electrons will genrate $2^n$ configurations.
However, Clebsch-Gordan coefficients only works for addition of two spins.
Thus, an recursive approach is needed to deal with the $n$ spin case.

\section{Results of Calculation}

In this section, I will present a benchmark calculation of the EA-ADC(2) method.
The calculated results are compared with HF, DFT with B3LYP functional and CCSD.
CCSD is used for comparison because CCSD is a widely used Post-Hartree-Fock method to include consideration of electron correlation, which is very reliable.
HF and DFT are used for comparison to show that ADC can perform better than these low-level results, otherwise we don't need these high-level methods, which are computationally more expensive.
For this benchmark calculation, small moleculars
$\text{H}_2\text{O}$, $\text{NH}_3$, $\text{CO}_2$, $\text{LiH}$,
$\text{LiF}$, $\text{N}_2\text{O}$, $\text{CH}_4$,
$\text{C}_2\text{H}_6$, $\text{CH}_2\text{O}$, $\text{HCN}$ are chosen, and the basis set is chosen to be 6-31g.
The results of HF and DFT are obtained by subjecting the ground state energy of original molecule and anion with -1 charge.
Since both HF and DFT are size consistent, it will not cause problem of size consistency.

\begin{table}
	\centering
	\caption{first EA of small molecules under cc-pVTZ basis}
	\label{firstEA}
\begin{tabular}{|c|c|c|c|c|}
	\hline
	Molecules              & HF    & B3LYP & ADC(2) & CCSD  \\ \hline
	$\text{CO}_2$          & -4.64 & -3.46 & -3.47  & -3.73 \\ \hline
	LiF                    & 0.14  & 0.29  & 0.18   & 0.17  \\ \hline
	LiH                    & 0.08  & 0.25  & 0.12   & 0.13  \\ \hline
	$\text{N}_2\text{O}$   & -2.95 & -2.8  & -2.8   & -3.22 \\ \hline
	$\text{H}_2\text{O}$   & -3.72 & -2.93 & -3.27  & -3.31 \\ \hline
	$\text{NH}_3$          & -3.62 & -2.78 & -3.08  & -3.13 \\ \hline
	$\text{CH}_4$          & -3.82 & -2.96 & -3.27  & -3.31 \\ \hline
	$\text{C}_2\text{H}_2$ & -4.17 & -2.93 & -3.33  & -3.5  \\ \hline
	HCN                    & -3.71 & -2.99 & -3.09  & -3.29 \\ \hline
	$\text{CH}_2\text{O}$  & -2.15 & -1.47 & -1.62  & -1.92 \\ \hline
\end{tabular}
\end{table}
\begin{figure}[ht]
	\centering
	\begin{subfigure}[b]{0.42\linewidth}
		\centering
		\includegraphics[width=\linewidth]{./figures/higher/C2H2.pdf}
	\end{subfigure}
	\begin{subfigure}[b]{0.42\linewidth}
		\centering
		\includegraphics[width=\linewidth]{./figures/higher/CH2O.pdf}
	\end{subfigure}
	\begin{subfigure}[b]{0.42\linewidth}
		\centering
		\includegraphics[width=\linewidth]{./figures/higher/CH4.pdf}
	\end{subfigure}
	\begin{subfigure}[b]{0.42\linewidth}
		\centering
		\includegraphics[width=\linewidth]{./figures/higher/CO2.pdf}
	\end{subfigure}
	\begin{subfigure}[b]{0.42\linewidth}
		\centering
		\includegraphics[width=\linewidth]{./figures/higher/H2O.pdf}
	\end{subfigure}
	\begin{subfigure}[b]{0.42\linewidth}
		\centering
		\includegraphics[width=\linewidth]{./figures/higher/HCN.pdf}
	\end{subfigure}
	\begin{subfigure}[b]{0.42\linewidth}
		\centering
		\includegraphics[width=\linewidth]{./figures/higher/LiF.pdf}
	\end{subfigure}
	\begin{subfigure}[b]{0.42\linewidth}
		\centering
		\includegraphics[width=\linewidth]{./figures/higher/LiH.pdf}
	\end{subfigure}
	\begin{subfigure}[b]{0.42\linewidth}
		\centering
		\includegraphics[width=\linewidth]{./figures/higher/N2O.pdf}
	\end{subfigure}
	\begin{subfigure}[b]{0.42\linewidth}
		\centering
		\includegraphics[width=\linewidth]{./figures/higher/NH3.pdf}
	\end{subfigure}
	\caption{First 10 EAs calculated by ADC(2) and CCSD on small molecules}
	\label{higher}
\end{figure}
On the other hand, choice of appropriate basis is very important for EA calculation.
The reason is that anions bind their outermost electrons rather weakly, and hence their valence-range electron densities are diffuse, which means that the use of diffuse functions is an important requirement for basis in EA calculation, and also means that a large basis space is needed to achieve accurate results. One of the basis that satisfying this requirement is cc-pVXZ basis (X=D/T/Q/5 \dots). \cite{newpaper}

We first calculate the first EA energy for these molecules.
In this calculation, cc-pVTZ basis and CCSD are used for an accurate geometry optimization and then cc-PVTZ basis is used for EA calculation.
The results are shown in Table \ref{firstEA}

We can see from the results that ADC(2) matches well with CCSD (mean derivation is 0.15 eV), while both HF and DFT give great derivation (mean derivations are 0.34 eV for HF and 0.39 eV for DFT).
We can also find that HF tends to underestimate EA, and DFT tends to overestimate EA.

Then we compare higher order electron affinities calculated by ADC(2) and CCSD on these small molecules.
Results are shown in Fig. \ref{higher}
As is shown in the figure, ADC(2) results match well with CCSD for the first few EAs among all the molecules.
However, they give large errors for higher orders of EAs, and ADC almost always gives lower EAs for these states.
We can also find that generally, disagreements comes later for large molecules (CO$_2$, N$_2$O and C$_2$H$_2$).

If we look into the amplitudes of 1p and 1h-2p parts, we can find that when a state is dominated by 1p part, ADC(2) and CCSD usually give similar results; however,when it is dominated by 1h-2p part, two theories disagree with a large probability.
This means that the ADC(2) give good results for 1p part but bad result for 1h-2p part, and it usually overestimate the energies for 1h-2p part.
Then we will analyze the reason for this.
Eq \ref{generalTEO} in ISR theory tells us that in the case of ADC(2), TEO of 1p part is second order, while TEO of 1h-2p part is zeroth order.
Thus, ADC(2) gives larger error for 1h-2p results while describes 1p part accurately, compared with CCSD.

On the other hand, we can see from the structure of ADC(2) secular matrix \ref{ADC2} that only $K_2$ is included for 1h2p-1h2p part, thus there is no mix between 1h2p states, which can not effectively decrease energies.
This problem will be solved in ADC(3), since $C^1_{11}$ is also included in 1h2p-1h2p part, and thus TEO for 1h2p states is second order, which is much better.

Another problem for these results is that first EAs of all molecules except Li$_H$ and Li$_F$ are negative, which means that such states are not stable and the anions will automatically decompose into a neutral and a free electron, which is totally unphysical.

This problem comes from two parts.
The first part is finite basis, and the second part is incorrect result of electron correlation.
The first part can principly be solved by complete basis set (CBS).
One famous CBS is cc-pVXZ basis \cite{CBS}, which should give asymptotic results when $X \rightarrow \infty$.
Then we perform calculations for LiH, LiF, CO$_2$ and N$_2$O under these basis.
The results are shown in Table \ref{ccpVXZ}.
\begin{table}
	\centering
	\caption{first EA calculated by ADC(2) and CCSD for cc-pVDZ, cc-pVTZ cc-pVDZ}
	\label{ccpVXZ}
	\begin{tabular}{|c|c|c|c|c|c|c|}
		\hline
		\multirow{2}{*}{Molecules} & \multicolumn{3}{c|}{ADC(2)} & \multicolumn{3}{c|}{CCSD}   \\ \cline{2-7}
								   & cc-pVDZ & cc-pVTZ & cc-pVQZ & cc-pVDZ & cc-pVTZ & cc-pVQZ \\ \hline
		LiH                        & 0.06    & 0.12    & 0.20    & 0.07    & 0.13    & 0.20    \\ \hline
		LiF                        & 0.12    & 0.18    & 0.26    & 0.11    & 0.17    & 0.25    \\ \hline
		CO$_2$                     & -4.97   & -3.47   & -2.52   & -5.21   & -3.73   & -2.73   \\ \hline
		N$_2$O                     & -3.55   & -2.80   & -2.48   & -3.92   & -3.22   & -2.92   \\ \hline
	\end{tabular}
\end{table}
As we can see in the results, the first EA increases when $X$ increases among all of the molecules, which means basis set plays a very imporant role to get positive EAs.
However, cc-pVXZ is a really large basis set, thus a cc-pV5Z is difficult for these molecules and is not given in this benchmark calculation.

For the second part of the reason, which is incorrect electron correlation, it still remains unsolved until now and is one of the center problems of electronic structure.
Many methods are also under development to solve this problem.
Thus, it is not surprising to see that both ADC(2) and CCSD sometimes give bad results.


The analysis strategy relies on the use of a binary classification algorithm to create regions of high \ttH purity.
A boosted decision tree (BDT) is chosen as the binary classification algorithm.
Deep neural networks (DNNs) were also explored; however, these were found to outperform BDTs only when a very high number of training examples are available (discussed in further detail in Sec.~\ref{sec:tth_dnns}).
The BDT-bkg algorithms are trained with the \textsc{xgboost}~\cite{xgboost} framework, using MC simulation of \ttH for the signal events and a combination of MC simulation and data-driven descriptions of the relevant background processes as background.
In order to avoid bias, the BDT-bkg algorithms are trained and optimized on completely separate samples from those used in the measurement of $\mu_{\ttH}$.
The various background processes are weighted according to their cross sections, while the weights of signal events are scaled such that the total number of signal events is equal to the total number of background events.
The signal weights are scaled in such a fashion to avoid issues in training related to imbalanced classes.
The features used in training the BDT-bkg algorithms are discussed in the following subsections.

\subsection{High-Level Features} \label{sec:tth_hlf}
In order to effectively separate \ttH from the SM backgrounds, we construct a number of high-level physics variables which are expected to have discriminating power between the two.
The full list of features is shown in Table~\ref{tab:tth_hlf}.
Plots showing the distributions for both data and simulation for each input feature are shown in Appendix~\ref{app:hlf}.

\begin{table}[h] \scriptsize
\centering
\caption{High level features used in training BDTs. The fourth jet kinematics are given as input features only to the BDT-bkg in the hadronic channel, while the lepton kinematics are given as input features only to the BDT-bkg in the leptonic channel.}
\renewcommand{\arraystretch}{1.5}
\begin{tabular}{|c| c c c|}
\multicolumn{4}{c}{Input Features to BDTs} \\ \hline
Category & \multicolumn{3}{c|}{Features} \\ \hline
\multirow{3}{*}{Photon Kinematics} & $\gamma_1$ $p_T$/$m_{\gamma \gamma}$ & $\gamma_1$ $\eta$ & $\gamma_1$ Pixel Seed Veto \\
& $\gamma_2$ $p_T$/$m_{\gamma \gamma}$ & $\gamma_2$ $\eta$ & $\gamma_2$ Pixel Seed Veto \\
& Max $\gamma$ ID MVA & Min $\gamma$ ID MVA & \\ \hline
\multirow{7}{*}{Jet Kinematics} & Jet 1 $p_T$ & Jet 1 $\eta$ & Jet 1 b-tag score \\
& Jet 2 $p_T$ & Jet 2 $\eta$ & Jet 2 b-tag score \\
& Jet 3 $p_T$ & Jet 3 $\eta$ & Jet 3 b-tag score \\
& Jet 4 $p_T$ & Jet 4 $\eta$ & Jet 4 b-tag score \\
& Max b-tag score & 2nd max b-tag score & \\
& $N_{\text{jets}}$ & $H_T$ & \\ \hline
\multirow{2}{*}{DiPhoton Kinematics} & $p_T^{\gamma \gamma} / m_{\gamma \gamma}$ & $Y_{\gamma \gamma}$ & $|\cos (\Delta \phi)_{\gamma \gamma}|$ \\
& $\Delta R_{\gamma \gamma}$ & $|\cos(\text{helicity angle} (\theta))|$ & \\ \hline
Lepton Kinematics & lepton $p_T$ & lepton $\eta$ & $N_{\text{leptons (tight ID)}}$ \\ \hline
\multirow{1}{*}{Event-level Kinematics} & $E_T^{\text{miss}}$ & & \\ \hline
\end{tabular}
%\caption{High level features used in training BDTs. The fourth jet kinematics are given as input features only to the BDT-bkg in the hadronic channel, while the lepton kinematics are given as input features only to the BDT-bkg in the leptonic channel.}
\label{tab:tth_hlf}
\end{table}

The physics motivations for the inclusion of each feature are enumerated below:
\begin{enumerate}
    \item Photon Kinematics
    \begin{itemize}
        \item Photon \pT divided by \mgg: prompt photons tend to have higher transverse momentum than fake photons from hadronic jets. The photon \pT is normalized by \mgg to prevent the BDT from learning \mH.
        \item Photon $\eta$: prompt photons tend have smaller $|\eta|$ than fake photons.
        \item Photon pixel seed veto: \ttplusX events often have one or more electrons from leptonically decaying \PW bosons which are identified as photons at reco-level. The pixel seed veto helps identify these events.
        \item Photon ID MVA: the primary means of separating between prompt and fake photons. Described in further detail in Sec.~\ref{sec:evt_photon_idmva}.
    \end{itemize}
    \item Jet Kinematics
    \begin{itemize}
        \item Jet \pT: jets from \ttH events tend to have higher transverse momentum than those from background processes (both multi-jet + X and \ttplusX), as the \ttb system recoils against the Higgs.
        \item Jet $\eta$: jets from \ttH tend to be more central than those from multi-jet + X events.
        \item Jet b-tag scores: jets from \ttH have higher b-tag scores than those from non-\ttb backgrounds, as two b-jets are expected 2 from the \ttb decay.
        \item Number of jets: we expect at least 6 (4) jets in a \ttH event in the Hadronic (Leptonic) channel.
        \item $H_{\text{T}}$: \ttH events tend to have higher values of $H_{\text{T}}$, due to the fact that we expect higher $p_T$ of individual jets as well as more total jets in the event.
    \end{itemize}
    \item DiPhoton Kinematics
    \begin{itemize}
        \item DiPhoton \pT divided by \mgg: the recoil of the Higgs against the \ttb system results in higher expected values for the diphoton momentum. The DiPhoton \pT is normalized by \mgg to prevent the BDT from learning \mH.
        \item DiPhoton $Y$: the DiPhoton rapidity is expected to be closer to zero for \ttH events.
        \item DiPhoton $\Delta R$: the angle between the two photons is expected to be smaller for \ttH events due to the fact that the Higgs tends to be boosted from recoil against the \ttb system.
        \item Helicity angle ($\theta$): defined by boosting to the rest frame of the DiPhoton pair and calculating the angle between the photons in that frame. Since the SM Higgs is a scalar, we expect a uniform distribution in $\cos(\theta)$ for \ttH (at generator-level), while backgrounds may peak closer to 1.
        \item $\cos (\Delta \phi)$ of DiPhoton pair: similar argument to above.
    \end{itemize}
    \item Lepton Kinematics
    \begin{itemize}
        \item Lepton \pT: prompt leptons tend to have higher \pT than fake leptons from hadronic jets.
        \item Lepton $|\eta|$: prompt leptons tend to be more central than fake leptons.
        \item Number of leptons passing tight ID: again, helpful in discriminating between prompt and fake leptons. The preselection requires that leptons pass medium ID, the number of leptons passing tight ID is given as an input to the BDT.
    \end{itemize}
    \item Event-level Kinematics
    \begin{itemize}
        \item \met: in the Leptonic channel, we expect nonzero \met in \ttb events due to the neutrino from the $\PW\to l\nu$ decay, while no \met is expected for multi-jet + X events. In the Hadronic channel, this may also be useful in identifying leptonically decaying \ttH events in which the lepton is not reconstructed.
    \end{itemize}
\end{enumerate}

\subsection{Deep Neural Networks for \dipho and \ttgg Backgrounds} \label{sec:tth_dnns}
The dominant SM backgrounds in regions of high \ttH purity (i.e. similar to the signal regions) are: \dipho and \ttgg in the hadronic channel, and \ttgg in the leptonic channel.
To further reduce these backgrounds, additional methods and variables designed to specifically target these backgrounds were explored.
Among the most successful of these methods were deep neural networks (DNNs) trained to reject these backgrounds specifically.
Another notable susccessful tool is the top tagger BDT, discussed in Sec.~\ref{sec:tth_top_tagger_bdt}.

The DNNs exploit low-level information in each event which is lost in the process of summarizing the event in terms of the high-level features described in Sec.~\ref{sec:tth_hlf}.
An example of such low-level information is the azimuthal angle $\phi$ of the reconstructed jets and leptons.
Any physics process should obey azimuthal symmetry in the CMS detector -- any value of $\phi$ is equally likely for any physics object from any physics process.
For this reason, the $\phi$ value of a given jet or lepton on its own provides no means to discriminate between \ttH and other SM backgrounds.
However, the $\phi$ value of a jet or lepton may provide discriminatory power when considered in the context of the rest of the event: angles between jets and leptons likely have different distributions in \ttH events and SM background events.
Providing this type of low-level input to a BDT would be of limited use, as making a splitting on the $\phi$ value of a jet or lepton is not useful on its own.
Deep neural networks were then explored as an algorithm which could potentially make better use of such low-level features.

\subsubsection{Training Features}
The high-level features described in Sec.~\ref{sec:tth_hlf} do not retain the full information of the original event.
To provide a more complete description of each event, the four-vectors of the leading jets and leptons (``physics objects'') are given as inputs to the DNNs.
The four-vector includes the physics object's \pT, $\eta$, $\phi$, and total energy ($E$).
In addition to the four-vectors, four jet flavor scores and lepton ID flags, indicating whether a given lepton is a muon or electron are provided for each physics object.
In total, for up to the leading eight (six) jets in each event and the leading zero (two) leptons in each event in the Hadronic (Leptonic) channel, the following nine features are provided for each physics object:
\begin{itemize}
    \item Four-vector: $p_T$, $\eta$, $\phi$, $E$
    \item 4 DeepCSV scores: $b$, $c$, $udsg$, $bb$
    \item Lepton ID flag: 0 for muons, 1 for electrons
\end{itemize}
In the hadronic channel, there are no leptons and so the Lepton ID flag is omitted, resulting in 8 features per physics object.
In the leptonic channel, jets (leptons) are assigned ``dummy'' values for the lepton ID flag (DeepCSV scores) of -2.
The ``dummy'' values are necessary as each physics object that is input to the DNN must have the same number of input features.
Photons are not included in the list of physics objects to prevent the DNN from learning \mH.

In addition to the low-level object features, a set of high-level features, shown in Table~\ref{tab:tth_hlf_dnn}, is given as inputs to the DNN to allow it to learn correlations between the physics objects and the rest of the event, including the diphoton kinematics and missing transverse momentum.
\begin{table}[h] \scriptsize
	\centering
	\caption{High level features used in training DNNs. The lepton kinematics features are only given as inputs to the leptonic channel DNN.}
	\renewcommand{\arraystretch}{1.5}
	\begin{tabular}{c| c c c c}
		\multicolumn{5}{c}{High-level features for DNNs} \\ \hline
		Category & \multicolumn{4}{c}{Features} \\ \hline
		\multirow{3}{*}{Photon Kinematics} & $\gamma_1$ \pT/\mgg & $\gamma_1$ $\eta$ & $\gamma_1$ $\phi$ & $\gamma_1$ Pixel Seed Veto \\
		& $\gamma_2$ \pT/\mgg & $\gamma_2$ $\eta$ & $\gamma_2$ $\phi$ & $\gamma_2$ Pixel Seed Veto \\
		& Max $\gamma$ ID MVA & Min $\gamma$ ID MVA & & \\ \hline
		\multirow{2}{*}{Jet Kinematics} & Max b-tag score & 2nd max b-tag score & & \\
		& $N_{\text{jets}}$ & & & \\ \hline
		\multirow{1}{*}{DiPhoton Kinematics} & $\pT^{\gamma \gamma} / \mgg$ & $Y_{\gamma \gamma}$ & $\Delta R_{\gamma \gamma}$ & \\ \hline
		\multirow{1}{*}{Lepton Kinematics} & $N_{\text{leptons (tight ID)}}$ & & & \\ \hline
		\multirow{1}{*}{Event-level Kinematics} & \met & $\met \phi$ & & \\ \hline
	\end{tabular}
	%\caption{High level features used in training DNNs. The lepton kinematics features are only given as inputs to the leptonic channel DNN.}
	\label{tab:tth_hlf_dnn}
\end{table}

\subsubsection{Architecture}
Arguably the simplest approach would be to feed the low-level and high-level features to a fully-connected deep neural network.
However, the question arises of how to organize the physics objects in the event: how does one identify the ``first'' jet in an event?
One solution is to order the physics objects by their \pT, meaning that the first jet in the event is the one with the highest \pT.
This strategy is far from perfect: in some \ttH events the first jet may be a b jet from a top decay, while in others the first jet may be a c jet from a \PW decay.
A long-short term memory (LSTM) architecture~\cite{lstm} is employed to address this challenge.
The physics objects are treated as a one-dimensional sequence (ordered by \pT) that is given to the LSTM.
This choice of architecture is motivated by other successful applications of LSTMs to physics objects, notably the DeepCSV and DeepJet architectures which classify jet flavor in part from a one-dimensional sequence of PF candidates given to an LSTM.
The output of the LSTM network is then merged with the high-level features in a set of fully-connected layers, as depicted in Fig.~\ref{fig:tth_dnn_diagram}.
\begin{figure}[h]
    \centering
    \includegraphics[width=0.7\linewidth]{figures/tth/leptonic_dnn_diagram.png}
    \caption{Schematic of deep neural network architecture, shown for the leptonic channel.}
    \label{fig:tth_dnn_diagram}
\end{figure}

\subsubsection{Training Details}
The DNN is implemented in \texttt{keras}~\cite{keras} with \texttt{tensorflow}~\cite{tensorflow} backend.
It uses the Adam optimizer~\cite{adam} with a learning rate of of $10^{-3}$ and with a binary cross-entropy loss function.
It is trained with an early-stopping procedure in which the batch size is increased over the course of training~\cite{increase_batch}.
Starting with a batch size of 1024, the DNN is trained until the improvement in 1-AUC, as calculated on the validation set, after each epoch is less than 1\%, at which point the batch size is quadrupled.
This procedure is repeated until the batch size exceeds 50,000, at which point it is capped and the training is stopped when the validation AUC ceases to improve.
The batch size is capped due to technical limitations of the GPU: large batch sizes take up a large amount of memory.
DNN hyperparameters are summarized in Table~\ref{tab:tth_dnn_hyperparams}.

\begin{table} [h]
    \centering
    \caption{Hyperparameters for the deep neural networks used in both the hadronic and leptonic channels.}
    \begin{tabular}{l l}
        Hyperparameter & Value(s) \\ \hline
        Number of nodes (fully connected layers) & 300, 200, 200, 200, 200 \\
        Number of nodes (LSTM layers) & 150, 150, 150 \\
        L2-normalization constraint (``maxnorm'') & 3 \\
        Dropout rate & 0.1 \\
        Learning rate & $10^{-3}$ \\
        Batch momentum & 0.99 \\
        Activation function (LSTM) & hyperbolic tangent \\
        Activation function (fully-connected layers) & exponential linear unit \\
        Activation function (output layer) & sigmoid \\
    \end{tabular}
    %\caption{Hyperparameters for the deep neural networks used in both the hadronic and leptonic channels.}
    \label{tab:tth_dnn_hyperparams}
\end{table}

A number of regularization methods are employed, which were found to improve performance and/or convergence speed during training.
First, training features are preprocessed with a ``Z-score'' normalization procedure, subtracting the mean and dividing by the standard deviation of each feature such that all features have zero mean and unit variance.
In addition to the Z-score transformation, input features with units of GeV are given in terms of their logarithm, with the log taken before the Z-score transformation.
The aim of preprocessing is to provide a uniform scale for all input features, as this results in faster convergence during training and even improved performance~\cite{lecun_efficient_backprop}.
In the same spirit as feature preprocessing, batch normalization~\cite{ioffe2015batch} is applied between the fully-connected layers of the DNN, normalizing each layer's inputs.
Batch normalization is not applied in between the LSTM layers.
Both feature preprocessing and batch normalization resulted in faster convergence and improved performance of the DNNs.
In addition, dropout~\cite{dropout} is applied between the fully-connected layers in order to reduce overfitting and improve performance.
Between layers which apply both batch normalization and dropout, batch normalization is applied first and dropout is applied second.

\subsubsection{Performance}
Three separate DNNs are trained:
\begin{itemize}
    \item Hadronic channel: \ttH vs. \dipho
    \item Hadronic channel: \ttH vs. \ttgg
    \item Leptonic channel: \ttH vs. \ttgg
\end{itemize}
The output of each DNN is shown for both data and simulation in Fig.~\ref{fig:tth_dnn_datamc}.
\begin{figure} [htbp!]
    \centering
    \begin{tabular}{c c}
        \includegraphics[width=0.48\linewidth,page=62]{{figures/tth/ttHHadronic_RunII_MVA_Presel_v4.11_9Jun2020_impute_histogramsRunIIstd}.pdf} &
        \includegraphics[width=0.48\linewidth,page=63]{{figures/tth/ttHHadronic_RunII_MVA_Presel_v4.11_9Jun2020_impute_histogramsRunIIstd}.pdf} \\
        \includegraphics[width=0.48\linewidth,page=73]{{figures/tth/ttHLeptonic_RunII_MVA_Presel_v4.11_9Jun2020_histogramsRunIIstd}.pdf} &
    \end{tabular}
    \caption{Agreement between data and MC description of background for the various DNNs used as input features to BDT-bkg, for the hadronic channel (top) and the leptonic channel (bottom).}
    \label{fig:tth_dnn_datamc}
\end{figure}
The DNNs are used as inputs to the BDT-bkg, rather than in place of the BDT-bkg because superior performance from DNNs was only observed in the case of a high number of events available for training.
The simulation samples describing \ttH, \dipho, and \ttgg processes each have a high number of individual events ($\geq 10^5$) passing the preselection requirements, so DNNs are trained to distinguish between these processes.
The improvement brought to each channel by the DNNs is shown in Fig.~\ref{fig:tth_dnn_za}.
\begin{figure} [htbp!]
    \centering
    \begin{tabular}{c c}
        \includegraphics[width=0.48\linewidth]{figures/tth/za_comparison_data_Hadronic_DNN_2May2020.pdf} &
        \includegraphics[width=0.48\linewidth]{figures/tth/za_comparison_data_Leptonic_DNN_2May2020.pdf}
    \end{tabular}
    \caption[Expected significance ($Z_A$) shown as a function of the number of \ttH events passing a given cut on BDT-bkg for versions of BDT-bkg trained with (red) and without (black) the DNN scores as training features.]{Expected significance ($Z_A$) shown as a function of the number of \ttH events passing a given cut on BDT-bkg for versions of BDT-bkg trained with (red) and without (black) the DNN scores as training features. Shaded bands show the $\pm 1\sigma$ statistical uncertainty in $Z_A$. The background yield is estimated from events in data in the \mgg sidebands.}
    \label{fig:tth_dnn_za}
\end{figure}
The improvement in expected sensitivity in the hadronic channel is about 10\%.
The improvement in the leptonic channel is significantly smaller than the statistical uncertainty in $Z_A$.
The fact that the improvement is greater in the hadronic channel is likely attributed to the fact that it benefits from the DNN trained against the \dipho background.
Not only is this the largest background in the \ttH hadronic channel signal regions, but the simulation sample has the largest number of events entering the preselection, allowing for aggressive DNN training with lower risk of overfitting.

\subsection{Top Tagger BDT} \label{sec:tth_top_tagger_bdt}
The dominant backgrounds in the hadronic channel at both preselection level and signal region level are the multi-jet, \gjets, and \dipho processes.
An obvious difference between these processes and \ttH is the fact that there are two top (anti-)quarks in the latter, while there are none in the former.
This motivates the use of methods which can identify the presence of top quarks as a tool for further rejecting the multi-jet, \gjets, and \dipho backgrounds.

The chosen method is a top tagger BDT, originally developed in a search for supersymmetric partners of the top quark~\cite{stop_search} and later updated for \ttH.
The BDT is trained using \textsc{xgboost}~\cite{xgboost}.
The BDT takes jet triplets as inputs, with triplets that are matched as coming from a top quark (using generator truth-level information) designated as signal and all other triplets designated as background.
Jets are required to have $p_T > 25$ GeV and $|\eta| < 2.4$.
In addition, the jets are cleaned such that they are not overlapping with leptons.
The truth matching enforces the following additional requirements:
\begin{itemize}
    \item $|m_{jjj} - m_t| < 80$ GeV
    \item All three reco jets are matched to generator-level quarks from a hadronically decaying top ($\Delta R(\text{jet, quark}) < 0.4$).
\end{itemize}
The training features are shown in Table \ref{tab:tth_top_tagger_features}.
\begin{table}
	\centering
    \caption{Input features used in training the Top Tagger BDT.}
	\begin{tabular}{c |c c c c} \hline \hline
		Category & \multicolumn{4}{c}{Features} \\ \hline
		\multirow{3}{*}{Single Jet Quantities} & $p_T$ & mass & & \\
		& DeepCSV b & DeepCSV c vs. light & DeepCSV c vs. b & \\
		& ptD & axis1 & multiplicity & \\ \hline
		\multirow{1}{*}{Di-Jet Quantities} & $\Delta R(\text{j,j})$ & $m_{\text{jj}}$ & & \\
		\multirow{1}{*}{Tri-Jet Quantities} & $\Delta R(\text{b,W})$ & $m_{\text{jjj}}$ & &  \\ \hline \hline
	\end{tabular}
    %\caption{Input features used in training the Top Tagger BDT.}
    \label{tab:tth_top_tagger_features}
\end{table}
The training features are defined as follows:
\begin{itemize}
    \item DeepCSV scores: for each jet, three DeepCSV quantities are provided: the b-tag score, and the c-tag score, given in terms of c vs. light and c vs. b.
    \item ptD, axis1, multiplicity: standard quark-gluon discrimination variables. The fragmentation function is defined as $\text{ptD} \equiv \frac{\sqrt{\sum p_{\text{T}_i}^2}}{\sum p_{\text{T}_i}}$, axis1 is the jet shape variable describing the jet's long axis, and multiplicity provides the number of constituents in the jet.
\end{itemize}
The jet in the triplet with the highest DeepCSV b score is labeled as the b-jet, while the other two jets are labeled as $\PW$-jet 1 and 2, with $\pT(\PW_{j1}) > \pT(\PW_{j2})$.

The ouput of the top tagger BDT is shown for both data and simulation in Fig.~\ref{fig:tth_top_tagger_datamc}.
\begin{figure} [htbp!]
    \centering
    \includegraphics[width=0.48\linewidth,page=48]{{figures/tth/ttHHadronic_RunII_MVA_Presel_v4.11_9Jun2020_impute_histogramsRunIIstd}.pdf}
    \caption{Agreement between data and MC description of background for the top tagger BDT score.}
    \label{fig:tth_top_tagger_datamc}
\end{figure}
Similar to the DNN scores, the top tagger BDT is given as an additional training feature to BDT-bkg.
The improvement in expected sensitivity gained by adding the top tagger to BDT-bkg is shown in Fig.~\ref{fig:tth_top_tagger_za} and is about 5\%.
\begin{figure} [htbp!]
    \centering
    \includegraphics[width=0.7\linewidth]{{figures/tth/za_comparison_data_Hadronic_TopTagger_2May2020}.pdf}
    \caption[Expected significance ($Z_A$) shown as a function of the number of \ttH events passing a given cut on BDT-bkg for versions of BDT-bkg trained with (red) and without (black) the top tagger BDT scores as a training feature.]{Expected significance ($Z_A$) shown as a function of the number of \ttH events passing a given cut on BDT-bkg for versions of BDT-bkg trained with (red) and without (black) the top tagger BDT as a training feature. Shaded bands show the $\pm 1\sigma$ statistical uncertainty in $Z_A$. The background yield is estimated from events in data in the \mgg sidebands.}
    \label{fig:tth_top_tagger_za}
\end{figure}

\subsection{BDT-bkg}
The final BDT-bkg algorithms for each channel use the high-level features listed in Table~\ref{tab:tth_hlf}, the DNN scores described in Sec.~\ref{sec:tth_dnns}, and the top tagger BDT score described in Sec.~\ref{sec:tth_top_tagger_bdt} as the training features.
The outputs of the BDT-bkg algorithms are shown in Fig.~\ref{fig:tth_bdt-bkg}, where agreement between data and simulation is observed within statistical and systematic uncertainties.
\begin{figure} [htbp!]
    \centering
    \begin{tabular}{c c}
        \includegraphics[width=0.48\linewidth]{figures/tth/Figure_001-a.pdf} &
        \includegraphics[width=0.48\linewidth]{figures/tth/Figure_001-b.pdf}
    \end{tabular}
    \caption[Output of the BDT-bkg algorithm for the hadronic channel (left) and the leptonic channel (right). Taken from~\cite{tth_observation}.]{Output of the BDT-bkg algorithm for the hadronic channel (left) and the leptonic channel (right). Events from the \mgg sidebands are shown for both data and simulation. The statistical (statistical $\oplus$ systematic) uncertainties in simulation are shown with black (red) shaded bands. The thinly dashed lines show the boundaries of each signal region used for the cross section measurement, while the thickly dashed lines show the boundaries of signal regions used for a measurement of the CP structure. Events in the gray shaded region are discarded. Taken from~\cite{tth_observation}.}
    \label{fig:tth_bdt-bkg}
\end{figure}

\subsubsection{Validation in \ttZ Events}
As an additional check of the agreement between data and simulation in the output of the BDT-bkg algorithms, a validation in a control region targeting \ttZ (\Zee) events is performed.
The rationale for using \ttZ events is the following: \ttZ and \ttH should have very similar kinematic distributions for most components.
Therefore, BDT-bkg should similarly assign high scores to \ttZ events as it does for \ttH events and a region of high \ttZ purity should be present at high scores.
Because the cross section times branching fraction of \ttZ (\Zee) is much higher than that of \ttH (\Hgg), this control region provides a test of the agreement between data and simulation for high scores of BDT-bkg with much smaller statistical uncertainty than obtained in the \mgg sidebands.
Lastly, it is important to emphasize that this check is qualitative in nature; no formal estimate of the compatibility between the distributions in data and simulation is performed.

The selection for the \ttZ control region is the same as the preselection, with the exception that the cut on the conversion-safe electron veto is inverted to select for \Zee events.
The reconstructed ``diphoton pairs'' in these events are then primarily composed of electrons that were reconstructed as photons.

Additional cuts are next applied on top of the preselection in order to increase the \ttZ purity:
\begin{itemize}
    \item $|m_{\text{Z}} - m_{\gamma \gamma}| < 10$ GeV
    \item $N_{\text{jets}} \geq 5 (3)$ for hadronic (leptonic)
    \item $N_{\text{b-jets}} \geq 2$, using the tight (medium) working point for hadronic (leptonic)
\end{itemize}

One subtlety in validating the BDT-bkg performance in \ttZ events is the fact that BDT-bkg is specifically trained to reject events in which electrons are reconstructed as photons through use of the pixel seed veto (defined in Sec.~\ref{sec:evt_photon_vardef}).
This challenge is addressed by manually changing the value of the pixel seed veto for these events before evaluating their score with the BDT-bkg algorithm: although most of these events fail the pixel seed veto, BDT-bkg is told that they all pass the pixel seed veto.
This hard-coding of the pixel seed veto ensures that \ttZ events are not assigned lower scores due to the fact that the typical diphoton pair in these events is suspiciously electron-like.

Comparisons of BDT-bkg between data and simulation are shown for events entering the \ttZ control regions in Fig.~\ref{fig:tth_bdt-bkg_ttz}.

\begin{figure} [htbp!]
    \centering
    \begin{tabular}{c c}
        \includegraphics[width=0.48\linewidth]{figures/tth/ttHHadronic_MVA_ttZ_12Jun2020.pdf}
        \includegraphics[width=0.48\linewidth]{figures/tth/ttHLeptonic_MVA_ttZ_12Jun2020.pdf}
    \end{tabular}
    \caption{Output of the BDT-bkg algorithm for the hadronic channel (left) and the leptonic channel (right) in the \ttZ control region. The statistical (statistical $\oplus$ systematic) uncertainties in simulation are shown with black (red) shaded bands. Events in the gray shaded region are discarded. Taken from ~\cite{tth_observation}.}
    \label{fig:tth_bdt-bkg_ttz}
\end{figure}

The purity of \ttZ events is much higher in the leptonic channel, where good agreement between data and simulation is observed at high BDT-bkg scores.
Good agreement is also observed in the hadronic channel of the \ttZ control region, though the purity of \ttZ events is lower.
Disagreement is present at lower values of BDT-bkg; however, these events do not enter the analysis selection.