\chapter{Object Reconstruction and Event Generation}
\label{chap:event_sim}
\epigraph{Any one who considers arithmetical methods of producing random digits is, of course, in a state of sin. For, as has been pointed out several times, there is no such thing as a random number --- there are only methods to produce random numbers, and a strict arithmetic procedure of course is not such a method.}{\textit{Jon von Neumann}}
\vskip 0.5in
\section{Introduction}
\label{intro}
This chapter is divided into two parts. In the first part, the procedure for the generation of simulated events is described. This is done in several distinct stages with the output of one stage serving as an input for the next. A suite of software packages, developed mostly by the particle and nuclear physics communities, is used to achieve this. This part concludes by detailing the simulated datasets used in the analyses described in this thesis. In the second part of this chapter, the reconstruction of physics objects is described in detail. It starts with a description of the particle-flow algorithm which is a global event reconstruction scheme for the entire event. This is followed by descriptions of track, muon and electron reconstructions. Reconstruction of jets is described next followed by description of composite objects used in the analysis such as collinear mass and transverse mass.

\section{Event simulation}
A $pp$ collision at the LHC, like any hadronic collision, is more complex than the hard interaction of two participating partons. The proton being a composite object, the colliding partons from the hard interaction are accompanied by other quarks and gluons that interact and rearrange themselves into color singlets due to color confinement. A $pp$ collision thus consists of: the Hard Scattering which represents the part of the collision where two partons in the initial state interact by exchanging high transverse momentum, and the Underlying Event that represent the interaction of the everything else in the collision except the partons in hard scattering. In addition to the implementing the above, i.e. physics of a $pp$ collision  that produces a bunch of final state particles, the event simulation also has to include interactions of these particles with the CMS detector. Monte Carlo methods, that use generation of random numbers to simulate sampling from a given probability distribution, are used to model the above event simulations~\cite{mc_evtsim}.

\subsection{Monte Carlo method}

Monte Carlo (MC) methods (named after a famous casino in the city state of Monaco) are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results~\cite{mcwiki}. In particle physics, these methods play a key role in generation of events and are used primarily for : generation of samples from specified probability distributions, and the calculation of integrals. Programs which implement the above method, called MC event generators, use generation of random numbers to make decisions about physics processes. These can range from selection of processes that are generated in the collision, to which decay channel a particle decays in, to making decisions on how the particle interacts with detector material. Usually, each such decision is the result of a draw from a distribution which depends only on the current state the process is in, and not on previous states. The MC generator is provided as input the distributions that represent the physics of the generated particles, their production, their decay modes and their couplings. A MC generator starts by using a pseudo-random number generator that usually outputs a random number between 0 and 1. Although, true random number generation can only be done by physical processes, modern pseudo-random number generators are known to generate numbers with a high degree of randomness. Starting from this distribution, the MC event generator uses one of the various methods such as the inverse-transform method, or the rejection sampling method to convert this uniform distribution into a desired probability distribution, $p(x)$. It is then possible to generate random numbers according to this distribution to simulate physical processes.

\subsection{CMS simulation pipeline}

The MC simulation of events in CMS consists of the following sequential steps. The first step is simulation of the Hard Scattering. As mentioned earlier, this represents the primary hard interaction in a collision where two partons in the initial state interact by exchanging high transverse momentum resulting in a final state with two or more partons. The parton density function (pdf) which parametrizes the distributions of the partons inside each hadron are used to model the momenta of incoming partons. It represents the probability of finding a parton of a certain flavor at a certain longitudinal momentum fraction, when the hadron, that contains it, is probed at a certain scale. The PDF are extracted from fits to the data, mainly from ep collisions, and various PDF sets are available for each parton flavor. Commonly used pdf sets include ones provided by the  CTEQ, HERA (H1 and ZEUS) and NNPDF collaborations. The LHAPDF library provides a unified C++ interface to all major PDF sets. The matrix element formulation is used to model the hard scattering process to leading order in perturbative QCD, or to higher orders depending on the generator. The next step is simulation of the parton shower. The hadronization and radiation of quarks and gluons in the initial and final states cannot be feasibly encapsulated in the matrix element computation. Parton shower describes these missing parts. The matrix element calculations are combined with the parton shower by one of the different matching schemes which ensure that there is no double counting of terms present in both the matrix element and the parton shower expansion. The matching schemes that are most often used are MLM~\cite{mlm}, CKKW~\cite{ckkw} and FxFx~\cite{Frederix:2012ps}. The simulation of the Underlying Event comes next. Underlying event includes everything in the collision that is not associated with the primary hard scattering process. This consists mostly of soft QCD interactions, and implemented using the MC event generators and interfaced with the matrix element simulation. The hadronization of the quarks and gluons is simulated next and it consists of recombination of individual partons into colorless hadrons. Lastly, the decay of short-lived particles is simulated.

An important part of the event generation chain is the simulation of pileup. The protons circulate inside the LHC not as a continuous beam but in discrete closely packed bunches. This leads to more than one proton-proton collision per bunch crossing, i.e. pileup both in-time and out-of-time (see chapter~\ref{chap:exper_setup}). Event generators add  pile-up events to the hard scattering samples by randomly simulating soft inelastic collisions and overlapping them. The distribution of the number of pileup interactions in data is hard to predict. MC event generators usually produce events for a scenario with a higher number of pileup vertices, and with a flat distribution of number of vertices . This is afterwards reweighted to match the observed distribution of pileup interactions in data.

Several MC generators have been developed. Some of these can produce all components of the above simulation pipeline while some calculate only the matrix element and need to be interfaced with other generators for the simulation of remaining parts. Pythia~\cite{Sjostrand:pythia8} and Herwig~\cite{herwig} can produce the entire chain while Powheg~\cite{Nason:2004rx,Frixione:2007vw, Alioli:2010xd, Alioli:2010xa, Alioli:2008tz, Bagnaschi:2011tu}, aMC@NLO~\cite{Alwall:2014} and Madgraph~\cite{Alwall:2011uj} produce up to matrix element stage. Powheg and aMC@NLO can perform next-to-leading order calculations.

Finally, the Geant4 (GEometry ANd Tracking)~\cite{GEANT4} package is used to simulate the interaction of physical particles after the collision, produced by pipeline described  above, with a sophisticated and complex simulation of the detector itself. This simulated detector response is used as input for the same physics reconstruction algorithms (described in the next section), that are used to reconstruct the data, thus enabling a direct comparison of the two. If differences are observed in the behavior of these reconstruction algorithms for MC events in comparison to observed data, the MC events are tuned to the behavior observed in data.

\section{MC samples used for the analyses}
\label{samples_mc}

The {ggH} and VBF Higgs boson samples are generated with POWHEG 2.0 while an extension of POWHEG 2.0~\cite{Luisoni:2013kna} is used for the $\PW\PH$ and $\PZ\PH$ simulated samples. For the \Hmue analysis, only the gluon fusion (ggH) production mode has been considered. Samples are generated for a range of H masses from 200 to 900 GeV.

The $\zjets$ and $\wjets$ processes are simulated using the \aMCATNLO generator at leading order (LO) with the MLM jet matching and merging scheme. The same generator is also used for diboson production which is simulated at  next-to-LO (NLO) with the FxFx jet matching and merging scheme. POWHEG 2.0 and 1.0 are used for top quark-antiquark ($\ttb$) and single top quark production, respectively. The POWHEG and MADGRAPH generators are interfaced with PYTHIA 8 for parton showering, fragmentation, and decays.

As mentioned earlier in this chapter, additional pileup interactions are also a part of the MC generation pipeline. All simulated samples are reweighted to the pileup distribution observed in data. An event weight is applied based on the number of simulated pileup events and the instantaneous luminosity per bunch-crossing, averaged over the run period. Several other scale factors  are used to reweight the events in order to get the MC simulation to match the data closely. These include scale factors based on trigger, lepton identification, lepton isolation and b-jet tagging efficiencies.

\section{Physics object reconstruction}
\label{p_ob_recon}
This section begins with the description of the particle-flow algorithm followed by reconstruction of tracks and vertices, electrons, muons, jets and other physics objects.
\subsection{Particle flow}
\label{p_flow}
The overarching algorithm used by CMS to produce a unified global (synchronized for all sub-detectors) description of an event is the particle-flow (PF) algorithm~\cite{Sirunyan:2017ulk}. The idea behind the PF algorithm is that if the basic building blocks or elements from the various sub-detectors can be correlated in a well-defined way, then the description of the event and that of each particle in it can be refined by using the global information from the entire detector. The ALEPH experiment at the CERN LEP collider was the first experiment to use such a holistic approach towards event reconstruction. The CMS experiment, owing to its very  granular layers of sub-detector, is the first hadron collider experiment to successfully use particle-flow. The first step of the PF algorithm is the linking of the several building-blocks or PF elements that a single particle can give rise to, across different sub-detector layers. The link algorithm tests pairs of neighbors in the $\eta-\phi$ plane and combines (links) them to form PF blocks. Reconstruction and identification algorithms are run according to a predefined sequence in each of these PF blocks. First, muon candidates are reconstructed and identified. If a muon candidate successfully passes PF quality criterion, the PF elements associated with it are removed from the block. Electron reconstruction proceeds next with electron candidates successfully becoming PF electrons  if their tracks in the tracker, when extrapolated,  have a corresponding energy deposit in the ECAL. The reconstruction procedure of muons, electrons and tracks are discussed in detail in the sections ~\ref{mu_recon},~\ref{track_recon},~\ref{e_recon}. The PF block now consists of photons and hadrons. To reduce fake track identification, tracks with momentum uncertainty larger than the resolution of the calorimeters are removed at this stage. The remaining tracks are then associated with charged hadrons. The remaining calorimeter energy deposits are then associated with photons (ECAL) and hadrons (HCAL). In this manner, PF finally produces a list of all electrons, photons, muons, charged hadrons and neutral hadrons in the event with optimally determined direction, charge and energy.

\subsection{Track and primary vertex reconstruction}
\label{track_recon}

Tracks of charged particles, that traverse the CMS tracker (described in section~\ref{tracker}), are reconstructed~\cite{track_reconstruction} using hits from the pixel and strip detectors in the tracker. Hits are reconstructed by clustering signals above specified thresholds in the pixel and strip channels, and then estimating the cluster positions and uncertainties in a local orthogonal system plane of each sensor. During track reconstruction, a translation is made between the local coordinate system of these hits to the global coordinate system of the tracks. The software used to reconstruct tracks by CMS is called the Combinatorial Track Finder (CTF) and is adaptation of the Kalman filter~\cite{kalman_filter}. Tracks are reconstructed using a iterative procedure with the basic idea being, tracks that are easiest to find (e.g., high $\pt$ tracks, and tracks produced near the interaction region) are searched in the initial iterations, with subsequent iterations looking for more difficult sets of tracks (e.g., low $\pt$ tracks , or tracks produced far from the interaction region). Hits unambiguously assigned to the track in the previous iterations are removed for the subsequent ones, thus reducing the combinatorial complexity. Each iteration can be divided into four sequential steps.

The first step is seed generation which provides initial track candidates that define the starting trajectory parameters and associated uncertainties of potential tracks. Charged particles follow helical paths in the quasi-uniform magnetic field of the tracker, requiring a total of five parameters to determine the trajectory. These five parameters are extracted using two or three hits in the inner region of the tracker. The seeds are constructed in the inner part (and then tracks constructed outwards, and not in the opposite manner) because the high granularity of pixel detectors (in contrast to outer strip layers)  ensure that low fraction of channels are hit. Also, particles like pions and electrons interact inelastically with tracker material or lose energy due to bremsstrahlung radiation as they traverse through the tracker to its outer regions, making the idea of constructing seeds in the inner region a better choice.

The second step in track generation is track finding which is closely based on the Kalman filter. It extrapolates the seed trajectories along the expected path of a charged particle, beginning with an estimate of the track parameters provided by the trajectory seeds generated in the last step. It then uses the location and uncertainty of detected hits, and estimations of effects such as Coulomb scattering, at successive detector layers, to build track candidates, updating the parameters at each layer. First, using the parameters of the track candidate, evaluated at the current layer, an analytical extrapolation is done that determines which adjacent layers of the detector the trajectory can intersect. This takes into account the current uncertainty in that trajectory just like a Kalman filter. Secondly, a search is performed for silicon modules in these layers that are compatible with the extrapolated trajectory. All compatible modules in each layer are then grouped into mutually exclusive groups, such that no two modules in each group overlap. The collection of all hits from from one such module group forms a group of hits. Finally, new track candidates are formed by adding exactly one of the compatible hits from each group, to each original track candidate. The modules in a given group are mutually exclusive and a contribution of more than one hit from each group is not expected. The trajectory parameters of the new candidates are then updated by combining the information from the added hits with the extrapolated trajectory of the original track candidates. Figure~\ref{fig:trackrecon} illustrates the reconstruction efficiency of tracks in case of isolated muons.

The third step in track generation track fitting. In this step the collection of hits from the last step are refitted using a Kalman filter and smoother, to provide a best possible estimate of parameters for each track trajectory. The procedure described above, in conditions as challenging as the LHC, can yields several fake tracks that are not associated with any charged particle passing through the tracker. The fourth and final step applies several quality requirements to the set of reconstructed tracks and substantially reduces the fake contribution. The requirements are based on criteria such as the minimum number of layers the track has hits in, how compatible its origin is with a primary vertex, how good a fit it yields etc.

\begin{figure*}
\begin{center}
\captionsetup{width=0.9\textwidth,justification=centering}
\includegraphics[width=0.9\textwidth,keepaspectratio]{plots_and_figures/chapter4/trackrecon.png}
\caption{Track reconstruction efficencies for single isolated muons as a function of $\eta$ and \pt~\cite{track_reconstruction}.}
\label{fig:trackrecon}
\end{center}
\end{figure*}

Proton-proton interaction vertices are reconstructed by selecting tracks that are produced promptly in the primary interaction region. The selected tracks are then clustered on the basis of their z-coordinates at their point of closest approach to the centre of the beam spot, which represents a 3-D profile of the region where the LHC beams collide inside the CMS detector. The exact positions of the vertices are then obtained from these clustered candidates, by using a fitting procedure, called the adaptive vertex fitter~\cite{vertex_fitting}. The vertex which has the largest sum of squared transverse momenta of tracks originating from it is considered the primary interaction vertex.

\subsection{Muon reconstruction}
\label{mu_recon}
Hits in the muon system (described in section~\ref{muon_system}) and tracks (muons being charged particles leave tracks in the tracker) from the tracker are used to reconstruct muons~\cite{muon_recon2018}. When muons traverse a muon subdetector (such as RPC, CSC or DT) in the muon system, they ionize the gas in the chambers. The electrical signals produced on the wires and strips as a consequence of the ionization are read out by electronics systems that associate these ``hits'' with well-defined locations in the detector. Various algorithms depending on the subdetector technology are used to reconstruct these hits. Reconstruction of muon tracks using these hits first proceeds independently of track reconstruction in the tracker. These tracks, called \textit{standalone-muon tracks}, are built using these reconstructed hits from the muon system using a Kalman filter. Muon tracks are also built inside-out by propagating tracker tracks (described in previous section) with transverse momentum above 0.5 GeV to the muon system and matching them to (straight-line) segments of hits in DT or CSC. If a match is found, the tracker track qualifies as a \textit{tracker muon track}. Muon tracks are also  built outside-in  by matching standalone-muon tracks with tracker tracks, and combining information from both using a Kalman filter fit. These are called \textit{global muon tracks}. The global muon reconstruction is especially efficient for muons leaving hits in several muon stations. The \textit{tracker muon} reconstruction is more efficient for low $\pt$ muon candidates but it can cause fake muon tracks due to hadronic particles which \textit{punch-through} to the innermost muon stations. The \textit{global muon} reconstruction has  high efficiency for muons penetrating through more than one muon station, and reduces the muon misidentification rate compared to tracker muons. Combining both \textit{tracker muon tracks} and \textit{global muon tracks}, the efficiency for reconstructing a muon is as high as 99\%. The particle-flow algorithm applies a set of requirements, based on various quality parameters from muon reconstruction as well as information from other sub-detectors, to reconstructed candidates. The PF muon candidates used in the analyses described in this thesis were required to satisfy the following set of criterion to be identified as a muon:

\begin{itemize}
\item Must be a global muon or a tracker muon.
\item Must have at least one hit in the pixel subdetector of the tracker
\item $\chi^2$ of the compatibility between the position of the standalone and trackers tracks $<12$
\item Transverse impact parameter of the associated tracker track with respect to the primary vertex $d_{xy}< 2 mm$
\item Longitudinal distance of the (origin of )associated tracker track with respect to the primary vertex $d_z <5 mm$
\item constraints on muon segment matching compatibility between tracker and muon system dependent on if it is a global muon
\end{itemize}
The efficiency of the above selection for muon identification is illustrated using a plot from a study performed by the CMS Muon Physics Object group in Fig.~\ref{fig:muoneff}. As can be seen from the plots, there is a difference in the efficiencies in data and MC simulation. This is corrected using a set a of scale-factors applied as a function $\eta$ and $\pt$ to adjust the efficiency in simulation to get it to match the efficiency in data.

\begin{figure*}[!htpb]
\centering
\captionsetup{width=0.98\textwidth,justification=centering}
\includegraphics[width=0.47\textwidth]{plots_and_figures/chapter4/muoneffveta.png}
\includegraphics[width=0.49\textwidth]{plots_and_figures/chapter4/muoneffvpt1.png} \\
\includegraphics[width=0.49\textwidth]{plots_and_figures/chapter4/muoneffvpt2.png}
\includegraphics[width=0.49\textwidth]{plots_and_figures/chapter4/muoneffvpt3.png}
\caption{Efficiency of muon identification as a function of $\eta$ and $\pt$, for data (black) and simulation (blue)~\cite{muon_pog}}
 \label{fig:muoneff}
\end{figure*}

The momentum of muons is measured by CMS using one among different possible ways involving the tracker and muon system~\cite{muon_recon2012} and then using the PF algorithm to refine this measurement exploiting information from the full event.

\subsection{Electron reconstruction}
\label{e_recon}
Besides muons, electron form the other primary part of the final state of the decay we are searching for in this thesis. Electrons, in the CMS, are reconstructed using clusters of energy formed in the ECAL (described in section ~\ref{Ecal}) and associating them with tracks from the tracker~\cite{e_recon}. The reconstruction of electrons is made complicated by the fact that they can radiate a significant amount of energy before reaching the ECAL. This happens due to the radiation of bremsstrahlung photons caused by the interaction of electrons with atoms as they pass through the tracker. This loss can range from 33\% to as high as 86\% depending on $\eta$ (as a consequence of the fact that the amount of detector material the electron has to cross is $\eta$ dependent). In order to measure the an electron's energy,  clustering algorithms thus need to take into account the energy from these bremsstrahlung photon showers, together with the deposit made in the ECAL by the electron. The energy from these radiated photons spreads primarily in the $\phi$ direction, owing to bend in electron trajectory in the magnetic field of CMS. The spread in the $\eta$ direction is relatively small. These facts are used by the clustering algorithms.

The algorithm used to cluster the electron energy deposit in the ECAL barrel is called the \textit{hybrid} algorithm. It exploits the above property of the electron shower shape, and uses the geometry of the ECAL to form clusters that are narrow in $\eta$ direction but wide in $\phi$ direction. Starting with a (seed) crystal containing the largest amount of energy deposited in a considered region above a certain threshold (1 GeV), it adds 5x1 arrays of crystals in $\eta\times\phi$ around the seed crystals in both directions of $\phi$ if the energy contained in the arrays is above another predefined threshold (0.1 GeV). Contiguous arrays are merged into clusters, and finally a electron supercluster is formed from all such strip clusters which have at least one seed strip with energy above another predefined threshold (0.35 GeV). The position of the supercluster is computed as the energy-weighted mean of the cluster positions, whereas its energy is simply taken as the sum of the energy of all its constituent clusters. In the ECAL endcap a different clustering algorithm is used owing to different geometrical arrangement of the crystals. This algorithm called the 5x5 algorithm starts similarly with a seed crystal with maximum energy in a local region, and satisfying the minimum energy requirement of 0.18 GeV. Clusters of 5x5 crystals are progressively grouped around the seed crystal, making a supercluster, if the total cluster energy exceeds 1 GeV and they are within $\pm 0.7$  and $\pm 0.3$ respectively in $\eta$ and $\phi$ around the seed crystal. The position and energy of the supercluster is calculated in the same manner as the barrel. The energy from the preshower is also added into the supercluster, using it's most energetic cluster and it's maximum distance in $\phi$ to other clusters, and extrapolating it to the preshower plane to define the spread in the preshower. The thresholds used in the above algorithms were optimized using simulation and adjusted during data taking periods.

The standard track reconstruction (section~\ref{track_recon}) is not efficient for electrons. This is because the standard approach is compromised by the large radiative losses in the tracker leading to a poor estimation of track parameters~\cite{track_reconstruction}. Therefore, a dedicated tracking procedure is used for electron candidates that uses information not only from the tracker but also the ECAL. Just like the standard track reconstruction procedure, the first step in electron track reconstruction is seeding. This is done in two ways and the results are then combined. In the first method, superclusters from ECAL are used. As mentioned earlier, owing to strong magnetic field, the bremsstrahlung photons emitted by the electrons deposit energy in the ECAL at $\eta$ values similar to that of the electron, but at different $\phi$ leading to a spread. The ECAL supercluster algorithms described above recover this energy. The position and  energy of these reconstructed superclusters along with the assumption that the electrons originated close to the center of the beam spot can be used to constrain the trajectory of the electron through the tracker. Hits in the first layers of the trackers compatible with these trajectories are deemed electron seeds. In the second method of seeding, the ``opposite'' is done. Tracks constructed by the regular tracking algorithm are extrapolated to the ECAL and matched with a supercluster. The seeds corresponding to such matching tracks are retained as electron seeds. The seed collections from these two methods are merged leading to a increase in overall efficiency of the seeding procedure. These seeds are then used to initiate electron track finding and fitting phases. This track finding procedure is similar to that used in standard tracking except for small adjustments. The $\chi^2$ fit thresholds used by the Kalman filter to decide whether a hit is compatible with a trajectory (see section~\ref{track_recon}) is weakened to accommodate tracks that deviate from their expected trajectory because of bremsstrahlung. Similar adjustments are made to the penalties assigned to track candidates for passing through a tracker layer without being assigned a hit. The final track fit uses a modified version of the Kalman filter, called the Gaussian Sum Filter (GSF), to account for the fact that the energy loss of an electron traversing the tracker material is non-Gaussian. This  makes it unsuitable to use a conventional Kalman filter algorithm which assumes gaussian distribution. The GSF technique deals with this by approximating this non-Gaussian energy-loss distribution as the sum of several Gaussian functions, and is found to perform much better than the regular fitting procedure.

Finally, electron candidates are constructed by associating a electron track (called GSF track) produced by the above procedure with a supercluster in the ECAL. For ECAL-seeded candidates this association is made by a geometrical matching in $\eta-\phi$, while for tracker-seeded candidates a multivariate (MVA) technique that combines information from supercluster and GSF track is used. The electron charge is estimated using a combination of three procedures involving the use of the GSF track curvature, use of ECAL supercluster position and its relative position in $\phi$ to that of the first hit in the GSF track, and also by using KF tracks that have common hits with the GSF tracks. The combination of a best vote of three methods reduces the charge misidentification probability to 1.5\%  compared to 10\% when using just the GSF track curvature method. Like other variables, the momentum of electrons is also estimated using a combination of tracker and ECAL measurements.

Further, several quality requirements are used on reconstructed electron candidates to identify (real/signal) electrons to suppress fake sources such as photon conversions, jets misidentified as electrons etc.
These requirements are based on variables that fall into three broad categories: variables that compare measurements from ECAL and the tracker, variables that come only from ECAL (such as transverse shape of electromagnetic showers, ratio of energy fractions deposited in the HCAL to the ECAL ) and purely tracking based variables (such as information from GSF track, difference between the information from GSF and KF-fitted tracks). These variables can be used in two ways: a cut-based method that uses the variables above directly to apply threshold requirements, or a multivariate (MVA) technique that uses all these variables as an input to a Boosted Decision Trees classifier to obtain a combined discriminator variable on which a threshold is applied. The BDT based method has much better performance as is illustrated in Fig~\ref{fig:elec_eff}. Two separate BDTs are trained depending on whether electron is required to pass a HLT triggering requirement or it is not. The trigger selection used in the analyses described in this thesis uses trigger based on muons. The BDT based identification criterion for non-triggering electrons is thus used in this analysis. The threshold corresponding to the working point used in this analysis has an efficiency of approximately 80\%. The difference in efficiencies  of electron identification based on the above criteria in data and MC simulation is corrected using a set a of scaled factors, applied as a function $\eta$ and $\pt$. This adjusts the efficiency in simulation to get it to match the efficiency in data.

\begin{figure*}[!htpb]
\centering
\captionsetup{width=0.95\textwidth,justification=centering}
\includegraphics[width=0.95\textwidth]{plots_and_figures/chapter4/elec_eff.png}
\caption{Performance of the BDT-based electron identification algorithm (red dots) compared with results from several working points of cut-based selection for electron candidates in the ECAL barrel (left), and endcaps (right)~\cite{e_recon}.}
 \label{fig:elec_eff}
\end{figure*}

\subsection{Hadronic tau leptons}
\label{tau_recon}
Tau leptons decay hadronically in several ways or decay modes. The primary decay modes consist of: one charged hadron and up to two neutral pions, or three charged hadrons. The algorithm that is used to reconstruct hadronically decaying tau leptons in CMS is called the hadrons-plus-strips (HPS) algorithm~\cite{Khachatryan:2015dfa,Sirunyan:tau16}. This algorithm proceeds in two steps. In the first step, the topology of the candidate is checked to match the topology of one of the decay modes. The second step consists of a MVA-based discriminator (built using variables such as lifetime information, decay mode etc.), that is used to reject electrons, muons, quarks or gluon jets wrongly identified as hadronic taus. The reconstruction efficiency is also improved, in the case of converted photons from a neutral-pion decay, by considering PF photons and electrons from a strip along the $\phi$ direction. The final states in the analyses presented here do not contain hadronically decaying taus. Hence, all events which contain hadronically decaying taus are rejected.

\subsection{Jet reconstruction}
\label{jet_recon}
Jets are clusters of particles that are experimental signatures of quark and gluons which hadronize (due to color confinement) producing a narrow spray or ``jet'' of particles~\cite{jet_recon}. Jets are reconstructed in CMS by clustering PF objects. In order to group together objects into a jet, CMS uses the anti-$k_{T}$ clustering algorithm. This belongs to a broader class of clustering algorithms called sequential clustering algorithms which cluster objects into jet in a sequential order following a predefined set of rules. The general form of a sequential clustering algorithm is based on the quantities  $d_{ij}$, which represents the distance between two entities, and $d_{iB}$ which represents the distance of the i-th entity from the beam axis. These distances are defined as:
\begin{equation}
  d_{ij}=min(k_{ti}^{2p},k_{tj}^{2p})\frac{\Delta_{ij}^{2}}{R^2}
\end{equation}
\begin{equation}
  d_{iB}=k_{ti}^{2p}
\end{equation}
where $\Delta_{ij}^{2}=(\eta_i-\eta_j)^2+(\phi_i-\phi_j)^2$, $k_{ti}$ is the transverse momentum of the i-th entity and R is the radius parameter which is set as 0.4. The parameter p governs the relative power of energy versus geometrical scales and the particular value of $p=-1$ defines the anti-$k_{T}$ clustering algorithm. The algorithm first computes distance $d_{ij}$ between all entity pairs present at that stage. If the minimum of those distances (say between entity i and j) is smaller than the minimum distance $d_{iB}$ of any entity from the beam axis, those entities i and j are combined into a single entity. Otherwise, the entity closest to the beam axis is considered a jet and removed from the list of entities to be further clustered. This continues until all entities are clustered. The anti-$k_{T}$ is dominated by high $\pt$ particles which it clusters first, subsequently including softer and softer constituents. Soft particles tend to cluster with hard particles before they tend to cluster among themselves. A hard particle that has no hard neighbors within a distance 2R accumulates all the soft particles within a circle of radius R. It tries to produce jets with fairly conical shapes that are centered around the hardest particles of the event, and with boundaries that resilient to the effect of soft radiation.

Jets being complex objects suffer from several effects that cause their energy, reconstructed as described above, to differ from their true values. Multiplicative correction factors are applied to calibrate their $\pt$ and to ensure a uniform response in $\eta$~\cite{jet_recon1,jet_recon2}. A total of four multiplicative corrections are applied. Firstly, the energy coming from pileup that has been clustered into the jet needs to be corrected for. A correction is applied based on the \textit{hybrid jet area} method which is a combination of two methods viz., the average offset method and the jet area method. The average offset method uses zero bias events to measure the average amount of energy added to the event due to pileup. The assumption is that averaging over zero bias events makes this measurement insensitive to high $\pt$ objects and primarily represents soft pileup contributions. The average offset is measured in bins of $\eta$ and number of pileup vertices ($N_{PV}$) averaged over $\phi$. The correction is then given by $1-\frac{<Offset(N_{PV},\eta)>}{p_{T}^{RAW}}$, where $p_{T}^{RAW}$  is the uncorrected jet $\pt$. The drawback of this method is that it assumes that every jet contains the same amount of pileup contribution. The jet area method, on the other hand , calculates corrections on a jet-by-jet basis. It calculates an energy density  per event  by clustering jets using the $k_{T}$ algorithm (this has a value of parameter p=1 and favors clustering soft jets as opposed to hard ones) and dividing the $\pt$ by jet area, which is defined as the region in $\eta-\phi$ occupied by soft particles clustered in the jet. The median of this distribution ($\rho$) for an event is expected to be insensitive to hard particles and thus $\rho A_{j}$ is a good approximation of pileup contribution to the i-th jet. The drawback of this second approach, however is that it doesn't take into account the fact that the detector response is $\eta$ dependent. The \textit{hybrid jet area} method combines these two methods to calculate a jet-by-jet correction depending on $\eta$ and $N_{PV}$. Secondly, a MC calibration factor, which corrects the energy of reconstructed jets to match the generated MC particle jet energy on average, is applied. This factor is based on simulated events. Finally, two other factors are used that each calibrate the energy response of reconstructed jets to be uniform with respect to $\eta$ and $\pt$. These are also measured using simulated events. A QCD dijet sample is used to uniformize the dependence in $\eta$. Conservation of momentum in the transverse plane tells us that the sum of momentum in the transverse plane should be zero. Using jets that are approximately back-to-back in the azimuthal direction but at different $\eta$ regions of the detector, the difference in response between these two $\eta$ regions can be ascertained and corrected/uniformized. Using the same method of measuring residual response in the transverse direction in $\gamma + jets$ or $Z +jets$  events, the absolute jet energy scale as a function of $\pt$ can be made uniform.

\subsection{Missing transverse energy: \ptvecmiss}
\label{mt_met_recon}
The CMS detector is unable to detect neutrinos (and other hypothetical particles) that are weakly interacting. However, the momentum balance (or imbalance) in the plane transverse to beam direction can be used to infer their presence. This ``missing'' transverse momentum vector is referred to as the \ptvecmiss and its magnitude is referred to as \ptmiss. It is defined as the negative vector sum of the $\pt$ of entire list of objects reconstructed in the event by the above reconstruction algorithms and refined by particle-flow (PF objects):
\begin{equation}
  \ptvecmiss=-\Sigma\ptvec
\end{equation}
The \ptvecmiss plays an important role in this analysis as it helps gauge the momentum of the neutrinos from the decaying tau lepton. The \ptvecmiss reconstruction is directly dependent on the reconstruction of all the other objects in the event, from jets to muons to electrons. Consequently, it is sensitive to all the effects that influence the precise reconstruction and calibration of these objects. The largest effects come from biases in jet reconstruction and pileup (which are interconnected). Jet energy corrections described in previous section and pileup mitigation techniques discussed earlier can help significantly reduce the bias in \ptvecmiss reconstruction~\cite{Sirunyan:2019kia}.

\subsection{Relative isolation}
\label{isolation}
The isolation of an object is the measure of the absence of other objects in its vicinity. In other words, it is the measure of how ``isolated'' an object is. It is calculated by summing up the $\pt$ of all objects in a cone with predefined radius $\Delta R=0.4$ around the lepton. The relative isolation which is obtained by dividing the isolation by the $\pt$ of the lepton is then given by:

\begin{equation}
I_\text{rel}^{\ell} = \left( \sum  \pt^\text{charged} + \text{max}\left[ 0, \sum \pt^\text{neutral}
                                 +  \sum \pt^{\gamma} - \pt^\text{PU}\left(\ell\right)  \right] \right) /  \pt^{\ell},
\end{equation}
where $\pt^\text{charged}$, $\pt^\text{neutral}$, and $\pt^{\gamma}$  indicate the $\pt$ of a charged particle, a neutral particle, and a photon within the cone, respectively. The contribution to isolation sum from charged particles in the cone coming from pileup is excluded by requiring the track origins be consistent with the primary vertex associated with the hard interaction. However, the same procedure cannot be adopted for neutral particles in the cone coming from pileup. This contribution, $\pt^\text{PU}\left(\ell\right)$, is estimated using a jet area method for electrons~\cite{isolation_1,isolation_2}, and simply as half of the scalar $\pt$ sum of charged particles from pileup inside the cone for muons. The definition ensures that the total contribution to the isolation cone sum for neutral particles is always greater than or equal to 0.

The relative isolation is an useful variable in this analysis as prompt objects are usually isolated. More importantly, for an analyses with electrons and muons in the final state such as this, a tightened isolation requirement helps to reject those events where a jet is misidentified as either one of these leptons. Strict isolation requirements are thus used in the event selection in this analysis as described in sections ~\ref{h125_presel_cat} and ~\ref{H_presel_cat}.

\subsection{Collinear mass: \mcol and transverse mass: $M_{T}$}
\label{col_mass}
An important variable of interest in this analysis is the collinear mass, $\mcol$. As mentioned in later chapters, $\mcol$ is used as the signal variable in the $\Hmue$ analysis and in the $\mcol$ fit method of the $\hmue$ analysis. The visible mass of the muon-electron system, $\mvis$ is not a very good estimator of the Higgs boson mass. This is because the neutrinos from the tau decay do not interact with the detector and the energy they carry is ``lost''. $\mcol$ provides a better estimate of the Higgs mass by approximating the neutrino momenta using the collinear approximation~\cite{Ellis:1987xu}. The primary idea is the following. The mass of the Higgs being much larger than that of the tau lepton causes the tau lepton to become highly boosted. Consequently, the decay products of the tau lepton, i.e. the electron, the tau neutrino and the electron neutrino, are produced in a highly collimated region around the direction of the tau lepton momentum. The momenta of the neutrinos ($\pt^{\nu,\,\text{est}}$) can thus be approximated from the projection of \ptvecmiss in the direction of momenta of the visible tau decay product (electron), i.e. $\vec{\pt^{\Pe}}$. The visible fraction of the tau lepton momentum is then given as $x_{\Pgt}^\text{vis}={\pt^{\vec{\Pgt}^{\text{vis}}}}/{(\pt^{\vec{\Pgt}^\text{vis}}+\pt^{\nu,\,\text{est}})}$. Finally, $\mcol$ is given as $M_{\text{col}}= M_{\text{vis}} / \sqrt{\smash[b]{x_{\Pgt}^\text{vis}}}$. A very simple illustration in Figure~\ref{fig:mcol_v_mvis} shows the superimposition  the $\mcol$ and $\mvis$ spectrums for a 300 GeV Higgs boson. Evidently, $\mcol$ is a better estimator of the mass and also has a sharper peak.

\begin{figure*}[!htpb]
 \centering
 \captionsetup{width=0.8\textwidth,justification=centering}
 \includegraphics[width=0.9\textwidth]{plots_and_figures/chapter4/mcol_v_mvis.pdf}
\caption{\mcol and \mvis distributions for Higgs mass of 300 GeV.}
 \label{fig:mcol_v_mvis}
\end{figure*}

Another variable that is used in this analyses and is useful to discriminate signal from background (see chapter~\ref{evt_sel}) is the transverse mass, $\mt(\ell)$ ($\ell = \Pgm, \Pe$). It is defined as follows: $ {\mt}(\ell)=\sqrt{\smash[b]{2|\ptvec^{\ell}||\ptvecmiss|(1-\cos{\Delta \phi_{\ell - \ptmiss}})}} $, where $\Delta \phi_{\ell - \ptmiss}$ is the angle between the lepton transverse momentum and \ptvecmiss.

% % uncomment the following lines,
% if using chapter-wise bibliography
%
% \bibliographystyle{ndnatbib}
% \bibliography{example}

% % uncomment the following lines,
% if using chapter-wise bibliography
%
% \bibliographystyle{ndnatbib}
% \bibliography{example}


\documentclass[addpoints, 11pt]{exam}
\usepackage{url}
\usepackage{amsmath,amsthm,enumitem,amssymb}
\usepackage{graphicx}
%\input myfonts

\newtheorem*{claim}{Claim}
\title{CS 6150: HW5 -- Randomized algorithms, optimization}
\date{Submission date: Monday, Nov 29, 2021 (11:59 PM)}
\begin{document}
\maketitle
\begin{center}
\fbox{\fbox{\parbox{5.5in}{\centering
%This assignment has \numquestions\ questions, for a total of \numpoints
points. % and \numbonuspoints\ bonus points.
Unless otherwise specified, complete and reasoned arguments will be expected for all answers. }}}
\end{center}

\qformat{Question \thequestion: \thequestiontitle\dotfill \textbf{[\totalpoints]}}
\pointname{}
\bonuspointname{}
\pointformat{[\bfseries\thepoints]}

\begin{center}
  \gradetable
\end{center}
\newpage
\paragraph{Instructions.}  For all problems in which you are asked to develop an algorithm, write down the pseudocode, along with a rough argument for correctness and an analysis of the running time (unless specified otherwise). Failure to do this may result in a penalty. If you are unsure how much detail to provide, please contact the instructors on Piazza.

\begin{questions}

\titledquestion{Trade-offs in sampling}
For this problem, you need to run some basic experiments and write down the results you obtained. You {\bf do not need to submit your code}, but if you prefer, you may add a publicly accessible link to the code (e.g., on github).

Suppose we have a population of size 10 million, and suppose 52\% of them vote for $A$ and 48\% of them vote for $B$.

\begin{parts}
\part[4] Randomly pick samples of size (a) 20, (b) 100, (c) 400, and evaluate the probability that $A$ is majority even in your sample (by running the experiment say 100 times and taking the count of times $A$ is the majority in your sample). Write down the values you observe for these probabilities in the cases (a-c).
\part[4] What is the size of the sample you need for the probability above to become 0.9? (Your answer can be within 20\% of the `best' value.)
\part[4] Suppose the population is more biased ---55\% of them vote for $A$ and 45\% of them vote for $B$--- and re-solve part (b).
\end{parts}

\titledquestion{Satisfying ordering constraints}
This problem is meant to illustrate a rather cool paradigm for solving problems, which is picking a {\em random solution}, and understanding how well it does. Indeed, there are many problems ("3-SAT", a version of boolean satisfiability, being the chief of them) where doing better than a random solution is actually NP-hard!

Suppose we have $n$ elements, labeled $1, 2, \dots, n$. Our goal is to find an ordering of these elements that satisfies as many ``constraints'' as possible, of the kind described below. We are given $m$ constraints, where each constraint is given by a triple $(a, b, c)$, where $a, b, c \in \{1, 2, \dots, n\}$. The constraint is said to be {\em satisfied} if in the ordering we output, $a$ does {\bf not} lie ``between'' $b$ and $c$ (but the order between $b$ and $c$ doesn't matter).  For example, if $n=4$ and we consider the ordering $2 4 3 1$, then the constraint $(1, 4, 3)$ is satisfied, but $(3, 1, 2)$ is not.

Given the constraints, the goal is to find an ordering that satisfies as many constraints as possible (for simplicity, assume in what follows that {\bf $m$ is a multiple of $3$}). For large $m, n$, this problem becomes very difficult.

\begin{parts}
\part[6] As a baseline, let us consider a {\em uniformly random} ordering. What is the expected number of constraints that are satisfied by this ordering?  [{\em Hint: } define appropriate random variables whose sum is the quantity of interest, and apply the linearity of expectation.]
\part[4] Let $X$ be the random variable which is the number of constraints satisfied by a random ordering, and let $E$ denote its expectation (which we computed in part (a)). Now, Markov's inequality tells us, for example, that $\Pr[ X \ge 2E ] \le 1/2$. But it does not directly imply that $\Pr[ X \ge E ]$  is ``large enough'' (which we need if we want to say that generating a few random orderings and picking the best one leads to many constraints being satisfied with high probability).
Use the definition of $X$ above to conclude that $\Pr[ X \ge E ] \ge 1/m$.

[{\em Hint:} $X$ is a non-negative integer!]
\end{parts}

\titledquestion{Writing constraints}[6]
We will see how writing down constraints can be tricky when formulating problems as optimization.

Recall the traveling salesman problem (TSP): we are given a \underline{directed} graph $G = (V, E)$ with edge lengths $w(e)$.  The goal is to find a {\em directed cycle} that (a) visits every vertex exactly once, and (b) minimizes the total length (sum of lengths of the edges of the cycle).

Consider the following optimization formulation. We have variables $x_{uv} \in \{0,1\}$, one for each edge $(u,v)$ (remember that the graph is directed). The objective is to minimize $\sum_{(u,v) \in E} w(uv) x_{uv}$. The constraints are as follows: let $N_+(u)$ and $N_-(u)$ denote the out-neighbors and in-neighbors of $u$; then we consider the constraints:
\begin{align*}
\forall ~u,~ \sum_{v \in N_+(u)} x_{uv} = 1, \\
\forall ~u,~ \sum_{w \in N_-(u)} x_{wu} = 1.
\end{align*}
(Intuitively, the constraints say that exactly one incoming edge and one outgoing edge must be chosen --- as in a directed cycle.) Does an optimal solution to this optimization problem {\em always yield} an optimal solution to TSP? Provide a proof or a counterexample with a full explanation.

[{\em Hint:} consider a feasible solution to the optimization problem and focus on the edges with $x_{uv}=1$. Do they have to form a {\em single} cycle?]

\titledquestion{LP Basics}
Consider the following two-variable linear program. The variables are $x, y \in \mathbb{R}$. And the constraints are:
\begin{align*}
2x + y & \le 10, \\
x &\ge 0, \\
y &\ge -1, \\
x+3y &\le 8.
\end{align*}
\begin{parts}
\part[4] Plot the feasible region for the linear program. (Diagram can be drawn approximately to scale + scanned).
\part[2] Suppose the objective function is to {\em maximize} $x+y$. Find the maximum value and the point at which this is attained.
\part[4] Say the maximum value found in part (b) is $C$. Then could you have concluded that $x+y \le C$ by just ``looking at'' the equations? [{\em Hint:} does adding equations, possibly with constants, yield a bound?]
\end{parts}

\titledquestion{Identifying Corners}[6]
Consider the following linear program, with variables $x_1, x_2, \dots, x_n \in \mathbb{R}$. Suppose that the constraints are:
\begin{align*}
0 \le x_i \le 1 \text{ for all $i$}, \\
x_1 + x_2 + \dots + x_n \le k,
\end{align*}
where $k$ is some integer between $1$ and $n$. Consider any point $(a_1, a_2, \dots, a_n)$ where $a_i \in \{0,1\}$, and exactly $k$ of the $\{a_i\}$ are $1$. Prove that any such point is (a) a feasible point for the LP, and (b) a corner point of the polytope defining the feasible set.

[{\em Hint:} To prove that a point is a corner point, we use the definition we mentioned in class: a feasible point $x$ is a corner point if and only if for {\em all} nonzero vectors $z$, either $x + z$ or $x-z$ is infeasible.]

[{\em Remark:} It turns out that these are the only corner points of the polytope, and so it is an example of a polytope defined by $2n+1$ constraints and having $\binom{n}{k}$ corner points.]

\titledquestion{Checking feasibility vs optimization}[6]
Some of the algorithms for linear programming (e.g. simplex) start off with one of the corner points of the feasible set.  This turns out to be tricky in general. In this problem, we will see that {\bf in general}, finding one feasible point is as difficult as actually performing the optimization!

Consider the following linear program (in $n$ variables $x_1, \dots, x_n$, represented by the vector $x$):
\begin{align*}
\text{minimize } &c^T x ~~\text{subject to} \\
a_1^T x &\ge b_1 \\
a_2^T x &\ge b_2 \\
&\cdots \\
a_m^T x &\ge b_m.
\end{align*}
Suppose you know that the optimum value (i.e. the minimum of $c^T x$ over the feasible set) lies in the interval $[-M, M]$ for some real number $M$ (this is typically possible in practice).  Suppose also that you have an {\bf oracle} that can take any linear program and say whether it is feasible or not.  Prove that using $O(\log (M/\epsilon))$ calls to the oracle, one can determine the optimum value of the LP above up to an error of $\pm \epsilon$, for any given accuracy $\epsilon > 0$.  [\emph{Hint: } can you write a new LP that is feasible only if the LP above has optimum value $\le z$, for some $z$?]

\end{questions}
\end{document}


\chapter{Liquid Nitrogen Cooling in IR Thermography applied to steel specimen}
The results of this study were firstly presented at an oral session of SPIE Commercial+ Scientific Sensing and Imaging. International Society for Optics and Photonics, 2017.
Then it was accepted for publication in Proceedings Volume 10214, Thermosense: Thermal Infrared Applications XXXIX; 102140T (2017).

%
\section{Résumé}
La thermographie pulsée (PT) est l'une des méthodes les plus courantes dans les procédures de thermographie active de la thermographie pour NDT \& E (essais non destructifs  \& évaluation), en raison de la rapidité et la commodité de cette technique d'inspection. Des éclairs ou des lampes sont souvent utilisés pour chauffer les échantillons dans la PT traditionnelle. Cet article explore principalement exactement la stimulation externe opposée en IR Thermographie: refroidissement au lieu de chauffage. Un échantillon d'acier avec des trous à fond plat de différentes profondeurs et tailles a été testé. De l'azote liquide (LN$_2$) est répandu sur la surface de l'échantillon et l'ensemble du processus est capturé par une caméra thermique. Pour obtenir une bonne comparaison, deux autres techniques classiques de CND, la thermographie pulsée et la thermographie verrouillée, sont également utilisées. En particulier, la méthode Lock-in est implémentée avec trois fréquences différentes. Dans la procédure de traitement d'image, la méthode de thermographie en composantes principales (PCT) a été effectuée sur toutes les images thermiques. Pour les résultats Lock-in, les images de phase et d'amplitude sont générées par la transformée de Fourier rapide (FFT). Les résultats montrent que toutes les techniques présentaient en partie les défauts tandis que la technique LN $_2$ affichait les défauts seulement au début du test. De plus, un poste-traitement de seuil binaire est appliqué aux images thermiques, et en comparant ces images à une carte binaire de l'emplacement des défauts, les courbes caractéristiques de fonctionnement du récepteur (ROC) correspondantes sont établies et discutées. Une comparaison des résultats indique que la meilleure courbe ROC est obtenue en utilisant la technique flash avec la méthode de traitement PCT.

%Cette recherche étudie une stimulation externe - refroidissement au lieu de chauffer en thermographie infrarouge pour NDT \& E. Un spécimen en acier est utilisé afin de  tester trois stimulations différentes sur les images thermiques et également une comparaison d'analyse ROC. Les résultats montrent que toutes les techniques mettent en évidence une partie des défauts de l'échantillon, alors que la technique LN $ _2 $ ne représente les défauts qu'au début; ceci peut être dû à la conductivité élevée de l'acier. Dans les résultats thermiques, la méthode de post-traitement PCT affiche de meilleurs résultats pour toutes les procédures. Plus de défauts sont exposés dans la stimulation Flash avec le traitement PCT.
% Les résultats de cette étude ont d'abord été présentés lors d'une session orale de SPIE Thermosense: Thermal Infrared Applications XXXIX 2017, aux Etats-Unis.

\section{Abstract}
Pulsed Thermography (PT) is one of the most common methods in Active Thermography procedures of the Thermography for NDT \& E (Nondestructive Testing \& Evaluation), due to the rapidity and convenience of this inspection technique. Flashes or lamps are often used to heat the samples in the traditional PT. This paper mainly explores exactly the opposite external stimulation in IR Thermography: cooling instead of heating. A steel sample with flat-bottom holes of different depths and sizes has been tested. Liquid nitrogen (LN$_2$) is sprinkled on the surface of the specimen and the whole process is captured by a thermal camera. To obtain a good comparison, two other classic NDT techniques, Pulsed Thermography and Lock-In Thermography, are also employed. In particular, the  Lock-in  method  is  implemented  with  three  different  frequencies.  In  the  image  processing  procedure,  the Principal Component Thermography (PCT) method has been performed on all thermal images. For Lock-In results, both Phase and Amplitude images are generated by Fast Fourier Transform (FFT). Results show that all techniques presented part of the defects while the LN$_2$ technique displays the flaws only at the beginning of the test. Moreover, a binary threshold post-processing is applied to the thermal images, and by comparing these images to a binary map of the location of the defects, the corresponding Receiver Operating Characteristic (ROC) curves are established and discussed. A comparison of the results indicates that the better ROC curve is obtained using the Flash technique with PCT processing method.

\newpage
\textbf{\texttt{Contributing authors:}}

\textbf{\textsf{Lei Lei}} (Ph.D candidate): developing protocol, experiment preparation and planning, data analysis,  personnel coordination and manuscript preparation.

\textbf{Giovanni Ferrarini} (Researcher of CNR-ITC): discussion in developing protocol, experiment preparation.

\textbf{Alessandro Bortolin} (Ph.D student of CNR-ITC): discussion and experiment preparation.

\textbf{Gianluca Cadelano} (Ph.D student of CNR-ITC): data collection, discussion and experiment preparation.

\textbf{Paolo Bison} (Research supervisor of CNR-ITC): student supervision, revision and correction of the manuscript.

\textbf{Xavier Maldague} (Research director of LVSN in University Laval): student supervision, revision and correction of the manuscript.

% \phantomsection\addcontentsline{lof}{table}{5.1\quad Experimental set-up in the reflection mode}
% \phantomsection\addcontentsline{lof}{table}{5.2\quad  Steel sample dimension details with Flat-Bottom Holes of different depths and sizes.}
% \phantomsection\addcontentsline{lof}{table}{5.3\quad Experimental set-up for LN$_2$ cooling}
% \phantomsection\addcontentsline{lof}{table}{5.4\quad One example of ROC analysis (LN$_2$ results) and binary map of defect locations}
% \phantomsection\addcontentsline{lof}{table}{5.5\quad Thermal Raw Images of PT and LN$_2$ stimulation techniques}
% \phantomsection\addcontentsline{lof}{table}{5.6\quad FFT in amplitude and Phase results for LIT}
% \phantomsection\addcontentsline{lof}{table}{5.7\quad PCT results of corresponding technique}
% \phantomsection\addcontentsline{lof}{table}{5.8\quad ROC curves obtained from above results}

% \includepdf[pages=-]{Thermosense2017_Lei}
\newpage
\section{Introduction}
\label{sect:intro}  % \label{} allows reference to this section
In the Nondestructive Testing \& Evaluation (NDT \& E) field, active InfraRed (IR) thermography (\citet{Maldague2001theory}) is a technique widely used in assessing the conditions of parts of material components, with an extremely broad range of applications (\citet{Vavilov2017Thermal, Cadelano2016Corrosion}). Traditionally, Pulsed Thermography (PT) deploys a thermal stimulation pulse (flash or lamp heating) to produce a thermal contrast between the feature of interest and the background, then monitors the time evolution of the surface temperature by a thermal camera. With this rapidity and convenience, numerous studies have been devoted to this technique, that is now a standard procedure for thermal testing (\citet{Maldague1993Nondestructive,Maldague1994bInfra,2007-Ibarra-Castanedo,2011-ClementeIbarra-Castanedo,duan2013quantitative,Vavilov2015Review}).
%\added [id=GF] {that is now a standard procedure for thermal testing.}

However, if the temperature of the material to inspect is already higher than the ambient temperature, it can be of interest to make use of a cold thermal source such as a line of air jets (or water jets; sudden contact with ice, snow, etc.). In fact, a thermal front propagates the same way whether being hot or cold: what is important is the temperature differential between the thermal source and the specimen. An advantage of a cold thermal source is that it does not induce spurious thermal reflections into the IR camera as in the case of a hot thermal source. The main limitations of cold stimulation sources are related to practical considerations, as it is generally easier and more efficient to heat rather then to cool a part.
% \replaced [id=GF] {, as it is generally easier and more efficient to heat rather then to cool a part.} {as for instance it is generally easier and more efficient, to heat rather then to cool a part.}
Thus, the advantage and convenience of using the cold stimulation in active infrared thermography  still remains to be investigated in detail and better understood.

Using a cold source could extend the range of application of infrared thermography to cases where the specimen under analysis could not be heated, due to safety issues or physical reasons~(\citet{Livingston2018High}). A cold source could be required while handling biological or organic materials   that could not withstand a temperature increase, such as food. Another field of investigation is the survey of concrete and composite materials in order to find cracks and delaminations due to the presence of water or ice. Also in this case, a cold source would significantly decrease the risk of altering the physical characteristics of the specimen.
% \added [id=GF] {Using a cold source could extend the range of application of infrared thermography to cases where the specimen under analysis could not be heated, due to safety issues or physical reasons. A cold source could be required while handling biological or organic materials that could not withstand a temperature increase, such as food. Another field of investigation is the survey of concrete and composite materials in order to find cracks and delaminations due to the presence of water or ice. Also in this case, a cold source would significantly decrease the risk of altering the physical characteristics of the specimen}

Nonetheless,in the past in the scientific community, only a limited number of studies investigating a cold approach (cooling as the external stimulation in active infrared thermography) have been performed on industrial product inspection~(\citet{endohdynamical2012,2012-LewisHom}). A study by Burleigh~(\citet{Burleigh1989Thermographic}) showed that the cooling method with refrigerating liquids is feasible but requires caution to ensure the safety.
% \added [id=GF] {A study by Burleigh(\citet{1989Burleigh} showed that the cooling method with refrigerating liquids is feasible but requires caution to ensure the safety.}
The work of (\citet{lei2017detection}) chose instead to use air cooling to survey refrigerated vehicles. All the available works do not give a quantitative information about the reliability of the cooling technique, especially in comparison with the traditional heating procedure.
% \replaced [id=GF] {chose instead to use air cooling to survey refrigerated vehicles.} {demonstrates an example that air cooling approach applied in detection of the truck panel.}
% \added [id=GF] {All the available works do not give a quantitative information about the reliability of the cooling technique, especially in comparison with the traditional heating procedure.}

Therefore, the aim of this paper is improving the current knowledge on cooling thermal stimulation
% \replaced [id=GF] {improving the current knowledge on cooling thermal stimulation} {then the continuous investigation of cold approach}
in IR thermography. In order to perform a reliable comparison, three methods (two traditional techniques, Pulsed Thermography and Lock-in Thermography act as the reference) will be applied on a steel slab with different sizes of flat-bottom-holes. The thermographic images of the experiments will be  treated to  eventually produce a binary map of the location of the defects. This map will be statistically evaluated in terms of sensitivity and specificity~(\citet{Fawcett2006}) by comparison with the `true' map of the defects, furnishing a rank of the three stimulation methods.

\section{Experimental setup} % (fold)
\label{sec:experimental_setup}
One side stimulation approach is often used in the Infrared Thermography NDT \& E field, which is also known as the reflection scheme: both the stimulation device and the camera stay on the same side of the sample being tested. This approach applied in reality is shown in Figure~\ref{Exp_setup}.

\begin{figure}[ht]
   % \centering
   % \begin{center}
   \hspace{-0.95cm}
   \begin{tabular}{c}
   \includegraphics[scale=0.95]{chp5/Exp_setup.png}
   \\
   \footnotesize{(a) Pulsed Thermography set-up} \hspace{4cm} \footnotesize{(b) Lock-in Thermography set-up}
   \end{tabular}
   % \end{center}
   \caption{Experimental set-up in the \textit{reflection} mode}
   \label{Exp_setup}
\end{figure}

% \begin{figure}[ht]
%    \centering
%    \subfloat[Pulsed Thermography set-up]
%    {
%       \includegraphics[scale=0.3]{chp5/Flash_Setup.png}
%    }
%    %\hspace{5pt}
%    \subfloat[Lock-in Thermography set-up]
%    {
%       \includegraphics[scale=0.3]{chp5/LIT_setup.png}
%    }
%    \caption{Experimental set-up in the \textit{reflection} mode}
%    \label{Exp_setup}
% \end{figure}

The following equipment was set up for this study:
\begin{itemize}
   \item Infrared Camera FLIR SC3000 (spatial resolution equal to 320$\times$240 pixels, frame rate up to 50Hz, GaAs sensor, spectral range 8-9 $\mu m$)
%    \replaced [id=GF] {spatial resolution equal to 320$\times$240 pixels, framerate up to 50Hz, GaAs sensor, spectral range 8-9 $\mu m$)}{320$\times$240 pixels, framerate 50Hz, GaAs , 8-9 $\mu m$)}
   \item Two pairs of flash
%    \replaced [id=GF] {}{halogen}
    lamps for a total of 10 kJ (electric) released in 5 $ms$
   \item One pair of modulated halogen lamps with 1kW each served as Lock-in stimulation
   \item An isolated bottle (500 ml)
%   ??? \added [id=LL] {need to be rechecked, thanks Giovanni~}
filled with
%    \replaced [id=GF] {(500 ml???) filled with}{full of}
   Liquid Nitrogen.
\end{itemize}

\subsection{Specimen} % (fold)
\label{sub:specimen}
In this study, a steel specimen comprising flat-bottom holes of different depths and sizes will be examined. Their dimensions are depicted in Figure~\ref{specimen}.
   \begin{figure}[ht]
   \centering
   % \begin{tabular}{c} %% tabular useful for creating an array of images
   \includegraphics[scale=0.4]{chp5/specimen_schema.pdf}
   % \end{tabular}
   \caption{Steel sample dimension details with Flat-Bottom Holes of different depths and sizes (mirror image)}
%    \added [id=LL] {Mirror image}
%>>>> use \label inside caption to get Fig. number with \ref{}
    \label{specimen}
   \end{figure}

The steel plate has seventeen holes whose diameters vary from 0.4 $cm$  to 3 $cm$, and whose depths (from bottom) vary from 0.3 $cm$ to 0.9 $cm$. The entire thickness is 1 $cm$. This specimen is painted before the test, in order to increase its emissivity and to obtain a homogeneous external stimulation.

% subsection specimen (end)
\subsection{Stimulation Techniques} % (fold)
\label{sub:stimulation_techniques}
Three external stimulations were deployed on the sample, for the sake of  obtaining a good comparison in results:
\begin{itemize}
   \item Pulse Thermography (PT)
   \item Lock-in Thermography (LIT)
   \item Liquid Nitrogen cooling (LN$_2$)
\end{itemize}
Known as the traditional and fast technique in NDT \& E, Pulse Thermography (PT) acts as the reference during this test. One reason for this popularity is the quickness of the inspection relying on a thermal  stimulation pulse, with duration going from a few ms for high thermal conductivity material inspection (such as metal specimen in this study) to a few seconds for low thermal conductivity specimens.  Such quick thermal stimulation allows direct deployment on the plant floor with convenient heating sources.

% When neglecting heat exchange with the environment, the pulse of energy $Q$, delivered on a layer of thickness $L$, characterized by a density $\rho$, a specific heat $C_p$ and a thermal conductivity $\lambda$ (or a thermal diffusivity $\alpha$) produces a temperature increment behavior on the heated surface given by:
% \begin{equation}
%    T(t) = \frac{Q}{\rho C_p L}[1+2\sum_{n=1}^{\infty} e^{-\frac{n^2 \pi ^2\alpha t}{L^2}}]
%    \label{eq_pt}
% \end{equation}

Lock-in thermography (LIT) is based on thermal waves generated inside the inspected specimen and detected remotely. Wave generation is for instance performed by periodically depositing heat on the specimen surface (e.g. through sine-modulated lamp heating) while the resulting oscillating temperature field in the stationary regime is remotely recorded through its thermal infrared emission.

In this study, 3 different frequencies of 0.0625~$Hz$ (LIT16--one period is equivalent to 16 seconds), 0.125~$Hz$ (LIT8--one period is equivalent to 8 seconds) and 0.25~$Hz$ (LIT4--one period is equivalent to 4 seconds) are performed in Lock-in Thermography.

% By the convolution integral, Eq~(\ref{eq_pt}) becomes:
% \begin{equation}
%    T(t) = \frac{W}{\lambda}\frac{\alpha}{L}\int_0^t d\tau \Big(1+\sin(\omega \tau - \frac{\pi}{2})\Big)\Big\{1+2\sum_{n=1}^{\infty} e^{-\frac{n^2 \pi ^2\alpha(t-\tau)}{L^2}}\Big\}
% \end{equation}
% where $W$ is the absorbed heating power.

The Liquid Nitrogen is applied in the test by means of pouring it out directly onto the surface in order to cool the sample. LN$_2$ was sprinkled onto the specimen center and allowed to spread out towards the edges. The whole capture duration is 500 frames with 50~$Hz$ of image frequency, ie. 10 seconds of recording. The pouring time (cooling time) is about 2 seconds, due to the bottle with a volume of 500 ml, with 100 frames in the results sequence. This way of cooling has a limit of that the central part will be first and mainly cooled, then the cold front propagates from center to the edges. This can lead to a drawback that in the final thermograms defects closer to the edges will be difficulty to detected. The experimental set-up is shown in Figure~\ref{Exp_LN2}.

\begin{figure}[ht]
   \centering
   \includegraphics[scale=0.4]{chp5/LN2_setup.png}
   \caption{Experimental set-up for LN$_2$ cooling}
   \label{Exp_LN2}
\end{figure}

% subsection stimulation_techniques (end)

% section experimental_setup (end)

\section{Processing Methods} % (fold)
\label{sec:processing_methods}
The following image-processing techniques and data-analysis methods were employed for this study:
\begin{itemize}
   \item Principal Component Thermography (PCT)
   \item Phase and Amplitude images by Fast Fourier Transform (FFT)
   \item Receiver Operating Characteristic curves (ROC Curves)
\end{itemize}

\subsection{Principal Component Thermography (PCT)}
The Principal Component Thermography technique~(\citet{Rajic2002}) uses ``singular value decomposition (SVD) to reduce the matrix of observations to a highly compact statistical representation of the spatial and temporal variations relating to contrast information associated with underlying structural flaws".

\subsection{FFT in Phase and Amplitude for LIT}
In addition to the common technique for NDT \& E, Fast Fourier Transform in LIT~(\citet{wu1998lock}) is also one of the most applied techniques in IR Thermography, which is based on the periodic heating of the object being tested. A thermal wave is likewise generated and propagates inside the material. In real experimental cases the thermal wave is composed by a principal frequency and several harmonics where the amplitude of the Fast Fourier Transform is a function of frequency. By selecting the component with the highest amplitude it is possible to produce a phase map at the corresponding frequency where the defect appears enhanced.

\subsection{ROC Curve analysis} % (fold)
\label{sub:roc_curve_analysis}
The Receiver operating characteristic (ROC) curve is a technique in statistics which helps visualize, organize and select classifiers based on their performance. The curve graph is created by plotting the the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings. More details about concepts and definitions can be found in (\citet{Fawcett2006}). While being frequently chosen a standard method in several scientific fields, ROC are rarely applied in the thermographic field~(\citet{Bison2014a}).

Implemented in this study, a binary map of the defect locations is built and correlated to the post-processed images in gray scale, and the lay-out is provided in Figure~\ref{binary}.
% \deleted [id=LL]{, and the lay-out is provided in Figure~\ref{binary}}.
The main algorithm in the calculation of TPR and FPR is clarified as:
\begin{enumerate}
   \item Choose the post-processed thermogram in which most defects shown as the test image;
%    \replaced [id=LL] {Choose the post-processed thermogram in which most defects shown as the test image;}{Identify the best gray-scale result from the post-processing thermal images as the test image;}
   \item Resize the defect map to the same size as that of the test image;
   \item Choose a thresholding step number $N$($N=1000$ in this study) and establish the step value of thresholding [from $0$, $\frac{1}{N}$, $\frac{2}{N}$ till $1 (=\frac{N}{N})$];
   \item For each thresholding step, binarize the test image with the thresholding and then compare it to the defect map, in order to obtain the corresponding TPR and FPR values;
   \item Iterate the binarization and comparison so as to plot a whole curve.
\end{enumerate}

% \begin{figure}[ht]
%    \begin{center}
%    \begin{tabular}{c}
%    \includegraphics[scale=1.0]{ROC_exm.png}
%    \\\footnotesize{(a) Binary map of defects} \hspace{4cm} \footnotesize{(b) Cool map}
%    \end{tabular}
%    \end{center}
%    \caption{One example of ROC analysis (LN$_2$ results) and binary map of defect locations}
%    \label{binary}
% \end{figure}

\begin{figure}[ht]
   \centering
   \subfloat[Binary map of defects]
   {
      \includegraphics[scale=0.195]{chp5/Schema_done.png}
      \label{bin_map}
   }
   \subfloat[Cool map]
   {
      \includegraphics[scale=0.8]{chp5/Cool_ROC.png}
      \label{cool_map}
   }
   \caption{One example of ROC analysis (LN$_2$ results) and binary map of defect locations}
   \label{binary}
\end{figure}

%% subsection roc_curve (end)

% section methods (end)
\section{Results \& Discussion} % (fold)
\label{sec:results_&_discussion}
Figure~\ref{raw_results} illustrates the thermal raw images of PT and LN$_2$. The Lock-In FFT results can be found in Figure~\ref{LIT_results}.

% \begin{figure}[ht]
%    \begin{center}
%    % \begin{tabular}{c}
%    \includegraphics[scale=0.60]{chp5/Raw_results.png}
%    % \\\footnotesize{(a) Binary map of defects} \hspace{4cm} \footnotesize{(b) Cool map}
%    % \end{tabular}
%    \end{center}
%    \caption{Thermal raw images of PT and LN$_2$ stimulation techniques}
%    \label{raw_results}
% \end{figure}

\begin{figure}[htpb]
   \centering
   \subfloat[Flash raw frame 23]
   {
      \includegraphics[scale=0.55]{chp5/flash_raw23_2.png}
      \label{Flash_raw23}
   }
   \hspace{10pt}
   \subfloat[Flash raw frame 60]
   {
      \includegraphics[scale=0.55]{chp5/flash_raw60_2.png}
      \label{Flash_raw60}
   }
   \hspace{10pt}
   \subfloat[LN$_2$ raw frame 41]
   {
      \includegraphics[scale=0.55]{chp5/cool_raw_2.png}
      \label{LN2_raw}
   }
   \caption{Thermal raw images of PT and LN$_2$ stimulation techniques}
   \label{raw_results}
\end{figure}

% \begin{figure}[ht]
%    \begin{center}
%    % \begin{tabular}{c}
%    \includegraphics[scale=0.6]{chp5/LIT_results.png}
%    % \\\footnotesize{(a) Binary map of defects} \hspace{4cm} \footnotesize{(b) Cool map}
%    % \end{tabular}
%    \end{center}
%    \caption{FFT in amplitude and phase results for LIT}
%    \label{LIT_results}
% \end{figure}

\begin{figure}[htpb]
   \centering
   \subfloat[LIT4 FFT in amplitude]
   {
      \includegraphics[scale=0.57]{chp5/LIT4_AMP.png}
   }
   \hspace{10pt}
   \subfloat[LIT4 FFT in phase]
   {
      \includegraphics[scale=0.57]{chp5/LIT4_PHA.png}
   }
   \hspace{10pt}
   \subfloat[LIT8 FFT in amplitude]
   {
      \includegraphics[scale=0.57]{chp5/LIT8_AMP.png}
   }
   \hspace{10pt}
   \subfloat[LIT8 FFT in phase]
   {
      \includegraphics[scale=0.57]{chp5/LIT8_PHA.png}
      \label{LIT8_ph}
   }
   \hspace{10pt}
   \subfloat[LIT16 FFT in amplitude]
   {
      \includegraphics[scale=0.57]{chp5/LIT16_AMP.png}
      \label{LIT16_ph}
   }
   \hspace{10pt}
   \subfloat[LIT16 FFT in phase]
   {
      \includegraphics[scale=0.57]{chp5/LIT16_PHA.png}
   }
   \caption{FFT in amplitude and phase results for LIT}
   \label{LIT_results}
\end{figure}

\subsection{Thermal images comparison}

From the results above, raw images in Figure~\ref{raw_results} indicate that the most detectable flaws are the ones with a high aspect ratio (ie. diameter-to-depth). In addition, for the PT stimulation, a small hole with diameter 0.4 $cm$, depth\footnote{It should be noted that the depth values mentioned here and after are from the bottom of the sample, therefore the real corresponding depths should be these values subtracted from the thickness.} 0.9 $cm$ (left-upper in Figure~\ref{Flash_raw23}) appeared in frame 23, and disappeared after. Other holes, two with diameter 2 $cm$ and depth 0.9 $cm$, and two with diameter 3 $cm$ and depth 0.5 $cm$,
% \replaced [id=GF]{. Other holes, two with diameter 2 $cm$ and depth 0.9 $cm$, and two with diameter 3 $cm$ and depth 0.5 $cm$,} {, while another two holes with diameter 2 $cm$, depth 0.9 $cm$ and two with diameter 3 $cm$, depth 0.5 $cm$}
(center in Figure~\ref{Flash_raw60}) appeared in frame 60.  Nonetheless, in LN$_2$ raw results, same defects as the one in Flash frame 60 appeared in LN$_2$ frame 41 (however, it should be noted that the first 100 frames of cooling time has been removed from the these final raw images. The reason is because there was much noise from nitrogen gases in the thermal images).
% \added [id=LL] {The reason is because there was much noise from nitrogent gases in the thermal images})
After this, there was no more defect that emerged. Another remark is that since the LN$_2$ is sprinkled onto the center of the surface, there could be the situation that the center part of specimen was over cooled while the `cold front' (opposite to heat front) might have not be able to propagate to the edges. Due to the high conductivity of steel, defects only showed up at the beginning
% \replaced [id=LL] {end} {beginning}
of the cooling procedure. These may be the main issues of pouring-out method.

For LIT results, FFT in amplitude has a better flaw detection capability than FFT in phase, as there is some noise in all of the phase images. LIT8 and LIT16 have about four more detected flaws than LIT4. Whereas, for FFT in phase for LIT8 and LIT16, there some inverse gray-scale values occur. This might be due to the reverse image question.

Following PCT processing, figure~\ref{PCT_results} exhibits a clearer result. It can be observed that most of the flaws are visible, especially in the flash image (Figure~\ref{PCT_Flash}). Less flaws are visible in the LIT4 PCT third image.
%The corresponding PCT results are represented in Figure~\ref{PCT_results}.
% \begin{figure}[ht]
%    \begin{center}
%    % \begin{tabular}{c}
%    \includegraphics[scale=0.6]{chp5/PCT_results.png}
%    % \\\footnotesize{(a) Binary map of defects} \hspace{4cm} \footnotesize{(b) Cool map}
%    % \end{tabular}
%    \end{center}
%    \caption{PCT results of corresponding technique}
%    \label{PCT_results}
% \end{figure}

\begin{figure}[htpb]
    \centering
    \subfloat[Flash PCT 2nd Image]{
      \includegraphics[scale=0.57]{chp5/Flash_PCT_2.png}
      \label{PCT_Flash}
      }
    \hspace{10pt}
    \subfloat[LN$_2$ PCT 2nd Image]{
      \includegraphics[scale=0.57]{chp5/Cool_PCT_2.png}
      }
    \hspace{10pt}
    \subfloat[LIT4 PCT 3rd Image]{
      \includegraphics[scale=0.57]{chp5/LIT4_PCT_3.png}
      }
    \\ %\hspace{10pt}
    \subfloat[LIT8 PCT 3rd Image]{
      \includegraphics[scale=0.57]{chp5/LIT8_PCT_3.png}
      }
    \hspace{10pt}
    \subfloat[LIT16 PCT 3rd Image]{
      \includegraphics[scale=0.57]{chp5/LIT16_PCT_3.png}
      }
    % \includegraphics[scale=0.4]{graph/LIT4_PCT_3.png}
    % \includegraphics[scale=0.4]{graph/LIT8_PCT_3.png}
    % \includegraphics[scale=0.4]{graph/LIT16_PCT_3.png}
    \caption{PCT results of corresponding technique}
    \label{PCT_results}
\end{figure}

Comparing the processed thermal images, the following observations can be made:
\begin{itemize}
    \item All techniques present part of the flaws in the sample;
    \item The PCT post-processing method displays a better results for all images;
    \item More defects are exhibited in Flash stimulation with PCT processing;
    %\item For LIT8 and LIT16, there are some reverse image question.
\end{itemize}

\subsection{Corresponding ROC curves comparison}
The ROC curves obtained from comparing the binary map of defect locations to the above results are represented in Figure \ref{roc_pct} and ~\ref{ROC_curve}.
\begin{figure}[htbp]
   \centering
   \includegraphics[scale=0.7]{chp5/ROC_PCT_2017.pdf}
   \caption{ROC curves for PCT processing results}
   \label{roc_pct}
\end{figure}

% \begin{figure}[htbp]
%    % \centering
%    \begin{center}
%    \begin{tabular}{c}
%    \includegraphics[scale=0.75]{chp5/ROC_LIT_results.png}
%    \\\footnotesize{(a) ROC from LIT FFT in amplitude results} \hspace{2.1cm} \footnotesize{(b)  ROC from LIT FFT in phase results}
%    \end{tabular}
%    \end{center}
%    \caption{ROC curves for LIT processing results}
%    \label{roc_lit}
% \end{figure}

\begin{figure}[ht]
    \centering
    % \subfloat[ROC from PCT Results]
    % {
    % \includegraphics[scale=0.75]{chp5/ROC_PCT_2017.pdf}
    % }
    % %\hspace{10pt}
    % \\
    \subfloat[ROC from LIT FFT in amplitude results]
    {
    \includegraphics[scale=0.70]{chp5/ROC_LIT_AMP_2017.pdf}
    \label{roc_amp}
    }
    \hspace{10pt}
    \subfloat[ROC from LIT FFT in phase results]
    {
    \includegraphics[scale=0.70]{chp5/ROC_LIT_PHA_2017.pdf}
    \label{roc_pha}
    }
    \caption{ROC curves for LIT processing results}
    \label{ROC_curve}
\end{figure}

%\added[id=GF] {Before you called the true positive rate TPR (and the false FPR). From here you call them tp rate and fp rate, maybe it is better to choose the same nomenclature }

In the ROC curves definitions, the \textit{sensitivity} is the True Positive Rate (\textit{TPR}), and \textit{1-specificity} is the False Positive Rate (\textit{FPR}).
From the curves of the PCT results (Figure~\ref{roc_pct}), one can easily notice that the five curves have almost the same performance of classification in the beginning. When the \textit{TPR} arrives at $0.3$, the PT curve becomes the nearest to the northwest (where the \textit{TPR} is higher, the \textit{FPR} is lower or both). The second one is the LN$_2$ curve that has a slightly higher \textit{TPR} than PT after the \textit{FPR} reaches $0.4$. The LIT8 and LIT16 curves have almost the same performance before the \textit{TPR} attains $0.7$. After that the LIT16 has a higher \textit{TPR} than LIT8. The worst one is the LIT4 curve as it is the closest one to the diagonal line $y=x$, which represents the strategy of randomly guessing a class.

The situation is the same for the curves of LIT amplitude (Figure~\ref{roc_amp}), in which LIT4 has an unfavorable performance in classification. LIT16 also has a slightly higher \textit{TPR} than LIT8. However, in the curves of LIT phase results (Figure~\ref{roc_pha}), LIT16 shows a poor performance while LIT4 displays a better classification
% \added[id=GF] {while LIT4 displays a better classification}
, contrary to the former results.
% \replaced[id=GF] {} {While LIT4 displays a better classification.}
Because of the reverse image problem, the result LIT8 presents a region (top-right) which is under the equal diagonal line.

\subsection{Area under curve analysis and comparison}
In ROC analysis, a common method to compare different classifiers is to calculate the area under the ROC curve, known as AUC~(\citet{Fawcett2006}). In this way, a single scalar value will represent the expected performance of the classifiers. As in the ROC curve profile, the AUC is a part of the unit square are, therefore, its value will be between 0 and 1. Thus, random guessing classifier produces a diagonal line between (0,0) and (1,1), which has an area of 0.5. It can be then indicated that a classifier AUC value less than 0.5 is even worse than a random guessing.

The corresponding AUC obtained from the above ROC curves is found in Tab. \ref{tab_auc}. Their comparison in form of the bar chart is illustrated in Fig. \ref{fig_auc}. A straight comparison shows that the Flash method with PCT second component has the highest AUC value of 0.9. The second one is the Liquid Nitrogen method, that is very close to the flash method with an AUC value of 0.87. At the bottom of the ranking, LIT4 with PCT third component and its amplitude one have the worst values as  0.67 and 0.65 respectively, which are just a bit better and a random guessing.
On the contrary, LIT8 and LIT16 technique have a better result both in PCT processing and their amplitude images. This might be caused by the fact that during one period, 4 seconds of heating (LIT4) were not enough to penetrate the steel specimen deeply as those of 8 seconds (LIT8) and 16 seconds (LIT16). On the other hand, all the Lock-in Thermography techniques have similar AUC value in their phase images, which indicates that for phase image results, heating time does not make an effective influence in Lock-in Thermography in this study. Considering all these results, though the best approach is Flash technique with PCT processing, the LN$_2$ technique can be considered as a valid alternative method

% Figure AUC to be added.
% Table to be added.
\begin{table}[htbp]
    \centering
    \begin{tabular}{lr}%ccccccccc}
    \toprule
    \textbf{Method} & \textbf{Value} \\
    \midrule
    Flash-2nd & 0.90 \\
    LN2-2nd & 0.87 \\
    \midrule
    LIT4-3rd & 0.67 \\
    LIT8-3rd & 0.80 \\
    LIT16-3rd & 0.82 \\
    \midrule
    LIT4-AMP & 0.65 \\
    LIT8-AMP & 0.79 \\
    LIT16-AMP & 0.82 \\
    \midrule
    LIT4-PHA & 0.71 \\
    LIT8-PHA & 0.69 \\
    LIT16-PHA & 0.67 \\

%     0.90 & 0.87 & 0.67 & 0.80 & 0.82 & 0.65 & 0.79 & 0.82 & 0.71 & 0.69 & 0.67 \\
    \bottomrule
    \end{tabular}
    \caption{AUC value comparison}
    \label{tab_auc}
\end{table}

% $x = ['Flash_2nd', 'LN2_2nd', 'LIT4_3rd', 'LIT8_3rd', 'LIT16_3rd',
%      'LIT4_AMP', 'LIT8_AMP', 'LIT16_AMP', 'LIT4_PHA', 'LIT8_PHA', 'LIT16_PHA']\\
% y = np.array([0.90, 0.87, 0.67, 0.80, 0.82,
%               0.65, 0.79, 0.82, 0.71, 0.69, 0.67])$

\begin{figure}[htpb]
   \centering
   \includegraphics[scale=0.48]{chp5/AUC.png}
   \caption{AUC value comparison}
   \label{fig_auc}
\end{figure}

% section results_&_discussion (end)

\section{Conclusion} % (fold)
\label{sec:conclusion}
This study investigates an external stimulation--cooling instead of heating in IR Thermography for NDT \& E.
A steel specimen is used to test three different stimulations for thermal images and also ROC analysis comparison.
Results shows that all techniques highlight part of the flaws in the sample, whereas the LN$_2$ technique represents the defects at the end of cooling process; this maybe due to the high conductivity of steel.
In thermal results, the PCT post-processing method displays a better results for all procedures. More defects are exhibited in Flash stimulation with PCT processing.
ROC curve analysis and its AUC analysis have elucidated a straightforward classification comparison, in which the best values are obtained with the Flash technique with PCT processing, trailed narrowly by the Liquid Nitrogen method.

The LN$_2$ technique should therefore be considered as a valid option for the survey of objects that should not be heated, such as biological tissues, organic materials, dry or iced samples. With this purpose, future work will analyze different kinds of specimens.
% \replaced[id=GF] {The LN$_2$ technique should therefore be considered as a valid technique for the survey of objects that should not be heated, such as biological tissues, organic materials, dry or iced samples. With this purpose, future work will analyze different kinds of specimens.} {In future work, other common composite materials such as CFRP (Carbon fiber reinforced polymer), GFRP (Glass Fiber Reinforced Polymer) will be chosen as specimen.}
The method of Liquid Nitrogen pouring may be replaced by spraying onto the sample surface, which can reduce the inhomogeneous cooling problem. To enhance the penetration of heat inside the sample, a proposition involving heating on one side of the specimen and cooling the other side might be taken into consideration.
% \replaced[id=GF] {} {Even though LN$_2$ technique in this study has not shown enough advantages,}
% \replaced[id=GF] {The} {this}
The exploration of an opposite means of external stimulation in InfraRed Thermography might favor new ideas
% \replaced[id=GF] {} {of approaches}
for NDT \& E.

\section*{Acknowledgments}

This research was supported by the governments of Italy and Quebec (Minist\`{e}re des Relations internationales et de la Francophonie) through the Joint Subcommittee Qu\'{e}bec-Italy, project n$^{\circ}$08.0203. It was also supported by  the Natural Sciences and Engineering Research Council of Canada (NSERC). We are also thankful to our collaborative institute CNR-ITC Padova which provided expertise that greatly helped in this research.

\bibliographystyle{unsrtnat}              % style de la bibliographie
\bibliography{U:/Desktop/Bibliography/Biblio_th}