%-----------------------------------------------------------------------------------------------

\begin{fullwidth}
\textit{Development Research in Practice} focuses on tools, workflows, and practical
guidance for implementing research projects.
While not central to the content of this book,
We think it is essential for all research team members
-- including field staff and research assistants --
to understand research design,
and specifically how research design choices impact data work.
Without going into too much technical detail,
as there are many excellent resources on impact evaluation design,
this appendix presents a brief overview
of the most common causal inference methods,
focusing on implications for data structure and analysis.
This appendix is intended to be a reference,
especially for junior team members,
to obtain an understanding of the way in which each
causal inference method constructs treatment and control groups,
the data structures needed to estimate the corresponding effects,
and specific code tools designed for each method.

It is essential for the research team members who will do the data work
to understand the study design, for several reasons.
If you do not know how to calculate the correct estimator for your study,
you will not be able to assess the statistical power of your research design.
You will also be unable to make decisions in the field
when you inevitably have to allocate scarce resources
between tasks like maximizing sample size
and ensuring follow-up with specific individuals.
You will save time by understanding the way your data needs to be organized
to produce meaningful analytics throughout your projects.
Just as importantly, familiarity with each of these approaches
will allow you to keep your eyes open for research opportunities:
many of the most interesting projects occur because people in the field
recognize the opportunity to implement one of these methods
in response to an unexpected event.

This appendix is split into two sections.
The first covers causal inference methods
in experimental and quasi-experimental research designs.
The second discusses how to measure treatment effects and structure data for specific methods,
including cross-sectional randomized control trials, difference-in-difference designs,
regression discontinuity, instrumental variables, matching, and synthetic controls.

\end{fullwidth}

%-----------------------------------------------------------------------------------------------
%-----------------------------------------------------------------------------------------------

\section{Causality, inference, and identification}

When we are discussing the types of inputs -- ``treatments'' -- commonly referred to as
``programs'' or ``interventions'', we are typically attempting to obtain estimates
of program-specific \textbf{treatment effects}.
These are the changes in outcomes attributable to the treatment.\sidenote{\citet{abadie2018econometric}}
  \index{treatment effect}
The primary goal of research design is to establish \textbf{causal identification} for an effect.
Causal identification means establishing that a change in an input directly altered an outcome.
  \index{identification}
When a study is well-identified, then we can say with confidence
that our estimate of the treatment effect would,
with an infinite amount of data,
give us a precise estimate of that treatment effect.
Under this condition, we can proceed to draw evidence from the limited samples we have access to,
using statistical techniques to express the uncertainty of not having infinite data.
Without identification, we cannot say that the estimate would be accurate,
even with unlimited data, and therefore cannot attribute it to the treatment
in the small samples that we typically have access to.
More data is not a substitute for a well-identified experimental design.
Therefore it is important to understand how exactly your study
identifies its estimate of treatment effects,
so you can calculate and interpret those estimates appropriately.

All the study designs we discuss here use the potential outcomes framework\sidenote{\citet{athey2017state}}
to compare a group that received some treatment to another, counterfactual group.
Each of these approaches can be used in two types of designs:
\textbf{experimental} designs, in which the research team
is directly responsible for creating the variation in treatment,
and \textbf{quasi-experimental} designs, in which the team
identifies a ``natural'' source of variation and uses it for identification.
Neither type is implicitly better or worse,
and both types are capable of achieving causal identification in different contexts.

%-----------------------------------------------------------------------------------------------
\subsection{Estimating treatment effects using control groups}

The key assumption behind estimating treatment effects is that every
person, facility, or village (or whatever the unit of intervention is)
has two possible states: their outcomes if they do not receive some treatment
and their outcomes if they do receive that treatment.
Each unit's treatment effect is the individual difference between these two states,
and the \textbf{average treatment effect (ATE)} is the average of all
individual differences across the potentially treated population.
  \index{average treatment effect}
This is the parameter that most research designs attempt to estimate,
by establishing a \textbf{counterfactual}\sidenote{
  \textbf{Counterfactual:} A statistical description of what would have happened to specific individuals in an alternative scenario, for example, a different treatment assignment outcome.}
for the treatment group against which outcomes can be directly compared.
  \index{counterfactual}
There are several resources that provide more or less mathematically intensive
approaches to understanding how various methods do this.
\textit{Impact Evaluation in Practice}\sidenote{\citet{gertler2016impact}}
is a strong general guide to these methods.
\textit{Causal Inference}\sidenote{\citet{hernan2010causal}} and
\textit{Causal Inference: The Mixtape}\sidenote{\citet{cunningham2018causal}}
provides more detailed mathematical approaches to the tools.
\textit{Mostly Harmless Econometrics}\sidenote{\citet{angrist2008mostly}}
and \textit{Mastering Metrics}\sidenote{\citet{angrist2014mastering}}
are excellent resources on the statistical principles behind all econometric approaches.

Intuitively, the problem is as follows: we can never observe the same unit
in both their treated and untreated states simultaneously,
so measuring and averaging these effects directly is impossible.\sidenote{\citet{rubin2003basic}}
Instead, we typically make inferences from samples.
\textbf{Causal inference} methods are those in which we are able to estimate the
average treatment effect without observing individual-level effects,
but through some comparison of averages with a \textbf{control} group.
  \index{causal inference}\index{control group}
Every research design is based on a way of comparing another set of observations --
the ``control'' observations -- against the treatment group.
They all work to establish that the control observations would have been
identical \textit{on average} to the treated group in the absence of the treatment.
Then, the mathematical properties of averages imply that the calculated
difference in averages is equivalent to the average difference:
exactly the parameter we are seeking to estimate.
Therefore, almost all designs can be accurately described
as a series of between-group comparisons.

Most of the methods that you will encounter rely on some variant of this strategy,
which is designed to maximize their ability to estimate the effect
of an average unit being offered the treatment you want to evaluate.
The focus on identification of the treatment effect, however,
means there are several essential features of causal identification methods
that are not common in other types of statistical and data science work.
First, the econometric models and estimating equations used
do not attempt to create a predictive or comprehensive model
of how the outcome of interest is generated.
Typically, causal inference designs are not interested in predictive accuracy,
and the estimates and predictions that they produce
will not be as good at predicting outcomes or fitting the data as other models.
Second, when control variables or other variables are used in estimation,
there is no guarantee that the resulting parameters are marginal effects.
They can only be interpreted as correlative averages,
unless there are additional sources of identification.
The models you will construct and estimate are intended to do exactly one thing:
to express the intention of your project's research design,
and to accurately estimate the effect of the treatment it is evaluating.
In other words, these models tell the story of the research design
in a way that clarifies the exact comparison being made between control and treatment.

%-----------------------------------------------------------------------------------------------
\subsection{Experimental and quasi-experimental research designs}

Experimental research designs explicitly allow the research team
to change the condition of the populations being studied,\sidenote{
  \url{https://dimewiki.worldbank.org/Experimental_Methods}}
often in the form of government programs, NGO projects, new regulations,
information campaigns, and many more types of interventions.\sidenote{\citet{banerjee2009experimental}}
The classic experimental causal inference method
is the \textbf{randomized control trial (RCT)}.\sidenote{
  \url{https://dimewiki.worldbank.org/Randomized_Control_Trials}}
  \index{randomized control trials}
In randomized control trials, the treatment group is randomized --
that is, from an eligible population,
a random group of units are given the treatment.
Another way to think about these designs is how they establish the control group:
a random subset of units are \textit{not} given access to the treatment,
so that they may serve as a counterfactual for those who are.
A randomized control group, intuitively, is meant to represent
how things would have turned out for the treated group
if they had not been treated, and it is particularly effective at doing so
as evidenced by its broad credibility in fields ranging from clinical medicine to development.
Therefore RCTs are very popular tools for determining the causal impact
of specific programs or policy interventions,
as evidenced by the awarding of the 2019 Nobel Prize in Economics
to Abhijit Banerjee, Esther Duflo and Michael Kremer
``for their experimental approach to alleviating global poverty''.\cite{nobel2019}
However, there are many other types of interventions that are impractical or unethical
to effectively approach using an experimental strategy,
and therefore there are limitations to accessing ``big questions''
through RCT approaches.\sidenote{\citet{deaton2009}}

Randomized designs all share several major statistical concerns.
The first is the fact that it is always possible to select a control group,
by chance, which is not in fact very similar to the treatment group.
This feature is called randomization noise, and all RCTs share the need to assess
how randomization noise may impact the estimates that are obtained.
(More detail on this later.)
Second, take-up and implementation fidelity are extremely important,
since programs will by definition have no effect
if the population intended to be treated
does not accept or does not receive the treatment.\sidenote{
  See \citet{jung2016impact} for an example.}
Loss of statistical power occurs quickly and is highly nonlinear:
70\% take-up or efficacy doubles the required sample, and 50\% quadruples it.\sidenote{
  \url{https://blogs.worldbank.org/impactevaluations/power-calculations-101-dealing-with-incomplete-take-up}}
Such effects are also very hard to correct ex post,
since they require strong assumptions about the randomness or non-randomness of take-up.
Therefore a large amount of field time and descriptive work
must be dedicated to understanding how these effects played out in a given study,
and may overshadow the effort put into the econometric design itself.

\textbf{Quasi-experimental} research designs,\sidenote{
  \url{https://dimewiki.worldbank.org/Quasi-Experimental_Methods}}
by contrast, are causal inference methods based on events not controlled by the research team.
Instead, they rely on ``experiments of nature'',
in which natural variation can be argued to approximate
the type of exogenous variation in treatment availability
that a researcher would attempt to create with an experiment.\sidenote{\citet{dinardo2016natural}}
Unlike carefully planned experimental designs,
quasi-experimental designs typically require the extra luck
of having access to data collected at the right times and places
to exploit events that occurred in the past,
or having the ability to collect data in a time and place
where an event that produces causal identification occurred or will occur.
Therefore, these methods often use either secondary data,
or they use primary data in a cross-sectional retrospective method,
including administrative data or other new classes of routinely-collected information.

Quasi-experimental designs therefore can access a much broader range of questions,
and with much less effort in terms of executing an intervention.
However, they require in-depth understanding of the precise events
the researcher wishes to address in order to know what data to use
and how to model the underlying natural experiment.
Additionally, because the population exposed
to such events is limited by the scale of the event,
quasi-experimental designs are often power-constrained.
Since the research team cannot change the population of the study
or the treatment assignment, power is typically maximized by ensuring
that sampling for data collection is carefully designed to match the study objectives
and that attrition from the sampled groups is minimized.

%-----------------------------------------------------------------------------------------------
%-----------------------------------------------------------------------------------------------
\section{Obtaining treatment effects from specific research designs}

%-----------------------------------------------------------------------------------------------
\subsection{Cross-sectional designs}

A cross-sectional research design is any type of study
that observes data in only one time period
and directly compares treatment and control groups.
This type of data is easy to collect and handle because
you do not need to track individuals across time.
If this point in time is after a treatment has been fully delivered,
then the outcome values at that point in time
already reflect the effect of the treatment.
If the study is experimental, the treatment and control groups are randomly constructed
from the population that is eligible to receive each treatment.
By construction, each unit's receipt of the treatment
is unrelated to any of its other characteristics
and the ordinary least squares (OLS) regression
of outcome on treatment, without any control variables,
is an unbiased estimate of the average treatment effect.

Cross-sectional designs can also exploit variation in non-experimental data
to argue that observed correlations do in fact represent causal effects.
This can be true unconditionally -- which is to say that something random,
such as winning the lottery, is a true random process and can tell you about the effect
of getting a large amount of money.\sidenote{\citet{imbens2001estimating}}
It can also be true conditionally -- which is to say that once the
characteristics that would affect both the likelihood of exposure to a treatment
and the outcome of interest are controlled for,
the process is as good as random:
like arguing that once risk preferences are taken into account,
exposure to an earthquake is unpredictable and post-event differences
are causally related to the event itself.\sidenote{\citet{callen2015catastrophes}}

For cross-sectional designs, what needs to be carefully maintained in data
is the treatment randomization process itself (whether experimental or not),
as well as detailed information about differences
in data quality and attrition across groups.\sidenote{\citet{athey2017econometrics}}
Only these details are needed to construct the appropriate estimator:
clustering of the standard errors is required at the level
at which the treatment is assigned to observations,
and variables which were used to stratify the treatment
must be included as controls (in the form of strata fixed effects).\sidenote{
  \url{https://blogs.worldbank.org/impactevaluations/impactevaluations/how-randomize-using-many-baseline-variables-guest-post-thomas-barrios}}
\textbf{Randomization inference} can be used
to estimate the underlying variability in the randomization process
(more on this in the next chapter).
\textbf{Balance checks}\sidenote{
  \textbf{Balance checks:} Statistical tests of the similarity of treatment and control groups.}
are often reported as evidence of an effective randomization,
and are particularly important when the design is quasi-experimental
(since then the randomization process cannot be simulated explicitly).
However, controls for balance variables are usually unnecessary in RCTs,
because it is certain that the true data-generating process
has no correlation between the treatment and the balance factors.\sidenote{
  \url{https://blogs.worldbank.org/impactevaluations/should-we-require-balance-t-tests-baseline-observables-randomized-experiments}}

Analysis is typically straightforward \textit{once you have a strong understanding of the randomization}.
A typical analysis will include a description of the sampling and randomization results,
with analyses such as summary statistics for the eligible population,
and balance checks for randomization and sample selection.
The main results will usually be a primary regression specification
(with multiple hypotheses\sidenote{
  See \citet{armand2017public} for an example.} appropriately adjusted for),
and additional specifications with adjustments for non-response, balance, and other potential contamination.
Robustness checks might include randomization-inference analysis or other placebo regression approaches.
There are a number of user-written code tools that are also available
to help with the complete process of data analysis,
including to analyze balance\sidenote{
  \url{https://dimewiki.worldbank.org/iebaltab}}
and to visualize treatment effects.\sidenote{
  \url{https://dimewiki.worldbank.org/iegraph}}
Extensive tools and methods for analyzing selective non-response are available.\sidenote{
  \url{https://blogs.worldbank.org/impactevaluations/dealing-attrition-field-experiments}}

%-----------------------------------------------------------------------------------------------
\subsection{Difference-in-differences}

Where cross-sectional designs draw their estimates of treatment effects
from differences in outcome levels in a single measurement,
\textbf{differences-in-differences}\sidenote{
  \url{https://dimewiki.worldbank.org/Difference-in-Differences}}
designs (abbreviated as DD, DiD, diff-in-diff, and other variants)
estimate treatment effects from \textit{changes} in outcomes
between two or more rounds of measurement.
  \index{difference-in-differences}
In these designs, three control groups are used –
the baseline level of treatment units,
the baseline level of non-treatment units,
and the endline level of non-treatment units.\sidenote{\citet{torres2015}}
The estimated treatment effect is the excess growth
of units that receive the treatment, in the period they receive it:
calculating that value is equivalent to taking
the difference in means at endline and subtracting
the difference in means at baseline
(hence the singular ``difference-in-differences'').\sidenote{\citet{mckenzie2012beyond}}
The regression model includes a control variable for treatment assignment,
and a control variable for time period,
but the treatment effect estimate corresponds to
an interaction variable for treatment and time:
it indicates the group of observations for which the treatment is active.
This model depends on the assumption that,
in the absense of the treatment,
the outcome of the two groups would have changed at the same rate over time,
typically referred to as the \textbf{parallel trends} assumption.\sidenote{
  \url{https://blogs.worldbank.org/impactevaluations/often-unspoken-assumptions-behind-difference-difference-estimator-practice}}
Experimental approaches satisfy this requirement in expectation,
but a given randomization should still be checked for pre-trends
as an extension of balance checking.\sidenote{
  \url{https://blogs.worldbank.org/impactevaluations/revisiting-difference-differences-parallel-trends-assumption-part-i-pre-trend}}

There are two main types of data structures for differences-in-differences:
\textbf{repeated cross-sections} and \textbf{panel data}.
In repeated cross-sections, each successive round of data collection contains a random sample
of observations from the treated and untreated groups;
as in cross-sectional designs, both the randomization and sampling processes
are critically important to maintain alongside the data.
In panel data structures, we attempt to observe the exact same units
in different points in time, so that we see the same individuals
both before and after they have received treatment (or not).\sidenote{
  \url{https://blogs.worldbank.org/impactevaluations/what-are-we-estimating-when-we-estimate-difference-differences}}
This allows each unit's baseline outcome (the outcome before the intervention) to be used
as an additional control for its endline outcome,
which can provide large increases in power and robustness.\sidenote{
  \url{https://blogs.worldbank.org/impactevaluations/another-reason-prefer-ancova-dealing-changes-measurement-between-baseline-and-follow}}
When tracking individuals over time for this purpose,
maintaining sampling and tracking records is especially important,
because attrition will remove that unit's information
from all points in time, not just the one they are unobserved in.
Panel-style experiments therefore require a lot more effort in field work
for studies that use original data.\sidenote{\citet{torres2007}}
Since baseline and endline may be far apart in time,
it is important to create careful records during the first round
so that follow-ups can be conducted with the same subjects,
and attrition across rounds can be properly taken into account.\sidenote{
  \url{https://blogs.worldbank.org/impactevaluations/dealing-attrition-field-experiments}}

As with cross-sectional designs, difference-in-differences designs are widespread.
Therefore there exist a large number of standardized tools for analysis.
Our \texttt{ietoolkit} Stata package includes the \texttt{ieddtab} command
which produces standardized tables for reporting results.\sidenote{
  \url{https://dimewiki.worldbank.org/ieddtab}}
For more complicated versions of the model
(and they can get quite complicated quite quickly),
you can use an online dashboard to simulate counterfactual results.\sidenote{
  \url{https://blogs.worldbank.org/impactevaluations/econometrics-sandbox-event-study-designs-co}}
As in cross-sectional designs, these main specifications
will always be accompanied by balance checks (using baseline values),
as well as randomization, selection, and attrition analysis.
In trials of this type, reporting experimental design and execution
using the CONSORT style is common in many disciplines
and will help you to track your data over time.\sidenote{\citet{schulz2010consort}}

%-----------------------------------------------------------------------------------------------
\subsection{Regression discontinuity}

\textbf{Regression discontinuity (RD)} designs exploit sharp breaks or limits
in policy designs to separate a single group of potentially eligible recipients
into comparable groups of individuals who do and do not receive a treatment.\sidenote{
  \url{https://dimewiki.worldbank.org/Regression_Discontinuity}}
These designs differ from cross-sectional and diff-in-diff designs
in that the group eligible to receive treatment is not defined directly,
but instead created during the treatment implementation.
  \index{regression discontinuity}
In an RD design, there is typically some program or event
that has limited availability due to practical considerations or policy choices
and is therefore made available only to individuals who meet a certain threshold requirement.
The intuition of this design is that there is an underlying \textbf{running variable}
that serves as the sole determinant of access to the program,
and a strict cutoff determines the value of this variable at which eligibility stops.\sidenote{\citet{imbens2008regression}}\index{running variable}
Common examples are test score thresholds and income thresholds.\sidenote{
  \url{https://blogs.worldbank.org/impactevaluations/regression-discontinuity-porn}}
The intuition is that individuals who are just above the threshold
will be very nearly indistinguishable from those who are just under it,
and their post-treatment outcomes are therefore directly comparable.\sidenote{\citet{lee2010regression}}
The key assumption here is that the running variable cannot be directly manipulated
by the potential recipients.
If the running variable is time (what is commonly called an ``event study''),
there are special considerations.\sidenote{\citet{hausman2018regression}}
Similarly, spatial discontinuity designs are handled a bit differently due to their multidimensionality.\sidenote{
  \url{https://blogs.worldbank.org/impactevaluations/spatial-jumps}}

Regression discontinuity designs are, once implemented,
very similar in analysis to cross-sectional or difference-in-differences designs.
Depending on the data that is available,
the analytical approach will center on the comparison of individuals
who are narrowly on the inclusion side of the discontinuity,
compared against those who are narrowly on the exclusion side.\sidenote{\citet{cattaneo2019}}
The regression model will be identical to the matching research designs,
i.e., contingent on whether data has one or more time periods
and whether the same units are known to be observed repeatedly.
The treatment effect will be identified, however, by the addition of a control
for the running variable -- meaning that the treatment effect estimate
will only be applicable for observations in a small window around the cutoff:
in the lingo, the treatment effects estimated will be ``local'' rather than ``average''.
In the RD model, the functional form of the running variable control and the size of that window,
often referred to as the choice of \textbf{bandwidth} for the design,
are the critical parameters for the result.\sidenote{\citet{calonico2019regression}}
Therefore, RD analysis often includes extensive robustness checking
using a variety of both functional forms and bandwidths,
as well as placebo testing for non-realized locations of the cutoff.

In the analytical stage, regression discontinuity designs
often include a large component of visual evidence presentation.
These presentations help to suggest both the functional form
of the underlying relationship and the type of change observed at the discontinuity,
and help to avoid pitfalls in modeling that are difficult to detect with hypothesis tests.\sidenote{\citet{pischke2018}}
Because these designs are so flexible compared to others,
there is an extensive set of commands that help assess
the efficacy and results from these designs under various assumptions.\sidenote{\citet{calonico2014robust}}
These packages support the testing and reporting
of robust plotting and estimation procedures,
tests for manipulation of the running variable,
and tests for power, sample size, and randomization inference approaches
that will complement the main regression approach used for point estimates.

%-----------------------------------------------------------------------------------------------
\subsection{Instrumental variables}

\textbf{Instrumental variables (IV)} designs, unlike the previous approaches,
begin by assuming that the treatment delivered in the study in question is
linked to the outcome in a pattern such that its effect is not directly identifiable.
Instead, similar to regression discontinuity designs,
IV attempts to focus on a subset of the variation in treatment take-up
and assesses that limited window of variation that can be argued
to be unrelated to other factors.\sidenote{\citet{angrist2001instrumental}}
To do so, the IV approach selects an \textbf{instrument}
for the treatment status -- an otherwise-unrelated predictor of exposure to treatment
that affects the take-up status of an individual.\sidenote{
  \url{https://dimewiki.worldbank.org/instrumental_variables}}
Whereas regression discontinuity designs are ``sharp'' --
treatment status is completely determined by which side of a cutoff an individual is on --
IV designs are ``fuzzy'', meaning that they do not completely determine
the treatment status but instead influence the \textit{probability} of treatment.

As in regression discontinuity designs,
the fundamental form of the regression
is similar to either cross-sectional or difference-in-differences designs.
However, instead of controlling for the instrument directly,
the IV approach typically uses the \textbf{two-stage-least-squares (2SLS)} estimator.\sidenote{\citet{bond2020}}
This estimator forms a prediction of the probability that the unit receives treatment
based on a regression against the instrumental variable.
That prediction will, by assumption, be the portion of the actual treatment
that is due to the instrument and not any other source,
and since the instrument is unrelated to all other factors,
this portion of the treatment can be used to assess its effects.
Unfortunately, these estimators are known
to have very high variances relative other methods,
particularly when the relationship between the instrument and the treatment is small.\sidenote{\citet{young2017consistency}}
IV designs furthermore rely on strong but untestable assumptions
about the relationship between the instrument and the outcome.\sidenote{\citet{bound1995problems}}
Therefore IV designs face intense scrutiny on the strength and exogeneity of the instrument,
and tests for sensitivity to alternative specifications and samples
are usually required with an instrumental variables analysis.
However, the method has special experimental cases that are significantly easier to assess:
for example, a randomized treatment \textit{assignment} can be used as an instrument
for the eventual take-up of the treatment itself,\sidenote{
  See \citet{iacovone2019improving} for an example.}
especially in cases where take-up is expected to be low,
or in circumstances where the treatment is available
to those who are not specifically assigned to it (``encouragement designs'').

In practice, there are a variety of packages that can be used
to analyse data and report results from instrumental variables designs.
While the built-in Stata command \texttt{ivregress} will often be used
to create the final results, the built-in packages are not sufficient on their own.
The \textbf{first stage} of the design should be extensively tested,
to demonstrate the strength of the relationship between
the instrument and the treatment variable being instrumented.\sidenote{\citet{stock2005weak}}
This can be done using the \texttt{weakiv} and \texttt{weakivtest} commands.\sidenote{\citet{pfluegerwang2015}}
Additionally, tests should be run that identify and exclude individual
observations or clusters that have extreme effects on the estimator,
using customized bootstrap or leave-one-out approaches.\sidenote{\citet{young2017consistency}}
Finally, bounds can be constructed allowing for imperfections
in the exogeneity of the instrument using loosened assumptions,
particularly when the underlying instrument is not directly randomized.\sidenote{\citet{clarke2018}}

%-----------------------------------------------------------------------------------------------
\subsection{Matching}

\textbf{Matching} methods use observable characteristics of individuals
to directly construct treatment and control groups to be as similar as possible
to each other, either before a randomization process
or after the collection of non-randomized data.\sidenote{
  \url{https://dimewiki.worldbank.org/Matching}}
  \index{matching}
Matching observations may be one-to-one or many-to-many;
in any case, the result of a matching process
is similar in concept to the use of randomization strata
in simple randomized control trials.
In this way, the method can be conceptualized
as averaging across the results of a large number of ``micro-experiments''
in which the randomized units are verifiably similar aside from the treatment.

When matching is performed before a randomization process,
it can be done on any observable characteristics,
including outcomes, if they are available.
The randomization should then record an indicator for each matching set,
as these become equivalent to randomization strata and require controls in analysis.
This approach is stratification taken to its most extreme:
it reduces the number of potential randomizations dramatically
from the possible number that would be available
if the matching was not conducted,
and therefore reduces the variance caused by the study design.
When matching is done ex post in order to substitute for randomization,
it is based on the assertion that within the matched groups,
the assignment of treatment is as good as random.
However, since most matching models rely on a specific linear model,
such as \textbf{propensity score matching},\sidenote{
  \textbf{Propensity Score Matching (PSM):} An estimation method that controls for the likelihood
  that each unit of observation would recieve treatment as predicted by observable characteristics.}
they are open to the criticism of ``specification searching'',
meaning that researchers can try different models of matching
until one, by chance, leads to the final result that was desired;
analytical approaches have shown that the better the fit of the matching model,
the more likely it is that it has arisen by chance and is therefore biased.\sidenote{\citet{king2019propensity}}
Newer methods, such as \textbf{coarsened exact matching},\sidenote{\citet{iacus2012causal}}
are designed to remove some of the dependence on linearity.
In all ex-post cases, pre-specification of the exact matching model
can prevent some of the potential criticisms on this front,
but ex-post matching in general is not regarded as a strong identification strategy.

Analysis of data from matching designs is relatively straightforward;
the simplest design only requires controls (indicator variables) for each group
or, in the case of propensity scoring and similar approaches,
weighting the data appropriately in order to balance the analytical samples on the selected variables.
The \texttt{teffects} suite in Stata provides a wide variety
of estimators and analytical tools for various designs.\sidenote{\citet{sscc2015}}
The coarsened exact matching (\texttt{cem}) package applies the nonparametric approach.\sidenote{\citet{blackwell2009cem}}
DIME's \texttt{iematch} command in the \texttt{ietoolkit} package produces matchings based on a single continuous matching variable.\sidenote{
  \url{https://dimewiki.worldbank.org/iematch}}
In any of these cases, detailed reporting of the matching model is required,
including the resulting effective weights of observations,
since in some cases the lack of overlapping supports for treatment and control
mean that a large number of observations will be weighted near zero
and the estimated effect will be generated based on a subset of the data.

%-----------------------------------------------------------------------------------------------
\subsection{Synthetic controls}

\textbf{Synthetic control} is a relatively new method
for the case when appropriate counterfactual individuals
do not exist in reality and there are very few (often only one) treatment units.\sidenote{\citet{abadie2015comparative}}
  \index{synthetic controls}
For example, state- or national-level policy changes
that can only be analyzed as a single unit
are typically very difficult to find valid comparators for,
since the set of potential comparators is usually small and diverse
and therefore there are no close matches to the treated unit.
Intuitively, the synthetic control method works
by constructing a counterfactual version of the treated unit
using an average of the other units available.\sidenote{\citet{abadie2010synthetic}}
This is a particularly effective approach
when the lower-level components of the units would be directly comparable:
people, households, business, and so on in the case of states and countries;
or passengers or cargo shipments in the case of transport corridors, for example.\sidenote{\citet{gobillon2016regional}}
This is because in those situations the average of the untreated units
can be thought of as balancing by matching the composition of the treated unit.

To construct this estimator, the synthetic controls method requires
retrospective data on the treatment unit and possible comparators,
including historical data on the outcome of interest for all units.\sidenote{
  See \citet{fernandes2016expediting} for an example.}
The counterfactual blend is chosen by optimizing the prediction of past outcomes
based on the potential input characteristics,
and typically selects a small set of comparators to weight into the final analysis.
These datasets therefore may not have a large number of variables or observations,
but the extent of the time series both before and after the implementation
of the treatment are key sources of power for the estimate,
as are the number of counterfactual units available.
Visualizations are often excellent demonstrations of these results.
The \texttt{synth} package provides functionality for use in Stata and R,
although since there are a large number of possible parameters
and implementations of the design it can be complex to operate.\sidenote{\citet{abadie2014synth}}


\section{The Master File}
The master file is the file that controls the model calculation. This file decides what calculations are used and  what is done with the calculations (which is not the scope of this manual).
In other words, the master file combines all the calculations and theory from the previous sections and puts it all together to form a model. As mentioned earlier, this structure enables the
user to create their own version of the model. If one has their own calculations, or wants to use an older version of the calculations in this manual, then the user can define it themselves
and call it instead of the calls that we use. The model is meant to be customisable, which this structure enables.

\subsection{Adding a Spin-Up Time}
Instead of having a static start (having the planet start from rest, so no rotations allowed) we will have the model start up for some time before we start simulating the climate extensively.
To accomodate for this, we have to make some changes in the code. First we need to add two booleans (variables that can only take two values, either \texttt{TRUE} or \texttt{FALSE}) that we use
to indicate to the model whether we want to simulate the wind and whether we want to simulate advection. This means that the main loop will have some changes made to it. After performing the
calculations in \autoref{alg:temperature with density} we would calculate the velocities and afterwards we would calculate the advection. Instead let us change it to what is shown in
\autoref{alg:stream4v1}.

\begin{algorithm}
    \While{\texttt{TRUE}}{
        \autoref{alg:temperature with density} \;
        \If{$velocity$}{
            \autoref{alg:stream3} \;
            \If{$advection$}{
                \autoref{alg:advectionfix} \;
            }
        }
    }
    \caption{Main loop that can simulate velocities and advection conditionally}
    \label{alg:stream4v1}
\end{algorithm}

Now to dynamically enable/disable the simulation of flow and advection we need to add the spin-up calculations to the main loop. So in \autoref{alg:stream4v1}, before
\autoref{alg:temperature with density} we add \autoref{alg:spinup}. What it does is it changes the timestep when spinnning up and disables flow simulation, and when a week has passed it reduces
the timestep and enables flow simulation. At this point in time, the advection is not dynamically enabled/disabled but it is done by the programmer.

\begin{algorithm}
    \eIf{$t < 7day$}{
        $\delta t \leftarrow 60 \cdot 47$ \;
        $velocity \leftarrow \texttt{FALSE}$ \;
    }{
        $\delta t \leftarrow 60 \cdot 9$ \;
        $velocity \leftarrow \texttt{TRUE}$ \;
    }
    \caption{The spin-up block dynamically enabling or disabling flow simulation}
    \label{alg:spinup}
\end{algorithm}

\subsection{Varying the Albedo}
The albdeo (reflectiveness of the planet's surface) is of course not the same over the whole planet. To account for this, we instead vary the albedo slightly for each point in the latitude
longitude grid. The algorithm that does this is shown in \autoref{alg:albedo variance}. The uniform distribution basically says that each allowed value in the interval has an equal chance of
being picked \cite{uniformdist}.

\begin{algorithm}
    $V_a \leftarrow 0.02$ \;
    \For{$lat \in [-nlat, nlat]$}{
        \For{$lon \in [0, nlon]$}{
            $R \leftarrow \text{ Pick a random number (from the uniform distribution) in the interval } [-V_a, V_a]$ \;
            $a[lat, lon] \leftarrow a[lat, lon] + V_a \cdot R$\;
        }
    }
    \caption{Varying the albedo of the planet}
    \label{alg:albedo variance}
\end{algorithm}

\subsection{Non-uniform air density}
While air density on the surface is in general consistent, this does not hold if you move up through the atmosphere. The planet will pull air down due to gravity, which means that more air is at
the planet surface than at the top of the atmosphere. Hence the air density changes throughout the atmosphere and we need to account for that. This is done in \autoref{alg:density}. Because this
is used in radiation, velocity and advection, we initialise this in the master file. Though one could argue it could be part of the control panel, we opt not to include any code other than
variable declarations in the control panel for greater clarity. This also means that we give the user the option to have only one layer (by skipping implementing this algorithm). Note that the
$\rho[:,: i]$ notation means that for every index in the first and second dimension, only change the value for the index $i$ in the third dimension.

\begin{algorithm}
    $\rho[:, :, 0] \leftarrow 1.3$ \;
    \For{$i \in [1, nlevels-1]$}{
        $\rho[:, :, i] \leftarrow 0.1\rho[:, :, i - 1]$
    }
    \caption{Initialisation of the air density $\rho$}
    \label{alg:density}
\end{algorithm}

\subsection{Interpolating the Air Density}
In order to interpolate (see \autoref{sec:interpolation}) the air density, to have a better estimation at each grid cell, we need data. However currently we are just guessing the air density at
higher levels, instead of taking real values. So let us change that. For that we are going to use the U.S. Standard Atmosphere, an industry standard measure of the atmosphere on Earth
\cite{usatmosp}. This data was provided in a text (\texttt{TXT}) file which of course needs to be read in order for the data to be used in the model. Here we only care for the density and the
temperature at a specific height. So the text file only contains those two columns of the data (and the height in km of course as that is the index of the row, the property that uniquely
identifies a row).

With that in mind, let's get coding and importing the data. We do this in \autoref{alg:usatmosp}. As one can see we do not specify how to open the file or how to split the read line, as this
is language specific and not interesting to describe in detail. I refer you to the internet to search for how to open a text file in the language you are working in. Keep in mind in which
magnitude you are working and in which magnitude the data is. If you work with $km$ for height and the data is in $m$, you need to account for that somewhere by either transforming the imported
data or work in the other magnitude.

\begin{algorithm}
    $data \leftarrow \text{open text file containing the us standard atmosphere data}$ \;
    \ForEach{$line \in data$}{
        Split $line$ into three components, $sh, st$ and $sd$, representing the height, temperature and density respectively \;
        $standardHeight.add(sh)$ \;
        $standardTemperature.add(st)$ \;
        $standardDensity.add(sd)$ \;
    }

    $densityProfile \leftarrow \texttt{interpolate}(heights, standardHeight, standardDensity)$ \;
    $temperatureProfile \leftarrow \texttt{interpolate}(heights, standardHeight, standardTemperature)$ \;

    \For{$alt \in [0, nlevels]$}{
        $\rho[:, :, alt] \leftarrow densityProfile[alt]$ \;
        $T_a[:, :, alt] \leftarrow temperatureProfile[alt]$ \;
    }
    \caption{Loading in the U.S. Standard Atmosphere}
    \label{alg:usatmosp}
\end{algorithm}

Note that the function \texttt{interpolate} takes three arguments, the first one being the data points that we want to have values for, the second one is the data points that we know and the
third one is the values for the data points that we know. This function may or may not exist in your programming language of choice, which might mean that you have to write it yourself.
The formula that we use for interpolation can be found in \autoref{eq:interpolation}, though you still need to figure out what value you need for $\lambda$ (see \autoref{sec:interpolation}).
This is left as an exercise for the reader.

\subsection{Clamping the Velocities}
Due to the boundaries in the advection calculations (see \autoref{sec:adv}) we get weird instabilities as the velocity calculations are executed on more cells. Which means that air is trying to
displace temperature (advect it) by flowing faster to those cells, but actually don't carry any temperature because we turned it off for those cells. This is something that we need to fix to get
rid of weirdness around the edges. This is done in \autoref{alg:velocity clamped}. Here the $bla:$ means from $bla$ to the last valid index, if the $:$ is in front of $bla$ then it means from
the first valid index to $bla$.

\begin{algorithm}
    \autoref{alg:velocity}
    $u[(adv\_boun, -adv\_boun - 1), :, :] \leftarrow 0.5u[(adv\_boun, -adv\_boun - 1), :, :]$ \;
    $v[(adv\_boun, -adv\_boun - 1), :, :] \leftarrow 0.5v[(adv\_boun, -adv\_boun - 1), :, :]$ \;
    $w[(adv\_boun, -adv\_boun - 1), :, :] \leftarrow 0.5w[(adv\_boun, -adv\_boun - 1), :, :]$ \;

    $u[:adv\_boun, :, :] \leftarrow 0 $\;
    $v[:adv\_boun, :, :] \leftarrow 0 $\;
    $w[:adv\_boun, :, :] \leftarrow 0 $\;

    $u[-adv\_boun:, :, :] \leftarrow 0$ \;
    $v[-adv\_boun:, :, :] \leftarrow 0$ \;
    $w[-adv\_boun:, :, :] \leftarrow 0$ \;
    \caption{Clamping the Velocities}
    \label{alg:velocity clamped}
\end{algorithm}

\subsection{Smoothing all the things}
On a planet wide scale, you have a lot of variety in the data. To counteract that we filter out the high frequency data. Which means that we filter out the data that occurs sporadically. So we
do not consider the data that occurs so infrequently that it means nothing. We do this for the radiation (temperature) and the velocity which is shown in \autoref{alg:smootht} and
\autoref{alg:smoothv} respectively. It is worth mentioning that \autoref{alg:smootht} is executed after we do the calculations for $T_a$ (shown in \autoref{alg:optical depth}).
\autoref{alg:smoothv} is done after \autoref{alg:velocity} but before \autoref{alg:velocity clamped}.

\begin{algorithm}
    $T_a \leftarrow \texttt{Smooth}(T_a, smooth_t)$ \;
    \caption{Smoothing the atmospheric temperature}
    \label{alg:smootht}
\end{algorithm}

\begin{algorithm}
    $u \leftarrow \texttt{Smooth}(u, smooth_u)$ \;
    $v \leftarrow \texttt{Smooth}(v, smooth_v)$ \;
    $w \leftarrow \texttt{Smooth}(w, smooth_w, smooth_{vert})$ \;
    \caption{Smoothing the velocity}
    \label{alg:smoothv}
\end{algorithm}


The energy deposit measured per event by CASTOR can be shown as a 2D histogram with 16x14 pixels. The information of the amount of energy deposit is transmitted through the intensity of the colour. With a convolutional neural network it should be possible to learn the distinctive patterns of different types of particles and therefore to classify events correctly.

\section{Data}
To train the network a sufficiently high amount of data is needed. For training and validation Monte Carlo events generated by the PYTHIA8 and the EPOS generator are used. To separate actual events from background noise a typical minimum bias event selection was made. Only events which trigger an energetic response higher than 5 GeV in the CaloTowers with a pseudorapidity range of 3.1 < $|\eta|$ < 5 are used in the analysis. To simplify the work for the neural network, only events containing isolated particles are used for differentiation. Here an isolated particle is defined by a minimum distance of $\Delta$R = $\sqrt{(\Delta \phi)^2 + (\Delta \eta)^2}$ = 0.8 to the next particles generated by the same event. Therefore every generated event is checked for isolated particles travelling through CASTOR.

Within CASTOR different types of particles leave different characteristic signatures which can be used to identify the particles. In fig. \ref{signals} the different types of signals can be seen. Electromagnetic light particles, such as electrons, positrons and photons, have a very large cross section and interact with the electromagnetic part of the calorimeter. In the energy deposit histogram very high energies are measured within the first two sections while nearly no energy is deposited in the hadronic rear sections.
When hadrons reach the calorimeter they interact and produce several lighter secondary particles which in turn interact again. The energy deposit is therefore distributed wider and reaches its peak within the hadronic sections of the calorimeter.
Muons very sparsely interact with the material of the calorimeter and pass it nearly without losing energy. A muon track is normally contained within one tower but can be seen as a very small energy deposit in every section.

Every histogram containing the energy deposit has a corresponding label vector. This is a binary vector, where the first index corresponds to isolated electron, positron or photon, the second to isolated hadron and the third to background. Here background is everything besides an isolated electron/photon or hadron. Therefore the categories do not exclude each other, since it is possible that an isolated electron and an isolated hadron travel through CASTOR during the same event. In table \ref{particlecounts} the particles and their frequency are listed. For this 500.000 Monte Carlo events were evaluated and only those particles with the beforementioned minimum distance were counted. For a neural network typically at least 1000 samples per class are needed to pick up the identifying features. As can be seen most particles are too few to be correctly identified, whereas the differentiation between electrons/photons (which produce nearly the same signal) and hadrons is possible. Even though muons have an especially distinctive signal, their number is too small to include them as a separate category.

\begin{figure}
\centering
\begin{minipage}[t]{0.95\textwidth}
\begin{minipage}{0.45\textwidth}
\includegraphics[width=\textwidth]{electronhist.pdf}
\end{minipage}
\begin{minipage}{0.45\textwidth}
\includegraphics[width=\textwidth]{muonhist.pdf}
\end{minipage}
\end{minipage}
\begin{minipage}[b]{0.95\textwidth}
\begin{minipage}{0.45\textwidth}
\includegraphics[width=\textwidth]{pionhist.pdf}
\end{minipage}
\begin{minipage}{0.45\textwidth}
\includegraphics[width=\textwidth]{protonhist.pdf}
\end{minipage}
\end{minipage}
\caption{Energy deposit of single particles travelling through CASTOR. An electron or photon (upper left) deposits its whole energy in the electromagnetic sections, while a muon travels with only a little energy loss through every section (upper right). The signals of pions and protons is spread wider and reaches into the hadronic sections of the calorimeter (lower left and right). The data is generated by EPOS.}
\label{signals}
\end{figure}

As seen in fig. \ref{signals}, the energy deposited in CASTOR by different particles varies much. In different events particles with very different energies are created. For example the kinetic energy of a proton can be between one and several thousand GeV. To learn that these very different signals are generated by the same particle type would pose a challenge for the neural network. Every histogram is therefore normalized to 1 in signal amplitude.
\begin{table}
\centering
\caption{Number of isolated particles travelling through CASTOR in 500.000 Monte Carlo generated events. If not specified otherwise, the numbers include the particle and the antiparticle. As can be seen, hadrons and electrons/photons dominate the spectrum.}
\sffamily
\begin{tabular}{cc |cc }
\hline
Leptons/Bosons & Count & Hadrons & Count \\ \hline
Photon & 170047 & $\pi^+$ & 106904 \\
Positron & 243 & $\pi^-$ & 103648 \\
Electron & 250 & $K^+ / K^-$ & 27591 \\
Muon & 29 & Proton & 15782\\
$\nu_{\mu}$ & 27 & Neutron & 15232 \\
$\nu_e$ & 11 & $K^0_L$ & 13604 \\
$\nu_{\tau}$ & 1 &  $\Lambda$ & 5308\\
& & $\Sigma^+$ & 2016 \\
& & $\Sigma$ & 1993 \\
& & $\Sigma^-$ & 1993 \\
& & $K^0_s$ & 1323 \\
& & $\Xi^0$ & 511 \\
& & $\Xi^-$ & 471 \\
& & $\Omega^-$ & 16 \\
 \hline
\end{tabular}
\label{particlecounts}
\end{table}
\section{Network design}
The energy deposit histograms of CASTOR can be treated as pictures with 16 to 14 pixels and varying amounts of colour intensity. To design a neural network which can classify events into particle categories techniques in image recognition should be used. In normal classification problems convolutional layers with a small and quadratic kernel size are used. Here the information given by the underlying physics can be used to the advantage of the network.

As particles reach the calorimeter from the left, it is not useful to use small kernel sizes to cover the whole image. Small, light particles are normally contained in one tower or at most two. Therefore convolutional layers with kernel sizes of one or two towers, meaning one or two pixels in the height and 7 or 14 in the width, should be able to recognize the important features of the tracks. Several layers, each with kernel sizes with a width of 14 pixels but varying heights are concatenated in the beginning of the network. In the deeper layers of the network, smaller kernel sizes can be used, since the convolution shrinks the size of the signal travelling through the neurons. In fig. \ref{convnet} the network structure is shown. Because of the large kernel sizes in the beginning, the size of the data volume shrinks quickly. Max pooling was therefore omitted as too much information was lost due to the rather small input size. Deeper networks with more hidden layers have been tested but showed no advantage in accuracy. Deeper networks only need more computing time and have a higher tendency to overfit as more free parameters are available.

\begin{figure}
\includegraphics[scale=0.85]{convnet_fig.png}
\caption{The network design for the convolutional neural network. The first convolutional layer is composed out of kernel sizes of (1, 7), (2, 7) and (3, 7). This was omitted for a clearer picture. To reduce overfitting, dropout is used between after the first convolutional layer and before and after the first dense layer, each with a probability of 0.5. \protect \footnotemark}
\label{convnet}
\end{figure}

\section{Monitoring}
For monitoring the progress of a neural network, the first important feature is the loss function. As the classification problem is a multi-label classification, the loss function used here is the binary crossentropy loss. Every epoch it is calculated for the training and for the validation data set. Both variables need to decrease during the training. To determine if the input is correctly put into the network the first step was to train without any dropout or pooling layers. The network should overfit on the training set if given enough time. In fig. \ref{overfitacc} the loss and the accuracy of the training while overfitting can be seen.
\footnotetext{This figure is generated by adapting the code from \url{https://github.com/gwding/draw_convnet}}
\begin{figure}
\centering
\includegraphics[scale=0.8]{trainvalacc.png}
\caption{Training and validation accuracy. Both start at roughly the same percentage, but as training goes on, the training accuracy increases constantly. The validation accuracy does not change or even decreases a little. This is a classic overfitting effect.}
\label{overfitacc}
\end{figure}
As standard networks normally deal with multi-class and not multi-label problems, it proved to be useful to additionally monitor variables commonly used in classification problems. Here recall and precision were implemented \cite{rijsbergen}. With recall, the false negatives are taken into account, while precision monitors the false positives. Recall is also called the sensitivity of a network and is calculated by
\begin{equation}
\mathrm{Recall} = \frac{tp}{tp + fn} \quad ,
\label{recall}
\end{equation}
where tp stands for true positive, tn for false negative. The recall therefore is the fraction of true positive labels over all actually positive labels. In the case of all labels being correctly predicted, the recall is 1. Precision is also referred to as the positive predictive value. It is the rate of all correctly classified positive labels over all positively predicted labels.
\begin{equation}
\mathrm{Precision} = \frac{tp}{tp + fp}
\end{equation}
The nomenclature is the same, tp for true positives and fp for false positives. When all labels are correctly predicted, the precision also equals 1.

The precision and the recall can be calculated separately for the different categories. It can be easily monitored if the networks has more problems recognizing one category over the others.

%Another number which was monitored was the number of falsely labelled electrons. It was obvious after a few training sessions that the network had most problems classifying electrons. A counter of the falsely labelled electrons was therefore implemented in the callbacks. With y$_{\mathrm{true}}$ as the true labels and y$_{\mathrm{pred}}$ as the predicted labels, the counter worked as seen below.
%\begin{equation}
%\mathrm{count} = (1 - y_{\mathrm{true}}) \cdot y_{\mathrm{pred}} - (1 - y_{\mathrm{pred}}) \cdot y_{\mathrm{true}}
%\end{equation}
%It is the same as the crossentropy loss without the logarithm, but with two main differences. The first one is that here not the predicted probabilities are used as input put the absolute prediction. That means if y$_{\mathrm{pred}}$ is higher than 0.5 it is rounded to 1, lower than 0.5 means being rounded to 0. The second one is it being a difference rather than a sum. This way if the number is negative, the network has more false negatives than false positives, if it is positive, more false positives than negatives.
\section{Evaluation}
In the process of training it became apparent that no real physical features were being learned by the network. The training loss decreased but the validation remained constant over several hundred epochs (fig. \ref{trainvalloss}). As mentioned before, without  regulations the network overfitted on the training data without problem. This lead to reevaluating the input data, as it seemed to be difficult for the network to generalize the underlying physical laws.

\begin{figure}
\centering
\begin{minipage}{0.45\textwidth}
\centering
\includegraphics[scale=0.45]{trainloss.png}
\end{minipage}
\begin{minipage}{0.45\textwidth}
\centering
\includegraphics[scale=0.45]{valloss.png}
\end{minipage}
\caption{Training and validation loss for single particles. The training loss behaves exactly as it is supposed to, decreasing gradually during training. The validation loss shows that the network is not generalizing properly. It is remaining constant or even increasing, showing that the error rates get higher.}
\label{trainvalloss}
\end{figure}

One problem was the definition of background events. Since at first only isolated particles were considered, background events could also contain hadrons or electrons and photons, just not in the right distance to other particles travelling through CASTOR. The background category therefore contained input data which were not distinguishable from an photon or hadron event. To solve this problem, two different approaches were chosen. The first one was to completely ignore the background category and turn the classification into a binary problem, only distinguishing between electromagnetic particles, photons and electrons, and hadrons. The second was the redefinition of the background category, where only true background was used. True background meant, that no particle with an energy higher than 3 GeV had travelled through CASTOR in the corresponding signal. To further simplify the classification, only single particles were used, meaning that one particle per event crossed CASTOR. In tab. \ref{singleparticles} are the particles belonging to this data set. The background category consists mostly of no particles, which is not shown here. As single particles are rarer than isolated particles, two million events were evaluated. Still the number of useful histograms is smaller than with isolated particles, where 500.000 events were used.

\begin{table}
\centering
\caption{Single particles crossing through CASTOR. The particles in the background category are no single particles but have an energy less than 3 GeV. Two million events were evaluated.}
\sffamily
\begin{tabular}{cccc}\hline
Particle & Electromagnetic Class & Hadronic Class & Background Class \\ \hline
$\Lambda$ & 0 & 641 & 0 \\
Proton & 0 & 1915 & 0 \\
Neutron & 0 & 2355 & 0 \\
Electron  & 64 & 0 & 11 \\
Photon & 18327 & 0 & 80 \\
Pi$^{+-}$ & 0 & 25292 & 3 \\
K$^{+-}$ & 0 & 3608 & 0 \\
K$_l$ & 0 & 1769 & 0 \\
K$_s$ & 0 & 1812 & 0 \\
$\Omega^-$ & 0 & 3 & 0 \\
$\Xi^0$ & 0 & 53 & 0 \\
$\Xi^-$ & 0 & 52 & 0 \\
$\Sigma^+$ & 0 & 263 & 0 \\
$\Sigma^-$ & 0 & 384 & 0 \\
\hline
\end{tabular}
\label{singleparticles}
\end{table}

As precision and recall were implemented as callbacks, their value was returned after every epoch. In the first few training sessions the monitoring of these showed that while the mean loss decreased and the accuracy increased, the recall especially of electrons remained really low. To counteract this, the binary crossentropy loss was defined as a separate function. The crossentropy is the sum of two parts, one evaluating and penalizing the number of false positives, the other watching the number of false negatives. As recall, as seen in eq. \enquote{\ref{recall}}, evaluates the false negatives, they were stronger penalized than the false positives.

\subsection{Comparison between different data inputs}

To evaluate the possible applications and limits of the event classification with the help of a neural network, several data sets have been used and compared. As the first idea was to classify isolated particles, one dataset is composed of isolated electromagnetic particles, isolated hadronic particles and background, in which no isolated particle crosses CASTOR. All particles included in this data can be seen in table \ref{particlecounts}. As the background definition for this data type was difficult and the distinguishing between events with more than one particle to those with only one also seemed to worsen the performance of the network, single particles were introduced. Only events in which just one particle had crossed through CASTOR were counted here. To also better define the background class, events with particles no higher than 3 GeV or no particle in CASTOR	were included in this class. At last, a binary classification was tried, excluding the background completely. For every data type, the training and the validation loss function can be seen in fig. \ref{lossall}. The expected result would be that the network performs worse on isolated particles since the distinction between background and electromagnetic/hadronic particles was not clear. Several events could also fall in both the electromagnetic and the hadronic class, which was more difficult to recognize. As can be seen in fig. no analysis behaves at it should. The training loss decreases exponentially, which is the expected behaviour. The validation loss remains constant or increases. This should not happen as dropout of 0.5 is used during all training time. Increasing dropout had no effect on the training. As the training loss functions correctly, it can be assumed that the implementation of the network is right. It seems to be unable to identify physical features which correctly classify the events. A hadronic process is more easily identified judging by the relatively small loss function.

\begin{figure}
\centering
\begin{minipage}{0.45\textwidth}
\includegraphics[width=7cm]{trainlossbinary.png}
\end{minipage}
\begin{minipage}{0.45\textwidth}
\includegraphics[width=7cm]{vallossbinary.png}
\end{minipage}
\\
\begin{minipage}{0.45\textwidth}
\includegraphics[width=7cm]{trainloss.png}
\end{minipage}
\begin{minipage}{0.45\textwidth}
\includegraphics[width=7cm]{valloss.png}
\end{minipage}
\\\begin{minipage}{0.45\textwidth}
\includegraphics[width=7cm]{trainlossisolated.png}
\end{minipage}
\begin{minipage}{0.45\textwidth}
\includegraphics[width=7cm]{vallossisolated.png}
\end{minipage}
\caption{Training and validation loss for different data sets. At first the binary classification is shown, then the single particles and at last the isolated particles. The correct tendency for the loss function to decrease is seen in every training figure. The validation loss behaves not as it should. It seems to forget what it learned.}
\end{figure}

\begin{figure}
\centering
\begin{minipage}{0.45\textwidth}
\includegraphics[width=7cm]{precisionsingle.png}
\end{minipage}
\begin{minipage}{0.45\textwidth}
\includegraphics[width=7cm]{recallsingle.png}
\end{minipage}
\caption{Precision and recall for single particles. The precision is rather high, false positives are therefore seldom. In comparison, recall is low for electrons and hadrons. Many times the network falsely labels an event as not belonging to one of these categories but rather to the background or no class.}
\label{lossall}
\end{figure}

The single particles have the most constant performance on the validation data set but it still seems to increase during training time. The last set in fig. \ref{lossall} is the training with isolated particles. For unknown reasons the background loss function behaves exactly the same as the hadron loss function, during training as well as during validation.
\subsection{Performance on actual data}
To see how the network performed on actual data after training, two runs with proton-proton collisions at $\sqrt{s}$ = 13 TeV with very low luminosity were evaluated. The low luminosity was useful to ensure that a high number of events contained isolated or single particles. To assess the performance of the network, the ratio of electromagnetic energy deposit contained in the first two sections of every tower to hadronic energy deposit in the following 14 sections was calculated. The energy ratio E$_{\mathrm{em}}$/(E$_{\mathrm{em}}$ + E$_{\mathrm{had}}$)is shown in the histograms in fig. \ref{ratio}, in comparison to actual electromagnetic and hadronic events and their ratio made by Monte Carlo simulations by EPOS.
\begin{figure}
\centering
\begin{minipage}{0.45\textwidth}
\includegraphics[width=7cm]{ratio.png}
\end{minipage}
\begin{minipage}{0.45\textwidth}
\includegraphics[width=7cm]{emhadbinary.png}
\end{minipage}
\\
\begin{minipage}{0.45\textwidth}
\includegraphics[width=7cm]{emhadsingle.png}
\end{minipage}
\begin{minipage}{0.45\textwidth}
\includegraphics[width=7cm]{emhadisolated.png}
\end{minipage}
\caption{Electromagnetic to hadronic energy in the different categories. The figure on the upper left are simulated events with single electromagnetic and hadronic particles. On the upper right hand side are the ratios based on the prediction trained on binary particles. On the lower left hand side are the predictions based on single particles, on the lower right hand side the predictions based on the isolated particles.}
\label{ratio}
\end{figure}

For electromagnetic events this ratio should be nearly 1, while for hadronic events a peak at 0 is to be expected. In the simulated events this behaviourism is very obvious. There are two peaks in the histogram, one for electromagnetic and one for hadronic events. In the predicted events the tendency cannot be seen. In the real events classified by the network the energy distribution is the same for all predicted categories. As most events do not contain single or isolated particles the energy ratio is centered around 0.5. This is due to several particles per event travelling through CASTOR. Interesting is the varying amount of events predicted as CASTOR by the network trained on single and on isolated particles. As events with isolated particles can also contain several particle tracks at once more events fall into the electromagnetic or hadronic category. For single particles more events are classified as belonging to background since several particle events were excluded as training events for the other two categories.

The neural network is unable to correctly identify a high number of events. As in real data single or isolated particles can not be separated, many events contain several particles crossing CASTOR. This is difficult for a network to classify correctly. To see if the network is classifying at least some events accurately, events belonging to one class with a probability of at least 80 \% have been picked out. These events were then again displayed as a two dimensional histogram. An electromagnetic and a hadronic event is shown in fig. \ref{electron}. Both have been identified by the neural network trained on single particles, the networks trained on binary and isolated particles identified similar events. The signature tracks of both particle types are clearly visible. The electromagnetic particle leaves a short track, depositing its energy in the first two sections of one or two towers. A hadronic particle leaves a wider track, decaying into lighter particles, which deposit their energy throughout the detector.
\begin{figure}
\centering
\begin{minipage}{0.45\textwidth}
\includegraphics[width=6cm]{electron.png}
\end{minipage}
\begin{minipage}{0.45\textwidth}
\includegraphics[width=6cm]{hadron.png}
\end{minipage}
\caption{One electromagnetic and one hadronic event with a probability higher than 80 \%. On the left the short track of an electron or photon can be seen, depositing nearly all energy in the first two sections. On the right is the wider signature of a hadron.}
\label{electron}
\end{figure}

At least a few events can be correctly identified by the neural network. The performance is not dependent on the data type used for training. Obvious particle tracks which are left by single particles can be recognized by a trained network. Overlapping signals are difficult for the network to discern and are then randomly put into one category. In real data isolated or single particles happen seldom. This explains the incorrectly behaving electromagnetic to hadronic ratio.

\subsection{Error analysis}
After several training cycles the loss function of the network still did not converge. Several possible causes for this failure have been mentioned and rectified, such as the definition of background events, very low recall and the network design. But even with only two categories, electron/photon and hadronic events, the loss function continued to show erratic behaviour. The network only performed well on very clear tracks. Events with several particles were not identified correctly. There are many possibilities which caused the poor classification, a few are going to be evaluated here.

The first one is the lack of training time. Neural networks which perform as well or even exceed human performance have an average training time of three weeks. This was not possible during this thesis as a continuous evaluation of the network's performance was needed.

The second one is the actual distinction between hadronic and electromagnetic events with only the energy deposit as input data. As hadronic events can also cause electromagnetic showers the input signals can seem similar. If several particles cross CASTOR in one event, the overlapping signals can be difficult to distinguish.

At last using conventional convolutional layers, important physical information gets lost. As the particles always enter the calorimeter from the left side, layers with higher weights and therefore more emphasis on the first sections could perform better than those used here. Convolutional layers share the same weights across the whole input, so no part is highlighted. This is advantageous in classic image recognition, but not for event classification in a calorimeter.


\chapter{New Chapter}
\section{New Section}
\begin{theorem}
    \lipsum[1]
\end{theorem}
\begin{proof}
    \lipsum[1]
\end{proof}

\begin{definition}
    \lipsum[1]
\end{definition}

\begin{note}
    \lipsum[1]
\end{note}

\begin{example}
    \lipsum[1]
\end{example}
\begin{solution}
    \lipsum[1]
\end{solution}

\begin{problem}
    \lipsum[1]
\end{problem}
\begin{solution}
    \lipsum[1]
\end{solution}

\section{New Section}
\begin{theorem}
    \lipsum[1]
\end{theorem}
\begin{proof}
    \lipsum[1]
\end{proof}

\begin{definition}
    \lipsum[1]
\end{definition}

\begin{note}
    \lipsum[1]
\end{note}

\begin{example}
    \lipsum[1]
\end{example}
\begin{solution}
    \lipsum[1]
\end{solution}

\begin{problem}
    \lipsum[1]
\end{problem}
\begin{solution}
    \lipsum[1]
\end{solution}

\section{New Section}
\begin{theorem}
    \lipsum[1]
\end{theorem}
\begin{proof}
    \lipsum[1]
\end{proof}

\begin{definition}
    \lipsum[1]
\end{definition}

\begin{note}
    \lipsum[1]
\end{note}

\begin{example}
    \lipsum[1]
\end{example}
\begin{solution}
    \lipsum[1]
\end{solution}

\begin{problem}
    \lipsum[1]
\end{problem}
\begin{solution}
    \lipsum[1]
\end{solution}


% --- [ Informal grammar for µC ] ----------------------------------------------

\subsection{Informal grammar for µC}
\label{app:informal_grammar_for_uc}

This is an informal context-free grammar for uC (included without modification from \url{https://www.it.uu.se/katalog/aleji304/CompilersProject/uc.html}):

\begin{itemize}
	\item The start symbol is \texttt{program}.
	\item Keywords and special symbols are written within double-quotes.
	\item \texttt{/empty/} denotes the empty string.
	\item \texttt{intconst} and \texttt{ident} denote classes of lexical elements.
	\item Associativity and precedence for expression operators is not expressed.
	\item The grammar has not been adjusted to fit any particular parsing method.
\end{itemize}

\begin{verbatim}
program         ::= topdec_list
topdec_list     ::= /empty/ | topdec topdec_list
topdec          ::= vardec ";"
                  | funtype ident "(" formals ")" funbody
vardec          ::= scalardec | arraydec
scalardec       ::= typename ident
arraydec        ::= typename ident "[" intconst "]"
typename        ::= "int" | "char"
funtype         ::= typename | "void"
funbody         ::= "{" locals stmts "}" | ";"
formals         ::= "void" | formal_list
formal_list     ::= formaldec | formaldec "," formal_list
formaldec       ::= scalardec | typename ident "[" "]"
locals          ::= /empty/ | vardec ";" locals
stmts           ::= /empty/ | stmt stmts
stmt            ::= expr ";"
                  | "return" expr ";" | "return" ";"
                  | "while" condition stmt
                  | "if" condition stmt else_part
                  | "{" stmts "}"
                  | ";"
else_part       ::= /empty/ | "else" stmt
condition       ::= "(" expr ")"
expr            ::= intconst
                  | ident | ident "[" expr "]"
                  | unop expr
                  | expr binop expr
                  | ident "(" actuals ")"
                  | "(" expr ")"
unop            ::= "-" | "!"
binop           ::= "+" | "-" | "*" | "/"
                  | "<" | ">" | "<=" | ">=" | "!=" | "=="
                  | "&&"
                  | "="
actuals         ::= /empty/ | expr_list
expr_list       ::= expr | expr "," expr_list
\end{verbatim}


\section{Analysis}
\subsection{Overview}
The code is implemented such that the only difference between the solving algorithms lie in there heuristic values. As the rest of the code is shared. Though an important impact of the heuristic is how many iterations the main part of the program goes through. The iterations tend to take longer to calculate than the heuristics, thus is ideal to take a bit longer to calculate at to result in fewer iterations. Such is rather apparent with the various experiments, in which depending on the implementation some algorithms would perform better than others. Such can be seen in figures of the difficulties section.
\newline
\newline

During the solving of the puzzle, how the board is represented varies. Thus it is important to minimize conversion time. The indonesian\_dot/spaces/node.py \#touch(self, action) function performs said conversion to an integer. Initially the approach was to perform string concatenation then convert the binary representation to an integer. However it was found that directly calculating the integer value from the binary was faster. Most likely because the same calculations would still be performed in addition to the overhead of performing the concatenations.
\newline
\newline

With the aim of minimizing memory usage, indonesian\_dot/spaces/node.py has the minimal number of features. As a simple way to have code run faster is for both the memory reading and the memory writing operations done by the operating system to take the minimal amount of time. Which requires the the operations to be taken out in the highest level of the memory hierarchy possible.
\newline
\newline
\subsection{Algorithm Comparison}

Regarding the three solving algorithms, in terms of time to complete, typically A* and BFS tended to have rather similar times with A* being slightly faster than BFS most of the time. Of the three DFS tended to be the slowest. The difference in times tended to increase as the size of the puzzles increased.
\newline
\newline

The calculating of g heuristic in terms of operation complexity and function calls is essentially the same, thus the h heuristic value is what sets them a part for speed. Though both A* and BFS have the exact same code for calculating h values; while DFS has a simple, though irrelevant, way to calculate DFS's h value.
\newpage
\subsection{Algorithm Contrast}

In terms of function calls (raw data shown in Appendix section), DFS tended to have about 80 times as many as A*. Such helps to explain why DFS is so slow. Another reason for DFS's slowness is that the algorithm is essentially a brute force approach solution, while the others algorithms used tend to take educated guesses (reduces iteration count) to speed up their solving of the puzzle.
\newline
\newline

Due to DFS's minimalist approach in calculating heuristics, in the event that it would take the same search path as the other algorithms, it would have the best time. Though such would merely be the result of a specific puzzle configuration, than by design. As DFS is a rather brutish approach for solving.
\newline
\newline

To solve the puzzle no repeated moves are required, as they would simply end up canceling out. Also the order of the moves does not matter. The program is implemented so that for search paths, moves cannot be repeated, however the result is that the order that the moves are selected in is fixed. The drawback of the ordering is that the resulting search space is the power set of the remaining possible moves (excluding the empty set), starting from the last move in the order. While subsequent search spaces will be smaller, it can still be rather costly to located the first move in the order. As every move before it will be a waste of time, amounting only in determining that those starting moves are not in the solution. Therefore solutions that have consecutive moves in the ordering are ideal for both DFS and BFS, while A* prefers paths formed from sparse moves.


    \filetitle{plus}{Create time-recursive lead of tseries object}{trec/plus}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
X{T+K}
\end{verbatim}

\paragraph{Input arguments}\label{input-arguments}

\begin{itemize}
\item
  \texttt{X} {[} tseries {]} - Tseries object whose time-recursive lead
  will be created.
\item
  \texttt{T} {[} trec {]} - Initialized trec object.
\item
  \texttt{K} {[} numeric {]} - Integer scalar specifying the lead.
\end{itemize}

\paragraph{Description}\label{description}

The tseries object, \texttt{X}, referenced by \texttt{T+K} in a
time-recursive expression will, in each iteration, return a value that
corresponds to period \texttt{t+K}, where \texttt{t} is the currently
processed date from the vector of dates (or date range) associated with
the trec object, \texttt{T}.


\section{Analysis on requester information}
In this section, we perform analysis on requester information attributes. Such analysis will give us some idea of how the information available in Reddit about the requesters could effect whether a request receive a pizza.

\subsection{Contrast analysis}
In our first analysis, we perform comparison between successful and unsuccessful requests on each requester information attribute. For example, with the attribute \textit{requester\_account\_age\_in\_days\_at\_request}, we want to see whether an older account holder will have more chance at receiving a free pizza or not. In order to do that, for each attribute, we collect values associate with successful and unsuccessful requests and visualize them for comparison. To have more precise analysis and better visualization, we removed outliers by dropping all value that are ``too extreme''. We used the standard rule to identify outliers. A value is considered as outliers if it greater than $1.5 \times IQR$ above the third quantile or less than $1.5 \times IQR$ below the first quantile. Figure \ref{boxplot} shows boxplots of requester information attributes based on the success of requests. We can see that, the median values for successful requests are slightly higher than for unsuccessful requests for all analyzed attributes but the there is not much different between the range of values and the shape of box plots. This result suggests that the requester information attributes do not have much predictive power because the distributions of attribute values are similar in both successful and unsuccessful requests.

\begin{figure}
	\includegraphics[width=0.16\textwidth]{data/requester_account_age_in_days_at_request}
	\includegraphics[width=0.16\textwidth]{data/requester_number_of_comments_at_request}
	\includegraphics[width=0.16\textwidth]{data/requester_number_of_posts_at_request}
	\includegraphics[width=0.16\textwidth]{data/requester_number_of_subreddits_at_request}
	\includegraphics[width=0.16\textwidth]{data/requester_upvotes_minus_downvotes_at_request}
	\includegraphics[width=0.16\textwidth]{data/requester_upvotes_plus_downvotes_at_request}
	\vspace*{-0.6cm}
	\caption{Boxplots of requester information attributes based on the success of requests}
	\vspace*{-0.6cm}
	\label{boxplot}
\end{figure}

\begin{table}[]
	\centering
	\caption{The confusion matrix for a fold in the cross validation}
	\label{confusion1}
	\begin{tabular}{|c|c|c|c|}
		\hline
		\multicolumn{2}{|c|}{\multirow{2}{*}{}}                & \multicolumn{2}{c|}{Actual}                            \\ \cline{3-4}
		\multicolumn{2}{|c|}{}                                 & \multicolumn{1}{c|}{TRUE} & \multicolumn{1}{c|}{FALSE} \\ \hline
		\multicolumn{1}{|c|}{\multirow{2}{*}{Predict}} & TRUE  & 0                         & 0                          \\ \cline{2-4}
		\multicolumn{1}{|c|}{}                         & FALSE & 92                        & 312                        \\ \hline
	\end{tabular}
	\vspace*{-0.6cm}
\end{table}

\subsection{Outcome prediction}
In second analysis, we perform a request classification task using requester information attributes to verify our assumption. There are several prediction models can be used in this classification task such as decision trees, logistic regression, SVM, etc. In our experiment, we used a simple logistic regression as the prediction method. We also used $10$-fold cross validation to measure the accuracy of the prediction model. At the iteration $i$ in the cross validation, we used fold $i$ as test set and the remaining data as training set. To measure the accuracy of the prediction model, we used the $F$-measure\footnote{\url{https://en.wikipedia.org/wiki/F1_score}}. The overall $F$-measure is 0. The reason why the value of the $F$-measure is zero is that the classifier always predict that a request will not receive a pizza. Table \ref{confusion1} show here a confusion matrix in a fold of the cross validation. We can see that the classifier predict \texttt{FALSE} values. From the experiment, we see that prediction models based requester information attributes do not perform well. It suggests that the decision whether or not a request receive a pizza do not based on the information about requester.


GWF Model observations include the simulated groundwater head (\texttt{head}), calculated drawdown (\texttt{drawdown}) at a node, and the flow between two connected nodes (\texttt{flow-ja-face}). The data required for each GWF Model observation type is defined in table~\ref{table:gwfobstype}. For \texttt{flow-ja-face} observation types, negative and positive values represent a loss from and gain to the \texttt{cellid} specified for ID, respectively.

\subsubsection{Structure of Blocks}
\vspace{5mm}

\noindent \textit{FOR EACH SIMULATION}
\lstinputlisting[style=blockdefinition]{./mf6ivar/tex/utl-obs-options.dat}
\lstinputlisting[style=blockdefinition]{./mf6ivar/tex/utl-obs-continuous.dat}

\subsubsection{Explanation of Variables}
\begin{description}
\input{./mf6ivar/tex/utl-obs-desc.tex}
\end{description}

\begin{longtable}{p{2cm} p{2.75cm} p{2cm} p{1.25cm} p{7cm}}
\caption{Available GWF model observation types} \tabularnewline

\hline
\hline
\textbf{Model} & \textbf{Observation type} & \textbf{ID} & \textbf{ID2} & \textbf{Description} \\
\hline
\endhead

\hline
\endfoot

\input{../Common/gwf-obs.tex}
\label{table:gwfobstype}
\end{longtable}

\vspace{5mm}
\subsubsection{Example Observation Input File}

An example GWF Model observation file is shown below.

\lstinputlisting[style=inputfile]{./mf6ivar/examples/utl-obs-example-obs.dat}


\section{Investment Saving - Liquidity preference - Money supply (IS-LM)}