\documentclass{bioinfo}
\copyrightyear{2019} \pubyear{2019}

\access{Advance Access Publication Date: Day Month Year}
\appnotes{Application note}

\begin{document}
\firstpage{1}

\subtitle{Effect is more effective than p}

\title[short Title]{A reproducible effect size is more useful than an irreproducible hypothesis test to analyze high throughput sequencing datasets}
\author[Fernandes \textit{et~al}.]{Andrew D. Fernandes\,$^{\text{\sfb 2}}$ , Michael Vu\,$^{\text{\sfb 1}}$, Lisa-Monique Edward\,$^{\text{\sfb 1}}$,  Jean M. Macklaim,$^{\text{\sfb 3}}$ and Gregory B. Gloor\,$^{\text{\sfb 1,}*}$}
\address{$^{\text{\sf 1}}$Department of Biochemistry, University of Western Ontario, London, N6A 5C1, Canada and \\
$^{\text{\sf 2}}$San Diego CA, 92130.\\
$^{\text{\sf 3}}$DNA Genotek, Ottawa, K2V 1C2, Canada.}

\corresp{$^\ast$To whom correspondence should be addressed.}

\history{Received on XXXXX; revised on XXXXX; accepted on XXXXX}

\editor{Associate Editor: XXXXXXX}

\abstract{
\textbf{Motivation:} High throughput sequencing is analyzed using a combination of null hypothesis significance testing and ad-hoc cutoffs. This framework is strongly affected by sample size, and is known to be irreproducible in underpowered studies, yet no suitable non-parameteric alternative has been proposed. \\
\textbf{Results:} Here we present implementations of non-parametric standardized median effect size estimates,  $\mathcal{E}_{d}$, for high-throughput sequencing datasets. Case studies are shown for transcriptome and amplicon-sequencing datasets.  The  $\mathcal{E}_{d}$ statistic is shown to be more reproducible and robust than p-values and requires sample sizes as small as 5 to reproducibly identify  differentially abundant features.\\
\textbf{Availability:} Source code and binaries freely available at:\\
 https://bioconductor.org/packages/ALDEx2.html, omicplotR, and  https://github.com/ggloor/CoDaSeq. Datasets can be found at doi://10.6084/m9.figshare.8132216\\
\textbf{Contact:} \href{ggloor@uwo.ca}{ggloor@uwo.ca}\\
\textbf{Supplementary information:} Supplementary data and code are available at \textit{Bioinformatics} online.
}

\maketitle

\section{Introduction}

%A well-powered large-scale experiment that examined variation in the constituents of the human microbiome was found to have multivariate effect sizes $<0.05$, and the largest univariate effect size of any taxon with any clinical covariate was less than 0.5 \citep{Falony:2016aa}. This sets an upper bound for expected effect sizes in microbiome studies and convincingly demonstrated that the vast majority of microbiome studies are dramatically underpowered.

High throughput sequencing (HTS) datasets for transcriptomics, metagenomics and 16S rRNA gene sequencing are high dimensional, commonly conducted with pilot-scale  experiments and analyzed using a null hypothesis significance testing framework \citep{Schurch:2016aa}. Much effort has been spent identifying the best approaches and tools to determine what is `significantly different' between groups \citep{Soneson:2013,Schurch:2016aa}, but the answer seems to depend on the specific dataset and associated model parameters \citep{Thorsen:2016aa,hawinkel2017,Weiss:2017aa}. As commonly conducted the investigator determines what is `significantly different' using a null hypothesis significance approach and then decides what level of difference is `biologically meaningful' among the `significantly' different features. Graphically, this approach is  represented by the Volcano plot \citep{Cui:2003aa} where the magnitude of change (difference) is plotted vs the p-value.  One under-appreciated consequence of pilot scale research is that features with significant p-values   will have dramatically exaggerated  apparent effect sizes \citep{Halsey:2015aa}. This explains in part why so many observations  of apparent large effect fail to replicate in larger datasets \citep{Ioannidis:2005aa}. In fact, both p-values and absolute differences are poor predictors of which differences would reproduce if the experiment were conducted again \citep{Cumming:2008aa,Halsey:2015aa}

%In part, the irreproducibility of p-value based hypothesis tests is due to the low statistical power of pilot-scale experimental designs coupled with the very large number of features (genes, transcripts, etc.) used for testing.  Paradoxically, when power is low, any features (genes, transcripts, etc) that are observed to be `significantly different' must have an inflated effect size \citep{Halsey:2015aa}.  Thus, `significant' features are expected to regress to their true difference between groups when the experiment is replicated, and as usually happens if the replication set is itself underpowered, replication is therefore unlikely to identify the same set of significant features.

%However, taking inter-sample variation into account by plotting the standardized effect size vs. p-values results in a much better fit \citep{gloor:effect}.

As many have pointed out, p-values are not useful proxies for biological relevance since p-values are designed, colloquially speaking, to estimate the likelihood of no true difference; p-values are not a test that the alternate hypothesis is true. They can only be used to estimate false-discovery rates if the p-value is calculated from a distribution that is appropriate for the experimental data, if reasonable estimates for the statistical power exist and a if reasonable estimate of the a priori probability that the null hypothesis is false \citep{Colquhoun:2014aa, Halsey:2015aa}. Simply put, p-values can only be used to test \emph{if} there is no difference between groups, not to measure the \emph{magnitude} of change between groups \citep{coe2002s,Colquhoun:2014aa}. The tension between the information that p-values provide and what the investigator needs is why magnitude of change cutoffs \citep{Cui:2003aa}, or other ad-hoc methods are used when deciding what is biologically relevant. Null-hypothesis significance based testing methods  also have the property that the number of significant features identified  is affected by the number of samples being compared. Thus leads to the concept of statistical power, where the experiment is designed such that statistical power is prioritized over biological significance.

On the other hand, a standardized effect size addresses the issues of interest to the biologist:``what is reproducibly different?'' or ``would I identify the same true positive features as differential if the experiment were repeated?"  \citep{coe2002s,shinichi:2004,Colquhoun:2014aa,gloor:effect}. Standardized effect size statistics start from the assumption that there is a difference, but that the difference can be arbitrarily close to zero. Unfortunately,  standardized effect size metrics are not routinely used when analyzing HTS datasets, and one potential barrier to their use is that parametric effect size statistics may not be suitable for  HTS datasets.

Here we introduce a simple non-parametric standardized effect size statistic for distributions, $\mathcal{E}_{d}$, that is implemented in the ALDEx2, omicplotR  and CoDaSeq R packages.  The $\mathcal{E}_{d}$ statistic has been used in both meta-transcriptome and microbiome studies, for example see \citep{macklaim:2013, bian:2017}, and has been shown to give remarkably reproducible results even with extremely small sample sizes \citep{nelson:2015vaginal}.   $\mathcal{E}_{d}$  has a near monotonic relationship with p-values (Supplement Figure 1). However, it is unknown how  $\mathcal{E}_{d}$  compares with  p-values or other effect size estimates, how many samples are required, and its sensitivity and specificity characteristics.
%\enlargethispage{12pt}

\section{Approach}

High throughput sequencing (HTS) machines output thousands to billions of `reads', short nucleotide sequences that are derived from a DNA or RNA molecule in the sequencing `library'. The library is a subset of the nucleic acid molecules that have been collected from an environment and made compatible with a particular HTS platform. The HTS instruments deliver these reads as integer `counts' per genomic feature---gene, location, etc. However, the counts are actually a single proxy for the probability of observing the particular read in a sample under a repeated sampling model; this is clear since technical replicates of the same library return different counts. The difference between technical replicates is consistent with multivariate Poisson sampling \citep{fernandes:2013, gloorAJS:2016} The probability estimate is delivered by the instrument as an integer representation of the probability multiplied by the number of reads  \citep{fernandes:2013, gloorAJS:2016}. Thus, the data returned by HTS are a type of count compositional data, where only the relationships between the features have meaning \citep{aitchison:1986, Lovell:2015,fernandes:2014,gloorFrontiers:2017,Kaul:2017aa}.

The ALDEx2 tool uses a combination of probabilistic modelling and compositional data analysis to determine the features that are different between groups, where that difference is insensitive to random sampling. Technical replicate variance estimation and conversion of the count data to probabilities is accomplished by Monte-Carlo sampling from the Dirichlet distribution \citep{fernandes:2013, gloorAJS:2016}, which is conveniently also the conjugate prior for the multivariate Poisson process. The differences between features is linearized by applying a log-ratio transformation to the Dirichlet Monte-Carlo realizations and analyzed according to the rules of compositional data analysis \citep{aitchison:1986,fernandes:2013,Tsilimigras:2016aa,gloorFrontiers:2017}.

Starting with two vectors $\vec{a}$ and $\vec{b}$  that correspond to the concatenated log-ratio transformed Dirichlet Monte-Carlo realizations of a feature in two groups, we need a method to determine the standardized effect size  for the log-ratio transformed posterior probability estimates;  that is, the difference between groups relative to an estimate of within-group dispersion. Since these posterior distributions can have heavy tails, be multimodal, and be skewed, any useful statistic should be insensitive to even extreme non-Normality and provide sensible answers even if the posterior picture distributions are almost Cauchy in one or both groups \citep{fernandes:2013}. Below and in the Supplement we define the properties of the approach used.

Cohen's d is a parametric standardized effect size for the difference between the means of two groups, and a general formulation is given in Equation~\ref{eq:cohen}. Cohen's d is the difference between the means of the two distributions divided by the pooled standard deviation, denoted as \(\sigma_{a,b}\). However, this metric depends upon the data being relatively Normal, which cannot be guaranteed for HTS data.

\begin{equation}
\mathrm{d} = \frac{\mathrm{mean}(\vec{a} )- \mathrm{mean}(\vec{b})}{\sigma_{a,b}}
\label{eq:cohen}
\end{equation}

We can define a non-parametric  \emph{difference} vector  in Equation~(\ref{eq:diff}) as the signed difference between the two groups. We can further define a non-parametric  \emph{dispersion} vector as in Equation~(\ref{eq:disp}), where the notation $\boldsymbol{\rho}\vec{a}$ indicates one or more random permutations of the vector. Finally, we can define an \emph{effect} vector as in Equation~(\ref{eq:ff}) that is the ratio of these two non-parametric statistics.

\begin{equation}
\vec{\delta} = \vec{a} - \vec{b}
\label{eq:diff}\vspace*{-10pt}
\end{equation}

\begin{equation}
\vec{\sigma} = max \{ \lvert \vec{a} - \boldsymbol{\rho} \vec{a}  \rvert ,\lvert \vec{b} -\boldsymbol{\rho} \vec{b} \rvert \}
\label{eq:disp}\vspace*{-10pt}
\end{equation}

\begin{equation}
\vec{\varepsilon} = \frac{\vec{\delta}}{\vec{\sigma}}
\label{eq:ff}\vspace*{0pt}
\end{equation}

Taking the median of $\vec{\delta}, \vec{\sigma}$ and $\vec{\varepsilon}$ returns a robust estimate of the central tendency of these statistics ($\tilde{D}$, MMAD (median of the maximum absolute deviation), and $\mathcal{E}_{d}$), and these are the `diff.btw', `diff.win' and `effect' statistics reported by ALDEx2. $\tilde{D}$ is the same as the difference between the means or the difference between medians in a Normal distribution as shown in Supplementary Figure  2. The MMAD metric is novel and the Supplement shows it has a Gaussian efficiency of 52\%, a breakdown point of 20\% (Supplementary Figure 3), and is 1.42 times the size of the standard deviation on a Normal distribution. The $\mathcal{E}_{d}$ statistic is a standardized effect size and is approximately 0.7 of Cohen's D when comparing the difference between two Normal distributions.  This is simply a Monte-Carlo estimate of a function of the respective random variables. Below and in Supplementary Figure 4 we show that this metric returns sensible values even with a Cauchy distribution.

\begin{methods}
\section{Methods}

We used  simple simulated datasets to determine baseline characteristics in a number of different distributions. Then we use the data  from a highly replicated RNA-seq experiment \citep{Schurch:2016aa} and examined 100 random subsets of the data with between 2 and 40 samples in each group. For each random subset we collected the set of features that were called as differentially abundant at thresholds of $\mathcal{E}_{d} \ge 1$, or with an expected Benjamini-Hochburg adjusted p-value of $\le 0.1$ calculated using either the parametric Welch's t-test, or the non-parametric Wilcoxon test in the ALDEx2 R package. These are output as `we.eBH' and 'wi.eBH' by the ALDEx2 tool. These were compared to a `truth' set determined by identifying those features that were identified in all of 100 independent tests of the full dataset with outliers removed using the same tests and cutoffs. Note that this is simply a measure of consistency and is congruent with the approach taken in \citep{Schurch:2016aa}.

The median values and 99\% confidence intervals for the number of true positives, false positives, true negatives and false negatives were tabulated and plotted in Fig. 1.

\enlargethispage{6pt}

\end{methods}

\section{Results and Discussion}

Measuring differential abundance in high throughput sequencing datasets is difficult for a variety of reasons. First, almost all experiments are underpowered. Second, the true distribution of the data is unknown. Third, when sample sizes are large almost all features are identified as `significantly different' by null hypothesis significance testing frameworks.

We began by examining behaviour of   $\mathcal{E}_{d}$ and Cohen's~d statistics in simulated distributions as shown in Figure \ref{fig:01}. For this, we generated 100000 distributions for sample sizes  of between 2 and 40 in each group, and calculated the median and 99\textsuperscript{th} percentile effect size. When there is no difference between groups as illustrated in the top two panels, all four distributions had a median effect size of 0 at all sample sizes. However, the  99\textsuperscript{th} percentiles were different between the two statistics. We observe that the 99\textsuperscript{th} of the $\mathcal{E}_{d}$ metric increases more rapidly than does Cohen's~d at very small sample sizes, and that the  99\textsuperscript{th} percentile of the effect size of the Cauchy distribution is always somewhat less than the other three distributions for the $\mathcal{E}_{d}$ metric. In contrast, the  99\textsuperscript{th} percentile of the Cauchy distribution is much larger than the other distributions at low sample sizes, and tends to become smaller than the other distributions at large sample sizes. This behaviour is perhaps not surprising since the parametric Cohen's~d effect size should be undefined in a Cauchy distribution.

\begin{figure}[tpb]
\centerline{\includegraphics[scale=0.50]{F1-null-model.eps}}
\caption{Characteristics of $\mathcal{E}_{d}$ and when detecting differential features in two groups in random Normal, random uniform, random Cauchy and a heavily skewed random $\beta$ distribution as a function of the per group sample size. The top panels show the behaviour of the $\mathcal{E}_{d}$ and Cohen's~d effect sizes when the two groups are drawn from the same distribution. The simulation included one-hundred thousand random distributions for each. The bottom two panels show the behaviour when there is a difference between groups. Single examples of the distributions are shown in the Supplement. The dashed lines show the median effect size, the solid lines show the 99\textsuperscript{th} percentile of the statistics.  }
\label{fig:01}
\end{figure}

The bottom two panels in Figure \ref{fig:01} show the behaviour when there is a small difference between groups; for reference; Supplementary Figure 5 shows single examples of the distributions used  for comparison.  Again, we can see that the distribution of the median and 99\textsuperscript{th} percentile difference at low sample sizes and are remarkably similar when using the  non-parametric $\mathcal{E}_{d}$ effect size metric, but diverge substantially when using the parametric Cohen's~d. Thus, we conclude that if the data were drawn from a Normal distribution that Cohen's~d would be preferred. However, given that distributions from actual sequencing datasets are unknown, and can be nearly Cauchy, multimodal or skewed \citep{fernandes:2013},  the non-parametric $\mathcal{E}_{d}$ effect size statistic gives a more stable and robust estimate of the standardized difference between distributions than does Cohen's~d.

\begin{figure}[tpb]
\centerline{\includegraphics[scale=0.5]{F2-yeast_TPFP.eps}}
\caption{Comparing $\mathcal{E}_{d}$ and adjusted p-values when detecting differential features between two groups. The top four panels show the   median and 95\% confidence interval of the true positive rate (TPR) and false positive rate (FPR) at different cutoff values.  The top two panels show the values for the q-value, the Benjamini-Hochberg corrected p-value, as a function of sample size. The middle two panels show the values for $\mathcal{E}_{d}$. Data were summarized using either a  0 or 2-fold difference cutoff (D). The number of features of the 6236 non-0 features that are identified as significantly different in the full dataset are shown in the top right corner of each plot. The bottom two panels show the effect size distribution of features identified as false positives by $\mathcal{E}_{d}$ at four different sample sizes. The dashed grey lines show the cutoff for a 10\% FDR and an 80\% power to detect.}
\label{fig:02}
\end{figure}

With this null behaviour information, we can examine an example dataset of a highly replicated RNA-seq dataset geneated by \citep{Schurch:2016aa}. In this dataset, the edgeR tool identified over 4600 out of 6349 genes as `significant'  (Benjamini-Hochberg adjusted p-value $< 0.05$) when all samples were included using either the glm or exact test modes (Supplementary Table 1).  Other widely used tools gave similar results \citep{Schurch:2016aa}. The null hypthesis testing framework in ALDEx2 also returned at least 4300 genes in the same dataset. Thus, identifying such a large proportion of genes as differentially abundant indicates that statistical significance is not informative for this type of experiment. Schurch et al. (and others) recommend adding a secondary threshold such as a fold-change cutoff to identify genes of interest for follow-up analyses \citep{Cui:2003aa,Schurch:2016aa}. When sample sizes are sufficiently large, we would expect that the fold-change cutoff itself would be the primary determinant of difference; however, this approach would not include either the biological variance or the uncertainty of measurement in the analysis.

\begin{figure}[tpb]
\centerline{\includegraphics[scale=0.4]{F3-yeast_volcano.eps}}
\caption{Volcano and Effect plots of features identified by  $\mathcal{E}_{d}$,  adjusted p-values  and absolute differences when detecting differential features between two groups. These plots compare the features identified by $\mathcal{E}_{d}$ and by q-scores (FDR) and a 2-fold fold-change thresholds in the full dataset.  In these plots all features that have a q score less than 0.1 also have an effect size greater than 1. Thus, the features in magenta are only identified as significantly different by q scores, those in orange are significantly different by both their q score and their effect size, and features in blue are significant by their q score, their effect size and their absolute difference.  The dashed grey lines in the two plots demarcate the 2-fold difference location; note that the difference is in a log2 scale. The Volcano plot is rotated from its usual orientation and the vertical solid line in the Volcano plot indicates a q score of 0.1. The diagonal solid lines in the Effect plot indicate the boundary where the difference equals the dispersion; ie, where the effect size is 1.
}
\label{fig:03}
\end{figure}

The relationship between sample size and the number of features identified as significantly different using a null hypothesis testing framework in this dataset is shown in the top  panels of Figure \ref{fig:02}.  Here we are testing for the ability to detect all features that would have been observed as differentially expressed in the full dataset when using a random subset of the data. The top two panels were generated using the expected value of the Benjamini-Hochberg corrected Welch's t-test by ALDEx2, but similar plots can be observed using the edgeR tool  or others \citep{Schurch:2016aa}. As expected we observe that the power of a t-test is strongly affected by sample size when the minimum absolute difference between groups is 0. However, when the minimum feature difference is a 2-fold change (D=2), the number of features identified is reduced approximately 10 fold, and a relatively small number of samples is required for acceptable power. The tradeoff here is that applying both the q-score and difference cutoffs results in an increase in the FDR at small numbers of samples. Note that all tools have difficulty estimating the actual FDR in many datasets \citep{Thorsen:2016aa,hawinkel2017}.

The middle two panels of Figure \ref{fig:02} show the behaviour of the $\mathcal{E}_{d}$ statistic in the same random datasets. Note that even when only two samples are used, the $\mathcal{E}_{d}$ statistic identifies over 80\% of the features as different as are identified by the same statistic in the full dataset. Thus, the simple metric outlined here  can correctly identify the `true positive' set even when the number of samples is very small. The tradeoff when using this statistic is that at very low sample sizes the False Discovery Rate (fdr) is extreme; in this dataset and with and with a cutoff of $\mathcal{E}_{d} > 1$, the fdr is 40\% with two samples, but falls to less than 10\% only when there are 15 or more samples. Interestingly, applying a fold-change cutoff to the $\mathcal{E}_{d}$ metric reduces the false discovery rate dramatically and also reduces the number of features identified as significantly different.

The bottom two panels of Figure \ref{fig:02} show the effect size in the full dataset of false positive features identified as different in subsets of the dataset. We can see that at a sample size of 2 the false positive features have true effect-sizes that are nearly randomly distributed, but that at a sample size of 10 or more, the vast majority of false positive features have true effect sizes that are at least 50\% of the desired effect size. When applying the difference cutoff, we observe that the majority of false positive features identified at small sample sizes have an effect size greater than the cutoff when the sample size is 5 or more. Thus, this implies that false positive features in this case pass the effect size cutoff but fail on the absolute difference cutoff. Supplementary Figure 6 shows a similar analysis for a 16S rRNA gene sequencing dataset \citep{bian:2017} with similar results, even thought he abundance/variance relationship between the features is very different than in the transcriptome experiment (Supplementary Figure 7). Supplementary Figure 8 shows that the effect thresholds from the simulated data in Figure 1 are appropriate, and perhaps even conservative, for real HTS data, and provide further evidence that the features identified as different in Figure 2 are likely to be reproducible.

Taking the data from Figures \ref{fig:01} and \ref{fig:02}, and the Supplement together, we can provide guidance as to appropriate cutoff values when using the $\mathcal{E}_{d} $ statistic in HTS datasets. First,  point estimates of $\mathcal{E}_{d} $ in real datasets and in simulated distributions are highly congruent. Thus, we can estimate that for every 100 features in a HTS dataset, we can use the curves in Figures \ref{fig:01} and  Supplementary Figure 8 to determine the number of false positive features expected if there is truly no difference between groups. The CoDaSeq R package contains a function that can be used to empirically determine thresholds for any desired percentile cutoff. As an example, the plots in Figure \ref{fig:01} that as a rule of thumb for the experimentalist to be 99\% confident in a true positive, effect sizes should be greater than 2 when the sample size is  4, greater than 1 when the sample size is 10, and greater than 0.5 for sample sizes larger than 40. This rule of thumb holds true regardless of the true underlying distribution and is appropriate whenever point estimates are computed. When computing the expected $\mathcal{E}_{d} $ as does the aldex.effect function, these thresholds are much lower at small sample sizes, and effect sizes can be about 30\% smaller.

Figure \ref{fig:03} shows how the different threshold cutoffs relate to each other when plotted as a difference vs. q-score in a Volcano plot \citep{Cui:2003aa} and in a dispersion vs difference Effect plot \citep{gloor:effect}. These plots demonstrate the advantage of using a standardized effect metric over a q-score either with or without an absolute difference cutoff. In the Volcano plot, the q-score only features coloured in magenta, are exclusively features that have q scores near the upper bound of statistical significance. These include features with both large and small absolute differences, and the reason that a feature may have a large difference but a marginally significant q-score is not revealed on the Volcano plot. However, examination of the Effect plot shows that features that have a marginal q-score and large difference are features with very large dispersion; that is the difference between features, $\tilde{D}$, is much smaller than the within-group dispersion, MMAD, calculated as in Equation \ref{eq:ff}. Such features would not be expected to reproduce well in a new dataset because of their intrinsically high variance. In fact, these features are exactly those that are excluded by the $\mathcal{E}_{d} $ statistic.  Furthermore, adding the constraint that features have at least a 2-fold difference does not exclude these high-dispersion features from consideration.

Examination of the orange features in the Volcano and Effect plots, we can see that both q-scores and $\mathcal{E}_{d} $ can exhibit arbitrarily small absolute differences if the dispersion is very small. In the case of the $\mathcal{E}_{d} $ statistic, adding in the requirement for at least a 2-fold change reduces the number of features to only those that are both different and reproducible.

\section{Conclusion}

By default, we want to know both `what is significant' and `what is different' \citep{Cui:2003aa}.  Both of these questions can be addressed with a standardized effect size statistic that scales the difference between features by their dispersion. We have found plots of difference and dispersion to be an exceeding useful tool when examining HTS datasets \citep{gloor:effect}. Furthermore, datasets analyzed by this approach have proven to be remarkably reproducible as shown by independent lab validation \citep{macklaim:2013, nelson:2015vaginal}

The $\mathcal{E}_{d}$ statistic outlined here is a relatively robust statistic with the attractive property that it consistently  identifies almost all the same set of true features regardless of the underlying distribution as shown in Figure \ref{fig:01}, and the number of samples as shown in Figure \ref{fig:02}. In marked contrast, even the best p-value based  approaches can identify only a small proportion of the features at small samples sizes that would have been found in the full dataset \citep{Schurch:2016aa}. Thus, the simple metric outlined here  can correctly identify the `true positive' set even when the number of samples is very small. Note that fold-change thresholds as is commonly used, is not the same as an standardized effect statistic, and applying the threshold values of \citep{Schurch:2016aa} while reducing the features that are found does not necessarily enhance reproducibility (Figures \ref{fig:02} and \ref{fig:03} ).

The tradeoff when using the $\mathcal{E}_{d}$ statistic is that at very low sample sizes the False Discovery Rate can be extreme; in this dataset and with and with a cutoff of $\mathcal{E}_{d} > 1$, the FDR is 40\% with two samples, but falls to less than 10\% only when there are 15 or more samples. Adding in an absolute fold-change restriction reduces the FDR substantially. Further tempering this, is the observation that the false positive features are frequently  close to the cutoff in the full dataset: that is, false positive features typically are true positives at slightly lower effect sizes or absolute fold changes. This is in contrast to the well-known random uniform distribution of false positive p-values. The Supplement shows additional evidence that the  $\mathcal{E}_{d}$ statistic is generally useful, having essentially the same characteristics in a 16S rRNA gene sequencing dataset which has much larger per feature dispersion.

This work  describes the  $\mathcal{E}_{d}$ statistic for examining high throughput sequencing datasets. The statistic is relatively robust and efficient, and answers the question most desired by the biologist, namely `what is reproducibly different'.   $\mathcal{E}_{d}$ is computed in the ALDEx2 R package as the `effect' output where it is the median of the inferred technical and biological data, and in the CoDaSeq R package where it acts only the point estimates of the data. Interactive exploration of effect sizes can be done in the omicplotR Bioconductor package \citep{omicplot}.
\section*{Acknowledgements}

To be determined

\vspace{-12pt}
\section*{Funding}

This work has been supported by NSERC, CIHR.\vspace*{-12pt}

%\bibliographystyle{natbib}
%\bibliographystyle{achemnat}
%\bibliographystyle{plainnat}
%\bibliographystyle{abbrv}
\bibliographystyle{bioinformatics}
%
%\bibliographystyle{plain}
%
%\bibliography{Document}

%\bibliography{bibdesk_refs}
\begin{thebibliography}{}

\bibitem[Aitchison(1986)Aitchison]{aitchison:1986}
Aitchison, J. (1986).
\newblock {\em The Statistical Analysis of Compositional Data\/}.
\newblock Chapman \& Hall.

\bibitem[Bian {\em et~al.}(2017)Bian, Gloor, Gong, Jia, Zhang, Hu, Zhang,
  Zhang, Zhou, Zhang, Burton, Reid, Xiao, Zeng, Yang, and Li]{bian:2017}
Bian, G., Gloor, G.~B., Gong, A., Jia, C., Zhang, W., Hu, J., Zhang, H., Zhang,
  Y., Zhou, Z., Zhang, J., Burton, J.~P., Reid, G., Xiao, Y., Zeng, Q., Yang,
  K., and Li, J. (2017).
\newblock The gut microbiota of healthy aged chinese is similar to that of the
  healthy young.
\newblock {\em mSphere\/}, {\bf 2}(5), e00327--17.

\bibitem[Coe(2002)Coe]{coe2002s}
Coe, R. (2002).
\newblock It's the effect size, stupid: What effect size is and why it is
  important.

\bibitem[Colquhoun(2014)Colquhoun]{Colquhoun:2014aa}
Colquhoun, D. (2014).
\newblock An investigation of the false discovery rate and the
  misinterpretation of p-values.
\newblock {\em R Soc Open Sci\/}, {\bf 1}(3), 140216.

\bibitem[Cui and Churchill(2003)Cui and Churchill]{Cui:2003aa}
Cui, X. and Churchill, G.~A. (2003).
\newblock Statistical tests for differential expression in cdna microarray
  experiments.
\newblock {\em Genome Biol\/}, {\bf 4}(4), 210.1 -- 210.10.

\bibitem[Cumming(2008)Cumming]{Cumming:2008aa}
Cumming, G. (2008).
\newblock Replication and p intervals: p values predict the future only
  vaguely, but confidence intervals do much better.
\newblock {\em Perspect Psychol Sci\/}, {\bf 3}(4), 286--300.

\bibitem[Fernandes {\em et~al.}(2013)Fernandes, Macklaim, Linn, Reid, and
  Gloor]{fernandes:2013}
Fernandes, A.~D., Macklaim, J.~M., Linn, T.~G., Reid, G., and Gloor, G.~B.
  (2013).
\newblock Anova-like differential expression (aldex) analysis for mixed
  population rna-seq.
\newblock {\em PLoS One\/}, {\bf 8}(7), e67019.

\bibitem[Fernandes {\em et~al.}(2014)Fernandes, Reid, Macklaim, McMurrough,
  Edgell, and Gloor]{fernandes:2014}
Fernandes, A.~D., Reid, J.~N., Macklaim, J.~M., McMurrough, T.~A., Edgell,
  D.~R., and Gloor, G.~B. (2014).
\newblock Unifying the analysis of high-throughput sequencing datasets:
  characterizing {RNA}-seq, 16{S} r{RNA} gene sequencing and selective growth
  experiments by compositional data analysis.
\newblock {\em Microbiome\/}, {\bf 2}, 15.1--15.13.

\bibitem[Giguere {\em et~al.}(2019)Giguere, Macklaim, and Gloor]{omicplot}
Giguere, D., Macklaim, J., and Gloor, G. (2019).
\newblock omicplotr: Visual exploration of omic datasets using a shiny app.
\newblock Bioconductor v1.4.0.

\bibitem[Gloor {\em et~al.}(2016a)Gloor, Macklaim, Vu, and
  Fernandes]{gloorAJS:2016}
Gloor, G.~B., Macklaim, J.~M., Vu, M., and Fernandes, A.~D. (2016a).
\newblock Compositional uncertainty should not be ignored in high-throughput
  sequencing data analysis.
\newblock {\em Austrian Journal of Statistics\/}, {\bf 45}, 73--87.

\bibitem[Gloor {\em et~al.}(2016b)Gloor, Macklaim, and Fernandes]{gloor:effect}
Gloor, G.~B., Macklaim, J.~M., and Fernandes, A.~D. (2016b).
\newblock Displaying variation in large datasets: Plotting a visual summary of
  effect sizes.
\newblock {\em Journal of Computational and Graphical Statistics\/}, {\bf
  25}(3C), 971--979.

\bibitem[Gloor {\em et~al.}(2017)Gloor, Macklaim, Pawlowsky-Glahn, and
  Egozcue]{gloorFrontiers:2017}
Gloor, G.~B., Macklaim, J.~M., Pawlowsky-Glahn, V., and Egozcue, J.~J. (2017).
\newblock Microbiome datasets are compositional: And this is not optional.
\newblock {\em Front Microbiol\/}, {\bf 8}, 2224.

\bibitem[Halsey {\em et~al.}(2015)Halsey, Curran-Everett, Vowler, and
  Drummond]{Halsey:2015aa}
Halsey, L.~G., Curran-Everett, D., Vowler, S.~L., and Drummond, G.~B. (2015).
\newblock The fickle p value generates irreproducible results.
\newblock {\em Nat Methods\/}, {\bf 12}(3), 179--85.

\bibitem[Hawinkel {\em et~al.}(2018)Hawinkel, Mattiello, Bijnens, and
  Thas]{hawinkel2017}
Hawinkel, S., Mattiello, F., Bijnens, L., and Thas, O. (2018).
\newblock A broken promise : microbiome differential abundance methods do not
  control the false discovery rate.
\newblock {\em BRIEFINGS IN BIOINFORMATICS\/}.

\bibitem[Ioannidis(2005)Ioannidis]{Ioannidis:2005aa}
Ioannidis, J. P.~A. (2005).
\newblock Why most published research findings are false.
\newblock {\em PLoS Med\/}, {\bf 2}(8), e124.

\bibitem[Kaul {\em et~al.}(2017)Kaul, Mandal, Davidov, and
  Peddada]{Kaul:2017aa}
Kaul, A., Mandal, S., Davidov, O., and Peddada, S.~D. (2017).
\newblock Analysis of microbiome data in the presence of excess zeros.
\newblock {\em Front Microbiol\/}, {\bf 8}, 2114.

\bibitem[Lovell {\em et~al.}(2015)Lovell, Pawlowsky-Glahn, Egozcue, Marguerat,
  and B{\"a}hler]{Lovell:2015}
Lovell, D., Pawlowsky-Glahn, V., Egozcue, J.~J., Marguerat, S., and B{\"a}hler,
  J. (2015).
\newblock Proportionality: a valid alternative to correlation for relative
  data.
\newblock {\em PLoS Comput Biol\/}, {\bf 11}(3), e1004075.

\bibitem[Macklaim {\em et~al.}(2013)Macklaim, Fernandes, Di~Bella, Hammond,
  Reid, and Gloor]{macklaim:2013}
Macklaim, J.~M., Fernandes, A.~D., Di~Bella, J.~M., Hammond, J.-A., Reid, G.,
  and Gloor, G.~B. (2013).
\newblock Comparative meta-rna-seq of the vaginal microbiota and differential
  expression by lactobacillus iners in health and dysbiosis.
\newblock {\em Microbiome\/}, {\bf 1}(1), 12.

\bibitem[Nakagawa(2004)Nakagawa]{shinichi:2004}
Nakagawa, S. (2004).
\newblock A farewell to bonferroni: the problems of low statistical power and
  publication bias.
\newblock {\em Behavioral Ecology\/}, {\bf 15}(6), 1044--1045.

\bibitem[Nelson {\em et~al.}(2015)Nelson, Borgogna, Brotman, Ravel, Walk, and
  Yeoman]{nelson:2015vaginal}
Nelson, T.~M., Borgogna, J.-L.~C., Brotman, R.~M., Ravel, J., Walk, S.~T., and
  Yeoman, C.~J. (2015).
\newblock Vaginal biogenic amines: biomarkers of bacterial vaginosis or
  precursors to vaginal dysbiosis?
\newblock {\em Frontiers in physiology\/}, {\bf 6}.

\bibitem[Schurch {\em et~al.}(2016)Schurch, Schofield, Gierli{\'n}ski, Cole,
  Sherstnev, Singh, Wrobel, Gharbi, Simpson, Owen-Hughes, Blaxter, and
  Barton]{Schurch:2016aa}
Schurch, N.~J., Schofield, P., Gierli{\'n}ski, M., Cole, C., Sherstnev, A.,
  Singh, V., Wrobel, N., Gharbi, K., Simpson, G.~G., Owen-Hughes, T., Blaxter,
  M., and Barton, G.~J. (2016).
\newblock How many biological replicates are needed in an rna-seq experiment
  and which differential expression tool should you use?
\newblock {\em RNA\/}, {\bf 22}(6), 839--51.

\bibitem[Soneson and Delorenzi(2013)Soneson and Delorenzi]{Soneson:2013}
Soneson, C. and Delorenzi, M. (2013).
\newblock A comparison of methods for differential expression analysis of
  {RNA-seq} data.
\newblock {\em BMC Bioinformatics\/}, {\bf 14}, 91.

\bibitem[Thorsen {\em et~al.}(2016)Thorsen, Brejnrod, Mortensen, Rasmussen,
  Stokholm, Al-Soud, S{\o}rensen, Bisgaard, and Waage]{Thorsen:2016aa}
Thorsen, J., Brejnrod, A., Mortensen, M., Rasmussen, M.~A., Stokholm, J.,
  Al-Soud, W.~A., S{\o}rensen, S., Bisgaard, H., and Waage, J. (2016).
\newblock Large-scale benchmarking reveals false discoveries and count
  transformation sensitivity in 16{S} r{RNA} gene amplicon data analysis
  methods used in microbiome studies.
\newblock {\em Microbiome\/}, {\bf 4}(1), 62.

\bibitem[Tsilimigras and Fodor(2016)Tsilimigras and Fodor]{Tsilimigras:2016aa}
Tsilimigras, M. C.~B. and Fodor, A.~A. (2016).
\newblock Compositional data analysis of the microbiome: fundamentals, tools,
  and challenges.
\newblock {\em Ann Epidemiol\/}, {\bf 26}(5), 330--5.

\bibitem[Weiss {\em et~al.}(2017)Weiss, Xu, Peddada, Amir, Bittinger, Gonzalez,
  Lozupone, Zaneveld, V{\'a}zquez-Baeza, Birmingham, Hyde, and
  Knight]{Weiss:2017aa}
Weiss, S., Xu, Z.~Z., Peddada, S., Amir, A., Bittinger, K., Gonzalez, A.,
  Lozupone, C., Zaneveld, J.~R., V{\'a}zquez-Baeza, Y., Birmingham, A., Hyde,
  E.~R., and Knight, R. (2017).
\newblock Normalization and microbial differential abundance strategies depend
  upon data characteristics.
\newblock {\em Microbiome\/}, {\bf 5}(1), 27.

\end{thebibliography}

\end{document}


%--------------------------------------------------------------------------
% Conclusions and Recommendations
%--------------------------------------------------------------------------

\chapter{Conclusions and Recommendations}
\label{cha: conclusions}

This study found multi-year ALOS/PALSAR mosaic data to be temporally consistent and capable of classifying between forest and non-forest areas; hence making it suitable tool for mapping and monitoring forest extent. Unlike fully polarimetric SAR data which require more complex image processing, the mosaic data is less complicated and is provided to the public free of charge, which makes it a viable data source for operational, wall-to-wall forest mapping and monitoring. In terms of discriminating forest types, ALOS/PALSAR was not suitable for discriminating between forest types based on canopy characteristics or a forest typology such as the FAO Global Forest Resource Assessment classification scheme.

Cluster analysis provided information on structure and hierarchy of forest types in aid of classification. The analysis can similarly be applied for other classification systems to reveal hierarchy and structure of land cover types, which can guide subsequent classification processes. Decision trees were successfully used in past research for classifying land and forest cover types using SAR data, and even outperformed other statistical classification techniques. A decision tree classifier was also applied in this study, which provided information on classification accuracies to assess the degree to which different forest types can be discriminated using a combination of feature attributes. A combination of hierarchical cluster analysis and decision tree classification techniques is recommended as a viable approach for determining forest hierarchy by finding structure in the data and using the output dendrogram to guide the subsequent decision tree classification.

However, while the multi-level hierarchical classification approach is recommended, low classification accuracies (or high error rates) were obtained, which may be primarily due to the incompatibility of the sensitivities of the PALSAR sensor to forest structural attributes with the forest classification scheme used, particularly the forest types of the FAO Global Forest Resources Assessment. Hence, a different forest classification system such as the one devised by Fernando et al. \citeyearpar{fernando_forest_2008}, which described Philippine forest formations typified based on their physical attributes, particularly physiognomy, elevation, and soil substrate may be a explored as a plausible forest type classification scheme, and a topic of subsequent research for mapping Philippine forest cover types using SAR data.

An object-based approach also allowed the use of multiple sources of data to increase the dimensionality of the input data for classification. Topographic and texture feature attributes helped to decrease classification error rates in contrast to using solely polarimetric attributes for classifying land and forest cover types. In turn, the availability of multiple data sources was maximised by the decision tree classifier through its selection of the most parsimonious feature attributes to improve the classification outcome. A subject of future research can be to test the sensitivity of specific texture measures to forest type classification, and exploring the effects of varying image segmentation parameters in terms of assessing temporal consistency, multi-level classification hierarchy, and classification accuracy.

Discriminating and mapping forest cover types is a key piece of information to support forest management. Combining multi-frequency SAR data such as C-band and L- band radar sensors is suggested as the subject of future research to improve the discrimination of forest types. C-band data such as Sentinel-1A sensor, which is sensitive to forest canopy components can complement L-band PALSAR data, which is more sensitive to forest structure components. Since SAR overcomes limitations of optical data, combining the data from these two SAR platforms have the potential to provide spatially explicit information in support of an operational, wall-to-wall forest mapping and monitoring system.


\chapter{DCM for Phase Coupling \label{Chap:data:dcm_phase}}

This chapter presents an extension of the Dynamic Causal Modelling (DCM) framework to the analysis of phase-coupled data. A weakly coupled oscillator approach is used to describe dynamic phase changes in a network of oscillators.
The influence that the phase of one oscillator has on the change of phase of another is characterised in terms of a Phase Interaction Function (PIF) as described in \cite{dcm_phase}. SPM supports PIFs specified using arbitrary order Fourier series. However, to simplify the interface, one is restricted to simple sinusoidal PIFs with the GUI.

\section{Data}

We will use the merged epoched MEG face-evoked dataset\footnote{Multimodal face-evoked dataset: \url{http://www.fil.ion.ucl.ac.uk/spm/data/mmfaces/}} saved in the files:

\begin{verbatim}
cdbespm12_SPM_CTF_MEG_example_faces1_3D.mat
cdbespm12_SPM_CTF_MEG_example_faces1_3D.dat
\end{verbatim}

%See ~\ref{multimodal:data:meg:preproc} for instructions on how to generate this file from raw MEG data.
DCM-Phase requires a head model and coregistration. If you have been following the previous chapters of this tutorial, these should already be available in the dataset. Otherwise, you should perform the 'Prepare' and '3D Source reconstruction' steps described earlier in the chapter, with the latter
comprising the MRI, Co-register, Forward and Save sub-steps.

\section{Getting Started}

You need to start SPM and toggle ``EEG'' as the modality (bottom-right of SPM main window), or start SPM with \texttt{spm eeg}. In order for this to work you need to ensure that the main SPM directory is on your \matlab\ path.
After calling \texttt{spm eeg}, you see SPM's graphical user interface, the top-left window. The button for calling the DCM-GUI is found in the second partition from the top, on the right hand side. When pressing the button, the GUI pops up (Figure~\ref{dcm-ir:fig:1}).

\section{Data and design}

 You should switch the DCM model type to ``PHASE'' which is the option for DCM-Phase. Press ``new data'' and select the data file \texttt{cdbespm12\_\-SPM\_\-CTF\_\-MEG\_\-example\_\-faces1\_\-3D.mat}. This is an epoched data file with multiple trials per condition.  On the right-hand side enter the trial indices
\begin{verbatim}
1 2
\end{verbatim}
for the 'face' and 'scrambled' evoked responses (we will model both trial types).
 The box below this list allows for specifying experimental effects on connectivity. Enter
\begin{verbatim}
1 0
\end{verbatim}
in the first row of the box. This means that ``face'' trial types can have different connectivity parameters than ''scrambled'' trial types. If we now click somewhere outside the box, a default name will be assigned to this effect - ``effect1''. It will appear in the small text box next to the coefficients box. It is possible to change this name to something else e.g. ``face''. Now we can select the peristimulus time window we want to model.
Set it to:
\begin{verbatim}
1 300
\end{verbatim}
ms. Select \texttt{1} for ``detrend'', to  remove the mean from each data record.  The \texttt{sub-trials} option makes it possible to select just a subset of trials for the analysis (select 2 for every second trial, 3 - for every third etc.). This is useful because DCM-Phase takes quite a long time to invert for all the trials and you might want to first try a smaller subset to get an idea about the possible results. Here we will assume that you used all the trials (sub-trials was set to 1). You can now click on the $>$ (forward) button, which will bring you to the next stage \textit{electromagnetic model}. From this part, you can press the red $<$ button to get back to the data and design part.

\section{Electromagnetic model}

With DCM-Phase, there are two options for how to extract the source data. Firstly, you can use 3 orthogonal single equivalent current dipoles (ECD) for each source, invert the resulting source model to get 3 source waveforms and take the first principal component. This option is suitable for multichannel EEG or MEG data. Alternatively, you can treat each channel as a source (LFP option). This is appropriate when the channels already contain source data either recorded directly with intracranial electrodes or extracted (e.g. using a beamformer).

Note that a difference to DCM for evoked responses is that the parameters of the spatial model are not optimized. This means that DCM-Phase (like DCM-IR) will project the data into source space using the spatial locations you provide.

We will use the ECD option and specify just two source regions. This requires specifying a list of source names in the left large text box and a list of MNI coordinates for the sources in the right large text box. Enter the following in the left box:

\begin{verbatim}
LOFA
LFFA
\end{verbatim}

Now enter in the right text box:

\begin{verbatim}
-39 -81 -15
-39 -51 -24
\end{verbatim}

These correspond to left Occipital Face Area, and left Fusiform Face Area.
The onset-parameter is irrelevant for DCM-Phase. Now hit the $>$ (forward) button and proceed to the \textit{neuronal model}. Generally, if you have not used the input dataset for 3D source reconstruction before you will be asked to specify the parameters of the head model at this stage.

\section{Neuronal model}

We will now define a coupled oscillator model for investigating network synchronization of alpha activity. To this end, we first enter the values 8 and 12 to define the frequency window. The wavelet number is irrelevant for DCM-Phase. After source reconstruction (using a pseudo-inverse approach), source data is bandpass filtered and then the Hilbert transform is used to extract the instantaneous phase. The DCM-Phase model is then fitted used standard routines as described in \cite{dcm_phase}.

Figure ~\ref{dcm-phase:fig:1} shows the four models we will apply to the
M/EEG data. We will first fit model 4. This model proposes that alpha activity in region LOFA changes its phase so as to synchronize with activity in region LFFA. In this network LFFA is the master and LOFA is the slave. Moreover, the
connection from LFFA to LOFA is allowed to be different for scrambled versus unscrambled faces.

\begin{figure}
\begin{center}
\includegraphics[width=160mm]{dcm_phase/figures/phase_models}
\caption{\em Four different DCM-Phase models  \label{dcm-phase:fig:1}}
\end{center}
\end{figure}

The connectivity for Model 4 can be set up by configuring the radio buttons as shown in Figure~\ref{dcm-phase:fig:2}. You can now press the \texttt{Invert DCM} button. It can take up to an hour to estimate the model parameters depending on the speed of your computer.

\begin{figure}
\begin{center}
\includegraphics[width=50mm]{dcm_phase/figures/model4_conn}
\caption{\em Radio button configurations for DCM-Phase model 4  \label{dcm-phase:fig:2}}
\end{center}
\end{figure}

\section{Results}

After estimation is finished, you can assess the results by choosing from the pull-down menu at the bottom (middle). The \texttt{Sin(Data)-Region i} option will show the sin of the phase data in region $i$, for the first 16 trials. The blue line corresponds to the data and the red to the DCM-Phase model fit. The
\texttt{Coupling(As)} and \texttt{Coupling(Bs)} buttons display the estimated endogenous and modulatory activity shown in Figure~\ref{dcm-phase:fig:3}.

\begin{figure}
\begin{center}
%\includegraphics[width=50mm]{dcm_phase/figures/endogenous}
\includegraphics[width=50mm]{dcm_phase/figures/modulatory}
\caption{\em The figure shows the estimated parameters for endogenous coupling (left column) and modulatory parameters (right column) for the 4th DCM.  \label{dcm-phase:fig:3}}
\end{center}
\end{figure}

If one fits all the four models shown in Figure~\ref{dcm-phase:fig:1} then they can be formally compared using Bayesian Model Selection. This is implemented by pressing the \texttt{BMS} button. You will need to first create a directory for the results to go in e.g. \texttt{BMS-results}. For 'Inference Method' select FFX (the RFX option is only viable if you have models from a group of subjects). Under 'Data', Select 'New Subject' and under 'Subject' select 'New Session'. Then under 'Models' select the \texttt{DCM.mat} files you have created.  Then press the green play button. This will produce the results plot shown in Figure~\ref{dcm-phase:fig:4}. This leads us to conclude that
LFFA and LOFA act in master slave arrangement with LFFA as the master.

\begin{figure}
\begin{center}
\includegraphics[width=50mm]{dcm_phase/figures/bms_phase}
\caption{\em Bayesian comparison of the four DCM-Phase models shown in Figure~\ref{dcm-phase:fig:1}.\label{dcm-phase:fig:4}}
\end{center}
\end{figure}

\section{Extensions}

In the DCM-Phase model accessible from the GUI, it is assumed that the phase interaction functions are of simple sinusoidal form ie. $a_{ij} \sin(\phi_j - \phi_i)$. The coefficients $a_{ij}$ are the values shown in the endogenous
parameter matrices in eg. Figure~\ref{dcm-phase:fig:3}. These can then be changed by an amount $b_{ij}$ as shown in the modulatory parameter matrices.
It is also possible to specify and estimate DCM-Phase models using matlab scripts. In this case it is possible to specify more generic phase interaction functions, such as arbitrary order Fourier series. Examples are given in \cite{dcm_phase}.


\section{WIMPs pair production}
    \label{sec:WIMPsProduction}

In this section, the results of the monojet analysis are converted into limits on the pair production of WIMPs.
Samples for several models based on an effective theory (D5, D8 and D9, see Table~\ref{tab:WIMPsEffectiveOperators}) corresponding to the process $\wimpwimp$ have been generated.
They are implemented using LO matrix elements in \madgraph{}.
The WIMP pair production plus one or two additional partons from ISR/FSR is considered.
For each operator, a sample is generated requiring at least one parton with $\pt>\unit[80]{GeV}$, and another sample is generated with at least one parton with a minimum $\pt$ of 300~GeV.
The latter samples are needed to populate those signal regions with $\met$ requirements larger than 350~GeV.
Only initial states of the four lightest quarks are considered, assuming equal coupling strengths for all quark flavors to the WIMPs.
The generated events are interfaced to \pythia{} for the parton showering and hadronization.
The MLM prescription is used for matching the matrix element calculations to the parton shower evolution.
The PDF set CTEQ6L1 is used for the event simulation, and the renormalization and factorization scales are set to the geometric average of $m_{\chi}^2 + \pt^2$ for the two WIMPs, being $m_{\chi}$ is the mass of the WIMP.
Events with WIMP masses between $\unit[10]{GeV}$ and $\unit[1.3]{TeV}$ are simulated for the different effective operators considered.

To study the transition between the effective field theory and a physical renormalizable model for Dirac fermion WIMPs coupling to SM particles via a new mediator particle $Z'$, a simplified model is generated with \madgraph{}.
For each WIMP mass, mediator particle masses $M_{\text{med}}$ between $\unit[50]{GeV}$ and $\unit[13]{TeV}$ are considered, for two different mediator particle width each ($\Gamma_{\text{med}} = M_{\text{med}}/3$ and $\Gamma_{\text{med}} = M_{\text{med}}/8\pi$).

For each effective model, the limits are extracted from the signal region M3, since it has the best expected sensitivity.
This is translated into corresponding 90\% CL limits\footnote{The limits are extracted at 90\% CL instead of 95\% CL, in order for them to be compared to direct dark matter search experiments.} on the suppression scale $M^{\ast}$ as a function of $m_{\chi}$.
To derive the lower limits in $M^{\ast}$, the $CL_s$ approach described in Section~\ref{sec:ComputationOfLimits} is used.

The systematic uncertainties for these models are computed as described in Chapter~\ref{chapter:Interpretations}.
Experimental uncertainties on jets and $\met$ scale and resolution, lepton efficiency, and luminosity translate into a 5\% to 3\% uncertainty on the signal yields for D5 and D8 models for a WIMP masses between 50~GeV and 1.3~TeV.
For the model D9, the experimental uncertainty varies from 1.5\% to 4\% for the same WIMP mass range.
The theoretical uncertainties include: the uncertainty on the renormalization, factorization scales; the uncertainty on the matrix element to parton shower matching scales; the uncertainty on the modeling of the ISF/FSR and the uncertainty on the PDFs.
These uncertainties translate to an effect between 3\% to 8\% in the signal yield, depending on the operator and the WIMP mass.

The 90\% CL limits on $M^{\ast}$ for the operators D5 (vector), D8 (axial-vector) and D9 (tensor) are reported in Table~\ref{tab:WIMPsEffective_limit_mstar} and shown in Figure~\ref{fig:WIMPsEffectiveMstar}, down to WIMP masses of 10~GeV.
These limits are extrapolated even further to smaller $m_{\chi}$ values, since for such low-mass WIMPs there is a negligible change in the fiducial cross section and kinematic distributions.

%--- \ref{tab:WIMPsEffective_limit_mstar}
\input{Interpretations/Tables/WIMPsEffectiveMstar.tex}

The effective theories used are based on the assumption that a new mediator particle couples SM particles to pairs of WIMPs, and that the mass of the mediator is much larger than the scale of the interaction.
If this is the case, the mediator cannot be produced directly in the collisions, and therefore it can be integrated out by the effective formalism.
However, this assumption is not always correct at the LHC, where the momentum transfer can reach the TeV energies.
For a given operator, one possible validity criterion would be that the momentum transferred in the hard interaction, $Q_{\text{tr}}$ is below the mediator mass, $M_{\text{med}}$, defined as $M_{\text{med}} = \sqrt{g_{q} g_{\chi}} M^{\ast}$, where $g_{q}$ and $g_{\chi}$ are the couplings of the mediator to the SM particles and the WIMPs, respectively.
Figure~\ref{fig:WIMPsEffectiveMstar} also shows the 90\% CL upper limit on $M^{\ast}$ when this ``truncation'' criteria ($Q<M_{\text{med}}$) is imposed, assuming $\sqrt{g_{q} g_{\chi}} = 1$.
The truncated limits fulfill the respective validity criteria wherever the lines are drawn in the figure.
For D5 (D9) for example, the criterion is fulfilled for WIMP masses up to 100~GeV (200~GeV).
No attempt is made to consider different couplings than $\sqrt{g_{q} g_{\chi}} = 1$ in these effective models.
The limits on $M^{\ast}$ for the truncated effective models are also quoted in Table~\ref{tab:WIMPsEffective_limit_mstar_trunc}.
The thermal relic lines are also included from Ref.~\cite{Goodman:2010ku} to this figure, corresponding to a coupling, set by $M^{\ast}$, of WIMPs to quarks such that WIMPs have the correct relic abundance as measured by the WMAP satelite~\cite{Larson:2010gs}, in the absense of any other interaction, apart from the one considered.

%--- \ref{tab:WIMPsEffective_limit_mstar_trunc}
\input{Interpretations/Tables/WIMPsEffectiveMstar_trunc.tex}

\begin{figure}[!t]
   \centering
   \mbox{
   \includegraphics[width=0.495\textwidth]{Interpretations/Figures/WIMPmStar_D5.eps}
   \includegraphics[width=0.495\textwidth]{Interpretations/Figures/WIMPmStar_D8.eps}
   }
   \mbox{
   \includegraphics[width=0.495\textwidth]{Interpretations/Figures/WIMPmStar_D9.eps}
   }
   \caption[Observed and expected 90\% CL limits on $M^*$ as a function of the WIMP mass $m_\chi$ for different operators in the selection M3.]{Expected and observed 90\% CL limits on $M^{\ast}$ as a function of the WIMP mass $m_\chi$ for an integrated luminosity of 20.3~\ifb for the D5 (vector), D8 (axial-vector) and D9 (tensor) operators in the M3 signal region. The Expected and observed limits are shown as dashed blue and solid black lines, respectively.
     The $\pm 1\sigma$ error band in yellow around the expected limit is due to the acceptance uncertainties (both experimental and theoretical). The rising green lines are the $M^{\ast}$ values at which WIMPs of the given mass result in the relic density as measured by WMAP~\cite{Larson:2010gs}, assuming annihilation in the early universe proceeded exclusively via the given operator.
     The thermal relic line for D8 has a bump feature at the top quark mass where the annihilation channel to top quarks opens.
     The purple dot-dashed  line is the 95\% CL observed limit on $M^{\ast}$ imposing a validity criterion with a coupling strength of 1.}
   \label{fig:WIMPsEffectiveMstar}
\end{figure}

A way to avoid the validity issues of the effective theories is to use a simplified model which includes a mediator explicitly.
Here as an example, a vector-like $Z'$ boson is considered, corresponding to the D5 operator in the effective theory framework, of a given mass, $M_{\text{med}}$, and width, $\Gamma_{\text{med}}$.
With this approach, the product of the coupling constants of the $Z'$ can be constrained.
Figure~\ref{fig:WIMPsSimplifiedCouplingLimit} shows the 90\% CL limits on $\sqrt{g_{q}g_{\chi}}$ as a function of $M_{\text{med}}$ for different values of the WIMP mass and the width of the mediator.
Couplings above 1 and 1.5 are excluded for WIMP masses between 25~GeV and 400~GeV, respectively, for mediator masses between 25~GeV and around 1~TeV.
For higher mediator masses, the limits on the couplings have higher values.
In particular, for mediator masses around 10~TeV, the couplings enter the non-perturbative regime and the theory is not anymore valid, for any WIMP mass and mediator width configuration.

\begin{figure}[!t]
  \begin{center}
    \mbox{
      \includegraphics[width=0.795\textwidth]{Interpretations/Figures/WIMPsimplified_MmedCoupling.eps}
    }
  \end{center}
  \caption[Observed 90\% CL limits on the product of the coupling constants as a function of the mediator mass, assuming a $Z'$-like simplified model.]{Observed 90\% CL limits on the product of the coupling constants, $\sqrt{g_q g_\chi}$, as a function of the mediator mass, $M_\text{med}$, assuming a $Z'$-like simplified model and a DM mass of 50~GeV and 400~GeV. The width of the mediator is varied between $M_\text{med}/3$ and $M_\text{med}/8\pi$.}
  \label{fig:WIMPsSimplifiedCouplingLimit}
\end{figure}

These limits can be translated into 90\% CL limits on $M^{\ast}$ as a function of $M_{\text{med}}$.
Figure~\ref{fig:WIMPsSimplifiedMstarLimit} demonstrates how for a given mediator particle mass and two values of the width, the real value of the mass suppression scale would compare to the $M^{\ast}$ derived assuming a contact interaction (shown as dashed line in the figure).
This contact interaction regime is reached by values of the $M_{\text{med}}$ larger than 5~TeV.
In the $\unit[700]{GeV} < M_\text{med} < \unit[5]{TeV}$ the mediator would be produced resonantly, and therefore the actual $M^{\ast}$ value is higher than in the contact interaction regime.
For lower mediator masses, the limit on $M^{\ast}$ is very low, since the WIMP would be heavier than the mediator, and the WIMP pair production via this mediator would be kinematically suppressed.
In this region, the contact interaction limits would be optimistic and overestimate the actual $M^{\ast}$ values.

\begin{figure}[!t]
  \begin{center}
    \mbox{
      \includegraphics[width=0.795\textwidth]{Interpretations/Figures/WIMPsimplified_MstarMmed_vector.eps}
    }
  \end{center}
  \caption[Observed limits on $M^{\ast}$ as a function of the mediator mass assuming a $Z'$-like simplified model.]{Observed limits on $M^{\ast}$ as a function of the mediator mass, $M_\text{med}$, assuming a $Z'$-like simplified model and a DM mass of 50~GeV and 400~GeV.
  The width of the mediator is varied between $M_\text{med}/3$ and $M_\text{med}/8\pi$.
  The corresponding limits of the effective model D5 are shown as dashed lines.
  Contour lines indicating a range of values of the product of the coupling constants, $\sqrt{g_q g_\chi}$, are also shown.}
  \label{fig:WIMPsSimplifiedMstarLimit}
\end{figure}

In the effective operator approach, the bounds on $M^{\ast}$ for a given $m_{\chi}$ can be converted to bounds on WIMP-nucleon scattering cross section, $\sigma_{\chi N}$, probed by direct DM experiments, using the transformation equations \ref{eq:WIMP-Nucleon}:

\begin{equation}
\begin{split}
\sigma^{D5}_{\chi N} &= 1.38 \times \unit[10^{-37}]{cm^2} \times \left(\frac{\mu_{\chi}}{\unit[1]{GeV}}\right)^2\left(\frac{\unit[300]{GeV}}{M^{\ast}}\right)^4 \\
\sigma^{D8}_{\chi N} &= 4.70 \times \unit[10^{-40}]{cm^2} \times \left(\frac{\mu_{\chi}}{\unit[1]{GeV}}\right)^2\left(\frac{\unit[300]{GeV}}{M^{\ast}}\right)^4 \\
\sigma^{D9}_{\chi N} &= 4.70 \times \unit[10^{-40}]{cm^2} \times \left(\frac{\mu_{\chi}}{\unit[1]{GeV}}\right)^2\left(\frac{\unit[300]{GeV}}{M^{\ast}}\right)^4. \\
\end{split}
\label{eq:WIMP-Nucleon_2}
\end{equation}

These bounds describe the scattering of WIMPs from nucleons at a very low momentum transfer, of the order of the keV.
Depending on the type of interaction, contributions to spin-dependent and spin-independent WIMP-nucleon interactions are expected.
The 90\% CL lower limits on the WIMP-nucleon scattering cross section are shown in Figure~\ref{fig:WIMPsNucleonXSection}.
Under the assumption made by the effective approach, these limits are relevant in the low DM mass region, and remain important in the full $m_{\chi}$ range covered.
Cross sections above $\unit[2.7\times10^{-40}]{cm^{2}}$ ($\unit[7.0\times10^{-38}]{cm^{2}}$) are excluded for WIMP masses of 1~GeV (1.3~TeV), respectively.
The spin-dependent limits in this figure are based on D8 and D9, where for D8 the limits have been calculated with the D5 acceptances, since they are identical, together with the D8 production cross section.
Both limits are significantly stronger than those from direct-detection experiments.
For D8, cross sections above $\unit[1.0\times10^{-41}]{cm^{2}}$ ($\unit[1.2\times10^{-38}]{cm^{2}}$) are excluded at 90\% CL for WIMP masses of 1~GeV (1.3~TeV), respectively, while for D9, cross sections above $\unit[1.1\times10^{-42}]{cm^{2}}$ ($\unit[9.8\times10^{-40}]{cm^{2}}$) are excluded in the same WIMP mass range.
The limits on the non-truncated and truncated $\sigma_{\chi N}$ are also shown in Tables~\ref{tab:WIMPsEffective_limit_sigma} and~\ref{tab:WIMPsEffective_limit_sigma_truncated}, respectively.

%--- \ref{tab:WIMPsEffective_limit_sigma}
\input{Interpretations/Tables/WIMPsEffectiveCrossSection.tex}

%--- \ref{tab:WIMPsEffective_limit_sigma_truncated}
\input{Interpretations/Tables/WIMPsEffectiveCrossSectionTruncated.tex}

\begin{figure}[ht]
   \centering
   \mbox{
     \includegraphics[width=0.795\textwidth]{Interpretations/Figures/WIMPnucleonXsection_spinIndependent.eps}
   }
   \mbox{
     \includegraphics[width=0.795\textwidth]{Interpretations/Figures/WIMPnucleonXsection_spinDependent.eps}
   }
\caption[90\% CL lower limits on spin-independent and spin-dependent the WIMP-nucleon scattering cross section.]{The 90\% CL lower limits on spin-independent (top) and spin-dependent (bottom) WIMP-nucleon scattering cross section for different masses of $\chi$ in M3 signal region.
 Results from direct detection experiments for the spin-independent~\cite{Angloher:2011uu,Akerib:2013tjd,Agnese:2013rvf,Agnese:2014aze,Aalseth:2014jpa,Bernabei:2008yi,Bernabei:2010mq,Aprile:2012nq,Aprile:2013doa} and spin-dependent~\cite{Archambault:2012pm,Desai:2004pq,Abbasi:2009uz,Behnke:2010xt,Felizardo:2011uw} cross section, and the CMS (untruncated) limits~\cite{CMS:rwa} are also shown for comparison.}
\label{fig:WIMPsNucleonXSection}
\end{figure}

%--- 90 % confidence level limits (to be compared to the 95% CL of the monojet exotics!!!)
%--- Only the simplified models have no theoretical uncertainties on the acceptance.


\section{Conclusion}
\label{sec:conclusion}

In this paper, we intended to disentangle the relationship between adversarial robustness and generalization by initially adopting the hypothesis that robustness and generalization are contradictory \cite{TsiprasARXIV2018,SuARXV2018}. By considering adversarial examples in the context of the low-dimensional, underlying data manifold, we formulated and experimentally confirmed four assumptions. First, we showed that regular adversarial examples indeed leave the manifold, as widely assumed in related work \cite{GilmerICLRWORK2018,TanayARXIV2016,IlyasARXIV2017,SamangoueiICLR2018,SchottARXIV2018}. Second, we demonstrated that adversarial examples can also be found on the manifold, so-called on-manifold adversarial examples; even if the manifold has to be approximated, \eg, using \VAEGANs \cite{LarsenICML2016,RoscaARXIV2017}. Third, we established that robustness against on-manifold adversarial examples is clearly related to generalization. Our proposed on-manifold adversarial training exploits this relationship to boost generalization using an approximate manifold, or known invariances. Fourth, we provided evidence that \red{robustness against regular, unconstrained adversarial examples and generalization are not necessarily contradicting goals}: for any arbitrary but fixed model, better generalization, \eg, through more training data, does not reduce robustness.


\chapter{Literature Study}
% put these two lines after every \chapter{} command
\vspace{-2em}
\minitoc

The implementation of FDIR on satellites have multiple complications with regards to the type of data generated by a satellite and the methodologies that can be implemented within the time and memory constraint of a cube-sat processor.

\section{Strategies for Satellite Constellations}
\cite{Castel2006}

\subsection{Individual Strategy}
Each satellite performs it's own FDIR without communication with other satellites. Consequently, the satellite has its own algorithms and knowledge of when a fault occurs within it's own system and is only required to communicate with the ground station.

\subsection{Centralised Strategies}
Centralised strategies focus on the FDIR of the entire constellation performed by a single entity. This can either be one of the satellites or the ground station.
\subsubsection{Master Centralised Strategy}

\subsubsection{Opportunistic Centralised Strategy}

\subsubsection{Global Centralised Strategy}

\subsection{Mixed Strategy}

\subsection{Distributed Strategies}
\subsubsection{Common Distributed Strategy}

\subsubsection{Individual Distributed Strategy}

\section{Anomaly Detection on Satellites}
Various methodologies have been tested on different component of satellites. Therefore a summary of these research articles are provided in this section.

\subsection{Analysis and Prediction of Satellite Anomalies}
\textcite{Wintoft}

\subsection{Agent-based algorithm for fault detection and recovery of gyroscope's drift in small satellite missions}
To ensure that the ADCS of satellites are autonomous every aspect of the control must be able to recover from faults. \textcite{carvajal2017agent} developed an algorithm to evaluate the control of a gyroscope and detect whether drifting exists. If drifting is detected another algorithm is deployed to ensure the recovery of the gyroscope drift by updating the error state vector.

\subsection{Multivariate Anomaly Detection in Discrete and Continuous Telemetry Signals Using a Sparse Decomposition in a Dictionary}

\cite{Pilastre2020}

\subsection{Fault isolation of reaction wheels onboard three-axis controlled in-orbit satellite using ensemble machine learning}
\cite{rahimi2020fault}

\subsection{Fault tolerant control for satellites with four reaction wheels}
\cite{jin2008fault}

\subsection{Innovative Fault Detection, Isolation and Recovery Strategies On-Board Spacecraft: State of the Art and Research Challenges}
\cite{wander2013innovative}

\subsection{Machine learning methods for spacecraft telemetry mining}
\cite{ibrahim2018machine}

\subsection{Machine learning techniques for satellite fault diagnosis}
\cite{ibrahim2020machine}

\section{Statistical Methods}
\subsection{Pearson Correlation}
Vectors of certain sensors are highly correlated. For instance the vector of the earth sensor is highly correlated since the magnitude of the vector remains more or less constant. To detect anomalies the correlation of vectors can be measured and with a specified threshold the correlation can be indicated as a anomaly or nor.

The squared Pearson correlation coefficient (SPCC) for vectors depicted as
\linebreak
\\
\centerline{$a = [a_1, a_2, \ldots, a_L]^T,$}
\linebreak
\centerline{$b = [b_1, b_2, \ldots, b_L]^T,$}
\\
is defined as \cite{benesty2009pearson}
\begin{equation}
	\rho^2 (a,b) = \frac{E^2 (a,b)}{E(a^Ta)E(b^Tb)}.
\end{equation}
The correlation coefficient is proven to be constraint as
\begin{equation}
	0 \leq \rho \leq 1,
\end{equation}
where $\rho = 1$ is perfect linear correlation.

\subsection{Variance}
Within a sequential data sample of the satellite, the variance of the variables should be within a given threshold if the satellite is in a stable condition. The variance of the data sample is defined as
\begin{equation}
	S^2 = \frac{\sum(x_i + \bar{x})^2}{n-1}
\end{equation}
where $x$ defines the variable within the dataset.

\subsection{Kalman-Filter}
The Kalman-filter application would require the state-space matrices to be provided in the log file.

\subsection{Multivariate Guassian Distribution}
The assumption that the error of our data is generated with a Guassian distribution with a specific mean, $\mu$, and variance, $\sigma^2$, provides the opportunity for using multi-variate Gaussian distribution to determine the probability of a data-sample within a dataset.
\begin{equation}
	\label{mean}
	\mu_j = \frac{1}{m} \sum_{i=1}^{m}x_j^{(i)}
\end{equation}

\begin{equation}
	\label{variance}
	\sigma_j^2 = \frac{1}{m} \sum_{i=1}^{m}(x_j^{(i)} - \mu_j)^2
\end{equation}

\begin{equation}
	\label{guassian distribution}
	p(x) = \prod_{j=1}^{n} \frac{1}{\sqrt{2\pi}\sigma_j}exp(-\frac{(x_j-\mu_j)^2}{2\sigma_j^2})
\end{equation}

For multi-variate Guassian distribution \cite{do2008multivariate}.

\begin{equation}
	\label{sum}
	\sum = \frac{1}{m}\sum_{i=1}^{m}(x^{(i)}-\mu)(x^{(i)}-\mu)^T
\end{equation}

\begin{equation}
	\label{multi-variate guassian distribution}
	p(x) = \frac{1}{(2\pi)^{\frac{n}{2}}{\lvert \sum \rvert}^\frac{1}{2}} exp(-\frac{1}{2}(x-\mu)^T{\sum}^{-1}(x-\mu))
\end{equation}

The Anomalies will be classified based on probabilities smaller than a given threshold $p(x) < \epsilon$.

\begin{algorithm}
	\SetKwInOut{Input}{Input}
	\SetKwInOut{Output}{Output}

	\SetKwData{Left}{left}
	\SetKwData{This}{this}
	\SetKwData{Up}{up}
	\SetKwFunction{Union}{Union}
	\SetKwFunction{FindCompress}{FindCompress}

	\Indm
	\Input{Data sample from satellite orbit.}
	\Output{Whether dataset contains anomaly.}
	\Indp
	\BlankLine

	Determine feature vectors $x_i$ \\
	Determine threshold probabilty, $\epsilon$ \\
	Calculate $\mu_j$ with Eq~\ref{mean} \\
	Calculate $\sigma_j$ with Eq~\ref{variance} \\
	Calculate $p(x)$ with Eq~\ref{guassian distribution} \\
	\If{$p(x) < \epsilon$}{Anomaly $= True$}
	\Else{Anomaly $= False$}

	\caption[Multi-variate Guassian Distribution]{Multi-variate Guassian Distribution Algorithm}
	\label{alg}
\end{algorithm}

\subsection{Kullback-Leibler Divergence}
The Kullback-Leibler divergence quantifies the difference between two probability density functions, denoted as $p(x)$ and $q(x)$ \cite{hershey2007approximating}. Satellites are systems that are predictable within a time-series. The divergence between two sequential data buffers from the satellite will have a very similar probability distribution. Therefore calculating the difference between two datasets can be used to detect an anomaly based on a given threshold.

The difference between the probability distributions from datasets, $a$ and $b$, in Figure~\ref{Guassian plot} cannot simply be calculated as the difference in the mean or the difference in the variance. To overcome this, the divergence between the two distributions can be calculated. Intuitively a point $x$ with a high probability in the dataset $a$ should have a high probability in the dataset $b$ if the two datasets have a small divergence.

\pgfmathdeclarefunction{gauss}{3}{%
	\pgfmathparse{1/(#3*sqrt(2*pi))*exp(-((#1-#2)^2)/(2*#3^2))}%
}
\begin{figure}[!h]
	\centering
	\textbf{Difference Between Probability Distributions}
	\begin{tikzpicture}
		\begin{axis}[
			no markers,
			domain=-3:6,
			samples=100,
			ymin=0,
			axis lines*=left,
			xlabel=$x$,
			every axis y label/.style={at=(current axis.above origin),anchor=south},
			every axis x label/.style={at=(current axis.right of origin),anchor=west},
			height=5cm,
			width=12cm,
			xtick=\empty,
			ytick=\empty,
			enlargelimits=false,
			clip=false,
			axis on top,
			grid = major,
			hide y axis
			]

			\addplot [very thick,cyan!50!black] {gauss(x, 3, 1)};

			\pgfmathsetmacro\valueA{gauss(3,3,1)}
			\draw [gray] (axis cs:3,0) -- (axis cs:3,\valueA);

			\node[below] at (axis cs:3, 0)  {$\mu_p$};

			\addplot [very thick,red!50!black] {gauss(x, 1.5, 1.5)};

			\pgfmathsetmacro\valueB{gauss(1.5,1.5,1.5)}
			\draw [gray] (axis cs:1.5,0) -- (axis cs:1.5,\valueB);

			\node[below] at (axis cs:1.5, 0)  {$\mu_q$};
		\end{axis}

	\end{tikzpicture}
	\caption{Guassian Distributions}
	\label{Guassian plot}
\end{figure}

The divergence can be expressed as

\begin{equation}
	KL(P\lvert\lvert Q) = \int p(x) \log \left( \frac{q(x)}{p(x)} \right)dx.
\end{equation}

\subsection{Canonical Correlation Analysis}
Due to the orbital nature of satellites there exist a correlation between various sensors. For instance the sun sensor, magnetometer and earth sensor are correlated based on the desired orientation and orbit of the satellite. This correlation might not be of linear nature, but with non-linear correlation methods such as kernel canonical correlation the correlation can be measured.

However, canonical correlation provides the measure of correlation between a multi-dimensional variable with another multi-dimensional variable. Although this seems profitable for satellite fault detection, it will only be applicable for each the comparison between individual sensors. This will indicate the non-linear correlation of the sun sensor with regards to the magnetometer. The problem however, according to \textcite{chen2017fault} is to, determine the appropriate threshold for which to classify a fault. \textcite{chen2017fault} proposed a method for determining the appropriate threshold on page 5, algorithm 1.
\cite{fukumizu2007statistical}
\cite{zhu2017quality}

Python - Pyrcca package

\subsubsection{K-means-based}
\subsubsection{Guassian Mixture Model}
\subsubsection{Just-In-Time-Learning}
\cite{chen2020just}

\section{Feature Extraction}
To
https://towardsdatascience.com/feature-extraction-techniques-d619b56e31be
\subsection{Prony's Method}
\subsection{Convolutional Networks}
\subsection{K-means Clustering}
K-clustering: Clustering multiple points with similar features.
\subsection{Principal Component Analysis}
\cite{choi2005fault}
\cite{ding2010application}
\subsection{Partial Least Square}
\subsection{Independent Component Analysis}
\subsection{Locally Linear Embedding}
\subsection{Linear Discriminant Analysis}
\subsection{Autoencoder}
\subsection{t-Distributed Stochastic Neighbor Embedding}
\subsection{Dynamic Mode Decomposition}
Dynamic mode Decomposition was developed in the fluids community by Professor Peter J. Schmid \cite{schmid2010dynamic}. Further adjustments and developments thereof has been done since the development in 2010.

Eigen values from smaller $\boldsymbol{A}$ while only computing that are equal to large A.
\subsection{Koopman Operator}

\section{Supervised Learning}
Supervised learning consists of models that are trained on labelled data. This is not a problem with simulation, but with the real data, it is a problem and to provide tests on the real data to label it must be proficient. If unsupervised learning and statistical methods are not sufficient in their accuracy, a method for labelling the real data must be provided.

\section{Physics-informed machine learning for sensor fault detection with flight test data}
To approximate the sensor of the next timestep a model
\begin{equation}
	x_{k+1} = Ax_k + By_k
\end{equation}
is used. DMD is used to predict matrix $\boldsymbol{A}$ and $\boldsymbol{B}$

From $\boldsymbol{A}$ and $\boldsymbol{B}$ the Kalman filter can be implemented to predict $x_{k+1}$
\begin{equation}
	\hat{x}_{k+1} = A\hat{x}_k + By_k + K(x_k - \hat{x}_k)
\end{equation}

Moving average of the innovation covariance
\begin{equation}
	V_k = \frac{1}{N} \sum_{i=k-N}^k (x_i - \hat{x}_i)(x_i - \hat{x}_i)^T
\end{equation}

\subsection{Random Forests}
\cite{Shi2006, Paul2018, Primartha2018}

\subsection{Long Short Term Memory}
Time-series data: LSTM or DLSTM

\subsection{Support Vector Machines}
Support Vector Machines

\subsection{Naive Bayes}
Naive Bayes

\subsection{K-nearest neighbours}
K-nearest neighbours

\subsection{Artificial Neural Networks}
Artificial Neural Networks

\section{Unsupervised Learning}
Density-based, distance, Clustering

\subsection{Isolation Forests}
This unsupervised learning methods is based on the principle of isolating data points by slicing the data with random conditions \cite{TonyLiu2008}. The data is randomly split into specified sample sizes with a randomly selected dimension and a randomly selected cut-off value. For each sample size the data must be split until each data point within the sample is isolated from all other data points. Training of a single tree is completed when all the data points are isolated and this training must be repeated for all the data samples, however many are predefined.

The distance measured from the first split the \emph{tree top} to the isolated data point is used to determine whether a data point is anomalous or not \cite{Hariri2021}. The logical reasoning for support of this algorithm is that data points which are non-anomalous will be more closely related and hence have more splits to separate the data points until isolation is achieved. Therefore, the distance from the tree top for non-anomalous data points will be longer than anomalous data points which will have a shorter distance from the tree top. Therefore non-anomalous data points are closer to the \emph{root}.

Figure~\ref{Figure-Isolation_Forest} demonstrates the splitting of the data points until isolated. Each split or \emph{branch} only splits the data into two groups. After training multiple trees, a single data point is "sent through the forest" and the distance from the tree top for each tree is calculated and the average of all the trees are used to calculated the average distance for the data point. Using a threshold for the distance, the data point is classified as anomalous or not.

\begin{figure}[h!tb]
	\centering

	\includegraphics[width=10cm]{fig/Isolation_Forests} % not necessary to give extension - now you can shift between compiling to ps or to pdf without any problems

	\caption[Isolation Forest]{Isolation Forests \cite{Chen2020}}
	\label{Figure-Isolation_Forest}
\end{figure}

The anomaly score is calculated with Eq~\ref{Eq-Isolation_anomaly}
\begin{equation}
s(x,n) = 2^{-E(h(x))/c(n)}
\label{Eq-Isolation_anomaly}
\end{equation}
where $E(h(x))$ is the average value of the distance measured from the tree top for a single data point in all the trees \cite{Hariri2021} and $n$ is the size of a data sample used to train a single tree. For the distance to be normalized, $c(n)$ --- the mean distance from the tree top in an unsuccessful search in a \emph{Binary Search Tree} (BST) --- is used and is calculated as
\begin{equation}
c(n) = 2H(n-1) - \frac{2(n-1)}{n}.
\label{Eq-normalizing_isolation}
\end{equation}
$H(i)$ in Eq~\ref{Eq-normalizing_isolation} is the harmonic number and is estimated with Euler's constant as
\begin{equation}
H(i) \approx ln(i) + 0.5772156649.
\label{Eq-H_i}
\end{equation}
Isolation Forests, however have multiple issues, since it splits data in rectangles as seen in Figure~\subref{Figure-ExtendedvsNormal_first}. This is due to the slicing algorithm selecting a feature, $x$ and a cut-off value, $v$. Consequently, the data is either split vertically or horizontally --- if seen as a two dimensional dataset. This split method is unable to categorise complex data structures. These issues however are addressed by \textcite{Hariri2021} and led to the \emph{Extended Isolation Forest} algorithm.

The extended isolation forest algorithm generalises the isolation forest algorithm by applying a slope to each slice. Data points are therefore divided into two groups depending on the "side" of the plane or slice as seen in Figure~\subref{Figure-ExtendedvsNormal_second}.

\begin{figure}[h!tb]
	\centering

	\subfloat[][Isolation Forest Slicing example]{
		\includegraphics[trim = 0 0 725 0, clip = true, width=7cm]{fig/Isolation_forest_slicing}
		\label{Figure-ExtendedvsNormal_first}
	}
	\quad
	\subfloat[][Extended Isolation Forest Slicing example]{
		\includegraphics[trim = 0 0 725 0, clip=true,width=7cm]{fig/Extended_Isolation_forest_slicing}
		\label{Figure-ExtendedvsNormal_second}
	}
	\caption[Slicing of Isolation Forest]{The slicing of Isolation Forest vs Extended Isolation Forest}
	\label{Figure-ExtendedvsNormal}
\end{figure}

It is evident that applying an angle of $0\degree$ to all the slices the general algorithm of the extended isolation forest produces the standard isolation forest algorithm where planes or slices are perpendicular to the axis of the randomly selected feature, $x$.

\subsection{Local Outlier Factor}
Most algorithms for anomaly detection are based on a metric which accounts for the entire dataset~\cite{breunig2000lof}. However, many anomalies are identifiable in relation to the local neighbourhood of data points and not the overall dataset. Therefore, \textcite{breunig2000lof} developed the local outlier factor \(LOF\) algorithm that provides a measure of a data point's "outlierness". This implies that a data point is not classified as an anomaly or not, but a local outlier factor is calculated to determine how much a data point is distantiated from it's $k$-nearest neighbours. This is clearly demonstrated in Figure~\ref{Figure-LOF_measure} where the data points which are clustered together have smaller LOF's than data points which are removed from the highly dense areas.

\begin{figure}[h!tb]
	\centering

	\includegraphics[width=10cm]{fig/LOF_measure} % not necessary to give extension - now you can shift between compiling to ps or to pdf without any problems

	\caption[Local outlier measure]{LOF measure}
	\label{Figure-LOF_measure}
\end{figure}

To calculate the LOF, the $k$-distance must be calculated and also the local reachability density \(lrd\). The $k$-distance, is the $k^{th}$ ranked $distance(o,p_i)$. Where $distance(o,p_i)$ is the distance between data point $o$ and any data point $p_i$, with $i \in N$, where $N$ is the number of data points within the dataset with a minimum value of $MinPts$. To reduce fluctuations in the $distance(o,p_i)$ the distance between $o$ and $p_i$ is replaced with
\begin{equation}
max \{distance(o,p_i), k\text{-distance}\}
\end{equation}
and will henceforth be referred to as the reachability distance~\cite{breunig2000lof}. The $lrd$ of a data point, $p$, is calculated as
\begin{equation}
lrd_{MinPts}(p) = 1/\left(\frac{\sum\limits_{o \in N_{MinPts}(p)}^{} reach\-dist_{MinPts}(p,o)}{|N_{MinPts}(p)|}\right)
\label{Eq-lrd}
\end{equation}
and denotes "the inverse of the average reachability distance based on the $MinPts$-nearest neighbours of the $p$" --- \textcite{breunig2000lof}. Eq~\ref{Eq-lrd} enables the calculation for the $LOF$ of point $p$ as shown in Eq~\ref{Eq-LOF}
\begin{equation}
LOF_{MinPts}(p) = \frac{\sum\limits_{o \in N_{MinPts}(p)}^{}\frac{lrd_{MinPts}(o)}{lrd_{MinPts}(p)}}{|N_{MinPts}(p)|}
\label{Eq-LOF}
\end{equation}
The rule of thumb for detecting an outlier is that when the LOF is larger than 1, then the point is considered an outlier with respect to its neighbourhood. This however is not fixed and the threshold can be changed depending on the application.
This method is aimed at producing a measure of the "outlierness" of a data point within a local neighbourhood and not for all the data points. This method will thus be implemented for the satellite anomaly detection, since it will detect anomalies within the two neighbourhoods produced by the eclipse during orbit. This method will also be able to detect measurements of earth sensors, sun sensors and magnetometers that drastically change from the previous orbital data. For example in Fig~\ref{Figure-Satellite_orbit} it is evident that the LOF will be comparatively larger for the red data points, which are anomalies, to the blue data points that are the normal orbit of the satellite.

\begin{figure}[h!tb]
\centering
\begin{tikzpicture}
\begin{axis}[title = Earth Sensor During Multiple Orbits]
\addplot3 table [x =Earth x, y = Earth y, z=Earth z, col sep=comma, only marks, scatter, blue, fill opacity=0.1, draw opacity = 0]{Data/3D_orbit.csv};
\addplot3 table [x =Earth_anomaly x, y = Earth_anomaly y, z=Earth_anomaly z, col sep=comma, only marks, scatter, red, fill opacity=0.1, draw opacity = 0]{Data/3D_orbit.csv};
\end{axis}
\label{Figure-Satellite_orbit}
\end{tikzpicture}
\end{figure}

\subsection{Kernel Adaptive Density-based}
Kernel adaptive density-based: Is an algorithm that uses the density factor of a data point relative to other data points to determine whether the data point is an outlier or not.

\subsection{Loda}
Loda: Is a fast and efficient anomaly detection algorithm that used histograms to evaluate data points to determine whether a data point is an outlier. Loda is an on-line method and not a batch method.

\subsection{Robust-kernel Density Estimation}
Robust-kernel density estimation

\section{Reinforcement Learning}
Active Anomaly detection with meta-policy (Meta-AAD) is a deep reinforcement learning approach that is based on the actor-critic model. The agent must query data points within the given dataset (where the queried point is the data top 1 data point). The query is given to a human

\section{Summary}