\chapter{EasyDD v2.0}
\label{c:easydd}

EasyDD can be found in \href{https://github.com/TarletonGroup/EasyDD}{\texttt{https://github.com/TarletonGroup/EasyDD}}.

As mentioned in \cref{s:objectives}, this project aims to integrate multidisciplinary skills for creating increasingly faithful recreations of reality in a user-friendly way. This chapter details the software engineering that moved us closer to this goal and yielded a much improved version of EasyDD.

\section{Bug fixes}
\label{s:bugs}

\subsection{Correct time-adaptive integrator}
\label{ss:integrator}

The first obvious hurdle to overcome when making more complex simulations possible was the unreasonably long computational time taken for a dislocation plasticity simulation to sufficiently advance past the elastic regime. A simple microcantilever bending simulation with a few Frank-Read sources would take a whole night to reach the plastic regime.

The unacceptable slowness prompted a closer examination of the adaptive-time Euler-trapezoid predictor-corrector solver described in algorithm 10.2 and equations (10.42, 10.45 and 10.46) in \cite[p.~214--216]{ddlab}, reproduced here by \cref{alg:trapezoid,eq:trapezoid},
\begin{align}\label{eq:trapezoid}
  \vec{r}_i^{\rvar{P}}(t + \Delta t) & = \vec{r}_i(t) + \vec{g}_i\left(\left\{ \vec{r}_i(t) \right\}\right) \Delta t\,,                                                                     \\
  \vec{r}_i(t + \Delta t)            & = \vec{r}_i + \dfrac{\vec{g}_i(\left\{\vec{r}_i(t)\right\}) + \vec{g}_i\left(\left\{\vec{r}_i^{\rvar{P}}(t + \Delta t)\right\}\right)}{2}\Delta t\,, \\
  \vec{v}_i                          & \coloneqq \dfrac{\rvar{d}\vec{r}_i}{\rvar{d}t} = \vec{g}_i\left(\left\{\vec{r}_i\right\}\right)\,,
\end{align}
where $\vec{r}_i$ are nodal coordinates, $\vec{v}_i$ are nodal velocities, $\vec{g}_i$ is the function that uses the mobility model and nodal forces to compute the nodal velocities, and the superscript $\rvar{P}$ denotes the predictor. This solver is fast and accurate for a small enough timestep $\Delta t$. The algorithm contains two free parameters, $\Delta t_{\rvar{max}}$ and $\epsilon$, which respectively denote the maximum allowed timestep and accuracy.
\begin{algorithm}
  \caption{Adaptive Euler-trapezoid predictor-corrector algorithm.}
  \label{alg:trapezoid}
  \begin{enumerate}
    \item Initialise time step $\Delta t = \Delta t_{\rvar{max}}$.
    \item $\Delta t_0 = \Delta t$.
    \item Compute $\vec{r}_i^{\rvar{P}}(t + \Delta t)$ and corrector $\vec{r}_i(t + \Delta t)$ from \cref{eq:trapezoid}.
    \item If $\max_i\left(\lVert \vec{r}_i^{\rvar{P}}(t + \Delta t) - \vec{r}_i(t + \Delta t)\rVert\right) > \epsilon$, reduce time step $\Delta t = \Delta t / 2$ and go to 3.
    \item $t = t + \Delta t$.
    \item If $\Delta t = \Delta t_0$, increase time step to $\Delta t = \min(1.2 \Delta t, \Delta t_{\rvar{max}})$.
    \item Return to 2, unless total number of cycles is reached.
  \end{enumerate}
\end{algorithm}

The original, supposed implementation of this is found in commit \href{https://github.com/TarletonGroup/EasyDD/blob/780a6c41b35687b443d3241674af7393d2140639/int_trapezoid.m}{\texttt{65907b0}} under the name \texttt{int\_trapezoid.m}.

However, this was not the case. There were a number of problems with the implementation. Firstly, it only used $\Delta t_\rvar{max}$ on the first timestep. It also only increased the timestep once every timestep. Ideally, an adaptive time algorithm should increase the time step to the maximum that is still under the error criteria. The way the old version of the integrator worked meant that the time step would take multiple steps to ramp back up after decreasing. Take for example, a case where the time step had to decrease once for two consecutive steps, $\Delta t = \Delta t_0/4$ would have taken $\lceil\log_{1.2}(4)\rceil = 8$ subsequent time steps to surpass what it was originally, $\Delta t_0$. In general, $N$ time step halvings, would require $\log_b(2^N)$ steps get back to the original value, using $b$ as the time-step increase factor. Since only one increase in time step was allowed per simulation step, simulations were taking many more timesteps than necessary.

An improved algorithm is described in \cref{alg:trapezoid_improved}. This algorithm maximises the time step given absolute and relative error criteria, ($\epsilon_{\rvar{max}}$, $\eta_{\rvar{max}}$), absolute maximal and minimal timesteps, ($\Delta t_{\rvar{max}}$, $\Delta t_{\rvar{min}}$) and number of iterations, $\rvar{iter}_{\rvar{max}}$. It also uses a modulated timestep increase by how far under the absolute accuracy criteria we are. It also fixes a bug that made the error calculation only account for the largest magnitude of a single component of the 3D vectors, rather than the largest norm. The new implementation can be found in the most current version of \href{https://github.com/TarletonGroup/EasyDD/blob/master/src/int_trapezoid.m}{\texttt{int\_trapezoid.m}}.
\begin{algorithm}
  \caption{Improved adaptive timestep algorithm.}
  \label{alg:trapezoid_improved}
  \begin{algorithmic}
    \State $\rvar{Convergent} = \rvar{false}$, $\Delta t_{\rvar{valid}} = 0$, $\rvar{iter} = 0$, $\rvar{flag} = \rvar{false}$
    \While{$\rvar{Convergent} == \rvar{false}$}
    \State Compute $\vec{r}_i^{\rvar{P}}(t + \Delta t)$ and corrector $\vec{r}_i(t + \Delta t)$ from \cref{eq:trapezoid}.
    \State $\Delta \vec{r}_i = \vec{r}_i(t + \Delta t) - \vec{r}_i^{\rvar{P}}(t + \Delta t)$
    \State $\vec{\overline{r}}_i = \dfrac{\vec{g}_i(\left\{\vec{r}_j(t)\right\}) + \vec{g}_i\left(\left\{\vec{r}_i^{\rvar{P}}(t + \Delta t)\right\}\right)}{2}\Delta t$
    \State $\epsilon = \max_i\left(\lVert \Delta \vec{r}_i \rVert \right)$
    \State $\eta = \max_i\left(\lVert \Delta \vec{r}_i - \vec{\overline{r}}_i \rVert\right)$
    \If{$\epsilon < \epsilon_{\rvar{max}} $ and $\eta < \eta_{\rvar{max}}$}
    \State $\Delta t_{\textrm{valid}} = \Delta t$
    \State $\gamma = 1.2\left(\dfrac{1}{\left[1 + (1.2^{20} - 1) (\epsilon / \epsilon_{\textrm{max}})\right]}\right)^{1/20}$
    \State $\rvar{flag} = \rvar{true}$
    \State $\rvar{iter} = \rvar{iter} + 1$
    \State $\Delta t = \max\left(\gamma \Delta t,\, \Delta t_\rvar{max}\right)$
    \ElsIf{$\rvar{flag} == \rvar{true}$}
    \State $\Delta t = \Delta t_{\rvar{valid}}$
    \State $\rvar{iter} = \rvar{iter}_\rvar{max}$
    \ElsIf{$\Delta t < \Delta t_\rvar{min}$}
    \State $\Delta t = \Delta t / 2$
    \State $\rvar{iter} = \rvar{iter} + 1$
    \ElsIf{$\Delta t == \Delta t_\rvar{min}$}
    \State $\rvar{iter} = \rvar{iter}_\rvar{max} + 1$
    \Else
    \State $\Delta t = \Delta t / 2$
    \EndIf
    \If{$\rvar{iter} > \rvar{iter}_{\rvar{max}}$ or $\Delta t == \Delta t_{\rvar{max}}$}
    \State $\rvar{Convergent} = \rvar{true}$
    \EndIf
    \EndWhile
    \State $t = t + \Delta t$
    \State Proceed with the rest of the simulation.
  \end{algorithmic}
\end{algorithm}

\Cref{alg:trapezoid_improved} is more computationally expensive per iteration, but can increase the timestep more rapidly, optimally and is more robust than \cref{alg:trapezoid}. The original algorithm also wasn't correctly implemented. Together, these changes let the new version of the software move through the elastic regime of our simulations orders of magnitude faster than before. Simulations that took hours to days to reach the plastic regime, started doing so in minutes to hours. It also narrowed the gap between different loading conditions, as simulations were now more capable of effectively adjusting the timestep to more efficiently match the error tolerances. However, as discussed in \cref{c:simulations}, there is still a danger of overloading simulations. Fixing the integrator was the first major bottleneck we fixed, allowing us to identify and resolve downstream issues that were previously unkown.

\subsection{Matrix conditioning}
\label{ss:matrix}

A common problem with discrete dislocation dynamics is the tendancy for some networks to have a large degree of variation in the speed of dislocation nodes \cite{bertin2019gpu,ddlab,arsenlis2007enabling}. This is a consequence of the fact that nodes are discretised representations of dislocation lines. As such, nodal motion must be constrained to the motion of dislocations, i.e. nodal movement must be consistent with how a dislocation glides, climbs and cross-slips, accounting for dislocation character and slip system. Dislocation motion can be assumed to follow a linear, anisotropic, viscous drag model \cite{ddlab},
\begin{align}\label{eq:dragCoef}
  \vec{F}_\textrm{drag}(\vec{x}) & = -\mathcal{B}(\vec{\xi}(\vec{x})) \cdot \vec{v}(\vec{x})\,,
\end{align}
where $\mathcal{B}$ is a tensor constructed from the anisotropic drag that a dislocation node at $\vec{x}$, experiences as a result of its constrained velocity $\vec{v}(\vec{x})$ due to being part of a discritesed line segment with direction $\vec{\xi}(\vec{x})$. This means a node can experience very different resitances to moving along its allowed movement vectors.

Different mobility laws can have different parametrisations for their permitted movement. A problem that often plagues traditional mobility laws\footnote{Bruce Bromage, of \cite{bromage2018calculating} has developed a new BCC mobility law, \href{https://github.com/TarletonGroup/EasyDD/blob/master/src/mobbcc_bb1b.m}{\texttt{mobbcc1\_bb.m}} which can be found in the most current version of EasyDD. It solves many of the issues with traditional laws and is our new go-to for BCC materials. Among the things it fixes are: unreasonable cross-slip, pencil glide, and the problematic line-direction parametrisation. The drag matrix, $\mtx{B}$ in \cref{eq:nodeVel}, can still be singular however, so this law also includes the scale-averaged Levenberg-Marquardt regularisation discussed in this chapter.} is the movement of a node along a line segment itself. \Cref{f:lineMovement} shows how a node moving along a line segment does not change its topology, and should therefore have no effect on the energy of the network. Consequently, nodal movement along the line direction must not contribute to the total drag, so the line drag component of the stress tensor, $\mathcal{B}$, should tend to zero. On the other end of the spectrum we have climb, which is orders of magnitude slower than glide, so that component of the drag tensor should be much larger than the rest. However, these requirements can make the drag matrix, $\mtx{B}$ in \cref{eq:nodeVel}, singular. Keeping the matrix invertible while simultaneously meeting the physical requirements is a careful balancing act. Moreover, the low drag along the line direction often causes nodal speeds along these vectors to be much higher than other directions, often limiting the integration time step as a result.
\begin{figure}
  \centering
  \includegraphics[width=0.5\linewidth]{lineDirMov.pdf}
  \caption{Node movement along dislocation line should have no drag contribution.}
  \label{f:lineMovement}
\end{figure}

In the overdamped regime, $\dfrac{\rvar{d}\vec{v}}{\rvar{d}t}(\vec{x}) \to \vec{0}$, known driving forces (external forces + segment-segment interaction force + self-force) and the unkown drag force sum to give,
\begin{align}\label{eq:drivePlusDrag}
  \vec{F}_\rvar{drag}(\vec{x}) + \vec{F}_\rvar{drive}(\vec{x}) = \vec{0}\,.
\end{align}
Substituting \cref{eq:dragCoef} into \cref{eq:drivePlusDrag} and rearranging gives,
\begin{align}\label{eq:nodeMob}
  \mathcal{B}(\vec{\xi}(\vec{x})) \cdot \vec{v}(\vec{x}) & = \vec{F}_\rvar{drive}(\vec{x})\,.
\end{align}
Strictly speaking, solving \cref{eq:nodeMob} requires global knowledge of the network. Fortunately, the discretisation makes it possible to distribute $\mathcal{B}$ along a line segment $i-j$,
\begin{align}\label{eq:dragMatrix}
  \mtx{B}_{ij} & = \oint_{C} N_i(\vec{x}) \mathcal{B}(\vec{\xi}(\vec{x})) N_j(\vec{x}) \rvar{d}L\,,
\end{align}
where $C$ is the whole network, $N_i$ and $N_j$ are simply shape functions that interpolate quantities given the relative position of a node to a segment, and $\rvar{d}L$ is the infinitesimal line segment. Thus giving individual expressions for node $i$ connected to node $j$. Using this discretisation and assuming the nodes connected to $i$ are subject to similar conditions, \cref{eq:nodeMob} can be broken up into an approximate expression for the velocity of a single node,
\begin{align}\label{eq:nodeVel}
  \vec{v}_i \approx \left(\sum\limits_j \mtx{B}_{ij}\right)^{-1} \cdot \vec{f}_i,\,
\end{align}
where $i$ is the node in question, $j$ are the nodes it shares segments with, and $\vec{f}_i$ is the local driving force on the node. From now on, we will drop the Einstein notation and refer to the local frame as $\vec{v} = \mtx{B}^{-1} \vec{f}$.

The immediate implication of the orders of magnitude difference in the drag coefficients used to construct $\mathcal{B}$ (and therefore $\mtx{B}$) means the condition number of $\mtx{B}$ has the potential to be very large. Given that $\mtx{B}$ is symmetric and therefore normal, its condition number is given by,
\begin{align}\label{eq:conditionNumb}
  \kappa(\mtx{B}) = \dfrac{\lvert \lambda_\rvar{max} \rvert}{\lvert \lambda_\rvar{min} \rvert} \,,
\end{align}
where $\lambda_\rvar{min}$, $\lambda_\rvar{max}$ are the minumum and maximum eigenvalues.

The condition number of a matrix codifies how much collinearity exists in its basis. The larger the condition number, the greater its basis' collinearity, and the more sensitive the sytem is to small perturbations. Such systems are said to be ill-conditioned. As a general rule, a condition number, $\kappa(\mtx{A}) = 10^k$, represents a loss of up to $k$ digits of precision on top of what is already lost from arithmetic methods under limited precision \cite{cheney2012numerical}.

True to Murphy's law, $\mtx{B}$, is frequently ill-conditioned---even in the simplest simulations. The way these cases were handled relied on the integrator reducing the timestep enough to sidestep the issue. We call this ``strongly coupled'' behaviour. Strong coupling between functionally independent functions or subroutines is a \emph{very} bad practice in software engineering.

The ``buginess'' of a function $\mathcal{X}$ with a set of $\mathbb{X}$ bugs whose phase space is defined by $\chi(x)$, can be mathematically expressed as, $\forall x \in \mathbb{X}~ \chi(x)$. That is, for all inputs, $x$, some process, $\chi(x)$, produces a set of bugs $\mathbb{X}$. In other words, how buggy a function is, depends on the severity and frequency with which the process, $\chi(x)$, produces bugs, $\mathbb{X}$, for different inputs.

Bugs that evade tests and make it into production code are often either low impact with broad phase spaces, or high impact with narrow phase spaces. The way by which both types of bugs commonly make it into production code differs. Usually, low impact, broad phase space bugs make it into production due to a lack of proper unit testing on ``obvious'' or ``trivial'' cases. High impact, narrow phase space bugs are usually down to rare corner cases such as catastrophic cancellation, loss of precision, division by zero, memory over or underflows, memory leaks, silent errors/crashes due to poor API (Application Programming Interface) design, etc. Both types of bugs can be avoided with a combination of proper software design, and testing protocols. Unfortunately, mitigation strategies go out the window when functions are strongly coupled. We want functions to be independent variables in our process. Just like in probability, phase spaces are multiplicative, and we want their products to be linearly independent. By having strongly coupled functions we make them non-linear, as one function depends on the other to solve its problems.

The way our old mobility functions dealt with ill-conditioned $\mtx{B}$, relied on the integrator picking up the slack. There are several flaws with the approach found in \cref{alg:bTotalOld}, an example of a mobility function with this problem can be found under commit \href{https://github.com/TarletonGroup/EasyDD/blob/65907b022d1fe408fc1b2e5c5ca2bd1797ccae04/mobbcc1.m#L68}{\texttt{65907b0}} under the name \texttt{mobbcc1.m}. Every mobility law in that commit has this issue.
\begin{algorithm}
  \caption[Bad way of avoiding drag matrix inversion singularity.]{Avoiding the inversion of a singular matrix by making $\mtx{B}$ extremely wrong and hoping the integrator error bounds pick it up and decrease the timestep.}
  \label{alg:bTotalOld}
  \begin{algorithmic}
    \State Diagonalise via eigenvalue decomposition, $\mtx{B} = \mtx{P} \mtx{D} \mtx{P}^{-1} \to \mtx{D} = \mtx{P}^{-1} \mtx{B} \mtx{P}$.
    \State{\Comment{$\lambda_{\rvar{max}}$ and $\lambda_{\rvar{min}}$ are the maximum and minimum eigenvalues respectively.}}
    \State $\kappa(\mtx{B}) = \lvert \lambda_{\rvar{max}} \rvert/ \lvert \lambda_{\rvar{min}} \rvert$
    \If{$\kappa(\mtx{B}) < \kappa_{\rvar{min}}$}
    \State{\Comment{Normalise $\mtx{D}$ and $\vec{f}$ to the maximum eigenvalue.}}
    \State $\mtx{D} = \mtx{D} / \lambda_{\rvar{max}}$
    \State $\vec{f} = \vec{f} / \lambda_{\rvar{max}}$
    \State{\Comment{Invert $\mtx{D}$ by inverting each non-maximal eigenvalue.}}
    \ForAll{$(\lambda_k \in \mtx{D}) \neq \lambda_{\rvar{max}}$}
    \State{\Comment{$\lambda_\rvar{crit} \to 0$ is some arbitrary threshold for the minimum value of $\lambda_k$.}}
    \If{$\lambda_k > \lambda_{\rvar{crit}}$}
    \State $\lambda_k = 1/\lambda_k$
    \Else
    \State{\Comment{The inverse of 0 is not 0.}}
    \State $\lambda_k = 0$
    \EndIf
    \EndFor
    \State{\Comment{$\mtx{D}$ is already inverted.}}
    \State $\vec{v} = \mtx{P} \mtx{D} \mtx{P}^{-1} \vec{f}$
    \Else
    \State $\vec{v} = \mtx{B}^{-1} \vec{f}$
    \EndIf
  \end{algorithmic}
\end{algorithm}
There are various problems with \cref{alg:bTotalOld}.

Formost of which, is assuming that an ill-conditioned matrix can be fixed by diagonalising and normalising. This is simply not true, it can help with numerical stability in some algorithms for solving linear systems, but it may not be enough for some systems. Perturbing the smallest eigenvalue in a direction such that $\kappa$ improves is a good way to renormalise a system without adding much error. However, perturbations of this type are often not enough when the condition number is too large. Furthermore, one wants to keep the overall behaviour of the system mostly intact, i.e. we want to maintain the relative sizes of the eigenvalues with respect to one another. If other eigenvalues are close in magnitude to the smallest one, the dynamics of the system can very easily change, even a very small perturbation may have undesireable consequences. Not to mention the fact that $\kappa$ may still be too large. In a system as stiff and dynamic as discrete dislocation dynamics, this approach is not robust enough.

The assumption that the case where $\lambda_k < \lambda_{\rvar{crit}}$ for non-maximal eigenvalues is so rare that it won't have much of an effect. Unfortunately, it's common in junctions where nodal mobility is limited, as well as in volumes with large stress gradients. Although to be fair, these are also a consequence of the assumption of locality in \cref{eq:nodeVel}. Moreover, when every non-maximal eigenvalue is smaller than $\lambda_{\rvar{crit}}$, the error is much greater. For every eigenvalue under the threshold of $\lambda_{\rvar{crit}} = 10^{-k}$, the upper bound of the error is $k$ digits in the corresponding mobility component.

Finally, assuming the integrator will not allow obviously wrong solutions because it will fail the error check and reduce the time step is not something to hang our hats on. Erroneous velocities only get caught within the integrator, not when nodal mobility is calculated outside the time integration, which happens in every topological operation where a new node is spawned. Spawning a new node with extremely incorrect velocities can prove to be quite a serious problem, especially if this happens in a region with a high dislocation density.

The regularisation of ill-conditioned systems is an active field of research \cite{regularisation1,regularisation2,regularisation3}. However, they require some knowledge of the matrix structure. Regularlisation is most often applied to high-rank matrices because poorly conditioned, low-rank matrices are often very sensitive to perturbations. In this case, $\mtx{B}$ is positive definite of rank equal to the number of orthogonal crystallographic directions, so it is always of low rank.

One of the most effective regularisations for low rank matrices is dampening the inversion by adding a small value, $c$---called the Levenberg-Marquardt coefficient---to a diagonal element of the matrix. This is roughly equivalent to perturbing the smallest eigenvalue as previously mentioned. If this factor is added to the diagonal elements of $\mtx{B}$, the inversion can be sufficiently dampened, whilst keeping the main contribution from each basis vector roughly equal in relation to each other. This amounts to adding a small multiple of the identity matrix, $\mtx{I}$, to an ill-conditioned matrix. For a more accurate solution, one can iteratively refine $c$ to find the smallest value that keeps the matrix inversion from blowing up. A good heuristic for the value of the Levenberg-Marquardt coefficient is,
\begin{align}\label{eq:marquardt}
  c = \max\left(\lvert\mtx{A}\rvert\right) \sqrt{\epsilon}\,,
\end{align}
where $\epsilon$ is machine precision and $\lvert \mtx{A}\rvert$ denotes the absolute values of matrix $\mtx{A}$. This ensures the perturbation is scaled to a number that will not underflow during arithmetic operations. Simply put, it ensures the perturbation is small but does not disappear during matrix inversion.

It is worth noting that this renormalisation is a source of error. Adding noise to a system---even if small and along the diagonal---moves the system away from its true behaviour. However, the degree to which it is wrong is many orders of magnitude smaller than what is done in \cref{alg:bTotalOld}. Indeed, \cref{alg:bTotalNew} uses $\pm c $ and both perturbed solutions are averaged to give a much more accurate and simpler solution even under the traditionally problematic scenarios mentioned above. This new algorithm is now present in the mobility laws we use, such as the most current version of \href{https://github.com/TarletonGroup/EasyDD/blob/372984499dd60136fc7badabd6cee192058d55d9/src/mobbcc_bb1b.m#L199}{\texttt{mobbcc\_bb.m}}.
\begin{algorithm}
  \caption[Dampening the drag matrix inversion singularity.]{Improved regularisation of $\mtx{B}$ by way of perturbing the diagonal.}
  \label{alg:bTotalNew}
  \begin{algorithmic}
    \If{$\lVert \mtx{B} \rVert < \epsilon$}
    \State $\vec{v} = \vec{0}$
    \ElsIf{$\lvert \lambda_{\rvar{max}} \rvert/ \lvert \lambda_{\rvar{min}} \rvert > 1/\epsilon$}
    \State $\mtx{B}_+ = \mtx{B} + \sqrt{\epsilon} \max(\mtx{\lvert B\rvert}) \mtx{I}$
    \State $\mtx{B}_- = \mtx{B} - \sqrt{\epsilon} \max(\mtx{\lvert B\rvert}) \mtx{I}$
    \State $\vec{v}_+ = \mtx{B}_+^{-1} \vec{f}$
    \State $\vec{v}_- = \mtx{B}_-^{-1} \vec{f}$
    \State $\vec{v} = (\vec{v}_- + \vec{v}_+)/2$
    \Else
    \State $\vec{v} = \mtx{B}^{-1} \vec{f}$
    \EndIf
  \end{algorithmic}
\end{algorithm}

This change enabled other team members, namely Haiyang Yu and Fengxian Liu to get further with their simulations. Haiyang's nanoindentation simulations have particularly complex dislocation structures with a substantial amount of junction formation \cite{YU2018,yu2020simulating,yu2019influence}; while Fengxian's have large localised stress gradients as a result of inclusions \cite{fengxian,fengxian2}. Both require nodal velocities to be very accurately calculated, else the dislocations move erratically, readily cross slip, and cause massive slowdowns as simulations advance.

\section{Research software engineering}
\label{s:rse}

Many scientists and researchers spare no thought for good software. However, with the presently ongoing SARS-CoV 2 pandemic, the importance of research software has been made clear. Modelling has played a large role in informing government policy in the UK and the world \cite{covidScotland,covidUK1,covidUK2}. With modelling thrust into the spotlight, and the public release of the Imperial College modelling code, which was used to inform so much policy \cite{covidUK2}, \href{https://github.com/mrc-ide/covid-sim}{\texttt{https://github.com/mrc-ide/covid-sim}}, criticism was levied at the lack of care taken to ensure the correctness of the code \cite{natureModelCritique}.

In experimental research, a lot of care is taken to ensure the methodoogy used is reproducible, robust, and sound. There are standards, both external and internal, which ensure the quality of the results. But even with these measures in place, there is already a common issue with reproducibility in science \cite{mede2020replication,randall2018irreproducibility,bolli2015reflections}. In the modern age, software is ubiquitous in the obtention, analysis and communication of scientific research. Anyone hoping to do science will interface with software at some point. This software should therefore be as high quality as possible, \emph{especially} if the science is modelling-based.

Along this vein, one of the primary goals of this project was to develop a competent codebase that non-experts can take advantage of with minimal preparation. There were multiple sub-objectives that help us achieve this.
\begin{enumerate}
  \item Provide a generic framework upon which the software can be modularly expanded with minimal change to the source: prevents increasingly divergent and incompatible source code between researchers.
  \item Provide an easy way to autocomplete inputs with sensible default parameters: prevents the software from crashing when---inevitably---one of the required arguments for a function is undefined.
  \item Improve readability and organisation: makes it easy to identify, fix bugs and know what the code does.
  \item Compartmentalise variables: makes the code more generic and increases usability and readability by reducing the number of function arguments.
  \item Optimise memory and computation: improves computational speed and reduces resource use.
\end{enumerate}

Akin to the Ship of Theseus, it is hard to tell whether the new version of the code is the same code as when this project began. While \href{https://github.com/TarletonGroup/EasyDD}{\texttt{EasyDD v2.0}} is ``complete'', the process of improving the code is ongoing. The amount of initial technical debt since its inception as the infinite-domain discrete dislocation dynamics code, \href{http://micro.stanford.edu/wiki/Main_Page}{\texttt{DDLab}} \cite{ddlab}, plus what it accrued as the Tarleton Group expanded its functionality and turned it into \href{https://github.com/TarletonGroup/EasyDD/tree/65907b022d1fe408fc1b2e5c5ca2bd1797ccae04}{\texttt{EasyDD}}, is quite large and needs to be carefully chipped away so as not to introduce regressions. However, while the quality and capabilities of the code can be greatly improved, there is only so much that can be done without fundamentally redesigning it completely. This is in part the subject of \cref{c:future}.

Other team members have made significant contributions to the release of \href{https://github.com/TarletonGroup/EasyDD}{\texttt{EasyDD v2.0}}. Some of which remain to be added, but should slowly make their way in. Given the code is the amalgam of the research group's work, we will credit other members' contributions where relevant but will leave the details for them to explain.

As this is actively developed research software, the work described herein will continue past the end of this project, for there are still many improvements to be made to the old codebase on top of whatever functionality is added by future researchers.

\subsection{Organisation}

An oft-overlooked aspect of code useability is its organisation. However, as a codebase grows, it becomes more important for code to be properly organised and source controlled. This makes it easier to browse, understand, and provides a crucial safey blanket against things going wrong.

\subsubsection{Source control}

Source control is a crucial aspect of industrial software development. It is an industry standard and is the very first thing to set up when starting a software development project. Academia is \emph{very} far behind in this regard. Source control software can perform a wide variety of tasks but among the most important are:
\begin{enumerate}
  \item provide a history of restore points in case things break;
  \item keep a history of changes, who and when they made them;
  \item lets developers and users make changes independently and request their changes be added to the main code as patches or new versions;
  \item detect conflicts between versions to be merged; and
  \item a robust versioning system integrated to the history of changes.
\end{enumerate}

With regard to reproducibility, source control is a crucial tool. In both \cref{ss:integrator,ss:matrix} we took advantage of this to provide a specific commit and a file name. Source control can be used in local and remote services via private LAN (Local Area Network), as well as webhosting services such as Gitlab, Bitbucket, or GitHub as in our case. Webhosted services are a popular choice because they simplify and enhance the user and developer experience. For example, the visibility of code can be easily set to:
\begin{enumerate}
  \item private, where only those explicitly given access have read and write access; or
  \item public, where anyone can view the code and propose changes which can then be reviewed and either accepted or rejected by those with write access.
\end{enumerate}
They also simplify the use of addons and webhooks that can perform myriad tasks like:
\begin{enumerate}
  \item automated remote testing on different platforms;
  \item code coverage reports (what lines and how many times each of them was hit during automated testing);
  \item automated documentation generation;
  \item automated merge checks; and even
  \item automated encryption/installation key generation.
\end{enumerate}

LAN-hosted version control software can make use of everything webhosted implementations can\footnote{Although man quality of life improvements such as GUIs (Graphical User Interfaces) and code editor integration might be unaccessible.}. Indeed, LAN-hosting is used for things pertaining to defense and national security, but the use of webhosted services is an easy choice for academia, industry, and private users. In fact, the digital version of this document is found in \href{https://github.com/dcelisgarza/DPhil_Thesis}{\texttt{https://github.com/dcelisgarza/DPhil\_Thesis}}, among many other repositories from recreational and past work. Moreover, whenever this document references a specific file, the digital version directly links to the relevant information (including relevant lines).

Publically available, webhosted source control is a boon for open, reproducible science. It can be used to provide a snapshot of the specific version of the code and data used to generate the results of a manuscript when submitting to a journal. Such platforms can be used to generate funding and copyright, and even enable independendant publishing through the ISBN registry and crowd-sourced peer-review. Through the use of tools such as Jupyter, one can write interactive publications, dubbed ``notebooks''. These notebooks give the power of peer-review back to the community, as well as being great tools for science communication, teaching, and even interactive techincal talks.

In summary, source control is the solution to multiple problems that inhibit scientific progress. On one hand, it provides a safety blanket with many bells and whistles that routinely saves researchers and companies time, money, effort, and heartache. On the other, it democratises scientific research by sidestepping the politics, costs, and ransoming of \emph{publically} funded research that is, at its core, for the benefit of humanity.

\subsubsection{Folder structure}

An ongoing task is improving the folder structure, which in the past was almost non-existant. Having a default place to place inputs, outputs and source code is of great import if a code is to be used by non-experts. It is trivial to do a large quality of life improvement for developers and users.

\subsection{Modularisation}

Modern software practices usually make heavy use of some form of variable encapsulation and code modulirisation to make code safer, simpler to expand, and easier to use. This enables users and developers to take advantage of a set of toolboxes that can be chosen according to their needs without having to directly modify the source code, this concept is called ``modularisation''.

\subsubsection{Encapsulation}\label{ss:encapsulation}

The act of keeping variables neatly stored in a way that fits a purpose is called encapsulation. By encapsulating variables, we greatly improve the usability of software by increasing readability, reducing the risk of human error by reducing the number of things to keep track of, and---if done properly---can even improve performance. How we go about encapsulating code is tightly bound to the design philosophy and programming paradigm of a software product, and the capabilities of the language it is written in. Common programming paradigms are outlined below.
\begin{itemize}
  \item Object-Oriented Programming (OOP): where variables are viewed as objects upon which the logic acts. Functions can take these objects and operate on them. This paradigm keeps related variables as part of a single entity and reduces the number of things users and developers have to keep track of, thus reducing the chance of implementation and user error. There are many advanced concepts and pitfalls in OOP that unecessarily increase complexity, so it is best to apply the KISS (keep it simple, stupid) approach for best results.
  \item Data-Oriven Programming (DDP): where the data dictates the structure of the programme. It can be similar to OOP in some ways, where the object is made to fit the data, not the data made to fit the object. It often leads to more performant code and avoids many of the issues that arise from OOP because it forces developers to really think about how to manage the data.
  \item Functional programming: where everything is a function. There is no state, but also no side effects. This is the strongest form of encapsulation but also the most inflexible.
  \item Procedural programming: this is the most ancient form of programming (that makes use of functions) where there are only functions and variables. It is the form most scientific software packages take, and it is the most difficult to work with both, as a developer and as a user. Even modern computers which can make use of branch prediction and deletion, cache pre-fetching, and compiler optimisations can be hindered by this approach.
\end{itemize}

Procedural programming is what gives us ``spagghetti'' code\footnote{Code with many branches and disparate procedures without a clear purpose. It is difficult to work with as a user and developer. It also tends to interfere with branch prediction and deletion, as well as many other compiler optimisations.}. Unfortunately, education in programming or software engineering in science is sorely lacking. As a result, a large amount of scientific software tends to be written in this way (particularly legacy software). Thus, we strive to separate ourselves from it as much as possible.

Functional programming is very useful where safety and fault-tolerance is a concern. It sees a lot of use in telecommunications, social media, cryptography, finance \cite{haskell,functionalProg}, but usually as part of a solution. It is however, impractical for most applications.

Object-oriented programming can be very useful if used properly. But it has the potential to unecessarily increase complexity and reduce performance. Fortunately, \mintinline{matlab}{MATLAB} does not truly have object-oriented capabilities, so it forces either a procedural or data-driven approach. This often leads to software being written in a procedural manner.

However, we can take a data driven approach and leverage \mintinline{matlab}{struct} to simplify the process. These are more of an ordered dictionary (Hash table), than a true object, but they are what we have available. Converting the code to use them is an ongoing process, but they have so far allowed us to make use of generic functions for different mobility functions, loading conditions, and various auxiliary variables. All of which have a simplifying effect on the code without sacrifing performance. While this project pioneered this task, Daniel Hortelano-Roig has been doing much in the way of increasing its scope. His changes already greatly simplify the code, and will slowly be ported over to the main branch.

\subsubsection{Generic functions}\label{ss:genFun}

Generic functions are implementation-agnostic functions that can be given by the user as any other variable. \mintinline{matlab}{MATLAB} has an old method that enables generic programming using \mintinline{matlab}{feval()}, where the first argument is the name of the file whose function is being called, and subsequent arguments are passed on to the function. However, this has some limitations, mainly their performance, and function handles are now preferred, e.g.
\begin{minted}[frame=lines, linenos]{matlab}
    % myFunction is defined in 'myFunction.m' and is called
    % like so myFunction(foo, bar)
    genericFunction = 'myFunction.m';
    foo = 1;
    bar = 2;
    feval('genericFunction', foo, bar);
\end{minted}
as opposed to,
\begin{minted}[frame=lines, linenos]{matlab}
    % myFunction is defined in 'myFunction.m' and is called
    % like so myFunction(foo, bar)
    genericFunction = @myFunction;
    foo = 1;
    bar = 2;
    genericFunction(foo, bar);
\end{minted}
which are not dissimilar, but the latter is more performant, easier to follow, can be returned as part of a \mintinline{matlab}{struct} (thus providing a poor-man's way of making classes), and can be nested. Generic functions were already in use for dislocation mobility, but there are now functions for calculating numeric and analytic tractions (see \cref{c:tractions}), loading conditions (multi-stage loading, constant loading, cyclic loading), post-processing functions, boundary conditions, and various miscelaneous functions that support different simulation types.

The main advantage of generic functions is reducing the need for direct source code modification. Changing source code, especially when there are no automated tests to speak of, is a dangerous proposition. It greatly increases the probability of introducing regressions (new bugs) and code divergeance between researchers. Regressions are always problematic, but code divergeance is a big issue that still affects the Tarleton Group. If experts within the same research group find it difficult to incorporate each others' additions to our work, there is little hope for the general user.

The introduction of regressions makes it so code must be thoroughly and exhaustively tested before merging two people's work. The use of generic functions makes it so if there is a problem with a new addition, the problem is localised and can be quickly identified and fixed because it is isolated. We can be sure the changes are only found in the new piece of code. And since it is generic, we can choose whether to use it by changing an input file rather than directly altering the source code. If instead we were to directly modify the source code, we would need to account for new inputs, outputs and perhaps create new branching behaviour. Doing so every time new functionality is added will lead to an explosion of complexity that spaggettifies code, making it difficult to work with and innefficient to run.

\subsection{Optimisation}
\renewcommand{\epigraphflush}{flushright}
\renewcommand{\textflush}{flushright}
\setlength{\epigraphwidth}{0.75\linewidth}
\epigraph{We \emph{should} forget about small efficiencies, say about 97\% of the time: pre-mature optimisation is the root of all evil. Yet we should not pass up our opportunities in that critical 3\%.}{\textit{--- Donald E. Knuth \cite[p.~268]{knuth1974structured}}}

\subsubsection{Spurious memory allocation}\label{ss:memAlloc}

\mintinline{matlab}{MATLAB} is infamous for the way it will happily grow an array dynamically when going out of bounds. This \emph{terrible} practice is even fundmental to the operation of \texttt{EasyDD}. However, changing this fundamental part of the codebase requires a complete rewrite and not deemed worthwhile, there are worse bottlenecks.

For example, the amount of memory unecessarily reallocated in the function that couples DDD and FEM was $12(N_x + 1)(N_y + 1)(N_z + 1)$ double precision floating point numbers, where $N_i$ is the number of elements in dimension $i$ of the finite domain. The cubic scaling is a performance killer at larger mesh sizes. In a simulation with hundreds or thousands of steps and a mesh with a few thousand FE nodes, the cost added up.

More memory allocation also means more garbage collection (GC) \cite{hanson1990fast} i.e. automatically freeing memory, based on a set of criteria. The cost of GC can be significantly higher than that of memory allocation. Timing the equivalent of allocating a $20 \times 20 \times 20$ mesh in \mintinline{matlab}{MATLAB 2020b} takes on the order of \SI{70}{\micro\sec} on x86 CPU architectures, which is in-line with what it costs in \mintinline{C}{C}. However, deallocating memory takes about $5 \times$ longer. In a function that otherwise takes a few \si{\milli\sec}, adding almost \SI{0.5}{\milli\sec} because of spurious allocation is an overhead that can be very easily remedied.

\subsubsection{Finite element optimisation}\label{ss:feOpt}

The use of sparse arrays is a must when working with finite elements, where matrices often take structured sparse forms. Taking advantage of this sparsity significantly reduces memory footprint and computational expense. The code already used these special data structures, along with Cholesky factorisation for faster solves of linear systems. However, \mintinline{matlab}{MATLAB} has a special form of it that exploits sparsity. This reduced the time required for the factorisation by around $50\times$, reduced the memory footprint of the final sparse arrays by a similar amount, and reduced the memory footprint of the actual factorisation procedure by approximately $10 \times$. The whole point of this factorisation is to make solving linear systems faster, in this case the calculation of the corrective displacements, $\vec{\hat{u}}$, on the fixed boundaries, $S_u$, as a result of the forces acting upon them,
\begin{align}
  \vec{\hat{u}} & = \mtx{K}^{-1} \vec{f}\quad \text{on } S_u                                               \\
                & = \mtx{P_l}^{-1} \mtx{U}^{-1} \mtx{L}^{-1} \mtx{P_u}^{-1} \vec{f}\quad \text{on } S_u\,.
\end{align}
Where $\mtx{P_l}$ and $\mtx{P_u}$ are sparse permutation matrices of the sparse upper and lower $\mtx{U}$, $\mtx{L}$ matrices given by the sparse factorisation.

The sparse factorisation improved the solution speed by a factor of 3. Thus ensuring the cost of coupling DDD to FEM is bottle-necked by the computation of dislocation-induced tractions and displacements, even for large finite element meshes.

The final two optimisations also deal with memory. The first, removes unecessary allocations in the calculation of the global stiffness matrix and the introduction of a reduced stiffness matrix that only includes relevant degrees of freedom.

In aggregate, these changes have given us the ability to use $15 \times$ more nodes than previously, as well as letting us build meshes over two orders of magnitude faster, mostly thanks to improved memory access patterns inside the mesh building loops.

\subsection{Misc quality of life improvements}\label{ss:qol}

Formerly, initialising a simulation would involve a series of non-obvious steps, particularly when compiling the C and CUDA code for the first time. This is now done automatically on a need-to-compile basis and has a fallback in case no \mintinline{CUDA}{CUDA} compiler is found.

There was also a recurring problem with simulations suddenly failing when a function was called with an undefined argument. This extremely common scenario is unbecoming of good software, especially if the failure happened a few minutes after starting a simulation. This has been remedied with the automatic calculation of a set of reasonable defaults for each and every undefined variable, aside from the arrays defining a dislocation network. Any new developments get added to this very simple, yet much needed script.

We also separated variable saving to minimise memory footprint and speed up simulations. Variables that remain constant throughout the simulation are saved only once before the main simulation loop, the rest are saved at whatever save frequency is provided in the input file. Plotting was also changed such that plotting functions are modular and provide more user control.

\section{Conclusions}

Scientific modelling is inherently an interdisciplinary skill. EasyDD is now faster, more efficient and more capable than ever before, in large part thanks to the work---ongoing or otherwise---described within this chapter. Subsequent chapters explore aspects more specific to discrete dislocation dynamics. However, it is important not to overlook the work that enables the obtention of results. Modern academia deems the final result to be paramount. But what is a result if it cannot be scrutinised because nobody can understand how it came about? Moreover, by developing good software, we bring others into the fold. We deem this to be one of the main contributions of the present project. We feel safe our ability to claim that we now have the capacity for providing more faithful recreations of reality in a more user-friendly way.
\savearabiccounter
% 5434 words


\section{Result}
\label{sec:result}
We have tested our method on a corpus provided by teacher Xu. For detailed
description of the corpus, please see former report.

All the tests are conducted serval times (depending on computation cost,
vary from 5 to 20) with random selected training and testing speakers.
The average over these tests are considered as the final
result.

\subsection{Efficiency Test of our GMM}
We have extensively examined the efficiency of our implementation of GMM
compared to scikit-learn version. Test is conducted using real MFCC data with
13 dimensions. We consider the scenario when training a UBM with 256 mixtures.
We examine the time used for ten iteration.  For comparable results, we diabled
the K-means initialization process of both scikit-learn GMM implementation and
ours.  Time used for ten iterations under different data size and concurrency
is recorded.

\begin{figure}[!ht]
	\begin{minipage}{0.48\linewidth}
		\centering
		\includegraphics[width=\linewidth]{res/time-comp.pdf}
		\caption{Comparison on efficiency\label{fig:gmm_efficiency}}
	\end{minipage}
	\hfill
	\begin{minipage}{0.48\linewidth}
		\centering
		\includegraphics[width=\linewidth]{res/time-comp-small.pdf}
		\caption{Comparison on efficiency when number of MFCC features is small\label{fig:gmm_efficiency_small}}
	\end{minipage}
\end{figure}

From \figref{gmm_efficiency}, we can immediately infer that our method
is much-much more efficient than the widely used version of GMM provided
by scikit-learn when the data size grows sufficiently large.

We shall analyze in two aspect:
\begin{itemize}
	\item No concurrency
		\begin{itemize}
			\item When the number of MFCC features is below 6000, which is a typical
				number of features generated by 60 seconds utterances (1ms frame shift),
				our method is slightly slower; but this is trivial since
				1 minute utterance is too small.
			\item When the number of MFCC features grows sufficiently large, our method
				shows great improvement. When training 512,000 features, our method
				is 5 times faster than comparing method.
		\end{itemize}
	\item With concurrency \\
		Our method shows considerable concurrency scalability that the running time
		is approximately lineary to the number of cores using.

		When using 8-cores, our method is \textbf{$19$ times} faster than comparing
		method.
\end{itemize}

\subsection{Effect Of Number Of Mixtures}
We examined our GMM compared to GMM from scikit-learn.
Test is conducted on 30-speaker corpus, 30 seconds training utterance
and 100 random sampled 5 seconds test utterance for each speaker.

As \figref{mixture} illustrates, when number of mixtures is small,
our GMM outperforms scikit-learn version by $10\%$, which indicates our
GMM models the distribution more accurately. The maximum accuracy
happens when the number of mixtures is around 32, reaching $0.965$. As
the number of mixtures increases, the decrease in accuracy

\begin{figure}
	\centering
	\includegraphics[width=\linewidth]{res/mixture-both.pdf}
	\caption{Accuracy curve on different number of mixtures\label{fig:mixture}}
\end{figure}

\section{Effect On Number Of Speakers}
An apparent trade-off in speaker recognition task is the number of speakers
enrolled and the accuracy of recognizing a person. We've conducted experiments
examining the effect of number of speakers enrolled on the performance of the
system.

The configurations of the test is as followed:
\begin{itemize}
	\item Number of mixtures is set to 32, the optimal number we found previously
	\item GMM from scikit-learn, compared to our GMM.
	\item 30s training utterance and 5s test utterance
	\item 100 sampled test utterance for each user
\end{itemize}

\begin{figure}[!ht]
	\centering
	\includegraphics[width=\linewidth]{res/nperson.pdf}
	\caption{Accuracy curve on different number of speakers enrolled\label{fig:nspk_enrolled}}
\end{figure}

Scrunitizing \figref{nspk_enrolled} we would see that, our GMM performs better than
scikit GMM in general. When number of speakers is small, due to the the random
selection, the variance of the tests is significantly high, as we can see from the curve fluctuants.
When number of speakers increases, it is clear that the
accuracy of our GMM is above scikit version. As the more speaker, the more
difficult the recognition task will be, this result suggests that our
optimization on GMM takes effect.


\chapter{Models, input data and analysis method}\label{chap:methods}

\section{FLEXDUST}\label{sec:flexdust}
FLEXDUST is an offline global dust emission model developed to be used together
with FLEXPART to study long-range dust transport and deposition. FLEXDUST relies on common
formulations for dust emissions found in global climate models \parencite{flexdust_ref_2016}.
FLEXDUST uses a bulk emission scheme as described in \Cref{sec:dust_emission_modelling}.
The version of FLEXDUST used is the latest as of July 17th 2021.
I modified the FLEXDUST source code slightly to allow the start date, duration and output directory of the simulation to be set dynamically.
To avoid having to recompile the code each time I changed the starting date of the simulation.

\subsection{Identification of dust sources}
FLEXDUST identifies the possible dust sources as regions with a high bare soil fraction.
The bare soil fraction is obtained from the satellite-based Global Land Cover map version 3, by National Mapping Organisations (GLCNMO), with a
resolution of 15 arcseconds \parencite{shirahata2017production}.
In addition to bare soil regions, FLEXDUST also considers partly vegetated areas as possible
dust sources. The available soil fraction for partly vegetated areas is determined by
subtracting the bare soil fraction from the vegetation cover, which is included in the ECWMF meteorological forcing.
FLEXDUST considers depressions to be more favourable to dust emissions as sediments are more easily gathered there \parencite{zender2003mineral}. Therefore FLEXDUST applies the erodibility scaling
\Cref{eq_ero_soil_frac} according to \textcite{dust_dist_Ginoux2001}.
\begin{equation}\label{eq_ero_soil_frac}
    S = \left(\frac{z_{max} - z_i}{z_{max} - z_{min}}\right)^5
\end{equation}
Here $z_i$ is the local elevation, $z_{max}$ and $z_{min}$ is the minimum and
maximum elevation in a 10\degree $\times$ 10\degree area.  The erodibility $S$ is
then scaled by the bare soil fraction to get the erodible soil fraction. The erodible soil fraction over East Asia is shown in \Cref{fig:erodible_soil_fraction_EA}.
\begin{figure}[hptb]
    \centering
    \includegraphics[width=\textwidth]{../figs/erodible_soil_fraction.pdf}
    \caption{The erodible soil fraction from FLEXDUST over East Asia}
    \label{fig:erodible_soil_fraction_EA}
\end{figure}
\subsection{Dust mobilisation threshold}
The threshold friction velocity follows the idealised expression of \textcite{shao2000simple} in \Cref{eq:treshold_fric_vel}, and is determined by the \SI{75}{\micro\metre} particles.
For regions with clay and silt present, but no sand, \SI{10}{\micro\metre}
particles are used to determine the threshold friction velocity. The clay and
silt fractions can either be obtained from ISRIC 250m resolution Soil Grids dataset
\parencite{soil-grid_ref}, or using the default Global soil Data \cite{task2014global} dataset.
The effect of soil moisture on the threshold friction velocity is parametrized according to \textcite{fecan1998parametrization}:
\begin{equation}
    \begin{cases}
    \frac{u_{*tw}}{u_{*t}}=1, & \text{if } w < w' \\
    \frac{u_{*tw}}{u_{*t}}=\sqrt{1+1.21(w-w')^{0.68}}, & w \geq w'
    \end{cases}
\end{equation}
Where $w$ is the volumetric water content of the soil (\%). If the soil moisture exceeds $w'$, the capillary
forces will start affecting the threshold friction velocity. Here, $w'$ is given by \Cref{eq:moisture_clay}
and depends on the clay content $c$ (\%).
\begin{equation} \label{eq:moisture_clay}
    w' = 0.17c + 0.0014c^2
\end{equation}
The soil moisture content is retrieved from the ECWMF input fields.
Furthermore, if the surface is covered by snow, dust emissions are assumed to be inhibited.
Snow cover is available from the ECWMF input fields.
\par In partly vegetated areas, non-erodible surface elements would increase the surface
roughness affecting the dust emissions. In FLEXDUST, the drag partition to non-erodible
surface elements for partly vegetated areas are accounted for following the description in
\textcite{zender2003mineral}, by scaling the mobilisation threshold by $f_d$ according to \Cref{eq:drag_partition}:
\begin{equation}\label{eq:drag_partition}
    f_d = \left[1 - \frac{\ln (z_{0m}/z_{0s})}{0.35\ln (0.1/z_{0s})^{0.8}}\right]^{-1}
\end{equation}
Here $z_{0m}$ is the roughness length of momentum transfer, and $z_{0s}$ is the smooth roughness length which
describes the roughness of a bed of potentially erodible particles without any non-erodible elements. Wind tunnel
experiments have shown that:
\begin{equation}
    z_{0s} \approx D/30
\end{equation}
where $D$ is the particle diameter. In FLEXDUST it is assumed globally uniform values of  $z_{0m}=\SI{100.0}{\micro\metre}$ and $z_{0s}=\SI{33.3}{\micro\metre}$.

\subsection{Dust flux parameterisation}

After the mobilisation threshold is calculated,  the dust emission flux is derived following the paremeterisation of the vertical dust flux as described in \textcite{MB95_dust_emission}:
\begin{equation}
    F=c\alpha \frac{\rho u_{*}^3}{g}\left(1-\frac{u^2_{*t}}{u^2_*}\right)\left(1+ \frac{u_{*t}}{u_*}\right)
\end{equation}
where $u_*$ is the friction velocity, c is a constant scaling factor ($4.8\cdot 10^{-4}$) and $\alpha$ is the sand blasting efficiency:
\begin{equation}\label{eq:sand_blasing_eff}
    \alpha = 100\exp{(13.4f_{clay}-6)\ln 10}
\end{equation}
\Cref{eq:sand_blasing_eff} is only valid for clay fraction $\leq$ 0.2, so if the clay fraction exceed 0.2, it is set to 0.2 \parencite{zender2003mineral}.

The small dust particles (< \SI{20}{\micro\metre}) has a greater potential to experience long-range transport due to their smaller settling velocities. Since FLEXDUST was developed to study long-range dust transport, the emitted dust is assumed to have a volume size distribution varying between \SI{0.2}{\micro\metre} and \SI{18.2}{\micro\metre}. This size distribution is based on brittle fragmentation theory described in \textcite{kok_scaling_2011}. The log-normal volume size distribution is determined according to \Cref{eq:size_dist}.

\begin{equation}\label{eq:size_dist}
    \frac{\text{d} V_d}{\text{d} \ln D_d} = \frac{D_d}{c_v}\left[1 + \text{erf}\left(\frac{\ln(D_d/\overline{D_s})}{\sqrt{2}\ln \sigma_s}\right)\right]\text{exp}\left[-\left(\frac{D_d}{\lambda}\right)^3\right]
\end{equation}
Where $c_v = \SI{12.62}{\micro\metre}$ is a normalisation constant, $\overline{D_s}=\SI{3.4}{\micro\metre}$ and $\sigma_s = 3.0$ is the median diameter and the geometric standard deviation respectively.

\section{FLEXPART}
\par The dust transport and deposition is calculated by the \acrfull{flexpart} \parencite{Flexpart10.4_ref}. FLEXPART is an offline \acrshort{lpdm} that has been applied to study the transport of a wide range of atmospheric tracers, e.g. mineral dust, black carbon and volcanic ash \parencite{flexdust_ref_2016,choi_investigation_2020, eckhardt2008estimation}.
The version of FLEXPART used is 10.4 and includes the improved wet deposition scheme based on cloud information from the ECWMF input fields \parencite{flexpart_wetdep}.
\par FLEXPART calculates trajectories for a large number of computational particles (from here on referred to as particles and not be confused with real dust particles) that move according to the wind-fields resolved in the meteorological forcing with parameterizations of the turbulent motions, subgrid-scale convection, and gravitational settling imposed on the trajectories.
Each particle represents a dust-aerosol population with a log-normal mass size distribution with a specified density and mean particle diameter.
In addition, ice nucleation efficiency,$IN_{eff}$, cloud condensation nuclei efficiency $CCN_{eff}$ and efficiency of below cloud scavenging by rain $C_{rain}$ and snow $C_{snow}$, can be assigned to the particle regulating the strength of the wet deposition in the model.

FLEXPART can be either run forward in time from the source to the receptor or backwards from the receptor to the source. The main difference between forward and backward simulations lies in how the emission strength is constrained.
In a forward simulation, the particles are released at the source.
The emitted mass which the particles carry is defined before the particle release, directly yielding the concentration and deposition on a regular longitude-latitude grid at each time step.
However, in a backward simulation, the particle release happens at the receptor, and the emission strength is unknown.
The purpose of the particles in a backward simulation is to probe for the possible source regions and establish the emission sensitivity, i.e. how sensitive the deposition at the receptor would be to a potential source element.
For a more in-depth derivation of source receptor relationships within the framework of \acrshort{lpdm} see Appendix \labelcref{appendix:lpdm_SRR}. The final output corresponds to a distribution of possible source areas at each time step.
The emission sensitivity is constructed such that when multiplied by the emission flux with units \si{\kg\per\cubic\metre\per\s} would produce a map of how much each source element is contributing to the concentration or deposition at the receptor.
In a backward simulation, concentration, wet and dry deposition must be obtained through separate simulations as the particle release is set up differently depending on the configuration.
\subsection{Dry deposition}
In FLEXPART, the dry deposition is represented by a dry deposition velocity $v_d(z)$, and when multiplied with the concentration at a specified height, yields the deposition flux.
FLEXPART includes both a dry deposition scheme for particulate matter as well as gasses and is calculated based on a two-layer resistance model \parencite{Flexpart-2005_ref_paper}.

In backward mode, dry deposition is calculated at the start of the simulation by releasing the particles in a shallow layer close to the ground. This layer corresponds to the layer in which particles in forward mode would be subjected to dry deposition, and is by default is set to 30m. The "mass" of the particles are then scaled by the deposition velocity and tracked backwards in time as in regular backward simulation \parencite{eckhardt2017source}.

\subsection{Wet deposition}
The wet deposition scheme in FLEXPART was recently improved in \textcite{flexpart_wetdep} to take advantage of the cloud information available from the ECWMF forcing. The wet deposition scheme in FLEXPART consists of two steps.
First, the particle's location in relation to the cloud is determined using the 3D specific \acrfull{ctwc}. Then based on the particle's location in relation to the cloud, the particle can either be subjected to nucleation scavenging inside the cloud or impaction scavenging below the cloud. If the particle is above the cloud, no scavenging can occur.

In backward mode, the wet deposition is calculated at the time of the particle release. Since wet deposition can occur throughout the whole atmospheric column, the particles are released over the whole atmospheric column to check if any particles are subjected to wet deposition. FLEXPART then calculates the trajectories for wet deposited particles starting at the height where the particles were scavenged, and the remaining particles are terminated.
\subsubsection{In cloud scavenging}
The in-cloud scavenging scheme in FLEXPART is only activated for particles residing within a precipitating cloud. Precipitation does not necessarily occur over the whole grid cell in the input data. Accordingly, FLEXPART uses an empirical relation to derive the fraction of a gridcell experiencing precipitation; see \textcite{Flexpart-2005_ref_paper} for details. The subgrid precipitating cloud water (PCW) is calculated according to:
\begin{equation}
    PCW = CTWC\frac{F}{cc}
\end{equation}
Here cc is the surface cloud cover, and F is the precipitating fraction of the gridcell.

The aerosol scavenging coefficient $\Lambda$ (\si{\per\s}) for in cloud scavenging is given by:
\begin{equation}
    \Lambda = F_{nuc}\left(I/PWC\right)ic_r
\end{equation}
Where $F_{nuc}$ is the nucleation efficiency, I is the precipitation intensity, which is derived from the accumulated precipitation during one time interval in the ECWMF input data, and $ic_r$ is the cloud water replenishment rate. However, $ic_r$ cannot be determined from the ECWMF input data and was determined based on testing in FLEXPART and is considered a tuning parameter in the model.

In reality, $F_{nuc}$ depends on many different parameters such as aerosol size, chemical composition, temperature and cloud phase.
A complete parameterization of $F_{nuc}$ in FLEXPART is currently not possible based on the limited information available in the input forcing.
Still, FLEXPART can account for the fact that aerosols have different nucleation efficiency depending on whether they serve as cloud condensation nuclei (CCN) or ice nuclei (IN).
% For insoluble aerosols like mineral dust $CCN_{eff}$ primarily depends on the particle size. By itself mineral dust is usually not a very efficient CCN. However, by weathering during transport, the mineral dust aerosols might acquire a coating of some soluble material, e.g. sulphate, increasing its potential to act as a CCN \textcite{Dust_aerosols_coating2001}.

\subsubsection{Below cloud scavenging}
Dust aerosols residing below the cloud might be scavenged by falling raindrops or snowflakes, called impaction scavenging.
In FLEXPART, the probability of an aerosol being scavenged by liquid droplets is parameterised according to \textcite{laakso2003ultrafine}.
The removal of aerosols by falling snowflakes is parameterised in \acrshort{flexpart} according to \textcite{kyro2009snow}.
The efficiency of scavenging by falling rain and snow can be adjusted in the model by changing the $C_{rain}$ and $C_{snow}$ parameter.

\section{Model input}
\subsection{flex\_extract}
Both FLEXPART and FLEXDUST depend on external meteorological input data. \verb|Flex_extract| is a package developed to retrieve and prepare the necessary meteorological input data from the servers of the \acrfull{ecwmf} \parencite{tipka_flex_extract_2020}.
\verb|flex_extract| version was 7.0.4 used to retrieve the necessary input fields from the ERA5 reanalysis at a 3hourly temporal resolution and $0.3\degree \times 0.3\degree$ spatial resolution for over the subdomain of East Asia from March until May for the years from 1999 to 2019.
The retrieval process can be slow, as, in addition to the large amount of data that needs to be downloaded, the request also has to be processed by the ECWMF servers.

\subsection{ERA5}
ERA5 is the latest reanalysis produced by the ECWMF.
The ERA5 reanalysis assimilates large amounts of observational data to provide the best estimate of the state of the atmosphere extending back to 1979 \parencite{hersbach_era5_2020}. Compared to its predecessor ERA-Interim, ERA5 offers a higher spatial resolution of $0.25\degree \times 0.25\degree$, up to hourly temporal resolution, and is based on a more recent version of the \acrfull{ifs}.

\section{Model setup}\label{sec:Model_setup}

In this thesis, the dust emission model FLEXDUST and the \acrshort{lpdm} FLEXPART was used to
simulate the springtime dust deposition between 1999 and 2019 at 7 sites across the \acrshort{clp}.

\textcite{flexdust_ref_2016} used the same two models to study the transport and deposition of high latitude dust, where they simulated the dust transport from a source-oriented viewpoint, following the dust from the source to deposition.
Here, the dust transport and deposition are simulated from a receptor viewpoint.
All the dust deposited at the receptor location is tracked backwards in time to probe for possible sources.
The emission field produced by FLEXDUST then constrains the distribution of possible source regions.
The flow chart in \Cref{fig: flow_chart} summarises the modelling workflow. The receptor oriented backward approach was chosen over the forward approach due to being more computationally efficient when the number of source elements greatly exceeds the number of receptors.
In addition, the backward approach allows for dealing with point receptors naturally \parencite{seibert2004source}.
% In a forward simulation, the deposition at the receptor location must be interpolated from the model output grid.
The backward approach also makes it possible to determine exactly the source regions that contribute to deposition at the receptor, namely the sites over CLP.

\begin{figure}[!ht]
  \centering
\begin{tikzpicture}[thick,node distance=2cm, scale=0.6, every node/.style={scale=0.8}]
    \node (start) [startstop, align=center] {\Large \verb|Flex_extract| \\Data retrieval from ECWMF servers};
    \node (in1) [io, below of=start, align=center] {ERA5 input fields \\ 3hourly, \ang{0.3} $\times$ \ang{0.3}};
    \node (flexpart) [process, below right=0.6cm and 0.8cm of in1, align=center] {\Large \verb|FLEXPART| \\ Dust transport and deposition, \\ backward simulations starting from \\ the specified receptor location.};
    \node (flexdust) [process, below left=1cm and 0.8cm of in1, align=center] {\Large \verb|FLEXDUST|};
    \node (emssens_wetdep) [io, below of=flexpart, align=center, xshift=60, node distance=2.5cm] {Wet deposition \\ sensitivity };
    \node (emssens_drydep) [io, below of=flexpart, align=center, xshift=-70, node distance=2.5cm] {Dry deposition \\ sensitivity};
    \node (dust_emissions) [io, below of=flexdust, align=center, node distance=2.9cm] {Dust emission flux \\ \si{\kg\per\square\metre}};
    \node (pprocess) [process, below of=start, align=center, node distance=10cm] {\Large Post processing \\ Emissions sensitivities are multiplied \\ with corresponding dust emissions.};
    \node (source_contrib) [io, below of=pprocess, align=center, node distance=3.5cm, xshift=-50] {\Large Source contribution \\ i.e. how much each source \\ element is contributing to \\ dry and wet deposition at \\ the receptor.};
    \node (stop) [startstop, right of= source_contrib,align=center,  node distance=9cm] {\Large Dust deposition at \\ \Large the receptor \\ the total contribution \\ from all source elements.};
    \draw [arrow] (start) -- (in1);
    \draw [arrow] (in1) -| (flexpart);
    \draw [arrow] (in1) -| (flexdust);
    \draw [arrow] (flexpart) -- (emssens_wetdep);
    \draw [arrow] (flexpart) -- (emssens_drydep);
    \draw [arrow] (flexdust) -- (dust_emissions);
    \draw [arrow] (emssens_wetdep) |- (pprocess);
    \draw [arrow] (dust_emissions) -- (pprocess);
    \draw [arrow] (emssens_drydep) -- (pprocess);
    \draw [arrow] (pprocess) -- (source_contrib);
    \draw [arrow] (source_contrib) -- (stop);
\end{tikzpicture}
\caption{Flow chart showing the workflow for the modelling analysis. }
\label{fig: flow_chart}
\end{figure}

The simulations have been conducted on the Saga high performance computing cluster operated by the Norwegian high performance
computing centre, Sigma2. For storage and postanalysis, the data is transferred to the
\acrfull{nird} computing infrastructure.

FLEXPART and FLEXDUST were run at a 3hourly temporal resolution, with $0.1\degree \times 0.1\degree$ output grid and driven by  ERA5 meteorological forcing data.
% FLEXDUST was set up at the same temporal and output grid as FLEXPART.
As described in \Cref{chap:east_asia_dust} dust events are primarily occurring in spring.
Therefore, I only simulated the spring months from March until May.  I ran the models over a 20 year period to provide insight on the interannual variation in the dust source, transport and deposition to the \acrshort{clp}.

\subsection{FLEXPART}
To make it possible to multiply the emission sensitivity with the dust emission flux, FLEXPART has to be set up such that particle release at each time step is kept separate in the output.
Therefore, I  specified a new particle release in the \verb|RELEASES| file for each forward time step. I limited the length of the trajectory to five days by specifying a time limit in the \verb|NAGECLASSES| file.
The FLEXPART output file then contains two temporal dimensions, one \verb|time| dimension going from the last date backwards to the start date of the simulation and the \verb|pointspec| dimension representing each release.
To allow for all the backward trajectories to complete, I let the last release happen five days after the starting date of the simulation.
The particle release is specified at 3rd hourly intervals. The number of particles released in each release is 50000 particles and 200000 particles for dry and wet deposition simulations, respectively.
More particles are needed in the wet deposition simulation to achieve statically robust results due to how the particle release is set up.
A copy of the exact FLEXPART and FLEXDUST source code used can be accessed at \url{10.5281/zenodo.5128338}.
\begin{figure}[htpb]
    \centering
    \includegraphics[width=0.7\textwidth]{texfiles/figs/map_loess.pdf}
    \caption{Map of the receptor points, specified in the FLEXPART simulations}
    \label{fig:maps_clp_location}
\end{figure}

To provide insight on the spatial variation in the dust source and transport to the \acrshort{clp}, seven sites distributed across the \acrshort{clp} were selected as receptor points in FLEXPART  (\Cref{fig:maps_clp_location}), namely: SACOL, Shapotou and Yinchuan in the north-west, Lingtai, Luochuan and Lantian in the south-east and Baode in the north-east.
The exact coordinates of the receptor locations are listed in \Cref{tab:coordinates_clp}.

Moreover, two particle size bins, \SI{1.7}{\micro\metre}-\SI{2.5}{\micro\metre} representing clay particles and  \SI{15}{\micro\metre}-\SI{20}{\micro\metre} size bin representing silt sized particles were used in the FLEXPART simulations, respectively.
The two particle size bins do not only differ in size, but they also have different assumptions on the $CCN_{eff}$.
The parameters for the two particle species are listed in \Cref{tab:particle_params}.
I included only two size bins as a trade-off to allow running multiple years and keep the computational demand reasonable.

\begin{table}[htpb]
\caption{FLEXPART species parameters for "clay" and "silt" particle size bin}
\label{tab:particle_params}'
\centering
% \resizebox{\textwidth}{!}{%
\begin{tabular}{@{}lll@{}}
\toprule
 & Clay  & Silt \\ \midrule
\verb|PCRAIN_AERO| & 1.00  & 1.00 \\
\verb|PCSNOW_AERO| & 1.00  &  1.00 \\
\verb|PCC_AERO| & 0.45 \parencite{flexdust_ref_2016}   &  0.90\parencite{flexdust_ref_2016} \\
\verb|PIN_AERO| & 0.10 \parencite{flexdust_ref_2016} & 0.10 \parencite{flexdust_ref_2016} \\
\verb|PDENSITY| & \SI{2500.0}{\kg\per\cubic\metre}    & \SI{2500.0}{\kg\per\cubic\metre} \\
\verb|PDQUER| & \SI{2.057E-6}{\metre}    &  \SI{17.32E-6}{\metre}   \\
\verb|PDSIGMA| & 1.21   &  1.15    \\ \bottomrule
\end{tabular}%
% }
\end{table}

Given that all the FLEXPART simulations are independent of each other, it is easy to take advantage of the parallel computing capabilities of Saga. By splitting the simulation into several independent FLEXPART simulations, each covering one month, the simulation could be run across multiple CPUs simultaneously.
All in all, this added up to a total of 1680 individual FLEXPART simulations.
To help set up the FLEXPART simulations, a python script\footnote{https://github.com/Ovewh/FLEXPART-script} was developed.

\subsection{FLEXDUST}
Compared to \acrshort{flexpart}, FLEXDUST is much less demanding to run.
The FLEXDUST simulation was split into sub simulations covering one spring, and the whole simulation took a few hours to complete on the supercomputer.
The soil texture dataset used was the ISRIC 250m Soil grids dataset.
The ISRIC dataset was proven based on model sensitivity experiments to represent the East Asian dust sources better than the default soil texture dataset used in FLEXDUST.

\subsection{Model validation}\label{sec:model_eval_discription}
There are very few published observations of dust deposition fluxes that separate both dry and wet deposition over East Asia and have a sufficient temporal resolution to evaluate the performance of FLEXPART/FLEXDUST. \textcite{osada2014wet} published an observation data set of monthly dry and wet deposition fluxes from 6 locations by the coast of Japan, covering the time period from November 2008 until December 2010. They used a 4-stage filtration approach to separate the deposited dust into four size bins  >\SI{20}{\micro\metre}, \SI{20}{\micro\metre}-\SI{10}{\micro\metre}, \SI{10}{\micro\metre}-\SI{5}{\micro\metre} and \SI{5}{\micro\metre}-\SI{1}{\micro\metre}.
The four particle size bins were simulated separately in FLEXPART and assumed to follow the same size distribution as the main model experiment. The Hedo, Fukuoka, Tottori and Toyama sites were selected as receptor points for FLEXPART simulations in this study.

\begin{figure}[htpb]
    \centering
    \includegraphics[scale=0.45]{texfiles/figs/Osada_locations.PNG}
    \caption{Map of the locations where dust deposition was sampled \parencite{osada2014wet}}
    \label{fig:map_japan}
\end{figure}

\subsection{Model sensitivity experiments}
To better evaluate the sensitivity of the model results to the uncertainties in the input data and model parameters. Four sensitivity experiments were conducted to investigate the impact of (1) different meterological forcing data, (2) varying particle density, (3) choice of soil texture data and (4) below cloud scavenging. All the sensitivity experiments were done for 2019 except experiment (1), which was done for 2015. Except for the parameters listed in \Cref{tab:sensitivity_exp}, all other parameters are equal to the main model experiment described previously. The control run, which the sensitivity simulations are compared to is the main model experiment.

% The reason for testing the model with different forcing data is to examine how sensitive the results are to the choice of input data. The two forcing dataset compared in the sensitivity analysis is the latest generation of \acrshort{ecwmf} reanalysis, ERA5 against the previous generation of reanalysis ERA-Interim. This analysis would allow to whether using a the new higher resolution ERA5 as input data improves the representation of dust in the models. Moreover, if the difference between the two forcing dataset can reasonably explain based on there known differences, it means that
\begin{table}[htpb]\label{tab:sensitivity_exp}
\caption{The model sensitivity experiments performed in this study}.
\centering

\resizebox{\textwidth}{!}{%
\begin{tabular}{@{}lllll@{}}
\toprule
 Sensitivity experiment & Soil map  & Forcing & Below cloud scavenging & Particle density \\ \midrule
 Model forcing & ISRIC  & ERA-Interim & Yes & \SI{2500}{\kg\per\cubic\cm}  \\
 Particle density & ISRIC & ERA5  & Yes & \SI{2200}{\kg\per\cubic\cm}-\SI{2800}{\kg\per\cubic\cm}  \\
 Soil texture & Default & ERA5 & Yes & \SI{2500}{\kg\per\cubic\cm}   \\
 Cloud scavenging& ISRIC  & ERA5 & No & \SI{2500}{\kg\per\cubic\cm} \\ \bottomrule

\end{tabular}%
}
\end{table}

\section{Analysis tools}

The analysis and postprocessing of the model output was achieved using a handful of Python libraries. More details on the specific software used and developed to facilitate the analysis are provided in Appendix \labelcref{appendix:software}.
\subsection{Post-processing of model output}
The FLEXDUST emission flux has to be matched with the corresponding emission sensitivity from FLEXPART to calculate the source contribution and deposition at the receptor.
However, there is currently no software available that can match output from a backward simulation with an emission inventory.
Therefore, Python code was developed to combine the FLEXPART and FLEXDUST output automatically. FLEXPART output data was reshaped to a format that would easily match with FLEXDUST.
% Therefore, a new backward time dimension with a length equal to the maximum particle age (in this case, 5 days) was created and the
%  \verb|pointspec| dimension was set as the forward time dimension.
% This format proved to be a more efficient way of storing the output from backward simulations since the raw output format would only contain zeros after the maximum age was exceeded.
% When the output was converted to the new format, it was straightforward to match the FLEXDUST emission inventory with the FLEXPART emission sensitivity.

After the FLEXPART and FLEXDUST outputs have been combined, the final step is to scale the source contribution by the fraction of the size distribution corresponding to the specific size bin listed in \Cref{tab:size_distribution}.
The assumed size distribution in \Cref{tab:size_distribution} is the same size distribution used in FLEXDUST to generate RELEASE files for forward FLEXPART simulation and follow the log-normal distribution of \Cref{eq:size_dist}.

\begin{table}[htpb]
    \caption{Assumed size distribution of emitted dust in FLEXPDUST}
    \label{tab:size_distribution}
    \centering
    \begin{tabular}{@{}ll@{}}
    \toprule
        bin width (\si{\micro\metre}) &  fraction \\ \midrule
         0.02 –  0.1 & 0.0030\\
         0.1 – 0.5 &  0.0163\\
         0.5 - 1.0 &0.0327\\
         1.0 – 1.7 & 0.0556\\
         1.7 – 2.5 & 0.0820\\
         2.5 – 5.0 &  0.1605\\
         5.0 – 7.5 & 0.2209\\
         7.5 – 10 & 0.2425\\
         10 – 15 & 0.1516 \\
         15 – 20 & 0.0349
    \end{tabular}

\end{table}

\subsubsection{Snakemake workflow}
To manage the considerable amount of data produced by \acrshort{flexpart}, a \verb|snakemake| workflow was implemented.
\verb|Snakemake| is a workflow management system \parencite{molder2021sustainable} that can keep track of all dependencies and steps required to generate a specific result.
This means that \verb|snakemake| would notice changes made to a file at the beginning of the workflow and regenerate all the results depending on that file.
\verb|Snakemake| also keeps track of which steps of the analysis have been completed, so if anything happens while the analysis is running, it does not have to start from the beginning.
\verb|Snakemake| works by defining a sequence of rules that takes predefined input files based on a specified filename pattern.
Based on this filename pattern \verb|snakemake| knows the sequence of rules that generates the desired output file. \Cref{fig:rule_graph_eks} show an example of how this sequence of rules might look like.
 \begin{figure}[htpb]
     \centering
     \includegraphics[width=0.4\textwidth]{texfiles/figs/rulegraph_eks.pdf}
     \caption{Example rulegraph for generating \Cref{fig:source_contrib2mmu_anomalies} and \Cref{fig:source_contrib20mmu_anomalies}}
     \label{fig:rule_graph_eks}
 \end{figure}

\subsection{FLEXPART trajectory analysis}\label{chap:method_trajec_analysis}
In addition to emission sensitivity, FLEXPART can also output information about the particle trajectories. Including calculating the centroid trajectory and cluster trajectories of the particle cloud.
Unfortunately, the clusters produced by FLEXPART are not very informative, as the clustering routine only considers the position of the particles at the output time and does not consider which cluster it was previously assigned to. However, the centroid trajectory can still provide insight into the dust transport paths.
The original idea was to take the centroid trajectories of every particle release and then cluster the centroid trajectories to obtain a set of trajectories representing the main dust transport pathways.
This procedure is described in more detail in Appendix \labelcref{chap:trajec_analysis}.
However, this did not work out for two reasons, (1) it was challenging to define a suitable distance metric, and (2) the centroid trajectories are an average of both dust loading and non-dust loading trajectories, thus smooths out the dust transport paths.
Therefore, making a clever selection and averaging of the trajectories proved to be more informative.

% The optimal approach would be do clustering based on the particle position, however this was not infeasible do here since the amount of data would become too much. Analysing the particle positions is better suited for case studies of transport during of single dust storms.
The current trajectory analysis was done by first filtering out the non-dust centroid trajectories based on whether dust was deposited at the receptor at the time of arrival.
Then all the dust loading trajectories were given a weight based on their dust loading and averaged to make one weighted average centroid trajectory.
This averaged centroid dust loading trajectory was then applied to diagnose the averaged paths of dust transport to the receptors.

\subsection{Statistical methods}

\subsubsection{Pearson's correlation}
Pearson's correlation was used to examine the potential connection of climatic factors with the interannual variability of the dust cycle over the CLP.
The interannual correlations were calculated between spring dust emissions, wet and dry deposition simulated by FLEXDUST and FLEXPART and several climatic variables characterizing different aspects of large-scale circulation. The following climate indices were used:
\acrshort{ao}, \acrshort{nao}, the south-north temperature gradient across East Asia (refered to as he \acrshort{eatg}), \acrshort{eawmi}, \acrshort{sh} and \acrshort{mo}. The definition of each of the indices is listed in \Cref{tab:climate_vars}.
The indices were calculated based on monthly ERA5 data for both winter and spring to investigate the influence of winter conditions on the spring \acrshort{eadc}.
The decadal trends were removed from the time series before correlation analysis to avoid any influence of low-frequency variations.

\subsubsection{Composite analysis}
Composite analysis is a useful method for identifying climatic features that are difficult to observe in its totally.
A composite analysis involves collecting several cases of a given situation, e.g. years with strong deposition, and creating an average representation of meteorological conditions in that given situation, e.g. surface winds during strong deposition years.
The change can then be enlarged further by subtracting the weak composite from the strong composite to show the composite anomalies.
In the composite analysis, each year is ranked based on the spring deposition rate, and then the four strongest and weakest years are composited. It should be noted that the frequency of extreme years varied greatly among the sites over this relatively short period. Therefore, it did not make sense to use a more objective criteria for selecting the strong and weak deposition years, e.g. standard deviation.

\begin{table}[htpb]
    \footnotesize
    \centering
    \caption{Definition of the climatic indices used in the correlation analysis.}
    \begin{tabular}{ C{3.7cm}  C{10cm} }
    \toprule
    Indices & Definition \\ \midrule
    \acrfull{ao} \parencite{thompson1998arctic}   &
    Leading \acrshort{eof} of the 1000hPa geopotential height anomalies poleward of 20\degree N   \\
    \acrfull{eawmi} \parencite{jhun2004new} &
    Difference in area averaged 300hPa zonal winds  $U_{300}(27.5\degree-37.5\degree N, 110\degree-170\degree E)- U_{300}(50\degree-60\degree N, 80-140\degree E)$  \\
    \acrfull{mo} \parencite{sakai2009remote} & Difference in \acrfull{mslp} between two grid points (105\degree E, 52.5°N) near Irkutsk and (145\degree E, 43.75\degree N) near Nemuro. \\
    \acrfull{shi} \parencite{wang2010well} &
    Area averaged \acrshort{mslp} (40\degree-65\degree N, 80\degree-120\degree E) \\
    \acrfull{nao} & Leading \acrshort{eof} of \acrshort{mslp} anomalies of the cutout (40\degree-90\degree N, 20\degree-80\degree E) \\
    \acrfull{eatg} \parencite{liu2020impact} & Gradient in zonally averaged temperature between 30\degree - 120\degree E, and taken the averaged gradient between 40\degree and 50\degree N \\
    \acrfull{apvi} \parencite{yu2021conditional} & Area averaged geopotential height anomalies over 45\degree–90\degree N, 60\degree–150\degree E \\
    \bottomrule

    \end{tabular}

    \label{tab:climate_vars}
\end{table}

\subsubsection{Empirical orthogonal function}

\acrfull{eof} analysis is often used in climate studies to identify the regions with the largest variability and study how they change with time.
Given a space-time field \acrshort{eof} analysis ﬁnds a set of orthogonal (independent) spatial patterns along associated with a time series of Principal components (\acrshort{pc}s) \parencite{hannachi2007empirical}. The \acrshort{eof} analysis ﬁnds a new set of variables that capture most of the observed variance from the data through linear combinations of the original variables. However, due to sampling issues, the eigenvalues of \acrshort{pc}s might be mixed.
\textcite{north1982sampling} provides a rule of thumb for determining whether the eigenvalues are distinct. \textcite{north1982sampling} states that if the neighbouring \acrshort{pc}s lays within the typical error of the respective \acrshort{pc}s $\lambda$, the two EOFs might be mixed in their signal, thus not representing true patterns.
The \acrshort{ao} is an example of a climatic feature that was first described based on an \acrshort{eof} analysis \parencite{thompson1998arctic}. Here the \acrshort{eof} analysis was done using \verb|eofs| python package \parencite{dawson2016eofs}.

were


% Copyright 2017-2018 Dan Foreman-Mackey and the co-authors listed below.

\documentclass[modern]{aastex61}

\usepackage{microtype}
\usepackage{url}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{natbib}
\usepackage{multirow}
\bibliographystyle{aasjournal}

% ------------------ %
% end of AASTeX mods %
% ------------------ %

% Projects:
\newcommand{\project}[1]{\textsf{#1}}
\newcommand{\kepler}{\project{Kepler}}
\newcommand{\ktwo}{\project{K2}}
\newcommand{\lsst}{\project{LSST}}
\newcommand{\tess}{\project{TESS}}
\newcommand{\celerite}{\project{celerite}}
\newcommand{\emcee}{\project{emcee}}

\newcommand{\foreign}[1]{\emph{#1}}
\newcommand{\etal}{\foreign{et\,al.}}
\newcommand{\etc}{\foreign{etc.}}
\newcommand{\ie}{\foreign{i.e.}}

\newcommand{\figureref}[1]{\ref{fig:#1}}
\newcommand{\Figure}[1]{Figure~\figureref{#1}}
\newcommand{\figurelabel}[1]{\label{fig:#1}}

\newcommand{\Table}[1]{Table~\ref{tab:#1}}
\newcommand{\tablelabel}[1]{\label{tab:#1}}

\renewcommand{\eqref}[1]{\ref{eq:#1}}
\newcommand{\Eq}[1]{Equation~(\eqref{#1})}
\newcommand{\eq}[1]{\Eq{#1}}
\newcommand{\eqalt}[1]{Equation~\eqref{#1}}
\newcommand{\eqlabel}[1]{\label{eq:#1}}

\newcommand{\sectionname}{Section}
\newcommand{\sectref}[1]{\ref{sect:#1}}
\newcommand{\Sect}[1]{\sectionname~\sectref{#1}}
\newcommand{\sect}[1]{\Sect{#1}}
\newcommand{\sectalt}[1]{\sectref{#1}}
\newcommand{\App}[1]{Appendix~\sectref{#1}}
\newcommand{\app}[1]{\App{#1}}
\newcommand{\sectlabel}[1]{\label{sect:#1}}

\newcommand{\T}{\ensuremath{\mathrm{T}}}
\newcommand{\dd}{\ensuremath{\,\mathrm{d}}}
\newcommand{\unit}[1]{{\ensuremath{\,\mathrm{#1}}}}
\newcommand{\bvec}[1]{{\ensuremath{\boldsymbol{#1}}}}

% TO DOS
\newcommand{\todo}[3]{{\color{#2}\emph{#1}: #3}}
\newcommand{\dfmtodo}[1]{\todo{DFM}{red}{#1}}

% \shorttitle{}
% \shortauthors{}
% \submitted{Submitted to \textit{The AAS Journals}}

% typography obsessions
\setlength{\parindent}{3.0ex}

\begin{document}\raggedbottom\sloppy\sloppypar\frenchspacing

\title{%
Inferring stellar rotation periods using K2 at scale
}

\author[0000-0002-9328-5652]{Daniel Foreman-Mackey}
\affil{Center for Computational Astrophysics, Flatiron Institute, New York, NY}

\author[0000-0003-4540-5661]{Ruth Angus}
\affil{Simons Fellow}
\affil{Department of Astronomy, Columbia University, New York, NY}

\author[0000-0002-0296-3826]{Rodrigo Luger}
\affil{Department~of~Astronomy, University~of~Washington, Seattle, WA}

\begin{abstract}

We measure all the periods.

\end{abstract}

\keywords{%
 %methods: data analysis
 %---
 %methods: statistical
 %---
 %asteroseismology
 %---
 %stars: rotation
 %---
 %planetary systems
}

\section{Introduction}

Some words\ldots \citep{Luger:2017}

\citep{Foreman-Mackey:2017} demonstrated that \celerite\ could be used to
compute the likelihood of a GP model where the kernel is a mixture of
stochastically-driven, damped simple harmonic oscillators (SHOs).
The power spectral density of each term in this model is:
\begin{eqnarray}
S_k(\omega) = \sqrt{\frac{2}{\pi}}\,
\frac{s_k\,\omega_k}{{(\omega^2 - {\omega_k}^2)}^2 +
    {\omega_k}^2\,\omega^2/{q_k}^2}
\end{eqnarray}
where $s_k$ is the driving power, $\omega_k$ is the un-damped frequency, and
$q_k$ is the quality factor of the oscillator.
We find that a restricted mixture of three SHO terms is flexible enough to
capture the astrophysical variability while remaining simple enough to be
easily interpreted.
We set the quality factor of the first oscillator to $Q_1 = 1/\sqrt{2}$ to
capture the stellar granulation using a ``Harvey model''.
We constrain the other two terms with
\begin{eqnarray}
a_2 &\ge& a_3, \\
P_2 &=& 2\,P_3,\quad \mathrm{and} \\
q_2 &\ge& q_3 > 1/2
\end{eqnarray}
where $a_k = s_k\,\omega_k\,q_k$ is the amplitude of the oscillator and
$P_k = 4\,\pi\,q_k / \sqrt{4\,{q_k}^2-1}\,\omega_k$ is the oscillation period.

\acknowledgments\
It is a pleasure to thank
\dfmtodo{many people}
for helpful discussions informing the ideas and code presented here.

This research made use of the NASA \project{Astrophysics Data System} and the
NASA Exoplanet Archive.
The Exoplanet Archive is operated by the California Institute of Technology,
under contract with NASA under the Exoplanet Exploration Program.

This paper includes data collected by the \kepler\ Mission. Funding for the
\kepler\ Mission is provided by the NASA Science Mission directorate.
We are grateful to the entire \kepler\ team, past and present.
These data were obtained from the Mikulski Archive for Space Telescopes
(MAST).
STScI is operated by the Association of Universities for Research in
Astronomy, Inc., under NASA contract NAS5\-26555.
Support for MAST is provided by the NASA Office of Space Science via grant
NNX13AC07G and by other grants and contracts.

%This research made use of Astropy, a community-developed core Python package
%for Astronomy \citep{Astropy-Collaboration:2013}.

\facility{Kepler}
\software{%
     %\project{AstroPy} \citep{Astropy-Collaboration:2013},
     \project{corner.py} \citep{Foreman-Mackey:2016},
     \project{Eigen} \citep{Guennebaud:2010},
     \project{emcee} \citep{Foreman-Mackey:2013},
     %\project{george} \citep{Ambikasaran:2016},
     %\project{Julia} \citep{Bezanzon:2012},
     %\project{LAPACK} \citep{Anderson:1999},
     \project{matplotlib} \citep{Hunter:2007},
     \project{numpy} \citep{Van-Der-Walt:2011},
     %\project{transit} \citep{Foreman-Mackey:2016a},
     \project{scipy} \citep{Jones:2001}.
}

%\vspace{5ex}
\appendix

There's always an appendix.

\bibliography{rotate}

\end{document}