\section{Materials and Methods}
\label{methods}

\subsection{Input Graph}

We build the network model using data provided by the Lebanese Ministry of Energy and Water. This dataset contains information about every power plant generator (sources for power), high-to medium voltage transmission substations, medium to low voltage distribution stations that disseminate power directly to subscribers, as well as transmission lines through which power dissipates. Transmission lines exist between generators and transmission substations, as well as among transmission substations, distribution substations, and finally, between transmission and distribution substations.

\secondedited{
The power grid consists of generating stations responsible for the power production, which is transmitted through high voltage lines to demand centers, which then, in turn, distribute power to the consumers.  This design entails the emergence of a large number of connected components which is a universal feature of grids with a characteristic power-law degree distribution~\cite{RosasCasals:2007td,Bompard:2009ga,Brummitt:2013jj,Daqing:2014bp,Albert:2004bw,Wang:2011js,Sole:2008cv}.}
%
There are five major power plants in Lebanon. Transmission and distribution substations are such that they receive power from all of the major power plants, and so, if all but one such plant is hit, the entire network can still receive power. Because of this redundancy, the only way to achieve total failure is through the obvious choice of attacking all five power plants.
%
\edited{
To exclude this obvious scenario from our contingency analysis, the data received directly from the Lebanese Ministry of Energy and Water does not include all five power plants from the associated graph model. This renders our contingency analysis completely focused on the irredundant nodes constituting transmission and distribution substations, where the corresponding graph representation of the power grid consists of several connected components. }
%
\secondedited{
We should note that in the original dataset these five vertices were assigned a tag directly by the Ministry of Energy and Water, which described them as redundant. Therefore no extra computational work was done to identify them. In the event that the power network fails to manifest a large number of connected components that permit for the kind of distribution we are adopting in the present manuscript, one can attempt to distribute/parallelise the low-level graph Betweenness Centrality and Single Source Shortest Paths algorithms employed, using, for example, a number of distributed and parallel tools available in~\cite{Bertolucci16,Djidev14,Edmonds10,Jin10,Kumbhare14,Redekopp13,Solomonik13}, to cite a few.}
%
Our analysis simulates attacks on both transmission as well as distribution substations. It also pursues inter-dependencies as failures propagate within a single connected component, and addresses the overall loss of connectivity to all subscribers as a result. Finally, our analysis adopts the idealised (and simplified) view that capacity across the transmission lines is never jeopardised, and thus failure of the network is a result of attacks on substations only.

\thirdedited{
Our resulting network model consists of an undirected, unweighted graph\footnote{Please note that this bears no impact on the performance if the graph is directed. First off, all of the algorithms used (e.g. strongly connected component and betweenness centrality) have variants that can tackle directed graphs. Moreover, these variants have the same work complexity as those for the undirected graph.} with $679965$ edges, representing transmission/distribution lines, and $899162$ vertices, representing transmission/distribution substations. This is one order of magnitude larger than the graph treated in \cite{2000Natur.406..378A}, and is a consequence of the fact that the given Lebanese power grid representation captures extremely fine grained spatial coordinates, yielding all distribution substations no matter how minor. Our distribution analysis of the degrees of vertices confirms that indeed, the resulting graph exhibits a power law distribution (Fig. \ref{fig:degree-vertices}).}
%Also, our analysis is holistic in the sense that it simulates attacks on both transmission as well as distribution substations.
%insert figure here

\begin{figure}[!tbp]
  \centering
     {\includegraphics[scale=0.35]{bench/generated/degree-counter-crop.pdf}}
  \caption{Degree of vertices}
    \label{fig:degree-vertices}
\end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/frequency20-crop.pdf}
\caption{Distribution of components w.r.t. the number vertices (range from $2$ to $20$ vertices per component)}
\label{fig:vertices-components-20}
\end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/frequencyall-crop.pdf}
\caption{Distribution of components w.r.t. the number vertices}
\label{fig:vertices-components-all}
\end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/frequency-selected-crop.pdf}
\caption{Distribution of components w.r.t. the number vertices (range from 40 to 140 vertices per component)}
\label{fig:vertices-components-40}
\end{figure}

% Because of this redundancy, the only way to achieve total failure is through the obvious choice of attacking all ten power plants. To exclude this obvious scenario from our contingency analysis, we remove all ten power plants from the associated graph model. This renders our contingency analysis completely focused on the irredundant nodes of the power grid. What remains are the high to medium step down stations that transform high voltage to medium voltage nodes as follows. As treated in \cite{2000Natur.406..378A}, a distribution substation is one that receives power from a single, high voltage transmission line, and distributes this on smaller voltage to consumers. However, when modeling the power grid here, we make no distinction between nodes representing transmission substations versus distributing substations. Since our distributed algorithm is aimed for big graphs, an exhaustive analysis of the power grid should be attempted, and as such, we preserve the power grid almost in its entirety with a few exceptions.

\subsection{Random and Cascading Contingency Analysis}

Our baseline approach for navigating through node attacks follows that of \cite{2000Natur.406..378A}. A high level representation of the algorithm for implementing the random and cascading attacks is depicted in Algorithm~\ref{algo:random-cascading}. It captures both notions of node attack, followed by an evaluation of the loss in power flow capacity across the entire network.
%\begin{lstlisting}[language=java]
%attackGraph(Graph graph) {
%   for(i = 0 until |V|) {
%      select victim vertex v
%      remove vertex v
%      update loss with respect to v
%   }
%}
%\end{lstlisting}
%
\begin{algorithm}[ht]
\KwIn{Graph graph = (V, E)}
\KwOut{random and cascading attacks of the input graph}
    \SetKwFunction{FMain}{attackGraph}
    \SetKwProg{Fn}{Function}{:}{}
    \Fn{\FMain{graph}}{
        \For{$i \gets 0$ \KwTo $|V|$}
        {
        select victim vertex v\;
        remove vertex v\;
        update loss with respect to v\;
        }
}
\textbf{End Function}
 \caption{\edited{Random and cascading attacks}}
 \label{algo:random-cascading}
\end{algorithm}

Here, a victim vertex is chosen upon an examination of some measure of its centrality. Centrality is a quantitative measure that aims at revealing the importance of a node. Several indices of centrality tackled in the literature are based on geometric, spectral, as well as path-based measures. We refer the reader to \cite{BoldiVigna14} for a comprehensive survey. For undirected graphs, the geometric measure related to the indegree of a given node, as well as the path-based measure based on its betweenness, have both yielded a contextual understanding of connectivity loss in power grids (\cite{2000Natur.406..378A, JinAl10}). In line with this body of work, we select the victim vertex according to one of the following four scenarios:
\begin{itemize}
\item \emph{Random attacks}: a vertex is selected at random.
\item \emph{Attacks based on geometric centrality measure}: the vertex with the highest degree is selected.
\item \emph{Attacks based on path-based centrality measure}: the vertex with the highest betweenness centrality is selected. The betweenness centrality of the nodes is only computed once (i.e., is not be updated after removing a vertex).
\item \emph{Cascading attacks}: This is similar to the betweenness centrality scenario; however, the betweenness centrality of the nodes is updated at each iteration (i.e., after removal of a victim vertex).
\end{itemize}
Algorithm~\ref{algo:scenario-selection} depicts a more detailed snippet captures these four scenarios.
%\begin{lstlisting}[language=java]
%Compute betweenness centrality for each v in V
%
%attackGraph(Graph G = (V,E), bool Cascading) {
%   for(i = 0 until |V|) {
%      select victim vertex v in V
%      remove vertex v
%      if Cascading == TRUE
%         for each node not removed so far
%            recompute betweenness centrality
%      update loss with respect to v
%   }
%}
%\end{lstlisting}

\begin{algorithm}[ht]
\KwIn{Graph graph = (V, E), boolean cascading}
Compute betweenness centrality for each v in V\;
    \SetKwFunction{FMain}{attackGraph}
    \SetKwProg{Fn}{Function}{:}{}
    \Fn{\FMain{graph,  cascading}}{
        \For{$i \gets 0$ \KwTo $|V|$}
        {
         select victim vertex $v$ in $V$\;
               remove vertex $v$\;
        \If{$ cascading $}
            {
             \For{each node $u$ not removed so far}
             {
                 recompute betweenness centrality for $u$\;
             }
            }
      update loss with respect to $v$\;
        }
}
\textbf{End Function}
 \caption{\edited{Scenario-based victim selection}}
 \label{algo:scenario-selection}
\end{algorithm}

The choice behind node betweenness centrality is motivated by the following. Let $\sigma(s,t)$ denote the number of shortest paths between two vertices $s$ and $t$, and let $\sigma(s,t \left \vert \right. v)$ denote the number of shortest paths between $s$ and $t$ that pass through $v$. A most recent betweenness centrality (BC) score of $v$ is suggested by Brandes in \cite{Brandes01} as follows:
\begin{equation}
C(v) = \sum_{s,t \in V} \frac{\sigma(s,t \left \vert \right. v)}{\sigma(s,t)}
\label{brandesBC}
\end{equation}
where this assumes $s \neq t \neq v$, and considers $\sigma(s,s)=1$, $\sigma(s,t \left \vert \right. v)=0$ if $v = s$ or $v=t$, and $0/0 = 0$. With this definition, a high centrality score $C(v)$ implies that a vertex can reach others on relatively short paths, or that a vertex lies on a large number of shortest paths connecting other vertices. This particular choice of index is the basis for a quadratic running time, linear space algorithm by Brandes which improves on former cubic running time algorithms. The algorithm follows an accumulation technique that invokes a single-source shortest paths (SSSP) algorithm in multiple iterations starting from each vertex in the graph whose score needs to be produced. When the graph is unweighted, SSSP can be solved using breadth first traversals. To illustrate, let
\begin{equation*}
\gamma(s, t \left \vert \right. v) = \frac{\sigma(s, t \left \vert \right. v)}{\sigma(s,t)}
\end{equation*}
denote the dependency of $s$ and $t$ on $v$, captured by the ratio of shortest paths between $s$ and $t$ that go through $v$. Also, let
\begin{equation*}
\gamma(s \left \vert \right. v ) = \sum_{t \in V} \gamma(s,t \left \vert \right. v )
\end{equation*}
denote the dependency of $s$ on $v$, which is the sum of dependencies of $s$ and $t$ on $v$ for all possible targets $t$.
We then have the following main result:
\begin{equation*}
\gamma(s \left \vert \right. v ) = \sum_{w: (v,w) \in E \land d(s,w) = d(s,v)+1} \frac{\sigma(s,v)\times (1+\gamma(s\left \vert \right. w))}{\sigma(s,w)}
\end{equation*}
where $d(s,w)$ is the shortest path length from $s$ to $w$ \cite{Brandes01}. The resulting algorithm now revolves around two steps:
\begin{enumerate}
\item For each $s \in V$, perform a breadth first traversal of the graph. In each traversal, compute the number of shortest paths from $s$ to all $t \in V$ going through a given other node $v \in V$.
\item After each traversal, accumulate $\gamma (s,t \left \vert \right. v)$ into $\gamma(s \left \vert \right. v)$.
\end{enumerate}

\subsection{Connected Components: A localised view}
In contrast to the relatively smaller graphs treated in the literature on power grid resilience analysis, we have maintained the bulk of the power grid down to the finest spatial coordinates. Our graph with $\left \vert V \right \vert \approx 9 \times 10^6 $ represents a challenging size for serial computers and opportunities for distributed algorithm design ought to be explored. The fact that the high voltage power plants have been removed from the grid representation render the graph a disconnected one, consisting of 199004 connected components, where each component consists of transmission/distribution substations as nodes, and transmission lines between them as edges. Our Spark algorithm will distribute the work over these components, and for this, we begin by explaining the contextual implications of a decentralised version of the grid as such. This distribution of the network leads us to readdress the connectivity/loss and betweenness centrality in a manner that explores the effects of local changes on the global graph properties. We address those two issues separately below.

\subsubsection{Local versus global connectivity loss}
In \cite{2000Natur.406..378A}, the notion of loss manifests itself when less and less paths connect a power plant to a distribution station. In contrast, the Lebanese power grid has adapted to decades of power cuts that have resulted in a significant dependence on diesel power generators which, in theory, can connect to any transmission substation, say, within districts, or distribution substations, at the level of neighborhoods. These diesel generators are mobile and so their location may remain ``unknown'' to an attacker, and can be reasonably easily replaced. As such, the notion of loss in our scenario associates with attacks on transmission/distribution substations, and the removal of transmission lines between them, excluding the status of power plant generators or diesel generators. A node retains a power supply in the network so long as there exists a path of edges connecting it to any other node. Each time a victim vertex is attacked, all of its incident edges are removed along with it. As a result, we adopt the following definitions for connectivity and its loss. Given an undirected graph $G=(V,E)$, we define its connectivity as follows:
$$
\mathtt{connectivity}(G) = \sum_{v \in V} \mid \mathtt{reachable}(v) \mid,
$$
where $\mathtt{reachable}(v)$ is the set of all reachable nodes from $v$ via some path.
This notion can be expanded as follows, where $G$ denotes an undirected graph:
\begin{equation}
\begin{array}{llr}
\mathtt{connectivity}(G) && \\
=  \mid V \mid \times \mid V-1 \mid & \mbox{if $G$ is connected,}& \\
 =  \sum_{i \in \{1,\ldots,n\}} \mathtt{connectivity}(G_i) & \mbox{otherwise.}&
\end{array}
\label{def}
\end{equation}
The following formula captures the percentage of loss as a result of attacking (removing) a node $x$ from $G$:
$$
\mathtt{loss}(G, x) = 1 - \frac{\mathtt{connectivity}(G \setminus \{x\})}{\mathtt{connectivity}(G)}
$$
where $G \setminus \{x\}$ is a graph defined by removing vertex $x$ in $G$ and all of its incident edges. For simplicity, we will track only the numerator appearing in this definition and consider hereafter that $\mathtt{loss}(G,x) = \mathtt{connectivity}(G)-\mathtt{connectivity}(G \setminus \{x\})$. The following proposition reveals how the local loss within a component translates to global loss:
\begin{proposition}
Let $G$ denote an undirected graph, and let $G_i$ denote the connected component to which $x$ belonged prior to its removal from $G$. We then have:
\begin{equation}
\begin{array}{ll}
\mathtt{connectivity}(G) - \mathtt{connectivity}(G \setminus \{x\}) &\\
= \mathtt{connectivity}(G_i) - \mathtt{connectivity}(G_i \setminus \{x\})
\end{array}
\label{loss}
\end{equation}
where $\mathtt{connectivity}(G_i \setminus \{x\})$ can be computed as in Eq. (\ref{def}) above.
\label{connectivity}
\end{proposition}
\begin{proof}
Since $G_i$ is a connected component of $G$ and $x \in G_i$, we have:
\begin{equation*}
\begin{array}{ll}
\mathtt{connectivity}(G \setminus \{x\})  &  \\
 =  \mathtt{connectivity}(G_i \setminus \{x\}) + \mathtt{connectivity}(G \setminus G_i)&\\
 =  \mathtt{connectivity}(G_i \setminus \{x\}) + \sum_{j = 1, j \neq i}^{n} \mathtt{connectivity}(G_j)&
\end{array}
\end{equation*}
We now use this equation in:
\begin{equation*}
\begin{array}{ll}
\mathtt{connectivity}(G) - \mathtt{connectivity}(G \setminus \{x\})  &  \\
 =  \sum_{j = 1}^{n} \mathtt{connectivity}(G_j) & \\
- \left ( \mathtt{connectivity}(G_i \setminus \{x\}) + \sum_{j = 1, j \neq i}^{n} \mathtt{connectivity}(G_j)\right)& \\
 =  \mathtt{connectivity}(G_i) - \mathtt{connectivity}(G_i \setminus \{x\})
\end{array}
\end{equation*}
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsubsection{Local versus global centrality}

We now express the relationship between the global and local centrality measures for a given node within its connected component. We begin with the measure denoting the degree of a given vertex:
\begin{proposition}
Let $\mathtt{CC}(G) = \{G_1, \ldots, G_n\}$ denote the connected components of the undirected power grid graph $G$. Let $\deg_{G}(v)$ denote the degree of a node $v$ in $G$, and $\deg_{G_i}(v)$ denote its degree in $G_i$, for some connected component $G_i \in \mathtt{CC}(G)$ containing $v$. We then have:
\begin{equation*}
\deg_{G}(v) = \deg_{G_i}(v).
\end{equation*}
\label{degs}
\end{proposition}
\begin{proof}
The proof is straightforward: As $G$ is undirected, the edges incident on $v$ all belong to its connected component, and no edge outside its component can be incident on $v$.
\end{proof}
We now address the betweenness centrality measure:
%Given a graph $G$, the betweeness centrality of a node $v$ is equal to $\mathtt{bc}_G(v) = \sum_{s \neq v \neq t} \frac{\sigma_{st}(v)}{\sigma_{st}}$, where $\sigma_{st}$ is the total number of shortest paths from node $s$ to node $t$ and $\sigma_{st}(v)$ is the number of those paths that pass through $v$.
\begin{proposition}
Let $\mathtt{CC}(G) = \{G_1, \ldots, G_n\}$ denote the connected components of the undirected power grid graph $G$. Let $C_{G}(v)$ denote the BC score of a node $v$ in $G$, and $C_{G_i}(v)$ denote its BC score in $G_i$, for some connected component $G_i \in \mathtt{CC}(G)$ containing $v$. We then have:
\begin{equation*}
\mathtt{C}_{G}(v) = \mathtt{C}_{G_i}(v).
\end{equation*}
\label{BCs}
\end{proposition}
\begin{proof}
Let $s$, $t$, and $v$ denote distinct vertices in $V$. Since $G$ is undirected, we have $G_i \cap G_j = \emptyset$, $\forall i \neq j$. Hence, if $\exists i$ such that $s, t \in G_i$, then $\sigma(s,t)$, representing the number of paths between $s$ and $t$ in $G$, is identical to the number of paths between $s$ and $t$ in the subgraph $G_i$. Otherwise, $\sigma(s,t) = 0$. Similarly, if $\exists i$ such that $s, t, v \in G_i$, then $\sigma(s,t \left \vert \right. v)$, representing the number of paths between $s$ and $t$ in $G$ that pass through $v$, is identical to the number of paths between $s$ and $t$ in the subgraph $G_i$ that pass through $v$. Otherwise, $\sigma(s,t \left \vert \right. v) = 0$. Using Eq. \ref{brandesBC} for the BC score, we obtain that $\mathtt{C}_{G}(v) = \mathtt{C}_{G_i}(v)$ in all cases.
\end{proof}

\subsection{Spark-based Implementation}
We now present our Spark-based algorithm, guided by the propositions above. The many connected components of the power grid provide for an inherent data distribution of the graph which, as we will see in Sec. \ref{results}, allows for a balanced workload as well as locality of reference. Spark employs a storage abstraction called Resilient Distributed Datasets (RDDs). Each RDD is a distributed, fault-tolerant vector on which we can perform a set of vectorised operations. One can define a data partitioning scheme on a RDD. The execution engine can co-schedule tasks on those RDDs to avoid data movement. In what follows, we refer to threads that may be referenced within one single multithreaded machine or across several machines.
%
%Spark~\cite{spark} is new programming model supported by an execution engine for big data processing. Spark is based on RDD (Resilient Distributed Data Set). RDDs are big parallel collections that can be distributed across a cluster. RDDs are created through parallel transformations (e.g., map, group by, filter, join, create from file systems). Moreover, RDDs can be cached (in-memory) by allowing to keep data sets of interest in Memory across operations and thus contribute to a substantial speedup. At the same time, Spark uses lineage to support fault tolerance, i.e., record all the operations/transformations that yield to create RDDs from a source data. That is, in case of failure, an RDD can be reconstructed given the transformation functions yielding to that RDD. Additionally, after creating RDDs, it is possible to do analytics on them by running actions on them such as count, reduce, collect and save. Note that all operations/transformations are lazy until you run an action. Then, the Spark execution engine pipelines operations and finds an execution plan. GraphX~\cite{graphx} is a platform built on top of Spark that provides APIs for parallel and distributed processing on large graphs.
%
\begin{enumerate}
\item Given a file (read from local or distributed file system, i.e., HDFS) containing information about the graph, build the corresponding graph RDD. Here, the input file is partitioned for distribution over the multiple threads. GraphX in Spark represents the graph internally as a pair of vertex and edge collections built on RDDs.
\item Compute the connected components on the graph RDD using GraphX's built in strongly connected component distributed kernel. This creates an RDD, \texttt{rddCC}, of items, where each item corresponds to a connected component.
\item Partition \texttt{rddCC} into several $p$ partitions, for $p = 4, \ldots, 32$, $p \leftarrow 2\times p$. Each partition is allocated to a single thread, and contains a number of connected components.
\edited{
Note that, we use a customized partitioning scheme to get balanced partitions, i.e., large strongly connected components are spread among different partitions.}

\item The connected components within each partition are now distributed over the multiple threads on a single core (or multiple cores on a single thread). For each connected component, a thread $t$ selects a victim node according to one of the four scenarios, and locally updates the corresponding component by removing the victim vertex and its incident edges. Only the fourth scenario stipulates an update on the BC score of each remaining node. A tuple $(\rho,\lambda)$ is appended to the end of a list ${\bf L}_t$. Here, $\rho$ denotes its rank in the removal process if the vertex has been chosen in random order. Otherwise, $\rho$ denotes its centrality (degree or load based). Also, $\lambda$ denotes the loss percentage associated with the given victim node.
\item Repeat Step 4 until all the nodes of a thread's connected component are identified.
\item All threads now communicate their pairs in the list ${\bf L}_t$ to each other.
\item In the case where vertices have been removed in random order, define a reduce operation that concatenates the generated lists into a unified list ${\bf L}$. Else, define a reduce operation that merges all the generated lists on $b$ into ${\bf L}$. This step is performed serially.
\item Simulate the attack on $G$ by removing the nodes in ${\bf L}$ starting from the head. For each node $x_i$ removed from ${\bf L}$, return $loss(G,x_i) = \sum_{j = 1}^{i} \lambda_i$. This step is also performed serially.
\end{enumerate}
We now have the following:
\begin{proposition}
The above algorithm is correct, and returns $loss(G\setminus {\bf L})$, where ${\bf L}$ denotes the list of all nodes attacked in $G$.
\end{proposition}
\begin{proof}
We first address Step 4. By Prop. \ref{degs} and \ref{BCs}, each local measure of a centrality by one thread is also the global measure, and as such, computing it locally yields the answer independently of the other connected components within the same thread or on other threads. Also by a direct result of Prop. \ref{BCs}, updating the betweenness centrality scores after reach removal in the cascading scenario requires information about the paths connecting each vertex to vertices only in its own local component and thus can be performed independently of other threads. This allows to parallelise the computation of all betweeness centralities after attacking nodes in the cascading scenario. At the end of Step 4, the loss upon removal of each vertex is computed locally on each connected component item. By Prop. \ref{connectivity}, this local loss computed independently of other threads also indicates the global loss.

We now address Step 7. If the scenario applied is the random hit scenario, concatenating the vertices from each thread violates no specific ordering on them, and thus one can proceed with the attacks as indicated by the concatenation. If the scenario applied relies on some measure of centrality, the merge procedure ensures that the locally ordered lists of vertices produced by each thread now yield a totally ordered sorted list on all the centrality measures, after which the attacks on nodes can begin to take place.

We conclude with Step 8. Write ${\bf L} = \{x_1,\ldots,x_n\}$, where $n = \left \vert L \right \vert$. Let $x$ and $y$ denote any two arbitrary nodes of $G$, and assume, without loss of generality, that $x$ was removed prior to $y$. Let $loss(G\setminus \{x,y\})$ denote the loss in the graph after removing $x$ followed by $y$. The losses returned in Step 7 are thus captured by $loss(G\setminus \{x_1,\ldots,x_n\})$:
\begin{itemize}
\item{Suppose that $x$ and $y$ do not belong to the same connected component. This can happen either when both of them are tackled by the same thread or otherwise. Whether or not the failures of $x$ and $y$ are happening simultaneously, the loss incumbent on removing $x$ is independent of the corresponding loss due to $y$, and so, $loss(G\setminus \{x,y\}) = loss (G \setminus \{x\}) + loss (G \setminus \{y\})$, i.e. the reduction operator on the individual losses can take place after the computation of individual losses locally in Step 4.}
\item{Otherwise, suppose that $x$ and $y$ belong to the same connected component. Then they certainly are tackled by the same thread. In this case, $x$ is failed before $y$ is failed, and so $loss(G\setminus \{x,y\})$ is computed correctly using the formula $loss(G \setminus \{x\}) + loss \left (\left(G \setminus \{x\}\right)\setminus \{y\}\right)$, which has already been computed locally in Step 4.}
\end{itemize}
\end{proof}

\secondedited{We are now ready to conclude the analysis of our parallel design and we follow the exposition in \cite{Jordan02, Parhami99} to define the following terms. We call a parallel algorithm {\it work-optimal} if it has the smallest possible work, defined to be the smallest possible sequential work divided by the total number of processors. This excludes the cost of partitioning, which occurs at the beginning and thus is counted towards pre-processing costs, as well as the resulting data shuffling, which we regard as being memory or communication operations, not computation. We call a parallel algorithm {\it work-time-optimal}, if in addition to being work-optimal, its time is best possible. In the following proposition, we show that our resulting parallel design is work-optimal according to the definition above. However, due to partitioning and data shuffling costs, our algorithm fails to be work-time optimal.}

\begin{proposition}
The parallelised block in Steps 1--6 of the above algorithm maintains optimal parallel work, and requires linear time communication costs and a constant number of synchronisation barriers in the BSP model. Particularly, let $p$ denote the number of threads, $W_s$ denote the serial work of the algorithm in Steps 1--6, $g$ denote the machine-dependent cost in flops to communicate one data work, and $\ell$ denote the machine-dependent cost in flops for all threads to synchronise. We then have:
\[
T_p = \Theta\left(\frac{W_s}{p}\right) + \Theta \left(\frac{\left \vert V \right \vert}{p}\right) + 2 \ell \quad \mbox{flops.}
\]
\end{proposition}
\begin{proof}
Here, we are addressing the costs of the parallelised block in Steps 1--6. In Steps 1 and 2, the data is distributed equally among all partitions, and within each partition, the Spark scheduler assigns available threads to the tasks required by Step 4. Let $W_s$ denote the serial work required by the algorithm in Steps 1--6. Because Step 4 engages all of the threads on the equally distributed data, its cost is accounted for by $\Theta\left(\frac{W_s}{p}\right)$.
%
\edited{A parallel algorithm is work-optimal if it does not run more instructions than the sequential version and as such our algorithm is work-optimal.}
%
Step 6 is a communication step in which all threads in one partition communicate a pair of integers associated with each vertex they have been tasked with, and so, by the balanced data distribution of the earlier steps, the communication cost is $\Theta \left (g \cdot \frac{\left \vert V \right \vert}{p} \right)$ flops. The only synchronisation barriers needed are at the end of supersteps 1--4, and after the communication step in 6. This concludes the proof.
\end{proof}

\subsection{Spatial Correlation Analysis}
\thirdedited{
Here we study the spatial correlation between the targeted nodes in the cascading attacks scenario in an attempt to uncover any underlying spatial pattern. For this purpose, and using the geocoding of the input vertices, we examine the subset of nodes $F$ whose removal lead to a $90 \%$ connectivity loss. We are interested in measuring the the relation between failures separated by some given distance $r$. Following closely the spatial analysis of the 1996 blackout of Western Systems Coordinating Council area conducted in \cite{DaqingAl14}, we adopt the spatial correlation function $C(r)$ that measures the relation between failures that are $r$ distance apart:
\begin{equation}
C(x) \propto \frac{\sum_{{i,j}\in F} (x_i- \bar{x} )(x_j - \bar{x}) \delta(r_{ij} - r)}{\sum_{i,j \in F} \delta (r_{ij} - r)},
\end{equation}
Here, $x_i=1$ if node is failed, and $0$ otherwise. Also, $\bar{x}$ denotes the average number of cascading failures over the whole network, $\delta$ denotes the function that singles out the $j$ nodes that are $r$ distance apart from $i$, and $r_{ij}$ denotes the Euclidean distance between nodes $i$ and $j$. Positive values for $C(r)$ indicate a tendency of failures to be close to each other, while negative values indicate anti-correlations.}

%%% end


\section{Interface Design}
\label{theory:interface-design}
Interfaces in a virtual environment comes with a new set of challenges when compared to a traditional interface designed for a screen, these will be discussed in this section.
When designing an interface it's important to evaluate the speed of selection. Fitts' law is a fundamental and proven way to evaluate pointing to real-world objects by measuring the distant to the object and its size\cite{interface:Fitts1954}.

This is however based on a real-world scenario, which does not translate into a virtual environment, were the user need a tool to interact with objects. Despite these differences, Fitts' law can be applied to pointing in a virtual environment using the following formula  \cite{interface:card1978evaluation}
\begin{equation}
ID = log(2) * ( 2 * D / W )
\end{equation}

where ID is the index of difficulty, D is the distance from startingpoint to the middle of the correct target. W is the width of the target (caluculated on the axis where the pointer will travel).

\subsection{Interface Design for Virtual Reality}
When designing UI for VR there are new challenges and choices to make in order to please the user and keep the functionality of the UI. The interface of a VR system is explained in Sherman's extensive book about VR\cite{interface:sherman2002understanding}
 \begin{quote}
   The access point through the boundary between the recipient and the virtual world is the user interface
 \end{quote}
 Sherman explains how virtual worlds are designed for a specific medium and interface and later adapted to facilitate for different user interfaces. This often brings a lower quality to the adapted system according to Sherman who later concludes that the critical part of the process lies in selecting a suiting medium for the intended goal and content\cite{interface:sherman2002understanding}.

\subsubsection{Placement and Head-Up Display}
In order to present information to the user in VR approaches like in-world UI's or Heads-Up-Displays (HUD) can be appended. An in-world UI act like a physical object in the VE and has a world-related position. An example of a in-world UI is a selector\label{theory:toolsandtech:selector} which has a position relative to the selected object. Interactions with this type of UI can be achieved using techniques like direct manipulation\label{theory:toolsandtech:direct} and ray-casts\label{theory:toolsandtech:raycast}. A HUD is a interface that is attached to the user's field of view and does not have a position related to the VE. HUD usage in VR applications consist mostly from displaying information about the user, virtual object or the environment\cite{interface:sherman2002understanding}.


\section{Classifier neural architectures}\label{sec:Architectures}

Risk estimators are generally classifier agnostic.  They can be coupled with whatever learning paradigm imparts the most advantageous inductive bias. When labeled data is limited (including being non\-/existent for some classes), transfer learning often improves a classifier's performance by leveraging portable feature information learned from other (labeled) datasets.

This section proposes two 20~newsgroups neural classifier architectures.  Section~\ref{sec:Architectures:LSTM}'s LSTM model only makes limited use of transfer learning but is more flexible overall.  Section~\ref{sec:Architectures:ELMo}'s ELMo~model depends almost entirely on transfer learning but is more rigid. A comparison of each architecture's empirical results is provided in Section~\ref{sec:ExperimentalResults}.

\subsection{End-to-end LSTM network}\label{sec:Architectures:LSTM}

This architecture is the ``classic'' deep recurrent NLP classifier. Document tokens are encoded using an embedding (e.g,~GloVe). These dense representations are input to a recurrent block composed of one or more bidirectional LSTM layers.  The final recurrent block output is passed to a feedforward network with one or more hidden layers and ReLU activations.

This model's only transferred knowledge is the embedding matrix. All recurrent and feedforward parameters must be learned from scratch.  Given the limited labeled data, this may be onerous, but as a positive, this paradigm allows the learner significant flexibility.  We experimented with freezing and unfreezing the embedding weights, but saw no major performance difference with either approach.

\subsection{ELMo preprocessed vectors}\label{sec:Architectures:ELMo}

The traditional word embedding model treats each token as context independent and yield token-specific representations. Proposed in~2018 by Peters\etal~\cite{Peters:2018}, ELMo (\underline{e}mbeddings for \underline{l}anguage \underline{mo}dels) extends the word embedding model such that each token representation is \textit{context dependent} --- a function of the current token as well as the entire preceding input.

ELMo uses unsupervised learning allowing it to be pretrained on a large scale.  ELMo representations easily integrate into many existing NLP~architectures, and often can be swapped for GloVe or other word embeddings.

The Allen Institute for AI provides multiple pretrained ELMo models.  We selected the best performing ELMo network --- the original, complete architecture trained on the extended 5.5~billion token corpus, specifically 1.9~billion tokens from Wikipedia and 3.6~billion from the WMT~monolingual news crawl. At over 93~million parameters, combining this ELMo model with long 20~newsgroups documents causes GPU memory overflow errors on the University of Oregon's K80~GPUs.  This limited our of use of ELMO to just creating a single static vector for each document. In essence, ELMo serves as a fixed ``document embedding.''

The ELMo embedder model has three layers --- a character CNN (for improved unknown word robustness) followed by two bidirectional LSTMs.  Each layer's output is a 1,024\-/dimension vector.  Given an input stream of $m$~tokens, the embedder's output is a tensor of size~${\langle \textnormal{\#Layers} \times d_{\textnormal{layer}} \times \textnormal{\#Tokens} \rangle}$ --- in this case~${\langle 3 \times 1024 \times m \rangle}$.

We used R\"{u}ckl\'{e}\etal's~\cite{Ruckle:2018} paradigm where a sentence representation is formed by taking the minimum, maximum, and average value along each layer's output dimension for a tensor like above. Therefore, our model turns each document into a vector of dimension

\begin{equation*}
  \abs{\set{\max,\min,\textnormal{avg}}} \cdot \textnormal{\#Layers} \cdot d_{\textnormal{layer}} = 3 \cdot 3 \cdot 1024 = 9,216\text{.}
\end{equation*}

Each aforementioned static document representation is input into a feedforward network with two hidden layers of dimension~300 and ReLU~activations. This feedforward block contains the only volatile parameters making each epoch very fast~(<500ms in our experiments).\footnote{This ELMo\-/based architecture was first proposed by Hsieh\etal~\cite{Hsieh:2018}.}

\subsection{Implementation}

Irrespective of the classifier architecture, each newsgroup document was tokenized using~\texttt{nltk}. Both architectures are implemented in PyTorch with the source code available on~\href{\sourceCode}{Github}.  In our implementation, the LSTM~architecture is the default. To instead use preprocessed ELMo~vectors, set command line flag~\texttt{\mbox{\-/\-/preprocessed}}.


\subsection{PostProcessor}
\label{sec:models_postProcessor}
A Post-Processor (PP) can be considered as an action performed on a set of data
or other type of objects.
%
Most of the post-processors contained in RAVEN, employ a mathematical operation
on the data given as ``input''.
%
RAVEN supports several different types of PPs.

Currently, the following types are available in RAVEN:
\begin{itemize}
  \itemsep0em
  \item \textbf{BasicStatistics}
  \item \textbf{ComparisonStatistics}
  \item \textbf{ImportanceRank}
  \item \textbf{SafestPoint}
  \item \textbf{LimitSurface}
  \item \textbf{LimitSurfaceIntegral}
  \item \textbf{External}
  \item \textbf{TopologicalDecomposition}
  \item \textbf{DataMining}
  \item \textbf{ParetoFrontier}
  \item \textbf{Metric}
  \item \textbf{CrossValidation}
  \item \textbf{ValueDuration}
  \item \textbf{FastFourierTransform}
  \item \textbf{SampleSelector}
  \item \textbf{Validation}
  \item \textbf{EconomicRatio}
  \item \textbf{HistorySetDelay}
  \item \textbf{HStoPSOperator}
  \item \textbf{HistorySetSampling}
  \item \textbf{HistorySetSync}
  \item \textbf{HistorySetSnapShot}
  \item \textbf{HS2PS}
  \item \textbf{TypicalHistoryFromHistorySet}
  \item \textbf{dataObjectLabelFilter}
  \item \textbf{TSACharacterizer}
\end{itemize}

The specifications of these types must be defined within the XML block
\xmlNode{PostProcessor}.
%
This XML node needs to contain the attributes:
\vspace{-5mm}
\begin{itemize}
  \itemsep0em
  \item \xmlAttr{name}, \xmlDesc{required string attribute}, user-defined
  identifier of this post-processor.
  %
  \nb As with other objects, this is the name that can be used to refer to this
  specific entity from other input XML blocks.
  \item \xmlAttr{subType}, \xmlDesc{required string attribute}, defines which of
  the post-processors needs to be used, choosing among the previously reported
  types.
  %
  This choice conditions the subsequent required and/or optional
  \xmlNode{PostProcessor} sub nodes.
  %
\end{itemize}
\vspace{-5mm}

As already mentioned, all the types and meaning of the remaining sub-nodes
depend on the post-processor type specified in the attribute \xmlAttr{subType}.
%
In the following sections the specifications of each type are reported.

%%%%% PP BasicStatistics %%%%%%%
\input{PostProcessors/BasicStatistics.tex}

%%%%% PP ComparisonStatistics %%%%%%%
\input{PostProcessors/ComparisonStatistics.tex}

%%%%% PP ImportanceRank %%%%%%%
\input{PostProcessors/ImportanceRank.tex}

%%%%% PP SafestPoint %%%%%%%
\input{PostProcessors/SafestPoint.tex}

%%%%% PP LimitSurface %%%%%%%
\input{PostProcessors/LimitSurface.tex}

%%%%% PP LimitSurfaceIntegral %%%%%%%
\input{PostProcessors/LimitSurfaceIntegral.tex}

%%%%% PP External %%%%%%%
\input{PostProcessors/External.tex}

%%%%% PP TopologicalDecomposition %%%%%%%
\input{PostProcessors/TopologicalDecomposition.tex}

%%%%% PP DataMining %%%%%%%
\input{PostProcessors/DataMining.tex}

%%%%%%%%%%%%%% ParetoFrontier PP %%%%%%%%%%%%%%%%%%%
\input{PostProcessors/ParetoFrontier.tex}

%%%%%%%%%%%%%% Metric PP %%%%%%%%%%%%%%%%%%%
\input{PostProcessors/Metric.tex}

%%%%%%%%%%%%%% Cross Validation PP %%%%%%%%%%%%%%%%%%%
\input{PostProcessors/CrossValidation.tex}

%%%%%%%%%%%%%% ValueDuration %%%%%%%%%%%%%%%%%%%
\input{PostProcessors/ValueDuration.tex}

%%%%%%%%%%%%%% FastFourierTransform %%%%%%%%%%%%%%%%%%%
\input{PostProcessors/FastFourierTransform.tex}

%%%%%%%%%%%%%% SampleSelector %%%%%%%%%%%%%%%%%%%
\input{PostProcessors/SampleSelector.tex}

%%%%% PP Validation %%%%%%%
\input{PostProcessors/Validation.tex}

%%%%% PP EconomicRatio %%%%%%%
\input{PostProcessors/EconomicRatio.tex}

%%%%% HistorySetDelay %%%%%%
\input{PostProcessors/HistorySetDelay.tex}

%%%%% HStoPSOperator %%%%%%
\input{PostProcessors/HStoPSOperator.tex}

%%%%% HistorySetSampling %%%%%%
\input{PostProcessors/HistorySetSampling.tex}

%%%%% HistorySetSync %%%%%%
\input{PostProcessors/HistorySetSync.tex}

%%%%% HistorySetSnapShot %%%%%%
\input{PostProcessors/HistorySetSnapShot.tex}

%%%%% HS2PS %%%%%%
\input{PostProcessors/HS2PS.tex}

%%%%% TypicalHistoryFromHistorySet %%%%%%
\input{PostProcessors/TypicalHistoryFromHistorySet.tex}

%%%%% dataObjectLabelFilter %%%%%%
\input{PostProcessors/dataObjectLabelFilter.tex}

%%%%%%%%%%%%%%% TSACharacterizer %%%%%%%%%%%%%%%%%%%
\input{PostProcessors/TSACharacterizer.tex}

%%%%%%%%%%%%%% InterfacedPostProcessors %%%%%%%%%%%%%%%%
% To be replaced by the PostProcessor Plugin
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\input{PostProcessors/InterfacedPostProcessors.tex}


\section{Introduction}
\label{sec:rul_intro}

Traditionally, maintenance of mechanical systems has been carried out based on scheduling strategies, nevertheless such strategies are often costly and less capable of meeting the increasing demand of efficiency and reliability \citep{Gebraeel2005, Zaidan2013}. Condition based maintenance (\gls{cbm}) also known as intelligent prognostics and health management (\gls{phm}) allows for maintenance based on the current health of the system, thus cutting costs and increasing the reliability of the system \citep{Zhao2017}. To avoid confusion, here we define prognostics as the estimation of remaining useful component life. The remaining useful life (\gls{rul}) of a system can be estimated based on history trajectory data, this approach which we refer here as data-driven can help improve maintenance schedules to avoid engineering failures and to save costs \citep{Lee2014}.

The existing \gls{phm} methods can be grouped into three different categories: model-based \citep{Yu2001} , data-driven \citep{Liu2009, Mosallam2013} and hybrid approaches \citep{Pecht2010, Liu2012}.

Model-based approaches attempt to incorporate physical models of the system into the estimation of the \gls{rul}. If the system degradation is modeled  precisely, model-based approaches usually exhibit better performance than data-driven approaches \citep{Qian2017}, nevertheless this comes at the expense of having extensive a priori knowledge of the underlying system and having a fine-grained model of such system (which usually involve expensive computations). On the other hand, data-driven approaches tend to use pattern recognition to detect changes in system states. Data-driven approaches are appropriate when the understanding of first principles of system operation is not comprehensive or when the system is sufficiently complex (i.e. jet engines, car engines, complex machinery) such that developing an accurate model is prohibitively expensive. Common disadvantages for the data-driven approaches are that they usually exhibit wider confidence intervals than model-based approaches and that a fair amount of data is required for training. Many data-driven algorithms have been proposed and good prognostics results have been achieved, among the most popular algorithms we can find artificial neural networks (\gls{ann}) \citep{Gebraeel2004}, support vector machine (\gls{svm}) \citep{Benkedjouh2013} and Markov hidden chains (\gls{mhc}) \citep{Dong2007}.

Over the past few years, data-driven approaches have gained more attention in the \gls{phm} community. A number of machine learning techniques, especially neural networks have been successfully applied to estimate the \gls{rul} of diverse mechanical systems. \glspl{ann} have demonstrated good performance when applied for modeling highly nonlinear, complex, multi-dimensional system without any prior expertise on the system's physical behavior \citep{Li2018}. While the confidence limits for the \gls{rul} predictions can not be naturally provided \citep{Sikorska2011}, the neural network approaches are promising on prognostic problems.

Neural networks for estimating the \gls{rul} of jet engines has been previously explored in \citep{Lim2016} where the authors propose a multi-layer perceptron (\gls{mlp}) coupled with a feature extraction (\gls{fe}) method and a time window for the generation of the features for the \gls{mlp}. In the publication the authors demonstrate that a moving window combined with a suitable feature extractior can improve the \gls{rul} prediction reported by other similar methods in the literature. In \citep{Li2018} the authors explore an even newer \gls{ann} architecture, the so-called convolutional neural networks (\glspl{cnn}), where they demonstrate that by using a \gls{cnn} without any pooling layers coupled with a time window the predicted \gls{rul} is further improved.

In this paper, we propose a novel framework for estimating the \gls{rul} of complex mechanical systems. The framework consists of a \gls{mlp} to estimate the \gls{rul} of the system at hand, coupled with an evolutionary algorithm for the fine tuning of data-related parameters, i.e. parameters that define the shape and quality of the features used by the \gls{mlp}. The publicly available NASA \gls{cmaps} dataset \citep{CMAPS2008} is used to assess the efficiency and reliability of the proposed framework. This approach allows for simple and small \glspl{mlp} to obtain better results than those reported in the current literature while using less computing power.

The remainder of this paper is organized as follows: The \gls{cmaps} dataset is presented in Section \ref{sec:rul_dataset}, then the framework and all of its components are thoroughly reviewed in Section \ref{sec:method}. The method is evaluated using the \gls{cmaps} dataset in Section \ref{sec:rul_eval}, a comparison with the state-of-the-art is also provided. Finally, our conclusions are presented in Section \ref{sec:conclusions}.


\chapter{Numerical Results}
\label{chap:results}

In this chapter we present a series of convergence and preconditioning experiments. The principle aim is to check the scalability performance of the preconditioned iterative methods for the MHD problem and the two decoupling schemes proposed in Chapter \ref{chap:precond}. All numerical experiments have been carried out in the Python programming language.

\section{Software}

Due to the complex nature of the MHD problem, a number of different libraries have been used both to discretise and then to solve the resulting systems.

The finite element software that was used to discretise \eqref{eq:mhd} was FEniCS \cite{wells2012automated}. The core libraries used within \fenics are DOLFIN \cite{LoggWells2010a,LoggWellsEtAl2012a} the problem-solving interface, FFC \cite{KirbyLogg2006a,LoggOlgaardEtAl2012a,OlgaardWells2010b} the compiler for finite element variational forms, FIAT \cite{Kirby2012a,Kirby2004a} the finite element  tabulator, Instant the just-in-time compiler, UFC \cite{AlnaesLoggEtAl2009a,AlnaesLoggEtAl2012a} the code generator and UFL \cite{AlnaesEtAl2012,Alnaes2012a} the form language.

Along with \fenics we have used a number of purely linear algebra libraries. The main one that has been used is PETSc4PY which is the python interface for PETSc \cite{petsc-web-page,petsc-user-ref}. PETSc has been used mainly for the iterative solvers as well as the blockwise preconditioning setup. The following packages were used to solve the precondition system: HYPRE \cite{falgout2002hypre} as a multigrid solver and for the sparse direct solvers we use UMFPACK \cite{Davis:2004:CPS:992200.992205,Davis:2004:AUV:992200.992206,Davis:1999:CUM:305658.287640,davis1997unsymmetric}, PASTIX \cite{henon2002pastix}, SuperLU \cite{superlu_ug99,li05} and MUMPS \cite{amestoy2000multifrontal,amestoy2001fully,amestoy2006hybrid}.

\section{Problem setup}

When numerically solving a set of linear or non-linear equations the problem needs to be initialised. By this we mean the mesh sequence, non-linear iteration stopping criteria and initial guess setup needs to be defined. In this small section we will briefly cover these aspects.

\paragraph{Mesh sequence} ~\\

To test the performance of both the preconditioners from  chapter \ref{chap:precond} and to validate the code produced by \fenics a sequence of meshes are required. Verifying the code  produces the correct convergence rates, we consider levels, $l$, of uniform grids. The levels define the number of edges between the nodes along the boundary edge to be  $2^l$. For example, consider the third level, $l = 3$, hence the grid generated is the $8\times 8$ grid given in figure \ref{fig:2Dmesh}.
\begin{figure}[h!]
  \centering
    \includegraphics[scale=.5]{../Results/Figures/dolfin_plot_1}
    \caption{Level $3$ grid on unit square domain}
  \label{fig:2Dmesh}
\end{figure}
The first column of all convergence and iteration tables below will show the grid level.

\paragraph{Stopping criteria} ~\\

Recall that both the Navier-Stokes equations as well as the full MHD problem are a set of non-linear equations. Section \ref{sec:nonlinear} outlines the process in which we linearise the problem and then solve for the updates. The stopping criteria that we enforce is the following:
\begin{itemize}
  \item NS: $\|\delta {u} \|_{\mathpzc{p}}+\|\delta p \|_{\mathpzc{p}} < \mbox{tol}$,
  \item MHD: $\|\delta {u} \|_{\mathpzc{p}}+\|\delta p \|_{\mathpzc{p}}+\|\delta {b} \|_{\mathpzc{p}}+\|\delta r \|_{\mathpzc{p}} < \mbox{tol}$,
\end{itemize}
where $\mathpzc{p}=\infty$ or $2$. The tolerance is stated for each example run within this thesis.

\paragraph{Initial guess} ~\\

To start the Picard iterations given in section \ref{sec:nonlinear} we require an initial guess. To form the initial guess, we solve the decoupled Stokes problem
$$
\left(
\begin{array}{cc}
A & B^T \\
B & 0 \\
\end{array}
\right)
\left(
\begin{array}{c}
u \\
p \\
\end{array}
\right)=\left(
\begin{array}{c}
f \\
0 \\
\end{array}
\right),
$$
then the Maxwell subproblem
$$\left(
\begin{array}{cc}
M-C & D^T \\
D & 0 \\
\end{array}
\right)
\left(
\begin{array}{c}
b \\
r \\
\end{array}
\right)=\left(
\begin{array}{c}
g \\
0 \\
\end{array}
\right).$$
Here the term $C$ corresponds the the coupling term using $u$ (the initial guess for the velocity field).

When solving these two systems, the convergence tolerance which we use is important. In particular, inaccurate solutions cause problems with non-homogeneous boundary conditions. This is because if the matrix system with the boundary conditions applied is not solve to a sufficient accuracy then there will be errors in the boundary data. When we solve for the updates we enforce homogeneous Dirichlet boundary conditions (i.e. zero boundary conditions) on the velocity ($u$), magnetic ($b$) and multiplier ($r$) fields. This therefore means that if the Stokes and Maxwell solves are not done accurately enough then when solving for the updates errors will remain within the boundary conditions. Thus the errors norms for the full MHD system will be capped by the accuracy of the Stokes and Maxwell solve for the initial guess.

\section{Navier-Stokes equations}
\label{sec:NS_validation}
Before considering the full discretised MHD problem, we require that the Navier-Stokes subproblem performs as expected in terms of the error estimates and preconditioner scalability. To check the validity of the FEniCS code, we introduce two smooth test problems, the first in 2D and the second in 3D.

\subsection{2D smooth solution}

Consider the domain $\Omega =(0, 1)^2$ with boundary $\partial \Omega$. Since we are considering a purely Dirichlet problem then $\Gamma_D = \partial \Omega$ and $\Gamma_N = \emptyset$. Taking the kinematic viscosity to be $\nu = 1$, then we choose the source terms $\uu{f}$ from the analytical solution
\begin{subequations} \nonumber
\begin{eqnarray}\nonumber
\uu{u}(x,y) &=&
\begin{pmatrix}
\sin(x)\exp(x+y)+\cos(y)\exp(x+y)\\
-\sin(y)\exp(x+y)
\end{pmatrix},\\
\nonumber
p(x,y) &=& x^3\sin(y)+\exp(x+y).
\end{eqnarray}
\end{subequations}

Running the \fenics code produces the following table of results:
\begin{table}[h!] \small
\begin{center}
\begin{tabular}{cccccccc}
\hline
$l$ &    Dofs $\uu{u}_h/p_h$ & $\|\uu{u}-\uu{u}_h\|_{L^2(\Omega)}$ & order & $\|\uu{u}-\uu{u}_h\|_{H^1(\Omega)}$ & order &        $\|{p}-{p}_h\|_{L^2(\Omega)}$ & order \\
\hline
 3 &     578/81 &  2.2100e-04 &     - &  1.5645e-02 &     - &  7.2758e-03 &      - \\
 4 &    2,178/289 &  2.7598e-05 &     3.00 &  3.9133e-03 &     2.00 &  1.8190e-03 &      2.00 \\
 5 &    8,450/1,089 &  3.4465e-06 &     3.00 &  9.7822e-04 &     2.00 &  4.5025e-04 &      2.01 \\
 6 &   33,282/4,225 &  4.3072e-07 &     3.00 &  2.4455e-04 &     2.00 &  1.1341e-04 &      1.99 \\
 7 &  132,098/16,641 &  5.3836e-08 &     3.00 &  6.1137e-05 &     2.00 &  2.8639e-05 &      1.99 \\
 8 &  526,338/66,049 &  6.7291e-09 &     3.00 &  1.5284e-05 &     2.00 &  7.0146e-06 &      2.03 \\
\hline
\end{tabular}
\caption{Convergence table for 2D Navier-Stokes smooth solution - Picard tolerance 1e-10}
\label{tab:NS_2D_smooth}
\end{center}
\end{table}

\noindent For Taylor-Hood elements the expected convergence rates for the velocity field in the $L^2$ and $H^1$ integral norms are third and second order and for the pressure field the $L^2$ norm is second order. From table x\ref{tab:NS_2D_smooth} we can see that we obtain third order in $L^2$, second order FOR $H^1$ in the velocity fields and second order in $L^2$ for the pressures. This is precisely what we expect from Taylor-Hood elements.

\subsection{3D smooth solution}

The three dimensional set up is very similar as with the 2D case. This time the domain is the unit cube, i.e. $\Omega =(0, 1)^3$ with boundary $\partial \Omega$. Again the test problem has only Dirichlet boundary conditions, so that $\Gamma_D = \partial \Omega$ and $\Gamma_N = \emptyset$. As before the kinematic viscosity is $\nu = 1$, then the source term $\uu{f}$ is calculated from the analytical solution
\begin{subequations} \nonumber
\begin{eqnarray}\nonumber
\uu{u}(x,y,z) &=&
\begin{pmatrix}
-\exp(x+y+z)\sin(y) + \exp(x+y+z)\sin(z)\\
\exp(x+y+z)\sin(x) - \exp(x+y+z)\sin(z)\\
-\exp(x+y+z)\sin(x) + \exp(x+y+z)\sin(y)
\end{pmatrix}, \\
\nonumber
p(x,y,z) &=&  \exp(x+y+z) + \sin(y).
\end{eqnarray}
\end{subequations}
Running the 3D code produces table \ref{tab:NS_3D_smooth} which shows the errors and convergence rates.
\begin{table}[h!] \small
\begin{center}
    \begin{tabular}{cccccccc}
    \hline
    $l$ &    Dofs $\uu{u}_h/p_h$ & $\|\uu{u}-\uu{u}_h\|_{L^2(\Omega)}$ & order & $\|\uu{u}-\uu{u}_h\|_{H^1(\Omega)}$ & order &        $\|{p}-{p}_h\|_{L^2(\Omega)}$ & order \\
    \hline
    1 &    375/27 &  2.6211e-02 &     - &  4.5804e-01 &     - &  1.7725e+00 &      - \\
    2 &   2,187/125 &  3.2997e-03 &     2.99 &  1.1547e-01 &     1.99 &  2.8602e-01 &      2.63 \\
    3 &   14,739/729 &  4.1267e-04 &     3.00 &  2.8944e-02 &     2.00 &  4.0587e-02 &      2.82 \\
    4 &  107,811/4,913 &  5.1565e-05 &     3.00 &  7.2416e-03 &     2.00 &  6.4794e-03 &      2.65 \\
    5 &  823,875/35,937 &  6.4443e-06 &     3.00 &  1.8108e-03 &     2.00 &  1.2724e-03 &      2.35 \\
    \hline
    \end{tabular}
\caption{Convergence table for 3D Navier-Stokes smooth solution - Picard tolerance 1e-5}
\label{tab:NS_3D_smooth}
\end{center}
\end{table}

\noindent From the table we can see that the convergence rates are the same as the 2D test solution in the previous section for the velocity field. However, we see that for the pressure field we get a slightly higher than expected order, namely about 2.5. We have seen this trend for multiple different examples that have been run but never have we seen an order lower than 2.

\subsection{Least Squares Commutator}

Section \ref{sec:NSprecond} outlines the two main preconditioning techniques for the Navier-Stokes equations. In order to have a scalable preconditioner for the full MHD system we need to check the performance of both LSC and PCD for our Navier-Stokes subproblem.

First considered the LSC preconditioner. Recall that to apply the LSC preconditioner we need to apply
\begin{equation} \label{eq:NS_precond_results}
  \left(
\begin{array}{cc}
F & B^T \\
0 & M_{S}
\end{array}
\right)^{-1}
\end{equation}
where
$$M_{S} = (B Q^{-1} B^T)(BQ^{-1}FQ^{-1}B^T)^{-1}(B Q^{-1} B^T)$$
at each GMRES iteration. This amounts to solving systems associated with $F$ and $B Q^{-1} B^T$, which has been outlined in section \ref{sec:NSprecond}. We do this by using a direct solve for $F$ and approximate $B Q^{-1} B^T \approx B \, \mbox{diag}(Q)^{-1} B^T$ then use a direct solve. Since $B \, \mbox{diag}(Q)^{-1} B^T$ is singular (the constant vector is in the null space) then we apply a small shift to the diagonal to ensure the invertability for the direct solver. Using GMRES with a relative tolerance stopping criteria of 1e-5 produces table \ref{tab:LSC_2D} of iteration results.
\begin{table}[h!] \small
\begin{center}
    \begin{tabular}{cccccc}
    \hline
    l &    Dofs &  \multicolumn{4}{ c }{Average iterations}\\
     & $\uu{u}_h/p_h$ & $\nu=10$ &  $\nu=1$ &  $\nu=0.1$&  $\nu=0.01$ \\
    \hline
     1 &      50/9 &     5 &     5 &    6 &    8 \\
     2 &     162/25 &    10 &    10 &   10 &   21 \\
     3 &     578/81 &    16 &    16 &   15 &   30 \\
     4 &    2,178/289 &    25 &    24 &   24 &   31 \\
     5 &    8,450/1,089 &    48 &    47 &   46 &   39 \\
     6 &   33,282/4,225 &   103 &    86 &   77 &   63 \\
     7 &  132,098/16,641 &   190 &   190 &  141 &  131 \\
    \hline
    \end{tabular}
\caption{Iteration table for LSC preconditioner  for 2D example -  Picard tolerance 1e-10}
\label{tab:LSC_2D}
\end{center}
\end{table}

Table \ref{tab:LSC_2D} shows that as the grid level increases the number of iterations GMRES takes to converge to the set tolerance increases. This is a relatively unknown problem when using Taylor-Hood elements. This therefore requires us to consider using PCD as the Navier-Stokes subproblem preconditioner instead.

\subsection{Pressure Convection-Diffusion}

As for LSC an application of \eqref{eq:NS_precond_results} is applied at each GMRES iteration. However, for PCD the Schur complement approximation $M_S$ is given by:
$$M_S = A_p F_p^{-1}W.$$
Instead of applying the  PCD preconditioner exactly (direct solves) we will use approximate solves. We therefore use a single multigrid V-cycle to approximately solve both $F$ and $A_p$ and Conjugate Gradient (CG) \cite{hestenes1952methods} with a block Jacobi preconditioner and outer tolerance of 1e-5 for $W$. With the outer tolerance of GMRES the same as before we obtain the tables of iterations for both the 2D and 3D examples.

\subsubsection{2D example}
Running the 2D example gives table \ref{tab:PCD_2D}.
\begin{table}[h!] \small
\begin{center}
    \begin{tabular}{cccccc}
    \hline
    l &    Dofs &  \multicolumn{4}{ c }{Average iterations}\\
     & $\uu{u}_h/p_h$ & $\nu=10$ &  $\nu=1$ &  $\nu=0.1$&  $\nu=0.01$ \\
    \hline
      5 &     8450/1089 &    17 &  15 &  18 &  5053 \\
      6 &    33,282/4,225 &    18 &  15 &  18 &   144 \\
      7 &   132,098/16,641 &    18 &  15 &  18 &    41 \\
      8 &   526,338/66,049 &    18 &  15 &  18 &    40 \\
      9 &  2,101,250/263,169 &    18 &  15 &  18 &    40 \\
    \hline
    \end{tabular}
\caption{Iteration table for PCD preconditioner  for 2D example -  Picard tolerance 1e-10}
\label{tab:PCD_2D}
\end{center}
\end{table}

\noindent From table \ref{tab:PCD_2D} we can see that as the mesh level increases then the iteration numbers stay roughly constant. This is what we expect and require for the Navier-Stokes subproblem preconditioner. Also, the table shows iteration number for multiple different values of the kinematic viscosity $\nu$. Again as expected, as the viscosity decreases (i.e. the equations become more convection dominated) the iterations increase.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{3D example}

Running the 3D code produces the following table.
\begin{table}[h!] \small
\begin{center}
    \begin{tabular}{cccccc}
    \hline
    l &    Dofs &  \multicolumn{4}{ c }{Average iterations}\\
     & $\uu{u}_h/p_h$ & $\nu=10$ &  $\nu=1$ &  $\nu=0.1$&  $\nu=0.01$ \\
    \hline

    \hline
    \end{tabular}
\caption{Iteration table for PCD preconditioner  for 3D example -  Picard tolerance 1e-10}
\label{tab:PCD_3D}
\end{center}
\end{table}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Maxwell's equations}
\label{sec:Maxwell_validation}

As for the Navier-Stokes subproblem (section \ref{sec:NS_validation}), we consider a 2D and 3D test solution to validated the \fenics code for the Maxwell subproblem.

\subsection{2D smooth solution}
\label{sec:2D_maxwell}
Again, consider the domain $\Omega =(0, 1)^2$ with boundary $\partial \Omega$. Since we are considering a purely Dirichlet problem then $\Gamma_D = \partial \Omega$ and $\Gamma_N = \emptyset$. Now choose the source terms $\uu{g}$ that satisfy the analytical solution
\begin{subequations} \nonumber
\begin{eqnarray}\nonumber
\uu{b}(x,y) &=&
\begin{pmatrix}
\exp(x + y)\cos(x)\\
\exp(x + y)\sin(x) - \exp(x + y)\cos(x)
\end{pmatrix},\\
\nonumber
r(x,y) &=& \sin(2\pi x)\sin(2\pi y).
\end{eqnarray}
\end{subequations}

\begin{table}[h!] \small
\begin{center}
\begin{tabular}{cccccc}
\hline
l &    Dofs $\uu{b}_h/r_h$ & $\|\uu{b}-\uu{b}_h\|_{L^2(\Omega)}$ & order & $\|\uu{b}-\uu{b}_h\|_{H(curl,\Omega)}$ & order \\
\hline
1 &      48/25 &  9.3833e-02 &    - &  1.0696e-01 &       - \\
2 &     176/81 &  2.3350e-02 &    2.01 &  2.7088e-02 &       1.98 \\
3 &     672/289 &  5.8372e-03 &    2.00 &  6.4220e-03 &       2.08 \\
4 &    2,624/1,089 &  1.4597e-03 &    2.00 &  1.4586e-03 &       2.14 \\
5 &   10,368/4,225 &  3.6497e-04 &    2.00 &  3.5066e-04 &       2.06 \\
6 &   41,216/16,641 &  9.1246e-05 &    2.00 &  8.6688e-05 &       2.02 \\
7 &  164,352/66,049 &  2.2812e-05 &    2.00 &  2.1609e-05 &       2.00 \\
\hline
\end{tabular}
\caption{Convergence table for 2D Maxwell smooth solution - Magnetic field}
\label{tab:2D_maxwell_magnetic}
\end{center}
\end{table}
\begin{table}[h!] \small
\begin{center}
\begin{tabular}{cccccc}
\hline
l &    Dofs $\uu{b}_h/r_h$ & $\|{r}-{r}_h\|_{L^2(\Omega)}$ & order & $\|{r}-{r}_h\|_{H^1(\Omega)}$ & order\\
\hline
 1 &      48/25 &  2.7761e-01 &    - &  2.8932e+00 &     - \\
 2 &     176/81 &  4.3540e-02 &    2.67 &  9.3299e-01 &     1.63 \\
 3 &     672/289 &  4.8633e-03 &    3.16 &  2.5904e-01 &     1.85 \\
 4 &    2,624/1,089 &  5.6724e-04 &    3.10 &  6.6810e-02 &     1.96 \\
 5 &   10,368/4,225 &  6.9363e-05 &    3.03 &  1.6841e-02 &     1.99 \\
 6 &   41,216/16,641 &  8.6203e-06 &    3.01 &  4.2193e-03 &     2.00 \\
 7 &  164,352/66,049 &  1.0760e-06 &    3.00 &  1.0554e-03 &     2.00 \\

\hline
\end{tabular}
\caption{Convergence table for 2D Maxwell smooth solution - multiplier field}
\label{tab:2D_maxwell_multiplier}

\end{center}
\end{table}

For Maxwell equations we use second order \nedelec elements  of the first kind \cite{nedelec1980mixed} and polynomial elements of the same order for the magnetic and multiplier fields respectively. Therefore, we exact second order convergence in $\|\uu{b}-\uu{b}_h\|_{L^2(\Omega)}$, $\|\uu{b}-\uu{b}_h\|_{H(curl,\Omega)}$ and $\|{r}-{r}_h\|_{H^1(\Omega)}$ and third order in $\|{r}-{r}_h\|_{L^2(\Omega)}$. Tables \ref{tab:2D_maxwell_magnetic} and \ref{tab:2D_maxwell_multiplier} show that we do obtain the expected rates of convergence for both the magnetic and multiplier fields.

\subsection{3D smooth solution}

As for the 3D Navier-Stokes problem the domain is the unit cube, i.e. $\Omega =(0, 1)^3$ with boundary $\partial \Omega$. Again the test problem has only Dirichlet boundary conditions then $\Gamma_D = \partial \Omega$ and $\Gamma_N = \emptyset$. Using the anayltical solution
\begin{subequations} \nonumber
\begin{eqnarray}\nonumber
\uu{b}(x,y,z) &=&
\begin{pmatrix}
-\exp(x + y + z)\sin(y) + \exp(x + y + z)\sin(z)\\
\exp(x + y + z)\sin(x) - \exp(x + y + z)\sin(z)\\
-\exp(x + y + z)\sin(x) + \exp(x + y + z)\sin(y)
\end{pmatrix},\\
\nonumber
r(x,y,z) &=& \sin(2\pi x)\sin(2\pi y)\sin(2\pi z),
\end{eqnarray}
 the source term $\uu{g}$ is defined.
\end{subequations}
\begin{table}[h!] \small
\begin{center}
    \begin{tabular}{cccccc}
    \hline
l &    Dofs $\uu{b}_h/r_h$ & $\|\uu{b}-\uu{b}_h\|_{L^2(\Omega)}$ & order & $\|\uu{b}-\uu{b}_h\|_{H(curl,\Omega)}$ & order \\
    \hline
 1 &     436/125 &  5.3312e-02 &    0.00 &  3.5258e-01 &       0.00 \\
 2 &    2,936/729 &  1.4192e-02 &    1.91 &  8.9944e-02 &       1.97 \\
 3 &   21,424/4,913 &  3.5801e-03 &    1.99 &  2.2690e-02 &       1.99 \\
 4 &  163,424/35,937 &  8.9697e-04 &    2.00 &  5.7585e-03 &       1.98 \\
    \hline
    \end{tabular}
\caption{Convergence table for 3D Maxwell smooth solution - magnetic field}
\label{tab:3D_maxwell_magnetic}
\end{center}
\end{table}

\begin{table}[h!] \small
\begin{center}
    \begin{tabular}{cccccc}
    \hline
l &    Dofs $\uu{b}_h/r_h$ & $\|{r}-{r}_h\|_{L^2(\Omega)}$ & order & $\|{r}-{r}_h\|_{H^1(\Omega)}$ & order\\
    \hline
 1 &     436/125 &  2.2689e-01 &    0.00 &  2.8923e+00 &     0.00 \\
 2 &    2,936/729 &  5.9452e-02 &    1.93 &  1.1782e+00 &     1.30 \\
 3 &   21,424/4,913 &  6.9068e-03 &    3.11 &  3.3901e-01 &     1.80 \\
 4 &  163,424/35,937 &  7.5913e-04 &    3.19 &  9.0082e-02 &     1.91 \\

    \hline
    \end{tabular}
\caption{Convergence table for 3D Maxwell smooth solution - multiplier field}
\label{tab:3D_maxwell_multiplier}
\end{center}
\end{table}

Tables \ref{tab:3D_maxwell_magnetic} and \ref{tab:3D_maxwell_multiplier} show the same orders of convergence as in the 2D case from section \ref{sec:2D_maxwell}.

\subsection{Preconditioning}

Section \ref{sec:MaxwellPrecond} introduces the preconditioning strategy that will be employed in this thesis. Recall that the preconditioner is:
\begin{equation} \nonumber
\mathcal{M}_{\rm MX} =
\begin{pmatrix}
N& 0 \\
0 & L
\end{pmatrix},
\end{equation}
where $N = M+X$ as defined in chapters \ref{sec:discretization} and  \ref{chap:precond}.  It is possible to form a scalable (i.e. iteration count stays the same as $l$ increase) application of $\mathcal{M}_{\rm MX}^{-1}$  using the algebraic multigrid method described in \cite{hiptmair2007nodal} for $N$ and standard multigrid for $L$. This, however, is very complex to implement and hence application of $\mathcal{M}_{\rm MX}^{-1}$ will be done with direct solves for the numerical results in this section.

\subsubsection{2D example}

Table \ref{tab:Maxwell_2D} shows the performance of using the preconditioner $\mathcal{M}_{\rm MX}$  with a MINRES tolerance of $1e-6$.
\begin{table}[h!] \small
\begin{center}
  \begin{tabular}{cccccc}
  \hline
   l &     Dofs $\uu{b}_h/r_h$ &  \multicolumn{4}{ c }{Number of iterations} \\
    &                 &  $\nu_m=10$ &  $\nu_m=100 $&  $\nu_m=1000 $&  $\nu_m=10000$ \\
  \hline
   5 &    10,368/4,225 &   5 &    4 &     6 &      6 \\
   6 &    41,216/16,641 &   5 &    6 &     6 &      6 \\
   7 &   164,352/66,049 &   5 &    6 &     6 &      6 \\
   8 &   656,384/263,169 &   5 &    6 &     6 &      8 \\
   9 &  2,623,488/1,050,625 &   4 &    6 &     6 &      8 \\
   10 &  10,489,856/4,198,401 &   4 &    6 &     8 &     10 \\

  \hline
\end{tabular}
\caption{Iteration table for Maxwell preconditioner  for 2D example - direct application of preconditioner}
\label{tab:Maxwell_2D}
\end{center}
\end{table}

From table \ref{tab:Maxwell_2D} we can see that as the number of MINRES iterations stays about constant as the mesh level, $l$, increases. Note that as the magnetic viscosity ($\nu_m$) increases then the number of iterations remains roughly the same.

\subsubsection{3D example}

Table \ref{tab:Maxwell_3D} presents the iteration numbers when using  $\mathcal{M}_{\rm MX}$ as the preconditioner to MINRES with tolerance $1e-6$.
\begin{table}[h!] \small
\begin{center}
\begin{tabular}{cccccc}
\hline
  l &     Dofs $\uu{b}_h/r_h$ &  \multicolumn{4}{ c }{Number of iterations} \\
    &                  &  $\nu_m=10$ &  $\nu_m=100 $&  $\nu_m=1000 $&  $\nu_m=10000$ \\
    \hline
  2 &    2,936/729 &   4 &    4 &     6 &      5 \\
  3 &   21,424/49,13 &   4 &    4 &     6 &      5 \\
  4 &  163,424/35,937 &   4 &    6 &     6 &      5 \\
\hline
\end{tabular}
\caption{Iteration table for Maxwell preconditioner  for 3D example - direct application of preconditioner}
\label{tab:Maxwell_3D}
\end{center}
\end{table}
Again, as with the 2D example we see that the number of iterations remains about constant with increasing mesh level and value of $\nu_m$.

\section{MHD problem}

Sections \ref{sec:NS_validation} and \ref{sec:Maxwell_validation} shows results for the Navier-Stokes and Maxwells equations in isolation. The next step is to incorporate these two subproblems  into the full MHD system.

\subsection{2D smooth solution}

To validate the code, we consider the following $2D$ test problem. For this problem, let the domain be the unit square $\Omega = (0, 1)^2$ with purely Dirichlet boundary conditions on $\partial \Omega$. Let $\nu = \kappa =1$, $\nu_m=\rm{1e4}$ then the source terms $\uu{f}$ and $\uu{g}$ are defined from the analytical solution:
\begin{align*} \nonumber
\uu{u}(x,y)& =\begin{pmatrix}
xy\exp(x + y) + x\exp(x + y)\\
-xy\exp(x + y) - y\exp(x + y)
\end{pmatrix},\\
p(x,y)& =\exp(y)\sin(x),\\[0.1cm]
\uu{b}(x,y)& =\begin{pmatrix}
 \exp(x + y)\cos(x) \\
 \exp(x + y)\sin(x) - \exp(x + y)\cos(x)
\end{pmatrix},\\
 r(x,y)& =x\sin(2\pi x)\sin(2\pi y).
\end{align*}

The asymptotic rates of convergence are given in tables \ref{tab:MHD_2D_smooth_fluids} to \ref{tab:MHD_2D_smooth_multiplier}.
\begin{table}[h!] \small
\begin{center}
\begin{tabular}{cccccccc}
\hline
$l$ &    Dofs $\uu{u}_h/p_h$ & $\|\uu{u}-\uu{u}_h\|_{L^2(\Omega)}$ & order & $\|\uu{u}-\uu{u}_h\|_{H^1(\Omega)}$ & order &        $\|{p}-{p}_h\|_{L^2(\Omega)}$ & order \\
\hline
 1 &      50/9 &  8.1347e-02 &     0.00 &  1.2748e+00 &     0.00 &  2.8866e-01 &      0.00 \\
 2 &     162/25 &  1.0566e-02 &     2.94 &  3.3018e-01 &     1.95 &  3.7372e-02 &      2.95 \\
 3 &     578/81 &  1.3354e-03 &     2.98 &  8.3246e-02 &     1.99 &  4.8723e-03 &      2.94 \\
 4 &    2,178/289 &  1.6744e-04 &     3.00 &  2.0840e-02 &     2.00 &  7.2809e-04 &      2.74 \\
 5 &    8,450/1,089 &  2.0998e-05 &     3.00 &  5.2110e-03 &     2.00 &  1.5307e-04 &      2.25 \\
 6 &   33,282/4,225 &  2.6528e-06 &     2.98 &  1.3028e-03 &     2.00 &  3.7059e-05 &      2.05 \\
 7 &  132,098/16,641 &  3.4518e-07 &     2.94 &  3.2570e-04 &     2.00 &  9.2136e-06 &      2.01 \\
 8 &  526,338/66,049 &  4.9471e-08 &     2.80 &  8.1425e-05 &     2.00 &  2.3009e-06 &      2.00 \\

\hline
\end{tabular}
\caption{Convergence table for 2D MHD smooth solution - Picard tolerance 1e-8}
\label{tab:MHD_2D_smooth_fluids}
\end{center}
\end{table}

\begin{table}[h!] \small
\begin{center}
\begin{tabular}{cccccc}
\hline
l &    Dofs $\uu{b}_h/r_h$ & $\|\uu{b}-\uu{b}_h\|_{L^2(\Omega)}$ & order & $\|\uu{b}-\uu{b}_h\|_{H(curl,\Omega)}$ & order \\
\hline
 1 &      48/25 &  1.1206e-01 &     0.00 &  1.7807e-01 &        0.00 \\
 2 &     176/81 &  2.8439e-02 &     1.98 &  4.5429e-02 &        1.97 \\
 3 &     672/289 &  7.1388e-03 &     1.99 &  1.1414e-02 &        1.99 \\
 4 &    2,624/1,089 &  1.7867e-03 &     2.00 &  2.8571e-03 &        2.00 \\
 5 &   103,68/4,225 &  4.4679e-04 &     2.00 &  7.1450e-04 &        2.00 \\
 6 &   41,216/16,641 &  1.1171e-04 &     2.00 &  1.7864e-04 &        2.00 \\
 7 &  164,352/66,049 &  2.7927e-05 &     2.00 &  4.4661e-05 &        2.00 \\
 8 &  656,384/263,169 &  6.9818e-06 &     2.00 &  1.1165e-05 &        2.00 \\
\hline
\end{tabular}
\caption{Convergence table for 2D MHD smooth solution - Magnetic field}
\label{tab:MHD_2D_smooth_magnetic}
\end{center}
\end{table}

\begin{table}[h!] \small
\begin{center}
\begin{tabular}{cccccc}
\hline
l &    Dofs $\uu{b}_h/r_h$ & $\|{r}-{r}_h\|_{L^2(\Omega)}$ & order & $\|{r}-{r}_h\|_{H^1(\Omega)}$ & order\\
\hline
 1 &      48/25  &  1.8438e-01 &     0.00 &  1.9449e+00 &     0.00 \\
 2 &     176/81  &  2.8307e-02 &     2.70 &  6.0694e-01 &     1.68 \\
 3 &     672/289  &  3.1214e-03 &     3.18 &  1.6510e-01 &     1.88 \\
 4 &    2,624/1,089  &  3.6035e-04 &     3.11 &  4.2417e-02 &     1.96 \\
 5 &   103,68/4,225  &  4.3890e-05 &     3.04 &  1.0683e-02 &     1.99 \\
 6 &   41,216/16,641  &  5.4485e-06 &     3.01 &  2.6757e-03 &     2.00 \\
 7 &  164,352/66,049  &  6.7988e-07 &     3.00 &  6.6923e-04 &     2.00 \\
 8 &  656,384/263,169  &  8.4948e-08 &     3.00 &  1.6733e-04 &     2.00 \\
\hline
\end{tabular}
\caption{Convergence table for 2D MDH smooth solution - multiplier field}
\label{tab:MHD_2D_smooth_multiplier}
\end{center}
\end{table}

The results shown in tables \ref{tab:MHD_2D_smooth_fluids} to \ref{tab:MHD_2D_smooth_multiplier} agree with the optimal rates for $\|\uu{u}-\uu{u}_h\|_{L^2(\Omega)}$, $\|\uu{u}-\uu{u}_h\|_{H^1(\Omega)}$, $\|\uu{b}-\uu{b}_h\|_{L^2(\Omega)}$, $\|\uu{b}-\uu{b}_h\|_{H(curl,\Omega)}$, $\|{r}-{r}_h\|_{L^2(\Omega)}$ and $\|{r}-{r}_h\|_{H^1(\Omega)}$. The pressure field  exhibits a little higher than expected rate of convergence for $l<5$ but settles down to second order for the higher levels.

% ======================================
%                3D
% ======================================

\subsection{3D smooth solution}

As for the 3 dimensional examples we consider a test problem discretised on a unit cube domain with Dirichlet boundary conditions. Let $\nu = \kappa =1$, $\nu_m=\rm{1e4}$ and the analytical solution as:
\begin{align*} \nonumber
\uu{u}(x,y,z)& =\begin{pmatrix}
-xy\exp(x + y + z) + xz\exp(x + y + z))\\
xy\exp(x + y + z) - yz\exp(x + y + z)\\
-xz\exp(x + y + z) + yz\exp(x + y + z)
\end{pmatrix},\\
p(x,y,z)& =\exp(x + y + z)\sin(y),\\[0.1cm]
\uu{b}(x,y,z)& =\begin{pmatrix}
 -\exp(x + y + z)\sin(y) + \exp(x + y + z)\sin(z) \\
xy\exp(x + y + z) - yz\exp(x + y + z) \\
-\exp(x + y + z)\sin(x) + \exp(x + y + z)\sin(y)
\end{pmatrix},\\
 r(x,y,z)& =\sin(2\pi x)\sin(2\pi y)\sin(2\pi z),
\end{align*}
then the source terms $\uu{f}$ and $\uu{g}$ are defined.

\begin{table}[h!] \small
\begin{center}
\begin{tabular}{cccccccc}
\hline
$l$ &    Dofs $\uu{u}_h/p_h$ & $\|\uu{u}-\uu{u}_h\|_{L^2(\Omega)}$ & order & $\|\uu{u}-\uu{u}_h\|_{H^1(\Omega)}$ & order &        $\|{p}-{p}_h\|_{L^2(\Omega)}$ & order \\
\hline
  1 &     375/27 &  4.2196e-03 &     - &  7.8282e-02 &     - &  5.0614e-02 &      - \\
  2 &    2187/125 &  5.2733e-04 &     3.00 &  1.9495e-02 &     2.01 &  9.0306e-03 &      2.49 \\

 3 &   14739/729 &  6.5749e-05 &     3.00 &  4.8664e-03 &     2.00 &  1.9035e-03 &      2.25 \\

 4 &  107811/4913 &  8.2092e-06 &     3.00 &  1.2161e-03 &     2.00 &  4.5311e-04 &      2.07 \\

\hline
\end{tabular}
\caption{Convergence table for 3D MHD smooth solution - Picard tolerance 1e-8}
\label{tab:MHD_3D_smooth_fluids}
\end{center}
\end{table}

\begin{table}[h!] \small
\begin{center}
\begin{tabular}{cccccc}
\hline
l &    Dofs $\uu{b}_h/r_h$ & $\|\uu{b}-\uu{b}_h\|_{L^2(\Omega)}$ & order & $\|\uu{b}-\uu{b}_h\|_{H(curl,\Omega)}$ & order \\
\hline
1 &     436/125 &  3.9931e-03 &     - &  2.4817e-02 &        - \\
2 &    2936/729 &  1.1211e-03 &     1.83 &  7.4101e-03 &        1.74 \\
3 &   21424/4913 &  2.9642e-04 &     1.92 &  1.9439e-03 &        1.93 \\
4 &  163424/35937 &  7.5494e-05 &     1.97 &  4.9180e-04 &        1.98 \\
\hline
\end{tabular}
\caption{Convergence table for 3D MHD smooth solution - Magnetic field}
\label{tab:MHD_3D_smooth_magnetic}
\end{center}
\end{table}

\begin{table}[h!] \small
\begin{center}
\begin{tabular}{cccccc}
\hline
l &    Dofs $\uu{b}_h/r_h$ & $\|{r}-{r}_h\|_{L^2(\Omega)}$ & order & $\|{r}-{r}_h\|_{H^1(\Omega)}$ & order\\
\hline
 1 &     436/125 &       7.4750e-04 &     - &  1.0113e-02 &     - \\
 2 &    2936/729 &      9.2457e-05 &     3.02 &  2.9368e-03 &     1.78 \\
 3 &   21424/4913 &         1.1182e-05 &     3.05 &  7.7197e-04 &     1.93 \\
 4 &  163424/35937 &       1.3829e-06 &     3.02 &  1.9597e-04 &     1.98 \\
\hline
\end{tabular}
\caption{Convergence table for 3D MDH smooth solution - multiplier field}
\label{tab:MHD_3D_smooth_multiplier}
\end{center}
\end{table}

From tables \ref{tab:MHD_3D_smooth_fluids} to \ref{tab:MHD_3D_smooth_multiplier} it can be seen that we obtain the same convergence orders as with the $2D$ case.

\subsection{Parameter decoupling tests}
\label{sec:ParaTest}

There are two different decoupling schemes considered in thesis. These effectiveness of these approaches is likely to be limited by the parameter set up of the problem, i.e. the fluid viscosity ($\nu$), magnetic viscosity ($\nu_m$) and the coupling number ($\kappa$). In this section we will look at how these three schemes perform when fixing two parameters and varying the second. We will only look at varying $\kappa$ and $\nu$ since $\nu_m$ appears within all three schemes. In the tables we denote (P) as the full MHD system given in \cref{eq:picard}, (MD) is the magnetic decoupling scheme \cref{eq:picard_explicit_MD} and (CD) is complete decoupling \cref{eq:picard_explicit_CD}. Also, if the Picard iterations do not converge then this is denoted by - in the table.

\subsubsection{Viscosity test}

The first test we will consider is to vary the fluid viscosity results are represented in table \ref{tab:MHD_2D_viscosity_decouple_test}. The parameter set up here is that the non-linear Picard tolerance is $1e-5$, coupling number $\kappa = 1$ and magnetic viscosity $\nu_m = 10$.

{\setlength{\tabcolsep}{.2em} \begin{table}[h!] \small
\begin{center}
\begin{tabular}{|cc|ccc|ccc|ccc|ccc|}
\hline
  \multicolumn{2}{|c }{} &  \multicolumn{3}{|c| }{$\nu=1$} &  \multicolumn{3}{|c| }{$\nu=0.1$} &  \multicolumn{3}{|c| }{$\nu=0.01$}&     \multicolumn{3}{|c| }{$\nu=0.001$} \\
l &     DoF &  (P) &  (MD) &  (CD) &  (P) &  (MD) &  (CD)  &   (P) &  (MD) &  (CD) &   (P) &  (MD) &  (CD)    \\
\hline
 3 &     1,620 &   5 &   7 &  15 &  8 &  10 &  - &  12 &  14 & -  &  40 &  40 & -  \\
 4 &     6,180 &   5 &   7 &  15 &  8 &  10 &  - &  12 &  14 &  - &  40 &  40 &  - \\
 5 &    24,132 &   5 &   7 &  15 &  8 &  10 & -  &  12 &  14 &   -&  19 &  19 & -  \\
 6 &    95,364 &   5 &   7 &  15 &  8 &  10 &  - &  12 &  14 &  - &  17 &  19 & -  \\
 7 &   379,140 &   5 &   7 &  15 &  8 &  10 &  - &  12 &  14 &   -&  17 &  19 &  - \\
 8 &  1,511,940 &   5 &   7 &  15 &  8 &  10 &  - &  12 &  14 & -  &  17 &  19 & -  \\
\hline
\end{tabular}
\caption{Number of non-linear iterations for various values of $\nu$. Let the Picard tolerance to be $1e-5$, $\kappa = 1$ and $\nu_m = 10$.}
\label{tab:MHD_2D_viscosity_decouple_test}
\end{center}
\end{table}}
As the fluid viscosity ($\nu$) decreases then the equations that determine the fluid flow become more convection dominated. Thus we would expect that the complete decoupling scheme might break down for small $\nu$ as for this decoupling scheme we remove the convection term. This is precisely what we what we see from the results in table \ref{tab:MHD_2D_viscosity_decouple_test}. However, the full MHD problem and magnetic decoupling cases perform very similarly in terms of the number of Picard iterations it takes to converge to the solution for $\kappa = 1$. For $\nu = 0.0001$ all schemes break down and do not converge. This maybe due to the stability of the convection discretisation.

\subsubsection{Coupling number test}

The other parameter test that we exam is to vary the coupling term.
{\setlength{\tabcolsep}{.15em} \begin{table}[h!] \small
\begin{center}
\begin{tabular}{|cc|ccc|ccc|ccc|ccc|ccc|}
\hline
  \multicolumn{2}{|c }{} &  \multicolumn{3}{ |c| }{$\kappa=0.1$} &  \multicolumn{3}{|c| }{$\kappa=1$} &  \multicolumn{3}{|c| }{$\kappa=10$}&     \multicolumn{3}{|c| }{$\kappa=100$}&     \multicolumn{3}{|c| }{$\kappa=1000$} \\
l &     DoF &  (P) &  (MD) &  (CD) &  (P) &  (MD) &  (CD)  &   (P) &  (MD) &  (CD) &   (P) &  (MD) &  (CD)  &   (P) &  (MD) &  (CD)    \\
\hline
3 &    1,620 &  5 &  5 &  13 &   5 &   5 &  13 &   6 &   7 &  17 &    7 &   13 &   26 &     7 &    93 &     - \\
4 &    6,180 &  5 &  5 &  13 &   5 &   5 &  15 &   6 &   7 &  18 &    7 &   13 &   26 &     7 &    93 &     - \\
5 &   24,132 &  5 &  5 &  13 &   5 &   5 &  15 &   6 &   7 &  18 &    7 &   13 &   26 &     7 &    93 &     - \\
6 &   95,364 &  5 &  5 &  14 &   5 &   5 &  15 &   6 &   7 &  18 &    7 &   13 &   26 &     7 &    93 &     - \\
7 &  379,140 &  5 &  5 &  14 &   5 &   5 &  15 &   6 &   7 &  18 &    7 &   13 &   26 &     7 &    93 &     - \\
8 &  1,511,940 &  5 &  5 &  14 &   5 &   5 &  15 &   6 &   7 &  18 &    7 &   13 &   26 &     7 &    93 &     - \\
\hline
\end{tabular}
\caption{Number of non-linear iterations for various values of $\nu_m$. LLet the Picard tolerance to be $1e-5$, $\nu = 1$ and $\nu_m = 1e2$.}
\label{tab:MHD_2D_kappa_decouple_test}
\end{center}
\end{table}}
This test shows how increasing the coupling term effects the number of non-linear iterations for each of the three schemes. We expect that the full Picard (P) scheme would perform best with both the magnetic and complete decoupling schemes breaking down as $\kappa$ increases. Table \ref{tab:MHD_2D_kappa_decouple_test} shows that this is what happens. Notice that the complete decoupling scheme completely breaks down for $\kappa > 100$ whereas the number of non-linear iterations for magnetic decoupling stays roughly constant for each mesh level but is larger than for the full Picard iterations.

\section{Preconditioned MHD problem}

% In section \ref{sec:ParaTest} we have performed a parameter test to see how the decoupling techiques performed with respect to the parameters $\kappa$ and $\nu$. From tables \ref{tab:MHD_2D_viscosity_decouple_test} and \ref{tab:MHD_2D_kappa_decouple_test} it is possible to see that the decoupling schemes perform

Sections \ref{sec:MHDprecond} and \ref{sec:MDprecond} introduced the three preconditioning techniques that will be applied to each of the decoupling schemes and the full MHD system. In this section we will look at the performances of these strategies with respect to time and iteration count for various tolerances.

\subsection{MHD problem}

Recall form section \ref{sec:MHDprecond} that the preconditioner for the full MHD problem ${\mathcal K}_{\rm MH}$ given in \eqref{eq:mhd_saddle} is
\begin{equation} \nonumber
\mathcal{M}_{\rm MH} =
\left(
\begin{array}{cccc}
F & B^T & C^T & 0\\
0 & -S & 0 & 0 \\
-C & 0 & N & 0\\
0 & 0 & 0 & L
\end{array}
\right),
\end{equation}
where $S$ will be the PCD approximation to the Schur complement of the Navier-Stokes subproblem. To apply $\mathcal{M}_{\rm MH}$ as the preconditioner we employ an inner Krylov solver where the preconditioner is the same as  $\mathcal{M}_{\rm MH}$ but importantly we drop the coupling terms, i.e.
\begin{equation} \nonumber
\mathcal{M}_{\rm innerMH} =
\left(
\begin{array}{cccc}
F & B^T & 0 & 0\\
0 & -S & 0 & 0 \\
0 & 0 & N & 0\\
0 & 0 & 0 & L
\end{array}
\right).
\end{equation}
There are four solves ($F$, $S$, $N$ and $L$) to apply $\mathcal{M}_{\rm innerMH}$ as an inner preconditioner. Each will be a direct solve apart from the PCD approximation in which apply $S^{-1} = W^{-1}F_p A_p^{-1}$ where $W^{-1}$ and $A_p^{-1}$ will be direct solves as well.

\subsection{Magnetic decoupling}
\begin{table}[h!] \small
\begin{center}
\begin{tabular}{cccccc}
\hline
 l &     DoF &  Soln Time &  picard iterations &  NS iters &  M iters \\
\hline
 3 &    1620 &   0.039161 &                  8 &        52 &        3 \\
 4 &    6180 &   0.097946 &                  8 &        48 &        3 \\
 5 &   24132 &   0.447303 &                  8 &        52 &        3 \\
 6 &   95364 &   2.109198 &                  8 &        52 &        3 \\
 7 &  379140 &   9.697754 &                  8 &        50 &        3 \\
 8 &  1511940 &  46.675882                  &8&        51&        3 \\
 9&  6038532&  539.038188&                  8&        51&        3 \\
\hline
\end{tabular}
\caption{$tol = 1e-5    kappa = 1.0/2 Mu_m =1e2 MU = 0.1$}
\end{center}
\end{table}

\subsection{Complete decoupling}

\begin{table}[h!] \small
\begin{center}
\begin{tabular}{ccccccc}
\hline
 l &      DoF &  AV solve Time &  Total picard time &  picard iterations & Av NS iters & Av M iters \\
\hline
 3 &     1620 &          0.030 &               0.32 &                  5 &        29.4 &        2.6 \\
 4 &     6180 &          0.072 &               0.72 &                  5 &        29.4 &        2.6 \\
 5 &    24132 &          0.290 &               2.36 &                  5 &        29.0 &        2.8 \\
 6 &    95364 &          1.552 &              11.14 &                  5 &        29.0 &        2.8 \\
 7 &   379140 &          7.164 &              48.96 &                  5 &        29.0 &        2.8 \\
 8 &  1511940 &         36.180 &             232.27 &                  5 &        29.0 &        2.8 \\
 9 &  6038532 &        159.17 &            1618.24 &                  5 &        29.0 &        2.8 \\
\hline
\end{tabular}
\caption{$tol = 1e-5    kappa = 1.0/2 Mu_m =1e2 MU = 10$}
\end{center}
\end{table}


\documentclass[12pt,parskip=full]{article}
\usepackage{lmodern}
\usepackage{amsmath}
\usepackage[left=1.0in,right=1.0in,top=0.5in,bottom=1.0in]{geometry}
\geometry{letterpaper}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{longtable}
\usepackage{float}
\usepackage{wrapfig}
\usepackage{soul}
\usepackage{textcomp}
\usepackage{marvosym}
\usepackage{wasysym}
\usepackage{latexsym}
\usepackage{amssymb}
\usepackage{apacite}
\usepackage{tabu}
\usepackage[svgnames]{xcolor}
\usepackage{tikz}
\usepackage[linktoc=all]{hyperref}
\usepackage{cleveref}
\usepackage{listings}
\usepackage{setspace}
\usepackage{parskip}
\usepackage{array}
\usepackage{apacite}
\usepackage{natbib}
\usepackage{multicol}
\usepackage{subcaption}
\usepackage{mathtools}
\usetikzlibrary{arrows}

\pgfdeclarelayer{edgelayer}
\pgfdeclarelayer{nodelayer}
\pgfsetlayers{edgelayer,nodelayer,main}

\tikzstyle{none}=[inner sep=0pt]
\tikzstyle{waypt}=[circle,fill=Black,draw=Black,scale=0.4]
\tikzstyle{Helobody}=[circle,fill=White,draw=Black,scale=4.0]
\tikzstyle{Tailrotor}=[circle,fill=White,draw=Black,scale=1.0]
\tikzstyle{ForceVector}=[->,draw=Indigo,fill=Indigo]
\tikzstyle{Coordinate}=[->,draw=Red,fill=Red,fill opacity=1.0]
\tikzstyle{angle}=[->]
\tikzstyle{MeasureMark}=[|-|]
\newlength{\imagewidth}
\newlength{\imagescale}

\setlength{\parskip}{11pt}
%\setlength{\parindent}{15pt}
\usepackage{bookmark}
\makeatletter
%\renewcommand\@seccntformat[1]{}
\makeatother

\lstset
{
	language=c,
	keywords={break,case,catch,continue,else,for,
		if,return,switch,try,while,int,void},
	basicstyle=\ttfamily,
	keywordstyle=\color{blue},
	commentstyle=\color{ForestGreen},
	stringstyle=\color{purple},
	numbers=left,
	numberstyle=\tiny\color{gray},
	stepnumber=1,
	numbersep=10pt,
	backgroundcolor=\color{white},
	tabsize=4,
	showspaces=false,
	showstringspaces=false
}

\renewcommand{\thesection}{\arabic{section}}

\renewcommand{\thesubsection}{\thesection\alph{subsection}}
\renewcommand{\theequation}{\thesubsection\arabic{equation}}
\newcommand*\circled[1]{\tikz[baseline=(char.base)]{
			\node[shape=circle,draw,inner sep=1pt] (char) {#1};}}

\numberwithin{subsection}{section}

\begin{document}
	\vspace{-4ex}
	\title{EbbCFD Theory Guide\vspace{-3.5ex}}
	\author{Rob Rau\vspace{-4ex}}
	\date{\today\vspace{-4ex}}
	\maketitle

	\section{Introduction}
		EbbCFD is a computational fluid dynamics package intended to solve the compressible Euler equations.
		It uses an unstructured, limited, second order finite volume spatial scheme and supports numerous
		time integration schemes. This document covers the theory behind EbbCFD's implementation

	\section{Unstructured Finite Volume}
		EbbCFD operates on completely unstructured meshes. In theory it supports cell geometries of up to six faces,
		though only triangular meshes have been extensively tested. For a first order spatial update, the numerical flux
		is computed on each edge in the mesh, than, for each cell in the mesh, those fluxes are summed and called the
		flux residuals.
		\begin{equation}
			\mathbf{R}_i = \sum_{e = 1}^{N_{f_i}}{m\mathbf{\hat{F}}(\mathbf{U}_i, \mathbf{U}_{N(i,e)}, \vec{n}_{i,e}) \Delta l_{i,e}}
		\end{equation}
		where $\mathbf{\hat{F}}(\mathbf{U}_i, \mathbf{U}_{N(i,e)}, \vec{n}_{i,e})$ is the numerical flux function,
		$\Delta l_{i,e}$ is the length of the cell edge, $N_{f_i}$ is the number of faces of cell $i$, and $m$ is $\pm 1$
		depending on which cell the normal $\vec{n}_{i,e}$ belonged to. This possible because the flux function is homogeneous, i.e.
		$\mathbf{\hat{F}}(\mathbf{U}_i, \mathbf{U}_{N(i,e)}, \vec{n}_{i,e}) = -\mathbf{\hat{F}}(\mathbf{U}_i, \mathbf{U}_{N(i,e)}, -\vec{n}_{i,e})$.
		This allows the code to compute the flux only once per edge, and then depending on the cell, multiply by $\pm 1$.
		With this flux residual we can write the semi-discrete form of the first order finite volume update.
		\begin{equation}
			\frac{\partial \mathbf{U}_i}{\partial t} = -\frac{1}{A_i}\mathbf{R}_i
		\end{equation}
		where $A_i$ is the area of cell $i$. Currently, EbbCFD only supports the Roe flux function, but others are planned.

	\section{Second Order}
		To achieve second order accuracy, the numerical flux is no longer computed using cell averaged state values.
		Instead, the edge state values are computed and used. To obtain edge state values, an estimate of the cell gradients
		is necessary. To reduce solution oscillations at discontinuities in the flow, EbbCFD also supports a couple of
		different schemes for gradient limiting.

		\subsection{Gradients}
			To estimate cell gradients a least squares approximation was used. The least squares problem is set up like so
			\begin{equation}
				\mathbf{r} = \begin{bmatrix} x_1 - x_M & y_1 - y_M \\ \vdots & \vdots \\ x_{N_{f_i}} - x_M & y_{N_{f_i}} - y_M \end{bmatrix} \begin{bmatrix}D_x \\ D_y \end{bmatrix} - \begin{bmatrix} u_1 - u_M \\ \vdots \\ u_{N_{f_i}} - u_M \end{bmatrix}
			\end{equation}
			where $x_M$ and $y_M$ are the current cell centroid coordinates, $x_1..x_{N_{f_i}}$ and $y_1..y_{N_{f_i}}$
			are the cell centroid coordinates for the neighboring cells, $u_M$ is a conserved state variable of the current
			cell, and $u_1..u_{N_{f_i}}$ are the conserved state variables of the neighboring cells. In 2D, four
			least squares problems need to be solved in order to get a gradient for each conserved quantity (density,
			x/y momentum, and energy for the Euler equations). For cells located at a boundary, a ghost cell approach is
			used. The cell centroid of the boundary cells is reflected across the boundary edge and its state values
			are updated in accordance with what type of boundary it is. A wall boundary for instance, will reflect the
			flow momentum across the boundary in order to cancel out the normal momentum component.

		\subsection{Limiters}
			EbbCFD currently supports two different limiting techniques. The first is a scalar limiter, where a single scalar
			value limits both x and y components of a gradient. The other is a vector limiter, where x and y gradient components
			are limited separately. Both scalar and vector limiters used in EbbCFD are implementations of ideas laid out by
			May and Berger. Specifically EbbCFD uses what is referred to as the \textit{relaxed formulation} as its gradient
			limit constraints \cite{doi:10.1137/120875624}. The means the limiters must obey the following conditions
			\begin{equation}
				\min{(u_M, u_1, \dots, u_N)} \le u_M + \begin{bmatrix}(x_j - x_M)D_x \\ (y_j - y_M)D_y\end{bmatrix} \begin{bmatrix} \phi_x \\ \phi_y \end{bmatrix} \le \max{(u_M, u_1, \dots, u_N)}
			\end{equation}
			where $\phi_x$ and $\phi_y$ are the limiters. This method finds the min and max values of all neighboring cells and
			ensures that when the solution is reconstructed to the neighboring cell centroids, monotonicity constraints aren't
			violated.

			\subsubsection{Scalar limiter}
				For the scalar limiter, $\phi_x = \phi_y$. The limiter value is computed as follows
				\begin{align}
					u_r &= u_M + (x_j - x_M)D_x + (y_j - y_M)D_y \\
					s_j &= \begin{cases}
						\frac{u_{min} - u_M}{(x_j - x_M)D_x + (y_j - y_M)D_y} & \mathrm{if}\ u_r < u_{min} \\
						\frac{u_{max} - u_M}{(x_j - x_M)D_x + (y_j - y_M)D_y} & \mathrm{if}\ u_r > u_{max}
					\end{cases} \quad j = 1..N_{f_i} \\
					\phi &= \min{(s_0, \dots, s_{N_{f_i}})}
				\end{align}

			\subsubsection{Vector limiter}
				The vector limiting scheme presented by May and Berger frames the limiting problem as a linear programming problem.
				Basically, the goal is to minimize the limited gradient error, i.e.
				\begin{align}
					\mathrm{minimize}\quad &|D_x - \phi_x D_x| + |D_y - \phi_y D_y| \\
									=& -|D_x|\phi_x - |D_y|\phi_y
				\end{align}
				subject to the monotonicity constraints presented earlier. The algorithm EbbCFD uses to solve this linear
				programming problem is the All-inequality simplex method outlined in \cite{doi:10.1137/120875624}. The vector
				limiter is still an area of ongoing development as it currently seems to get in the way of convergence.

		\begin{figure}[H]
			\centering
			\begin{subfigure}[H]{0.6\textwidth}
				\includegraphics[width=\textwidth,trim={7cm 2cm 10cm 2cm},clip]{FirstOrderShock.pdf}
				\caption{First order shock capture}
			\end{subfigure}
			\begin{subfigure}[H]{0.6\textwidth}
				\includegraphics[width=\textwidth,trim={7cm 2cm 10cm 2cm},clip]{SecondOrderShock.pdf}
				\caption{Second order shock capture. Over/under shoot is visible at shock boundary}
			\end{subfigure}
			\begin{subfigure}[H]{0.6\textwidth}
				\includegraphics[width=\textwidth,trim={7cm 2cm 10cm 2cm},clip]{SecondOrderScalarLimitedShock.pdf}
				\caption{Second order with scalar limiter shock capture. Over/under shoot eliminated}
			\end{subfigure}
			\caption{Mach 2 diamond wedge airfoil}
		\end{figure}

	\section{Numerical Viscosity}
		The Euler equations neglect a dissipative term and thus are not suitable for simulating viscous flows. However, when
		discretized, the approximation of the Euler equations that are actually solved add some dissipation back into
		the solution. An attempt was made to characterize this dissipation to see if any relation between mesh size and
		viscosity could be made. The thought behind this study was to: first, at low Mach number ($M = 0.3$) generate vortices
		from the flow past a cylinder at various mesh sizes, and second, find a correlation between shedding frequency and
		the average hydraulic diameter of the mesh.

		Unfortunately, EbbCFD could not be made to shed vortices around a cylinder at low Mach number, so instead a half cylinder
		was substituted. While the half cylinders did produce vortex shedding, it did not show any sort of meaningful
		relationship between average hydraulic diameter and shedding frequency. Instead all runs snapped to one of two different
		modes as show below.

		\begin{figure}[H]
			\centering
			\includegraphics[width=\textwidth]{SheddingFrequency.pdf}
			\caption{Vortex shedding frequencies on different mesh sizes}
		\end{figure}
		\begin{figure}[H]
			\centering
			\includegraphics[width=\textwidth]{ForceTime.pdf}
			\caption{Lift force time history}
		\end{figure}

		\begin{figure}[H]
			\centering
			\includegraphics[width=\textwidth,trim={7cm 2cm 10cm 2cm},clip]{HalfCylinderVortex.pdf}
			\caption{A sample half cylinder shedding vortices}
		\end{figure}

		\bibliographystyle{siam}
		\bibliography{siam_sjoce335_A2163}
\end{document}