The function \textit{ScenarioLoading} takes as input such scenario.mat file and generate the map X and Y coordinates useful for reference tracking merging all the pieces of road that compose the whole scenario.

The sections Simulation and Reference Signal defines the time span for the simulation and initialise the vectors to allocate all the map waypoints and car orientation, after the function \textit{reference\_generator} up-sample the map according to the prediction horizon and the vehicle speed generating a map with higher density of points to optimize the path following algorithm.

The script named \textit{MPC\_controller} contains the code required to launch the MPC algorithm.
The first line of code initialise the controller defining all the parameters needed for its execution and the reference trajectory that the path planner will follow.

The function \textit{loadParameters} takes as input the vehicle ID, an integer from 1 to 5 that is used as selector for car data as shown in Table \ref{tab:vehicle_data}.

Assuming that the car is already traveling at a speed \textit{V}, the initial conditions for the other states and for the two inputs are all set equal to zero.

next the function \textit{odometer} compute the total distance covered by the car following the scenario in order to let the simulation terminates exactly when the road ends.

A few lines of code are used to create a discrete time state space model for the plant and assign the value for the prediction and control horizon as well as the weights of the output and manipulated variables.

In the successive section of code are defined the constraint matrices, which correctly tuned, guarantee that the vehicle follows the reference without going off the road  i.e. stays within the lane width limits and is capable to avoid an obstacle when is inside the sensors detection range.

\subsection{seconda parte} % ALE
Finally, the last section in the main script is the one where the simulation actually occurs. At each time$-$step, different instructions are repeated in a \textit{for} loop to simulate the controller in a closed$-$loop fashion.\\
First, the current state and the inputs are used in the function \textit{obstacleVehicleModelDT} to generate a discrete$-$time model, using the Zero$-$Order Hold (ZOH) method, and consequently to generate the nominal conditions for the discrete plant and the measurements. Then, the reference path is updated: starting from an index that is increased at each cycle, the trajectory data are read along the prediction horizon length. Once the new plant model and nominal conditions are calculated, they are used as inputs for the \textit{mpcmoveAdaptive} Matlab command that computes the Adaptive MPC control action at current time. This same control action is subsequently used as input to the \textit{VehicleModelCT\_DYN\_ode} function that outputs the differential equations relative to the vehicle dynamic model; these equations are integrated from 0 to the time$-$step using the \textit{ode45} function. In the end, the loop is concluded assigning to the next state\footnote{The \textit{next} state in the current cycle will correspond to the \textit{current} state in the successive cycle as the index is increased} the result of the integration.\\
The pattern described above is repeated \textit{n} times, where \textit{n} represent the ratio between the set simulation duration and resolution.

\subsection{MPC Algorithm}
Once the controller has been initialised, all is set to begin the simulation.\\
% \\At each time$-$step, different instructions are repeated in a \textit{for} loop to simulate the controller in a closed$-$loop fashion.
% Figure \ref{fig:MPC_scheme} graphically shows the general concept behind a model predictive control; below it is described the MATLAB algorithm that follows the same scheme.
% \begin{figure}[H]
%     \centering
%     \includegraphics[width=0.9\textwidth]{Figures/MPC_scheme_basic.png}
%     \caption{MPC algorithm scheme}
%     \label{fig:MPC_scheme}
% \end{figure}
First, the current state and the inputs are used in the function \textit{obstacleVehicleModelDT} to generate a discrete$-$time model, using the Zero$-$Order Hold (ZOH) method, and consequently to generate the nominal conditions for the discrete plant and the measurements. Then, the reference path is updated: starting from an index that is increased at each cycle, the trajectory data are read along the prediction horizon length. Once the new plant model and nominal conditions are calculated, they are used as inputs for the \textit{mpcmoveAdaptive} MATLAB command that computes the Adaptive MPC control action at current time. This same control action is subsequently used as input to the \textit{VehicleModelCT\_DYN\_ode} function that outputs the differential equations relative to the vehicle dynamic model; these equations are integrated from 0 to the time$-$step using the \textit{ode45} function.
\\In the end, the loop is concluded assigning to the next state\footnote{The \textit{next} state in the current cycle will correspond to the \textit{current} state in the successive cycle as the index is increased} the result of the integration.\\
The pattern described above is repeated \textit{n} times, where \textit{n} represent the ratio between the set simulation duration and resolution.


\section{April 27 Review Questions}
\begin{QandA}
\item What's a tradeoff mentioned in Tensorflow between synchronous and asynchronous coordination?
\begin{answered}
Synchronous coordination never has stale parameter values in training steps. However, this comes with the cost of
limited throughput. On the other hand, asynchronous coordination may have stale parameter values and additional training
steps but we can have increased throughput. Machine learning algorithm like SGD and neural network is robust to stragglers
and thus, asynchronous coordination is considered to be the better coordination protocol.
\footnote{We can asychronously write to files because we don't care about synchronous because machine learning algorithmcan afford the cost to
compute one more training iteration}
\end{answered}

\item How does the parameter server model reduce network usage?
\begin{answered}
Parameter server uses \texttt{push} and \texttt{pull} operations to push its entire local gradient into the servers, and
then pull the updated weights back. By updating parameters in batch, parameter server avoids the network overhead of each individual
parameter update. This updating scheme naturally fits with the batch update form of machine learning algorithms. In addition,
parameter server compresses the messages sent across the nodes, which further reduces network usage.
\end{answered}
\end{QandA}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%  develop.tex
%
%  Further information for delvelopers
%
%  Edinburgh Soft Matter and Statistical Physics Group and
%  Edinburgh Parallel Computing Centre
%
%  (c) 2014-2017 The University of Edinburgh
%
%  Contributing authors:
%  Kevin Stratford (kevin@epcc.ed.ac.uk)
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Further Information for Developers}

\subsection{Address models for 3-dimensional fields}

The code allows for different order of storage associated with 3-dimensional
fields (scalars, vectors, and so on). These storage schemes are usually
referred to as array of structures (AOS), structure of arrays (SOA), and
for the blocked version array of structures of (short) arrays (AOSOA).
For historical reasons these are
also sometimes referred to as `Model' and `Model~R' options, which correspond
to array-of-structures and structure-of-arrays layout, respectively. Early
versions of the code for CPU were written to favour summation of LB
distributions on a per lattice site basis in operations such as
$\rho(\mathbf{r}) = \sum_i f_i(\mathbf{r})$. This is array-of-structures,
where the $f_i$ are stored contiguously per site. Introduction of GPU
versions, where memory coalescing favours the opposite memory order,
were then referred to as `Model~R', the `R' standing for `reverse'.

The memory layouts are discussed below. In all cases, a 3-d field occupies
lattice sites with a unique spatial index determine by position, and
computed via \texttt{coords\_index()}. These position indices will be denoted
$r_0, r_1, \ldots, r_{n_s-1}$ where $n_s$ is the total number of lattice
sites (including halo points).

\subsubsection{Rank 1 objects (to include scalar fields)}
\label{subsection:addressing-model-rank1}

\textbf{ADDR\_AOS}: The array-of-structures order for an
$n-$vector field
with components $ v_\alpha = (v_0, v_1, \ldots, v_{n-1})$ is, schematically:
\[
\underbrace{ \boxed{v_0\vphantom{v_{n-1}}} \boxed{v_1\vphantom{v_{n-1}}}
\boxed{\ldots\vphantom{v_{n-1}}} \boxed{v_{n-1}} }_{r_0}
\underbrace{ \boxed{v_0\vphantom{v_{n-1}}} \boxed{v_1\vphantom{v_{n-1}}}
\boxed{\ldots\vphantom{v_{n-1}}} \boxed{v_{n-1}} }_{r_1}
\underbrace{ \boxed{v_0\vphantom{v_{n-1}}} \boxed{v_1\vphantom{v_{n-1}}}
\boxed{\ldots\vphantom{v_{n-1}}} \boxed{v_{n-1}} }_{r_2} \ldots
\underbrace{ \boxed{v_0\vphantom{v_{n-1}}} \boxed{v_1\vphantom{v_{n-1}}}
\boxed{\ldots\vphantom{v_{n-1}}} \boxed{v_{n-1}} }_{r_{n_s-1}}
\]

\textbf{ADDR\_SOA}: The structure-of-arrays version is:
\[
\overbrace{
\begin{array}{cc} \boxed{v_0} \\  _{r_0} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{v_0} \\  _{r_1} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{\ldots\vphantom{v_0}} \\ \phantom{r_2}\end{array} \mkern-22mu
\begin{array}{cc} \boxed{v_0} \\  _{r_{n_s-1}} \end{array}
}^{component_0}
\overbrace{
\begin{array}{cc} \boxed{v_1} \\  _{r_0} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{v_1} \\  _{r_1} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{\ldots\vphantom{v_0}} \\ \phantom{r_2}\end{array} \mkern-22mu
\begin{array}{cc} \boxed{v_1} \\  _{r_{n_s-1}} \end{array}
}^{component_1}
\begin{array}{cc} \ldots \\ \phantom{r_2} \end{array}
\overbrace{
\begin{array}{cc} \boxed{v_{n-1}} \\  _{r_0} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{v_{n-1}} \\  _{r_1} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{\ldots\vphantom{v_{n-1}}} \\ \phantom{r_2}\end{array} \mkern-17mu
\begin{array}{cc} \boxed{v_{n-1}} \\  _{r_{n_s-1}} \end{array}
}^{component_{n-1}}
\]

A scalar field has $n=1$.

% ADDR_MACRO(n_s, n, i, \alpha) in this notation corresponds to
% ADDR_MACRO(nsites, nfield, index, ifld)

\subsubsection{Rank 2 objects (to include dyadic tensor fields)}

\textbf{ADDR\_AOS}:
A general rank 2 tensor $t_{\alpha\beta}$ with components $(t_{0,0},
\ldots, t_{m-1,n-1})$ is stored as:
\begin{gather*}
\underbrace{
\boxed{t_{0,0}\vphantom{t_{m-1,n-1}}} \boxed{t_{1,0}\vphantom{t_{m-1,n-1}}}
\boxed{\ldots\vphantom{t_{m-1,n-1}}} \boxed{t_{m-1,0}\vphantom{t_{m-1,n-1}}}
\boxed{t_{0,1}\vphantom{t_{m-1,n-1}}} \boxed{t_{1,1}\vphantom{t_{m-1,n-1}}}
\boxed{\ldots\vphantom{t_{m-1,n-1}}} \boxed{t_{m-1,1}\vphantom{t_{m-1,n-1}}}
\ldots
\boxed{t_{0,n-1}\vphantom{t_{m-1,n-1}}}\boxed{t_{1,n-1} \vphantom{t_{m-1,n-1}}}
\boxed{\ldots\vphantom{t_{m-1,n-1}}} \boxed{t_{m-1,n-1}\vphantom{t_{m-1,n-1}}}
}_{r_0}
\ldots
\\
\ldots
\underbrace{
\boxed{t_{0,0}\vphantom{t_{m-1,n-1}}} \boxed{t_{1,0}\vphantom{t_{m-1,n-1}}}
\boxed{\ldots\vphantom{t_{m-1,n-1}}} \boxed{t_{m-1,0}\vphantom{t_{m-1,n-1}}}
\boxed{t_{0,1}\vphantom{t_{m-1,n-1}}} \boxed{t_{1,1}\vphantom{t_{m-1,n-1}}}
\boxed{\ldots\vphantom{t_{m-1,n-1}}} \boxed{t_{m-1,1}\vphantom{t_{m-1,n-1}}}
\ldots
\boxed{t_{0,n-1}\vphantom{t_{m-1,n-1}}}\boxed{t_{1,n-1} \vphantom{t_{m-1,n-1}}}
\boxed{\ldots\vphantom{t_{m-1,n-1}}} \boxed{t_{m-1,n-1}\vphantom{t_{m-1,n-1}}}
}_{r_{n_s-1}}
\end{gather*}
Dyadic tensors, for example the gradient of a vector field
$\partial_\alpha v_\beta$ in three dimensions, are stored in corresponding
fashion with $m=3$ and
$\partial_\alpha = (\partial_x, \partial_y, \partial_z)$.

\textbf{ADDR\_SOA}: The structure-of-arrays version is
\[
\overbrace{
\begin{array}{cc} \boxed{t_{0,0}} \\  _{r_0} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{t_{0,0}} \\  _{r_1} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{\ldots\vphantom{t_{0,0}}} \\ \phantom{r_2}\end{array} \mkern-18mu
\begin{array}{cc} \boxed{t_{0,0}} \\  _{r_{n_s-1}} \end{array}
}^{component_{0,0}}
\overbrace{
\begin{array}{cc} \boxed{t_{1,0}} \\  _{r_0} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{t_{1,0}} \\  _{r_1} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{\ldots\vphantom{t_{0,0}}} \\ \phantom{r_2}\end{array} \mkern-18mu
\begin{array}{cc} \boxed{t_{1,0}} \\  _{r_{n_s-1}} \end{array}
}^{component_{1,0}}
\begin{array}{cc} \ldots\vphantom{t_{n-1}} \\ \phantom{r_2}\end{array}
\overbrace{
\begin{array}{cc} \boxed{t_{m-1,n-1}} \\  _{r_0} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{t_{m-1,n-1}} \\  _{r_1} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{\ldots\vphantom{t_{0,0}}} \\ \phantom{r_2}\end{array} \mkern-17mu
\begin{array}{cc} \boxed{t_{m-1,n-1}} \\  _{r_{n_s-1}} \end{array}
}^{component_{m-1,n-1}}
\]

% ADDRRES_MACRO(n_s, m, n, i, alpha, beta) corresponds to, e.g.,
% ADDRESS_MACRO(nsites, NVECTOR, NQAB, index, ia, ib)

\subsubsection{Compressed rank 2 objects}

A symmetric tensor $S_{\alpha\beta}$ in three dimensions has six independent
components. It may be convenient to store this in compressed form as a
rank 1 vector $(S_{xx}, S_{xy}, S_{xz}, S_{yy}, S_{yz}, S_{zz})$ to eliminate
redundent storage.

A symmetric traceless rank 2 tensor --- for example, the Landau-de Gennes
liquid crystal order parameter $Q_{\alpha\beta}$ --- has five independent
components. This is stored as a rank 1 vector with five components
$(Q_{xx}, Q_{xy}, Q_{xz}, Q_{yy}, Q_{yz})$ to eliminate redundant
storage. API calls are provided to expand the compressed format to the
full rank-2 representation $Q_{\alpha\beta}$ and, conversely, to compress
the full representation to five components.

\subsubsection{Rank 3 objects (to include triadic tensor fields)}

The general rank 3 object $t_{\alpha\beta\gamma}$ with components
$(t_{0,0,0}, \ldots, t_{m-1,n-1,p-1})$ is stored in a manner which
generalises from the above, i.e., with the rightmost index running
fastest. Diagrams are omitted, but AOS and SOA storage patterns
follow the same form as seen above.

A triadic tensor, for example the general second derivative of a vector
field $\partial_\alpha \partial_\beta v_\gamma$ may be stored as a rank
3 object.

\subsubsection{Compressed rank 3 objects}

Symmetry properties may be used to reduce the storage requirement
associated with rank 3 tensors. For example, the gradient of the
liquid crystal order parameter $\partial_\gamma Q_{\alpha\beta}$
may be stored as a rank 2 object. The exact requirement will depend
on the construction of the tensor.

%
% The generalised macro would be
% ADDR_MACRO(nsites, m, n, p, i, alpha, beta, gamma)

\subsubsection{LB distribution data}

\textbf{ADDR\_AOS}: For the LB distributions, up to two distinct
distributions can be accommodated to allow for a binary fluid
implementation, although the usual situation is to have only one.
The AOS order for two $N$-velocity distributions $f$ and $g$ is:
\[
\underbrace{
\boxed{f_0\vphantom{f_{N-1}}}    \boxed{f_1\vphantom{f_{N-1}}}
\boxed{\ldots\vphantom{f_{N-1}}} \boxed{f_{N-1}} \,
\boxed{g_0\vphantom{f_{N-1}}}        \boxed{g_1\vphantom{f_{N-1}}}
\boxed{\ldots\vphantom{f_{N-1}}} \boxed{g_{N-1}\vphantom{f_{N-1}}}}_{r_0}
\ldots
\underbrace{
\boxed{f_0\vphantom{f_{N-1}}}    \boxed{f_1\vphantom{f_{N-1}}}
\boxed{\ldots\vphantom{f_{N-1}}} \boxed{f_{N-1}} \,
\boxed{g_0\vphantom{f_{N-1}}}        \boxed{g_1\vphantom{f_{N-1}}}
\boxed{\ldots\vphantom{f_{N-1}}} \boxed{g_{N-1}\vphantom{f_{N-1}}}}_{r_{n_s=1}}
\]

\textbf{ADDR\_SOA}: The strucuture-of-arrays order is:

\[
\overbrace{
\overbrace{
\begin{array}{cc} \boxed{f_0} \\  _{r_0} \end{array} \mkern-18mu
\begin{array}{cc} \boxed{\ldots\vphantom{f_0}} \\ \phantom{r_2}\end{array} \mkern-22mu
\begin{array}{cc} \boxed{f_0} \\  _{r_{n_s-1}} \end{array}
}^{f_0}
\begin{array}{cc} \ldots \\ \phantom{r_2} \end{array}
\overbrace{
\begin{array}{cc} \boxed{f_{N-1}} \\  _{r_0} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{\ldots\vphantom{f_{N-1}}} \\ \phantom{r_2}\end{array} \mkern-17mu
\begin{array}{cc} \boxed{f_{N-1}} \\  _{r_{n_s-1}} \end{array}
}^{f_{N-1}}
}^{f_i}
\overbrace{
\overbrace{
\begin{array}{cc} \boxed{g_0\vphantom{f_{N-1}}} \\ _{r_0} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{\ldots\vphantom{f_{N-1}}} \\ \phantom{r_2}\end{array} \mkern-22mu
\begin{array}{cc} \boxed{g_0\vphantom{f_{N-1}}} \\  _{r_{n_s-1}} \end{array}
}^{g_0} \ldots
\overbrace{
\begin{array}{cc} \boxed{g_{N-1}\vphantom{f_{N-1}}} \\ _{r_0} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{\ldots\vphantom{f_{N-1}}} \\ \phantom{r_2}\end{array} \mkern-17mu
\begin{array}{cc} \boxed{g_{N-1}\vphantom{f_{N-1}}} \\  _{r_{n_s-1}} \end{array}
}^{g_{N-1}\vphantom{f_{N-1}}}
}^{g_i}
\]
For the single-distribution case, this is equivalent to the rank 1 vector
field with $n=N$.

\subsection{Generalised model for SIMD vectorisation}

To promote parallelism at the instruction level, it is necessary to
insure the innermost loop of any loop construct has an extent which
is the vector length for the target architecture. The memory layout
should therefore be adjusted accordingly. This means the MODEL format
is generalised to a (rather clumsily named) array of structures of
short vectors. The length of the short vectors is the SIMD vector
length.

The outermost loop in this context is always to be the loop over
lattice sites, which is adjusted according to the vector length;
see Section \ref{subsection:addressing-how-to} below for practical
examples.

The SOA picture, which targets coalescence, can be viewed as
naturally supporting SIMD vectorisation by virtue of allowing
contiguous access to individual quantities as a function of lattice
index. (If SIMD vectorisation is wanted at all on GPU architecture,
it can be as a means to relieve register pressure.) SOA therefore
remains unaltered and we concentrate on the AOS picture.

\subsubsection{Rank 1 objects}

\textbf{ADDR\_AOSOA}: The structure of short vectors is based on the
SIMD vector length which we here denote $V$:
\[
\underbrace{
\overbrace{
\begin{array}{cc} \boxed{v_0} \\  _{r_0} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{v_0} \\  _{r_1} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{\ldots\vphantom{v_0}} \\ \phantom{r_2}\end{array} \mkern-20mu
\begin{array}{cc} \boxed{v_0} \\  _{r_{V-1}} \end{array}
}^{component_0}
\overbrace{
\begin{array}{cc} \boxed{v_1} \\  _{r_0} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{v_1} \\  _{r_1} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{\ldots\vphantom{v_0}} \\ \phantom{r_2}\end{array} \mkern-20mu
\begin{array}{cc} \boxed{v_1} \\  _{r_{V-1}} \end{array}
}^{component_1}
%\begin{array}{cc} \ldots \\ \phantom{r_2} \end{array}
\overbrace{
\begin{array}{cc} \boxed{v_{n-1}} \\  _{r_0} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{v_{n-1}} \\  _{r_1} \end{array} \mkern-17mu
\begin{array}{cc} \boxed{\ldots\vphantom{v_{n-1}}} \\ \phantom{r_2}\end{array}
\mkern-17mu
\begin{array}{cc} \boxed{v_{n-1}} \\  _{r_{V-1}} \end{array}
}^{component_{n-1}}}_{\mathrm{SIMD\ BLOCK}}
\quad\ldots
\]
Subsequent SIMD blocks involve lattice sites $r_V \ldots r_{2V-1}$,
$r_{2V} \ldots r_{3V-1}$, and so on. If the SIMD vector length is unity,
this collapses to the standard Model picture shown in
Section~\ref{subsection:addressing-model-rank1}.

The generalisation of this approach to rank 2 and rank 3 objects
follows the corresponding Model implementation.

\subsection{Addressing: How to?}
\label{subsection:addressing-how-to}

\subsubsection{Compilation}

A number of options exist to control at compilation time the choice
of the address model. If no options are specified the default
addressing model is AOS. Two additional C preprocessor directives
exist:

\texttt{-DADDR\_SOA} selects structure-of-arrays order (e.g., for
coallescing on GPU platforms).

\texttt{-DADDR\_AOSOA} block SOA pattern (usually combined with
a non-unit vector length).

For development purposes, the address functions allow the array indices
to be checked against the correct range. For production runs, the
relevant functions are replaced by macros if \texttt{-DNDEBUG} is
specified.

The targetDP vector length is specified via, e.g.,  \texttt{-DVVL=4}.

\subsubsection{Kernels}

To provide a transparent interface for addressing vector fields,
a single API is provided which implements the addressing order
selected at compile time. This interface is always based
on a fixed order of indices which may include the
vector index of the innermost SIMD loop if present. This
allows both vectorised and non-vectorised loops to be constructed
in a consistent manner.

The usual model for constructing a loop involving all lattice sites
(in this case without haloes) would be, in the absence of vectorisation:
\begin{lstlisting}
for (ic = 1; ic <= nlocal[X]; ic++) {
  for (jc = 1; jc <= nlocal[Y]; jc++) {
    for (kc = 1; kc <= nlocal[Z]; kc++) {

      index = coords_index(ic, jc, kc);
      /* Perform operation per lattice site ... */
    }
  }
}
\end{lstlisting}
where final array indexing is based on the coordinate index and is
specific to the object at hand. To allow transparent switching
between different addressing models, the indexing must be abstracted.
The abstraction is provided in \texttt{memory.h}, where functions
(or function-like macros) are provided to make the appropriate
address computation.
As a concrete example, the following considers a rank 1 3-vector:
\begin{lstlisting}
for (ic = 1; ic <= nlocal[X]; ic++) {
  for (jc = 1; jc <= nlocal[Y]; jc++) {
    for (kc = 1; kc <= nlocal[Z]; kc++) {

      index = coords_index(ic, jc, kc);
      for (ia = 0; ia < 3; ia++) {
        iref = addr_rank1(nsites, 3, index, ia);
        array[iref] = ...;
      }
    }
  }
}
\end{lstlisting}
Here, the function \texttt{addr\_rank1()} performs the appropriate
arithmetic to reference the correct 1-d array element in either
address model. We note that
this can be implemented to operate appropriately when the SIMD vector
length is an arbitrary number.

This will usually be carried out in computational kernels, where the
\texttt{kernel.h} interface can be used to help to manage the
one-dimensional loop structure required the the targetDP threading
model. In this way, the kernel launch stage will specify the limits
of the kernel in terms of local $x$-coordinates, $y$-coordinates, and
$z$-coordinates in the usual way. The kernel itself is then written
in terms of a one-dimensions kernel index.

In the following schematic, the launch stage defines a kernel context
to be passed to the kernel along with the other actual arguments:
\begin{lstlisting}
  kernel_info_t limits;
  kernel_ctxt_t * ctxt;

  limits.xmin = 1; limits.xmax = nlocal[X];
  limits.ymin = 1; limits.ymax = nlocal[Y];
  limits.zmin = 1; limits,zmax = nlocal[Z];

  kernel_ctxt_create(NSIMDVL, limits, &ctxt);

  __host_launch(kernel_function, nblk, ntpb, ktx, ...);
\end{lstlisting}

An suitable  vectorised kernel may be constructed as follows with the
appropriate base index for the loop computed via the utility
\texttt{kernel\_baseindex()}:
\begin{lstlisting}
for (kindex = 0; kindex < kiterations; kindex += SIMDVL) {
  index = kernel_baseindex(ctxt, kindex);
  for (ia = 0; ia < 3; ia++) {
    for (iv = 0; iv < SIMDVL; iv++) {
      iref = addr_rank1(nsites, 3, index + iv, ia);
      array[iref] = ...;
    }
  }
}
\end{lstlisting}
Note that the same function \texttt{addr\_rank1()} with the
argument which is the index of the vector loop is used. Note also that
vectorised versions must take into account the extent that the kernel
extends into the halo region, which involves
logic to avoid lattice sites which are not required (this type of
masking operation is not shown in this
example).

\vfill
\pagebreak


\begin{homeworkProblem}
	\section{Types}
	There are two kinds of types in the Dice programming language: primitive types and non-primitive types.  There are, correspondingly, two kinds of data values that can be stored in variables, passed as arguments, returned by methods, and operated on: primitive values and non-primitive values.\\
	\begin{minted}{java}
		Type:
		PrimitiveType
		NonprimitiveType
	\end{minted}
	There is also a special null type, the type of the expression $null$, which has no name. Because the null type has no name, it is impossible to declare a variable of the null type. The null reference is the only possible value of an expression of null type. The null reference can always undergo a widening reference conversion to any reference type. In practice, the programmer can ignore the null type and just pretend that $null$ is merely a special literal that can be of any reference type.\\

	\subsection{Primitive Types and Values}
	A primitive type is predefined by the Dice programming language and named by its reserved keyword.
	\begin{minted}{java}
		PrimitiveType:
		NumericType
		bool
		NumericType:
		IntegralType
		float
		IntegralType: one of
		int char
	\end{minted}

	\subsubsection{int}
	A value of type $int$ is stored as a 32-bit signed two's-complement integer. The $int$ type can hold values ranging from -2,147,483,648 to 2,147,483,647, inclusive.

	\subsubsection{float}

	The float type stores the given value in 64 bits. The $float$ type can hold values ranging from 1e-37 to 1e37. Since all values are represented in binary, certain floating point values must be approximated.

	\subsubsection{char}

	The $char$ data type is a 8-bit ASCII character. A $char$ value maps to an integral ASCII code. The decimal values 0 through 31, and 127, represent non-printable control characters. All other characters can be printed by the computer, i.e. displayed on the screen or printed on printers, and are called printable characters.

	The character 'A' has the code value of 65, 'B' has the value 66, and so on. The ASCII values of letters 'A' through 'Z' are in a contiguous increasing numeric sequence. The values of the lower case letters 'a' through 'z' are also in a contiguous increasing sequence starting at the code value 97. Similarly, the digit symbol characters '0' through '9' are also in an increasing contiguous sequence starting at the code value 48.

	\subsubsection{bool}

	A variable of type $bool$ can take one of two values, $true$ or $false$. A bool could also be $null$.\\

	\subsection{Non-Primitive Types}
	Non-primitive types include arrays and classes.
	\subsubsection{Arrays}
	An array stores one or more values of the same type contiguously in memory. The type of an array can be any primitive or an array type. This allows the creation of an n-dimensional array, the members of which can be accessed by first indexing to the desired element of the outermost array, which is of type $array$, and then accessing into the desired element of the immediately nested array, and continuing n-1 times.

	\subsubsection{Classes}

	Classes are user-defined types. See chapter 7 to learn about their usage.

	\subsection{Casting}
	Casting is not supported in this language. There are behaviors between ints and float defined in the section on operators that imitate casting, but there is no syntax to support casting between types directly.

\end{homeworkProblem}


\chapter{SHA: The Secure Hash Algorithm}
\label{cha:sha:-secure-hash}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../main/Cryptol"
%%% End:


\section{Objectives}
\label{sec:objective}

In this work, our objective will be to compare this new reinforcement learning framework for active network management to a more classical framework based on mathematical optimization.
We will compare the two methods on the three following points :
\begin{itemize}
\item Assumptions and scope of the model.
\item Execution time and computational complexity.
\item Experimental results.
\end{itemize}


\documentclass[12pt]{article}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{geometry}
\usepackage{enumerate}
\usepackage{natbib}
\usepackage{float}%稳定图片位置
\usepackage{graphicx}%画图
\usepackage[english]{babel}
\usepackage{a4wide}
\usepackage{indentfirst}%缩进
\usepackage{enumerate}%加序号
\usepackage{multirow}%合并行
\title{\large UM-SJTU JOINT INSTITUTE\\PHYSICS LABTORATORY\\(VE216)\\\ \\\ \\\ \\\ \\\ \\\ \\\ \\\ \\\ \\\ \\\
LABTORATORY REPORT\\\ \\\ LAB 3\\\ Feedback Control \\\ \\\ \\\ \\\ \\\ }
\author{Name: Pan Chongdan\\ID: 516370910121\\Name Xiang Zhiyuan\\ID: 516370910126}
\date{Date: \today}

\begin{document}
\maketitle
\newpage
\section{Objectives}
\begin{enumerate}
\item Understand feedback control.
\item Build Simple Open Loop Control Plant
\end{enumerate}
\section{Theoretical Background}
\subsection{Preliminary Information}
\subsubsection{A Closed-Loop Feedback Model}
The block diagram of a generic closed-loop feedback control system is shown in Fig. 1 In this figure,
the “plant” is the system whose output we wish to control. A measurement is performed on the output
by the measurement sub-system, which is then compared with the desired output, i.e., the input to the
control system. The output of the comparator, which computes the difference between the desired and the
actual output, is fed into a controller whose output serves as the input to the plant. If the comparator and
measurement system were removed from the above diagram then one would be left with an open-loop (i.e.,
no feedback) controller.
\begin{figure}[H]
\centering
\includegraphics[scale=0.4]{P1.jpg}
\caption{Closed-Loop Control System. The plant is the generic term for the system to be controlled}
\end{figure}
\subsubsection{Closed-Loop Transfer Function}
As mentioned earlier, we will restrict our discussion to the case where all of the blocks shown in Fig. 1
are LTI systems. Thus the input/output relationship of each of these blocks can be described by a system
transfer function. Furthermore, we will realize the comparator as an adder and change the gain on the
measurement to -1. With these changes, the block diagram of Fig. 2 becomes:
\begin{figure}][H]
\centering
\includegraphics[scale=0.4]{P2.jpg}
\caption{Closed-Loop Feedback Control System.}
\end{figure}
It is a simple matter of algebra to compute the closed-loop transfer function, $G_{cl}(s)=Y(s)/X(s)$, of the
systems shown in Fig. 1 as indicated below.
\begin{equation}
Y(s)=E(s)C(s)P(s)
\end{equation}
\begin{equation}
E(s)=X(s)-H(s)Y(s)
\end{equation}
Combining Eqs. (1) and (2) yields
\begin{equation}
G_{cl}=\frac{Y(s)}{X(s)}=\frac{C(s)P(s)}{1+C(s)P(s)H(s)}
\end{equation}
\begin{equation}
\frac{E(s)}{X(s)}=\frac{1}{1+C(s)P(s)H(s)}
\end{equation}
\subsection{Examples of Feedback Control Systems}
\subsubsection{DC Motor Model}
Let us assume that we want to control the shaft position of a DC motor. To help design a controller for
this DC motor, we mathematically model the angular position, (t) of the shift by the following differential
equation
\begin{equation}
\frac{d^2\theta(t)}{dt^2}+\frac{d\theta(t)}{dt}=V(t)
\end{equation}
where V (t) is the voltage applied to the motor. Thus the plant (i.e., the motor) is a LTI system that has
the following system transfer function
\begin{equation}
P(s)=\frac{1}{s(s+1)}
\end{equation}
This transfer function corresponds to a second-order LTI system with no zeros, and a pair of real poles
located at the origin and at -1.
\subsubsection{No Controller}
Suppose we want the shaft of the motor to rotate by one radian per second by using a unit step as the input, then
\begin{equation}
\theta(s)=V(s)P(s)=\frac{1}{s}\frac{1}{s(s+1)}=-\frac{1}{s}+\frac{1}{s^2}+\frac{1}{s+1}
\end{equation}
where the right-most side of Eq. (5) is obtained by a partial fraction expansion. Consequently,
\begin{equation}
\theta(t)=(t-1+e^{-t})u(t)
\end{equation}
which is clearly not going to achieve the desired rotation by one radian per second!
\subsubsection{Controller Without Feedback}
Shown below is the block diagram of an open-loop controller. We choose a differentiator as the controller,
\begin{figure}[H]
\centering
\includegraphics[scale=0.5]{P3.jpg}
\caption{Open-Loop Controller}
\end{figure}
\begin{equation}
C(s)=s
\end{equation}
of a unit-step input. Thus
\begin{equation}
\theta(s)=V(s)C(s)P(s)=\frac{1}{s}s\frac{1}{s(s+1)}=\frac{1}{s}-\frac{1}{s+1}
\end{equation}
Therefore
\begin{equation}
\theta(t)=(1-e^{-t})u(t)
\end{equation}
Clearly,$\lim_{t\to\inf}\theta(t)=1$, so that this controller results in no steady-state error when the input is a unit step. The response of the DC motor to the unit step, however, is rather slow. It takes approximately 2.3 s for the shaft angle to reach $90\%$ of its final value of 1 radian.
\par We might ask how well this same controller would work if the input were a ramp rather than a step.
Under such circumstances
\begin{equation}
V(s)=\frac{1}{s^2}
\end{equation}
\begin{equation}
\theta(s)=V(s)C(s)P(s)=\frac{1}{s^2}s\frac{1}{s(s+1)}=-\frac{1}{s}+\frac{1}{s^2}+\frac{1}{s+1}
\end{equation}
Therefore
\begin{equation}
\theta(t)=(t-1+e^{-t})u(t)
\end{equation}
and the steady-state error is 1 radian/second. Thus the angular position of the shift deviates from the
desired position by 1 radian as $t\to 1$, and this differentiator-controller is inadequate for a ramp input.
\subsubsection{Sensitivity of an Open-Loop Controller to Plant Changes}
Suppose that overheating changes the transfer function of the mtoro by a small amount to become
\begin{equation}
P(s)=\frac{1}{(s+0.01)(s+1)}
\end{equation}
one of the poles has moved along the real axis from the origin into the left-hand plane (LHP). The step
response of the motor now becomes
\begin{equation}
\theta(s)=V(s)C(s)P(s)=\frac{1}{s}s\frac{1}{(s+0.01)(s+1)}=\frac{100/99}{s+0.01}-\frac{100/99}{s+1}
\end{equation}
Therefore
\begin{equation}
\theta(t)=(100/99)(e^{-0.01t}-e^{-t})u(t)
\end{equation}
and in steady state $\lim_{t\to\inf}=0$, yielding a steady state error of one radian. It is clear that this open-loop
controller is quite sensitive to variations in the plant transfer function.
\subsubsection{Feedback Control}
Consider now the feedback control system shown in Fig. 2 with $H(s)=1,C(s)=Ks$ and $P(s)=1/[s(s+1)]$ as before. Then according to Eq. (3) the closed-loop transfer function
is given by
\begin{equation}
G_{cl}(s)=\frac{Y(s)}{X(s)}=\frac{C(s)P(s)}{1+C(s)P(s)H(s)}=\frac{KsP(s)}{1+KsP(s)}=\frac{K}{s+(K+1)}
\end{equation}
and the step response becomes
\begin{equation}
\theta(s)=V(s)G_{cl}(s)=\frac{1}{s}\frac{K}{s+(K+1)}
\end{equation}
or equivalently
\begin{equation}
\theta(t)=\frac{K}{K+1}(1-e^{-(K+1)t})u(t)
\end{equation}
If $K$ is sufficiently large, then the steady-state error, $lim_{t\to\inf}(V(t)−\theta(t))=1/(K+1)$ will be small. Also
note that the response time is much improved over that obtained by an open-loop controller. In particular,
the 90\% rise-time has been reduced from 2.3 s to 2.3/(K+1)s.
\subsubsection{Sensitivity of the Closed-Loop Controller to Plant Changes}
As before we will consider the situation where the transfer function of the motor changes by a small amount
to become that given by Eq. (15). Using the same feedback controller described above, the closed-loop
response becomes
\begin{equation}
G_{cl}=\frac{Ks}{s^2+(1.01+K)s+0.01}
\end{equation}
Thus the step response is given by
\begin{equation}
\theta(s)=\frac{K}{s^2+(1.01+K)s+0.01}=\frac{K/(a_+-a_-)}{s+a_+}-\frac{K/(a_+-a_-)}{s+a_-}
\end{equation}
where
\begin{equation}
a_\pm=\frac{(1.01+K)\mp\sqrt{(1.01+K)^2-0.04}}{2}
\end{equation}
Therefore for $K$ large $a_+\approx0$ and $a_-\approx K$, yielding the step response
\begin{equation}
\theta(t)\approx(1-e^{-Kt})u(t)
\end{equation}
Notice that with this closed-loop controller the step response is relatively insensitive to small changes in the
plant.
\subsubsection{Using Feedback to Stabilize Unstable Systems}
Consider a plant that has the following transfer function
\begin{equation}
P(s)=\frac{1}{s-1}
\end{equation}
This system is not bounded-input/bounded-output (BIBO) stable, since its transfer function has a pole in
the right-half plane. It is easy to verify, for example, that the unit step response of this system is given by
$(−1 + e^t)u(t)$ and hence is not bounded. By implementing the feedback system shown in Fig. 2 with
$H(s)=1$ and $C(s)=K$, the closed-loop transfer function becomes
\begin{equation}
G_{cl}(s)=\frac{C(s)P(s)}{1+C(s)P(s)}=\frac{1}{s+(K-1)}
\end{equation}
Thus for $K>1$, the pole has been moved into the LHP and the system has become BIBO stable.
\section{Experimental Procedure}
\subsection{Open Loop Control Plant}
\begin{figure}[H]
\centering
\includegraphics[scale=0.5]{P4.jpg}
\caption{Plant Circuit}
\end{figure}
Steps:
\begin{enumerate}
\item Construct the plant circuit according to Figure 4. Where $R_0=10k\Omega,C_1=100\mu F,C_2=0.22\mu F.$
\item Impulse response: A=1V, width=0.1s, f=1Hz.
\item Step Response: A=1V, f=1Hz.
\end{enumerate}
For example, for the step response, you should try to get something like the waveform shown in following
gure.
\begin{figure}[H]
\centering
\includegraphics[scale=0.1]{P5.jpg}
\caption{Plant Result}
\end{figure}
\subsection{Feedback Control}
\begin{figure}[H]
\centering
\includegraphics[scale=0.5]{P6.jpg}
\end{figure}
Steps:
\begin{enumerate}
\item Add the feedback control circuit to the plant according to Figure 5. Where $R1=R3=150k\Omega,R2=3k\Omega
,C3=0.47\mu F$.
\item Impulse response: A=1V, width=0.1s, f=1Hz.
\item Step Response: A=1V, f=1Hz.
\end{enumerate}
For example, for step response, you should try to get something like the waveform shown in the following
figure.
\begin{figure}[H]
\centering
\includegraphics[scale=0.45]{P7.jpg}
\caption{Feedback Control Result}
\end{figure}
\section{Experiment Results}
\subsection{Open Control Plant}
\begin{figure}[H]
\centering
\includegraphics[scale=0.22]{P8.jpg}
\caption{Impulse Response}
\includegraphics[scale=0.22]{P9.jpg}
\caption{Step Response}
\end{figure}
\subsection{Feedback Control}
\begin{figure}[H]
\centering
\includegraphics[scale=0.22]{P10.jpg}
\caption{Impulse Response}
\includegraphics[scale=0.22]{P11.jpg}
\caption{Step Response}
\end{figure}
\section{Error Analysis and Discussion}
\subsection{Error Analysis}
In the first part of the lab, we were supposed to construct a Plant Circuit, which was the system whose output we wish to control in the whole feedback control system.
If we did it right, we were supposed to get the figure below, which was the case when the input signal was a square wave.
\begin{figure}[H]
\centering
\includegraphics[scale=0.1]{P5.jpg}
\end{figure}
\begin{equation}
Y(s)=\frac{1/(C_1C_2R^2)}{s^2+(2/C_1R)s+1/(C_1C_2R^2)}
\end{equation}
\begin{equation}
Y(s)=\frac{1/(C_1C_2R^2)}{s^3+(2/C_1R)s^2+1/(C_1C_2R^2)S}
\end{equation}
Equation (27),(28) are the impulse response and step response in frequency domain respectively. According to prelab exercise, the two responses in time domain is

\begin{equation}
\frac{1}{C_1 C_2 R^2}\cdot\frac{1}{2\sqrt{\frac{1}{C_1^2R^2}-\frac{1}{C_1C_2R^2}}}\cdot e^{-\frac{t}{C_1R}}(e^{\sqrt{\frac{1}{C_1^2R^2}-\frac{1}{C_1C_2R^2}}t}-e^{-\sqrt{\frac{1}{C_1^2R^2}-\frac{1}{C_1C_2R^2}}t})u(t)
\end{equation}
\begin{equation}
y(t)=(A_0+A_1e^{s_1t}+A_2e^{s_2t})u(t)
\end{equation}
\par And comparing to the ideal result, our figure was quite the same, which means that we did it correctly, and the error was neglectable.
\par In the second part of the lab, we would use the Plant Circuit we made in part 1 to make up a Feedback Control Circuit, and then we set the input signal as impulse signal and square wave signal to check the output. According to equation prelab assignment 3.5
\begin{equation}
W(s)=K_p(X(s)-Y(s))-K_DsY(s)
\end{equation}
where W(s) is the input of P(s)
$$K_p=\frac{R_3}{R_2}=0.5\quad K_D=CR_3=0.0705$$
\par Also, if we did it right, we were supposed to get the figure below, which was the case when the input signal was a square wave.
\begin{figure}[H]
\centering
\includegraphics[scale=0.5]{P7.jpg}
\end{figure}
Comparing to the ideal result, our figure is quite the same, which means that we did it correctly and the error was neglectable. But during the lab, actually many groups’ figures had some “noises”, and also they did nothing wrong. And though in this lab, our group had no error or only the error that was neglectable, but we'll try to analyze the cause of the “noises” and something that you’d better be careful of.
\subsection{Discussion}
For the first part, actually at first, our group just burned the entire circuit, and we had to renew all the units. And the reason was that, we need a DC voltage to help the op amp work, but since the circuit was quite complex, we needed to use lots of wires, and when we were connecting the circuit, two wires just connected accidently, then the whole circuit was burned.
\par So we suggested that when we were connecting a complex circuit, you’d better use the long wires, and keep the nodes as separate as possible, it would help you see the circuit clearly and it could avoid short circuit.
\par And also, if you find your circuit is right, but you can’t get the right figure, you’d better check and change your units(like the capacitor, resistor), if you check and change the units, but it doesn’t work, then you’d better change your board.
\par For the second part, this circuit was far more complex than the first one, and since we burned the circuit in part 1, this time we were more careful, and we used lots of long wires to prevent us from making the same mistake.
\par And we think the reason why some groups’ figure had the “noises” was that: since the circuit is quite complex, sometimes you might let two wires cross each other. Since we had learned that the current will produce a magnetic field, and the changing magnetic field will also cause current. So the crossing wires might affect each other, then the “noises” appeared.
\par So, to prevent this phenomenon happened, we suggested that students should use long wire, and keep the nodes as separate as possible, and try to separate the crossing wires.
\section{Conclusion}
\subsection{Prelab}
We learned some basic knowledge about the Closed-Loop Feedback Model and Closed-Loop Transfer Function, and we know what is a Feedback Control System, and what’s the differences between different kinds of Control Systems.
\par By doing the Prelab assignments, we calculated the relationship between output and input of a Control System, so we can have a deeper understanding of the knowledge we learned before, and by using the matlab to draw the output figure, we got a general impression of the lab results.
\par We learned how to connect the circuit clearly and correctly, that is using long wire, and keeping the nodes as separate as possible.
\subsection{During the lab}
We did learn lots of things during the lab.It’s absolutely the most complex lab we have ever met. And since board had something wrong within it, it took us an hour to finish the first part. This “accident” did help us to improve our patience, and we learned the way to debug, that is check one layer at a time, when you find your circuit doesn’t work, don’t be nervous, just calm down, and first check the circuit, then check the units, then check the board, calm down and check it step by step, then you will figure it out. Also, by connecting the circuit ourselves and see the output figure, we had a deeper understanding of the basic concepts of the Feedback Control System.
\subsection{After the lab}
After the lab, we compared our results to the ideal results, and we were glad to find our results were very close to the ideal one, and by discussing with other teams, we analyzed the cause of the “noises”, and we review the relationship between current and magnetic field to solve this problem, so by doing this, we reviewed the knowledge we learned before and tried to use it to solve real problem, it improved our skills to apply knowledge to real life.
\par All in all, this lab is quite difficult but we finished it perfectly, and we not only learned new knowledge, but also review the old knowledge and improve ourselves in many aspects at the same time.
\par This lab is very interesting and helpful to us!

\end{document}


\chapter{Parallelisation of the Analytical Forces Induced by Dislocations on Surface Elements}
\label{c:para_f_dln_se}
%
\section{Parallelisation on Graphics Processing Units}
Efficient parallelisation requires coalesced memory access, which means we have to be extremely careful when mapping CPU memory to device memory. The fact that threads work ``simultaneously''\footnote{Not quite but essentially simultaneously.} means that in order to obtain good performance, data which is to be ``simultaneously'' loaded into each thread must be contiguous. This maximises cache memory use and therefore reduces slow memory fetch operations to global or shared memory.

The parallelisation was done only over the surface elements in order to avoid the undesirable and inefficient GPU branching that would occur under other schemes.

The most natural form of parallelisation is to have blocks of $ 4n $ threads where $ n \leq 8 \in \mathbb{N}$. This also fits nicely into the 32 thread per warp paradigm.
%
\subsection{Data Mapping}
%
\begin{figure}
	\centering
	\includegraphics[width=\linewidth]{lrse_thread_map.pdf}
	\caption[Linear rectangular surface element mapping.]{Each linear rectangular surface element is mapped to one thread.}
	\label{f:lrse_map}
\end{figure}
%
\subsubsection{Elements: Host $ \mapsto $ Device}
%
\begin{algorithm}
	\caption{Elements in host $ \mapsto $ device.}
	\label{a:ehd}
	\begin{algorithmic}[1]
		\Function{element\_host\_device\_map}{\hvar{\vec{X}[N][3\times E]}}
		\State{\Comment{Input, temporary, and output indices set to zero.}}
		\State{$ i,\, j,\, k \gets 0 $}
		\State{\Comment{\dvar{\vec{X}} accomodates all 3 coordinates of all $ N $ nodes in all $ E $ elements.}}
		\State{$ \dvar{\vec{X}} \gets $ malloc($ 3\times E\times N $)}
		\For{$(n = 0;\, n < N;\, n++)$} \Comment{Loop over nodes in element.}
		\State{\Comment{Set output index to point at the $ n\textsuperscript{th} $ node of the first coordinate of the first element.}}
		\State{$ k \gets j $}
		\For{$ (c = 0;\, c < 3;\, c++) $} \Comment{Loop over coordinates.}
		\State{\Comment{Set input index to point at the $ c\textsuperscript{th} $ coordinate of the first element of the $ n\textsuperscript{th} $ node.}}
		\State{$ i \gets c $}
		\For{$ (e = 0;\, e < E;\, e++) $} \Comment{Loop over elements.}
		\State{$ \dvar{\vec{X}[k + e]} \gets \hvar{\vec{X}[n][i]} $}
		\State{\Comment{Advance output index to point at the $ n\textsuperscript{th} $ node of the $ c\textsuperscript{th} $ coordinate of the first element.}}
		\State{$ i \gets i + 3 $}
		\EndFor
		\State{\Comment{Advance output index to point at the $ n\textsuperscript{th} $ node of the $ c\textsuperscript{th} $ coordinate of the first element.}}
		\State{$ k \gets k + E \times N $}
		\EndFor
		\State{\Comment{Advance temporary index to point at the $ (n+1)\textsuperscript{th} $ node of the first coordinate of the first element.}}
		\State{$ j \gets j + E $}
		\EndFor
		\State{\Return{\dvar{\vec{X}}}}
		\EndFunction
	\end{algorithmic}
\end{algorithm}
\subsubsection{Elements: Device $ \mapsto $ Thread}
\begin{algorithm}
	\caption{Elements in device $ \mapsto $ thread.}
	\label{a:edt}
	\begin{algorithmic}[1]
		\GPUFunction[element\_device\_thread\_map]{\dvar{\vec{X}[3\times E\times N]}, \tvar{\vec{X}[N][3]}}
		\State{\Comment{Thread, input indices.}}
		\State{$ \rvar{idx},\, i \gets \rvar{threadIdx}.x + \rvar{blockIdx}.x\times \rvar{blockDim}.x $}
		\For{$ (c = 0;\, c < 3;\, c++) $}\Comment{Loop over elements.}
		\For{$ (n = 0;\, n < N;\, n++) $}\Comment{Loop over nodes in element.}
		\State{$ \tvar{\vec{X}[n][c]} \gets \dvar{\vec{X}[i + n\times E]}$}
		\EndFor
		\State{$ i \gets i + E\times N $}
		\Comment{Advance input index to the $(n+1)\textsuperscript{th}$ node of element $i$.}
		\EndFor
		\State{\Return{\tvar{\vec{X}[n][c]}}}
		\EndGPUFunction
	\end{algorithmic}
\end{algorithm}
\subsubsection{Force: Thread $ \xmapsto{+} $ Device}
\begin{algorithm}
	\caption{Force in thread $ \xmapsto{+} $ device.}
	\begin{algorithmic}[1]
		\GPUFunction[add\_force\_thread\_device]{\tvar{\vec{F_{n}}[N][3]}, \tvar{\vec{F_{e}}[3]}, \dvar{\vec{F_{n}}[3\times E\times N]},\newline{}\dvar{\vec{F_{e}}[3\times E]}, \rvar{idx}}
		\State{\Comment{Nodal force.}}
		\State{\Comment{Set output index to whatever parallelisation index is given to the function. In this case the same index we use for the main parallelisation.}}
		\State{$ i \gets \rvar{idx} $}
		\For{$ (n = 0;\, n < N;\, n++) $} \Comment{Loop over nodes.}
		\For{$ (c = 0;\, c < 3;\, c++) $} \Comment{Loop over coordinates.}
		\State{\Comment{Ensure nodal forces are correctly added and mapped from local thread memory to global device memory.}}
		\State{atomicAdd(\dvar{\vec{F_{n}}[i + c\times E\times N]}, \tvar{\vec{F_{n}}[n][c]})}
		\EndFor
		\State{\Comment{Displace output index to point at the first coordinate of the $ (n+1)\textsuperscript{th} $ node of the \rvar{idx}\textsuperscript{th} surface element.}}
		\State{$ i \gets i + E $}
		\EndFor
		\State{\Comment{Total force.}}
		\State{$ i \gets \rvar{idx} $} \Comment{Reset output index.}
		\For{$ (c = 0;\, c < 3;\, c++) $} \Comment{Loop over coordinates.}
		\State{atomicAdd(\dvar{\vec{F_{e}}[i]}, \tvar{\vec{F_{e}}[c]})}
		\State{\Comment{Advance the output index to point at the $ (i+1)\textsuperscript{th} $ coordinate of the $ \rvar{idx}\textsuperscript{th} $ surface element.}}
		\State{$ i \gets i + E $}
		\EndFor
		\State{\Return{\dvar{\vec{F_{n}}}, \dvar{\vec{F_{e}}}}}
		\EndGPUFunction
	\end{algorithmic}
\end{algorithm}
\subsubsection{Force (Parallelise over Dislocations): Thread $ \xmapsto{+} $ Device}
\begin{algorithm}
	\caption{Force (parallelise over dislocations) in thread $ \xmapsto{+} $ device.}
	\begin{algorithmic}[1]
		\GPUFunction[dln\_add\_force\_thread\_device]{\tvar{\vec{F_{n}}[N][3]}, \tvar{\vec{F_{e}}[3]},\newline{}\dvar{\vec{F_{n}}[3\times E\times N]}, \dvar{\vec{F_{e}}[3\times E]}, $ k $}
		\State{\Comment{Nodal force.}}
		\State{\Comment{Set output index to correspond to whatever surface element we're on. By parallelising over dislocation lines, we must loop through surface elements in the main code, this is the value of $ k $ we provide. Multiply by 3 and $ N $ because each surface element has 3 coordinates and $ N $ nodes.}}
		\State{$ i \gets 3\times N\times k $}
		\State{$ j \gets 0 $} \Comment{Auxiliary index.}
		\For{$ (n = 0;\, n < N;\, n++) $} \Comment{Loop over nodes.}
		\For{$ (c = 0;\, c < 3;\, c++) $} \Comment{Loop over coordinates.}
		\State{\Comment{Ensure nodal forces are correctly added and mapped from local thread memory to global device memory.}}
		\State{atomicAdd(\dvar{\vec{F_{n}}[i + j + c]}, \tvar{\vec{F_{n}}[n][c]})}
		\EndFor
		\State{\Comment{Displace auxiliary index to point at the first coordinate of the $ (n+1)\textsuperscript{th} $ node of the $ k \textsuperscript{th}$ surface element.}}
		\State{$ j \gets j + 3 $}
		\EndFor
		\State{\Comment{Total force.}}
		\State{\Comment{Set auxiliary index to point at the first coordinate of the $ k\textsuperscript{th} $ surface element.}}
		\State{$ j \gets 3\times k $}
		\For{$ (c = 0;\, c < 3;\, c++) $} \Comment{Loop over coordinates.}
		\State{atomicAdd(\dvar{\vec{F_{e}}[j+c]}, \tvar{\vec{F_{e}}[c]})}
		\EndFor
		\State{\Return{\dvar{\vec{F_{n}}}, \dvar{\vec{F_{e}}}}}
		\EndGPUFunction
	\end{algorithmic}
\end{algorithm}
\subsubsection{Nodal Force: Device $ \mapsto $ Host}
\begin{algorithm}
	\caption{Nodal force in device $ \mapsto $ host.}
	\begin{algorithmic}[1]
		\Function{fx\_device\_host\_map}{\dvar{\vec{F_{n}}[3\times E\times N]}, \hvar{\vec{F_{n}}[N][3\times E]}}
		\State{$ i,~j \gets 0 $} \Comment{Set input and output indices to zero.}
		\For{$ n = 0;\, n < N;\, n++ $} \Comment{Loop over nodes.}
		\State{$ j \gets 0 $} \Comment{Reset output index to point at the first element.}
		\For{$ e = 0;\, e < E;\, e++ $} \Comment{Loop over elements.}
		\For{$ c = 0;\, c < 3;\, c++ $} \Comment{Loop over coordinates.}
		\State{$ \hvar{\vec{F_{n}}[n][j + c]} = \dvar{\vec{F_{n}}[i + e + c\times E\times N]} $}
		\EndFor
		\State{\Comment{Advance output index to point at the first coordinate of the $ (e+1)\textsuperscript{th} $ element.}}
		\State{$ j \gets j + 3 $}
		\EndFor
		\State{\Comment{Advance input index to point at the first coordinate of the $ (n+1)\textsuperscript{th} $ node of the first element.}}
		\State{$ i \gets i + E $}
		\EndFor
		\State{\Return{\hvar{\vec{F_{n}}}}}
		\EndFunction
	\end{algorithmic}
\end{algorithm}
\subsubsection{Total Force: Device $ \mapsto $ Host}
\begin{algorithm}
	\caption{Total force in device $ \mapsto $ host.}
	\begin{algorithmic}[1]
		\Function{ftot\_device\_host\_map}{\dvar{\vec{F_{e}}[3\times E]}, \hvar{\vec{F_{e}}[3\times E]}}
		\State{$ i \gets 0 $} \Comment{Set output index to zero.}
		\For{$ e = 0;\, e < E;\, e++ $} \Comment{Loop over elements.}
		\For{$ c = 0;\, c < 3;\, c++ $} \Comment{Loop over coordinates.}
		\State{$ \hvar{\vec{F_{e}}[i + c]} = \dvar{\vec{F_{e}}[e + c\times E]} $}
		\EndFor
		\State{\Comment{Advance the output index to point at the first coordinate of the $ (e+1)\textsuperscript{th} $ surface element.}}
		\State{$ i \gets i + 3 $}
		\EndFor
		\State{\Return{\hvar{\vec{F_{e}}}}}
		\EndFunction
	\end{algorithmic}
\end{algorithm}
\begin{subequations}\label{eq:fcne}
	\begin{align}
		\vec{X}_{en}          & \coloneqq	\left[x_{en},\, y_{en},\, z_{en}\right]                                                     \\
		\vec{X}_{(1\to E)n}   & \mapsto 	\vec{X}_{n} \nn
		\vec{X}_{n}           & \coloneqq	\left[x_{1n},\, y_{1n},\, z_{1n},\ldots,x_{En},\, y_{En},\, z_{En}\right]\label{seq:xn_arr} \\
		\vec{X}_{1\to N}      & \mapsto	\vec{X}^{\textrm{SE}}\nn
		\vec{X}^{\textrm{SE}} & \coloneqq
		\begin{aligned}
			 & \left[x_{11},\, \ldots,\, x_{E1},\, x_{12},\, \ldots,\, x_{E2},\, \ldots,\, x_{1N},\, \ldots,\, x_{EN}, \right.   \\
			 & \left.\,y_{11},\, \ldots,\, y_{E1},\, y_{12},\, \ldots,\, y_{E2},\, \ldots,\, y_{1N},\, \ldots,\, y_{EN}, \right. \\
			 & \left.\,z_{11},\, \ldots,\, z_{E1},\, z_{12},\, \ldots,\, z_{E2},\, \ldots,\, z_{1N},\, \ldots,\, z_{EN}  \right]
		\end{aligned}
	\end{align}
\end{subequations}
where $ e $ is the surface element label, $ n $ the node label, $ E $ the number of surface elements in the scope. $ N $ the number of nodes in a surface element.

Data-mapping according to \cref{a:fcne} and relabelling the nodes so they go from $ 0 \to N-1 $, the data from \cref{f:lrse_map} would be arranged in device memory like \cref{eq:fcne_eg},
\begin{align}\label{eq:fcne_eg}
	\vec{X}^{\textrm{SE}} & = \begin{aligned}
		\left[\underbrace{h_{x},\, i_{x},\, f_{x},\, g_{x}}_{\mathbf{x_{0}}},\,
		\underbrace{a_{x},\, b_{x},\, i_{x},\, h_{x}}_{\mathbf{x_{1}}},\,
		\right.  & \underbrace{i_{x},\, d_{x},\, e_{x},\, f_{x}}_{\mathbf{x_{2}}},\,
		\underbrace{b_{x},\, c_{x},\, d_{x},\, i_{x}}_{\mathbf{x_{3}}},              \\
		\ldots~y & \textrm{-coord}~\ldots,                                           \\
		\ldots~z & \textrm{-coord}~\ldots]
	\end{aligned}
\end{align}
%

In the GPU, each thread will cater to one surface element at a time. This means that each thread will have to extract the relevant data from the 1D array with length $ 3\times E\times N $ into four 1D arrays of length $ 3 $. The purpose of CNE mapping is to provide the warp with coalesced memory access. This is achieved via \cref{a:bcne}.

Since \cref{a:bcne} is performed in a CUDA GPU, threads in a single block execute sequentially from,
\begin{align}
	\rvar{threadIdx}.x & = 0 \to \rvar{threadIdx}.x = \rvar{blockDim}.x - 1\,,
\end{align}
while threads in different blocks execute in parallel. Coalesced memory access is ensured by having each block load a cache line whose entries are contiguously accessed by the threads in the block. Using the same notation as \cref{a:fcne}, full cache line utilisation (optimal cache use) is achieved if cache lines can accomodate $ l $ entries given by \cref{eq:opt_cache_len_bcne},
\begin{align}
	\label{eq:opt_cache_len_bcne}
	l =
	\begin{cases}
		a \times N \times E                & ,\, a > 0 \in \mathbb{N}                                                \\
		                                   & \textrm{or}                                                             \\
		\dfrac{1}{2^{a}} \times N \times E & ,\, a \geq 0 \in \mathbb{N},\, N \times E \equiv 0\; (\bmod\; 2^{a})\,.
	\end{cases}
\end{align}
\Cref{f:fcne_eg} shows an example of \cref{a:bcne} up to the $ y $-coordinate.
\begin{figure}
	\centering
	\includegraphics[width=\linewidth]{node_bmap.pdf}
	\caption[Example of the backward coodinate-node-element-map.]{Minimum working complete example of the backward coodinate-node-element-map. The explicit calculations of global indices correspond to subsituting the values for ``idx'' and ``idxi'' as in \cref{a:bcne}. $\vec{X_{a}^{b}}$ denotes a 1D array of length three containing the $ xyz $-coordinates of node $ a $ of element $ b $. Each thread concerns itself with only one element at a time. The dash-dot and dashed boxes represent cache lines for a thread block, the dotted line represents a memory fetch, and the dashed lines in $ \vec{X^{E}} $ represent steps of $ E $ entries (the start of the data for the next node of the element type we're dealing with). The memory operations are not shown twice to minimuse redundancy.}
	\label{f:fcne_eg}
\end{figure}
% Start: Code for generating figure e:opt_cache_len_bcne
%\begin{align}%\label{f:fcne_eg}
%	\vec{X}^{\textrm{SE}} &=
%							\begin{aligned}
%								\left[\overbrace{h_{x},\, i_{x},\, f_{x},\, g_{x}}^{\mathbf{x_{3}}},\,
%								\overbrace{a_{x},\, b_{x},\, i_{x},\, h_{x}}^{\mathbf{x_{4}}},\,
%								\right.&\overbrace{i_{x},\, d_{x},\, e_{x},\, f_{x}}^{\mathbf{x_{5}}},\,
%								\overbrace{b_{x},\, c_{x},\, d_{x},\, i_{x}}^{\mathbf{x_{6}}},\\
%								\overbrace{h_{y},\, i_{y},\, f_{y},\, g_{y}}^{\mathbf{x_{3}}},\,
%								\overbrace{a_{y},\, b_{y},\, i_{y},\, h_{y}}^{\mathbf{x_{4}}},\,
%								&\overbrace{i_{y},\, d_{y},\, e_{y},\, f_{y}}^{\mathbf{x_{5}}},\,
%								\overbrace{b_{y},\, c_{y},\, d_{y},\, i_{y}}^{\mathbf{x_{6}}},\\
%								\overbrace{h_{z},\, i_{z},\, f_{z},\, g_{z}}^{\mathbf{x_{3}}},\,
%								\overbrace{a_{z},\, b_{z},\, i_{z},\, h_{z}}^{\mathbf{x_{4}}},\,
%								&\left.\overbrace{i_{z},\, d_{z},\, e_{z},\, f_{z}}^{\mathbf{x_{5}}},\,
%								\overbrace{b_{z},\, c_{z},\, d_{z},\, i_{z}}^{\mathbf{x_{6}}}\right]
%\end{align}
%\begin{align}
%	\textrm{Parallel execution}
%		&\begin{cases}
%			&\vectorsym{X_{0}^{0}}[0] \gets \vectorsym{X^{E}}[(0+0\times 2) + (0\times 4 \times 4) + (0\times 4)]\nonumber\\
%			&\vectorsym{X_{0}^{1}}[0] \gets \vectorsym{X^{E}}[(1+0\times 2) + (0\times 4 \times 4) + (0\times 4)]\nonumber\\
%		\end{cases}\nonumber\\
%	\textrm{Parallel execution}
%		&\begin{cases}
%			&\vectorsym{X_{0}^{2}}[0] \gets \vectorsym{X^{E}}[(0+1\times 2) + (0\times 4 \times 4) + (0\times 4)]\nonumber\\
%			&\vectorsym{X_{0}^{3}}[0] \gets \vectorsym{X^{E}}[(1+1\times 2) + (0\times 4 \times 4) + (0\times 4)]\nonumber\\
%		\end{cases}\nonumber\\
%	\textrm{Parallel execution}
%		&\begin{cases}
%			&\vectorsym{X_{2}^{0}}[0] \gets \vectorsym{X^{E}}[(0+0\times 2) + (0\times 4 \times 4) + (2\times 4)]\nonumber\\
%			&\vectorsym{X_{2}^{1}}[0] \gets \vectorsym{X^{E}}[(1+0\times 2) + (0\times 4 \times 4) + (2\times 4)]\nonumber\\
%		\end{cases}\nonumber\\
%	\textrm{Parallel execution}
%		&\begin{cases}
%			&\vectorsym{X_{2}^{2}}[0] \gets \vectorsym{X^{E}}[(0+1\times 2) + (0\times 4 \times 4) + (2\times 4)]\nonumber\\
%			&\vectorsym{X_{2}^{3}}[0] \gets \vectorsym{X^{E}}[(1+1\times 2) + (0\times 4 \times 4) + (2\times 4)]\nonumber\\
%		\end{cases}\nonumber\\
%	%
%	\textrm{Parallel execution}
%		&\begin{cases}
%			&\vectorsym{X_{0}^{0}}[1] \gets \vectorsym{X^{E}}[(0+0\times 2) + (1\times 4 \times 4) + (0\times 4)]\nonumber\\
%			&\vectorsym{X_{0}^{1}}[1] \gets \vectorsym{X^{E}}[(1+0\times 2) + (1\times 4 \times 4) + (0\times 4)]\nonumber\\
%		\end{cases}\nonumber\\
%	\textrm{Parallel execution}
%		&\begin{cases}
%			&\vectorsym{X_{0}^{2}}[1] \gets \vectorsym{X^{E}}[(0+1\times 2) + (1\times 4 \times 4) + (0\times 4)]\nonumber\\
%			&\vectorsym{X_{0}^{3}}[1] \gets \vectorsym{X^{E}}[(1+1\times 2) + (1\times 4 \times 4) + (0\times 4)]\nonumber\\
%		\end{cases}\nonumber\\
%	\textrm{Parallel execution}
%		&\begin{cases}
%			&\vectorsym{X_{2}^{0}}[1] \gets \vectorsym{X^{E}}[(0+0\times 2) + (1\times 4 \times 4) + (2\times 4)]\nonumber\\
%			&\vectorsym{X_{2}^{1}}[1] \gets \vectorsym{X^{E}}[(1+0\times 2) + (1\times 4 \times 4) + (2\times 4)]\nonumber\\
%		\end{cases}\nonumber\\
%	\textrm{Parallel execution}
%		&\begin{cases}
%			&\vectorsym{X_{2}^{2}}[1] \gets \vectorsym{X^{E}}[(0+1\times 2) + (1\times 4 \times 4) + (2\times 4)]\nonumber\\
%			&\vectorsym{X_{2}^{3}}[1] \gets \vectorsym{X^{E}}[(1+1\times 2) + (1\times 4 \times 4) + (2\times 4)]\nonumber\\
%		\end{cases}\nonumber
%\end{align}
% End: Code for generating figure e:opt_cache_len_bcne
\subsubsection{Node Coordinate Element Map}
%
The node-coordinate-element (NCE) data mapping in \cref{eq:fnce} is carried out by \cref{a:fnce}, each thread looks after a given surface element.
\begin{algorithm}
	\caption{NCE data mapping.}
	\label{a:fnce}
	\begin{algorithmic}
		\ForAll{n nodes $ \in $ surface element}
		\ForAll{c coordinates $ \in [x,\, y,\, z] $}
		\ForAll{e surface elements $ \in $ surface mesh section}
		\State list.append(data of the $ n\textsuperscript{th}$ node with $ c\textsuperscript{th} $ coordinate of the $ e\textsuperscript{th} $ SE)
		\EndFor
		\EndFor
		\EndFor
	\end{algorithmic}
\end{algorithm}
\begin{subequations}\label{eq:fnce}
	\begin{align}
		\vec{X}_{en}          & \coloneqq \left[x_{en},\, y_{en},\, z_{en}\right]                                                    \\
		\vec{X}_{(1\to E)n}   & \mapsto \vec{X}_{n} \nn
		\vec{X}_{n}           & \coloneqq \left[x_{1n},\ldots,\, x_{En},\, y_{1n},\ldots,\, y_{En},\, z_{1n},\ldots,\, z_{En}\right] \\
		\vec{X}_{1\to N}      & \mapsto \vec{X}^{\textrm{SE}}\nn
		\vec{X}^{\textrm{SE}} & \coloneqq
		\begin{aligned}
			\left[x_{11},\ldots,\, x_{E1},\, y_{11},\right. & \ldots,\, y_{E1},\, z_{11},\ldots,\, z_{E1}              \\
			,                                               & \ldots,                                                  \\
			x_{1M},\ldots,\, x_{EN},\, y_{1N},              & \left.\ldots,\, y_{EN},\, z_{1N},\ldots,\, z_{EN}\right]
		\end{aligned}
	\end{align}
\end{subequations}
where $ e $ is the surface \textbf{e}lement, $ n $ the \textbf{n}ode, $ E $ the total number of surface elements in scope, $ N $ the total number of nodes in each surface element.

Data-mapping according to \cref{a:fnce} and relabelling the nodes so they go from $ 0 \to N-1 $, the data from \cref{f:lrse_map} would be arranged in device memory like \cref{eq:fnce_eg},
\begin{align}\label{eq:fnce_eg}
	\vec{X}^{\textrm{SE}} & = \begin{aligned}
		 & \left[\underbrace{h_{x},\, i_{x},\, f_{x},\, g_{x},\,
			h_{y},\, i_{y},\, f_{y},\, g_{y},\,
		h_{z},\, i_{z},\, f_{z},\, g_{z}}_{\mathbf{x_{0}}}\right., \\
		 & ~\ldots~\mathbf{x_{1}},\, xyz\textrm{-coords}~\ldots,   \\
		 & ~\ldots~\mathbf{x_{2}},\, xyz\textrm{-coords}~\ldots,   \\
		 & ~\ldots~\mathbf{x_{3}},\, xyz\textrm{-coords}~\ldots]
	\end{aligned}
\end{align}
%
\subsubsection{Resolving Data Write Conflicts}
Data write conflicts can be a problem in parallel applications where global data is changed by multiple threads within the same clock tick. This can be avoided with atomic operations.
%
\subsection[Parallel Dislocation Line Segments to Surface Elements]{Resolving Parallel Dislocation Line Segments to Surface Elements}
%
Dealing with the special case when the dislocation line segment $ \vec{t} $ is parallel to the surface element is relatively trivial when in a serial program. By the mean value theorem, we can slightly perturb $ \vec{t} $ by rotating it by a small angle around the midpoint of $ \vec{t} $ with respect to the axis of rotation defined by $ \vec{t} \times \vec{n} $. In contrast to serial code, program branches can have a serious effect on parallel performance due to warp divergence. Due to the complexity of the force calculation and relative rarity of the edge case, there is no branching behaviour in the parallel code until \emph{after} the calculation is performed. Where \cref{a:plse_p} is performed.
\begin{algorithm}
	\caption{Resolving cases when $ \vec{t} \parallel \vec{n} $ on GPUs.}
	\label{a:plse_p}
	\begin{algorithmic}
		\ForAll{surface elements \emph{and} line segments}
		\State $ \ldots $
		\If{$ \vec{t}_{\textrm{thread}} \parallel \vec{n}_{\textrm{thread}} $}
		\State $ \vec{x}_{\textrm{buffer}} \gets \vec{x}_{\textrm{thread}} $
		\State $ \vec{t}_{\textrm{buffer}} \gets \vec{t}_{\textrm{thread}} $
		\Else
		\State $ \vec{F}_{\textrm{total}} += \vec{F}_{\textrm{thread}} $
		\EndIf
		\EndFor
		\State return to serial code
		\If{$ \vec{x}_{\textrm{buffer}} \neq $ empty}
		\ForAll{surface elements \emph{and} line segments}
		\State perform calculation for $ \vec{t} \parallel \vec{n} $
		\EndFor
		\EndIf
	\end{algorithmic}
\end{algorithm}
\subsubsection{Implementation}
The node mapping would be the same as \cref{a:fcne}. The parallelisation would be on individual packets of a surface element with a slightly rotated dislocation line segment, as in a single iteration of the serial code where the dislocation line segment is rotated. The forces would be averaged with atomic operations and \mintinline{c}{__syncthreads()} before they are added to the total nodal forces.

\subsection{Test Case}
\begin{figure}
	\includegraphics[width=\linewidth]{cuda_test.pdf}
	\caption[Test case for CUDA development.]{Test case for CUDA development. Dislocation line not shown in node separation as the parallelisation happens across surface elements. Not to perspective or scale.}
	\label{f:cuda_test}
\end{figure}
According to \cref{a:fcne} and the node labelling scheme of \cref{f:flrs} the data mapping would look like:
\begin{align}\label{eq:cuda_test}
	\vec{X}^{\textrm{SE}} =
	\begin{split}
		\left[\overbrace{\underbrace{1,1,0,0}_{\mathbf{x_{3}}}, \underbrace{2,2,1,1}_{\mathbf{x_{4}}},
		\underbrace{1,1,0,0}_{\mathbf{x_{5}}}, \underbrace{2,2,1,1}_{\mathbf{x_{6}}}}^{x\textrm{-coord}},\right.\\
		\overbrace{\underbrace{0,1,1,0}_{\mathbf{x_{3}}}, \underbrace{0,1,1,0}_{\mathbf{x_{4}}},
		\underbrace{1,2,2,1}_{\mathbf{x_{5}}}, \underbrace{1,2,2,1}_{\mathbf{x_{6}}}}^{y\textrm{-coord}},\\
		\left.\overbrace{\underbrace{0,0,0,0}_{\mathbf{x_{3}}}, \underbrace{0,0,0,0}_{\mathbf{x_{4}}},
		\underbrace{0,0,0,0}_{\mathbf{x_{5}}}, \underbrace{0,0,0,0}_{\mathbf{x_{6}}}}^{z\textrm{-coord}}\right]
	\end{split}
\end{align}
In general the number of indices in the array would be $ 3 \times e \times n $ where $ e $ is the number of surface elements, $ n $ the number of nodes per element and 3 the number of spatial dimensions.
%
\section{Thread Block Size Optimisation}
\href{http://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#memory-optimizations}{http://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html\#memory-optimizations}
\href{http://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#execution-configuration-optimizations}{http://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html\#execution-configuration-optimizations}
\savearabiccounter


\chapter{Auxiliary-Field Quantum Monte Carlo}
\label{chap:afqmc}
The Auxiliary-Field Quantum Monte Carlo (AFQMC) method is an orbital-space formulation of the imaginary-time propagation algorithm. We refer the reader to one of the review articles on the method \cite{AFQMC_review,PhysRevLett.90.136401,PhysRevE.70.056702}, for a detailed description of the algorithm. It uses the Hubbard-Stratonovich transformation to express the imaginary-time propagator, which is inherently a 2-body operator, as an integral over 1-body propagators which can be efficiently applied to an arbitrary Slater determinant. This transformation allows us to represent the interacting many-body system as an average over a non-interacting system (e.g. Slater determinants) in a time-dependent fluctuating external field (the Auxiliary fields). The walkers in this case represent non-orthogonal Slater determinants, whose time average represent the desired quantum state. QMCPACK currently implements the phaseless AFQMC algorithm of Zhang and Krakauer \cite{PhysRevLett.90.136401}, where a trial wave-function is used to project the simulation to the real axis, controlling the fermionic sign problem at the expense of a bias. This approximation is similar in spirit to the fixed-node approximation in real-space DMC, but applied in the Hilbert space where the AFQMC random walk occurs.

\section{Theoretical Background}
... Coming Soon ...

\section{Input}

The input for an AFQMC calculation is fundamentally different to the input for other real-space algorithms in QMCPACK. The main source of input comes from the Hamiltonian matrix elements in an appropriate single particle basis. This must be evaluated by an external code and saved in a format that QMCPACK can read. More details about file formats are found below. The input file has six basic xml-blocks: \texttt{AFQMCInfo}, \texttt{Hamiltonian}, \texttt{Wavefunction}, \texttt{WalkerSet}, \texttt{Propagator}, and \texttt{execute}. The first five define input structures required for various types of calculations. The \texttt{execute} block represents actual calculations and takes as input the other blocks.
Non-execution blocks are parsed first, followed by a second pass where execution blocks are parsed (and executed) in order. Listing 13.1 shows an example of a minimal input file for an AFQMC calculation. Table \ref{table:afqmc_basic} shows a brief description of the most important parameters in the calculation. All xml sections contain a ``name'' argument used to identify the resulting object within QMCPACK. For example, in the example, multiple Hamiltonian objects with different names can be defined. The one actually used in the calculation is the one passed to ``execute'' as ham.

\begin{lstlisting}[caption=Sample input file for AFQMC.]
<?xml version="1.0"?>
<simulation method="afqmc">
  <project id="Carbon" series="0"/>

  <AFQMCInfo name="info0">
    <parameter name="NMO">32</parameter>
    <parameter name="NAEA">16</parameter>
    <parameter name="NAEB">16</parameter>
  </AFQMCInfo>

  <Hamiltonian name="ham0" type="SparseGeneral" info="info0">
    <parameter name="filetype">hdf5</parameter>
    <parameter name="filename">../fcidump.h5</parameter>
  </Hamiltonian>

  <Wavefunction name="wfn0" info="info0">
    <ImpSamp name="impsamp0" type="PureSD" init="ground" >
      <parameter name="filetype">none</parameter>
    </ImpSamp>
  </Wavefunction>

  <WalkerSet name="wset0" type="distributed">
  </WalkerSet>

  <Propagator name="prop0" info="info0">
  </Propagator>

  <execute wset="wset0" ham="ham0" wfn="wfn0" prop="prop0" info="info0">
    <parameter name="timestep">0.005</parameter>
    <parameter name="blocks">10000</parameter>
    <parameter name="nWalkers">20</parameter>
  </execute>

</simulation>
\end{lstlisting}

%The following table lists some of the most practical parameters in the \texttt{execute} block
%The following table lists some of the practical parameters
\begin{table}[h]
\begin{center}
\begin{tabularx}{\textwidth}{l l l l l l }
\hline
\multicolumn{6}{l}{\texttt{afqmc} method} \\
\hline
\multicolumn{6}{l}{parameters in \texttt{AFQMCInfo}} \\
   &   \bfseries name     & \bfseries datatype & \bfseries values & \bfseries default   & \bfseries description \\
%   &   \texttt{name            } &  argument    &         & no & unique name that identifies the xml-block \\
   &   \texttt{NMO             } &  integer     & $\ge 0$ & no & number of molecular orbitals \\
   &   \texttt{NAEA            } &  integer     & $\ge 0$ & no & number of active electrons of spin up \\
   &   \texttt{NAEB            } &  integer     & $\ge 0$ & no & number of active electrons of spin down \\
%   &   \texttt{NCA            } &  integer     & $\ge 0$ & 0 & number of core electrons of spin up \\
%   &   \texttt{NCB            } &  integer     & $\ge 0$ & 0 & number of core electrons of spin down \\
\multicolumn{6}{l}{parameters in \texttt{Hamiltonian}}  \\
%   &   \bfseries name     & \bfseries datatype & \bfseries values & \bfseries default   & \bfseries description \\
%   &   \texttt{name            } &  argument   &               & no   & unique name that identifies the xml-block \\
   &   \texttt{type            } &  argument   & SparseGeneral & no   & type of \texttt{Hamiltonian} \\
   &   \texttt{info            } &  argument   &               &      & name of \texttt{AFQMCInfo} block \\\\
   &   \texttt{filename        } &  string     &               & no   & name of file with the hamiltonian \\
   &   \texttt{filetype        } &  string     & fcidump       & no   & ASCII based format of Molpro, VASP, PySCF, etc. \\
   &   \texttt{                } &             & hdf5          &      & native HDF5-based format of QMCPACK  \\
%   &   \texttt{cutoff\_1bar            } &  real     &  & 1e-8 & cutoff applied to integrals during reading \\
%   &   \texttt{cutoff\_decomposition   } &  real     &  & 1e-6 & cutoff used to stop the iterative cycle in the generation of the Cholesky decomposition of the 2-electron integrals \\
\multicolumn{6}{l}{parameters in \texttt{Wavefunction}}\\
%   &   \bfseries name     & \bfseries datatype & \bfseries values & \bfseries default   & \bfseries description \\
%   &   \texttt{name            } &  argument   &             & no   & unique name that identifies the xml-block \\
   &   \texttt{info            } &  argument   &             &      & name of \texttt{AFQMCInfo} block \\
   &   \texttt{ImpSamp         } &  xml-block  &             & no   & defines the trial wave-function \\
%   &   \texttt{name            } &  argument   &             & no   & unique name that identifies the xml-block \\
   &   \texttt{type            } &  argument   & PureSD      & no   & single slater determinant trial wave-function \\
   &   \texttt{                } &             & MultiPureSD &      & linear combination of Slater Determinants \\
   &   \texttt{                } &             & GenSD       &      & single determinant w/o rotated 2-e integrals \\
   &   \texttt{filetype        } &  string     & ascii       & no   & ASCII data file type \\
   &   \texttt{                } &             & hdf5        &      & HDF5 data file type \\
   &   \texttt{                } &             & none        &      & no input file needed, requires init=``ground'' \\
\multicolumn{6}{l}{parameters in \texttt{WalkerSet}} \\
 %  &   \bfseries name     & \bfseries datatype & \bfseries values & \bfseries default   & \bfseries description \\
%   &   \texttt{name            } &  argument   &             & no   & unique name that identifies the xml-block \\
   &   \texttt{type            } &  argument   & distributed & no   & type of walker container: standard \\
\multicolumn{6}{l}{parameters in \texttt{Propagator}} \\
%   &   \bfseries name     & \bfseries datatype & \bfseries values & \bfseries default   & \bfseries description \\
%   &   \texttt{name            } &  argument   &       & no    & unique name that identifies the xml-block \\
   &   \texttt{type            } &  argument   & afqmc & afqmc & type of propgator \\
   &   \texttt{                } &             &  vmc  &       & not yet functioning \\
   &   \texttt{info            } &  argument   &       &       & name of \texttt{AFQMCInfo} block \\
\multicolumn{6}{l}{parameters in \texttt{execute}} \\
%   &   \bfseries name     & \bfseries datatype & \bfseries values & \bfseries default   & \bfseries description \\
%   &   \texttt{name            } &  argument    &         & no   & unique name that identifies the xml-block \\
   &   \texttt{wset            } &  argument    &         &      &  \\
   &   \texttt{ham             } & argument     &         &      &  \\
   &   \texttt{wfn             } & argument     &         &      &  \\
   &   \texttt{prop            } & argument     &         &      &  \\
   &   \texttt{info            } &  argument    &         &      & name of \texttt{AFQMCInfo} block \\
   &   \texttt{nWalkers        } &  integer     & $\ge 0$ & 5    & initial number of walkers per task group   \\
   &   \texttt{timestep        } &  real        & $> 0$   & 0.01 & time step in 1/a.u. \\
   &   \texttt{blocks          } &  integer     & $\ge 0$ & 100  & number of blocks            \\
   &   \texttt{step            } &  integer     & $> 0$   & 1    & number of steps within a block \\
   &   \texttt{substep         } &  integer     & $> 0$   & 1    & number of substeps within a step \\
  \hline
\end{tabularx}
\end{center}
\caption{Input options for AFQMC in QMCPACK}
\label{table:afqmc_basic}
\end{table}

Below is a list of all input sections for AFQMC calculations, along with a detailed explanation of accepted parameters. Since the code is under active development, the list of parameters and their interpretation can change in the future.\\

\texttt{AFQMCInfo}: input block that defines basic information about the calculation. It is passed to all other input blocks to propagate the basic information:
\texttt{<AFQMCInfo name="info0">}
\begin{itemize}
\item \textbf{NMO}. Number of molecular orbitals, i.e., number of states in the single particle basis.
\item \textbf{NAEA}. Number of Active Electrons-Alpha, i.e., number of spin-up electrons.
\item \textbf{NAEB}. Number of Active Electrons-Beta, i.e., number of spin-down electrons.
\item \textbf{NCA}. number of core electrons of spin up (only meaningful if reading ASCII FCIDUMPs). Default: 0.
\item \textbf{NCB}. Number of core electrons of spin down (only meaningful if reading ASCII FCIDUMPs). Default: 0.\\
\end{itemize}

\texttt{Hamiltonian}: controls the object that reads, stores and manages the hamiltonian.
  \texttt{<Hamiltonian name="ham0" type="SparseGeneral" info="info0">}
\begin{itemize}
\item \textbf{filename}. Name of file with the \texttt{Hamiltonian}. This is a required parameter.
\item \textbf{filetype}. The type of file found in filename. Options are: fcidump, hdf5. fcidump is the ASCII-based format of Molpro, VASP, PySCF, etc.; hdf5 is the native HDF5-based format  of QMCPACK and can also be generated by modified versions of PySCF and VASP. This is a required parameter.
\item \textbf{cutoff\_1bar}. Cutoff applied to integrals during reading. Any term in the hamiltonian smaller than this value is set to zero. (For filetype=``hdf5'', the cutoff is only applied to the 2-electron integrals). Default: 1e-8
\item \textbf{cutoff\_decomposition}. Cutoff used to stop the iterative cycle in the generation of the Cholesky decomposition of the 2-electron integrals. The generation of Cholesky vectors is stopped when the maximum error in the diagonal reaches this value. In case of an eigenvalue factorization, this becomes the cutoff applied to the eigenvalues. Only eigenvalues above this value are kept. Default: 1e-6
\item \textbf{hdf\_write\_file}. If a value is given, an hdf5 restart file for the \texttt{Hamiltonian} is generated. Default: ``'' (no file is generated)
\item \textbf{hdf\_write\_type}. What to store in the restart file. Options: ``integrals'': write directly the 1- and 2-electron integrals; ``factorized'': write the cholesky decomposition of the integrals; ``default'': Default value, write the type found in filetype.
\item \textbf{ascii\_write\_file} If a value is given, an ASCII restart file for the \texttt{Hamiltonian} is generated. Default: ``'' (no file is generated)
\item \textbf{nblocks}. This parameter controls the distribution of the 2-electron integrals among processors. In the default behavior (nblocks=1), all nodes contain the entire list of integrals. If nblocks $>$ 1, the of nodes in the calculation will be split in nblocks groups. Each node in a given group contains the same subset of integrals and subsequently operates on this subset during  any further operation that requires the hamiltonian. The maximum number of groups is NMO. Currently only works for filetype=``hdf5'' and the file must contain integrals.  Not yet implemented for input hamiltonians in the form of Cholesky vectors or for ASCII input. Coming soon!
    Default: No distribution
\item \textbf{printEig}. If ``yes'', prints additional information during the Cholesky decomposition.
    Default: no
\item \textbf{fix\_2eint}.  If this is set to ``yes'', orbital pairs that are found not to be positive definite are ignored in the generation of the Cholesky factorization. This is necessary if the 2-electron integrals are not positive definite due to round-off errors in their generation.
    Default: no \\
\end{itemize}

\texttt{Wavefunction}: controls the object that manages the trial wave-functions. This block expects a list of xml-blocks defining actual trial-wave functions for various roles.
\texttt{<Wavefunction name="wfn0" info="info0">}
  \texttt{<ImpSamp name="impsamp0" type="PureSD">}
\begin{itemize}
\item \textbf{init}. Argument in ImpSamp. Selects pre-defined trial wave-functions if desired.
Value: ground. Only for PureSD. Sets the trial wave-function to a block diagonal form. Diagonal in the occupied-occupied block and zero outside. This option corresponds to a ground state determinant (assuming orbitals ordered by increasing eigenvalue) in the orbital space that defines the hamiltonian. For a \texttt{Hamiltonian} generates based on HF orbitals, this generates a HF trial wave-function. (Only useful situation.)
\item \textbf{filename}. Name of file with wave-function information.
\item \textbf{cutoff}. cutoff applied to the terms in the calculation of the local energy. Only terms in the hamiltonian above this cutoff are included in the evaluation of the energy.
      Default: 1e-6
\item \textbf{nnodes}. Defines the parallelization of the local energy evaluation and the distribution of the \texttt{Hamiltonian} matrix (not to be confused with the list of 2-electron integrals managed by \texttt{Hamiltonian}. These are not the same.) If nnodes $>$ 1, the nodes in the simulation are split into groups of nnodes, each group works collectively in the evaluation of the local energy of their walkers. This helps distribute the effort involved in the evaluation of the local energy among the nodes in the group, but also distributes the memory associated with the wave-function among the nodes in the group.
      Default: No distribution
\item \textbf{hdf\_write\_file}. If provided, a restart file for the wave-function will be generated.
      Default: ``''
\item \textbf{runtype}. Unique to MultiPureSD with orthogonal expansions.
      Values: 0.  Use low memory algorithm. Much slower but has small memory requirements;
      1. Use high-memory algorithm. Faster but with significant memory requirements.
\item \textbf{fast}. Unique to MultiPureSD. If ``yes'', use fast-update algorithm with orthogonal expansions.
      Default: No
\item \textbf{Estimator}. Optional xml-block which defines a trial wave-function used for the calculation of energy averages.  It is used only during the collection of averages and doesn't influence the propagation. Note that this block must be set if an Estimator xml-block in the execute xml-block is specified. This must also be defined in order to calculate and print the energy in the case of hybrid propagation (see below). \\
\end{itemize}

\texttt{WalkerSet}: Controls the object that handles the set of walkers.
\texttt{<WalkerSet name="wset0" type="distributed">}
\begin{itemize}
\item \textbf{pop\_control}. Population control algorithm. Options: ``simple'': Uses a simple branching scheme with a fluctuating population. Walkers with weight above max\_weight are split into multiple walkers of weight reset\_weight. Walkers with weight below min\_weight are killed with probability (weight/min\_weight); ``pair'': Fixed-population branching algorithm, based on QWalk's branching algorithm. Pairs of walkers with weight above/below max\_weight/min\_weight are combined into 2 walkers with weights equal to $(w_1+w_2)/2$. The probability of replicating walker w1 (larger weight) occurs with probability $w_1/(w_1+w_2)$, otherwise walker w2 (lower weight) is replicated; ``comb'': Fixed-population branching algorithm based on the Comb method. Will be available in the next release. Default: ``pair''
\item \textbf{min\_weight}. Weight at which walkers are possibly killed (with probability weight/min\_weight). Default: 0.05
\item \textbf{max\_weight}. Weight at which walkers are replicated. Default: 4.0
\item \textbf{reset\_weight}. Weight to which replicated walkers are reset to. Default: 1.0
\item \textbf{extra\_spaces}. Number of empty spaces kept in the list. Useful to avoid constant reallocation. Default: 10 \\
\end{itemize}

\texttt{Propagator}: Controls the object that manages the propagators.
\texttt{<Propagator name="prop0" info="info0">}
\begin{itemize}
\item \textbf{cutoff}. Cutoff applied to Cholesky vectors. Elements of the Cholesky vectors below this value are set to zero.
    Default: 1e-6
\item \textbf{substractMF}. If ``yes'', apply mean-field substraction based on the ImpSamp trial wave-function. Must set to ``no'' to turn it off.
    Default: yes
\item \textbf{useCholesky}. If ``yes'', use iterative Cholesky decomposition to factorize 2-electron integral matrix. If ``no'', use eigenvalue decomposition (which is very slow and memory intensive).
    Default: yes
\item \textbf{vbias\_bound}. Upper bound applied to the vias potential. Components of the vias potential above this value are truncated there. The bound is currently applied to $\sqrt{\tau} v_{bias}$, so a larger value must be used as either the time-step or the fluctuations increase (e.g. from running a larger system or using a poor trial wave-function).
    Default: 3.0
\item \textbf{apply\_constrain}. If ``yes'', apply the phaseless constrain to the walker propagation. Currently, setting this to ``no'' produces unknown behavior, since free propagation algorithm has not been tested.
    Default: yes
\item \textbf{hybrid}. If ``yes'', use hybrid propagation algorithm. This propagation scheme doesn't use the local energy during propagation, leading to significant speed ups when its evaluation  cost is high. The local energy of the ImpSamp trial wave-function is never evaluated. To obtain energy estimates in this case, you must define an Estimator xml-block with the \texttt{Wavefunction} block. The local energy of this trial wave-function is evaluated and printed. It is possible to use a previously defined trial wave-function in the Estimator block, just set its ``name'' argument to the name of a previously defined wave-function. In this case, the same object is used for both roles.
    Default: no
\item \textbf{nnodes}. Controls the parallel propagation algorithm. If nnodes $>$ 1, the nodes in the simulation are split into groups of nnodes nodes, each group working collectively to propagate their walkers.
    Default: 1 (Serial algorithm)
\item \textbf{hdf\_read\_file}. If provided, initializes the object from the given file.
    Default: ``''
\item \textbf{hdf\_write\_file}. If provided, generates a restart file.
    Default: ``''
\item \textbf{parallel\_factorization}. If ``yes'', calculates Cholesky decomposition in parallel. Mostly for testing purposes.
    Default: yes
\item \textbf{parallel\_propagation}. If ``yes'', uses parallel propagation algorithm even with nnodes=1. This is set to ``yes'' if nnodes $>$ 1. If ``no'', uses a serial propagation algorithm. Only possible if nnodes=1 and ncores=1 (in \texttt{execute} block).
    Default: yes
\item \textbf{dense}. If ``yes'', use a dense representation of the factorized hamiltonian. The default behavior uses a sparse representation. Which option is better depends on the sparsity of the problem.
    Default:  no \\
\end{itemize}

\texttt{execute}: Defines an execution region.
\texttt{<execute wset="wset0" ham="ham0" wfn="wfn0" prop="prop0" info="info0">}
\begin{itemize}
\item \textbf{nWalkers}. initial number of walkers per core group (see ncores). This sets the number of walkers for a given gorup of "ncores" on a node, the total number of walkers in the simulation depends on the total number of nodes and on the total number of cores on a node in the following way: $ \#_walkers_total = nWalkers * \#_nodes * \#_cores_total / ncores $. \\
    Default: 5
\item \textbf{timestep}. time step in 1/a.u. \\
    Default: 0.01
\item \textbf{blocks}. number of blocks. Slow operations occur once per block, e.g. write to file, slow observables, checkpoints, etc. \\
    Default: 100
\item \textbf{step}. number of steps within a block. Operations that occur at the step level include: load balance, orthogonalization, branching, etc. \\
    Default: 1
\item \textbf{substep}. number of substeps within a step. Only walker propagation occurs in a substep. \\
    Default: 1
\item \textbf{ortho}. number of steps between orthogonalization.
    Default: 1
\item \textbf{ncores}. Number of nodes in a task group. This number defines the number of cores on a node that share the parallel work associated with a distributed task. This number is used in the \texttt{Wavefunction} and \texttt{Propagator} task groups. The walker sets are shares by the ncores on a given node in the task group.
\item \textbf{checkpoint}. Number of blocks between checkpoint files are generated. If a value smaller than 1 is given, no file is generated. If \textbf{hdf\_write\_file} is not set, a default name is used. \textbf{Default: 0}
\item \textbf{samplePeriod}. Number of blocks between sample collection. \textbf{Default: 0}
\item \textbf{hdf\_write\_file}. If set (and checkpoint>0), a checkpoint file with this name will be written.
\item \textbf{hdf\_read\_file}. If set, the simulation will be restarted from the given file.\\
\end{itemize}

Within the \texttt{Estimators} xml block has an argument \textbf{name}: the type of estimator we want to measure. Currently only ``basic'' or ``Basic'' or ``standard'' are allowed, , which are synonymous with one another. The basic estimator has the following optional parameters:
\begin{itemize}
\item \textbf{EstimEloc} or \textbf{estim\_eloc} or \textbf{estimeloc}. print the local energy of the ``Estimator'' wave-function. Default: false
\item \textbf{timers}. print timing information. Default: true
\item \textbf{nwalk}. print population control information ({\bf max \# branching events etc?}). Default: false \\
\end{itemize}

\section{File formats}
... Coming Soon ...

\section{Advice/Useful Information}

AFQMC calculations are computationally expensive and require some care in order to obtain reasonable performance.
Below is a growing list of useful advice for new users followed by a sample input for a large calculation.
\begin{itemize}
\item Generate Cholesky-decomposed integrals with external codes instead of the 2-electron integrals directly. The generation of the Cholesky factorization is faster and consumes less memory.
\item Use the hybrid algorithm for walker propagation. Set steps/substeps to adequate values to reduce the number of energy evaluations. This is essential when using large multi-determinant expansions.
\item Adjust cutoffs in the wave-function and propagator bloxks until desired accuracy is reached. The cost of the calculation will depend on these cutoffs.
\item Adjust ncores/nWalkers to obtain better efficiency. Larger nWalkers will lead to more efficient linear algebra operations, but will increase the time per step. Larger ncores will reduce the time per step, but will reduce efficiency due to inefficiencies in the parallel implementation. For large calculations, values between 6-12 for both quantities should be reasonable, depending on architecture.
\end{itemize}

\begin{lstlisting}[caption=Example of sections of an AFQMC input file for a large calculation.]
...

  <Hamiltonian name="ham0" type="SparseGeneral" info="info0">
    <parameter name="filetype">hdf5</parameter>
    <parameter name="filename">fcidump.h5</parameter>
    <parameter name="cutoff_1bar">1e-6</parameter>
    <parameter name="cutoff_decomposition">1e-5</parameter>
  </Hamiltonian>

  <Wavefunction name="wfn0" info="info0">
    <ImpSamp name="impsamp0" type="PureSD" init="ground" >
      <parameter name="filetype">none</parameter>
      <parameter name="cutoff">1e-6</parameter>
    </ImpSamp>
    <Estimator name="impsamp0" />
  </Wavefunction>

  <WalkerSet name="wset0" type="distributed">
  </WalkerSet>

  <Propagator name="prop0" info="info0">
    <parameter name="hybrid">yes</parameter>
  </Propagator>

  <execute wset="wset0" ham="ham0" wfn="wfn0" prop="prop0" info="info0">
    <parameter name="ncores">8</parameter>
    <parameter name="timestep">0.01</parameter>
    <parameter name="blocks">10000</parameter>
    <parameter name="steps">10</parameter>
    <parameter name="substeps">5</parameter>
    <parameter name="nWalkers">8</parameter>
  </execute>
\end{lstlisting}


\section{Introduction}

Autonomous vehicles are considered to be one of the most challenging
types of reactive systems currently under development~\cite{AlurMT16,
  WongpiromsarnKF11, RamanDSMS15}. They need to interact reliably with
a highly reactive environment and crashes cannot be tolerated.  Life
critical decisions have to be made instantaneously and need to be
executed at the right point in time.

The development of autonomous vehicles and other cyberphysical systems
is supported by a wide spectrum of
programming and modeling methodologies,
including synchronous programming languages like Lustre~\cite{conf/popl/CaspiPHP87}
and Esterelle~\cite{conf/concur/BerryC84},
hardware-oriented versions of imperative programming languages like
SystemC~\cite{open2006ieee}, and visual languages like MSCs and Stateflow-charts~\cite{harel2003message,journals/scp/Harel87}.
The question of which programming paradigm is best-suited to write
easy-to-understand, bug-free code is still largely unresolved.

In the development of other forms of critical software, outside the
embedded domain, developers increasingly turn to functional
programming (cf.~\cite{frankau2009commercial}).  The strong type system in functional
languages largely eliminates runtime errors~\cite{cardelli1996type}.
Higher-order functions like \texttt{map} often eliminate the need for
explicit index counters, and, hence, the risk of ``index out of
bounds'' errors.  Functional purity reduces the possibility of
malformed state that can cause unexpected behavior.

While mathematical models of embedded and cyberphysical systems often
rely on functional notions such as stream-processing
functions~\cite{series/mcs/BroyS01,conf/csdm/Broy12}, the
application of functional programming in the practical development of
such systems has, so far, been limited. One of the most advanced
programming language in this direction is Ivory, which was used in the
development of autonomous vehicles~\cite{pike2014}.  Ivory is a
restricted version of the C programming language, embedded in Haskell.
It provides access to the low level operations necessary for embedded
system programming, but still enforces good programming practice, such
as disallowing pointer arithmetic, with a rich type system.

Ivory does not, however, have an explicit notion of time.
It cannot deal directly with the integration of
continuous and discrete time, which is fundamental for the
development of a cyberphysical system. For example,
in a car, continuous signals, such as the velocity or acceleration,
mix with the discrete steps of the digital controller.

In this paper, we investigate the use of functional programming
in a domain where the interaction between continuous and discrete signals
is of fundamental importance. We build a vehicle controller capable
of both autonomous vehicle control and multi-vehicle communication,
such as the coordination in platooning situations.

Our approach is based on Functional Reactive Programming
(FRP)~\cite{hudak2003arrows,hudak2000haskell}.
The fundamental idea of FRP is to extend the classic building blocks
of functional programming (\eg monads, arrows,
applicatives)
with the abstraction of a \emph{signal} to
describe time-varying values. FRP programs can be exceptionally
efficient.  For example, a network controller recently implemented as
an FRP program on a multicore processor outperforms any other such
controller existing today~\cite{Voellmy:2012:SSD:2377677.2377735}.

We have built a library, \ourLib, to use FRP to control a vehicle inside a simulation.
The library interfaces Haskell FRP programs to TORCS, The Open Racing Car Simulator, an open-source vehicle simulator~\cite{torcs}.
TORCS has been used in the Simulated Car Racing Championship competition~\cite{SCRC}, as well as other autonomous vehicle research projects~\cite{xu2016experimental,OnievaPAMP09,conf/cig/CardamoneLL09,conf/cig/MunozGS10}.
Through \ourLib, the Haskell program has access to the sensors and actuators of the car, as well
as communication channels between different vehicles.
Such a simulator is a  critical component of modern autonomous vehicle research, especially towards the goal of safe platooning algorithms~\cite{kamali2016formal}.

We report on our experience with two case studies, one in which we implement a controller for a solo car, and another for multi-vehicle platooning using a communication channel between the cars.
Our controller successfully navigates the TORCS preloaded tracks with reasonable speed and finesse while avoiding collisions (see Fig.~\ref{fig:race}).
Furthermore, the case study illustrates that the functional approach indeed leads to elegant, easily understandable, and safe code.
The ability to run full simulations for solo and platooning vehicles is a critical piece to advancing the state of the art in using FRP for autonomous vehicle control.

\begin{figure}[t]
\includegraphics[width=0.45\textwidth]{figs/racing.jpg}
\caption{A screenshot of Haskell controlling the autonomous vehicle in the TORCS simulator.}
\label{fig:race}
\end{figure}