\chapter{Bundle Adjustment}
\label{ch:bundle_adjustment}

\section{Overview}

\newenvironment{myindentpar}[1]
               {\begin{list}{}
                   {\setlength{\leftmargin}{#1}}
                 \item[]
               }
               {\end{list}}

\definecolor{lgray}{gray}{0.95}

Satellite position and orientation errors have a direct effect on the
accuracy of digital elevation models produced by the Stereo Pipeline.
If they are not corrected, these uncertainties will result in
systematic errors in the overall position and slope of the \ac{DEM}.
Severe distortions can occur as well, resulting in twisted or ``taco
shaped'' \acp{DEM}, though in most cases these effects are quite
subtle and hard to detect. In the worst case, such as with old mission
data like Voyager or Apollo, these gross camera misalignments can
inhibit Stereo Pipeline's internal interest point matcher and block
auto search range detection.
\begin{figure}[bt]
  \centering
  \includegraphics[width=8cm]{images/ba_orig.pdf}
  \includegraphics[width=8cm]{images/ba_adjusted.pdf}
  \caption{Bundle adjustment is illustrated here using a color-mapped,
    hill-shaded DEM mosaic from Apollo 15, Orbit 33, imagery. (a)
    Prior to bundle adjustment, large discontinuities can exist between
    overlapping DEMs made from different images. (b) After bundle
    adjustment, DEM alignment errors are minimized and no longer visible.}
  \label{fig:bundle_adjustment}
\end{figure}

Errors in camera position and orientation can be corrected using a
process called \emph{bundle adjustment}. Bundle adjustment is the
process of simultaneously adjusting the properties of many cameras and
the 3D locations of the objects they see in order to minimize the error
between the estimated, back-projected pixel locations of the 3D objects
and their actual measured locations in the captured images.

This complex process can be boiled down to this simple idea: bundle
adjustment ensures that the observations in multiple images of a
single ground feature are self-consistent. If they are not consistent,
then the position and orientation of the cameras as well as the 3D
position of the feature must be adjusted until they are.  This
optimization is carried out along with thousands (or more) of similar
constraints involving many different features observed in other
images.  Bundle adjustment is very powerful and versatile: it can
operate on just two overlapping images, or on thousands. It is also a
dangerous tool. Careful consideration is required to insure and
verify that the solution does represent reality.

Bundle adjustment can also take advantage of \acp{GCP}, which are
3D locations of features that are known a priori (often by measuring
them by hand in another existing \ac{DEM}). \acp{GCP} can improve the internal
consistency of your \ac{DEM} or align your \ac{DEM} to an existing data
product. Finally, even though bundle adjustment calculates the
locations of the 3D objects it views, only the final properties of
the cameras are recorded for use by the Ames Stereo Pipeline. Those
properties can be loaded into the \texttt{stereo} program which
uses its own method for triangulating 3D feature locations.

When using the Stereo Pipeline, bundle adjustment is an optional step
between the capture of images and the creation of \acp{DEM}. The bundle
adjustment process described below should be completed prior to
running the \texttt{stereo} command.

Although bundle adjustment is not a required step for generating
\acp{DEM}, it is {\em highly recommended} for users who plan to
create \acp{DEM} for scientific analysis and publication.  Incorporating
bundle adjustment into the stereo work flow not only results in
\acp{DEM} that are more internally consistent, it is also the correct
way to co-register your \acp{DEM} with other existing data sets and
geodetic control networks.

At the moment however, Bundle Adjustment does not automatically work
against outside \acp{DEM} from sources such as laser altimeters.
Hand-picked \acp{GCP} are the only way for \ac{ASP} to register to those
types of sources.

\section{Bundle adjustment using ASP}
\label{baasp}

Stereo Pipeline provides its own bundle adjustment
tool, named \texttt{bundle\_adjust}. Its usage is described in section
\ref{bundleadjust}.

Here is an example of using this tool on a couple of Apollo 15 images,
and its effect on decreasing the stereo triangulation error.

\begin{figure}[h!]
\centering
  \includegraphics[width=3.0in]{images/examples/before_ba.jpg}
  \includegraphics[width=3.0in]{images/examples/after_ba.jpg}
\caption{Illustration of the triangulation error map for a pair of
  images before (left) and after (right) using Stereo Pipeline's
  \texttt{bundle\_adjust}. Red and black colors suggest higher error.}
\label{fig:asp-ba-example}
\end{figure}

Running \texttt{stereo} without using bundle-adjusted camera models.
\begin{verbatim}
  stereo AS15-M-1134.cub AS15-M-1135.cub run_noadjust/run
\end{verbatim}

Performing bundle adjustment.
\begin{verbatim}
  bundle_adjust AS15-M-1134.cub AS15-M-1135.cub -o run_ba/run
\end{verbatim}

Running stereo while using the bundle-adjusted camera models.
\begin{verbatim}
  stereo AS15-M-1134.cub AS15-M-1135.cub run_adjust/run \
    --bundle-adjust-prefix run_ba/run
\end{verbatim}

A comparison of the two ways of doing stereo is shown in figure \ref{fig:asp-ba-example}.

ASP also offers the tool \texttt{parallel\_bundle\_adjust} which can
be much faster bundle adjusting many images at once.

\subsection{Floating intrinsics and using a lidar or DEM ground truth}
\label{floatingintrinsics}

This section documents some advanced functionality, and it suggested the
reader study it carefully and invest a certain amount of time to fully
take advantage of these concepts.

When the input cameras are of pinhole type, it is possible to optimize
the intrinsic parameters, in addition to the extrinsics. It is also
possible to take advantage of an existing terrain ground truth, such as a lidar file or a DEM,
to correct imperfectly calibrated intrinsic parameters, which can result in greatly
improved results, such as creating less distorted DEMs that agree much better
with the ground truth.

\subsubsection{A first attempt at floating the intrinsics}

We recommend that first bundle adjustment is run with the intrinsics fixed,
to get the extrinsics mostly correct, as optimizing for both of them at
the same time may result in a non-convex problem which may lead to a
suboptimal local minimum. Then, we will jointly optimize the intrinsics
and extrinsics.

Note that when solving for intrinsics, \texttt{bundle\_adjust} will by default
optimize all intrinsic parameters and will share them across all cameras (which must
be the same type).  You can control this behavior with the
\texttt{-\/-intrinsics-to-float} and \texttt{-\/-intrinsics-to-share} parameters.

Hence, the first invocation of camera optimization should be like:

\begin{verbatim}
  bundle_adjust -t nadirpinhole --inline-adjustments      \
    left.tif right.tif left.tsai right.tsai -o run_ba/run
\end{verbatim}

It is suggested that one run \texttt{stereo} with the obtained cameras,
and then examine the intersection error:

\begin{verbatim}
  stereo -t nadirpinhole --alignment-method epipolar left.tif right.tif \
     run_ba/run-left.tsai run_ba/run-right.tsai run_stereo/run
  point2dem --tr RESOLUTION --errorimage run_stereo/run-PC.tif
  gdalinfo -stats run_stereo/run-IntersectionErr.tif
  colormap run_stereo/run-IntersectionErr.tif
  stereo_gui run_stereo/run-IntersectionErr_CMAP.tif
\end{verbatim}

If desired, fancier stereo correlation algorithms can be used, such as MGM, as detailed in
chapter \ref{ch:correlation}. For \texttt{colormap}, \texttt{-\/-min} and \texttt{-\/-max}
bounds can be specified if the automatic range is too large. We also suggest inspecting
the interest points:
\begin{verbatim}
  stereo_gui left.tif right.tif run_ba/run
\end{verbatim}
and then viewing the interest points from the menu.

If the interest points are not well-distributed, this may result in large ray intersection errors
where they are missing. If so, they can be re-created by modifying \texttt{-\/-ip-detect-method}
and \texttt{-\/-ip-per-tile}. Or, one can take advantage of the just-completed stereo run
and invoke \texttt{stereo\_tri} with the additional option
\begin{verbatim}
  --num-matches-from-disp-triplets 10000
\end{verbatim}

to create dense and uniformly distributed interest points with desired density
(the latter creates a .match file that needs to be copied to the name
\texttt{bundle\_adjust} expects). This option also ensures that if three
images are present, and stereo is invoked on the first and second image,
and then on the second and the third, followed by interest point generation,
many interest points will be triplets, that is, the same feature will often
will be identified in all three images, which
can be a very good constraint on bundle adjustment later.

If the interest points are good and the mean intersection error is
acceptable, but this error shows an odd nonlinear pattern, that means
it may be necessary to optimize the intrinsics. We do so by using the
cameras with the optimized extrinsics found earlier, that is:

\begin{verbatim}
  bundle_adjust -t nadirpinhole --inline-adjustments               \
    --solve-intrinsics --camera-weight 1                           \
    left.tif right.tif run_ba/run-left.tsai run_ba/run-right.tsai  \
    -o run_ba_intr/run
\end{verbatim}

It is important to note that only the non-zero intrinsics will be
optimized, and the step size used in optimizing a certain intrinsic
parameter is proportional to it. Hence, if an intrinsic is 0 and it is
desired to optimize it, it should be set to small non-zero value
suggestive of its final estimated scale. If the algorithm fails to give
a good solution, perhaps different initial values for the intrinsics
should be tried. For example, one can try changing the sign of the
initial distortion coefficients, or make their values much smaller.

Sometimes the camera weight may need to be decreased, even all the way
to 0, if it appears that the solver is not aggressive enough, or it may
need to be increased if perhaps it overfits. This will become less of a
concern if there is some ground truth, as discussed later.

Next, one can run stereo as before, with the new cameras, and
see if the obtained solution is more acceptable, that is, if the
intersection error is smaller. It is good to note that a preliminary
investigation can already be made right after bundle adjustment, by
looking at the residual error files before and after bundle
adjustment. They are in the output directory, with names containing the
strings
\begin{verbatim}
  initial_residuals_no_loss_function_pointmap
  final_residuals_no_loss_function_pointmap
\end{verbatim}

If desired, these csv files can be converted to a DEM with
\texttt{point2dem}, which can be invoked with
\begin{verbatim}
  --csv-format 1:lon,2:lat,4:height_above_datum
\end{verbatim}

then one can look at their statistics, also have them colorized,
and viewed in \texttt{stereo\_gui}.

This file also shows how often each feature is seen in the images, so, if three
images are present, hopefully many features will be seen three times.

\subsubsection{Using ground truth when floating the intrinsics}

If a ground truth lidar file (or DEM) is present, say named \texttt{lidar.csv}, it can be used
as part of bundle adjustment. For that, the DEM obtained with the earlier
stereo pass needs to be first aligned to this ground truth, such as:

\begin{verbatim}
  pc_align --max-displacement VAL run_stereo/run-DEM.tif lidar.csv -o run_align/run
\end{verbatim}

(see the manual page of this tool in section \ref{pcalign} for more details).

This alignment can then be applied to the cameras as well:

\begin{verbatim}
  bundle_adjust -t nadirpinhole --inline-adjustments --max-iterations 0  \
    --initial-transform run_align/run-inverse-transform.txt              \
    left.tif right.tif run_ba/run-left.tsai run_ba/run-right.tsai        \
    -o run_align/run
\end{verbatim}

Here we have used 0 iterations because we simply want to move the cameras
without any optimization. Note that your lidar file may have some conventions as to what each column
means, and then any tools that use this cloud must set \texttt{-\/-csv-format}
and perhaps also \texttt{-\/-datum} and/or \texttt{-\/-csv-proj4}.

If \texttt{pc\_align} is called with the clouds in reverse order
(the denser cloud should always be the first), when applying the
transform to the cameras in \texttt{bundle\_adjust} one should use
\texttt{transform.txt} instead of \texttt{inverse-transform.txt} above.

Next, we will need to create a disparity from the left and right images
that we will use during bundle adjustment. For that we will take the disparity obtained
in stereo and remove any intermediate transforms stereo applied to the
images and the disparity. This can be done as follows:

\begin{verbatim}
  stereo_tri -t nadirpinhole --alignment-method epipolar left.tif right.tif  \
    run_ba/run-left.tsai run_ba/run-right.tsai run_stereo/run                \
    --unalign-disparity
\end{verbatim}

and then bundle adjustment can be invoked with this disparity and
the lidar/DEM file. Note that we use the cameras obtained after alignment:

\begin{verbatim}
  bundle_adjust -t nadirpinhole --inline-adjustments --solve-intrinsics              \
    left.tif right.tif run_align/run-run-left.tsai run_align/run-run-right.tsai      \
    --reference-terrain lidar.csv --disparity-list run_stereo/run-unaligned-D.tif    \
    --camera-weight 0 --max-disp-error 50 --max-num-reference-points 1000000         \
    --parameter-tolerance 1e-12 --reference-terrain-weight 5 -o run_ba_intr_lidar/run
\end{verbatim}

Here we set the camera weight all the way to 0, since it is hoped that having
a reference terrain is a sufficient constraint to prevent over-fitting.

This tool will write some residual files of the form
\begin{verbatim}
  initial_residuals_no_loss_function_reference_terrain.txt
  final_residuals_no_loss_function_reference_terrain.txt
\end{verbatim}

which may be studied to see if the error-to-lidar decreased. Each
residual is defined as the distance, in pixels, between a terrain point
projected into the left camera image and then transferred onto the right
image via the unaligned disparity and its direct projection into the
right camera.

If the initial errors in that file are large to start with, say more than 2-3 pixels,
there is a chance something is wrong. Either the cameras are not well-aligned to each
other or to the ground, or the intrinsics are off too much. In that case it is possible
the errors are too large for this approach to reduce them effectively.

We strongly recommend that for this process one should not rely on bundle adjustment
to create interest points, but to use the dense and uniformly distributed ones
created with stereo, as suggested earlier.

The hope is that after these directions are followed, this will result in a smaller intersection
error and a smaller error to the lidar/DEM ground truth (the later can be evaluated by
invoking \texttt{geodiff -\/-absolute} on the ASP-created aligned DEM
and the reference lidar/DEM file).

When the lidar file is large, in bundle adjustment one can use the flag
\texttt{-\/-lon-lat-limit} to read only a relevant portion of it. This
can speed up setting up the problem but does not affect the
optimization.

\subsubsection{Using the heights from a reference DEM}

In some situations the DEM obtained with ASP is, after alignment, quite
similar to the reference DEM, but the heights may be off. This can
happen, for example, if the focal length is not accurately known. It is
then possible after triangulating the interest point matches in bundle
adjustment to replace their heights above datum with values obtained
from the reference DEM, which are presumably more accurate. These
triangulated points can be kept fixed while the extrinsics and
intrinsics of the cameras are varied. The option for this is
\texttt{-\/-heights-from-dem arg}. To allow these triangulated
points to vary somewhat, one can pass a positive value to
\texttt{-\/-heights-from-dem-weight}. The larger its value is,
the more constrained those points will be.

This option can be used instead of the \texttt{-\/-reference-terrain}
option or together with it, and the DEM provided need not be the same for the two options.

It is important to note that here we assume that a simple height correction
is enough. Hence this option is an approximation, and perhaps it should be used
iteratively, and a subsequent pass of bundle adjustment should be done without it,
or one should consider using a smaller weight above.
This option can however be more effective than using \texttt{-\/-reference-terrain}
when there is a large uncertainty in camera intrinsics.

\subsubsection{Using multiple images}

Everything mentioned earlier works with more than two images, in fact,
having more images is highly desirable, and ideally the images overlap a
lot. For example, one can create stereo pairs consisting of first and
second images, second and third, third and fourth, etc., invoke the
above logic for each pair, that is, run stereo, alignment to the ground
truth, dense interest point generation, creation of unaligned
disparities, and transforming the cameras using the alignment transform
matrix. Then, a directory can be made in which one can copy the dense
interest point files, and run bundle adjustment with intrinsics
optimization jointly for all cameras. Hence, one should use a command as
follows (the example here is for 4 images):

\begin{verbatim}
  disp1=run_stereo12/run-unaligned-D.tif
  disp2=run_stereo23/run-unaligned-D.tif
  disp3=run_stereo34/run-unaligned-D.tif
  bundle_adjust -t nadirpinhole --inline-adjustments         \
    --solve-intrinsics  --camera-weight 0                    \
    img1.tif img2.tif img3.tif img4.tif                      \
    run_align_12/run-img1.tsai run_align12/run-img2.tsai     \
    run_align_34/run-img3.tsai run_align34/run-img4.tsai     \
    --reference-terrain lidar.csv                            \
    --disparity-list "$disp1 $disp2 $disp3"                  \
    --max-disp-error 50 --max-num-reference-points 1000000   \
    --overlap-limit 1 --parameter-tolerance 1e-12            \
    --reference-terrain-weight 5                             \
    -o run_ba_intr_lidar/run
\end{verbatim}
% $ %  Need this line to avoid confusing the syntax highlighting in XEmacs
In case it is desired to omit the disparity between one pair
of images, for example, if they don't overlap, instead of
the needed unaligned disparity one can put the word \texttt{none} in this list.

Notice that since this joint adjustment was initialized from several stereo pairs,
the second camera picked above, for example, could have been either the second camera
from the first pair, or the first camera from the second pair, so there was a choice
to make. In section \ref{sec:skysat} an example is shown where a preliminary bundle
adjustment happens at the beginning, without using a reference terrain, then those
cameras are jointly aligned to the reference terrain, and then one continues as
done above, but this time one need not have dealt with individual stereo pairs.

The option \texttt{-\/-overlap-limit} can be used to control which images should be
tested for interest point matches, and a good value for it is say 1 if
one plans to use the interest points generated by stereo, though a value
of 2 may not hurt either. One may want to decrease
\texttt{-\/-parameter-tolerance}, for example, to 1e-12, and set a value
for \texttt{-\/-max-disp-error}, e.g, 50, to exclude unreasonable
disparities (this last number may be something one should experiment with,
and the results can be somewhat sensitive to it). A larger value of
\texttt{-\/-reference-terrain-weight} can improve the alignment
of the cameras to the reference terrain.

Also note the earlier comment about sharing and floating the intrinsics individually.

\subsubsection{RPC lens distortion}
If it is realized that the optimized intrinsics still do not make the
ASP-generated DEMs agree very well with the ground truth, and some
residual and systematic error can be seen either by comparing these two
or in intersection error files, it may be convenient to convert the
current camera models to ones with the distortion given by rational
function coefficients (RPC) of a desired degre (section
\ref{pinholemodels}). An RPC model can have a lot more coefficients to
optimize, hence a better fit can be found. However, it is suggested to
use low-degree polynomials as those are easy to fit, and go to higher
degree only for refinement if needed.

An example showing how to convert a camera model to RPC is given in
section \ref{convertpinholemodel}.

\subsubsection{Working with map-projected images}

If stereo was done with map-projected images, one can still extract
dense interest point matches and the unaligned disparity from such a
run, and these can be applied with the original unprojected images for
the purpose of bundle adjustment (after being renamed
appropriately). This may be convenient since while bundle adjustment
must always happen with the original images, stereo could be faster and
more accurate when images are map-projected. It is suggested that the
unaligned disparity and interest points obtained this way be examined
carefully. Particularly the grid size used in mapprojection should be
similar to the ground sample distance for the raw images for best
results.

\section{Bundle adjustment using ISIS}

In what follows we describe how to do bundle adjustment using
\ac{ISIS}'s tool-chain. It also serves to describe bundle adjustment in more
detail, which is applicable to other bundle adjustment tools as well,
including Stereo Pipeline's own tool.

In bundle adjustment, the position and orientation of each camera
station are determined jointly with the 3D position of a set of image
tie-points points chosen in the overlapping regions between
images. Tie points, as suggested by the name, tie multiple camera images
together. Their physical manifestation would be a rock or small crater
than can be observed across more than one image.

Tie-points are automatically extracted using \ac{ISIS}'s
\texttt{autoseed} and \texttt{pointreg} (alternatively one could use a
number of outside methods such as the famous SURF\citep{surf08}).
Creating a collection of tie points, called a {\it control network}, is
a three step process. First, a general geographic layout of the points
must be decided upon. This is traditionally just a grid layout that has
some spacing that allows for about 20-30 measurements to be made per
image. This shows up in slightly different projected
locations in each image due to their slight misalignments. The second step
is to have an automatic registration algorithm try to find the same feature
in all images using the prior grid as a starting location. The third
step is to manually verify all measurements visually, checking to insure
that each measurement is looking at the same feature.

\begin{figure}[b!]
  \begin{center}
  \includegraphics[trim=20mm 20mm 20mm 15mm,clip,width=6in]{images/ba_feature_observation.pdf}
  \end{center}
  \caption{ A feature observation in bundle adjustment, from \citet{moore09} }
  \label{fig:ba_feature}
\end{figure}

Bundle Adjustment in \ac{ISIS} is performed with the \texttt{jigsaw}
executable. It generally follows the method described
in~\cite{triggs00} and determines the best camera parameters that
minimize the projection error given by ${\bf \epsilon} =
\sum_k\sum_j(I_k-I(C_j, X_k))^2$ where $I_k$ are the tie points on the
image plane, $C_j$ are the camera parameters, and $X_k$ are the 3D
positions associated with features $I_k$. $I(C_j, X_k)$ is an image
formation model (i.e. forward projection) for a given camera and 3D
point. To recap, it projects the 3D point, $X_k$, into the camera with
parameters $C_j$. This produces a predicted image location for the 3D
point that is compared against the observed location, $I_k$. It then
reduces this error with the Levenberg-Marquardt algorithm (LMA). Speed
is improved by using sparse methods as described in \citet{hartley04},
\citet{konolige:sparsesparse}, and \citet{cholmod}.

Even though the arithmetic for bundle adjustment sounds clever, there
are faults with the base implementation. Imagine a case where all
cameras and 3D points were collapsed into a single point. If you
evaluate the above cost function, you'll find that the error is indeed
zero. This is not the correct solution if the images were taken
from orbit. Another example is if a translation was applied equally to
all 3D points and camera locations. This again would not affect the
cost function. This fault comes from bundle adjustment's inability to
control the scale and translation of the solution. It will correct the
geometric shape of the problem, yet it cannot guarantee that the solution
will have correct scale and translation.

\ac{ISIS} attempts to fix this problem by adding two additional cost
functions to bundle adjustment. First of which is ${\bf \epsilon} =
\sum_j(C_j^{initial}-C_j)^2$. This constrains camera parameters to
stay relatively close to their initial values. Second, a small handful
of 3D ground control points can be chosen by hand and added to the
error metric as ${\bf \epsilon} = \sum_k(X_k^{gcp}-X_k)^2$ to
constrain these points to known locations in the planetary coordinate
frame. A physical example of a ground control point could be the
location of a lander that has a well known location. \acp{GCP} could also be
hand-picked points against a highly regarded and prior existing map
such as the THEMIS Global Mosaic or the LRO-WAC Global Mosaic.

Like other iterative optimization methods, there are several
conditions that will cause bundle adjustment to terminate. When
updates to parameters become insignificantly small or when the error,
${\bf \epsilon}$, becomes insignificantly small, then the algorithm
has converged and the result is most likely as good as it will get.
However, the algorithm will also terminate when the number of
iterations becomes too large in which case bundle adjustment may or
may not have finished refining the parameters of the cameras.

\subsection{Tutorial: Processing Mars Orbital Camera Imagery}
\label{sec:ba_example}

This tutorial for ISIS's bundle adjustment tools is taken from
\cite{lunokhod:controlnetwork} and \cite{lunokhod:gcp}. These tools
are not a product of NASA nor the authors of Stereo Pipeline. They
were created by USGS and their documentation is available at
\cite{isis:documentation}.

What follows is an example of bundle adjustment using two \ac{MOC}
images of Hrad Vallis. We use images E02/01461 and M01/00115, the same
as used in Chapter~\ref{ch:moc_tutorial}. These images are available from
NASA's \ac{PDS} (the \ac{ISIS} \texttt{mocproc} program will operate
on either the IMQ or IMG format files, we use the \texttt{.imq} below
in the example).  For reference, the following \ac{ISIS} commands are
how to convert the \ac{MOC} images to \ac{ISIS} cubes.

\begin{verbatim}
  ISIS 3> mocproc from=e0201461.imq to=e0201461.cub mapping=no
  ISIS 3> mocproc from=m0100115.imq to=m0100115.cub mapping=no
\end{verbatim}

Note that the resulting images are not map-projected. Bundle
adjustment requires the ability to project arbitrary 3D points into
the camera frame. The process of map-projecting an image dissociates
the camera model from the image. Map-projecting can be perceived as
the generation of a new infinitely large camera sensor that may be
parallel to the surface, a conic shape, or something more
complex. That makes it extremely hard to project a random point into
the camera's original model. The math would follow the transformation
from projection into the camera frame, then projected back down to
surface that ISIS uses, then finally up into the infinitely large
sensor. \texttt{Jigsaw} does not support this and thus does not
operate on map-projected imagery.

Before we can dive into creating our tie-point measurements we must
finish prepping these images. The following commands will add a vector
layer to the cube file that describes its outline on the globe. It
will also create a data file that describes the overlapping sections
between files.

\begin{verbatim}
  ISIS 3> footprintinit from=e0201461.cub
  ISIS 3> footprintinit from=m0100115.cub
  ISIS 3> echo *cub |  xargs -n1 echo > cube.lis
  ISIS 3> findimageoverlaps from=cube.lis overlaplist=overlap.lis
\end{verbatim}

At this point, we are ready to start generating our measurements. This
is a three step process that requires defining a geographic pattern
for the layout of the points on the groups, an automatic registration
pass, and finally a manual clean up of all measurements. Creating the
ground pattern of measurements is performed with \texttt{autoseed}. It
requires a settings file that defines the spacing in meters between
measurements. For this example, write the following text into a
\textit{autoseed.def} file.

\begin{verbatim}
  Group = PolygonSeederAlgorithm
        Name = Grid
        MinimumThickness = 0.01
        MinimumArea = 1
        XSpacing = 1000
        YSpacing = 2000
  End_Group
\end{verbatim}

The minimum thickness defines the minimum ratio between the sides of
the region that can have points applied to it. A choice of 1 would
define a square and anything less defines thinner and thinner
rectangles. The minimum area argument defines the minimum square
meters that must be in an overlap region. The last two are the spacing
in meters between control points. Those values were specifically
chosen for this pair so that about 30 measurements would be produced
from \texttt{autoseed}. Having more control points just makes for more
work later on in this process. Run \texttt{autoseed} with the
following instruction.

\begin{figure}[ht]
  \centering
  \includegraphics[width=5in]{images/qnet/Qnet_AfterAutoseed_400px.png}
  \caption{A visualization of the features laid out by
    \texttt{autoseed} in \texttt{qnet}. Note that the marks do not
    cover the same features between images. This is due to the poor
    initial spice data for MOC imagery.}
  \label{fig:after_autoseed}
\end{figure}

\begin{verbatim}
  ISIS 3> autoseed fromlist=cube.lis overlaplist=overlap.lis    \
            onet=control.net deffile=autoseed.def networkid=moc \
            pointid=???? description=hrad_vallis
\end{verbatim}

The next step is to perform auto registration of these features
between the two images using \texttt{pointreg}. This program also
requires a settings file that describes how to do the automatic
search. Copy the text box below into a \textit{autoRegTemplate.def}
file.

\begin{verbatim}
   Object = AutoRegistration
    Group = Algorithm
      Name         = MaximumCorrelation
      Tolerance    = 0.7
    EndGroup

    Group = PatternChip
      Samples = 21
      Lines   = 21
      MinimumZScore = 1.5
      ValidPercent = 80
    EndGroup

    Group = SearchChip
      Samples = 75
      Lines   = 1000
    EndGroup
  EndObject
\end{verbatim}

The search chip defines the search range for which \texttt{pointreg}
will look for matching imagery. The pattern chip is simply the kernel
size of the matching template. The search range is specific for this
image pair. The control network result after \texttt{autoseed} had a
large vertical offset in the ball park of 500 px. The large
misalignment dictated the need for the large search in the lines
direction. Use \texttt{qnet} to get an idea for what the pixel shifts
look like in your stereo pair to help you decide on a search range. In
this example, only one measurement failed to match automatically. Here
are the arguments to use in this example of \texttt{pointreg}.

\begin{verbatim}
  ISIS 3> pointreg fromlist=cube.lis cnet=control.net             \
             onet=control_pointreg.net deffile=autoRegTemplate.def
\end{verbatim}

The third step is to manually edit the control and verify the
measurements in \texttt{qnet}. Type \texttt{qnet} in the terminal and
then open \textit{cube.lis} and lastly
\textit{control\_pointreg.net}. From the Control Network Navigator
window, click on the first point listed as \textit{0001}. That opens a
third window called the Qnet Tool. That window will allow you to play
a flip animation that shows alignment of the feature between the two
images. Correcting a measurement is performed by left clicking in the
right image, then clicking \textit{Save Measure}, and finally
finishing by clicking \textit{Save Point}.

In this tutorial, measurement \textit{0025} ended up being
incorrect. Your number may vary if you used different settings than
the above or if MOC spice data has improved since this writing. When
finished, go back to the main Qnet window. Save the final control
network as \textit{control\_qnet.net} by clicking on \textit{File},
and then \textit{Save As}.

\begin{figure}[ht]
  \centering
  \includegraphics[width=5in]{images/qnet/Qnet_AfterQnetManual_400px.png}
  \caption{A visualization of the features after manual editing in
    \texttt{qnet}. Note that the marks now appear in the same location
    between images.}
  \label{fig:after_manual}
\end{figure}

Once the control network is finished, it is finally time to start
bundle adjustment. Here's what the call to \texttt{jigsaw} looks like:

\begin{verbatim}
  ISIS 3> jigsaw fromlist=cube.lis update=yes twist=no radius=yes \
             cnet=control_qnet.net onet=control_ba.net
\end{verbatim}

The update option defines that we would like to update the camera
pointing, if our bundle adjustment converges. The \textit{twist=no}
says to not solve for the camera rotation about the camera bore. That
property is usually very well known as it is critical for integrating
an image with a line-scan camera. The \textit{radius=yes} means that
the radius of the 3D features can be solved for. Using no will force
the points to use height values from another source, usually LOLA or
MOLA.

The above command will spew out a bunch of diagnostic information from
every iteration of the optimization algorithm. The most important
feature to look at is the \textit{sigma0} value. It represents the mean of
pixel errors in the control network. In our run, the initial error was
1065 px and the final solution had an error of 1.1 px.

Producing a DEM using the newly created camera corrections is the same
as covered in the Tutorial on page \pageref{ch:moc_tutorial}. When using
\texttt{jigsaw}, it modifies a copy of the spice data that is stored
internally to the cube file. Thus when we want to create a DEM using
the correct camera geometry, no extra information needs to be given to
\texttt{stereo} since it is already contained in the file. In the
event a mistake has been made, \texttt{spiceinit} will overwrite the
spice data inside a cube file and provide the original uncorrected
camera pointing.

\begin{verbatim}
  ISIS 3> stereo E0201461.cub M0100115.cub bundled/bundled
\end{verbatim}

%% \subsection{Processing with Ground Control Points}

%% Ground control point files describe a single point in the world
%% that is seen by 1 or more cameras. How they are measured in the
%% first place is up to the user. We use a manual process of comparing
%% each image to a respected map-projected image and then recording
%% the latitude, longitude, and altitude of the point(s). The maps to
%% register against can be anything, but it is recommended to register
%% against a product with a high amount of cartographic stability and
%% accuracy.  For terrestrial work, we would use a \ac{USGS} product
%% that can provide imagery that is registered to LIDAR height
%% measurements.

%% Unlike match files, ground control points must specifically be given
%% to \texttt{isis\_adjust} from the command line, but in no particular
%% order. Ground control point files are written with the extension
%% \texttt{.gcp}. Below is an example of a ground control point file that
%% was created to control a series of Apollo Metric Camera images from
%% several Apollo 15 orbits.

%% \begin{verbatim}
%%     -52.8452 27.2561 1735999 300 300 500
%%     sub4-AS15-M-2086.cub     210.9   3565.0
%%     sub4-AS15-M-2087.cub     1476.9  3579.0
%%     sub4-AS15-M-2088.cub     2798.9  3586.8
%%     sub4-AS15-M-2089.cub     4133.5  3588.6
%%     sub4-AS15-M-2344.cub     906.9   3874.8
%%     sub4-AS15-M-2345.cub     2204.2  3913.9
%%     sub4-AS15-M-2482.cub     939.8   4348.0
%%     sub4-AS15-M-2483.cub     2282.0  4340.7
%%     sub4-AS15-M-2484.cub     3642.1  4330.9
%% \end{verbatim}

%% The first line of a \texttt{.gcp} file is like a header line and
%% is different from the remaining lines.  The first line defines the
%% world location of the ground control point, and the rest of the
%% lines define the image locations of the ground control points. Here
%% are what the columns mean for the first line.

%% \begin{myindentpar}{2cm}
%% \begin{description}
%%   \item[Column 1:] Longitude in degrees
%%   \item[Column 2:] Latitude in degrees
%%   \item[Column 3:] Radius in meters
%%   \item[Column 4:] Sigma (or uncertainty) in meters for Local X axis
%%   \item[Column 5:] Sigma (or uncertainty) in meters for Local Y axis
%%   \item[Column 6:] Sigma (or uncertainty) in meters for Local Z axis
%% \end{description}
%% \end{myindentpar}

%% The other lines describe where this \ac{GCP} is found in each image:

%% \begin{myindentpar}{2cm}
%% \begin{description}
%%   \item[Column 1:] Image name
%%   \item[Column 2:] Sample (X) image measurement
%%   \item[Column 3:] Line (Y) image measurement
%% \end{description}
%% \end{myindentpar}

%% Make a {\tt .gcp} file for every ground control point, then be sure
%% to feed them as an input to {\tt isis\_adjust}. Remember that you
%% can scale the sigma of all ground control points by using the {\tt
%% -\/-gcp-scalar} flag. This can save time by allowing you to make
%% adjustments without needing to edit all of the files individually.


\section{Introduction to data science and machine learning}

Machine learning is a data science technique, which enables computers to learn.
It means that machines use algorithms to learn from data how to find a particular information and
they are not explicitly programmed to do that.
That type of algorithms improves its operation when the number of training samples increases so they are used to analyze large volumes of data.

With the rise of big data machine learning algorithms are used in such areas: natural language processing, image processing, recommendation systems and computational biology. Machine learning is applied in instances that we use in ordinary life like online recommendation systems --- friends suggestions on Facebook, Netflix movies suggestions based on our earlier choices.
It is also used in automatic article tagging on Wikipedia thanks to learned
associations between articles and categories assigned to them.

Machine learning uses two popular methods: supervised learning and unsupervised learning.
Supervised learning uses known input and output data so that it can predict outputs for future inputs.
It uses classification and regression methods to develop models.
Classification techniques classify input data into categories, they are used in medical imaging and speech recognition.
Regression techniques predict continuous responses, example usage can be predicting housing prices. Unsupervised learning groups and interprets data based only on given input.
The most common unsupervised learning technique is clustering,
which analyses input data to find patterns in it.
Typical usage of clustering method are recommendation systems.

In our thesis we consider problems with a large number of labels and we implement supervised learning classification algorithms.


\documentclass[11pt]{article}

\title{Data Simulation and Program Verification Using MATLAB and Maple V}
\author{Greg Ward \and Mark Wolforth}
\date{9 September, 1993}

\def\code#1{{\tt #1}}

% \def\simuldir{~greg/matlab/testing}
\newcommand{\units}[1]{\mbox{$\rm{#1}$}}
\newcommand{\funits}[2]{\mbox{$\rm\frac{#1}{#2}$}}

\begin{document}

\maketitle

\tableofcontents

%--------------------------------------------------------------
% \newpage
\section{Introduction}

This document is meant to provide a basic guide to performing data
simulation (and subsequent program verification) using Maple V and
MATLAB.  It was developed in conjunction with the rCBF analysis
package and its documentation, and uses the simulation of rCBF data
and verification of the analysis package as illustrative tools.
Furthermore, much of the information is specific to the Silicon
Graphics workstation environment at the McConnell Brain Imaging
Centre, particularly the presence of current versions of Maple,
MATLAB, and the rCBF analysis programs for MATLAB.

%--------------------------------------------------------------
% \newpage
\section{Software Tools}

Maple and MATLAB are both commercially available mathematical software
packages with rather different focuses: Maple deals with mathematics
almost entirely symbolically, while MATLAB is entirely numeric.  Maple
is more concerned with traditional algebraic entities like expressions
and functions, whereas MATLAB's primary data structure is the matrix
(and the special cases of vectors and scalars).  Thus, MATLAB is
better suited to dealing with scientific data, and Maple is
well-adapted to dealing with the theoretical functions which model the
data.

EMMA and rCBF are both sets of tools written at the MNI for MATLAB,
largely using MATLAB's built-in language.  EMMA consists of functions
to read both image and ancillary data from MINC files; rCBF uses EMMA
to read this data, and MATLAB's collection of mathematical tools (with
some enhancements) to calculate the kinetic parameters of either
the one- or two-compartment model of regional cerebral blood flow.
See the documents ``EMMA User Manual'' and ``rCBF Analysis Using
MATLAB'' for more information.

\subsection{A Brief Introduction to Maple V}

Since Maple V is not currently a heavily-used tool at McBIC, we will
first present a brief introduction to its capabilities.

\subsubsection{Starting Maple}

There are currently two interfaces to Maple accessible on the SGI
workstations at McBIC.  Typing \code{maple} at the shell prompt starts
up a {\em very} basic TTY-style interface to Maple.  This interface
provides minimal command-line editing, no command-line history, and
poor access to graphics.  (The easiest method is to use
\code{xterm(1)} on an X Windows display---either an SGI console or Sun
X terminal will work---and activate the Tektronix window {\em before}
issuing the Maple plot command.)

Alternatively, you can type \code{xmaple} at the shell prompt, as long
as you are using an X Windows display.  This opens up a new window for
Maple's use.  The interface is essentially the same as the TTY version,
but here you can at least edit command lines, recall previous command
lines (albeit in a genuinely bizarre way that must be experienced to be
believed), and put plots into their own windows automatically.  Neither
interface to Maple is really satisfactory, unless you make no typing
errors and have no objections to retyping nearly identical command
lines.

\subsubsection{Creating and Assigning Expressions}

Creating mathematical expression in Maple is quite easy.  For example,
try typing---and the semi-colon {\em is} important---
\begin{verbatim}
     sin (x) / x^2;
\end{verbatim}
and see that Maple ``prettyprints'' the expression in a form somewhat
more legible than the standard computer-readable format.  If we now wish
to manipulate this function in some way without retyping it, it is
necessary to assign it to a variable.  Actually, this is not strictly
true: you can access the most recent result of a Maple statement with
the \code{"} symbol, the second most recent result with \code{""}, and
the third most recent with \code{"""}.  In fact, we will use this
method to call the above expression \code{f}:
\begin{verbatim}
     f := ";
\end{verbatim}
%
Note the use of \code{:=} for assignment.  If you had typed
\begin{verbatim}
     f = ";
\end{verbatim}
then Maple would have simply created an ``equation'' object, which
would be accessible only through \code{"}.  In the former case,
\code{f} is logically equivalent to \verb|sin (x) / x^2|; that is,
using \code{f} in an expression is the same as using \verb|sin (x) / x^2|.
In the second case, \code{f} is only mathematically equivalent,
and only within the context of the expression itself. {\em This}
expression could be passed to a Maple function such as \code{solve} to
find the inverse of the expression to which \code{f} is equated.
(Don't bother trying it with the current expression---it's much too
complicated for Maple to solve symbolically for
\code{x}.  However, if you type \code{y~=~sin (x);} and then \code{solve (",~
x);}, Maple should solve correctly for \code{x}.  Or, to do the same thing in a
single step, try \code{solve (y = sin (x), x);}.)

Remember that you must always end a Maple statement with a semicolon
or a colon.  The only difference between the two is that using a colon
will cause Maple to suppress the output of the command.  This can be
handy in scripts (see below) when you do not want to mess up the
display with intermediate steps.

If you now want to make sure that f is in fact {\em logically} equivalent
to \verb|sin (x) / x^2|, just type \code{f;} and Maple will report on
the ``value'' of \code{f}.

Now, as an example of something Maple {\em can} do with \code{f}, type
\begin{verbatim}
     fsolve (f, x);
\end{verbatim}
The \verb|fsolve| function, as opposed to \verb|solve|, solves
equations numerically and thus will be able to find a solution more
often than \verb|solve|.  Note that \verb|fsolve| expects an equation
as its first argument, and we have passed a simple expression with no
\verb|=| anywhere in it.  In this case, \verb|fsolve| interprets
\verb|f| as the equation \verb|f = 0|---thus, we have just computed a
zero of $\frac{sin(x)}{x^{2}}$.

\subsubsection{Using Maple's Built-in Functions}

Some other built-in functions that will come in very handy are
\verb|subs|, \verb|evalf|, \verb|int|, and \verb|simplify|.
\verb|subs| allows you to substitute a portion of one expression
with another expression.  For example, try entering
\begin{verbatim}
     subs (x=y, f);
\end{verbatim}
and observe that Maple replaces \verb|x| in \verb|sin (x) / x^2| with \verb|y|.
(Note that this does {\em not} change the value of \verb|f|.  You can
generally only do that with another assignment.)  We can combine
\verb|evalf| and \verb|subs| to evaluate the expression at certain
values of \verb|x|.  For instance,
\begin{verbatim}
     evalf (subs (x=Pi, f));
\end{verbatim}
assures us that $\pi$ is indeed a zero of $\frac{sin(x)}{x^{2}}$, at
least to the precision Maple is capable of.

Next, let's symbolically integrate \verb|f| with respect to x:
\begin{verbatim}
     int (f, x);
\end{verbatim}
(That peculiar looking expression \verb|Ci| in the output is one of
Maple's built-in mathematical functions.  Since $\frac{sin(x)}{x^{2}}$
is not easily integrated, Maple uses \verb|Ci|, which involves the
gamma function and some other non-elementary functions, to express
this integral.  Type \verb|? Ci| for more information, including a
list of all of Maple's built-in mathematical functions).  Alternately,
we can compute a definite integral:
\begin{verbatim}
     int (f, x=Pi/2..Pi);
\end{verbatim}
We can find out what this is as a floating point number using
\verb|evalf| again:
\begin{verbatim}
     evalf (");.
\end{verbatim}

\subsubsection{Control Structures}

Maple provides a number of traditional programming structures,
including \verb|for/while/do| looping and \verb|if/then/else|
statements.  For detailed help, type \verb|? for| or \verb|? if|;
since we will be using the \verb|for| loop extensively in our data
simulation, a brief example is provided here.

A simple example using \verb|for| is:
\begin{verbatim}
     for i from 1 to 10 do
        lprint (i);
        lprint (evalf (sin (i)));
     od;
\end{verbatim}
which simply prints the integers from 1 to 10 and the sine of each
integer.  Although \verb|for| is generally used within programs, you
can type the above example into Maple interactively.

\subsubsection{Outputting Data}

Maple provides several built-in functions for saving expressions
(which can, of course, be either symbolic expressions or the numerical
output of \verb|evalf| or \verb|fsolve|) to text files.
Unfortunately, none of them work.  The option that appears most
convenient is the \verb|open| and \verb|writeln| pair.
Hypothetically, one could enter:
\begin{verbatim}
     readlib (write);                     # to have access to open/writeln
     open (foobar);                       # prepare to write to file foobar
     writeln (`Hello There`);             # and write to it
     writeln (sin (x));
     writeln (int (sin (x), x=0..Pi));
     writeln (evalf (subs (x=Pi/2, sin(x))));
\end{verbatim}
to write a string, a simple numerical expression, and two numbers (2
and 1) to the text file \verb|foobar|.  Unfortunately---as of this
writing, at least---Maple's \verb|open| function does not work: it
appears to do nothing, and the Maple prompt never returns.

Since it is (apparently) impossible to coax Maple into directly
writing to a text file, we will make use of the input/output
redirection provided by the Unix shell.  To do this, we first write a
text file containing the list of Maple commands whose output we wish
to capture.  This file can be terminated by a Maple ``\verb|quit;|''
command if we desire, but this is not necessary: when Maple is
finished reading from the file, it then switches to reading from your
terminal, where you can continue the session or type ``\verb|quit;|'' to
end it.  For instance, create a file called \verb|makesin| in your
current directory with the following contents:
\begin{verbatim}
     f := sin (x):
     for i from 0 by 0.5 to 10 do
        lprint (evalf (subs (x=i, f)));
     od;
\end{verbatim}
From the Unix shell prompt, type
\begin{verbatim}
     maple < makesin
\end{verbatim}
and note that we get $sin(x)$ evaluated at $x$ = 0, 0.5, 1, \ldots, 10.  To
save that output to a file called \verb|sinoutput|, we would have
typed \verb|maple < makesin > sinoutput|.  If we wish to use these
numbers in any way, the output file will most likely have to be edited
to remove the Maple prompts and commands read from \verb|makesin|.

%\subsection{A Very Brief Introduction to MATLAB, EMMA, and rCBF}

%Please consult the documents ``EMMA User Manual'' and ``rCBF Analysis
%Using MATLAB'' if you are not yet conversant with MATLAB and the EMMA
%and rCBF sets of functions.

%--------------------------------------------------------------
% \newpage
\section{Data Simulation}

\subsection{Mathematical Background}
For a more detailed mathematical exposition, please consult ``rCBF
Analysis Using MATLAB'' or one of its references.  Briefly, we wish
to generate a set of plasma data; pick reasonable values of $K_{1}$,
$k_{2}$, and $V_{0}$; and generate brain activity data $A^{*}(t)$ according
to the model equation
\begin{equation}
A^{*}(t) = K_{1} (C_{a}(t) \otimes e^{-k_{2}t}) + C_{a}(t)V_{0}.
\label{eq:pet_model}
\end{equation}
Note, however, that PET data is in reality
\begin{equation}
A(t_{mid,i}) = \frac{\int_{t_{start,i}}^{t_{end,i}} A^{*}(t) \,dt}{t_{end,i} - t_{start,i}}
\label{eq:frameint}
\end{equation}
where $t_{start,i}$ and $t_{end,i}$ are the start and end times of
frame $i$, and $t_{mid,i}$ is the mid-frame time ($ = (t_{start,i} +
t_{end,i}) / 2$).  Thus, in order to model real data, we must integrate
both sides of Eq. \ref{eq:pet_model} across each frame to obtain
\begin{equation}
A(t_{i}) = \frac{\int_{t_{start,i}}^{t_{end,i}} [K_{1} (C_{a}(t) \otimes e^{-k_{2}t}) + C_{a}(t)V_{0}] \,dt}{t_{end,i} - t_{start,i}}.
\label{eq:pet_data_model}
\end{equation}

Also, it has been found that the formula
\begin{equation}
C_{a}(t) = \sum_{i=1}^{4} a_{i} t^{n_{i}} e^{-t/b_{i}}
\label{eq:gen_ca}
\end{equation}
creates realistic-looking simulated plasma data when $a_{i}$ = (0.055,
10, 0.7, 0.2), $n_{i}$ = (6, 1, 1, 1), and $b_{i}$ = (1.8, 22, 180, 300)
[REF!!!].  The implementation of this formula will thus be the first step of
our simulation.  Also, the ``data'' created with this formula is ideal
plasma data, in that it is subject to neither dispersion or delay.
Thus, we can skip the blood correction step when analysing the
simulated data.

\subsection{Generating the Simulated Data in Maple}

Most of the Maple commands described in this section are also
available in various files in the directory
\verb|/local/matlab/toolbox/local/simul|.  In the remainder of this
document, it will be assumed that you have created a symbolic link to
this directory using the shell command
\begin{verbatim}
     ln -s /local/matlab/toolbox/local/simul simul
\end{verbatim}
from one of your own directories.  That way, you can refer to files in
the simulation directory using ``\verb|simul/filename|'' rather than
the unwieldy ``\verb|/local/matlab/toolbox/local/simul/filename|''.
Thus, any time that this document refers to the directory
\verb|simul/|, we are referring to \\
\verb|/local/matlab/toolbox/local/simul|.

The commands are stored as text files which are read into Maple with
the \verb|read| command.  For example, to read the file
\verb|makeplasma| (assuming you are in the directory where you created
the symbolic link \verb|simul|), enter
\begin{verbatim}
     read `simul/makeplasma`;
\end{verbatim}
in Maple.  Note the use of backquotes \verb|`| to denote a string in Maple.

There are a few minor differences between the commands presented here
and those available on-line; particularly, most commands in the
on-line version are terminated with colons rather than semicolons to
suppress Maple's output.  Also, none of the unit conversions described
herein are done by the scripts \verb|makeplasma|, \verb|makepet|, or
\verb|intframes|.  The reason for this is that there is also a script
\verb|makeall| which calls the three ``workhorse'' scripts.
\verb|makeall| outputs all required data (the calculated plasma
and brain activity, and the semi-arbitrarily chosen frame times and
lengths) in the appropriate units, as well as performing the unit
conversions necessary to the next step in the process.  After you have
worked through and understand the example code presented here, it
would be best to run the automated version to generate the output.

\subsubsection{Units}

In order to avoid headaches, we will do all calculations to generate
the simulated data using units of \units{counts}, \units{mL_{blood}},
\units{g_{tissue}}, and \units{sec}.  This will necessitate converting
the input \verb|Ca| from \units{\frac{counts}{g_{blood}\:sec}} to
\funits{counts}{mL_{blood}\:sec}, and the output \verb|A| from
\funits{counts}{g_{tissue}\:sec} to
\funits{nCi}{mL_{tissue}}, the standard units for PET data.
Furthermore, we must be sure to supply values of $K_{1}$, $k_{2}$, and
$V_{0}$ that make sense in units of
\funits{mL_{blood}}{g_{tissue}\:sec}, \units{sec^{-1}}, and
\funits{mL_{blood}}{g_{tissue}}.

\subsubsection{Generating Plasma Activity}

First, we wish to generate $C_{a}(t)$ according to Eq. \ref{eq:gen_ca}.  The
following Maple statements create 4-element vectors of $a_{i}$, $n_{i}$, and
$b_{i}$, and perform the summation:

\begin{verbatim}
     a := array (1..4, [0.055, 10, 0.7, 0.2]);
     n := array (1..4, [6, 1, 1, 1]);
     b := array (1..4, [1.8, 22, 180, 300]);

     Ca := 0;
     for i from 1 to 4 do
        Ca := Ca + 500 * a[i] * t ^ (n[i]) * exp (-t / (b[i]));
     od;
\end{verbatim}

Note the scaling of \verb|Ca| by a factor of 500.  This is simply to
get values of the appropriate order of magnitude, namely a function
peaking at around 150,000.  Since this is comparable to what is
experimentally observed in rCBF studies (neglecting dispersion
correction, which results in a slightly higher peak), we shall assign to the
generated \verb|Ca| units of \funits{counts}{g_{blood}\:sec},
which are the units that experimental plasma data is recorded in---and
therefore what the rCBF programs expect it to be in.

Since we will have to multiply \verb|Ca| by a scale factor in order to
create $A(t)$ in the expected units, it would be wise to save
\verb|Ca| to a file now, while it is in the desired units.  As
discussed above, however, Maple does {\em not} make this terribly
easy.  Thus, there is a Maple script in
\verb|simul/| called \verb|makeall| which
automates data generation; this file can be redirected to Maple's
input, and the resulting output redirected to a file which can then be
manually edited and split up for converting to MINC files.  This
procedure will be discussed below.

\subsubsection{Generating Brain Activity}

The first step in calculating $A(t)$ is to convert \verb|Ca| to
\funits{counts}{mL_{blood}\:sec}.  Since $1.05 \units{g} = 1
\units{mL}$ is assumed for both blood and tissue, this is done by
simply multiplying by 1.05:
\begin{verbatim}
     Ca := Ca * 1.05:
\end{verbatim}
Now, we wish to pick values of $K_{1}$, $k_{2}$, and $V_{0}$.  Note
that the ratio $\frac{K_{1}}{k_{2}}$ should be between 0.6
and 1.0 \funits{mL_{blood}}{g_{tissue}}; $V_{0}$ should be around 0.01 to
0.05 \funits{mL_{blood}}{g_{tissue}}.  With this in mind, we pick the
following parameters:
\begin{eqnarray*}
K_{1} &=& 75 \: \funits{mL_{blood}}{100\:g_{tissue}\:{min}} \\
k_{2} &=& 1 \: \units{min^{-1}} \\
V_{0} &=& 4 \: \funits{mL_{blood}}{100\:g_{tissue}}
\end{eqnarray*}
and enter them into Maple, with conversions to the desired units:
\begin{verbatim}
     K1 := 75 / 60 / 100;
     k2 := 1 / 60;
     V0 := 4 / 100;
\end{verbatim}
Of course, you should try a fairly wide range of parameter values,
even if they don't necessarily make physiological sense: as long as
$k_{2}$ is not out of the range of the main lookup tables, the MATLAB
rCBF programs should be able to recover all three values fairly well.
These assignment statements are in the file \verb|makepet|; you should
copy this file to your directory and edit it to reflect whatever
parameter values you wish to use.

Now we will perform the steps of Eq. \ref{eq:pet_model} to find
$A^{*}(t)$.  First, recall that the convolution operator~$\otimes$
denotes the following operation:
\begin{equation}
f(t) \otimes g(t) \equiv \int_{0}^{t} f(\tau) g(t-\tau) \: d\tau.
\label{eq:convolution}
\end{equation}
In Maple, we perform this operation with the following statements:
\begin{verbatim}
     arg1 := subs (t=tau, Ca);
     arg2 := exp (-k2 * (t-tau));
     convo := int (arg1 * arg2, tau=0..t);
\end{verbatim}
(\verb|arg1| and \verb|arg2| are used simply for clarity; the whole
operation is performed in one line in \verb|makepet|)

Unfortunately, Maple is unable to symbolically integrate the entire
expression.  We will still be able to numerically evaluate it, but we
cannot use Maple to generate a plot of the function (or rather, the
eventual end product $A^{*}(t)$) to ensure that it ``looks right'' for PET
brain activity.

Now, we complete the calculations of Eq. \ref{eq:pet_model}, and find
$A^{*}(t)$:
\begin{verbatim}
     Astar := K1 * convo + V0 * Ca;
\end{verbatim}

\subsubsection{From Theoretical Functions to Simulated Data}

To convert $A^{*}(t)$ to $A(t)$, we must create a set of frames across
which to integrate $A^{*}(t)$, and then numerically integrate
\verb|Astar| across each one.  Unfortunately, because of the $\int
\tau^{6} e^{(-\tau)} d\tau$ in \verb|Astar|, Maple is unable to integrate
the entire expression.  It can {\em evaluate} \verb|Astar| at a given
value of t---that is,
\begin{verbatim}
     evalf (subs (t=5, Astar));
\end{verbatim}
successfully calculates $A^{*} (5)$---but it cannot integrate
\verb|Astar|, even numerically.  That is,
\begin{verbatim}
     evalf (int (Astar, t=0..5));
\end{verbatim}
fails to calculate $\int_{t=0}^{5} A^{*} (t) dt$.

Note, however, that the ``frame-by-frame integration'' in
Eq.~\ref{eq:frameint} simply averages the function $A^{*}(t)$ across
each frame.  Thus, as long as the function is nearly linear over each
frame, then simply evaluating \verb|Astar| at each mid-frame time will
be a good approximation to integrating across the frame and dividing
by the frame length.  (In fact, this is the method used to perform
Eq.~\ref{eq:frameint} in previous implementations of the rCBF
analysis, where linear interpolation between data points was used to
``evaluate'' the function.)

We will pick the frame start times and frame lengths (all in seconds)
as follows (note that splitting the input across lines poses no
problems, because Maple only looks for the terminating semicolon):
\begin{verbatim}
     FrameStarts := array (1..21,
         [0,5,10,15,20,25,30,35,40,45,50,55,60,70,80,90,100,110,120,140,160]);
     FrameLengths = array (1..21,
         [4,5, 5, 4, 5, 4, 4, 5, 4, 5, 4, 5, 9,10, 9,10, 10, 10, 19, 19, 20]);
\end{verbatim}
(Actually, these are taken from study \verb|ARNAUD_20547| with the
decimal portion of the frame start times truncated.  Again, feel free
to copy the Maple script---this code is in \verb|intframes|---and
modify it.  Note that gaps between frames are allowed, since they do
occur in real data; overlapping frames, however, are not handled by
the rCBF software.)

Before creating an array of values of $A(t)$, we will scale
\verb|Astar| in order to convert from its present units of
\funits{counts}{g_{tissue}\:sec} to \funits{nCi}{mL_{tissue}}.  This
is done by simply multiplying by the cross-correlation coefficient,
which relates blood-count data to brain activity data.  This
coefficient is generally taken to be 0.11 \funits{nCi / mL}{count / g
\: sec}, therefore:
\begin{verbatim}
    Astar := Astar * 0.11;
\end{verbatim}

To create 21 values of $A(t)$ to simulate actual PET data, then, the
following \verb|for| loop is used:
\begin{verbatim}
     for i from 1 to 21 do
         MidTime := FrameStarts [i] + (FrameLengths [i] / 2)
         A [i] := evalf (subs (t=MidTime, Astar));
     od
\end{verbatim}

\subsection{Saving the Simulated Data in MINC Files}

At this stage, you should have the Maple scripts \verb|makeplasma|,
\verb|makepet|, \verb|intframes|, and \verb|makeall| in your own
directory.  Also, if you wish to make any modifications to the scripts
(e.g., to change the values of $K_1$, $k_2$, or $V_0$, or the frame
times or lengths) you should do so now.

Since Maple's outputting facilities are non-functional, we will have
to make use of some of the tools provided by Unix to make the job of
getting sensible numerical output out of Maple easier.  The first is
to use Maple as a pipe, reading its input from the file \verb|makeall|
(instead of from a terminal) and writing output to some other file.
Then, we will use a simple \verb|awk(1)| script to strip out
unnecessary output and split the output into several files.

Finally, we will convert the processed Maple output (which are
human-readable numbers) to machine-readable ``raw'' format, and then
to a MINC file.

\subsubsection{Getting Numbers Out of Maple}

To run the \verb|makeall| script and send the output to a file called
\verb|output|, type the following from the shell prompt:
\begin{verbatim}
     maple < makeall > output
\end{verbatim}
Since \verb|makeall| does not end with a \verb|quit| command, Maple
will simply wait for input from the terminal.  Just type \verb|quit;|
and Maple will exit.

The output file now contains all the data we need, as well as Maple's
prompts, input from the script, and occasional Maple status reports.
To get rid of the unneeded text and split the file into several
smaller files---one for each of \verb|Ca|, \verb|A|, \verb|FrameTimes|,
and \verb|FrameLengths|, as well as for the times at which \verb|Ca|
was sampled---we will use \verb|parse-output.awk| (also in
\verb|simul/|).  The command
\begin{verbatim}
     awk -f parse-output.awk output
\end{verbatim}
will scan the file \verb|output| for lines starting with
``\verb|START|'' followed by a filename; write all lines that are
numeric data (i.e. start with a digit) to the file named on the
\verb|START| line; and stop writing to that file when a line starting
with ``\verb|END|'' is encountered.  This is repeated for as many
\verb|START-END| pairs are found in \verb|output|.  If you have not
modified either \verb|makeall| or \verb|output| from the copies in
\verb|simul/|, then \verb|output| will contain
\verb|START-END| pairs to create the following files:
\begin{description}
\item \verb|plasma.txt| - the simulated plasma activity data.
\item \verb|sample-times.txt| - the start times of simulated ``blood samples''
\item \verb|sample-lengths.txt| - the length of simulated blood samples.
Together, \verb|sample-times.txt| and \verb|sample-lengths.txt|
specify the mid-sample times, which are the times at which \verb|Ca|
was sampled to create \verb|plasma.txt|.  However, we must store the
sample start times and lengths because they are what the rCBF software
expects to find in the MINC file.
%This is just
%the Maple expression \verb|Ca| evaluated at {\em mid-\/}sample times; i.e.,
%if the sample start times are 0, 2, 4, \ldots, 180 sec and all sample lengths
%are 2 sec, then the mid sample times will be 1, 3, \ldots, 181 sec.
\item \verb|frame-times.txt| - the start times of every frame of the
simulated PET study.
\item \verb|frame-lengths.txt| - the lengths of every frame in the
simulated study.
\item \verb|pet.txt| - the simulated PET data, sampled at the
mid-frame times (which are specified by the frame timing data in
\verb|frame-times.txt| and \verb|frame-lengths.txt|).
\end{description}

\subsubsection{Putting Text Data Into a MINC File}

While it is quite straightforward to read simple text files contain
columns of numbers into MATLAB (using the \verb|load| command), it
will be far more convenient to put the data into a MINC file as
expected by the rCBF programs.

This is done by means of a number of utility programs provided with
the EMMA package: \verb|micreate|, \verb|micreateimage|,
\verb|micreatedim|, and \verb|micreatevar|.  (Note: in order to
improve conformance to the MINC standard and simplify certain
operations within MATLAB, \verb|micreate| and \verb|micreateimage|
will eventually be merged into one program.  As of this writing,
however, they are distinct.)  Since these programs are really intended
to be run by other programs rather than by humans, they have rather
inflexible command line syntaxes: generally speaking, all arguments
must be provided in the exact order specified by the program's help.
The help is obtained by simply running each program with no arguments.
If you type the commands shown here exactly as is, you should have no
problems.

First, we must create the MINC file.  Type the following at the shell
prompt:
\begin{verbatim}
     micreate - simul.mnc
\end{verbatim}
The first argument to \verb|micreate| is a ``parent file,'' which is
generally used when creating MINC files of results calculated from a
particular study---this allows the new file to inherit attributes such
as the patient name or slice spacing.  Supplying a single \verb|-|
tells \verb|micreate| there is no such parent file.

Next, we create the image dimensions and variable.  Since we have
created only one brain activity curve, the image will be 1 by 1, with
1 slice and 21 frames.  The following command will create such an
image variable:
\begin{verbatim}
     micreateimage simul.mnc 21 1 1 1
\end{verbatim}

Now, we create the dimension and variables required to store the
plasma activity data.  There is only one dimension involved, and its
length is the number of simulated blood samples taken, or the number
of lines in the file \verb|plasma.txt|.  If \verb|makeall| was not
modified, this will be 91.  Thus, to create a NetCDF dimension
\verb|sample| with a length of 91, enter
\begin{verbatim}
     micreatedim simul.mnc sample 91
\end{verbatim}
If you {\em have} changed \verb|makeall|, you may need to re-count the
number of lines in \verb|plasma.txt|.  Do this by running the command
\begin{verbatim}
     wc plasma.txt
\end{verbatim}
The first number given is the number of lines.  If applicable, use
this number for the dimension length (in place of 91) in the above
call to \verb|micreatedim|.

Now, we must create several variables for the blood data.  The
variable \verb|blood_analysis| only exists as a tag to indicate that
the blood data is present in the MINC file.  (Actually, it is also
used to set up a hierarchy of MINC variables, but this aspect of MINC
files is ignored by this simulation.) The variables
\verb|sample_start|, \verb|sample_length|, and
\verb|corrected_activity| will contain useful data.  To create these
variables, type:
\begin{verbatim}
     micreatevar simul.mnc blood_analysis NC_LONG 0 -
     micreatevar simul.mnc sample_start NC_DOUBLE 1 sample
     micreatevar simul.mnc sample_length NC_DOUBLE 1 sample
     micreatevar simul.mnc corrected_activity NC_DOUBLE 1 sample
\end{verbatim}

Finally, we must convert the human-readable text files to
machine-readable ``raw'' format, and then put the data into the
newly-created MINC file.  There is a little conversion utility called
\verb|atod| (for ASCII to double) which we will run on our .txt files
as follows:
\begin{verbatim}
     simul/atod < plasma.txt > plasma.dat
     simul/atod < sample-times.txt > sample-times.dat
     simul/atod < sample-lengths.txt > sample-lengths.dat
     simul/atod < pet.txt > pet.dat
     simul/atod < frame-times.txt > frame-lengths.dat
     simul/atod < frame-lengths.txt > frame-lengths.dat
\end{verbatim}
Again, if you have changed any of the filenames, these commands should
be adapted appropriately.

We first write the raw file of image data to all 21 frames of the MINC
file.  Note that \verb|miwriteimages| expects slice and frame numbers
to be zero-based, so that frame numbers run from 0 to 20:
\begin{verbatim}
     miwriteimages simul.mnc - 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20 pet.dat
\end{verbatim}
Also, do {\em not} include spaces in the list of frame numbers.

Writing the other variables is done as follows (again, be sure to
change 91 or 21 to the number of samples or frames if you have changed
\verb|makeall|):
\begin{verbatim}
     miwritevar simul.mnc sample_start 0 91 sample-times.dat
     miwritevar simul.mnc sample_length 0 91 sample-lengths.dat
     miwritevar simul.mnc corrected_activity 0 91 plasma.dat
     miwritevar simul.mnc time 0 21 frame-times.dat
     miwritevar simul.mnc time-width 0 21 frame-lengths.dat
\end{verbatim}

\subsection{Running rCBF on the Simulated Data}

\verb|[K1,k2,V0] = rcbf2 ('simul.mnc', 1, 1, 0)|.

\end{document}


\section{Object tracking} \label{sec:tracking}

Analogous to the case of traditional 2D video consisting of separate discrete frames of pixels, a dynamic stream of 3D data is, in a simple case, individual ``frames'' of point sets.

The dynamic case requires tracking of individual points or objects in order to usefully handle the moved object(s).
Non-static human motion capture cases in video gaming or movies where post-processing time is available rely on manual work to perfect the quality.
A realtime case would need to e.g.~automatically register each frame between the previous one to use the geometry's dynamic properties.

The simplest tracking step is to leave tracking out completely: depending on the application, tracking might not be needed if the work done on the three-dimensional data does not need e.g. topological continuity in the time domain, but recomputes its work on each new point cloud.

Registration and tracking of point clouds or meshes is a large topic on its own; this section reviewes only some of the most common methods presented in the literature. Some work with the reconstructed 3D structure; others use the source images and do additional work.

Disorganised nature of the result data topology in frame-per-frame capture make it challenging to track individual points.
In three dimensions, a template model is often scanned beforehand that is then morphed to match the target object to keep the topology (i.e.~vertex neighborhood connectivity) constant.
This technique adapts to both static and rigid cases.
\cite{bojsen2012tracking,li2009robust}

Common method also for the entertainment industry is to use a pre-determined model of the scanned target or a completely separate character, and deform it on each frame based on the current state of the object, fitting the model's vertices to the scanned set.

Facial animation does not necessarily even need multi-view stereo for tracking and driving a predesigned character \cite{chuang2002performance,deng2007computer}.

\subsection{Registration}

When reconstructing an object all over again in consecutive frames, the points might not be fully aligned, and more importantly, they do not share the same model topology, i.e. the vertices have no other relation to the vertices in previous frames than being spatially near each other, accompanied with noise. \cite{zhao2005alignment}

Registration aligns disoriented models (e.g. point clouds, surfaces or triangular meshes, depending on the application) together with a rigid transformation between them so that one becomes the other from a reference coordinate frame's viewpoint.
Aligning two geometrically \textit{same} objects may not hold if the object deforms in a non-rigid way; when tracking the motion/deformation of a surface, the point sets describe a different geometry and there will always be some error.

A common method is Iterative Closest Point (ICP): iterative refinement approach to move the meshes closer to each other a small step at a time.
When ICP is used locally, it can be applied to non-rigid cases too. \cite{brown2007global}

\subsection{2D features}

In two dimensions, the full reconstruction step can be skipped when using only features in image space, providing real-time performance. \cite{pilet2005real}
When assuming that a precise object stays in the cameras, features can be detected and tracked locally with no reprojection, while the tracked object is assumed to keep the same topology as a previously scanned model.

\subsection{Optical flow}

Motion in an image shows as changes in pixel color; it can be perceived as similar pixels flowing to the direction of movement in a pattern.
A motion vector can be calculated for each pixel, resulting in a vector field; following and interpolating the movement is another way to detect three-dimensional movement.
\cite{gibson1950perception,horn1981determining,beauchemin1995computation}

By combining the movement tracking to a mesh whose dynamics are modeled physically, motion can be estimated from a single image.
\cite{decarlo1996integration}


\documentclass[]{BasiliskReportMemo}
\usepackage{AVS}

\newcommand{\submiterInstitute}{Autonomous Vehicle Simulation (AVS) Laboratory,\\ University of Colorado}

\newcommand{\ModuleName}{extPulsedTorque}
\newcommand{\subject}{Module to apply a cyclic pulsed disturbance torque}
\newcommand{\status}{First Version}
\newcommand{\preparer}{H. Schaub}
\newcommand{\summary}{This module allows the user to setup a cyclic pulsed external disturbance torque.  The pulses are symmetrically applying $\pm\bm\tau_{pulsed}$ followed by a specified off period before repeating.     }

\begin{document}

\makeCover

%
%	enter the revision documentation here
%	to add more lines, copy the table entry and the \hline, and paste after the current entry.
%
\pagestyle{empty}
{\renewcommand{\arraystretch}{1.1}
\noindent
\begin{longtable}{|p{0.5in}|p{4.5in}|p{1.14in}|}
\hline
{\bfseries Rev}: & {\bfseries Change Description} & {\bfseries By} \\
\hline
v1.0 & Initial document & H. Schaub \\
\hline

\end{longtable}
}

\newpage
\setcounter{page}{1}
\pagestyle{fancy}

\tableofcontents
~\\ \hrule ~\\

\begin{figure}[htb]
	\centerline{
	\includegraphics[]{Figures/TorqueDiagram1}
	}
	\caption{Illustration of Disturbance Torque acting on a rigid body}
	\label{fig:extPulse1}
\end{figure}
\section{Introduction}
This module allows a special pulsed external disturbance torque $\bm \tau$ to be applied onto a rigid body.   The torque is taken about the body-fixed point $B$, and the vector components are given in the body frame $\mathcal{B}$ as illustrated in Figure~\ref{fig:extPulse1}.

\begin{figure}[htb]
	\centerline{
	\includegraphics[]{Figures/TorqueDiagram2}
	}
	\caption{Illustration of Pulsed Disturbance Torque }
	\label{fig:extPulse2}
\end{figure}
\section{Specifying the Pulsed Disturbance Torque}
The module creates a cyclic disturbance torque which is applied to the rigid body.  The torque vector $\bm\tau$ is applied for equal time periods as $+\bm\tau$ and $-\bm\tau$.  This is followed by a specified off period before repeating.  This pattern is illustrated in Figure~\ref{fig:extPulse2}.

Note that the pulse and off periods are specified through integer counts of the simulation integration time.

\section{Module Parameters}
The external disturbance torque vector and pulsing parameters are set directly from python.

\subsection{{\tt pulsedTorqueExternalPntB\_B} Parameter}
This vector sets the external torque, about point $B$, in $\mathcal{B}$ body-frame vector components.

\subsection{{\tt countOnPulse} Parameter}
This integer represents the duration of both the $+\bm\tau$ and $-\bm\tau$ pulses.  The integer value represents how many integration time steps the pulse is on.

\subsection{{\tt countOff} Parameter}
This integer represents the off period duration between $\pm$ pulsing.  The integer value represents how many integration time steps the pulse is off.

\end{document}


%---------------------------------------------
% This document is for pdflatex
%---------------------------------------------
\documentclass[11pt]{article}

\usepackage{amsmath,amsfonts,amssymb,graphicx,natbib,setspace,authblk}
\usepackage[running]{lineno}
\usepackage[vmargin=1in,hmargin=1in]{geometry}
\usepackage{rotating}

\usepackage[compact]{titlesec}

\usepackage{graphicx}

\usepackage{times}

\usepackage{setspace}

\usepackage{enumitem}
\setlist{topsep=.125em,itemsep=-0.15em,leftmargin=0.75cm}

\usepackage[textsize=footnotesize,textwidth=2.1cm]{todonotes}
\newcommand{\smalltodo}[2][]
    {\todo[caption={#2}, #1]
    {\begin{spacing}{0.5}#2\end{spacing}}}
% There is nothing \smalltodo{here is a comment} here.
% \todo[inline]{Here is a comment}

\renewcommand{\floatpagefraction}{0.98}
\renewcommand{\topfraction}{0.99}
\renewcommand{\textfraction}{0.05}

\clubpenalty = 10000
\widowpenalty = 10000

\newcommand{\be}{\begin{equation}}
\newcommand{\ee}{\end{equation}}
\newcommand{\ba}{\begin{equation} \begin{aligned}}
\newcommand{\ea}{\end{aligned} \end{equation}}

\def\X{\mathbf{X}}
\def\A{\mathbf{A}}
\def\B{\mathbf{B}}
\def\C{\mathbf{C}}
\def\D{\mathbf{D}}
\def\G{\mathbf{G}}
\def\H{\mathbf{H}}
\def\N{\mathbf{N}}
\def\M{\mathbf{M}}
\def\W{\mathbf{\Omega}}
\def\P{\mathbf{P}}
\def\V{\mathbf{V}}

\title{Detecting and predicting the relationship between climate and population dynamics}

\author[1]{Andrew T. Tredennick\thanks{Corresponding author: atredenn@gmail.com}}
\author[1]{Peter B. Adler}
\author[2]{Giles Hooker}
\author[3]{Stephen P. Ellner}
\author[1]{Brittany J. Teller}
\affil[1]{Department of Wildland Resources and the Ecology Center, Utah State University, Logan Utah}
\affil[2]{Department of Ecology and Evolutionary Biology, Cornell University, Ithaca, New York}
\affil[3]{Department of Biological Statistics and Computational Biology, Cornell University, Ithaca, New York}
\renewcommand\Authands{, and }

\date{Last compile: \today}

\renewcommand{\baselinestretch}{1.2}
\linespread{1.5}

\begin{document}

\maketitle

\vspace{1in}

\large
{\emph{Running head}: Predicting climate effects}
\normalsize

\newpage

\renewcommand\linenumberfont{\normalfont\tiny\sffamily\color{gray}}
\linenumbers

\section*{Abstract}
Populations fluctuate in response to internal and external forces.
Quantifying the influence of external forces on population dynamics is a classic focus of population ecology, and it has taken on new relevance as we face the challenge of predicting species' responses to climate change.
However, detecting climate--population growth relationships is difficult and, once detected, the relationship may not improve out of sample prediction --- the gold standard of forecasting.
Here, we review empirical studies to assess how often including climate, or climate-related, covariates improves out of sample prediction of population abundances and dynamics.
We find that...
Drawing on examples from our literature review and our own work (failures?), we outline ways to determine in advance whether one can predict out of sample.
We then use a simulation approach to show how to identify the most efficient ways to improve predictive skill and, in turn, allocate research effort.

\subsection*{Keywords:} Prediction, climate, forecast, population, validation

\newpage

\section*{Introduction}
Why populations fluctuate has fascinated generations of ecologists.
Do populations fluctuate because of strong internal regulation and density-dependence?
Or, do external forces, like weather, determine population growth and decline from year-to-year?
Finding answers to these questions is intellectually satisfying, and doing so has taken on new relevance as we confront the challenge of predicting the impacts of climate change.
Whether our motivation is fundamental or applied, discriminating between internal and external population regulation requires quantifying the effects of external covariates on population growth, either directly or through demographic rates.

Unfortunately, detecting such climate--population growth relationships is not easy.
One issue is the high deminsionality of the problem: there are generally a handfull of plausible environmental or climate covariates that can potentially affect a number of fitness components, and each covariate must be aggregated over some time period to be tractable in a statistical model.
Another issue is that few population time series are long enough and replicated enough to meet the demands of estimating more than a couple covariate effects (Teller et al. 2016).
In other words, it is nontrivial to decide which covariates to use and difficult to detect significant effects of those covariates given data limitations.

Even when detecting a significant climate--population growth relationship is possible, doing so does not always lead to significantly better predictions.

\end{document}


\chapter{Shelley}

\section{Update proposals}
\label{shelley:hardfork}

\subsection{Moment of the hard fork}
\label{shelley:hardfork:moment}

Similar to the Byron ledger (\cref{byron:hardfork:moment}), the Shelley ledger
provides a ``current protocol version'', but it is a two-tuple (not a
three-tuple), containing only a \emph{hard fork} component and \emph{soft fork}
component:
%
\begin{lstlisting}
_protocolVersion :: (Natural, Natural)
\end{lstlisting}
%
in \lstinline!PParams!. The hard fork from Shelley to its successor will be
initiated once the hard fork component of this version gets incremented.

\subsection{The update mechanism for the protocol version}

The update mechanism in Shelley is simpler than it is in Byron. There is no
distinction between votes and proposals: to ``vote'' for a proposal one merely
submits the exact same proposal. There is also no separate endorsement step
(though see \cref{shelley:hardfork:initiating}).

The procedure is as follows:

\begin{enumerate}

\item
As in Byron, a proposal is a partial map from parameters to their values.

\item
During each epoch, a genesis key can submit (via its delegates) zero, one, or
many proposals; each submission overrides the previous one.

\item
``Voting'' (submitting of proposals) ends $6k/f$ slots before the end of the
epoch (i.e., twice the stability period, called \lstinline!stabilityWindow! in
the Shelley ledger implementation).

\item
At the end of an epoch, if the majority of nodes (as determined by the
\lstinline!Quorum! specification constant, which must be greater than half the
nodes) have most recently submitted the same exact proposal, then it is adopted.

\item
The next epoch is always started with a clean slate, proposals from the
previous epoch that didn't make it are discarded.\footnote{Proposals \emph{can}
be explicitly marked to be for future epochs; in that case, these are simply
not considered until that epoch is reached.}

\end{enumerate}

The protocol version itself is also considered to be merely another parameter,
and parameters can change without changing the protocol version, although a
convention could be established that the protocol version must change if any of
the parameters do; but the specification itself does not mandate this.

\subsection{Initiating the hard fork}
\label{shelley:hardfork:initiating}

The timing of the hard fork in Shelley is different to the one in Byron: in
Byron, we \emph{first} vote and then wait for people to get ready
(\cref{byron:hardfork:initiating}); in Shelley it is the other way around.

Core node operators will want to know that a significant majority of the core
nodes is ready (supports the hard fork) before initiating it. To make this
visible, Shelley blocks contain a protocol version. This is not related to the
current protocol version as reported by the ledger state
(\lstinline!_protocolVersion! as discussed in the previous section), but it is
the \emph{maximum} protocol version that the node which produced that block can
support.

Once we see blocks from all or nearly all core nodes with the `hard fork`
component of their protocol version equal to the post-hard-fork value, nodes
will submit their proposals with the required major version change to initiate
the hard fork.\footnote{This also means that unlike in Byron
(\cref{byron:unnecessary-restarts}), in Shelley there is no need to restart the
node merely to support a particular parameter change (such as a maximum block
size).}

\section{Forecasting}
\label{shelley:forecasting}

Discuss the fact that the effective maximum rollback in Shelley is $k - 1$,
not $k$; see also \cref{ledger:forecasting}.


\chapter{Open Research Directions} \label{Chap:Bench}

\section{Benchmarking of Task-based Heterogeneous Applications}

\lettrine[findent=2pt]{\textbf{T}}{}he benchmark is composed of six parallel applications. Five of them have been generated from Chameleon, a dense linear algebra software which is part of the MORSE project~\cite{agullo2012morse}, while the sixth has been generated with GGen, a library for generating directed acyclic graphs~\cite{GGen:simutools10}, and it corresponds to a more irregular application.

\subsection{Chameleon Benchmark}
This benchmark have been generated for the purpose of two scientific papers:
\begin{itemize}
 \bibitem[Mommessin {\rm{\em et~al.}} (2017)Mommessin, Amaris, Lucarelli, e
  Trystram]{Mommessin:2017}
Clement~Mommessin, {\bf Marcos~Amaris}, Giorgio~Lucarelli and Denis~Trystram.
\newblock Generic algorithms for scheduling applications on hybrid multi-core
  machines.
\newblock In \emph{2017 Euro-Par: International Conference on Parallel and
  Distributed Computing}, pages 220--231.

  \bibitem[Mommessin {\rm{\em et~al.}} (2018)Mommessin, Amaris, Lucarelli, e
  Trystram]{mommesin:2017:CCPE}
Clement~Mommessin, {\bf Marcos~Amaris}, Giorgio~Lucarelli and Denis~Trystram.
\newblock Generic algorithms for scheduling applications on heterogeneous multi-core platforms
\newblock \emph{arXiv preprint arXiv:1711.06433} in revision. Submmited in
\emph{Concurrency and Computation: Practice and Experience}
\end{itemize}

Five applications were chosen of Chameleon software to create this benchmark, these are  \texttt{getrf\_nopiv}, \texttt{posv}, \texttt{potrs}, \texttt{potri} and \texttt{potrf}, these aplication are composed of multiple sequential basic tasks of linear algebra. Task or kernels among are \texttt{SYRK} (symetric rank update), \texttt{GEMM} (general matrix-matrix multiply) and \texttt{TRSM} (triangular matrix equation solver), among other. Figure~\ref{fig:DAGspotrf} shows the DAG of the application Cholesky Factorization, which is the application \texttt{potrf}. For both workloads with 2 and 3 resource types the Chameleon applications were executed with the runtime StarPU~\citep{agullo2012morse} and the traces of executions were collected.

At first, all the applications were executed on CPUs and then were forced to execute on GPUs to have the processing times of each task of the application for different resources.

The five applications of Chameleon, named \emph{getrf$\_$nopiv}, \emph{posv}, \emph{potrs}, \emph{potri} and \emph{potrs}, are composed of multiple sequential basic tasks of linear algebra such as \emph{SYRK} (symetric rank update), \emph{GEMM} (general matrix-matrix multiply) and \emph{TRSM} (triangular matrix equation solver), as shown in Table~\ref{tab:kernels}.

\begin{figure}[htpb]
	\centering
    \includegraphics[scale=.3]{images/DAGspotrf.pdf}
    \caption{DAGspotrf}
    \label{fig:DAGspotrf}
\end{figure}

\begin{table}[htpb]
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|}
	\hline
	\backslashbox{\bf Kernels}{\bf Apps} & {\bf getrf$\_$nopiv} & {\bf posv} & {\bf potrf} & {\bf potri} & {\bf potrs}\\
	\hline
	{\bf syrk}& & x & x & x & \\
	\hline
	{\bf gemm}& x & x & x & x & x \\
	\hline
	{\bf trsm}& x & x & x & x & x \\
	\hline
\end{tabular}
\caption{Basic kernel of linear algebra of each application}
\label{tab:kernels}
\end{center}
\end{table}

To generate the applications, different tilings of the matrices have been used, varying the number of sub-matrices denoted by $nb\_blocks$ and the size of the sub-matrices denoted by $block\_size$. The different values of $nb\_blocks$ were 5, 10 and 20 and the different values of $block\_size$ were 64, 128, 320, 512, 768 and 960, for a total of 18 configurations per application. Table \ref{fig:nb_tasks} shows the total number of tasks for each application and each value of $nb\_blocks$. Notice that the value of $block\_size$ does not impact the number of tasks.

\begin{table}[htpb]
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|}
	\hline
	\backslashbox{\bf Nb$\_$blocks}{\bf Apps} & {\bf getrf$\_$nopiv} & {\bf posv} & {\bf potrf} & {\bf potri} & {\bf potrs} \\
	\hline
	{\bf 5} & 55 & 65 & 35 & 105 & 30 \\
	\hline
	{\bf 10} & 385 & 330 & 220 & 660 & 110 \\
	\hline
	{\bf 20} & 2870 & 1960 & 1540 & 4620 & 420 \\
	\hline
\end{tabular}
\caption{Total number of tasks in function of the number of blocks}
\label{fig:nb_tasks}
\end{center}
\end{table}
The Chameleon applications were executed with the runtime StarPU~\cite{Augonnet:StarPU} and the traces of executions were collected. At first, all the applications were executed on CPUs and then were forced to execute on GPUs to have the processing times of each task of the application for both computing units.

The machine where this data was collected had the following hardware characteristics: Dual core Xeon E7 v2, with a total of 20 physical cores  with hyper-threading of 3 GHz of processor base Frequency, 256 GB of RAM, operative system Linux Ubuntu 14.04. This machine had 4 GPUs NVIDIA Tesla K20 with each 4 GB of global memory, 200 GB/s of bandwidth and 2496 cores divided in 13 multiprocessors.

%The SWF was defined in order to ease the use of workload logs and models. With it, programs that analyze workloads or simulate system scheduling need only to be able to parse a single format, and can be applied to multiple workloads.
% The data set and other information are available\footnote{Hosted at: \url{https://github.com/marcosamaris/heterogeneous-SWF} [Accessed on 19 October 2016]} under Creative Commons Public License for the sake of reproducibility.

\subsection{Fork-joint Benchmark}
The application generated with GGen is \emph{fork-join}. It represents a real application that starts by executing sequentially and then forks to be executed in parallel with a specific diameter (number of parallel tasks), when the parallel execution has completed, results are aggregated by performing a join operation. This procedure can be repeated several times depending on the number of phases. For our experiments, we used 2, 5 and 10 phases with a diameter of 100, 200, 300, 400 and 500 for a total of 15 different configurations. Table \ref{fig:nb_tasksFork} shows the total number of tasks for each configuration of the fork-join application.

\begin{figure}[htpb]
	\centering
    \includegraphics[scale=.5]{images/forkjoin-3-5.png}
    \caption{Fork-join application with 3 phases and 5 parallel and concurrent tasks}
    \label{fig:forkjoin}
\end{figure}

\begin{table}[htpb]
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|}
	\hline
	\backslashbox{\bf Nb$\_$phases}{\bf Diameter} & {\bf 100} & {\bf 200} & {\bf 300} & {\bf 400} & {\bf 500} \\
	\hline
	{\bf 2} & 203 & 403 & 603 & 803 & 1003 \\
	\hline
	{\bf 5} & 506 & 1006 & 1506 & 2006 & 2506 \\
	\hline
	{\bf 10} & 1011 & 2011 & 3011 & 4011 & 5011 \\
	\hline
\end{tabular}
\caption{Total number of tasks in function of the number of phases and the width of the phase}
\label{fig:nb_tasksFork}
\end{center}
\end{table}
The running time of each task was computed using a Gaussian distribution with center $p$ and standard deviation $\frac{p}{4}$, where $p$ is the number of phases. We have decided to establish various acceleration factors in each diameter of fork. In this way, for all the configurations there are five parallel tasks with an acceleration factor between $0.1$ and $0.5$ while the others have an acceleration factor between $0.5$ and $50$.

% \subsection{SWF e Dependencies Files}
The execution time of each task and their respective dependencies were collected and formatted in different files, in SWF~(Standard Workload Format). The SWF standard was defined in order to ease the use of workload logs and models. With it, programs that analyze workloads or simulate systems scheduling need only to be able to parse a single format, and can be applied to multiple workloads.

% \subsection{Online Scheduling Techniques}

\section{Performance Predictions on Hybrid CPU-GPU 2D Stencil 5-point Application}

\input{sections/streams.tex}