The SPOT sensor placement solvers are launched with the
\doxyref{sp}{p.}{spExecutable} command. The {\ttfamily sp}
command reads in one or more IMPACT files, and computes a sensor
placement. Command-\/line options for {\ttfamily sp} can specify any
of a set of performance or cost goals as the objective to be optimized,
as well as constraints on performance and cost goals.

The {\ttfamily sp} command currently interfaces with three different
sensor placement optimizers:
\begin{itemize}

\item {\bfseries MIP solvers} -\/ Several different MIP solvers can be
used by the {\ttfamily sp} command: the commercial CPLEX solver and the
open-\/source PICO solver. These optimizers use the MIP formulations to
find globally optimal solutions. However, this may be a computationally
expensive process (especially for large problems), and the size of the
MIP formulation can become prohibitively large in some cases.

Two different MIP solvers can be used: the public-\/domain glpk solver
and the commercial  solver. PICO is included in distributions of SPOT.

\item {\bfseries GRASP heuristic} -\/
The GRASP heuristic performs
sensor placement optimization without explicitly creating a MIP
formulation. Thus, this solver uses much less memory, and it usually
runs very quickly. Although the GRASP heuristic does not guarantee that
a globally optimal solution is found, it has proven effective at finding
optimal solutions to a variety of large-\/scale applications.

Two different implementations of
the GRASP solvers can be used: an ATT commercial solver (att\_\-grasp)
and an open-\/source implementation of this solver (snl\_\-grasp).

\item {\bfseries Lagrangian Heuristic} -\/ The Lagrangian heuristic uses the structure of the p-\/median MIP formulation (eSP) to find near-\/optimal solutions while computing a lower bound on the best possible solution.

\end{itemize}

The following sections provide examples that illustrate the use of the
{\ttfamily sp} command. A description of {\ttfamily sp} command line options is
available in the \doxyref{Appendix}{p.}{spExecutable}.

The {\ttfamily sp} command has many different options. The following
examples show how different sensor placement optimization problems can
be solved with {\ttfamily sp}. Note that these examples can be run in
the {\ttfamily C:$\backslash$spot$\backslash$examples$\backslash$simple}
directory. The user needs to generate IMPACT files for these examples
with the following commands:
\lstinputlisting[firstline=11,lastline=12]{examples/simple/solver1.sh} % anchor=prelim

\section{A Simple Example}\label{solvers_solvers2}

The following simple example illustrates the way that SPOT has been most
commonly used. In this example, SPOT minimizes the extent of contamination
(ec) while limiting the number of sensors (ns) to no more than 5. This
problem formulation (eSP) can be efficiently solved with all solvers
for modest-\/size distribution networks, and heuristics can effectively
perform sensor placement on very large networks.

We begin by using the PICO solver to solve this problem, with the following command line:
\lstinputlisting[firstline=23,lastline=23]{examples/simple/solver1.sh} % anchor=sp
This specifies that network {\ttfamily Net3} is analyzed. The objective is to minimize {\ttfamily ec}, the extent of contamination, and an upper-\/bound of 5 is placed on {\ttfamily ns}, the number of sensors. The solver selected is {\ttfamily pico}, the PICO MIP solver.

This execution of the {\ttfamily sp} command uses the {\ttfamily Net3\_\-ec.impact} file and creates the following files: {\ttfamily Net3.log}, a logfile for the optimization solver, and Net3.sensors, a file with the sensor placement locations. Also, {\ttfamily sp} generates the following output:
\lstinputlisting[]{examples/simple/solver1.txt}

The initial information up to the statement \char`\"{}... PICO done\char`\"{} is simply output about what solver is run and information from the solver output. The next information beginning with \char`\"{}Sensor placement id:\char`\"{} is generated by \doxyref{evalsensor}{p.}{evalsensorExecutable}. This is a summary that describes the sensor placement and the performance of this sensor placement with respect to the impact measure that was minimized. This includes the following data:
\begin{itemize}
\item {\bfseries Sensor placement id} -\/ an integer ID used to distinguish this sensor placement
\item {\bfseries Number of sensors} -\/ the number of sensors in the sensor placement
\item {\bfseries Total cost:} -\/ the cost of the sensor placement, which may be nonzero if cost data is provided
\item {\bfseries Sensor node IDs} -\/ the internal node indexes used by {\ttfamily sp}
\item {\bfseries Sensor junctions} -\/ the EPANET junction labels for the sensor locations
\end{itemize}The performance of the sensor placement is summarized for each IMPACT file used with {\ttfamily sp}. The impact statistics included are:
\begin{itemize}

\item {\bfseries min} -\/ The minimum impact over all contamination
events. If we make the assumption that a sensor protects the node at
which it is placed, then this measure will generally be zero.

\item {\bfseries mean} -\/ The mean (or average) impact over all
contamination events.

\item {\bfseries lower quartile} -\/ 25\% of contamination events,
weighted by their likelihood, have an impact value less than this
quartile.

\item {\bfseries median} -\/ 50\% of contamination events, weighted by
their likelihood, have an impact value less than this quartile.

\item {\bfseries upper quartile} -\/ 75\% of contamination events,
weighted by their likelihood, have an impact value less than this
quartile.

\item {\bfseries VaR} -\/ The value at risk (VaR) uses a user-\/defined
percentile. Given $ 0.0 < \beta < 1.0$, VaR is the minimum value for
which $100*(1-\beta)$\% of contamination events have a smaller impact.

\item {\bfseries TCE} -\/ The tailed-\/conditioned expectation (TCE)
is the mean value of the impacts that are greater than or equal to VaR.

\item {\bfseries worst} -\/ The value of the worst impact.

\end{itemize}

Finally, a greedy sensor placement is described by {\ttfamily evalsensor}, which takes the five sensor placements and places them one-\/at-\/a-\/time, minimizing the mean impact as each sensor is placed. This gives a sense of the relative priorities for these sensors.
The {\ttfamily evalsensor} command can evaluate a sensor placement for a wide variety of different objectives. For example, the command
\lstinputlisting[firstline=27,lastline=28]{examples/simple/solver2.sh} % anchor=evalsensor
will summarize the solution in the {\ttfamily Net3.sensors} file for the {\ttfamily ec}, {\ttfamily mc} and {\ttfamily nfd} impact measures.

The following example shows how to solve this same problem with the GRASP heuristic. This solver finds the same (optimal) solution as the MIP solver, though much more quickly.
The command
\lstinputlisting[firstline=23,lastline=23]{examples/simple/solver3.sh} % anchor=sp
generates the following output:
\lstinputlisting[]{examples/simple/solver3.txt}

Finally, the following example shows how to solve this problem with the Lagrangian heuristic. This solver does not find as good a solution as the GRASP heuristic.
The command
\lstinputlisting[firstline=23,lastline=23]{examples/simple/solver4.sh} % anchor=sp
generates the following output:
\lstinputlisting[]{examples/simple/solver4.txt}

\section{Computing a Bound on the Best Sensor Placement Value}\label{solvers_solvers2a}

The following example shows how a lower bound can be computed on the best possible sensor placement. That is, any sensor placement would have an expected impact greater than this value. A bound is computed with the following syntax:
\lstinputlisting[firstline=23,lastline=24]{examples/simple/solver5.sh} % anchor=sp
This command generates the following output:
\lstinputlisting[]{examples/simple/solver5.txt}

\section{Minimizing the Number of Sensors}\label{solvers_solvers2b}

The sensor placement problem can be inverted by minimizing the number of sensors subject to a constraint on the extent of contamination. Note that the following example finds a solution with a single sensor that meets our goal of 40000 mean extent of contamination.
The command
\lstinputlisting[firstline=23,lastline=24]{examples/simple/solver6.sh} % anchor=sp
generates the following output:
\lstinputlisting[]{examples/simple/solver6.txt}

\section{Fixing Sensor Placement Locations}\label{solvers_solvers2c}

Properties of the sensor locations can be specified with the {\ttfamily -\/-\/sensor-\/locations} option. This options specifies a \doxyref{Placement Locations File}{p.}{formats_locationsFile} that can control whether sensor locations are feasible or infeasible, and fixed or unfixed. For example, suppose the file {\ttfamily locations} contains
\lstinputlisting[]{examples/simple/locations}
The following example shows how these restrictions impact the solution. Compared to the first example above, we have a less-\/optimal solution, since the sensor locations above cannot be used and junction {\ttfamily 161} must be included.
The command
\lstinputlisting[firstline=23,lastline=24]{examples/simple/solver7.sh} % anchor=sp
generates the following output:
\lstinputlisting[]{examples/simple/solver7.txt}

\section{Robust Optimization of Sensor Locations}\label{solvers_solvers4}

The following example demonstrates the optimization of sensor placements
using the TCE measure. TCE is the mean value of the worst incidents in
the ensemble being evaluated. Given a confidence level $1-\beta$, TCE is
the expectation of the worst impacts with probability $\beta$. Compared
with our first example, this finds a better solution in
terms of TCE, although the mean performance is slightly worse.

The command
\lstinputlisting[firstline=23,lastline=23]{examples/simple/solver8.sh} % anchor=sp
generates the following output:
\lstinputlisting[]{examples/simple/solver8.txt}

Note that the greedy ordering of sensors is less useful in this
case. Although we optimized to minimize TCE, the greedy ordering uses
the mean value to select each sensor location.

\section{Multi-\/Criteria Analysis}\label{solvers_solvers7}

The {\ttfamily sp} command supports multi-\/objective analysis through an iterative process. SPOT does not have a general \char`\"{}pareto search\char`\"{} optimizer. Instead, users can specify constraints with {\ttfamily sp} that ensure that previously optimized objectives are \char`\"{}close\char`\"{} to their previous values. In this way, the user can explicitly search for trade-\/offs between one-\/or-\/more performance objectives.

The examples above consider the extent-\/of-\/contamination objective.  The sensor placements generated above can be assessed as to how they minimize other objectives like the expected mass of contaminant consumed using {\ttfamily evalsensor}. Consider the solution generated by the previous example (which minimized {\ttfamily ec\_\-tce}), which was copied into the file {\ttfamily Net3\_\-ec.sensors}.

The command
\lstinputlisting[firstline=29,lastline=29]{examples/simple/solver9.sh} % anchor=evalsensor
generates the following output:
\lstinputlisting[]{examples/simple/solver9.txt}

The mean mass consumed is 70907, which is far from the optimal value of 21782 (which we computed separately). The robust optimization example in the previous section is revisited here; \char`\"{}extent of contamination -\/ tce\char`\"{} is still the primary objective, but now a \char`\"{}side constraint\char`\"{} is imposed that precludes any solution that admits an average mass consumed of worse than 30,000 units.
The command
\lstinputlisting[]{examples/simple/solver10.sh} % anchor=evalsensor
generates the following output:
\lstinputlisting[]{examples/simple/solver10.txt}

Note that the primary objective, minimizing the TCE of the \char`\"{}extent of contamination\char`\"{} measure, has gotten worse: it is now 50267 rather than 26376. However, our side constraint has been honored, and the mean mass consumed value is now 29350 rather than 70907.

\section{Sensor Placements without Penalties}\label{solvers_solvers9}

A fundamental issue for sensor placement is how to handle the fact
that a limited budget of sensors will not be able to cover all possible
incidents. SPOT addresses this issue by providing impact measures that
integrate an impact 'penalty' for incidents that are not detected by a CWS
design. Thus, in the previous examples there was an implicit trade-\/off
between impact reduction and reduction in the number of contamination
incidents that are detected.

SPOT also includes impact measures that do not contain these penalties,
which allows a user to more directly assess the performance of a CWS
design in the context where detections have occurred. For example, the
time-\/to-\/detection measure (td) includes a penalty for undetected
incidents, but the detected-\/time-\/to-\/detection measure (dtd) has
no penalty (or, more precisely, a zero penalty).

For example, consider the simple example above, which minimizes the
extent of contamination. The {\ttfamily evalsensors} command is applied to the final
solution to evaluate the {\bfseries ec}, {\bfseries dec} and {\bfseries
nfd} impact measures:
\lstinputlisting[firstline=22,lastline=23]{examples/simple/solver11.sh} % anchor=evalsensor
This generates the following output:
\lstinputlisting[]{examples/simple/solver11.txt}

In this example, the final sensor placement fails to detect 25\% of
the incidents. It is noteworthy that this does not impact the mean
performance very much, since the impact penalty has led to a final
solution that fails to detect few incidents with high penalties.

Note that minimizing {\bfseries dtd} does not really make sense. With
zero sensors, you detect no incidents, which means that the final
impact measurement is zero! Thus, minimizing {\bfseries dtd} requires
the additional constraint on the number of failed detections (nfd)
as well as a limit on the number of sensors (or total sensor costs).

Only the 'pico' SPOT solver currently supports optimization with 'detected' impact measures. For example:
\lstinputlisting[firstline=16,lastline=17]{examples/simple/solver12.sh}
generates the following output:
\lstinputlisting[]{examples/simple/solver12.txt}

\section{Limited-\/Memory Sensor Placement Techniques}\label{solvers_solvers5}

Controlling memory usage is a critical issue for solving large sensor
placement formulations. This is a particular challenge for MIP methods,
but both the GRASP and Lagrangian heuristics can have difficultly solving
very large problems on standard workstations. A variety of mechanisms have
been integrated into {\ttfamily sp} to reduce the problem representation
size while preserving the structure of the sensor placement problem.  These techniques include: scenario aggregation and filtering, feasible locations, witness aggregation, skeletonization, and memory management.

The \doxyref{scenarioAggr}{p.}{scenarioAggrExecutable}
method described in the previous section is one possible
strategy. This tool compresses the impact file while preserving
the fundamental structure of the impact file and it is appropriate
when optimizing for mean performance objectives. Similarly, the
\doxyref{filter\_\-impacts}{p.}{filter_impactsExecutable} script can
limit the sensor placement to only consider contamination incidents that
are \char`\"{}sufficiently bad\char`\"{} in the worst-\/case.

Another strategy is to limit the number of sensor placements, using the {\ttfamily
-\/-\/sensor-\/locations} option described above.  Eliminating
feasible locations reduces the problem representation used by the
{\ttfamily sp} solvers.

The {\itshape witness aggregation\/} technique can be used to
limit the size of the sensor placement formulation. This term refers to
the variables in the MIP formulation that \char`\"{}witness\char`\"{}
a contamination event. By default, variables that witness contamination
events with the same impact are aggregated, and this typically reduces
the MIP constraint matrix by a significant amount. Further reductions
can be performed with more aggressive aggregations.

To illustrate the use of witness aggregation,
we generated impact files with the {\ttfamily
C:$\backslash$spot$\backslash$etc$\backslash$tsg$\backslash$hourly.tsg}
TSG file. The following table illustrates the use of the two
witness aggregation options when optimizing the mean extent
of contamination: {\ttfamily -\/-\/aggregation-\/percent} and
{\ttfamily -\/-\/aggregation-\/ratio} (used with the {\ttfamily
-\/-\/distinguish-\/detection} option, which helps this aggregation
option). The second line of data in this table is the default aggregation,
which has about half as many non-\/zero values in the MIP constraint
matrix. Both the percent and ratio aggregation strategies effectively
reduce the problem size while finding near-\/optimal solutions.

\begin{center} \begin{TabularC}{5}
\hline
Aggr Type&Aggr Value&Binary Vars&MIP Nonzeros&Solution Value  \\\cline{1-5}
None&NA&97&220736&8525  \\\cline{1-5}
Percent&0.0&97&119607&8525  \\\cline{1-5}
Percent&0.125&97&49576&9513  \\\cline{1-5}
Ratio&0.125&97&12437&10991  \\\cline{1-5}
\end{TabularC}
\end{center}

Another option to reduce the memory requirement for sensor placement is to reduce the size of the network model through skeletonization.  Skeletonization groups neighboring nodes based on the topology of the network and pipe attributes.  The TEVA-SPOT Skeleton code, \doxyref{spotSkeleton} {p.}{skelExecutable}, provides techniques for branch trimming, series pipe merging, and parallel pipe merging.  Skeletonization eliminates pipes and junctions that have little impact on the overall hydraulics of the system.  This effectively contracts a connected piece of the network into a single node, called a supernode.  Skeletonized networks can be used to define geographic proximity in a two-tiered sensor placement approach for large network models.  Details on the two-tiered sensor placement approach can be found in \doxyref{Section 5.9}{p.}{solvers_solvers5a}.

Additionally, the GRASP heuristic has several options for controlling how memory is managed. The {\ttfamily -\/-\/grasp-\/representation} option can be used to control how the local search steps are performed. By default, a dense matrix is precomputed to perform local search steps quickly, but a sparse matrix can be used to perform local search with less memory. Also, the GRASP heuristic can be configured to use the local disk to store this matrix. It should be noted that the Lagrangian heuristic requires less memory than the GRASP heuristic, and thus similar techniques have not been developed for it.

\section{Two-\/tiered Sensor Placement Approach}\label{solvers_solvers5a}

The two-tiered sensor placement approach uses geographic aggregation on the large impact files to select candidate locations for secondary optimization.  While the approach uses detailed hydraulic calculations, it avoids solving sensor placement using the large impact file directly, a process that can exceed the memory available on a personal computer.   Geographic aggregation combines witness locations over all scenarios based on their geographic proximity.  To maintain hydraulic equivalence as much as possible, skeletonization is used to define geographic proximity.  This process is similar to witness aggregation, where witness locations for a single scenario are combined based upon their witness quality.  In both cases, the number of possible sensor locations is reduced.

Geographic aggregation reduces the impact file by grouping nodes that are co-located in the skeletonization process.  In essence, this reduces the impact file to include only skeletonized nodes (i.e. supernodes) defined in the skeletonized network.  From the skeletonized viewpoint, the contaminant hits a supernode when it first crosses the curve defining the supernode.  Depending upon the direction the contaminant approaches, it will hit the real nodes inside the supernodes in different orders.  With geographic aggregation, the minimum, maximum, mean, or median could be used to group the time and impact for the real nodes in each supernode.  For a sensor on a single real node in the supernode, the maximum statistic is pessimistic for incidents that the node detects.  Similarly, using the minimum is always optimistic.  For example, consider one supernode that represents four nodes in the original network.  The impact values for those nodes are 100, 200, 250, and 300.  After geographic aggregation, the impact value is 100 using the minimum, 300 using the maximum, 212.5 using the mean, and 225 using the median.  The time associated with each of these impacts is grouped the same way.  The new time and impact pair is the aggregated representation of the impact file for that node.  For this supernode, four lines of the impact file are reduced to one.  This aggregation is performed for each scenario and each supernode. While this greatly reduces the size of the impact file, the number of scenarios remains the same.  This method is not influenced by the modified hydraulics of the skeletonized network; rather, the original impact file is modified to reflect geographic aggregation from the skeletonized network.

In the first-tier (Tier 1), sensor placement filters candidate sensor locations based on a geographic aggregation of the impact file. Heuristically, sensor placement in this course search identifies supernodes that are promising regions for sensors.  Only real nodes within these selected supernodes are considered candidate sensor locations. All other locations from the impact file generated from the original, most-refined network model are removed.  The second-tier (Tier 2) sensor placement uses this refined impact file to place sensors in the original network.  Since sensor placement in each tier runs on a smaller impact file than a direct placement on the original network, the two-tiered approach reduces the maximum memory requirement.

The two-tiered sensor placement approach can be run using \doxyref{sp-2tier} {p.}{sp2tierExecutable}.   The method calls \doxyref{spotSkeleton} {p.}{skelExecutable} to define geographic proximity from skeletonized networks.  The two-tiered method has been shown to maintain solution quality while greatly reducing the memory footprint need for sensor placement.  For additional details, see Klise, Phillips, and Janke \cite{KliPhiJan11review}.

The {\bfseries sp-2tier} executable is run with the following command line:
\lstinputlisting[firstline=16,lastline=17]{examples/simple/solver16.sh}

This command generates aggregated and refined impact files, along with output files from {\bfseries sp} based on Tier 1 and Tier 2 sensor placement.

\section{Evaluating a Sensor Placement}\label{solvers_solvers6}

The \doxyref{evalsensor}{p.}{evalsensorExecutable} executable
takes sensor placements in a \doxyref{Sensor Placement
File}{p.}{formats_sensorPlacementFile} and evaluates them using data from
an \doxyref{Impact File}{p.}{formats_impactFile} (or a list of impact
files). This executable measures the performance of each sensor placement
with respect to the set of possible contamination locations. This analysis
assumes that probabilities have been assigned to these contamination
locations, and if no probabilities are given then uniform probabilities
are used by {\ttfamily evalsensor}.

The following example illustrates the use of {\ttfamily evalsensor}
after running the first sensor placement optimization example.
The command
\lstinputlisting[firstline=30,lastline=30]{examples/simple/solver13.sh}
generates the following output:
\lstinputlisting[]{examples/simple/solver13.txt}

The {\ttfamily evalsensors} command can also evaluate a sensor placement in the case where sensors can fail, and there is some small number of different classes of sensors (grouped by false negative probability). This information is defined by a \doxyref{Sensor Class File}{p.}{formats_sensorClass} and a \doxyref{Junction Class File}{p.}{formats_junctionClass}.  Consider the sensor class or sc file, {\ttfamily Net3.imperfectsc}, which defines different categories of sensor failures:
\lstinputlisting[]{examples/Net3/Net3.imperfectsc}
Sensors of class \char`\"{}1\char`\"{} give false negative readings 25\% of the time, sensors of class \char`\"{}2\char`\"{} give them 50\% of the time, etc.

Once failure classes have been defined, the junctions of the network are assigned to classes. This is done with a junction class or jc file, like {\ttfamily Net3.imperfectjc} (here we show just the beginning of this file):
\lstinputlisting[firstline=0,lastline=10]{examples/Net3/Net3.imperfectjc}
Given the junction classes, we can run {\ttfamily evalsensor} to determine the expected impact of a sensor placement, given that sensors may fail. Again, using the solution from the original example:
\lstinputlisting[firstline=30,lastline=31]{examples/simple/solver14.sh}
generates the following output:
\lstinputlisting[]{examples/simple/solver14.txt}

Note that the mean impact of this \char`\"{}extent of contamination\char`\"{} changes dramatically if sensors are allowed to fail. The original solution, 8499 pipe feet, was misleading if sensors fail according to the probabilities we have assigned. With sensor failures, the expected impact is 17193 pipe feet -\/-\/ more than twice the \char`\"{}perfect sensor\char`\"{} impact.

\section{Sensor Placement with Imperfect Sensors}\label{solvers_solvers3}

The GRASP heuristics in SPOT can optimize sensor placements that take into account sensor failures. For example, we can perform sensor placement optimization with imperfect sensors using the {\ttfamily Net3.imperfectsc} and {\ttfamily Net3.imperfectjc} files defined in the previous section.
The command
\lstinputlisting[firstline=16,lastline=17]{examples/simple/solver15.sh} % anchor=sp
generates the following output:
\lstinputlisting[]{examples/simple/solver15.txt}

After this optimization, the mean impact is 13469 pipe feet rather than the 17193 pipe feet value for the solution optimized with perfect sensors. Thus, it is clear that the GRASP heuristic makes different choices if the sensors are imperfect.

\section{Summary of Solver Features}\label{solvers_solversSummary}

The following table highlights the capabilities of the SPOT optimizers. The previous examples illustrate SPOT's capabilities, but the advanced features in SPOT are not available for all optimizers.

 \begin{TabularC}{4}
\hline
{\bfseries Solver Feature} &{\bfseries MIP} &{\bfseries GRASP} &{\bfseries Lagrangian}  \\\cline{1-4}
Minimize mean impact$^{1}$ &YES &YES &YES  \\\cline{1-4}
Minimize worst impact$^{2}$ &YES &YES &NO  \\\cline{1-4}
Minimize number of sensors$^{3}$ &YES &NO &NO  \\\cline{1-4}
Robust objectives$^{4}$ &YES &YES &NO  \\\cline{1-4}
Side-\/constraints$^{5}$ &YES &YES &YES  \\\cline{1-4}
Fixed/Invalid locations$^{6}$ &YES &YES &NO  \\\cline{1-4}
Witness aggregation$^{7}$ &YES &NO &YES  \\\cline{1-4}
Incident probabilities$^{8}$ &YES &NO &YES  \\\cline{1-4}
Incident aggregation$^{9}$ &YES &NO &YES  \\\cline{1-4}
Sparse data management$^{10}$ &NO &YES &NO  \\\cline{1-4}
Imperfect sensor model$^{11}$ &NO &YES &NO  \\\cline{1-4}
One imperfect witness$^{12}$ &YES &YES &NO \\\cline{1-4}
Computes lower bound$^{13}$ &YES &NO &YES  \\\cline{1-4}
\end{TabularC}

Footnotes:\\
1. The mean sensor placement formulation has been tested on a wide range of
network models. The GRASP heuristic reduces memory requirements for large
problems with reliable solutions.  The Lagrangian solver can be used to find a
lower bound. The Langrangian solver solutions generally have larger errors than
GRASP.  MIP is provably optimal but can be slower and require more memory.\\
2. Worst case is slow with GRASP.  GRASP has much larger error ratios than for
mean.  The standard MIP formulation is difficult and may not solve.
Improvements pending.\\
3. MIP can be slow for a mean side-constraint.  Improvements pending for
worst-case side constraints.  Other side constraints not tested.\\
4. VAR and CVAR are slow with GRASP.  GRASP has much larger error ratios than
for mean.  The standard MIP formulation is difficult and may not solve. \\
5. Not extensively tested.  MIP and GRASP side constraints are hard. Lagrangian
is soft/heuristic.  Side constraint may not be satisfied.  Likely worse for
main objective as well.  Developers recognize the need to tune this.\\
6. Fixed and invalid locations currently work with both MIP and GRASP solvers.
Pending for Lagrangian.\\
7. Witness aggregation significantly reduces solution quality for Lagrangian,
though does reduce memory.  No-error aggregation is on by default for MIP.\\
8. Incident weightings are only implemented for mean or CVAR.\\
9. Incident aggregation introduces no errors and can reduce space.  In initial
testing, the prefix property necessary for this aggregation is rare, so this is
off by default given the cost of searching and implementing.\\
10. This is not supported in the SNL GRASP implementation.\\
11. This is a nonlinear formulation.\\
12. This is a linear approximation for imperfect sensors implemented with the
mean solvers.  This is much faster than the GRASP solver for the nonlinear
formulation and seems to give values close to the GRASP solution value.\\
13. Lower bounds provably show how close a solution is to optimal.  MIP solvers
give gaps or give provably optimal solutions.  A lower bound from the Lagrangian
solver can validate the quality of a solution (upper bound) from GRASP.\\


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%-------------------------------------------------------------------
% Software Project Evaluation
%-------------------------------------------------------------------
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\selectlanguage{british}
\chapter{Software Project Evaluation}
\label{toc:evaluation}

In order to evaluate a software development project and its outcome, a
set of evaluation metrics is required. The evaluation of a project is
an important requirement for concrete software process improvement. An
abundant amount of methods exist for measuring the software itself,
but metrics for agile development processes are scarce. In this
thesis, a metric for measuring the software itself is presented
alongside with a metric for evaluating the adoption of Extreme
Programming practices.

First, the Chidamber-Kemerer metrics \citep{oodmetrics} for measuring
object-oriented software are presented. The \abbrev{CK} metric suite
is a known and used set of methods for measuring different quality
aspects of software. After that, the Extreme Programming Evaluation
Framework \citep{xpevaluationfw} is introduced. The \abbrev{XP-EF} is
a set of metrics that provide information on how well the \abbrev{XP}
methods have been taken into use and how successful the team has been.
The \abbrev{XP-EF} metrics have been designed so that they can be used
in a small team without a dedicated metrics specialist.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Chidamber-Kemerer Metrics
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Chidamber-Kemerer Metrics}
\label{toc:evaluation:ck}

To improve the software process used in a company and the quality of
the produced software, a method for measuring the software is
required. With object-oriented software engineering, the traditional
methods for evaluating software are no longer appropriate.
\cite{oodmetrics} report that the conventional software metrics are
lacking appropriate mathematical properties and are without a solid
theoretical base. They proceed to propose a set of methods that are
firmly rooted in theory and relevant to practitioners in
organisations. After almost ten years, the \abbrev{CK} metrics have
been used in measuring software and \cite{ckanalysis} present further
evidence in support of a correlation between the values obtained from
the \abbrev{CK} suite and software defects.

The \abbrev{CK} suite contains six metrics for measuring software
design. The metrics are taken from \citep{oodmetrics} and they are
summarised below.

\begin{enumerate}

\item \textbf{Weighted Methods per Class (\abbrev{WMC})}\\* This
metric is a weighted sum of all the methods in a class. In the
original \abbrev{CK} suite, a weight was assigned to each method based
on the complexity of the method. The actual implementation was not
specified, however, and some researchers suggest that the weight can
be ignored, and the metric can simply count the amount of methods in
the class \citep{ckanalysis}.

The number of methods in a class predicts how much time and effort is
required to develop and maintain the method. The method count has an
impact also on the children of the class, since all but private
methods are inherited to the children.

\item \textbf{Depth of Inheritance Tree (\abbrev{DIT})}\\* This metric
calculates the longest path from the measured class to a root class in
the inheritance hierarchy. Longer inheritance paths indicate greater
design complexity since more classes and methods are involved. Longer
paths also allow for greater potential reuse of inherited methods.

\item \textbf{Number of Children (\abbrev{NOC})}\\* This metric
calculates the number of immediate children that have inherited from
the measured class. Greater number of children indicates greater
reuse, since inheritance is a form of reuse. However, a large number
of children may also be caused by improper subclassing. In any case,
classes with large child counts have a greater influence on the
overall design of the system and thus require more testing.

\item \textbf{Coupling Between Object Classes (\abbrev{CBO})}\\* This
metric calculates the number of classes to which the measured class is
coupled. The value is an indication on how independent the class is.
Excessive coupling between classes is detrimental to modular design
and prevents reuse. Highly coupled classes are also more sensitive to
changes in other classes. Classes that have high coupling require more
testing than independent ones.

\item \textbf{Response For a Class (\abbrev{RFC})}\\* This metric
calculates the amount of methods that can potentially be called in
response to a message received by an instance of the measured class.
This set is defined to be the union of all the methods in the class
and all the methods that can be called from the methods of the class.
Large values for this metric indicate that the class is complex and
debugging it is more difficult since a greater level of understanding
is required. More testing time should also be allocated for classes
that have a large response value.

\item \textbf{Lack of Cohesion in Methods (\abbrev{LCOM})}\\* This
metric calculates the difference between the amount of non-similar
method pairs and the amount of similar method
pairs\footnote{Similarity of method pairs is the amount of same
instance variables that the methods operate on; a similar method pair
is therefore a method pair whose similarity is non-zero.} in a class.
Negative values are truncated to zero. Classes with more similar
method pairs than non-similar ones are desirable, since they are
cohesive. A large \abbrev{LCOM} value indicates that the class
contains separate functionality and should therefore be partitioned
into multiple cohesive classes.

\end{enumerate}

The \abbrev{CK} metrics provide a suite of measurements that can be
used to evaluate object-oriented software. The metrics can be used to
track down key problem areas in a software system and provide a way to
compare different projects.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% XP Evaluation Framework
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{XP Evaluation Framework}
\label{toc:evaluation:xpef}

As discussed before, Extreme Programming has been gaining popularity,
but empirical evidence concerning the use of \abbrev{XP} has been
scarce. One reason for this is the lack of proper metrics. One such
metric is the Extreme Programming Evaluation Framework
\citep{xpevaluationfw}. The \abbrev{XP-EF} comprises of three parts,
namely, the \abbrev{XP} \textsl{context factors} (\abbrev{XP-cf}), the
\abbrev{XP} \textsl{adherence metrics} (\abbrev{XP-am}) and the \abbrev{XP}
\textsl{outcome measures} (\abbrev{XP-om}). Of these, the \abbrev{XP}
context factors are used to track down the exact context of the
project, from software classification and sociological factors to
ergonomic and geographic factors. These factors are important when
considering the differences between two projects.

The \abbrev{XP} adherence metrics are used to evaluate how well the
project has adopted the \abbrev{XP} practices. The evaluation of
adherence is based on both subjective and objective measures. The
subjective measures consist of a fifteen-part Shodan Adherence Survey
questionnaire \citep{xpevaluationfw}, which is presented to each team
member. The questions are all linked to single \abbrev{XP} practices,
and the answers form a subjective measurement on how well the
practices have been adopted. All fifteen measurements can be combined
to form a single value that indicates how well \abbrev{XP} has been
adopted on the whole. The measurements of the questions are weighted
with the weight of the corresponding \abbrev{XP} practice. The weight
of a practice is defined to be the amount of other \abbrev{XP}
practices the specific practice supports. These weights are originally
calculated from the \abbrev{XP} practice support diagram presented in
\citep{xpexplained}.

The objective measures consist of nine metrics that can be measured
either automatically or manually. These contain metrics such as the
ratio between test code and source code; and unit test runs per day.
Both the subjective and objective metrics are listed in
\citep{xpevaluationfw}.

The \abbrev{XP} outcome measures is a set of measures that evaluate
the produced software. The \abbrev{XP-om} includes calculating the
\abbrev{CK} metrics and McCabe Complexity\footnote{An introduction to
the McCabe Complexity can be found, for example, in \citep{mccabe}.}
of the software, and furthermore requires the calculation of six more
\abbrev{XP}-specific metrics. These metrics are described in detail in
\citep{xpevaluationfw}.

Although the \abbrev{XP} Evaluation Framework is a relatively new
framework, it has been already used in some projects.
\cite{evaluatingxp} report that the \abbrev{XP-EF} has been
comprehensive enough in a case study while not imposing too much
burden on the team members. They acknowledge, however, that there is
still work to be done in validating and extending the framework.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Summary
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Summary}
\label{toc:evaluation:summary}

In this chapter, the need for measuring software systems and software
projects has been identified. The chapter also presented two methods
for these purposes.

Section~\ref{toc:evaluation:ck} introduced the known and used
\abbrev{CK} metric suite. The suite has been used in measuring
object-oriented software and has proven to be useful in identifying
problem areas in software systems.

In section~\ref{toc:evaluation:xpef}, a relatively new framework for
evaluating Extreme Programming was presented. The \abbrev{XP}
Evaluation Framework consists of several subjective and objective
metrics that can be easily calculated alongside a project. The
metrics provide information on how well the \abbrev{XP} practices have
been adopted and how high the quality of the produced software is.

In conclusion, it is important to realise that to gain the most value
out of any software metrics, a comparison point should be available.
This can be another project, or an earlier version of the same
project. The metric values can still be used by themselves to locate
problem areas in software, or to obtain subjective evaluations of the
project.


\subsection{Rock Paper Scissors Lizard Spock (RPS-LS)}
This game is a bit more advanced. The number of outcomes remain the same, but there are 2 more possible inputs: spock and lizard, examples of both of these images are visible in figure \ref{fig:rpsls_input}. There are 2400 training examples and 600 examples are used to calculate the accuracy.
\begin{figure}
    \centering
    \includegraphics[width=.3\textwidth]{figures/input/lizard.jpg}
    \includegraphics[width=.3\textwidth]{figures/input/spock.jpg}
    \caption{Example of input pictures of the hand gestures: lizard and spock\cite{RPSLS-database}.} %%TODO import ref of pictures source
    \label{fig:rpsls_input}
\end{figure}

\paragraph{Implementation details:} For the DeepProbLog implementation, a similar approach has been taken to the RPS model. The learning rate has been increased to 0.001. The model now has 5 outputs instead of 3 because of the addition of 2 new hand gestures: lizard and spock. In listing \ref{lst:rpsls-logic} the ProbLog model can be found that is used, it is similar to the RPS model. The CNN model still has the same number of outputs: tie, player1 wins and player2 wins and here we also increased the learning to 0.001 for better results. Alongside this CNN model, a new CNN has been introduced for this game and it is described in the following section.

\begin{lstlisting}[label={lst:rpsls-logic},language=Prolog,frame=single,caption={Rock paper scissors lizard spock DeepProbLog model},captionpos=b]
nn(rpsls_net,[X],Y,[paper,scissors,rock,lizard,spock]) :: sign(X,Y).

rpsls(X,Y,0) :- sign(X,Z), sign(Y,Z).

rpsls(X,Y,1) :- sign(X,paper), sign(Y,rock).
rpsls(X,Y,2) :- sign(X,paper), sign(Y,scissors).
rpsls(X,Y,2) :- sign(X,paper), sign(Y,lizard).
rpsls(X,Y,1) :- sign(X,paper), sign(Y,spock).

rpsls(X,Y,1) :- sign(X,scissors), sign(Y,paper).
rpsls(X,Y,2) :- sign(X,scissors), sign(Y,rock).
rpsls(X,Y,1) :- sign(X,scissors), sign(Y,lizard).
rpsls(X,Y,2) :- sign(X,scissors), sign(Y,spock).

rpsls(X,Y,1) :- sign(X,rock), sign(Y,scissors).
rpsls(X,Y,2) :- sign(X,rock), sign(Y,paper).
rpsls(X,Y,1) :- sign(X,rock), sign(Y,lizard).
rpsls(X,Y,2) :- sign(X,rock), sign(Y,spock).

rpsls(X,Y,2) :- sign(X,lizard), sign(Y,scissors).
rpsls(X,Y,1) :- sign(X,lizard), sign(Y,paper).
rpsls(X,Y,2) :- sign(X,lizard), sign(Y,rock).
rpsls(X,Y,1) :- sign(X,lizard), sign(Y,spock).

rpsls(X,Y,1) :- sign(X,spock), sign(Y,scissors).
rpsls(X,Y,2) :- sign(X,spock), sign(Y,paper).
rpsls(X,Y,1) :- sign(X,spock), sign(Y,rock).
rpsls(X,Y,2) :- sign(X,spock), sign(Y,lizard).
\end{lstlisting}
\subsubsection{Results}
The loss and accuracy over the iterations of both models are plotted in figure \ref{fig:rpsls_output_loss} and figure \ref{fig:rpsls_output_acc} respectively. Notice that the scales are different from those we have seen in figure \ref{fig:rps_output}. We can clearly see that by keeping the same models and increasing the complexity, the CNN model struggles more compared to the DeepProbLog model. It takes the CNN model 38000 iterations to reach an accuracy of 100\% while the DeepProbLog model takes only 3700 iterations to reach the same level of accuracy. The CNN takes up to 661 seconds until it reaches this accuracy while the DeepProbLog model takes 246.2 seconds.
\\
After excessive tuning and trying different hyperparameters for this CNN model \footnote{These tuning methods include altering the learning rate, gradually increasing or decreasing the learning rate \cite{lr-decay}, gradually increasing the batch-size \cite{increase-batch}, choosing different optimizer, L2 regularisation, altering the CNN structure (adding layers and/or nodes)}, the model was able to achieve a 100\% accuracy a little faster.  Setting the learning rate to $0.0002$, this novel CNN model was able to achieve in 23000 iterations and in 399.2 seconds.

\begin{figure}[h]
    \begin{tikzpicture}[yscale=0.9,xscale=0.9]
        \begin{axis}[xlabel=Iterations,ylabel=Loss,xscale=2]
            \addplot[thin,red] table [x=i, y=loss, col sep=comma] {results/RPSLS/RPSLS_BaseLine_loss.log};
            \addplot[thin,blue]  table [x=i, y=loss, col sep=comma] {results/RPSLS/RPSLS_Problog_loss.log};
            \addplot[ultra thin,green]  table [x=i, y=loss, col sep=comma] {results/RPSLS/RPSLS_BaseLine_losslr0.0002lrm1.log};
        \end{axis}
    \end{tikzpicture}
    \caption{Loss of RPS-LS networks over the number of iterations: blue for DeepProbLog, red for CNN, green for the novel CNN with a learning rate of 0.002}
    \label{fig:rpsls_output_loss}
\end{figure}
\begin{figure}[h]
    \begin{tikzpicture}[yscale=0.9,xscale=0.9]
        \begin{axis}[xlabel=Iterations,ylabel=Accuracy,xscale=2]
            \addplot[thin,red] table [x=i, y=Accuracy, col sep=comma] {results/RPSLS/RPSLS_BaseLine_accuracy.log};
            \addplot[thin,blue]  table [x=i, y=Accuracy, col sep=comma] {results/RPSLS/RPSLS_Problog_accuracy.log};
            \addplot[ultra thin,green]  table [x=i, y=Accuracy, col sep=comma] {results/RPSLS/RPSLS_BaseLine_accuracylr0.0002lrm1.log};
        \end{axis}
    \end{tikzpicture}
    \caption{Accuracy of RPS-LS networks over the number of iterations: blue for DeepProbLog, red for CNN, green for the novel CNN with a learning rate of 0.002}
    \label{fig:rpsls_output_acc}
\end{figure}