\chapter{Reinforcement Learning}
\label{RL}
\index{reinforcement learning}
\index{preference!numeric-indifferent}
\index{rl}

Soar has a reinforcement learning (RL) mechanism that tunes operator selection knowledge based on a given reward function.
This chapter describes the RL mechanism and how it is integrated with production memory, the decision cycle, and the state stack.
We assume that the reader is familiar with basic reinforcement learning concepts and notation. If not, we recommend first reading \emph{Reinforcement Learning: An Introduction} (1998) by Richard S. Sutton and Andrew G. Barto.
The detailed behavior of the RL mechanism is determined by numerous parameters that can be controlled and configured via the \soarb{rl} command.
Please refer to the documentation for that command in section \ref{rl} on page \pageref{rl}.

\section{RL Rules}
\label{RL-rules}

Soar's RL mechanism learns Q-values for state-operator\footnote{
In this context, the term ``state'' refers to the state of the task or environment, not a state identifier.
For the rest of this chapter, bold capital letter names such as \soarb{S1} will refer to identifiers and italic lowercase names such as $s_1$ will refer to task states.}
pairs.
Q-values are stored as numeric indifferent preferences created by specially formulated productions called \emph{RL rules}.
RL rules are identified by syntax.
A production is a RL rule if and only if its left hand side tests for a proposed operator, its right hand side creates a single numeric indifferent preference, and it is not a template rule (see \ref{RL-templates}).
These constraints ease the technical requirements of identifying/updating RL rules and makes it easy for the agent programmer to add/maintain RL capabilities within an agent.
We define an \emph{RL operator} as an operator with numeric indifferent preferences created by RL rules.

The following is an RL rule:

\begin{verbatim}
sp {rl*3*12*left
   (state <s> ^name task-name
              ^x 3
              ^y 12
	          ^operator <o> +)
   (<o> ^name move
	    ^direction left)
-->
   (<s> ^operator <o> = 1.5)
}
\end{verbatim}

Note that the LHS of the rule can test for anything as long as it contains a test for a proposed operator.
The RHS is constrained to exactly one action: creating a numeric indifferent preference for the proposed operator.

The following are not RL rules:

\begin{verbatim}
sp {multiple*preferences
   (state <s> ^operator <o> +)
-->
   (<s> ^operator <o> = 5, >)
}

sp {variable*binding
    (state <s> ^operator <o> +
               ^value <v>)
-->
    (<s> ^operator <o> = <v>)
}
\end{verbatim}

The first rule proposes multiple preferences for the proposed operator and thus does not comply with the rule format.
The second rule does not comply because it does not provide a \emph{constant} for the numeric indifferent preference value.

In the typical RL use case, the user intends for the agent to learn the best operator in each possible state of the environment.
The most straightforward way to achieve this is to give the agent a set of RL rules, each matching exactly one possible state-operator pair.
This approach is equivalent to a table-based RL algorithm, where the Q-value of each state-operator pair corresponds to the numeric indifferent preference created by exactly one RL rule.

In the more general case, multiple RL rules can match a single state-operator pair, and a single RL rule can match multiple state-operator pairs.
all numeric indifferent preferences for an operator are summed when calculating the operator's Q-value\footnote{
This is assuming the value of \soarb{numeric-indifferent-mode} is set to \soarb{sum}.
In general, the RL mechanism only works correctly when this is the case, and we assume this case in the rest of the chapter.
See page \pageref{numeric-indifferent-mode} for more information about this parameter.}.
In this context, RL rules can be interpreted more generally as binary features in a linear approximator of each state-operator pair's Q-value, and their numeric indifferent preference values their weights.
In other words,
$$Q(s, a) = w_1 \phi_2 (s, a) + w_2 \phi_2 (s, a) + \ldots + w_n \phi_n (s, a)$$
where all RL rules in production memory are numbered $1 \dots n$, $Q(s, a)$ is the Q-value of the state-operator pair $(s, a)$, $w_i$ is the numeric indifferent preference value of RL rule $i$, $\phi_i (s, a) = 0$ if RL rule $i$ does not match $(s, a)$, and $\phi_i (s, a) = 1$ if it does.
This interpretation allows RL rules to simulate a number of popular function approximation schemes used in RL such as tile coding and sparse coding.

\section{Reward Representation}
\label{RL-reward}

RL updates are driven by reward signals.
In Soar, these reward signals are given to the RL mechanism through a working memory link called the \soarb{reward-link}.
Each state in Soar's state stack is automatically populated with a \soarb{reward-link} structure upon creation.
Soar will check this structure for a numeric reward signal for the last operator executed in the associated state at the beginning of every decision phase.
Reward is also collected when the agent is halted or a state is retracted.
% What happens when an agent with multiple states is halted? Do the rewards in the substates get collected?

In order to be recognized, the reward signal must follow this pattern:

\begin{verbatim}
(<r1> ^reward <r2>)
(<r2> ^value [val])
\end{verbatim}

where \verb=<r1>= is the \soarb{reward-link} identifier, \verb=<r2>= is some intermediate identifier, and \verb=[val]= is any constant numeric value.
Any structure that does not match this pattern is ignored.
If there are multiple valid reward signals, their values are summed into a single reward signal.
As an example, consider the following state:

\begin{verbatim}
(S1 ^reward-link R1)
  (R1 ^reward R2)
    (R2 ^value 1.0)
  (R1 ^reward R3)
    (R3 ^value -0.2)
\end{verbatim}

In this state, there are two reward signals with values 1.0 and -0.2.
They will be summed together for a total reward of 0.8 and this will be the value given to the RL update algorithm.

There are two reasons for requiring the intermediate identifier.
The first is so that multiple reward signals with the same value can exist simultaneously.
Since working memory is a set, multiple WMEs with identical values in all three positions (identifier, attribute, value) cannot exist simultaneously.
Without an intermediate identifier, specifying two rewards with the same value would require a WME structure such as

\begin{verbatim}
(S1 ^reward-link R1)
  (R1 ^reward 1.0)
  (R1 ^reward 1.0)
\end{verbatim}

which is invalid. With the intermediate identifier, the rewards would be specified as

\begin{verbatim}
(S1 ^reward-link R1)
  (R1 ^reward R2)
    (R2 ^value 1.0)
  (R1 ^reward R3)
    (R3 ^value 1.0)
\end{verbatim}

which is valid.
The second reason for requiring an intermediate identifier in the reward signal is so that the rewards can be augmented with additional information, such as their source or how long they have existed.
Although this information will be ignored by the RL mechanism, it can be useful to the agent or programmer.
For example:

\begin{verbatim}
(S1 ^reward-link R1)
  (R1 ^reward R2)
    (R2 ^value 1.0)
    (R2 ^source environment)
  (R1 ^reward R3)
    (R3 ^value -0.2)
    (R3 ^source intrinsic)
    (R3 ^duration 5)
\end{verbatim}

The \verb=(R2 ^source environment)=, \verb=(R3 ^source intrinsic)=, and \verb=(R3 ^duration 5)= WMEs are arbitrary and ignored by RL, but were added by the agent to keep track of where the rewards came from and for how long.

Note that the \soarb{reward-link} is not part of the \soarb{io} structure and is not modified directly by the environment.
Reward information from the environment should be copied, via rules, from the \soarb{input-link} to the \soarb{reward-link}.
Also note that when collecting rewards, Soar simply scans the \soarb{reward-link} and sums the values of all valid reward WMEs.
The WMEs are not modified and no bookkeeping is done to keep track of previously seen WMEs.
This means that reward WMEs that exist for multiple decision cycles will be collected multiple times.

\section{Updating RL Rule Values}
\label{RL-algo}

Soar's RL mechanism is integrated naturally with the decision cycle and performs online updates of RL rules.
Whenever an RL operator is selected, the values of the corresponding RL rules will be updated.
The update can be on-policy (Sarsa) or off-policy (Q-Learning), as controlled by the \soarb{learning-policy} parameter of the \soarb{rl} command.
Let $\delta_t$ be the amount the Q-value of an RL operator changes in an update.
For Sarsa, we have
$$ \delta_t = \alpha \left[ r_{t+1} + \gamma Q(s_{t+1}, a_{t+1}) - Q(s_t, a_t) \right] $$
where
\begin{itemize}
\item $Q(s_t, a_t)$ is the Q-value of the state and chosen operator in decision cycle $t$.
\item $Q(s_{t+1}, a_{t+1})$ is the Q-value of the state and chosen RL operator in the next decision cycle.
\item $r_{t+1}$ is the total reward collected in the next decision cycle.
\item $\alpha$ and $\gamma$ are the settings of the \soarb{learning-rate} and \soarb{discount-rate} parameters of the \soarb{rl} command, respectively.
\end{itemize}

Note that since $\delta_t$ depends on $Q(s_{t+1}, a_{t+1})$, the update for the operator selected in decision cycle $t$ is not applied until the next RL operator is chosen.
For Q-Learning, we have
$$ \delta_t = \alpha \left[ r_{t+1} + \gamma \underset{a \in A_{t+1}}{\max} Q(s_{t+1}, a) - Q(s_t, a_t) \right] $$
where $A_{t+1}$ is the set of RL operators proposed in the next decision cycle.

Finally, $\delta_t$ is divided by the number of RL rules comprising the Q-value for the operator and the numeric indifferent values for each RL rule is updated by that amount.

An example walkthrough of a Sarsa update with $\alpha = 0.3$ and $\gamma = 0.9$ (the default settings in Soar) follows.

\begin{enumerate}

\item In decision cycle $t$, an operator \soarb{O1} is proposed, and RL rules \soarb{rl-1} and \soarb{rl-2} create the following numeric indifferent preferences for it:
\begin{verbatim}
   rl-1: (S1 ^operator O1 = 2.3)
   rl-2: (S1 ^operator O1 =  -1)
\end{verbatim}
	The Q-value for \soarb{O1} is $Q(s_t, \soarb{O1}) = 2.3 - 1 = 1.3$.

\item \soarb{O1} is selected and executed, so $Q(s_t, a_t) = Q(s_t, \soarb{O1}) = 1.3$.

\item In decision cycle $t+1$, a total reward of 1.0 is collected on the \soarb{reward-link}, an operator \soarb{O2} is proposed, and another RL rule \soarb{rl-3} creates the following numeric indifferent preference for it:
\begin{verbatim}
	rl-3: (S1 ^operator O2 = 0.5)
\end{verbatim}
	So $Q(s_{t+1}, \soarb{O2}) = 0.5$.

\item \soarb{O2} is selected, so $Q(s_{t+1}, a_{t+1}) = Q(s_{t+1}, \soarb{O2}) = 0.5$
	Therefore,
	$$\delta_t = \alpha \left[r_{t+1} + \gamma Q(s_{t+1}, a_{t+1}) - Q(s_t, a_t) \right] = 0.3 \times [ 1.0 + 0.9 \times 0.5 - 1.3 ] = 0.045$$
	Since \soarb{rl-1} and \soarb{rl-2} both contributed to the Q-value of \soarb{O1}, $\delta_t$ is evenly divided amongst them, resulting in updated values of
\begin{verbatim}
   rl-1: (<s> ^operator <o> = 2.3225)
   rl-2: (<s> ^operator <o> = -0.9775)
\end{verbatim}

\item \soarb{rl-3} will be updated when the next RL operator is selected.
\end{enumerate}

\subsection{Gaps in Rule Coverage}
\label{RL-gaps}

Call an operator with numeric indifferent preferences an RL operator.
The previous description had assumed that RL operators were selected in both decision cycles $t$ and $t+1$.
If the operator selected in $t+1$ is not an RL operator, then $Q(s_{t+1}, a_{t+1})$ would not be defined, and an update for the RL operator selected at time $t$ will be undefined.
We will call a sequence of one or more decision cycles in which RL operators are not selected between two decision cycles in which RL operators are selected a \emph{gap}.
Conceptually, it is desirable to use the temporal difference information from the RL operator after the gap to update the Q-value of the RL operator before the gap.
There are no intermediate storage locations for these updates.
Requiring that RL rules support operators at every decision can be difficult for agent programmers, particularly for operators that do not represent steps in a task, but instead perform generic maintenance functions, such as cleaning processed output-link structures.

To address this issue, Soar's RL mechanism supports automatic propagation of updates over gaps.
For a gap of length $n$, the Sarsa update is
$$\delta_t = \alpha \left[ \sum_{i=t}^{t+n}{\gamma^{i-t} r_i} + \gamma^{n+1} Q(s_{t+n+1}, a_{t+n+1}) - Q(s_t, a_t) \right]$$
and the Q-Learning update is
$$\delta_t = \alpha \left[ \sum_{i=t}^{t+n}{\gamma^{i-t} r_i} + \gamma^{n+1} \underset{a \in A_{t+n+1}}{\max} Q(s_{t+n+1}, a) - Q(s_t, a_t) \right]$$

Note that rewards will still be collected during the gap, but they are discounted based on the number of decisions they are removed from the initial RL operator.

Gap propagation can be disabled by setting the \soarb{temporal-extension} parameter of the \soarb{rl} command to \soarb{off}.
When gap propagation is disabled, the RL rules preceding a gap are updated using $Q(s_{t+1}, a_{t+1}) = 0$.
The \soarb{rl} setting of the \soarb{watch} command (see Section \ref{watch} on page \pageref{watch}) is useful in identifying gaps.

\subsection{RL and Substates}
\label{RL-substates}

When an agent has multiple states in its state stack, the RL mechanism will treat each substate independently.
As mentioned previously, each state has its own \soarb{reward-link}.
When an RL operator is selected in a state \soarb{S}, the RL updates for that operator are only affected by the rewards collected on the \soarb{reward-link} for \soarb{S} and the Q-values of subsequent RL operators selected in \soarb{S}.

The only exception to this independence is when a selected RL operator forces an operator-no-change impasse.
When this occurs, the number of decision cycles the RL operator at the superstate remains selected is dependent upon the processing in the impasse state.
Consider the operator trace in Figure \ref{fig:rl-optrace}.

\begin{itemize}
\item At decision cycle 1, RL operator \soarb{O1} is selected in \soarb{S1} and causes an operator-no-change impass for three decision cycles.
\item In the substate \soarb{S2}, operators \soarb{O2}, \soarb{O3}, and \soarb{O4} are selected and applied sequentially.
\item Meanwhile in \soarb{S1}, reward values $r_2$, $r_3$, and $r_4$ are put on the \soarb{reward-link} sequentially.
\item Finally, the impasse is resolved by \soarb{O4}, the proposal for \soarb{O1} is retracted, and RL operator \soarb{O5} is selected in \soarb{S1}.
\end{itemize}

\begin{figure}
\insertfigure{Figures/rl-optrace}{1.5in}
\insertcaption{Example Soar substate operator trace.}
\label{fig:rl-optrace}
\end{figure}

In this scenario, only the RL update for $Q(s_1, \soarb{O1})$ will be different from the ordinary case.
Its value depends on the setting of the \soarb{hrl-discount} parameter of the \soarb{rl} command.
When this parameter is set to the default value \soarb{on}, the rewards on \soarb{S1} and the Q-value of \soarb{O5} are discounted by the number of decision cycles they are removed from the selection of \soarb{O1}.
In this case the update for $Q(s_1, \soarb{O1})$ is
$$\delta_1 = \alpha \left[ r_2 + \gamma r_3 + \gamma^2 r_4 + \gamma^3 Q(s_5, \soarb{O5}) - Q(s_1, \soarb{O1}) \right]$$
which is equivalent to having a three decision gap separating \soarb{O1} and \soarb{O5}.

When \soarb{hrl-discount} is set to \soarb{off}, the number of cycles \soarb{O1} has been impassed will be ignored.
Thus the update would be
$$\delta_1 = \alpha \left[ r_2 + r_3 + r_4 + \gamma Q(s_5, \soarb{O5}) - Q(s_1, \soarb{O1}) \right]$$

For impasses other than operator no-change, RL acts as if the impasse hadn't occurred.
If \soarb{O1} is the last RL operator selected before the impasse, $r_2$ the reward received in the decision cycle immediately following, and \soarb{O}$_\soarb{n}$, the first operator selected after the impasse, then \soarb{O1} is updated with
$$\delta_1 = \alpha \left[ r_2 + \gamma Q(s_n, \soarb{O}_\soarb{n}) - Q(s_1, \soarb{O1}) \right]$$

If an RL operator is selected in a substate immediately prior to the state's retraction, the RL rules will be updated based only on the reward signals present and not on the Q-values of future operators.
This point is not covered in traditional RL theory.
The retraction of a substate corresponds to a suspension of the RL task in that state rather than its termination, so the last update assumes the lack of information about future rewards rather than the discontinuation of future rewards.
To handle this case, the numeric indifferent preference value of each RL rule is stored as two separate values, the expected current reward (ECR) and expected future reward (EFR).
The ECR is an estimate of the expected immediate reward signal for executing the corresponding RL operator.
The EFR is an estimate of the time discounted Q-value of the next RL operator.
Normal updates correspond to traditional RL theory (showing the Sarsa case for simplicity):
\begin{align*}
\delta_{ECR} &= \alpha \left[ r_t - ECR(s_t, a_t) \right] \\
\delta_{EFR} &= \alpha \left[ \gamma Q(s_{t+1}, a_{t+1}) - EFR(s_t, a_t) \right] \\
\delta_t &= \delta_{ECR} + \delta_{EFR} \\
&= \alpha \left[ r_t + \gamma Q(s_{t+1}, a_{t+1}) - \left( ECR(s_t, a_t) + EFR(s_t, a_t) \right) \right] \\
&= \alpha \left[ r_t + \gamma Q(s_{t+1}, a_{t+1}) - Q(s_t, a_t) \right]
\end{align*}
During substate retraction, only the ECR is updated based on the reward signals present at the time of retraction, and the EFR is unchanged.

Soar's automatic subgoaling and RL mechanisms can be combined to naturally implement hierarchical reinforcement learning algorithms such as MAXQ and options.

\subsection{Eligibility Traces}
\label{RL-et}
The RL mechanism supports eligibility traces, which can improve the speed of learning by updating RL rules across multiple sequential steps.
The \soarb{eligibility-trace-decay-rate} and \soarb{eligibility-trace-tolerance} parameters control this mechanism.
By setting \soarb{eligibility-trace-decay-rate} to \soarb{0} (default), eligibility traces are in effect disabled.
When eligibility traces are enabled, the particular algorithm used is dependent upon the learning policy.
For Sarsa, the eligibility trace implementation is \emph{Sarsa($\lambda$)}.
For Q-Learning, the eligibility trace implementation is \emph{Watkin's Q($\lambda$)}.

\subsubsection{Exploration}

The \soarb{indifferent-selection} command (page \pageref{indifferent-selection}) determines how operators are selected based on their numeric indifferent preferences.
Although all the indifferent selection settings are valid regardless of how the numeric indifferent preferences were arrived at, the \soarb{epsilon-greedy} and \soarb{boltzmann} settings are specifically designed for use with RL and correspond to the two most common exploration strategies.
In an effort to maintain backwards compatibility, the default exploration policy is \soarb{softmax}.
As a result, one should change to \soarb{epsilon-greedy} or \soarb{boltzmann} when the reinforcement learning mechanism is enabled.

\section{Automatic Generation of RL Rules}

The number of RL rules required for an agent to accurately approximate operator Q-values is usually infeasibly large to write by hand, even for small domains.
Therefore, several methods exist to automate this.

\subsection{The gp Command}
The \soar{gp} command can be used to generate productions based on simple patterns.
This is useful if the states and operators of the environment can be distinguished by a fixed number of dimensions with finite domains.
An example is a grid world where the states are described by integer row/column coordinates, and the available operators are to move north, south, east, or west.
In this case, a single \soar{gp} command will generate all necessary RL rules:

\begin{verbatim}
gp {gen*rl*rules
   (state <s> ^name gridworld
              ^operator <o> +
              ^row [ 1 2 3 4 ]
              ^col [ 1 2 3 4 ])
   (<o> ^name move
        ^direction [ north south east west ])
-->
   (<s> ^operator <o> = 0.0)
}
\end{verbatim}

For more information see the documentation for this command on page \pageref{gp}.

\subsection{Rule Templates}
\label{RL-templates}

Rule templates allow Soar to dynamically generate new RL rules based on a predefined pattern as the agent encounters novel states.
This is useful when either the domains of environment dimensions are not known ahead of time, or when the enumerable state space of the environment is too large to capture in its entirety using \soar{gp}, but the agent will only encounter a small fraction of that space during its execution.
For example, consider the grid world example with 1000 rows and columns.
Attempting to generate RL rules for each grid cell and action a priori will result in $1000 \times 1000 \times 4 = 4 \times 10^6$ productions.
However, if most of those cells are unreachable due to walls, then the agent will never fire or update most of those productions.
Templates give the programmer the convenience of the \soar{gp} command without filling production memory with unnecessary rules.

Rule templates have variables that are filled in to generate RL rules as the agent encounters novel combinations of variable values.
A rule template is valid if and only if it is marked with the \soarb{:template} flag and, in all other respects, adheres to the format of an RL rule.
However, whereas an RL rule may only use constants as the numeric indifference preference value, a rule template may use a variable.
Consider the following rule template:

\begin{verbatim}
sp {sample*rule*template
    :template
    (state <s> ^operator <o> +
               ^value <v>)
-->
    (<s> ^operator <o> = <v>)
}
\end{verbatim}

During agent execution, this rule template will match working memory and create new productions by substituting all variables in the rule template that matched against constant values with the values themselves.
Suppose that the LHS of the rule template matched against the state

\begin{verbatim}
(S1 ^value 3.2)
(S1 ^operator O1 +)
\end{verbatim}

Then the following production will be added to production memory:

\begin{verbatim}
sp {rl*sample*rule*template*1
    (state <s> ^operator <o> +
               ^value 3.2)
-->
    (<s> ^operator <o> = 3.2)
}
\end{verbatim}

The variable \soar{<v>} is replaced by \soar{3.2} on both the LHS and the RHS, but \soar{<s>} and \soar{<o>} are not replaced because they matches against identifiers (\soar{S1} and \soar{O1}).
As with other RL rules, the value of \soar{3.2} on the RHS of this rule may be updated later by reinforcement learning, whereas the value of \soar{3.2} on the LHS will remain unchanged.
If \soar{<v>} had matched against a non-numeric constant, it will be replaced by that constant on the LHS, but the RHS numeric indifference preference value will be set to zero to make the new rule valid.

The new production's name adheres to the following pattern:
\soarb{rl*template-name*id}, where \soarb{template-name} is the name of the originating rule template and \soarb{id} is monotonically increasing integer that guarantees the uniqueness of the name.

If an identical production already exists in production memory, then the newly generate production is discarded.
It should be noted that the current process of identifying unique template match instances can become quite expensive in long agent runs.
Therefore, it is recommended to generate all necessary RL rules using the \soar{gp} command or via custom scripting when possible.

\subsection{Chunking}
Since RL rules are regular productions, they can be learned by chunking just like any other production.
This method is more general than using the \soar{gp} command or rule templates, and is useful if the environment state consists of arbitrarily complex relational structures that cannot be enumerated.


\section{Future Work and Conclusions}
\label{sec:future}

The main contribution of this paper is to pose the problem of DSP-PBE.
While we have presented a prototype implementation of a DSP-PBE tool, this primarily functions as a proof-of-concept.
There remains significant room for optimization in both the distance calculation and the search algorithm.
In future work, we also plan to expand beyond commutative filters to be able to synthesize effects such as delay lines.

We briefly revisit the motivating conceptual example from Sec.~\ref{sec:background} where we want to synthesize the filter that transforms a trumpet into a trombone and apply that filter to a violin.
In order to achieve this we need more complex filters than high-pass, low-pass, and amplitude adjustment.
Despite a lack of a formal approach for non-commutative filters, we added a pitch shift filter and the \texttt{ringz} delay line filter from SuperCollider into our tool using the current approach.
Though we had no reason to believe that our current approach should produce useful results when allowing more complex filters in the search space of synthesis, we found the results to be at least interesting, and even reasonable.
The synthesized filter to transform a trumpet into a trombone was as follows:

\begin{lstlisting}
SetVolume: 1.00%
LoPass: freq@1000 amp@0.91 >>>
HiPass: freq@100 amp@0.00 >>>
PitchShift: freq@-1600 amp@0.91 >>>
Ringz: freq@9100 delay@0.10 amp@0.23 >>>
WhiteNoise: amp@0.00
\end{lstlisting}

The trumpet and trombone input audio files, as well as the audio of a violin with this filter applied is available as a Soundcloud playlist~\cite{soundcloudAudio}.

Although with the current tool, synthesis times presented might be prohibitively slow for many use cases, especially on such small programs, we should be encouraged by progress in other domains of program synthesis.
In the SyGuS program synthesis competition, which has run for four years, tools have seen an exponential speed up and increase in the range of programs that can be synthesized.
As one example, in the 2014 competition the \texttt{LinExpr\_eq1.sl} benchmark was only solved by one tool, and took 1128 seconds~\cite{sygus2014}.
In the 2017 competition, the same benchmark was solved by all tools, with the fastest taking only 199 seconds~\cite{sygus2017}.


\subsection{Grid Search}
\label{chp:study:sec:execution:subsec:gridsearch}
Training \ac{ML} models involves the setting of hyperparameters.
Those have a direct impact on the performance of a trained model and the user must set them appropriately to optimize the learning routine and with that the model \parencite{Claesen:2015}.
We perform a grid search to find a suitable parameter combination for each model.
We base the parameter grid for each model on the corresponding recommendations of the models' authors.

\subsubsection{Hyperparameter Grid}
\label{chp:study:sec:execution:subsec:gridsearch:hyperparameter_grid}
All our models are transformer-based and involve multiple hyperparameters.
Transformer-based models have the \textit{maximum length} of their input sequence as hyperparameter.
Sequences are truncated when they exceed this limit and are padded with zeros if they are shorter than this hyperparameter.
Due to our limited datasets we want to perform $k$-fold cross validation.
In order to determine how many \textit{folds} we want to use we add this as hyperparameter to our grid as well.
For the training of the models we use imbalanced datasets with the majority being not vague.
To take that imbalance into account we consider different \textit{re-sampling} strategies.
\textit{Random down-sampling} randomly discards datapoints of the majority class to create a balanced dataset for the training phase.
\textit{Random up-sampling} achieves the same by randomly duplicating requirements of the minority class.
The hyperparameters for \textit{learning rate}, \textit{epochs} and \textit{batch size} follow the the recommendations of the base models' authors.
Since all models are transformer-based the hyperparameter grids are similar.
The hyperparameter grid for \ac{BERT} and \ac{DistilBERT} is shown in \cref{tab:study:execution:paramter_grid:BERT}.
\begin{table}[htpb]
    \centering
    \begin{tabular}{ p{3.5cm} p{1.5cm} p{2cm} p{1.5cm} p{1.6cm} p{1.5cm} }
        \toprule
         Re-sampling & Folds & Learning Rate & Epochs & Max Length & Batch Size \\
        \midrule
            \begin{itemize}[noitemsep,topsep=0pt,leftmargin=15pt]
                \item {random

                down-sampling}
                \item {random

                up-sampling}
            \end{itemize}
            &\begin{itemize}[noitemsep,topsep=0pt,leftmargin=15pt]
                \item 4
                \item 8
            \end{itemize}
            & \begin{itemize}[noitemsep,topsep=0pt,leftmargin=15pt]
                \item $1\mathrm{e}{-06}$
                \item $5\mathrm{e}{-06}$
                \item $1\mathrm{e}{-05}$
                \item $5\mathrm{e}{-05}$
            \end{itemize}
            & \begin{itemize}[noitemsep,topsep=0pt,leftmargin=15pt]
                \item 1
                \item 2
                \item 3
            \end{itemize}
            & \begin{itemize}[noitemsep,topsep=0pt,leftmargin=15pt]
                \item 64
                \item 128
            \end{itemize}
            & \begin{itemize}[noitemsep,topsep=0pt,leftmargin=15pt]
                \item 16
                \item 32
            \end{itemize}\\
        \bottomrule
    \end{tabular}
    \caption[Hyperparameter Grid for \ac{BERT} and \ac{DistilBERT}]{Hyperparameter grid for \ac{BERT} and \ac{DistilBERT}.}\label{tab:study:execution:paramter_grid:BERT}
\end{table}

The Hyperparameter grid for \ac{ERNIE2.0} is similar, however \textcite{Sun:2019a} recommend different learning rates.
This leads to the Hyperparameter grid for \ac{ERNIE2.0} shown in \cref{tab:study:execution:paramter_grid:ERNIE2.0}
\pagebreak %Todo check page break
\begin{table}[htpb]
    \centering
    \begin{tabular}{ p{3.5cm} p{1.5cm} p{2cm} p{1.5cm} p{1.6cm} p{1.5cm} }
        \toprule
         Re-sampling & Folds & Learning Rate & Epochs & Max Length & Batch Size \\
        \midrule
            \begin{itemize}[noitemsep,topsep=0pt,leftmargin=15pt]
                \item {random

                down-sampling}
                \item {random

                up-sampling}
            \end{itemize}
            &\begin{itemize}[noitemsep,topsep=0pt,leftmargin=15pt]
                \item 4
                \item 8
            \end{itemize}
            & \begin{itemize}[noitemsep,topsep=0pt,leftmargin=15pt]
                \item $2\mathrm{e}{-05}$
                \item $3\mathrm{e}{-05}$
                \item $4\mathrm{e}{-05}$
                \item $5\mathrm{e}{-05}$
            \end{itemize}
            & \begin{itemize}[noitemsep,topsep=0pt,leftmargin=15pt]
                \item 1
                \item 2
                \item 3
            \end{itemize}
            & \begin{itemize}[noitemsep,topsep=0pt,leftmargin=15pt]
                \item 64
                \item 128
            \end{itemize}
            & \begin{itemize}[noitemsep,topsep=0pt,leftmargin=15pt]
                \item 16
                \item 32
            \end{itemize}\\
        \bottomrule
    \end{tabular}
    \caption[Hyperparameter Grid for \ac{ERNIE2.0}]{Hyperparameter grid for \ac{ERNIE2.0}.}\label{tab:study:execution:paramter_grid:ERNIE2.0}
\end{table}

\subsubsection{Grid Search Execution}
\label{chp:study:sec:execution:subsec:gridsearch:execution}

Now we perform grid search using the crowdsourced dataset $D_{crowd}$.
For that we split $D_{crowd}$ and use $90\%$ of its data for training whereas the remaining $10\%$ are merely used for the final evaluation of the trained models.
We ensure that the resulting datasets $D_{crowd_{train}}$ and $D_{crowd_{test}}$ consist of the same proportion of vague datapoints of approximately $19\%$.
Finding a suitable learning rate is very challenging task, \textcite{Zeiler:2012} goes even further and states \textit{"[d]etermining a good learning rate becomes more of an art than science for many problems"}.
However, \textcite{Smith:2018} introduced the 1cycle learning rate schedule to dynamically adjust the learning rate during the training phase.
This policy achieved remarkable results in his experiments.
We use this approach and set the initial learning rate to the value of the corresponding hyperparameter grid.
For a detailed description of the 1cycle policy we refer to \citeauthor{Smith:2018}'s original paper \parencite{Smith:2018}.

\subsubsection{Found Hyperparameters}
\label{chp:study:sec:execution:subsec:gridsearch:found_hyperparameters}

Performing the grid search with the given hyperparameter grids the hyperparameter combinations which achieved the best $F_1$ score are listed in \cref{tab:study:execution:grid_search:results}.
All three models achieved an $F_1$ score around $0.5$.
\pagebreak %Todo check page break
\begin{table}[htpb]
    \centering
\begin{tabular}{l | p{2.9cm} l p{1.5cm} l p{1.6cm} l }
        \toprule
         Model & Re-sampling & Folds & Learning Rate & Epochs & Max Length & Batch Size \\
        \midrule
        \ac{BERT} & random\newline down-sampling & 4 & $1\mathrm{e}{-05}$ & 2 & 128 &16\\
        \ac{DistilBERT} & random\newline down-sampling & 4 & $5\mathrm{e}{-06}$ & 1 & 64 &32\\
        \ac{ERNIE2.0} & random\newline down-sampling & 4 & $4\mathrm{e}{-05}$ & 2 & 128 &32\\
        \bottomrule
    \end{tabular}
    \caption[Grid Search Results]{The best hyperparameter combinations per model found with grid search.}\label{tab:study:execution:grid_search:results}
\end{table}


Input to the Ghost-Node Correction (GNC) Package is read from the file that has type ``GNC6'' in the Name File.  Only one GNC Package can be used per GWF Model.

The GNC Package has two options for adding the correction terms to the system of equations.  The implicit option, which is the default, adds the terms on both the left-hand and right-hand sides of the equations.  When this default option is used, the BICGSTAB linear acceleration option should be specified within the LINEAR block of the Sparse Matrix Solver.  The BICGSTAB acceleration option is designed to handle the asymmetry in the conductance matrix.  When the EXPLICIT option is specified for the GNC Package, then the correction terms are added to the right-hand side, and either the CG or BICGSTAB acceleration methods can be used.

\vspace{5mm}
\subsubsection{Structure of Blocks}
\vspace{5mm}

\noindent \textit{FOR EACH SIMULATION}
\lstinputlisting[style=blockdefinition]{./mf6ivar/tex/gwf-gnc-options.dat}
\lstinputlisting[style=blockdefinition]{./mf6ivar/tex/gwf-gnc-dimensions.dat}
\lstinputlisting[style=blockdefinition]{./mf6ivar/tex/gwf-gnc-gncdata.dat}

\vspace{5mm}
\subsubsection{Explanation of Variables}
\begin{description}
\input{./mf6ivar/tex/gwf-gnc-desc.tex}
\end{description}

\vspace{5mm}
\subsubsection{Example Input File}
\lstinputlisting[style=inputfile]{./mf6ivar/examples/gwf-gnc-example.dat}


\renewcommand{\thefigure}{S\arabic{figure}}
\chapter{Supplement}
\label{sec:supplement}
\begin{figure}[H]
    \centering
        \includegraphics[width=15.7463cm,height=15.7463cm]{./content/figures/fig_plot_shae_pathways_kirp.png} \caption{Enriched hallmarks of cancer pathways for each of the 64 latent dimension of \gls{shae} on the best split of \glsxtrfull{kirp}.}\label{fig:KIRP-SHAE-pathways}
\end{figure}
\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=7.87315cm]{./content/figures/fig_plot_shae_permutation_importance_blca.png} \caption{Group permutation feature importance for each input group of \gls{shae} on the best split of \glsxtrfull{blca} using 10 permutations.}\label{fig:SHAE-BLCA-permutation-importance}
\end{figure}

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=15.7463cm]{./content/figures/fig_plot_shae_residual_heatmap_blca.png} \caption{Correlation heatmap of all features which had the highest correlation with at least one of the 64 latent space dimensions of \gls{shae} residual on the best test split of \glsxtrfull{blca}.}\label{fig:BLCA-SHAE-residual-heatmap}
\end{figure}

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=15.7463cm]{./content/figures/fig_plot_shae_residual_heatmap_kirp.png} \caption{Correlation heatmap of all features which had the highest correlation with at least one of the 64 latent space dimensions of \gls{shae} residual on the best test split of \glsxtrfull{kirp}.}\label{fig:KIRP-SHAE-residual-heatmap}
\end{figure}

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=15.7463cm]{./content/figures/fig_plot_surrogate_performance_favor_lasso_by_cancer.png} \caption{Test concordance of all methods trained on clinical and gene expression only across the 10 test splits of each of the 17 considered TCGA cancers. Clin. Cox PH trained on clinical only. Surrogates trained on the predictions $\hat \varphi(\tilde C)$ from multi-omics \gls{shae} (residual) using clinical and gene expression data only. Lasso unpenalized clinical trained on clinical and gene expression only to predict survival while leaving clinical variables unpenalized. Surrogate models also left clinical variabes unpenalized.}\label{fig:surrogate-performance-favor-clinical-by-cancer}
\end{figure}

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=7.87315cm]{./content/figures/fig_plot_shae_surrogate_sparsity.png} \caption{Model sparsity of the \gls{shae} surrogate model across the 10 test splits of each of the 17 \gls{tcga} cancers.}\label{fig:shae-residual-surrogate-sparsity-by-cancer}
\end{figure}

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=7.87315cm]{./content/figures/fig_plot_lasso_unpenalized_clinical_sparsity.png} \caption{Model sparsity of the \gls{lasso} model with clinical variables unpenalized across the 10 test splits of each of the 17 \gls{tcga} cancers when trained on clinical and gene expression to predict survival.}\label{fig:lasso-sparsity-by-cancer-unpenalized}
\end{figure}

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=15.7463cm]{./content/figures/fig_plot_feature_selection_performance_by_cancer.png} \caption{Test concordance of the \gls{shae} residual surrogate model, \gls{cox} and a \gls{rsf} and Ridge regularized Cox PH model across the 10 test splits of each of the 17 considered TCGA cancers. Clin. Cox PH trained on clinical only. Surrogate trained on the predictions $\hat \varphi(\tilde C)$ from multi-omics \gls{shae} residual using clinical and gene expression data only. \gls{rsf} and Ridge regularized Cox PH trained using the same feature set as the \gls{shae} residual model but to predict survival directly.}\label{fig:feature-selection-performance-by-cancer}
\end{figure}

\begin{figure}
    \centering
        \includegraphics[width=15.7463cm,height=7.87315cm]{./content/figures/fig_plot_feature_selection_performance_take_cancers_out.png} \caption{Test concordance of the \gls{shae} residual surrogate model, \gls{cox} and a \gls{rsf} and Ridge regularized Cox PH model across the 10 test splits of 14 TCGA cancers (\glsxtrfull{kirp}, \glsxtrfull{lgg} and \glsxtrfull{sarc} were removed for this figure). Clin. Cox PH trained on clinical only. Surrogate trained on the predictions $\hat \varphi(\tilde C)$ from multi-omics \gls{shae} residual using clinical and gene expression data only. \gls{rsf} and Ridge regularized Cox PH trained using the same feature set as the \gls{shae} residual model but to predict survival directly.}\label{fig:feature-selection-performance-cancers-excluded}
\end{figure}


\subsection{Vectors}
\label{Vectors}
A vector is essentially a map \(f(i)\) such that given \(i, 0 \leq i < n\), it
returns a value of a given type. The main types that \Q\ supports are four variants
of integers (1, 2, 4, and 8 byte) and two variations of floating point (single
and double precision). In addition, it supports bit vectors and constant length
strings. There is limited support for variable length strings, which are used
primarily as dictionaries.

Note that Q has 6 types of numbers, in contrast with Lua which uses a
single type {\tt number}, internally a double precision floating point.
This is because data bandwidth plays a significant role in determining
performance, as illustrated by Nvidia's
introduction of half precision floating point  \cite{nvidia2017}. The user is
encouraged (but not required) to use the smallest type that supports the actual
dynamic range required. It is eye-opening to realize that this is often  less
than what one expects. For example, while analyzing LinkedIn's endorsement feature
(e.g. User A endorses User B's proficiency in Java),
the number of unique endorsements that captured more than 99\% of the traffic
was less than 32K, making a 2-byte integer adequate for the task at hand.

It is relatively easy to add other types to Q, as long as these
are fixed width types. The real cost is in making sure that all relevant
operators interpret that data meaningfully. For example, currently, {\tt add}
accepts all combinations of number types, but we do not (yet)
support adding a bit to a double.

We
encourage ``inexact'' types where justified. For example, a common use case is
to trace a user's activity through a web session. This requires joining on a
user ID across different log tables in the data warehouse. The user ID is
often provided as a long alphanumeric string. While Q could represent this as a
constant length string, it is better represented as an unsigned 64-bit integer
containing the hash of the string, since join requires only equality comparisons.
The number of distinct keys being hashed is often less than \(2^{32}\). When
hashed to a \(2^{64}\) space, the probability of collisions is vanishingly
small. Most business analyses are insensitive to this level of imprecision.

When a vector is created, we need to specify (i) its type (ii) whether it has
null values (iii) how it will be populuated. We can either (a) ''push'' data to
it, much like writing to a file in append mode or (b) we can provide a generator function,
which is invoked with the chunk index, \(i\), as parameter and which knows how to
generate the \(i^{th}\) chunk.
%% TODO Have not introduced chunk as yet

Vectors are evaluated lazily. Hence, a statement like
{\tt x = Q.const(\{len = 10, qtype = I4, val = 0\})} does not actually create
ten
4-byte integers with value 0 as one might suspect. Data is populated only when
{\tt eval()} is explicity invoked on the vector or the data is
implicitly required by some other operator e.g. {\tt Q.print\_csv(x)}

Vectors are processed in chunks. Consider an expression  like \(\sum (a + b\times
c)\), written in Q as {\tt d = Q.sum(Q.add(Q.mul(b,c), a))}.
When {\tt d} is eval'd, computation alternates between the {\tt mul, add, sum}
operators
processing chunks of data at a time until there are
no more.
We had originally chosen co-routines to effect the {\tt resume} and
{\tt yield} capabilities needed. However, Lua's support of closures allowed for
a simpler implementation.
The chunk size, \(n_C\), is chosen large enough that it is amenable to
vectorization and parallelization and small enough that its memory consumption
is low.

Vectors are not mutable (with few exceptions)
and must be produced sequentially. In other words, the \(i^{th}\) element must
be produced before the \({i+1}^{th}\). Vectors
operate in ``chunks'' of a fixed size. Let us say that the chunk size is 64K and
that we have produced 65K elements. In that case, the current chunk would have
only 1K elements. Whether one can get access to an element in the previous chunk
depends on whether the vector has been ``memo-ized''. The default behavior, with
a concomitant performance hit, is to memo-ize. However, when the programmer is
aware that the vector will be consumed in a streaming fashion, they set memo
to false.

A vector is fully materialized in one of two ways (1) If it was setup with a a generator function and invocation of the function produces
fewer elements than the chunk size (2) it was set up for streaming and {\tt
eov()} was explicitly signaled.

Memo-izing is done by appending previous chunks in binary format to a file.
Subsequent reads of this vector are done by mmap-ing the file. Not all
algorithms are readily transformed into streaming operations e.g. sort. There
are a few cases where we support modifying a vector after it has been fully
materialized by opening it in write mode and mmap-ing it.

Mmap-ing gives us the illusion of a linear address space. This is useful to
incoporate algorithms and libraries that have not been written with streaming in
mind. For example, we have used this to call functions in the GNU Scientific
Library and LAPACK from Q.

Q's run time is an alternate approach to ``stream fusion''
\cite{Mainland2017}. In
that paper, the authors identify this as a technique that allows a compiler to
`` cope with boxed numeric types, handle lazy evaluation, and
eliminate intermediate data structures''.
\subsection{Vectors versus Tables}
\label{vectors_versus_tables}

A deliberate choice in Q's design was that Vectors, not tables, were the basic
data type. It is a design choice that we were (and continue to be) conflicted with. On the one hand,
it led to simplicity and performance. On the other hand, it put the burden of
table semantics on the programmer. In particular, consider a
Lua table \(\{f, g\}\) containing 2 Vectors \(f, g\). Maintaining table
semantics means that \(T(i) = (f(i), g(i))\)  i.e., the \(i^{th}\) element of
the table is the \(i^{th}\) element of \(f\) and the \(i^{th}\) element of \(g\). An
operation like {\tt Q.sort(T.f)} would destroy that contract. Q
provides higher level functions (Figure~\ref{sort_tbl}) which are invoked when the entire table needs to
be sorted e.g. {\tt Q.sort(T, f, "ascending")}

\begin{figure}[hbtp]
\centering
\fbox{
\begin{minipage}{14 cm}
\centering
\verbatiminput{sort_tbl.lua}
\caption{Re-ordering table based on sort of column f in ascending order}
\label{sort_tbl}
\end{minipage}
}
\end{figure}
%% TODO make sure permute is consistent with Terra code


\title{Formalization of Methods}

\date{\today}

\author{Alex Wallar}

\documentclass{article}

\usepackage{amsfonts}

\usepackage{amsmath}

\usepackage[pdftex]{graphicx}

\usepackage{relsize}

\usepackage[margin=1.4in]{geometry}

\usepackage{float}

\usepackage{fancyvrb}

\usepackage{algorithm}

\usepackage[noend]{algorithmic}

\floatstyle{ruled} \newfloat{program}{thp}{lop} \floatname{program}{Structure}

\newcommand{\Normal}[2]{\mathcal{N}_k(#1, #2)}

\newcommand{\Acronym}[1]{\ensuremath{{\small{\texttt{#1}}}}}
\newcommand{\Constant}[1]{\ensuremath{\small{\texttt{#1}}}}
\newcommand{\Rover}{\Acronym{Rover}}
\newcommand{\Keeper}{\Acronym{Keeper}}
\newcommand{\False}{\Constant{false}}
\newcommand{\True}{\Constant{true}}
\newcommand{\Symbol}[1]{\ensuremath{\mathcal{#1}}}
\newcommand{\Function}[1]{\ensuremath{{\small \textsc{#1}}}}
\newcommand{\Var}[1]{\ensuremath{{\small{\textsl{#1}}}}}
\newcommand{\argmax}[1]{\underset{#1}{\operatorname{arg}\,\operatorname{max}}\;}
\newcommand{\Max}[1]{\underset{#1}{\operatorname{max}}\;}
\newcommand{\grad}[1]{\underset{#1}{\operatorname{\Function{GradientDecent}}}\;}
\newcommand{\s}{\textbf{s}}
\newcommand{\fig}[1]{\textbf{Fig.~\ref{fig:#1}}}

\newcommand{\n}[2]{\ensuremath{n^{(#2)}_{#1}}}

\newcommand{\PA}{\ensuremath{P^{(\n{i}{t},\;\n{j}{t})}_{A}}}
\newcommand{\PD}{\ensuremath{P^{(t_0,\;t_m)}_{a}}}
\newcommand{\PAll}{\ensuremath{P^{(t_0,\;t_m)}_{A}}}

\begin{document}

\maketitle

\section{Temporal Probabilistic Roadmap}

\paragraph{Construction} A temporal probabilistic roadmap (tPRM) is similar to
other probabilistic roadmaps but instead of just representing the configuration
space of the robot, a tPRM also incorporates the time at which the robot would
be at that configuration. For instance, in a two dimensional setting, the nodes
of the tPRM would be of the form $(x, y, t)$ where $t$ is the time at which the
robot would be at $(x, y)$. The addition of time into the tradition approach
allows more constraints to be set on the construction of the roadmap that
include time. The tPRM is represented by a weighted directed graph, $(V, E,
W)$, such that an edge can only exist from $n_i$ to $n_j$ if $\n{i}{t} <
\n{j}{t}$ along with other constraints set during the construction.  The
temporal probabilistic roadmap is constructed by sampling random points within
the search space and connecting nodes that are feasible for the robot to move
between.  More formally, an edge exists between $n_i$ and $n_j$ if $F_R(n_i,
n_j) = 1$ where the function, $F_R: V \times V \to \{0, 1\}$ represents whether
or not it is feasible to move between two vertices in the roadmap for a given
robot model, $R$.  Other than the feasibility constraint for an edge between
two nodes, there are also constraints that limit the size of edges. We assume
that there is a maximum time $T_m > 0$ such that if $(n_i, n_j) \in E$, then
$\n{j}{t} - \n{i}{t} \leq T_m$. Likewise, we assume that there is a maximum
distance that can exist between nodes, $D_m > 0$, such that if $(n_i, n_j) \in
E$, then $\Var{dist}(n_i, n_j) \leq D_m$ where $\Var{dist}(n_i, n_j)$ is the
distance between $n_i$ and $n_j$ in Euclidean space. Algo.~\ref{algo:tprm}
shows pseudocode for the construction of the tPRM.

\paragraph{Search} In order to get a set of sub-goals that can be used to lead
the robot to the goal, a graph search algorithm is used. The current
implementation uses Dijkstra's shortest path algorithm because it is provided
in the \verb|NetworkX| graph library by default. Given the constructed roadmap,
the algorithm is used to find the shortest path in the directed graph from the
robot's initial configuration to it's goal configuration.

\begin{algorithm}[ht]

\caption{Pseudocode for tPRM construction}

\label{algo:tprm}

\begin{algorithmic}[1]

\setcounter{ALC@line}{0}

\vspace*{1mm}

\STATE $G \leftarrow \Function{InitDirectedGraph}()$

\STATE $S \leftarrow \{\Var{start}\}, i \leftarrow 0$

\WHILE{$i < \Var{MaxPoints}$}

\STATE $i \leftarrow i + 1$

\STATE $\rho \leftarrow \Function{RandomChoice}(S)$

\STATE $\sigma \leftarrow \Function{Sample}(\rho, F_R)$

\IF{$\Function{WithinSearchSpace}(\sigma)$}

\STATE $S \leftarrow S \cup \{\sigma\}$

\ENDIF

\FORALL{$s \in S$}

\IF{$\Var{dist}(s, \sigma) \leq D_m \wedge |\sigma^{(t)} - s^{(t)}| \leq
T_m$}

\IF{$s^{(t)} < \sigma^{(t)} \wedge F_R(s, \sigma) = 1$}

\STATE $\Function{AddEdge}(G, s, \sigma, P^{(s^{(t)},\;\sigma^{(t)})}_{A}(s,
\sigma))$

\ENDIF

\IF{$s^{(t)} > \sigma^{(t)} \wedge F_R(\sigma, s) = 1$}

\STATE $\Function{AddEdge}(G, \sigma, s, P^{(\sigma^{(t)},\;\s^{(t)})}_{A}(
\sigma, s))$

\ENDIF

\ENDIF

\ENDFOR

\ENDWHILE

\RETURN $G$

\end{algorithmic}
\end{algorithm}

\section{Agent Position Probability Distribution}

For a given set of agents, $A$, in the environment and their associated
trajectories, $\{\zeta_a : a \in A\}$, where $\zeta_a: \mathbb{R} \to
\mathbb{R}^k$, the probability of an agent being given position is represented
between a given time interval is given by the function, $\PD: \mathbb{R}^k \to
(0, 1)$, where $k$ is the dimension of the search space. $\PD$ is defined as,

$$\PD(\mathbf{x}^k) = \mathlarger{\int_{t_0}^{t_m}}(t_m - t + 1)^r
\cdot \Normal{\zeta_a(t)}{(t_m - t)^2 \cdot \alpha}(\mathbf{x}^k)dt$$

Where $\mathbf{x}^k$ represents the $k$-dimensional input vector, $\alpha$ is a
scaling constant, and $\mathcal{N}_k(\mu, \sigma^2)$ is a $k$-dimensional
Gaussian distribution centered at $\mu$ with a variance $\sigma^2$. For all of
the agents, the probability density function becomes

$$\PAll(\mathbf{x}^k) = \frac{\mathlarger{\sum_{a \in A}}
\PD(\mathbf{x}^k)}{|A|}$$

\section{Edge Weights}

For two nodes, $n_i$ and $n_j$, in the temporal probabilistic roadmap (tPRM),
where $\n{i}{t} < \n{j}{t}$, where $n^{(t)}$ represents the time at which robot
would be at node $n$, the weight of the edge between those two nodes is,

$$w(n_i, n_j) = \gamma \cdot \mathlarger{\max_{t \in [\n{i}{t},
    \n{j}{t}]}}(\PA \circ E)(t)$$

Where $E(t)$ is a function, $E: \mathbb{R} \to \mathbb{R}^k$, that represents
the position along the edge between $n_i$ and $n_j$ given a time, $t
\in [\n{i}{t},\;\n{j}{t}]$ and $k$ is the dimension of the search space. In the
equation, $\circ$ represents the function composition, and $\gamma$ is a
scaling constant.

\end{document}


The correctness of the algorithm relies on the fact that we only compute with
well-typed terms.  This guarantees the existence of normal forms, and hence,
ensures the termination of the type checking algorithm.

The proof will be done in two stages: first we prove soundness in the absence
of constraint solving, and then we prove that constraint solving is sound.

\subsection{Soundness without constraint solving}

There are a number of things we need to prove: that type checking preserves
well-formed signatures, that it produces well-typed terms, that conversion
checking is sound, and that new signatures respect the old signatures.
Unfortunately these properties are all interdependent, so we cannot prove them
separately.

\begin{definition}[Signature extension] \label{defSigExt}
    We say that $\Sigma'$ {\em extends} $\Sigma$ if for any {\Core}
    judgement $J$, $\vdash_\Sigma J$ implies $\vdash_{\Sigma'} J$.
\end{definition}

Note that this definition admits both simple extensions--adding a new
constant--and refinement, where we give a definition to a constant.
\if \DetailedProofs 1
This is expressed by the following two lemmas.

\begin{lemma}[Signature weakening] \label{lemWeakenSig}
    If $\Sigma' = \Ext \Sigma {c:A}$ and $\IsSigCS {\Sigma'}$ then $\Extends
    {\Sigma'} \Sigma$.
\end{lemma}

\begin{lemma}[Signature refinement] \label{lemRefineSig}
    Giving a definition to a constant in a signature is an extension of the
    signature.
\if \DetailedProofs 1
    More precisely, if
    \begin{itemize}
	\item $\HasTypeCS {\Sigma_1} {} M A$
	\item $\Sigma ~ = ~ \Ext {\Sigma_1} \Ext {\MetaDecl c A} \Sigma_2$
	\item $\Sigma' ~ = ~ \Ext {\Sigma_1} \Ext {\IMetaDecl c A M} \Sigma_2$
    \end{itemize}
    then $\Extends {\Sigma'} \Sigma$.
\fi
\end{lemma}
\fi

\if \DetailedProofs 1
\begin{proof}[Proofs]
    In both lemmas any derivation using $\Sigma$ is immediately valid also with $\Sigma'$.
\end{proof}
\fi

\if \DetailedProofs 1
To express the soundness of conversion checking we need to define when a
constraint is well-formed. Note that this is not the same as being true. For
instance, $\TypeConstr {\mathit{Nat}} {\mathit{Bool}}$ is a well-formed
constraint given $\mathit{Nat} : \SET$ and $\mathit{Bool} : \SET$ in the
signature.

\begin{definition}[Well-formed constraint]
    A constraint if well-formed if the terms (or types) under consideration are
    well-typed.
    \if \DetailedProofs 1
    \[
	\infer{ \ValidConstr \Sigma {\TypeConstr A B}}
	{\begin{array}{l}
	    \IsTypeCS \Sigma \Gamma A \\
	    \IsTypeCS \Sigma \Gamma B
	\end{array}}
    \qquad
	\infer{ \ValidConstr \Sigma {\TermConstr M N A}}
	{\begin{array}{l}
	    \HasTypeCS \Sigma \Gamma M A \\
	    \HasTypeCS \Sigma \Gamma N A \\
	\end{array}}
    \]
    \else
    For instace, $\TermConstr M N A$ is well-formed if $\HasTypeC
    \Gamma M A$ and $\HasTypeC \Gamma N A$.
    \fi
\end{definition}
\fi

Now we are ready to state the soundness of the type checking algorithm in the
absence of constraint solving.

\begin{theorem}[Soundness of type checking] \label{thmSoundNoCs}
    Type checking produces well-typed terms, conversion checking produces
    well-formed constraints and if no constraints are produced, the conversion
    is valid in {\Core}. Also, all rules produce well-formed extensions of the
    signature.
\if \DetailedProofs 1
    \begin{itemize}

	% is type
	\item
	    $\begin{array}[t]{l}
		\ExplicitJudgement \Sigma {\IsType e A} {\Sigma'}
		~ \wedge ~ \IsCtxCS \Sigma \Gamma \\
		{} \implies \Extends {\Sigma'} {\Sigma}
		~ \wedge ~ \IsTypeCS {\Sigma'} \Gamma A
	    \end{array}$

	% check type
	\item
	    $\begin{array}[t]{l}
		\ExplicitJudgement \Sigma {\CheckType e A M} {\Sigma'}
		~ \wedge ~ \IsTypeCS \Sigma \Gamma A \\
		{} \implies \Extends {\Sigma'} {\Sigma}
		~ \wedge ~ \HasTypeCS {\Sigma'} \Gamma M A
	    \end{array}$

	% infer type
	\item
	    $\begin{array}[t]{l}
		\ExplicitJudgement \Sigma {\InferType e A M} {\Sigma'}
		~ \wedge ~ \IsCtxCS \Sigma \Gamma \\
		{} \implies \Extends {\Sigma'} {\Sigma}
		~ \wedge ~ \HasTypeCS {\Sigma'} \Gamma M A
	    \end{array}$

	% equal types
	\item
	    $\begin{array}[t]{l}
		\ExplicitJudgement \Sigma {\EqualType A B \Cs} {\Sigma'}
		~ \wedge ~ \IsTypeCS \Sigma \Gamma A
		~ \wedge ~ \IsTypeCS \Sigma \Gamma B
		\\
		{} \implies \Extends {\Sigma'} {\Sigma}
		~ \wedge ~ \ValidConstr {\Sigma'} \Cs
		~ \wedge ~ (\Cs = \emptyset \implies \EqualTypeCS {\Sigma'} \Gamma A B)
	    \end{array}$

	% equal terms
	\item
	    $\begin{array}[t]{l}
		\ExplicitJudgement \Sigma {\Equal M N A \Cs} {\Sigma'}
		~ \wedge ~ \HasTypeCS \Sigma \Gamma M A
		~ \wedge ~ \HasTypeCS \Sigma \Gamma N A
		\\
		{} \implies \Extends {\Sigma'} {\Sigma}
		~ \wedge ~ \ValidConstr {\Sigma'} \Cs
		~ \wedge ~ (\Cs = \emptyset \implies \EqualCS {\Sigma'} \Gamma M N A)
	    \end{array}$

    \end{itemize}

    The statements for weak head normal form conversion ($\EqualWhnf M N A
    \Cs$) and term sequence conversion ($\Equal {\bar M} {\bar N} \Delta \Cs$)
    are equivalent to that of term conversion.
\fi
\end{theorem}

\begin{proof}
    By induction on the derivation.
\if \DetailedProofs 1
    Some interesting cases:
    \begin{itemize}

	\item {\em (TODO: Not so interesting)} In the type conversion case for
	function spaces where the domains produce constraints, we have to use
	Lemma~\ref{lemCoreSubstType}.

	\item In the term conversion case where the terms are weak head
	normalised we need subject reduction for weak head normalisation
	(Lemma~\ref{lemCoreSubjectReduction}).

	\item When checking conversion of terms with the same head we need an
	inversion principle for application (Lemma~\ref{lemCoreAppInv}).

	\item The most interesting case is the meta-variable instantiation
	case, so let us spell that out in more detail.

	The instantiation rule does not produce any constraints, so the only
	thing we have to prove is that it constructs a valid extension of the
	signature. This follows from the signature refinement lemma
	(Lemma~\ref{lemRefineSig}) which can be applied if we prove that if
	$\Sigma = \Ext {\Sigma_1} \Ext {\MetaDecl \alpha {B'}} \Sigma_2$ then
	$\LAM {\bar x} M : {B'}$.

	We have $\HasTypeCS \Sigma \Gamma {\alpha \, \bar x} A$ so $B'$ must
	have the form $\PI {\bar x} \Delta B$. By Lemma~\ref{lemCoreAppInv} we
	conclude that $\HasTypeCS \Sigma \Gamma {\bar x} \Delta$ and thus
	$\HasTypeCS \Sigma \Gamma {\alpha \, \bar x} B$. Then by
	Lemma~\ref{lemCoreEqType} $\EqualTypeCS \Sigma \Gamma A B$.

	From $\HasTypeCS \Sigma \Gamma M A$ we get $\HasTypeCS \Sigma \Gamma M
	B$ and using Lemma~\ref{lemCoreShadow}  $\HasTypeCS \Sigma \Gamma {\LAM
	{\bar x} M} {\Delta \to B}$. We know that $\Delta \to B$ is a closed
	type, and since $\FV{M} \subseteq \bar x$, $\LAM {\bar x} M$ is also
	closed. Thus by strenghtening (Lemma~\ref{lemCoreStrengthen})
	$\HasTypeCS \Sigma {} {\LAM {\bar x} M} {\Delta \to B}$. We have
	$\IsTypeCS {\Sigma_1} {} {\Delta \to B}$ and $\InScope \alpha M$ so
	$\HasTypeCS {\Sigma_1} {} {\LAM {\bar x} M} {\Delta \to B}$ which is
	what we set out to prove.

    \end{itemize}
\else
    The most interesting case is the meta-variable instantiation case where we
    have to prove that the instantiation constructs a valid extension of the
    signature. This is proven by showing that the instantiation is well-typed.
%     This follows from Lemma~\ref{lemRefineSig} once we show that
%     the instantiation is well-typed.
\fi
\end{proof}

Since well-typed terms in {\Core} have normal forms we get the existence of
normal forms for type checked terms and hence the type checking algorithm is
terminating.

\begin{corollary}
    The type checking algorithm is terminating.
\end{corollary}

Note that type checking terminates with one of three answers: {\em yes it is
type correct}, {\em no it is not correct}, or {\em it might be correct if the
meta-variables are instantiated properly}. The algorithm is not complete, since
finding correct instantiations to the meta-variables is undecidable in the
general case.

\subsection{Soundness of constraint solving}

In the previous section we proved type checking sound and decidable in the
absence of constraint solving. We also mostly ignored the constraints, only
requiring them to be well-formed. In this section we prove that the terms
produced by the type checker stay well-typed under constraint solving. This is
done by showing that constraint solving is a signature extension operator in
the sense of Definition~\ref{defSigExt}.

Previously we only ensured that the {\Core} restriction of the signature was
well-formed. Now, since we are going to update and remove the constraints of
guarded constants we have to strengthen the requirements and demand {\em
consistent} signatures. A signature is consistent if the solution of a guard is
a sufficient condition for the well-typedness of the definition it is guarding.

\if \DetailedProofs 1

\begin{definition}[Ensures] \label{defEnsures}
    A set of constraints $\Cs$ {\em ensures} a {\Core} judgement $J$ in a signature
    $\Sigma$ if, for any $\Sigma_1$ such that
    $\Extends {\Sigma_1} \Sigma$ and
    $\ExplicitJudgement {\Sigma_1} {\CheckConstr \Cs \emptyset} {\Sigma_2}$
    it is the case that $\vdash_{\CoreSig{\Sigma_2}} J$.
\end{definition}

\begin{remark} \label{lemExtendEnsures}
    If $\Cs$ ensures $J$ in $\Sigma$ and $\Extends {\Sigma'} \Sigma$ then $\Cs$
    ensures $J$ in $\Sigma'$.
\end{remark}

\begin{definition}[Consistent signature] \label{defConsistentSig}
    A signature $\Sigma$ is said to be {\em consistent} if $\Sigma = \Ext
    {\Sigma_1} \Ext {\ConstDecl p A M \Cs} \Sigma_2$ implies that $\Cs$ ensures
    $\HasTypeC {} M A$ in $\Sigma_1$.
\end{definition}

\fi

In order to prove that type checking preserves consistency, we first need to
know that the constraints we produce are sound.

\begin{lemma}[Soundness of generated constraints] \label{lemSoundConstraints}
    The constraints generated during conversion checking ensures that the
    checked terms are convertible. For instance, if $\EqualType A B \Cs$, then
    solving $\Cs$ guarantees that $\EqualTypeC \Gamma A B$ in {\Core}.
\if \DetailedProofs 1
    More precisely,
    \begin{itemize}
	\item \( \begin{array}[t]{l}
		    \IsTypeCS \Sigma \Gamma A
		    ~ \wedge ~ \IsTypeCS \Sigma \Gamma B
		    ~ \wedge ~ \ExplicitJudgement \Sigma {\EqualType A B \Cs} {\Sigma'}
		    \\ {} \implies \mbox{$\Cs$ ensures $\EqualTypeC \Gamma A B$ in $\Sigma'$}
	      \end{array} \)
	\item \( \begin{array}[t]{l}
		    \HasTypeCS \Sigma \Gamma M A
		    ~ \wedge ~ \HasTypeCS \Sigma \Gamma N A
		    ~ \wedge ~ \ExplicitJudgement \Sigma {\Equal M N A \Cs} {\Sigma'}
		    \\ {} \implies \mbox{$\Cs$ ensures $\EqualC \Gamma M N A$ in $\Sigma'$}
	      \end{array} \)
    \end{itemize}
\fi
\end{lemma}

\if \DetailedProofs 1
\begin{proof}
    Again we highlight some interesting cases.
    \begin{itemize}
	\item

	The only difficult case is the case of conversion for function types
	where a new constant $p$ is introduced. There we need to prove
	$\EqualTypeCS {\Sigma_2} {\Ext \Gamma x : A_1} {B_1} {B_2}$ from
	$\EqualTypeCS {\Sigma_2} {\Ext \Gamma x : A_1} {B_1} {\SubstD {B_2} {p
	\, \Gamma \, x}}$

	If $\Sigma_2$ has an empty guard for $p$ then $p \, \Gamma \, x$
	reduces to $x$ and we are done. If the guard is non-empty we can apply
	Lemma~\ref{lemCoreEqualDummySubst}, since $p \, \Gamma \, x$ is on weak
	head normal form, and $p$ is a fresh constant which does not appear in
	$B_1$.

	\item In the case where $\Cs$ is known (for instance, in the rule for
	blocked terms), we can apply soundness of conversion checking
	(Theorem~\ref{thmSoundNoCs}) to get $\TrueConstr {\Sigma_2} \Cs$.

    \end{itemize}
\end{proof}
\fi

\if \DetailedProofs 1
\begin{lemma} \label{lemRefineConsistent}
    Refinement preserves consistent signatures.
    More precisely, if
    \begin{itemize}
	\item $\HasTypeCS {\Sigma_1} {} M A$
	\item $\Sigma ~ = ~ \Ext {\Sigma_1} \Ext {\MetaDecl c A} \Sigma_2$
	\item $\Sigma' ~ = ~ \Ext {\Sigma_1} \Ext {\IMetaDecl c A M} \Sigma_2$
	\item $\Sigma$ is consistent
    \end{itemize}
    then $\Sigma'$ is consistent.
\end{lemma}

\begin{proof}
    There are two cases to consider: refinement to the left and to the right of
    a guard. In the latter case the proof is trivial, and in the former case
    consistency follows from the fact that refinement extends a signature
    (Lemma~\ref{lemRefineSig}).
\end{proof}
\fi

\begin{lemma}[Type checking preserves consistency] \label{lemTypeCheckConsistent}
Type checking and conversion checking preserves consistent signatures.
\if \DetailedProofs 1
More precisely,
    \begin{itemize}

	% is type
	\item
	    $\begin{array}[t]{l}
		\ExplicitJudgement \Sigma {\IsType e A} {\Sigma'}
		~ \wedge ~ \IsCtxCS \Sigma \Gamma
		~ \wedge ~ \mbox{$\Sigma$ is consistent}
		\\ {} \implies \mbox{$\Sigma'$ is consistent}
	    \end{array}$

	% check type
	\item
	    $\begin{array}[t]{l}
		\ExplicitJudgement \Sigma {\CheckType e A M} {\Sigma'}
		~ \wedge ~ \IsTypeCS \Sigma \Gamma A
		~ \wedge ~ \mbox{$\Sigma$ is consistent}
		\\ {} \implies \mbox{$\Sigma'$ is consistent}
	    \end{array}$

	% infer type
	\item
	    $\begin{array}[t]{l}
		\ExplicitJudgement \Sigma {\InferType e A M} {\Sigma'}
		~ \wedge ~ \IsCtxCS \Sigma \Gamma
		~ \wedge ~ \mbox{$\Sigma$ is consistent}
		\\ {} \implies \mbox{$\Sigma'$ is consistent}
	    \end{array}$

	% equal types
	\item
	    $\begin{array}[t]{l}
		\ExplicitJudgement \Sigma {\EqualType A B \Cs} {\Sigma'}
		~ \wedge ~ \IsTypeCS \Sigma \Gamma A
		~ \wedge ~ \IsTypeCS \Sigma \Gamma B \\
		~ \wedge ~ \mbox{$\Sigma$ is consistent}
		\\ {} \implies \mbox{$\Sigma'$ is consistent}
	    \end{array}$

	% equal terms
	\item
	    $\begin{array}[t]{l}
		\ExplicitJudgement \Sigma {\Equal M N A \Cs} {\Sigma'}
		~ \wedge ~ \HasTypeCS \Sigma \Gamma M A
		~ \wedge ~ \HasTypeCS \Sigma \Gamma N A \\
		~ \wedge ~ \mbox{$\Sigma$ is consistent}
		\\ {} \implies \mbox{$\Sigma'$ is consistent}
	    \end{array}$

    \end{itemize}

    The statements for weak head normal form conversion ($\EqualWhnf M N A
    \Cs$) and term sequence conversion ($\Equal {\bar M} {\bar N} \Delta \Cs$)
    are equivalent to that of term conversion.
\fi
\end{lemma}

\begin{proof}
\if \DetailedProofs 1
    By induction on the derivation. We only need to consider the cases where
    the signature changes. Adding a (non-guarded) constant trivially preserves
    consistency, instantiating a meta-variable preserves consistency by
    Lemma~\ref{lemRefineConsistent}. What remains is to check that new guarded
    constants are consistent. There are two cases: the conversion rule and
    conversion checking of function types. In both cases consistency follows
    from soundness of conversion checking (Lemma~\ref{lemSoundConstraints}).
\else
    We have to prove that when we introduce a guarded constant the guard
    ensures the well-typedness of the value. This follows from
    Lemma~\ref{lemSoundConstraints}.
\fi
\end{proof}

\begin{lemma}[Constraint solving is sound] \label{lemSolveConsistent}
    If $\Sigma$ is consistent and the solving the constraints yields a
    signature $\Sigma'$, then $\Sigma'$ is consistent and $\Extends {\Sigma'}
    \Sigma$.
\end{lemma}

\begin{proof}
    Follows from Theorem~\ref{thmSoundNoCs}, Lemma~\ref{lemSoundConstraints},
    and Lemma~\ref{lemTypeCheckConsistent}.
\end{proof}

From this follows that we can mix type checking and constraint solving freely,
so we can add a constraint solving rule to the type checking algorithm. In
order to obtain optimal approximations we have to solve constraints eagerly,
i.e as soon as a meta-variable has been instantiated.

\subsection{Relating user expressions and checked terms}

An important property of the type checking algorithm is that the type correct terms
produced correspond to the expressions being type checked. The correspondance
is expressed by stating that the only operations the type checker is allowed
when constructing a term is replacing a $?$ by a term ({\em refinement}) and
replacing a term by a guarded constant ({\em approximation}).

\if \DetailedProofs 1

\begin{definition}[Approximation]
    A term $M$ {\em approximates} $M'$ if $M$ can be obtained by replacing
    subterms of $M'$ by guarded constants $p \, \bar x$.
\end{definition}

\begin{definition}[Refinement]
    A term $M$ is a {\em refinment} of a user expression $e$ if $M$ can be
    obtained by replacing the $?$ in $e$ by concrete terms.
\end{definition}

\fi

\begin{lemma} \label{lemApproxRefine}
    If $\CheckType e A M$ then $M$ approximates a refinement of $e$. This
    property is preserved when unfolding instantiated meta-variables and
    guarded constants in $M$.
\end{lemma}

\begin{lemma}
    If $\CheckType e A M$ then $M$ is an {\em optimal approximation} of a
    refinement $M'$ of $e$.
\end{lemma}

\begin{proof}
The proof relies on the fact that we only introduce guarded constants when
absolutely necessary and solve the constraints eagerly. This is proven by
showing that the constraints produced by conversion checking are not only
sufficient but also necessary for the validity of the judgement.
\end{proof}

\subsection{Main result}

We now prove the main soundness theorem stating that if all meta-variables are
instantiated and all guards solved, then the term produced by the type checker
(extended with constraint solving) is valid in the original signature after
unfolding the definitions of the meta-variables and guarded constants
introduced during type checking.

\begin{theorem}[Soundness of type checking] \label{thmMain}
    If $\Sigma$ is a well-formed {\Core} signature and $\ExplicitJudgement
    \Sigma {\CheckType e A M} {\Sigma'}$, then if all meta-variables have been
    instantiated and all guards are empty in $\Sigma'$, then $\HasTypeCS \Sigma
    \Gamma {M\sigma} A$ where $\sigma$ is the substitution inlining the meta
    variables and constants in $\Sigma'$. Moreover, $M\sigma$ is a refinement
    of $e$.
\end{theorem}

\if \DetailedProofs 1
\begin{proof}
    From Theorem~\ref{thmSoundNoCs} follows that $\HasTypeCS {\Sigma'} \Gamma M
    A$ and Lemma~\ref{lemTypeCheckConsistent} and
    Lemma~\ref{lemSolveConsistent} give that $\Sigma'$ is a consistent
    extension of $\Sigma$.  Thus $\sigma$ is well-typed and we have $\HasTypeCS
    {\Sigma'} \Gamma {M\sigma} A$.
    Since $M\sigma$ only uses constants from $\Sigma$ we can strengthen the
    signature to obtain $\HasTypeCS \Sigma \Gamma {M\sigma} A$.

    By Lemma~\ref{lemApproxRefine} we have that $M\sigma$ approximates a
    refinement of $e$. Since $M\sigma$ does not contain any guarded constants
    it is a refinement of $e$.
\end{proof}
\fi


\subsubsection{compare}

\texttt{compare} carries out the actual comparison of the ``Bag of Words''
histograms and is also capable of calculating the structural similarity index
for spatial verification of possible duplicates.

\begin{verbatim}
USAGE:

compare  [--bowmetric <str>] [--metric <str>] [-v] [-l <int>] [--]
         [--version] [-h] <file1> <file2>

Where:

   --bowmetric <str>
     [BOWHistogram] Metric for histogram comparison (CV_COMP_CHISQR
     ,CV_COMP_CORREL,CV_COMP_INTERSECT,CV_COMP_BHATTACHARYYA)

   --metric <str>
     [ImageHistogram] Metric for histogram comparison (CV_COMP_CHISQR
     ,CV_COMP_CORREL,CV_COMP_INTERSECT,CV_COMP_BHATTACHARYYA)

   -v,  --verbose
     Provide additional debugging output

   -l <int>,  --level <int>
     Comparison Level (1-4)

   --,  --ignore_rest
     Ignores the rest of the labeled arguments following this flag.

   --version
     Displays version information and exits.

   -h,  --help
     Displays usage information and exits.

   <file1>
     (required)  file 1 that should be compared

   <file2>
     (required)  file 2 that should be compared
\end{verbatim}