\section{Wavefunction Optimization}
\label{sec:optimization}

Optimizing wavefunction is critical in all kinds of real-space quantum Monte Carlo calculations
because it significantly improves both the accuracy and efficiency of computation.
However, it is very difficult to directly adopt deterministic minimization approaches due to the stochastic nature of evaluating quantities with Monte Carlo.
Thanks to the algorithmic breakthrough during the first decade of this century and the tremendous computer power available,
it becomes feasible to optimize tens of thousands of parameters in a wavefunction for a solid or molecule.
QMCPACK has multiple optimizers implemented based on the state-of-the-art linear method.
We are continually improving our optimizers for the robustness and friendliness and trying to provide a single solution.
Due to the large variation of wavefunction types carrying distinct characteristics, using several optimizer may be needed in some cases.
It is highly suggested to read the recommendation from the experts maintaining these optimizers.

A typical optimization block looks like the following. It starts with method=``linear" and contains three blocks of parameters.
\begin{lstlisting}[style=XML]
 <loop max="10">
  <qmc method="linear" move="pbyp" gpu="yes">
    <!-- Specify the VMC options -->
    <parameter name="walkers">              256 </parameter>
    <parameter name="samples">          2867200 </parameter>
    <parameter name="stepsbetweensamples">    1 </parameter>
    <parameter name="substeps">               5 </parameter>
    <parameter name="warmupSteps">            5 </parameter>
    <parameter name="blocks">                70 </parameter>
    <parameter name="timestep">             1.0 </parameter>
    <parameter name="usedrift">              no </parameter>
    <estimator name="LocalEnergy" hdf5="no"/>
    ...
    <!-- Specify the correlated sampling options and define the cost function -->
    <parameter name="minwalkers">            0.3 </parameter>
         <cost name="energy">               0.95 </cost>
         <cost name="unreweightedvariance"> 0.00 </cost>
         <cost name="reweightedvariance">   0.05 </cost>
    ...
    <!-- Specify the optimizer options -->
    <parameter name="MinMethod">    OneShiftOnly </parameter>
    ...
  </qmc>
 </loop>
\end{lstlisting}
\begin{itemize}
\item loop is helpful to execute identical optimization blocks repeatedly.
\item The first part is highly identical to a regular VMC block.
\item The second part is to specify the correlated sampling options and define the cost function.
\item The last part is used to specify the options of different optimizers. They can be very distinct from one optimizer to another.
\end{itemize}

\subsection{VMC run for the optimization}
The VMC calculation for the wavefunction optimization has a strict requirement
that \texttt{samples} or \texttt{samplesperthread} must be specified because of the optimizer needs for the stored samples.
The input parameters of this part are identical to the VMC method.

Recommendations:
\begin{itemize}
\item Run the inclusive VMC calculation correctly and efficiently, because the this part takes significant amount of time in optimization.
For example, make sure the derived steps per block is 1 and use larger substeps to control the correlation between samples.
\item A reasonable starting wavefunction is necessary. A lot of optimization fails because of a bad wavefunction starting point.
The sign of a bad initial wavefunction includes but not limited to very long equilibration time, low acceptance ratio and huge variance.
The first thing to do after a failed optimization is to check the information provided by the VMC calculation via *.scalar.dat files.
\end{itemize}

\subsection{Correlated sampling and Cost function}
After generating the samples with VMC, the derivatives of the wavefunction with respect to the parameters are computed for proposing a new set of parameters by optimizers.
And later, a correlated sampling calculation is performed to quickly evaluate values of the cost function on the old set of parameters and the new set for the further decisions.
The input parameters are listed in the following table.
\begin{table}[h]
\begin{center}
\begin{tabularx}{\textwidth}{l l l l l l }
\hline
\multicolumn{6}{l}{\texttt{linear} method} \\
\hline
\multicolumn{2}{l}{parameters}  & \multicolumn{4}{l}{}\\
   &   \bfseries name     & \bfseries datatype & \bfseries values & \bfseries default   & \bfseries description \\
   &   \texttt{nonlocalpp} &  text     & yes, no & no  & include non-local PP energy in the cost function\\
%   &   \texttt{GEVMethod} &  text     & mixed, H2 & mixed  & methods of generalized eigenvalue problem\\
%   &   \texttt{beta} &  real     & any value & 0.0  & a parameter for GEVMethod\\
%   &   \texttt{use\_nonlocalpp\_deriv} &  text     & yes, no & no  & include the derivatives of non-local PP\\
   &   \texttt{minwalkers} &  real     & 0--1   & 0.3 & lower bound of the effective weight\\
   &   \texttt{maxWeight} &  real     & $>1$   & 1e6 & Maximum weight allowed in reweighting\\
  \hline
\end{tabularx}
\end{center}
\end{table}

Additional information:
\begin{itemize}
\item \texttt{maxWeight}. The default should be good.
\item \texttt{nonlocalpp}. Non-local PP contribution to the local energy depends on the wavefunction.
When a new set of parameter is proposed, this contribution needs to be updated if the cost function consists of local energy.
Fortunately, non-local contribution is chosen small when making a PP for small locality error.
We can ignore its change and avoid the expensive computational cost.
GPU code has a implementation issue that large amount of memory is consumed with this option.
\item \texttt{minwalkers}. A CRITICAL parameter. When the ratio of effective samples to actual number of samples in a reweighting step goes lower than \texttt{minwalkers},
the proposed set of parameters is invalid. % The last set of acceptable parameters is kept.
\end{itemize}

The cost function consists of three components: energy, unreweighted variance and reweighted variance.
\begin{lstlisting}[style=XML]
     <cost name="energy">                   0.95 </cost>
     <cost name="unreweightedvariance">     0.00 </cost>
     <cost name="reweightedvariance">       0.05 </cost>
\end{lstlisting}

\subsection{Optimizers}
QMCPACK implements a few optimizers having different preference aiming for different priorities.
They can be switched among  `OneShiftOnly' (default), `adaptive' and `quartic' (old) by the following line in the optimization block.
\begin{lstlisting}
<parameter name="MinMethod"> THE METHOD YOU LIKE </parameter>
\end{lstlisting}

\subsubsection{OneShiftOnly}
The OneShiftOnly optimizer targets a fast optimization by moving parameters more aggressively. It works with OpenMP and GPU and can be considered for large systems.
This method relies on the effective weight of correlated sampling rather than the cost function value to justify a new set of parameters.
If effective weight is larger than \texttt{minwalkers}, the new set is taken no matter the cost function value decreases or not.
If a proposed set is rejected, the standard output prints the measured ratio of effective samples to the total number of samples
and adjustment on \texttt{minwalkers} can be made if needed.

\begin{table}[h]
\begin{center}
\begin{tabularx}{\textwidth}{l l l l l l }
\hline
\multicolumn{6}{l}{\texttt{linear} method} \\
\hline
\multicolumn{2}{l}{parameters}  & \multicolumn{4}{l}{}\\
   &   \bfseries name     & \bfseries datatype & \bfseries values & \bfseries default   & \bfseries description \\
   &   \texttt{shift\_i} &  real     & $>0$ & 0.01 & Direct stabilizer added to the Hamiltonian matrix\\
   &   \texttt{shift\_s} &  real     & $>0$ & 1.00 & Initial stabilizer based on the overlap matrix\\
  \hline
\end{tabularx}
\end{center}
\end{table}

Additional information:
\begin{itemize}
\item \texttt{shift\_i}. This is the direct term added to the diagonal of the Hamiltonian matrix.
                         More stable but slower optimization with a large value.
\item \texttt{shift\_s}. This is the initial value of the stabilizer based on the overlap matrix added to the Hamiltonian matrix.
                         More stable but slower optimization with a large value. The used value is auto-adjusted by the optimizer.
\end{itemize}

Recommendations:
\begin{itemize}
  \item Default \texttt{shift\_i}, \texttt{shift\_s} should be fine.
  \item For hard cases, increasing \texttt{shift\_i} (factor of 5 or 10) can significantly stabilize the optimization by reducing the pace towards the optimal parameter set.
  \item If the VMC energy of the last optimization iterations grows significantly, increase \texttt{minwalkers} closer to 1 and make the optimization stable.
  \item If the first iterations of optimization are rejected on a reasonable initial wavefunction,
        lower the \texttt{minwalkers} value based on the measured value printed in the standard output to accept the move.
\end{itemize}

It is recommended to use this optimizer in two sections with a very small \texttt{minwalkers} in the first and a large value in the second like the following.
In the very beginning, parameters are far away form optimal values and large changes are proposed by the optimizer.
Having a small \texttt{minwalkers} allows accepting these changes much easier.
When the energy gradually converges, we can have a large \texttt{minwalkers} to avoid risky parameter sets.
\begin{lstlisting}[style=XML]
 <loop max="6">
  <qmc method="linear" move="pbyp" gpu="yes">
    <!-- Specify the VMC options -->
    <parameter name="walkers">                1 </parameter>
    <parameter name="samples">            10000 </parameter>
    <parameter name="stepsbetweensamples">    1 </parameter>
    <parameter name="substeps">               5 </parameter>
    <parameter name="warmupSteps">            5 </parameter>
    <parameter name="blocks">                25 </parameter>
    <parameter name="timestep">             1.0 </parameter>
    <parameter name="usedrift">              no </parameter>
    <estimator name="LocalEnergy" hdf5="no"/>
    <!-- Specify the optimizer options -->
    <parameter name="MinMethod">    OneShiftOnly </parameter>
    <parameter name="minwalkers">           1e-4 </parameter>
  </qmc>
 </loop>
 <loop max="12">
  <qmc method="linear" move="pbyp" gpu="yes">
    <!-- Specify the VMC options -->
    <parameter name="walkers">                1 </parameter>
    <parameter name="samples">            20000 </parameter>
    <parameter name="stepsbetweensamples">    1 </parameter>
    <parameter name="substeps">               5 </parameter>
    <parameter name="warmupSteps">            2 </parameter>
    <parameter name="blocks">                50 </parameter>
    <parameter name="timestep">             1.0 </parameter>
    <parameter name="usedrift">              no </parameter>
    <estimator name="LocalEnergy" hdf5="no"/>
    <!-- Specify the optimizer options -->
    <parameter name="MinMethod">    OneShiftOnly </parameter>
    <parameter name="minwalkers">            0.5 </parameter>
  </qmc>
 </loop>
\end{lstlisting}

For each optimization step, you will see
\begin{lstlisting}
The new set of parameters is valid. Updating the trial wave function!
\end{lstlisting}
or
\begin{lstlisting}
The new set of parameters is not valid. Revert to the old set!
\end{lstlisting}
Occasional rejection is fine. Frequent rejection indicates potential problems and users should inspect the VMC calculation or change optimization strategy.
To track the progress of optimization, using command ``qmca -q ev *.scalar.dat'' to look at the VMC energy and variance for each optimization step.

\subsubsection{adaptive}

The default setting of the adaptive optimizer is to construct the linear method Hamiltonian and overlap matrices explicitly and add different shifts to the Hamiltonian matrix
as ``stabilizers''.
The generalized eigenvalue problem is solved for each shift to obtain updates to the wave function parameters.
Then a correlated sampling is performed for each shift's updated wave function and the initial trial wave function
using the middle shift's updated wave function as the guiding function.
The cost function for these wave functions is compared, and the update corresponding to the best cost function is selected.
In the next iteration, the median magnitude of the stabilizers is set to that that generated the best update in the current iteration, thus adapting the magnitude of
the stabilizers automatically.

When the trial wave function contains more than ten thousand parameters, constructing and storing the linear method matrices may become a memory bottleneck.
To avoid explicit construction of these matrices, the adaptive optimizer implements the block linear method (BLM) approach. \cite{Zhao:2017:blocked_lm}
The BLM tries to find an approximate
solution $\vec{c}_{opt}$ to the standard LM generalized eigenvalue problem by dividing the variable space into a number of blocks
and making intelligent estimates for which directions within those blocks will be most important for constructing $\vec{c}_{opt}$.
which is then obtained by solving a smaller, more memory-efficient
eigenproblem in the basis of these supposedly important block-wise directions.

\begin{table}[h]
\begin{center}
\begin{tabularx}{\textwidth}{l l l l l l }
\hline
\multicolumn{6}{l}{\texttt{linear} method} \\
\hline
\multicolumn{2}{l}{parameters}  & \multicolumn{4}{l}{}\\
   &   \bfseries name     & \bfseries datatype & \bfseries values & \bfseries default   & \bfseries description \\
   %&   \texttt{stepsize} &  real     & 0--1 & 0.25  & Step size for moving parameters\\
   &   \texttt{max\_relative\_change} &  real     & $>0$ & 10.0 & Allowed change in cost function\\
   &   \texttt{max\_param\_change} &  real     & $>0$ & 0.3 & Allowed change in wave function parameter\\
   &   \texttt{shift\_i} &  real     & $>0$ & 0.01 & Initial diagonal      stabilizer added to the Hamiltonian matrix\\
   &   \texttt{shift\_s} &  real     & $>0$ & 1.00 & Initial overlap-based stabilizer added to the Hamiltonian matrix\\
   &   \texttt{chase\_lowest} &  text   & yes, no & yes & Chase the lowest eigenvector in iterative solver\\
   &   \texttt{chase\_closest} &  text   & yes, no & no & Chase the eigenvector closest to initial guess\\
   &   \texttt{block\_lm} &  text   & yes, no & no & Use block linear method\\
   &   \texttt{nblocks} &  integer   & $>0$ &  & \# of blocks in BLM\\
   &   \texttt{nolds} &  integer   & $>0$ &  & \# of old update vectors used in BLM\\
   &   \texttt{nkept} &  integer   & $>0$ &  & \# of eigenvectors to keep per block in BLM\\
  \hline
\end{tabularx}
\end{center}
\end{table}

Additional information:
\begin{itemize}
  \item \texttt{shift\_i}.  This is the initial coefficient used to scale the diagonal stabilizer.
                            More stable but slower optimization is expected with a large value.
                            The adaptive method will automatically adjust this value after each linear method iteration.
  \item \texttt{shift\_s}.  This is the initial coefficient used to scale the overlap-based stabilizer.
                            More stable but slower optimization is expected with a large value.
                            The adaptive method will automatically adjust this value after each linear method iteration.
  \item \texttt{nblocks}.   This is the number of blocks used in block LM. The amount of memory required to store LM matrices decreases
                            with increased number of blocks. But the error introduced by BLM would increase with number of blocks.
  \item \texttt{nolds}.     In BLM, the inter-block correlation is accounted for by including a small number of wave function update vectors
                            outside the block. Larger \texttt{nolds} would include more inter-block correlation and more accurate results, but
                            also higher memory requirements.
  \item \texttt{nkept}.     This is the number of update directions retained from each block in the BLM. If all directions are retained in each block,
                            then the BLM becomes equivalent to the standard LM.  Retaining 5 or fewer directions per block is often sufficient.
\end{itemize}

Recommendations:
\begin{itemize}
  \item Default shift\_i, shift\_s should be fine.
  \item When there are fewer than about 5,000 variables being optimized, the traditional LM is preferred as it has a lower overhead than the BLM when the number of variables is small.
  \item Initial experience with the BLM suggests that a few hundred blocks and a handful of \texttt{nolds} and \texttt{nkept}
        often provide a good balance between memory use and accuracy.  In general, using fewer blocks should be more accurate but will require more memory.
\end{itemize}

\begin{lstlisting}[style=XML]
 <loop max="15">
  <qmc method="linear" move="pbyp">
    <!-- Specify the VMC options -->
    <parameter name="walkers">                1 </parameter>
    <parameter name="samples">            20000 </parameter>
    <parameter name="stepsbetweensamples">    1 </parameter>
    <parameter name="substeps">               5 </parameter>
    <parameter name="warmupSteps">            5 </parameter>
    <parameter name="blocks">                50 </parameter>
    <parameter name="timestep">             1.0 </parameter>
    <parameter name="usedrift">              no </parameter>
    <estimator name="LocalEnergy" hdf5="no"/>
    <!-- Specify the correlated sampling options and define the cost function -->
         <cost name="energy">               1.00 </cost>
         <cost name="unreweightedvariance"> 0.00 </cost>
         <cost name="reweightedvariance">   0.00 </cost>
    <!-- Specify the optimizer options -->
    <parameter name="MinMethod">adaptive</parameter>
    <parameter name="max_relative_cost_change">10.0</parameter>
    <parameter name="shift_i"> 1.00 </parameter>
    <parameter name="shift_s"> 1.00 </parameter>
    <parameter name="max_param_change"> 0.3 </parameter>
    <parameter name="chase_lowest"> yes </parameter>
    <parameter name="chase_closest"> yes </parameter>
    <parameter name="block_lm"> no </parameter>
    <!-- Specify the BLM specific options if needed
      <parameter name="nblocks"> 100 </parameter>
      <parameter name="nolds"> 5 </parameter>
      <parameter name="nkept"> 3 </parameter>
    -->
  </qmc>
 </loop>
\end{lstlisting}
%To activate this optimizer, add ``-D BUILD\_LMYENGINE\_INTERFACE=1'' in the CMake command line.

The adaptive optimizer is also able to optimize individual excited states directly. \cite{Zhao:2016:dir_tar}
In this case, it tries to minimize the following function:
\begin{equation*}
\Omega[\Psi]=\frac{\left<\Psi|\omega-H|\Psi\right>}{\left<\Psi|{\left(\omega-H\right)}^2|\Psi\right>}
\end{equation*}
The global minimum of this function corresponds to the state whose energy lies immediately above the shift parameter $\omega$ in the energy spectrum.
For example, if $\omega$ were placed in between the ground state energy and the first excited state energy and the wave function ansatz was capable of a good
description for the first excited state, then the wave function would be optimized for the first excited state.
It is important to note that, if the ansatz is not capable of a good description of the excited state in question, the optimization may converge to a different
state, as is known to occur in some circumstances for traditional ground state optimizations.
Note also that the ground state can be targeted by this method by choosing $\omega$ to be below the ground state energy, although we should stress that this
is not the same thing as a traditional ground state optimization and will in general give a slightly different wave function.
Excited state targeting requires two additional parameters, as shown in this table.

\begin{table}[h]
\begin{center}
\begin{tabularx}{\textwidth}{l l l l l l }
\hline
\multicolumn{6}{l}{Excited State Targeting} \\
\hline
\multicolumn{2}{l}{parameters}  & \multicolumn{4}{l}{}\\
   &   \bfseries name     & \bfseries datatype & \bfseries values & \bfseries default   & \bfseries description \\
   %&   \texttt{stepsize} &  real     & 0--1 & 0.25  & Step size for moving parameters\\
   &   \texttt{targetExcited} &  text   & yes, no      & no   & Whether to use the excited state targeting optimization\\
   &   \texttt{omega}         &  real   & real numbers & none & Energy shift used to target different excited states\\
  \hline
\end{tabularx}
\end{center}
\end{table}

Excited state recommendations:
\begin{itemize}
  \item Due to the finite variance in any approximate wave function, it is recommended to set $\omega=\omega_0-\sigma$, where $\omega_0$ is placed just
        below the energy of the targeted state and $\sigma^2$ is the energy variance.
  \item In order to obtain an unbiased excitation energy, one should optimize the ground state with the excited state variational principle as well by setting
        \texttt{omega} below the ground state energy.  Note that using the ground state variational principle for the ground state and the excited state variational
        principle for the excited state creates a bias in favor of the ground state.
\end{itemize}

\subsubsection{quartic}
\textbf{This is an older optimizer method retained for compatibility. We recommend starting with the newest OneShiftOnly or adaptive optimizers.}
The quartic optimizer fits a quartic polynomial to 7 values of the cost function obtained using reweighting along chosen direction and determines the optimal move.
This optimizer is very robust but a bit conservative to accept new steps especially when large parameters changes are proposed.
\begin{table}[h]
\begin{center}
\begin{tabularx}{\textwidth}{l l l l l l }
\hline
\multicolumn{6}{l}{\texttt{linear} method} \\
\hline
\multicolumn{2}{l}{parameters}  & \multicolumn{4}{l}{}\\
   &   \bfseries name     & \bfseries datatype & \bfseries values & \bfseries default   & \bfseries description \\
   %&   \texttt{stepsize} &  real     & 0--1 & 0.25  & Step size for moving parameters\\
   &   \texttt{bigchange} &  real     & $>0$ & 50.0  & Largest parameter change allowed\\
   &   \texttt{alloweddifference} &  real     & $>0$ & 1e-4 & Allowed increased in energy\\
   &   \texttt{exp0} &  real     & any value & -16.0 & Initial value for stabilizer\\
   &   \texttt{stabilizerscale} &  real     & $>0$ & 2.0 & Increase in value of exp0 between iterations\\
   &   \texttt{nstabilizers} &  integer     & $>0$ & 3 & Number of stabilizers to try\\
   &   \texttt{max\_its} &  integer   & $>0$ & 1 & Number of inner loops with same samples\\
  \hline
\end{tabularx}
\end{center}
\end{table}

Additional information:
\begin{itemize}
\item \texttt{exp0}. It is the initial value for stabilizer (shift to diagonal of H). The actual value of stabilizer is $10^{\textrm{exp0}}$.
\end{itemize}

Recommendations:
\begin{itemize}
  \item{For hard cases (e.g. simultaneous optimization of long MSD and 3-Body J), set exp0
to 0 and do a single inner iteration (max its=1) per sample of configurations.}
\end{itemize}

\begin{lstlisting}[style=XML]
    <!-- Specify the optimizer options -->
    <parameter name="MinMethod">quartic</parameter>
    <parameter name="exp0">-6</parameter>
    <parameter name="alloweddifference"> 1.0e-4 </parameter>
    <parameter name="nstabilizers"> 1 </parameter>
    <parameter name="bigchange">15.0</parameter>
\end{lstlisting}

\subsection{General recommendations}
Here are a few recommendations to make wavefunction optimization easier.
\begin{itemize}
\item All electron wavefunctions are typically more difficult to optimize than pseudopotential ones due to the importance of the wavefunction near the nucleus.
\item Two body Jastrow contributes the largest portion of correlation energy from bare Slater determinants. For this reason, the recommended order for optimizing wavefunction components is two-body, one-body, three-body Jastrow factors and MSD coefficients.
\item For two-body spline Jastrows, always start from a reasonable one. The lack of physically-motivated constraints in the functional form at large distances can cause slow convergence if starting from zero.
\item One-body spline Jastrow from old calculations can be a good starting point.
\item Three-body polynomial Jastrow can start from zero. It is beneficial to first optimize one-body and two-body Jastrow factors without adding three-body terms in the calculation and then add the three-body Jastrow and optimize all the three components together.
\end{itemize}


\section{Calculus of variations}


\subsubsection{Model 1 - Scalar}
\label{sec:scalar_model}
In this model, each mixture is represented by a scalar function of the carbon chain lengths of the mixture’s components.  Then some function of those scalars predicts generalization across the mixtures.  In this model different mixtures are treated identically, if they map to the same scalar.  For example, a mixture of 2-heptanol and 2-nonanol might be treated identically to solution of only 2-octanol (a midpoint in carbon chain length).

\subsubsection{Model 2 - All to all}
\label{sec:all2all_model}
At the other extreme, in the most complex model (``All to all''), mixtures are compared by computing a function of each component in the $S+$ to each component in the test mixture -- for $N$-component mixtures this means $N^2$ computations ($N^2 = 4$ since these are binary mixtures) -- with functions added to predict generalization.  In this model each component ``interacts'' with each other component, and any deviation in any combination of components across mixtures is potentially predictive of non-generalization.
[STILL NOT SURE HOW THIS WORKS EXACTLY]

\subsubsection{Model 3 - Nearest Neighbor}
\label{sec:nearest_neighbor_model}
This model takes each component of the $S+$, and compares it only to the ``nearest'' component in the test mixture, with nearest determined by carbon chain length.  Since there is only one function for each $S+$ component (a function of the component and its nearest neighbor in the test mixture), there are $N$ computations ($N=2$ here).

We found that the nearest neighbor model was the best fitting model (Fig \ref{fig:else}).  Because vapor pressure increases with decreasing carbon chain length, shorter molecules are perceived more intensely under the equal concentrations used here.  Thus we also added a vapor pressure correction, motivated by the exponential form of the vapor pressure vs carbon chain length relationship, to account for potential differences in salience across mixture components.  This helped account for asymmetries in the observed generalization data, improving model fit.


\section{Background}
\label{sec:lts-transformation:background}
In this section, we introduce the basic concepts required to introduce our preservation check.
First, finite state LTSs are introduced, which we use to define the behavior of individual processes as well as the behavior of models formed by interacting processes.
Then, we discuss networks of LTSs, which are used to specify models in terms of interacting processes.
Next, we give a definition of DSBB, an equivalence relation between LTSs.
Finally, a fragment of the modal $\mu$-calculus~\cite{Kozen-83} that is compatible with DSBB is discussed.

\subsection{Labeled transition system}
An LTS~\graph is a tuple~\ltstuple{\graph}, where \states{\graph} is the (finite) set of states, $\actions{\graph}$ is the set of actions (including the invisible action $\tau$), $\transitions{\graph} \subseteq \states{\graph} \times \actions{\graph} \times \states{\graph}$ is the transition relation, and $\initialstates{\graph} \subseteq \states{\graph}$ the set of initial states.
Usually, LTSs representing potential behavior of concurrent systems have only one initial state, as defined in Section~\ref{sec:reusable-correct-transformations:operational_semantics}.
Here, we support multiple initial states to enable representing transformation rule patterns in terms of LTSs, as described in Section~\ref{sec:lts-transformation:transformations}.
In all other cases, LTSs have a single initial state.
Actions in \actions{\graph} are denoted by $a$, $b$, $c$, etc.
We use $s_1 \xrightarrow{a}_\graph s_2$ as a shorthand for $\langle s_1, a, s_2 \rangle \in \transitions{\graph}$.
If a transition $s_1 \xrightarrow{a}_\graph s_2$ is an element of $\transitions{\graph}$, this means that the LTS~\graph can move from state~$s_1$ to state~$s_2$ by performing action $a$.
The reflexive transitive closure of $\xrightarrow{\tau}_\graph$ is denoted by $\tausteps_\graph$.

\subsection{Network of LTSs}
We represent models consisting of a finite number of concurrent processes by a number of LTSs and a set of rules that define how these LTSs interact.
For this, we use the concept of networks of LTSs~\cite{lang05}.
Given an integer $n>0$, $1..n$ is the set of integers ranging from $1$ to $n$.
A vector $\vectornot{v}$ of size $n$ contains $n$ elements indexed by $1..n$. For $i \in 1..n$, $\vectornot{v}[i]$ denotes element $i$ in $\vectornot{v}$.

\begin{definition}
A network of LTSs~\smodel of size~$n$ is a pair~\networktuple, where
\begin{itemize}
\item $\Pi$ is vector of $n$ ({\it process}) LTSs.
For each $i \in 1..n$, we write $\Pi[i] = \ltstuple{i}$, and $s_1\xrightarrow{b}_i s_2$ is shorthand for $s_1\xrightarrow{b}_{\Pi[i]} s_2$;
\item \synchrules is a finite set of synchronization rules.
A synchronization rule is a tuple \syncruletuple, where $a$ is an action label, and $\vectornot{t}$ is a vector of size $n$ called a synchronization vector, whose elements are action labels from $\bigcup_{i\in 1..n} \actions{i}$ and a special symbol $\bullet$ that does not occur as a label in the LTSs.
If~$\vectornot{t}[i] = \bullet$ for some synchronization rule, then process LTS~$i$ is not involved in this synchronization.
\end{itemize}
\end{definition}

The potential behavior of a network of LTSs can be described as a single LTS.
For a network~$\smodel = \networktuple$, combining the LTSs in $\Pi$ according to the rules in $\synchrules$ produces a new LTS~\ltstuple{\smodel}, where
\begin{itemize}
\item $\initialstates{\smodel} = \{ \langle s_{1},\ldots, s_{n} \rangle \mid \forall i \in 1..n . s_{i} \in \initialstates{i} \}$;
\item $\actions{\smodel} = \{a \mid \langle \vectornot{t}, a \rangle \in \synchrules \}$;
\item $\states{\smodel} = \states{1} \times \ldots \times \states{n}$;
\item $\transitions{\smodel}$ is the smallest transition relation satisfying:
\[
\langle \vectornot{t}, a \rangle \in \synchrules \wedge \forall i \in 1..n . (\vectornot{t}[i] = \bullet \wedge s'[i]=s[i]) \vee (\vectornot{t}[i] \neq \bullet \wedge s[i]\xrightarrow{\vectornot{t}[i]}_i s'[i]) \implies s\xrightarrow{a}_{\smodel} s'.
\]
\end{itemize}

\noindent
In the remainder, we refer to such an LTS as a network LTS.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%With $\vectornot{s}\xrightarrow{b}_\smodel^{\vectornot{t}} \vectornot{s}'$, we denote that $\vectornot{t}$ enables $\vectornot{s}\xrightarrow{b}_\smodel \vectornot{s}'$.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{figure}[hbt]
\centering
\includegraphics[scale=0.2]{lts-transformation/figs/network}
\caption{A network of LTSs and the corresponding network LTS}
\label{fig:lts-transformation:network}
\end{figure}

On the left of Figure~\ref{fig:lts-transformation:network}, a network consisting of three process LTSs and four synchronization rules is shown.
The network LTS representing the behavior of this network is shown on the right of the figure.
The figure demonstrates the expressiveness of networks of LTSs.
It shows, for example, that multi-party synchronization is offered, as illustrated with the synchronization rule~$\langle\langle f, f, f\rangle, f\rangle$.
This rule specifies that the action~$f$ in the network LTS is the result of the synchronization of the actions~$f$ of the three processes.
Rule~$\langle\langle b, d, \bullet \rangle, e\rangle$ specifies a synchronization between processes~$\Pi[1]$ and~$\Pi[2]$, rule~$\langle\langle a, \bullet, \bullet \rangle, a \rangle$ specifies that action~$a$ of process~$\Pi[1]$ can be executed independently, and rule~$\langle\langle \bullet, c, \bullet \rangle, c \rangle$ specifies the same for action~$c$ of process~$\Pi[2]$.
Synchronization rules can also be used to introduce non-deterministic behavior, by specifying multiple rules involving the same actions.
For example, by adding the rule~$\langle\langle a, c, \bullet \rangle, g \rangle$ to the network of Figure~\ref{fig:lts-transformation:network}, $\Pi[1]$ and~$\Pi[2]$ can either synchronize or perform action~$a$ and~$c$ independently.

To abstract from certain actions, we define the hiding operator $\abstr$, which renames all actions in~$H$ to~$\tau$.
This operator can be extended to networks of LTSs.

\begin{definition}
\label{def:lts-transformations:hiding}
Let \actions{} be a set of actions and $H \subseteq \actions{}$.
The hiding of an action $a \in \actions{}$ w.r.t.\ $H$ is defined as follows.
\[
\abstr (a) =
\left\{\begin{array}{ll}
a    & {\rm if} ~ a \not\in H \\
\tau & {\rm if} ~ a \in H
\end{array}\right.
\]
The hiding of a network $\smodel = \langle \Pi, \synchrules \rangle$ w.r.t.\ $H$ is defined as follows.
\[
	\abstr (\langle \Pi, \synchrules \rangle) =
	\langle
		\Pi,
		\{\langle \vectornot{t}, \abstr(a)\rangle \mid \langle \vectornot{t}, a \rangle \in \synchrules \}
	\rangle
\]
\end{definition}

\subsection{Divergence-Sensitive Branching Bisimilarity}
We use the equivalence relation DSBB~\cite{GlaLutTrc09} to relate LTSs, which preserves $\tau$-cycles and branching-time properties such as inevitable reachability.

\begin{definition}
\label{def:lts-transformation:dsbbsim}
For LTSs $\graph_1 = \ltstuple{\graph_1}$ and~$\graph_2 = \ltstuple{\graph_2}$, a binary relation $B \subseteq \states{\graph_1} \times \states{\graph_2} $ is a divergence-sensitive branching bisimulation if for all $s \in \states{\graph_1}$ and $t \in \states{\graph_2}$ such that $s\ B\ t$, the following conditions hold.
\begin{enumerate}
\item If $s \xrightarrow{a}_{\graph_1} s'$ then
  \begin{itemize}
  \item either $a=\tau$ with $s'\ B\ t$;
  \item or $t \tausteps_{\graph_2} {\hat t} \xrightarrow{a}_{\graph_2} t'$ with $s\ B\ {\hat t}$ and $s'\ B\ t'$.
  \end{itemize}
\item Symmetrically, if $t \xrightarrow{a}_{\graph_2} t'$ then
  \begin{itemize}
  \item either $a=\tau$ with $s\ B\ t'$;
  \item or $s \tausteps_{\graph_1} {\hat s} \xrightarrow{a}_{\graph_1} s'$ with ${\hat s}\ B\ t$ and $s'\ B\ t'$.
  \end{itemize}
\item If for all $k\geq 0$ and $s=s_0$, $s_k \xrightarrow{\tau}_{\graph_1} s_{k+1}$ then for all $\ell\geq 0$ and $t=t_0$, $t_\ell \xrightarrow{\tau}_{\graph_2} t_{\ell+1}$ and $s_k\ B\ t_{\ell}$, for all $k, \ell$.
\item Symmetrically, if for all $k\geq 0$ and $t=t_0$, $t_k \xrightarrow{\tau}_{\graph_2} t_{k+1}$ then for all $\ell\geq 0$ and $s=s_0$, $s_\ell \xrightarrow{\tau}_{\graph_1} s_{\ell+1}$ and $s_{\ell}\ B\ t_k$, for all $k, \ell$.
\end{enumerate}
Two states $s$ and $t$ are divergence-sensitive branching bisimilar, denoted by $s\dsbbis t$, if there is a divergence-sensitive branching bisimulation~$B$ with $s\ B\ t$.
\end{definition}

Two LTSs~$\graph_1 = \ltstuple{\graph_1}$ and~$\graph_2 = \ltstuple{\graph_2}$ are divergence-sensitive branching bisimilar, denoted by $\graph_1 \dsbbis \graph_2$, iff $\forall s_1 \in \initialstates{\graph_1}. \exists s_2 \in \initialstates{\graph_2}. s_1 \dsbbis s_2$ and vice versa.
Furthermore, a state $s$ is diverging, denoted by $s\diverge$, iff an infinite $\tau$-path is reachable from~$s$.
For finite LTSs, this means that a $\tau$-cycle is reachable via $\tau$-transitions.

If the constituting LTSs satisfy the following conditions related to $\tau$-transitions, DSBB is a congruence for networks of LTSs,
which means that replacing a process LTS in a network with a process LTS that is divergence-sensitive branching bisimilar leads to a network for which the corresponding network LTS is divergence-sensitive branching bisimilar with the network LTS that corresponds to the original network.

\begin{definition}
\label{def:lts-transformation:admissibility}
A network $\smodel = \networktuple$ is called admissible iff the following holds.
\begin{enumerate}
\item \textbf{No synchronization and renaming:}
\[
\forall \syncruletuple \in \synchrules. \vectornot{t}[i] = \tau \implies {\it Ac}(\vectornot{t}) = \{i\} \wedge a=\tau;
\]
\item \textbf{No cut:} $\exists s_1, s_2 \in \states{i}. s_1\xrightarrow{\tau}_i s_2 \implies \exists \langle \vectornot{t}, \tau \rangle \in \synchrules. \vectornot{t}[i] = \tau$,
\end{enumerate}
\end{definition}

\noindent
where the set~${\it Ac}(\vectornot{t})$ of indices of processes active for a synchronization vector~$\vectornot{t}$ is defined as ${\it Ac}(\vectornot{t})= \{i \mid i \in 1..n \wedge \vectornot{t}[i] \neq \bullet \}$.
The first condition states that a $\tau$-action may not synchronize with other actions and that it may not be renamed, and the second condition states that there must exist synchronization rules that enable the $\tau$-transitions of each process.
In the remainder of this chapter, only admissible networks are considered.

\subsection{The modal $\mu$-calculus \dsbrLmu}
\label{subsec:lts-transformation:lmu}
Mateescu and Wijs~\cite{mateescu.wijs.propred} identified a fragment of the modal $\mu$-calculus~\cite{Kozen-83} that is fully compatible with the maximal hiding technique~\cite{mateescu.wijs.propred} and DSBB.
The fragment is called~\dsbrLmu and is suitable for expressing safety, liveness, and fairness properties.
In~\dsbrLmu, properties are expressed in terms of action formulas, denoted by $\Af$, and state formulas, denoted by $\Sf$ and $\Sp$.
The syntax of these formulas and the semantics of the action formulas are defined as follows.

\[
\begin{array}{lcl}
\Af & {::=} & a \mid \FALSE \mid \neg \Af_1 \mid \Af_1 \vee \Af_2
\end{array}
\]

\[
\begin{array}{rcl}
\sem a \antic_{\actions{}} & = & \{ a \} \\
\sem \FALSE \antic_{\actions{}} & = & \emptyset \\
\sem \neg \Af_1 \antic_{\actions{}} & = & {\actions{}} \setminus \sem \Af_1 \antic_{\actions{}} \\
\sem \Af_1 \vee \Af_2 \antic_{\actions{}} & = & \sem \Af_1 \antic_{\actions{}} \cup \sem \Af_2 \antic_{\actions{}}
\end{array}
\]

\[
\begin{array}{rcl}
\Sf & {::=} & \FALSE
 \mid \neg \Sf_1
 \mid \Sf_1 \vee \Sf_2
 \mid \langle (\Sf_1 ? . \Af_h)^{*} \rangle \Sp
 \mid \langle \Sf_1 ? . \Af_h \rangle @
 \mid X
 \mid \mu X . \Sf_1 \\
\Sp & {::=} & \Sf
 \mid \langle \Af_v \rangle \Sf
 \mid \neg \Sp_1
 \mid \Sp_1 \vee \Sp_2,
\end{array}
\]

\noindent
where $\actions{}$ is a set of actions, $X$ is a set of propositional variables, $\tau \in \sem \Af_h \antic_{\actions{}}$, and~$\tau \not\in \sem \Af_v \antic_{\actions{}}$.

Interpretation $\sem \Af \antic_\actions{}$ of $\Af$ on the set of actions $\actions{}$ denotes the set of actions satisfying~$\Af$.
An action $a$ satisfies a formula $\Af$, denoted by $a \models_\actions{} \Af$, iff $a \in \sem \Af \antic_\actions{}$.
Furthermore, a transition~$s_1\ \smash{\xrightarrow{a}_\graph}\ s_2$ with~$a \models_\actions{\graph} \Af$ is called an $\Af$-transition.

We do not provide the formal semantics of the state formulas, as it is not essential for understanding the current work.
Compared to the standard modal $\mu$-calculus, the fragment~\dsbrLmu differs in two respects.
First, it introduces the weak possibility modality and the weak infinite looping operator.
Informally, the weak possibility modality~$\langle (\Sf_1 ? . \Af_h)^{*} \rangle \Sp$ characterizes the states having an outgoing sequence of zero or more $\Af_h$-transitions whose intermediate states satisfy~$\Sf_1$ and whose terminal state satisfies~$\Sp$.
The weak infinite looping operator~$\langle \Sf_1 ? . \Af_h \rangle @$ characterizes the states having an infinite outgoing sequence of~$\Af_h$-transitions whose intermediate states satisfy~$\Sf_1$.
In both cases, $\Af_h$ must capture~$\tau$.
Second, the occurrence of the strong modality~$\langle \Af_v \rangle \Sf$ is restricted syntactically such that it can appear only after a weak possibility modality, and the action formula~$\Af_v$ must denote visible actions only.
For further details, we refer to the work of Mateescu and~Wijs~\cite{mateescu.wijs.propred}.

The fact that a state~$s$ of an LTS~\graph satisfies a closed state formula~$\Sf$ is denoted by~$s \models_\graph \Sf$.
A formula is closed if all propositional variables are bound.
Additionally, we denote~$\forall s \in \initialstates{\graph} . s \models_\graph \Sf$ by~$\models_\graph \Sf$.

When checking a state formula $\Sf$ on an LTS, some actions can be hidden (renamed to~$\tau$) without disturbing the interpretation of~$\Sf$.

\begin{definition}
\label{def:lts-transformation:hiding-set}
Let $\Af$ be an action formula interpreted over a set of actions $\actions{}$.
The hiding set of $\Af$ w.r.t.\ $\actions{}$ is defined as follows.
\[
h_\actions{} (\Af) = \left\{
\begin{array}{ll}
\sem \Af \antic_\actions{}                    & {\rm if} ~ \tau \models \Af \\
\actions{} \setminus \sem \Af \antic_\actions{} & {\rm if} ~ \tau \not\models \Af
\end{array}
\right.
\]
The hiding set of a state formula $\Sf$ w.r.t.\ $\actions{}$, denoted by $h_\actions{} (\Sf$), is defined as the intersection of $h_\actions{} (\Af)$ for all action subformulas $\Af$ of $\Sf$.
\end{definition}

We denote maximal hiding in an LTS~$\graph = \ltstuple{\graph}$ as $\maxabstr(\graph) = \smash{\abstrset{h_{\actions{\graph}}(\Sf)}(\graph)}$.
Maximal hiding preserves~\dsbrLmu properties: $\models_\graph \Sf \iff \smash{\models_{\maxabstr(\graph)} \Sf}$.
Furthermore, for closed $\Sf$, \dsbrLmu is compatible with the $\dsbbis$ relation.
Let $\graph = \ltstuple{\graph}$ be an LTS and let $s_1, s_2 \in \states{\graph}$ such that $s_1 \dsbbis s_2$.
Then, $s_1 \models_\graph \Sf \iff s_2 \models_\graph \Sf$ for any closed state formula $\Sf$ of \dsbrLmu.
This allows reducing an LTS (after maximal hiding) modulo DSBB before verifying a closed~\dsbrLmu formula~\cite{mateescu.wijs.propred}.
It also allows reasoning about LTSs w.r.t.\ properties.
Given a~\dsbrLmu formula $\Sf$ and LTSs~$\graph_1$ and $\graph_2$ with $\maxabstr(\graph_1) \dsbbis \maxabstr(\graph_2)$, then, if~$\models_{\graph_1} \Sf$, also $\models_{\graph_2} \Sf$.


\documentclass[/../base.tex]{subfiles}

\begin{document}
\section{Result Tables}
\label{a:tables}

\begin{table}[H]
	\centering
	\begin{threeparttable}
		\caption{Contagion Coefficients - OLS}
		\centering
		\begin{tabular}{p{1.5cm}*{8}{c}}
			\toprule
			\toprule
			Crisis Dummy & ES & FR & GR & IE & IT & NL & PT \\
			\midrule
			\multirow{2}{*}{ES} & - & \textbf{0.007} & 0.09 & \textbf{0.059} & \textbf{0.072} &\textbf{0.007}& 0.063 \\
			& - & (0.005) & (0.067) & (0.018) & (0.012) & (0.003) & (0.025) \\
			\multirow{2}{*}{FR} & -0.001 & - & \textbf{-0.021} & \textbf{-0.001} & \textbf{0.026} & \textbf{-0.001} & -0.002 \\
			& (0.002)& - & (0.023) & (0.006) & (0.004) & (0.001) & (0.009) \\
			\multirow{2}{*}{GR} & -0.005 & \textbf{0.057} & - & -0.007 & 0.019 & 0.002 & \textbf{-0.003} \\
			& (0.002) & (0.021) & - & (0.005) & (0.004) & (0.001) & (0.008)\\
			\multirow{2}{*}{IE} & 0.004 & 0.033 & 0.006 & - & -0.002 & \textbf{-0.002} & \textbf{0.026} \\
			& (0.001) & (0.020) & (0.006) & - & (0.004) & (0.001) & (0.009)\\
			\multirow{2}{*}{IT}  & \textbf{0.014} & \textbf{0.041} & 0.008 & \textbf{0.023} & - & \textbf{0.002} & 0.059 \\
			& (0.002) & (0.025) & (0.006) & (0.003) & - & (0.001) & (0.007)\\
			\multirow{2}{*}{NL} & 0.008 & -0.010 & \textbf{0.015} & \textbf{0.011} &\textbf{ 0.003} & - & 0.0010 \\
			& (0.001) & (0.018) & (0.008) & (0.003) & (0.001) & - & (0.009)\\
			\multirow{2}{*}{PT} & -0.002 & 0.11 & \textbf{0.03} & \textbf{0.005} & -0.001 & 0.007 & - \\
			& (0.002) & (0.022) & (0.006) & (0.004) & (0.001) & (0.007) & -\\
			\bottomrule
			\bottomrule
		\end{tabular}
		\begin{tablenotes}
			\small
			\item Notes: Coefficients on a crisis dummy country equation (listed in columns).
			\item Bold indicates significant at 5\%
		\end{tablenotes}
		\label{tab:contagion_ols_a}
	\end{threeparttable}
\end{table}

\begin{table}[H]
	\centering
	\begin{threeparttable}
		\caption{Contagion Coefficients - GIVE}
		\centering
		\begin{tabular}{p{1.5cm}*{8}{c}}
			\toprule
			\toprule
			Crisis Dummy & ES & FR & GR & IE & IT & NL & PT \\
			\midrule
			\multirow{2}{*}{ES} & - & -0.06 & 0.822 & 0.358 & 0.191 & 0.004 & 0.131 \\
			& - & (0.428) & (0.720) & (0.277) & (0.115) & (0.024) & (0.344)\\
			\multirow{2}{*}{FR} & 0.262 & - & 0.933 & 0.453 & 0.335 & -0.093 & -0.320 \\
			& (1.426) & - & (3.67) & (0.803) & (0.479) & (0.262) & (0.746) \\
			\multirow{2}{*}{GR} & -0.004 & 1.645 & - & 0.201 & 0.074 & -0.017 & 0.245 \\
			& (0.233) & (1.09) & - & (0.363) & (0.182) & (0.05) & (0.472)\\
			\multirow{2}{*}{IE} & 0.048 & 0.297 & -0.162 & - & -0.099 & 0.029 & \textbf{0.317}\\
			& (0.178) & (0.424) & (0.192) & - & (0.151) & (0.080) & (0.07)\\
			\multirow{2}{*}{IT} & 0.172 & -0.695 & -0.738 &  0.011 & - & 0.017 & 0.231\\
			& (0.856) & (1.893) & (0.454) & (0.081) & - & (0.022) & (0.078)\\
			\multirow{2}{*}{NL} & 0.242 & -0.696 & -0.302 & -0.091 & 0.024 & - & -0.040\\
			& (1.3627) & (0.815) & (0.235) & (0.133) & (0.085) & - & (1.00)\\
			\multirow{2}{*}{PT} & -0.709 & -3.03 & 0.209 & -0.484 & 0.109 & 0.025 & -\\
			& (3.122) & (3.666) & (1.02) & (0.529) & (0.273) & (0.343) & -\\
			\bottomrule
			\bottomrule
		\end{tabular}
		\begin{tablenotes}
			\small
			\item Notes: Notes: Coefficients on a crisis dummy country equation (listed in columns).
			\item Bold indicates significant at 5\%
		\end{tablenotes}
		\label{tab:contagion_give_a}
	\end{threeparttable}
\end{table}

\end{document}


\subsection{Parameters}
Parameters are set with  commands of the form
\begin{verbatim}
:- set_sc(<parameter>,<value>).
\end{verbatim}
The available parameters are:
\begin{itemize}
\item \verb|specialization|: (values: \verb|{bottom,mode}|,  default value: \texttt{bottom}, valid for SLIPCOVER) specialization mode.
\item \verb|depth_bound|: (values: \verb|{true,false}|,  default value: \texttt{true}) if \texttt{true}, the depth of the derivation of the goal is limited to the value of the \texttt{depth} parameter.
\item \verb|depth| (values: integer, default value: 2): depth of derivations if  \verb|depth_bound|  is set to \verb|true|
\item \verb|single_var| (values: \verb|{true,false}|, default value: \verb|false|): if set to \verb|true|, there is a random variable for each clause, instead of a different random variable for each grounding of each clause
\item \verb|epsilon_em| (values: real, default value: 0.1): if the difference in the log likelihood in two successive parameter EM iteration is smaller
than \verb|epsilon_em|, then EM stops
\item \verb|epsilon_em_fraction| (values: real, default value: 0.01): if the difference in the log likelihood in two successive parameter EM iteration is smaller
than \verb|epsilon_em_fraction|*(-current log likelihood), then EM stops
\item \verb|iter| (values: integer, defualt value: 1): maximum number of iteration of EM parameter learning. If set to -1, no maximum number of iterations is imposed
\item \verb|iterREF| (values: integer, defualt value: 1, valid for
 SLIPCOVER and LEMUR):
 maximum number of iteration of EM parameter learning for refinements. If set to -1, no maximum number of iterations is imposed.
\item \verb|random_restarts_number| (values: integer, default value: 1, valid for EMBLEM, SLIPCOVER and LEMUR): number of random restarts of parameter EM learning
\item \verb|random_restarts_REFnumber| (values: integer, default value: 1, valid for  SLIPCOVER and LEMUR): number of random restarts of parameter EM learning for refinements
\item \verb|seed| (values: seed(integer) or seed(random), default value \verb|seed(3032)|): seed for the Prolog random functions, see \href{http://www.swi-prolog.org/pldoc/man?predicate=set_random/1}{SWI-Prolog manual}
\item \verb|c_seed| (values: unsigned integer, default value 21344)): seed for the C random functions
\item \verb|logzero| (values: negative real, default value $\log(0.000001)$): value assigned to $\log 0$
\item \verb|max_iter| (values: integer, default value: 10, valid for  SLIPCOVER): number of interations of beam search
\item \verb|max_var| (values: integer, default value: 4, valid for
SLIPCOVER and LEMUR): maximum number of distinct variables in a clause
\item \verb|beamsize|  (values: integer, default value: 100, valid for SLIPCOVER): size of the beam
\item \verb|megaex_bottom| (values: integer, default value: 1, valid for SLIPCOVER): number of mega-examples on which to build the bottom clauses
\item \verb|initial_clauses_per_megaex| (values: integer, default value: 1, valid for SLIPCOVER):
 number of bottom clauses to build for each mega-example (or
 model or interpretation)
\item \verb|d| (values: integer, default value: 1, valid for SLIPCOVER):
 number of saturation steps when building the bottom clause
\item \verb|mcts_beamsize|  (values: integer, default value: 3, valid for LEMUR): size of the Monte-Carlo tree search beam
\item \verb|mcts_visits|  (values: integer, default value: +1e20, valid for LEMUR): maximum number of visits
\item \verb|max_iter_structure| (values: integer, default value: 10000, valid for SLIPCOVER):
maximum  number of theory search iterations
\item \verb|background_clauses| (values: integer, default value: 50, valid for SLIPCOVER):
 maximum numbers of background clauses
\item \verb|maxdepth_var| (values: integer, default value: 2, valid for SLIPCOVER and LEMUR): maximum depth of
variables in clauses (as defined in \cite{DBLP:journals/ai/Cohen95}).
\item \verb|mcts_max_depth|  (values: integer, default value: 8, valid for LEMUR): maximum depth of default policy search
\item \verb|mcts_c|  (values: real, default value: 0.7, valid for LEMUR): value of parameter $C$ in the computation of UCT
\item \verb|mcts_iter|  (values: integer, default value: 20, valid for LEMUR): number of Monte-Carlo tree search iterations
\item \verb|mcts_maxrestarts|  (values: integer, default value: 20, valid for LEMUR): maximum number of Monte-Carlo tree search restarts
\item \verb|neg_ex| (values:  \verb|given|, \verb|cw|, default value: \verb|cw|): if  set to \verb|given|, the negative examples in testing
are taken from the test folds interpretations, i.e., those examples \verb|ex| stored as \verb|neg(ex)|; if set to \verb|cw|, the negative examples are generated according to the closed world assumption, i.e., all atoms for target predicates that are not positive examples. The set of all atoms is obtained by collecting the set of constants for each type of the arguments of the target predicate.
\item \verb|alpha| (values: floating point $\geq 0$, default value: 0): parameter of the
symmetric Dirichlet distribution used to initialize the parameters. If it takes value 0, a truncated Dirichlet process is
used to sample parameters: the probability of being true of  each Boolean random variable
used to represent multivalued
random variables is sampled uniformly and independently in [0,1]. If it takes a value $\geq 0$, the parameters are sampled from
a symmetric Dirichlet distribution, i.e. a Dirichlet distribution  with vector of parameters $(\alpha,\ldots,\alpha)$.
\item \verb|verbosity| (values: integer in [1,3], default value: 1): level of verbosity of the algorithms.
\end{itemize}


The module {\module{blocktree}} ({\file{blocktree.ml}} and {\file{blocktree.mli}})
contains code related to keep up with the current tree of blocks
and the current best block.
Other various information is also included here.

\section{Block Tree}

The main data type in the module {\module{blocktree}} is the type {\type{blocktree}}
with one constructor {\constr{BlocktreeNode}} (described below).
Each node can be thought of as between a block and possible successor blocks.
Enough information is included in the node to determine if a new block (header) is a valid successor block (header).
The root of the tree is the ``genesis node'' which contains information about
the initial ledger tree, initial stake modifier and initial target.
Qeditas has no ``genesis block'' in the traditional sense. The first block would be
a valid child of the genesis node.

Nodes to the tree are added as headers are received (or staked) even if the corresponding block delta
has not yet been received.
In order to distinguish between these cases there is a type {\type{validationstatus}}
with three constructors: {\constr{Waiting}} (meaning the block delta is being requested from peers),
{\constr{ValidBlock}} (meaning a delta has been received and checked to be part of a valid block when
combined with the header)
or {\constr{InvalidBlock}} (meaning there is no delta making the header part of a valid block, though it is not clear how this could be determined).

The single constructor {\constr{BlocktreeNode}} of the type {\type{blocktree}} takes 13 arguments:\footnote{This probably should be a record type.}
\begin{itemize}
\item an optional {\type{blocktree}} giving the parent, which is {\val{None}} for the genesis node (and possibly for some checkpoint nodes).
\item a reference to a list of hash values, giving a list of addresses which have staked in some child of this node up to six levels deep. This is to be used to identify double-staking which can be punished by proof-of-forfeiture.
\item an optional hash of the previous block header, which is {\val{None}} for the genesis node.
\item an optional root of the current theory tree (see {\type{ttree}} in Chapter~\ref{chap:math}).
\item an optional root of the current signature tree (see {\type{stree}} in Chapter~\ref{chap:math}).
\item the current ledger hash root.
\item the current target information (including the current and future stake modifier).
\item the current time stamp.
\item the current cumulative stake.
\item the current block height.
\item a reference to the current validation status (of type {\type{validationstatus}}).
\item a reference to a boolean indicating if the block above the node has been blacklisted, so children should not be considered valid.
\item a reference to a list of children, which gets updated as successor blocks are found.
\end{itemize}
There are a variety of functions for obtaining the information above from a node in the tree:
{\func{node\_recent\_stakers}},
{\func{node\_prevblockhash}},
{\func{node\_theoryroot}},
{\func{node\_signaroot}},
{\func{node\_ledgerroot}},
{\func{node\_targetinfo}},
{\func{node\_timestamp}},
{\func{node\_cumulstk}},
{\func{node\_blockheight}},
{\func{node\_validationstatus}} and
{\func{node\_children\_ref}}.

The function {\func{eq\_node}} tests if two nodes are equal by simply checking
if they have the save previous block hash.
The usual equality operation cannot be used since the target is of type {\type{big\_int}}
whose values are (often) abstract.

The variable {\var{genesisblocktreenode}} is initialized to be the node with the
genesis ledger root, genesis target information and genesis time stamp.
A child of this node corresponds to a block at height $1$.

The hash table {\var{blkheadernode}} associates hash values with their corresponding
node in the block tree.

The functions {\func{process\_new\_header}},
{\func{process\_new\_header\_a}},
{\func{process\_new\_header\_aa}},
{\func{process\_new\_header\_ab}}
and {\func{process\_new\_header\_b}}
are used to check a header is valid and, if so, create
a node for it and enter the node into the block tree.
If the previous block hash in the header does not correspond
to a node in the tree (according to {\var{blkheadernode}}),
the new header is considered an orphan and put into
the hash table {\var{orphanblkheaders}}.
The parent header of an orphan block header are requested using {\func{find\_and\_send\_requestdata}}, if possible.
When a new header is processed, it may mean previously orphaned headers are no longer orphaned,
at which point the headers should be removed from {\var{orphanblkheaders}}
and new nodes in the block tree should be added. (The handling of orphan blocks has not been tested.)

In addition, there is a list {\var{earlyblocktreenodes}} which contains headers which have
been received but whose time stamps are in the future. These are intended to be delayed until
the appropriate time arrives (although the code to handle them later does not seem to be written).

The hash table {\var{tovalidate}} is a set of hashes of headers where we are still waiting to validate the corresponding block delta.

\section{Checkpoints}

The checkpoint public key is given in this module
by {\var{checkpointspubkeyx}}
and {\var{checkpointspubkeyy}}.
If the corresponding private key is given on the command line or in the configuration file,
then {\var{checkpointsprivkeyk}} is set to the corresponding integer.
(The key is for the uncompressed address, and this fact is hard-wired into the code.)

The value {\var{lastcheckpointnode}} is set to the most recent checkpoint, which is the
genesis node by default.

The hash table {\var{checkpoints}} associates header hashes with pairs $(h,\sigma)$
where $h$ is a block height and $\sigma$ is a signature of the hash by the checkpoint key.

\section{Best Node}

The current block chain is determined by finding the ``best node''
in the block tree, i.e., the node with the highest cumulative stake.
We sometimes allow the best node to be awaiting validation and sometimes require the best node
to be fully validated.

The variable {\var{bestnode}} is set to the current best node.
This gets reset when a new block is staked, if a new header is received
or if Qeditas gives up waiting for a node to be validated.
In the last case, we may want to find the best validated node.
The value of {\var{bestnode}} should never be updated directly,
but instead should be updated by calling {\func{update\_bestnode}}.
Given a node $n$, {\func{update\_bestnode}} sets {\var{bestnode}} to $n$,
sets {\var{netblkh}} to the current block height (technically, the
height of the next block to be found)
and, if the private key for signing checkpoints is known, the block which
is now 36 blocks deep is signed as a checkpoint,
{\var{lastcheckpointnode}} is updated
and the checkpoint is broadcast (as inventory) to all peers.

The function {\func{initblocktree}} creates a genesis block tree node
and initializes the values
{\var{genesisblocktreenode}},
{\var{lastcheckpointnode}}
and {\var{bestnode}} accordingly.
Then {\func{init\_headers}} is called to traverse all known block headers from
the block header database {\module{DbBlockHeader}}
and create corresponding nodes in the block tree.

The function {\func{find\_best\_validated\_block\_from}}
finds the descendent node with validation status {\constr{ValidBlock}}
(and which is not blacklisted)
with the highest cumulative stake.
The best node and cumulative stake found so far are extra arguments to compare against.
The function {\func{find\_best\_validated\_block}} calls {\func{find\_best\_validated\_block\_from}}
starting from the current value of {\var{lastcheckpointnode}}
and calls {\func{update\_bestnode}} with the result.

The function {\func{print\_best\_node}} gives the hash of the last block header of the current best node (or endicates it is the genesis node if there is no previous header).

The function {\func{record\_recent\_staker}} climbs an indicated number of parents
from a given node and inserts a given address into the recent stakers of the node.
This information is then used by {\func{is\_recent\_staker}} to determine if
an address has staked on some descendent of a node (up to some depth).
The relevant depth is $6$, but the functions are recursive, so the depth arguments $i$ are generic.

\section{Other Local Data}

The hash table {\var{stxpool}} associates hash values with corresponding signed transactions
and {\var{published\_stx}} records which of these transactions have been published (broadcast to peers).

The hash table {\var{thytree}} associates hash values with the root of a corresponding theory tree {\type{ttree}}.
The function {\func{lookup\_thytree}} looks up the theory tree given an optional hash value, where the default value of {\val{None}} returns the empty theory tree {\val{None}}.
For the moment, Qeditas has only been tested with empty theory trees.

Similarly, there is a hash table {\var{sigtree}} which associates hash values with the root of a corresponding signature tree {\type{stree}}.
The function {\func{lookup\_sigtree}} looks up the signature tree given an optional hash value, where the default value of {\val{None}} returns the empty signature tree {\val{None}}.
Again, Qeditas has only been tested with empty signature trees.

\section{Networking Code}

Since the {\module{blocktree}} module has access to the current state of the block chain,
a function {\func{send\_inv}} is defined to collect the most recent inventory data
(transactions, block headers and block deltas) to send to new peers.
The networking code obtains access to the {\func{send\_inv}} function via the reference
{\var{send\_inv\_fn}} which is set to {\func{send\_inv}} in {\module{blocktree}}.

The functions {\func{publish\_stx}} and {\func{publish\_block}}
add transactions and blocks to the local database
and broadcast them as new inventory to peers.

In addition, most of the implemented message handlers for the networking code (see Chapter~\ref{chap:net})
are given in the {\module{blocktree}} module by associating functions with message types in
the hash table
{\var{msgtype\_handler}}.
Here are the message types handled in {\module{blocktree}}:
\begin{itemize}
\item {\constr{Inv}}: The payload of an inventory message should be a 32-bit integer $n$ indicating
how many inventory items there are followed by $n$ triples $(\tau,h,k)$
where $\tau$ is a message type (as a byte, indicating the kind of object in the inventory), $h$ is a block height (as a 64-bit integer)
and $k$ is a hash value (indicating the identifier of the object in the inventory).
The handler reads these triples and determines whether or not to request the corresponding objects.
Headers will be requested if they are new and those to be requested are collect into a list to be requested in batches of at most 255 headers (by {\func{req\_header\_batches}}).
Block deltas are requested if they are new and the corresponding header is waiting to be validated (as indicted by {\var{tovalidate}}).
If a signed transaction is new, then the corresponding unsigned transaction is requested.
If the transaction of a signed transaction is known but the signatures are not, the signatures are requested.
A checkpoint is requested unless it is a known checkpoint or would be an ancestor of the current checkpoint.
No other inventory is immediately requested.
All inventory is inserted into the {\field{rinv}} field of the connection state in case
we need to request something later.
\item {\constr{GetHeader}}: The payload contains a hash value.
If the corresponding header has already been sent to the node (as recorded in {\field{sentinv}}),
then it is not resent. Otherwise, the header is loaded from the database and sent to the peer in a {\constr{Headers}} message. If the header is not in the database, the message is ignored.
\item {\constr{GetHeaders}}: The payload contains a byte $n$ indicating the number of headers being requested, followed by $n$ hash values.
The corresponding headers which have not yet been sent are collected and send to the peer in a {\constr{Headers}} message.
\item {\constr{Headers}}: The payload contains a byte $n$ indicating the number of headers being sent, followed by $n$ hash values and block headers.
Each header must hash to the given hash value, otherwise this is logged and the connection is marked as banned by setting {\field{banned}} in the {\type{connstate}} to {\val{true}}.
Each new header is checked to be valid and, if so, processed by {\func{process\_new\_header\_ab}} (which assumes the given header is valid).
If a received header is not valid, this is logged and the connection is marked as banned by setting {\field{banned}} in the {\type{connstate}} to {\val{true}}.
\item {\constr{GetBlockdelta}}: The payload contains a hash value identifying a block.
If the corresponding block delta has not yet been sent to the peer, it is sent in a {\constr{Blockdelta}} message.
\item {\constr{Blockdelta}}: The payload contains a hash value and a block delta.
If the hash value is of a block for which the block delta is already known, then
the handler stops (before deserializing the block delta).
Otherwise, the node for the corresponding header is found in the block tree.
If there is no such node in the block tree, the message is ignored.
Assume there is such a node $n$.
We check its validation status.
The case we are interested in is when the validation status is {\constr{Waiting}}
but there is not yet a candidate {\type{blockdelta}} to validate.
In this case we find the parent node $p$ of $n$.
If the parent node is already known to correspond to a valid block (or there is no parent node),
then the block delta is deserialized
and the function {\func{validate\_block\_of\_node}} is called to check
that the new block is valid and update the validation status of the node.
(Assuming the block is valid, {\func{validate\_block\_of\_node}} also traverses children
which have block deltas waiting to be validated and tries to validate these.)
If the parent node has not yet been validated,
then the block delta is deserialized and stored (along with the connection state of the peer)
with the {\constr{Waiting}} status. In principle it will be validated after its ancestors
have been validated.
\item {\constr{GetTx}}: The payload is a hash value which should be a transaction id.
Unless the transaction has already been sent to the peer, the (unsigned) transaction
is looked up in {\var{stxpool}} or {\module{DbTx}} and sent as a {\constr{Tx}} message.
\item {\constr{Tx}}: The payload is a hash value $h$ followed by an unsigned transaction (of type {\type{tx}}).
If $h$ is the transaction id of a transaction in the {\module{DbTx}} database, we ignore it.
Likewise, if the transaction was not explicitly requested from the peer, we ignore it.
Otherwise, we deserialize the transaction and check that it hashes to $h$ and
put it into the {\module{DbTx}} database.
\item {\constr{GetTxSignatures}}: The payload is a hash value which should be a transaction id.
Unless the transaction signatures have already been sent to the peer, the transaction signatures
are looked up in {\var{stxpool}} or {\module{DbTxSignatures}} and sent as a {\constr{TxSignatures}} message.
\item {\constr{TxSignatures}}: The payload is a hash value $h$ and a collection of transaction signatures (see Chapter~\ref{chap:assetstx}).
If $h$ is the transaction id of a transaction in the {\module{DbTxSignatures}} database, we ignore the message.
Likewise, if the transaction signatures were not explicitly requested from the peer, we ignore the message.
Otherwise, the signatures are deserialized and checked to be complete and valid (for some block height).
Assuming the signatures are complete and valid, the signatures are saved in {\module{DbTxSignatures}}
and added to the transaction in {\var{stxpool}}.
\item {\constr{GetCheckpoint}}: The payload is a hash value, which should be the hash of a block header which has
been signed as a checkpoint.
If the checkpoint exists and has not previously been sent to the peer,
it is sent as a {\constr{Checkpoint}} message.
\item {\constr{Checkpoint}}: The payload is a hash value $h$ (identifying a block header), a block height $i$ and a signature $\sigma$.
The signature $\sigma$ is checked to be a valid signature of $h$ for the public key for checkpoints.
Assuming it is valid, the pair $(i,\sigma)$ is added to {\var{checkpoints}}.
If the hash value $h$ was previously blacklisted in {\module{DbBlacklist}}, the entry is removed from the blacklist.
Assume there is a node $n$ corresponding to $h$ in the block tree.
If the node $n$ is marked as blacklisted, this is reset to {\val{false}}
while all the siblings of $n$ (if there are any) are marked as blacklisted.
The node is set to have validation status {\constr{Valid}} (independent of whether a block delta was ever received).
If there is no node corresponding to $h$ in the block tree,
a node is simply created (with no parent).\footnote{This could cause confusing with the genesis node in some parts of the code, and should probably be handled differently.}
\end{itemize}

\section{Dumping the State}

The function {\func{dumpblocktreestate}} writes a large amount of information about
the current block tree state into a channel (presumably of an open text file).
This is for debugging purposes.


%%
\chapter{About the Author}

Dominic Widdows is a mathematician, computational linguist, and software engineer.
He has worked on differential geometry and Oxford, natural language processing at Stanford,
and many projects at MAYA Design, Google, Microsoft, Grab, and LivePerson, where he works
particularly on conversational AI and internationalization.

As an author, his work includes the book \href{https://www.amazon.com/-/dp/B00J4FZOOY/}{Geometry and Meaning},
and over 50 scientific papers, in areas including pure mathematics,
computer science, language processing, bioinformatics, information extraction, logistics, and quantum computing.
As a developer, his open source contributions include work on the
\href{https://github.com/sky-map-team/stardroid}{SkyMap Planetarium App},
the \href{https://github.com/semanticvectors}{SemanticVectors} project, and
\href{https://github.com/dwiddows/pilmaps}{PILMaps} for drawing maps.

Enthusiasm for scientific publishing and open source development combined to make this book.


\documentclass{report}
\usepackage{graphicx}

\usepackage{amsmath}
\DeclareMathOperator*{\argmax}{argmax}

\usepackage{hyperref}
\hypersetup{
    colorlinks=true,
    linkcolor=blue,
    filecolor=magenta,
    urlcolor=cyan,
}

\begin{document}

\title{Banana World Project Report}
\author{Denis Sergeev}

\section*{Problem definition}

The goal of the project was to create a RL Agent which is able to learn from scratch by interacting with the Tennis environment for a series of episodes. The episode is divided into discrete time intervals. After each time interval the the agent receives the environment state. The agent is able to perform some actions after each time interval which may change the environment state.

\subsection*{Environment description}

In this environment, two agents control rackets to bounce a ball over a net. If an agent hits the
ball over the net, it receives a reward of \(+0.1\). If an agent lets a ball hit the ground or hits the ball out of bounds, it receives a reward of -0.01. Thus, the goal of each agent is to keep the ball in play.

The observation space consists of 8 variables corresponding to the position and velocity of the
ball and racket. Each agent receives its own, local observation. Two continuous actions are available, corresponding to movement toward (or away from) the net, and jumping.

The task is episodic, and in order to solve the environment, the agents must get an average score of \(+0.5\) (over 100 consecutive episodes, after taking the maximum over both agents). Specifically,

\begin{itemize}
\item After each episode, we add up the rewards that each agent received (without discounting), to get a score
for each agent. This yields 2 (potentially different) scores. We then take the maximum of these 2 scores.
\item This yields a single score for each episode.
\end{itemize}

\section*{Solution}
\subsection*{Deep Deterministic Policy Gradient Algorithm}
\subsubsection*{Network}

Since the environment is not competitive but collaborative and both agents have same task and observe mirrored state of the environment, I decided to use simple DDPG algorithm and train both agents on single pair of actor/critic network, instead of training them separately.

The DDPG algorithm uses two neural networks, an actor and a critic. Each one has two copies, a local and a target one. The Actor is trained to give the best possible action, given the current state of the environment. \(s => a\). The Critic is trained to give an estimate of the reward that'd be obtained if an action A is applied at the state \(s\). \((s, a) => V\) Local networks are trained to get the "labels" given by the target.

Both networks consist of fully connected units with batch normalization and ReLu activation. The input to the actor network is the environment state, which goes through 3 fully connected layers. The final activation function is \(tanh\), since the action value should be in-between \(-1\) and \(1\).

The input to the critic is the environment state, which goes through the first fully connected layer. After batch normalization and ReLu activation it is concatenated with the action tensor and goes through the remaining 2 layers.

\subsection*{Training}
\subsubsection*{Loss function and optimizing method}

DDGP algorithm uses following loss function for critic network \ref{critic-loss}:
\begin{equation} \label{critic-loss}
L \equiv (R_t + \gamma Q(S_{t+1}, a_{t+1}; \theta'_t) - Q(S_t, a_t; \theta_t))^{2}
\end{equation}

Where \(Q(S_t, a_t; \theta_t)\) is output (value of action) of the critic network with internal parameters \(\theta_t\) given input \(S_t\) and \(a_t\). \(a_{t+1}\) is obtained from \(\pi(S_t; \eta'_t)\) (target actor) prediction with internal parameters \(\eta'_t\).

Actor loss function is following \ref{actor-loss}:
\begin{equation} \label{actor-loss}
L \equiv -(Q(S_t, \pi(S_t; \eta_t); \theta_t))
\end{equation}

Both loss functions used 2 sets of network parameters: target \(\theta'_t\), \(\eta'_t\) and local \(\theta_t\), \(\eta_t\). Target network parameters are updating through soft update \(\theta'_t = (1 - \tau) \theta'_t + \tau \theta_t\).
As optimizing method for both networks I used Adam with learning rate \(\alpha\).

\subsubsection*{Experience replay}
The input for training did not come directly from episodes. Instead I used experience replay technique (see \href{https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf}{Human-level control through deep reinforcement learning}).
Also I implemented \href{https://arxiv.org/pdf/1511.05952.pdf}{prioritized experience replay}. But I did not notice any improvement in training while using it instead of simple replay.

\subsubsection*{Exploratory noise}
In order to make the agent explore the environment DDPG algorithm adds noise in it actions during training. I used \href{https://en.wikipedia.org/wiki/Ornstein-Uhlenbeck_process}{Ornstein–Uhlenbeck process} for this purpose.

\section*{Results}

I came up using following training hyper parameters:
\begin{itemize}
	\item Buffer size: 10e6
	\item Batch size: 1024
	\item \(\gamma\): 0.9
	\item \(\tau\): 0.001
	\item \(\alpha_{actor}\): 0.0001
	\item \(\alpha_{critic}\): 0.001
\end{itemize}
They were used across all training experiments below.

\subsection*{Training DDPG 64x32 64x32}

DDPG actor: 64x32, critic 64x(32+31).

Episode 100.	Average Score: 0.01.	Time elapsed: 1:04.

Episode 200.	Average Score: 0.01.	Time elapsed: 2:21.

Episode 300.	Average Score: 0.00.	Time elapsed: 3:35.

Episode 400.	Average Score: 0.02.	Time elapsed: 5:01.

Episode 500.	Average Score: 0.03.	Time elapsed: 6:36.

Episode 600.	Average Score: 0.05.	Time elapsed: 8:34.

Episode 700.	Average Score: 0.06.	Time elapsed: 10:39.

Episode 800.	Average Score: 0.04.	Time elapsed: 12:38.

Episode 900.	Average Score: 0.06.	Time elapsed: 14:56.

Episode 1000.	Average Score: 0.11.	Time elapsed: 18:18.

Episode 1100.	Average Score: 0.09.	Time elapsed: 20:58.

Episode 1200.	Average Score: 0.09.	Time elapsed: 23:42.

Episode 1300.	Average Score: 0.11.	Time elapsed: 26:57.

Episode 1400.	Average Score: 0.10.	Time elapsed: 29:56.

Episode 1500.	Average Score: 0.11.	Time elapsed: 33:29.

Episode 1600.	Average Score: 0.30.	Time elapsed: 43:18.

Episode 1700.	Average Score: 0.40.	Time elapsed: 56:31.

Episode 1800.	Average Score: 0.47.	Time elapsed: 72:25.

Environment solved in 1722 episodes!	Average Score: 0.50.	Time elapsed: 76:52. See \ref{fig:DDPG_64x32}.

\begin{figure}
	\includegraphics[width=0.9\linewidth]{res/ddpg_64x32/score.png}
	\caption{DDPG 64x32: Rewards per episode}
	\label{fig:DDPG_64x32}
\end{figure}

\subsection*{Training DDPG 64x32 64x32 with prioritized replay}

DDPG actor: 64x32, critic 64x(32+31).

Episode 100.	Average Score: 0.02.	Time elapsed: 1:05.

Episode 200.	Average Score: 0.02.	Time elapsed: 2:33.

Episode 300.	Average Score: 0.02.	Time elapsed: 3:59.

Episode 400.	Average Score: 0.02.	Time elapsed: 5:24.

Episode 500.	Average Score: 0.04.	Time elapsed: 7:10.

Episode 600.	Average Score: 0.05.	Time elapsed: 9:21.

Episode 700.	Average Score: 0.09.	Time elapsed: 12:26.

Episode 800.	Average Score: 0.11.	Time elapsed: 15:57.

Episode 900.	Average Score: 0.11.	Time elapsed: 19:52.

Episode 1000.	Average Score: 0.17.	Time elapsed: 25:28.

Episode 1100.	Average Score: 0.19.	Time elapsed: 31:33.

Episode 1200.	Average Score: 0.22.	Time elapsed: 39:04.

Episode 1300.	Average Score: 0.32.	Time elapsed: 50:28.

Episode 1400.	Average Score: 0.29.	Time elapsed: 60:57.

Episode 1500.	Average Score: 0.25.	Time elapsed: 69:48.

Episode 1600.	Average Score: 0.19.	Time elapsed: 77:12.

Episode 1700.	Average Score: 0.21.	Time elapsed: 84:53.

Episode 1800.	Average Score: 0.30.	Time elapsed: 96:14.

Episode 1900.	Average Score: 0.27.	Time elapsed: 106:36.

Episode 2000.	Average Score: 0.34.	Time elapsed: 119:37.

Episode 2100.	Average Score: 0.26.	Time elapsed: 130:05.

Episode 2200.	Average Score: 0.32.	Time elapsed: 143:45.

Episode 2300.	Average Score: 0.28.	Time elapsed: 155:33.

Episode 2400.	Average Score: 0.35.	Time elapsed: 170:28.

Episode 2500.	Average Score: 0.33.	Time elapsed: 185:15.

Episode 2600.	Average Score: 0.32.	Time elapsed: 199:51.

Episode 2700.	Average Score: 0.39.	Time elapsed: 218:28.

Episode 2800.	Average Score: 0.35.	Time elapsed: 235:13.

Episode 2900.	Average Score: 0.36.	Time elapsed: 252:47.

Environment solved in 2885 episodes!	Average Score: 0.51.	Time elapsed: 275:05 See \ref{fig:DDPG_64x32_pri}.

\begin{figure}
	\includegraphics[width=0.9\linewidth]{res/ddpg_64x32_2/score.png}
	\caption{DDPG 64x32 with prioritized replay: Rewards per episode}
	\label{fig:DDPG_64x32_pri}
\end{figure}

\section*{Areas of improvement}

One possible improvement could be implementing \href{https://arxiv.org/pdf/1707.06347.pdf}{PPO algorithm} instead of DDPG. It should be working more stably.

Another one improvement could be staying with DDPG and play with hyperparameters to achieve faster training.

\section*{Conclusion}

From the above results we can summarize, that the Tennis environment is not a real multi-agent environment and it is possible to solve it just using regular DDPG algorithm. However the training process is very unstable. We can see this on the scores plot \ref{fig:DDPG_64x32}.

For a competitive and non-symmetric environment I would implement \href{https://arxiv.org/pdf/1706.02275.pdf}{MADDPG algorithm}.

\end{document}


% -*- root: ../main.tex -*-
\subsection{Formal Representation of a Program}

This \gls{SPL} and its formal representation is the language chosen to write the programs to be formally verified.
%
It has been chosen by \citep{thesisAle} because its simplicity and expressiveness in order to write concurrent programs.
%
Because its simplicity it is a great option to do formal verification with it.

\subsubsection{Preliminaries (Notation, Definition)}

The semantics of programs are given as a sequence of states.
%
The \pc\; variable (\concept{Program\IS counter}) has the information of the line to execute next.
%
Additionally, there are the steps of the program. Each step modifies the state of the program by modifying the values of the variables including the \pc.
%
A step can be easily expressed in \gls{FOL} using \concept{post-state} variables, which are the new values the variables will have after the execution of the line.
%
The formula gathering the information of the execution of a line, using the \pc and \pc' (the post-state \pc) and all the other variables is called \concept[Transition relation]{transition relation}.

\subsubsection{Possible instructions}
%
In order to express correctly the transition relation corresponding to certain line, we need to know how to translate program statements into \gls{FOL}.
%
As we are going to work with programs used by more than one thread, we need one program counter for each thread executing.
%
We parametrize the program counter by thread identifiers.
%
That is, we define the \textbf{program counter} as a \textbf{function} that given a thread, returns its program counter.
%
We could have introduced one variable per thread.
%
It would be an equivalent formulation.

We proceed to define in general terms how to build transition relation for a \gls{SPL} statement.
%
We only show two types of statements. All the others statements are defined in a similar way.
%
A complete definition can be found in \cite{SPL}.
%
For these definitions we use the letter $T$ to refer a thread.
%

\begin{description}
\item [Assignments:]
		The transition relation for a variable assignment consists of the
		update of the program counter for the running thread and the
		corresponding modification to the variable being assigned.
%
			\[
			\begin{array}[t]{p{8em}@{\hspace{6em}}p{\longtablesize}}
				\hline
				Statement & Transition relation \\ \hline\hline
				$\begin{array}[t]{l@{\hspace{0.3em}}c@{\hspace{1em}}l}
					l_1 & : & \mathtt{v := 2} \\
					l_2 & : & \mathtt{\cdots}
				\end{array}$
				&
				$\begin{array}[t]{ll}
					 \pc(T) = l_1 \andcond
					 \pc'(T) = l_2 \andcond
					 \mathtt{v}' = 2
				 \end{array}$ \\
				 \hline
			\end{array}
			\]

%	\item [Conditionals:]
%		We present now the two possible kinds of conditional statements in
%		SPL.
%
%		In the first case, if condition $c$ does not hold, the execution
%		proceeds from the statement following the end of the conditional.
%
%		\[
%       \begin{array}[t]{p{8em}@{\hspace{6em}}p{\longtablesize}}
%				\hline
%				Statement & Transition relation \\ \hline\hline
%				$\begin{array}[t]{l@{\hspace{0.3em}}c@{\hspace{1em}}l}
%					\ell_1 & : & \mathtt{\textbf{if } c \textbf{ then }} \\ \\
%					\ell_2 & : & \mathtt{\cdots} \\
%						&& \mathtt{\vdots} \\
%					\ell_n & : & \mathtt{\textbf{end if}} \\
%					\ell_{n+1} & : & \mathtt{\cdots}
%				\end{array}$
%				&
%				$\begin{array}[t]{ll}
%					(\pc(T) = \ell_1 \andcond \;\;\mathtt{c} \andcond \pc'(T) = \ell_2) \; \Or \\
%					(\pc(T) = \ell_1 \andcond \lnot \mathtt{c} \andcond \pc'(T) = \ell_{n+1})
%				 \end{array}$ \\ \hline
%			 \end{array}
%		 \]
%
%		In the second case, if condition $c$ does not hold, the execution
%		continues at the first statement in the \textbf{else} section of the
%		conditional statement.
%%
%		\[
%				\begin{array}[t]{p{8em}@{\hspace{6em}}p{\longtablesize}}
%				\hline
%				Statement & Transition relation \\ \hline\hline
%				$\begin{array}[t]{l@{\hspace{0.3em}}c@{\hspace{1em}}l}
%					\ell_1 & : & \mathtt{\textbf{if } c \textbf{ then }} \\ \\
%					\ell_2 & : & \mathtt{\cdots} \\
%					\mathtt{\vdots} \\
%					\ell_n & : & \mathtt{\textbf{else}} \\
%					\ell_{n+1} & : & \mathtt{\cdots} \\
%					\mathtt{\vdots} \\
%					\ell_m & : & \mathtt{\textbf{end if}} \\
%					\ell_{m+1} & : & \mathtt{\cdots}
%				\end{array}$
%				&
%				$\begin{array}[t]{ll}
%					(\pc(T) = \ell_1 \andcond \;\;\mathtt{c} \andcond \pc'(T) = \ell_2) \; \Or \\
%					(\pc(T) = \ell_1 \andcond \lnot \mathtt{c} \andcond \pc'(T) = \ell_{n+1})
%						& \text{for line } \ell_1 \\ \\ \phantom{\vdots} \\
%
%						\pc(T) = \ell_n \andcond \pc'(T) = \ell_{m+1} & \text{for line } \ell_n
%				 \end{array}$ \\ \hline
%			\end{array}
%		\]
%%
	\item [Loops:]
		We consider the only loop statement available in SPL which executes
	the statements in the body as long as the loop condition holds.
%
		\[
				\begin{array}[t]{p{8em}@{\hspace{6em}}p{\longtablesize}}
				\hline
				Statement & Transition relation \\ \hline\hline
				$\begin{array}[t]{l@{\hspace{0.3em}}c@{\hspace{1em}}l}
					\ell_1 & : & \mathtt{\textbf{while } c \textbf{ do }} \\
					\ell_2 & : & \mathtt{\cdots} \\
					\mathtt{\vdots} \\
					\ell_n & : & \mathtt{\textbf{end while}} \\
					\ell_{n+1} & : & \mathtt{\cdots}
				\end{array}$
				&
				$\begin{array}[t]{ll}
						(\pc(T) = \ell_1 \andcond \;\;\texttt{c} \andcond \pc'(T) = \ell_2) \;
						\Or \\
						(\pc(T) = \ell_1 \andcond \lnot \texttt{c} \andcond \pc'(T) =
					\ell_{n+1})
					& \text{for line $\ell_1$} \\ \\
					\pc(T) = \ell_n \andcond \pc'(T) = \ell_1 &
						\text{for line $\ell_n$}
				 \end{array}$ \\ \hline
			 \end{array}
		\]
%
%			\item [Non deterministic choice:]
%		The transition relation for the non-deterministic choice statement can
%		be expressed as follows:
%
%		\[
%			\begin{array}[t]{p{8em}@{\hspace{6em}}p{\longtablesize}}
%				\hline
%				Statement & Transition relation \\ \hline\hline
%				$\begin{array}[t]{l@{\hspace{0.3em}}c@{\hspace{1em}}l}
%					\ell_1 & : & \mathtt{\Nondet} \\
%					\ell_2 & : & \mathtt{\;\;\;\;\;\; \cdots} \\
%					\ell_3 & : & \mathtt{\textbf{or } \; \cdots} \\
%					\mathtt{\vdots} \\
%					\ell_n & : & \mathtt{\textbf{or } \; \cdots} \\
%					\ell_{n+1} & : & \mathtt{\NondetEnd} \\
%				\end{array}$
%				&
%				$\begin{array}[t]{ll}
%					\pc(T) = \ell_1 \andcond
%					\bigvee\limits_{i = 2..n} \pc'(T) = \ell_i
%				 \end{array}$ \\ \hline
%			 \end{array}
%		\]
%
%	\item [Lock and unlock:]
%	Even though these are not SPL statements, as they will be widely used, it is necessary to define theirs transition relations.
%%
%		\[
%			\begin{array}[t]{p{8em}@{\hspace{6em}}p{\longtablesize}}
%				\hline
%				Statement & Transition relation \\ \hline\hline
%				$\begin{array}[t]{l@{\hspace{0.3em}}c@{\hspace{1em}}l}
%					\ell_1 & : & \mathtt{\fLock(l)} \\
%					\ell_2 & : & \mathtt{\cdots}
%				\end{array}$
%				&
%				$\begin{array}[t]{ll}
%					\pc(T) = \ell_1 \andcond
%						\mathtt{l} = \oslash \andcond
%						\mathtt{l}' = T \andcond \pc'(T) = \ell_2
%				 \end{array}$ \\ \hline\hline
%				$\begin{array}[t]{l@{\hspace{0.3em}}c@{\hspace{1em}}l}
%					\ell_1 & : & \mathtt{\fUnlock(l)} \\
%					\ell_2 & : & \mathtt{\cdots}
%				\end{array}$
%				&
%				$\begin{array}[t]{ll}
%					\pc(T) = \ell_1 \andcond
%						\mathtt{l}' = \oslash \andcond \pc'(T) = \ell_2
%				 \end{array}$ \\ \hline
%			 \end{array}
%		\]
%
\end{description}