%# -*- coding: utf-8-unix -*-
%%==================================================
%% chapter06.tex for SJTU Master Thesis
%%==================================================

%\bibliographystyle{sjtu2}%[此处用于每章都生产参考文献]
\chapter{MULTI-MODAL DEEP AUTO-ENCODER BASED EMOTION RECOGNITION}
\label{chap:english_summary}

	Emotion plays an important part in the daily life of everyone. Though we could recognize emotions from facial expressions, voice or body movement, with the development of technology and equipment which could collect signals direct from human brains, we are able to recognize emotions with the help of EEG signals. Compared with the original method, EEG signals have a rather direct connection with emotions. Therefore, EEG-based emotion recognition is now the state-of-the-art research area which got the attention of lots of researchers. However, the performance and accuracy are limited if we can access only the single EEG modal.  Now we try to make use of not only the EEG data but the eye motion data, we believe these two data sources can complement each other and thus we can have a better result if we create an algorithm to use them simultaneously. As a result, our algorithm is based on multimodal deep autoencoder, which will be explained below:

	A deep autoencoder is composed of two, symmetrical deep-belief networks that typically have four or five shallow layers representing the encoding half of the net, and second set of four or five layers that make up the decoding half.

	The layers are restricted Boltzmann machines, the building blocks of deep-belief networks, with several peculiarities that we’ll discuss below. Here’s a simplified schema of a deep autoencoder’s structure, which will be  explained below. a deep autoencoder would use binary transformations after each RBM. Deep autoencoders can also be used for other types of datasets with real-valued data, on which you would use Gaussian rectified transformations for the RBMs instead.

	For the encoding process, If, say, the input fed to the network is 784 pixels, then the first layer of the deep autoencoder should have 1000 parameters; i.e. slightly larger.
This may seem counterintuitive, because having more parameters than input is a good way to overfit a neural network.

	In this case, expanding the parameters, and in a sense expanding the features of the input itself, will make the eventual decoding of the autoencoded data possible.
This is due to the representational capacity of sigmoid-belief units, a form of transformation used with each layer. Sigmoid belief units can’t represent as much as information and variance as real-valued data. The expanded first layer is a way of compensating for that.

	The layers will be 1000, 500, 250, 100 nodes wide, respectively, until the end, where the net produces a vector 30 numbers long. This 30-number vector is the last layer of the first half of the deep autoencoder, the pretraining half, and it is the product of a normal RBM, rather than an classification output layer such as Softmax or logistic regression, as you would normally see at the end of a deep-belief network.
For the decoding process, those 30 numbers are an encoded version of the original input data. The second half of a deep autoencoder actually learns how to decode the condensed vector, which becomes the input as it makes its way back.

	The decoding half of a deep autoencoder is a feed-forward net with layers 100, 250, 500 and 1000 nodes wide, respectively. Those layers initially have the same weights as their counterparts in the pretraining net, except that the weights are transposed; i.e. they are not initialized randomly.).

	The decoding half of a deep autoencoder is the part that learns to reconstruct the image. It does so with a second feed-forward net which also conducts back propagation. The back propagation happens through reconstruction entropy.

Deep autoencoder can be used in many cases, as lisited below:
\begin{itemize}
\item Image search

	deep autoencoders are capable of compressing images into 30-number vectors. Image search, therefore, becomes a matter of uploading an image, which the search engine will then compress to 30 numbers, and compare that vector to all the others in its index.Vectors containing similar numbers will be returned for the search query, and translated into their matching image.
\item Data compression

	A more general case of image compression is data compression. Deep autoencoders are useful for semantic hashing, as discussed in this paper by Geoff Hinton
\item .Topic modeling and information retrieval

	Deep autoencoders are useful in topic modeling, or statistically modeling abstract topics that are distributed across a collection of documents.

	In brief, each document in a collection is converted to a Bag-of-Words (i.e. a set of word counts) and those word counts are scaled to decimals between 0 and 1, which may be thought of as the probability of a word occurring in the doc.

	The scaled word counts are then fed into a deep-belief network, a stack of restricted Boltzmann machines, which themselves are just a subset of feedforward-backprop autoencoders. Those deep-belief networks, or DBNs, compress each document to a set of 10 numbers through a series of sigmoid transforms that map it onto the feature space.

	Each document’s number set, or vector, is then introduced to the same vector space, and its distance from every other document-vector measured. Roughly speaking, nearby document-vectors fall under the same topic.

	For example, one document could be the “question” and others could be the “answers,” a match the software would make using vector-space measurements.
\end{itemize}

	Deep networks have already been successfully applied to unsupervised feature learning for single modalities such as text, images or audio. In this work, we propose a novel application of deep networks to learn features over EEG data and eye motion data. We present multiple algorithm and a series of tasks for multimodal learning and illustrate how to train deep networks that learn features to solve these tasks. In particular, we demonstrate cross modality feature learning, where better features for one modality such as EEG data can be learned if multiple modalities such as EEG data and eye motion data are present at feature learning time.

	This work focus on the emotion recognition tasks based on the EEG data, to improve the current result of our emotion recognition research, we have applied the multimodal deep learning algorithm containing deep autoencoder to this thesis. With the related multimodal deep learning algorithm created by the researchers, we have tried different activation function and extract feature from different layers. It turned out to be that the rectified linear unit has a better performance than sigmoid function unit while feature from different layer also differs from each other.

	Concretely,

	However, we see that we are a long way to go before we can make this algorithm really brilliant, so that it can have advantages in both speed and accuracy. So far, what we can see to be done in the future are listed below:
\begin{enumerate}
\item As we tried many libraries before we found this suitable one, we spent a lot of time on it so that we have not enough time to tune the parameters including the number of the hidden units, the learning rate or the core function of our SVM classifier. We believe the result can get better if there is more time for tuning.
\item Our lab usually list the results of five bands(Alpha, Beta, Gamma and so on) and the result of the combination of all five bands respectively for comparison. However, as we don’t have enough time for this work, we just present and analyze the result of the combination of all five bands. We can analyze the five bands respectively to find the best band for this deep autoencoder algorithm.
\item Now the final result represents for the output of multimodal deep autoencoder, and we use this result to compare with the feature that has not been learned or trained by any network, and it is not fair.  In the future, we need to use this algorithm to extract the feature of only EEG data and eye motion data respectively, then use the results to compare with the result of multimodal autoencoder to illustrate the difference between multi-modal and single-modal. Also we need to concatenate the feature of two modals and compare with the result of our algorithm to illustrate the importance of the deep network. Use the method of controlled variable, we can state the advantages of our algorithm clearly.
\item To show that the feature we learned is useful intuitively, we can cluster the learned feature and original feature using K-means cluster algorithm and see the result of the cluster algorithm.
\item Now we just use SVM to classify the feature we learned, we can try different classifier in the future such as multiple perceptron.
\item We can try shared representation task, in which we will use data of one modal to train and data of another modal to test. In this case, the classification can work because the classifier has learned the shared representation of these two modal.
\end{enumerate}

	Finally, the result we get is compatible with the one we anticipated, naturally they are both better than the baseline algorithm. Moreover, after testing so many times, the library we use for the work has a decent speed and can help with the future work.

Furthermore, we expect to show how to learn a shared representation between modalities and evaluate it on a unique task in the future, where the classifier is trained with audio-only data but tested with video-only data and vice-versa.


\newpage
\section{Discussion and Conclusion}

This work described the simulation of an experiment conducted in ref.~\cite{heeg}. Graphene was used as a Raman active material for surface-enhanced Raman spectroscopy with a gold nanodimer as a plasmonic nanostructure. Because the laser spot size is much bigger than the area of the dimer cavity, the calculated local enhancement cannot be directly compared to the experimental enhancement.

It was the scope of this work to draw a comparison between the local simulated enhancement and the total measured enhancement. The shape of the graphene layer, that was suspended on top of the gold nanodimer, was simulated with a 3D modelling software. The local electromagnetic enhancement was calculated with the finite-difference time-domain method, using the commercial software Lumerical FDTD solutions. By averaging the local electromagnetic enhancement at the graphene layer over the size of the laser spot the total enhancement was calculated.

The calculated average enhancement within the simulated area was $60.2$  both for a \SI{2}{nm} corner radius and a \SI{5}{nm} corner radius (both in the plane of graphene) and $59.2$ for the \SI{5}{nm} corner radius for a planar slice at a height of $z=\SI{40}{nm}$. Taking the laser geometry into account the total enhancement was calculated as $38.1$ for the \SI{2}{nm} corner radius, $37.8$ for the \SI{5}{nm} corner radius and $37.2$ for the planar slice. Using the spatial coherence theory of ref.~\cite{coherence} the enhancement of the \SI{5}{nm} planar slice decreased to $23.0$. The corner radius of the gold nanostructure only had a minor influence on the total enhancement, although it led to significant changes in $xz$ contour plots of the near-field.

The simulated local electric field enhancement nicely reproduced the simulation of ref.~\cite{heeg}. Comparing the resulting total enhancement for an accurately modeled geometry of the graphene layer to a slice at the height of \SI{40}{nm} confirmed the approximation used in ref.~\cite{heeg}. Taking spatial coherence effects into account, the resulting total enhancement of $23.0$ reproduced the measured value of $12.8$ in ref.~\cite{heeg} within a factor of two.

Further simulations should include even larger corner radii to find an experimental setup that results in the same near-field at the edges of the nanostructure. Additionally, the presented method can be adapted for other Raman-active materials suspended on nanostructures for
surface-enhanced Raman spectroscopy.


\SetPicSubDir{ch-Main}
\SetExpSubDir{ch-Main}

\chapter{Main Text}
\label{ch:main}
\vspace{2em}

The main contribution of this work is to test a learning-based approach to smoothing the motion profile of a DRL model. We do this in the following sequence:
\begin{enumerate}
\item Introduce smooth motion profiles
\item Present the network architecture used to apply this learning-based approach
\item Discuss the empirical findings of this learning-based approach as well as compare it with a non-learning based approach
\end{enumerate}

\section{Smooth Motion Profile}
In map-less navigation, we deal with point-to-point motion. That is, the goal of the robot is to move from its starting position and reach some target end position while avoiding obstacles. In terms of motion, point-to-point means that from a stop (zero velocity), the load is accelerated to a constant velocity, and then decelerated such that the final acceleration, and velocity, are zero at the moment the load arrives at the programmed destination. However, achieveing fast motions without residual vibration is a challenge that pervades many applications of point-to-point motion~\cite{meckl_optimized_1998}, including deep reinforcement learning for map-less navigation.

To perform point-to-point motion with minimal vibration, previous works have focused on optimization of the motion profile itself~\cite{meckl_optimized_1998}. For example, ~\autoref{main:fig:motion_profile_1} shows a comparison between a typical trapezoidal motion profile and an (improved) S-curve motion profile.

\begin{figure}[H]
  \centering
  \includegraphics[width=.9\linewidth]{\Pic{png}{motion_profile_1}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Comparison of Trapezoidal and S-Curve Motion Profiles. Image from Meckl and Arestides, 1998~\cite{meckl_optimized_1998}}
  \label{main:fig:motion_profile_1}
\end{figure}

As can be seen from ~\autoref{main:fig:motion_profile_1}, smoothing a motion profile involves jerk reduction (or some manipulation of the jerk profile). Jerk is the time-derivative of acceleration, that is:
\begin{equation}
j(t) = \frac{da(t)}{dt} = \frac{d^2 v(t)}{dt^2}
\end{equation}

\section{Proposed Approach}
\subsection{Three Different Models}
\label{main:sec:3_models}
In this work, we compare the motion profiles of the following three models:
\begin{enumerate}
\item \textbf{Normal Model}: The default end-to-end neural network trained with deep reinforcement learning to perform map-less navigation
\item \textbf{Model with Velocity Smoother}: The same network as Normal Model but trained with a simple velocity smoother
\item \textbf{Model with Jerk-Learning}: A network very similar to Normal Model but with an added reward function that penalizes the robot for the jerk generated. The inputs of this model are also slightly different (for the reward function to work properly).
\end{enumerate}

\subsection{State}
\label{main:sec:state}
\textit{Normal Model} and \textit{Model with Jerk-Learning} have different features included in their inputs (states) but they both have the same output (action).

At a certain time step $t$, the state of the agent (robot) trained with \textit{Normal Model} is described by:
\begin{itemize}
\item a stack of three 270-degree sparse laser scans y ($y_{t}, y_{t-1}, y_{t-2}$)
\item the current linear and angular velocity of the robot ($v_{t}, \omega_{t}$)
\item x and y components of the robot's displacement from its target position ($d_{x,t}, d_{y,t}$)
\end{itemize}

In the \textit{Model with Jerk-Learning}, the acceleration and jerk of the robot are included in the input layer. The idea is to allow the robot to use information about its motion profile to reduce the jerk generated (in combination with the reward function to incentivize this behaviour). The inputs for this model are:
\begin{itemize}
\item a stack of three 270-degree sparse laser scans y ($y_{t}, y_{t-1}, y_{t-2}$)
\item a stack of linear and angular velocities of the robot ($v_{t}, \omega_{t}, v_{t-1}, \omega_{t-1}, v_{t-2}, \omega_{t-2}$)
\item a stack of linear and angular accelerations of the robot ($accel_{t}, \alpha_{t}, accel_{t-1}, \alpha_{t-1}, accel_{t-2}, \alpha_{t-2}$)
\item a stack of linear and angular jerks of the robot ($j_{t}, \zeta_{t}, j_{t-1}, \zeta_{t-1}, j_{t-2}, \zeta_{t-2}$)
\item x and y components of the robot's displacement from its target position ($d_{x,t}, d_{y,t}$)
\end{itemize}

\subsection{Action}
For all 3 models, the output (action) is a linear velocity ($v_t$) and an angular velocity ($\omega_{t}$). However, the action for the \textit{Model with Velocity Smoother} is slightly different as it is constrained by a simple velocity smoother. In this work, we use the following algorithm for simple velocity smoothing:

\begin{algorithm}[H]
\caption{Simple Velocity Smoother}
\label{main:simple_v_smoother}
\If{Target Velocity > Control Velocity}{Control Velocity = $\min$(Target Velocity, Control Velocity + $v_c$)}
\ElseIf{Target Velocity < Control Velocity}{Control Velocity = $\max$(Target Velocity, Control Velocity - $v_c$)}
\Else{Control Velocity = Target Velocity}
\end{algorithm}
Where:
\begin{itemize}
\item $v_c$ is some constant velocity value
\end{itemize}

Essentially, ~\autoref{main:simple_v_smoother} is an acceleration limiting approach to motion control where a maximum acceleration is assumed. This generates a trapezoidal motion profile similar to the one found in ~\autoref{main:fig:motion_profile_1}.

\subsection{Simulation Setup}
To enable the DRL agent to learn from trial and error, we use a simple robot simulation environment called Stage. Stage provides a virtual world populated by mobile robots and sensors, along with various objects for the robots to sense and manipulate. We use Stage in ROS (Robot Operating System) to train our DRL agent~\cite{gerkey_playerstage_nodate}.

\subsection{Reward Function}
The reward function, $r(s, a)$, serves as a training signal to encourage or discourage behaviours in the context of a desired task~\cite{zhu_safe_2019}. Thus, the features chosen for the reward function must capture the relevant objectives of the task. In this work, the following reward function was used for the \textit{Normal Model}:
\begin{equation}
Reward, r_{t} =
\begin{cases}
  -5& \text{if robot crashes},\\
  5& \text{if robot reaches the goal},\\
  \gamma{\bigg((d_{t-1} - d_{t})\Delta{t} - C\bigg)}& \text{otherwise}.
  \\
\end{cases}
\end{equation}
Where:
\begin{itemize}
\item $\gamma$ acts as a discount factor (for infinite-horizon discounted return)
\item $d_{t-1}$ and $d_{t}$ indicate the distance between the robot and its target at two consecutive time stamps
\item $\Delta{t}$ represents the time for each step
\item C is a constant used as a time penalty.
\end{itemize}

Note that the policy learnt with this reward function may be \textit{sub}-optimal. This is due to the fact that the agent is driven by something else other than reaching the target in the shortest time possible. Ideally, a sparse reward function would be better as it would only reward the robot when it actually reaches the target; however, a sparse reward function is challenging for random exploration.

For the \textit{Model with Jerk-Learning}, a new feature which penalizes the robot for jerky movements is added to the reward function. We have chosen to use a dense penalty function for the new feature as it would alleviate the challenges of learning the appropriate motion profile to minimize jerk:
\begin{equation}
Reward, r_{t} =
\begin{cases}
  -5& \text{if robot crashes},\\
  5& \text{if robot reaches the goal},\\
  \gamma{\bigg((d_{t-1} - d_{t})\Delta{t} - C\bigg)} -\\
  \frac{\gamma}{25} \Bigg(\bigg(\frac{j_{t}}{\max{j_{t}}}\bigg)^{2} + \bigg(\frac{\zeta_{t}}{\max{\zeta_{t}}}\bigg)^2\Bigg)& \text{otherwise}.
  \\
\end{cases}
\end{equation}
Where:
\begin{itemize}
\item $j_t$ is linear jerk at time $t$
\item $\max{j_t}$ is the theoretical maximum linear jerk for each state transition
\item $\zeta_{t}$ is angular jerk at time $t$
\item $\max{\zeta_{t}}$ is the theoretical maximum angular jerk for each state transition
\end{itemize}

\subsection{Network Architecture}
The network architecture we use is largely similar to Xie et. al's Assisted DDPG network~\cite{xie_learning_2018}. The difference, however, is that we have discarded the convolutional neural network (CNN) layers and used a sparser laser beam to allow the agent to learn the policy faster. We reiterate: the goal here is not to produce a state-of-the-art model for map-less navigation, but to empirically study a learning-based approach to jerk reduction. ~\autoref{main:fig:network_architecture} shows the network architecture of the \textit{Normal Model}.

\begin{figure}[H]
  \centering
  \includegraphics[width=.9\linewidth]{\Pic{png}{network_architecture}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Network architecture. Network layers are demonstrated by rectangles. Orange arrows indicate the connectivity between network layers and some other components, e.g. input state and the output of the simple controller. Blue arrows indicate the operation of concatenation (i.e. tf.concat). The final action is selected based on the Q-value predicted by the critic-DQN.}
  \label{main:fig:network_architecture}
\end{figure}

The network architecture of ~\autoref{main:fig:network_architecture} consists of two parts:
\begin{enumerate}
\item \textbf{Policy network and assistive controller (red)}: The policy network consists of a fully-connected layers which estimates the optimal linear and angular velocities for the robot based only on the input state (i.e., laser scans, current speed, and target position in the local frame). This is a very important feature of DDPG as sampling actions from a policy is computationally less expensive than exhaustively evaluating as many discrete actions as possible (since our action space is continuous). Note that the activation functions of the outputs of the policy are sigmoid (linear velocity) and tanh (angular velocity), respectively. The assisive controller is an alternative, switchable policy used to speed up training of the model. It is only used to speed up training, and is discarded eventually.
\item \textbf{Augmented critic network (green)}: The critic-DQN is also constructed with fully-connected layers. It has two branches: one is the critic branch where the action is concatenated into the second layer; the other is the DQN branch where (similar to Xie et. al~\cite{xie_learning_2018}) we apply dueling and double network architecture to speed up training and avoid overestimation. Note that there is no nonlinear activation for its output layers.
\end{enumerate}

The network architecture of the \textit{Model with Velocity Smoother} is exactly the same as the \textit{Normal Model} but with a velocity smoother at the end. It is shown in ~\autoref{main:fig:network_architecture_jerk}:

\begin{figure}[H]
  \centering
  \includegraphics[width=.9\linewidth]{\Pic{png}{network_architecture_vsmooth}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Network architecture of Model with Velocity Smoother. Output layer has a velocity smoother.}
  \label{main:fig:network_architecture_vsmooth}
\end{figure}

The network architecture of the \textit{Model with Jerk-Learning} is slightly different in terms of its inputs (as mentioned in ~\autoref{main:sec:state}). It is shown in ~\autoref{main:fig:network_architecture_jerk}:

\begin{figure}[H]
  \centering
  \includegraphics[width=.9\linewidth]{\Pic{png}{network_architecture_jerk}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Network architecture of Model with Jerk-Learning. Input layer now includes stacks of velocities, accelerations, and jerks of the robot.}
  \label{main:fig:network_architecture_jerk}
\end{figure}

\subsection{Training Methodology}
The training procedure of both networks is according to ~\autoref{main:training_procedure} ~\cite{xie_learning_2018}:
\begin{algorithm}[H]
\caption{Assisted DDPG}
\label{main:training_procedure}
Initialize $A(s, a|\theta^A), Q(s, \sigma|\theta^Q),$ and $\pi(s, \theta^\pi)$\;
Initialize target network $\theta^{A'}, \theta^{Q'},$ and $\theta^{\pi'}$\;
Initialize replay buffer $R$ and exploration noise $\epsilon$\;
\While{Episode is not terminal}{
  Reset the environment\;
  Obtain the initial observation\;
  \For{step = 1}{
    Infer switching $[Q_{policy}, Q_P] = Q(s_t, \sigma|\theta^Q)$
    $\sigma_t = \argmax([Q_{policy}, Q_P])$
    \If{$\sigma_t == 1$}{Sample policy action $a_t = \pi(s_t|\theta^\pi) + \epsilon$}
    \Else{Sample action $a_t$ from external controller}
    Execute $a_t$ and obtain $r_t, s_{t+1}$\;
    Store transition ($s_t, a_t, r_t, s_{t+1}, \sigma_t$) in $R$\;
    Sample N transitions ($s_i, a_i, r_i, s_{i+1}, \sigma_i$) in $R$\;
    Optimize critic-DQN by minimizing loss function\;
    Update policy\;
    Update target networks\;
  }}
\end{algorithm}

All 3 models were trained in 2 different Stage simulation worlds~\autoref{appsec:simple_world},~\autoref{appsec:complex_world}. In each simulation world, the networks were trained until the average performance reached a plateau (i.e. when gradient of average performance with respect to time is approximately zero).

% Here, average performance of the network is obtained according to~\autoref{main:eq:average_performance}. To our knowledge, ~\autoref{main:eq:average_performance} has not been explicitly mentioned in the literature; however, our understanding is that this is what is typically meant by average performance in the mapless navigation literature ~\cite{xie_learning_2018}.
% \begin{equation}
% \text{Average Performance} = \underset{Ep \sim (100 Eps)}{\mathrm E}\bigg[\underset{(s, a) \sim Ep}{\mathrm E} \big(\underset{a \sim \pi}{\max} Q^{\pi}(s, a)\big)\bigg]
% \label{main:eq:average_performance}
% \end{equation}

\section{Experiment Results}
In this section, we discuss and compare the results of training the 3 different models. Visualizing these results, however, remains a difficulty. Consider ~\autoref{main:visualization_difficulty_1}:

\begin{figure}[H]
  \centering
  \includegraphics[width=.9\linewidth]{\Pic{png}{paper_Simple World_Performance_Raw_line}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Graph of Performance vs Training Episodes (Raw).}
  \label{main:visualization_difficulty_1}
\end{figure}

It is difficult to see how well the model is performing over training episodes from ~\autoref{main:visualization_difficulty_1} purely based on the raw results. Therefore, we have chosen to use an exponentially weighted moving average (EWMA) to better visualize how the agent's performance is improving over time (as well as other variables which we will look at in this work). EWMA is calculated as follows:
\begin{equation}
y(t) = \frac{x_t + (1-\alpha_{EWMA})x_{t-1} + (1-\alpha_{EWMA})^{2}x_{t-2} + ... + (1-\alpha_{EWMA})^{t}x_0}{1 + (1-\alpha_{EWMA}) + (1-\alpha_{EWMA}^2) + ... + (1-\alpha_{EWMA})^t}
\end{equation}

Where:
\begin{itemize}
\item $x_t$ is the input
\item $y_t$ is the result
\item $\alpha_{EWMA}$ is the smoothing factor.
\end{itemize}

While it is possible to pass $\alpha_{EWMA}$ directly, it's often easier to think about the span of an EW moment. In this work, unless otherwise specified, we use a span of 1000 for our smoothed results. This is also known as an "N-day EW moving average". That is:
\begin{equation}
\alpha_{EWMA} = \frac{2}{s + 1} \text{ for span s $\geq$ 1}
\end{equation}

Using EWMA, our graph of performance vs training episodes is now smoothened and we are able to see the long-term trends:

\begin{figure}[H]
  \centering
  \includegraphics[width=.9\linewidth]{\Pic{png}{paper_Simple World_Performance_Smooth_line}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Graph of Performance vs Training Episodes (Smooth).}
  \label{main:visualization_difficulty_2}
\end{figure}

% \begin{enumerate}
% \item \textbf{Average Performance over Time}: How fast does the model learn? How well does it perform in the end?
% \item \textbf{Average Jerk over Time}: Does the jerk penalty lead to a smoother motion profile than using a simple velocity smoother?
% \item \textbf{Motion Profile}: What motion profile does the trained model generate? Trapezoidal or S-Curve?
% \end{enumerate}

\subsection{Performance of Models}
\label{main:subsec:performance}
In this section, we compare the performance of the 3 models. We measure performance as the return of a trajectory (which is a measure of cumulative reward along the trajectory). The way this is computed is according to ~\autoref{main:eq:performance}:

\begin{equation}
\label{main:eq:performance}
\text{Performance} = R(\tau) = \sum^{\infty}_{t=0} \gamma^{t} r_t
\end{equation}

~\autoref{main:eq:performance} (infinite-horizon discounted return) is explained in ~\autoref{background:subsec:reward}.

~\autoref{main:performance_1} and ~\autoref{main:performance_2} show that all 3 models display performance improvements at approximately equal rates. All 3 models also plateau at roughly the same performance level. We also observe that the \textit{Model with Jerk-Learning} generally outperforms the \textit{Model with Velocity Smoother}.
\begin{figure}[H]
  \centering
  \includegraphics[width=.8\linewidth]{\Pic{png}{paper_Simple World_Performance_Smooth_line}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Performance vs Training Episodes (Simple World)}
  \label{main:performance_1}
\end{figure}

\begin{figure}[H]
  \centering
  \includegraphics[width=.8\linewidth]{\Pic{png}{paper_Complex World_Performance_Smooth_line}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Performance vs Training Episodes (Complex World)}
  \label{main:performance_2}
\end{figure}

~\autoref{main:performance_3} and ~\autoref{main:performance_4} show that all 3 models display improvements in success rate at approximately equal rates. All 3 models also end up plateuing at the same success rate. Again, we observe that the \textit{Model with Jerk-Learning} generally outperforms the \textit{Model with Velocity Smoother}.

\begin{figure}[H]
  \centering
  \includegraphics[width=.8\linewidth]{\Pic{png}{paper_Simple World_Success Rate_Smooth_line}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Success Rate vs Training Episodes (Simple World)}
  \label{main:performance_3}
\end{figure}

\begin{figure}[H]
  \centering
  \includegraphics[width=.8\linewidth]{\Pic{png}{paper_Complex World_Success Rate_Smooth_line}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Success Rate vs Training Episodes (Complex World)}
  \label{main:performance_4}
\end{figure}

\subsection{Motion Profile of Models}
\label{main:subsec:motion}
In this section, we compare the motion profiles of the 3 models. We do this by looking at histograms of linear and angular jerk for each model in each world.

\begin{figure}[H]
  \centering
  \includegraphics[width=.9\linewidth]{\Pic{png}{paper_simple_Linear Jerk_hist}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Histograms of linear jerk for 500 episodes across all 3 models in Simple World ~\autoref{appsec:simple_world}. Model with Jerk-Learning only slightly outperforms the other models here in terms of jerk reduction.}
  \label{main:motion_1}
\end{figure}

\begin{figure}[H]
  \centering
  \includegraphics[width=.9\linewidth]{\Pic{png}{paper_simple_Angular Jerk_hist}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Histograms of angular jerk for 500 episodes across all 3 models in Simple World ~\autoref{appsec:simple_world}. Model with Velocity Smoother outperforms the other models here in terms of jerk reduction.}
  \label{main:motion_1}
\end{figure}

\begin{figure}[H]
  \centering
  \includegraphics[width=.9\linewidth]{\Pic{png}{paper_complex_Linear Jerk_hist}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Histograms of linear jerk for 500 episodes across all 3 models in Complex World ~\autoref{appsec:complex_world}. Model with Jerk-Learning only slightly outperforms the other models here in terms of jerk reduction.}
  \label{main:motion_1}
\end{figure}

\begin{figure}[H]
  \centering
  \includegraphics[width=.9\linewidth]{\Pic{png}{paper_complex_Angular Jerk_hist}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Histograms of angular jerk for 500 episodes across all 3 models in Complex World ~\autoref{appsec:complex_world}. Model with Velocity Smoother outperforms the other models here in terms of jerk reduction.}
  \label{main:motion_1}
\end{figure}

\section{Discussion}
From ~\autoref{main:subsec:performance}, we observe that neither the velocity smoother nor the jerk-learning reward feature seemed to have a significant impact on the training time required.

~\autoref{main:subsec:motion} showed that the \textit{Model with Velocity Smoother} outperformed the other 2 models in terms of jerk reduction. The only cases where it seemed to (very) slightly underperform the \textit{Model with Jerk-Learning} were for linear jerk. We can understand why this is case when we look at the histograms of linear velocities:

\begin{figure}[H]
  \centering
  \includegraphics[width=.9\linewidth]{\Pic{png}{paper_simple_Linear Velocity_hist}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Histograms of linear velocity for 500 episodes across all 3 models in Simple World ~\autoref{appsec:simple_world}}
  \label{main:discussion:linear_velocity:simple}
\end{figure}

\begin{figure}[H]
  \centering
  \includegraphics[width=.9\linewidth]{\Pic{png}{paper_complex_Linear Velocity_hist}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Histograms of linear velocity for 500 episodes across all 3 models in Complex World ~\autoref{appsec:complex_world}}
  \label{main:discussion:linear_velocity:complex}
\end{figure}

From ~\autoref{main:discussion:linear_velocity:simple} and ~\autoref{main:discussion:linear_velocity:complex} we see that to avoid generating any linear jerk the \textit{Model with Jerk-Learning} only produces the highest linear velocity possible in every state. That is, it avoids changes in velocities to avoid generating jerk. This is also the case for angular velocities as can be seen from ~\autoref{main:discussion:angular_velocity:simple} and ~\autoref{main:discussion:angular_velocity:complex}

\begin{figure}[H]
  \centering
  \includegraphics[width=.9\linewidth]{\Pic{png}{paper_simple_Angular Velocity_hist}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Histograms of angular velocity for 500 episodes across all 3 models in Simple World ~\autoref{appsec:simple_world}}
  \label{main:discussion:angular_velocity:simple}
\end{figure}

\begin{figure}[H]
  \centering
  \includegraphics[width=.9\linewidth]{\Pic{png}{paper_complex_Angular Velocity_hist}}
  \vspace{\BeforeCaptionVSpace}
  \caption{Histograms of angular velocity for 500 episodes across all 3 models in Complex World ~\autoref{appsec:complex_world}}
  \label{main:discussion:angular_velocity:complex}
\end{figure}

Furthermore, ~\autoref{main:discussion:angular_velocity:simple} and ~\autoref{main:discussion:angular_velocity:complex} also show why the \textit{Model with Velocity Smoother} outperforms the other 2 models in terms of angular jerk: the angular velocities generated by \textit{Model with Velocity Smoother} is more spread out.


\section[Effects on downstream analysis]{Effects on downstream analysis - not quite the missing link, but close}
\label{variantcalling-sec:downstream}

The ability to find additional shared variants has significant impact on our understanding of cancer evolution and the timing of initiation and metastatic seeding. Recent work has shown, that similar to the well known genetic heterogeneity, there is heterogeneity when it comes to the timing of metastatic seeding. While traditionally it was thought that tumours only metastasise after they reach a certain size, to escape the restrictions of the niche, like reduced nutrition, recent publications showed, there is also very early metastatic seeding \cite{Hu2019}.
But all methods analysing heterogeneity and evolutionary timing and history are ultimately based on the somatic variants found in the data, so if we improve on the input of the analysis methods, we can expect a clearer and possibly more granular result.

In the following sections, I will highlight how big the effect can be for methods, like phylogenetic reconstruction and clonal decomposition, which use somatic variants as input.

\subsection[Phylogenetic reconstruction]{Phylogenetic reconstruction}
\label{variantcalling-sec:phylo}
As this work is not about the advantages and shortcomings of different phylogenetic reconstruction tools, I will not show a comprehensive amount of these tools, but rather focus on the results.  For this reason, I chose to use neighbour joining (NJ) \cite{Saitou1987}, because it is fast, readily available in most phylogenetic reconstruction tool kits and if the input distance is correct, the output will be correct. And even, if the distance is not 100\% correct, if the distance is ``nearly additive`` and the input distances are not far off from the real distance, the tree topology will still be reconstructed correctly \cite{Mihaescu2007}. Lastly, in contrast to many other methods like UPGMA and WPGMA \cite{Sokal1958}, NJ does not assume an equal mutation rate of each sample, because we know, that the molecular clock hypothesis \cite{Zuckerkandl1962} is not valid for different lineages of cancers \cite{Shibata2010}.

The only thing that NJ requires as an input is a distance matrix of all samples, so the next step was the selection of the right distance metric. While there are lots of distance measures for DNA sequences, which allow accounting for different probabilities of transitions and transversions as well as uneven base composition, models like F81 \cite{Felsenstein1981} or HKY85 \cite{Hasegawa1985} are only really designed for germline mutations and are not easily applicable for subclonal somatic mutations, which is why I decided to first transform the variants present in all samples into a binary occurrence vector and then calculating the Hamming distance \cite{Hamming1950} between all samples. This generates a maximum parsimony approach and the branch length of the trees will be directly translatable to the amount of variants which are different between samples.

\autoref{fig:ca9phylo} shows both the reconstructed phylogenies of the autopsy samples of the late stage melanoma patient ``CA-F`` from the manuscript (\autoref{ch:appendixManuscript}, \autoref{A:tab:S1}), using the variants found with the default tumour-normal method on the left and our improved joint method on the right. The exact same reconstruction methodology was used otherwise, such that only the difference inputs lead to the final difference.

\begin{figure}[!ht]
\centering
\includegraphics[width=.99\linewidth]{Figures/phyloCA9.pdf}
\caption[Reconstructed phylogenies of joint samples]{Reconstructed phylogenies of a patient with multiple spatially distinct samples; Neighbour joining on Hamming distance on variant occurrence vector. Tip labels describe the location of the sample in the patient. Trees are shown as unrooted with germline as fixated origin point; black line ruler shows the length of an edge with 2000 mutations}\label{fig:ca9phylo}
\end{figure}

\todo[inline, color=green]{Maybe adjust the font size in the trees to make it more readable}

There are several obvious changes, first, the longer edge connecting the germline, which we consider as the state of no somatic variants, to all other samples. This shows that there are many more shared mutations in all samples, than what would have been anticipated with the default method, which corresponds to an overestimation of the heterogeneity of the samples. As the accumulation of somatic variants is still used as a proxy for timing and cell divisions, when assuming a high mutation rate for lung cancer ($5.3 \cdot 10^{-8}$ from \citeauthor*{Werner2020} \cite{Werner2020}) this difference of $\approx 36000$ variants is equivalent to $\approx 2000$ cell divisions. While the cell doubling rate of lung cancers is highly dependent on the type \cite{Arai1994}, this difference makes a huge difference when assessing the timing of the tumour initiation and further evolution.

Secondly, there have been topological changes, which generate a longer bifuricating edge between the olive coloured ``r. liver lobe`` and the ``r. pariet. lobe`` showing a bottle neck in cancer evolution, which fits very well with the clinical history, where the patient lived with stable disease for almost ten years, before progressing and dying. The extreme distance of the primary/diagnostic sample from the rest of the samples could be either a difference in sequencing quality, or due to the exposure to FFPE for the ten years between tumour diagnosis and death. However, as this feature is preserved between both the joint and the pairwise analysis, it is no result of our new method.

\begin{figure}[!ht]
\centering
\includegraphics[width=.99\linewidth]{Figures/tanglePhyloCA9.pdf}
\caption[Tanglegram of the reconstructed phylogenies]{Side by side view of the reconstructed trees from \autoref{fig:ca9phylo}; internal edges, which are distinct between trees are shown as dotted lines; common subtree is shown in red  Tree labels have been sorted to minimise distance between labels; Visualisation generated with dendextend \cite{Galili2015}}\label{fig:tanglePhyloCA9}
\end{figure}
\todo[inline, color=green]{maybe increase the line width of the edges}

\autoref{fig:tanglePhyloCA9} shows a topology focused view of the two trees, which highlights the breaks which are needed to morph one tree into the other with dotted edges \cite{Vienne2018}. The common subtrees are coloured the same on both sides and connecting lines show identical labels. This format shows that while the trees look quite similar at first glance, they show vastly different topologies.

One example of this is ``small bowel`` which was connected to the red common subtree, but is now much closer to the ``r. cereb. lobe`` and forms a parallel clade with the ``r.liver lobe``. In general, where the pairwise tree shows a very linear topology, which leaves only branching out of the main with no disjunct subclades, which are clearly present in the joint reconstruction.  (\autoref{fig:tanglePhyloCA9}).

\section[Longitudinal analysis]{Longitudinal analysis - something for the ages }
\label{variantcalling-sec:longitudinal}

While the initial motivation for the development of these workflows was the analysis of multi-region, so spatial, samples from the same patient coming from the CASCADE rapid autopsy program, a longitudinal application of these methods for the joint analysis of diagnostic and relapse sample, or even the repeated testing of ctDNA are quite worth thinking about. In this part, I will present work using the published workflows on a longitudinal dataset, which highlights the flexibility and widespread usability of the new methods.

In addition to their autopsy, Patient ``CA-F`` also had three longitudinal blood samples taken, from which ctDNA was extracted and WES performed. In a study of late stage melanoma patients, \Citeauthor{Tan2019} identified ctDNA sequencing as a way to stratify patients into high and low risk of relapse and therefore inform adjuvant therapy \cite{Tan2019}, which makes this patient a prime example to show the improvement with joint variant calling. Similar to the spatially related samples, the joint analysis can improve the performance, which then in term enables the detection of lower allele frequency variants, either through lower tumour burden or through the limited availability of DNA fragments from brain lesions due to the blood brain barrier \cite{2014}.

To show that even in longitudinal data, the joint analysis can boost the signal, we jointly variant called the diagnostic biopsy with only the three ctDNA samples and compared them with the results from the pairwise analysis. On average, we found 2905 additional variants in each of the ctDNA samples, which is more than double than the average number of variants found with the pairwise analysis (2414). Out of those, we found 534 variants in the ctDNA samples, which were found as a high confidence variant in the diagnostic sample, indicating that these findings are high quality calls.

\begin{figure}[!ht]
\centering
\includegraphics[width=.99\linewidth]{Figures/longitudinalCA9ctDNAVafs.pdf}
\caption[Improved somatic variant calling in longitudinal data]{Improved somatic variant calling in longitudinal data: Variant allele frequency (VAF) of variants found additionally through joint variant calling which were found as high confidence variants in the primary sample; Variants with less than 0.1 VAF in the primary are coloured grey; ``T1 recovered`` shows variants, which were high confidence in all ctDNA samples but T1 and were only found through joint calling there; Axis label show the date of blood collection }\label{fig:longitudinalVAFsctDNA}
\end{figure}

Exactly like in the spatially different samples, in longitudinal data lower tumour purity samples benefit more from the joint analysis. We see that time point 2 (T2) has the highest amount of recovered variants (377) which are found as high confidence variants in both other time points (\autoref{fig:longitudinalVAFsctDNA} A vs. B vs. C) and T2 also has the lowest tumour purity in the cfDNA recorded (T1: 60\%; T2: 20\%; T3: 60\%) however, there are still 106 variants, which were not found in the ctDNA samples at all with the pairwise analysis at all, even though they were high confidence variants in the primary sample (\autoref{fig:longitudinalVAFsctDNA}F). These variants usually show a lower depth of coverage (dp) in the ctDNA samples, which might indicate a problematic region in the genome, but rather than it not being called a variant, it is just a sign of incomplete data, which can be used with our joint approach.

\begin{figure}[!ht]
\centering
\includegraphics[width=.99\linewidth]{Figures/longitudinalCA9primaryVafs.pdf}
\caption[Longitudinal data informs diagnostic variant calling]{Longitudinal data informs diagnostic variant calling: Vafs of variants additionally found through joint calling in the primary samples; Primary recovered by PASS shows variants which were high confidence in at least one ctDNA sample; Primary recovered joint shows variant which were low confidence in all samples in the pairwise analysis; Axis label show the date of blood collection}\label{fig:longitudinalVAFsprimary}
\end{figure}

Finally, we can also find 398 additional variants in the   primary sample. 398 were discarded due to missing data in the tissue sequencing, but could be found with a high confidence in the longitudinal data and two of the variants were included, as all 4 samples had this variant below the detection threshold (\autoref{fig:longitudinalVAFsprimary}). The missing depth in the primary also leads to the occasional very high allele frequency of the variant, as all available reads show the variant, but their numbers are below the threshold normal variant callers will report variants.

This shows that both spatially and longitudinal related samples should be analysed jointly, as it substantially increases the amount of true variants found, which as shown before have a big impact on downstream analysis of the samples.

\subsection[Clonal deconvolution]{Clonal deconvolution}
\label{variantcalling-sec:clonal}

Finally, the holy grail of analysis of multiple related samples from the same patient is the clonal deconvolution, where subclonal reoccurring patterns of mutations (clones) are resolved both spatially and longitudinally. These reoccurring clones can be linked to either parallel evolution through positive selection pressure, like a targeted drug, or to the process of developing Metastases where a piece of the cancer ``breaks`` off and grows at a different site.
Surprisingly, as it shares the same issue as the joint somatic variant calling of needing deeply sequenced data from multiple samples of the same organism, there is a plethora of algorithms and methods available for clonal deconvolution. Since 2015 PhyloWGS \cite{Deshwar2015}, Canopy \cite{Jiang2016}, CLOE \cite{Marass2016}, CloneFinder \cite{Miura2018}, MACHINA \cite{ElKebir2018} and MOBSTER \cite{Caravagna2020} were published, to name a few. Underlying all these models is a form of clustering variants with similar variant allele frequency together, to reduce the combinatorial space and enhance the confidence in the signal \cite{Tarabichi2021}. Due to the high number of tools, the challenge to select the right tool is substantial, especially since all of them have up and downsides \cite{Miura2020}. In this work I decided to use PhylogicNDT \cite{Leshchiner2018} as it has been shown to work well on clinical samples \cite{Gerstung2020} and does not have the restriction for the input to be from copy number neutral areas which many of the other tools have.

The analysis was conducted by transforming the variants called with the joint workflows as well as the default pairwise analysis into the required file format without the cancer cell fraction part, in order to let PhylogicNDT calculate those on the fly. The local copy numbers for each variant is generated by intersecting the called segments from sequenza with the position of the variants. Variants which could not be assigned a local copy number change were discarded from the analysis.

\begin{figure}[!ht]
\centering
\includegraphics[width=.99\linewidth]{Figures/clonalDeconv.pdf}
\caption[Reconstructed clonal trees for joint and pairwise variant calling]{Reconstructed clonal trees from PhylogicNDT; Blue circle at top depicts the germline/normal state. The coloured edges with the same coloured circle represents a distinct subclone of the parent from which the edge emerges; The number in braces next to the edge is the number of mutations which define this subclone with an added gene symbol added, if there is a cancer driver gene mutation. The left part shows the result when using the default pairwise method of Strelka2 and the right side uses the results from the Strelka2Pass workflow}\label{fig:clonaldeconv}
\end{figure}

\autoref{fig:clonaldeconv} shows the highest parsimony clonal tree reconstructed by PhylogicNDT for the pairwise as well as the joint variant calling. As the copynumber calling information is the same for both inputs, the only difference is in the called variants. While there was no subclonal structure detected at all for the pairwise analysis, there is a highly variable structure detected using the jointly called variants. As this is a clinical sample, we cannot 100\% be sure that the more branched model is the actual truth, but its biologically more logical that a late stage cancer has developed several subclones, rather than it being a very homogeneous disease at all of the 10 sites at autopsy with no evolution over ten years of disease \cite{Gerstung2020}.
It is of particular interest, that the \textit{CDC27} gene got mutated at different time points in different clones (clone 8 vs. clone 4), which a clear sign of parallel evolution, which would definitely be missed without the joint analysis.

\subsection{Longitudinal enriched phylogeny}
\label{variantcalling-sec:fullphylo}
Of course it is finally also possible to build a phylogeny with the spatial tissue samples and the longitudinal ctDNA samples. However, as the ctDNA give a holistic view of all cancer metastases (\autoref{intro-sec:ctDNA}) the interpretation needs to accommodate for that.

\begin{figure}[!ht]
\centering
\includegraphics[width=.99\linewidth]{Figures/phyloCA9_withctDNA.pdf}
\caption[Reconstructed phylogeny with longitudinal ctDNA samples]{Reconstructed phylogeny with longitudinal ctDNA samples: Tree from \autoref{fig:ca9phylo} with three additional ctDNA samples from different time points about one year prior to death. The ruler shows the equivalent of 2000 mutations} \label{fig:phyloCA9ctDNA}
\end{figure}

The maybe most surprising thing is that the more temporally distant ctDNA samples from 17.09.2012 and 30.04.2013 are in a subclade together, away from the ``ctDNA 30.11.2012`` sample. Secondly, the addition of the ctDNA samples also lead to a further bipartition edge, which separates ``r. liver lobe``, ``small bowel`` and ``r. cereb. lobe`` from the rest of the tree (\autoref{fig:phyloCA9ctDNA}). This was already inferable from the topology of the previous tree in \autoref{fig:tanglePhyloCA9} ``joint``, but is even more pronounced with the inclusion of the ctDNA samples.

This shows that the addition of more samples helps to refine and improve the trajectory and history of cancer samples and it is vital to do this analysis jointly to generate the optimal result.