%% This is an example first chapter.  You should put chapter/appendix that you
%% write into a separate file, and add a line \include{yourfilename} to
%% main.tex, where `yourfilename.tex' is the name of the chapter/appendix file.
%% You can process specific files by typing their names in at the
%% \files=
%% prompt when you run the file main.tex through LaTeX.
\chapter{Experiments}\label{chap:exp}

\section{Evaluation criteria} \label{sec:evacri}
\subsection{Object detection evaluation metrics}
Average Precision is a standard and popular metric in measuring the performance of the object detection and segmentation algorithms. The Precision (Prcn) and Recall (Rcll) are defined as follow:
\begin{equation}
	Prcn = \frac{TP}{TP+FP}
\end{equation}
\begin{equation}
	Rcll = \frac{TP}{TP+FN}
\end{equation}
where, TP, FP, and FN are number of True Positive, False Positive, and False Negative, respectively. A predicted bounding box is considered a TP if its intersection over union (IoU), or Jaccard Index, with a ground truth box is larger than 0.5. Equation \ref{eq:IoU} shows how the IoU between a predicted box P and a ground truth box G is calculated.
\begin{equation}
	\label{eq:IoU}
	IoU(P,G) = \frac{|P\cap G|}{| P \cup G |}
\end{equation}
F1 score is a metric that combines recall and precision into a single score by calculating the harmonic mean of precision and recall:
\begin{equation}
	F1 = \frac{2TP}{2TP+FP+FN}
\end{equation}
Latest research papers tend to give results in the COCO dataset format. In COCO mAP, a 101-point interpolated AP definition is used in the calculation. For COCO, AP is the average over multiple IoU. AP@[. 5: .95] corresponds to the average AP for IoU from 0.5 to 0.95 with a step size of 0.05. The following are some other metrics collected for the COCO dataset:
\subsection{Object tracking evaluation metrics}
Performance is measured according to the framework presented in MOT16 \cite{DBLP:journals/corr/MilanL0RS16} and in the same manner as performance is measure in the MOTChallange. The authors of \cite{DBLP:journals/corr/MilanL0RS16} provide publicly available code for evaluation, this code is used to calculate the different performance metrics. MOT16 was chosen since it is a compilation of other many metrics developed in an attempt to standardize evaluation of multiple object tracking. MOT16 contains a wide array of metrics for evaluation of multiple object tracking, some which are quite similar to each other. Hence, metrics that were considered to similar to other have not been included in this thesis.
\begin{itemize}
	\item Identification Recall (IDR) and Identification Precision (IDP): IDR and IDP are similar to the metrics Recall and Precision for object detection. The metrics will however differ since objects are considered tracked only if they can be assigned an identity, which will not be the case for all detected objects. Another difference is that inconsistencies in identity assignments will lower the IDTP score. For each ground truth identity, the predicted identity most similar to it is found. Any other identity assigned to the ground truth identity is then considered a mismatch (IDFP) and will be counted as a false positive instead of a true positive.
	\begin{equation}
		IDR = \frac{IDTP}{IDTP+IDFP}
	\end{equation}
	\begin{equation}
		IDP = \frac{IDTP}{IDTP+IDFN}
	\end{equation}
where, IDTP is the sum of TP in detection and the number of correctly labeled objects in the tracking; IDFP/IDFN is the sum of FP/FN in detection and the number of correctly predicted objects for positive class in detection but incorrectly labeled in tracking.
	\item IDF1-score(IDF1): Similar to F1 score for object detection, IDF1 combines both IDR and IDP into a single score to facilitate comparisons of different trackers. The higher IDF1 is, the better tracker is.
	\begin{equation}
		IDF1 = \frac{2IDTP}{2IDTP+IDFP+IDFN}
	\end{equation}
	\item Mostly Tracked (MT): The number of ground truth identities that are tracked for 80\% or more of their existence.
	\item Partly Tracked (PT): The number of ground truth identities that are tracked between 20\% and 80\% of their existence.
	\item Mostly Lost (ML): The number of ground truth identities that are tracked for less than 20\% of their existence.
	\item Identity Switches (IDs): The number of identity switches. An identity switch is counted every time an already tracked ground truth identity is assigned a new tracking identity.
	\item Track Fragmentations (FM): The number of track fragmentations. A track fragmentation is counted every time a tracked ground truth identity is lost and then found again in a later frame.
	\item Multiple Object Tracking Accuracy (MOTA): MOTA combines false negatives, false positives and identity switches into a single score in order to express overall performance with a single value. This is the most important metric for object tracking evaluation and it is defined as:
	\begin{equation}
		MOTA = 1 - \frac{\sum _t (IDFN_t+IDFP_t+IDS_t)}{\sum _t GT_t}
	\end{equation}
where, t is the index of frame, GT is the number of observed objects in the real-world. It is worth to note that MOTA would be a negative value if there are many errors in the tracking process and the number of these errors is larger than that of observed objects.
	\item Multiple Object Tracking Precision(MOTP): MOTP measures how well correctly predicted bounding boxes TP fit their respective ground truth boxes GT. This is done by calculating the average overlap between true positives and their corresponding ground truth object.
	\begin{equation}
		MOTP = \frac{\sum _t d_{t,i}}{\sum _t c_t}
	\end{equation}
where, \(c_t\) denotes the number of matches found in the frame t and \(d_{t,i}\) is the sum of distances between all true positive and their corresponding ground truth i. This metric indicate the ability of the tracking in estimating precise object positions.
\end{itemize}
\section{Experimental results}
The following sub-sections will present result for different object detection and tracking algorithms tested in this thesis. All experiments are conducted on 32 videos of Micand32 datasets. The average inference speed is measured on a Workstation super-micro with Intel® Xeon®, CPU E5-2620 v2@2.1GHz, 6 cores, 12 threads, RAM 12GB, GPU GTX 1080.
\subsection{Egocentric hand detection and segmentation result}\label{subsec:det_res}
This subsection presents results for the different object detection algorithms consider in this thesis. The algorithms to be evaluated are: Yolov3\_spp, Yolov4x, FasterRCNN\_R\_50\_FPN\_3x, MaskRCNN\_R\_50\_FPN\_3x. Detection evaluation is show in Table \ref{tab:detres_ap}, Table \ref{tab:detres_ar} and Table \ref{tab:detres_sp}.
\begin{table}[]
	\centering
	\label{tab:detres_ap}
	\caption{Object detection and segmentation Average Precision following the COCO standard.}
	\begin{tabular}{|c|c|c|c|c|c|c|}
		\hline
		Algorithm                  & AP            & AP50          & AP75          & \(AP^{small}\) & \(AP^{medium}\)      & \(AP^{large}\)       \\ \hline
		Yolov3                & 89.2          & 92.4          & 92.1          & 1.1     & 66.4          & 54.1          \\ \hline
		Yolov4x                    & 93.1          & 95.6          & 94.6          & 3.2     & 72.5          & 42.9          \\ \hline
		FasterRCNN & \textbf{96.2} & 97.9          & \textbf{97.9} & 0.9     & \textbf{75.8} & 6.3           \\ \hline
		MaskRCNN  & 92.1          & \textbf{98.9} & 97.9          & 0.0     & 32.4          & \textbf{92.2} \\ \hline
	\end{tabular}
\end{table}
\begin{table}[]
	\centering
	\caption{Object detection and segmentation Average Recall following the COCO standard.}
	\label{tab:detres_ar}
	\resizebox{\textwidth}{!}{%
		\begin{tabular}{|c|c|c|c|c|c|c|}
			\hline
			Algorithm                  & \(AR^{max=1}\)      & \(AR^{max}=10\)      & \(AR^{max}=100\)     & \(AR^{small}\) & \(AR^{medium}\)      & \(AR^{large}\)       \\ \hline
			Yolov3                & 6.5          & 53.6          & 76.4          & 3.2     & 32.5          & 75.9          \\ \hline
			Yolov4x                    & 8.7          & 65.8          & 89.7          & 7.1     & 40.1          & 82.7          \\ \hline
			FasterRCNN & \textbf{9.6} & 76.8          & \textbf{97.6} & 10.0    & \textbf{77.8} & \textbf{97.6} \\ \hline
			MaskRCNN   & 9.2          & \textbf{73.9} & 94.6          & 0.0     & 50.8          & 94.7          \\ \hline
		\end{tabular}%
	}
\end{table}
\begin{table}[hbt!]
	\centering
	\caption{Object detection and segmentation average training and inference time, speed and machine requirements.}
	\label{tab:detres_sp}
	\resizebox{\textwidth}{!}{%
		\begin{tabular}{|c|c|c|c|c|c|}
			\hline
			\multirow{2}{*}{Algorithm} & \multicolumn{2}{l|}{Average time}         & \multirow{2}{*}{Speed (Hz)} & \multicolumn{2}{l|}{Memory requirement (MB)} \\ \cline{2-3} \cline{5-6}
			& Training (hour) & Inference (millisecond) &                             & RAM                   & GPU                  \\ \hline
			Yolov3                & \textbf{36}     & 45                      & 22                          & \textbf{1989}         & 2260                 \\ \hline
			Yolov4x                    & 40              & \textbf{38}             & \textbf{25}                 & 2152                  & \textbf{2076}        \\ \hline
			FasterRCNN & 46              & 84                      & 12                          & 2351                  & 3297                 \\ \hline
			MaskRCNN   & 50              & \textbf{215}            & \textbf{5}                  & \textbf{2461}         & \textbf{3674}        \\ \hline
		\end{tabular}%
	}
\end{table}
\subsection{Egocentric hand tracking result}
Tracking results for SORT and DeepSORT with different object detection strategy are presented in this section. As described in section 3.4, the tests are performed in a tracking by detection system where tracking and detection algorithms are completely separated. Therefore, different results for combinations of tracking and detection algorithms are reported. Further, ground-truth detection are also test with SORT and DeepSORT. This is done so as to give an insight how much of the error is due to the object detection algorithm and how much of it is due to the tracking algorithm. Tracking performance with ground-truth detection should only have errors introduced by the tracking algorithms and can thereby give an upper limit for how much better the tracking by detection system can become by changing object detection algorithm. Thus, this frameworks consists of 6 detction and segmentation algorithms, they are: Yolov3\_SPP, Yolov4x, Faster\_RCNN\_R\_50\_FPN\_3x, Mask\_RCNN\_R50\_FPN\_3x, Mask\_RCNN\_R50\_FPN\_3x with region based and detection ground-truth. Their combination with 2 tracking algorithms, SORT and DeepSORT, we obtain a total of 12 options for the pipeline. However, the combination of MaskRCNN with region based and SORT scheme is identical to the combination of Mask and SORT scheme because in theory, the SORT algorithm does not use visual information fields. To conclude, we have 11 approaches conventionally defined as in the Table \ref{tab:notation} for brief presentation.
\begin{table}[]
	\centering
	\label{tab:notation}
	\caption{Notation of the 11 approaches mentioned and used in the experiments.}
	\begin{tabular}{|l|l|c|l|}
		\hline
		No & Detector              & Tracker                                        & Notation \\ \hline
		1  & Yolov3                & \multirow{5}{*}{SORT}                          & Y3S      \\ \cline{1-2} \cline{4-4}
		2  & Yolov4                &                                                & Y4S      \\ \cline{1-2} \cline{4-4}
		3  & FasterRCNN            &                                                & FS       \\ \cline{1-2} \cline{4-4}
		4  & MaskRCNN              &                                                & MS       \\ \cline{1-2} \cline{4-4}
		5  & Groundtruth           &                                                & GS       \\ \hline
		6  & Yolov3                & \multicolumn{1}{c|}{\multirow{6}{*}{DeepSORT}} & Y3DS     \\ \cline{1-2} \cline{4-4}
		7  & Yolov4                & \multicolumn{1}{c|}{}                          & Y4DS     \\ \cline{1-2} \cline{4-4}
		8  & FasterRCNN            & \multicolumn{1}{c|}{}                          & FS       \\ \cline{1-2} \cline{4-4}
		9  & MaskRCNN              & \multicolumn{1}{c|}{}                          & MS       \\ \cline{1-2} \cline{4-4}
		10 & MaskRCNN+Region based & \multicolumn{1}{c|}{}                          & RDS      \\ \cline{1-2} \cline{4-4}
		11 & Groundtruth           & \multicolumn{1}{c|}{}                          & GDS      \\ \hline
	\end{tabular}
	\vspace{-0.7cm}
\end{table}
\\In this thesis, I conduct the experiments to evaluate the performance of these approaches on the custom dataset Micand32 which contains 2 sub-datasets: Micand32S and Micand32E. The properties of this dataset is reported in chapter \ref{chap:method}, sub-section \ref{subsec:micand32}.
\subsubsection{Short-term tracking on a standard dataset: Micand32S}
The Mican32S sub-datasets contains 26 sequences containing in total 3162 frames as per the MOT16 Challenge rules and recommendations, each containing from 100 to 300 images. These recordings are chosen so every video has an excursion span of precisely one patient’s activity, for instance opening the cap of the water bottle, getting the circle, playing with the ball, turning left of right, etc. Simultaneously, every video generally contains just 1 or 2 active hands. This sub-dataset can be characterized into short-term tracking datasets. The desire set for the framework proposed in this theory is to recognize, segment, and track this entire fundamental excursion of the hand. Since the quantity of recordings is up to 26 recordings, in this part I will list the overall evaluation for each approach on the whole 26 recordings, which is a weighted average of the quantity of frames per video. Besides, I likewise measure a case of 26 recordings running on 1 approach to see the impact of the sort of activity on the presentation of the framework. Note that the hand in the 26 recordings is utilized to train the deep appearance descriptor for DeepSORT as referenced in chapter \ref{chap:framework}, sub-section \ref{subsec:train_deep}. Table \ref{tab:short_overall} reports short-term tracking overall result on Micand32S following the MOT16 evaluation protocol.
\begin{table}[]
	\caption{Short-term tracking \textbf{overall} result on Micand32S following the MOT16 evaluation protocol.}
	\label{tab:short_overall}
	\centering
	\resizebox{\textwidth}{!}{%
		\begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|}
			\hline
			Method & \begin{tabular}[c]{@{}l@{}}IDF1\\ (\%)$\uparrow$\end{tabular} &
			\begin{tabular}[c]{@{}l@{}}IDP\\ (\%)$\uparrow$\end{tabular} &
			\begin{tabular}[c]{@{}l@{}}IDR\\ (\%)$\uparrow$\end{tabular} &
			\begin{tabular}[c]{@{}l@{}}Rcll\\$\uparrow$    \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}Prcn\\$\uparrow$    \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}GT\\$\uparrow$      \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}MT\\$\uparrow$      \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}PT\\$\uparrow$      \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}ML\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}FP\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}FN\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}IDs\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}FM\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}\textbf{MOTA}\\ (\%)$\uparrow$\end{tabular} &
			\begin{tabular}[c]{@{}l@{}}MOTP\\ $\downarrow$\end{tabular}            \\ \hline
			Y3S    & 97.0                                                 & 99.6                                                & 94.6                                                & 94.8                                                   & 99.8                                                   & 35                                                   & 33          & \textbf{1}                                           & 1          & 6          & 207        & 1          & 1          & 94.6                                                 & 0.087          \\ \hline
			Y4S    & 98.6                                                 & 99.4                                                & 97.7                                                & 97.8                                                   & 99.4                                                   & 35                                                   & \textbf{34} & \textbf{1}                                           & \textbf{0} & 23         & 89         & \textbf{0} & 5          & 97.2                                                 & 0.108          \\ \hline
			FS     & 97.1                                                 & 99.7                                                & 94.6                                                & 94.8                                                   & 99.9                                                   & 35                                                   & 33          & \textbf{1}                                           & 1          & 4          & 206        & 1          & 1          & 94.7                                                 & 0.087          \\ \hline
			MS     & 98.8                                                 & 99.7                                                & 98.0                                                & 98.2                                                   & 99.9                                                   & 35                                                   & 34          & \textbf{1}                                           & \textbf{0} & 3          & 73         & 1          & 1          & 98.1                                                 & 0.072          \\ \hline
			GS     & \textbf{99.9}                                        & 99.9                                                & \textbf{99.9}                                       & \textbf{99.9}                                          & \textbf{100.0}                                         & 35                                                   & \textbf{34} & \textbf{1}                                           & \textbf{0} & 1          & 4          & \textbf{0} & \textbf{0} & \textbf{99.9}                                        & 0.089          \\ \hline
			Y3DS   & 97.1                                                 & 99.6                                                & 94.7                                                & 94.9                                                   & 99.8                                                   & 35                                                   & 33          & \textbf{1}                                           & 1          & 7          & 203        & 1          & 2          & 94.7                                                 & 0.088          \\ \hline
			Y4DS   & 98.8                                                 & 99.6                                                & 97.9                                                & 98.1                                                   & 99.8                                                   & 35                                                   & \textbf{34} & \textbf{1}                                           & \textbf{0} & 7          & 74         & 1          & 2          & 97.9                                                 & 0.106          \\ \hline
			FDS    & 98.2                                                 & 99.9                                                & 96.5                                                & 96.5                                                   & \textbf{100.0}                                         & 35                                                   & 33          & \textbf{1}                                           & 1          & 1          & 140        & \textbf{0} & \textbf{0} & 96.5                                                 & 0.077          \\ \hline
			MDS    & 98.2                                                 & 99.9                                                & 96.5                                                & 96.5                                                   & \textbf{100.0}                                         & 35                                                   & 33          & \textbf{1}                                           & 1          & 1          & 140        & \textbf{0} & \textbf{0} & 96.5                                                 & 0.076          \\ \hline
			RDS    & 99.3                                                 & 99.3                                                & 99.3                                                & 99.3                                                   & 99.4                                                   & 35                                                   & \textbf{34} & \textbf{1}                                           & \textbf{0} & 25         & 27         & \textbf{0} & 6          & 98.7                                                 & 0.089          \\ \hline
			GDS    & \textbf{99.9}                                        & \textbf{100}                                        & 99.8                                                & 99.8                                                   & \textbf{100.0}                                         & 35                                                   & \textbf{34} & \textbf{1}                                           & \textbf{0} & \textbf{0} & \textbf{0} & \textbf{0} & 1          & 99.8                                                 & \textbf{0.062} \\ \hline
		\end{tabular}%
	}
\end{table}
The GDS presents an almost perfect performance on Micand32S, detailed metrics of this method is revealed in Figure \ref{fig:GDS_short}. Of all ground-truth-free approaches, the Y4DS approach reaches a remarkable result shown in Figure \ref{fig:y4ds_short}.
\subsubsection{Long-term tracking on a realistic dataset: Micand32E}
In this part, I will validate the effectiveness of the 11 approaches on the Micand32E dataset. As referenced in chapter \ref{chap:method}, sub-section \ref{subsec:micand32}, the Micand32E contains 6 sequences, the sequence length of each is in range of 1000 frames and 2000 frames, and total the volume of this sub-dataset is 7857 frames. Also, this sub-dataset fully contains 4 types of patient’s action: (5) practice with a ball, (6) practice with water bottles, (7) practice with wooden blocks, (8) practice with cylinders. This sub-dataset is very realistic, interesting and quite challenging because it is specifically selected with careful observations  by developers in the NAFOSTED’s project who are experienced in computer vision problems.
Table \ref{tab:long_overall} presents the overall tracking results referenced on Micand32E of the 11 approaches.
% Please add the following required packages to your document preamble:
% \usepackage{graphicx}
% Please add the following required packages to your document preamble:
% \usepackage{graphicx}
\begin{table}[]
	\caption{Long-term tracking \textbf{overrall} results on Micand32E following the MOT16 protocol.}
	\label{tab:long_overall}
	\centering
	\resizebox{\textwidth}{!}{%
		\begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|}
			\hline
			Method & \begin{tabular}[c]{@{}l@{}}IDF1\\ (\%)$\uparrow$\end{tabular} &
			\begin{tabular}[c]{@{}l@{}}IDP\\ (\%)$\uparrow$\end{tabular} &
			\begin{tabular}[c]{@{}l@{}}IDR\\ (\%)$\uparrow$\end{tabular} &
			\begin{tabular}[c]{@{}l@{}}Rcll\\$\uparrow$    \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}Prcn\\$\uparrow$    \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}GT\\$\uparrow$      \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}MT\\$\uparrow$      \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}PT\\$\uparrow$      \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}ML\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}FP\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}FN\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}IDs\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}FM\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}\textbf{MOTA}\\ (\%)$\uparrow$\end{tabular} &
			\begin{tabular}[c]{@{}l@{}}MOTP\\ $\downarrow$\end{tabular}            \\ \hline
			Y3S    & 51.4          & 59.4          & 45.2          & 75.3                                                   & 99.4                                                   & 24   & 7                                                    & 8                                                    & 9                                                    & 68                                                   & 3630                                                 & 123                                                   & 174                                                  & 74.1          & 0.133          \\ \hline
			Y4S    & 56.7          & 60.7          & 53.0          & 86.4                                                   & 99.4                                                   & 24   & 9                                                    & \textbf{11}                                          & 4                                                    & 81                                                   & 1996                                                 & 134                                                   & 159                                                  & 85.0          & 0.127          \\ \hline
			FS     & 74.5          & 73.9          & 74.8          & 97.9                                                   & 97.1                                                   & 24   & 17                                                   & 7                                                    & \textbf{0}                                           & 426                                                  & 306                                                  & 115                                                   & 91                                                   & 94.2          & 0.082          \\ \hline
			MS     & 74.5          & 73.9          & 74.8          & 97.9                                                   & 97.2                                                   & 24   & 17                                                   & 7                                                    & \textbf{0}                                           & 420                                                  & 304                                                  & 114                                                   & 90                                                   & 94.3          & 0.082          \\ \hline
			GS     & 89.1          & 89.3          & 88.7          & 98.5                                                   & 99.6                                                   & 24   & 21                                                   & 3                                                    & \textbf{0}                                           & 62                                                   & 220                                                  & 91                                                    & 50                                                   & 97.5          & 0.059          \\ \hline
			Y3DS   & 58.7          & 66.0          & 52.6          & 78.4                                                   & 98.7                                                   & 24   & 9                                                    & 7                                                    & 8                                                    & 149                                                  & 3176                                                 & 123                                                   & 202                                                  & 76.6          & 0.151          \\ \hline
			Y4DS   & 65.0          & 68.1          & 61.9          & 89.3                                                   & 98.5                                                   & 24   & 11                                                   & 9                                                    & 4                                                    & 194                                                  & 1581                                                 & 122                                                   & 192                                                  & 87.1          & 0.142          \\ \hline
			FDS    & 79.4          & 79.0          & 79.5          & 98.1                                                   & 97.8                                                   & 24   & 17                                                   & 7                                                    & \textbf{0}                                           & 320                                                  & 282                                                  & 117                                                   & 75                                                   & 95.1          & 0.060          \\ \hline
			MDS    & 83.5          & 83.5          & 83.3          & 98.1                                                   & 98.7                                                   & 24   & 18                                                   & 5                                                    & 1                                                    & 184                                                  & 275                                                  & 95                                                    & 61                                                   & 96.2          & 0.054          \\ \hline
			RDS    & \textbf{89.6} & \textbf{89.5} & \textbf{89.5} & 98.2                                                   & 98.6                                                   & 24   & 16                                                   & 8                                                    & \textbf{0}                                           & 212                                                  & 258                                                  & 79                                                    & 74                                                   & 96.3          & 0.072          \\ \hline
			GDS    & 88.5          & 88.5          & 88.1          & \textbf{99.1}                                          & \textbf{99.9}                                          & 24   & \textbf{23}                                          & 1                                                    & \textbf{0}                                           & \textbf{12}                                          & \textbf{135}                                         & 82                                                    & \textbf{43}                                          & \textbf{98.4} & \textbf{0.052} \\ \hline
		\end{tabular}%
	}
%\vspace{-1cm}
\end{table}
To analyze the complexity and the difficulty of the four types of action, I analyzed the results of the tracking algorithm that achieves highest MOTA, the GDS, across these four action types. This result is shown in the Table \ref{tab:long_gds}.
\begin{table}[]
	\caption{GDS \textbf{detail} results on Micand32E.}
	\label{tab:long_gds}
	\centering
	\resizebox{\textwidth}{!}{%
		\begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|}
			\hline
			Method & \begin{tabular}[c]{@{}l@{}}IDF1\\ (\%)$\uparrow$\end{tabular} &
			\begin{tabular}[c]{@{}l@{}}IDP\\ (\%)$\uparrow$\end{tabular} &
			\begin{tabular}[c]{@{}l@{}}IDR\\ (\%)$\uparrow$\end{tabular} &
			\begin{tabular}[c]{@{}l@{}}Rcll\\$\uparrow$    \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}Prcn\\$\uparrow$    \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}GT\\$\uparrow$      \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}MT\\$\uparrow$      \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}PT\\$\uparrow$      \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}ML\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}FP\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}FN\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}IDs\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}FM\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}\textbf{MOTA}\\ (\%)$\uparrow$\end{tabular} &
			\begin{tabular}[c]{@{}l@{}}MOTP\\ $\downarrow$\end{tabular}            \\ \hline
			GH010354\_5\_17718\_19366 & 64.1          & 64.8          & 63.5          & 97.7                                                   & 99.6                                                   & 3          & 3                                                    & 0                                                    & \textbf{0}                                           & 7          & 41         & 5                                                     & 7                                                    & 97.0          & 0.096          \\ \hline
			GH010373\_5\_1284\_2724   & 89.1          & 89.1          & 88.7          & 98.8                                                   & 99.9                                                   & \textbf{6} & \textbf{6}                                           & 0                                                    & \textbf{0}                                           & 4          & 40         & 19                                                    & 14                                                   & 98.1          & 0.051          \\ \hline
			GH010358\_6\_10208\_11900 & 81.4          & 81.3          & 81.1          & 99.3                                                   & \textbf{100.0}                                         & 4          & 4                                                    & 0                                                    & \textbf{0}                                           & \textbf{0} & 24         & 8                                                     & 13                                                   & 99.0          & \textbf{0.038} \\ \hline
			GH010373\_6\_3150\_4744   & \textbf{99.7} & \textbf{99.8} & \textbf{99.5} & \textbf{99.8}                                          & \textbf{100.0}                                         & 4          & 4                                                    & 0                                                    & \textbf{0}                                           & \textbf{0} & 8          & \textbf{1}                                            & 1                                                    & \textbf{99.7} & 0.048          \\ \hline
			GH010358\_7\_2490\_3390   & 97.6          & 97.4          & 97.2          & 99.0                                                   & 99.9                                                   & 4          & 3                                                    & \textbf{1}                                           & \textbf{0}                                           & 1          & 19         & 9                                                     & 8                                                    & 98.5          & 0.050          \\ \hline
			GH010358\_8\_8000\_8547   & 97.2          & 97.3          & 97.0          & 99.7                                                   & \textbf{100.0}                                         & 3          & 3                                                    & 0                                                    & \textbf{0}                                           & \textbf{0} & \textbf{3} & 40                                                    & \textbf{0}                                           & 96.1          & 0.048          \\ \hline
			Overall                   & 88.5          & 88.5          & 88.1          & 99.1                                                   & 99.9                                                   & 24         & 23                                                   & 1                                                    & 0                                                    & 12         & 135        & 82                                                    & 43                                                   & 98.4          & 0.052          \\ \hline
		\end{tabular}%
	}
\end{table}
%\raggedbottom
Especially, the approach proposed by the author of this thesis performs an impressive result as in the Table \ref{tab:long_rds}.
\begin{table}[]
	\caption{RDS \textbf{detail} results on Micand32E.}
	\label{tab:long_rds}
	\centering
	\resizebox{\textwidth}{!}{%
		\begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|}
			\hline
			Method & \begin{tabular}[c]{@{}l@{}}IDF1\\ (\%)$\uparrow$\end{tabular} &
			\begin{tabular}[c]{@{}l@{}}IDP\\ (\%)$\uparrow$\end{tabular} &
			\begin{tabular}[c]{@{}l@{}}IDR\\ (\%)$\uparrow$\end{tabular} &
			\begin{tabular}[c]{@{}l@{}}Rcll\\$\uparrow$    \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}Prcn\\$\uparrow$    \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}GT\\$\uparrow$      \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}MT\\$\uparrow$      \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}PT\\$\uparrow$      \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}ML\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}FP\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}FN\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}IDs\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}FM\\$\downarrow$     \end{tabular} &
			\begin{tabular}[c]{@{}l@{}}\textbf{MOTA}\\ (\%)$\uparrow$\end{tabular} &
			\begin{tabular}[c]{@{}l@{}}MOTP\\ $\downarrow$\end{tabular}            \\ \hline
			GH010354\_5\_17718\_19366 & 63.1       & 63.1      & 63.1      & 95.4                                                   & 95.5                                                   & 3    & 1                                                    & 2                                                    & 0                                                    & 79   & 80   & 0                                                     & 10                                                   & 90.7       & 0.112  \\ \hline
			GH010373\_5\_1284\_2724   & 79.4       & 79.2      & 79.2      & 97.5                                                   & 98.2                                                   & 6    & 4                                                    & 2                                                    & 0                                                    & 60   & 82   & 18                                                    & 27                                                   & 95.2       & 0.072  \\ \hline
			GH010358\_6\_10208\_11900 & 97.0       & 96.4      & 97.2      & 98.6                                                   & 98.3                                                   & 4    & 3                                                    & 1                                                    & 0                                                    & 9    & 32   & 9                                                     & 12                                                   & 97.4       & 0.063  \\ \hline
			GH010373\_6\_3150\_4744   & 99.6       & 99.8      & 99.5      & 99.7                                                   & 100.0                                                  & 4    & 3                                                    & 1                                                    & 0                                                    & 1    & 10   & 1                                                     & 2                                                    & 99.6       & 0.069  \\ \hline
			GH010358\_7\_2490\_3390   & 97.5       & 97.4      & 96.9      & 98.3                                                   & 99.5                                                   & 4    & 3                                                    & 1                                                    & 0                                                    & 9    & 32   & 9                                                     & 12                                                   & 97.4       & 0.063  \\ \hline
			GH010358\_8\_8000\_8547   & 96.8       & 96.8      & 96.8      & 99.5                                                   & 99.5                                                   & 3    & 2                                                    & 1                                                    & 0                                                    & 5    & 6    & 40                                                    & 1                                                    & 95.4       & 0.071  \\ \hline
			Overall                   & 89.6       & 89.5      & 89.5      & 89.5                                                   & 98.2                                                   & 24   & 16                                                   & 8                                                    & 0                                                    & 212  & 258  & 79                                                    & 74                                                   & 96.3       & 0.072  \\ \hline
		\end{tabular}%
	}
\end{table}
\section{Discussions}
\subsection{Object detection: tradeoff between accuracy and speed}
Concerning egocentric hand detection performance, from Table \ref{tab:detres_ap}, it’s evident that, on the Micand32 dataset, two-stage detectors, represented by the RCNN family models generally outperform the one-state detector, the YOLO family models. Concerning the average precision, the AP50 obtained by the MaskRCNN and FastrCNN are 98,9\% and 97,9\% respectively. This points that MaskRCNN performs an impressive segmentation result, while the FastRCNN additionally presents an awesome detection quality where the AP is pretty high, 96.2\%. Yolov4 performs a remarkable performance with 95,6\% AP50 and 72,5\% \(AP^{medium}\). Yolov3 adapts well in predicting large hands, it obtains 54,1\% \(AP^{large}\), but is still worse than the MaskRCNN which acquires 92,2\% \(AP^{large}\).

In term of average recall, again, the FastRCNN performs the best performance with 97,6\% for both \(AR^{large}\) and \(AR^{max}=100\), this means that this models can detect almost large hands in the datasets.

Metrics in Table \ref{tab:detres_sp} is measured in order to evaluate the machine memory requirements and the processing rate of detection phase when inference models. As theory researched in chapter \ref{chap:method}, the one-stage detectors represented by Yolo family models completely outperforms the two-stage detectors. Yolov4 performs a real-time speed, 25 frames per second on a normal machine which requires only 2Gb RAM and 2GB GPU memory in an inference stage. Processing rate is slightly reduced in case of Yolov3spp with 22 frames per second. The MaskRCNN takes up to 215 milliseconds computing time for each images, and thus the speed is only 5 frames per second and it’s not adaptable for real-time processing application, and also, it requires a huge memory, 3,5GB GPU and 2,4GB CPU.
From this investigation result, three recommendations on the choice of egocentric hand detection and segmentation can be provided. First, the Yolov4x is suggested for the application that requires real-time processing and machine resources are restricted. Second, in case the accuracy is essential, the MaskRCNN should be employed. Third, the FastRCNN seems to get the balance trade-off between precision and speed in remaining cases.

\subsection{The superiority of DeepSORT over SORT}
To compare the effectiveness of the 2 tracking method, overall tracking evaluation on Micand32E reported in Table \ref{tab:long_overall} is very essential. On the same detection algorithm, DeepSORT generates a higher MOTA in almost cases, for example: Y3DS 76,6\%  while Y3S 74,1\%, MDS 96,2\% while MS 94,3\%. Figure \ref{fig:pgf_mota} and Figure \ref{fig:pgf_idf1} clearly demonstrates the dominance of DeepSORT over SORT. This result is expected because the DeepSORT tracking inherits the benefits of deep appearance descriptor trained on the egocentric hand re-identification dataset generated by this thesis which is mentioned in the background theory. Also, the identity switches IDs has significantly decreased between SORT and DeepSORT, 91 for GS and only 79 for GDS.
\begin{figure}
\resizebox{0.5\textwidth}{!}{%
\begin{tikzpicture}
	\label{fig:pgf_mota}
	\begin{axis}[
		symbolic x coords={Y3,Y4,FR,MR,GT},
		xlabel=Detector,
		ylabel=MOTA,
		legend style={at={(0.6,0.15)},
			anchor=north,legend columns=-1},
		]
		\addplot [cyan,mark=*, nodes near coords,every node near coord/.append style={xshift=19pt,yshift=-7pt,anchor=east,font=\footnotesize},]
		coordinates {(Y3,74.1) (Y4,85)
			(FR,94.2) (MR,94.3) (GT,97.5)};
		\addplot [blue,mark=square,nodes near coords,every node near coord/.append style={xshift=-6pt, font=\footnotesize}]
		coordinates {(Y3,76.6) (Y4,87.1)
			(FR,95.1) (MR,96.2) (GT,98.4)};
		\legend{SORT,DeepSORT}
	\end{axis}
\end{tikzpicture}}
\resizebox{0.5\textwidth}{!}{%
	\begin{tikzpicture}
		\label{fig:pgf_idf1}
		\begin{axis}[
			symbolic x coords={Y3,Y4,FR,MR,GT},
			xlabel=Detector,
			ylabel=IDF1,
			legend style={at={(0.4,1.0)},
				anchor=north,legend columns=-1},
			]
			\addplot [mark=*, nodes near coords,every node near coord/.append style={xshift=19pt,yshift=-7pt,anchor=east,font=\footnotesize},]
			coordinates {(Y3,51.4) (Y4,56.7)
				(FR,74.5) (MR,74.5) (GT,89.1)};
			\addplot [red,mark=square,nodes near coords,every node near coord/.append style={xshift=-6pt, font=\footnotesize}]
			coordinates {(Y3,58.7) (Y4,65)
				(FR,79.4) (MR,83.5) (GT,89.6)};
			\legend{SORT,DeepSORT}
		\end{axis}
	\end{tikzpicture}
}
\caption{Overall MOTA and IDF1 metric on Micand32E.}
\end{figure}
\subsection{Impact of detection method over tracking result}
In the sub-section \ref{subsec:det_res}, the accuracy of the Yolo family models and RCNN family models are reported and we insists that in term of precision, the MaskRCNN and FastRCNN completely outperforms the Yolos. The impact of choosing detection tactic on the tracking performance is also shown clearly in Figure 4 1 and Figure 4 2. On the same tracking method, SORT, MOTA results of Yolov3, Yolov4, FasterRCNN, MaskRCNN and detection groundtruth are respectively 74,1\%, 85\%, 94,2\%, 94,3\% and 97,5\%. Also it’s remarkable that the number of FN decreases, 3176 for Y3DS, 1581 for Y4DS, 282 for FDS, 184 for MDS and 82 for GDS. After researching theory and evaluating experiments, it’s quit intuitive but reasonable to conclude that the better detection method is, the higher tracking quality is.
\subsection{Complexity of 4 types of patients's actions}
The diversity and complexity of 4 type of patient’s actions are analyzed via metrics in Table \ref{tab:long_gds}, the highest results achieved in this experiment. In term of MOTA metric, the GDS obtains best performance on the action type 6, with 2 sequences:  GH010373\_6\_3150\_4744 99,7\% and GH010358\_6\_10208\_11900 99\%. On the contrary, the MOTA for GH010358\_8\_8000\_5447 is 96,1\% with GDS.
On the other hand, the sequence’s volume also has impacts on the tracking qualities. All videos in Micand32S are short-term and the results of all the 11 approaches is almost perfect on this datasets. But in Micand32E, the sequences is long and then many weaknesses of tracking algorithms appear, it’s visible via the metrics like IDP and IDR. However, on both datasets, almost approaches can still track hands well, this is shown by metric GT and MT in Table \ref{tab:long_overall}.
\subsection{Challenging cases}
According to my personal coding experience through this thesis, multiple object tracking is quit difficult to debug errors. The evaluation process is not reversible, when the results are available, the metrics can be calculated, but by just looking at the metrics, it’s hard to know where the errors occur, we can only visualize the result through visible video to find the bug. In this way, I find some challenging cases in some sequences that make almost of 11 approaches yield tracking errors. Some typical cases are enumerated as follow:
\begin{itemize}
	\item When hands are occluded by other obstacles. Trackers lost information about these hands due to the detector naturally can't find those hands. In figure \ref{fig:occluded}, in the first, from frame \(200^{th}\) to \(250^{th}\), SORT are tracing the right hand with ID 1. In the second row, from frame \(256^{th}\) to frame \(261^{th}\), an experimental equipment - a "toy house", has significantly obscured the right hand part. This makes FasterRCNN can't recognize that hand (false positive), and therefore, SORT lost that hand's trajectory. Until frame \(267^{th}\), the hand goes out of the equipment and can be seen clearly. In the third row, FasterRCNN can predict again that right hand, but the shape of the hand seems to be too different for the first row, and of course, SORT initialize a new ID 7 for the hand. A remarkable point in this case is that, with the same condition, the left hand isn't overlapped by the "toy house", so both FaterRCNN and SORT worked perfectly, it keeps the ID 4 for the left hand through all the time.
	\item When a hand moves rapidly so that the cameras can’t catch, this make the motion blur phenomenon occur. As a result, detectors fail to predict the existence of the hand. For example, in figure \ref{fig:blur}, in the frame \(119^{th}\), the trackers are tracing the ID 1, but in the following frames from \(120^{th}\) to \(142^{th}\) Yolov3 can't predict the hand's existence (false positive), until the frame \(143^{th}\) it find that hand again but at this time the tracker can't retrieve the hand's identity, it initialize a new ID 2 for the hand.
	\item When the shape of the hand changes heavily. Intuitively, the deep appearance descriptor doesn't work in this case, it simply can't recognize that these different shapes belong to a same id. Consequently, a new ID is initialized and then ID switches occur. In figure \ref{fig:shape}, DeepSORT works stability in frames \(1582^{th}, 1592^{th}, 1602^{th}, 1612^{th}, 1622^{th}\) with the hand ID 5. But in frame \(1630^{th}\), the shape of the right hand changes heavily, DeepSORT initialize a new ID 7 for that hand. Figure \ref{fig:occluded} also proves this: the shape of the right hand in the first row and the third row is too different, and certainly, SORT assign 2 identities for just 1 hand.
	\item The ambiguous  definition of the "hand" causes difficulty. Detection model is training on both GTEA family ("hand" includes wrist, arm) and EgoHand ("hand" just includes hand) so the during inference stage, the detector may be quite tough. In figure \ref{fig:definition}, FasterRCNN and SORT keep the left hand awesomely through 283 frames for \(1123^{th}\) to \(1406^{th}\) with ID 22, presented in the first row. Suddenly, in frame \(1407^{th}\), FasterRCNN predicts this left hand including the wrist. This makes SORT assign a new ID 24 for that hand as show in the second row. This issue is reported and will be discussed in the future work in the section \ref{sec:futurework}.
	\item When the hands disappear from the camera’s view and return after a long time, this makes the tracker "forget" that instance.
\end{itemize}
\begin{figure*}[ht!]
	\includegraphics[width=.16\textwidth]{Figs/Occluded/200.png}
	\includegraphics[width=.16\textwidth]{Figs/Occluded/210.png}	\includegraphics[width=.16\textwidth]{Figs/Occluded/220.png}
	\includegraphics[width=.16\textwidth]{Figs/Occluded/230.png}	\includegraphics[width=.16\textwidth]{Figs/Occluded/240.png}
	\includegraphics[width=.16\textwidth]{Figs/Occluded/250.png}
	\includegraphics[width=.16\textwidth]{Figs/Occluded/256.png}
	\includegraphics[width=.16\textwidth]{Figs/Occluded/257.png}	\includegraphics[width=.16\textwidth]{Figs/Occluded/258.png}
	\includegraphics[width=.16\textwidth]{Figs/Occluded/259.png}	\includegraphics[width=.16\textwidth]{Figs/Occluded/260.png}
	\includegraphics[width=.16\textwidth]{Figs/Occluded/261.png}
	\includegraphics[width=.16\textwidth]{Figs/Occluded/267.png}
	\includegraphics[width=.16\textwidth]{Figs/Occluded/270.png}	\includegraphics[width=.16\textwidth]{Figs/Occluded/273.png}
	\includegraphics[width=.16\textwidth]{Figs/Occluded/276.png}	\includegraphics[width=.16\textwidth]{Figs/Occluded/279.png}
	\includegraphics[width=.16\textwidth]{Figs/Occluded/282.png}
	\caption{Illustration of hand occluded by an obstacle. Pay attention to the patient's right hand. Frames extracted from GH010373\_5\_1284\_2724 using FasterRCNN+SORT, ordered from left to right, up to down: (200, 210, 220, 230, 240, 250), (256, 257, 258, 259, 260, 261), (267, 270, 273, 276, 279, 282).}
	\label{fig:occluded}
\end{figure*}
\begin{figure*}[ht!]
	\includegraphics[width=.16\textwidth]{Figs/blur/119.png}
	\includegraphics[width=.16\textwidth]{Figs/blur/120.png}	\includegraphics[width=.16\textwidth]{Figs/blur/125.png}
	\includegraphics[width=.16\textwidth]{Figs/blur/131.png}	\includegraphics[width=.16\textwidth]{Figs/blur/137.png}
	\includegraphics[width=.16\textwidth]{Figs/blur/143.png}
	\caption{Motion blur phenomenon due to hand's rapid movement. Frames extracted from GH010354\_5\_17718\_19366 using Yolov3+SORT, ordered from left to right: 119, 125, 130, 131, 137 and 143.}
	\label{fig:blur}
\end{figure*}
\begin{figure*}[ht!]
	\includegraphics[width=.16\textwidth]{Figs/ShapeChange/1582.png}
	\includegraphics[width=.16\textwidth]{Figs/ShapeChange/1592.png}	\includegraphics[width=.16\textwidth]{Figs/ShapeChange/1602.png}
	\includegraphics[width=.16\textwidth]{Figs/ShapeChange/1612.png}	\includegraphics[width=.16\textwidth]{Figs/ShapeChange/1622.png}
	\includegraphics[width=.16\textwidth]{Figs/ShapeChange/1630.png}
	\caption{Shape changing illustration. Frames extracted from GH010358\_6\_10208\_11900 using MaskRCNN+DeepSORT, ordered from left to right: 1582, 1592, 1602, 1612, 1622 and 1630.}
	\label{fig:shape}
\end{figure*}
\begin{figure*}[ht!]
	\includegraphics[width=.16\textwidth]{Figs/definition/1123.png}
	\includegraphics[width=.16\textwidth]{Figs/definition/1173.png}	\includegraphics[width=.16\textwidth]{Figs/definition/1223.png}
	\includegraphics[width=.16\textwidth]{Figs/definition/1273.png}	\includegraphics[width=.16\textwidth]{Figs/definition/1323.png}
	\includegraphics[width=.16\textwidth]{Figs/definition/1406.png}
	\includegraphics[width=.16\textwidth]{Figs/definition/1407.png}
	\includegraphics[width=.16\textwidth]{Figs/definition/1408.png}	\includegraphics[width=.16\textwidth]{Figs/definition/1409.png}
	\includegraphics[width=.16\textwidth]{Figs/definition/1410.png}	\includegraphics[width=.16\textwidth]{Figs/definition/1411.png}
	\includegraphics[width=.16\textwidth]{Figs/definition/1412.png}
	\caption{The unclear "hand" definition illustration. Pay attention to the patient's left hand. Frames extracted from GH010373\_5\_1284\_2724 using FasterRCNN+SORT, ordered from left to right, up to down: (1123, 1173, 1223, 1273, 1323, 1406), (1407, 1408, 1409, 1410, 1411, 1412).}
	\label{fig:definition}
\end{figure*}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%  Copyright (c) 2016, Perry L Miller IV
%  All rights reserved.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%  Discussion about equations.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Equations}
\label{sec:discussion_equations}

All work and no play makes Jack a dull boy.
Make sure you look at equations
\ref{eq:newton_force} and
\ref{eq:einstein_energy}.
Here's a short inline equation $P_i = F_1$.
All work and no play makes Jack a dull boy.

\begin{equation}
\label{eq:newton_force}
F = m a
\end{equation}

\begin{equation}
\label{eq:einstein_energy}
e = m c ^ 2
\end{equation}


\chapter{Models}
\label{sec:chapterlabel3}

This chapter will develop a reinforcement learning agent definition to solve the recommendation problem using deep learning techniques. After describing the main motivation behind this approach, details of the baseline model derived from previous work on the DDPG algorithm is presented, and finally, as the contribution to this MSc. project, a bayesian personalised ranking model based on factorization machines is introduced into the deep learning algorithm to try to enhance the policy architecture and therefore the system recommendations.

\section{Motivation}

Previously, reinforcement learning agents have received little attention to solve the recommendation tasks due to their action domain becomes intractable in current large-scale recommender systems. However, the great performance reached in several control tasks by end-to-end RL agents based on deep neural networks \cite{mnih2015human} have demonstrated the capacity to continuously adapt their behaviour without any human intervention and solve challenging human tasks. More interestingly, a single trained general-purpose deep reinforcement agent (using the same hyperparameters) was able to converge and perform well on several task environments with different action space definitions. Consequently, a new opened research area has been using recent breakthroughs in training deep neural networks to derive RL agents models that can adapt to other kind of existing tasks under both discrete and continuous domains and solve tasks that were previously considered intractable.

Another reason that motivates our approach is the sequential nature of the recommendation process. In general, RS usually work in a sequential manner: once the system suggests items to the user, she can accept one of the recommendations. Then, at the next stage a new list of recommended items is calculated based on the user feedback and then presented again to the user. This sequential nature allows us to reformulate the recommendation process as a sequential optimization process where optimal recommendations can depend either on the previous items purchased, and in the order in which those items were purchased. For example, Zimdars et al. \cite{zimdars2001using} suggested the use of a k-order Markov chain model (with k=3) to represent this behaviour by dividing a sequence of transactions $X_1, . . . ,X_T$ into cases $(X_{t-k}, . . . ,X_{t-1},X_t$) for $t = 1, . . . ,T$. and then, build a model to predict the item at the time step t given the other columns.

As now deep reinforcement learning approaches can converge to good estimations under tasks considered intractable before, it motivates us to evaluate and improve these models under recommendation tasks, which this thesis project is focused.

\section{Baseline Agent Definition}
\label{sec:baseline}

Therefore, the recommender problem can be considered as a sequence of Markov Decision Processes $\langle \mathcal{S}, \mathcal{A}, \mathcal{R}, \mathcal{P}, \gamma \rangle$, where the action space $\mathcal{A}$ is defined as the set of items to recommend, $\mathcal{S}$ represent the current state or item a user is currently consuming, thus both $\mathcal{S}$ and $\mathcal{A}$ represent items as feature vectors $\mathbf{a}$ and $\mathbf{s} \in \mathbb{R}^n$ respectively. $\mathcal{P}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow \mathbb{R}$ is the transition probability distribution to move from state ${s_t}$ to $s_{t+1}$,  $\mathcal{R}: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ is the reward value function $\mathcal{R}$ associated with an item accepted by the user, and $\gamma \in [0, 1]$ is the discount factor for future rewards.

The reinforcement learning agent is then defined as a model-free approach with the main objective to find the optimal policy $\pi$ that maximizes the expected total future reward $\mathbb{E}[R_1]$ over all episodes $R_t = \sum^{T}_{i=t} \gamma^{i-t}r(s_i, a_i)$. The expectation can be obtained using the Bellman equation (presented in section \ref{sec:model-free}):
\begin{equation}
Q^{\pi}(\mathbf{s}, \mathbf{a}) = r(\mathbf{s}, \mathbf{a}) + \gamma \sum_{\mathbf{s}'}P(\mathbf{s}'|\mathbf{s}, \mathbf{a}) Q^{\pi}(\mathbf{s}', \pi(\mathbf{s}')
\end{equation}

In order to approximate $\pi$ and $Q$ under large-scale action spaces, standard actor-critic approaches need to define a policy by a parametrized actor function approximator $\pi_{\theta} = \arg\max_{\mathbf{a \in \mathcal{A}}} Q(\mathbf{s}, \mathbf{a}) : \mathcal{S} \rightarrow \mathcal{A}$ that would be able to generalize and that does not scales linearly in relation to the number of actions (turning the task into an intractable problem). Based on recent work presented by Lillicrap et al. \cite{lillicrap2015continuous} and Dulac-Arnold et al. \cite{Dulac-Arnold2015}, it is now possible to define generalizable reinforcement agents using a policy architecture with a sub-linear complexity relative to the action space that avoids the heavy cost of evaluating all actions. This actor-critic framework uses a multi-layer neural network as function approximators and is trained using Deep Deterministic Policy Gradient (DDPG).

The action generation function is based on the \textit{Wolpertinger} policy defined in \cite{Dulac-Arnold2015}. First, an initial function $f_{\theta^{\pi}}(s) = \hat{\mathbf{a}}$ provides a proto-action in $\mathbb{R}^n$ for a given state s. $\hat{\mathbf{a}}$ is then mapped to the discrete action set $\mathcal{A}$ by applying the k-nearest-neighbor\footnote{The size of the generated action set k is task specific, and allows for an explicit trade-off between policy quality and speed} function from equation \ref{eq:knnDDPG}, which returns the k closest actions in $\mathcal{A}$ by computing the $L_2$ distance between them.
\begin{equation}
\label{eq:knnDDPG}
g_k(\hat{\mathbf{a}}) = \arg\min^k_{a \in \mathcal{A}} | \mathbf{a} - \mathbf{\hat{\mathbf{a}}} |_2
\end{equation}

Even if $g_k$ has the same complexity as the \textit{argmax} in the Q action-value function, each step of evaluation of the $L_2$ distance is faster than a full value-function evaluation as the lookup is performed in logarithmic time. Finally, the choice of the actual action to be applied to the environment is set according to the highest-scoring action obtained according $Q_{\theta^Q}$ defined in equation \ref{eq:qDDPG}. As a result, the full Wolpertinger policy makes the algorithm significantly more robust to imperfections in the choice of action representation.
\begin{equation}
\label{eq:qDDPG}
\pi_{\theta}(s) = \arg\max_{a \in g_k \circ f_{\theta^{\pi}}(s) } Q_{\theta^Q}(s, a)
\end{equation}

On the other hand, the DDPG implementation maintains two functions: (1) the actor function $\mu(s|\theta_{\mu})$ that obtains the current policy by deterministically mapping states to specific actions; and (2) the critic function $Q(s,a)$ learned by applying the Bellman equation. It then updates the actor by applying the chain rule to the expected return from a start distribution $J = \mathbb{E}_{r_i, si \sim E, a_i \sim \pi}[R_1]$ and with respect to the actor parameters:
\begin{equation}
\label{}
\begin{aligned}
\nabla_{\theta^{\mu}}J & \approx \mathbb{E}_{s_t \sim \rho^{\beta}} [\nabla_{\theta^{\mu}} Q(s, a | \theta^Q)|_{s=s_t, a=\mu(s_t|\theta^{\mu})} ] \\
& = \mathbb{E}_{s_t \sim \rho^{\beta}} [\nabla_a Q(s, a | \theta^Q)|_{s=s_t, a=\mu(s_t)}  \nabla_{\theta^{\mu}} \mu(s|\theta^{\mu})|_{s=s_t}]
\end{aligned}
\end{equation}

In order to maintain stability in the learning algorithm, a separate target networks $\mu'(s|\theta_{\mu'})$ and $Q'(s, a|\theta_{Q'})$ for the actor and critic are used for calculating the target values, where their weight parameters are updated by using an smooth function update $\theta' \leftarrow \tau\theta + (1 - \tau)\theta'$ with $\tau \ll 1$ to main an exponential moving average of the trained parameters $\theta^{\mu}$ and $\theta^{Q}$.

Among the main advantages of using this deep learning approach, DDPG introduces an experience replay buffer of size N that helps to reduce the correlation among the sequence of samples used by the RL agent. The mechanism allows the behaviour distribution to be averaged over many of its previous states, smoothing out learning and avoiding instability or divergence in the parameters.

When executing the DDPG algorithm, the critic is trained from samples generated and stored in the replay buffer by the full policy, while the policy gradient $\nabla_a Q_{\theta^Q}(s,a)$ is taken at the true output $\hat{\mathbf{a}} = f_{\theta^{\pi}}(s)$. The target action in the Q-update is generated by the full policy.

\section{FM Agent Model}

Our implementation attempts to replace the k-nearest neighbour computation in the Wolpertinger policy for a Bayesian Personalized Ranking (BPR) model using Factorization Machines to find the top-K items from a given state to then select the item that yields to the highest Q value in the long term. The deep reinforcement agent will finally learn using the same DDPG architecture as the baseline model.

The Bayesian formulation presented by Rendle et al. in \cite{rendle2009bpr} finds the correct personalized ranking for all items $i \in I$ by maximizing the posterior probability $p(\theta | >_u) \approx p(>_u | \theta) p(\theta)$ where $\theta$ represents the parameter vector of an arbitrary model class (e.g. matrix factorization) and $>_u$ is the desired but latent preference structure for user $u$. In order to guarantee a personalized order, totality, antisymmetry and transitivity properties are fulfilled by defining the individual probability that a user really prefers item $i$ to $j$ as $p(i >_u j | \theta ) := \sigma(\hat{x}_{uij}(\theta))$ where $\sigma$ is the sigmoid function and $\hat{x}$ is an is  an  arbitrary  real-valued  function of the model parameter vector which captures the special relationship between user $u$, item $i$ and item $j$, such as Matrix Factorization of adaptive kNN.

Rendel S. showed in \cite{rendle2012factorization} that factorization models, such as the BPR model for top-K recommendation, can be expressed using Factorization Machines, where vectors $x$ are ordered by the score of $\hat{y}(x)$ and optimization is done over pairs of instance vectors $(x_i, x_j) \in \mathcal{D}$.

\subsection{Proposed Approach}

The new policy architecture called \textit{FM-policy} tries to improve how an agent selects the action with the highest Q value by evaluating the top-k returned items from a given state (i.e. the item returned by the actor function and last item consumed by the user) by a pre-trained Factorization Machine model, rather than obtaining the k-nearest neighbors using the euclidean distance. As a result, the FM-policy maintains the sublinear complexity of the Wolpertinger policy but it introduces an stronger (factorization) model. The FM-policy algorithm is described in Algorithm \ref{alg:fmpolicy}.

\begin{center}
 \begin{algorithm}[t]
    \SetAlgoLined\DontPrintSemicolon
    \caption{FM policy}
    \label{alg:fmpolicy}

    State $\mathbf{s}$ previously received from environment.\;
    $\hat{\mathbf{a}} = f_{\theta^{\pi}}(\mathbf{s})$ {Receive proto-action from actor}.\;
    $\mathcal{A}_k = g_k(\hat{\mathbf{a}}, \mathbf{s})$ {Retrieve Top-k recommendation using BPR-FM-model}\;
    $\mathbf{a}_{TOP} = \arg \max_{\mathbf{a}_j \in \mathcal{A}_k} Q_{\theta^{Q}}(\mathbf{s}, \mathbf{a}_j)$\;
%    \If{$\hat{\mathbf{a}} == \mathbf{a}_{TOP}$}{
%	$updateBPRModel(\mathbf{s}, \mathbf{a})$\;
%    }
    Apply $\mathbf{a}_{TOP}$ to environment; receive $r$, $\mathbf{s}'$\;
  \end{algorithm}
%\end{minipage}
\end{center}
After the actor function provides a proto-action $\hat{\mathbf{a}}$ in $\mathbb{R}$ given the current state or last consumed item by the user, a pre-trained BPR-FM model obtains the the Top-K recommendations based on the current observation: $(\hat{\mathbf{a}}, \mathbf{s})$. Then, the item in $\mathcal{A}_k$ that yields to the highest reward is applied to the environment.

Similarly to the baseline model, when executing the DDPG algorithm, the critic is trained from samples generated and stored in the replay buffer by the FM-policy, while the policy gradient is taken at the true output in the actor function $f_{\theta^{\pi}}(s)$. As a reference, the pseudocode of the DDPG algorithm is included in Appendix \ref{app:ddpgfm}.


\documentclass[a4paper,11pt]{article}
\usepackage{graphicx}
\usepackage{enumerate}

\begin{document}

\begin{flushright}

\vspace{1.1cm}

{\bf\Huge Problem Set 7}

\rule{0.25\linewidth}{0.5pt}

\vspace{0.5cm}
%Put Authors
Justin Ely
\linebreak
\newline
%Put Author's affiliations
\footnotesize{615.202.81.FA15 Data Structures \\}
\vspace{0.5cm}
% Date here below
20 October, 2015
\end{flushright}

\noindent\rule{\linewidth}{1.0pt}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{1a)}
\begin{itemize}
    \item The oth level (root) node has exactly 0 ancestors
    \item the first level will have exactly 1 ancestor, the root node
    \item the second level will have exactly 2 ancestors (1 node at level 1 and the root)
    \item ...
\end{itemize}

Following this trend, a node at level n will have exactly n ancestors.

\section*{1b)}
\begin{itemize}
    \item A tree with 1 leaf has 1 node
    \item A tree with 2 leaves has 3 total nodes
    \item A tree with 3 leaves has 5 nodes
    \item A tree with 4 leaves has 7 nodes
    \item ...
\end{itemize}

Following this trend, a regular binary tree with n leaves has 2n-1 total nodes.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{2)}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{3)}
For a general m-ary tree, the total number of pointers set aside will be equal to n x m.  However, each node n (apart from the root) will remove 1 null pointer from this count as the pointer will connect to it's parent.

\begin{eqnarray}
N_{null} &=& (n * m) - (n - 1) \\
                &=& (n * m) - n + 1 \\
                &=& n * (m - 1) + 1
\end{eqnarray}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{4)}
\begin{verbatim}
class Tree
    data = Array()

    def maketree(value)
        newNode = Node()

        newNode.value = value
        newNode.left = Null
        newNode.right = Null

        return newNode

    def traverse(Node)
        If not Node == null:
            traverse(Node.left)
            print Node.value
            traverse(Node.right)

    def setleft(value, parent)
        if parent.left != null:
            raise Error

        child = maketree(value)
        child.right = parent

        data[IndexOf(parent) * 2] = child

    def setright(value, parent)
        child = maketree(value)
        child.right = parent.right

        data[IndexOf(parent)*2 +1] = child

\end{verbatim}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{5a)}
\begin{verbatim}
def fib(n)
    if n = 1:
        Here.left = 0
    else:
        Here.left = fib(n-1)
        Here.right = fib(n-2)
\end{verbatim}

\section*{5b)}
No, the 1 node will only have an n-1 child, and not an n-2 child.
\section*{5c)}
An order N fibonacci tree will have fib(n-1) + fib(n-2) leaves.
\section*{5d)}
A fibonacci tree of order n has depth n.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\end{document}


\documentclass[a4paper,11pt]{article}

%\usepackage[utf8]{inputenc}
\usepackage[frenchb,english]{babel}
%\usepackage[T1]{fontenc}
%\usepackage{lmodern}
\usepackage{hyperref}
\hypersetup{colorlinks=false}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{longtable,geometry}
%\pagestyle{headings}
%\geometry{dvips,a4paper,margin=2cm}
\selectlanguage{english}

\geometry{verbose,a4paper,tmargin=30mm,bmargin=20mm,lmargin=15mm,rmargin=15mm}

\begin{document}

\title{Using the OpenFEC.org Library: a Simple User Guide}

\author{Vincent Roca (INRIA), Jonathan Detchart (INRIA), Mathieu Cunche (INRIA)\\
	Valentin Savin (CEA-LETI), J\'er\^ome Lacan (ISAE)\\
	\emph{http://openfec.org}}
\date{\today}

\maketitle
\tableofcontents

\vfill

\noindent version: \date{\verb;$Id: howto_use_openfec_lib.tex 110 2014-04-08 09:21:58Z roca $;
\newpage

\section{Introduction}
%---------------------
% \begin{flushright}
% \framebox{
% \begin{minipage}{10cm}
% 	\emph{$\rightarrow$ In short:
% 	...
% 	}
% \end{minipage}
% }
% \end{flushright}

This document is a user guide meant to facilitate the use of the OpenFEC.org library.
It is recommended for developpers that need to integrate one or several codecs within
their own application.

\subsection{Important Files}
%---------------------
% \begin{flushright}
% \framebox{
% \begin{minipage}{10cm}
% 	\emph{$\rightarrow$ In short:
% 	...
% 	}
% \end{minipage}
% }
% \end{flushright}

The various methods of the API are defined in file \verb+src/lib_common/of_openfec_api.h+.
This is the main file and \emph{the only one that needs to be included by any application}.

In addition, to accomodate code and codec specificities, additional API files are defined.
For instance, with the LDPC-staircase codec, the
\verb+src/lib_stable/ldpc_staircase/of_ldpc_staircase_api.h+
file defines additional specificities.
This file (and the similar ones for the other codecs) are automatically included.

\section{Main concepts}
%---------------------
% \begin{flushright}
% \framebox{
% \begin{minipage}{10cm}
% 	\emph{$\rightarrow$ In short:
% 	...
% 	}
% \end{minipage}
% }
% \end{flushright}

\subsection{Codec identification: of\_codec\_id}
%\subsection{Codec identification}
%---------------------

The various codecs are identified by means of a constant:
\begin{verbatim}
typedef enum
{
        OF_CODEC_NIL = 0,
        OF_CODEC_REED_SOLOMON_GF_2_8_STABLE = 1,
        OF_CODEC_REED_SOLOMON_GF_2_M_STABLE = 2,
        OF_CODEC_LDPC_STAIRCASE_STABLE = 3,
        OF_CODEC_2D_PARITY_MATRIX_STABLE = 5,
        OF_CODEC_LDPC_FROM_FILE_ADVANCED = 6,
	[...]
} of_codec_id_t;
\end{verbatim}
Since several codecs may implement the same code, for instance using different
algorithms, the constant identifies first of all a codec, which provides an
implementation of a class of codes.

\section{LDPC from file codec}
%---------------------------------------

The LDPC from file codec requires that the LDPC code be specified
in a separate file whose format is the following:

\begin{verbatim}
number of rows
number of columns
number of source symbols
number of repair symbols
line_x1 col_y1 col_y2 col_y3 ...
line_x2 col_y1 col_y2 col_y6 ...
\end{verbatim}

You only write 1 entries.

Example for a k=10, r=10 matrix :

\begin{verbatim}
 10
 20
 10
 10
     0        0         1         3         4         5        10
     1        2         3         6         7         8        10        11
     2        2         5         6         7         9        11        12
     3        3         4         6         8         9        12        13
     4        4         5         6         8         9        13        14
     5        0         1         4         5         8        14        15
     6        0         2         4         7         8        15        16
     7        0         1         3         5         7        16        17
     8        1         2         3         6         9        17        18
     9        0         1         2         7         9        18        19
\end{verbatim}

\bibliographystyle{abbrv}
\bibliography{biblio}

\end{document}