\chapter{Methodology}
\label{methods}
%\pagenumbering{arabic}
In this section, we discuss our novel framework implementation and also discuss different methods which are used in our framework. Our framework is based on three steps. In the first step, visual descriptors are extracted from videos using Improved Dense Trajectory method (IDT)~\cite{wang2013action}. Second, Gaussian Mixture Model (GMM) and Fisher Vector are used for feature representation. In the last, we jointly learn the dictionary and projection matrix via transfer learning to get the discriminative and transferable features for cross-view action recognition. The Joint Dictionary and Transfer Learning (JDTL) framework is an iterative process. First, the new projection matrix is obtained by using different transfer learning methods, then a dictionary is learned by fixing the projection matrix and coefficients, and finally coefficients are learned by fixing the projection matrix and updated dictionary.

\section{Feature Extraction}
The first step of our framework is to extract the features from videos. Improved Dense Trajectories (IDT) is a state-of-the-art method used for feature extraction in our study. In brief, IDT was introduced by Wang et al.~\cite{wang2013action} and is extended from dense trajectories~\cite{wang2013dense,bilinski2014human}. Dense Trajectory densely sample the feature points in video frame and track those detected features points in upcoming video frames, while IDT improves dense trajectories by explicitly estimating camera motion. %\\

\noindent
\textbf{Dense Trajectories (DT). }
It is difficult to determine the best scale to track feature points, and therefore dense trajectories are extracted from multiple spatial scales. Feature points are sampled in each frame of a video on a grid spaced by ${W}$ pixels. To obtain enough dense trajectories and catch significant motion information, the parameter $W$ is set to 5 pixels. Once feature points are extracted, these are tracked in each video frame in each spatial scale separately. Eight spatial scales are used, and each spatial scale is increased by a factor of  $\frac{1}{\sqrt{2}}$.

Dense optical flow field $w_{t}$ is computed from frame $I_{t}$ to the following frame $I_{t+1}$, using the Farneback algorithm~\cite{farneback2003two}. Each point $P_{t} = (x_{t}, y_{t})$ at frame $t$ is tracked to the next frame $t+1$ by median filtering in dense optical flow field $w_{t} = (u_{t}, v_{t})$, where $u_{t}$, and $v_{t}$ are the horizontal and vertical components of optical flow respectively.
\begin{equation}
\begin{aligned}
P_{t+1}\ =\ (x_{t+1},\ y_{t+1})\ =(x_{t},\ y_{t})\ +\ (M\ast\ w)\ |_{(\bar{x}_t,\ \ \bar{y}_t)}\
\end{aligned}
\end{equation}

\noindent
\textbf{DT Descriptors: Trajectory shape, HOG, HOF, and MBH. }
Wang et al.~\cite{wang2013action} proposed the trajectory shape descriptor to encode a form of the extracted dense trajectories. A sequence of displacement vectors normalized by the sum of displacement vectors magnitudes describe the trajectory shape.

In a space-time volume around a trajectory, three descriptors (HOG, HOF, and MBH) are calculated. Each local volume is subdivided into a grid with $ n_{x} \times n_{y} \times n_{t}$ spatio-temporal cells in order to embed structure information. A histogram descriptor is calculated for each cell of the grid. The histograms are normalized with $L_{2}$ norm, and then normalized cells are concatenated into the final descriptors.

Similarly, the HOG and HOF descriptors are calculated for spatio-temporal interest points. In this case, edge and optical flow orientations are quantized into 8 bins using full orientations, with the HOF descriptor having an additional zero bin. The example of HOG is shown in Figure~\ref{fig:HOG-Example}, where in (a) the image is the input image, (b) input image with extracted HOG features at different locations on input image, and in (c) extracted HOG features from input image are illustrated.

\begin{figure}[!ht]
	\centering
	\includegraphics[width=.55\textheight]{figures/HOG-final2}
	\linebreak
	\caption{(a) Input image. (b) HOG features extracted from the image at different locations. (c) HOG features illustration.}
	\label{fig:HOG-Example}
\end{figure}

Dalal et al.~\cite{dalal2006human} proposed the Motion Boundary Histogram (MBH) descriptor to detect human. Optical flow field $I_{w} = (I_{x}, I_{y})$ is separated into its $x$ and $y$ components. Spatial derivatives for the horizontal and vertical components of the optical flow field are computed separately, and the orientation information is quantized into histograms. Background motion is eliminated by suppressing the constant motion information and keeping the information about changes in the flow field.

IDT boosts the recognition performance of dense trajectories by taking camera motion into account.\ It assumes that background motion of two consecutive frames can be characterized by a homography matrix.\ To estimate the homography matrix, the first step is to find the correspondence between two consecutive frames. They resort to SURF~\cite{bay2008} feature matching and optical flow based matching, as these two kinds of matching schemes are complementary to each other. Then RANSAC~\cite{fischler1981random} algorithm is used to estimate the homography matrix.\ Based on the homography matrix, they rectify the frame image to remove camera motion and re-calculate the optical flow called \textit{wrapped flow}. There are two major advantages for dense trajectories to cancel out camera motion from optical flow. First, this brings advantages to the descriptor calculated from optical flows, in particular for HOF. Second, trajectories generated due to camera motion can be removed.

It describes the deviation of orignal set of features from average distribution of features modeled by parametric generative model,
\section{Feature Representation}
Once local features are extracted, these are used to represent actions in videos. As the length of each video is different, we refer to \textbf{Fisher Vector (FV)}, one of most efficient vector quantization method to represent each video and achieve a fixed length vector~\cite{perronnin2010improving,perronnin2007fisher}. FV is an extension of bag-of-visual word representation and its encoding represents differences between features and visual words. FV has achieved remarkable results as a global descriptor both for image classification and for image retrieval, outperforming the conventional bag-of-features approach.
Given a large set of vectors, the FV method assembles them into a high dimensional representation.
So, in the first stage diagonal covariance was used to train the Gaussian Mixture Model (GMM), and derivatives are considered with respect to gaussian mean and variance~\cite{simonyan2013fisher}. It gives the representation, which captures the average first and second order differences between the original feature and each of the GMM centers:
\begin{equation}
\begin{aligned}
\Phi_k^{(1)} & = \frac{1}{N  \sqrt[]{W_k}} \sum_{p=1}^{N}\ \alpha_p(k)\Bigg(\frac{x_p -\mu_k}{\sigma_k}\Bigg), \\
\Phi_k^{(2)} & = \frac{1}{N \sqrt[]{2W_k}} \sum_{p=1}^{N}\ \alpha_p(k)\Bigg(\frac{(x_p -\mu_k)^2}{\sigma_k^2} -1  \Bigg),
\end{aligned}
\end{equation}
where $\{w_k, \mu_k, \sigma_k\}_k$ are mixture weights, means, and diagonal covariances of GMM.  $\alpha_p(k)$ is the soft assignment weight of the $p$-th feature $x_p$ to the $k$-th Gaussian. An FV $\Phi$ is obtained by stacking the differences: $ \Phi = [{\Phi_1^{(1)},\Phi_1^{(2)},.....,\Phi_K^{(1)},\Phi_K^{(2)} }]$. The final encoded vector shows the difference between distribution of video features and distribution fitted to the features of all training videos.
\begin{table}[h]
	\centering
	\caption{Details of the variables used in our method}
	\begin{tabular}{@{\extracolsep{12pt}}|c|l|}
		\hline
		\textbf{Variable} & \textbf{Description} \\ \hline
		$P$	&  Projection Matrix           \\ \hline
		$Y$	&  Training Sample            \\ \hline
		$D_0$	& Shared Dictionary             \\ \hline
		$D_j$	& Class-specific Dictionary            \\ \hline
		$\bar{D}$	& Total Dictionary that includes class-specific and shared dictionary            \\ \hline
		$X_0$	& Shared Coefficients            \\ \hline
		$X_c$	& Class-Specific Coefficients             \\ \hline
		$M$	&  Each Dictionary Column Mean Vector            \\ \hline
		$M_0$	&  Shared Coefficient Mean Vector            \\ \hline
		$M_c$	& Class-specific Coefficient Mean Vector             \\ \hline

		$J$	& Input Data Matrix (Transfer Learning)             \\ \hline
		%$D_s$    &  Source Domain            \\ \hline
		%$D_t$	 & Target Domain            \\ \hline
		$J_s$	 & Source Task  \\ \hline
		$J_t$  	&  Target Task  \\ \hline
		$L_t$		& Target Label            \\ \hline
		$L_s$		& Source Label            \\ \hline
		$I$	& Identity Matrix             \\ \hline
		%$m,c$	& Shared Feature/Classes             \\ \hline
		$M_d$	& Maximum Mean Discrepancy (MMD) Matrix             \\ \hline
		$M_{d_c}$ & Maximum Mean Discrepancy (MMD) Matrices involving class labels \\ \hline
		$K$	& Input Kernel Parameter             \\ \hline
		%$\phi$	& Feature Map induced by Kernel             \\ \hline
		$\mathcal{P}$	&  Marginal Distribution           \\ \hline
		$\mathcal{Q}$	&  Conditional function           \\ \hline
		$H$	&  Centering Matrix           \\ \hline
		$\triangleq$ & equal by definition \\ \hline
		$\mu ,\lambda$	&  Trade-off Parameter/Regularize parameter\\ \hline
	\end{tabular}
	\label{tb:TLVariable}
\end{table}
\noindent
\section{Transfer Learning}
Transfer learning is employed in this chapter to map the source (one view) and target (another view) features into a common subspace. The transfer learning methods are categorized in supervised, semi-supervised and unsupervised as we described in related work, and we detail four methods below which will be incorporated into the joint learning framework. The variables and  basic formulation in this chapter can be found in Table~\ref{tb:TLVariable} and~\ref{table:tcl}.
\begin{table}[h]
	\footnotesize
	%\large
	\centering
	\caption{Transfer learning methods}
	%	\resizebox{\textwidth}{!} {%
	\begin{tabular}{|m{2cm}|m{9cm}|m{2.9cm}|}
		\hline
		Method & Learning Objective & Constraints  \\
		\hline
		Transfer Component Analysis&  $\min_W \textbf{tr}(W^\T W) + \mu \ \textbf{tr}(W^\T KLKW)$ &$W^\T KHKW = I$\\ \hline
		Joint Distribution Adaptation&$\min\ \sum_{C}^{c=0} \textbf{tr}(J^\T JM_{d_c}J^\T A) \ + \ \lambda ||A||_F^2 $ & $A^\T JHJ^\T A= I$           \\ \hline
		Balance Distribution Adaptation&  $\min \  \textbf{tr} \bigg(A^\T J \bigg((1-\mu )M_{d_0} \ + \mu \sum_{c=1}^{C}M_{d_c}\bigg)J^\T A \bigg)\ +\lambda||A||_F^2$&$A^\T JHJ^\T A= I$ , $0\leq \mu \leq 1$         \\ \hline
		Transfer Joint Matching & $\min \textbf{tr}(A^\T KM_dK^\T A)\ +\lambda \big(||A_s||_{2,1} + ||A_t||_F^2 \big)$ &$A^\T KHK^\T A= I $           \\  \hline
		%\hline
	\end{tabular}%
	%	}
	\label{table:tcl}
\end{table}


%https://www.sharelatex.com/project/55c47dd369b600da349c3e75
\documentclass[12pt, a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage{mdwlist}
\usepackage{enumerate}
\usepackage{paralist}
\usepackage{array}
\usepackage{mathtools}
\usepackage{longtable}

\usepackage[numbers,square]{natbib} %% CUSTOM[square]
\usepackage[utf8]{inputenc} %% CUSTOM (instead of latin1)
\usepackage{amsfonts} %% CUSTOM
\usepackage{amssymb} %% CUSTOM
\usepackage{tabulary} %% CUSTOM
\usepackage{multirow} %% CUSTOM
\usepackage{dcolumn} %% CUSTOM
\usepackage{cprotect} %% CUSTOM
\usepackage[hang,bottom]{footmisc} %%
\usepackage{subcaption}
\usepackage{hyperref}

\usepackage{multirow}

%% use colors
\usepackage{color}

%% make fancy math
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{textcomp}
\usepackage{yhmath} % fr die adots

%http://tex.stackexchange.com/questions/128050/add-equation-name-underneath-equation-number
\newcommand{\eqname}[1]{\\[-\baselineskip] \tag*{#1}}% Tag equation with name
\newcommand{\eqnameH}[1]{\\[-0.3\baselineskip] \tag*{#1}}% Tag equation with name
\newcommand{\eqnameT}[1]{\\[-0.3\baselineskip] \tag*{#1}}% Tag equation with name

\newcommand{\itemref}[1]{\item[\ref{#1}]{\makebox[6.5cm][l]{\nameref{#1}}}}
\newcommand{\clabel}[2]{\caption{#2}\label{#1}}
\newcommand{\eqlabelname}[2]{\label{eq:#1} \\[-\baselineskip] \tag*{#2} \label{eqname:#1}}

\title{Multiview ICP}
\author{Adrian Haarbach}
\date{August 2015}

\begin{document}

\maketitle
The goal of this project was to solve the multiview rigid point cloud registration problem using multiview levenberg-marquardt ICP implemented with Ceres solver.

\section{ICP}
The Iterative Closest Points algorithm (ICP) \cite{besl1992method,chen1991object,zhang1994iterative} is a two-step iterative algorithm to rigidly align two point clouds. The following two steps are applied in alternation until convergence:
\begin{enumerate}
	\item compute closest point \textbf{correspondences} $\{p_i \rightarrow q_i\}_1^N$ between the two clouds
	\item update the current \textbf{transformation} estimate $g \in SE(3)$ so that it minimizes a cost function defined on these correspondences.
\end{enumerate}
In the following, we concentrate on step 2, meaning the update of the transformation estimate while leaving the correspondences fixed, and extend it to the multiview setting.

\section{Pairwise registration}
Let $p_i \leftrightarrow q_i ( p_i, q_i\in \mathbb{R}^3) $ be a set of point correspondences. The goal is to find a rigid body motion $g \in SE(3)$ that minimizes the following error:
\begin{equation}
	E(g) = \sum_{i=1}^N l(||d(g(p_i),q_i)||^2)
	\label{eq:icp}
\end{equation}
Where $d$ is a CostFunction while $l$ is a LossFunction.
The solution $g = argmin(E)$ is the relative transformation that aligns the model cloud (the $p_i$) to the data cloud (the $q_i$) in a least squares sense.

\section{Multiview registration}
In the multiview setting, the roles of model and data cloud are no longer fixed, since one cloud can take on the roles of both of them. Let $C_1, \dots C_M$ be the set of point clouds that are brought to be in alignment.
%In general, it doesn't make sense to try to align each point cloud with each other, since normally one frame only overlaps with some of its neighbors, but not with a shot taken from the other side of the object.
%Remember the alignment error from the pairwise setting,
To generalize and formalize the notation of which cloud gets registered to which other cloud, we can encode these relations as a directed graph, with the adjacency matrix $A \in \{0,1\}^{M \times M}$, such that $A(h,k) = 1$ if cloud $C_h$ can be registered to cloud $C_k$. Let $g_1, \dots g_M$ be the absolut camera poses of each view in the global reference frame. The alignment error between two clouds $C_h$ and $C_k$ then is:
\begin{equation}
	E(g_h, g_k) = A(h,k) \sum_{i=1}^{N_h} l(||d(g_h(p_i^h),g_k(q_i^h))||^2) \label{eq:mvlmicp-pair}
\end{equation}
where $\{p_i^h \rightarrow q_i^h\}$ are the $N_h$ closest point correspondences obtained from the clouds $C_h$ and $C_k$.
The pairwise formulation \eqref{eq:icp} can be obtained by setting $g=g_k * g_h^{-1}$. The overall alignment error, which we want to minimize at this stage, is obtained by summing up the contribution of every pair of overlapping views:
\begin{equation}
E(g_1, ..., g_M) = \sum_{h=1}^M \sum_{k=1}^M A(h,k) \sum_{i=1}^{N_h} l(||d(g_h(p_i^h),g_k(q_i^h))||^2) \label{eq:mvlmicp-general}
\end{equation}
The solutions $g_1, ..., g_M = argmin(E)$ are the absolute camera poses that align the $M$ clouds in a least squares sense. In contrast to the pairwise registration error \eqref{eq:icp}, which has closed form solutions for the relative transformation $g$ that aligns the two clouds, there are no closed form solutions in the multiview setting. However, rigid point cloud registration is just an instance of Non-linear least squares optimization, which can be solved efficiently using Ceres Solver \footnote{\url{http://ceres-solver.org/nnls_tutorial.html}}.

\subsection{Common ICP Cost Functions}
\subsubsection{Pairwise}
The most prominent cost functions one minimizes using closed form solutions in the pairwise setting are:

\begin{align}
E(g) = \sum_{i=1}^N || R \cdot p_i + t - q_i ||^2 \label{eq:pointToPoint} \eqname{point to point \cite{besl1992method,zhang1994iterative}} \label{eqname:pointToPoint}
\end{align}
\begin{gather}
E(g) = \sum_{i=1}^N || (R \cdot p_i + t - q_i)^T n_{q_i} || ^2 \label{eq:pointToPlane}
\eqname{point to plane \cite{chen1991object}} \label{eqname:pointToPlane}
\end{gather}
where $R \in SO(3), t \in \mathbb{R}^3$, so $g=(R,t) \in SE(3)$. In these cases, the loss function $l$ is just the identity, and the cost functions $d$ are
\begin{align}
	d_{point-point}(g,p,q) & = R \cdot p + t - q  \in \mathbb{R}^3 \label{dist:pointToPoint} \\
	d_{point-plane}(g,p,q,n_q) & = (R \cdot p + t - q)^T n_q \in \mathbb{R}^1 \label{dist:pointToPlane}
\end{align}
Even though there are closed form solutions in the pairwise setting for the above cost functions \cite{eggert1997estimating,low2004linear}, one can also use Ceres to solve the pairwise error \eqref{eq:icp} yielding a relative pose update g. Note that the evaluations of the cost functions, called the residuals, don't have to be in $\mathbb{R}^3$.

\subsubsection{Multiview}
The multiview setting needs only a slight modification of above cost functions. In the pairwise setting, $d$ depends on the relative pose $g$. In the multiview setting, $d$ depends on the two absolute poses $g_h=(R_h,t_h)$ and $g_k=(R_k,t_k)$.
\begin{align}
	d_{point-point}(g_h,g_k,p,q) & = (R_h \cdot p + t_h ) - (R_k \cdot q + t_k )  \label{dist:pointToPointGlobal} \\
	d_{point-plane}(g_h,g_k,p,q,n_q) & = d_{point-point}(g_h,g_k,p,q)^T ( R_k \cdot n_q)  \label{dist:pointToPlaneGlobal}
\end{align}

With this slight modification of the cost functions, we can model \eqref{eq:mvlmicp-general} using Ceres and let it solve for $g_1, ..., g_M$. Notice that in the point-to plane case, the normals of the points in the destination cloud are only rotated, not translated, since they represent directions rather than actual points.

\section{Parametrization of rigid body motions}
In the above, we just wrote $g=(R,t)$ and especially assumed R to be a 3x3 rotation matrix $R \in SO(3)$, which must fulfill $RR^T=I$ and $|R|=1$. If used in an iterative optimization algorithm, one has to optimize over 9 parameters constrained to above orthogonality conditions, while only having 3 degrees of freedom. It is much more efficient to optimize over minimal representations of rotations such as AngleAxis, Unit Quaternions or the Lie Algebra representation.

\subsection{Angle Axis}
According to Euler's rotation theorem, any rotation in 3D space can be expressed as a single rotation around some axis by a certain angle. The angle-axis (or axis-angle) representation of a rotation parametrizes a rotation by a unit vector $n \in \mathbb{R}^3$ as the axis of the rotation and an angle $\phi$ describing the magnitude of the rotation around the axis.\footnote{\url{https://en.wikipedia.org/wiki/Axis\%E2\%80\%93angle_representation}}

Furthermore, the angle of rotation $\theta$ can be absorbed into the norm of the unit axis vector $n$, so that we only have 3 parameters for the 3 DoF: $\omega = n * \theta$. In the theory of three-dimensional rotation, Rodrigues' rotation formula, named after Olinde Rodrigues, is an efficient algorithm for rotating a vector in space, given an axis and angle of rotation.\footnote{\url{https://en.wikipedia.org/wiki/Rodrigues\%27_rotation_formula}}

Ceres AngleAxisRotatePoint function\footnote{\url{https://github.com/ceres-solver/ceres-solver/blob/master/include/ceres/rotation.h\#L566}} uses Rodrigues rotation formula to directly rotate a point. One first has to recover the angle and the unit axis from the minimal representation, the vector $\omega \in \mathbb{R}^3$, which represents rotational velocity, by $ \phi = ||\omega||,  n=\frac{\omega}{\phi}$.

\begin{equation}
p_{rot} = p \cos{\phi} + (n \times p) \sin{\phi} + n(n^Tp)(1 - \cos{\phi})
\end{equation}

Another way is to first convert from the Angle-Axis formulation to a rotation matrix $R$, and then rotate the point by matrix - vector multiplication. This second form of the Rodrigues rotation formula can be derived from the first one.
\[
R  = I + \sin{\phi} [n]_x + (1-\cos\phi) 	[n]_x^2
\]
This is closely related to the Lie Algebra representation, where we also define the $[.]_x$ cross product matrix operator \eqref{eq:crossmatrix}.

\subsubsection{Cost function}
With the parametrization of $g$ as $(w,t)$ and $w,t \in \mathbb{R}^3$, we just replace all occurrences of $R \cdot p$ (or q) in the cost functions \eqref{dist:pointToPoint}, \eqref{dist:pointToPlane}, \eqref{dist:pointToPointGlobal}, \eqref{dist:pointToPlaneGlobal} with $p_{rot}$. The translational component is still represented as a 3-dimensional vector.

\subsubsection{Jacobian}
We rely on Ceres AutoDiffCostFunction functionality to automatically compute the necesarry derivatives of our Cost function parametrized by the angle axis representation and a translation vector.

\subsection{Unit Quaternions}
Yet another way to represent rotations are unit quaternions.

A quaternion $q$ is basically the extension of a complex number $c=a+bi, i^2 = -1$ from 2 to 4 dimensions: $q = w + xi + yj + zk, i^2 = j^2 = k^2 = ijk = -1$. Just as complex numbers can be used to represent rotations in $\mathbb{R}^2$ (remember Euler's formula $\mathrm{e}^{\mathrm{i}\,\varphi} = \cos\left(\varphi \right) + \mathrm{i}\,\sin\left( \varphi\right)$ ), quaternions can be used to represent rotations in $\mathbb{R}^3$.

Formally, a quaternion $q \in \mathbb{H}$ may be represented by a vector $q = [q_w,q_x,q_y,q_z]^T = [q_w, q_{x:z}]^T$ together with the definitions:
\begin{align*}
	\text{adjoint}: & \quad \bar{q}= [q_w, -q_{x:z}]^T \\
	\text{norm}:	& \quad ||q|| = \sqrt{q_w^2+q_x^2+q_y^2+q_z^2} \\
	\text{inverse}:	& \quad q^{-1} = \frac{\bar{q}}{||q||}
\end{align*}

A unit quaternion is a quaternion with unity norm, $||q||=1$ and can be used to represent the orientation of a rigid body in 3D Euclidean space. Specifically, a unit quaternion can be retrieved from the axis-angle representation, with a rotation $\phi$ about the normalized rotation axis $n, ||n||=1$ via
\[
q(\phi,n)=[\cos(0.5 \phi) , n \sin(0.5 \phi)]^T
\]
Moreover, there are closed form solutions for converting a unit quaternion into a rotation matrix as well as the other way around which we will use extensively. Since they don't look as neat as the previous formula, we refer to \cite{diebel2006representing} for the details.

Even though Ceres comes with an implementation of unit quaternions in the same file as AngleAxisRotatePoint, we instead used Eigen's quaternions \footnote{\url{http://eigen.tuxfamily.org/dox/classEigen_1_1Quaternion.html}}, which allows for much nicer syntax due to operator overloading. It is important to note that the storage order of the two differ: it is  $[x, y, z, w]$ for eigen quaternions, but  $[w, x, y, z]$ for the ones built into ceres.

\subsubsection{Local Parametrization}
Unit Quaternions are not a minimal representation, they have 4 components. Nevertheless, the 4th component, typically the scalar part $w$, can be recovered from the other three because the whole 4-vector must have unit norm.

This is why Ceres gives you a 3-dimensional incremental update step $\delta$ that needs to be applied to the current 4-dimensional unit quaternion. This has to be done in a class extending Local Parametrization which must implement the Plus function. For Ceres in-built quaternions, this is implemented in the \href{https://github.com/ceres-solver/ceres-solver/blob/master/internal/ceres/local_parameterization.cc#L157}{QuaternionParameterization::Plus function}. For our Eigen Quaternion, we copied the Plus function but adapted/permuted the indices to the different storage order:

\[
Plus(q, \delta) = [sin(|\delta|) \delta / |\delta|, cos(|\delta|)] * q
\]
with * being the quaternion multiplication operator. Here we assume that the last element of the quaternion vector is the real (cos theta) part ($[x, y, z, w]$ Eigen quaternion storage order).

Since we did not yet implement automatic differentiation of this local parameterization (as for our Lie Algebra using the Sophus Library), one also has to provide the correct Jacobian. For Eigen Quaternions storage order they look as follows:

\begin{equation}
J = \frac{\partial q}{\partial \delta} =
\begin{bmatrix}
 w &  z & -y \\
-z &  w &  x \\
 y & -x &  w \\
-x & -y & -z \\
\end{bmatrix}
\label{eq:quaternionJacobian}
\end{equation}

\subsubsection{Cost function}
With the parametrization of $g$ as $(q,t)$ and $q \in \mathbb{R}^4, t \in \mathbb{R}^3$, we just replace all occurrences of $R \cdot p$ (or q) in the cost functions \eqref{dist:pointToPoint}, \eqref{dist:pointToPlane}, \eqref{dist:pointToPointGlobal}, \eqref{dist:pointToPlaneGlobal} with either a direct quaternion rotation (Eigen storage order $p = [x, y, z, w]$):
\[
[p_{rot},0]^T = q \cdot [p,0] \cdot q^{-1}
\]
or by first converting the quaternion to a Rotation matrix R as in \cite{diebel2006representing}.
The translational component is still represented as a 3-dimensional vector.

\subsubsection{Jacobian}
In Ceres, it is possible to mix the analytic derivatives for our Local Parametrization as stated above with automatic derivatives for the complete cost function. This drastically lowered the complexity, while still providing decent efficiency, since we were able to use only well tested analytic Jacobians for the Local Parametrization, but did not have to compute analytic Jacobians for the complete cost function.

\subsection{Lie Algebra of Twists}
Each rigid body transformation matrix T in the Lie group $T \in$ SE(3) has a minimal representation as a twist $\hat\xi$ in its associated Lie algebra $\hat\xi \in se(3)$. Each twist is uniquely defined by its twist coordinates $\xi \in \mathbb{R}^6$:
\[
\xi = (\xi_1, \dots, \xi_6)^T = (u_1, u_2, u_3,\omega_1, \omega_2, \omega_3)^T = (u^T \omega^T)^T
\]
where $u$ represents the translational velocity and $\omega$ the rotational velocity.

Let us first define the operator $[.]_x$, which is an isomorphism between $\mathbb{R}^3$ and the space $so(3)$ of all 3x3 skew symmetric matrices ($[\omega]_x = -[\omega]_x^T$).
\begin{equation}
[.]_x : \mathbb{R}^3 \rightarrow so(3) \subset \mathbb{R}^{3 \times 3} ;
[\omega]_x =
\begin{bmatrix}
 0 & -\omega_3 & \omega_2\\
 \omega_3 & 0  & -\omega_1\\
 -\omega_2 & \omega_1 & 0
\end{bmatrix}
\label{eq:crossmatrix}
\end{equation}
It allows to express the cross product $p \times q$ with $p,q \in \mathbb{R}^3$ as a matrix-vector multiplication: $ p \times q = [p]_x * q$. Furthermore, it allows to convert from twist coordinates to a twist using the hat operator:
\[
\wedge : \mathbb{R}^6 \rightarrow se(3) \subset \mathbb{R}^{4 \times 4} ;
\hat\xi =
\xi^\wedge =
\begin{bmatrix}
 u \\
 \omega \\
\end{bmatrix}^\wedge
=
\begin{bmatrix}
 [\omega]_x & u\\
 0 & 0\\
\end{bmatrix}
\]
The mapping from the twist in the Lie algebra to the transformation matrix in the Lie group is done by matrix exponentiation:
\begin{equation}
\label{eq:lieexp}
\begin{aligned}
exp : se(3) &\rightarrow SE(3) \\
exp(\hat\xi) = %e^{\hat\xi} =
exp\left(
\begin{bmatrix}
 [\omega]_x & u\\
 0 & 0\\
\end{bmatrix}
\right)
&=
\begin{bmatrix}
 e^{[\omega]_x} & Vu\\
 0 & 1\\
\end{bmatrix}
=
\begin{bmatrix}
 R & t\\
 0 & 1\\
\end{bmatrix}
\end{aligned}
\end{equation}
We can convert the vector $\omega \in \mathbb{R}^3$, which represents rotational velocity, into an axis angle representation by $ \phi = ||\omega||,  n=\frac{\omega}{\phi}$, to obtain a closed form solution for the Taylor series expansion $e^{[w]_x} = \sum_{i=0}^\infty \frac{[\omega]_x^i}{!i}$ using Rodrigues' rotation formula:
\[
e^{[w]_x} = e^{[n]_x\phi} = I + 	sin\phi [n]_x + (1-cos\phi) 	[n]_x^2
\]
and similarly for $V$
\[
V = I + \frac{1-cos\phi}{\phi}[n]_x + \frac{\phi-sin\phi}{\phi} [n]_x^2
\]
To get from twist coordinates $\xi \in \mathbb{R}^6$ to a transformation matrix $T \in SE(3) \subset \mathbb{R}^{4\times4}$ one first has to apply the hat operator $\wedge$ and then the exponential map $exp$. The other direction is possible as well, by first applying the inverse of the exponential map, called the logarithmic map $log$, and then the inverse of the hat operator, called the vee operator $\vee$. For our application however, we only need the forward direction, which is why just summarize the different operators here:

\begin{displaymath}
\xi \in \mathbb{R}^6 \xrightleftharpoons[\vee (vee)]{\wedge (hat) } \hat{\xi} \in se(3) \xrightleftharpoons[log]{exp} T \in SE(3)
\end{displaymath}

Every Lie group, such as SO(3) and SE(3), is a group that is also a smooth manifold, with the property that the group operations of multiplication and inversion are smooth maps. They can locally be approximated by their corresponding Lie algebras $so(3)$ and $se(3)$, which form the tangent space of the group at the identity. This allows one to do calculus on the elements of the Lie algebra, such as calculating derivatives, which we will need for numerical minimization.

%The whole reason why we need the Lie algebra of twists $se(3)$ is that in numerical minimization algorithms, we iterate towards the minimum by applying a series of small updates additively to our pose parameterization, but our corresponding Lie group $SE(3)$ only has the multiplication operator. Let us now derive the Jacobian of th

\subsubsection{Cost function}
With the parametrization of $g$ by its twist coordinates $\xi \in \mathbb{R}^6$, we have a truly minimal representation of both rotation and translation in just one 6 dimensional vector.

In the Lie algebra, a point $p \in \mathbb{R}^3$ is transformed (rotated and translated) to the point $y$ by a twist via \cite{ma2005invitation}[p.32] :
\[
\begin{bmatrix}
 y\\
 0\\
\end{bmatrix}
=
\begin{bmatrix}
 [\omega]_x & u\\
 0 & 0\\
\end{bmatrix}
\begin{bmatrix}
 p\\
 1\\
\end{bmatrix}
=
\begin{bmatrix}
   		   &  -\omega_3 p_2  & +\omega_2 p_3 	& + u_1\\
\;\;\omega_3 p_1 &                &  -\omega_1 p_3 	& + u_2 \\
-\omega_2 p_1 &	+\omega_1 p_2	   &     			& + u_3 \\
               & 			   & 0  &
\end{bmatrix}
\Leftrightarrow
y = w \times p + u
\]

\subsubsection{Jacobian}
The Jacobian of the twist $\hat\xi$ is calculated by partial derivates of $y=[y_1,y_2,y_3]^T$ and the twist coordinates $\xi = (u_1, u_2, u_3,\omega_1, \omega_2, \omega_3)^T$ \cite{slavcheva2015unified} :
\begin{equation}
J = \frac{\partial y}{\partial \xi} =
\begin{bmatrix}
\frac{\partial{y_1}}{\partial{\xi_1}} & \dots & \dots & \frac{\partial{y_1}}{\partial{\xi_6}}	\\
\vdots 								 &      & & \vdots 							   		\\
\frac{\partial{y_3}}{\partial{\xi_1}} & \dots & \dots & \frac{\partial{y_3}}{\partial{\xi_6}}
\end{bmatrix}
=
\begin{bmatrix}
1 & 0 & 0 &   0  & p_3  & -p_2 \\
0 & 1 & 0 & -p_3 &   0  &  p_1 \\
0 & 0 & 1 &  p_2 & -p_1 &  0   \\
\end{bmatrix}
\label{eq:lieJacobian}
\end{equation}
\medskip

\bibliographystyle{unsrt}%Used BibTeX style is unsrt
\bibliography{references}

\end{document}


\RequirePackage[l2tabu, orthodox]{nag}
\documentclass{article}

\usepackage[letterpaper, margin=1.3cm]{geometry}
\usepackage{siunitx}
\usepackage{multicol}
\usepackage{pgfplots}
\usepackage{mathtools}
\usepackage{amssymb}
\usepackage{mathrsfs}
\usepackage{graphicx}
\usepackage{epstopdf}
\usepackage{float}
\usepackage{minted}
\usepackage{pdflscape}
\usepackage{subcaption}

\pgfplotsset{compat=1.16}

\begin{document}

\begin{titlepage}
     \begin{center}
          \vspace*{1cm}

          \textbf{Lab 1}

          \vspace{0.5cm}

          \Large{Introduction to Discrete-time Signals and Systems}
          \vspace{1.5cm}

          \textbf{Michael Kwok (1548454)}

          \vfill
          ECE 340 Lab Discrete Time Signals and Systems\\
          Department of Electrical and Computer Engineering\\
          University of Alberta\\
          30 September 2020
     \end{center}
\end{titlepage}
\begin{multicols*}{2}
     \section{Signal Generation and Plotting}
     2 signals were generated in this part of the lab, with the 2nd signal getting split into the real and imaginary part.
     \subsection{Signal plot}
     \begin{figure}[H]
          \centering
          \includegraphics[width=\linewidth]{siggenres}
          \caption{Question 1a and b}
     \end{figure}
     \subsection{Periodicity of signals}
     In this lab, signal \(x_1\) is periodic while signal \(x_2\) is not periodic as \(x_1\) is linearly composed of two sinusoid signals while \(x_2\) is a damped sinusoid, which by definition is not periodic.

     Calculation of period:
     \begin{align*}
           & \omega_{\sin} = 0.1\pi, & f_{\sin} & = 0.05, & T_{\sin} = 20 \\
           & \omega_{\cos} = 0.4\pi, & f_{\cos} & = 0.2,  & T_{\cos} = 5
     \end{align*}
     Find LCM of  $T_{\sin}$ and  $T_{\cos}$ for base period
     \[LCM(T_{\sin}, T_{\cos})= 20\]
     Period of \(x_1\) is 20
     \subsection{Energy of signals}
     \begin{itemize}
          \item Energy of \(x_1\): \(698.5335\)
          \item Energy of \(x_2\): \(5.2632\)
     \end{itemize}

     \section{Audio Processing}
     \subsection{Song lyrics}
     Ho ey Ho ey baila baila comigo baila baila mi amor.
     \subsection{Audio signal}
     From the Matlab calculated result of 1000000 \(\times \) 1 which is a total of 1000000 samples in the audio file, which is almost 23 seconds \( \frac{1000000}{44100} \approx 22.68\). Screenshot of stem plot in appendix.
     \section{Image Processing}

     The image is 512 \(\times \) 512

     \begin{figure}[H]
          \centering
          \begin{subfigure}{0.4\linewidth}
               \centering
               \includegraphics[width=\linewidth]{lena}
          \end{subfigure}
          \begin{subfigure}{0.4\linewidth}
               \centering
               \includegraphics[width=\linewidth]{lena_bright}
          \end{subfigure}
          \caption{lena vs lena\_bright}
     \end{figure}

     \section{Audio Processing}
     \subsection{Difference equation}
     \(y(k) = a y(k-1) + x(k)\)
     \subsection{Calculation}
     \begin{enumerate}
          \item a = 0.5; \begin{align*}
                     y[0] & = 0 + 1                 \\
                          & = 1                     \\
                     y[1] & = 0.5 \cdot y[0] + x[1] \\
                          & = 1.5                   \\
                     y[2] & = 0.5 \cdot y[1] + x[2] \\
                          & = 1.75                  \\
                     y[3] & = 0.5 \cdot y[2] + x[3] \\
                          & = 1.875                 \\
                     y[4] & = 0.5 \cdot y[3] + x[4] \\
                          & = 1.9375                \\
                     y[5] & = 0.5 \cdot y[4] + x[5] \\
                          & = 1.96875
                \end{align*}
                \begin{figure}[H]
                     \centering
                     \includegraphics[width=\linewidth]{a5}
                     \caption{Graph at \(a=0.5\)}
                \end{figure}
          \item a = 2; \begin{align*}
                     y[0] & = 0 + 1               \\
                          & = 1                   \\
                     y[1] & = 2 \cdot y[0] + x[1] \\
                          & = 3                   \\
                     y[2] & = 2 \cdot y[1] + x[2] \\
                          & = 7                   \\
                     y[3] & = 2 \cdot y[2] + x[3] \\
                          & = 15                  \\
                     y[4] & = 2 \cdot y[3] + x[4] \\
                          & = 31                  \\
                     y[5] & = 2 \cdot y[4] + x[5] \\
                          & = 65
                \end{align*}
                \begin{figure}[H]
                     \centering
                     \includegraphics[width=\linewidth]{a2}
                     \caption{Graph at \(a=2\)}
                \end{figure}
          \item Another stable a = 0.1;
                \begin{figure}[H]
                     \centering
                     \includegraphics[width=\linewidth]{a1}
                     \caption{Graph at \(a=0.1\)}
                \end{figure}
     \end{enumerate}
     Any value \(|a| < 1\) will be stable. This is implied by the difference equation. A value \(|a| <1\) takes a fraction of the previous output before adding it to the current input. Eventually it reaches an unchanging value.

\end{multicols*}
\appendix
\begin{landscape}
     \section{Audio Plot}
     \begin{figure}[H]
          \centering
          \includegraphics{baila}
          \caption{Audio plot}
     \end{figure}
\end{landscape}
\section{Main Code}
\inputminted{Matlab}{Lab1.m}
\section{sysresp Code}
\inputminted{Matlab}{sysresp.m}
\end{document}