\documentclass[twocolumn,showpacs,preprintnumbers,amsmath,amssymb]{revtex4}
%\documentclass[preprint,showpacs,preprintnumbers,amsmath,amssymb]{revtex4}

% Some other (several out of many) possibilities
%\documentclass[preprint,aps]{revtex4}
%\documentclass[preprint,aps,draft]{revtex4}
%\documentclass[prb]{revtex4}% Physical Review Letter

\begin{document}

\preprint{ }

\title{ 6-dimensional Kaluza-Klein Theory for Basic Quantum Particles and Electron-Photon Interaction}
\author{\normalsize Xiaodong Chen}
\altaffiliation{2073B Vestavia Park Ct, Birmingham, AL 35216, U.S.A.}
\email{xiaodong.chen@gmail.com}

\date{\today}

\begin{abstract}
By extending original Kaluza-Klein theory to 6-dimension, the basic quantum field equations for 0-spin particle,
1-spin particle and 1/2 spin particle with mass $>$0 are directly derived from 6-dimensional Einstein equations.
It shows that the current quantum field
equations of basic particles become pure geometry properties under 6-dimension time-space. The field equations
of electron and photon can be unified in one 6-dimensional extended Maxwell equation.
The equations containing interactions between electron and photon will be derived from Einstein equation
under 6-dimension time-space. It shows that the interactions in QED can be considered as
the effect of local geometry curvature changing instead of exchange virtual photons.

\pacs{11.10.-z, 11.27.+d, 04.50.+h, 04.62.+v}
\end{abstract}

\maketitle

\section{Introduction} \label{INTRO}

Kaluza \cite{Kal21} first developed his method in 1919 in an attempt to unify Electromagnetism
and General Relativity. In fact, Kaluza's idea showed the possibility to unify not only
gravity and electromagnetism, but also matter and geometry \cite{Overduin}. It gave physicists the hope of extending
Einstein's vision of nature as pure geometry to quantum level. The original Kaluza metric can be
written as follows:
\begin{equation}
\left( \hat{g}_{AB} \right) = \left( \begin{array}{cc}
   g_{\alpha\beta} - \phi A_{\alpha} A_{\beta} \; \; & \; \;
     -\phi A_{\alpha} \\
   - \phi A_{\beta} \; \;                                & \; \;
     - \phi
   \end{array} \right) \; \; \;
\label{5dMetric}
\end{equation}
where the $\alpha\beta$-part of $\hat{g}_{AB}$ with $g_{\alpha\beta}$
(the four-dimensional metric tensor), the $\alpha 4$-part with $A_{\alpha}$ (the electromagnetic potential),
and the $44$-part with $\phi$  (a scalar field). The four-dimensional metric signature is taken
to be $(+ \, - \, - \, -)$

The metric above contains a massless scalar field and a massless 1-spin vector field. The method did not
include the mass part of quantum field (although people believes that the metric (\ref{5dMetric}) describe
a spin-2 graviton,
a spin-1 photon and a spin-0 boson which is thought to be connected with how particles acquire mass.);
it also can not be used to describe the particle with half integer spin. To completely extend Einstein's vision
of nature (as pure geometry) to quantum level, we need include quantum fields for particles of
 mass $>0$, and the particles of $\frac{1}{2}$ spin into the theory. In other efforts, a possible
``theory of everything''-- Superstring theory extended Kaluza's idea to ten-dimension,
 which contains all kinds of fields. But superstring theory is much complicate, it lost the beauty of
 simplicity in original Kaluza-Klein theory.

In this paper, one will find that by extending Kaluza-Klein theory to 6-dimension time-space,
the field equations of 0-spin particle with mass $>0$ (Klein-Gordon equation), equations of 1-spin particle with
mass $>0$ and equations of half integer spin particle (Dirac equation) can be derived directly from 6-dimensional
Einstein equations under new 6-dimensional metric, especially the equations of electron and photon can be unified
in one 6-dimensional Maxwell equation when we expand the components of fields of electron and photon
to 6-dimension.
It indicates that the current quantum field
equations of basic particles are pure geometry properties of 6-dimension time-space.
The coupling equations between electron and photon will be obtained from Einstein equation under 6-dimension
time-space. It shows that the interactions in QED can be considered as
the effect of local geometry curvature changing instead of exchange virtual photons.

\section{Equations of 0-spin free particle} \label{ZeroSpin}

Throughout this paper, the four-dimensional metric signature is taken to be $(+ \, - \, - \, -)$,
and the indices for 6-dimensional time-space to be 0,1,2,3,4,5.

The 6-dimensional Einstein equations keep the same format as 4-dimensional:
\begin{equation}
\hat{G}_{AB} = \kappa \hat{T}{AB} \; \; \; ,
\label{5dEFE1}
\end{equation}
where $\hat{T}{AB}$ is 6-dimensional energy momentum tensor,
$\hat{G}_{AB} \equiv \hat{R}_{AB} - \hat{R} \, \hat{g}_{AB} / 2$
is the Einstein tensor, $\hat{R}_{AB}$ and
$\hat{R} = \hat{g}_{AB} \hat{R}^{AB}$ are
the 6-dimensional Ricci tensor and scalar respectively,
and $\hat{g}_{AB}$ is the 6-dimensional metric tensor, A, B.. run over 0,1,2,3,4,5 .

The 6-dimensional Ricci tensor and Christoffel symbols are defined
in terms of the metric exactly as in four dimensions:
\begin{eqnarray}
\hat{R}_{AB}        & = & \partial_C \hat{\Gamma}^C_{AB} -
                          \partial_B \hat{\Gamma}^C_{AC} +
                          \hat{\Gamma}^C_{AB} \hat{\Gamma}^D_{CD} -
                          \hat{\Gamma}^C_{AD} \hat{\Gamma}^D_{BC}
                          \; \; \; , \nonumber \\
\hat{\Gamma}^C_{AB} & = & \frac{1}{2} \hat{g}^{CD} \left(
                          \partial_A \hat{g}_{DB} +
                          \partial_B \hat{g}_{DA} -
                          \partial_D \hat{g}_{AB} \right) \; \; \;
\label{6dChristRicci}
\end{eqnarray}
where A, B.. run over 0,1,2,3,4,5.

For 0-spin particle, we choose the 6-dimensional time-space metric as:
\begin{equation}
\left( \hat{g}_{AB} \right) = \left( \begin{array}{cc}
   g_{\alpha\beta} \; \; \; \; \; \;  \; \; \; \; \; \; \\
  \; \; \; \; \; \; \; \; \phi \; \; \; \; \; \; \; \; \\
   \; \; \; \;  \; \; \; \; \; \; \; \; \; \; -1 \\
   \end{array} \right)
\label{6dMetric_0}
\end{equation}
where metric elements $g_{\alpha\beta}$ is 4-dimensional metric. We concentrate on quantum field equations in
this paper, so ignore gravity field, then
$g_{\alpha\beta} = {\delta}_{\alpha\beta}$, $g_{55} = -1$, $g_{44} = \phi$,
with conditions:
\begin{eqnarray}
\partial_{4} \phi = 0  \;, \; \;  \;
\partial_{5} \phi =  i\frac{m_0}{\hbar} \phi \; \; \;  \;
\label{6DCondition}
\end{eqnarray}
where $\hbar$ is Planck constant, $m_0$ is rest mass of particle. or equivalently
\begin{equation}
\phi =  \phi_k e^{\frac{im_0}{\hbar} x_5}
\end{equation}
where $\phi_k$ is $\phi$ in Kaluza metric (\ref{5dMetric}). Start from here,
throughout this paper, capital Latin indices
A,B,C .. run over 0,1,2,3,5 (A,B $<> 4$),
Greek indices $\alpha, \beta$ ... run over 0, 1, 2, 3, and small Latin indices a, b, ... run over 1, 2, 3.
Also we have $g_{AB} g^{AB} = \delta_{AB} $, that makes $g^{44} = \frac{1}{\phi} $.

Using equations (\ref{6dMetric_0}), (\ref{6dChristRicci}), the $\alpha \beta-$, $\alpha 5- $, 55-, $\alpha 4- $,
and 44-components of Einstein equations (\ref{5dEFE1}) become:
\begin{equation}
 \frac{1}{\phi}\partial^{\alpha}\partial_{\beta}\phi = \kappa \hat{T}{\alpha\beta}
\label{5dMSTensor_0}
\end{equation}
\begin{eqnarray}
\frac{-im_0}{\hbar \phi} \partial_{\alpha} \phi = \kappa \hat{T}{5\alpha} = \kappa \hat{T}{\alpha 5} \\
\label{5dMSTensor_1}
-(\frac{m_0}{\hbar})^2 = \kappa \hat{T}{55}     \\
\label{5dMSTensor_2}
 \kappa \hat{T}{4 \beta} = \kappa \hat{T}{\alpha 4} = 0 \\
\label{5dSETensor_3}
\Box \phi + (\frac{m_0}{\hbar})^2\phi = 0
\label{5dSETensor_4}
\end{eqnarray}
Here $\hat{T}{44} = 0$, i.e. no 5th dimensional energy momentum tensor. Equation (\ref{5dSETensor_4})
is Klein-Gordon Equation for free 0-spin particle with mass $>0$. Notice that for free {\it single} particle:
\begin{equation}
\phi = e^{\frac{-i}{\hbar}(p^\alpha x_{\alpha} - m_0 x_5)}
\label{plan_wave}
\end{equation}
Then equation (\ref{5dMSTensor_0}) become:
\begin{equation}
 -\frac{1}{\hbar^2}p^{\alpha}p_{\beta} = \kappa \hat{T}{\alpha\beta}
\label{5dMSTensor_5}
\end{equation}
If we let $\kappa =\frac{-1}{\hbar^2} $, then $\hat{T}{\alpha\beta} = p^{\alpha}p_{\beta}$, i.e. energy momentum
tensor is the products of two 4-dimensional momentum. Let $p_5 = m_0, p_4 = 0$, then 6-dimensional
engergy momentum tensor is: $\hat{T}{AB} = p^{A}p_{B}$.  One can also see that as we describe quantum field
in {\it pure geometry}, $\frac{i}{\hbar}$ plays the similar {\it geometry} role as $\sqrt{8\pi G}$ 's
role in 4-dimensional Einstein equation, where G is gravational constant.

\section{Equations of 1-spin free particle} \label{OneSpin}

For 1-spin particle with mass $m_0 > 0 $, we let
\begin{equation}
\hat{A}_{\alpha} = A_{\alpha}e^{im_{0} x_5}
\label{hatA}
\end{equation}
where where $m_{0}$ is rest mass of particle, $x_5$ is 6th dimension coordinate,
and start from this section, we always choose $\hbar \equiv  1$.
Let
\begin{equation}
\hat{A}_{5} = 0
\label{hatA_5}
\end{equation}
6-dimensional metric for 1-spin free particle become:
\begin{equation}
\left( \hat{g}_{AB} \right) = \left( \begin{array}{cc}
   g_{\alpha\beta} + \hat{A}_{\alpha} \hat{A}_{\beta} \; \; & \; \;
       \hat{A}_{\alpha} \; \\
    \hat{A}_{\beta} \; \;                                & \; \;
     1 \; \; \; \; \; \; \\
  \; \; \; \; \; \; \; \; \;  & \; \; \; \; \; \; \; \; \; \; \;  \; \;   -1  \end{array} \right)
\label{6dMetric1S_m}
\end{equation}
Let $\hat{F}_{AB} \equiv \partial_{A} \hat{A}_{B} -
\partial_{B} \hat{A}_{A}$, and A, B runs over 0,1,2,3,5.
energy momentum tensor
\begin{equation}
\hat{T}_{AB} \equiv g_{AB} \hat{F}_{CD} \hat{F}^{CD}/4 - \hat{F}_{A}^{C} \hat{F}_{BC}
\label{1SpinT}
\end{equation}
where A,B run over (0,1,2,3,5).
so the $\alpha \beta-$, $\alpha 4- $, and 44-components of
6-dimensional Einstein equations (\ref{5dEFE1}) become:
\begin{eqnarray}
G_{\alpha\beta} = \frac{1}{2}
   \hat{T}_{\alpha\beta} \; , \; \; \;
\nabla^{\alpha} \, \hat{F}_{\alpha\beta} - m_0^{2} \hat{A}_{\beta} = 0 \; \nonumber \\
\frac{1}{4}\hat{F}_{\alpha\beta} \hat{F}^{\alpha\beta} - \frac{1}{2} m_{0}^{2} \hat{A}_{\alpha} \hat{A}^{\alpha} = 0 \; \; \;
\label{6dFieldEquns1S_m}
\end{eqnarray}
This is equations for 1-spin single particle with mass $>$ 0.
As we see above, the particle of 1-spin obtains its mass from derivative of 6th dimension.

When $m_0 = 0$, metric (\ref{6dMetric1S_m}) becomes usual Kaluza metric at the case of $\phi = -1$. The second and
third equation of equations (\ref{6dFieldEquns1S_m}) become
\begin{eqnarray}
\nabla^{\alpha} \, \hat{F}_{\alpha\beta} = 0 \;, \; \; \;
\frac{1}{4}\hat{F}_{\alpha\beta} \hat{F}^{\alpha\beta} = 0 \;
\label{Maxwell_1}
\end{eqnarray}
The first equation of (\ref{Maxwell_1}) is Maxwell equation in vacuum and the second equation of (\ref{Maxwell_1}) is true
for free photon (plane-wave).

\section{Equations of $\frac{1}{2}$-spin free particle} \label{halfSpin}

In section \ref{OneSpin} for 1-spin particle, we choose $\hat{A}_{5} = 0$, now we will see that if we
let $\hat{A}_{5} <> 0$, we will get field equations for $\frac{1}{2}$ particle. We choose the 6-dimensional metric
for  $\frac{1}{2}$ particle as:
\begin{equation}
\left( \hat{g}_{AB} \right) = \left( \begin{array}{cc}
   g_{\alpha\beta} + \hat{K}_{\alpha} \hat{K}_{\beta} \; \; & \; \;
      \hat{K}_{\alpha} \; \; \; \; \; \;  \; \;  \hat{K}_{\alpha} \hat{K}_{5} \\
    \hat{K}_{\beta} \; \;  & \; \;
     1 \; \; \; \; \; \;  \; \;   \hat{K}_5  \\
   \hat{K}_{5} \hat{K}_{\beta} \; \; & \; \;  \hat{K}_5 \; \; \; \;  \; \;  -1+\hat{K}_{5} \hat{K}_{5}  \end{array} \right) \;
\label{6dMetricHalfS}
\end{equation}
where $\hat{K}$ is 6-dimensional vector field in 6-dimensional time-space, with conditions
\begin{eqnarray}
\partial_5 \hat{K}_{A} = im \hat{K}_{A} \;, \; \; \hat{K}_5 <> 0 \;, \; \; \hat{K}_4 = 0 \;, \; \; \partial_4 \hat{K}_A = 0
\end{eqnarray}
and
\begin{equation}
\left( \hat{g}^{AB} \right) = \left( \begin{array}{cc}
   g^{\alpha\beta}  \; \;  & \; \;
    - \hat{K}^{\alpha} \; \; \;  \; \; \; \\
   - \hat{K}^{\beta} \; \;  & \; \;
     1+\hat{K}_{A}\hat{K}^{A} \; \; \; \;- \hat{K}^{5}\\
 \; \;  & \; \;  - \hat{K}^{5} \; \; \; \; \; \;   -1  \end{array} \right)
\label{6dMetricHalfS_i}
\end{equation}

Let $\hat{E}_{AB} \equiv \partial_{A} \hat{K}_{B} -
\partial_{B} \hat{K}_{A}$.
Define energy momentum tensor for half spin particle:
\begin{equation}
\hat{T}_{AB} \equiv g_{AB} \hat{E}_{CD} \hat{E}^{CD}/4 - \hat{E}_{A}^{C} \hat{E}_{BC}
\label{halfSpinT}
\end{equation}
so the AB-, A4-, and 44-components of
6-dimensional Einstein equations (\ref{5dEFE1}) become:
\begin{eqnarray}
G_{AB} = \frac{1}{2} \hat{T}_{AB} \;, \; \; \; \;
\partial_{A} (\partial^{C} \hat{K}_{C}) - \partial^{C} \partial_{C} \hat{K}_{A} = 0   \nonumber \\
\frac{1}{4}\hat{E}_{AB} \hat{E}^{AB} = 0
\label{halfspinEq}
\end{eqnarray}
To derive the equation above, we used the relation below:
\begin{equation}
\partial_{C} g^{CC} (\partial_{A} \hat{K}_{C} - \partial_{C} \hat{K}_{A} )
 = \partial_{A} (\partial^{C} \hat{K}_{C} ) - \partial^{C} \partial_{C} \hat{K}_{A}
 \label{relation}
 \end{equation}

For free particle, it is reasonable to assume that each components of $\hat{K}$ satisfied plane-wave condition:
\begin{equation}
\partial^{C} \partial_{C} \hat{K}_{A} = 0
\label{planewave_1}
\end{equation}
The second equation of equations (\ref{halfspinEq}) become
\begin{equation}
\partial_{A} (\partial^{C} \hat{K}_{C})  =  0  \; \; \; \; for \, all \, A = 0,1,2,3,5
\label{halfSpinEq2}
\end{equation}
i.e. $\partial^{C} \hat{K}_{C} $ does not depended on $x_0,x_1,x_2,x_3,x_5$,
 so it is reasonable to let
$\partial^{C} \hat{K}_{C} $ equals zero (Not likely equals a constant other than zero
becaues $\hat{K}$ contains
a plane-wave function part of condition (\ref{planewave_1})). Together with the third equation of
equations (\ref{halfspinEq}), we have
\begin{eqnarray}
\partial^{C} \hat{K}_{C}  = 0  \;, \; \; \; \;
\hat{E}_{AB} \hat{E}^{AB} = 0   \; \; \; \; \;
\label{halfSpinEq3}
\end{eqnarray}
Now let
\begin{eqnarray}
\hat{K_0} = Cg_{00}\phi_0 e^{im_0 x_5}   \; \; \; \;
\hat{K_1} = Cg_{11}\phi_3 e^{im_0 x_5}  \nonumber \\
\hat{K_2} = -iCg_{22} \phi_3 e^{im_0 x_5} \; \; \; \;
\hat{K_3} = Cg_{33}\phi_2 e^{im_0 x_5}  \nonumber \\
\hat{K_5} = Cg_{55}\phi_0 e^{im_0 x_5}
\label{HalfSpinVector}
\end{eqnarray}
where C is constant to be determined, $g_{\alpha \alpha}$ is element of usual 4-dimensional metric (1,-1,-1.-1), $g_{55} = -1$.
First equation of (\ref{halfSpinEq3}) becomes
\begin{equation}
\partial_{0} \phi_0 + \partial_{1} \phi_3 - i \partial_{2} \phi_3 + \partial_3 \phi_2 + im_0 \phi_0 = 0
\label{ModifiedDirac}
\end{equation}
It is first Dirac equation in $x_3$ representation. The two equations in (\ref{halfSpinEq3}) plus normalization
condition of $\phi_0$, $\phi_2$, $\phi_3$ keep the solution unique, plus we need choose constant C to make the energy momentum
tensor $\hat{T}_{AB}$ reasonable. The solutions of  (\ref{halfSpinEq3}) are:
\begin{eqnarray}
\hat{K}_0 = C \sqrt{\frac{m_0+p_0 }{2m_0}} e^{-i p^A x_{A}}\;, \; \; \; \hat{K}_5 = - \hat{K}_0 \;, \;  \; \; \ \nonumber \\
\hat{K}_1 = -C\sqrt{\frac{m_0+p_0 }{2m_0}} \frac{p_1 + ip_2}{m_0+p_0} e^{-i p^A x_{A}}  \;, \; \; \; \;  \nonumber \\
\hat{K}_2 = iC\sqrt{\frac{m_0+p_0 }{2m_0}} \frac{p_1 + ip_2}{m_0+p_0} e^{-i p^A x_{A}}  \;, \; \; \; \;  \nonumber \\
\hat{K}_3 = -C\sqrt{\frac{m_0+p_0 }{2m_0}} \frac{p_3}{m_0+p_0} e^{-i p^A x_{A}} \; \;  \; \; \;
\label{DComponents}
\end{eqnarray}
The corresponding $\phi_0$, $\phi_1 = 0$,
$\phi_2$, $\phi_3$ in relation (\ref{HalfSpinVector}) is the first solution of Dirac equation of $\frac{1}{2}$ spin particle with positive energy.
We choose the constant C in equations (\ref{HalfSpinVector}) as
\begin{equation}
C = \frac{\sqrt{(m_0+p_0)2m_0}}{p_3 \sqrt{p_0}}
\label{Constant}
\end{equation}
Substitute (\ref{Constant}), (\ref{DComponents}) into (\ref{halfSpinT}), then:
\begin{equation}
\hat{T}_{ab} = -\frac{p_{a}}{p_0}p_{b}e^{\frac{-2i}{\hbar}p^{c}x_{c}}
\label{TAB}
\end{equation}
The negative sign can be removed by adding a constant $\kappa$ in metric and let $\kappa = \frac{-1}{\hbar^2} $ as we did for
zero spin particle. In relativity, we know that $v_a = \frac{p_{a}}{p_0} $, so
the result is just we expected: the energy momentum tensor becomes
the 5-dimensional momentum vector $(p_{\alpha},m_0)$ multiply speed and times the square of plane wave function.

If we let
\begin{eqnarray}
\hat{K_0} = Cg_{00}\phi_1 e^{ im_0 x_5}  \;, \; \; \; \;
\hat{K_1} = Cg_{11}\phi_2 e^{ im_0 x_5}  \nonumber \\
\hat{K_2} = iCg_{22}\phi_2 e^{im_0 x_5}  \;, \; \; \; \;
\hat{K_3} = -Cg_{33}\phi_3 e^{im_0 x_5}  \nonumber \\
\hat{K_5} = Cg_{55}\phi_1 e^{im_0 x_5}
\label{HalfSpinVector_2}
\end{eqnarray}
We can get local time-space metric of half spin free particle with spin $-\frac{1}{2}$ and positive energy;
Easy to examine that, it is also satisfied both equations of (\ref{halfSpinEq3}) and Dirac equations.
Similarly we can derive the other two solutions for negative energy of Dirac equations.  As conclusion, equations
(\ref{halfspinEq}) is general field equations for particles with half integer spin. It is interesting seeing that:
equation (\ref{halfSpinEq3}) together with plane-wave condition (\ref{planewave_1}), we have:
\begin{equation}
\nabla^{A}(\hat{E}_{AB}) = 0
\label{Maxwell_ex}
\end{equation}
where $ \hat{E} \equiv \partial_{A} \hat{K}_{B} -
\partial_{B} \hat{K}_{A} $. It is similar to Maxwell equation. If we let $\hat{K}_5 = 0$, it becomes
the second equation of (\ref{6dFieldEquns1S_m}) for 1-spin particle; if we also let $m_0 = 0$, then the above equation
(\ref{Maxwell_ex}) turns into Maxwell equation. So 1-spin particle (both massless and mass $>0$), $\frac{1}{2}$-spin particle
are satisfied the same equations -- extended 6-dimensional Maxwell equation (\ref{Maxwell_ex}) in 6-dimensional time-space.
The field of $\frac{1}{2}$ particle becomes 6-dimensional vector field.

Compare the metrics of 1-spin particle with $\frac{1}{2}$-spin particle,
the only difference is that one without 6th-componet
of vector and the other with 6th-component. The metric of 1-spin particle has symmetry of
4-dimensional time-space, the metric
of $\frac{1}{2}$-spin particle has symmetry of 5-dimensnional time-space (0,1,2,3,5).

\section{Equations of Electron-Photon Interaction} \label{Interaction}

The similarity between the metric of single electron and the metric of single photon
makes us easier to combine eletron and photon into one metric, The metric $\hat{g}_{AB}$ of the coupling of
electron-photon is:
\begin{equation}
\left( \begin{array}{cc}
   g_{\alpha\beta} + \kappa^{2} \hat{B}_{\alpha} \hat{B}_{\beta} \; & \;
    \kappa \hat{B}_{\alpha} \; \; \; \; \kappa^{2}\hat{B}_{\alpha} \hat{K}_{5}  \\
    \kappa \hat{B}_{\beta}  \; & \;
     1 \; \; \; \; \;  \;  \; \;  \;  \kappa \hat{K}_5 \\
   \kappa^{2} \hat{K}_{5} \hat{B}_{\beta} \; & \; \; \; \kappa \hat{K}_5  \; \; \; \;  \;  -1+ \kappa^{2} \hat{K}_{5} \hat{K}_{5}  \end{array} \right)
\label{6dMetricCoupl_e}
\end{equation}
where $\hat{B}_\alpha \equiv A_{\alpha}+\hat{K}_{\alpha} $;  $A_{\alpha}$ for 4-dimensional photon vector field, $\hat{K}$ for
6-dimensional vector field (spinor in 4-dimension) which we defined in previous section for electron.
$\kappa$ is constant for coupling with gravity; $\kappa = 4\sqrt{\pi G}$ where G is gravational constant.
To describe interactions between electron and photon, we impliment Klein's idea \cite{Kle26a} to make the
derivative of 5th dimension $<>0$ , we can understand that in two ways: 1)
the interaction of Electron-Photon changed curvature of local time-space. 2) An Electron always associate with electromagnetic fields
which cause non-zero derivative of 5th dimension.
Let
\begin{eqnarray}
\hat{K}_{A} \rightarrow \hat{K}_{A}e^{in\gamma x_4}
\label{4thDeriv}
\end{eqnarray}
where $\gamma $ is a very small constant, n is integer, and there is no $\hat{K_4}$. Then $\partial_4 \hat{K}_{A} = i \gamma \hat{K}_{A} $.
Because the derivative of 5th dimension $<> 0$, 5th dimensional energy momentum tensor is not zero.

Using (\ref{6dMetricCoupl_e}), (\ref{4thDeriv}), through considerable algebra, the AB-, and 44-components of
6-dimensional Einstein equations (\ref{5dEFE1}) become:
\begin{eqnarray}
G_{AB} = \frac{\kappa^2}{2} \hat{T}_{AB} \;, \; \; \; \;
\frac{1}{4}\hat{L}_{AB} \hat{L}^{AB} = \hat{T}_{44}
\label{halfspinEq_coupl}
\end{eqnarray}
and the A4-components of Einstein equations can be write into 3 separate pieces:
\begin{eqnarray}
D^{C} \hat{K}_{C} = 0 \;, \; \; \; D^{C} D_{C} \hat{K}_{A} = 0 \;, \;\nonumber \\
\nabla^{\alpha} \, \hat{F}_{\alpha A} = 0
\label{halfspinEq_coup2}
\end{eqnarray}
where $D_{\alpha} =\partial_{\alpha} - in\gamma A_{\alpha} $.
We extended photon field
to 6-dimension with $A_4 = 0, A_5 = 0$; $F_{AB} \equiv \partial_{A} A_{B} - \partial_{B} A_{A}$,
$\hat{L}_{AB} \equiv \partial_{A} A_{B} -
\partial_{B} A_{A} + D_{A}\hat{K}_{B} - D_{B} \hat{K}_{A}$, energy momentum tensor of electron coupling with photon becomes:
\begin{eqnarray}
\hat{T}_{AB} \equiv \frac{g_{AB}}{4}\hat{L}_{CD}\hat{L}^{CD} - \hat{L}_{A}^{C} \hat{L}_{BC} + \nonumber\\
     + in\gamma(\hat{B}_{B}\hat{K}^C F_{AC} + \hat{B}_{A}\hat{K}^C F_{BC}) \nonumber \\
     + in\gamma (D_{A}\hat{K}_{B} + D_{B}\hat{K}_{A})
\label{halfSpinT_e}
\end{eqnarray}
where we dropped $\gamma^2$ items in energy momentum tensor since $\gamma$ is small; $\hat{E}_{AB} \equiv \partial_{A} \hat{K}_{B} -
\partial_{B} \hat{K}_{A}$.

If let $n=1, \gamma = -e$, e is charge of electron, and use relation from (\ref{HalfSpinVector}),
then first equation of (\ref{halfspinEq_coup2}) becomes:
\begin{eqnarray}
(\partial_{0} + ie A_0) \phi_0 + (\partial_{1} +ie A_1) \phi_3 - i (\partial_{2}+ie A_2) \phi_3 \; \; \; \; \nonumber \\
 + (\partial_3 + ie A_3) \phi_2 + im_0 \phi_0 = 0
\label{ModifiedDirac_2}
\end{eqnarray}
This is Dirac equation for electron-photon interaction.
The second equation of (\ref{halfspinEq_coup2}) is Klein-Gordon equation in electromagnetic field that
each components of electron field should satisfied individually. The 3rd equation of (\ref{halfspinEq_coup2})
is Maxwell equation.

Now let's look at energy momentum tensor (\ref{halfSpinT_e}). The first two parts are combined energy momentum tensor of electron
and photon. The third item $ie\hat{B}_{B}\hat{K}^C F_{AC}$ is related to Lorenz force. The third item is electric current density.

6-dimensional action  is
\begin{equation}
S = -\int \sqrt{-\hat{g}} \hat{R}
     \; d^6 x \; \; \;
\label{6dAction}
\end{equation}
where $\hat{R} = \hat{g}^{AB} \hat{R}_{AB} $ (A,B runs over 0,1,2,3,4,5)
is 6-dimensional Ricci scalar. The metric determinant $\hat{g}$ reduces in the simple manner:
\begin{equation}
\hat{G} = det(\hat{g}_{AB}) = - det(g_{\alpha\beta})
\label{metric_det}
\end{equation}
Using (\ref{6dMetricCoupl_e}), (\ref{halfspinEq_coupl}) and (\ref{halfspinEq_coup2}),
we also assume $\int dx_5 dx_6 = 1 $, then
\begin{equation}
S = -\int \sqrt{-\hat{g}}( R + \frac{1}{4} \hat{L}_{AB} \hat{L}^{AB} )
     \; d^4 x \; \; \;
\label{6dAction_coupl}
\end{equation}
where R is usual 4-dimensional Ricci scalar. The above equation contains a gravity part and kinetic energy part.
for interaction between electron and photon.

\section{Discussions and Conclusions} \label{SUM}

The kinetic energy part of (\ref{6dAction_coupl}) $\frac{1}{4} \hat{L}_{AB} \hat{L}^{AB}$
contains the combination of electron field and
photon field. Actually, metric (\ref{6dMetricCoupl_e}) can be used to describe:

1) {\it Eletron associate with its own static electromagnetic field}. Eletron is always accompany with photon (static
electromagnetic fields), that's why it can absort photon to gain energy, emits phton and lost energy, annihilation with
positron and turns into two photons, so it is reasonable to say that there is
always $A_{\alpha}$ field in electron's local geometry metric.

2) {\it Electron absorb a photon and gains energy}. If a free photon $A_0$ at time $t_0$, location $X(x_{01},x_{02},x_{03})$ which
local geometry metric is (\ref{6dMetric1S_m}), it is absorted by an electron at time t, then the local metric at time t becomes
metric (\ref{6dMetricCoupl_e}) with $\hat{B^{\prime}} = A^{\prime}_0 + A_{e} + \hat{K}$, where photon $A_0$ turns to $A^{\prime}_0$ after
interaction and $A_{e}$ is initial photon contained in electron, and finally electron gain the energy.

3) {\it The intermediate state of a photon interacts with electron}. If a free photon $A_0$ at time $t_0$, location $X(x_{01},x_{02},x_{03})$ which
local geometry metric is (\ref{6dMetric1S_m}), it interacts with electron at time t, then the local metric at time t becomes
metric (\ref{6dMetricCoupl_e}) with $\hat{B^{\prime}} = A^{\prime}_0 + A_{e} + \hat{K}$, after interaction, the photon and electron
will be separate. Of couse the energy and momentum of both particle changed.

In the discussion above, we only dealing with curvature changes. We do not need use any ``virtual'' photons.

One can see from section \ref{OneSpin}, U(1) field obtained its mass from derivative of 6th dimension without losing symmetry, so
we could avoid Higgs mechanism. If we treate interactions as changes of geometry curvature of local time-space, we could
avoid so called ``vacuum effects'' in current QED theory, the ``vaccum effects'' could be interpreted as local curvature changes too.

In addition, here we see a symmetry between gravity and electromagnetism: Mass is the source of gravity, it comes from the derivative
of 6th dimension; Charge is the source of electromagnetic interaction, it comes from the derivative of 5th dimension.
We can also use Klein's original idea that, the 5th and 6th have circular topology and periodic conditions, that can make
quantization of electric charge and quantization of mass (or energy).

As the conclusion, in this paper, we derived above equations directly from Einstein equations under 6-dimensional time-space metric.
The energy momentum tensor is
the products of extended 6-dimensional momentum vector $p_A$ (where $p_5 = m_0$, $p_4 = 0$, no 5 dimensional momentum);
Planck constant $\hbar$ plays the similar role as the role of gravational constant G in gravity equations.
The interactions between photon and electron can be treated as the effects of local time-space geometry curvature changing.
It indicates the pure geometry nature of quantum fields.
Indeed, by using metric (\ref{6dMetric_0}), one can find that the geodesic path of single free
 particle in 6-dimensional time-space is exact plane-wave function, the interpretion of basic quantum physics by using
 6-dimensional time-space is discussed in another paper \cite{xchen}.

\begin{thebibliography}{9}

\bibitem{Kal21} T. Kaluza, {\it Zum Unit\"atsproblem der Physik},
                 Sitz. Preuss. Akad. Wiss. Phys. Math. K1 (1921) 966.

\bibitem{Overduin}  J. M. Overduin, P.S.Wilson, {\it Kaluza-Klein Gravity}, \\
		http://arxiv.org/abs/gr-qc/9805018

\bibitem{xchen} X. Chen , {\it Modified Kaluza-Klein Theory, Quantum Hidden Variables and 3-Dimensional Time},
                http://xxx.lanl.gov/abs/quant-ph/0501034

\bibitem{Kle26a} O. Klein, {\it Quantentheorie und f\"unfdimensionale
                  Relativit\"atstheorie },
                  Zeits. Phys. 37 (1926) 895.

\end{thebibliography}

\end{document}


\chapter{Conclusion}

\section{Summary}
This work shows that GEMM for convolutional neural network inference can be accelerated by as much as 45\% with low-precision integer arithmetic. In addition, the experiment results demonstrate negligible loss in image recognition accuracy in the case of four popular neural network architectures. To achieve such a speed-up, quantization methods were employed. The work showcases the quantization technique for inference speed improvement in the context of the image recognition problem, whereby a lexical description is predicted given an image.

\section{System limitations and future work}
However, two architectures are concluded to be incompatible with the quantization method used. Namely, inference with residual neural networks fail with such a technique and therefore completely break down when quantized. This is speculated to be because of the nature of such networks, where inputs to a given convolution are not necessarily taken from the previous layer's output, but often from layers prior. As these networks can be considered equivalent to many shallower networks stacked together, where tight layer relationships are not necessarily a reality, individual quantized perturbations of low-precision calculations do not carry over from layer to layer.

As the size of convolutional neural networks grow, the demands for accurate classifications persist, and smaller and smaller devices like phones and other embedded systems are expected to run image recognition software, quantization of neural networks is a research field worthy of continuation. This work demonstrates that in order to reap the benefits of low-precision GEMM, total runtime must be reduced by optimizing or off-lining the online processes involved in quantization as much as possible: online parameter calculation (i.e. maxima and minima of input and output volumes), online quantization of GEMM components, and dequantization.


\chapter{Background} \label{chap:background}

\section{Introduction}
Many scientific subject areas (i.e. medicine, remote sensing) have
to deal with the problem of image correspondence (also known as
image registration or image matching). This problem arises when
there is a need to extract information about an object by
comparing a set of different images in which this object appears.
Most of the time this comparison cannot be performed easily
because of differences between images such as viewpoint changes,
use of different sensors, images taken at different times, and
changes in general imaging conditions. Given such differences, a
way to align different images of the same object is needed. In
medical applications, and specifically in mammography,
radiologists have to deal with all the above problems. For
example, a way to detect abnormal structures in the breast is to
compare temporal (images of the same breast taken at different
times) or contralateral (using left-right breast) mammograms.

As images are taken under different conditions (i.e. film
sensitivity, radiation exposure, breast compression and patient
movement) and/or time intervals (e.g. screening interval of three
years) the structure of the breast is likely to suffer some
changes, although an overall similarity will be maintained. A way
to detect those changes is to align the images and compare the
results using, for instance, simple image subtraction. In
addition, as described in the previous chapter, different imaging
modalities are being used to provide a better understanding of a
region of interest. The first step to incorporate information from
those modalities is to align them to be able to establish areas of
correspondence.

An image correspondence method is based on finding a mapping
function ($f(x,y)$) that maps each coordinate of one image ($A$)
into another image ($B$).

\begin{equation}
B(x',y')=A(f(x,y))
\end{equation}
\noindent which maps spatial coordinates $(x,y)$ in the reference
image $A$ to coordinates $(x',y')$ in the warped (also referred to
as target) image $B$. Usually the function $f$ is expressed as two
(or three in three dimensions) separate functions $f_x$ and $f_y$
(and $f_z$). This review describes image registration in two
dimensions. However extrapolation to 3D is often straightforward
and is described in detail where appropriate.

A general methodology for image alignment typically follows those
steps:

\begin{enumerate}
\item \emph{Selection and extraction of features.} A registration method can
be based on matching different image primitives. For instance
points, edges, ridges, surfaces, or a whole image. The selection
of a suitable primitive is a trade off between the information it
provides and its complexity. For instance, raw pixels with only
intensity information provide a straightforward comparison. On the
other hand, taking the whole image provides more (and more
complex) information. Primitives will be described by a set of
features extracted from them. That is the case, for instance, of
shape features extracted from regions or curvature values
extracted from linear structures. There is an extensive literature
about feature extraction, the reader is referred to Chapter 4
where a review on that subject is given.

\item \emph{Similarity metric.} A way to measure the similarity between
the extracted features is needed. Many different similarity
metrics have been proposed and their suitability depends on the
chosen feature. The reader is referred to Chapter 5 where
similarity measures are reviewed and a novel measure is proposed.

\item \emph{Selection of the mapping function and estimation of
its parameters.} The complexity of a mapping function should be
determined depending on the application field and type of
misalignment we are dealing with. Once a mapping function is
selected its parameters need to be estimated. This requires a
definition of an optimum search space and search strategy which is
often related to optimisation of a cost function related to a
similarity measure previously mentioned.

\item \emph{Alignment of images using the mapping function.} When the
function parameters are known we are able to transform an image in
order to minimise the initial misalignment.
\end{enumerate}

\subsection{Interpolation}
Usually transformed coordinates $(x',y')$ in the warped image do
not match a given coordinate grid (i.e. they are non-integer
values) and interpolation is needed. Different interpolation
methods have been used in image registration: nearest neighbour,
tri-linear and partial volume distribution. The interpolation
problem is graphically represented in Figure~\ref{fig:interp}.

\begin{figure}[htb]
        \centering
        \epsfxsize=4cm
        {\epsfbox{figures/interp.eps}}
  \caption{Interpolation of corresponding coordinates}
  \label{fig:interp}
\end{figure}

Nearest neighbour interpolation assigns the intensity value of
$m=(x',y')$ to its nearest point:
\begin{equation}
        B(m)=B(\_{\forall i}{min}(d_E[m,n_i]))
\end{equation}
\noindent where $d_E$ indicates Euclidean distance and $n_i$ are
the nearest neighbours. The nearest neighbour approach provides
good accuracy and it is easy to compute.

Trilinear interpolation assigns to $m$ the value of the weighted
($w_i$) sum of the neighbouring pixels.

\begin{equation}
        B(m)=\sum_iw_i B(n_i)
\end{equation}
\noindent where each weight $w_i$ is related to the distance from
$m$ (see Figure~\ref{fig:interp})

\begin{equation}
        w_1 = (1-dx)(1-dy) ~~~~ w_2 = (1-dx)dy ~~~~w_3 = dx(1-dy)
        ~~~~ w_4 = dx dy
        \nonumber
\end{equation}

This interpolation creates a new intensity value, which could
introduce undesirable effects in the intensity distribution. This
fact is regarded as its main drawback, and can be solved using a
partial volume distribution. This method adds the weight of each
neighbour ($w_i$ values) at the intensity value in the histogram
distribution, without creating an additional intensity value.


\documentclass[onecolumn, conference]{IEEEtran}
\IEEEoverridecommandlockouts
% The preceding line is only needed to identify funding in the first footnote. If that is unneeded, please comment it out.
\usepackage{cite}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{textcomp}
\usepackage{xcolor}
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
    T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
\begin{document}

\title{Tensor network and neural networks

}

\author{
    \IEEEauthorblockN{Junjie He}
\IEEEauthorblockA{\textit{School of Information Science and Technology} \\
\textit{ShanghaiTech University}\\
Shanghai, China \\
hejj1@shanghaitech.edu.cn}
\and
\IEEEauthorblockN{Jiaqiong Zhang}
\IEEEauthorblockA{\textit{School of Information Science and Technology} \\
\textit{ShanghaiTech University}\\
Shanghai, China \\
zhangjq@shanghaitech.edu.cn}
% \and
% \IEEEauthorblockN{3\textsuperscript{rd} Given Name Surname}
% \IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
% \textit{name of organization (of Aff.)}\\
% City, Country \\
% email address or ORCID}
% \and
% \IEEEauthorblockN{4\textsuperscript{th} Given Name Surname}
% \IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
% \textit{name of organization (of Aff.)}\\
% City, Country \\
% email address or ORCID}
% \and
% \IEEEauthorblockN{5\textsuperscript{th} Given Name Surname}
% \IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
% \textit{name of organization (of Aff.)}\\
% City, Country \\
% email address or ORCID}
% \and
% \IEEEauthorblockN{6\textsuperscript{th} Given Name Surname}
% \IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
% \textit{name of organization (of Aff.)}\\
% City, Country \\
% email address or ORCID}
}

\maketitle

% \begin{abstract}
% This document is a model and instructions for \LaTeX.
% This and the IEEEtran.cls file define the components of your paper [title, text, heads, etc.]. *CRITICAL: Do Not Use Symbols, Special Characters, Footnotes,
% or Math in Paper Title or Abstract.
% \end{abstract}

% \begin{IEEEkeywords}
% component, formatting, style, styling, insert
% \end{IEEEkeywords}

\section{Introduction}
Main topic for our project are tensor methods and neural network.Deep neural networks currently demonstrate state-of-the-art performance in several domains.such as computer vision, speech recognition, text processing, etc.These advances have become possible because of algorithmic advances, large amounts of available data,and modern hardware. For example, convolutional neural networks (CNNs) \cite{b1}\cite{b2}show by a large margin superior performance on the task of image classification.These models have thousands of nodes and millions of learnable parameters and are trained using millions of images\cite{b3} on powerful Graphics Processing Units (GPUs).The necessity of expensive hardware and long processing time are the factors that complicate the application of such models on conventional desktops and portable devices.
In tradition neural network,this layer has a linear transformation of a high dimension input signal to a high dimension output signal.For example,the data set CIFAR10 widely used in deep learning course is a collection of pictures.When it used as input signal into neural network,the pictures in it will be divided 32*32*3 pixels.particularly, 32*32 means a picture divided into 32*32 pixel blocks,and 3 means three channels(RGB).Then,the input signal will be reshaped a 32*32*3 dimensional vector .Obviously,such an operation will largely increase the dimension of input signal,and leads to  further complexity of the calculations.The data set CIFAR10\cite{b5} is already a very simple data set in the field of deep learning.However,in the convolutional neural network model used in practical application the dimensions of the input and output signals of the fully-connected layers are of the order of thousands, bringing the number of parameters of the fully-connected layers up to millions.This is undoubtedly a very demanding requirement for hardware facilities.
Consequently, a large number of works tried to reduce both hardware requirements (e. g. memory demands) and running times.To solve this problem,We consider the most frequently used layer of neural network:fully-connected layer.We use a compact tensor train data set to represent the matrix of the fully-connected layers using few parameters while keeping enough flexibility to perform signal transformations\cite{b6}.And,the layer transformed should be compatible with the existing training algorithms for neural network,because all the derivatives required by the back propagation algorithm\cite{b4} can be computed using the properties of Tensor train set.
Tensors are natural multidimensional generalizations of matrices and have attracted tremendous interest in recent years.Multilinear algebra,tensor analysis, and the theory of tensor approximations play increasingly important roles in computational mathematics and numerical analysis\cite{b7}\cite{b8}\cite{b9}\cite{b10}. An efficient representation of a tensor (by tensor we mean only an array with d indices) by a small number of parameters may give us an opportunity and ability to work with d-dimensional problems, with d being as high as 10, 100, 1000 or even one million.Problems of such sizes cannot be handled by standard numerical methods due to the curse of dimensionality, since everything (memory, amount of operations) grows exponentially in d.So,Tensor train decomposition will a effective way to solve this problem.
We will apply our method to popular network architectures proposed for data set CIFAR10.We will experimentally use the networks with tradition fully-connected layer and the tensor fully-connected layer to train a neural network model.Then,we will compare the performance of two models.

\section{Preliminary ideas}
In various fields, low-rank approximation was applied to reduce the computation cost and memory usage. In \cite{b11}, they generalize the idea of low-rank. The authors do not find low-rank approximation of weight matrix in fully-connected layers, they treat the matrix as multidimensional tensor and employ Tensor Train decomposition \cite{b6} to accelerate the computation.
Usally, wider neural network can achieve better performance than narrow neural network. But wide neural networks imply large dense matrix, amount of computation resources are used in per step when training neural networks. By using Tensor Train decomposition for weight matrix, wide neural network can be developed for applications with moderate computation cost and memory usage.
\cite{b11} shows that wide and shallow neural networks has competitive performance with the state-of-art deep neural networks by traing a shallow network oin the outputs of a trained deep neural network.
They report the improvement of performance with the increase of the layer size and used up to 30 000 hidden units while restricting the matrix rank of the weight matrix in order to be able to keep and to update it during the training. Restricting the TT-ranks of the weight matrix (in contrast to the matrix rank) allows to use much wider layers potentially leading to the greater expressive power of the model.

\par CP-decomposition algorithm was applied to compress convolution kernel in CNNs. And they also using properties of CP-decomposition to speed up the inference time.
tp speed up computation of matrix-by-vector, properties of the Kronecker product of matrices was exploited. These matrices have the same structure as TT-matrices with unit TT-ranks. We can generalize this idea to formulate the weight matrix with TT-matrices with unit TT-ranks.
The Tucker format and the canonical format will meet the curse of dimensionality, TT-format is immune to the cues of dimensionality and its algorithm are robust.
\par A $d$-dimensional array (tensor) $ \mathcal{A}$ is said to be TT-format if for each dimension $k=1,...,d$ and for each possible value of the $k$-th  dimension index $j_k=1,...,n_k$ there exists a matrix $\mathbf{G}_k[j_k]$ such that all elements of $\mathcal{A}$ can be computed as the following matrix product:
\begin{equation}
    \label{eq1}
    \mathcal{A}(j_1,...,j_d)=\mathbf{G}_1[j_1]\mathbf{G}_2[j_2]\cdots\mathbf{G}_d[j_d]
\end{equation}
All the matrices $\mathbf{G}_k[j_k]$ related to the same dimension $k$ are restricted to be of the same size $r_{k-1}\times r_k$.
The values $r_0$ and $r_d$ equal to 1 in order to keep the matrix product (\ref{eq1}) of size $1\times 1$.
In what follows we refer to the representation of a tensor in the TT-format as the TT-representation or d the TT-decomposition.
The sequence $\{r_k \}^d_{k=0}$ is referred to as the TT-ranks of the TT-representation of $\mathbf{A}$
(or the ranks for short), its maximum – as the maximal TT-rank of the TT-representation $n$ of $\mathcal{A}: r = \max_{k=0,...,d} r_k$ .
The collections of the matrices $(\mathbf{G}_k [j_k ])_{j_k}^{n_k} =1$ corresponding to the same dimension (technically, 3-dimensional arrays $\mathcal{G}_k$ ) are called the cores.

We use the symbols $\mathbf{G}_k[j_k](\alpha_{k-1},\alpha_k)$ to denote the element of the matrix $\mathbf{G}_k[j_k]$ in the position $(\alpha_{k-1},\alpha_k)$,

where $\alpha_{k-1}=1,...,r_{k-1},\alpha_k=1,...,r_k$. Equation (\ref{eq1}) can be equivalently rewritten as the sum of the products of the elements of the cores:
\begin{equation}
    \label{eq2}
    \mathcal{A}(j_1,...,j_d)=\sum_{\alpha_0,...,\alpha_d} \mathbf{G}_1[j_1](\alpha_{0},\alpha_1)\cdots \mathbf{G}_d[j_d](\alpha_{d-1},\alpha_d)
\end{equation}
The representation of a tensor $\mathcal{A}$ via the explicit enumeration of all its elements requires to store
$\prod_{k=1}^d n_k $ numbers compared with $\sum_{k=1}^d n_kr_{k-1}r_k$ numbers if the tensor is stored in the TT-format.
Thus, the TT-format is very efﬁcient in terms of memory if the ranks are small.

An attractive property of the TT-decomposition is the ability to efﬁciently perform several types of operations on tensors if they are in the TT-format: basic linear algebra operations, such as the addition of a constant and the multiplication by a constant, the summation and the entrywise product of tensors (the results of these operations are tensors in the TT-format generally with the increased ranks); computation of global characteristics of a tensor, such as the sum of all elements and the Frobenius norm.
See [17] for a detailed description of all the supported operations.
\par
Neural networks are usually trained with the stochastic gradient descent algorithm where the gradient is computed using the back-propagation procedure.
Back-propagation allows to compute the gradient of a loss-function $L$ with respect to all the parameters of the network.
The method starts with the computation of the gradient of $L$ w.r.t. the output of the last layer and proceeds sequentially through the layers in the reversed order while computing the gradient w.r.t. the parameters and the input of the layer making use of the gradients computed earlier.
Applied to tensorizing fully-connected layers  the back-propagation method computes the gradients w.r.t. the input $\bf{x}$ and the parameters $\bf{W}$ and $\bf{b}$ given the gradients $\frac{\partial L}{\partial {\bf y}}$ w.r.t to the output ${\bf y}$:
\begin{equation}
    \label{gradient}
    \frac{\partial L}{\partial {\bf x}}={\bf W}^T \frac{\partial L}{\partial {\bf y}},
    \frac{\partial L}{\partial {\bf W}}={\bf W}^T \frac{\partial L}{\partial {\bf y}}{\bf x}^T,
    \frac{\partial L}{\partial {\bf b}}={\bf W}^T \frac{\partial L}{\partial {\bf y}}
\end{equation}

In what follows we derive the gradients required to use the back-propagation algorithm with the tensorizing layers.
To compute the gradient of the loss function w.r.t. the bias vector ${\bf b}$ and w.r.t. the input vector ${\bf x}$ one can use equations (\ref{gradient}).
The latter can be applied using the matrix-by-vector product (where the matrix is in the TT-format) with the complexity of $\mathcal{O}(dr^2n\max\{m,n\}^d)=\mathcal{O}(dr^2n\max\{M,N\})$.
To perform a step of stochastic gradient descent, we can use traditional back-propagation in computational graph to compute gradient of loss function w.r.t the weight matrix $\mathbf{W}$, then we convert the gradient matrix into the TT-format using TT-SVD algorithm. Another way to learn the TensorNet parameters is to
compute gradient of loss function w.r.t the cores of the TT-representations of $\mathbf{W}$.

\par For high-dimensional matrices, the TT-SVD algorithm will meet curse of dimensionality, i.e., computation cost will increase quickly such as exponentially. Then we have difficulty to employ TT-format in neural networks. To
A Randomized Tensor Train Singular Value
Decomposition
Each of the existing TT decomposition algorithms,
including the TT-SVD and randomized TT-SVD, is successful in the field, but
neither can both accurately and efficiently decompose large-scale sparse tensors. \cite{b13} proposes a new quasi-best fast TT
decomposition algorithm for large-scale sparse tensors with proven correctness
and the upper bound of its complexity is derived. In numerical experiments,
authors verify that the proposed algorithm can decompose sparse tensors faster than
the TT-SVD, and have more speed, precision and versatility than randomized
TT-SVD\cite{b14}, and it can be used to decomposes arbitrary high-dimensional tensor
without losing efficiency when the number of non-zero elements is limited.
Faster TT-SVD algorithm can be integrated into tensorizing neural networks, and it should be more efficiently to solve the problem in large scale.

\section{Experiments}
In all experiments we will use MATLAB extension\footnote{https://github.com/Bihaqo/TensorNet} of the MatConvNet framework\footnote{http://www.vlfeat.org/matconvnet}. For the operations related to the TT-format we use the TT-Toolbox\footnote{https://github.com/oseledets/TT-Toolbox} implemented in MATLAB as well.
To show the properties of the TT-layer and compare different strategies for setting its parameters: dimensions of the tensors representing the input/output of the layer and the TT-ranks of the compressed weight matrix.
We run the experiment on the MNIST dataset for the task of handwritten-digit recognition.
As a baseline we use a neural network with two fullyconnected layers (1024 hidden units) and rectiﬁed linear unit (ReLU) and compute error on the test set.
For more reshaping options we resize the original 28 × 28 images to 32 × 32.

Futhermore, we will train several networks differing in the parameters of the single TT-layer. The networks contain the following layers: the TT-layer with weight matrix of size 1024×1024, ReLU, the fully-connected layer with the weight matrix of size 1024 × 10.
We test different ways of reshaping the input/output tensors and try different ranks of the TT-layer.
As a simple compression baseline in the place of the TT-layer we use the fully-connected layer such that the rank of the weight matrix is bounded (implemented as follows: the two consecutive fully-connected layers with weight matrices of sizes 1024 × r and r ×1024, where r controls the matrix rank and the compression factor).

% \section{Ease of Use}

% \subsection{Maintaining the Integrity of the Specifications}

% The IEEEtran class file is used to format your paper and style the text. All margins,
% column widths, line spaces, and text fonts are prescribed; please do not
% alter them. You may note peculiarities. For example, the head margin
% measures proportionately more than is customary. This measurement
% and others are deliberate, using specifications that anticipate your paper
% as one part of the entire proceedings, and not as an independent document.
% Please do not revise any of the current designations.

% \section{Prepare Your Paper Before Styling}
% Before you begin to format your paper, first write and save the content as a
% separate text file. Complete all content and organizational editing before
% formatting. Please note sections \ref{AA}--\ref{SCM} below for more information on
% proofreading, spelling and grammar.

% Keep your text and graphic files separate until after the text has been
% formatted and styled. Do not number text heads---{\LaTeX} will do that
% for you.

% \subsection{Abbreviations and Acronyms}\label{AA}
% Define abbreviations and acronyms the first time they are used in the text,
% even after they have been defined in the abstract. Abbreviations such as
% IEEE, SI, MKS, CGS, ac, dc, and rms do not have to be defined. Do not use
% abbreviations in the title or heads unless they are unavoidable.

% \subsection{Units}
% \begin{itemize}
% \item Use either SI (MKS) or CGS as primary units. (SI units are encouraged.) English units may be used as secondary units (in parentheses). An exception would be the use of English units as identifiers in trade, such as ``3.5-inch disk drive''.
% \item Avoid combining SI and CGS units, such as current in amperes and magnetic field in oersteds. This often leads to confusion because equations do not balance dimensionally. If you must use mixed units, clearly state the units for each quantity that you use in an equation.
% \item Do not mix complete spellings and abbreviations of units: ``Wb/m\textsuperscript{2}'' or ``webers per square meter'', not ``webers/m\textsuperscript{2}''. Spell out units when they appear in text: ``. . . a few henries'', not ``. . . a few H''.
% \item Use a zero before decimal points: ``0.25'', not ``.25''. Use ``cm\textsuperscript{3}'', not ``cc''.)
% \end{itemize}

% \subsection{Equations}
% Number equations consecutively. To make your
% equations more compact, you may use the solidus (~/~), the exp function, or
% appropriate exponents. Italicize Roman symbols for quantities and variables,
% but not Greek symbols. Use a long dash rather than a hyphen for a minus
% sign. Punctuate equations with commas or periods when they are part of a
% sentence, as in:
% \begin{equation}
% a+b=\gamma\label{eq}
% \end{equation}

% Be sure that the
% symbols in your equation have been defined before or immediately following
% the equation. Use ``\eqref{eq}'', not ``Eq.~\eqref{eq}'' or ``equation \eqref{eq}'', except at
% the beginning of a sentence: ``Equation \eqref{eq} is . . .''

% \subsection{\LaTeX-Specific Advice}

% Please use ``soft'' (e.g., \verb|\eqref{Eq}|) cross references instead
% of ``hard'' references (e.g., \verb|(1)|). That will make it possible
% to combine sections, add equations, or change the order of figures or
% citations without having to go through the file line by line.

% Please don't use the \verb|{eqnarray}| equation environment. Use
% \verb|{align}| or \verb|{IEEEeqnarray}| instead. The \verb|{eqnarray}|
% environment leaves unsightly spaces around relation symbols.

% Please note that the \verb|{subequations}| environment in {\LaTeX}
% will increment the main equation counter even when there are no
% equation numbers displayed. If you forget that, you might write an
% article in which the equation numbers skip from (17) to (20), causing
% the copy editors to wonder if you've discovered a new method of
% counting.

% {\BibTeX} does not work by magic. It doesn't get the bibliographic
% data from thin air but from .bib files. If you use {\BibTeX} to produce a
% bibliography you must send the .bib files.

% {\LaTeX} can't read your mind. If you assign the same label to a
% subsubsection and a table, you might find that Table I has been cross
% referenced as Table IV-B3.

% {\LaTeX} does not have precognitive abilities. If you put a
% \verb|\label| command before the command that updates the counter it's
% supposed to be using, the label will pick up the last counter to be
% cross referenced instead. In particular, a \verb|\label| command
% should not go before the caption of a figure or a table.

% Do not use \verb|\nonumber| inside the \verb|{array}| environment. It
% will not stop equation numbers inside \verb|{array}| (there won't be
% any anyway) and it might stop a wanted equation number in the
% surrounding equation.

% \subsection{Some Common Mistakes}\label{SCM}
% \begin{itemize}
% \item The word ``data'' is plural, not singular.
% \item The subscript for the permeability of vacuum $\mu_{0}$, and other common scientific constants, is zero with subscript formatting, not a lowercase letter ``o''.
% \item In American English, commas, semicolons, periods, question and exclamation marks are located within quotation marks only when a complete thought or name is cited, such as a title or full quotation. When quotation marks are used, instead of a bold or italic typeface, to highlight a word or phrase, punctuation should appear outside of the quotation marks. A parenthetical phrase or statement at the end of a sentence is punctuated outside of the closing parenthesis (like this). (A parenthetical sentence is punctuated within the parentheses.)
% \item A graph within a graph is an ``inset'', not an ``insert''. The word alternatively is preferred to the word ``alternately'' (unless you really mean something that alternates).
% \item Do not use the word ``essentially'' to mean ``approximately'' or ``effectively''.
% \item In your paper title, if the words ``that uses'' can accurately replace the word ``using'', capitalize the ``u''; if not, keep using lower-cased.
% \item Be aware of the different meanings of the homophones ``affect'' and ``effect'', ``complement'' and ``compliment'', ``discreet'' and ``discrete'', ``principal'' and ``principle''.
% \item Do not confuse ``imply'' and ``infer''.
% \item The prefix ``non'' is not a word; it should be joined to the word it modifies, usually without a hyphen.
% \item There is no period after the ``et'' in the Latin abbreviation ``et al.''.
% \item The abbreviation ``i.e.'' means ``that is'', and the abbreviation ``e.g.'' means ``for example''.
% \end{itemize}
% An excellent style manual for science writers is \cite{b7}.

% \subsection{Authors and Affiliations}
% \textbf{The class file is designed for, but not limited to, six authors.} A
% minimum of one author is required for all conference articles. Author names
% should be listed starting from left to right and then moving down to the
% next line. This is the author sequence that will be used in future citations
% and by indexing services. Names should not be listed in columns nor group by
% affiliation. Please keep your affiliations as succinct as possible (for
% example, do not differentiate among departments of the same organization).

% \subsection{Identify the Headings}
% Headings, or heads, are organizational devices that guide the reader through
% your paper. There are two types: component heads and text heads.

% Component heads identify the different components of your paper and are not
% topically subordinate to each other. Examples include Acknowledgments and
% References and, for these, the correct style to use is ``Heading 5''. Use
% ``figure caption'' for your Figure captions, and ``table head'' for your
% table title. Run-in heads, such as ``Abstract'', will require you to apply a
% style (in this case, italic) in addition to the style provided by the drop
% down menu to differentiate the head from the text.

% Text heads organize the topics on a relational, hierarchical basis. For
% example, the paper title is the primary text head because all subsequent
% material relates and elaborates on this one topic. If there are two or more
% sub-topics, the next level head (uppercase Roman numerals) should be used
% and, conversely, if there are not at least two sub-topics, then no subheads
% should be introduced.

% \subsection{Figures and Tables}
% \paragraph{Positioning Figures and Tables} Place figures and tables at the top and
% bottom of columns. Avoid placing them in the middle of columns. Large
% figures and tables may span across both columns. Figure captions should be
% below the figures; table heads should appear above the tables. Insert
% figures and tables after they are cited in the text. Use the abbreviation
% ``Fig.~\ref{fig}'', even at the beginning of a sentence.

% \begin{table}[htbp]
% \caption{Table Type Styles}
% \begin{center}
% \begin{tabular}{|c|c|c|c|}
% \hline
% \textbf{Table}&\multicolumn{3}{|c|}{\textbf{Table Column Head}} \\
% \cline{2-4}
% \textbf{Head} & \textbf{\textit{Table column subhead}}& \textbf{\textit{Subhead}}& \textbf{\textit{Subhead}} \\
% \hline
% copy& More table copy$^{\mathrm{a}}$& &  \\
% \hline
% \multicolumn{4}{l}{$^{\mathrm{a}}$Sample of a Table footnote.}
% \end{tabular}
% \label{tab1}
% \end{center}
% \end{table}

% \begin{figure}[htbp]
% \centerline{\includegraphics{fig1.png}}
% \caption{Example of a figure caption.}
% \label{fig}
% \end{figure}

% Figure Labels: Use 8 point Times New Roman for Figure labels. Use words
% rather than symbols or abbreviations when writing Figure axis labels to
% avoid confusing the reader. As an example, write the quantity
% ``Magnetization'', or ``Magnetization, M'', not just ``M''. If including
% units in the label, present them within parentheses. Do not label axes only
% with units. In the example, write ``Magnetization (A/m)'' or ``Magnetization
% \{A[m(1)]\}'', not just ``A/m''. Do not label axes with a ratio of
% quantities and units. For example, write ``Temperature (K)'', not
% ``Temperature/K''.

% \section*{Acknowledgment}

% The preferred spelling of the word ``acknowledgment'' in America is without
% an ``e'' after the ``g''. Avoid the stilted expression ``one of us (R. B.
% G.) thanks $\ldots$''. Instead, try ``R. B. G. thanks$\ldots$''. Put sponsor
% acknowledgments in the unnumbered footnote on the first page.

% \section*{References}

% Please number citations consecutively within brackets \cite{b1}. The
% sentence punctuation follows the bracket \cite{b2}. Refer simply to the reference
% number, as in \cite{b3}---do not use ``Ref. \cite{b3}'' or ``reference \cite{b3}'' except at
% the beginning of a sentence: ``Reference \cite{b3} was the first $\ldots$''

% Number footnotes separately in superscripts. Place the actual footnote at
% the bottom of the column in which it was cited. Do not put footnotes in the
% abstract or reference list. Use letters for table footnotes.

% Unless there are six authors or more give all authors' names; do not use
% ``et al.''. Papers that have not been published, even if they have been
% submitted for publication, should be cited as ``unpublished'' \cite{b4}. Papers
% that have been accepted for publication should be cited as ``in press'' \cite{b5}.
% Capitalize only the first word in a paper title, except for proper nouns and
% element symbols.

% For papers published in translation journals, please give the English
% citation first, followed by the original foreign-language citation \cite{b6}.

\begin{thebibliography}{00}
\bibitem{b1} A. Krizhevsky, I. Sutskever, and G. E. Hinton, ``Imagenet classification with deep convolutional neural networks,'' in Advances in Neural Information Processing Systems 25 (NIPS), 2012, pp. 1097–1105.
\bibitem{b2} K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,”in International Conference on Learning Representations (ICLR), 2015.
\bibitem{b3} O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla,M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” International Journal of Computer Vision (IJCV), 2015.
\bibitem{b4} D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,”Nature, vol. 323, no. 6088, pp. 533–536, 1986.
\bibitem{b5} A. Krizhevsky, “Learning multiple layers of features from tiny images,” Master’s thesis, Computer Science Department, University of Toronto, 2009.
\bibitem{b6} I. V. Oseledets, “Tensor-Train decomposition,” SIAM J. Scientific Computing, vol. 33, no. 5, pp. 2295–2317, 2011.
\bibitem{b7} L. de Lathauwer, B. de Moor, and J. Vandewalle, A multilinear singular value decomposition, SIAM J. Matrix Anal. Appl., 21 (2000), pp. 1253–1278.
\bibitem{b8} L. de Lathauwer, B. de Moor, and J. Vandewalle, On best rank-1 and rank-(R1, R2,...,RN ) approximation of high-order tensors, SIAM J. Matrix Anal. Appl., 21 (2000), pp. 1324–1342.
\bibitem{b9} R. Bro, PARAFAC: Tutorial and applications, Chemometrics Intell. Lab. Syst., 38 (1997), pp. 149–171.
\bibitem{b10} L. Grasedyck, Existence and computation of low Kronecker-rank approximations for large systems in tensor product structure, Computing, 72 (2004), pp. 247–265.
\bibitem{b11} Novikov, A., Podoprikhin, D., Osokin, A., Vetrov, D. P. (2015). Tensorizing neural networks. In Advances in neural information processing systems (pp. 442-450).
\bibitem{b12} J. Ba and R. Caruana, “Do deep nets really need to be deep?” in Advances in Neural Information Processing Systems 27 (NIPS), 2014, pp. 2654–2662.
\bibitem{b13} Li, L., Yu, W., Batselier, K. (2019). Faster Tensor Train Decomposition for Sparse Data. arXiv preprint arXiv:1908.02721.
\bibitem{b14} Huber, B., Schneider, R., Wolf, S. (2017). A randomized tensor train singular value decomposition. In Compressed Sensing and its Applications (pp. 261-290). Birkhäuser, Cham.
\end{thebibliography}
\vspace{12pt}
% \color{red}
% IEEE conference templates contain guidance text for composing and formatting conference papers. Please ensure that all template text is removed from your conference paper prior to submission to the conference. Failure to remove the template text from your paper may result in your paper not being published.

\end{document}


\section{Partial Adjustment Model}


    \filetitle{yearly}{Display tseries object one full year per row}{tseries/yearly}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
yearly(X)
\end{verbatim}

\paragraph{Input arguments}\label{input-arguments}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \texttt{X} {[} tseries {]} - Tseries object that will be displayed one
  full year of observations per row.
\end{itemize}

\paragraph{Description}\label{description}

\paragraph{Example}\label{example}


\section{Games}

\project{Games!}{Computers and electronics are not just about having fun---you can play games as well!}

\subsection*{Equipment Required}

The circuit built in Section~\ref{s:buzzer}.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.75\textwidth]{buzzer-circuit}
\end{figure}

\subsection*{Exercises}

In this worksheet, you're not going to be given any code.  You are just going to be given some ideas for games you can make and play using the CamJam EduKit, and some hints that you might need.

\subsubsection*{Hints}

Random numbers can be generated with `RND(1)'.  This will generate a random number between 0 and 1.  To generate a number between 1 and X, you can use `INT(RND(1)*X+1)' as in the following code:
\begin{basic}
10 PRINT "RANDOM NUMBER BETEWEEN 1 AND 10: ";
20 PRINT INT(RND(1)*10+1)
\end{basic}

Print out instructions for the user to the screen using the `PRINT' command:
\begin{basic}
10 PRINT "TELL THE USER WHAT TO DO"
\end{basic}

\subsubsection*{Game 1---Reaction Timer}

Light the LEDs one at a time, going from Red to Green with one second in between.  After the green has been lit, wait a random length of time (say, between 1 and 5 seconds) before sounding the buzzer.  Time how long it takes from the buzzer sounding to the player pressing the button.

\subsubsection*{Game 2---Eat the Orange}

Randomly choose which LED to light.  Light it for 0.2 seconds.  If the player presses the button while the Amber LED is lit, they get a point.  Time how long it takes for them to get 10 points.

\subsubsection*{Game 3---Segment of Orange}

Change the colour of the LEDs from red, to amber, to green, to amber, and back to red every 0.1s.  If the player presses the button when the amber LED is lit, they get a point.  Only give them 10 chances to hit the button when the amber is lit.


\section{Representation}

\subsection{Nodes}
Nodes for the main decision tree, or ADFs, comprise decision and end nodes. Decision nodes take a number of children and select one of them based on the value of a patient's attribute. There are 7 decision nodes - one for each attribute, excluding \emph{COMFORT} as it contains missing values. The number of children for decision nodes depends on the possible values for their associated attributes. The Blood Pressure (\emph{BP}) node, for instance, has 3 children - one for \emph{low}, one for \emph{mid} and one for \emph{high}. End nodes represent one of the three discharge decisions (\emph{I}, \emph{S} or \emph{A}); or, in the main decision-tree, end nodes may also reference one of 3 ADFs. The representation of ADFs is further explained in section \ref{sec:individuals}. The end nodes and decision nodes are described in table \ref{tab:nodes}.

\begin{table}[H]
\resizebox{\textwidth}{!}{\begin{tabular}{|l|l|l|}
\hline
\textbf{Symbol} & \textbf{Arity} & \textbf{Description}                                                                                \\ \hline
IT              & 3              & \begin{tabular}[c]{@{}l@{}}The internal temperature decision node\\ - If the patient's internal temperature is low then select the first child\\ - If the patient's internal temperature is mid then select the second child\\ - If the patient's internal temperature is high then select the third child\end{tabular}                                                                        \\ \hline
ITS             & 3              & \begin{tabular}[c]{@{}l@{}}The internal temperature stability decision node\\ - If the patient's internal temperature is unstable then select the first child\\ - If the patient's internal temperature is moderately stable then select the second child\\ If the patient's internal temperature is stable then select the third child\end{tabular}                                           \\ \hline
ST              & 3              & \begin{tabular}[c]{@{}l@{}}The surface temperature decision node\\ - If the patient's surface temperature is low then select the first child\\ - If the patient's surface temperature is mid then select the second child\\ - If the patient's surface temperature is high then select the third child\end{tabular}                                                                            \\ \hline
STS             & 3              & \begin{tabular}[c]{@{}l@{}}The surface temperature stability decision node\\ - If the patient's surface temperature is unstable then select the first child\\ - If the patient's surface temperature is moderately stable then select the second child\\ - If the patient's surface temperature is stable then select the third child\end{tabular}                                             \\ \hline
BP              & 3              & \begin{tabular}[c]{@{}l@{}}The blood pressure decision node\\ - If the patient's blood pressure is low then select the first child\\ - If the patient's blood pressure is mid then select the second child\\ - If the patient's blood pressure is high then select the third child\end{tabular}                                                                                \\ \hline
BPS             & 3              & \begin{tabular}[c]{@{}l@{}}The blood pressure stability decision node\\ - If the patient's blood pressure is unstable then select the first child\\ - If the patient's blood pressure is moderately stable then select the second child\\ - If the patient's blood pressure is stable then select the third child\end{tabular}                                                                 \\ \hline
OS              & 4              & \begin{tabular}[c]{@{}l@{}}The oxygen saturation decision node\\ - If the patient's oxygen saturation is poor then select the first child\\ - If the patient's oxygen saturation is fair then select the second child\\ - If the patient's oxygen saturation is good then select the third child\\ - If the patient's oxygen saturation is excellent then select the fourth child\end{tabular} \\ \hline
I               & 0              & The end node for the patient to be sent to an intensive care unit                                                                                \\ \hline
S               & 0              & The end node for the patient to be sent home                                                                                \\ \hline
A               & 0              & The end node for the patient to be sent to the general hospital floor                                                                                \\ \hline
ADF0            & 0              & Returns the result of executing the first ADF of the individual.                                                                                \\ \hline
ADF1            & 0              & Returns the result of executing the second ADF of the individual.                                                                                \\ \hline
ADF2            & 0              & Returns the result of executing the third ADF of the individual.                                                                                \\ \hline
\end{tabular}}
\caption{Decision Tree Nodes}
\label{tab:nodes}
\end{table}

\subsection{Individuals}\label{sec:individuals}
Each chromosome within a population consists of a 4 decision trees for determining the discharge decision for a patient. As per the architecture implemented by Bruce \cite{bruce1995application}, the first decision tree represents the main program while the other three trees represent ADFs.

The function set and terminal set for the main program are made up of all the available nodes in table \ref{tab:nodes}. For the ADFs the terminal sets all consist of only \emph{I}, \emph{S} and \emph{A} - hence, there are no references between the function-defining trees. In addition - the function set for the first ADF consists of \emph{BP} and \emph{BPS}, the function set for the second ADF consists of \emph{IT} and \emph{ITS}, and the function set for the third ADF consists of \emph{ST} and \emph{STS}.

Given the function and terminal sets described above for the main program and each ADF, an example of an individual is depicted in figure \ref{fig:individual}.

\begin{figure}[H]
\centering
\includegraphics[width=\textwidth]{report/02_representation/chromosome.png}
\caption{Example of an Individual}
\label{fig:individual}
\end{figure}

\subsection{Populations}
Population representation and generation is the same as that applied by the previous genetic programming approach that did not incorporate modularization. To reiterate, the genetic algorithm made use of a single population consisting of 2048 individuals. This means that only a single population was evolved and cross-population breeding was not applied.

The initial population was generated using the ramped-half-and-half method proposed by Koza \cite{koza1992genetic} to enhance population diversity of structure. The maximum tree depth allowed during the initial population generation was 4. So, in other words, the population is divided equally among individuals to be initialized with trees having depths 2, 3, and 4. For each depth group, half of the trees are initialized with the full technique and half with the grow technique.

In addition, to cover as much of the search space as possible an effort was made to reduce duplicates within the initial population. The strategy used to reduce the number of duplicates was as follows: if a duplicate individual is created as part of the initial population, it will be regenerated up to 100 times to try replace it with a new, original individual. After 100 attempts, if no unique individual was generated, the duplicate individual will be used.


\subsection{Raman Spectroscopy}

The inelastic scattering of light is referred to as Raman scattering. When photons exchange energy with the material during a scattering process they can either lose energy (Stokes process) or gain additional energy (anti-Stokes process). This energy in the material is vibrational energy, also known as phonons. The process of transferring energy can also be described as
the inelastic collision of a photon with a phonon.

By gaining or losing energy the photon changes it's frequency. This shift in the photon's
frequency is referred to as the Raman shift or Raman frequency. The Stokes shifted frequency is

\begin{equation}
  \omega_S=\omega_i-\omega_0
\end{equation}

with $\omega_i$ being the frequency of the incoming light and $\omega_0$ the phonon frequency.
The anti-Stokes shifted frequency is shifted by the same amount $\omega_0$ as

\begin{equation}
  \omega_{AS}=\omega_i+\omega_0.
\end{equation}


% !TeX program = pdflatex
% !BIB program = biber

\blindmathfalse

\section{Conclusion}
\label{sec:Conclusion}

Cite some more papers \citep{Yaari1965, Warner2001, Davidoff2005, Benartzi2011}. Let's cite a~book: \cite{Luce1959}. Let's cite a~contribution to a~collected volume: \cite{Harrison2008} and a~collection (an~edited volume) itself: \cite{Kagel2016}. Now let's cite presentations at conferences: \cite{Beute2012, Vosgerau2008}. \cite{Attema2016} propose a~method for ``measuring discounting without measuring utility''\footnote{The basic idea of their method is intriguingly simple: Imagine an~individual who is indifferent between, say, Option~A: \$10 today and Option~B: \$10 in one year plus \$10 in two years. With a~constant annual discount factor $\delta$, this indifference translates to $u(\$10) = \delta\,u(\$10) + \delta^2\,u(\$10)$, so that $u(\$10)$ cancels out, and $\delta$ can be readily calculated as the solution to $1 = {\delta + \delta^2}$.}.

\blindtext


\subsection{Formalization}

\begin{table}[ht]
\centering
\begin{spacing}{2.0}
\begin{tabular}{rll}
\v{program} &$::=$&
  $\overline{\v{s}}$
\\
\v{s} &$::=$&
    \k{Definition} \ \v{n} \ \k{:} \ t \ \k{:=} \ \v{e} \ \k{.} \\ &&
\|  \k{Signature} \ \v{n} \ \k{:=} \ t \ \k{.} \\ &&
\|  \k{Assumption} \ \v{n} \ \k{:} \ t \ \k{.}
\\
\v{e} &$::=$&
    \v{n}
\|  \v{primitive-value}
\|  \k{(} \ \v{e}$\p$ \ \v{e} \ \k{)}
\|  \k{(} \ \k{let} \ \v{n} \k{:=} \v{e} \ \k{in} \ \v{e} \ \k{)} \\ &&
\|  \k{(} \ \k{fun} \ \v{n} \ \k{\maparrow} \ \v{e} \k{)}
\|  \k{(} \ \k{rec} \ \v{n} \ \k{of} \ \v{n} \ \k{\maparrow} \ \v{e} \ \k{)}
\\
\v{t} &$::=$&
    \v{n}
\|  \v{primitive-type}
\|  \k{(} \ \v{t} \ \k{->} \ \v{t} \ \k{)}
\|  \k{(} \ \v{t} \ \v{t} \ \k{)} \\ &&
\|  \k{(} \ \k{forall} \ \v{n} \ \k{,} \ \v{t} \ \k{)}
\\
\v{n} &$::=$&
      \v{string}
\\
\end{tabular}
\end{spacing}
\end{table}

\subsection{Notation}

\begin{itemize}
\item An expression with an overline may be repeated any natural number of times.
\end{itemize}


\documentclass[a4paper]{article}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{geometry}
\usepackage{enumerate}
\usepackage{natbib}
\usepackage{float}%稳定图片位置
\usepackage{graphicx,subfig}%画图
\usepackage{caption}
\usepackage[english]{babel}
\usepackage{indentfirst}%缩进
\usepackage{enumerate}%加序号
\usepackage{multirow}%合并行
\usepackage{hyperref}
\hypersetup{hypertex=true, colorlinks=true, linkcolor=black, anchorcolor=black, citecolor=black}
\title{\Large \textbf{VP390 Problem Set 2}\\
\author{\textbf{Pan, Chongdan ID:516370910121}\\
}
}
\begin{document}
\maketitle
\section{Problem 1}
\begin{enumerate}[(a)]
    \item $m(u^2)=\frac{m}{\sqrt{1-u^2/c^2}}=\sqrt{3}m$
    \\Since $m_{ij}=m(u^2)[\delta_{ij}+\frac{1}{c^2-u^2}u_iu_j]=\sqrt{3}m(\delta_{ij}+\frac{3}{c^2}u_iu_j)$
    \\Then $mij=\sqrt{3}m\left(
    \begin{array}{ccc}
      1+1 & 0 & 1\\
      0 & 1 & 0\\
      1 & 0 & 1+1\\
    \end{array}
    \right)=\sqrt{3}m\left(
    \begin{array}{ccc}
        2 & 0 & 1\\
        0 & 1 & 0\\
        1 & 0 & 2\\
      \end{array}
  \right) $
  \item $F=ma=\sqrt{3}m\left(\begin{array}{ccc}
    2 & 0 & 1\\
    0 & 1 & 0\\
    1 & 0 & 2\\
  \end{array}\right)\left(\begin{array}{ccc}
    0\\
    0\\
    1\\
  \end{array}\right)=\sqrt{3}m\left(\begin{array}{ccc}
    1\\
    0\\
    2\\
  \end{array}\right)$
  \begin{figure}[H]
    \centering
    \includegraphics[scale=0.25]{P1.png}
    \caption{The y component is all 0, so the plot only show the x-z plane}
    \label{P1}
\end{figure}

\end{enumerate}
\section{Problem 2}
\quad
\\$E^2=p^2c^2+E_0^2$ where $E_0=mc^2$
\\Since $p=p_1,E=E_1+E_2$
\\Then $m^2=m_1^2+m_2^2$
\\So the new particle's rest mass m is $\sqrt{m_1^2+m_2^2}$
\\$E=K_1+m_1c^2+m_2c^2=\frac{mc^2}{\sqrt{1-u^2/c^2}}$
\\Then the new particle's velocity is $c\sqrt{1-\frac{m_1^2c^4+m_2^2c^4}{(K_1+m_1c^2+m_2c^2)^2}}$
\section{Problem 3}
\quad
\\If it's possible, then the total energy for the electron after collision is :
\\$E_{electron}=E_{photon}+E_{0}$ where $E_0=mc^2$
\\For the momentum $p=\frac{E_{photon}}{c}$
\\Since $E^2=p^2c^2+E_0^2$
\\Then $E_{electron}=\sqrt{E_{photon}^2+E_{0}^2}\neq E_{photon}+E_{0}$
\\So it's impossible.
\section{Problem 4}
\quad
\\$L_z=n^2\hbar^2$
\\$L_{\mathrm{ph}}=n'^2\hbar^2-n^2\hbar$
\section{Problem 5}
\quad
\\$R(\lambda)=\frac{1}{4}cu(\lambda)=\frac{2\pi}{\lambda^5}\frac{hc^2}{e^{hc/\lambda k_\mathrm{B}T}-1}$
\\Let $R(\lambda)'=0$, we can get $\frac{2c^3h^2\pi e^{ch/k_\mathrm{B}T\lambda}}{(-1+e^{ch/k_\mathrm{B}T\lambda})^2 k_\mathrm{B}T\lambda^7}-\frac{10c^2h\pi}{(-1+e^{ch/k_\mathrm{B}T\lambda})\lambda^6}=0$
\\Ignore the root where $\lambda=0$ we can get $\frac{che^{ch/k_\mathrm{B}T\lambda}}{(-1+e^{ch/k_\mathrm{B}T\lambda})\lambda k_\mathrm{B}T}=5$
\\Let $x=\frac{ch}{k_\mathrm{B}T\lambda}$ we get $\frac{xe^x}{e^x-1}=5$
\\The solution $x\approx4.96$, so $\lambda_mT=\frac{xk_\mathrm{B}}{ch}\approx2.9\times10^{-3}(m\cdot K)$
\section{Problem 6}
\begin{enumerate}[(a)]
    \item \quad
    \\Since $\lambda_2-\lambda_1=\frac{h}{mc}(1-\cos\theta)$
    \\Then $\frac{c}{\nu_2}=\frac{h}{mc}(1-\cos\theta)+\frac{c}{\nu}$
    \\The energy of scattered photon is $h\nu_2=\frac{hc}{\frac{h}{mc}(1-\cos\theta)+\frac{c}{\nu}}=\frac{hm\nu c^2}{h\nu+mc^2-h\nu\cos\theta}$
    \\\\For the electron, the total energy will be $E_e=h\nu+mc^2-h\nu_2$, and $K_c=E_e-mc^2=h\nu-h\nu_2$
    \\$K_c=E_e-mc^2=h\nu(1-\frac{mc^2}{h\nu+mc^2-h\nu\cos\theta})$
    \\When $\cos\theta=-1$, we can get the max $K_c=\frac{h\nu}{1+\frac{mc^2}{2h\nu}}$
    \item \quad
    \\$K_c=\frac{2E^2}{2E+mc^2}$, then $2E^2-2EK_c-mc^2K_c=0$
    \\$E=\frac{2K_c\pm\sqrt{4K_c^2+8mc^2K_c}}{4}$, keep the positive solution we can get
    \\$E=1.133\times10^{-13}J$
    \\$\lambda=\frac{c}{\nu}=\frac{hc}{E}=1.75\times10^{-12}m$
\end{enumerate}
\section{Problem 7}
    \quad
    \\$\Psi_{12}=|\Psi_1+\Psi_2|^2=|\Psi_1|^2+|\Psi_2|^2+2\mathrm{Re}(\Psi_1\Psi_2^*)$
    \\$\Psi_1=\frac{1}{\sqrt{2}}e^{-\frac{y^2}{2}}\cos(\omega t-ay)+\frac{1}{\sqrt{2}}e^{-\frac{y^2}{2}}\sin(\omega t-ay)i$
    \\$\Psi_1=\frac{1}{\sqrt{2}}e^{-\frac{y^2}{2}}\cos(\omega t-ay-by)+\frac{1}{\sqrt{2}}e^{-\frac{y^2}{2}}\sin(\omega t-ay-by)i$
    \\$|\Psi_1|^2=|\Psi_2|^2=\frac{e^{-y^2}}{2}$
    \\$\mathrm{Re}(\Psi_1\Psi_2^*)=\frac{e^{-y^2}}{2}\cos(\omega t-ay)\cos(\omega t-ay-by)+\frac{e^{-y^2}}{2}\sin(\omega t-ay)\sin(\omega t-ay-by)$
    \\$\mathrm{Re}(\Psi_1\Psi_2^*)=\frac{e^{-y^2}}{2}\cos by$
    \\$\Psi_{12}=e^{-y^2}(1+\cos by)$
\end{document}


\documentclass[submission,copyright,creativecommons]{eptcs}
\providecommand{\event}{TeaseLP} % Name of the event you are submitting to

\usepackage{amsmath}
\usepackage{amssymb}

\usepackage{breakurl}             % Not needed if you use pdflatex only.
\usepackage{underscore}           % Only needed if you use pdflatex.
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{cleveref}

\hypersetup{hidelinks}

\title{Accelerating Program Synthesis in miniKanren}
\author{Robert Zinkov
\institute{University of Oxford \\
Oxford, UK}
\email{zinkov@robots.ox.ac.uk}
\and
Michael Ballantyne
\institute{Northeastern University\\
Boston, MA, USA}
\email{ballantyne.m@northeastern.edu}
\and
Gregory L. Rosenblatt
\institute{University of Alabama at Birmingham\\
Birmingham, AL, USA}
\email{gregr@uab.edu}
\and
William E. Byrd
\institute{University of Alabama at Birmingham\\
Birmingham, AL, USA}
\email{webyrd@uab.edu}
}
\def\titlerunning{Accelerating Program Synthesis in miniKanren}
\def\authorrunning{R. Zinkov et al.}
\begin{document}
\maketitle

\begin{abstract}
  While many logic programming systems like miniKanren are highly
  expressive, they suffer from long and unpredictable running times.
  The challenge comes from the search algorithm being usually an
  uninformed search. Through the domain of program synthesis
  we show that it possible to greatly speedup this search by guiding
  it using example programs.

\end{abstract}

\section{Introduction}

In the miniKanren\cite{byrd2017unified} programming language, we often
pose relations in a compostional manner, where more complicated
relations are defined in terms of simpler relations. These simple
relations include $\equiv$ defining constraints and $\textbf{cond}^e$
for defining a logical disjunction over clauses. In $\textbf{cond}^e$
we have to make a decision over the order we explore each of the
clauses. If we pick well, we quickly find an answer that satisfies our
constraints. If we pick less well, we hope we quickly find a
constraint violation and backtrack immediately. If we pick badly, it
could take a long while to get an answer at all.

Now, it can feel like there might an optimal way to arrange the
clauses but this runs into several problems. Firstly, our instincts of
which clause to try first are in many settings not obvious or intuitive. Secondly,
what ordering might make sense in the absence of constraints might not make
sense in their presence. But most importantly, the best ordering is highly
contextual and when dealing with relations which are recursively-defined
the best ordering could be different from call to call.

To deal with this issue we take a cue from existing machine learning
work\cite{zaremba2014learning,lee2018accelerating} and learn a search
heuristic to guide the search. More specifically, as outlined in
\Cref{fig:overview} we learn a probabilistic model to choose which
clause to pick given the context the choices exist within.

As an example consider a grammar like the lambda-calculus. Suppose we
are trying to generate the operand $e_2$ of a function application. We
can use the context of being the operand in the expression to decide
what program we generate. To decide what expressions to generate we
might look at a set of example programs and count in the operand
position what is most likely a symbol, a function abstraction, or a
function application. Then we should bias our search to look more like
example programs. Particularly, if we are trying to generate programs
that resemble a given set of example programs.

\begin{figure}
\begin{eqnarray*}
  e & := & x \; | \; (e_1 \; e_2) \; | \; (\lambda \; (x) \; e_1) \\
\end{eqnarray*}
\caption{Example grammar}
\end{figure}

This can be reasonable for small grammars like the lambda calculus, but
what should we do if we end up in a context we have never seen before?
In that case, we don't want to say an expression shouldn't be generated
even if we don't possess an example. So we assume every choice was seen
at least once or what is sometimes called Laplacian smoothing.

We train this model by collecting a dataset of (context, choice)
pairs. These are created in a domain-specific way from example
programs that be converted into these (context, choice) pairs in an
entirely offline manner. For each pair we count what choices were made
per context. These counts are then saved into a table that our
implementation of $\textbf{cond}^e$ accesses selecting clauses from
most to least probable, thus guaranteeing the search stays complete.

\begin{figure}[htb!]
\centering
\includegraphics[scale=1.0]{drawing.pdf}
\caption{Overview of the system}
\label{fig:overview}
\end{figure}

\section{Experiments}

We show a preliminary evaluation in the table below. The following
results come from trying to complete a program synthesis
task. Synthesis is defined as satisfying a query in a relational
interpreter written in miniKanren. We first try to synthesise programs
where an expert chooses the ordering of the different rules in the
grammar, and then with our method where we use the probabilities of
a learned model to select the grammar rule.

The interpreter is for a subset of the Scheme language, and we are
thus able to train using a collection of programs sourced from the
Little Schemer and the Seasoned Schemer.

\begin{table}[ht]
\centering
\caption{Times for completing synthesis test in seconds}
\begin{tabular}[t]{lcc}
\toprule
Program&Expert ordering&$n$-gram directed ordering\\
\midrule
append&0.9809&0.5906\\
reverse&--&0.0115\\
rember&0.0103&0.007\\
foldr&2.3881&0.0064\\
\bottomrule
\end{tabular}
\end{table}

We show that in isolation this optimisation yields
significant speedups on several programs. For some programs, like
reverse, the original program did not find a solution even when given
several minutes.

\section{Future Work}

The present method is highly specialised to the program synthesis task
of Barliman, but there is a significant possibility that if we had a
dataset of queries and query answers, we could generalise our results
for miniKanren programs in general.

%\nocite{*}
\bibliographystyle{eptcs}
\bibliography{paper}
\end{document}


\section{Model}
In this work we use a UNet and attempt to perform image segmentation a set of
representative streamlines for two common flow types: ABC and Radial. Figure
\ref{fig:unet} shows an example of a UNet model. A UNet was chosen because it
has an encoding and decoding side. One the input it down samples an image and
on the output side it upsamples, training on different input and output images,
commonly called masks.

\begin{figure}[ht]
\centering
    \includegraphics[width=0.5\textwidth]{unet.jpg}
    \caption{Unet model~\cite{Ronneberger}}
    \label{fig:unet}
\end{figure}

In this work we generated dense and ``sparse" streamlines, using the sparse
as the image mask.

\subsection{Creating Streamlines}
Included in the code package is a C++ file called ``advect.cpp" that was written
to generate streamline files. At the top of this code are options to change
how and what streamlines are generated and where the files will be written to.
This file contains functions to generate ABC, Radial, and Gyre flow fields.
Generally we used 100 lines for the dense representation and 10 for the sparse.
Other sizes were tried but these sizes led to fast computation and did not
change the results of the network.

The C++ file generates files with the extension ``.lines". These files can be
used to visualize our streamlines in a program such as Visit~\cite{HPV:VisIt}
A python script, called ``visit\_script.py", was used to generate all images
from a specified directory and output them in another directory. Finally a
script, called ``convertToPIL.py", was used to convert these images into
Python's Imaging Library (PIL) format, which consists of 3-Channels and can be
easily read by PyTorch. This convert file was used because it simplified the
process and greatly reduced the size of the data, as the dense streamline
files can easily take a few hundred gigs of storage space.
These files also allowed us to modify images fairly easily.

\subsection{Training and Evaluation}
For the UNet model we used Pytorch-UNet~\cite{milesial} as a base and modified
the training and prediction algorithms. Changes were made to the network to both
update for the newer versions of PyTorch, changes to the network were made to
allow for use of our input images and changing kernel sizes, as well as options
were added to test different different hyperparameters, and of course
debugging the original author's work. This lead to a much
better understanding of PyTorch than the authors of this work previously had.

Initially the network was trained on large images, size 1024x1024. It was found
that this was taking substantial time, was not training well, and did not fit
on the GPU. Images were then tried at 512x512 and finally 184x184. The final
size was found to fit on the GPU, still contained images where the operators
could determine streamlines, and new test cases could be generated quickly.

The work started by focusing on Radial images, as these were the simplest.
Initial test sizes included about 5000 input images and corresponding masks.
A validation size of 20\% was constantly used throughout the experimentation.
An epoch size of 1000 was set to run over night and took many hours to complete.
Ultimately this did not prove fruitful and unfortunately the loss did not
vary significantly between epochs.
Figure~\ref{fig:Rad} shows a sample image from the input set~\ref{fig:Radial_in},
a sample mask~\ref{fig:Radial_mask}, and the resultant output from the test
\ref{fig:output}.

\begin{figure}[ht]
    \centering
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[width=0.4\linewidth]{input.png}
        \caption{Radial Input}
        \label{fig:Radial_in}
    \end{subfigure}
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[width=0.4\linewidth]{mask.png}
        \caption{Radial Mask}
        \label{fig:Radial_mask}
    \end{subfigure}
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[width=0.4\linewidth]{out.png}
        \caption{Output from Network}
        \label{fig:output}
    \end{subfigure}
    \caption{Input and mask of Radial Streamlines}
    \label{fig:Rad}
\end{figure}

As this did not go well an attempt was made using ABC inputs and masks. Similarly
this did not result in a good output. Figure~\ref{fig:adam} shows a
representative out output of the absolute value of the loss vs the number of
epochs the network was trained on. All test cases showed similar results.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.4\linewidth]{ADAM_Loss_lr0p05_batch20.png}
    \caption{Loss per epoch using ADAM}
    \label{fig:adam}
\end{figure}

After ABC and Radial failed an experiment was done just to see if have a
a combination of the two would produce different results. 500 images from each
were randomly selected and used to train, for a total of a thousand images.
Unfortunately this did not result in a change in the loss. We tried SGD as
well as ADAM and batched up a process to test vary learning rates, momentum,
and batch sizes. The number of epochs was reduced to 20 as it was judged that if
learning did not start by that point it was unlikely to learn in longer
iterations. We also tried to change the depth of the network but this did not
prove to be fruitful. Adding an extra layer did not change the loss in a
significant way nor did it show that there was any improvement between
epochs.
It was decided to not test the Gyre images as these were more
complicated and if the UNet could not learn a simple Radial streamline
it was not going to learn more complex features.