\documentclass[a4paper,12pt]{article}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{float}
\usepackage{subfigure}
\usepackage{geometry}
\pagestyle{plain}
\usepackage{listings}
\usepackage{multirow}
\geometry{left=2.2cm, right=2.2cm, top=3cm, bottom=3cm}
\usepackage{algorithm}
\usepackage{algpseudocode}
\renewcommand{\algorithmicrequire}{\textbf{Input:}}
\renewcommand{\algorithmicensure}{\textbf{Output:}}

\title{A Bayesian Perspective on Distributional Deep Q-Learning}
\date{\today}

\author{Ni Chengzhuo\footnote{This is the homework for the course Bayesian Theory and Computation.}\\
1500011343\\
School of Mathematical Science, Peking University\\
}

\usepackage{cite}
\begin{document}
\maketitle

\abstract{We propose a Bayesian framework for Q-value estimation. We utilize the Bayesian neural network to form a posterior distribution of the Q-value given the historical data. Under this framework, both the action and the estimation of the target value can be sampled from the posterior distribution, which allows for a more robust estimation as well as encouraging exploration. We show that using the posterior mean of the Bayesian Q-value distribution, rather than a deterministic network, improves the stability and performance of Q-Learning methods in several platforms.}

\section{Introduction}
One of the primary goals of artificial intelligence is to solve complex tasks from unprocessed, high-dimensional input. Recently, significant progress has been made combining advances in deep learning with reinforcement learning, resulting in the ``Deep Q Network'' (DQN)\cite{DQN} algorithm and its variations\cite{DDQN, PDQN, DuelDQN} that are capable of human level performance on many platforms.

However, as these methods usually deal with unstable environments with high variance, distributional perspectives of function approximations have begun to gain popularity.  Currently, such methods typically use many approximators to fit a distribution\cite{DisDQN, BDQN}. However, recent advances in Bayesian inference using deep neural networks have been successful in providing estimates with uncertainty\cite{Dropout, BNN, BNN2, BNN3}. Another problem in RL is the exploration-exploitation problem, which has become a main focus for researchers nowadays. One effective solution to the problem is the Thompson Sampling Method\cite{TS, TS2, TS3}, whose exploration strategy is based on the posterior distribution of the model parameters. However, since calculating the posterior is usually intractable for complex models (e.g. neural network), the method has not been widely applied in deep reinforcement learning so far.

In this work, we develop an approach where a Bayesian neural network (BNN) is utilized as a Q-value function approximator in Deep Q-Learning. We apply several methods\cite{BNN, Dropout} from previous work to get an approximate posterior distribution of the network weights and then form a posterior distribution of the Q-value estimation. With this estimated distribution, we develop a distributional perspective on Q-Learning that is different from the previous work\cite{DisDQN}. Based on our structure, we are also able to apply the Thompson Sampling method, which is shown to be more effective when compared to other exploration strategies such as $\epsilon$-greedy or Boltzmann method. We demonstrate the significance of using Bayesian method across a range of Q-Learning methods including DQN, Double DQN, DQN with prioritized replay memory on several toy platforms.

\section{Background and related work}
\subsection{Reinforcement Learning Background}
In the standard reinforcement learning setting, an agent interacts with an environment over a number of time steps. At each time step $t$, the agent receives a state $s_t$ and selects an action $a_t$ from a set of possible actions $\mathcal{A}$ according to its policy $\pi$. Then the agent receives the next state $s_{t+1}$ and a scalar reward $r_t$. The process continues until the terminal state is reached. The return $Z_t = \sum_{k=0}^\infty \gamma^kr^{t+k}$ is the discounted accumulated reward from time step $t$ with discount factor $\gamma \in (0, 1]$. The goal of the agent is to maximize the expected return from each state $s_t$.

The Q-value $Q^{\pi}(s, a) = \mathbb{E}[Z_t|s, a]$ is the expected return for selecting action $a$ in state $s$ and following policy $\pi$ afterwards. The optimal Q-value $Q^*(s,a) =\max_\pi Q^\pi(s,a)$ is the maximum value for state $s$ and action $a$ achievable by any policy. In DQN, the policiy $\pi(s, a)$ and the value function $Q(s, a)$ are represented with deep neural networks. At each step, given the current state, the agent selects an action $\epsilon$-greedily according to the action values, and adds the transition $(s_t , a_t , r_t, S_{t+1})$ to a replay buffer. The parameters of the neural network are optimized using stochastic gradient descent to minimize the loss

\begin{equation}
L = \mathbb{E}(r_t +\gamma \max_{a'}Q_{\bar{\theta}}(s_{t+1},a')-Q_\theta(s_t,a_t))^2
\end{equation}

In practice, the optimization is performed on mini batches sampled from the replay buffer.

\subsection{Distributional Perspectives in RL}
Several recent works have investigated the distributional perspectives in reinforcement learning. In \cite{BAC}, a Bayesian framework was applied to the actor-critic architecture by fitting a Gaussian Process (GP) for the critic which allowed for a closed-form derivation of update rules. In \cite{Dropout}, dropout uncertainty estimates of a Q-value function were used for Thompson sampling and optimistic exploration. In \cite{DisDQN}, both the structure and the update rule of the value function are based on a distributional perspective. Similarly, in \cite{BDQN}, the authors use k-heads on the Q-value to model a distribution rather than the dropout approach of \cite{Dropout}.

Here we use the Bayesian method to build a posterior distribution for the Q-value, which is similar to the work of \cite{BAC}, but we apply the procedure to the Q-learning framework rather than the actor-critic framework. Also, we build our model via a more complicated neural network rather than the relatively simple Gaussian Process, which is capable to fit more complex distributions.

\subsection{Bayesian Neural Network}
Given training inputs $X = \{x_1, \cdots , x_N\}$ and their outputs $Y = \{y_1 , \cdots, y_N\}$, in the Bayesian framework we would like to infer a distribution over parameters $w$ of a function $y = f^w(x)$ that could have generated the outputs. Following the Bayesian approach, we first put some prior distribution over the parameters $p_0(w)$. We further need to define a likelihood over the outputs given the inputs $p(y|x, w)$, which is usually assumed to be a softmax function for classification or a Gaussian distribution for regression. Given a dataset $X, Y$, we then look for the posterior distribution of the parameters: $p(w|X, Y)$. With it we can predict an output for a new input point $x^∗$ by integrating

\begin{equation}
p(y^* | x^*, X, Y) = \int p(y^*|x^*, w)p(w|X, Y)dw
\end{equation}

When the prior distribution is placed on a neural network's weights $w = \{W_i\}^L_{i=1}$ , the model is called a Bayesian neural network(BNN). In practice, given weight matrices $W_i$ at layer $i$, we often place Gaussian prior distributions over the weights, i.e., $p_0(W_i) = N (W_i ; 0, \sigma^2 I)$. Under this setting, both the parameters and the predictions of the network turn from deterministic values into variables from specific distributions. A comparison between the usual neural network and the Bayes neural network is shown in the following figure,

\begin{figure}[htb!]
\center
\includegraphics[scale=0.6]{bnn}
\caption{Left: each weight has a fixed value. Right: each weight is assigned a distribution.}
\end{figure}

\section{A Bayesian Perspective on Q-learning}
\subsection{The Bayesian Estimation of Q-value}
In the previous setting of Q-learning, the Q-value $Q^w(s_t, a_t)$ is a deterministic function parameterized by $w$ which estimates the expectation of the return $Z_t$. This deterministic output version is not compatible with the Bayesian method, since a likelihood is needed to compute a posterior distribution over the weights, which becomes a tricky delta function $\delta_{x^*}(\cdot)$ in the deterministic case. To apply the Bayesian method, the output of the Q-value function must first be modified from a single expectation value to a distributional estimation for $Z_t$,

\begin{equation}
z_t \sim p(\cdot | s_t, a_t, w)
\end{equation}

where $p(\cdot | s_t, a_t, w)$ is the c.d.f. of a continuous probability distribution parameterized by $w$. Clearly, $p(z | s_t, a_t, w)$ is the likelihood of $z$ when the parameter of the model is $w$ and the current state and action are $s_t$ and $a_t$, respectively. Suppose we have got a number of state-action pairs $\{(s_t, a_t)\}_{t=1}^T$ and their corresponding return $\{z_t\}_{t=1}^T$, and we denote the data we have got so far as $\mathcal{D} = {(s_t, a_t, z_t)}_{t=1}^T$. We further assume that the returns are independent between different time steps, then the likelihood of the whole data can be computed as below,

\begin{equation}
p(\mathcal{D} | w) = \prod_{t=1}^T p(z_t | s_t, a_t, w)
\end{equation}

In the frequentists' perspective, as the problem is usually handled before, the true value of $w$ exists which is usually estimated by the maximum likelihood estimation(MLE) method,

\begin{equation}
\hat{w} = \textrm{argmax}_w p(\mathcal{D} | w) =\textrm{argmax}_w\prod_{t=1}^T p(z_t | s_t, a_t, w)
\end{equation}

When the likelihood is assumed to be in the form of a Gaussian distribution, the above problem is equivalent to finding the minimizer of the mean squared loss,

\begin{equation}
\hat{w} = \textrm{argmin}_w \frac{1}{T}\sum_{t=1}^T (z_t - \mu_t(s_t, a_t, w))^2
\end{equation}

However, when the problem is looked into with a Bayesian perspective, where a prior $p(w)$ is usually placed over the parameter $w$, the parameter is assumed to be a random variable which can only be estimated via the posterior distribution,

\begin{equation}
\hat{w} \sim p(w | \mathcal{D}) \sim p(\mathcal{D} | w)p(w)
\end{equation}

when the posterior of $w$ is given, the posterior estimation of the return of some new state-action pair $(s, a)$ can be got by the following equation,

\begin{displaymath}
z \sim \int p(z | s, a, w)p(w | \mathcal{D}) dw
\end{displaymath}

To estimate the Q-value $\hat{q}$, the posterior expectation of $z$ is calculated via integration,

\begin{equation}
\hat{q} = \iint zp(z | s, a, w)p(w | \mathcal{D}) dwdz
\end{equation}

 In practice, however, the integral is usually hard to calculate. One solution to the problem is to first sample several $z$ from the posterior distribution, and replace the estimation by an empirical mean. But due to the randomness in $w$ and the complicated structure in neural network, it is still intractable to sample $z$ directly from its distribution. Therefore, we must sample $w$ as well, i.e., we first sample several $\{w_i\}_{i=1}^k \sim f(w | \mathcal{D})$ from the posterior distribution, then for each $w_i$, we get the mean value of the discounted cumulative rewards $\{\bar{z}_i\}_{j=1}^l$ from the c.d.f. $p(z | s, a, w_i)$, then the mean value can be estimated as

\begin{equation}
\hat{q}(s, a) = \frac{1}{k}\sum_{i=1}^k \bar{z}_i
\end{equation}

\subsection{Action Selection with Thompson Sampling Method}
In the previous Q-learning process, the action is selected $\epsilon$-greedily to be the one which maximizes the Q-value under the current state.

\begin{equation}
a_t = \textrm{argmax}_a \hat{q}(s_t, a)
\end{equation}

However, the above strategy has several weaknesses. The $\epsilon$-greedy strategy is shown to be insufficient for exploration in practice, and it is costly to calculate the posterior mean under the Bayesian setting. To achieve better exploration and reduce the cost, we instead use the Thompson sampling method. At each step, we first sample the weight $w_t$ from the current posterior $w_t \sim f(w | \mathcal{D}_t)$, then we choose the action $a_t$ to be the maximizer for the Q-value estimation under $w_t$,

\begin{equation}
a_t = \textrm{argmax}_a \bar{z}(s_t, a | w_t)
\end{equation}

The basic framework of the Thompson sampling method is written in the pseudocode below,

\begin{algorithm}[htb]
\begin{algorithmic}[1]
\caption{The Thompson sampling method}
\Require the posterior distribution $p(w | \mathcal{D})$, the likelihood function $p(\cdot | w)$, the current state $s$
\Ensure the selected action $a^*$
\State Sample $w$ from $p(w | \mathcal{D})$
\State Calculate $\bar{z}(s, a) = \mathbb{E}[z | s, a, w]$ for each $a\in\mathcal{A}$
\State Return $a^* = \textrm{argmax}_a\bar{z}(s, a)$
\end{algorithmic}
\end{algorithm}

\section{Implement the Bayesian Model}
To implement a specific update rule for the Bayesian network, we propose two methods, the first one based on the variational inference method while the second one applying a simpler procedure with a perspective related to the dropout method.

\subsection{Variational Inference Under Gaussian approximation}
\subsubsection{Variational Inference}
In practice, it is intractable to get an explicit expression for the posterior distribution of the network weights. One alternative way is to use the variational inference method to simplify calculation. Variational inference finds the parameters $\theta$ of a distribution on the weights $q(w|\theta)$ that minimises the Kullback-Leibler (KL) divergence with the true Bayesian posterior on the weights,

\begin{equation}
\begin{aligned}
\theta^* &= \textrm{argmin}_\theta \textrm{KL}[q(w|\theta)||p(w|\mathcal{D})]\\
& = \textrm{argmin}_\theta \int q(w|\theta)\ln \frac{q(w|\theta)}{p(w | \mathcal{D})p(w)}\\
& =  \textrm{argmin}_\theta \textrm{KL} [q(w|\theta) ||  p(w)] - \mathbb{E}_{q(w|\theta)} [\ln p(\mathcal{D}|w)]
\end{aligned}
\end{equation}

The resulting cost function is variously known as the variational free energy or the expected lower bound. We suppose $q(w | \theta)$ takes the simple form of independent Gaussian distribution where $\theta = (\mu, \rho)$ and the Gaussian distribution takes the form $w\sim N(\mu, \ln^2(1 + \exp(\rho)))$. The main procedure to update procedure is based on one conclusion proposed in \cite{BNN}.

\paragraph{Proposition 1.}
Let $\varepsilon$ be a random variable having a probability density given by $q(\varepsilon)$ and let $w = t(\theta,\varepsilon)$ where $t(\theta,\varepsilon)$ is a deterministic function. Suppose further that the marginal probability density of $w, q(w|\theta)$, is such that $q(\varepsilon)d\varepsilon = q(w|\theta)dw$. Then for a function f with derivatives in $w$:

\begin{displaymath}
\frac{\partial}{\partial\theta}\mathbb{E}_{q(w|\theta)}[f(w, \theta)] =  \mathbb{E}_{q(\varepsilon)}\left[\frac{\partial f(w, \theta)}{\partial w}\frac{\partial w}{\partial\theta} + \frac{\partial f(w, \theta)}{\partial\theta}\right]
\end{displaymath}

According to this proposition, $\theta$ can be updated by first sampling $w$ from its corresponding distribution and then use the ordinary gradient descent method to update its parameters, which leads to the following procedure,

\begin{algorithm}[htb]
\begin{algorithmic}[1]
\caption{Updating procedure for $\theta=(\mu,\rho)$}
\Require the data $\mathcal{D}$,  the initial point $(\mu,\rho)$, the learning rate $r$
\State Sample $\varepsilon\sim N(0,I)$
\State Let $w=\mu + \ln(1+\exp(\rho))\odot \varepsilon$
\State Let $f(w,\mu,\rho)=\ln q(w|\mu,\rho)-\ln p(w)p(\mathcal{D}|w)$
\State Calculate the gradient with respect to the mean
\begin{displaymath}
\Delta_\mu = \frac{\partial f(w,\mu,\rho)}{\partial w} + \frac{\partial f(w,\mu,\rho)}{\partial \mu}
\end{displaymath}
\State Calculate the gradient with respect to the standard deviation parameter $\rho$
\begin{displaymath}
\Delta_\rho = \frac{\partial f(w,\mu,\rho)}{\partial w}\frac{\varepsilon}{1 + \exp(-\rho)} + \frac{\partial f(w,\mu,\rho)}{\partial \rho}
\end{displaymath}
\State Update the variational parameters
\begin{displaymath}
\begin{aligned}
\mu &= \mu - r\Delta\mu\\
\rho &= \rho - r\Delta\rho
\end{aligned}
\end{displaymath}
\end{algorithmic}
\end{algorithm}

\subsubsection{Compute the target value}
An widely applied way for target estimation in Q-learning is the bootstrap method, which estimates the current Q-value via the current reward and an estimation of the return of next state,

\begin{equation}
z^{\textrm{target}}_t = r_t + \gamma \max_a\hat{z}(s_{t+1}, a)
\end{equation}

Here the equation means that the LHS and RHS are identically distributed. To sample from the right hand expression, we first sample the weight $w$ from the posterior distribution and calculate $\max_a\hat{z}(s_{t+1}, a, w)$. The target estimation is then written as below,

\begin{equation}
z^{\textrm{target}}_t = r_t + \max_a\hat{z}(s_{t+1}, a, w)
\end{equation}

After we have got the target value, the pair $(s_t, a_t, y_{\textrm{targ}})$ is added into the historical dataset to update the posterior distribution.

\subsubsection{Assumption on the likelihood and prior}
Like many previous methods, we assume that the likelihood for $z_t$ given $s_t, a_t$ and $w$ takes the form of Gaussian distribution. Meanwhile, the prior on the weights is also set to be an independent Gaussian distribution with equal variance for each component. The above assumptions lead to an explicit expression on the target loss of the variational method,

\begin{equation}
\begin{aligned}
f(w, \mu, \rho) &= \ln q(w | \mu, \rho) - \ln p(\mathcal{D}|w)\ln p(w)\\
&\sim -\frac{1}{2}1^T\ln\sigma - \frac{(w - \mu)^T(w - \mu)}{2\sigma^2}+\frac{\alpha}{2T} \sum_{t=1}^T(\hat{z}(s_t, a_t, w)-z^{\textrm{target}}_t)^2 + \frac{\beta}{2} w^Tw
\end{aligned}
\end{equation}

where $\alpha > 0, \beta > 0$ are different hyperparameters associated to the variance of these distributions and $\sigma= \ln(1 + \exp(\rho))$ is the standard error of $w$. Under this setting, we can write $\Delta_\mu, \Delta_\rho$ explicitly,

\begin{equation}
\Delta_\mu = \alpha\frac{\sum_{t=1}^T(\hat{z}(s_t, a_t, w)-z^{\textrm{target}}_t)}{T}\frac{\partial \hat{z}(s_t, a_t, w)}{\partial w} + \beta w - \frac{\mu - w}{\ln^2(1 + \exp(\rho))}\\
\end{equation}

\begin{equation}
\begin{split}
\Delta_\rho = \left(\alpha\varepsilon\odot\frac{\sum_{t=1}^T(\hat{z}(s_t, a_t, w)-z^{\textrm{target}}_t)}{T}\frac{\partial \hat{z}(s_t, a_t, w)}{\partial w} + \beta\varepsilon\odot w - \frac{1}{\ln(1 + \exp(\rho))} +  \frac{(w - \mu)^T(w - \mu)}{\ln^3(1 + \exp(\rho))} \right) \\
\odot\frac{1}{1 + \exp(-\rho)}
\end{split}
\end{equation}

\subsection{Dropout as a Bayesian Approximation}
In \cite{Dropout}, the authors propose a different viewpoint on posterior calculation, which results in a simpler procedure than the previous work. In this work, the authors assume that the approximator of the intractable posterior $q(w)$ is a distribution over matrices whose columns are randomly set to zero,

\begin{equation}
\begin{aligned}
W_i&=M_i\textrm{diag}([z_{i,j}]_{j=1}^{K_i})\\
z_{i,j} &\sim \textrm{Bernoulli}(p_i) \textrm{ for } i=1, \cdots, L, j = 1,\cdots, K_{i-1}
\end{aligned}
\end{equation}

given some probabilities $p_i$ and matrices $M_i$ as variational parameters. The objective function is then estimated by Monte Carlo integration with a single sample $\hat{w}_n \sim q(w)$ to get an unbiased estimate $-\ln p(y_n |x_n , \hat{w}_n)$. After some scaling, the estimated objective function can be written as,

\begin{equation}
\begin{aligned}
L \propto -\frac{1}{N}\sum_{n=1}^N\ln p(y_n|x_n,\hat{w}_n) + \frac{1}{2N}\sum_{i=1}^L(p_il^2\Vert M_i\Vert_2^2 + l^2\Vert m_i\Vert_2^2)
\end{aligned}
\end{equation}

which is exactly the same with minimization objective in dropout,

\begin{equation}
\begin{aligned}
L=\frac{1}{N}\sum_{i=1}^N E(y_i,\hat{y}_i)+ \lambda\sum_{i=1}^L(\Vert W_i \Vert_2^2 + \Vert b_i\Vert_2^2)
\end{aligned}
\end{equation}

Under this setting, getting the sampled Q-value with a sampled $\hat{w}$ from $q(w)$ is equivalent to applying the dropout method directly during the forward propagation, and the weights of the network can be updated in the same way as in the dropout method.

\section{Experiments}
The algorithm is applied on several toy environments. We compare the outcome of our algorithm with some previous proposed Q-learning algorithms\footnote{Codes available at https://github.com/hzxsnczpku/banrinochoujou}, our experiments are aimed to answer the following two questions,

\begin{itemize}
\item[-] Is the Thompson Sampling method helpful for efficient exploration?
\item[-] Can the Bayesian neural network outperform the previous algorithms under our environments?
\end{itemize}

\subsection{N-Chains}
We first test the methods in the environment called nChain. Under this environment, the agent starts at position 1, and at each step, the agent can choose to move one step forward or one step backward. When the agent reaches position 0, the game is ended and the agent receives a reward of 1. When the agent reaches position n, the game is also ended and the agent receives a reward of n. The aim of the agent is to receive a reward as high as possible throughout one game, which can be achieved if the agent always chooses to move forward. However, to achieve this, the agent needs to explore for a long time since there is no reward until the final state is reached. We calculate the mean episode reward between episode 90-110 under different algorithms and different chain lengths. The result is recorded in table 1. From the result we can see, the Bayesian method improves the performance of the agent, which shows that the Thompson sampling method can encourage efficient exploration under certain environments.

\begin{figure}[htb!]
\center
\includegraphics[scale=0.85]{nchain_demo}
\caption{A demo of the nChain environment. }
\end{figure}

\begin{table}[htb!]
\center
\caption{Mean episode reward between episode 90-110 under different algorithms and different chain lengths}
\begin{tabular}{c | c c c c c}
\hline
N & 20 & 30 & 50 & 80 & 100\\
\hline
Bayesian TS & \textbf{20.0} & 14.05 & \textbf{50.0} & \textbf{80.0} & 80.15\\
Bayesian Dropout & 19.05 & \textbf{30.00} & 45.10 & 76.05 & \textbf{80.20}\\
DQN no noise  & 14.30 & 6.00 & 15.50 & 4.00 & 60.00\\
DQN $\varepsilon$-greedy & 9.00 & 9.05 & 10.35 & 24.25 & 40.25\\
\hline
\end{tabular}
\end{table}

\subsection{Classical Control}
We further test our method on two classical control problems. The environments are the CartPole-v0 and the Acrobot-v1, respectively. In the first environment, the goal is to keep  the pole on a cart standing as long as possible. In the second environment, the goal is to make the end of the pole touch a horizontal line as quickly as possible. In both two environments, the state space is a two dimensional continuous vector and the action is an integer taking value in $\{0, 1\}$. We test two versions of posterior updater of the Bayesian method, the first one of which is utilizing the variational inference with Gaussian approximator to the posterior while the second one is applying directly the dropout method.  The original DQN algorithm with and without $\epsilon$-greedy exploration strategy are tested as baselines. The double DQN algorithm and DQN with a prioritized replay memory are also tested.

\begin{figure}[htb!]
\center
\includegraphics[height=4cm]{Cartpole_sample}
\includegraphics[height=4cm]{Acrobot_sample}
\caption{Left: A demo of the CartPole environment. Right: A demo of the Acrobot environment. }
\end{figure}

We plot the episode rewards of different algorithms performed in the two platforms at the following two figures, where each reward is smoothed as the mean reward of the nearby 10 episodes and the shadow area covers two times of standard error.

\begin{figure}[htb!]
\center
\includegraphics[height=5cm]{CartPole}
\includegraphics[height=5cm]{Acrobot}
\caption{Left: Experimental results on the CartPole environment, where the score is smoothed over a number of episodes.  Right: Experimental results on the Acrobot environment, where the score is smoothed over a number of episodes. }
\end{figure}

In both two environments, the agent behaves poorly when no exploration strategy is applied, staying at a low score for a very long time. We can see from our results that the Bayesian estimation with variational inference method achieves the best performance, which outperforms all of the other algorithms at the very beginning of the CartPole platform and is slightly better than the other algorithms at the Acrobot environment. One of our findings is that this method gets a higher score even at the beginning of the game, where the strategy is nearly random. This fact shows that the variational inference method combined with Thompson Sampling method achieves very high exploring efficiency. The dropout method behaves very similar to the previous Q-learning methods, but the result is still inspiring since it means that we have found a new method which is at least not worse than the existing methods.

\section{Conclusion and further work}
We use the Bayesian method to model the distribution of the return in Q-learning using the variational inference method and as well as the dropout-as-Bayesian-approximation method. We apply the Thompson sampling method given the posterior distribution of the parameters and show that the method achieves a better performance than the previous algorithms in several platforms. However, we find that our methods can sometimes be unstable and have a higher computation cost than the previous algorithms, which leads to the poor performance of our algorithms under some more complicated environments. Further works include but are not limited in improving the stability of the algorithms, cutting down the computation cost, doing experiments on some more complicated environments, etc.

\begin{thebibliography}{99}
	\bibitem{DQN}
	Mnih V, Kavukcuoglu K, Silver D, et al. Playing atari with deep reinforcement learning[J]. arXiv preprint arXiv:1312.5602, 2013.
	\bibitem{DDQN}
	Van Hasselt H, Guez A, Silver D. Deep Reinforcement Learning with Double Q-Learning[C]//AAAI. 2016: 2094-2100.
	\bibitem{PDQN}
	Schaul T, Quan J, Antonoglou I, et al. Prioritized experience replay[J]. arXiv preprint arXiv:1511.05952, 2015.
	\bibitem{DuelDQN}
	Wang Z, Schaul T, Hessel M, et al. Dueling network architectures for deep reinforcement learning[J]. arXiv preprint arXiv:1511.06581, 2015.
	\bibitem{DisDQN}
	Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. arXiv preprint arXiv:1707.06887, 2017.
	\bibitem{BDQN}
	Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Advances in Neural Information Processing Systems, pages 4026–4034, 2016.
	\bibitem{Dropout}
	Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050–1059, 2016.
	\bibitem{BNN}
	Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint arXiv:1505.05424, 2015.
	\bibitem{BNN2}
	José Miguel Hernández-Lobato and Ryan P. Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In Proceedings of the 32nd International Conference on Machine Learning (ICML), pages 1861–1869, 2015.
	\bibitem{BNN3}
	David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint arXiv:1710.04759, 2017.
	\bibitem{TS}
	Agrawal S, Goyal N. Analysis of Thompson sampling for the multi-armed bandit problem[C]//Conference on Learning Theory. 2012: 39.1-39.26.
	\bibitem{TS2}
	Chapelle O, Li L. An empirical evaluation of thompson sampling[C]//Advances in neural information processing systems. 2011: 2249-2257.
	\bibitem{TS3}
	Kaufmann E, Korda N, Munos R. Thompson Sampling: An Asymptotically Optimal Finite-Time Analysis[C]//ALT. 2012, 12: 199-213.
	\bibitem{BAC}
	Mohammad Ghavamzadeh, Yaakov Engel, and Michal Valko. Bayesian policy gradient and actor-critic algorithms. Journal of Machine Learning Research, 17(66):1–53, 2016.
	\bibitem{RL}
	Sutton R S, Barto A G. Reinforcement learning: An introduction[M]. Cambridge: MIT press, 1998.
	\bibitem{DL}
	LeCun Y, Bengio Y, Hinton G. Deep learning[J]. Nature, 2015, 521(7553): 436-444.
\end{thebibliography}

\end{document}


\chapter{Jet smearing method}
    \label{app:JetSmearingMethod}

The large $\met$ in the multijet background originates mainly from the misreconstruction of the energy of the jets in the calorimeter, and to a lesser extent, due to the presence of neutrinos in the decays of heavy flavor hadrons.
The multijet processes represent a small contribution to the total background in the selection M1, and are almost negligible in the other selections.

The method to estimate the multijet processes relies on the assumption that the $\met$ of these events has its origin in the fluctuations in the response of the calorimeter.
This method, also known as the \emph{jet smearing method}, proceeds in four steps, detailed in the following:

\begin{enumerate}
\item{\textbf{Seed sample}} A sample of multijet events is selected with a lower cut on the leading jet $\pt$ of 100~GeV.
In order to select well-balanced events, a cut on the significance of the missing transverse energy,
\begin{equation}
\met / \sqrt{\sum{\et}} < 1.0,
\label{eq:JetSmearingMetSignificance}
\end{equation}

\noindent is required.
This seed sample is then used in steps 3 and 4.

\item{\textbf{Response function}}  Two jet \emph{response functions}, one for jets with b-veto and another for $b$-tagged jets\footnote{The MV1 $b$-tagger is used.}, are extracted from the simulation to estimate the fluctuations in the measured jet transverse momenta and the contributions from neutrinos from heavy flavor decays.
They are estimated by comparing the generator level to reconstructed jet transverse momenta distribution.
This is the only part where the MC simulation is used.
The two response functions are shown in Figure~\ref{fig:JetSmearingResponseFunctions}.

\begin{figure}[!ht]
  \begin{center}
    \mbox{
      \includegraphics[width=0.495\textwidth]{Appendix_JetSmearingMethod/Figures/jetsmearing25_R_btag.eps}
      \includegraphics[width=0.495\textwidth]{Appendix_JetSmearingMethod/Figures/jetsmearing25_R_bveto.eps}
    }
  \end{center}
  \caption{Response functions for $b$-tag jets and $b$-veto jets}
  \label{fig:JetSmearingResponseFunctions}
\end{figure}

\item{\textbf{Adjusted response function}} The response function from step 2 is used to smear the seed events from step 1.
In the analysis, a total of 500 smeared events are produced from each seed event.
Further corrections as a function of the $\phi$-direction of the jet are also applied to the smeared jets.

\item{\textbf{Extrapolation to SR}} The seed events from step 1 are now smeared with the adjusted response function from step 3 to estimate the distributions of relevant variables in the control and signal regions.
After the seed events are smeared, the multijet contribution needs to be normalized in a dedicated control region.
The multijet control regions are defined with exactly the same cuts as the different signal regions M1-M6 (see Table \ref{tab:SignalRegionCuts}), but reverting the $\Delta\phi(\text{jet}, \mathbf{p_{T}}^\text{miss})$ cut to be below 0.4.
The contribution from the other backgrounds ($W/Z$+jets, single top and $\ttbar$, dibosons) are subtracted using MC simulations, normalized with the normalization factors from the global fit (see Section~\ref{sec:Fit}), when appropriate.
As an example, the relevant distributions of the smeared events for the region M1 are shown in Figure \ref{fig:JetSmearingCheckPlots}.

\end{enumerate}

The normalization factors and the estimation of the multijet background for the signal regions M1 to M6 are shown in Table~\ref{tab:qcd_est}.
This table also shows the fraction of multijet events in the total background prediction.

Finally, in order to determine the systematic uncertainty on the multijet prediction, the multijet yields calculated with different response functions, are compared.
Variations in the multijet yield of the order of 100\% have been observed.
For this reason, a 100\% systematic uncertainty is assigned to the estimated number of events.

%--- \ref{tab:qcd_est}
\input{Appendix_JetSmearingMethod/Tables/qcdEstimation.tex}

\begin{figure}[!ht]
  \begin{center}
    \mbox{
      \includegraphics[width=0.495\textwidth]{Appendix_JetSmearingMethod/Figures/plot_Stop_A6_CRqcd_dPhi_min_fitted.eps}
      \includegraphics[width=0.495\textwidth]{Appendix_JetSmearingMethod/Figures/plot_Stop_A6_CRqcd_met_fitted.eps}
    }
    \mbox{
      \includegraphics[width=0.495\textwidth]{Appendix_JetSmearingMethod/Figures/plot_Stop_A6_CRqcd_pt1_fitted.eps}
      \includegraphics[width=0.495\textwidth]{Appendix_JetSmearingMethod/Figures/plot_Stop_A6_CRqcd_eta1_fitted.eps}
    }
    \mbox{
      \includegraphics[width=0.495\textwidth]{Appendix_JetSmearingMethod/Figures/plot_Stop_A6_CRqcd_pt2_fitted.eps}
      \includegraphics[width=0.495\textwidth]{Appendix_JetSmearingMethod/Figures/plot_Stop_A6_CRqcd_eta2_fitted.eps}
    }
  \end{center}
  \caption[Distributions of the minimum $\Delta\phi$ between the $\met$ and the leading jet $\pt$, the $\met$, the leading jet $\pt$ and $\eta$, and the second leading jet $\pt$ and $\eta$ for the multijet normalization region corresponding to the M1 selection cuts.]{Distributions of the minimum $\Delta\phi$ between the $\met$ and the leading jet $\pt$ (top left), the $\met$ (top right), the leading jet $\pt$ (middle left) and $\eta$ (middle right), and the second leading jet $\pt$ (bottom left) and $\eta$ (bottom right) for the multijet normalization region for the M1 selection cuts.}
  \label{fig:JetSmearingCheckPlots}
\end{figure}


\documentclass{tufte-handout}
\usepackage{./fall2018_preamble_v1.0}

\IfSubStr{\jobname}{\detokenize{Solutions}}{\toggletrue{solutions}}{\togglefalse{solutions}}

\fancypagestyle{firstpage}

{\rhead{How Can Images Be Real If Our Cameras Aren't Real? \linebreak \textit{Version: \today} \linebreak \href{https://github.com/newsch/wonky-bobbin/blob/master/overnight/main.tex}{source}}}

\title{Overnight Overnight: How Can Images Be Real If Our Cameras Aren't Real?}
\author{QEA}
\date{Fall 2018}

\toggletrue{solutions}
% \togglefalse{solutions}

\begin{document}

\maketitle
\thispagestyle{firstpage}

% % in the first problem block, make sure to use the series option for enumerate
% \begin{enumerate}[series=exercises, label=\textbf{Exercise} (\arabic*)]
% \item The text of the first exercise
% \solution{Here is the solution.}
% \end{enumerate}
% % when adding a second block of problems make sure to use "resume" instead of "series"
% \begin{enumerate}[resume=exercises, label=\textbf{Exercise} (\arabic*)]
% \item The text of the second exercise.
% \iftoggleverb{solutions}
% \textbf{Solution:}
% \begin{lstlisting}
% x = linspace(0, 2*pi, 100);
% y = 5;
% \end{lstlisting}
% \fi
% \end{enumerate}

\section{Overview}

The goal of this assignment is to get familiar with the concepts of digital image processing and the mathematical concepts involved with them. In this assignment you'll implement some common photo manipulation methods and learn some concepts that will prepare you for further work in this area.

\section{Learning Goals}

\begin{itemize}
\item Understand the relationship between the physical equipment used to capture
images and the digital representation of them, including:
    \begin{itemize}
        \item CCDs and Bayer grids
        \item Linear vs Nonlinear space and Gamma correction
        \item White Balance adjustment
    \end{itemize}
\item Understand how data is stored in bits/bytes and how that impacts various
operations and manipulations
% \item Feel comfortable working with matrix operations in Python and Jupyter notebooks
\item Be able to implement basic linear interpolation debayering in Python
\item Become comfortable with concepts needed for more advanced algorithms.
\item Re-familiarize with PCA and Eigenspaces.
\end{itemize}

\section{Image manipulation}

\subsection{Review of Image Manipulation (now in Python) [1 hr]}

Re-familiarize yourself with some of the image manipulation we did in the faces module, specifically the brightness and contrast section (problems 9-14) of \href{https://drive.google.com/file/d/0B7LNBbaxYFujOHpEU2FDdkF6VEE/view
}{Module 2, Night 3}.

\begin{enumerate}[series=exercises, label=\textbf{Exercise} (\arabic*)]
    \item Implement problems 9-14 in Python, using numpy operations.
\end{enumerate}

\subsection{2D Convolution w/ Kernels [2 hrs]}

You already were introduced to the concept of convolution in \href{https://drive.google.com/file/d/1_eEtwNDHhF-Izfcf6l00PvG-b6whwety/view}{FitBit Night 2}. In this overnight assignment we're going to use it with discrete 2-dimensional signals: images!

We'll do this by convolving the fairly complex and discrete matrix of an image with a much smaller and simpler matrix commonly called a kernel. This kernel can be used to quickly perform algebraic operations on the pixels of an image relative to their neighbors.

Take the discrete convolution of these two 3x3 matrices for example:
\begin{align}
    \threebythree{0}{0}{0}{0}{2}{0}{0}{0}{0} * \threebythree{1}{1}{1}{1}{1}{1}{1}{1}{1} =
    \threebythree{2}{2}{2}{2}{2}{2}{2}{2}{2}
\end{align}

One way to read think of this (with the left matrix as our "kernel" and the right matrix as our "image") is \emph{"For every element in the image, sum the element multiplied by 2 and each of its neighbors multiplied by 0"}.

In practice, this is equivalent to both of the following operations:
\begin{align}\label{eqn:convolution1}
    \begin{bmatrix}2\end{bmatrix} * \threebythree{1}{1}{1}{1}{1}{1}{1}{1}{1} =
    \threebythree{2}{2}{2}{2}{2}{2}{2}{2}{2}
\end{align}

\begin{align}
    2 \threebythree{1}{1}{1}{1}{1}{1}{1}{1}{1} = \threebythree{2}{2}{2}{2}{2}{2}{2}{2}{2}
\end{align}

To write this generally, the resulting matrix $G$ of the correlation of matrices $H$ (our kernel) and $F$ (our image),
\begin{align}
    H * F = G
\end{align}
is calculated as:
\begin{align}
    G[i, j] = \sum_{u=-k}^{k} \sum_{v=-k}^{k} H[u, v] F[i - u, j - v]
\end{align}
\emph{Note: the actual operation performed here involves flipping the rows and columns of the kernel. All of the examples that we're looking at use symmetric kernels, so you won't need to take that into consideration.}

However, the kernel we used in the initial example is not a particularly exciting one. As you saw, it's actually equivalent to an element-wise multiplication. The really exciting stuff starts to happen when you use kernels with non-zero surrounding elements.

Let's look at the following example:
\begin{align}\label{eqn:convolution2}
    \threebythree{0}{1}{0}{1}{2}{1}{0}{1}{0} * \threebythree{1}{1}{1}{1}{1}{1}{1}{1}{1} =
    \threebythree{4}{5}{4}{5}{6}{5}{4}{5}{4}
\end{align}
Again, you can read this as \emph{"For every element in the image, sum the element multiplied by 2 with the values of its cardinal neighbors"}.

One thing we haven't talked about yet is how to treat the elements on the edges of the matrix that aren't bordered by 8 other elements. There are several methods that can be used when calculating the convolution, including:
\begin{itemize}
    \item ignoring values in the matrix that aren't completely bordered (this results in a matrix with $n-2$ fewer rows and columns)
    \item "wrapping" the matrix edges around PAC-MAN style
    \item considering all none-present elements equal to 0 (or alternatively, "padding" the matrix borders with zeros)
\end{itemize}
As you may have discerned from the above example \eqref{eqn:convolution2}, we're going to be the using the third method of treating these elements as if they are 0. In the context of image manipulation, we don't want to reduce the size of our images with every operation, and wrapping the image around doesn't really make sense (unless you're working with some sort of 360-degree panorama, where the left/right and top/bottom of the image actually do touch).

\bex
\item Calculate the convolutions of these three kernels:
    \begin{enumerate}
        \item \threebythree{0}{1}{0}{1}{1}{1}{0}{1}{0}
        \item \threebythree{0}{1}{0}{1}{-4}{1}{0}{1}{0}
        \item \threebythree{0}{\frac{1}{5}}{0}{\frac{1}{5}}{\frac{1}{5}}{\frac{1}{5}}{0}{\frac{1}{5}}{0}
    \end{enumerate}
    with these three matrices:
    \begin{enumerate}
        \item \threebythree{0}{1}{0}{0}{1}{0}{0}{1}{0}
        \item \threebythree{1}{2}{1}{2}{3}{2}{1}{2}{1}
        \item \threebythree{5}{4}{3}{4}{3}{2}{3}{2}{1}
    \end{enumerate}
    What do you notice about their effects? How do the results of kernel (a)
    differ from kernels (b) and (c)?
    \solution{Kernels (b) and (c) preserve the amount of "energy"/brightness in the image, while (a) increases it.}  % TODO: math out the convolutions
\item Can you think of an "identity" kernel, that returns the same image?
\solution{\threebythree{0}{0}{0}{0}{1}{0}{0}{0}{0} or similar (could be
different dimensions)}
\eex

\subsection{Data Types, Numpy, and Matplotlib [30 min]}

Before we turn you loose to start working with convolution and images in Python, we're going to give you some tips for working with images as matrices. These mainly revolve around how Python/Numpy store data and how Matplotlib displays matrix data.

First, lets talk about data types. You've probably heard about integers and floats in ModSim. The main difference between them is that integers store, well, integer values (numbers without a fractional component), while floats can represent fractions (or at least estimates of them). Generally Python is pretty good about knowing when to work with integers vs floats and converting between them, but Numpy can be a little more tricky (it introduces its own types, including (but not limited to) \lstinline{np.float32}, \lstinline{np.int16}, and \lstinline{np.uint8}).

The numbers at the ends of those types are the number of bits that they use to store their values. A \emph{bit} (a portmanteau of \emph{binary digit}) can store either 0 or 1 for a value. Together, multiple bits can encode larger values. 8 bits are commonly called a \emph{byte}\footnote{But not always! In the past, \emph{byte} could refer to smaller or larger numbers of bits, depending on the computer. Technically, \emph{octet} the more specific and correct term.} and can represent 256, or $2^8$ different values.

You might have noticed that we listed the \lstinline{np.uint8} type earlier and are asking yourself, \emph{"What does the 'u' mean?"}. In this case, \emph{u} means \emph{unsigned}, in the mathematical sense of +/-. If we want to store a number's sign, we need to use another bit. In most modern computers, it's preferable to use multiples of 8 for data types, so generally the signed variant of an integer loses a bit\footnote{Get it?} of value in order to store the sign. For example, the maximum value of the \lstinline{np.int8} data type is 127, and the minimum value is -128, while the maximum and minimum values of \lstinline{np.uint8} are 255 and 0, respectively. Note that both types can represent 256 different values. The best one to use varies from situation to situation.

Why do we bring this up? Without understanding how the data types you use behave in certain situations, your code may exhibit some very peculiar behavior. One such case to consider about is \emph{integer overflow}. When a numerical data type exceeds its representable range it is known as \emph{overflow}\footnote{You might also come across the term \emph{underflow}, which can (confusingly) refer to either a data type exceeding the lower bound of its representable range, or a case in floating-point arithmetic where the value's precision cannot be properly represented.}. Depending on the language and system, this can cause the value to \emph{saturate} at the limit (\emph{e.g.} \lstinline{255 + 1 = 255} for an unsigned 8-bit integer) or to \emph{wrap} to the other limit (\emph{e.g.} \lstinline{-128 - 1 = 127} for a signed 8-bit integer).

% TODO: fig: wikipedia's odometer physical overflow example

You may be fortunate enough to encounter a warning or error when this case occurs. In other cases you won't have any indication except for funky behavior, and you'll be in a much better place for debugging if you know to check for overflows. At the time of this writing, numpy/scipy do not provide warnings when overflow occurs in convolution operations.

\bex
\item Create a number of type \lstinline{np.int8}. Cause it to overflow. Is this \emph{wrapping} or \emph{saturating} behavior?
\solution{Numpy overflows wrap (\lstinline{np.int8(127) + np.int8(1)} yields \lstinline{-128}).}
\eex

In digital images, pixel values are commonly represented with 8 bits of precision, ranging in value from 0 to 255. Unsigned 8-bit integers are the perfect data type for storing images of this kind, but when working with them (adding, multiplying, \emph{etc.}) you should use a data type that can handle any potential changes in value from the operations. One such type is a signed integer with a higher precision such as \lstinline{np.int32}, but you don't get the benefits of floating-point arithmetic. We'll discuss this more after the break.

\bex
\item What kind of behavior could integer overflow cause when working with images?
\solution{One example: sections of the image that should appear dark could appear light and \emph{vice versa}.}
\eex

% TODO: explain floats

For image manipulation, we recommend using the \lstinline{np.float32} type, which will allow you to use decimal precision and work with images to your heart's content.

An important thing to note is that the \lstinline{matplotlib.pyplot.imshow} function behaves differently for integers and floats. When displaying integer data, \lstinline{imshow} interprets values \emph{less than or equal to 0 as black}, and values \emph{greater than or equal to 255} as completely white/red/green/blue, depending on the context (2D arrays are interpreted as black/white and 3D arrays with a depth of size 3 are interpreted as RGB images). With float data on the other hand, \lstinline{imshow} interprets values \emph{less than or equal to 0.0 as black}, and values \emph{greater than or equal to 1.0} as completely white/red/green/blue.

% TODO: link to scipy documentation

This means that when converting between integer and float types, you'll want to scale between $[0,\ 255]$ and $[0.0,\ 1.0]$. Another function that may be useful is \lstinline{np.clip}, which you can you to lop off values outside of a given range (like $[0,\ 255]$).

% TODO: talk about considerations with astype

\subsection{Convolution for Image Manipulation [1 hr]}

Convolution can be implemented on two-dimensional arrays for a number of applications. Specific kernels can produce desired effects when convolved with digital images. Unlike the matrix-wide scaling we used for brightness and contrast adjustment, convolution allows us to perform operations relative to a pixel's neighbors, like increasing local contrast between pixels for example.

If you'd like more information on how kernels and convolution work with images, I would highly recommend \href{http://setosa.io/ev/image-kernels/}{Explained Visually's interactive page on image kernels}.

\bex
\item Read the \href{https://en.wikipedia.org/wiki/Kernel_(image_processing)}{Wikipedia page on image kernels}.

\item Implement the sharpening kernel in the Wikipedia article above, and try it out on a grayscale image (make sure the image isn't too large, or the results will be hard to see).
\item Implement the edge detection kernel. How does it compare to the kernels you worked with earlier?
\eex

\subsection{Gamma Correction [1 hr]}  % TODO: (Hadleigh)

\href{https://en.wikipedia.org/wiki/Gamma_correction}{link to Wikipedia page on Gamma Correction}
Gamma correction is a technique to remap the pixel values of a digital image according to a logarithmic scale to range more readable to human eyes.
\begin{enumerate}
    \item Use the numpy power function to raise each element in an array to a power between one and 0.
    \item Read the Wikipedia page and try using the value suggested there.
\end{enumerate}

\section{Demosaicing}

% \subsection{Background on Bayer filters}  % TODO: (Evan)

\subsection{Simple Demosaicing w/ Convolution [1 hr]}

% TODO: do grayscale first, then RGB

The most basic method for debayering a digital color image is to simply calculate the average of the neighboring values, a form of \emph{bilinear interpolation}.

\bex
\item Look back at problems 2 and 7 of \href{https://drive.google.com/file/d/1_eEtwNDHhF-Izfcf6l00PvG-b6whwety/view}{FitBit Night 2} to remind yourself of how moving average filters and convolution work.
\item Think through how you could use convolution to compute the components necessary to calculate the averages (for each pixel you'll need the sum of its neighbors of the same color, and the total number of neighbors).
% TODO: provide array of Bayer grid data
\item Implement this form of debayering in Python.
\eex

% note artifacts
% what else is needed?
% - vignetting
% - gamma correction
% - white balance

% \section{Advanced Demosaicing/White Balance foundations}

% \subsection{Prep for other algorithm}  % TODO: decide on another algorithm to discuss

\subsection{PCA denoising algorithm [1 hr]}  % TODO: (Hadleigh)

Local Pixel Grouping and Principal Component Analysis are commonly used in image denoising algorithms. Image noise is mostly spatially independent, meaning that by analyzing the principal components of small groups of pixels, the noise patterns can be determined. \href{https://www4.comp.polyu.edu.hk/~cslzhang/paper/PR_10_x_3.pdf}{Here is a paper detailing a LPG and PCA based denoising algorithm}.

\bex
\item Scan the paper, paying most attention to the procedure needed to implement this process. Write out pseudo-code for the procedure. %TODO: Write and scan pseudocode (Hadleigh)
\eex

\begin{align}
 \mu_{d} = \frac{1}{N} \sum_{i=1}^{N} d_{i} \label{eqn:Mean}
\end{align}

\begin{align}
\Phi = \mathbf{MM}^{T} \label{eqn:Covariance}
\end{align}

\eqref{eqn:Mean} is the equation for the mean value of an image. This value should be subtracted from the matrix of pixel values to mean center the image for co-variance \eqref{eqn:Covariance} and eigen decomposition.

%\begin{figure}
%\includegraphics[width=11cm]{figs/Saddle_point_contour.pdf}
%\caption{Contour plot of $f(\x)$. }
%\label{figSaddlePointContour}
%\end{figure}

%Figure \ref{figSaddlePointSurf}

\end{document}


    \filetitle{ww}{IRIS serial date number for weekly date}{dates/ww}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
Dat = ww(Year,Week)
Dat = ww(Year,Month,Day)
\end{verbatim}

\paragraph{Input arguments}\label{input-arguments}

\begin{itemize}
\item
  \texttt{Year} {[} numeric {]} - Years.
\item
  \texttt{Week} {[} numeric {]} - Week of the year.
\item
  \texttt{Month} {[} numeric {]} - Calendar month.
\item
  \texttt{Day} {[} numeric {]} - Calendar day of the month
  \texttt{Month}.
\end{itemize}

\paragraph{Output arguments}\label{output-arguments}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \texttt{Dat} {[} numeric {]} - IRIS serial date number representing
  the weekly date.
\end{itemize}

\paragraph{Description}\label{description}

The IRIS weekly dates comply with the ISO 8601 definition:

\begin{itemize}
\item
  every week starts on Monday and ends on Sunday;
\item
  the month or year to which the week belongs is determined by its
  Thurdsay.
\end{itemize}

\paragraph{Example}\label{example}


\documentclass[simplex.tex]{subfiles}
% NO NEED TO INPUT PREAMBLES HERE
% packages are inherited; you can compile this on its own
\begin{document}
\subsection{Reduced Dimension Clustering}
%
%We develop a new statistical method, named ``automatic repulsive
%clustering'' (ARC, to handle the clustering problem when the
%dimensionality outgrows the number of the data points. The standard
%dimension reduction tool such as principle component analysis (PCA) is
%not optimal for finding the subspace best suitable for clustering.
%Therefore, we seek the method to do dimension reduction and clustering
%at the same time: we develop a regularization to encourage good
%separation of the clusters, forcing the subspace to an angle so that
%each point can belong to a cluster completely. The plots below shows the
%different performances in clustering a equally proportioned 3
%components: the subspace found by PCA leads to incorrect clustering of
%the data, due to the orientation of the space; whereas ARC successfully
%identifies the ideal clustering direction in one dimensional space,
%hence generating satisfactory clustering result.
%
%
%\begin{figure}[h!]
%\begin{cframed}
%\centering
%\includegraphics[width=0.45\textwidth]{../../figs/pcaBased.png}
%\includegraphics[width=0.45\textwidth]{../../figs/ARC.png}
%\caption{PCA (left) versus Reduced Dimension Clustering (right).}
%\label{fig:arc}
%\end{cframed}
%\end{figure}
%
%
\end{document}


\subsubsection{\tt compiotaAKD}
\label{benchmarkcompiotaakd}

This AKD synthetic benchmark measures the performance of the
APL { \em compress-iota} idiom, {\tt booleanVector\qslash\qiota\0N}
against compiled versions of the same expression.
This idiom appears frequently in application code, as a
way to compute a subset of the indices of a vector.
For example, tab characters in a text vector might be
changed to blanks by a statement such as:

{\tt txt\qlbr (txt\qeps\0TAB)\qslash\qiota\qrho\0txt\qrbr\qlarrow'\qblank'}

Most APL interpreters special-case this idiom because of its
frequency of appearance in application code and the potential
for significant savings of CPU time and memory, by not having
to materialize the result of {\tt \qiota\qrho\0txt}. We expect
that compiled \sac code, should be able to match or outperform the APL code,
due to the use of \wlf and \awlf.

The benchmarks {\tt compiota}, {\tt compiotaAKD}, {\tt compiotad}, and
{\tt compiotadAKD} form a clique, as they
all perform similar tasks.


\documentclass[11pt,twocolumn]{article}
\usepackage[margin=1.5cm,columnsep=1cm]{geometry}
\usepackage{titling}
\usepackage[tiny]{titlesec}
\usepackage[utf8]{inputenc}
\usepackage{times}
\usepackage{listings, listings-rust}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{float}
\usepackage{tikz}

\newcommand{\codesize}{\fontsize{8pt}{9pt}\selectfont}
\newcommand{\textcode}[1]{{\codesize\ttfamily{#1}}}
\newcommand{\divider}[0]{\begin{center}\rule{0.48\linewidth}{0.48pt}\end{center}}
\titlespacing{\paragraph}{0pt}{0.5\baselineskip}{1em}
\renewcommand{\section}[1]{\paragraph{#1.}}

\lstset{
basicstyle=\codesize\ttfamily,
escapeinside={\%*}{*}
}

\newcommand*\circled[1]{\,\tikz[baseline=(char.base)]{
            \node[shape=circle,draw,inner sep=.7pt] (char) {\textcode{#1}};}\,}
\newcounter{clnumcounter}
% \counterwithin{clnumcounter}{figure}

\renewcommand{\theclnumcounter}{\Alph{clnumcounter}}
\newcommand{\clnum}{\refstepcounter{clnumcounter}\circled{\theclnumcounter}}
\newcommand{\clref}[1]{\circled{\ref{#1}}}

\predate{}
\postdate{}
\pretitle{\begin{flushleft}\Large\vspace{-3em}}
\posttitle{\end{flushleft}}
\preauthor{\begin{flushleft}}
\postauthor{\end{flushleft}\vspace{-1.2em}\rule{2cm}{0.2pt}\vspace{-.8em}}
\date{}

\title{Verus --- SMT-based verification of low-level systems code\\\small github.com/secure-foundations/verus}

\author{Andrea Lattuada -- ETH Zurich \\
Chris Hawblitzel -- Microsoft Research \\
Bryan Parno, Yi Zhou, Chanhee Cho, Travis Hance -- Carnegie Mellon University \\
Jon Howell -- VMware Research}

\begin{document}

\maketitle

This \textbf{30 minute} talk will introduce Verus, an SMT-based tool we are building to
verify full functional correctness of systems code written in Rust. I am
planning to demonstrate Verus via live demos interspersed in the talk:
the examples in this proposal are samples of the snippets that I will
verify with Verus in the demos.

\section{Introduction}\label{introduction}

The Rust programming language, with its combination of high-level
features, memory safety, and low-level ``unsafe'' primitives has proven
to be an excellent tool to write high-performance systems software, with
direct interactions with the hardware.

When looking at Rust in the context of software verification
we observed that functions using the safe subset of Rust without
interior mutability are pure, deterministic transformations, even when
they manipulate mutable references (as they can be logically rewritten
as consuming the initial value of the reference, and returning the new
value). By statically preventing aliased mutable references Rust
relieves the burden on the developer of reasoning about data races; when
verifying imperative code in languages that do not restric aliasing, the
developer similarly has to explicitly manage the heap and potential
aliasing. In our experience, a Rust-like linear type system can be an
excellent aid to Floyd-Hoare SMT-based verification of executable code.

We leverage these intuitions in the design of our new tool for SMT-based
verification of Rust code, Verus.
We are building Verus to efficiently verify full functional correctness
of low-level systems code written in a safe Rust dialect that supports
expressing specifications and proofs.

Figure~\ref{fig:fibo} shows a snippet of a Rust
function~\clref{clnum:fn-fibo-impl} verified with Verus.

\section{Verus' design}\label{verus-design}

Verus leverages the fact that safe Rust has a natural, direct encoding
in SMT (for code without interior mutability, which we need to address
separately). Algebraic data types (ADTs) and sound generics complement
safe Rust to make it a solid basis for a mathematical language for
expressing specifications and proofs. In Figure~\ref{fig:fibo},
the \textcode{fibo} function~\clref{clnum:fn-fibo} is a pure mathematical definition of
the n-th Fibonacci number. When functions represent theorems
(like \textcode{lemma\_fibo\_is\_monotonic} \clref{clnum:fn-lemma-monotonic}),
expressing complex proofs like lemmas-by-induction is straightforward
via a simple recursive call~\clref{clnum:lemma-monotonic-induct}.

\begin{figure}[H]

\begin{lstlisting}[language=Rust]
#[spec]
fn fibo(n: nat) -> nat { %*\clnum\label{clnum:fn-fibo}*
  decreases(n);
  if n == 0 { 0 } else if n == 1 { 1 }
  else { fibo(n - 2) + fibo(n - 1) }
}

#[proof]
fn lemma_fibo_is_monotonic(i:nat, j:nat) { %*\clnum\label{clnum:fn-lemma-monotonic}*
  requires(%*\clnum\label{clnum:requires-clause}* i<=j);
  ensures(%*\clnum\label{clnum:ensures-clause}* fibo(i) <= fibo(j));
  decreases(j-i);

  if i<2 && j<2 { } else if i==j { }
  else if i==j-1 {
    reveal_with_fuel(fibo, 2);
    lemma_fibo_is_monotonic(i, j-1); %*\clnum\label{clnum:lemma-monotonic-induct}*
  } else {
    lemma_fibo_is_monotonic(i, j-1);
    lemma_fibo_is_monotonic(i, j-2);
  }
}

#[spec]
fn fibo_fits_u64(n: nat) -> bool { %*\clnum\label{clnum:fn-fibo-fits}*
  fibo(n) <= u64::MAX }

#[exec]
fn fibo_impl(n: u64) -> u64 { %*\clnum\label{clnum:fn-fibo-impl}*
  requires(%*\clnum\label{clnum:requires-clause-impl}* fibo_fits_u64(n));
  ensures(%*\clnum\label{clnum:ensures-clause-impl}* |result: u64| result == fibo(n));
  if n == 0 { return 0; }
  let mut prev: u64 = 0; let mut cur: u64 = 1;
  let mut i: u64 = 1;
  while i < n {
    invariant([%*\clnum\label{clnum:invariant-clause}*
      i > 0, i <= n,
      fibo_fits_u64(n as nat), fibo_fits_u64(i as nat),
      cur == fibo(i), prev == fibo(i as nat - 1),
    ]);
    let new_cur = cur + prev;
    prev = cur; cur = new_cur; i = i + 1;
    lemma_fibo_is_monotonic(i, n);
  }
  cur
}
\end{lstlisting}

\caption{A proof of correcntess of a function computing the n-th
Fibonacci number. We use circled letters, similar to \protect\circled{Z},
to mark points of interest in the code.}
\label{fig:fibo}
\end{figure}

Other tools, like Dafny, need complex encodings in SMT to support the
semantics of the language, and in our
experience this can be the cause of unpredictable prover performance.
We choose to prioritize efficient, straighforward encoding of Rust to Z3 in
Verus (without an intermediate language like Boogie).
Thanks to Rust's linear typing guarantees we can
often encode potentially mutable, complex objects, as immutable SMT
datatypes, rather than having to rely on complex rules and axioms. Even
objects with interior mutability that expose an ``immutable''
interface can often be directly encoded.

Additionally, Rust's memory management story removes a significant
challenge of other tools, like Dafny, that need to encode and manage the
heap to handle potential aliasing. Heap-reasoning is notoriously costly in SMT-based tools.
For example, in our experience building and verifying VeriBetrKV, a verified crash-safe
high-performance key-value store, we were able to reduce proof burden
(in terms of LOCs and verification time) by up to 30\% by integrating a
linear type system into a fork of Dafny.

This design principle of prioritizing efficient Z3 queries
by relying on safe Rust and the linear type system
comes with a necessary compromise:
we do not intend to support all Rust features and libraries,
instead excluding those that would
require complicated encoding.
\footnote{In the talk, I will highlight which benefits and
limitations this may bring in contrast with other Rust verification
tools like RustBelt and Prusti, and how these tools may complement each other.}
We do, however, plan to provide
verification-aided safe alternatives for patterns where ``unsafe''
would otherwise be required in Rust.

\section{Borrow-checking proofs}\label{borrow-checking-proofs}

As part of recent work, we recognized the value
of tracking ownership of ghost variables, which do not appear in the
emitted code, but aid verification. For example, in concurrent code,
linear ghost variables can facilitate tracking of ghost permissions to
access non-atomic memory, represent a thread's knowledge of a
concurrent, shared-memory protocol state, or represent limited
aliasing domains, for example as a mechanism to encapsulate and prove
non-linear data-structures that require aliased references such as
\texttt{Rc}. To enable using linearity to track these assertions,
Verus supports three modes for variables and functions,
\textbf{\textcode{\#{[}spec{]}}}, \textbf{\textcode{\#{[}proof{]}}}, and
\textbf{\textcode{\#{[}exec{]}}}, as shown in Figure~\ref{fig:fibo} and
Figure~\ref{fig:linear-ghost}.

Code and variables in \textbf{\textcode{\#{[}spec{]}} mode} are always
ghost and make up the pure mathematical language used to write specifications.
Examples of \textcode{spec} are the predicates \clref{clnum:fn-fibo},
\clref{clnum:fn-fibo-fits}, and the expressions in
\textcode{requires} \clref{clnum:requires-clause}\clref{clnum:requires-clause-impl},
\textcode{ensures} \clref{clnum:ensures-clause}\clref{clnum:ensures-clause-impl},
and \textcode{invariant} clauses \clref{clnum:invariant-clause}.
Code in this mode is not checked for linearity and borrow rules,
and is always erased before compilation.

Code in \textbf{\textcode{\#{[}proof{]}} mode} is ghost code used to
define lemmas (e.g. \clref{clnum:fn-lemma-monotonic}) and prove postconditions
(e.g. \clref{clnum:lemma-monotonic-induct}).
To support using linear ghost variables to track permissions and
other singly-owned tokens that can aid verification, \textcode{proof}
code supports linear types and is checked by the borrow checker; it
is also erased, like \textcode{spec}. Figure~\ref{fig:linear-ghost}
shows an example of the borrow-checker preventing \clref{clnum:linear-error}
an illegal use \clref{clnum:linear-illegal} of a linear ghost permission
\clref{clnum:linear-permission}
whose ownership had been transferred \clref{clnum:linear-transfer}.

Executable code (\textbf{\textcode{\#{[}exec{]}}} mode) is used for the
actual implementation (e.g. \clref{clnum:fn-fibo-impl}),
and abides by the same rules as regular Rust code.

\begin{figure}
\begin{lstlisting}[language=Rust]
#[exec]
fn increment(
  // a shared memory location
  counter: &PCell<u64>,
  // a linear ghost permission to access it
  #[proof] permission: &mut Permission<u64>) {
  requires(...); ensures(...);

  let cur = counter.borrow(permission); // read
  counter.put(cur + 1, permission);     // write
}

#[exec]
fn release(#[proof] permission: Permission<u64>)
{ ... }

#[exec]
fn start_thread(
  counter: &PCell<u64>,
  #[proof] mut permission: Permission<u64> %*\clnum\label{clnum:linear-permission}*) {
  requires(...);

  release(permission); %*\clnum\label{clnum:linear-transfer}*

  // error: borrow of moved value: `permission` %*\clnum\label{clnum:linear-error}*
  increment(counter, %*\clnum\label{clnum:linear-illegal}* &mut permission);

  assert(...); // that the counter was incremented
}
\end{lstlisting}

\caption{An example of a linear ``ghost'' permission used to access
a shared memory counter. This code is rejected by the borrow checker
because \textcode{permission} is a linear ``ghost'' value.}
\label{fig:linear-ghost}
\end{figure}

\section{Tailored solvers}

In our experience verifying large scale system software, we needed
the flexibility to write proofs in a variety of ways (e.g. by induction,
using ghost variables, using linearity). Verus enables
switching the solver's strategy for some proofs, for example by
selectively leveraging the SMT solver's bit-vector theory
while using the integer theory for the rest of the system.
We are also experimenting with support for dedicated solvers
(in addition to Z3) to dispatch certain verification conditions
for which they are more effective.
% is an important part of this for low-level OS
% coding, packet processing, run-time systems, crypto, etc.
\section{Well-engineered foundation}

An explicit goal of the project is to build a solid, well-engineered
foundation with excellent diagnostics, to aid onboarding of new users,
in the spirit of the Rust's compiler error reporting (I will demonstrate
this in the demos). A well-engineered foundation should also allow us
and others to use Verus as the basis to experiment with new verification
language features and techniques.

\end{document}


\documentclass[10pt]{article}
\input{articlepackage}
\usepackage[labelfont=bf]{caption}
\usepackage[margin=0.8in]{geometry}
\usepackage{fancyhdr}
\pagestyle{fancy}
\lhead{\small\textsc{Three Body Decays}}
\chead{}
\rhead{}
\lfoot{}
\cfoot{\textit{\thepage}}
\rfoot{}

\renewcommand*{\thefootnote}{\fnsymbol{footnote}}
\setcounter{footnote}{1}

\begin{document}
\hrule
\vspace{1pt}
\hrule
\begin{center}
\large\textsc{\color{darkblue}\textbf{The Three Body Decay $\pi_0 \rightarrow \gamma\chi\chi$}}
\vspace{5pt}

\footnotesize\textit{\textbf{J.B.G. Alvey}, Theoretical Particle Physics and Cosmology, King's College London\footnote{\href{mailto:james.alvey@kcl.ac.uk}{james.alvey@kcl.ac.uk}}}
\end{center}
\begin{abstract}
\noindent Set of notes describing the kinematics for the two subsequent on-shell two-body decays $\pi_0 \rightarrow \gamma V$, $V \rightarrow \chi\chi$. The kinematics constrain the masses to lie in the ranges $2 m_\chi < m_V < m_\pi$.
\end{abstract}
\hrule
\tableofcontents
\vspace{5pt}
\hrule
\vspace{1pt}
\hrule
\vspace{10pt}
\renewcommand{\thefootnote}{\tiny\textbf{\arabic{section}.\arabic{footnote}}}
%\begin{multicols*}{2}
\section{The First Decay}
The first decay in the process involves $\pi_0 \rightarrow \gamma V$ where $V$ is some dark vector mediator that kinetically mixes with the standard model photon. In the situation of interest, we have a distribution of pion energies which we wish to translate into a distribution of dark matter energies. Let $p_\mu^\pi$ be the 4-momentum of the pion in the lab frame. We choose our co-ordinate frame such that the pion is propagating along the $z$-axis. In the lab frame, this 4-momenta is given by;
\begin{equation}
  p^\pi_\mu = (E_\pi, 0, 0, \sqrt{E_\pi^2 - m_\pi^2})
\end{equation}
whilst in the rest frame of the pion we have,
\begin{equation}
  p^{\pi, c}_\mu = (m_\pi, \mathbf{0})
\end{equation}
In the rest frame, we can consider the decay as shown in Figure \ref{fig:decay}. The angles $\theta$ and $\phi$ parametrise the decay and should be sampled from suitable distributions. In the implementation of what follows, we take $\phi \in U[0, 2\pi]$ and $\cos\theta \in U[-1, 1]$. Note that this of course does not mean the angles are uniformly distributed in the lab frame since they are boosted in the direction of the pion.
\begin{center}
\begin{figure}
  \centering
  \includegraphics[width=0.5\linewidth]{diag}
  \caption{We parametrise the decay of the $\pi_0$ using two angles, $\theta$ and $\phi$ which ar distributed accordingly.}
  \label{fig:decay}
\end{figure}
\end{center}
\subsection{Energy and Momentum Conservation}
\noindent Conserving energy and momentum in this frame;
\begin{align}
  \sqrt{\Abs{\mathbf{p}_V^c}^2 + m_V^2} + \Abs{\mathbf{p}_\gamma^c} &= m_\pi \\
  \mathbf{p}_V^c &= -\mathbf{p}_\gamma^c
\end{align}
Here the superscript indicates that the quantities are measured in the rest frame of the pion. Substituting the second relation into the first,
\begin{equation}
  \sqrt{\Abs{\mathbf{p}_V^c}^2 + m_V^2} + \Abs{\mathbf{p}_V^c} = m_\pi
\end{equation}
Solving this for $\Abs{\mathbf{p}_V^c}$, we find,
\begin{equation}
  \label{eq:pv}
  \Abs{\mathbf{p}_V^c} := p_V = \frac{m_\pi^2 - m_V^2}{2m_\pi}
\end{equation}
\subsection{Going Back to the Lab Frame}
We now have the 4-momentum for the massive mediator in the pion rest frame via the parametrisation,
\begin{equation}
  p^{V,c}_\mu = (\sqrt{p_V^2 + m_V^2}, p_V\sin\theta\cos\phi, p_V\sin\theta\sin\phi, p_V\cos\theta)
\end{equation}
We can then boost to the lab frame by making a Lorentz transformation with respect to the pion 3-momentum. This leads to a Lorentz transformation of the form,
\begin{equation}
  \Lambda_1 = \begin{pmatrix} \gamma_1 & 0 & 0 & \gamma_1\beta_1 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ \gamma_1\beta_1 & 0 & 0 & \gamma_1 \end{pmatrix}
\end{equation}
where the $\gamma$ and $\beta$ factors are given as;
\begin{equation}
  \label{eq:gammabeta}
  \gamma_1 = \frac{E_\pi}{m_\pi}, \, \beta_1 = \left(1 - \frac{1}{\gamma^2}\right)^{\frac{1}{2}}
\end{equation}
The 4-momentum of the mediator is then given via $p_\mu^V := (E^V, \mathbf{p}^V) = (\Lambda_1 p^{V, c})_\mu$;
\begin{align}
  E^V &= \gamma_1\sqrt{p_V^2 + m_V^2} + \gamma_1\beta_1p_V\cos\theta \\
  p^V_x &= p_V \sin\theta\cos\phi \label{eq:pvx} \\
  p^V_y &= p_V \sin\theta\sin\phi \\
  p^V_z &= \gamma_1\beta_1\sqrt{p_V^2 + m_V^2} + \gamma_1 p_V \cos\theta \label{eq:pvz}
\end{align}
where $p_V$, $\gamma_1$ and $\beta_1$ are as defined in \eqref{eq:pv} and \eqref{eq:gammabeta}.
\section{The Second Decay}
We need to repeat the process above but for the decay $V \rightarrow \chi\chi$. There will be two differences. Firstly, both the products of the decay have the same mass. Secondly, the mediator $V$ no longer has a $3$-momentum along the $z$ axis so the form of the Lorentz boost will change. We construct an identical diagram to Figure \ref{fig:decay} except we relabel the angles to be $\theta^V$ and $\phi^V$. We let the $4$-momentum of the dark matter particles in the rest frame of $V$ be;
\begin{equation}
  p^{1,c}_\mu = (E_{1}^c, \mathbf{p}^c_{1}), \, p^{2,c}_\mu = (E_{1}^c, -\mathbf{p}^c_{1})
\end{equation}
Now conservation of energy in the rest frame of $V$ gives;
\begin{align}
  \label{eq:pchi}
  2\sqrt{\Abs{p_{1}^c}^2 + m_\chi^2} = m_V \Rightarrow \Abs{\mathbf{p}_{1}^c} := p_\chi = \sqrt{\frac{1}{4}m_V^2 - m_\chi^2}
\end{align}
With the parameterisation given, the 4-momentum of the dark matter particles in the rest frame of $V$ is now given by;
\begin{equation}
  E_{1}^c = \sqrt{p_\chi^2 + m_\chi^2}, \mathbf{p}^c_1 = (p_\chi\sin\theta^V\cos\phi^V, \ldots)
\end{equation}
\subsection{Boosting to the Lab Frame}
The final step is to boost back to the lab frame using a suitable Lorentz transformation. This is determined by the $3$-momentum of the mediator. The Lorentz matrix then takes the form;
\begin{equation}
  \Lambda_2 = \begin{pmatrix} \gamma_2 & \gamma_2\beta_2 \frac{p^V_x}{q_V} & \gamma_2\beta_2 \frac{p^V_y}{q_V} & \gamma_2\beta_2 \frac{p^V_z}{q_V} \\ \gamma_2\beta_2 \frac{p^V_x}{q_V} & 1 + (\gamma_2 - 1)\frac{(p^V_x)^2}{q_V^2} & (\gamma_2 - 1)\frac{p^V_x p^V_y}{q_V^2} & (\gamma_2 - 1)\frac{p^V_x p^V_z}{q_V^2} \\ \gamma_2\beta_2 \frac{p^V_y}{q_V} & (\gamma_2 - 1)\frac{p^V_y p^V_x}{q_V^2} & 1 + (\gamma_2 - 1)\frac{(p^V_y)^2}{q_V^2} & (\gamma_2 - 1)\frac{p^V_y p^V_z}{q_V^2} \\ \gamma_2\beta_2 \frac{p^V_z}{q_V} & (\gamma_2 - 1)\frac{p^V_z p^V_x}{q_V^2} & (\gamma_2 - 1)\frac{p^V_z p^V_y}{q_V^2} & 1 + (\gamma_2 - 1)\frac{(p^V_x)^2}{q_V^2} \end{pmatrix}
\end{equation}
where we have,
\begin{equation}
  \label{eq:g2b2}
  \gamma_2 = \frac{E^V}{m_V}, \, \beta_2 = \left(1 - \frac{1}{\gamma_2^2}\right)^{\frac{1}{2}}, \, q_V = \sqrt{(p_x^V)^2 + (p_y^V)^2 + (p_z^V)^2}
\end{equation}
The energies and 3-momenta of the dark matter particles, in the lab frame of the original $\pi_0$ are then obtained via;
\begin{equation}
p_\mu(\chi_1) = (\Lambda_2 p^{1, c})_\mu, \, p_\mu(\chi_2) = (\Lambda_2 p^{2, c})_\mu
\end{equation}
The resulting energies, $E_\chi^{1, 2}$ can be obtained from the zeroth components, as well as the kinetic energies $T^{1, 2}_\chi = E_\chi^{1, 2} - m_\chi$. Explicitly, we find;
\begin{equation}
  E_{\chi}^{1, 2} = \gamma_2\sqrt{p_\chi^2 + m_\chi^2} \pm \frac{\gamma_2\beta_2p_\chi}{q_V}\left(p^V_x\sin\theta^V \cos\phi^V + p^V_y \sin\theta^V \sin\phi^V + p^V_z\cos\theta^V\right)
\end{equation}
where $p_\chi$ is defined in \eqref{eq:pchi}, $p_V^{x, y, z}$ in \eqref{eq:pvx} - \eqref{eq:pvz}, and $q_V$, $\gamma_2$, $\beta_2$ in \eqref{eq:g2b2}.
%\end{multicols*}
\end{document}


\label{statements}

A Program line is composed of a line number and one or more statements. Multiple statements are separated by colons \code{:}.

\section{IF}

\index{IF (statement)}\codeline{\textbf{IF} TRUTH\_VALUE \textbf{THEN} TRUE\_EXPRESSION [\textbf{ELSE} FALSE\_EXPERSSION]}

If \code{TRUTH\_VALUE} is truthy, executes \code{TRUE\_EXPRESSION}. If \code{TRUTH\_VALUE} is falsy and \code{FALSE\_EXPERSSION} is specified, executes that expression; otherwise, the next line or next statement will be executed.

\subsubsection*{Notes}

\begin{itemlist}
\item \codebf{IF} is both statement and expression. You can use IF-clause after \codebf{ELSE}, or within functions as well, for example.
\item \codebf{THEN} is \emph{not} optional, this behaviour is different from most of the BASIC dialects.
\item Also unlike the most dialects, \codebf{GOTO} cannot be omitted; doing so will make the number be returned to its parent expression.
\end{itemlist}

\section{ON}

\index{ON (statement)}\codeline{\textbf{ON} INDEX\_EXPRESSION \{\textbf{GOTO}|\textbf{GOSUB}\} LINE0 [\textbf{,} LINE1]\ldots}

Jumps to the line number returned by the \code{INDEX\_EXPRESSION}. If the result is outside of the range of the arguments, no jump will be performed.

\subsubsection*{Parameters}

\begin{itemlist}
\item \code{LINEn} can be a number, numeric expression (aka equations) or a line label.
\item When \code{OPTIONBASE 1} is used within the program, \code{LINEn} starts from 1 instead of 0.
\end{itemlist}

\section{DEFUN}

\index{DEFUN (statement)}\emph{There it is, the} DEFUN. \emph{All those new-fangled parser\footnote{A computer program that translates program code entered by you into some data bits that only it can understand.} and paradigms\footnote{A guidance to in which way you must think to assimilate your brain into the computer-overlord.} are tied to this very statement on \tbas, and only Wally knows its secrets\ldots}

\codeline{\textbf{DEFUN} NAME \textbf{(} [ARGS0 [\textbf{,} ARGS1]\ldots] \textbf{)} \textbf{=} EXPRESSION }

With the aid of other statements\footnote{Actually, only the IF is useful. Use closure expression for more sophisticated function definition.} and functions, DEFUN will allow you to ascend from traditional BASIC and do godly things such as \emph{recursion}\footnote{See recursion.} and \emph{functional programming}.

Oh, and you can define your own function, in traditional \code{DEF FN} sense.

\subsubsection*{Parameters}

\begin{itemlist}
\item \code{NAME} must be a valid variable name.
\item \code{ARGSn} must be valid variable names, but can be a name of variables already used within the BASIC program; their value will not be affected nor be used.
\end{itemlist}


\subsection{Identifiers, keywords and types}
SDIR introduces only one additional identifier with the keyword \codeword{sym} (see Section \ref{section:symbol}).\\
Every op and region specific to SDIR is prefixed by the keyword \codeword{sdir} followed by a dot to clearly mark the end of the prefix. Some examples are: \codeword{sdir.tasklet\{\}}, \codeword{sdir.return}, \codeword{sdir.alloc()}. \\
SDIR contains only four additional types on top of the builtin types: \codeword{!sdir.array<>}, \codeword{!sdir.memlet<>} (See Section \ref{section:memlet}), \codeword{!sdir.stream_array<>} and \codeword{!sdir.stream<>} (See Section \ref{section:stream}).

\subsubsection{Symbols} \label{section:symbol}
The op \codeword{sdir.alloc_symbol} creates a new symbol, but does not return anything. In order to access symbols the identifier \codeword{sym()} can be used. See Figure \ref{fig:symbol}. One can describe arithmetic expressions inside the identifier. Example:\codeword{sym("2*N - 1")}.
\smc{symbol}{Symbol used for range in map}

 \subsubsection{Symbolic sizes}
 Symbols (see Section \ref{section:symbol}) can be used to define the size of an \codeword{sdir.memlet}. One simply replaces the constants in the type by the symbol identifier \codeword{sym()}.\\
 Example:\codeword{\%A = sdir.alloc() : !sdir.memlet<sym("2*N")xi32>}.

 \subsubsection{Write conflict resolution}
In some cases an op might write to a location that contains the result of an other op. To resolve this conflict any op that writes to a location has an optional attribute called \codeword{wcr}. Examples:\codeword{sdir.copy\{wcr="add"\} \%a -> \%c} or \codeword{sdir.store\{wcr="max"\} \%1, \%a : i32}. "overwrite" is the default if no \codeword{wcr} is provided. A function may be provided as well. The function must have the signature \codeword{old, new -> resolved} and the types must match the memlet types. See Figure \ref{fig:wcr}.
\codeblock{wcr}{Write conflict resolution with custom function}

\subsubsection{Multiple basic blocks and branching}
Tasklets (see Section \ref{section:tasklet}) and \codeword{func} are the only regions that may have multiple basic blocks and branching. Every other region has only one basic block and therefore terminator operations can be omitted. SDFG regions (see Section \ref{section:multistate}) contain branches, but these are provided as attributes and therefore there is no need for branching with \codeword{br} or \codeword{cond_br}.

\subsubsection{Unnamed outputs}
Any output that is unnamed will be auto-generated with the prefix \codeword{__} (double underscore). Therefore explicitly naming any output with the same prefix is strictly forbidden for the user.


\section{Divine Power}\label{prayer:divinePower}
\textbf{Cost:} 50 CP\\
\textbf{Requirements:} Disciple of a God, 3 Piety Points \\
\textbf{Active, Prayer, Weapon, Repeatable, Source(50 Gold)}\\
While this prayer is active, you can add \sfrac{1}{4} of your EM to any damage or healing rolls you make with prayers or wonders if you want.
\\
Level Progression:\\
II: 500 CP, 7 Piety Points, \sfrac{1}{3} of your EM\\
III: 900 CP, 12 Piety Points, \sfrac{1}{2} of your EM\\
IV: 1,500 CP, 18 Piety Points, Your full Empathy\\


    \filetitle{dbredate}{Redate all tseries objects in a database}{dbase/dbredate}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
D = redate(D,OldDate,NewDate)
\end{verbatim}

\paragraph{Input arguments}\label{input-arguments}

\begin{itemize}
\item
  \texttt{D} {[} struct {]} - Input database with tseries objects.
\item
  \texttt{OldDate} {[} numeric {]} - Base date that will be converted to
  a new date in all tseries objects.
\item
  \texttt{NewDate} {[} numeric {]} - A new date to which the base date
  \texttt{OldDate} will be changed in all tseries objects;
  \texttt{newDate} need not be the same frequency as \texttt{OldDate}.
\end{itemize}

\paragraph{Output arguments}\label{output-arguments}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \texttt{d} {[} struct {]} - Output database where all tseries objects
  have identical data as in the input database, but with their time
  dimension changed.
\end{itemize}

\paragraph{Description}\label{description}

\paragraph{Example}\label{example}


\subsection{Elasticity of substitution}


\subsubsection{\tt ipbd}
\label{benchmarkipbd}

{\tt ipbd} is a synthetic AKS benchmark that performs
a Boolean-Double ({\apl B\qplus\qdot\qtimes\qtran\0D}) matrix product on
arrays of shape {\tt [500, 500]}.

We expect that \wlf and \awlf should exhibit similar performance
on this benchmark.


%% Introduction
\chapter{Introduction}\label{sec:introduction}
%% - Motivating examples
Consider the following scenario:
\begin{quote}
  John and Mary are working from home one morning when they get a craving for a
  slice of cake. Being denizens of the web, they quickly find the nearest store
  which does home deliveries.
  Unfortunately for them, they both order their cake at the \emph{same} store,
  which has only one slice left. After that, all it can deliver is
  disappointment.
\end{quote}
This is an example of a \emph{race condition}. We can model this scenario in the
\textpi-calculus, assuming \john, \mary and \store are three processes
modeling John, Mary and the store, and \sliceofcake and \nope are two channels
giving access to a slice of cake and disappointment, respectively.
As expected, this process has two possible outcomes: either John gets the cake,
and Mary gets disappointment, or vice versa.
\[
  \begin{array}{c}
    \tm{(\piPar{%
    \piSend{x}{\sliceofcake}{\piSend{x}{\nope}{\store}}
    }{%
    \piPar{\piRecv{x}{y}{\john}}{\piRecv{x}{z}{\mary}}
    })}
    \\[1ex]
    \rotatebox[origin=c]{270}{$\Longrightarrow^{\star}$}
    \\[1ex]
    \tm{(\piPar{\store}{\piPar{\piSub{\sliceofcake}{y}{\john}}{\piSub{\nope}{z}{\mary}}})}
    \quad
    \text{or}
    \quad
    \tm{(\piPar{\store}{\piPar{\piSub{\nope}{y}{\john}}{\piSub{\sliceofcake}{z}{\mary}}})}
  \end{array}
\]
While John or Mary may not like all of the outcomes, it is the store which is
responsible for implementing the online delivery service, and the store is happy
with either outcome. Thus, the above is an interaction we would like to be able to
model.

Now consider another scenario, which takes place \emph{after} John has already
bought the cake:
\begin{quote}
  Mary is \emph{really} disappointed when she finds out the cake has sold out.
  John, always looking to make some money, offers to sell the slice to her for a
  profit. Mary agrees to engage in a little bit of back-alley cake resale, but
  sadly there is no trust between the two.
  John demands payment first.
  Mary would rather get her slice of cake before she gives John the money.
\end{quote}
This is an example of a \emph{deadlock}. We can also model this scenario in the
\textpi-calculus, assuming that \bill\ is a channel giving access to some
adequate amount of money.
\[
  \begin{array}{c}
    \tm{(\piPar{%
    \piRecv{x}{z}{\piSend{y}{\sliceofcake}{\john}}
    }{%
    \piRecv{y}{w}{\piSend{x}{\bill}{\mary}}
    })}
    \quad
    \centernot\Longrightarrow^{\star}
  \end{array}
\]
The above process does not reduce. As both John and Mary would prefer the
exchange to be made, this interaction is desired by \emph{neither}. Thus, the
above is an interaction we would \emph{somehow} like to exclude.

Session types~\parencite{honda1993} statically guarantee that concurrent
programs, such as those above, respect communication protocols.
Session-typed calculi with logical foundations, such as
\piDILL~\parencite{caires2010} and CP~\parencite{wadler2012}, obtain deadlock freedom as a
result of a close correspondence with logic.
The same correspondence, however, also rules out non-determinism and race
conditions.

In this dissertation, we present \nodcap (nodcap), an extension of
CP~\parencite{wadler2012} with a novel account of non-determinism and races.
Inspired by bounded linear logic~\parencite{girard1992}, we introduce a form of
shared channels, in which the type of a shared channel tracks how many times it
is reused.
As in the untyped $\pi$-calculus, sharing introduces the potential for
non-determinism.
We show that our approach is sufficient to capture practical examples of races,
such as the web store, as well as other formal characterizations of
non-determinism, such as non-deterministic choice.  However, \nodcap does not
lose the metatheoretical benefits of CP: we show that it enjoys termination and
deadlock-freedom.

This dissertation proceeds as follows.
In \cref{sec:background}, we introduce \cp and reproduce the proofs of some of
its meta-theoretical results, and discuss related work which also addresses the
addition of non-determinism to logic-inspired session typed process calculi.
In \cref{sec:cppi}, we introduce an alternative reduction strategy for \cp,
which more closely resembles reduction in the \textpi-calculus.
In \cref{sec:main}, we introduce \nodcap.
Finally, in \cref{sec:discussion}, we conclude with a discussion of the work
done in this dissertation and potential avenues for future work.
%%% Local Variables:
%%% TeX-master: "main"
%%% End:


\documentclass[aps,prl]{revtex4-2}
\usepackage{graphicx,amssymb,amsmath}
\bibliographystyle{apsrev4-2}
\renewcommand*{\citenumfont}[1]{S#1}
\renewcommand*{\bibnumfmt}[1]{[S#1]}

\begin{document}

\title{Supplementary Material\\Vicissitudes of Cities Driven by Re-distributive Growth}
\date{\today}
\author{Gezhi Xiu, Jianying Wang, Lei Dong}
% \email{xiugz@pku.edu.cn}
% \affiliation{IRSGIS, Peking University}

\author{Yu Liu}
\email{liuyu@urban.pku.edu.cn}
\affiliation{Institute of Remote Sensing and Geographic Information Systems (IRSGIS), Peking University}
% \altaffiliation{CAMS (CNRS/EHESS) 190-198, avenue de France, 75244 Paris Cedex 13, France}

\pacs{}

% If your reference list includes text notes as well as references,
% include the following line; otherwise, comment it out.

\maketitle
\tableofcontents
\vspace{1cm}

\section{Mathematical Proofs}

\subsection{Average Age within the Memory Kernel}

We consider the active population over the whole region as working population. The generation speed of population is $N^*\beta_2 + k\beta_1$. Thus the population is refreshing, leading to a constant expected age of \[N^*/\beta'\] for $\beta' := (N^*\beta_2 + k\beta_1)/N^*+k$. This leads to a practical implication that the working age allowed in cities shall be related with the sum of working offer from central government of a region. For instance, in the United States, the regressive value of $\beta$ is around $0.04$. We assume that the working years of a person is 40 years, the expected $N^*$ for the United States is 1,600 units, i.e., 4 millions distributive working opportunities in all cities.

\section{Details on the simulations}

The simulation results presented here are obtained in the following way. Instead of conducting the designed protocol, we do it in an equivalent way by stretching timeline to events labeled in integer. At each time step, we first decide if we add a new city, with probability $p(S)$, or a new meta-population to the existing city, with probability $1-p(S)$. The probability $p(S)$ is determined by the total regional population and number of cities, $\frac{k\beta_1}{k\beta_1+N\beta_2}$. If a city is to be built, the place of it is randomly chosen at an empty spot. By empty we mean the block of the spot contains no existing node. If it is a new node to be generated, we first determine its ancestor node, and land it at $(r,\theta)$ of the ancestor node, where $r$ is constant default as $0.5$ and $\theta$ is a realization of a uniform distributed random variable $\mathcal{U}[0,2\pi]$. The new generations of nodes are accompanied with a record in the memory kernel (MK).

% \bibliographystyle{plain}
\bibliography{refs.bib}

\end{document}