% \newcommand{\R}{\mathbb{R}}
\newcommand{\V}{\mathbb{V}}
% \newcommand{\E}{\mathop{\mathbb{E}}}
\newcommand{\Prob}{\mathbb{P}}

\section{Introduction}

% \begin{itemize}
%     \item Offline RL/ prior work

%     \item Summary of benchmark results (guidance)

%     \item What is holding multi-step back? evaluation error caused by distribution shift and iterative error exploitation

%     \item When can multi-step be more effective? Very broad coverage to estimate bad states. When behavior is good, one-step more effective.

%     \item Summarize contributions
% \end{itemize}

An important step towards effective real-world RL is to improve sample efficiency. One avenue towards this goal is offline RL (also known as batch RL) where we attempt to learn a new policy from data collected by some other behavior policy without interacting with the environment. Recent work in offline RL is well summarized by \citet{levine2020offline}.

In this paper, we challenge the dominant paradigm in the deep offline RL literature that primarily relies on actor-critic style algorithms that alternate between policy evaluation and policy improvement \citep{fujimoto2018off, fujimoto2019benchmarking, peng2019advantage, kumar2019stabilizing, kumar2020conservative, wang2020critic, wu2019behavior, kostrikov2021offline, jaques2019way, Siegel2020Keep, nachum2019algaedice}. All these algorithms rely heavily on off-policy evaluation to learn the critic. Instead, we find that a simple baseline which only performs one step of policy improvement using the behavior Q function often outperforms the more complicated iterative algorithms.
Explicitly, we find that our one-step algorithm beats prior results of iterative algorithms on most of the gym-mujoco \citep{brockman2016gym} and Adroit \citep{rajeswaran2017learning} tasks in the the D4RL benchmark suite \citep{fu2020d4rl}.

We then dive deeper to understand why such a simple baseline is effective. First, we examine what goes wrong for the iterative algorithms. When these algorithms struggle, it is often due to poor off-policy evaluation leading to inaccurate Q values. We attribute this to two causes: (1) distribution shift between the behavior policy and the policy to be evaluated, and (2) iterative error exploitation whereby policy optimization introduces bias and dynamic programming propagates this bias across the state space.
We show that empirically both issues exist in the benchmark tasks and that one way to avoid these issues is to simply avoid off-policy evaluation entirely.

Finally, we recognize that while the the one-step algorithm is a strong baseline, it is not always the best choice. In the final section we provide some guidance about when iterative algorithms can perform better than the simple one-step baseline. Namely, when the dataset is large and behavior policy has good coverage of the state-action space, then off-policy evaluation can succeed and iterative algorithms can be effective.
In contrast, if the behavior policy is already fairly good, but as a result does not have full coverage, then one-step algorithms are often preferable.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.6\textwidth]{figures/offline-rl/cartoon.png}
    \vspace{-0.2cm}
    \caption{A cartoon illustration of the difference between one-step and multi-step methods. All algorithms constrain themselves to a neighborhood of ``safe'' policies around $ \beta$. A one-step approach (left) only uses the \textcolor{blue}{on-policy} $ \widehat Q^\beta$, while a multi-step approach (right) repeatedly uses \textcolor{red}{off-policy} $\widehat Q^{\pi_i}$.
    % \joan{This is a nice illustration. I was wondering to what extent we could illustrate this neighborhood as a landscape (given by the objective function $J$) and show the two cartoon scenarios: (i) very smooth landscape, in which multi-step approaches would be worse off due to errors in estimating the gradient ($Q$), (ii) rugged landscape / valley, so that $\beta$ is in a basin that needs a non-linear path to keep descending. Also, we can illustrate what happens in off-policy updates, as the red arrows start drifting away from the true ones (not accessible)}
    }
    \label{fig:cartoon}
\end{figure}

Our main contributions are:
\begin{itemize}[leftmargin=*]
    \item A demonstration that a simple baseline of one step of policy improvement outperforms more complicated iterative algorithms on a broad set of offline RL problems.
    \item An examination of failure modes of off-policy evaluation in iterative offline RL algorithms.
    \item A description of when one-step algorithms are likely to outperform iterative approaches.
\end{itemize}

\vspace{0.2cm}
\section{Setting and notation}

We will consider an offline RL setup as follows. Let $ \mathcal{M} = \{\mathcal{S}, \mathcal{A}, \rho, P, R, \gamma \}$ be a discounted infinite-horizon  MDP.
In this work we focus on applications in continuous control, so we will generally assume that both $ \mathcal{S}$ and $ \mathcal{A}$ are continuous and bounded.
We consider the offline setting where rather than interacting with $\mathcal{M}$, we only have access to a dataset $ D_N$ of $ N $ tuples of $ (s_i, a_i, r_i)$ collected by some behavior policy $ \beta$ with initial state distribution $ \rho$.
Let $ r(s,a) = \E_{r|s,a}[r]$ be the expected reward.
Define the state-action value function for any policy $ \pi$ by $ Q^\pi(s,a) := \E_{P, \pi | s_0 = s,\ a_0 = a}[\sum_{t=0}^\infty \gamma^t r(s_t, a_t)]$.
The objective is to maximize the expected return $ J $ of the learned policy:
\begin{align}
    J(\pi) := \E_{\rho, P, \pi}\left[ \sum_{t=0}^\infty \gamma^t r(s_t, a_t) \right] = \E_{\substack{s\sim \rho\\ a \sim \pi|s}}[Q^\pi(s,a)]. % \frac{1}{1-\gamma} \E_{s\sim d^\pi_\rho} \E_{a\sim \pi|s}[r(s,a)]
\end{align}
Following \citet{fu2020d4rl} and others in this line of work, we allow access to the environment to tune a small (< 10) set of hyperparameters.
See \citet{paine2020hyperparameter} for a discussion of the active area of research on hyperparameter tuning for offline RL.
We also discuss this further in Appendix \ref{sec:app_exp_setup}.

\section{Related work}

Most prior work on deep offline RL consists of iterative actor-critic algorithms. The primary innovation of each paper is to propose a different mechanism to ensure that he learned policy does not stray too far from the data generated by the behavior policy. Broadly, we group these methods into three camps: policy constraints/regularization, modified of imitation learning, and Q regularization:

\begin{enumerate}[leftmargin=*]
\item The majority of prior work acts directly on the policy.
Some authors have proposed explicit constraints on the learned policy to only select actions where $ (s,a)$ has sufficient support under the data generating distribution \citep{fujimoto2018off, fujimoto2019benchmarking, laroche2019safe}.
Another proposal is to regularize the learned policy towards the behavior policy \citep{wu2019behavior} usually either with a KL divergence \citep{jaques2019way} or MMD \citep{kumar2019stabilizing}. This is a very straighforward way to stay close to the behavior with a hyperparameter that determines just how close. All of these algorithms are iterative and rely on off-policy evaluation.

\item \citep{Siegel2020Keep, wang2020critic, chen2020bail} all use algorithms that filter out datapoints with low Q values and then perform imitation learning.
\citep{wang2018exponentially, peng2019advantage} use a weighted imitation learning algorithm where the weights are determined by exponentiated Q values.
These algorithms are iterative.
\item Another way to prevent the learned policy from choosing unknown actions is to incorporate some form of regularization to encourage staying near the behavior and being pessimistic about unknown state, action pairs \citep{wu2019behavior, nachum2019algaedice, kumar2020conservative, kostrikov2021offline}.
However, properly being able to quantify uncertainty about unknown states is notoriously difficult when dealing with neural network value functions \citep{buckman2020importance}.
Again all of these algorithms are iterative.
\end{enumerate}

Some recent work has also noted that optimizing policies based on the behavior value function can perform surprisingly well \citep{gulcehre2020rl, goo2020you}. However, these papers propose complicated variants of the one-step approach involving ensembles, non-standard regularizers and paraterizations or ensembles and distributional Q functions. In contrast, we implement the simplest possible one-step algorithms without any modifications to the network architecture or standard regularizers/constraints.
Moreover, we focus on providing an analysis of when and why this simple baseline works.

There are also important connections between the one-step algorithm and the literature on conservative policy improvement \citep{kakade2002approximately, schulman2015trust, achiam2017constrained}, which we discuss in more detail in Appendix \ref{sec:app_improvement}.

\section{Defining the algorithms}

In this section we provide a unified algorithmic template for offline RL algorithms as offline approximate modified policy iteration. We show how this template captures our one-step algorithm as well as a multi-step policy iteration algorithm and an iterative actor-critic algorithm.
Then any choice of policy evaluation and policy improvement operators defines one-step, multi-step, and iterative algorithms.

%Full details about our implementation can be found in Appendix \warn{write up implementation}.

\subsection{Algorithmic template}

We consider a generic offline approximate modified policy iteration (OAMPI) scheme, shown in Algorithm \ref{alg:oapi}. Essentially the algorithm alternates between two steps. First, there is a policy evaluation step where we estimate the Q function of the current policy $ \pi_{k-1}$ by $ \widehat Q^{\pi_{k-1}}$ using only the dataset $ D_N$. Implementations also often use the prior Q estimate $ \widehat Q^{\pi_{k-2}}$ to warm-start the approximation process. Second, there is a policy improvement step. This step takes in the estimated Q function $ \widehat Q^{\pi_{k-1}}$, the estimated behavior $ \hat \beta$, and the dataset $ D_N$ and produces a new policy $ \pi_k$. Again an algorithm may use $ \pi_{k-1}$ to warm-start the optimization. Moreover, we expect this improvement step to be regularized or constrained to ensure that $ \pi_k$ remains in the support of $ \beta$ and $ D_N$. Choices for this regularization/constraint are discussed below. Now we discuss a few ways to instantiate the template.

\begin{figure*}[h]
\vspace{-0.5cm}
\begin{algorithm}[H]
    \caption{OAMPI} \label{alg:oapi}
    \begin{algorithmic}[1]
             \Require{$ K$, dataset $ D_N$,  estimated behavior $ \hat \beta$}
             \State Set $\pi_0 = \hat \beta$. Initialize $ \widehat Q^{\pi_{-1}}$ randomly.
             \For{k = 1, \dots, K}
                \State Policy evaluation: $ \widehat Q^{\pi_{k-1}} = \mathcal{Q}(\pi_{k-1}, D_N, \widehat Q^{\pi_{k-2}})$
                \State Policy improvement: $ \pi_{k} = \mathcal{I}(\widehat Q^{\pi_{k-1}}, \hat \beta, D_N, \pi_{k-1})$
             \EndFor
        \end{algorithmic}
    \end{algorithm}
\end{figure*}

\paragraph{One-step.}
The simplest algorithm sets the number of iterations $ K = 1$. We train the policy evaluation to estimate $ Q^\beta$, and then use one of the policy improvement operators discussed below to find the resulting $ \pi_1$.

\paragraph{Multi-step.}
The multi-step algorithm now sets $ K >1$. The evaluation operator must evaluate off-policy since $ D_N$ is collected by $ \beta$, but evaluation steps for $ K \geq 2$ require evaluating policies $ \pi_{k-1}\neq \beta$. Each iteration is trained to convergence in both the estimation and improvement steps.

\paragraph{Iterative actor-critic.}

An actor critic approach looks somewhat like multistep policy iteration, but does not attempt to train to convergence at each iteration. Instead, each iteration consists of one gradient step to update the Q estimate and one gradient step to improve the policy. Since all of the evaluation and improvement operators that we consider are gradient-based, this algorithm can adapt the same evaluation and improvement operators used by the multi-step algorithm. Most algorithms from the literature fall into this category \citep{fujimoto2018off, kumar2019stabilizing, kumar2020conservative, wu2019behavior,  wang2020critic, Siegel2020Keep}.

\subsection{Policy evaluation operators}

Following prior work on continuous state and action problems, we always evaluate by simple fitted Q evaluation \citep{fujimoto2018off, kumar2019stabilizing, Siegel2020Keep, wang2020critic, paine2020hyperparameter, wang2021instabilities}.
Explicitly the evaluation step for the one-step or multi-step algorithms looks like
\begin{align}
     \mathcal{Q}(\pi_{k-1}, D_N, \widehat Q^{\pi_{k-2}}) = \arg\min_Q \sum_{i=1}^N \left(r(s_i,a_i) + \gamma \E_{a'\sim \pi_{k-1}|s_i'}Q(s_i', a') - Q(s_i, a_i) \right)^2,
\end{align}
where the right hand side may depend on $\widehat Q^{\pi_{k-2}}$ to warm-start optimization.
In practice this is optimized by stochastic gradient descent with the use of a target network \citep{mnih2015human}.
For the iterative algorithm the $ \arg\min$ is replaced by a single stochastic gradient step.
We estimate the expectation over next state by a single sample from $ \pi_{k-1}$ (or from the dataset in the case when $ \pi_{k-1} = \hat \beta$). See \cite{voloshin2019empirical, wang2021instabilities} for more comprehensive examinations of this evaluation step.

\subsection{Policy improvement operators}

To instantiate the template, we also need to choose a specific policy improvement operator $ \mathcal{I}$. We consider the following improvement operators selected from those discussed in the related work section. Each operator has a hyperparameter controlling deviation from the behavior policy.

\paragraph{Behavior cloning.} The simplest baseline worth including is to just return $ \hat \beta$ as the new policy $ \pi$. Any policy improvement operator ought to perform at least as well as this baseline.

% \paragraph{Greedy.} The naive policy improvement operator just acts greedily with respect to $ \widehat Q^\beta$ so that $\hat \pi(a|s) = \one[a = \arg\max_{a'} \widehat Q^\beta(s, a')]$.
% Calculating this arg max may not feasible in a continuous action space, so we can instead amortize the computation by learning a neural network policy to maximize the estimated Q values such that
% \begin{align}
%     \pi_k = \mathcal{I}(\widehat Q, \hat \beta, D_N, \pi_{k-1}) =  \arg\max_{\pi} \sum_i \E_{a\sim \pi|s_i} [\widehat Q (s_i, a)]
% \end{align}
% where the optimization can be performed using stochastic optimization via the reparameterization trick. Specifically, at each iteration we (1) sample a minibatch of datapoints, (2) sample one reparameterized action for each state in the minibatch, and then (3) backpropogate to get a stochastic gradient estimate.

\paragraph{Constrained policy updates.}
Algorithms like BCQ \citep{fujimoto2018off} and SPIBB \citep{laroche2019safe} constrain the policy updates to be within the support of the data/behavior.
% The SPIBB algorithm requires uncertainty estimates that are difficult to quantify in continuous state and action spaces, and both of these algorithms are somewhat complicated.
In favor of simplicity, we implement a simplified version of the BCQ algorithm that removes the policy correction network which we call Easy BCQ.
We define a new policy $ \hat \pi^M_{k}$ by drawing $ M $ samples from $ \hat \beta$ and then executing the one with the highest value according to $ \widehat Q^\beta$. Explicitly:
\begin{align}
    \hat \pi^M_{k}(a|s) = \one[a = \arg\max_{a_j} \{\widehat Q^{\pi_{k-1}}(s, a_j): a_j \sim \pi_{k-1}(\cdot|s),\ 1 \leq j \leq M\}].
\end{align}

% Note, we can think about this policy in terms of pushforward distributions where we are taking an $ n$th order statistic of $ M $ samples from $ \widehat Q_\sharp^\beta (s, \hat \beta)$. If we let $ F$ be the CDF of $ \widehat Q_\sharp^\beta (s, \hat \beta)$ and $ F_M$ be the CDF of $ \widehat Q_\sharp^\beta (s, \hat \pi_M)$, it is straightforward to see that
% \begin{align}
%     F_M(v) = F(v)^M.
% \end{align}
% Thus, we know that we are moving mass in a strictly increasing way, and can come up with a policy improvement guarantee similar to the one we have for QFIL, but in terms of the difference between $ F(v) $ and $ F(v)^M$.

% This sort of analysis is novel since prior work on BCQ \citep{fujimoto2018addressing} proposes a more complicated algorithm and only provides a tabular analysis.

\paragraph{Regularized policy updates.}
Another common idea proposed in the literature is to regularize towards the behavior policy \citep{wu2019behavior, jaques2019way, kumar2019stabilizing, ma2019imitation}. For a general divergence $ D$ we can define an algorithm that maximizes a regularized objective:
\begin{align}
    \hat \pi^\alpha_k = \arg\max_\pi \sum_{i}  \E_{a\sim \pi|s}[\widehat Q^{\pi_{k-1}}(s_i, a)] - \alpha D(\hat \beta(\cdot|s_i), \pi(\cdot|s_i))
\end{align}

A comprehensive review of different variants of this method can be found in \cite{wu2019behavior} which does not find dramatic differences across regularization techniques.
In practice, we will use reverse KL divergence,  i.e. $ KL( \pi(\cdot|s_i) \| \hat \beta(\cdot|s_i))$.
%For the forward KL, i.e. $ KL(\beta(\cdot|s_i) \| \pi(\cdot|s_i))$, we can simply use the sampled action $ a_i$ from the dataset to estimate the divergence.
To compute the reverse KL, we draw samples from $ \pi(\cdot |s_i)$ and use the density estimate $ \hat \beta$ to compute the divergence. Intuitively, this regularization forces $ \pi$ to remain within the support of $ \beta$ rather than incentivizing $ \pi$ to cover beta.

\paragraph{Variants of imitation learning.}
Another idea, proposed by \citep{wang2018exponentially, Siegel2020Keep, wang2020critic, chen2020bail} is to modify an imitation learning algorithm either by filtering or weighting the observed actions so as to get a policy improvement. The weighted version that we implement uses exponentiated advantage estimates to weight the observed actions:
\begin{align}
    \hat \pi^\tau_k = \arg\max_\pi \sum_{i}  \exp(\tau (\widehat Q^{\pi_{k-1}}(s_i, a_i) - \widehat V(s_i))) \log \pi(a_i|s_i).
\end{align}
%This can also be derived as a variational implementation of the reverse-KL regularized update, as is done in \cite{wang2018exponentially}.

% \begin{align}
%     \hat \pi_b (a|s) = \arg\max_\pi \sum_{i}  \1[\widehat Q^\beta(s_i, a_i) \geq b(s_i)] \log \pi(a_i|s_i)
% \end{align}

\section{Benchmark Results}\label{sec:bench}

Our main empirical finding is that one step of policy improvement is sufficient to beat state of the art results on much of the D4RL benchmark suite \cite{fu2020d4rl}.
This is striking since prior work focuses on iteratively estimating the Q function of the current policy iterate, but we only use one-step derived from $ \widehat Q^\beta$. Results are shown in Table~\ref{tab:d4rl}. Full experimental details are in Appendix \ref{sec:app_exp_setup}.

\begin{table}[h]
    \centering
    \caption{Results of one-step algorithms on the D4RL benchmark. The first column gives the best results across several iterative algorithms considered in \cite{fu2020d4rl}. We run 3 seeds and each algorithm is tuned over 6 values of their respective hyperparameter. We report the mean and standard deviation over seeds on 100 evaluation episodes per seed. We \textbf{bold} the best result on each dataset and \textcolor{blue}{blue} any result where a one-step algorithm beat the best reported iterative result from \cite{fu2020d4rl}. We use m for medium, m-e for medium-expert, m-re for medium-replay, r for random, and c for cloned.}
    \begin{small}
    \begin{tabular}{lccccc}
        \toprule & Iterative & \multicolumn{4}{c}{One-step}\\
        \cmidrule(lr){2-2} \cmidrule(lr){3-6}
         & \cite{fu2020d4rl} & BC & Easy BCQ  & Rev. KL Reg & Exp. Weight \\
        \midrule
        halfcheetah-m & 46.3 &
                        41.9  $\pm$  0.1 &
                        \textcolor{blue}{52.6  $\pm$  0.2}  &
                        \textcolor{blue}{\textbf{55.2  $\pm$  0.4 }} &
                        \textcolor{blue}{48.4  $\pm$  0.1} \\
        walker2d-m & 81.1 &
                        68.6  $\pm$  6.3 &
                        \textcolor{blue}{\textbf{87.2  $\pm$  1.3}} &
                        \textcolor{blue}{85.9  $\pm$  1.4} &
                        \textcolor{blue}{81.8  $\pm$  2.2}  \\
        hopper-m & 58.8 &
                        49.9  $\pm$  3.1 &
                        \textcolor{blue}{74.5  $\pm$  6.2}&
                        \textcolor{blue}{\textbf{83.7  $\pm$  4.5}} &
                        \textcolor{blue}{59.6  $\pm$  2.5}\\
        \midrule
        halfcheetah-m-e & 64.7 &
                        61.1  $\pm$  2.7 &
                        \textcolor{blue}{78.2  $\pm$  1.6}  &
                        \textcolor{blue}{\textbf{93.8  $\pm$  0.5}} &
                        \textcolor{blue}{93.4  $\pm$  1.6} \\
        walker2d-m-e & 111.0 &
                        78.5  $\pm$  22.4 &
                        \textcolor{blue}{112.2  $\pm$  0.3}   &
                        \textcolor{blue}{111.2  $\pm$  0.2}  &
                        \textcolor{blue}{\textbf{113.0  $\pm$  0.4}}  \\
        hopper-m-e & \textbf{111.9} &
                        49.1  $\pm$  4.3 &
                        85.1  $\pm$  2.2 &
                        98.7  $\pm$  7.5 &
                        103.3  $\pm$  9.1 \\
        \midrule
        halfcheetah-m-re & \textbf{47.7} &
                         34.6  $\pm$  0.9 &
                         38.3  $\pm$  0.3  &
                        41.9  $\pm$  0.5 &
                         38.1  $\pm$  1.3 \\
        walker2d-m-re & 26.7 &
                        26.6  $\pm$  3.4  &
                         \textcolor{blue}{69.1  $\pm$  4.2} &
                        \textcolor{blue}{\textbf{74.9  $\pm$  6.6}} &
                         \textcolor{blue}{49.5  $\pm$  12.0}  \\
        hopper-m-re & 48.6 &
                       23.1  $\pm$  2.7 &
                        \textcolor{blue}{78.4  $\pm$  7.2} &
                        \textcolor{blue}{92.3  $\pm$  1.1} &
                        \textcolor{blue}{\textbf{97.5  $\pm$  0.7}}\\
        \midrule
        halfcheetah-r & \textbf{35.4} &
                        2.2  $\pm$  0.0 &
                        5.4  $\pm$  0.3 &
                        8.8  $\pm$  3.8   &
                        3.2  $\pm$  0.1 \\
        walker2d-r & \textbf{7.3} &
                        0.9  $\pm$  0.1 &
                        3.7  $\pm$  0.1  &
                        6.2  $\pm$  0.7 &
                        5.6  $\pm$  0.8 \\
        hopper-r & \textbf{12.2} &
                        2.0  $\pm$  0.1 &
                        6.6  $\pm$  0.1 &
                        7.9  $\pm$  0.7 &
                        7.5  $\pm$  0.4  \\
        \midrule
        pen-c & 56.9 &
                    46.9  $\pm$  11.0 &
                    \textcolor{blue}{\textbf{65.9  $\pm$  3.6}} &
                    \textcolor{blue}{57.4  $\pm$  3.5} &
                    \textcolor{blue}{60.0  $\pm$  4.1} \\
        hammer-c & 2.1 &
                    0.4  $\pm$  0.1  &
                    \textcolor{blue}{\textbf{2.9  $\pm$  0.5 }} &
                    0.2  $\pm$  0.1 &
                    2.1  $\pm$  0.7  \\
        relocate-c & -0.1 &
                    -0.1  $\pm$  0.0 &
                    \textcolor{blue}{\textbf{0.3  $\pm$  0.2}} &
                    \textcolor{blue}{0.2  $\pm$  0.1} &
                    \textcolor{blue}{0.2  $\pm$  0.1}   \\
         door-c & 0.4 &
                    0.0  $\pm$  0.1 &
                    \textcolor{blue}{\textbf{0.6  $\pm$  0.6}} &
                    0.2  $\pm$  0.7 &
                    0.2  $\pm$  0.3 \\
        \bottomrule
    \end{tabular}
    \end{small}
    \label{tab:d4rl}
\end{table}

% Goo:
% m: 35.6,  68.6, 76.4
% m-e: 16.6,  92.7, 106.4,
% m-re: 34.4,  41.5, 94.8
% r: 7.2, 8.1, 9.8
% c: 61.2, 0.4, -0.2, 1.1

As we can see in the table, all of the one-step algorithms usually outperform the best iterative algorithms tested by \citet{fu2020d4rl}.
The one notable exception is the case of random data (especially on halfcheetah), where iterative algorithms have a clear advantage. We will discuss potential causes of this further in Section \ref{sec:when}.

To give a more direct comparison that controls for any potential implementation details, we use our implementation of reverse KL regularization to create multi-step and iterative algorithms. We are not using algorithmic modifications like Q ensembles, regularized Q values, or early stopping that have been used in prior work. But, our iterative algorithm recovers similar performance to prior regularized actor-critic approaches. These results are shown in Table~\ref{tab:multi}.

\begin{table}[h]
\vspace{-0.5 cm}
    \centering
    \caption{Results of reverse KL regularization on the D4RL benchmark across one-step, multi-step, and iterative algorithms. Again we run 3 seeds and 6 hyperparameters and report the mean and standard deviation across seeds using 100 evaluation episodes.}
    \begin{small}
    \begin{tabular}{lccc}
        \toprule
         & One-step & Multi-step & Iterative \\
        \midrule
        halfcheetah-m &
                        55.2  $\pm$  0.4  &
                        \textbf{59.3  $\pm$  0.7} &
                        51.2  $\pm$  0.2\\
        walker2d-m &
                        \textbf{85.9  $\pm$  1.4} &
                        74.5  $\pm$  2.8    &
                        74.8  $\pm$  0.7  \\
        hopper-m &
                        \textbf{83.7  $\pm$  4.5} &
                        54.8  $\pm$  4.3  &
                        54.7  $\pm$  1.9\\
        \midrule
        halfcheetah-m-e &
                        93.8  $\pm$  0.5 &
                        \textbf{94.2  $\pm$  0.5} &
                        93.7  $\pm$  0.6 \\
        walker2d-m-e &
                        \textbf{111.2  $\pm$  0.2} &
                        109.8  $\pm$  0.3  &
                        108.7  $\pm$  0.6 \\
        hopper-m-e &
                        \textbf{98.7  $\pm$  7.5} &
                        90.6  $\pm$  18.8 &
                        94.5  $\pm$  11.9\\
        \midrule
        halfcheetah-r &
                        8.8  $\pm$  3.8  &
                        18.3  $\pm$  6.5 &
                        \textbf{21.2  $\pm$  5.2}\\
        walker2d-r &
                        \textbf{6.2  $\pm$  0.7} &
                        5.4  $\pm$  0.2 &
                        5.4  $\pm$  0.4\\
        hopper-r &
                        7.9  $\pm$  0.7 &
                        \textbf{21.9  $\pm$  8.9} &
                        9.7  $\pm$  0.4\\
        \bottomrule
    \end{tabular}
    \end{small}
    \label{tab:multi}
\end{table}

Put together, these results immediately suggest some guidance to the practitioner: it is worthwhile to run the one-step algorithm as a baseline before trying something more elaborate. The one-step algorithm is substantially simpler than prior work, but usually achieves better performance.

% \subsection{Benchmarking QFIL}

% \begin{figure}[h]
%     \centering
%     \includegraphics[width=0.3\textwidth]{figures/offline-rl/benchmark/hc-m.png}\includegraphics[width=0.3\textwidth]{figures/offline-rl/benchmark/hc-me.png}\includegraphics[width=0.3\textwidth]{figures/offline-rl/benchmark/hc-r.png}\\
%     \includegraphics[width=0.3\textwidth]{figures/offline-rl/benchmark/w-m.png}\includegraphics[width=0.3\textwidth]{figures/offline-rl/benchmark/w-me.png}\includegraphics[width=0.3\textwidth]{figures/offline-rl/benchmark/w-r.png}\\
%     \includegraphics[width=0.3\textwidth]{figures/offline-rl/benchmark/h-m.png}\includegraphics[width=0.3\textwidth]{figures/offline-rl/benchmark/h-me.png}\includegraphics[width=0.3\textwidth]{figures/offline-rl/benchmark/h-r.png}
%     \caption{Results on the D4RL benchmark, with state of the art (SOTA) taken from \cite{fu2020d4rl}. \qfil\ beats the state of the art on 6 out of 9 datasets shown in the figure and always makes a substantial improvement over behavior cloning (BC).}
%     \label{fig:benchmark}
% \end{figure}

\section{What goes wrong for iterative algorithms?}\label{sec:why}

% \subsection{Policy improvement perspective}

% We can modify the performance difference lemma to measure the difference between two policies $ \pi_1, \pi_2$ using the state distribution derived from the behavior $ \beta$.

% \begin{lemma}[Approximate three-way performance difference]
% For any three policies $ \pi_2, \pi_1$ and $ \beta$,
% \begin{align}
%     (1-\gamma) (J(\pi_2) &- J(\pi_1)) = \underbrace{\E_{s\sim d_\beta}[\widehat Q^{\pi_1}(s,\pi_2) - \widehat Q^{\pi_1}(s,{\pi_1})]}_{Estimated\ improvement} \\ &- \underbrace{\E_{s\sim d_\beta}\left[(\widehat Q^{\pi_1}(s,\pi_2) - Q^{\pi_1}(s, \pi_2)) + (Q^{\pi_1}(s,{\pi_1}) - \widehat Q^{\pi_1}(s, {\pi_1})) \right]}_{Evaluation\ error}\\
%     &- \underbrace{\E_{\substack{s\sim d_\beta \\ s' \sim d_{\pi_2}}}\left[(Q^{\pi_1}(s, \pi_2) - Q^{\pi_1}(s', \pi_2) ) + (Q^{\pi_1}(s', \pi_1) - Q^{\pi_1}(s, \pi_1)) \right]}_{State\ distribution\ mismatch}
% \end{align}
% \end{lemma}

The benchmark experiments show that one step of policy improvement often beats iterative and multi-step algorithms. In this section we dive deeper to understand why this happens. First, by examining the learning curves of each of the algorithms we note that iterative algorithms require stronger regularization to avoid instability. Then we identify two causes of this instability: \emph{distribution shift} and \emph{iterative error exploitation}.

Distribution shift causes evaluation error by reducing the effective sample size in the fixed dataset for evaluating the current policy and has been extensively considered in prior work as discussed below. Iterative error exploitation occurs when we repeatedly optimize policies against our Q estimates and exploit their errors. This introduces a bias towards overestimation at each step (much like the training error in supervised learning is biased to be lower than the test error). Moreover, by iteratively re-using the data and using prior Q estimates to warmstart training at each step, the errors from one step are amplified at the next. This type of error is particular to multi-step and iterative algorithms.

\subsection{Learning curves and hyperparameter sensitivity}

To begin to understand why iterative and multi-step algorithms can fail it is instructive to look at the learning curves. As shown in Figure \ref{fig:learning_curves}, we often observe that the iterative algorithm will begin to learn and then crash. Regularization can help to prevent this crash since strong enough regularization towards the behavior policy ensures that the evaluation is nearly on-policy.

\begin{figure}[h]
    \centering
    \includegraphics[width=\textwidth]{figures/offline-rl/learning curves/learning_curves.png}
    \caption{Learning curves and final performance on halfcheetah-medium across different algorithms and regularization hyperparameters. Error bars show min and max over 3 seeds. Similar figures for other datasets from D4RL can be found in Appendix~\ref{sec:app_extra_exp}.}
    \label{fig:learning_curves}
\end{figure}

In contrast, the one-step algorithm is more robust to the regularization hyperparameter. The rightmost panel of the figure shows this clearly. While iterative and multi-step algorithms can have their performance degrade very rapidly with the wrong setting of the hyperparameter, the one-step approach is more stable.
Moreover, we usually find that the optimal setting of the regularization hyperparameter is lower for the one-step algorithm than the iterative or multi-step approaches.

\subsection{Distribution shift}

Any algorithm that relies on off-policy evaluation will struggle with distribution shift in the evaluation step. Trying to evaluate a policy that is substantially different from the behavior reduces the effective sample size and increases the variance of the estimates. Explicitly, by distribution shift we mean the shift between the behavior distribution (the distribution over state-action pairs in the dataset) and the evaluation distribution (the distribution that would be induced by the policy $ \pi$ we want to evaluate).

\paragraph{Prior work.} There is a substantial body of prior theoretical work that suggests that off-policy evaluation can be difficult and this difficulty scales with some measure of distribution shift.
%\cite{jiang2016doubly} give an exponential (in horizon) lower bound on sample complexity in the tabular setting for any unbiased estimator from the Cramer-Rao lower bound.
\citet{wang2020statistical, Amortila2020AVO, zanette2021exponential} give exponential (in horizon) lower bounds on sample complexity in the linear setting even with good feature representations that can represent the desired Q function and assuming good data coverage.
Upper bounds generally require very strong assumptions on both the representation and limits on the distribution shift \citep{wang2021instabilities, duan2020minimax, chen2019information}. Moreover, the assumed bounds on distribution shift can be exponential in horizon in the worst case.
% The literature on policy improvement also suggests limiting distribution shift \citep{kakade2002approximately, schulman2015trust, achiam2017constrained}. We discuss how the guarantees from this literature apply directly to the one-step but not multi-step algorithms in more detail in Appendix \ref{sec:app_improvement}. \warn{TODO: link this in better}
On the empirical side, \citet{wang2021instabilities} demonstrates issues with distribution shift when learning from pre-trained features and provides a nice discussion of why distribution shift causes error amplification. \citet{fujimoto2018off} raises a similar issue under the name ``extrapolation error''. Regularization and constraints are meant to reduce issues stemming from distribution shift, but also reduce the potential for improvement over the behavior.

\paragraph{Empirical evidence.} Both the multi-step and iterative algorithms in our experiments rely on off-policy evaluation as a key subroutine. We examine how easy it is to evaluate the policies encountered along the learning trajectory. To control for issues of iterative error exploitation (discussed in the next subsection), we train Q estimators from scratch on a heldout evaluation dataset sampled from the behavior policy. We then evaluate these trained Q function on rollouts from 1000 datapoints sampled from the replay buffer. Results are shown in Figure~\ref{fig:mse}.

The results show a correlation betweed KL and MSE. Moreover, we see that the MSE generally increases over training. One way to mitigate this, as seen in the figure, is to use a large value of $ \alpha$. We just cannot take a very large step before running into problems with distribution shift. But, when we take such a small step, the information from the on-policy $ \widehat Q^\beta$ is about as useful as the newly estimated $ \widehat Q^\pi$. This is seen, for example, in Figure \ref{fig:learning_curves} where we get very similar performance across algorithms at high levels of regularization.

\begin{figure}[h]
    \centering
    \includegraphics[width=\textwidth]{figures/offline-rl/mse/mse_vs_kl.png}
    \caption{Results of running the iterative algorithm on halfcheetah-medium. Each checkpointed policy is evaluated by a Q function trained from scratch on heldout data. MSE refers to $\E_{s,a\sim \beta}[\hat Q^{\pi_i}(s,a) - Q^{\pi_i}(s,a)]$ and KL refers to $ \E_{s\sim \beta}[KL(\pi(\cdot|s)\| \beta(\cdot|s)]$. Left: 90 policies taken from various points in training with various hyperaparmeters and random seeds. Center: MSE learning curves. Right: KL learning curves. Error bars show min and max over 3 random seeds.}
    \label{fig:mse}
\end{figure}

\subsection{Iterative error exploitation}

The previous subsection identifies how any algorithm that uses off-policy evaluation is fundamentally limited by distribution shift, even if we were given fresh data and trained Q functions from scratch at every iteration. But, in practice, iterative algorithms repeatedly iterate between optimizing policies against estimated Q functions and re-estimating the Q functions using the \emph{same data} and using the Q function from the previous step to warm-start the re-estimation. This induces dependence between steps that causes a problem that we call iterative error exploitation.

\paragraph{Intuition about the problem.} In short, iterative error exploitation happens because $ \pi_i$ tends to choose overestimated actions in the policy improvement step, and then this overestimation propagates via dynamic programming in the policy evaluation step. To illustrate this issue more formally, consider the following: at each $ s,a$ we suffer some Bellman error $ \varepsilon_\beta^\pi(s,a)$ based on our fixed dataset collected by $ \beta$. Formally,
\begin{align}
    \widehat Q^\pi(s,a) = r(s,a) + \gamma \E_{\substack{s'|s,a \\ a'\sim \pi|s'}}[\widehat Q^\pi(s',a')] + \varepsilon_\beta^\pi(s,a).
\end{align}
Intuitively, $ \varepsilon_\beta^\pi$ will be larger at state-actions with less coverage in the dataset collected by $ \beta$. Note that $ \varepsilon_\beta^\pi $ can absorb all noise due to our finite dataset as well as function approximation error.

%So far this is all fully general, such an $ \varepsilon_\beta^\pi$ can always be defined for any estimated Q function. Now we will make the simplifying assumption.
All that is needed to cause iterative error exploitation is that the $ \epsilon_\beta^\pi$ are highly correlated across different $ \pi$, but for simplicity, we will assume that $ \varepsilon_\beta^\pi$ is \emph{the same} for all policies $ \pi$ estimated from our fixed offline dataset and instead write $ \varepsilon_\beta$. Now that the errors do not depend on the policy we can treat the errors as auxiliary rewards that obscure the true rewards and see that
\begin{align}
    \widehat Q^\pi(s,a) = Q^\pi(s,a) + \widetilde Q^\pi_\beta(s,a), \qquad \widetilde Q^\pi_\beta(s,a) := \E_{\pi| s_0, a_0 = s,a }\left[\sum_{t=0}^\infty \gamma^t \varepsilon_\beta(s_t,a_t) \right].
\end{align}
This assumption is somewhat reasonable since we expect the error to primarily depend on the data. And, when the prior Q function is used to warm-start the current one (as is generally the case in practice), the approximation errors are automatically passed between steps.

Now we can explain the problem. Recall that under our assumption the $ \varepsilon_\beta $ are fixed once we have a dataset and likely to have larger magnitude the further we go from the support of the dataset. So, with each step $ \pi_i$ is able to better maximize $ \varepsilon_\beta$, thus moving further from $ \beta$ and increasing the magnitude of $ \widetilde Q^{\pi_i}_\beta$ relative to $ Q^{\pi_i}$. Even though $ Q^{\pi_i}$ may provide better signal than $ Q^\beta$, it can easily be drowned out by $ \widetilde Q^{\pi_i}_\beta$. In contrast, $ \widetilde Q_\beta^\beta$ has small magnitude, so the one-step algorithm is robust to errors\endnote{We should note that iterative error exploitation is similar to the overestimation addressed by double Q learning \citep{van2016deep, fujimoto2018addressing}, but distinct. Since we are in the offline setting, the errors due to our finite dataset can be iteratively exploited more and more, while in the online setting considered by double Q learning, fresh data prevents this issue. We are also considering an algorithm based on policy iteration rather than value iteration.}.

\paragraph{An example.} Now we consider a simple gridworld example to illustrate iterative error exploitation. This example fits exactly into the setup outlined above since all errors are due to reward estimation so the $ \varepsilon_\beta$ is indeed constant over all $ \pi$. The gridworld we consider has one deterministic good state with reward 1 and many stochastic bad states that have rewards distributed as $ \mathcal{N}(-0.5, 1)$. We collect a dataset of 100 trajectories, each of length 100. One run of the multi-step offline regularized policy iteration algorithm is illustrated in Figure \ref{fig:gridworld}.

\begin{figure}[h]
    \centering
    %\includegraphics[width = 0.8\textwidth]{figures/offline-rl/gridworld/gridworld_onestep.png}
    \includegraphics[width=\textwidth]{figures/offline-rl/gridworld/gridworld_flat.png}
    \includegraphics[width=\textwidth]{figures/offline-rl/gridworld/gridworld_error.png}
    \caption{An illustration of multi-step offline regularized policy iteration. The leftmost panel in each row shows the true reward (top) or error $ \varepsilon_\beta$ (bottom). Then each subsequent panel plots $ \pi_i$ (with arrow size proportional to $ \pi_i(a|s)$) over either $ Q^{\pi_i}$ (top) or $ \widetilde Q^{\pi}_\beta $ (bottom), averaged over actions at each state. The one-step policy ($ \pi_1$) has the highest value. The behavior policy here is a mixture of optimal $ \pi^*$ and uniform $ u $ with coefficient 0.2 so that $\beta = 0.2 \cdot \pi^* + 0.8 \cdot u$. We set $ \alpha = 0.1$ as the regularization parameter for reverse KL regularization.}
    \label{fig:gridworld}
\end{figure}

In the example, like in the D4RL benchmark, we see that one step outperforms multiple steps of improvement. Intuitively, when there are so many noisy states, it is likely that a few of them will be overestimated. Since the data is re-used for each step, these overestimations persist and propagate across the state space due to iterative error exploitation. This property of having many bad, but poorly estimated states likely also exists in the high-dimensional control problems encountered in the benchmark where there are many ways for the robots to fall down that are not observed in the data for non-random behavior.
\begin{figure}[h]
\vspace{-0.5cm}
    \centering
    \includegraphics[width=0.6\textwidth]{figures/offline-rl/estimation/histograms.png}
    \caption{Histograms of overestimation error ($\widehat Q^{\pi_i}(s,a) - Q^{\pi_i}(s,a)$) on halfcheetah-medium with the iterative algorithm. Left: errors from the training Q function. Right: errors from an independently trained Q function.}
    \label{fig:over}
\end{figure}
Moreover, both settings have larger errors in areas where we have less data.
So even though the errors in the gridworld are caused by noise in the rewards, while errors in D4RL are caused by function approximation, we think this is a useful mental model of the problem.

\paragraph{Empirical evidence.}
In practice we cannot easily visualize the progression of errors. However, the dependence between steps still arises as overestimation of the Q values. We can track the overestimation of the Q values over training as a way to measure how much bias is being induced by optimizing against our dependent Q estimators. As a control we can also train Q estimators from scratch on independently sampled evaluation data. These independently trained Q functions do not have the same overestimation bias even though the squared error does tend to increase as the policy moves further from the behavior (as seen in Figure \ref{fig:mse}).
Explicitly, we track 1000 state, action pairs from the replay buffer over training. For each checkpointed policy we perform 3 rollouts at each state to get an estimate of the true Q value and compare this to the estimated Q value. Results are shown in Figure \ref{fig:over}.

\section{When are multiple steps useful?}\label{sec:when}

So far we have focused on why the one-step algorithm often works better than the multi-step and iterative algorithms. However, we do not want to give the impression that one-step is always better. Indeed, our own experiments in Section \ref{sec:bench} show a clear advantage for the multi-step and iterative approaches when we have randomly collected data. While we cannot offer a precise delineation of when one-step will outperform multi-step, in this section we offer some intuition as to when we can expect to see benefits from multiple steps of policy improvement.

As seen in Section~\ref{sec:why}, multi-step and iterative algorithms have problems when they propagate estimation errors. This is especially problematic in noisy and/or high dimensional environments. While the multi-step algorithms propagate this noise more widely than the one-step algorithm, they also propagate the signal. So, when we have sufficient coverage to reduce the magnitude of the noise, this increased propagation of signal can be beneficial. The D4RL experiments suggest that we are usually on the side of the tradeoff where the errors are large enough to make one-step preferable.

\begin{figure}[h]
\vspace{-0.3cm}
    \centering
    \includegraphics[width=0.4\textwidth]{figures/offline-rl/learning curves/r-m_mixture.png}
    \caption{Performance of all three algorithms with reverse KL regularization across mixtures between halfcheetah-random and halfcheetah-medium. Error bars indicate min and max over 3 seeds.}
    \label{fig:interp}
\end{figure}

In Appendix~\ref{sec:app_grid} we illustrate a simple gridworld example where a slight modification of the behavior policy from Figure~\ref{fig:gridworld} makes multi-step dramatically outperform one-step. This modified behavior policy (1) has better coverage of the noisy states (which reduces error, helping multi-step), and (2) does a worse job propagating the reward from the good state (hurting one-step).

We can also test empirically how the behavior policy effects the tradeoff between error and signal propagation. To do this we construct a simple experiment where we mix data from the random behavior policy with data from the medium behavior policy. Explicitly we construct a dataset $ D $ out of the datasets $ D_r$ for random and $ D_m$ for medium such that each trajectory in $ D $ comes from the medium dataset with probability $ p_m$. So for $ p_m = 0$ we have the random dataset and $ p_m = 1 $ we have the medium dataset, and in between we have various mixtures. Results are shown in Figure~\ref{fig:interp}. It takes surprisingly little data from the medium policy for one-step to outperform the iterative algorithm.

\section{Discussion, limitations, and future work}

This paper presents the surprising effectiveness of a simple one-step baseline for offline RL. We examine the failure modes of iterative algorithms and the conditions where we might expect them to outperform the simple one-step baseline. This provides guidance to a practitioner that the simple one-step baseline is a good place to start when approaching an offline RL problem.

But, we leave many questions unanswered.
One main limitation is that we lack a clear theoretical characterization of which environments and behaviors can guarantee that one-step outperforms multi-step or visa versa. Such results will likely require strong assumptions, but could provide useful insight. We don't expect this to be easy as it requires understanding policy iteration which has been notoriously difficult to analyze, often converging much faster than the theory would suggest \citep{sutton2018reinforcement, Agarwal2019ReinforcementLT}.
Another limitation is that while only using one step is perhaps the simplest way to avoid the problems of off-policy evaluation, there are possibly other more elaborate algorithmic solutions that we did not consider here. However, our strong empirical results suggest that the one-step algorithm is at least a strong baseline.

\printendnotes


\section{Formal Axiomatic Specification in Herd}
\label{sec:herd}

The tool \textsf{herd} takes a memory model and a litmus test as input and simulates the execution of the test on top of the memory model. Memory models are written in the domain specific language \textsc{Cat}. This section provides two \textsc{Cat} memory model of RVWMO. The first model, Figure~\ref{fig:herd2}, follows the \emph{global memory order}, Chapter~\ref{ch:memorymodel}, definition of~RVWMO, as much as is possible for a \textsc{Cat} model. The second model, Figure~\ref{fig:herd3}, is an equivalent, more efficient, partial order based RVWMO model.

The simulator~\textsf{herd} is part of the \textsf{diy} tool suite --- see \url{http://diy.inria.fr} for software and documentation. The models and more are available online at~\url{http://diy.inria.fr/cats7/riscv/}.

\begin{figure}[h!]
  {
  \tt\bfseries\centering\footnotesize
  \begin{lstlisting}
(*************)
(* Utilities *)
(*************)

(* All fence relations *)
let fence.r.r = [R];fencerel(Fence.r.r);[R]
let fence.r.w = [R];fencerel(Fence.r.w);[W]
let fence.r.rw = [R];fencerel(Fence.r.rw);[M]
let fence.w.r = [W];fencerel(Fence.w.r);[R]
let fence.w.w = [W];fencerel(Fence.w.w);[W]
let fence.w.rw = [W];fencerel(Fence.w.rw);[M]
let fence.rw.r = [M];fencerel(Fence.rw.r);[R]
let fence.rw.w = [M];fencerel(Fence.rw.w);[W]
let fence.rw.rw = [M];fencerel(Fence.rw.rw);[M]
let fence.tso =
  let f = fencerel(Fence.tso) in
  ([W];f;[W]) | ([R];f;[M])

let fence =
  fence.r.r | fence.r.w | fence.r.rw |
  fence.w.r | fence.w.w | fence.w.rw |
  fence.rw.r | fence.rw.w | fence.rw.rw |
  fence.tso

(* Same address, no W to the same address in-between *)
let po-loc-no-w = po-loc \ (po-loc?;[W];po-loc)
(* Read same write *)
let rsw = rf^-1;rf
(* Acquire, or stronger  *)
let AQ = Acq|AcqRel
(* Release or stronger *)
and RL = RelAcqRel
(* All RCsc *)
let RCsc = Acq|Rel|AcqRel
(* Amo events are both R and W, relation rmw relates paired lr/sc *)
let AMO = R & W
let StCond = range(rmw)

(*************)
(* ppo rules *)
(*************)

(* Overlapping-Address Orderings *)
let r1 = [M];po-loc;[W]
and r2 = ([R];po-loc-no-w;[R]) \ rsw
and r3 = [AMO|StCond];rfi;[R]
(* Explicit Synchronization *)
and r4 = fence
and r5 = [AQ];po;[M]
and r6 = [M];po;[RL]
and r7 = [RCsc];po;[RCsc]
and r8 = rmw
(* Syntactic Dependencies *)
and r9 = [M];addr;[M]
and r10 = [M];data;[W]
and r11 = [M];ctrl;[W]
(* Pipeline Dependencies *)
and r12 = [R];(addr|data);[W];rfi;[R]
and r13 = [R];addr;[M];po;[W]

let ppo = r1 | r2 | r3 | r4 | r5 | r6 | r7 | r8 | r9 | r10 | r11 | r12 | r13
\end{lstlisting}
  }
  \caption{{\tt riscv-defs.cat}, a herd definition of preserved program order (1/3)}
  \label{fig:herd1}
\end{figure}

\begin{figure}[ht!]
  {
  \tt\bfseries\centering\footnotesize
  \begin{lstlisting}
Total

(* Notice that herd has defined its own rf relation *)

(* Define ppo *)
include "riscv-defs.cat"

(********************************)
(* Generate global memory order *)
(********************************)

let gmo0 = (* precursor: ie build gmo as an total order that include gmo0 *)
  loc & (W\FW) * FW | # Final write after any write to the same location
  ppo |               # ppo compatible
  rfe                 # includes herd external rf (optimisation)

(* Walk over all linear extensions of gmo0 *)
with  gmo from linearisations(M\IW,gmo0)

(* Add initial writes upfront -- convenient for computing rfGMO *)
let gmo = gmo | loc & IW * (M\IW)

(**********)
(* Axioms *)
(**********)

(* Compute rf according to the load value axiom, aka rfGMO *)
let WR = loc & ([W];(gmo|po);[R])
let rfGMO = WR \ (loc&([W];gmo);WR)

(* Check equality of herd rf and of rfGMO *)
empty (rf\rfGMO)|(rfGMO\rf) as RfCons

(* Atomicity axiom *)
let infloc = (gmo & loc)^-1
let inflocext = infloc & ext
let winside  = (infloc;rmw;inflocext) & (infloc;rf;rmw;inflocext) & [W]
empty winside as Atomic
\end{lstlisting}
  }
  \caption{{\tt riscv.cat}, a herd version of the RVWMO memory model (2/3)}
  \label{fig:herd2}
\end{figure}

\begin{figure}[h!]
  {
  \tt\bfseries\centering\footnotesize
  \begin{lstlisting}
Partial

(***************)
(* Definitions *)
(***************)

(* Define ppo *)
include "riscv-defs.cat"

(* Compute coherence relation *)
include "cos-opt.cat"

(**********)
(* Axioms *)
(**********)

(* Sc per location *)
acyclic co|rf|fr|po-loc as Coherence

(* Main model axiom *)
acyclic co|rfe|fr|ppo as Model

(* Atomicity axiom *)
empty rmw & (fre;coe) as Atomic
\end{lstlisting}
  }
  \caption{{\tt riscv.cat}, an alternative herd presentation of the RVWMO memory model (3/3)}
  \label{fig:herd3}
\end{figure}


\subsection{GraphNets Library}
\label{sec:gnlib}

Our GraphNets library is an implementation of the GN framework introduced by \cite{deepmind:graphnets}. It can be used to implement and train GNs in combination with the deep learning platform PyTorch. Its primary use-case is research in the field of GNs. We have developed it within the scope of this work. The framework is, however, not influenced by the page rank aspects. It is generic and could, as-is, be used for any problem that requires GNs.

The researchers at DeepMind have implemented their own framework and have open-sourced it\footnote{GitHub repository by DeepMind: \url{https://github.com/deepmind/graph_nets}}. Opposed to our solution it is based on the deep learning framework TensorFlow. We chose not to use it because the development of a custom solution gave us more flexibility, a learning opportunity, and the free framework choice (PyTorch vs. TensorFlow).

In the remainder of this section we explain the components of our library and link it to the mathematical formulation of Section~\ref{sec:graphnetworks}. Furthermore, we mention specific details of the implementation which seem relevant. We close by naming potential future features and improvements.

\subsubsection{Graph Data Structures}

\begin{figure}\centering
    \includegraphics[scale=0.65]{resources/graphnets-datastructs}
    \caption[Class diagram of the GraphNets library data structure classes]{Class diagram of the GraphNets library data structure classes, representing a directed, attributed multigraph.}\label{fig:classdiagramgndatastructs}
\end{figure}

The graph data structures are four classes, namely \texttt{Graph}, \texttt{Edge}, \texttt{Node}, and \texttt{Attribute}. They are the implementation of a directed, attributed multigraph\footnote{A multigraph is a graph which may have multiple edges (also called parallel edges). For example, there may be two edges pointing from a given node $v_1$ another node $v_2$.} with a global attribute. Here we follow the mathematical specification of \cite{deepmind:graphnets} closely. Figure~\ref{fig:classdiagramgndatastructs} depicts the classes with attributes and methods.

The \texttt{Graph} class stores references to all contained nodes and edges. They are stored in a set and in a list for performance reasons: The set allows to determine in $\mathcal{O}(1)$ time whether a node/edge is contained in the graph. The list in turn allows for $\mathcal{O}(n)$ comparison of two isomorphic graphs.

The method \texttt{\_check\_integrity()} validates, whether the edges are exclusively connecting nodes contained in the \texttt{nodes} set. It also ensure that the elements contained in \texttt{nodes} and \texttt{ordered\_nodes} are equivalent (and for edges respectively). The \texttt{Graph}'s \texttt{\_\_del\_\_} method is overwritten because it traverses the graph and searches for PyTorch tensors which are placed on the GPU. Upon deletion it moves them to the CPU to ensure their garbage collection after destruction of the \texttt{Graph} class instance.

An \texttt{Edge} requires two nodes (sender and receiver) to be passed to its constructor. The edge adds itself to the \texttt{receiving\_edges} set of the receiver and \texttt{sending\_edges} set of the sender. \texttt{Edge} objects must therefore be created after \texttt{Node}s.

All three, \texttt{Graph}, \texttt{Node}, and \texttt{Edge} have in common that they have an attribute \texttt{attr} of type \texttt{Attr}. An attribute can hold any value in its \texttt{val} attribute. In the page rank example, the \texttt{val} of an attribute may for instance be a tuple of screenshots.

\subsubsection{Aggregation and Update Functions}

Aggregation functions serve the purpose of converting a set of attributes into a single attribute. Update functions exist for nodes, edges, and the global state and map their attributes to new, updated attributes. In the GN framework aggregation functions are denoted by $\rho$ and update functions by $\phi$. All aggregation functions and their relationships can be seen in the class diagram in Figure~\ref{fig:classdiagramgnfunctionsaggr}.

\begin{figure}\centering
    \includegraphics[scale=0.65]{resources/graphnets-functions-aggr}
    \caption[Class diagram of the GraphNets library aggregation functions]{Class diagram of the GraphNets library aggregation functions.}\label{fig:classdiagramgnfunctionsaggr}
\end{figure}

The \texttt{Aggregation} class is abstract and may be implemented by the user of our library. An aggregation can be used for nodes and edges alike. Several common, default implementations are provided, namely

\begin{itemize}
    \item \texttt{SumAggregation}: Sums up the vectors (PyTorch tensor objects) in the set of attributes (parameter \texttt{attrs}) element-wise and returns the resulting vector wrapped in a new \texttt{Attr} object.
    \item \texttt{AvgAggregation}: Works like \texttt{SumAggregation} with subsequent division by the length of \texttt{attrs}.
    \item \texttt{ScalarSumAggregation}: Operates on a list of attributes which hold scalar values, e.g. Python \texttt{float} or \texttt{int} values. It sums them up and returns the resulting value wrapped in a new \texttt{Attr} object.
    \item \texttt{ConstantAggregation}: Ignores the list of attributes entirely and returns a constant value which can be passed to the constructor. This aggregation is useful in a cases where the set of nodes or edges does not contain any useful attributes yet. For instance in the encoder of a GN.
\end{itemize}

The update functions are specific to edges, nodes, and the global state. Therefore, three abstract classes exist, namely \texttt{EdgeUpdate}, \texttt{NodeUpdate}, and \texttt{GlobalStateUpdate}, corresponding to $\phi^e$, $\phi^v$, and $\phi^u$, respectively. The classes are abstract because the update may be chosen by the user of the library. The relationships can be seen in the class diagram in Figure~\ref{fig:classdiagramgnfunctionsupdate}.

\begin{figure}\centering
    \includegraphics[scale=0.65]{resources/graphnets-functions-update}
    \caption[Class diagram of the GraphNets library update functions]{Class diagram of the GraphNets library update functions.}\label{fig:classdiagramgnfunctionsupdate}
\end{figure}

For all three types of update functions, the library comes with two default implementations: First, the independent update (\texttt{IndependentEdgeUpdate}, \texttt{IndependentNodeUpdate}, and \texttt{IndependentGlobalStateUpdate}). An independent update function takes a parameter called \texttt{mapping\_fn}. The result of a forward pass is the application of that mapping function $f_\text{map}$ to the previous edge (or node or global state) attribute. Mathematically that is \begin{align}
\phi^e\left(\bm{e}_k,\bm{v}_{r_k},\bm{v}_{s_k},\bm{u}\right)=&f_\text{map}\left(\bm{e}_k\right)\\
\phi^v\left(\bm{\overline{e}}'_i,\bm{v}_i,\bm{u}\right)=&f_\text{map}\left(\bm{v}_i\right)\\
\phi^u\left(\bm{\overline{e}}',\bm{\overline{v}},\bm{u}\right)=&f_\text{map}\left(\bm{u}\right)\,.
\end{align}Verbally described, everything but the previous attribute value is discarded and the new attribute value is computed with the mapping function.

The second default implementation are the identity update functions. They leave the attribute values unaltered.

\subsubsection{GN Block}

\begin{figure}\centering
    \includegraphics[scale=0.65]{resources/graphnets-block}
    \caption[Class diagram of the GraphNets library GNBlock class]{Class diagram of the GraphNets library GNBlock class.}\label{fig:classdiagramgnblock}
\end{figure}

The GN block is the core component of the library. It inherits from the PyTorch class \texttt{torch.nn.Module} (see class diagram in Figure~\ref{fig:classdiagramgnblock}). Its input is a graph and it outputs a transformed version of that graph, by passing it through the \texttt{forward} method. The implementation of \texttt{forward} is exactly matching the flow shown in Figure~\ref{fig:fullgraphblock}.

An instantiation of a \texttt{GNBlock} object requires update and aggregation functions. The correspondence between functions and attributes is shown in Table~\ref{tab:gnblockattrs}.

\begin{table}
    \centering
    \begin{tabular}{ l l l }
        \textbf{Function} & \textbf{Attribute} & \textbf{Default}\\
        \hline
        $\phi^e$ & \texttt{phi\_e} & \texttt{IdentityEdgeUpdate} \\
        $\phi^v$ & \texttt{phi\_v} & \texttt{IdentityNodeUpdate} \\
        $\phi^u$ & \texttt{phi\_u} & \texttt{IdentityGlobalStateUpdate} \\
        $\rho^{e\rightarrow v}$ & \texttt{rho\_ev} & \texttt{ConstantAggregation} \\
        $\rho^{e\rightarrow u}$ & \texttt{rho\_eu} & \texttt{ConstantAggregation} \\
        $\rho^{v\rightarrow u}$ & \texttt{rho\_vu} & \texttt{ConstantAggregation} \\
        \hline
    \end{tabular}
    \caption[Mapping from \texttt{GNBlock} attributes to GN functions]{Mapping from \texttt{GNBlock} attributes (see class diagram in Figure~\ref{fig:classdiagramgnblock}) to GN functions (described in Section~\ref{sec:graphnetworks}). The default value is chosen if the attribute is left unspecified.}
    \label{tab:gnblockattrs}
\end{table}

The GN block simplifies the creation of a GN significantly. After specifying aggregation and update functions, a graph can be passed to a \texttt{GNBlock} instance and is processed by it. Listing \ref{lst:gnblock} shows this flow.

\begin{lstlisting}[
    label={lst:gnblock},
    language=Python,
    caption={Sample usage of the \texttt{GNBlock} class},
    captionpos=b
]
core_block = GNBlock(
    phi_e=CoreEdgeUpdate(drop_p),
    phi_v=CoreNodeUpdate(drop_p),
    phi_u=CoreGlobalStateUpdate(drop_p),
    rho_ev=AvgAggregation(),
    rho_vu=AvgAggregation(),
    rho_eu=AvgAggregation())

g = Graph()
# add attributed nodes and edges to g ...
g_processed = core_block(g)
\end{lstlisting}

\subsubsection{Future Features and Improvements}
\label{sec:gnlibfuturefeatures}

The current implementation of the library is processing each graph individually. Batches can not be processed in parallel, which is due to the nature of graphs: A single batch may contain many graphs of different size and shape. A performance boost could be gained by providing GPU-optimized implementations (so called CUDA kernels) for the some common update and aggregation functions that handle batches.

A useful feature would be a greater toolbox of implemented update and aggregation functions. Also, composition of GN blocks into actual networks could be simplified with predefined modules.

In our current implementation the critical features of the components are being unit tested. There is also an end-to-end test which trains a GN to learn the identity function, thereby ensuring PyTorch's automatic differentiation works through GN blocks. Before performing a greater refactoring (e.g. the implementation of CUDA kernels), a higher test coverage would be desirable.


\title{\bf Feedback Effects in Galaxy Formation}

\section{Basics}

The discussion here follows the description found in the textbook of
Cimatti, Fraternali, \& Nipoti.

\subsection{General Concepts in Feedback}

The concept of star formation feedback in galaxy formation is
generally motivated by the need to explain low global star formation
efficiency within high and low mass dark matter halos. For that
reason, generally feedback refers to {\it negative feedback} effects;
that is, processes that inhibit star formation. These effects are
divided into {\it stellar feedback}, which is literally feedback in
the sense that star formation leads to stellar processes that inhibit
further star formation, and {\it AGN feedback}, which refers to AGN
processes that inhibit star formation. In both cases, some combination
of radiative energy and kinetic energy flux couples to gas in the
galaxy to heat it, add turbulence to it, or drive it in large scale
winds to the outskirts of or entirely out of the galaxy.

In the case of winds, the distinction is made between {\it
  momentum-driven} and {\it energy-driven} winds. Physically, these
cases are distinguished by whether the cooling time of the affected
gas is short relative to the dynamical time. If the cooling time is
short, then the energy quickly radiates away, but the momentum
imparted is not radiated away and the wind is momentum driven. If the
cooling time is not short, the thermal energy created by the feedback
helps to drive the wind. The distinction is important to the
dependence on galaxy potential depth of the relationship between a
star formation rate ($\dot M_\ast$) or black hole accretion rate
($\dot M_{\rm BH}$) to a mass outflow rate.

In the case of an energy-driven wind, the energy converted to kinetic
energy per unit time is proportional to the star formation or
black hole accretion rate:
\begin{equation}
\dot K \propto \dot M.
\end{equation}
Because in a time-invariant wind $\dot K = \dot M_{\rm w} v_{\rm w}^2$,
the rate of mass driven to some velocity $v_{\rm w}$ is:
\begin{equation}
\dot M_{\rm w} \propto \dot K v_{\rm w}^{-2} \propto \dot M v_{\rm
  w}^{-2}.
\end{equation}
The escape speed is proportional to the velocity dispersion of the
halo $\sigma$, so if we want to know how much mass is flowing out of
the galaxy (i.e. reaches $v_{\rm w} \sim \sigma$) it will scale as:
\begin{equation}
\dot M_{\rm out} \propto \dot M \sigma^{-2}.
\end{equation}

In the case of a momentum-driven wind, the momentum imparted per unit
time (due to mechanical energy or radiation pressure) will be
proportion to the star formation or black hole accretion rate:
\begin{equation}
\dot p \propto \dot M.
\end{equation}
Because in a time-invariant wind $\dot p = \dot M_{\rm w} v_{\rm w}$,
the rate of mass driven to some velocity $v_{\rm w}$ is:
\begin{equation}
\dot M_{\rm w} \propto \dot p v_{\rm w}^{-1} \propto \dot M v_{\rm
  w}^{-1}.
\end{equation}
The escape speed is proportional to the velocity dispersion of the
halo $\sigma$, so if we want to know how much mass is flowing out of
the galaxy (i.e. reaches $v_{\rm w} \sim \sigma$) it will scale as:
\begin{equation}
\dot M_{\rm out} \propto \dot M \sigma^{-1}.
\end{equation}

In the context of galaxy formation, these different scalings mean that
these two types of wind shape galaxy stellar mass function
differently.

\subsection{Stellar Feedback}

Stellar feedback comes in the form of supernovae and stellar winds,
primarily from massive stars. Individual supernovae generally do not
impart much of their energy to the interstellar medium as kinetic
energy, only a few percent. Stellar feedback generally is thought to
operate effectively through the combination of many massive stars or
supernovae creating {\it superbubbles} from which emanate winds. These
winds are usually taken to be energy driven

For either a single massive star, a group of massive stars, or a group
of supernovae, we can treat the feedback as a steady, very supersonic
wind acting for some period of time. We can calculate what fraction of
that power is converted into kinetic energy of the interstellar
medium. This calculation assumes some constant input of supersonic
wind energy under spherical symmetry. The wind shocks the surrounding
interstellar medium and creates a hot, expanding bubble of gas at
pressure $P_b$, with a shocked shell of interstellar medium gas
surrounding it. The bubble's thermal energy increases due to
mechanical energy being converted to thermal energy, and decreases due
to its expansion:
\begin{equation}
\label{eq:energy-bubble}
\frac{{\rm d}U_b}{{\rm d}t} \approx L_{\rm w} - P_b \frac{{\rm
    d}V}{{\rm d}t}.
\end{equation}
For a nonrelativistic ideal gas, we can write $U_b = (3/2) P_b V$.

The shell's momentum is driven by the pressure of the hot bubble
behind it, and we can write its equation of motion:
\begin{equation}
\label{eq:shell}
\frac{{\rm d}}{{\rm d}t} \left(M_{\rm sh} v_{\rm sh}\right) = 4\pi
r_{\rm sh}^2 P_b
\end{equation}
We can solve for $P_b$, and assuming the interstellar medium is of
constant density $\rho_0$ and is all swept up in the shell, we find:
\begin{equation}
\label{eq:shell-pb}
P_b = \rho_0\left(\frac{1}{3} r_{\rm sh} {\ddot r}_{\rm sh} + {\dot
r}_{\rm sh}^2\right)
\end{equation}
Solving for $r_{\rm sh}$ assuming a power law solution in time we
find:
\begin{equation}
\label{eq:shell-evolution}
r_{\rm sh}
= \left(\frac{125}{154\pi}\right)^{1/5} \left(\frac{L_w}{\rho_0}\right)^{1/5}
t^{3/5}
\end{equation}
We can calculate the kinetic energy of the shell's expansion:
\begin{equation}
\label{eq:shell-kinetic}
K_{\rm sh} = \frac{1}{2} M_{\rm sh} v_{\rm sh}^2 = \frac{15}{77}L_w
t \approx 0.2 L_w t
\end{equation}
This calculation shows that the conversion of wind energy to kinetic
energy is rather efficient for a steady, supersonic wind in a uniform
medium.

Numerically, if we consider massive O-stars, they lose mass at about
$10^{-6}$ $M_\odot$ yr$^{-1}$, in winds that reach thousands of km
s$^{-1}$. Such a wind will produce a mechanical power of:
\begin{equation}
L_{\rm w} = \frac{1}{2} \dot M_{\rm w} v_{\rm w}^2 \sim 10^{36}
\left(\frac{\dot M_{\rm w}}{10^{-6} M_\odot {\rm ~yr}^{-1}}\right)
\left(\frac{v_{\rm w}} {2\times 10^3 {\rm km~s}^{-1}} \right) {\rm ~erg~s}^{-1}.
\end{equation}
Using the equations above, this results over the lifetime of a single
O star (of order million years) a bubble of size $\sim 100$ pc.

For a given cluster of massive stars, the exercises show that a
similar amount of power is produced by supernovae, but over a longer
time (up to the lifetime of 8 $M_\odot$ stars, or about 30 million
years), which means that the feedback is dominated by the combined
effect of these supernovae producing superbubbles.

Per unit star formation the rate of core-collapse supernovae is:
\begin{equation}
R_{\rm SN} \approx 10^{-2} \left(\frac{\rm SFR}{M_\odot {\rm
~yr}^{-1}}\right) {\rm ~yr}^{-1}
\end{equation}
If we relate the mass outflow rate to the kinetic energy input rate by
$\dot K \sim {\dot M}_{\rm out} v_{\rm esc}^2 / 2$, and use an energy for
each supernova of $10^{51}$ erg, then:
\begin{equation}
\dot M_{\rm out} \sim \frac{\eta}{0.1} \left(\frac{{\rm SFR}}{M_\odot
{\rm ~yr}^{-1}}\right) \left(\frac{v_{\rm esc}}{300 {\rm
~km~s}^{-1}}\right)^{-2} M_\odot {\rm ~yr}^{-1}
\end{equation}
where $\eta$ is the fraction conversion of wind power into kinetic
energy (calculated as 0.2 above).

A momentum-driven model is also possible if the medium around the star
formation is optically thick. This situation would apply mostly to
starbursts. An order-of-magnitude estimate of the momentum-driven
winds yields:
\begin{equation}
\dot M_{\rm out} \sim \left(\frac{{\rm SFR}}{M_\odot
{\rm ~yr}^{-1}}\right) \left(\frac{v_{\rm esc}}{300 {\rm
~km~s}^{-1}}\right)^{-1} M_\odot {\rm ~yr}^{-1}
\end{equation}
which is similar to the energy-driven winds at moderate star formation
rates but much more effective for star bursts
(see \citealt{murray05b}).

For some of the observed wind velocities in star forming galaxies, the
high observed velocities ($500$--$1000$ km s$^{-1}$) and relatively
high inferred mass-loading factors suggest a momentum driven wind.

\subsection{AGN Feedback}

AGN feedback is driven by accretion onto a black hole. The luminosity
of an accreting black hole is limited (in the spherical case) to the
Eddington luminosity $L_{\rm Edd}$. This luminosity can be related to
an Eddington mass accretion rate:
\begin{equation}
{\dot M}_{\rm Edd} = \frac{L_{\rm Edd}} {\epsilon_{\rm rad} c^2},
\end{equation}
where $\epsilon_{\rm rad}$ is the radiative efficiency, which in
theory can be as high as 0.29 for a highly spinning black hole with
very radiatively efficient accretion, but is typically thought to be
$\sim 0.1$ even for AGN in radiatively efficient phases.

We can estimate whether AGN have sufficient energy to substantially
affect the galaxy by comparing the total energy emitted in the
formation of the black hole to the potential energy of the bulge:
\begin{equation}
\frac{E_{\rm BH}}{E_{\rm bulge}} \sim \frac{\epsilon_{\rm rad} M_{\rm BH}
c^2}{M_{\rm bulge} \sigma^2} \sim 100 \frac{\epsilon_{\rm
rad}}{0.1} \left(\frac{\sigma}{300 {\rm
~km~s}^{-1}}\right)^{-2} \frac{M_{\rm BH}/M_{\rm bulge}}{0.001}
\end{equation}
For a massive galaxy, then, there is amply energy available in
principle to affect the evolution of the bulge region.

It is believed that AGN feedback occurs in both radiatively efficient
and radiatively inefficient AGN phases. The {\it radiative} or {\it
quasar} mode feedback is associated with Eddington ratios of $L/L_{\rm
Edd} > 10^{-2}$. Feedback occurs through photoionization heating and
radiation pressure. The dominant effect on the galaxy as a whole is
thought to be through the radiation pressure, which can create a
galactic wind. The radiation pressure is proportional to the
luminosity and the momentum imparted per unit time to the wind is
dimensionally:
\begin{equation}
p_w \propto {\dot M}_{\rm BH}  v_w
\end{equation}
with the constant of proportionality estimated theoretically at about
0.5. This effect drives a momentum-driven wind. In theoretical models
the mechanical efficiency is:
\begin{equation}
\epsilon_w = \frac{L_w}{{\dot M}_{\rm BH} c^2} \sim 10^{-3}.
\end{equation}

The {\it jet} or {\it radio} mode feedback occurs primarily in the
radiatively inefficient regime (Eddington ratios $<10^{-2}$). The jets
of the AGN interact with the hot gas in the galaxy or surrounding
cluster mechanically. In a few systems, the effects of these
interactions can be seen as X-ray bubbles or cavities. The gas outside
the cavity is at the virial temperature and emits in the X-rays. The
gas inside the cavity is moving relativistically and its primary
emission is radio synchtron. The surrounding pressure can be estimated
from the X-ray data, and the total energy in the bubble is:
\begin{equation}
E = U + W = \frac{1}{\gamma - 1} PV + PV = \frac{\gamma}{\gamma - 1} PV,
\end{equation}
where $\gamma = 4/3$ for a relativistic gas. The kinetic power can be
estimated as $E/t$ where $t$ is some estimate of the age of the
bubble, for example based on its size and the sound speed
(\citealt{fabian12a}). This power is comparable in magnitude and tends
to scale with the X-ray luminosity, indicating that it can balance
cooling. In fact, the kinetic power associated with these bubbles is
thought to couple very efficiently to the interstellar or
intergalactic gas.

One of the reasons that AGN feedback began to be considered seriously
is that it is possible to construct a scaling relation based on the
Eddington limit that looks like $M_{\rm BH}$-$\sigma$ relation of
galaxies. In particular, if you set the total outward force due to
radiation pressure at the Eddington limit, $L_{\rm Edd} / c$, equal to
the total inward force due to gravity on some amount of gas $M_{\rm
gas} = f M_{\rm gal}$:
\begin{equation}
\frac{L_{\rm Edd}}{c} = \frac{4\pi G_{\rm BH} m_p}{\sigma_T}
= \frac{GM_{\rm gal}M_{\rm gas}}{r} = \frac{f
G}{r^2} \left(\frac{2\sigma^2 r}{G}\right)^2 = \frac{4f\sigma^2}{G}
\end{equation}
which implies that $M_{\rm BH} \propto \sigma^4$, relatively similar
to the relation observed. Thus, if the fraction of gas loss $f$ is
constant across galaxy mass, and the typical Eddington ratios are
constant across galaxy mass (or at least that the momentum imparted by
feedback processes scales linear with black hole mass), then the
$M_{\rm BH}$-$\sigma$ relation will follow.

The argument can be made more specific by supposing that the growth of
the surrounding galaxy is in fact Eddington limited by the action of
the quasar. This requires that the quasar radiation couple with the
interstellar gas much better than the Thomson cross-section with the
electrons. In fact, for galaxies with Milky Way-like dust to gas
ratios, the cross-section of dust per proton is about $\sigma_d \sim
1000 \sigma_T$. Thus for dusty neutral gas, the Eddington limit is
about 1000 times larger. Over the course of the lifetime of the galaxy
and black hole, if both grow at their respective Eddington limits (or
some equal fraction thereof) they will end up with $M_{\rm BH}/M_{\rm
gal} \sim \sigma_T / \sigma_d \sim 10^{-3}$ (\citealt{fabian12a}).
This same argument also would lead to $M_{\rm gal} \propto \sigma^4$,
as seen for elliptical galaxies in the Faber-Jackson
relation.

\section{Commentary}

Much of the literature on feedback is motivated by the desire to get
the theoretical predictions of galaxy formation simulations to agree
with observations. Although some of the arguments are suggestive
regarding AGN feedback and the $M_{\rm BH}$-$\sigma$ relation, it is
still fairly speculative that these processes are related and not a
numerical and dimensional coincidence.

\section{Key References}

\begin{itemize}
  \item Cimatti
  \item
    \href{https://ui.adsabs.harvard.edu/abs/2015ARA%26A..53...51S/abstract}
    {{\it Physical Models of Galaxy Formation in a Cosmological
        Framework} (\citealt{somerville15a})}
\end{itemize}

\section{Order-of-magnitude Exercises}

\begin{enumerate}
\item Estimate proportionality between the supernova rate and the star
  formation rate.
\item For an O/B association with 100 O and B stars (down to 8
$M_\odot$), what is the typical power output of supernovae?
\item For a single massive with a wind like that described above, in a
typical region of the interstellar medium, after about 100,000 years
what sized region will it have created around it?
\item Estimate of energy driven stellar feedback wind outflow rates
\item Estimate of momentum driven stellar feedback wind outflow rates
\end{enumerate}

\section{Analytic Exercises}

\begin{enumerate}
\item Sedov
\item Derive Equation \ref{eq:shell-pb} from \ref{eq:shell}. Using
this relationship along with Equation \ref{eq:energy-bubble} show that
Equation \ref{eq:shell-evolution} holds. Finally, show that the
kinetic energy of the shell obeys Equation \ref{eq:shell-kinetic}.
\item Show that about 20\% of the power input into an expanding bubble
this relationship along with Equation \ref{eq:energy-bubble} show that
Equation \ref{eq:shell-evolution} holds.
\item Energy from SNe (Cimatti S8.7.1)
\end{enumerate}

%\section{Numerics and Data Exercises}
%
%\begin{enumerate}
%\item Dealing with a numerical simulation result
%\end{enumerate}

\bibliographystyle{apj}
\bibliography{exex}


\subsubsection{\stid{3.13} CLOVER Sub-project SLATE}\label{subsubsect:slate}

\paragraph{Overview}

The Software for Linear Algebra Targeting Exascale (SLATE)
provides fundamental dense linear algebra capabilities
to DOE and the HPC community at large.
To this end, SLATE provides
parallel Basic Linear Algebra Subprograms (BLAS), norms,
linear systems solvers, least square solvers,
singular value and eigenvalue solvers.

The ultimate objective of SLATE is to replace the
venerable Scalable Linear Algebra PACKage (ScaLAPACK) library,
which has become the industry standard for dense linear algebra operations
in distributed-memory environments.
After two decades of operation,
ScaLAPACK is past the end of its life cycle and overdue for a replacement,
as it can hardly be retrofitted to support GPUs,
which are an integral part of today's HPC hardware infrastructure.

Primarily, SLATE aims to extract the full performance potential and maximum
scalability from modern HPC machines with large numbers nodes,
large numbers of cores per node, and multiple GPUs per node.
For typical dense linear algebra workloads, this means getting close
to the theoretical roofline peak performance and scaling to the full size of
the machine.
This is accomplished in a portable manner by relying on standards
like MPI and OpenMP.

SLATE functionalities will first be delivered to the ECP applications
that most urgently require SLATE capabilities
(NWChem, GAMESS, EXAALT, QMCPACK, CANDLE, etc.)
and to other software libraries
that rely on underlying dense linear algebra services
(STRUMPACK, SuperLU, etc.).
Figure~\ref{fig:slate-architecture} shows the role of SLATE
in the ECP software stack.

While the initial objective of SLATE is to serve as a successful,
drop-in replacement for ScaLAPACK with support for GPU accelerators,
the ultimate goal of SLATE is to deliver dense linear algebra capabilities
beyond the capabilities of ScaLAPACK.
This includes new features such as communication-avoiding
algorithms and randomization algorithms, as well as the potential to
support variable size tiles and block low-rank compressed tiles.

\begin{figure}[htb]
    \centering
    \includegraphics[width=0.75\textwidth]{projects/2.3.3-MathLibs/2.3.3.13-CLOVER/SLATE-architecture.jpg}
    \caption{\label{fig:slate-architecture}
    SLATE in the ECP software stack.}
\end{figure}

\paragraph{Key  Challenges}

\begin{enumerate}

\item
\textbf{Designing from the ground up:}
The SLATE project's primary challenge stems from the need to design the package
from the ground up, as no existing software package offers
a viable path forward for efficient support of GPUs
in a distributed-memory environment.

\item
\textbf{Facing harsh hardware realities:}
SLATE is being developed in a difficult hardware environment, where virtually
all the processing power is on the GPU side.
Achieving efficiency requires aggressive offload to GPU accelerators
and careful optimization of multiple bottlenecks, including
interconnect technology lagging behind the computing
capabilities of the GPUs.

\item
\textbf{Facing harsh software realities:}
SLATE is being developed using cutting-edge software technologies,
and relies on modern C++ features and recent extensions
to the OpenMP standard, many of which are not fully supported by compilers
and their runtime environments.
In terms of GPU acceleration, standardized solutions are still in flux.
Also, while complete parallel programming frameworks exist, at this stage
they have to be considered research prototypes.

\end{enumerate}

\paragraph{Solution Strategy}

\begin{enumerate}

\item
\textbf{Evolving design:}
Due to the inherent challenges of designing a software package
from the ground up, the SLATE project started
with a careful analysis of the existing and emerging
implementation technologies~\cite{abdelfattah2017roadmap},
and followed with a phase
of laying out the initial design~\cite{kurzak2017designing}.
Since then, the team rolls out new computational routines every quarter.
While we continue to refactor as needed to achieve high performance, the basic
design has solidified and been published~\cite{gates2019slate-design}.

\item
\textbf{Focus on GPUs:}
Efficient GPU acceleration is the primary focus of performance
engineering efforts in SLATE.
Where applicable, highly optimized vendor implementations of GPU operations
are used, such as the batched \texttt{gemm} routine.
Where necessary, custom GPU kernels are developed, as in the case of computing
matrix norms.
Care is taken to hide communication by overlapping it with GPU computations.

\item
\textbf{Community engagement:}
The SLATE team interacts on a regular basis with the OpenMP community,
represented in ECP by the SOLLVE project, and with the MPI community,
represented in ECP by the OMPI-X project and the Exascale MPI project.
The SLATE team also engages the vendor community through our contacts
at Cray, IBM, Intel, NVIDIA, AMD, and ARM.

\end{enumerate}

\paragraph{Recent Progress}

% During 2018, the SLATE team implemented parallel BLAS, parallel norms,
% linear system solvers (LU, Cholesky, Hermitian indefinite $LTL^T$),
% and least squares solvers (QR, LQ).
During 2019, the SLATE team continued to add to its suite:
mixed-precision linear solvers for LU and Cholesky in March,
matrix inversion for LU and Cholesky in June,
and Hermitian eigenvalue and singular value solvers in September.
Routines are available in the four standard precisions (single, double,
complex-single, complex-double).
% We also continued to implement performance improvements, such as a CPU--GPU
% coherency protocol inspired by the MOSI cache consistency protocol, and
% improved handling of row-major matrices for efficient row-swapping in LU.
In addition to SLATE's native C++ API, compatibility APIs for LAPACK
and ScaLAPACK users are also provided, so that SLATE can serve
as a drop-in replacement.
All developments are documented in SLATE Working Notes
\footnote{\url{http://www.icl.utk.edu/publications/series/swans}.}

\paragraph{Next Steps}

\begin{enumerate}

\item
\textbf{Users' Guide and Developers' Guide:}
These guides will supplement SLATE's online Doxygen documentation by thoroughly
explaining both SLATE's public interface---how to integrate it into an
application, with examples---as well as SLATE's internal design, to help
new developers and outside collaborators understand the codebase. As living
documents, they will be continually updated as SLATE matures.

\item
\textbf{Band solvers:}
To SLATE's existing general band LU solvers,
we are adding Hermitian band Cholesky factorization and solve,
along with Hermitian band BLAS and norm routines.

\item
\textbf{Performance improvements:}
We have observed several routines that are not performing as well as expected.
In some cases, such as in QR, LQ, and parallel norms, we have already identified
improvements to be made to the algorithm, such as refactoring loops to improve
parallelism. In other cases, we are analyzing traces to identify and
correct problems.

\item
\textbf{Generalized Hermitian eigenvalue solver:}
We will extend SLATE's Hermitian eigenvalue solver to
handle generalized Hermitian-definite eigenvalue problems, of the form
$Ax = \lambda Bx$, $ABx = \lambda x$, or $BAx = \lambda x$.
These are of strong interest in mechanics and chemistry applications,
among others.

\item
\textbf{New C++, C, and Fortran application programming interfaces (APIs):}
The BLAS and (Sca)LAPACK APIs have served well over the past 40 years, but are
rooted in out-dated FORTRAN 77 limitations such as cryptic 5--6 letter
abbreviations (e.g., \texttt{dgemm, dgesv}). For SLATE, we will design a new,
user-friendly C++ interface (e.g., \texttt{multiply, solve}, respectively). New
C and Fortran APIs will also give better direct access to SLATE's features and
Matrix objects, without requiring use of our ScaLAPACK compatibility interface.

\item
\textbf{Non-symmetric eigenvalue problem:}
Computing the non-symmetric eigenvalue problem is significantly more
computationally expensive than the Hermitian eigenvalue problem. Our
implementation will leverage the latest advances, such as aggressive early
deflation, to achieve high performance.

\end{enumerate}


\section{Introduction}

One in a billion, or $10^{-9}$, is the prescribed safety margin of a
catastrophic fault occurring in the avionics of a civil
aircraft~\cite{Rushby09:SEFM}.  The justification for the requirement
is to show that failures resulting in a catastrophic effect are ``so
unlikely that it is not anticipated to occur during the operational
life of an entire system or fleet''\cite{FAA2000}. Let us call systems
with reliability requirements on this order \emph{ultra-critical} and
those that meet the requirements \emph{ultra-reliable}.  Similar
reliability metrics might be claimed for other safety-critical
systems, like nuclear reactor shutdown systems or railway switching
systems.

Neither formal verification nor testing can ensure system reliability.
Contemporary ultra-critical systems may contain millions of lines of code; the
full functional correctness of approximately ten thousand lines of code
represents the state-of-the-art, taking 20 engineer-years to achieve~\cite{l4}.
Nearly 20 years ago, Butler and Finelli showed that testing alone cannot verify
the reliability of ultra-critical software~\cite{butler}.

Runtime verification (RV), where monitors detect and respond to property
violations at runtime, holds particular potential for ensuring that
ultra-critical systems are in fact ultra-reliable, but there are challenges.  In
ultra-critical systems, RV must account for both software and hardware faults.
Whereas software faults are design errors, hardware faults can also be a result of
random failure.

For the purposes of this paper, assume that characterizing a system as
being \emph{ultra-critical} implies it is a distributed system with replicated
hardware (so that the failure of an individual component does not cause system-wide
failure); also assume ultra-critical systems are embedded systems
responsible for
sensing and/or controlling some physical plant and that they are \emph{hard
  real-time}, meaning that deadlines are fixed and time-critical.  %% Another
%% challenge then is RV for distributed real-time systems.

%% The mean time to failure of a hardware component contributes to the
%% mean time to failure of the whole system, it depends on the margins that
%% a hardware component has been designed with, related to the inhospitality
%% of the environment within which ultra-critical systems are often deployed,
%% as well as the duration of their deployment.
%% Random hardware faults are inherent to every hardware component, have to
%% be anticipated and responded to.

\paragraph{Outline.}  In Section~\ref{sec:motivation}, we motivate the need for
runtime verification for ultra-critical systems.  In
Section~\ref{sec:constraintsApproaches}, we describe constraints and approaches
to RV for embedded systems.  In Section~\ref{sec:language}, we present the
language of Copilot, our approach for ultra-critical RV.  We describe the tools
(e.g., the interpreter and compiler back-ends) associated with Copilot in
Section~\ref{sec:tools}.  We present two case-studies applying Copilot to
avionic systems in Section~\ref{sec:case-study}.  Related work is given in
Section~\ref{sec:related}, and we make final remarks in
Section~\ref{sec:conclusions}.

\section{When Ultra-Critical Is \emph{Not} Ultra-Reliable}\label{sec:motivation}

Well-known, albeit dated, examples of the failure of critical systems include the
Therac-25 medical radiation therapy machine~\cite{therac} and the Ariane~5
Flight 501 disaster~\cite{ariane}.  However, more recent events show that
critical-system software safety, despite certification and extensive testing, is still an
unmet goal.  Below, we briefly overview three examples drawn from
faults in the Space Shuttle, a Boeing 777, and an Airbus A330, all occurring
between 2005 and 2008.

\paragraph{Space Shuttle.}
During the launch of shuttle flight Space Transportation System 124 (STS-124) on
May 31, 2008, there was a pre-launch failure of the fault diagnosis software due
to a ``non-universal I/O error'' in the Flight Aft (FA) multiplexer
de-multiplexer (MDM) located in the orbiter's aft avionics bay~\cite{mdm08}.
The Space Shuttle's data processing system has four general purpose computers
(GPC) that operate in a redundant set.  There are also twenty-three MDM units
aboard the orbiter, sixteen of which are directly connected to the GPCs via
shared buses.  The GPCs execute redundancy management algorithms that include a
fault detection, isolation, and recovery function.  In short, a diode failed on
the serial multiplexer interface adapter of the FA MDM.  This failure was
manifested as a \emph{Byzantine fault} (i.e., a fault in which different nodes
interpret a single broadcast message differently~\cite{lamport95byzantine}), which was not
tolerated and forced an emergency launch abortion.

\paragraph{Boeing~777.}
On August 1, 2005, a Boeing 777-120 operated as Malaysia Airlines Flight 124
departed Perth, Australia for Kuala Lumpur, Malaysia.  Shortly after takeoff,
the aircraft experienced an in-flight upset, causing the autopilot to
dramatically manipulate the aircraft's pitch and airspeed.  A subsequent
analysis reported that the problem stemmed from a bug in the Air Data Inertial
Reference Unit (ADIRU) software~\cite{ATSB07}.  Previously, an accelerometer
(call it \emph{A}) had failed, causing the fault-tolerance computer to take data
from a backup accelerometer (call it \emph{B}).  However, when the backup
accelerometer failed, the system reverted to taking data from \emph{A}.  The
problem was that the fault-tolerance software assumed there would not be a
simultaneous failure of both accelerometers. Moreover, the software
was not designed to report accelerometer \emph{A}'s failure so
maintenance was not performed.

\paragraph{Airbus~A330.}
On October 7, 2008, an Airbus~A330 operated as Qantas Flight~QF72 from Singapore
to Perth, Australia was cruising when the autopilot caused a pitch-down followed by a
loss of altitude of about 200 meters in 20 seconds (a subsequent less severe pitch was also
made)~\cite{ATSB08}.  The accident required the hospitalization of
fourteen people.  Like in the Boeing~777 upset, the source of this accident was
an ADIRU.  The ADIRU appears to have suffered a transient fault that was not
detected by the fault-management software of the autopilot system.


\section{Introduction}

How well can we infer shared divergence times given only a single locus?

How does a full-likelihood method that assumes unlinked characters
compare to an ABC method that correctly assumes linked sites?

\section{Methods}

We used
\ecoevolity \citep[Version 0.2.1 (commit 8ac8228);][]{Oaks2018ecoevolity}
for all full-likelihood analsyes and
\pymsbayes \citep[Version 0.2.4;][]{Oaks2014dpp}
for all ABC analyses.
We chose the settings for simulating and analyzing data with these tools to
maximize the comparability of the results.
For both methods, we used the same distributions from which parameter values
were drawn for simulating the data as priors in the subsequent analyses of the
simulated datasets.
As a result, there were no model violations except for \ecoevolity, where we
were violated the assumption that all characters are unlinked.

\subsection{Simulation-based analyses with \ecoevolity}

To simulate 500 \datasets with a single locus of length 500, 1000, 2000, and
10,000 sites,
we used the \simcoevolity tool.
For each \dataset, there were five pairs of populations with 20 copies of the
locus sampled from each population.
We assumed that all five pairs share the same rate of mutation, which was set
to 1.0, such that time and effective population sizes are scaled by the
mutation rate.
We treated the number and timing of divergence events and the assignment of the
pairs of populations to those events as a random variable under a Dirichlet
process with a concentration parameter of 2.22543 (a prior mean number of
events of three) and base distribution of $\etime \sim \dexponential{0.01}$;
the base distribution is the prior on divergence times.

The model implemented in \ecoevolity assumes biallelic characters that evolve
along gene trees according to a continutuous-time Markov chain (CTMC) model of
state substitution.
For this CTMC model, we constrained the equilibrium frequencies of the two
states to be equal.

We drew the effective size of each descendant population of each pair
from a gamma distribution,
$\epopsize[\descendantpopindex{}] \sim \dgamma{10}{0.0005}$.
The size of each ancestral population relative to the mean
size of the descendant populations was drawn from
another gamma distribution,
$\rootrelativepopsize \sim \dgamma{100}{1}$.

For each pair of populations, we assumed the locus was biparentally inherited
and diploid (i.e., we set the ploidy to 2.0).
However, this is directly translatable to a uniparentally inherited, haploid
locus (e.g., a mitochondrial locus) by simply multiplying the effective
population sizes by four.
In other words, we would get the same results if we set the ploidy to 0.5 and
changed the distribution on the descendant population size to
\dgamma{10}{0.002}.

We ran three independent MCMC chains for 30,000 generations, sampling every
20th generation for 1501 samples from each chain, including the initial state.
For each analyzed \dataset, we removed the first 401 samples from each chain
before summarizing the posterior across the remaining 3300 samples.
We confirmed this was an adequate burn-in by checking the potential scale
reduction factor \citep[PSRF; the square root of Equation 1.1 in][]{Brooks1998}
and effective sample size \citep[ESS;][]{Gong2014} of the log-likelihood and
parameters.

\subsection{Simulation-based analyses with \pymsbayes}

For all analyses with \pymsbayes, we used the \dppmsbayes model
\citep{Oaks2014dpp}.
We used the same distributions on parameters as we did for \ecoevolity, with
two exceptions.
First, \dppmsbayes parameterizes population sizes differently than \ecoevolity.
For \dppmsbayes, priors are specified for $4\epopsize{}\murate{}$, rather than
on \epopsize{} and \murate{}.
Accordingly, we used a distribution of \dgamma{10}{0.002} on
$4\epopsize{}\murate{}$ to ensure comparability with the \ecoevolity results.
Second, for \dppmsbayes, we used a Jukes-Cantor model of nucleotide
substitution \citep{JC1969}; the model we used in \ecoevolity is the two-state
equivalent.

We simulated 500,000 samples from the joint prior distribution, and used four
summary statistics calculated from the simulated locus alignment of each pair:
\begin{enumerate}
    \item The number of segregating sites \citep[$\theta_W$;][]{Watterson1975},
    \item the average number of pairwise differences across all gene copies
        \citep[$\pi$;][]{NeiLi1979},
    \item the net number of pairwise differences between the two populations
        \citep[Equation 25 in][]{NeiLi1979}, and
    \item the standard deviation in the difference between $\pi$ and $\theta_W$
        \citep{Tajima1989}.
\end{enumerate}
We used the mean and standard deviation from the prior to standardize each
statistic.
Using the same standardized statistics calculated from each simulated observed
\dataset, we then retained the 2,000 prior samples that had the smallest
Euclidean distance as the sample from the approximate posterior.

\section{Results}

In general, neither \ecoevolity nor \dppmsbayes is doing particularly well with
just one locus.
\ecoevolity is doing a bit better by most metrics
(\cref{fig:divtimes,fig:nevents,fig:leafsizes,fig:rootsizes}),
which suggests we are better off violating the assumption of unlinked sites
rather than resorting to ABC.

\ifembed{
\input{fig-div-times.tex}
}{}

\ifembed{
\input{fig-nevents.tex}
}{}

\ifembed{
\input{fig-leaf-sizes.tex}
}{}

\ifembed{
\input{fig-root-sizes.tex}
}{}

The PSRF
(Fig.\
S\ref{fig:psrflikelihood}
\&
S\ref{fig:psrfdivtimes})
and ESS
(Fig.\
S\ref{fig:esslikelihood},
S\ref{fig:essdivtimes},
\&
S\ref{fig:essrootsizes})
values for the log likelihood score and parameters
indicate that MCMC chains run with \ecoevolity converged and mixed well.

\section{Discussion}


\section{Age stratification}
% Note that this will vary for every application, so will need to be edited - not sure of how best to manage this:
All compartments of the base compartmental structure were stratified by age into the following age bands:
\begin{itemize}
    \item Zero to 14 years
    \item 15 to 24 years
    \item 25 to 49 years
    \item 50 to 69 years
    \item 70 years and above
\end{itemize}
Demographic processes, including births, ageing and non-infection-related deaths are not simulated, given the timeframes considered in this simulation.