\part{Lecture 10: Value-Based Control with Function Approximation}
\title[RL Lecture 10]{Lecture 10: Value-Based Control \\with Function Approximation}
\date{}
\frame{\titlepage}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Preface %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Preface}
Problem space: it is further assumed that
\begin{itemize}
	\item the states $\bm{x}$ are (quasi-)continuous and
	\item the actions $u$ are discrete.
\end{itemize}\pause
\vspace{0.5cm}
Today's focus:
\begin{itemize}
	\item \hl{valued-based control} tasks, i.e., transferring the established tabular methods to work with function approximation.
	\item Hence, we need to extend the previous prediction methods to action values
\end{itemize}
\begin{equation}
			\hat{q}(\bm{x}, u, \bm{w}) \approx q_\pi(\bm{x}, u) .
\end{equation}\pause
\begin{itemize}
	\item And apply the well-known generalized policy iteration scheme (GPI) to find optimal actions:
\end{itemize}
\begin{equation}
			\hat{q}(\bm{x}, u, \bm{w}) \approx q^*(\bm{x}, u) .
\end{equation}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Types of Action-Value Function Approximation %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Types of Action-Value Function Approximation}
\begin{figure}
\includegraphics[width=9.5cm]{fig/lec10/Model_types_action_value.pdf}
\label{fig:Model_types_action_value}
\caption{Possible function approximation settings for discrete actions}
\end{figure}
\begin{itemize}
	\item Left: one function with both states and actions as input
	\item Middle: one function with $i=1,2,\ldots$ outputs covering the action space (e.g., ANN with appropriate output layer)
	\item Right: multiple (sub-)functions one for each possible action $u_i$ (e.g., multitude of linear approximators in small action spaces)
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Feature Engineering %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Feature Engineering}
\begin{itemize}
	\item Also for action-value estimation a proper feature engineering (FE) is of vital importance.
	\item Compared to the state-value prediction, the action becomes part of the FE processing:
\begin{equation}
	\hat{q}(\bm{x}, u, \bm{w}) = \hat{q}\left(\bm{f}\left(\bm{x}, u\right), \bm{w}\right) .
\end{equation}
	\item Above, $\bm{f}(\bm{x},u)\in\mathbb{R}^\kappa$ is the FE function.\pause
	\item \hl{For sake of notation simplicity we write $\hat{q}(\bm{x}, u, \bm{w})$ and understand that FE has already been considered (i.e., is a part of $\hat{q}$).}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{On-Policy Control With (Semi-)Gradients}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Gradient-Based Action-Value Learning %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Gradient-Based Action-Value Learning}

\begin{itemize}
	\item Transferring the objective \eqref{eq:VE_J} from on-policy prediction to control yields:
\begin{equation}
	J(\bm{w})= \sum_k \left[q_{\pi}(\bm{x}_k, u_k) - \hat{q}(\bm{x}_k, u_k, \bm{w})\right]^2 .
\label{eq:VE_J_action}
\end{equation}\pause
\item Analogous, the (semi-)gradient-based parameter update from \eqref{eq:gradient_param_value} is also applied to action values:
 \begin{equation}
	 \bm{w}_{k+1} = \bm{w}_{k} + \alpha\left[q_\pi(\bm{x}_k, u_k) - \hat{q}(\bm{x}_k, u_k, \bm{w}_k)\right]\nabla_{\bm{w}} \hat{q}(\bm{x}_k, u_k, \bm{w}_k) .
	\label{eq:gradient_param_value_action}
 \end{equation}\pause
\item Depending on the control approach, the true target $q_\pi(\bm{x}_k, u_k)$ is approximated by:
\begin{itemize}
	\item Monte Carlo: full episodic return $q_\pi(\bm{x}_k, u_k) \approx g$,\pause
	\item Sarsa: one-step bootstrapped estimate $q_\pi(\bm{x}_k, u_k) \approx r_{k+1}+\gamma\hat{q}(\bm{x}_{k+1}, u_{k+1}, \bm{w}_k)$,\pause
	\item $n$-step Sarsa: $q_\pi(\bm{x}_k, u_k) \approx r_{k+1}+\gamma r_{k+2}+\cdots+\gamma^{n-1} r_{k+n}+\gamma^n\hat{q}(\bm{x}_{k+n}, u_{k+n}, \bm{w}_{k+n-1})$.
\end{itemize}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Houston: We have a Problem (1)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Houston: We have a Problem (1)}
\begin{itemize}
	\item Recall tabular \hl{policy improvement theorem} (\theoref{theo:Policy_improvement}): guarantee to find a globally better or equally good policy in each update step.\pause
	\item With parameter updates \eqref{eq:gradient_param_value_action} generalization applies.
	\item Hence, when reacting to one specific state-action transition other parts of the state-action space within $\hat{q}$ are affected too. \pause
\end{itemize}
\vspace{0.5cm}
\begin{columns}[t,onlytextwidth]
\begin{column}{0.4\textwidth}
\begin{minipage}[c]{\linewidth}
\begin{figure}
\includegraphics[height=2.5cm]{fig/lec03/GPI_02.pdf}
\caption{GPI}
\end{figure}
\end{minipage}
\end{column}
\hfill
\begin{column}{0.57\textwidth}
\begin{minipage}[c]{\linewidth}
\begin{block}{Loss of policy improvement theorem}
	\begin{itemize}
		\item Is not applicable with function approximation!
		\item We may improve and impair the policy at the same time!
	\end{itemize}
\end{block}
\end{minipage}
\end{column}
\end{columns}

}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Houston: We have a Problem (2)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Houston: We have a Problem (2)}
\begin{figure}
	\subfloat{
		\includegraphics[height=4.4cm]{fig/lec10/Sarsa_Brakeout.pdf}
	}
	\subfloat{
		\includegraphics[height=4.4cm]{fig/lec10/Sarsa_Seaquest.pdf}
	}
\caption{Learning curves with drastic performance dips when applying Sarsa with function approximation.  Left: Atari Breakout, right: Atari Seaquest (source: D. Zhao et al., \textit{Deep reinforcement learning with experience replay based on SARSA}, IEEE Symposium Series on Computational Intelligence, 2016)}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Gradient Monte Carlo On-Policy Control %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Algorithmic Implementation: Gradient MC Control}
\begin{itemize}
	\item Direct transfer from tabular case to function approximation
	\item Update target becomes the sampled return $q_\pi(\bm{x}_k, u_k)\approx g_k$
	\item If operating $\varepsilon$-greedy on $\hat{q}$: baseline policy (given by $\bm{w}_0$) must (successfully) terminate the episode!
\end{itemize}
\setlength{\algomargin}{0.5em}
\begin{algorithm}[H]
\SetKwInput{Input}{input}
\SetKwInput{Output}{output}
\SetKwInput{Init}{init}
\SetKwInput{Param}{parameter}
\Input{a differentiable function $\hat{q}:\mathbb{R}^\kappa\times\mathbb{R}^\zeta\rightarrow\mathbb{R}$}
\Input{a policy $\pi$ (only if estimating $q_\pi$)}
\Param{step size $\alpha\in\left\{\mathbb{R}|0<\alpha<1\right\}$, $\varepsilon\in\left\{\mathbb{R}|0<\varepsilon<<1\right\}$}
\Init{parameter vector $\bm{w}\in\mathbb{R}^\zeta$ arbitrarily}
 \For{$j=1,2,\ldots,$ episodes}{
		generate episode following $\pi$ or $\varepsilon$-greedy on $\hat{q}$: $x_{0}, u_{0}, r_{1},\ldots,x_{T}$ \;
		calculate every-visit return $g_k$\;
		\For{$k=0,1,\ldots, T-1$ time steps}{
			$\bm{w} \leftarrow \bm{w} + \alpha\left[g_k - \hat{q}(\bm{x}_k, u_k,\bm{w})\right]\nabla_{\bm{w}} \hat{q}(\bm{x}_k, u_k, \bm{w})$\;
		}
	}
\caption{Every-visit gradient MC-based action-value estimation (output: parameter vector $\bm{w}$ for $\hat{q}_\pi$ or $\hat{q}^*$)}
\label{algo:MC_gradient_control}
\end{algorithm}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Semi-Gradient Sarsa %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Algorithmic Implementation: Semi-Gradient Sarsa}
\setlength{\algomargin}{0.5em}
\begin{algorithm}[H]
\small
\SetKwInput{Input}{input}
\SetKwInput{Output}{output}
\SetKwInput{Init}{init}
\SetKwInput{Param}{parameter}
\Input{a differentiable function $\hat{q}:\mathbb{R}^\kappa\times\mathbb{R}^\zeta\rightarrow\mathbb{R}$}
\Input{a policy $\pi$ (only if estimating $q_\pi$)}
\Param{step size $\alpha\in\left\{\mathbb{R}|0<\alpha<1\right\}$, $\varepsilon\in\left\{\mathbb{R}|0<\varepsilon<<1\right\}$}
\Init{parameter vector $\bm{w}\in\mathbb{R}^\zeta$ arbitrarily}
 \For{$j=1,2,\ldots$ episodes}{
		initialize $\bm{x}_{0}$\;
		\For{$k=0, 1, 2 \ldots $ time steps}{
			$u_k \leftarrow$ apply action from $\pi(\bm{x}_k)$ or $\varepsilon$-greedy on $\hat{q}(\bm{x}_k, \cdot, \bm{w})$\;
			observe $\bm{x}_{k+1}$ and $r_{k+1}$\;
			\If{$\bm{x}_{k+1}$ is terminal}{
					$\bm{w} \leftarrow \bm{w} + \alpha\left[r_{k+1} - \hat{q}(\bm{x}_k, u_k, \bm{w})\right]\nabla_{\bm{w}} \hat{q}(\bm{x}_k, u_k, \bm{w})$\;
					go to next episode\;
			}
			choose $u'$ from $\pi(\bm{x}_{k+1})$ or $\varepsilon$-greedy on $\hat{q}(\bm{x}_{k+1},\cdot, \bm{w})$\;
			$\bm{w} \leftarrow \quad\bm{w} + \alpha\left[r_{k+1} + \gamma\hat{q}(\bm{x}_{k+1}, u', \bm{w})  - \hat{q}(\bm{x}_k, u_k, \bm{w})\right]\nabla_{\bm{w}} \hat{q}(\bm{x}_k, u_k, \bm{w})$\;
		}
	}
\caption{Semi-gradient Sarsa action-value estimation (output: parameter vector $\bm{w}$ for $\hat{q}_\pi$ or $\hat{q}^*$)}
\label{algo:Semi_gradient_Sarsa}
\end{algorithm}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Sarsa Application Example: Mountain Car (1)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Sarsa Application Example: Mountain Car (1)}
\begin{columns}[t,onlytextwidth]
\begin{column}{0.475\textwidth}
\begin{minipage}[c]{\linewidth}
\begin{figure}
	\includegraphics[width=4.8cm]{fig/lec10/mountain_car.png}
	\caption{Classic RL control example: mountain car (derivative work based on \href{https://github.com/openai/gym}{https://github.com/openai/gym}, MIT license)}
	\label{fig:Mountain_car_gym}
\end{figure}
\end{minipage}
\end{column}
\hfill
\begin{column}{0.54\textwidth}
\begin{minipage}[c]{\linewidth}
\begin{itemize}
	\item Two cont. states: position, velocity
	\item One discrete action: acceleration given by $\left\{\mbox{left},\mbox{none},\mbox{right}\right\}$
	\item $r_k=-1$, i.e., goal is to terminate episode as quick as possible
	\item Episode terminates when car reaches the flag (or max steps)
	\item Simplified longitudinal car physics with state constraints
	\item Position initialized randomly within valley, zero initial velocity
	\item Car is underpowered and requires swing-up
\end{itemize}
\end{minipage}
\end{column}
\end{columns}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Sarsa Application Example: Mountain Car (2)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Sarsa Application Example: Mountain Car (2)}
\begin{figure}
\includegraphics[width=10cm]{fig/lec10/Mountain_Car_BS.pdf}
\caption{Cost-to-go function $-\max_u \hat{q}(\bm{x},u,\bm{w})$ for mountain car task using linear approximation with Sarsa and tile coding (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Tile Coding %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Tile Coding}
\begin{itemize}
	\item Problem space is grouped into (overlapping) partitions / tiles.
	\item Performs a discretization of the problem space.
	\item Function approximation serves as interpolation between tiles.
	\item Find an example here: \href{https://github.com/MeepMoop/tilecoding}{https://github.com/MeepMoop/tilecoding} .
\end{itemize}
\begin{figure}
\includegraphics[width=10cm]{fig/lec10/Tile_Coding.pdf}
\caption{Tile coding example in 2D (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Sarsa Application Example: Mountain Car (3)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Sarsa Application Example: Mountain Car (3)}
\begin{figure}
\includegraphics[width=10cm]{fig/lec10/Mountain_Car_BS_Learning_Rate.pdf}
\caption{Mountain car learning curves with semi-gradient Sarsa for different learning rates $\alpha$ (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% n-step Semi-Gradient Sarsa %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{
\setlength{\algomargin}{0.5em}
\begin{algorithm}[H]
\footnotesize
\SetKwInput{Input}{input}
\SetKwInput{Output}{output}
\SetKwInput{Init}{init}
\SetKwInput{Param}{parameter}
\Input{a differentiable function $\hat{q}:\mathbb{R}^\kappa\times\mathbb{R}^\zeta\rightarrow\mathbb{R}$}
\Input{a policy $\pi$ (only if estimating $q_\pi$)}
\Param{$\alpha\in\left\{\mathbb{R}|0<\alpha<1\right\}$, $\varepsilon\in\left\{\mathbb{R}|0<\varepsilon<<1\right\}$, $n\in\mathbb{Z}^+$}
\Init{parameter vector $\bm{w}\in\mathbb{R}^\zeta$ arbitrarily}
 \For{$j=1,2\ldots$ episodes}{
		initialize and store $\bm{x}_{0}$\;
		select and store $u_0\sim\pi(\bm{x}_0)$ or $\varepsilon$-greedy w.r.t. $\hat{q}(\bm{x}_0, \cdot, \bm{w})$\;
		$T\leftarrow\infty$\;
		\Repeat( $k=0, 1, 2, \ldots$){$\tau=T-1$}{
			\If{$k<T$}{
				take action $u_k$ observe and store $\bm{x}_{k+1}$ and $r_{k+1}$\;
				\lIf{$\bm{x}_{k+1}$ is terminal}{$T\leftarrow k+1$} \lElse{select \& store $u_{k+1}\sim\pi(\bm{x}_{k+1})$ or $\varepsilon$-greedy w.r.t. $\hat{q}(\bm{x}_{k+1}, \cdot, \bm{w})$}
				}
			$\tau\leftarrow k-n+1$ ($\tau$ time index for estimate update)\;
			\If{$\tau \geq 0$}{
				$g\leftarrow \sum_{i=\tau+1}^{\min(\tau + n, T)}\gamma^{i-\tau-1} r_i$\;
				if $\tau+n < T$: $g\leftarrow g + \gamma^n \hat{q}(\bm{x}_{\tau+n}, u_{\tau+n},\bm{w})$\;
				$\bm{w} \leftarrow \bm{w} + \alpha\left[g - \hat{q}(\bm{x}_\tau, u_\tau, \bm{w})\right]\nabla_{\bm{w}} \hat{q}(\bm{x}_\tau, u_\tau, , \bm{w})$\;
			}
		}
	}
\caption{$n$-step semi-gradient Sarsa (output: parameter vector $\bm{w}$ for $\hat{q}_\pi$ or $\hat{q}^*$)}
\label{algo:Semi_gradient_Sarsa_nstep}
\end{algorithm}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Mountain Car: one vs eight-step semi-gradient Sarsa %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Mountain Car: One vs Eight-Step Semi-Gradient Sarsa}
\begin{figure}
\includegraphics[width=10cm]{fig/lec10/Mountain_Car_BS_n_step_comp.pdf}
\caption{Mountain car learning curves comparing one-step Sarsa ($\alpha=0.5/8$) and eight-step Sarsa ($\alpha=0.3/8$) based on same problem definition as in \figref{fig:Mountain_car_gym} (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Mountain Car: one vs eight-step semi-gradient Sarsa %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Mountain Car: Early Performance Evaluation}
\begin{figure}
\includegraphics[width=10cm]{fig/lec10/Mountain_Car_BS_early_performance.pdf}
\caption{Effect of learning rate $\alpha$ and step variable $n$ on early performance of $n$-step semi-gradient Sarsa and tile coding function approximation on the mountain car task. Abscissa is scaled with the number of used tiles, here equal to eight (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Average Reward: An Alternative for Continuing Tasks}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents[currentsection]
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Average Reward %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{The Average Reward Definition}
\begin{itemize}
	\item Is an alternative performance metric in continuing tasks.
	\begin{itemize}
		\item Recall: discounting helped us to limit $v<\infty$. \pause
	\end{itemize}
	\item Alternative viewpoint: why value rewards further in the future less?\pause
\end{itemize}
\setcounter{mydefi}{0}
\begin{defi}{Average reward}{avg_return}
In continuing tasks the average reward is defined as
\begin{equation}
	\begin{split}
		\overline{r}_\pi &= \lim_{h\rightarrow\infty}\frac{1}{h}\sum_{k=1}^{h}\E{R_k|\bm{X}_0, U_{0:k-1}\sim\pi},\\
		&=\lim_{k\rightarrow\infty}\E{R_k|\bm{X}_0, U_{0:k-1}\sim\pi},\\
		&=\int_\mathcal{X}\mu_\pi(\bm{x})\sum_u\pi(u|\bm{x})\int_{\mathcal{X},\mathcal{R}}p(\bm{x}', r|\bm{x},u)r.
	\end{split}
	\label{eq:avg_return}
\end{equation}
for policy $\pi$ with $\mu_\pi$ being the steady-state distribution $\mu_\pi(\bm{x})=\lim_{k\rightarrow\infty}\Pb{\bm{X}_k=\bm{x}|U_{0:k-1}\sim\pi}$ (long-term state distribution).
\end{defi}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Equivalence in terms of optimality (1)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Equivalence in Terms of Optimality (1)}
To compare the average reward against the usual discounted reward setting in continuing tasks when utilizing function approximation, we introduce the performance metric
\begin{equation}
	J_\pi = \int_\mathcal{X}\mu_\pi(\bm{x})v_\pi^\gamma(\bm{x}) .
	\label{eq:metric_avg_return}
\end{equation}
Here, $v_\pi^\gamma$ is the usual discounted value and, therefore, \eqref{eq:metric_avg_return} evaluates the long-term value following $\pi$. \pause Applying the Bellman equation we receive:
\begin{equation}
	J_\pi = \int_\mathcal{X}\mu_\pi(\bm{x})\sum_u\pi(u|\bm{x})\int_{\mathcal{X},\mathcal{R}}p(\bm{x}', r|\bm{x},u)\left[r + \gamma v_\pi^\gamma(\bm{x}')\right] .
\end{equation}\pause
Inserting the average reward $\overline{r}_\pi$ definition \eqref{eq:avg_return} produces:
\begin{equation}
	J_\pi = \overline{r}_\pi + \int_\mathcal{X}\mu_\pi(\bm{x})\sum_u\pi(u|\bm{x})\int_{\mathcal{X},\mathcal{R}}p(\bm{x}', r|\bm{x},u) \gamma v_\pi^\gamma(\bm{x}').
	\label{eq:prev_avg_return_comp}
\end{equation}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Equivalence in terms of optimality (2)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Equivalence in Terms of Optimality (2)}
In the last equation \eqref{eq:prev_avg_return_comp} the term
\begin{equation*}
	\int_\mathcal{X}\mu_\pi(\bm{x})\sum_u\pi(u|\bm{x})\int_{\mathcal{X},\mathcal{R}}p(\bm{x}', r|\bm{x},u)
\end{equation*}
combines the steady-state distribution $\mu_\pi(\bm{x})$ with the likelihood of transitioning to $\bm{x}'$ and receiving $r$ when taking an action $u\sim\pi(\bm{x})$.\pause If one assumes that the long-term distribution $\mu_\pi(\bm{x})$ is independent from the starting state $\bm{x}_0$, i.e., \hl{the underlying MDP is an ergodic process}, then
\begin{equation}
\int_\mathcal{X}\mu_\pi(\bm{x})\sum_u\pi(u|\bm{x})\int_{\mathcal{X},\mathcal{R}}p(\bm{x}', r|\bm{x},u) \gamma v_\pi^\gamma(\bm{x}') = \gamma\int_{\mathcal{X}}v_\pi^\gamma(\bm{x}')\mu_\pi(\bm{x}')
\label{eq:prev_avg_return_ergodic}
\end{equation}
holds since it does not make any difference if we start from $\bm{x}$ or $\bm{x}'$.
}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Equivalence in terms of optimality (3)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Equivalence in Terms of Optimality (3)}
Inserting \eqref{eq:prev_avg_return_ergodic} into \eqref{eq:prev_avg_return_comp} yields:
\begin{equation}
\begin{split}
	J_\pi &= \overline{r}_\pi +\gamma\int_{\mathcal{X}}v_\pi^\gamma(\bm{x}')\mu_\pi(\bm{x}'),\\
				&= \overline{r}_\pi +\gamma J_\pi,\\
				&=\overline{r}_\pi +\gamma \overline{r}_\pi+ \gamma^2 J_\pi,\\
				&=\cdots\\
				&=\frac{1}{1-\gamma}\overline{r}_\pi.
\end{split}
\end{equation}
\hl{Take away observations:}
\begin{itemize}
	\item The on-policy discounted value equals the scaled discounted average reward (for on-policy continuing task methods). \pause
	\item Hence, ordering of all policies based on $\overline{r}_\pi$ would be exactly the same as using the discounted value $v_\pi^\gamma$. \pause
	\item Discount factor $\gamma$ changes from a problem to a solution parameter.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Introducing the Differential Return %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Introducing the Differential Return}
\begin{itemize}
	\item Since discounting is not used in the average reward definition we have to reformulate the return and value definitions. \pause
	\item Intuitive approach: introduce \hl{differential quantities} describing the deviations towards $\overline{r}_\pi$. \pause
	\item Differential return:
	\begin{equation}
		G_k = R_{k+1}-\overline{r}_\pi + R_{k+2}-\overline{r}_\pi+R_{k+3}-\overline{r}_\pi+\ldots
		\label{eq:differential_return}
	\end{equation}\pause
	\item Differential value functions:
	\begin{equation}
		\begin{split}
				v_\pi(\bm{x})&=\El{G_k|\bm{X}_k=\bm{x}}{\pi},\\
				q_\pi(\bm{x}, u) &= \El{G_k|\bm{X}_k=\bm{x},U_k=u}{\pi}.
		\end{split}
	\end{equation}\pause
	\item Accordingly, the Bellman equations require adaption (cf. chapter 10.3 in lecture book of Barto/Sutton, 2018).
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Integrating the average reward into RL Solutions %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Integrating the Average Reward into RL Solutions}
\begin{itemize}
	\item Based on the differential return \eqref{eq:differential_return} we can reintroduce the TD error in combination with function approximation:
		\begin{equation}
		\begin{split}
				\delta_k &= r_{k+1}-\hat{\overline{r}}_k + \hat{v}(\bm{x}_{k+1},\bm{w}_k) - \hat{v}(\bm{x}_{k},\bm{w}_k),\\
				\delta_k &= r_{k+1}-\hat{\overline{r}}_k + \hat{q}(\bm{x}_{k+1},u_{k+1}, \bm{w}_k) - \hat{q}(\bm{x}_{k},u_{k}, \bm{w}_k).\\
		\end{split}
	\end{equation}
		\item Here, $\hat{\overline{r}}_k$ is an estimate of $\overline{r}_\pi$ at time step $k$ (e.g., by a moving average filter).\pause
		\item And finally the semi-gradient-based parameter update is:
		\begin{equation}
		\begin{split}
				\bm{w}_{k+1} &= \bm{w}_k + \alpha \delta_k\nabla_{\bm{w}} \hat{v}(\bm{x}_{k},\bm{w}_k),\\
				\bm{w}_{k+1} &= \bm{w}_k + \alpha \delta_k\nabla_{\bm{w}} \hat{q}(\bm{x}_{k},u_k,\bm{w}_k).
		\end{split}
	\end{equation}\pause
	\item In the following we provide details on the differential Sarsa implementation, but nevertheless, the TD modifications for state-value prediction are straightforward.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Differential Semi-Gradient Sarsa %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Algorithmic Impl.: Differential Semi-Gradient Sarsa}
\begin{itemize}
	\item For incremental average estimation of $\overline{r}_\pi$ we introduce $\beta$.
	\item Similar to the discussion around $\gamma$, the averaging step size $\beta$ is also a solution parameter of the RL algorithm.
\end{itemize}
\setlength{\algomargin}{0.5em}
\begin{algorithm}[H]
\small
\SetKwInput{Input}{input}
\SetKwInput{Output}{output}
\SetKwInput{Init}{init}
\SetKwInput{Param}{parameter}
\Input{a differentiable function $\hat{q}:\mathbb{R}^\kappa\times\mathbb{R}^\zeta\rightarrow\mathbb{R}$}
\Input{a policy $\pi$ (only if estimating $q_\pi$)}
\Param{step sizes $\left\{\alpha,\beta\right\}\in\left\{\mathbb{R}|0<\alpha,\beta<1\right\}$, $\varepsilon\in\left\{\mathbb{R}|0<\varepsilon<<1\right\}$}
\Init{parameter vector $\bm{w}\in\mathbb{R}^\zeta$ arbitrarily, average return estimate $\hat{\overline{r}}\in\mathbb{R}$}
\Init{$\bm{x}_{0}$ and $u_0$ from $\pi(\bm{x}_0)$ or $\varepsilon$-greedy on $\hat{q}(\bm{x}_0, \cdot, \bm{w})$}
		\For{$k=0, 1, 2 \ldots $ time steps}{
			apply $u_k$ and observe $\bm{x}_{k+1}$ and $r_{k+1}$\;
			choose $u_{k+1}$ from $\pi(\bm{x}_{k+1})$ or $\varepsilon$-greedy on $\hat{q}(\bm{x}_{k+1},\cdot, \bm{w})$\;
			$\delta \leftarrow  r_{k+1} -\hat{\overline{r}}+\hat{q}(\bm{x}_{k+1}, u_{k+1}, \bm{w})-\hat{q}(\bm{x}_{k}, u_k, \bm{w})$\;
			$\hat{\overline{r}}\leftarrow\hat{\overline{r}}+\beta\delta$\;
			$\bm{w} \leftarrow \bm{w}+\alpha\delta\nabla_{\bm{w}}\hat{q}(\bm{x}_{k}, u_k, \bm{w})$\;
		}
\caption{Differential semi-gradient Sarsa action-value estimation (output: parameter vector $\bm{w}$ for $\hat{q}_\pi$ or $\hat{q}^*$)}
\label{algo:Semi_gradient_Sarsa_diff}
\end{algorithm}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Remarks on Differential Semi-Gradient Sarsa %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Remarks on Differential Semi-Gradient Sarsa}
What is the \hl{key motivation to use the differential returns} and not the discounted returns in continuing tasks?\pause
\begin{itemize}
	\item If one is interested in the long-term control behavior setting the discount close to one ($\gamma \rightarrow 1$) is required.\pause
	\item Depending on the application (reward feedback), the estimated values might become unfeasibly large numbers.\pause
	\item Even if the numeric stability can be maintained, learning might become slow.
	\begin{itemize}
		\item Recall again: discounting in continuing tasks is numerically limiting $v$.
		\item Hence we have to face a trade off: numeric stability vs. long-term estimation capabilities.\pause
	\end{itemize}
	\item Average rewards can be considered numerically more robust.\pause
\end{itemize}
Moreover, $n$-step bootstrapping  can be implemented as well:
\begin{equation*}
	\begin{split}
				G_{k:k+n} &= r_{k+1}-\hat{\overline{r}}_{k+n-1} + \ldots + r_{n+1}-\hat{\overline{r}}_{k+n-1} + \hat{q}(\bm{x}_{k+n}, u_{k+n}, \bm{w}_{k+n-1}),\\
				\delta_k &= G_{k:k+n} - \hat{q}(\bm{x}_{k},u_{k}, \bm{w}_k).
		\end{split}
	\end{equation*}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Differential Semi-Gradient Sarsa %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Algorithmic Impl.: Differential $n$-Step Semi-Gradient Sarsa}
\setlength{\algomargin}{0.5em}
\begin{algorithm}[H]
\small
\SetKwInput{Input}{input}
\SetKwInput{Output}{output}
\SetKwInput{Init}{init}
\SetKwInput{Param}{parameter}
\Input{a differentiable function $\hat{q}:\mathbb{R}^\kappa\times\mathbb{R}^\zeta\rightarrow\mathbb{R}$}
\Input{a policy $\pi$ (only if estimating $q_\pi$)}
\Param{$\left\{\alpha,\beta\right\}\in\left\{\mathbb{R}|0<\alpha,\beta<1\right\}$, $\varepsilon\in\left\{\mathbb{R}|0<\varepsilon<<1\right\}$, $n\in\mathbb{Z}^+$}
\Init{parameter vector $\bm{w}\in\mathbb{R}^\zeta$ arbitrarily, average return estimate $\hat{\overline{r}}\in\mathbb{R}$}
\Init{$\bm{x}_{0}$ and $u_0$ from $\pi(\bm{x}_0)$ or $\varepsilon$-greedy on $\hat{q}(\bm{x}_0, \cdot, \bm{w})$}
		\For{$k=0, 1, 2 \ldots $ time steps}{
			apply $u_k$ and observe $\bm{x}_{k+1}$ and $r_{k+1}$\;
			choose $u_{k+1}$ from $\pi(\bm{x}_{k+1})$ or $\varepsilon$-greedy on $\hat{q}(\bm{x}_{k+1},\cdot, \bm{w})$\;
			$\tau\leftarrow k-n+1$\;
			\If{$\tau\geq 0$}{
				$\delta \leftarrow  \sum_{i=\tau+1}^{\tau+n}( r_{i} -\hat{\overline{r}})+\hat{q}(\bm{x}_{\tau+n}, u_{\tau+n}, \bm{w})-\hat{q}(\bm{x}_{\tau}, u_\tau, \bm{w})$\;
				$\hat{\overline{r}}\leftarrow\hat{\overline{r}}+\beta\delta$\;
				$\bm{w} \leftarrow \bm{w}+\alpha\delta\nabla_{\bm{w}}\hat{q}(\bm{x}_{\tau}, u_\tau, \bm{w})$\;
			}
		}
\caption{Differential $n$-step semi-gradient Sarsa action-value estimation (output: parameter vector $\bm{w}$ for $\hat{q}_\pi$ or $\hat{q}^*$)}
\label{algo:Semi_gradient_Sarsa_diff_n_step}
\end{algorithm}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Least Squares Policy Iteration (LSPI)}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents[currentsection]
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Transfering LSTD-Style Batch Learning to Action Values %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Transferring LSTD-Style Batch Learning to Action Values}
\begin{itemize}
	\item In the previous lecture we developed a closed-form batch learning tool, LSTD, which estimated the state values of a given policy if
	\begin{itemize}
		\item we use linear function approximation and
		\item a fixed, representative data set $\bm{\mathcal{D}}$ is given.
	\end{itemize}\pause
\item Same idea can be transferred to action values when bootstrapping with one-step Sarsa -- called \hl{LS-Sarsa} (or sometimes LSTD$Q$):
\begin{equation}
\begin{split}
	q_\pi(\bm{x}_k, u_k) &\approx r_{k+1}+\gamma\hat{q}(\bm{x}_{k+1}, u_{k+1}, \bm{w}_k),\\
	\hat{q}(\bm{x}_k, u_k, \bm{w}_k)&=\hat{q}(\tilde{\bm{x}}_k, \bm{w}_k)= \tilde{\bm{x}}\T_k\bm{w}_k .
\end{split}
\label{eq:bootstrapping_LSTDQ}
\end{equation}
\pause\vspace{-0.2cm}
\item The cost function for action-value prediction is then:
\begin{equation}
	J(\bm{w}) = \sum_k \left[r_{k+1} - \left(\tilde{\bm{x}}\T_{k} - \gamma \tilde{\bm{x}}\T_{k+1}\right)\bm{w}\right]^2.
\end{equation}\pause\vspace{-0.2cm}
\item Hence, the closed-form least squares solution for the action values is the same as for the state value case but the feature vector depends also on the actions:
\begin{equation*}
	\tilde{\bm{x}}_k = \bm{f}(\bm{x}_k, u_k).
\end{equation*}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% On and Off-Policy LSTDQ %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{On and Off-Policy LS-Sarsa}
With $b$ samples we can form a target vector $\bm{y}$ and regressor matrix $\bm{\Xi}$:
\begin{equation}
	\bm{y} = \begin{bmatrix} r_1 \\ r_2 \\ \vdots \\ r_b \end{bmatrix}, \quad \bm{\Xi}=\begin{bmatrix} \left(\tilde{\bm{x}}\T_{0} - \gamma \tilde{\bm{x}}\T_{1}\right) \\ \left(\tilde{\bm{x}}\T_{1} - \gamma \tilde{\bm{x}}\T_{2}\right)\\ \vdots \\\left(\tilde{\bm{x}}\T_{b-1} - \gamma \tilde{\bm{x}}\T_{b}\right) \end{bmatrix}\, .
	\label{eq:LSTDQ_data}
\end{equation}\pause
Regarding the data input to $\bm{\Xi}$ we can distinguish two cases: The actions $u_k$ and $u_{k+1}$ in the feature pair $\left(\tilde{\bm{x}}\T_{k} - \gamma \tilde{\bm{x}}\T_{k+1}\right)$ per row in $\bm{\Xi}$ either descends from the
\begin{itemize}
	\item \hl{same policy $\pi$ (on-policy learning)} or \pause
	\item the action $u_{k+1}$ in $\tilde{\bm{x}}_{k+1} = \bm{f}(\bm{x}_{k+1}, u_{k+1})$ is chosen based on an \hl{arbitrary policy $\pi'$ (off-policy learning)}.
\end{itemize}\pause
If we apply off-policy LS-Sarsa then
\begin{itemize}
	\item we retrieve the flexibility to collect training samples arbitrarily
	\item at the cost of an estimation bias based on the sampling distribution.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% On and Off-Policy LSTDQ %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{LS-Sarsa}
\begin{block}{LS-Sarsa solution}
Having arranged $i=1,\ldots,b$ samples $\left\langle \bm{x}_i, u_i, r_{i+1}, \bm{x}_{i+1}, u_{i+1}\right\rangle\sim\bm{\mathcal{D}}$ using one-step bootstrapping \eqref{eq:bootstrapping_LSTDQ} and linear function approximation as in \eqref{eq:LSTDQ_data}, the LS-Sarsa solution is
\begin{equation}
		\bm{w}^* = (\bm{\Xi}\T\bm{\Xi})^{-1}\bm{\Xi}\T\bm{y} .
\end{equation}
\end{block}\pause
Again, basic usage distinction:
\begin{itemize}
	\item If $\left\{u_i, u_{i+1}\right\}\sim\pi$: on-policy prediction (as in LSTD)
	\item If $u_i\sim\pi$ and $u_{i+1}\sim\pi'$: off-policy prediction (useful for control)
\end{itemize}\pause
Possible modifications:
\begin{itemize}
	\item To prevent numeric instability regularization is possible cf. \eqref{eq:Ridge_LSTD}
	\item Recursive implementation for online usage straightforward cf. \eqref{eq:RLS}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Least Squares Policy Iteration (LSPI) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Least Squares Policy Iteration (LSPI)}
General idea:
\begin{itemize}
	\item apply general policy improvement (GPI) based on data set $\bm{\mathcal{D}}$
	\item policy evaluation by off-policy LS-Sarsa
	\item policy improvement by greedy choices on predicted action values
\end{itemize}
\begin{figure}
\includegraphics[height=3.0cm]{fig/lec03/GPI_02.pdf}
\end{figure}\pause
Some remarks:
\begin{itemize}
	\item LSPI is an offline and off-policy control approach
	\item Exploration is required by feeding suitable sampling distributions in $\bm{\mathcal{D}}$
	\begin{itemize}
		\item Such as $\varepsilon$-greedy choices based on $\hat{q}$
		\item But also complete random samples are conceivable
	\end{itemize}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Semi-Gradient Sarsa %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Algorithmic Implementation: LSPI}
\setlength{\algomargin}{0.5em}
\begin{algorithm}[H]
\small
\SetKwInput{Input}{input}
\SetKwInput{Output}{output}
\SetKwInput{Init}{init}
\SetKwInput{Param}{parameter}
\Input{a feature representation $\tilde{\bm{x}}$ with $\tilde{\bm{x}}_T=0$ (i.e., $\hat{q}(\tilde{\bm{x}}_T,\cdot)=0$)}
\Input{a data set $\left\langle \bm{x}_i, u_i, r_{i+1}, \bm{x}_{i+1}\right\rangle\sim\bm{\mathcal{D}}$ with $i=1,\ldots, b$ samples}
\Param{an accuracy threshold $\Delta\in\left\{\mathbb{R}|0<\Delta\right\}$}
\Init{linear approximation function weights $\bm{w}\in\mathbb{R}^\zeta$ arbitrarily}

$\pi\leftarrow \argmax_{u}\hat{q}(\cdot, u, \bm{w})$ (greedy choices based on $\hat{q}(\bm{w})$)\;
\Repeat{$||\bm{w}'-\bm{w}||<\Delta$}{
		$\bm{w}' \leftarrow \bm{w}$\;
		$\bm{w}\leftarrow\mbox{LS-Sarsa}(\bm{\mathcal{D}}, u_{i+1}\sim\pi)$\;
		$\pi\leftarrow \argmax_{u}\hat{q}(\cdot, u, \bm{w})$\;
}
\caption{Least squares policy iteration (output: $\bm{w}$ for $\hat{q}^*$)}
\label{algo:LSPI}
\end{algorithm}\pause
\begin{itemize}
	\item In a (small) discrete action space the $\argmax_{u}$ operation is straightforward: just compare $\hat{q}$ for all possible actions given the state. \pause
	\item After one full LSPI evaluation the data set $\bm{\mathcal{D}}$ might be altered to include new data obtained based on the updated $\bm{w}$ vector. \pause
	\item Source: M. Lagoudakis and R. Parr, \textit{Least-Squares Policy Iteration}, Journal of Machine Learning Research 4, pp. 1107-1149, 2003
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% LSPI Application Example: Inverted Pendulum %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{LSPI Application Example: Inverted Pendulum (1)}
\begin{columns}[t,onlytextwidth]
\begin{column}{0.475\textwidth}
\begin{minipage}[c]{\linewidth}
\begin{figure}
	\includegraphics[width=4.8cm]{fig/lec10/Inv_pendulum.pdf}
	\caption{Classic RL control example: inverted pendulum (source: \href{https://commons.wikimedia.org/wiki/File:Cart-pendulum.svg}{www.wikipedia.org}, \href{https://creativecommons.org/publicdomain/zero/1.0/deed.en}{CC0 1.0})}
	\label{fig:Inv_pendulum}
\end{figure}
\end{minipage}
\end{column}
\hfill
\begin{column}{0.54\textwidth}
\begin{minipage}[c]{\linewidth}
\begin{itemize}
	\item Two continuous states: angular position $\theta$ and velocity $\dot{\theta}$
	\item One discrete action: acceleration force (i.e., torque at shaft)
	\item Action noise as disturbance
	\item Non-linear system dynamics
	\item State initialization randomly close to upper equilibrium
	\item $r_k=0$ if pendulum is above horizontal line
	\item $r_k=-1$ if below horizontal line and episode terminates
	\item $\gamma=0.95$
\end{itemize}
\end{minipage}
\end{column}
\end{columns}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% LSPI Application Example: Inverted Pendulum (2) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{LSPI Application Example: Inverted Pendulum (2)}
\begin{itemize}
	\item Initial training samples for $\bm{\mathcal{D}}$ following a policy selecting actions at uniform probability
	\item Additional samples have been manually added during the training
	\item Radial basis function as feature engineering
\end{itemize}
\begin{figure}
\includegraphics[height=4.2cm]{fig/lec10/LSPI_Inv_Pendulum.pdf}
\caption{Balancing steps before episode termination with a clipping of maximum 3000 steps (source: M. Lagoudakis and R. Parr, \textit{Least-Squares Policy Iteration}, Journal of Machine Learning Research 4, pp. 1107-1149, 2003)}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% RLS-TD Algorithmic Implementation  %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Algorithmic Implementation: Online LSPI}
\setlength{\algomargin}{0.5em}
\begin{algorithm}[H]
\small
\SetKwInput{Input}{input}
\SetKwInput{Output}{output}
\SetKwInput{Init}{init}
\SetKwInput{Param}{parameter}
\Input{a feature representation $\tilde{\bm{x}}$ with $\tilde{\bm{x}}_T=0$ (i.e., $\hat{q}(\tilde{\bm{x}}_T,\cdot, \cdot)=0$)}
\Param{forgetting factor $\lambda\in\left\{\mathbb{R}|0 < \lambda \leq 1\right\}$, $\varepsilon\in\left\{\mathbb{R}|0<\varepsilon<<1\right\}$, update factor $k_w\in\left\{\mathbb{N}|1\leq k_w\right\}$}
\Init{weights $\bm{w}\in\mathbb{R}^\zeta$ arbitrarily, policy $\pi$ being $\varepsilon$-greedy w.r.t. $\hat{q}(\bm{w})$, covariance $\bm{P}>0$ (e.g., $\bm{P}=\beta\bm{I}$)}
 \For{$j=1,2,\ldots$ episodes}{
		initialize $\bm{x}_{0}$ and set $u_0\sim\pi(\bm{x}_{0})$\;
		\For{$k=0, 1, 2 \ldots $ time steps}{
			apply action $u_k$, observe $\bm{x}_{k+1}$ and $r_{k+1}$, set $u_{k+1}\sim\pi(\bm{x}_{k+1})$\;
			$y \leftarrow r_{k+1}$\;
			$\bm{\xi}\T \leftarrow \tilde{\bm{x}}\T_{k}(\bm{x}_k, u_k) - \gamma \tilde{\bm{x}}\T_{k+1}(\bm{x}_{k+1}, u_{k+1})$\;
			$\bm{c} \leftarrow \left(\bm{P}\bm{\xi}\right)/\left(\lambda+\bm{\xi}\T\bm{P}\bm{\xi}\right)$\;
			$\bm{w} \leftarrow \bm{w} + \bm{c}\left(y-\bm{\xi}\T \bm{w}\right)$\;
			$\bm{P} \leftarrow \left(\bm{I} - \bm{c}\bm{\xi}\T\right)\bm{P}/\lambda$\;
			\If{$k \mod k_w=0$}{
					$\pi\leftarrow \varepsilon$-greedy w.r.t. $\hat{q}=\tilde{\bm{x}}\T(\bm{x},u)\bm{w}$\;
			}
			exit loop if $\bm{x}_{k+1}$ is terminal\;
		}
	}
\caption{Online LSPI with RLS-Sarsa (output: $\bm{w}$ for $\hat{q}^*$)}
\label{algo:LSPI_Online}
\end{algorithm}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Remarks on Online LSPI %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Remarks on Online LSPI}
\begin{itemize}
	\item $k_w$ depicts the number of steps between policy improvement cycles.\pause
	\item Forgetting factor $\lambda$ and $k_w$ require mutual tuning:
	\begin{itemize}
		\item After each policy improvement the policy evaluation requires sample updates to accurately predict the altered policy.
		\item Numerically instability may occur for $\lambda < 1$ and requires regularization.
	\end{itemize}\pause
	\item Hence, the algorithm is online-capable but its policy is normally not updated in a step-by-step fashion.\pause
	\item Alternative online LSPI with OLS-Sarsa can be found in L.~Bu{\c{s}}oniu~et~al., \textit{Online least-squares policy iteration for reinforcement learning control}, American Control Conference, 2010.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Online LSPI Application Example: Inverted Pendulum %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Online LSPI Application Example: Inverted Pendulum }
\begin{columns}[t,onlytextwidth]
\begin{column}{0.6\textwidth}
\begin{minipage}[c]{\linewidth}
\begin{figure}
	\includegraphics[width=4.8cm]{fig/lec10/Online_LSPI_Pendulum.pdf}
	\caption{Inverted pendulum with online LSPI using OLS-Sarsa ($k_w=k_\theta$, source: L. Bu{\c{s}}oniu et al., \textit{Online least-squares policy iteration for reinforcement learning control}, American Control Conference, 2010.)}
	\label{fig:Online_LSPI_Pendulum}
\end{figure}
\end{minipage}
\end{column}
\hfill
\begin{column}{0.4\textwidth}
\begin{minipage}[c]{\linewidth}
\begin{itemize}
	\item In principle same problem framework as before
	\item Altered reward: $r=-\bm{x}^T\bm{N}\bm{x}-m u^2$
	\item $\bm{N}=\mbox{diag}(\begin{bmatrix}5 & 0.1\end{bmatrix})$
	\item $m=1$
	\item $\bm{x}=\begin{bmatrix}\theta & \dot{\theta}\end{bmatrix}\T$
	\item $\bm{x}=\begin{bmatrix}0 & 0\end{bmatrix}\T$ equals pendulum pointing up without movement
\end{itemize}
\end{minipage}
\end{column}
\end{columns}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Deep \texorpdfstring{$Q$}{Q}-Networks (DQN)}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents[currentsection]
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% General Background on DQN  %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{General Background on DQN}
\begin{itemize}
	\item Recall incremental learning step from tabular $Q$-learning:
	\begin{equation*}
		\hat{q}(x, u) \leftarrow \hat{q}(x, u) + \alpha\left[r+\gamma \max_u \hat{q}(x', u) - \hat{q}(x, u)\right] .
		\end{equation*}\pause
	\item \hl{Deep $Q$-networks (DQN)}	transfer this to an approximate solution:
	\end{itemize}
	 \begin{equation}
	 \bm{w} = \bm{w} + \alpha\left[r+\gamma \max_u \hat{q}(\bm{x}', u, \bm{w}) - \hat{q}(\bm{x}, u, \bm{w})\right]\nabla_{\bm{w}} \hat{q}(\bm{x}, u, \bm{w}).
 \end{equation}\pause
However, instead of using above semi-gradient step-by-step updates, DQN is characterized by
	\begin{itemize}
		\item an \hl{experience replay buffer} for batch learning (cf. prev. lectures),\pause
		\item a separate set of \hl{weights $\bm{w}^-$ for the bootstrapped $Q$-target}.
	\end{itemize}\pause
Motivation behind:
\begin{itemize}
	\item Efficiently use available data (experience replay).
	\item Stabilize learning by trying to make targets and feature inputs more like i.i.d. data from a stationary process (prevent windup of values).
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Summary of DQN Working Principle  (1)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Summary of DQN Working Principle (1)}
\begin{itemize}
	\item Take actions $u$ based on $\hat{q}(\bm{x},u,\bm{w})$ (e.g., $\varepsilon$-greedy). \pause
	\item Store observed tuples $\left\langle \bm{x}, u, r, \bm{x}'\right\rangle$ in memory buffer $\bm{\mathcal{D}}$.\pause
	\item Sample mini-batches $\bm{\mathcal{D}}_b$ from $\bm{\mathcal{D}}$.
	\item Calculate bootstrapped $Q$-target with a delayed parameter vector $\bm{w}^-$ (so-called target network):
	\begin{equation*}
		 q_\pi(\bm{x},u)\approx r+\gamma \max_u \hat{q}(\bm{x}',u,\bm{w}^-).
	\end{equation*}\pause
	\item Optimize MSE loss between above targets and the regular approximation $\hat{q}(\bm{x},u,\bm{w})$ using $\bm{\mathcal{D}}_b$
	\begin{equation}
				\mathcal{L}(\bm{w}) = \left[\left(r+ \gamma \max_u \hat{q}(\bm{x}',u,\bm{w}^-)\right) - \hat{q}(\bm{x},u,\bm{w}) \right]^2_{\bm{\mathcal{D}}_b} \, .
	\end{equation}\pause
	\item Update $\bm{w}^-$ based on $\bm{w}$ from time to time.
 \end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Summary of DQN Working Principle  (2)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Summary of DQN Working Principle (2)}
\begin{figure}
\includegraphics[height=5.5cm]{fig/lec10/DQN.pdf}
\caption{DQN structure from a bird's-eye perspective (derivative work of \figref{fig:RL_Wiki} and \href{https://commons.wikimedia.org/wiki/File:Multi-Layer_Neural_Network-Vector.svg?uselang=de}{wikipedia.org}, \href{https://creativecommons.org/publicdomain/zero/1.0/deed.en}{CC0 1.0})}
\label{fig:DQN}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Algo DQN %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Algorithmic Implementation: DQN}
\setlength{\algomargin}{0.5em}
\begin{algorithm}[H]
\small
\SetKwInput{Input}{input}
\SetKwInput{Output}{output}
\SetKwInput{Init}{init}
\SetKwInput{Param}{parameter}
\Input{a differentiable function $\hat{q}:\mathbb{R}^\kappa\times\mathbb{R}^\zeta\rightarrow\mathbb{R}$ (including feature eng.)}
\Param{$\varepsilon\in\left\{\mathbb{R}|0<\varepsilon<<1\right\}$, update factor $k_w\in\left\{\mathbb{N}|1\leq k_w\right\}$}
\Init{weights $\bm{w}=\bm{w}^-\in\mathbb{R}^\zeta$ arbitrarily, memory $\bm{\mathcal{D}}$ with certain capacity}
 \For{$j=1,2,\ldots$ episodes}{
		initialize $\bm{x}_{0}$\;
		\For{$k=0, 1, 2 \ldots $ time steps}{
			$u_k \leftarrow$ apply action $\varepsilon$-greedy w.r.t $\hat{q}(\bm{x}_k, \cdot, \bm{w})$\;
			observe $\bm{x}_{k+1}$ and $r_{k+1}$\;
			store tuple $\left\langle \bm{x}_k, u_k, r_{k+1}, \bm{x}_{k+1}\right\rangle$ in $\bm{\mathcal{D}}$\;
			sample mini-batch $\bm{\mathcal{D}}_b$ from $\bm{\mathcal{D}}$ (after initial memory warmup)\;
			\For(calculate $Q$-targets){$i=1,\ldots,b$ samples}{
				\lIf{$\bm{x}_{i+1}$ is terminal}{$y_i=r_{i+1}$}
				\lElse{$y_i= r_{i+1}+ \gamma \max_u \hat{q}(\bm{x}_{i+1},u,\bm{w}^-)$}
			}
			fit $\bm{w}$ on loss $\mathcal{L}(\bm{w})=[y_i - \hat{q}(\bm{x}_i, u_i, \bm{w})]^2_{\bm{\mathcal{D}}_b}$\;
			\lIf{$k \mod k_w=0$}{$\bm{w}^-\leftarrow \bm{w}$ (update target weights)}
			}
}
\caption{DQN (output: parameter vector $\bm{w}$ for $\hat{q}^*$)}
\label{algo:DQN}
\end{algorithm}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Remarks on DQN Implementation %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Remarks on DQN Implementation}
\begin{itemize}
	\item General framework is based on V. Mnih et al., \textit{Human-level control through deep reinforcement learning}, Nature, pp. 529-533, 2015.\pause
	\item Often 'deep' artificial neural networks are used as function approximation for DQN.
	\begin{itemize}
		\item Nevertheless, other model topologies are fully conceivable.\pause
	\end{itemize}
	\item The fit of $\bm{w}$ on loss $\mathcal{L}$ is an intermediate supervised learning step.
	\begin{itemize}
		\item Comes with degrees of freedom regarding solver choice.
		\item Has own optimization parameters which are not depicted here in details (many tuning options).
	\end{itemize}\pause
	\item Mini-batch sampling from $\bm{\mathcal{D}}$ is often randomly distributed.
		\begin{itemize}
		\item Nevertheless, guided sampling with useful distributions for a specific control task can be beneficial (cf. Dyna discussion in 7th lecture).
	\end{itemize}\pause
	\item Likewise the simple $\varepsilon$-greedy approach can be extended.
	\begin{itemize}
		\item Often a scheduled/annealed trajectory $\varepsilon_k$ is used.
		\item Again referring to the Dyna framework, many more exploration strategies are possible.
	\end{itemize}
 \end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% DQN Application Example: Atari Games (1) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{DQN Application Example: Atari Games (1)}
\begin{itemize}
	\item End-to-end learning of $\hat{q}(\bm{x},u)$ from monitor pixels $\bm{x}$
	\item Feature engineering obtains stacking of raw pixes from last 4 frames
	\item Actions $u$ are 18 possible joystick/button combinations
	\item Reward is the change of highscore per step
	\item Interesting lecture from V. Minh with more details: \href{https://www.youtube.com/watch?v=fevMOp5TDQs&t=1499s}{YouTube}
\end{itemize}
\begin{figure}
\includegraphics[height=4.2cm]{fig/lec10/DQN_Network_Silver.pdf}
\caption{Network architecture overview used for DQN in Atari games \SilverLectureSource}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% DQN Application Example: Atari Games (2%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{DQN Application Example: Atari Games (2)}
\begin{figure}
\includegraphics[height=6.5cm]{fig/lec10/DQN_Atari_Results.pdf}
\caption{DQN performance results in Atari games against human performance \SilverLectureSource}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Summary %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Summary: What You've Learned Today}
\begin{itemize}
	\item From a simplified perspective, the procedures from the approximate prediction can simply be transferred to value-based control.\pause
	\item On the contrary, the policy improvement theorem no longer applies in the approximate RL case (generalization impact).\pause
	\begin{itemize}
		\item Control algorithms may diverge completely.\pause
		\item Or a performance trade-off between different parts of the problem space could emerge.\pause
	\end{itemize}
	\item Differential-returns allow an interesting alternative MDP formulation for continuing tasks.
		\begin{itemize}
		\item The usual discounted MDP framework may exhibit numerical problems for $\gamma\approx 1$ (slow learning, unfeasible large returns).\pause
		\item Discounting is not relevant for the optimal policy order.\pause
	\end{itemize}
	\item Off-policy batch learning approaches allow for efficient data usage.
	\begin{itemize}
		\item LSPI uses LS-Sarsa on linear function approximation.\pause
		\item DQN extends $Q$-learning on non-linear approximation with additional tweaks (experience replay, target networks,...).\pause
		\item However, a prediction bias results (off-policy sampling distribution).
	\end{itemize}
\end{itemize}
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Final Slide %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{The End for Today}
\vspace{-0.25cm}
\begin{figure}
\hspace*{-0.5cm}
\includegraphics[width=10cm]{fig/lec10/dilbert.jpg}
\end{figure}
\vspace{1cm}
\centering
Thanks for your attention and have a nice week!
}


\section{Cryptographic primitives}
\label{sec:crypto-primitives}

Figure~\ref{fig:crypto-defs} introduces the cryptographic abstractions used in
this document. Note that we define a family of serialization functions
$\serialised{\wcard}_\type{A}$, for all types $\type{A}$ for which such
serialization function can be defined. When the context is clear, we omit the
type suffix, and use simply $\serialised{\wcard}$.

\begin{figure}[htb]
  \emph{Abstract types}
  %
  \begin{equation*}
    \begin{array}{r@{~\in~}lr}
      \var{vk} & \SKey & \text{signing key}\\
      \var{vk} & \VKey & \text{verifying key}\\
      \var{hk} & \KeyHash & \text{hash of a key}\\
      \sigma & \Sig  & \text{signature}\\
      \var{d} & \Data  & \text{data}\\
    \end{array}
  \end{equation*}
  \emph{Derived types}
  \begin{equation*}
    \begin{array}{r@{~\in~}lr}
      (sk, vk) & \SkVk & \text{signing-verifying key pairs}
    \end{array}
  \end{equation*}
  \emph{Abstract functions}
  %
  \begin{equation*}
    \begin{array}{r@{~\in~}lr}
      \hash{} & \VKey \to \KeyHash
      & \text{hash function} \\
      %
      \fun{verify} & \VKey \times \Data \times \Sig
      & \text{verification relation}\\
      \serialised{\wcard}_\type{A} & \type{A} \to \Data
      & \text{serialization function for values of type $\type{A}$}\\
      \fun{sign} & \SKey \to \Data \to \Sig
      & \text{signing function}
    \end{array}
  \end{equation*}
  \emph{Constraints}
  \begin{align*}
    & \forall (sk, vk) \in \SkVk,~ m \in \Data,~ \sigma \in \Sig \cdot
      \sign{sk}{m} = \sigma \Rightarrow \verify{vk}{m}{\sigma}
  \end{align*}
  \emph{Notation}
  \begin{align*}
    & \mathcal{V}^\sigma_{\var{vk}}~{d} = \verify{vk}{d}{\sigma}
      & \text{shorthand notation for } \fun{verify}
  \end{align*}
  \caption{Cryptographic definitions}
  \label{fig:crypto-defs}
\end{figure}

\subsection{A note on serialization}
\label{sec:a-note-on-serialization}

\begin{definition}[Distributivity over serialization functions]
  For all types $\type{A}$ and $\type{B}$, given a function
  $\fun{f} \in \type{A} \to \type{B}$, we say that the serialization function
  for values of type $\type{A}$, namely $\serialised{ }_\type{A}$ distributes
  over $\fun{f}$ if there exists a function $\fun{f}_{\serialised{ }}$ such
  that for all $a \in \type{A}$:
  \begin{equation}
    \label{eq:distributivity-serialization}
    \serialised{\fun{f}~a}_\type{B} = \fun{f}_{\serialised{ }}~\serialised{a}_\type{A}
  \end{equation}
\end{definition}
The equality defined in \cref{eq:distributivity-serialization} means that the
following diagram commutes:

\[
  \begin{tikzcd}
    A \arrow{r}{\fun{f}} \arrow{d}{\serialised{ }_\type{A}}
    & B \arrow{d}{\serialised{ }_\type{B}} \\
    \Data \arrow{r}{\fun{f}_{\serialised{ }}} & \Data
  \end{tikzcd}
\]

Throughout this specification, whenever we use
$\serialised{\fun{f}~a}_\type{B}$, for some type $\type{B}$ and function
$\fun{f} \in \type{A} \to \type{B}$, we assume that $\serialised{ }_\type{A}$
distributes over $\fun{f}$ (see for example
Rule~\ref{eq:utxo-witness-inductive}). This property is what allow us to
extract a component of the serialized data (if it is available) without
deserializing it in the cases in which the deserialization function
($\serialised{\wcard}^{-1}_\type{A}$) doesn't behave as an inverse of
serialization:

\begin{equation*}
  \serialised{\wcard}^{-1}_\type{A} \cdot \serialised{\wcard}_\type{A} \neq \fun{id}_\type{A}
\end{equation*}

For the cases in which such an inverse exists, given a function $\fun{f}$, we
can readily define $\fun{f}_{\serialised{ }}$ as:

\begin{equation*}
  \fun{f}_{\serialised{ }} \dot{=} \serialised{\wcard}_\type{B}
                            . \fun{f}
                            . \serialised{\wcard}^{-1}_\type{A}
\end{equation*}


\section{BDS Model} \label{bds}

\begin{figure*}
	% \advance\leftskip-1cm
	\centering
	\includegraphics[width=1\textwidth]{img/chapter5/pipeline.pdf}
	\caption{ The overall architecture of BDS model in the $t_{th} $ time period. Each color refers to an individual model component. Dynamic Product Behavior Representation (for all products) and Community Distribution Pretraining (for seed products only) are optimized before training the main multi-task model (Neighbor Product Selection \& Summarization Generation). Red dashed lines show the workflows to calculate RL reward and in return to optimize RL policy. Data Prerequisite steps are omitted here for simplicity but are illustrated in detail in the main paper. }
	\label{fig:c5_pipeline}
\end{figure*}

The overall model architecture is sketched in Figure \ref{fig:c5_pipeline}. My final goal is to generate product dynamic summarizations using behavior data instead of product reviews. To optimize BDS model, the involved data are categorized into two groups: training \& validation data and seed product data. The training \& validation data contains products with both sufficient user behavior and temporal reviews in an individual time period. While seed product data are products required to have sufficient behaviors and reviews across all time periods. The training \& validation data helps to calculate dynamic product behavior presentations (\textit{Section \ref{sc:dpbr}}) and seed product data pretrains community distribution model (\textit{Section \ref{sc:spp}}). Subsequently, a multi-task model dynamically selects neighbor products (\textit{Section \ref{sc:rns}}) from seed products whose sentimental phrases filtered from reviews contribute to generate product summarization (\textit{Section \ref{sc:sgt}}).

\subsection{Data Prerequisite} \label{sc:dp}

Before optimizing BDS model, four following types of information need to be clarified and extracted as data prerequisite:

\textbf{Seed Products:} The top products having the most sufficient behaviors and reviews across all $t$ time periods are regarded as seed products. Their community distributions are prior-known in each time period, which are later used to guide neighbor product selection (Section \ref{sc:npst}).

\textbf{Product Descriptive Phrase:} AliNLP\footnote{English Tutorial: https://www.alibabacloud.com/help/product/57736.html \\ Chinese Tutorial: https://data.aliyun.com/product/nlp}, a paid NLP service by Alibaba, can extract name entities and sentimental phrases from E-commerce review contexts. Each product is associated with a description profile. After applying AliNLP Named Entity Recognizer (NER) on their profiles, I only keep the noun phrases to characterize related products.

\textbf{Review Sentimental Phrase:} The AliNLP sentiment analyzer can also extract sentimental phrases related to specific aspects from seed product reviews, i.e., `poor quality' is a sentimental phrases of `\textit{Quality}' aspect. Later on, these filtered sentimental phrases will be concatenated with product descriptive phrases as the model input for product summarization (Section \ref{sc:sgt}).

\textbf{Product Community Distribution:} In this paper, seed product' community distribution is prior knowledge which is learned from their reviews via AliNLP. As user behaviors has strong correlation to product characteristics, their behavior based communities should also be strongly correlated with communities calculated from product reviews which depicts product characteristics. It is a normalized vector in which each dimension represents the product affiliation towards each community, It is the ground truth to guide Community Distribution Pretraining (Section \ref{sc:spp}).

\subsection{Dynamic Product Behavior Representation} \label{sc:dpbr}

Matrix factorization is utilized on dynamic user behaviors to learn product behavior representation across all time periods. In the initial time period, I learn both user and product $m$ types of behavior representations via related user behavior $\{B^{(0)}_{1},...,B^{(0)}_{m}\}$, where a behavior type can refer to `\textit{Click}' or `\textit{Purchase}', etc. The $i_{th}$ type of user behavior $B^{(0)}_{i}$ is in a format of sparse matrix where each row denotes a user and each column denotes a product. The data points in $B^{(0)}_{i}$ denote users' $i_{th}$ type of behaviors on products. Considering all potential methods listed in \textit{Section \ref{sc:baseline}}, I empirically select Probabilistic Matrix Factorization (PMF) to decompose $B^{(0)}_{i}$ to user representation matrix $U^{(0)}_{i}$ and product representation matrix $P^{(0)}_{i}$ by satisfying the following equation:
\begin{equation}
B^{(0)}_{i} = U^{(0)}_{i} (P^{(0)}_{i})^\intercal, i = 1,...,m
\end{equation}

As user shopping preference is consistently stable, once user representation $U^{(0)}_{i}$ is calculated from the initial period, it remains unchanged and bridges product dynamic behavior representation in subsequent time periods.

In the $t_{th}$ time period, the matrix form of Least Squares Approximation \cite{nasrabadi2007pattern} helps to calculate the $i_{th}$ type of product behavior representation $P^{(t)}_{i}$ given related user behavior $B^{(t)}_{i}$ and user consistent representation $U^{(0)}_{i}$:

\begin{equation}
\begin{aligned}
& B^{(t)}_{i} = U^{(0)}_{i} (P^{(t)}_{i})^\intercal, i = 1,...,m \\
& \Rightarrow (U^{(0)}_{i})^\intercal (B^{(t)}_{i} - U^{(0)}_{i} (P^{(t)}_{i})^\intercal) = 0 \\
& \Rightarrow (U^{(0)}_{i})^\intercal U^{(0)}_{i} (P^{(t)}_{i})^\intercal = (U^{(0)}_{i})^\intercal B^{(t)}_{i} \\
& \Rightarrow (P^{(t)}_{i})^\intercal = (( U^{(0)}_{i})^\intercal U^{(0)}_{i})^{-1}(U^{(0)}_{i})^\intercal B^{(t)}_{i} \\
& \Rightarrow P_{i}^{(t)} = (B_{i}^{(t)})^\intercal U^{(0)}_{i}(( U^{(0)}_{i})^\intercal U^{(0)}_{i})^{-1}
\end{aligned}
\end{equation}

\subsection{Neighbor Product Selection Task}\label{sc:npst}

\subsubsection{Community Distribution Pretraining} \label{sc:spp}

This pretraining process is only leveraged on seed products. In the $t_{th}$ time period, I combine product behavior representation and category to estimate product community distribution via a hybrid CNN-MLP (Convolutional Neural Network and Multilayer Perceptron) approach. Let $c$ be the category of a target product, stored originally as one-hot embedding. To better represent its enriched information, I first apply an one-layer MLP with $\rm tahn(\cdot)$ activation function to convert it as a dense vector $v^{c}$.

\begin{equation}
v^{c} = {\rm tahn}(W^{c}c+b^{c})
\end{equation}

where $W^{c}$ and $b^c$ denote the weight matrix and bias respectively.

For the same product, I can also obtain its multi-type behavior representation $\{p^{(t)}_{1},...,p^{(t)}_{m}\}$, where $p^{(t)}_{m} \in P^{(t)}_{m}$ denotes its $m_{th}$ type behavior representation. As CNN kernel can help to filter out the most important dimensions (local features) from a vector, a CNN layer ${\rm cnn}(\cdot)$ with Max pooling mechanism ${\rm max\_pooling}(\cdot)$ is applied to capture product $i_{th}$ type local feature representation $l^{(t)}_{i}$.
\begin{equation}
% \vspace{-0.5em}
\begin{aligned}
&l^{(t)}_{i} = {\rm max\_pooling}({\rm tahn}({\rm cnn}(p^{(t)}_{i}))), i= 1,...,m
\end{aligned}
\end{equation}

Subsequently, I average all $m$ types of local feature representation to a single vector $l^{(t)}$. Similarly, I calculate the average product behavior vector $p^{(t)}$ as global feature representation of $m$ types of behaviors:
\begin{equation}
\begin{aligned}
&l^{(t)} = \frac{1}{m}\sum_{i = 1}^{m}l^{(t)}_{i} \\
&p^{(t)} = \frac{1}{m}\sum_{i = 1}^{m}p^{(t)}_{i}
\end{aligned}
\end{equation}

In the end, a product has three types of information representation, including global feature representation $p^{(t)}$, local feature representation $l^{(t)}$, and category representation $v^{c}$. I concatenate all these information to calculate the estimated product community distribution $d^{(t)}$ over all pre-selected aspects via one layer MLP normalized by $\rm softmax(\cdot)$ function:
\begin{equation}
d^{(t)} = {\rm softmax}(W^{d}[p^{(t)},l^{(t)},v^{c}]+b^{d})
\end{equation}
where $W^{d}$ and $b^{d}$ are related weight matrix and bias. $[\,, ]$ denotes the concatenation operation.

This pretraining process is optimized by minimizing the cross entropy between the estimated product community distribution $d^{(t)}$ and the actual product community distribution calculated from product reviews (Section \ref{sc:dp}). This optimization step has to be done ahead of the main multi-task model introduced in the following sections. In the later training process, for all products in the training \& validation dataset, their estimated community distribution can be calculated from the optimized pretraining model.

\subsubsection{Reinforcement Neighbor Selection} \label{sc:rns}

In the $t_{th}$ time period, among $h$ seed products with sufficient reviews, a policy gradient approach earns an action to select $s$ neighbor products $\mathcal{A} = \{\alpha_{1}^{(t)},\alpha_{2}^{(t)},...,\alpha_{s}^{(t)}\}$ out of $h$ candidates where $\alpha_{s}^{(t)}$ denotes the $s_{th}$ selected neighbor product. As I do not consider the sampling sequence on neighbor products, the reinforcement approach is a one-step Markov Decision Process (MDP) with single state $\mathcal{S}$ and single action $\mathcal{A}$.

Assume there are $n$ products in total across all time periods, given a target product, the initial observation $\mathcal{O}$ is the product itself, represented as a one-hot embedding $\in \mathbb{R}^{n}$. The state $\mathcal{S}$ is learned via a two-layer Multilayer Perception (MLP) on the initial observation $\mathcal{O}$:
\begin{equation}
\label{eq:state}
\mathcal{S} = {\rm softmax} (W_{2}{\rm tahn}(W_{1}\mathcal{O} + b_{1}))
\end{equation}
where $W_1$,$W_2$ and $b_1$ denote related weight matrices and bias, respectively.

The learned state $\mathcal{S} \in \mathbb{R}^{h}$ is the selection probability distribution over $h$ seed products. Assuming an action is taken to sample $s$ neighbor products with related probability weights $\{\omega_{1}^{(t)},\omega_{2}^{(t)},...,\omega_{s}^{(t)}\} \in \mathcal{S}$, the sampling policy $\pi_{\Theta_{1}}(\mathcal{A}|\mathcal{S})$ can be therefore calculated as:
\begin{equation}
\pi_{\Theta_{1}}(\mathcal{A}|\mathcal{S}) = s!\prod_{i=1}^{s}\omega^{(t)}_{i}
\end{equation}
$\Theta_{1}$ denote the parameters to be learned. The factorial of $s$ ($s!$) denotes the number of permutations for the selected neighbor products as the neighbor products are sequence insensitive. $\prod_{i=1}^{s}\omega^{(t)}_{i}$ is the generative probability of each permutation.

To assess the fitness of the $s$ selected neighbor products for the target product, I design two dynamic rewards: a community reward to measure the community similarity, and a semantic reward to calculate the content similarity between $s$ neighbor products and the target product.

\textbf{Community Reward:}
In the $t_{th}$ time period, I estimate the target product's community distribution as $d^{(t)}_{a}$ by the Community Distribution Pretraining (Section \ref{sc:spp}). The community distributions of all its selected neighbor products $\{d^{(t)}_{1},...,d^{(t)}_{s}\}$ are known as prior knowledge (Section \ref{sc:dp}). To evaluate pairwise distribution similarity, Pearson correlation calculates the community reward $ \mathcal{R}_{com,i}^{(t)}$ of the $i_{th}$ selected neighbor product $\alpha_{i}^{(t)}$ as follows:

\begin{equation}
\mathcal{R}_{com,i}^{(t)} = \frac{\mathbb{E}[(d_{a}^{(t)}-\mu(d_{a}^{(t)}))(d_{i}^{(t)}-\mu(d_{i}^{(t)}))]}{\sigma(d_{a}^{(t)})\sigma(d_{i}^{(t)})}, i = 1,...,s
\end{equation}

where $\mathbb{E}(\cdot)$ denotes the expectation, $\mu(\cdot)$ denotes the mean and $\sigma(\cdot)$ denotes the standard deviation. Larger similarity score offers a higher reward to the related neighbor product.

\textbf{Semantic Reward:} In the $t_{th}$ time period, the semantic reward of neighbor products is measured by the accuracy of generated product summarization. In this paper, we use the word level Jaccard Similarity as the indicator to calculate the semantic reward $\mathcal{R}_{sem}^{(t)}$ for all selected neighbor products, which contains two parts: the averaged Jaccard Similarity between all neighbor product original reviews and real product summarization, and the Jaccard Similarity between generated product summarization and actual product summarization:

\begin{equation}
\mathcal{R}_{sem}^{(t)} = \frac{1}{s}\sum_{i=1}^{s} \frac{|R_{i}^{(t)} \cap Y^{(t)}|}{|R_{i}^{(t)} \cup Y^{(t)}|} + \frac{|\hat{Y}^{(t)} \cap Y^{(t)}|}{|\hat{Y}^{(t)} \cup Y^{(t)}|}
\end{equation}

where $R_{i}^{(t)}$ denotes all original reviews of the $i_{th}$ neighbor product $\alpha_{i}^{(t)}$, $\hat{Y}^{(t)}$ denotes the generated summarization of target product, and $Y^{(t)}$ denotes its actual summarization. The total reward $\mathcal{R}^{(t)}_{\mathcal{A}}$ of neighbor products is the weighted sum between community reward and semantic reward controlled by a weighting factor $\gamma$:

\begin{equation}
\mathcal{R}^{(t)}_{\mathcal{A}} = \sum_{i=1}^{s} \omega_{i}^{(t)}\mathcal{R}_{com,i}^{(t)} +\gamma \mathcal{R}_{sem}^{(t)}
\end{equation}

\textbf{Task Optimization:} I use policy gradient method to optimize the sampling policy, aiming to maximize the expected total reward for neighbor products. The expected reward $\mathcal{J}_{sel}(\Theta_{1})$ in the $t_{th}$ time period is:

\begin{equation}
\begin{aligned}
\mathcal{J}_{sel}(\Theta_{1}) &= \mathbb{E}_{\mathcal{A} \sim \pi_{\Theta_{1}}(\mathcal{A}|\mathcal{S})}[\mathcal{R}^{(t)}_{\mathcal{A}}]
\end{aligned}
\end{equation}

Then, the gradient is estimated using the likelihood ratio trick:

\begin{equation}
\label{eq:loss_1}
\begin{aligned}
\nabla_{\Theta_{1}}\mathcal{J}_{sel}(\Theta_{1}) &= \nabla_{\Theta_{1}} \sum_{\mathcal{A}} \pi_{\Theta_{1}}(\mathcal{A}|\mathcal{S}) \mathcal{R}^{(t)}_{\mathcal{A}}\\
&\approx \frac{1}{N} \sum_{i=1}^{N}\nabla_{\Theta_{1}}log\pi_{\Theta_{1}}(\mathcal{A}_{i}|\mathcal{S})\mathcal{R}^{(t)}_{\mathcal{A}_{i}}
\end{aligned}
\end{equation}

where $\mathcal{A}_{i}$ denotes the $i_{th}$ of $N$ randomly sampled actions (selecting $s$ neighbor products from $h$ seed products).

\subsection{Summarization Generation Task} \label{sc:sgt}

In the $t_{th}$ time period, the filtered neighbor product sentimental phrases together with product own descriptive phrases are concatenated into a sequence $X^{(t)} = \{x_{1},...,x_{w}\}$. It is used as the input of Neural Machine Translation (NMT) model to generate product summarization sequence $Y^{(t)} = \{y_{1},...,y_{k}\}$. $w$ and $k$ denote the input and output sequence length, respectively. Filtering out most of emotional and other irrelevant words in advance can better map the input to aspect oriented summarizations instead of subjective review summaries.

The input sequence $X^{(t)} = \{x_{1},...,x_{w}\}$ is fed one-by-one into the encoder (a single-layer bidirectional LSTM), producing a sequence of encoder hidden states $\{e_{1},...,e_{w}\}$. In decoding step $i$, the decoder (a single-layer unidirectional LSTM) has a decoder hidden state $h_{i}$. Its context vector $u_{i}$ is generated via an Attention mechanism on all encoder hidden states and current decoder hidden state:

\begin{equation}
\begin{aligned}
& a_{ij} = {\rm attention}(h_{i},e_{j}), j = 1,...,w \\
& a_{ij}^{*} = \frac{{\rm exp}(a_{ij})}{\sum_{k=1}^{w} {\rm exp}(a_{ik})}\\
& u_{i} = \sum_{j=1}^{w}a_{ij}^{*}e_{j}
\end{aligned}
\end{equation}
where $a_{ij}$ denotes the attention weight of encoder hidden state $e_{j}$. $a_{ij}^{*}$ is the normalized weight by Softmax function. The weighted sum of all encoder hidden states, $u_{i}$, is the context vector for current step $i$, reflecting the auxiliary information from input sequences.

A one-layer MLP is subsequently utilized on the combination of context vector $u_{i}$ and decoder hidden state $h_{i}$ to generate the vocabulary probability distribution $P_{vocab}$:

\begin{equation}
P_{vocab} = {\rm softmax}(W^{o}[u_{i},h_{i}]+b^{o})
\end{equation}
where $W^{o}$ and $b^{o}$ are related weight and bias.

In decoding step $i$, the generation loss for target word $y_{i}$ is its negative log likelihood, $-logP_{vocab}(y_{i})$. The overall generation loss $\mathcal{J}_{gen}(\Theta_{2})$ is the average of all $k$ step generation losses and $\Theta_{2}$ are all related parameters to be optimized.

\begin{equation}
\mathcal{J}_{gen}(\Theta_{2}) = \frac{1}{k}\sum_{i=1}^{k} -logP_{vocab}(y_{i})
\end{equation}

In the multi-task model, $\mathcal{J}_{sel}(\Theta_{1})$ and $\mathcal{J}_{gen}(\Theta_{2})$ both need to be minimized during the model training process. Each task is learned separately and alternately after taking a certain number of training data batches in their optimization processes.


\subsection{Methods of Calculating Assurances}\label{sec:methods}
Recall that an assurance is defined as \emph{any} behavior or property of an AIA that affects a user's trust. As such, it is possible to design assuring properties into the system a priori. It is likewise possible to design assuring behaviors into an AIA. From the literature, several key high-level issues emerge pertaining to assurance implementation: (1) uncertainty quantification; (2) complexity reduction; (3) assurance by design; and (4) planning strategies for assurances.

\brettcomm{DISCUSS IMPLICIT VS EXPLICIT ASSURANCES HERE---PROBABLY SOMETHING LIKE A PARAGRAPH OR SO}

\subsubsection{Quantifying Uncertainty} %%nra: awkward sentence: Being able to quantify the different kinds of uncertainty in the AIA is necessary before attempting to express that uncertainty to a human user.
    An AIA must have some way to assess and quantify uncertainty before attempting to express those uncertainties to a human user. \brettcomm{Why? what is the argument for uncertainty helping? In order for AIAs to assure human users, they must be able to identify their own competence boundaries, this involves understanding their own limitations.} There are several different kinds of uncertainty that might be considered, e.g. uncertainty in sensing, uncertainty in planning, and uncertainty in locomotion. A model or method needs to be incorporated in the AIA that will represent the different kinds of uncertainty to the human user in some way. A human user could use such information to inform their trust in the `situational normality', `competence', and/or `predictability' of the AIA. One might imagine that, in the UGV road-network problem, the UGV expressing high uncertainty in its plan would influence the competence component of the user's trust.% Conversely, if an uncertainty measure is not available, the user might take this as an implicit assurance that the AIA is perfectly confident, or (based on the user's experience)  might conclude that since all AIA plans have been flawed in the past, the plan of this AIA must be flawed as well.

    The surveyed literature  featured several different uncertainty quantification approaches. In some cases uncertainty is already represented intrinsically by the algorithms and/or models being used in the AIA. Some use the built-in statistical representations of state transitions and sensor observations, e.g. for POMDPs, as the basis of quantifying uncertainty. This approach has straightforward analogs in other systems that use algorithms and/or models that inherently consider uncertainty. Models and methods that intrinsically represent uncertainty are frequently available. However, even when that it is the case, there are types of uncertainties that may still not be considered. For example, there are additional aspects of uncertainty beyond those intrinsically captured by some optimal planning approaches \cite{Aitken2016-cv,Kuter2015-qh}. Using the UGV road network problem as an example, what is the total probability distribution over all possible cumulative reward outcomes (i.e. beyond just the expected value maximized by the optimal policy)? Or, given a certain road network, how closely can a Monte Carlo-based POMDP solver for the UGV approximate the true optimal policy? These describe `uncertainties in application', which arise in applying certain algorithms and models in different contexts.

    Independent of the algorithms or representations used by an AIA, uncertainty quantification for assurance design also often focuses on assessing the probability of error for decision making as it pertains to any of the AIA capabilities from Fig. \ref{fig:AIcapabilities}. For instance, the problem of assessing the probability of error for classification and regression tasks for learning and perception are well-studied. In addition to standard training/validation data-based assessment techniques, some have approached this problem by learning models of errors for underlying learning models, e.g. by training a GP error `meta-model' for classifier errors  from empirical training data and using the meta-model to quantify uncertainty in different test scenarios \cite{Gurau2016-hs}. In contrast, we might quantify the uncertainty of a classifier based solely on the input data itself \cite{Zhang2014-he}. Of course, these methods share common drawbacks of being solely supervised learning approaches. However even in domains such as reinforcement learning, methods for avoiding highly uncertain (un-safe) rewards must use some form of external expert knowledge (see \cite{Garcia2015-rs, Lipton2016-dq}). Hence, the problem of defining a sensible `ground truth' against which uncertainties can be `correctly' assessed remains a fundamental challenge. %for developing AIA assurances.
    %
    %\edit{...merge this paragraph with end of previous somehow...}
    %    Uncertainty can be easier to assess if some kind of oracle, or reference is available for comparison.

    Some works have proposed quantifying the similarity between empirical experience and available reference data from a truth model as a measure of uncertainty \cite{Kaipa2015-hy}. Of course, this approach loses its appeal when a `truth model' isn't available. This shouldn't detract from the intent of finding some kind of reference (truth or otherwise) in which the reasoning, sensing, and other processes of an AIA can be compared to evaluate uncertainty. The evaluation of statistical models involves a very similar concept: given a statistical model as a reference does current empirical experience support or detract from the hypothesis that the model is still valid \cite{Laskey1995-jp,Ghosh2016-dl}? These approaches can quantify the degree to which the statistical models are still true, and this measurement can be used as an indication of uncertainty. However, these techniques can only provide binary indications that a model is statistically valid/invalid, without providing specific human-understandable causes or justifications as to why. Human expertise and designer judgment is thus required to carefully interpret such assurances, although a universal set of `best practices' for reporting such assurances and coping with cognitive biases in probabilistic/statistical data interpretation (framing effects, etc.) have yet to be established.
    %
    %%This paragraph repeats the first one, so merging in the last part with that and commenting out the rest:
    %%Generally, uncertainty quantification allows AIAs to express assurances related to the `situational normality', `competence', and `predictability' of the system in a given situation. One might imagine that, in the UGV road-network problem, the UGV expressing high uncertainty in its plan would influence the competence component of the user's trust. Conversely, if an uncertainty measure is not available, the user might take this as an implicit assurance that the AIA is perfectly confident, or (based on the user's experience)  might conclude that since all AIA plans have been flawed in the past, the plan of this AIA must be flawed as well.
    %%
\subsubsection{Reducing Complexity}
    Many researchers have attempted to remove complexity from the models and logic of the AIA to make the methods more interpretable (or comprehensible, or explainable, \ldots) to human users. As with uncertainty quantification, interpretability can also inform a user's trust regarding the AIA's `situational normality', `competence', and `predictability'. Of course, this presupposes that many of the methods used by AIAs are `complex' by some measure; we claim that the fact that highly trained and educated experts are required to understand and develop these methods %(and even then it may not be totally possible)
    proves this supposition. Complexity only exists in the presence of some reference frame, i.e. the designer's or the user's in this case.
    From the perspective of an assurance designer, the key challenges to making/discovering/learning assurances with scalable interpretability lie in defining and satisfying criteria for required depth of understanding on the part of the user, the level of user expertise, and the time required by the user to gain understanding.

    Generally, the `complexity' of an algorithm or more general process used by an AIA is said to increase with the number of variables, steps of reasoning, the size of data, etc. in the formal sense that is understood by computer scientists.
    In practice, \textit{reduction} of complexity has been addressed by approaches as simple as finding summary statistics, e.g. calculating averages \cite{Muir1994-ow,Muir1996-gt}, that help explain why/how an algorithm or AIA arrives at a particular result.
    This can also be accomplished by computing heuristic measures which reduce many complex algorithmic components into more manageable pieces\cite{Aitken2016-cv}. Creating variable fidelity models is another way by which complexity can be reduced (and increased when necessary). One might also elect to create and use models to support AIA reasoning, decision-making, etc. capabilities which are inherently more interpretable to humans. This could include constraining the feature space to be more simple, reducing dimensionality, learning more understandable features, and using physical theories to ground learned models (e.g. for interpretable science).  %%\edit{...merge? feels a bit out of place...}

    It is possible for `inherently interpretable models and algorithms' to compete with other non-interpretable models in terms of performance.
    However, this may not be the most viable long-term approach for tackling the problem of complexity reduction in assurance design, especially due to the amount of work required to design interpretable counterparts to the latest state of the art algorithms for implementing AIA capabilities.
    On the other hand, approaches that seek `post hoc' explanations of complex models and algorithms offer more promise.
    This is partly because the idea of `interpretability' is not universally well-defined, and thus does not offer a single tangible goal. Rather, there exists a continuum of interpretability based on the complexity of the problem, the time required for a user to interpret (e.g. a few seconds vs. a few months of study, depending on the application), the expertise of the user, and other factors. However, assurances that can be automatically generated to provide user-specific and model/method-agnostic explanations/interpretations of complex AIA processes would provide the best of both worlds. These ideas are much more aligned with the efforts of \cite{Ruping2006-xj} and others who seek models with scalable resolution and accuracy.

\subsubsection{Assurance by Design}
    No matter how much engineers and system designers like to think about automating everything, humans will realistically need to be involved at some level of AIA operation as well as assurance design for the foreseeable future, since the human-AIA trust cycle cannot be properly managed without taking human inputs into consideration.
    The problems of uncertainty quantification and complexity reduction can largely leverage existing methods to generate explicit assurances `on the fly' as needed for human users.
    We have also seen that several strategies are available to directly engineer AIAs to be as meaningful and transparent to humans as possible from the outset through direct `consequential' forms of interaction.

    One approach is to make the human user and human interpretability an intrinsic part of an AIA's decision making and learning processes, in order to modify their associated objective functions according to human user preferences \cite{Freitas2006-qo,Dragan2013-wd}. \brettcomm{good to add \cite{Huang_undated-ab}}
    Poorly designed objective functions are an important source of discord between how human users expect AIAs to behave and how AIAs actually behave (thus making them less predictable and less trustworthy) \cite{Amodei2016-xi}.
    `Myopic objectives' are said to be present when an AIA focuses on a specific objective to the extent that a human can no longer relate to the objectives of the AIA (so that the user will be correspondingly surprised by its actions). This suggests that significant time may be required to design objectives that align with those of humans, in order to make the AIA more predictable, and competent in the user's eyes.
    %
    `Human-in-the-loop' (HITL) methods can be used to implicitly encode many human qualities that cannot be expressly quantified or explained.
    It is interesting to note that using HITL can offer more interpretability to the result of a learning process, while at the same time making the learning process itself less complex and procedural.
    However, there are trade-offs that can be undesirable in many situations as well, as decision making and learning algorithms are often used to circumvent human biases and imperfect human reasoning.

    Aside from relying on HITL, it is also possible to modify standard decision making and learning approaches (e.g. as discussed in the previous section) to make the methods inherently more transparent and assuring to users. For example, one might restructure a neural net architecture or a decision tree to make it more interpretable \cite{Choi2016-by,Abdollahi2016-vn,Jovanovic2016-gw}. However, there are no universally established principles yet for how this can/should be done.

\subsubsection{Planning Explicit Assurances}
    Computational resources are needed for an AIA to formulate and communicate assurances, as well as assess whether such assurances are eliciting the desired TRBs from users. This naturally raises the question of whether/how AIAs can formulate plans to achieve the goal of effectively and appropriately expressing assurances to a human user.

    As mentioned earlier, not all AIAs are capable of planning.
    Such AIAs can be designed beforehand with some kind of static plan or policy to deliver assurances (e.g. using a fixed set of rules).
    Otherwise, more advanced AIAs might use TRBs as feedback signals to come up with assurance strategies designed to steer users into enacting appropriate TRBs.

    When planning assurances, the AIA must be able to account for limitations of users, and its own limitations in expressing assurances. For example a user may not be able to understand information needed to use the AIA more appropriately. Also, the AIA may need to take a longer-term strategy to teach or tutor the user, as opposed to only presenting the user with canned, static, assurances. Some important user considerations are discussed in Section~\ref{sec:express_assurances}.

    One must ask whether the human user can correctly process the information received. This is perhaps most easily illustrated by considering a non-expert user who cannot understand highly technical assurances regarding the AIA. However, less trivial manifestations, such as the existence of bias in the perception of assurances, or the inability of users to recognize or process important information due to cognitive overload, are also possible. This will also be addressed in Section~\ref{sec:express_assurances}.

    The problem of planning for assurances is nearly unexplored in the context of human-AIA trust relationships. However, there are several fairly new research programs that are interested in this question (i.e. DARPA's Explainable Artificial Intelligence (XAI) program \cite{Gunning2016-kb}) and related ones (e.g. DARPA's Assured Autonomy program \cite{Neema2017-bb}). Assuming an AIA can provide assurances, important questions arise, such as: what is the best way to present them? How can they be adapted for different kinds of users? How can the AIA teach or tutor the human to use the AIA appropriately over time and in different operational contexts? The problem of enabling AIAs to be aware that they must take action and formulate plans to deliver assurances is a large gap in the current assurances landscape; answers to these questions are critical to designing more robust and effective assurances.