\section{Modelling articulated subjects}

% Brief introduction and discuss 'non-shape/weak shape' methods. E.g. faces and hands.
% Discuss methods like dolphins or Angjoo birds which start

% face model paper: https://arxiv.org/pdf/1909.01815.pdf

The design of 3D morphable models (3DMMs) has a significant recent history in computer vision research. A 3DMM is a statistical model designed to represent the structure, deformation and apperance space of for a particular object category. Such a model can be constructed for any object category for which a dense point-to-point correspondence can be established between instances. For example, a 3DMM can be designed to represent medium-sized quadrupeds but perhaps not for general animal categories. How, for instance, would one sensibly determine correspondences between a dog and an octupus? 3DMMs have been used extensively as a strong 3D prior to aid various 3D reconstruction algorithms. They are, however, most influential for problems with the most ambiguity: particularly when dealing with articulated objects (e.g. animals or humans), when only a single monocular RGB image is available or when no paired 3D training data is available.

% cars:
% ears: https://core.ac.uk/download/pdf/158370989.pdf
% human bodies: http://grail.cs.washington.edu/projects/digital-human/pub/allen03space-submit.pdf
% human bodies:

Blanz and Vetter~\cite{blanz-vetter} presented the first 3DMM, which expressed a low-dimensional face space space learnt by aligning various face scans. This work, presented over two decades ago, has been recognized with an impact paper award for the continued applications for the ideas presented. Indeed, the approach introduced has found applications far beyond faces~\cite{face-warehouse, basel-old, basel-new}, including for cars~\cite{deformable-cars}, other human body parts including the hands~\cite{Khamis_2015_CVPR} and ears~\cite{deformable-ears}, the human body surface~\cite{anguelov05scape,loper15smpl} and a restricted set of animal categories~\cite{cafm, zuffi2017menagerie}.

This section will cover methods for modelling articulated subjects, focussing primarily methods for human bodies and animals.

% \subsection{Building 3D morphable models}

%     Of primary concern to this thesis are methods which represent articulated structures, such as human bodies or animals, with a 3D polygon mesh. A polygon mesh $M = (V, T)$ is a collection of vertices, edges bound by vertex pairs, and polygons bound by sequences of edges and vertices~\cite{smith2006vertex}. Although other convex shapes are allowed, this thesis only has need to discuss triangular mesh polygons, which henceforth will be referred to as \emph{triangles}. An example mesh is shown in Figure~\ref{fig:polygon_mesh}.

%         \begin{figure}[H] % Example image
%             \center{\includegraphics[width=0.5\linewidth]{dolphin_mesh}}
%             \caption{A polygon mesh~\cite{polygon_mesh}.}
%             \label{fig:polygon_mesh}
%         \end{figure}

%     \subsection{Mesh deformation}
%     The process of adapting a 3D mesh is known as \textit{mesh deformation} and has relevance to a multitude of computer graphics applications, particuarly those in which models are designed to represent dynamic objects. To constrain an optimization function (or simplify the animation process), it is useful to introduce priors that prevent unnatural mesh movement. Two methods for achieving this are discussed:

%         \subsubsection{As Rigid as Possible}
%         As Rigid as Possible (ARAP) surface deformation~\cite{sorkine2007rigid} is a distance function that measures similarity between two meshes with corresponding vertices. For two vertex sets~$V_{1}$ and~$V_{2}$, ARAP minimizes over~$N = |V|$ rotation matrices. Note~$j \sim i$ indicates vertex indices~$j$ adjacent to vertex index~$i$:

%         \begin{equation}
%             D(V_{1}, V_{2}) = \min_{R_{1..N}}\sum_{i=1}^{N}\sum_{j \sim i}|| (V_{1i} - V_{1j}) - R_{i}(V_{2i} - V_{2j}) ||^{2}
%         \end{equation}

%         This distance function can be incorporated into an energy-based optimizer as a regularization function. By considering how small vertex regions overlap, the function can be used to  discourage `unnatural movement', e.g.\ shearing effects, over mesh faces. ARAP regularizers are particularly useful in cases in which there is no prior knowledge of the mesh. Figure~\ref{fig:arap_dino} shows an example of a dinosaur mesh undergoing ARAP deformation, obtained by translating the highlighted yellow vertex.

%         \begin{figure}[H] % Example image
%             \center{\includegraphics[width=0.35\linewidth]{dino_arap}}
%             \caption{Dinosaur mesh undergoing ARAP deformation, obtained by translating the highlighted yellow vertex. Reprinted from~\cite{sorkine2007rigid}.}
%             \label{fig:arap_dino}
%         \end{figure}

%         \subsubsection{Skeletal Rigging and Linear Blend Skinning}
%         In cases that the mesh shape is known in advance, it is common to follow a process known as \textit{rigging}, in which the mesh is augmented with a hierarchical bone structure. The point at which two bones meet is called a \emph{joint}, and these can be used to define acceptable centres of rotation for mesh deformation. It is possible to describe a distribution of joint configurations, which could be used to constrain the mesh to (in the case of human / animal subjects) anatomically achievable poses. It is also simple to define conceptual `body parts' from a rigged mesh, by considering regions between pairs of joints; for example a lower leg region can be defined between a knee and ankle joint. A simple example of a rigged 2D mesh with joints indicated by green diamonds is shown in Figure~\ref{fig:finger_model}. Note how the mesh surface deforms naturally as the joints are displaced.

%         \begin{figure}[H]
%             \centering
%             \begin{subfigure}{0.48\linewidth}
%             \centering
%                 \includegraphics[width=1\linewidth]{finger/finger1}
%                 \caption{Default joint positions.}
%             \end{subfigure}
%             \begin{subfigure}{0.48\linewidth}
%             \centering
%                 \includegraphics[width=1\linewidth]{finger/finger2}
%                 \caption{Right-most joint displaced.}
%             \end{subfigure}
%             \begin{subfigure}{0.48\linewidth}
%             \centering
%                 \includegraphics[width=1\linewidth]{finger/finger3}
%                 \caption{Central joint displaced and right-most joint displaced and rotated.}
%             \end{subfigure}%
%             \caption{Web application demonstrating LBS on a 2D finger mesh. Joints are denoted as green diamonds.}
%             \label{fig:finger_model}
%         \end{figure}

%         \clearpage
%         Formally, a skinned mesh consists of a set of rigged vertices $V \subseteq \mathbb{R}^3 \times \mathbb{R}^{|J|}$, a set of faces $F \subseteq V^3$ and joints $J \subseteq R^{3\times3}$. Each vertex $v = (x, s) \in V$ consists of positional coordinate $x \in \mathbb{R}^{3}$ and a weight vector $s \in \mathbb{R}^{|J|}$ which describes the level of influence each joint $j \in J$ has over its movement. Many approaches exist for assigning weights, but perhaps the simplest is to build a vector with entries corresponding to the distance from the vertex to each joint centre. Skinning weight vectors are normalized such that their entries sum to one, and for computational reasons, the number of non-zero elements is typically limited to 2 or 4. The weakness of such models is that artifacts and other unrealistic deformations can occur around the model joints, particularly for meshes that model non-linear structures such as humans. However, the technique is frequently used in computer graphics and game design when a character's shape is known ahead of time.

%         To assist in explanation, Figure \ref{fig:rigged_cylinder} shows skinning weight influences from three joints within a rigged cylinder mesh. Here, $|J| = 3$ and each vertex $v_{i} = (x_{i}, s_{i}) \in V$ has a skinning weight vector $s_{i} \in \mathbb{R}^{3}$. Each model joint is assigned a distinct RGB value, shown separately in (a), (b) and (c), and together in (d) by linearly combining the colours. This linear blend colorization scheme will be frequently used in later sections of this report.

%         \begin{figure}[H]
%             \centering
%             \begin{subfigure}{0.25\linewidth}
%             \centering
%                 \includegraphics[width=1\linewidth]{wonky_pole/lower_bone}
%                 \caption{Lower joint.}
%             \end{subfigure}%
%             \begin{subfigure}{0.25\linewidth}
%             \centering
%                 \includegraphics[width=1\linewidth]{wonky_pole/middle_bone}
%                 \caption{Middle joint.}
%             \end{subfigure}%
%             \begin{subfigure}{0.25\linewidth}
%             \centering
%                 \includegraphics[width=1\linewidth]{wonky_pole/upper_bone}
%                 \caption{Upper joint.}
%             \end{subfigure}%
%             \begin{subfigure}{0.25\linewidth}
%             \centering
%                 \includegraphics[width=1\linewidth]{wonky_pole/linear_blend}
%                 \caption{Linear blend.}
%             \end{subfigure}%
%             \caption{A rigged cylinder with $|J| = 3$ and where each vertex $v_{i} = (x_{i}, s_{i}) \in V$ has a skinning weight vector $s_{i} \in \mathbb{R}^{3}$.}
%             \label{fig:rigged_cylinder}
%         \end{figure}

%         Figure \ref{fig:rigged_quadruped} shows a more complex rigged quadruped mesh with $|J| = 25$ with skinning weight influences again shown by the linear blend colorization scheme. Again, each joint is assigned a unique RGB value and a vertex's colour is calculated by linearly combining joint colours with skinning weight vectors given by the $\{s_{i}\}$. A triangle's colour is then generated by averaging the colours given for the three surrounding vertices.

%         \begin{figure}[H] % Example image
%             \center{\includegraphics[width=0.5\linewidth]{linear_blend_bold_bones}}
%             \caption{A rigged quadruped with $|J| = 25$ and where each vertex $v_{i} = (x_{i}, s_{i}) \in V$ has a skinning weight vector $s_{i} \in \mathbb{R}^{3}$. Visualization uses the linear blend colorization scheme in which each joint is assigned a unique RGB value.}
%             \label{fig:rigged_quadruped}
%         \end{figure}

%         Once a mesh has been suitably rigged, there are a number of options (e.g. Linear Blend Skinning (LBS), Dual Quaternions~\cite{kavan2007skinning} etc.) for applying a particular mesh deformation. Typically, a user assigns a transformation (in this case comprising a rotation and transformation) to each `joint' and the updated positions $\bar{x_{i}}$ of the remaining vertices $v_{i}$ with original positions $x_{i}$ are then calculated. The original transformation for each joint (i.e. before the deformation) is expressed as a matrix $U_{j}$. The transformation after the deformation has been applied is captured by $D_{j}$. Note that $s_{ij}$ denotes the skinning weight influence of joint~$j \in J$ on vertex $v_{i} \in V$.

%         The updated positions $\bar{x_{i}}$ can then be calculated by LBS:

%         \begin{equation}
%             \bar{x_{i}} = \sum_{j=1}^{|J|}s_{ij}D_{j}U_{j}^{-1}x_{i}
%         \end{equation}

% \subsection{Modelling human body parts}

% % Hands, faces etc.

% Given the availability of strong shape and pose priors, articulated hand tracking aptly demonstrates the advantage of model fitting approaches. Again, it is first necessary to decide how the human hand should be parameterized, i.e. what an optimizer should specifically aim to learn. Similar to the case with the full human body, the aim is again to adapt a mesh (although this time of a hand) to reproduce a performance given by a real human hand either in still frames or from an input video sequence. Many modern approaches follow a hand parameterization given by Khamis et al.~\cite{Khamis_2015_CVPR} using a pose vector $\theta \in \mathbb{R}^{28}$ that includes global translation and rotation, one adbuction and three flexion variables for each finger digit, and one abduction and flexion parameter for the wrist and forearm. An example hand tracking result can be seen in Figure~\ref{fig:hand_tracking}.

\subsection{Constructing 3D morphable models}

% As it relates to modelling rigged articulated objects, it is important to factor deformation into two categories. Firstly \emph{pose} deformations govern the positioning of articulated parts, for example arms and legs. Consequently, parameters for controlling pose will typically vary when reconstructing a sequence exhibiting an articulated subject in motion. \emph{Shape} deformations, however, control the relative lengths and sizes of articulated parts and indeed the global structure. In general, parameters for controlling shape should be invariant to motion and remain consistent for a single individual.

% A frustrating exception to this occurs in the literature discussing face reconstruction. Since models typicaly  since without a define face skeleton, all shape variations are handling face reconstruction, since without a defined face skeleton, all

% remain consistent In general,  control the relative length and sizing of body parts. In general, only one sizing and length of the model limbs (e.g. arms and legs). typically used to govern the positions of the limbs. are used to govern global and local body proportions (e.g. body part sizes and lengths), and are mostly consistent for a single subject. \emph{Pose deformations} on the other hand are used to govern the positioning of limbs. In order to consolidate various definitions used in the literature, this thesis will define \emph{pose} to be any mesh deformation affected by the movement of an internal skeleton. \emph{Shape} will therefore be any other kind deformation. As an example, then, of most concern to 3D morphable models representing human faces will be \emph{shape} parameters, since face models rarely have an internal skeletal structure. an internal skeleton constitute any deformation change affected by the typically which govern body proportions (e.g. body part sizes, lengths) and \emph{pose deformation} which governs the positioning of limbs. Of course, a typical image of a human or animal will comprise

3D deformable models are typically represented by a polygon mesh. A polygon mesh $M = (V, T)$ is a collection of vertices, edges bound by vertex pairs, and polygons bound by sequences of edges and vertices~\cite{smith2006vertex}. Although other convex shapes are allowed, this thesis only has need to discuss triangular mesh polygons, which henceforth will be referred to as \emph{triangles}. An example mesh is shown in Figure~\ref{fig:polygon_mesh}.

\begin{figure}[t] % Example image
    \center{\includegraphics[width=0.5\linewidth]{dolphin_mesh}}
    \caption{A polygon mesh. Reprinted from~\cite{polygon_mesh}.}
    \label{fig:polygon_mesh}
\end{figure}

A 3D morphable model can then be constructed by deforming a template mesh $M$ on $n$-vertices to a set of 3D training examples. Generally, this optimization will have a significant degrees of freedom so it is necesssary to employ techniques for regularization. One such regularizer is known as the \emph{As Rigid as Possible} scheme:

\begin{definition}[As Rigid as Possible]
As Rigid as Possible (ARAP) surface deformation~\cite{sorkine2007rigid} is a distance function that measures similarity between two meshes with corresponding vertices. For two vertex sets~$V$ and~$W$, ARAP minimizes over~$N = |V|$ rotation matrices. Note~$j \sim i$ indicates vertex indices~$j$ adjacent to vertex index~$i$:

\begin{equation}
    D(V, W) = \min_{R_{1..N}}\sum_{i=1}^{N}\sum_{j \sim i}|| (V_{i} - V_{j}) - R_{i}(W_{i} - W_{j}) ||^{2}
\end{equation}

This distance function can be incorporated into an energy-based optimizer as a regularization function. By considering how small vertex regions overlap, the function can be used to  discourage `unnatural movement', e.g.\ shearing effects, over mesh faces. ARAP regularizers are particularly useful in cases in which there is no prior knowledge of the mesh. Figure~\ref{fig:arap_dino} shows an example of a dinosaur mesh undergoing ARAP deformation, obtained by translating the highlighted yellow vertex.

\begin{figure}[t] % Example image
    \center{\includegraphics[width=0.35\linewidth]{dino_arap}}
    \caption{Dinosaur mesh undergoing ARAP deformation, obtained by translating the a vertex on the nose. Reprinted from~\cite{sorkine2007rigid}.}
    \label{fig:arap_dino}
\end{figure}

\end{definition}

Once aligned, a $d$-dimensional \emph{shape space} can then be defined (with $d << n$), where each $w \in \R{d}$ gives rise to a vertex configuration in $\R{3}$ (with unchanged triangulation). In this way, every plausible 3D example has a parameter vector $w \in \R{d}$ that generates it. This construction can then be interpreted as a \emph{generative} model. However, very few selections of $w \in \R{d}$ will generate a plausible-looking 3D mesh. This can then be interpreted probabilistically, by defining a density function $f(w)$ that defines the likelihood that a realistic 3D example would be represented by $w$ in shape space.

\subsection{Modelling shapes (e.g. faces)}

The concepts raised above were first introduced in the seminal work of Blanz and Vetter~\cite{blanz-vetter}. They define a linear generator function based on principle component analysis (PCA) in order to map $d$-dimensional parameter vectors to the set of $n$ vertex coordinates. In particular they use the mapping:

\begin{equation}
    g(\alpha) = \bar{c} + E\alpha
\end{equation}

where $g: \R{d} \mapsto \R{3n}$ is the generator function, $\bar{c} \in \R{3n}$ is the mean 3D face in the training dataset and $E \in \R{3n \times d}$ is a matrix containing the $d$ most dominant eigenvectors computed over shape residuals $\{c_i - \bar{c_i}\}$.

This construction assumes 3D faces in this $d$-dimensional parameter space follow a multivariate normal distribution (a design decision explored further in this thesis \Cref{chap:wldo}). In addition, the function $f(w)$ which defines the likelihood shape space vector $\alpha$ represents a plausible face, is therefore given by the Mahalanbois distance of $\alpha$ to the origin.

Note that this formulation additionally enables the definition of facial expressions. For example, Blanz and Vetter defined an expression (e.g. surprise) according to the difference in shape space between a expressive and neural face of the same subject. This then enabled the formulation above to be factored into identity and expression components:

\begin{equation}
    g(\alpha_{idt}, \alpha_{exp}) = \bar{c} + E_{idt}\alpha_{idt} + E_{exp}\alpha_{exp}
\end{equation}

where $E_{idt}, E_{exp}$ are the basis vectors of the identity and expression space and $\alpha_{idt}, \alpha_{exp}$ are the coefficients. As noted by Lewis et al.~\cite{xxx}, the basis vectors of the expression space above can be interpreted as a data-driven \emph{blendshape model}: a standard approach in the animation industry for representing facial expressions as a linear combination of target faces. This concept will later reemerge in a section discussing corrective blendshapes uses in SMPL~\cite{loper15smpl}.

%Blendshape Facial Animation
As identified by Blanz and Vetter, improved modelling of finer details (particularly around the eye or nose regions) can be obtained through local modelling. Various authors~\cite{xxx} began manually segmentating the face into parts and learning individual PCA representations for them. Later, segmentations were automatic and learned based on displacement patterns found in the training dataset. Next, approaches were adopted based on hierarchical, multi-scale frameworks~\cite{xxx, xxx}. Possibly the closest to later sections which require a focus on \emph{pose deformation}  is the work of Wu et al~\cite{xxx}. who combine a a local shape space model with an anatomical bone structure that helps regularize deformation.

% booth et al 2017
% pascal 2010
% egger et al 2016b
% zhou et al.
% xu et al (deep 3d portrait from a single image)

A standard challenge in face modelling is towards reconstructing appearance, typically incorporating albedo and illumination (although frequently these are not factored, in which case apperance is generally referred to as \emph{texture}). Early work modelled shape and texture independently~\cite{xxx, xxx}, although recent techniques show solving for these factors jointly enable constraints to be applied due to correlations present. Perhaps most interesting are the recent techniques among these~\cite{xxx, xxx}, who propose methods based on deep convolutional models to jointly model shape and texture.

% PROBLEMS
% the statistics of most models are limited to the face and do not include information on eyes, mouth interior or hair
% Second, the interpretability of the representations would benefit from being improved. PCA is the most commonly used method to perform statistics on 3D faces, and as it is an unsupervised method, the principal components do not coincide with attributes that humans would use to describe a face

\subsection{Modelling articulation (e.g. hands)}

% Perhaps the biggest difference with hand modelling are the complex modes of articulation among subjects.

3D morphable models have also influenced work in 3D hand tracking and modelling. Human hands servce multiple purposes in everyday life, serving a mechanism to handle tools/objects, expressing emotion and aiding (or even as the primary tool for) communication. As a result, hands (and particularly fingers) exhibit complex articulation patterns which are best characterized as 3D \emph{rotations}. Compared to the previous section, in which face shape variation could be represented as as an abstract linear basis learnt from scans, an advantage to modelling hands is the modes of articulation can be defined in advance.

In particular, human hand motion is controlled by a hierarchical bone structure referred to as a \emph{skeleton}. The point at which two bones meet is referred to as a \emph{joint} and can be used to define acceptable centers of articulation. The direction and magnitude of the articulation can then be neatly expressed as a 3D rotation.

This formulation helps provide insight into why the abstract linear basis (shape space) introduced in the previous section would be poor choice for modelling hands. Deformation is here characterized in terms of 3D rotations, and 3D rotations are non-linear with respect to the input angle. This is easily shown:

\begin{definition}[3D Rotations]

The simplest kind of 3D rotation is an \emph{elementary rotation} and involves a rotation around a single axis of a coordinate system. For example, the following matrix represents a rotation by an angle $\gamma$ around the $x$ axis:

\begin{equation}
    R_{x}(\gamma) = \begin{bmatrix}
        1 & 0 & 0 \\
        0 & \cos(\gamma) & -\sin(\gamma) \\
        0 & \sin(\gamma) & \cos(\gamma)
    \end{bmatrix}
\end{equation}

One can then apply this matrix to an input point $p \in \R{3}$ to compute the new position $p'$ after rotating $\gamma$ around the $x$ axis:

\begin{equation}
    p' = R_{x}(\theta)p
\end{equation}

This formulation can then be extended to represent any 3D rotation as the composition of elementary rotations. For example, a 3D rotation can be decomposed into a $\gamma$ rotation around the $x$-axis (pitch), followed by a $\beta$ rotation around the $y$-axis (yaw) and finally by an $\alpha$ rotation around the $z$-axis (roll).

\begin{equation}
    R = R_{z}(\alpha) R_{y}(\beta) R_{x}(\gamma)
\end{equation}

One can see immediately that affecting a 3D rotation (i.e. computing the new position of the points) is a non-linear function of the input angle. It is necessary, therefore to describe an alternative technique for low-dimensional and efficient mesh deformation, which relies on \emph{rigging} and \emph{linear blend skinning}.

\end{definition}

% Compared to facesCompared to faces, hands have a simpler space of shape variation (typically restricted to palm size, finger lengths etc.) but the modes of articulation are extremely complex. In the class of 3DMMs, hand models arguably have the strongest synergy to their biological counterparts; in many cases each of the 27 hand bones (occasionally except the capal bones which join the fingers to the wrists) are explicitly represented in the model skeleton.

\begin{definition}[Skeletal Rigging and Linear Blend Skinning (LBS)]

In cases that the articulated object is known in advance, it is common to augment a representative 3D mesh model with an internal skeleton that approximates the biological counterpart. This is achieved through a process known as \emph{rigging}.

% In general, the mesh skeleton approximates the real, biometric skeleton by allowing the important modes of deformation but often is a rough approximation of the real, biometric skeleton be detailed enough to allow the primary modes of deformation, but often it is not necessary that the 3D mesh skeletthat the articulated object is known In cases that the mesh shape is known in advance, it is common to follow a process known as \textit{rigging}, in which the mesh is augmented with a hierarchical bone structure. The point at which two bones meet is called a \emph{joint}, and these can be used to define acceptable centres of rotation for mesh deformation. It is possible to describe a distribution of joint configurations, which could be used to constrain the mesh to (in the case of human / animal subjects) anatomically achievable poses. It is also simple to define conceptual `body parts' from a rigged mesh, by considering regions between pairs of joints; for example a lower leg region can be defined between a knee and ankle joint. A simple example of a rigged 2D mesh with joints indicated by green diamonds is shown in Figure~\ref{fig:finger_model}. Note how the mesh surface deforms naturally as the joints are displaced.

% \begin{figure}[H]
%     \centering
%     \begin{subfigure}{0.48\linewidth}
%     \centering
%         \includegraphics[width=1\linewidth]{finger/finger1}
%         \caption{Default joint positions.}
%     \end{subfigure}
%     \begin{subfigure}{0.48\linewidth}
%     \centering
%         \includegraphics[width=1\linewidth]{finger/finger2}
%         \caption{Right-most joint displaced.}
%     \end{subfigure}
%     \begin{subfigure}{0.48\linewidth}
%     \centering
%         \includegraphics[width=1\linewidth]{finger/finger3}
%         \caption{Central joint displaced and right-most joint displaced and rotated.}
%     \end{subfigure}%
%     \caption{Web application demonstrating LBS on a 2D finger mesh. Joints are denoted as green diamonds.}
%     \label{fig:finger_model}
% \end{figure}

Formally, a skinned mesh consists of a set of rigged vertices $V \subseteq \R{3} \times \R{|J|}$, a set of faces $F \subseteq V^3$ and joint transformation matrices $J \subseteq \RR{3}{4}$. Each vertex $v = (x, s) \in V$ consists of positional coordinate $x \in \R{3}$ and a weight vector $s \in \R{|J|}$ which describes the level of influence each joint $j \in J$ has over its movement. Many approaches exist for assigning weights, but perhaps the simplest is to build a vector with entries corresponding to the distance from the vertex to each joint centre. Skinning weight vectors are normalized such that their entries sum to one, and for computational reasons, the number of non-zero elements is typically limited to 2 or 4. The weakness of such models is that artifacts and other unrealistic deformations can occur around the model joints, particularly for meshes that model non-linear structures such as humans. However, the technique is frequently used in computer graphics and game design when a character's shape is known ahead of time.

% To assist in explanation, Figure \ref{fig:rigged_cylinder} shows skinning weight influences from three joints within a rigged cylinder mesh. Here, $|J| = 3$ and each vertex $v_{i} = (x_{i}, s_{i}) \in V$ has a skinning weight vector $s_{i} \in \mathbb{R}^{3}$. Each model joint is assigned a distinct RGB value, shown separately in (a), (b) and (c), and together in (d) by linearly combining the colours. This linear blend colorization scheme will be frequently used in later sections of this report.

% \begin{figure}[H]
%     \centering
%     \begin{subfigure}{0.25\linewidth}
%     \centering
%         \includegraphics[width=1\linewidth]{wonky_pole/lower_bone}
%         \caption{Lower joint.}
%     \end{subfigure}%
%     \begin{subfigure}{0.25\linewidth}
%     \centering
%         \includegraphics[width=1\linewidth]{wonky_pole/middle_bone}
%         \caption{Middle joint.}
%     \end{subfigure}%
%     \begin{subfigure}{0.25\linewidth}
%     \centering
%         \includegraphics[width=1\linewidth]{wonky_pole/upper_bone}
%         \caption{Upper joint.}
%     \end{subfigure}%
%     \begin{subfigure}{0.25\linewidth}
%     \centering
%         \includegraphics[width=1\linewidth]{wonky_pole/linear_blend}
%         \caption{Linear blend.}
%     \end{subfigure}%
%     \caption{A rigged cylinder with $|J| = 3$ and where each vertex $v_{i} = (x_{i}, s_{i}) \in V$ has a skinning weight vector $s_{i} \in \mathbb{R}^{3}$.}
%     \label{fig:rigged_cylinder}
% \end{figure}

% Figure \ref{fig:rigged_quadruped} shows a more complex rigged quadruped mesh with $|J| = 25$ with skinning weight influences again shown by the linear blend colorization scheme. Again, each joint is assigned a unique RGB value and a vertex's colour is calculated by linearly combining joint colours with skinning weight vectors given by the $\{s_{i}\}$. A triangle's colour is then generated by averaging the colours given for the three surrounding vertices.

% \begin{figure}[H] % Example image
%     \center{\includegraphics[width=0.5\linewidth]{linear_blend_bold_bones}}
%     \caption{A rigged quadruped with $|J| = 25$ and where each vertex $v_{i} = (x_{i}, s_{i}) \in V$ has a skinning weight vector $s_{i} \in \mathbb{R}^{3}$. Visualization uses the linear blend colorization scheme in which each joint is assigned a unique RGB value.}
%     \label{fig:rigged_quadruped}
% \end{figure}

Once a mesh has been suitably rigged, Linear Blend Skinning (LBS) can be used to apply the mesh deformation. Typically, a user assigns a transformation (e.g. rotation and translation) to each `joint' and LBS computes the an updated position of vertex $v = (x, s)$ where $x$ is the original location and $s = \{s_j\}$ are the skinning weight influence of joints $j$. The matrix $U_{j}$ (occasionally referred to as a the \emph{reference pose}) is the mapping from the bone's "default" coordinate system to world coordinates. As a result, it need only be computed (and inverted) once since it is invariant to changing joint angles. In general, a user will supply the matrices $D_{j}$ to define the mapping from the bone's deformed coordinate system to world coordinates.

The updated position of an input point $x$ can then be calculated by LBS:

\begin{equation}
    LBS(D, x; U, s) = \sum_{j}s_{j}D_{j}U_{j}^{-1}x
\end{equation}

% TODO: Explain LBS for kinematic trees
Note this formulation is made slightly more complicated in the case of kinematic trees, since the

Similarly to the approach mentioned above, this formulation can again be seen as a generative 3D model. In particular, an output vertex positions $v'$ can be computed from a vector of 3D joint rotations $\theta$ and input vertex position $v$ as follows:

\begin{equation}
    v' = LBS(R(\theta), v)
\end{equation}

\end{definition}

With the above formulation, it is now possible to give an overview of recent hand tracking literature. In many cases, 3D morphable hands models are built to reflect the 27 biological hand bones (occasionally except the capal bones which join the fingers to the wrists). Of course, while the major source of hand deformation is due to 3D pose, some sources of variation are present due to hand size and finger propotions.

Allen et al~\cite{xxx} handle this variation adapting a 3D surface with displacement maps with various constraints designed to avoid self-intersections before adopting the linear blend skinning formulation defined above. Rhee et al~\cite{xxx} learn a shape deformation space (with a similar technique to that of faces) and user-specific skinning weights for LBS. Albrecht undergo a laborious process to create extremely detailed hand models through laser scanninng plaster casts~\cite{xxx}. A significant advance was presented by Taylor et al~\cite{xxx}. who presented a method to learn a personalized hand model (although not a shape basis) given an input video sequence (with depth) taken of a user slowly articulating their fingers. Ballan et al.~\cite{xxx} follow a similar process by making use of multi-view input.

% TODO: Allen et al. B. Allen, B. Curless, and Z. Popovic. Articulated body de-formation from range scan data. In ACM Transactions on Graphics (TOG). ACM, 2002. 2

% Represent the model as an adaptation of a standard subdivision surface model with linear blend skinning. Cruicially their adaptations are displacement maps on top of a base surface. The displacements must be limited in magnitude to avoid self-intersections and their shape basis is forced to coincide with the input scans.

% % TODO: Rhee et al. T. Rhee, U. Neumann, and J. P. Lewis. Human hand modeling from surface anatomy. In Proceedings of the 2006 symposium on Interactive 3D graphics and games, pages 27–34. ACM, 2006. 2

% Early work included the work of Rhee et al.~\cite{xxx} who learn a 3D shape deformation space from hands by fitting 3D model with user-specific skinning.

% % TODO: Albrecht et al. I. Albrecht, J. Haber, and H.-P. Seidel. Construction and animation of anatomically based human hand models. In Proc. Eurographics, 2003. 2

% Albrecht et al. go to the other extreme creating very detailed, physically-realistic hand models. However, the process is laborious requiring plaster casting of human hands, performing laser scans, and manually creating a physics-enabled hand model.

% % TODO: Taylor et al. J. Taylor, R. Stebbing, V. Ramakrishna, C. Keskin, J. Shotton, S. Izadi, A. Hertzmann, and A. Fitzgibbon. User-specific hand modeling from monocular depth sequences. In Proc.CVPR, 2014. 1, 2, 4, 5, 7

% A more automatic technique is presented by Taylor et al., which generates personalized hand models given noisy input depth sequences where the user’s hand rotates 180 degrees whilst articulating fingers. A continuous optimization that jointly solves for correspondences and model parameters across a smooth subdivision surface with as rigid as possible (ARAP) regularization leads to high-quality userspecific rigged hand models, though not a shape basis. Whilst the process is automatic, the hands are required to cover the full range of articulations, and longer sequences are required, leading to more complex capture requirements and more costly optimization.

% % TODO: Ballan et al. L. Ballan, A. Taneja, J. Gall, L. V. Gool, and M. Pollefeys. Motion capture of hands in action using discriminative salient points. In Proc. ECCV, 2012. 1, 2

% Ballan et al. construct a personalized hand
% mesh using a multiview camera rig and Poisson surface reconstruction, which is then manually skinned. They demonstrate high-quality results with complex two-handed and
% hand-object interactions, closely fitting the detailed mesh
% model to the data. However, this system focuses on pose
% estimation as opposed to the shape construction, which is
% performed in an time consuming manual manner.

% TODO: Khamis et al. Learning an efficient model of hand shape variation from depth images

% TODO: Fits Like a Glove: Tan et al. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/06/FitsLikeAGlove.pdf

% TODO: Taylor et al. Efficient and Precise Interactive Hand Tracking Through Joint, Continuous Optimization of Pose and Correspondences

\subsection{Modelling the human body surface}

% hasler et al. 2010

However, of all human categories, the works of most relevance to this thesis are those which represent the entire human body surface. It is first important to characterize these two deformation modes which must be overcome with modelling algorithms. Firstly, there is considerable variation in the \emph{shape} characteristics between different human subjects. Humans not only vary in their heights and weights, but also in their body part proportions, muscle density, fat etc. Secondly, humans exhibit significant \emph{pose} variation, characterized by the range of motion of body parts (e.g. arms and legs). In general, pose is likely to change for a individual subject over a sequence.

The earliest deformable 3D models of the human body was presented by Allen at al~\cite{xxx} (although ~\cite{xxx} came soon after with similar ideas). Allen et al. learnt a PCA shape space model from 250 registered body scans cound in the CAESAR dataset. The model was articulated through a set of pose parameters, which use linear blend skinning to interpolate rotation matrices assigned to the joint to transform model veertices. Unfortunately, this approach suffers from artefacts around joint locations, due to a loss of volume. For this reason, it is important to note that pose and shape are not entirely independent; in fact, body shape does indeed change due to pose variation. Imagine for example, how a fatty stomach region would deform during a walking sequence. SCAPE~\cite{anguelov05scape} improved over this by introducing a model equipped with both body shape variation and pose-dependent shape changes, expressed in terms of triangle deformations (rather than vertex displacements, see~\cite{loper15smpl} for a comprehensive overview). An important advance was made by Hasler et al.~\cite{xxx}, who learn two linear blend rigs: one for pose and one for body shape. In this model, shape change was controlled through the introduction of abstract bones that further deform the vertices.

Perhaps the most significant advance however, was the introduction of the Skinned Multi-Person Linear (SMPL) model of Loper et al.~\cite{loper15smpl}. SMPL follows a similar design philosophy to SCAPE by decomposing shape into identity-dependent and pose-dependent components. However, unlike scape, SMPL adopts a vertex-based skinning approach based on corrective blend shapes. The model's shape space is first taught how human beings deform through pose changes using 1786 high-resolution 3D scans of different subjects in a wide variety of poses. Following alignment to a template mesh, a linear model for each biological gender is created from the CAESAR dataset \cite{robinette2002civilian} using principal component analysis (PCA). SMPL can then be viewed as a function, which makes use of a shape basis and linear blend skinning to map a set of pose and shape parameters to a set of vertex locations. Precisely, \emph{pose} is given as a set of 3D rotations (per-joint and global) in axis-angle form $\pose \in \R{24}{3}$. \emph{Shape} is then given as coefficients for a learned shape basis $\shape \in \R{10}$. The SMPL function can then be viewed as:

\begin{equation}
    v = \SMPL(\pose, \shape) + \trans
\end{equation}

where $v \in \RR{6890}{3}$ and $\trans \in \R{3}$ is a global translation parameter. Further details on SMPL have been left to \Cref{chap:3dmulti} of this thesis, which makes use of the model to examine uncertainty when deriving 3D reconstructions of ambiguous input imagery.

% Another key ambition for the SMPL model was the motivatio to create a realistic data-driven human body model which can be rendered in real-time using standard engines, such as Unity~\cite{unity2017} or Blender~\cite{blender2017}. Having been designed for animation, SMPLs base template has a number of useful qualities for this work; the underlying mesh is a clean structure and comprises relatively few polygons. A novelty of this model is that it encodes explicit and meaningful body joint positions. Some sample SMPL meshes are shown in Figure \ref{fig:smpl_model}.

% TODO: GHUML & GHUM model
More recently, SMPL has been combined with face and hand models to add expressive capabilities~\cite{xiang19monocular, joo18total, pavlakos19expressive}. CAPE~\cite{CAPE:CVPR:20} also shows how to add a clothing parameter to effectively model humans in clothing, a challenge solved by learning a shape prior over freeform vertex deformations Techniques have also been developed to model human clothing -- a common challenge generally handled by allowing SMPL model vertices to vary independently to the provided blend shapes. SMPL has also been recently improved with STAR~\cite{STAR:2020} which constructs a part-based shape space (closely related to the local PCA space discussed in the earlier shape section of this literature review). They show this new parameterization is much more efficient (uses approximately 20\% of the model parameters of SMPL) and avoids capturing spurious long-range correlations present in the training dataset. They also show a method for learning shape-dependent pose-corrective blendshapes, that better model how individuals with different body shapes deform with motion. Tangential work of Xu et al.~\cite{ghum-ghuml} train an end-to-end network and learn 3D human body model parameters (including faces and hands) for an input artist model using variational auto-encoders and normalizing flows. This work will be further explored \Cref{chap:3dmulti} in which these generative models will be fully examined.

\begin{figure}[t] % Example image
    \center{\includegraphics[width=0.5\linewidth]{smpl_wbg}}
    \caption{SMPL model showing pose-invariant shape changes, reprinted from~\cite{loper15smpl}.}
    \label{fig:smpl_model}
\end{figure}

\subsection{Modelling animals}

There is still relevatively little work specifically focusing on the 3D scanning~\cite{xxx} and modelling of animal categories. The variation in animal shape and sizes combined with the practical challenges associated with scanning live animal subjects (particularly in attaching traditional motion capture equipment) make scanning a difficult task. As a result, there is a significant lack of real 3D animal training data available in the public domain which could otherwise have been employed to build 3D deformable models. As with humans, animals deformations can again be factored into shape (e.g. variation mostly due to identity) and pose (variation due to articulated motion). However, the enormous diversity among animal species and even between individual breeds results in a much more complex shape space.

Some early work by Favreau et al~\cite{xxx} describe a method for animating an artist-designed rigged 3D model, by tracking a 2D sequence. Chen et al.~\cite{xxx} learn a shape space by registering 11 3D shark models downloaded from the Internet. Cashman et al.~\cite{xxx} learn a morphable model of dolphin shapes by adapting a representative 3D model to 2D images. Ntouskos et al.~\cite{xxx} fit geometric primitives to manually-segemented animal parts generated from an input collection. Reinert et al.~\cite{xxx} demonstrates an effective method for fitting generalized cylinders to an input video sequence supplied with sketched limb tracks. They demonstrate reconstructed results with 3D texture on a few quadruped sequences. So far, none of these techniques for animal reconstruction explicitly factor shape and pose.

\subsubsection{SMAL}
A similar technique to that used to build the SMPL model has been recently used to build a Skinned Multi-Animal Linear Model (SMAL)~\cite{zuffi2017menagerie}, a generative animal model exhibiting realistic 3D shape (see Figure \ref{fig:smal_model_shape}) and pose (see Figure \ref{fig:smal_model_poses}). Due to the lack of available motion capture data for animal subjects, the SMAL model is learnt from a set of $41$ 3D scans of toy figurines in arbitrary poses. The figurines span five quadruped families, and included examples of lions, cats, tigers, dogs, horses, any many more, although notably for this work no rodent toys were included. The paper introduces a new technique to accurately align each toy scan to a common template, allowing the shape space to be learnt.

\begin{figure}[t]
    \centering
    \begin{subfigure}{0.3\linewidth}
    \centering
        \includegraphics[width=1\linewidth]{smal/default}
        \caption{Default SMAL mesh.}
    \end{subfigure}%
    \begin{subfigure}{0.3\linewidth}
    \centering
        \includegraphics[width=1\linewidth]{smal/horse}
        \caption{SMAL in horse shape.}
    \end{subfigure}%
    \begin{subfigure}{0.3\linewidth}
        \centering
            \includegraphics[width=1\linewidth]{smal/lion}
            \caption{SMAL in lion shape.}
    \end{subfigure}%
    \caption{SMAL with varying shape parameters.}
    \label{fig:smal_model_shape}
    \end{figure}

    \begin{figure}[t]
    \centering
    \begin{subfigure}{0.3\linewidth}
    \centering
        \includegraphics[width=1\linewidth]{smal/pose_1}
    \end{subfigure}%
    \begin{subfigure}{0.3\linewidth}
    \centering
        \includegraphics[width=1\linewidth]{smal/pose_2}
    \end{subfigure}%
    \begin{subfigure}{0.3\linewidth}
        \centering
            \includegraphics[width=1\linewidth]{smal/pose_3}
    \end{subfigure}%
    \caption{SMAL with varying pose parameters.}
    \label{fig:smal_model_poses}
\end{figure}

From the paper, SMAL is defined as a function $\SMAL(\pose, \shape)$ parameterized by pose-invariant shape $\shape \in \R{41}$ (again, coefficients of a low-dimensional shape space) and pose $\pose \in \RR{32}{3}$ (including global rotation). There are three pose parameters for each of the $32$ body joints and an additional three to express the global rotation. Global translation $\gamma$ is expressed by a further three parameters. The $\SMAL$ function returns a triangulated surface comprising $6890 \times 3$ vertex locations. \Cref{chap:cgas} and \Cref{chap:wldo} of this thesis make use of the SMAL model in order to reconstruct various quadruped categories.


\section{Related work}

This work is based on the work of Dan, Meshman, Vechev, and Yahav on predicate abstraction for relaxed memory models \cite{danetal13}, and it implements most of the techniques derived by the same. It also borrows heavily from the predicate abstraction rules defined by Ball, Majumdar, Millstein, and Rajamani \cite{balletal01}. Below, we describe these publications and the nature of their influence on this work, apart from general inspiration.

\subsection{Predicate Abstraction for Relaxed Memory Models}

The work by Dan et al. \cite{danetal13} has presented the two abstraction steps this work also employs. It has handled the problem of verifying a program $P$ running on an arbitrary memory model $M$ by embedding a description of said memory model into a program $P_M$, which is in its behaviour equivalent to $P$. The resulting abstraction $P_M$ has then been analysed for predicates to be generated in order to be used as correctness criteria for a successive predicate abstraction, which would then describe the conversion of $P_M$ into a boolean program ready to be verified by a model checker. Furthermore, the aforementioned methods have been implemented for TSO and PSO.\\

It is safe to say that this work primarily seeks to implement the ideas and concepts described in \cite{danetal13}, to reproduce its success in achieving feasibility in its implementation, and to improve on its performance and complexity wherever possible.

\subsection{Automatic Predicate Abstraction of C Programs}

The work by Ball et al. \cite{balletal01} has laid the foundation on which predicate abstraction in this work is implemented. It has described the difficulties model checkers face when confronted with systems with large state spaces, such as software, which is infinite-space. The authors have described the first algorithm to automatically construct a predicate abstraction of programs in C, and they have implemented it in a toolkit, which combined it with other methods such as model checking to statically verify input programs.\\

The final stage of this work, predicate abstraction, implements the algorithm defined by Ball et al., adjusted for the PSO/TSO language we use, and employs some of the optimizations described by the authors.


\section{Xiaodong's inputs}
\paragraph{Homework problem for classifcation}

Give some continuous boundaries, some domains ....

Training data:  some discrete sets within these subdomains.

Validation data:  given a big set of points, find the domain where each point belongs ...


\section{Validation on industrial systems}
\label{sec:exp2}
The choice of the new system was a device under development for a \acs{WPMS}, which configuration is reported in Table \ref{tab:conf2}. This new optical bench was more robust than the previous one: the lasers were more thin and less noisy, while all the components are robustly fixed to the box. Nevertheless, it was dial by two different laser-camera pair. As we said in Chapter \ref{ch:sys_cmp}, it is primary to avoid occlusions when we take the profile of a train wheel, because of the presence of the flange. For this reasons, the target used for the second sets of experiments, was a section of a wheel (referred to as \textit{DIMA} in the following), shown in Figure \ref{fig:exp2-dima}. The measures of interests are reported in Table \ref{tab:exp2-measures}, all of them evaluable with a precision of $0.01 \, mm$. The abbreviations used are:
  \begin{itemize}
    \item \textit{FH} - Flange Height
    \item \textit{FT} - Flange Thickness
    \item \textit{QR} - qR quote
    \item \textit{RW} - Rim Width
    \item \textit{RT} - Rim Thickness
  \end{itemize}

\input{./src/chapters/ch5-Analysis/tab-conf2.tex}

The first step was to compare the results of our calibration algorithm with the ones of the company. Differently from the previous case, the presence of two laser-camera pair required to calibrate both of them, independently one from the other. After that, a second round of calibration was needed to determine the relations to use to merge the two profiles. Note that this scenario gave us some issue later, in particular because of the two pairs
  \begin{figure}
    \centering
    \includegraphics[width=\textwidth]{./images/analysis/exp2/DIMA.jpg}
    \caption{Photo of the DIMA}
    \label{fig:exp2-dima}
  \end{figure}
\vfill
  \input{./src/chapters/ch5-Analysis/tab-exp2-meas.tex}
\vfill
  \begin{figure}
    \centering
    \includegraphics[width=\textwidth]{./images/analysis/exp2/wheel_steps.jpg}
    \caption{Steps used to merged the two profiles}
    \label{fig:exp2-merged}
  \end{figure}
\noindent
which used two different configurations, e.g. in terms of baselines length and triangulation angles. However, once again, all the parameters where comparable between the two algorithms, thus we continued with our analysis.

The second step, was the evaluation of the profile. Using the results of the previous set of experiments, we focused only on the \textit{center of mass}, \textit{Blais\&Rioux} and \textit{FIR} filters, ignoring the others. The windows used where of $16$ and $20$ pixels for each filter. After several attempts we correctly merged the two profiles collected by the two laser-camera pairs. To do that, we first converted the filtered laser in the world reference system, than we superimposed the two profiles and merged them in the same data structure. At the end we smoothed a bit the obtained profile, in order to reduce the effect of noise and the laser thickness, especially along the inner ad outer sides. The result reached is shown in Figure \ref{fig:exp2-merged}. \\

As shown in the last sub-plot, we had to pay attention when using \textsc{MatLab} functions: to increase the precision of this process, we decided to split the wheel's section in many parts, using the keypoints described above as extreme points of these curves; thus, we used a different mathematical model to smooth each part. During this process, it could happen that some gaps appear in the smoothed profile, reducing the precision of points location. As we will see, we can consider these errors negligible in our working conditions. \\

Finally, numerical results are shown. In Table \ref{tab:exp2:refereces} we reported an approximation of the errors committed at the center of the \acs{FOV}. It is already possible to see that, despite the improving given by the filters at pixel level, we haven't a same increase on the world reference system, i.e. in $mm$. Carefully analysing each step of the model, we noticed that the performance decreased on Equation \ref{eq:model:err:disc} and on Equation \ref{eq:det_a}. In the first case it is simple to understand why: the multiplication by the pixel size considerably reduces the weight of the error, especially if the pixel size is in the order of tenths of a millimiter. To be precise, this is not an error reduction, rather a change of scale, from pixel to millimeters; the smaller the pixel is, the smaller is the weight of the error. Regarding the second case, instead, we didn't find a good answer that can explain what happened. From our point of view, we could interpret this situation as an another change of reference system, from the camera to the world. However, the change of unit of measure was performed in case before. The only thing that can help us is that, a variation in the point in the space corresponds to a much smaller variation in the sensor, so the error is more apparent on the latter plane, rather than in the world.
  \input{./src/chapters/ch5-Analysis/tab-exp2-res-pt.tex}

Nevertheless, a comparison from theoretical and real results is shown in Table \ref{tab:exp2:means}. The values are the averages over the five measures mentioned above, evaluated using the points shown in Figure \ref{fig:exp2-cad}. As of the theoretical measures, they are estimated as differences between the values in column \textit{point error} of the Table \ref{tab:exp2:refereces}, but evaluated for the keypoints in figure. As we can see, the model is validated: the variation between theoretical and empirical results is comparable with the theoretical variance, so we can consider the results identical. There are only few consideration to take into account.
  \input{./src/chapters/ch5-Analysis/tab-exp2-res.tex}

  \begin{figure}[t!]
    \centering
    \includegraphics[width=0.88\textwidth]{./images/analysis/exp2/wheel_profile_mod.jpg}
    \caption{Analysed profile, with measure indication, CAD style.}
    \label{fig:exp2-cad}
  \end{figure}
As we said at the beginning of this section, we are using two laser-camera pairs, which results are merged to obtain the profile. The two systems used different configurations, in particular different triangulation angles and baselines. Despite this, differences are very small in terms of absolute values, we noticed that they lead to different output models, i.e. different error approximations. In Table \ref{tab:exp2:refereces}, as in the other phases of our analysis, the values in columns \textit{Y} and \textit{X} are the means of the two models. In this case we consider the means, instead of the error propagation for the variance in Equation \ref{eq:exp:var-prop}, because we didn't know to which piece of the profile the selected point belongs. Statistically, we had a probability of an half, thus we considered the mean as the most appropriate operator.

Another doubt we still had, regards the tuning of the model. Also in this case, we missed some informations, such as the precision of the Scheimpflug angle, or the orientation of the laser (it is a degree of freedom of the system). Thus, we needed to perform a tuning in order to balance the output with the empirical results. Notwithstanding the values used were reasonable with respect to the system structure, we thought that in this way, we mixed some errors due to the model and to the algorithm used to align and merge the two profiles. \\

To conclude, we can say that the model seems to be correct. This last set of tests allowed to underline the fact that it is a good approximation of the real scenarios, however, it has some degrees of freedom that must be analysed to get reasonable results. Also in this case, we remarked the importance of using windows comparable to the laser width, on the contrary the effects noise present in the acquired frames could be great enough to predominate on the laser. As far as the camera is concerned, we used here a sensor with pixels sizes smaller than the camera used in the previous section. As we can see, reducing pixels size reduces considerably the performances of the sub-pixel filters: this is reasonable if we think that a smaller pixel represents a smaller part of the world. \\
Two more considerations before continuing, concerning the resolution of the camera and the effects of noise in the frame. During our analysis we noticed that the empirical errors we obtained were very similar to the nominal resolution of the camera. This observation suggested us to control if a same thing happened in the previous set of experiments, and we noticed that it was quite true. Looking at the noise, however, we have noticed how to change the preprocessing of the image being analysed, changing the values of the empirical errors. More in-depth analyses are discussed in the sections bellow.
%


% \section{Bounded Model Checking}
\section{Encoding Scheme}

The main part of this work is implemented in the \texttt{solve} \CHANGE{mode},
% allowing the verification of concurrent assembly programs by bounded model checking based on register states in our abstract machine model.
% allowing bounded model checking of concurrent assembly programs based on register states in our abstract machine model.
% performing bounded model checking of concurrent software running on our abstract machine model.
allowing verification of concurrent software running on our abstract machine model by the means of bounded model checking \cite{ref:BMC}.
It takes an arbitrary number of programs plus the upper bound \CHANGE{$\BOUND$} as input and encodes them into a finite state machine, expressed as \CHANGE{an} SMT formula where each transition translates to the execution of a single thread.
{\SMTLIB} \cite{ref:SMT-Lib} and the novel {\BTOR} \cite{ref:BTOR2} word level model checking format can be generated, using the theories of arrays, uninterpreted functions and bit-vectors.
To simplify the definition of bad states directly in the program code, the possibility of encountering an exit code greater than zero is checked for per default, but custom properties \CHANGE{may} be defined in a separate file and added with the \texttt{-c} command line parameter.
The resulting formula is then evaluated by a state-of-the-art \CHANGE{SMT} solver.
If it is satisfiable, the corresponding execution trace is extracted from \CHANGE{a generated} model and stored for later inspection.

% \todo{intro
  % \begin{itemize}
    % \item submodule \texttt{solve}
    % \item verification of assembly programs by bounded model checking based on register states
    % \item property: there exists a trace in which $\mathbf{EG^?F^?}(\EXITCODE \neq 0)$ holds (exists globally)
    % \item using satisfiability modulo theories (arrays and bit-vectors)
    % \item problem defined by state and transition variables
    % \item one step corresponds to a specific thread's execution of a single instruction
  % \end{itemize}
% }

% \todo[inline]{Basic Definitions? sorts, functions, etc.}

% \subsection*{Basic Encoding Scheme}
\subsection*{Formal Model}

Let $\BVSORT[n]$ be the fixed size bit-vector sort of width $n$ and $\ASORT[n]$ the array sort with index and element sorts $\BVSORT[n]$.
The following variables are used to encode the machine state at a particular step \CHANGE{$k \leq \BOUND$}, where $k = 0$ is the initial state.

% \renewcommand{\arraystretch}{1.5}
\setlength{\tabulinesep}{3pt}
\begin{longtabu}{llX}
  \firsthline
  % \textbf{Machine} &&\\
  % \hline
  $\HEAP^k$ & $\in \ASORT$ & shared memory \\
  $\EXIT^k$ & $\in \BVSORT[1]$ & exit flag \\
  $\EXITCODE^k$ & $\in \BVSORT$ & exit code \\
  \hline
  % \textbf{Threads} &&\\
  % \hline
  $\ACCU^k_t$ & $\in \BVSORT$ & accumulator register of thread $t$ \\
  $\MEM^k_t$ & $\in \BVSORT$ & \lstASM{CAS} memory register of thread $t$ \\
  $\SBADR^k_t$ & $\in \BVSORT$ & store buffer address register of thread $t$ \\
  $\SBVAL^k_t$ & $\in \BVSORT$ & store buffer value register of thread $t$ \\
  $\SBFULL^k_t$ & $\in \BVSORT[1]$ & store buffer full flag of thread $t$ \\
  $\STMT^k_{t, pc}$ & $\in \BVSORT[1]$ & activation flag for statement at $pc$ of thread $t$ \\
  $\BLOCK^k_{id, t}$ & $\in \BVSORT[1]$ & block flag for checkpoint $id$ of thread $t$ \\
  $\HALT^k_t$ & $\in \BVSORT[1]$ & halt flag of thread $t$ \\
  \lasthline
  \caption{State Variables}
  \label{tbl:states}
\end{longtabu}

\newpage

\newcommand{\READ}{\texttt{read}}
\newcommand{\WRITE}{\texttt{write}}

% Shared memory is modelled using the array variable $\HEAP^k$ in combination with the functions $\READ^k \colon \BVSORT \mapsto \BVSORT$ for retrieving and $\WRITE^k \colon \BVSORT \times \BVSORT \mapsto \ASORT$ for updating the element at $adr \in \BVSORT$ with a given value $val \in \BVSORT$ in step $k$.
% Shared memory in step $k$ is modelled using the array variable $\HEAP^k$ in combination with the functions $\READ^k \colon \BVSORT \mapsto \BVSORT$ for retrieving an element and $\WRITE^k \colon \BVSORT \times \BVSORT \mapsto \ASORT$, returning an updated version of the shared memory state array with the given element set to a specific value.
Shared memory states are modelled using the array variables $\HEAP^k$ in combination with the functions $\READ^k \colon \BVSORT \mapsto \BVSORT$ for retrieving an element and $\WRITE^k \colon \BVSORT \times \BVSORT \mapsto \ASORT$ returning an updated version of the shared memory state array with the given element set to a specific value.
% Its initial state $\HEAP^0$ may contain input data according to a given memory map, but is assumed to be uninitialized in general.
Register states of a thread $t$ are determined by the bit-vector variables $\ACCU^k_t$, $\MEM^k_t$, $\SBADR^k_t$, $\SBVAL^k_t$ and the flag $\SBFULL^k_t$, signalling that the store buffer is full.
%, which are all initially set to zero.
% To aid solvers by reducing the formulas complexity, program flow is modelled without an explicit \CHANGE{program counter ``(PC)''}.
\CHANGE{To reduce the formula's complexity}, program flow is modelled without an explicit \CHANGE{program counter (``$pc$'')}.
Instead, an activation flag $\STMT^k_{t, pc}$ is added for every statement in the program of thread $t$, expressing that it is about to execute the statement at $pc$.
%To model the program flow without an explicit program counter, an activation flag $\STMT^k_{t, pc}$ is added for every statement in the program of thread $t$, expressing that it is about to execute the statement at $pc$.
Blocking a thread $t$ while it is waiting for all other threads to reach a checkpoint $id$ is achieved by a flag $\BLOCK^k_t$.
% Blocking a thread $t$ while waiting for all other threads to synchronize on a checkpoint $id$ is achieved by the $\BLOCK^k_{id, t}$ state flag.
Similarly, the flag $\HALT^k_t$ indicates that thread $t$ executed a \lstASM{HALT} instruction and is therefore also prevented from being scheduled.
% Termination due to an call to \texttt{EXIT}, or because all threads finished executing their programs, or \texttt{EXIT}  is captured by
Termination is captured by the flag $\EXIT^k$ and bit-vector variable $\EXITCODE^k$.

% The shared memory state array $\HEAP^0$ may be initialized with input data according to a given memory map, but is assumed to be uninitialized in general.
% All other states are initially set to zero, with the exception of the initial statement activation flag $\STMT^0_{t, 0}$ of every thread $t$.
All states are initially set to zero, except the initial statement activation flag $\STMT^0_{t, 0}$ of every thread $t$ and the shared memory state array $\HEAP^0$, which may be initialized with input data according to a given memory map, but is assumed to be uninitialized in general.

\bigskip

% Constraints for valid machine state transitions of the form $s_k \to^k s_{k + 1}$ are governed by the following variables.
Machine state transitions of the form $s_k \to^k s_{k + 1}$ are \CHANGE{encoded} by the following free variables.
% Machine state transitions of the form $s_k \to^k s_{k + 1}$ are governed by the free variables given in Table \ref{tbl:encoding:transitions}.
% Transitions $s_k \to^k s_{k + 1}$ of a state $s_k$ are governed by the following free variables and are allowed to be chosen non-deterministically.

\begin{longtabu}{llX}
  \firsthline
  $\THREAD^k_t$ & $\in \BVSORT[1]$ & thread $t$ is scheduled to execute an instruction in step $k$ \\
  $\FLUSH^k_t$ & $\in \BVSORT[1]$ & thread $t$ flushes its store buffer in step $k$ \\
  \lasthline
  \caption{Transition Variables}
  \label{tbl:encoding:transitions}
\end{longtabu}

% \todo[inline]{helper variables}
To simplify the definition of successor states, the following helper variables capture frequently used \CHANGE{conditions}.
% The following helper variables are introduced to simplify the latter definition of state transitions by capturing frequently used terms.

\begin{longtabu}{llX}
  \firsthline
  $\EXEC^k_{t, pc}$ & $\in \BVSORT[1]$ & thread $t$ is executing the statement at $pc$ in step $k$ \\
  $\CHECK^k_{id}$ & $\in \BVSORT[1]$ & all threads reached checkpoint $id$ in step $k$ \\
  \lasthline
  \caption{Helper Variables}
\end{longtabu}

% The statement execution variable $\EXEC^k_{t, pc}$,
The actual execution of a specific statement is encoded by $\EXEC^k_{t, pc}$ and
% The actual execution of a statement at $pc$ by thread $t$ in step $k$ is encoded by the execution variable $\EXEC^k_{t, pc}$ and
is defined as a conjunction of the corresponding statement and thread activation variables.

\[
  \EXEC^k_{t, pc} = \STMT^k_{t, pc} \land \THREAD^k_t
\]

\bigskip

Furthermore, we use $\CHECK^k_{id}$ to signal that all threads containing a call to checkpoint $id$ \CHANGE{(given in the set $C_{id} = \{ \, t \mid t \text{ contains \lstASM{CHECK} } id \, \}$)} have synchronized.% and the corresponding $\BLOCK^k_{id, t}$ flags set.
% from a set $C_{id}$ of threads containing a call to checkpoint $id$
\[
  \CHECK^k_{id} = \bigwedge_{t \in C_{id}} \BLOCK^k_{id, t}
\]

% \bigskip
\subsubsection*{Scheduling}

\newcommand{\CardLt}{\leq^1_n(x_1, \ldots, x_n)}
\newcommand{\CardLtSeq}{\text{LT}^{n, 1}_{\text{SEQ}}}

\newcommand{\ITE}{\texttt{ite}}
\newcommand{\ITEindent}{\;\;\;\;\;\;\;}

% Non-deterministic scheduling of a single thread per step is realized by a boolean cardinality constraint over all $\THREAD^k_t$ and $\FLUSH^k_t$ variables for a number of threads $n$.
Non-deterministic scheduling of at most one thread per step is realized by a boolean cardinality constraint over all transition variables and the exit flag $\EXIT^k$ to ensure satisfiability if the machine terminates in a step \CHANGE{$k < \BOUND$}.
Let $\CardLt$ be a predicate expressing that at most one out of $n$ variables is allowed to be true.
% The na\"{i}ve way of defining $\leq^1_n(x_1, \ldots, x_n)$ is by explicitly excluding all combinations of two variables being simultaneously true.
The intuitive way of encoding $\CardLt$ is by excluding all combinations of two variables being simultaneously true.
% The traditional way of defining such a constraint is by excluding all combinations of two variables being simultaneously true:
\[
  \bigwedge_{1 \leq i < j \leq n} (\neg x_i \lor \neg x_j)
\]
This na\"ive approach, however, consists of $\binom{n}{2}$ \CHANGE{binary} clauses.
% While this na\"ive approach consists of $\binom{n}{2}$ Horn clauses, a more compact formulation, based on a sequential counter circuit, is presented as $\text{LT}^{n, 1}_{\text{SEQ}}$ in \cite{ref:Sinz}.
A more compact formulation, based on a sequential counter circuit computing partial sums $s_i = \sum^i_{j = 1} x_j$ for increasing values of $i$ up to the final $i = n$, is presented as $\CardLtSeq$ in \cite{ref:Sinz} and defined as follows.
% It only requires $3n - 4$ clauses in contrast to the na\"ive approach
% It is based on a sequential counter circuit, only requiring $3n - 4$ clauses, but $n - 1$ additional auxiliary variables.
% Beside this na\"ive approach, consisting of $\binom{n}{2}$ Horn clauses, a more compact formulation with respect to the number of clauses is presented as $\text{LT}^{n, 1}_{\text{SEQ}}$ in \cite{ref:Sinz}.
% This approach consists of $\binom{n}{2}$ Horn clauses.
% consisting of $\binom{n}{2}$ Horn clauses.
% A more compact formulation in terms of the number of clauses, called , based on a sequential counter circuit, is  presented in \cite{ref:Sinz} as $\text{LT}^{n, 1}_{\text{SEQ}}$.
% Cardinality constraint predicate $\leq^1_n(x_1, \ldots, x_n)$ defined as:
\[
  (\neg x_1 \lor s_1) \land (\neg x_n \lor \neg s_{n-1}) \bigwedge_{1 < i < n} \big ( (\neg x_i \lor s_i) \land (\neg s_{i-1} \lor s_i) \land (\neg x_i \lor \neg s_{i-1}) \big )
\]
% $\CardLtSeq$ only requires $3n - 4$ Horn clauses and $n - 1$ additional auxiliary variables.
% It is therefore superior to the na\"ive encoding with regard to the number of clauses for all $n > 5$.
$\CardLtSeq$ is superior to the na\"ive encoding with regard to the number of clauses for all $n > 5$, as it only requires $3n - 4$ \CHANGE{binary} clauses and $n - 1$ additional auxiliary variables.
% \todo[inline]{constraint selection}
% Since we need to include two times the number of threads plus one variables in the constraint, the actual definition of $\leq^1_n(x_1, \ldots, x_n)$ is therefore determined by the number of threads involved and the na\"ive encoding only used for up to two threads, $\CardLtSeq$ otherwise.
The actual definition of the at most one constraint predicate $\CardLt$ is therefore determined by the number of threads involved.
Since we have to include two times the number of threads plus one variables in the constraint, the na\"ive encoding is only used for up to two threads and $\CardLtSeq$ otherwise.
% \[
  % \leq^1_n(x_1, \ldots, x_n) \equiv
    % \begin{cases}
      % \CardLtSeq & \text{if } n > 5 \\
      % \text{na\"ive} & \text{otherwise}
    % \end{cases}
% \]
% \todo[inline]{exactly one constraint}
An at most one constraint alone is not sufficient, as we need exactly one transition variable or the exit flag to be true in every step.
\CHANGE{This is because} our generated formula \CHANGE{should} not \CHANGE{be} trivially satisfiable by never scheduling a single thread.
Thus, we define the exactly one constraint predicate $=^1_n\!(x_1, \ldots, x_n)$ by simply adding a disjunction over all variables.% to $\CardLt$.
% The required exactly one constraint predicate $=^1_n(x_1, \ldots, x_n)$ as a conjunction of $\leq^1_n(x_1, \ldots, x_n)$ with a disjunction over all variables
% Let $=^1_n(x_1, \ldots, x_n)$ therefore be an exactly one constraint predicate.
\[
  %=^1_n(x_1, \ldots, x_n) \equiv
  (x_1 \lor \ldots \lor x_n) \; \land \CardLt
  % \CardLt \land (x_1 \lor \ldots \lor x_n)
\]
% \todo[inline]{actual constraint added}
% If we now redeclare $n$ to be the number of threads, the exactly one constraint used for non-deterministic scheduling of a single thread in step $k$ looks as follows.
If we redeclare $n$ as the number of threads, non-deterministic scheduling of a single thread in step $k$ can now be encoded by the following constraint.
% We are now able to state the actual exactly one constraint added to our formula as follows.
\[
  =^1_{2n + 1}\!(\THREAD^k_0, \ldots, \THREAD^k_{n-1}, \FLUSH^k_0, \ldots, \FLUSH^k_{n-1}, \EXIT^k)
\]
% \todo[inline]{scheduling influenced by constraints}
% restrictions by our machine model.
% To comply with our machine model, the cardinality constraint influenced under certain conditions.
% The scheduling constraint is influenced by  certain transitions
% influenced by disabling transition variables under certain conditions.
% to comply with our machine model,
% This cardinality constraint is further influenced by explicitly disabling transitions from certain states in order to meet the requirements of our machine model.

This cardinality constraint is further influenced by explicitly disabling transitions from certain states that are prohibited by our machine model.
% \todo[inline]{store buffer constraints}
% Blocking a thread $t$ not performing a store operation.
% Flushing an empty store buffer of a thread $t$, not containing any barrier \todo{atomic?} operation, can be prevented by a simple implication.
Flushing an empty store buffer of a thread $t$ can be prevented by a simple \CHANGE{relational constraint}.
\[
  \FLUSH^k_t \implies \SBFULL^k_t
\]
% If thread $t$ contains a write together with a barrier operation, execution of the latter has to be restricted while the store is full.
In case thread $t$ containing a write, execution of any barrier operation has to be delayed while the store buffer is full.
% In this case, both mutually exclusive store buffer constraints are combined in a single expression by using
% Both store buffer constraints may be combined in a single expression by using $\ITE: \BVSORT[1] \times \BVSORT[n] \times \BVSORT[n] \to \BVSORT[n]$, a functional if-then-else, returning the $a \in \BVSORT[n]$ if $x \in \BVSORT[1]$ is \emph{true}, else $b \in \BVSORT[n]$.
% If a write or barrier operation
% Otherwise,
Let $F_t$ be a set of statements requiring an empty store buffer
and $\ITE: \BVSORT[1] \times \BVSORT[n] \times \BVSORT[n] \to \BVSORT[n]$ be a functional if-then-else, returning $a \in \BVSORT[n]$ if $x \in \BVSORT[1]$ is \emph{true}, else $b \in \BVSORT[n]$.
% Let $\ITE: \BVSORT[1] \times \BVSORT[n] \times \BVSORT[n] \to \BVSORT[n]$ be a functional if-then-else, returning $a \in \BVSORT[n]$ if $x \in \BVSORT[1]$ is \emph{true}, else $b \in \BVSORT[n]$
\[
  \ITE(x, a, b) =
  \begin{cases}
    a \text{ if } x \text{ is } true \\
    b \text{ otherwise}
  \end{cases}
\]
% Blocking a thread $t$ about to execute a statement that requires the store buffer being empty.
% Set of statements requiring an empty store buffer $F_t$.
Since both store buffer related constraints mainly depend on mutually exclusive values of $\SBFULL^k_t$, we are able to encode them in a single expression.
\[
  \ITE(\SBFULL^k_t, ( \bigvee_{pc \in F_t} \STMT^k_{t, pc} \implies \lnot \THREAD^k_t \,), \lnot \FLUSH^k_t)
\]
% \todo[inline]{checkpoint constraints}
% Blocking a thread $t$, while it is waiting for all other threads reaching a checkpoint $id$ is expressed by a conjunction of the corresponding block flag $\BLOCK^k_{id, t}$ and helper variable $\CHECK^k_{id}$ implying a negated thread activation $\THREAD^k_t$.
Blocking a thread $t$, while it is waiting for all other threads reaching a checkpoint $id$ is implied by a conjunction of the corresponding block flag $\BLOCK^k_{id, t}$ and synchronization variable $\CHECK^k_{id}$.
% \todo[noline]{unsat if deadlock by CHECK}
\[
  \BLOCK^k_{id, t} \land \lnot \CHECK^k_{id} \implies \lnot \THREAD^k_t
\]
% \todo[inline]{halt constraints}
% Finally, a halted thread $t$ is also excluded from the cardinality constraint by disabling its activation variable.% by a simple implication.
% If a thread $t$ has finally halted, it is also excluded from the cardinality constraint.
Finally, if a thread $t$ has halted, it must also be stopped from being scheduled.
\[
  \HALT^k_t \implies \lnot \THREAD^k_t
\]

\CHANGE{
% If any of these conditions are violated, this does not correspond to a valid execution.
The purpose of this conditions is to restrict transitions which do not correspond to a valid execution.
% Since these constraints are defined relationally, special care must be taken such that they do not result in an unsatisfiable formula due to a violation of the primary cardinality constraint.
Since these constraints are defined relationally, special care must be taken in order to prevent a violation of the cardinality constraint, causing the formula to be unsatisfiable if all included variables are falsified simultaneously.
% Considering the previously defined constraints, the only way this situation might occur is if every thread's store buffer is empty ($\lnot \SBFULL^k_t \implies \lnot \FLUSH^k_t$) and
% all threads that are still running have their block flags enabled in case all are waiting for different checkpoint,
% all running threads are waiting for a different checkpoint, causing a \emph{deadlock} i
% at least a single thread $t$ is still running in step $k$ ($\lnot \HALT^k_t \implies \lnot \EXIT^k$).
Considering the previously defined constraints, the only way this situation might occur is if a \emph{deadlock} is introduced by an unfortunate interleaving of different \lstASM{CHECK} instructions, resulting in each thread waiting for another.
% This problem, common for all synchronization primitives, leads to alternating block flags $\BLOCK^k_{t, id}$ being set, violating the conditions of corresponding synchronization variables $\CHECK^k_{id}$ and thread activation variables $\THREAD^k_t$ blocked by the associated constraint.
% In this case, alternating block flags $\BLOCK^k_{id, t}$ are enabled, falsifying the corresponding synchronization variables $\CHECK^k_{id}$ and every running thread's activation $\THREAD^k_t$ being explicitly disabled by the relevant checkpoint constraint.
% This leads to alternate block flags $\BLOCK^k_{id, t}$ being enabled, making it impossible to fulfill the condition of corresponding synchronization variables $\CHECK^k_{id, t}$.
% preventing the condition of corresponding synchronization variables $\CHECK^k_{id}$
% In this case, $\EXIT^k$ will be \emph{false} and since all running threads are prevented from being scheduled by having their activation variables $\THREAD^k_t$ disabled through each individual block state $\BLOCK^k_{t, id}$ being enabled, while none of the corresponding sync
% Let $T$ be the set of threads involved and $C$ the set of checkpoint IDs, then the condition (state) causing the formula to be unsatisfiable can be summarized as follows.
% In this case, let $T$ be the set of threads involved and $C$ the set of checkpoint IDs, then the condition causing
% the scheduling constraints to be violated
% the formula to be unsatisfiable
Let $T$ be the set of threads involved and $C$ the set of checkpoint IDs, then the condition causing unsatisfiability can be summarized as follows.
% \[
  % \exists \, k \leq \BOUND, \forall \, t \in T \,[\, \lnot \SBFULL^k_t \land \lnot \HALT^k_t \land \exists \, id \in C \,[\, \BLOCK^k_{id, t} \land \lnot \CHECK^k_{id} \,]]
% \]
% \[
  % \exists \, k \leq \BOUND : \lnot \EXIT^k \land \forall \, t \in T : \lnot \SBFULL^k_t \land \lnot \HALT^k_t \land \exists \, id \in C : \BLOCK^k_{id, t} \land \lnot \CHECK^k_{id}
% \]
% \begin{align*}
  % & \lnot \EXIT^k \\
  % & \land \exists \, k \leq \BOUND : \forall \, t \in T : \lnot \SBFULL^k_t \land \lnot \HALT^k_t \\
  % & \land \exists \, id \in C : \BLOCK^k_{id, t} \land \lnot \CHECK^k_{id}
% \end{align*}
\begin{equation*}
  \begin{gathered}
  \exists \, k \leq \BOUND : \lnot \EXIT^k \\
  \land \\
  \forall \, t \in T : \lnot \SBFULL^k_t \land \lnot \HALT^k_t \\
  \land \\
  \exists \, id \in C : \BLOCK^k_{id, t} \land \lnot \CHECK^k_{id}
  \end{gathered}
\end{equation*}
% \[
  % \lnot \SBFULL^k_t \land \lnot \HALT^k_t \land \BLOCK^k_{id, t} \land \lnot \CHECK^k_{id} \implies \lnot \EXIT^k \land \lnot \FLUSH^k_t \land \lnot \THREAD^k_t
% \]
}

\subsubsection*{Memory Access}

% \newcommand{\READ}{\texttt{read}}
\newcommand{\LOAD}{\texttt{load}}

% Let $\READ^k: \BVSORT \to \BVSORT$ be a function, returning a given element from array $\HEAP^k$.
% Due to store forwarding in combination with indirect addressing, memory access can not be expressed by simple array lookups, but turns out to be the most complicated part in our encoding scheme.
Due store forwarding, memory access can not be expressed as simple array lookup, but is encoded by a separate function $\LOAD^k_t: \BVSORT \to \BVSORT$ for loading the shared memory element at address $adr \in \BVSORT$ with store forwarding from thread $t$. % and indirect addressing signalled by $indirect \in \BVSORT[1]$, being defined as follows.
\begin{align*}
  \LOAD^k_t(adr) = \ITE(& \SBFULL^k_t \land \SBADR^k_t = adr, \\
  & \SBVAL^k_t, \\
  & \READ^k(adr))
\end{align*}
In case of indirect addressing, $\LOAD^k_t$ is redefined to prevent dependency on certain features of the target language or the use of additional auxiliary variables.
% \todo[noline]{repeated, iterative, recursive?}
\begin{align}
  \LOAD^k_t(adr) = \ITE(& \SBFULL^k_t, \label{def:load:1} \\
  & \ITE(\SBADR^k_t = adr, \label{def:load:2} \\
  & \ITEindent \ITE(\SBVAL^k_t = adr, \label{def:load:3} \\
  & \ITEindent \ITEindent \SBVAL^k_t, \label{def:load:4} \\
  & \ITEindent \ITEindent \READ^k(\SBVAL^k_t)), \label{def:load:5} \\
  & \ITEindent \ITE(\SBADR^k_t = \READ^k(adr), \label{def:load:6} \\
  & \ITEindent \ITEindent \SBVAL^k_t, \label{def:load:7}\\
  & \ITEindent \ITEindent \READ^k(\READ^k(adr)))), \label{def:load:8} \\
  & \READ^k(\READ^k(adr))) \label{def:load:9}
\end{align}
% \todo[noline]{use tabbing evironment?}
First, we check if the store buffer contains an entry and store forwarding might apply (\ref{def:load:1}).
%, or the requested value can be directly retrieved from memory (\ref{def:load:9}).
If it is empty, the requested value has to be directly retrieved from memory (\ref{def:load:9}).
% Otherwise, if the immediate address can be forwarded (\ref{def:load:2}), we must further check if it is equal to the effective address (\ref{def:load:3}) and either both (\ref{def:load:4}), or just the immediate address is forwarded (\ref{def:load:5}).
Otherwise, if the store buffer contains an entry for the given address (\ref{def:load:2}), we must further check if it is equal to the effective address (\ref{def:load:3}) to determine if either both (\ref{def:load:4}), or just the given address can be forwarded (\ref{def:load:5}).
% Otherwise, if the store buffer contains an entry for the immediate address (\ref{def:load:2}), we check if it is equal to the effective address (\ref{def:load:3}) and either return the store buffer value (\ref{def:load:4}) or the element at
% Otherwise, if the store buffer contains an entry for the immediate address (\ref{def:load:2}) and
% Finally, in case the given address could not be forwarded, the store buffer might contain an entry
% Finally, in case the given address was not contained,
Finally, in case there is no entry for the given address,
%we test if and either the effective address can be forwarded (\ref{def:load:7}), or nothing after all (\ref{def:load:8}).
the effective address is fetched from memory and compared to the store buffer address (\ref{def:load:6}) to see if it
% can be forwarded (\ref{def:load:7}), or nothing after all (\ref{def:load:8}).
% might be subject to store forwarding (\ref{def:load:7}), or nothing after all (\ref{def:load:8}).
might be contained (\ref{def:load:7}), or nothing can be forwarded after all (\ref{def:load:8}).
% fetch the effective address from memory and
% or nothing after all.
% and nothing could be forwarded after all.
% If the immediate address cannot be forwarded,

% If store forwaring does not apply because either the store buffer being empty or it neither contains the immediate nor the effective address,
% In case of indirect addressing, we first need to check if the store buffer is full (\ref{def:load:indirect:sbfull}) and the given base address can be forwarded (\ref{def:load:indirect:sbadr}).
% If a value for the base address is contained in the store buffer, we then check if this value

% \begin{align}
  % \LOAD^k_t(adr, indirect) = \ITE(& indirect, \\
  % & \ITE(\SBFULL^k_t, \label{def:load:sbfull} \\
  % & \ITEindent \ITE(\SBADR^k_t = adr, \label{def:load:indirect:sbadr} \\
  % & \ITEindent \ITEindent \ITE(\SBVAL^k_t = adr, \\
  % & \ITEindent \ITEindent \ITEindent \SBVAL^k_t, \\
  % & \ITEindent \ITEindent \ITEindent \READ^k(\SBVAL^k_t)), \\
  % & \ITEindent \ITEindent \ITE(\SBADR^k_t = \READ^k(adr), \\
  % & \ITEindent \ITEindent \ITEindent \SBVAL^k_t, \\
  % & \ITEindent \ITEindent \ITEindent \READ^k(\READ^k(adr)))), \\
  % & \ITEindent \READ^k(\READ^k(adr))), \\
  % & \ITE(\SBFULL^k_t \land \SBADR^k_t = adr, \\
  % & \ITEindent \SBVAL^k_t, \\
  % & \ITEindent \READ^k(adr)))
% \end{align}

% \begin{itemize}
  % \item Nested array lookups if the store buffer is empty and store forwarding does not apply alltogether, or neither the indirect, nor the effective address can be forwarded.
  % \item The value contained in the store buffer if either the effective or both addresses can be forwarded.
% \end{itemize}

% store buffer empty: nested lookup

% \begin{enumerate}
  % \item nothing can be forwarded
  % \item only the indirect address can be forwarded
  % \item only the effective address can be forwarded
  % \item both addresses can be forwarded
% \end{enumerate}

% \[
  % \LOAD^k_t(adr) = \ITE(& \SBFULL^k_t \land \SBADR^k_t = adr, \SBVAL^k_t, \READ^k(adr)))
% \]

% \ref{def:load:sbfull}

% \paragraph{Store Forwarding:} $\sbfull \; \land \; \sbadr = adr \land \accu = \sbval \lor \lnot (\sbfull \land \sbadr = adr)$

% \bigskip

\subsubsection*{Frame Axioms}

\newcommand{\TRUE}{\textit{true}}
\newcommand{\FALSE}{\textit{false}}

\newcommand{\EFFECTIVE}{\texttt{effective}}
\newcommand{\MSB}{\texttt{msb}}

% The effect of a particular operation, determined by the transition variables $\FLUSH^k_t$ and $\EXEC^k_{t, pc}$
% The effect of a thread $t$ flushing its store buffer or executing the instruction at $pc$ in step $k$
Successor states of transitions for all possible types of operations, determined by $\FLUSH^k_t$ and $\EXEC^k_{t, pc}$, are defined by the frame axioms in the table below.
% \todo[inline]{block}
All state variables, not explicitly altered are assumed to be unchanged in the next step, except the block flags $\BLOCK^{k + 1}_{id, t}$, which are reset if all threads synchronized upon checkpoint $id$.
\[
  \BLOCK^{k + 1}_{id, t} = \ITE(\CHECK^k_{id}, \FALSE, \BLOCK^k_{id, t})
\]
To further simplify the definition of axioms, two additional functions are introduced:
% Let $\EFFECTIVE^k: \BVSORT \to \BVSORT$ be a function for transparently selecting the effective address during $\texttt{STORE}$ or $\texttt{CAS}$ instructions.
$\MSB: \BVSORT \to \BVSORT[1]$ for retrieving the most significant bit of a given bit-vector and $\EFFECTIVE^k: \BVSORT \to \BVSORT$ for transparently selecting the effective address during \lstASM{STORE} or \lstASM{CAS} instructions.
\vspace{-0.1cm}
\[
  \EFFECTIVE^k(adr) =
  \begin{cases}
    \READ^k(adr) \text{ if indirect} \\
    adr \text{ otherwise}
  \end{cases}
\]
% and $\MSB: \BVSORT \to \BVSORT[1]$ a function for retrieving the most significant bit of a given bit-vector.

% $\SLICE: \BVSORT \times \mathcal{N} \times \mathcal{N} \to \BVSORT[m]$

% \newcommand{\defaxiom}[1]{
    % \vspace{-0.4cm}
    % {\begin{align*}#1\end{align*}}
    % \vspace{-1.3cm}
% }
% \begin{longtabu}{lX[lm]}
  % \firsthline
    % \texttt{FLUSH} &
    % \vspace{-0.4cm}
    % {\begin{align*}
      % \HEAP^{k + 1} &= \WRITE^k(\SBADR^k_t, \SBVAL^k_t) \\
      % \SBFULL^{k + 1}_t &= false \\
    % \end{align*}}
    % \vspace{-1.3cm}
    % \\
  % \hline
  % \texttt{LOAD adr} &
    % \defaxiom{
      % \ACCU^{k + 1}_t &= \LOAD^k_t(\texttt{adr}) \\
      % \STMT^k_{t, pc} &= false \\
      % \STMT^k_{t, pc + 1} &= true \\
    % } \\
  % \hline
  % foo & bar \\
  % \lasthline
  % \caption{Frame Axioms}
  % \label{tbl:axioms}
% \end{longtabu}

% \doublerulesep=6pt
% \arrayrulewidth=1pt
% \doublerulesep=1.5 pt
% \everyrow{\hline\hline}

% \newcommand{\defaxiom}[2]{
  % \paragraph{#1} ~\\
  % \rule[0.5\baselineskip]{\textwidth}{0.1pt}\vspace{-1\baselineskip}\par\noindent
  % #2
% }

% \newpage
% \defaxiom
% {\texttt{FLUSH}}
% {
  % \begin{itemize}
    % \item[] $\HEAP^{k + 1} = \WRITE^k(\SBADR^k_t, \SBVAL^k_t)$
    % \item[] $\SBFULL^{k + 1} = false$
  % \end{itemize}
% }

% \defaxiom
% {\texttt{LOAD arg}}
% {
  % \begin{itemize}
    % \item[] $\ACCU^{k + 1}_t = \LOAD^k_t(\texttt{adr})$
    % \item[] $\STMT^k_{t, pc} = false$
    % \item[] $\STMT^k_{t, pc + 1} = true$
  % \end{itemize}
% }

% \defaxiom
% {\texttt{STORE arg}}
% {
  % \begin{itemize}
    % \item[] $\SBADR^{k + 1}_t =
      % \begin{cases}
        % \READ^k(adr) \text{ if indirect} \\
        % adr \text{ otherwise}
      % \end{cases}$
    % \item[] $\SBVAL^{k + 1}_t = \ACCU^k_t$
    % \item[] $\STMT^k_{t, pc} = false$
    % \item[] $\STMT^k_{t, pc + 1} = true$
  % \end{itemize}
% }

% \newsavebox{\STORESBADRAXIOM}
% \savebox{\STORESBADRAXIOM}{
  % \hspace{-0.3cm}
  % $\begin{cases}
    % \READ^k(adr) \text{ if indirect} \\
    % adr \text{ otherwise}
  % \end{cases}$
% }

\newsavebox{\CASHEAPAXIOM}
\savebox{\CASHEAPAXIOM}{
  \hspace{-0.3cm}
  % $\begin{cases}
    % \begin{aligned}
      % \ITE(& \MEM^k_t = \READ^k(\READ^k(\texttt{arg})), \\
      % & \WRITE^k(\READ^k(\texttt{arg}), \ACCU^k_t), \\
      % & \HEAP^k)
    % \end{aligned}
    % \hspace{0.2cm}
    % \text{if indirect} \\
    % \begin{aligned}
      % \ITE(& \MEM^k_t = \READ^k(\texttt{arg}), \\
      % & \WRITE^k(\texttt{arg}, \ACCU^k_t), \\
      % & \HEAP^k)
    % \end{aligned}
    % \hspace{1.5cm}
    % \text{otherwise}
  % \end{cases}$
  $\begin{aligned}
    \ITE(& \MEM^k_t = \READ^k(\EFFECTIVE^k(\texttt{arg})), \\
    & \WRITE^k(\EFFECTIVE^k(\texttt{arg}), \ACCU^k_t), \\
    & \HEAP^k)
  \end{aligned}$
}

\newsavebox{\CASACCUAXIOM}
\savebox{\CASACCUAXIOM}{
  \hspace{-0.3cm}
  % $\begin{cases}
    % \begin{aligned}
      % \ITE(& \MEM^k_t = \READ^k(\READ^k(\texttt{arg})), \\
      % & 1, \\
      % & 0)
    % \end{aligned}
    % % \ITE(\MEM^k_t = \READ^k(\READ^k(\texttt{arg})), 1, 0)
    % \hspace{0.2cm}
    % \text{if indirect} \\
    % \begin{aligned}
      % \ITE(& \MEM^k_t = \READ^k(\texttt{arg}), \\
      % & 1, \\
      % & 0) \\
    % \end{aligned}
    % % \ITE(\MEM^k_t = \READ^k(\texttt{arg}), 1, 0)
    % \hspace{1.5cm}
    % \text{otherwise}
  % \end{cases}$
  % $\begin{aligned}
    % \ITE(& \MEM^k_t = \READ^k(\EFFECTIVE^k(\texttt{arg})), \\
    % & 1, \\
    % & 0)
  % \end{aligned}$
  $\ITE(\MEM^k_t = \READ^k(\EFFECTIVE^k(\texttt{arg})), 1, 0)$
}

% \begin{longtabu}{llX[m]}
  % \firsthline
  % \multirow{2.5}{*}{\texttt{STORE adr}}
    % & $\SBADR^{k + 1}_t$    & $=$ \usebox{\mybox} \\
    % & $\SBVAL^{k + 1}_t$    & $= \ACCU^k_t$ \\
    % & $\STMT^k_{t, pc}$     & $= false$ \\
    % & $\STMT^k_{t, pc + 1}$ & $= true$ \\
    % \lasthline
% \end{longtabu}

% \begin{longtabu}{llX[m]}
  % \firsthline
  % \multirow{2}{*}{\texttt{FLUSH}}
    % & $\HEAP^{k + 1}$       & $= \WRITE^k(\SBADR^k_t, \SBVAL^k_t)$ \\
    % & $\SBFULL^{k + 1}$     & $= false$ \\
  % \lasthline
% \end{longtabu}
% \vspace{-0.9cm}
% \begin{longtabu}{llX[m]}
  % \firsthline
  % \multirow{3}{*}{\texttt{LOAD adr}}
    % & $\ACCU^{k + 1}_t$     & $= \LOAD^k_t(\texttt{adr})$ \\
    % & $\STMT^k_{t, pc}$     & $= false$ \\
    % & $\STMT^k_{t, pc + 1}$ & $= true$ \\
  % \lasthline
% \end{longtabu}
% \vspace{-0.9cm}
% \begin{longtabu}{llX[m]}
  % \firsthline
  % \multirow{2.5}{*}{\texttt{STORE adr}}
    % & $\SBADR^{k + 1}_t$    & $=$ \usebox{\mybox} \\
    % & $\SBVAL^{k + 1}_t$    & $= \ACCU^k_t$ \\
    % & $\STMT^k_{t, pc}$     & $= false$ \\
    % & $\STMT^k_{t, pc + 1}$ & $= true$ \\
  % \lasthline
% \end{longtabu}

% \setlength{\tabulinesep}{2pt}
% \setlength{\doublerulesep}{4pt}
\begin{longtabu}{llX[m]}
  \firsthline
  \multirow{2}{*}{\lstASM{FLUSH}}
    & $\HEAP^{k + 1}$   & $= \WRITE^k(\SBADR^k_t, \SBVAL^k_t)$ \\
    & $\SBFULL^{k + 1}$ & $= \FALSE$ \\
  \hline
  % \hhline{===}
  % \tabucline[2pt off 2pt]{-}
  \multirow{3.05}{*}{\lstASM{LOAD arg}}
    & $\ACCU^{k + 1}_t$           & $= \LOAD^k_t(\texttt{arg})$ \\
    & $\STMT^{k + 1}_{t, pc}$     & $= \FALSE$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$ & $= \TRUE$ \\
  \hline
  % \hhline{===}
  % \multirow{2.5}{*}{\texttt{STORE adr}}
    % & $\SBADR^{k + 1}_t$    & $=$ \usebox{\STORESBADRAXIOM} \\
    % & $\SBVAL^{k + 1}_t$    & $= \ACCU^k_t$ \\
    % & $\STMT^k_{t, pc}$     & $= false$ \\
    % & $\STMT^k_{t, pc + 1}$ & $= true$ \\
  \multirow{4}{*}{\lstASM{STORE arg}}
    & $\SBADR^{k + 1}_t$          & $= \EFFECTIVE^k(\texttt{arg})$ \\
    & $\SBVAL^{k + 1}_t$          & $= \ACCU^k_t$ \\
    & $\STMT^{k + 1}_{t, pc}$     & $= \FALSE$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$ & $= \TRUE$ \\
  \hline
  % \hhline{===}
  \multirow{2}{*}{\lstASM{FENCE}}
    & $\STMT^{k + 1}_{t, pc}$     & $= \FALSE$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$ & $= \TRUE$ \\
  \hline
  \multirow{3.05}{*}{\lstASM{ADD arg}}
    & $\ACCU^{k + 1}_t$           & $= \ACCU^k_t + \LOAD^k_t(\texttt{arg})$ \\
    & $\STMT^{k + 1}_{t, pc}$     & $= \FALSE$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$ & $= \TRUE$ \\
  \hline
  \multirow{3.05}{*}{\lstASM{ADDI arg}}
    & $\ACCU^{k + 1}_t$           & $= \ACCU^k_t + \texttt{arg}$ \\
    & $\STMT^{k + 1}_{t, pc}$     & $= \FALSE$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$ & $= \TRUE$ \\
  \hline
  \newpage
  \hline
  \multirow{3.05}{*}{\lstASM{SUB arg}}
    & $\ACCU^{k + 1}_t$           & $= \ACCU^k_t - \LOAD^k_t(\texttt{arg})$ \\
    & $\STMT^{k + 1}_{t, pc}$     & $= \FALSE$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$ & $= \TRUE$ \\
  \hline
  \multirow{3.05}{*}{\lstASM{SUBI arg}}
    & $\ACCU^{k + 1}_t$           & $= \ACCU^k_t - \texttt{arg}$ \\
    & $\STMT^{k + 1}_{t, pc}$     & $= \FALSE$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$ & $= \TRUE$ \\
  \hline
  \multirow{3.05}{*}{\lstASM{MUL arg}}
    & $\ACCU^{k + 1}_t$           & $= \ACCU^k_t * \LOAD^k_t(\texttt{arg})$ \\
    & $\STMT^{k + 1}_{t, pc}$     & $= \FALSE$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$ & $= \TRUE$ \\
  \hline
  \multirow{3.05}{*}{\lstASM{MULI arg}}
    & $\ACCU^{k + 1}_t$           & $= \ACCU^k_t * \texttt{arg}$ \\
    & $\STMT^{k + 1}_{t, pc}$     & $= \FALSE$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$ & $= \TRUE$ \\
  \hline
  \multirow{3}{*}{\lstASM{CMP arg}}
    & $\ACCU^{k + 1}_t$           & $= \ACCU^k_t - \LOAD^k_t(\texttt{arg})$ \\
    & $\STMT^{k + 1}_{t, pc}$     & $= \FALSE$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$ & $= \TRUE$ \\
  \hline
  \multirow{2}{*}{\lstASM{JMP arg}}
    & $\STMT^{k + 1}_{t, pc}$           & $= \CHANGE{\ITE(pc \neq arg, \FALSE, \TRUE)}$ \\
    & $\STMT^{k + 1}_{t, \texttt{arg}}$ & $= \CHANGE{\ITE(pc \neq arg, \TRUE, \FALSE)}$ \\
  \hline
  \multirow{3}{*}{\lstASM{JZ arg}}
    & $\STMT^{k + 1}_{t, pc}$           & $= \CHANGE{\ITE(pc \neq arg, \FALSE, \lnot \ACCU^k_t)}$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$       & $= \ACCU^k_t$ \\
    & $\STMT^{k + 1}_{t, \texttt{arg}}$ & $= \lnot \ACCU^k_t$ \\
  \hline
  \multirow{3}{*}{\lstASM{JNZ arg}}
    & $\STMT^{k + 1}_{t, pc}$           & $= \CHANGE{\ITE(pc \neq arg, \FALSE, \ACCU^k_t)}$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$       & $= \lnot \ACCU^k_t$ \\
    & $\STMT^{k + 1}_{t, \texttt{arg}}$ & $= \ACCU^k_t$ \\
  \hline
  \multirow{3}{*}{\lstASM{JS arg}}
    & $\STMT^{k + 1}_{t, pc}$           & $= \CHANGE{\ITE(pc \neq arg, \FALSE, \MSB(\ACCU^k_t))}$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$       & $= \lnot \MSB(\ACCU^k_t)$ \\
    & $\STMT^{k + 1}_{t, \texttt{arg}}$ & $= \MSB(\ACCU^k_t)$ \\
  \hline
  \newpage
  \hline
  \multirow{3}{*}{\lstASM{JNS arg}}
    & $\STMT^{k + 1}_{t, pc}$           & $= \CHANGE{\ITE(pc \neq arg, \FALSE, \lnot \MSB(\ACCU^k_t))}$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$       & $= \MSB(\ACCU^k_t)$ \\
    & $\STMT^{k + 1}_{t, \texttt{arg}}$ & $= \lnot \MSB(\ACCU^k_t)$ \\
  \hline
  \multirow{3}{*}{\lstASM{JNZNS arg}}
    & $\STMT^{k + 1}_{t, pc}$           & $= \CHANGE{\ITE(pc \neq arg, \FALSE, \ACCU^k_t \land \lnot \MSB(\ACCU^k_t))}$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$       & $= \lnot \ACCU^k_t \lor \MSB(\ACCU^k_t)$ \\
    & $\STMT^{k + 1}_{t, \texttt{arg}}$ & $= \ACCU^k_t \land \lnot \MSB(\ACCU^k_t)$ \\
  \hline
  \multirow{4}{*}{\lstASM{MEM arg}}
    & $\ACCU^{k + 1}_t$           & $= \LOAD^k_t(\texttt{arg})$ \\
    & $\MEM^{k + 1}_t$            & $= \LOAD^k_t(\texttt{arg})$ \\
    & $\STMT^{k + 1}_{t, pc}$     & $= \FALSE$ \\
    & $\STMT^{k + k}_{t, pc + 1}$ & $= \TRUE$ \\
  \hline
  % \multirow{2.35}{*}{\texttt{CAS arg}}
  \multirow{2.97}{*}{\lstASM{CAS arg}}
    & $\HEAP^{k + 1}$             & $=$ \usebox{\CASHEAPAXIOM} \\
    & $\ACCU^{k + 1}_t$           & $=$ \usebox{\CASACCUAXIOM} \\
    & $\STMT^{k + 1}_{t, pc}$     & $= \FALSE$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$ & $= \TRUE$ \\
  \hline
  \multirow{2}{*}{\lstASM{HALT}}
    % & $\EXIT^{k + 1}_t$ & $= \HALT^{k + 1}_0 \land \ldots \land \HALT^{k + 1}_{n - 1}$ \\
    & $\EXIT^{k + 1}_t$       & $= \displaystyle \bigwedge^{n - 1}_{i = 0} \HALT^{k + 1}_i$ \\
    & $\HALT^{k + 1}_t$       & $= \TRUE$ \\
    & $\STMT^{k + 1}_{t, pc}$ & $= \FALSE$ \\
  \hline
  \multirow{3}{*}{\lstASM{EXIT arg}}
    & $\EXIT^{k + 1}_t$       & $= \TRUE$ \\
    & $\EXITCODE^k$           & \vspace{.15cm} $= \texttt{arg}$ \\
    & $\STMT^{k + 1}_{t, pc}$ & $= \FALSE$ \\
  \hline
  \multirow{3}{*}{\lstASM{CHECK arg}}
    & $\STMT^{k + 1}_{t, pc}$     & $= \FALSE$ \\
    & $\STMT^{k + 1}_{t, pc + 1}$ & $= \TRUE$ \\
    & $\BLOCK^{k + 1}_{arg, t}$   & $= \TRUE$ \\
  \lasthline
  \caption{Frame Axioms}
  \label{tbl:axioms}
\end{longtabu}

% \vspace{0.2cm}
% \hlinE
% \VSPACE{0.1CM}
% \NOINDENT
% \BEGIN{TABU} TO \TEXTWIDTH {Lx[M]}
  % $\heap^{K + 1}$       & $= \write^K(\sbadr^K_T, \sbval^K_T)$ \\
  % $\sbfull^{K + 1}$     & $= FALSE$ \\
% \END{TABU}

% \PARAGRAPH{\TEXTTT{flush}}

% \VSPACE{0.2CM}
% \HLINE
% \VSPACE{0.1CM}
% \NOINDENT
% \BEGIN{TABU} TO \TEXTWIDTH {Lx[M]}
  % $\heap^{K + 1}$       & $= \write^K(\sbadr^K_T, \sbval^K_T)$ \\
  % $\sbfull^{K + 1}$     & $= FALSE$ \\
% \END{TABU}

% \[
% \texttt{compare}^k_t(adr) =
% \begin{cases}
  % \MEM^k_t = \READ^k(\READ^k(adr)) \text{ if indirect} \\
  % \MEM^k_t = \READ^k(adr) \text{ otherwise}
% \end{cases}
% \]

% $\HEAP^{k + 1} =
% \begin{cases}
  % \begin{aligned}
    % \ITE(& \MEM^k_t = \READ^k(\READ^k(\texttt{arg})), \\
    % & \WRITE^k(\READ^k(\texttt{arg}), \ACCU^k_t) \\
    % & \HEAP^k)
  % \end{aligned}
  % \text{ if indirect} \\
  % \begin{aligned}
    % \ITE(& \MEM^k_t = \READ^k(\texttt{arg}), \\
    % & \WRITE^k(\texttt{arg}, \ACCU^k_t) \\
    % & \HEAP^k)
  % \end{aligned}
  % \text{ otherwise}
% \end{cases}$

% Let $\texttt{define-accu}(k, t, pc)$ be the predicate defining the accumulator register state $\ACCU^k_t$ of thread $t$ in step $k$ and $pc$ the set of program counters, altering that state.% \todonote{refine}

% \paragraph{\texttt{LOAD adr}}

% \subsubsection{Statement Activation}

% \setlength{\tabulinesep}{2\lineskip}