\chapter{Code availability}
\label{ap2}

\newpage

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{table}[b]
\footnotesize
\begin{center}
\begin{threeparttable}
\caption[Learning Parameters for Active, Continual Fine-Tuning]{
Learning parameters used for training and fine-tuning of AlexNet for AFT in our experiments. $\mu$ is the momentum, $lr_{fc8}$ is the learning rate of the weights in the last layer, $\alpha$ is the learning rate of the weights in the rest layers, and $\gamma$ determines how $lr$ decreases over epochs. ``Epochs" indicates the number of epochs used in each step. For ACFT, all the parameters are set to the same as AFT except the learning rate $lr$, which is set to 1/10 of that for AFT.}
\label{ap2:tab:hyperparameter}
\begin{tabular}{p{0.38\textwidth}P{0.09\textwidth}P{0.09\textwidth}P{0.09\textwidth}P{0.09\textwidth}P{0.09\textwidth}}
\hline
Applications & $\mu$ & $lr$ & $lr_{fc8}$ & $\gamma$ & epoch \\
\hline
Colonoscopy frame classification & 0.9 & 1e-4 & 1e-3 & 0.95 & 8 \\
Polyp detection & 0.9 & 1e-4 & 1e-3 & 0.95 & 10 \\
Pulmonary embolism detection & 0.9 & 1e-3 & 1e-2 & 0.95 & 5 \\
\hline
\end{tabular}
\end{threeparttable}
\end{center}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Active Continual Fine-tuning}
We have investigated the effectiveness of ACFT in four applications: scene classification, colonoscopy frame classification, polyp detection, and pulmonary embolism (PE) detection. Ablation studies have been conducted to confirm the significant design of our majority selection and randomization, built upon conventional entropy and diversity based active selection criteria.
For all four applications, we set $\alpha$ to 1/4 and $\omega$ to 5.
The deep learning library Matlab and Caffe are utilized to implement active learning and transfer learning (more details can be found at \href{https://github.com/MrGiovanni/Active-Learning}{https://github.com/MrGiovanni/Active-Learning}). We based our experiments on AlexNet and GoogLeNet because their architectures offer an optimal depth balance, deep enough to investigate the impact of ACFT and AFT on pre-trained CNN performance, but shallow enough to conduct experiments quickly. The learning parameters used for training and fine-tuning of AlexNet in our experiments are summarized in \tableautorefname~\ref{ap2:tab:hyperparameter}. The Adam optimizer is utilized to optimize the objective functions described in our paper. The batch size is 512 in the learning procedure.

\section*{UNet++}

Our experiments are implemented in Keras with Tensorflow backend. We use {\em early-stop} mechanism on the validation set to avoid over-fitting and evaluate the results using Dice-coefficient and Intersection over Union (IoU). Adam is used as the optimizer with a learning rate of 3$e$-4. Both UNet+ and UNet++ are constructed from the original U-Net architecture. All the experiments are performed using three NVIDIA TITAN X (Pascal) GPUs with 12 GB memory each. To facilitate reproducibility and model reuse, we have released the implementation of U-Net, UNet+, and UNet++ for various traditional and modern backbone architectures at \href{https://github.com/MrGiovanni/UNetPlusPlus}{https://github.com/MrGiovanni/UNetPlusPlus}.

\section*{Models Genesis}

\textit{Pre-training Models Genesis:} Our Models Genesis are pre-trained from 623 Chest CT scans in LUNA~2016~\citep{setio2017validation} in a self-supervised manner. The reason that we decided not to use all 888 scans provided by this dataset was to avoid test-image leaks between proxy and target tasks, so that we can confidently use the rest of the images solely for testing Models Genesis as well as the target models, although Models Genesis are trained from only unlabeled images, involving no annotation shipped with the dataset. We first randomly crop sub-volumes, sized $64\times 64\times 32$ pixels, from different locations. To extract more informative sub-volumes for training, we then intentionally exclude those which are empty (air) or contain full tissues. Our Models Genesis 2D are self-supervised pre-trained from LUNA~2016~\citep{setio2017validation} and ChestX-ray14~\citep{wang2017chestx} using 2D CT slices in an axial view and X-ray images, respectively. For all proxy tasks and target tasks, the raw image intensities were normalized to the $[0,1]$ range before training. We use the mean square error (MSE) between input and output images as objective function for the proxy task of image restoration. As suggested by~\citet{pathak2016context} and \citet{chen2019self}, the MSE loss is sufficient for representation learning, although the restored images may be blurry.

When pre-training Models Genesis, we apply each of the transformations on sub-volumes with a pre-defined probability. That being said, the model will encounter not only the transformed sub-volumes as input, but also the original sub-volumes. This design offers two advantages:
\begin{itemize}
    \item First, the model must distinguish original versus transformed images, discriminate transformation type(s), and restore images if transformed. Our self-supervised learning framework, therefore, results in pre-trained models that are capable of handling versatile tasks.
    \item Second, since original images are presented in the proxy task, the semantic difference of input images between the proxy and target task becomes smaller. As a result, the pre-trained model can be transferable to process regular/normal images in a broad variety of target tasks.
\end{itemize}

\textit{Fine-tuning Models Genesis:} The pre-trained Models Genesis can be adapted to new imaging tasks through transfer learning or fine-tuning. There are three major transfer learning scenarios: (1) employing the encoder as a fixed feature extractor for a new dataset and following up with a linear classifier (\eg Linear SVM or Softmax classifier), (2) taking the pre-trained encoder and appending a sequence of fully-connected (\textit{fc}) layers for target classification tasks, and (3) taking the pre-trained encoder and decoder and replacing the last layer with a $1\times 1\times 1$ convolutional layer for target segmentation tasks. For scenarios (2) and (3), it is possible to fine-tune all the layers of the model or to keep some of the earlier layers fixed, only fine-tuning some higher-level portion of the model. We have evaluated the performance of our self-supervised representation for transfer learning by fine-tuning all layers in the network. In the following, we examine Models Genesis on five distinct medical applications, covering classification and segmentation tasks in CT and MRI images with varying levels of semantic distance from the source (Chest CT) to the targets in terms of organs, diseases, and modalities (see \tablename~\ref{ch5:tab:distance}) for investigating the transferability of Models Genesis.

\textit{Benchmarking Models Genesis:} For a thorough comparison, we used three different techniques to randomly initialize the weights of models: (1) a basic random initialization method based on Gaussian distributions, (2) a method commonly known as Xavier, which was suggested in~\citet{glorot2010understanding}, and (3) a revised version of Xavier called MSRA, which was suggested in~\citet{he2015delving}. They are implemented as \texttt{uniform}, \texttt{glorot\_uniform}, and \texttt{he\_uniform}, respectively, following the Initializers\footnote{\label{foot:Initializer}Initializers: \href{http://faroit.com/keras-docs/1.2.2/initializations/}{faroit.com/keras-docs/1.2.2/initializations}} in Keras. We compare Models Genesis with Rubik's cube~\citep{zhuang2019self}, the most recent multi-task and self-supervised learning method for 3D medical imaging. Considering that most of the self-supervised learning methods are initially proposed and implemented in 2D, we have extended five most representative ones~\citep{vincent2010stacked,pathak2016context,noroozi2016unsupervised,chen2019self,caron2018deep} into their 3D versions for a fair comparison (see detailed implementation in~\ref{sec:baseline_implementation_appendix}). To promote the 3D self-supervised learning research, we make our own implementation of the 3D extended methods and their corresponding pre-trained weights publicly available as an open-source tool that can effectively be used out-of-the-box. In addition, we have examined publicly available pre-trained models for 3D transfer learning in medical imaging, including NiftyNet\footnote{\label{foot:NiftyNet}NiftyNet Model Zoo: \href{https://github.com/NifTK/NiftyNetModelZoo}{github.com/NifTK/NiftyNetModelZoo}}~\citep{gibson2018niftynet}, MedicalNet\footnote{\label{foot:MedicalNet}MedicalNet: \href{https://github.com/Tencent/MedicalNet}{github.com/Tencent/MedicalNet}}~\citep{chen2019med3d}, and, the most influential 2D weights initialization, Models ImageNet. We also fine-tune I3D\footnote{\label{foot:I3D}I3D: \href{https://github.com/deepmind/kinetics-i3d}{github.com/deepmind/kinetics-i3d}}~\citep{carreira2017quo} in our five target tasks because it has been shown to successfully initialize 3D models for lung nodule detection in~\citet{ardila2019end}. The detailed configurations of these models can be found in \ref{sec:public_3d_model_appendix}.

3D U-Net architecture\footnote{\label{foot:3dunet}3D U-Net: \href{https://github.com/ellisdg/3DUnetCNN}{github.com/ellisdg/3DUnetCNN}} is used in 3D applications; U-Net architecture\footnote{\label{foot:densenet121}Segmentation Models: \href{https://github.com/qubvel/segmentation_models}{github.com/qubvel/segmentation\_models}} is used in 2D applications. Batch normalization~\citep{ioffe2015batch} is utilized in all 3D/2D deep models. For proxy tasks, SGD method~\citep{zhang2004solving} with an initial learning rate of $1e0$ is used for optimization. We use \texttt{ReduceLROnPlateau} to schedule learning rate, in which if no improvement is seen in the validation set for a certain number of epochs, the learning rate is reduced. For target tasks, Adam method~\citep{kinga2015method} with a learning rate of $1e-3$ is used for optimization, where $\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=1e-8$. We use \textit{early-stop} mechanism on the validation set to avoid over-fitting. Simple yet heavy 3D data augmentation techniques are employed in all five target tasks, including random flipping, transposing, rotating, and adding Gaussian noise. We run each method ten times on all of the target tasks and report the average, standard deviation, and further present statistical analysis based on an independent two-sample \textit{t}-test.

In the proxy task, we pre-train the model using 3D sub-volumes sized $64\times 64\times 32$, whereas in target tasks, the input is not limited to sub-volumes with certain size. That being said, our pre-trained models can be fine-tuned in the tasks with CT sub-volumes, entire CT volumes, or even MRI volumes as input upon user's need. The flexibility of input size is attributed to two reasons: (1) our pre-trained models learn generic image representation such as appearance, texture, and context feature, and (2) the encoder-decoder architecture is able to process images with arbitrary sizes.

\section*{Implementation Details of Revised Baselines}
\label{sec:baseline_implementation_appendix}

This work is among the first effort to create a comprehensive benchmark for existing self-supervised learning methods for 3D medical image analysis. We have extended the six most representative self-supervised learning methods into their 3D versions, including De-noising~\citep{vincent2010stacked}, In-painting~\citep{pathak2016context}, Jigsaw~\citep{noroozi2016unsupervised}, and Patch-shuffling~\citep{chen2019self}. These methods were originally introduced for the purpose of 2D imaging.
On the other hand, the most recent 3D self-supervised method~\citep{zhuang2019self} learns representation by playing a Rubik's cube.
We have reimplemented it because their official implementation is not publicly available at the time this dissertation is written. All of the models are pre-trained using the LUNA~2016 dataset~\citep{setio2017validation} with the same sub-volumes extracted from CT scans as our models (see \tablename~\ref{ch5:tab:terminology}). The detailed implementations of the baselines are elaborated in the following sections.

\textit{Extended 3D De-noising:} In our 3D De-noising, which is inspired by its 2D counterpart~\citep{vincent2010stacked}, the model is trained to restore the original sub-volume from its transformed one with additive Gaussian noise (randomly sampling $\sigma\in[0,0.1]$). To correctly restore the original sub-volume, models are required to learn Gabor-like edge detectors when denoising transformed sub-volumes. Following the proposed image restoration training scheme, the auto-encoder network is replaced with a 3D U-Net, wherein the input is a $64\times 64\times 32$ sub-volume that has undergone Gaussian noise and the output is the restored sub-volume. The L2 distance between input and output is used as the loss function.

\begin{figure*}[t]
\begin{center}
\includegraphics[width=1.0\linewidth]{Figures/AP2/fig_innercutout_inpainting.pdf}
\end{center}
\caption[Comparison Between Image In-painting and Inner-cutout]{
A direct comparison between image in-painting~\citep{pathak2016context} and our inner-cutout. (a) contrasts our inner-cutout with in-painting, wherein the model in the former scheme computes loss on the entire image and the model in the latter scheme computes loss only for the cutout area. (b) presents the performance on five target tasks, showing that inner-cutout is better suited for target classification tasks (\eg \texttt{NCC} and \texttt{ECC}), while in-painting is more helpful for target segmentation tasks (\eg \texttt{NCS}, \texttt{LCS}, and \texttt{BMS}).
}
\label{ap2:fig:innercutout_inpainting}
\end{figure*}

\textit{Extended 3D In-painting:} In our 3D In-painting, which is inspired by its 2D counterpart~\citep{pathak2016context}, the model is trained to in-paint arbitrary cutout regions based on the rest of the sub-volume. A qualitative illustration of the image in-painting task is shown in the right panel of \figurename~\ref{ap2:fig:innercutout_inpainting}(a). To correctly predict missing regions, networks are required to learn local continuities of organs in medical images via interpolation. Unlike the original in-painting, the adversarial loss and discriminator are excluded from our implementation of the 3D version because our primary goal is to empower models with generic representation, rather than generating sharper and realistic sub-volumes. The generator is a 3D U-Net, consisting of an encoder and a decoder. The input of the encoder is a $64\times 64\times 32$ sub-volume that needs to be in-painted.
Their decoder works differently than our inner-cutout because it predicts the missing region only, and therefore, the loss is just computed on the cutout region---an ablation study on the loss has been further presented in \figurename~\ref{ap2:fig:innercutout_inpainting}.

\textit{Extended 3D Jigsaw:} In our 3D Jigsaw, which is inspired by its 2D counterpart~\citep{noroozi2016unsupervised}, we utilize the implementation by~\citet{taleb20203d}\footnote{\label{foot:3d_self_learning}Self-Supervised 3D Tasks: \href{https://github.com/HealthML/self-supervised-3d-tasks}{github.com/HealthML/self-supervised-3d-tasks}}, wherein the puzzles are created by sampling a $3\times 3\times 3$ grid of 3D patches. Then, these patches are shuffled according to an arbitrary permutation, selected from a set of predefined permutations. This set with size $P=100$ is chosen out of the $(3\times 3\times 3)!$ possible permutations, by following the Hamming distance based algorithm, and each permutation is assigned an index. As a result, the problem is cast as a $P$-way classification task, \ie the model is trained to recognize the applied permutation index, allowing us to solve the 3D puzzles efficiently. We build the classification model by taking the encoder of 3D U-Net and appending a sequence of $fc$ layers. In the implementation, we minimize the cross-entropy loss of the list of extracted puzzles.

\begin{figure}[t]
\begin{center}
\includegraphics[width=1.0\linewidth]{Figures/AP2/fig_localshuffling_patchshuffling.pdf}
\end{center}
\caption[Comparison Between Global Patch Shuffling and Local Pixel Shuffling]{
A direct comparison between global patch shuffling~\citep{chen2019self} and our local pixel shuffling. (a) illustrates ten example images undergone local-shuffling and patch-shuffling independently. As seen, the overall anatomical structure such as individual organs, blood vessels, lymph nodes, and other soft tissue structures are preserved in the transformed image through local-shuffling.
(b) presents the performance on five target tasks, showing that models pre-trained by our local-shuffling noticeably outperform those pre-trained by patch-shuffling for cross-domain transfer learning (\texttt{BMS}).
}
\label{ap2:fig:localshuffling_patchshuffling}
\end{figure}

\textit{Extended 3D Patch-shuffling:} In our 3D Patch-shuffling, which is inspired by its 2D counterpart~\citep{chen2019self}, the model learns image representation by restoring the image context. Given a sub-volume, we randomly select two isolated small 3D patches and swap their context. We set the length, width, and height of the 3D patch to be proportional to those in the entire sub-volume by 25\% to 50\%. Repeating this process for $T=10$ times can generate the transformed sub-volume (see examples in \figurename~\ref{ap2:fig:localshuffling_patchshuffling}(a)). The model is trained to restore the original sub-volume, where L2 distance between input and output is used as the loss function. To process volumetric input and ensure a fair comparison with other baselines, we replace their U-Net with 3D U-Net architecture, where the encoder and decoder serve as analysis and restoration parts, respectively.

\textit{Extended 3D DeepCluster:} In our 3D DeepCluster, which is inspired by its 2D counterpart~\citep{caron2018deep}, we iteratively cluster deep features extracted from sub-volumes by $k$-means and use the subsequent assignments as supervision to update the weights of the model. Through clustering, the model can obtain useful general-purpose visual features, requiring little domain knowledge and no specific signal from the inputs. We replaced original AlexNet/VGG architecture with the encoder of 3D U-Net to process 3D input sub-volumes. The number of clusters that works best for 2D tasks may not be a good choice for 3D tasks. To ensure a fair comparison, we extensively tune this hyper-parameter in $\{10,20,40,80,160,320\}$ and finally set to 260 from the narrowed down search space of $\{240,260,280\}$. Unlike ImageNet models for 2D imaging tasks, there is no available pre-trained 3D feature extractor for medical imaging tasks; therefore, we randomly initialize the model weights at the beginning. Our Models Genesis, the first generic 3D pre-trained models, could potentially be used as the 3D feature extractor and co-trained with 3D DeepCluster.

\textit{Rubik's Cube:} We implement Rubik's Cube with respect to~\citet{zhuang2019self}, which consists of cube rearrangement and cube rotation. Like playing a Rubik's cube, this proxy task enforces models to learn translational and rotational invariant features from raw 3D data. Given a sub-volume, we partition it into a $2\times 2\times 2$ grid of cubes. In addition to predicting orders (3D Jigsaw), this proxy task permutes the cubes with random rotations, forcing models to predict the orientation. Following the original paper, we limit the directions for cube rotation, \ie only allowing 180$^{\circ}$ horizontal and vertical rotations, to reduce the complexity of the task. The eight cubes are then fed into a Siamese network with eight branches sharing the same weight to extract features. The feature maps from the last fully-connected or convolution layer of all branches are concatenated and given as input to the fully-connected layer of separate tasks, \ie cube ordering and orienting, which are supervised by permutation loss and rotation loss, respectively, with equal weights.

\section*{Configurations of Publicly Available Models}
\label{sec:public_3d_model_appendix}

For publicly available models, we do not re-train their proxy tasks and instead simply endeavor to find the best hyper-parameters for each of them in target tasks. We compare them with our Models Genesis in a user perspective, which might seem to be unfair in a research perspective because many variables are asymmetric among the competitors, such as programming platform, model architecture, number of parameters, etc. However, the goal of this section is to experiment with existing ready-to-use pre-trained models under different medical tasks; therefore, we presume that all of the publicly available models and their configurations have been carefully composed to the optimal setting.

\textit{NiftyNet:} We examine the effectiveness of fine-tuning from NiftyNet in five target tasks. We should note that NiftyNet is not initially designed for transfer learning but is one of the few publicly available supervised pre-trained 3D models. The model from~\citet{gibson2018automatic} has been considered as the baseline in our experiments because it has also been pre-trained on the chest region in CT modality and applied an encoder-decoder architecture that is similar to our work. We directly adopt the pre-trained weights of the dense V-Net architecture provided by NiftyNet, so it carries a smaller number of parameters than our 3D U-Net (2.60M vs. 16.32M). For target classification tasks, we use the dense V-Net encoder by appending a sequence of $fc$ layers; for target segmentation tasks, we use the entire dense V-Net. Since NiftyNet is developed in Tensorflow, all five target tasks are re-implemented using their build-in configuration. For each target task, we have tuned hyper-parameters (\eg learning rate and optimizer) and applied extensive data augmentations (\eg rotation and scaling).

\textit{Inflated 3D:} We download the Inflated 3D (I3D) model pre-trained from Flow streams in the Kinetics dataset~\citep{hara2018can} and fine-tune it on our five target tasks. The input sub-volume is copied into two channels to align with the required input shape. For target classification tasks, we take the pre-trained I3D and append a sequence of randomly initialized fully-connected layers. For target segmentation tasks, we take the pre-trained I3D as the encoder and expand a decoder to predict the segmentation map, resulting in a U-Net like architecture. The decoder is the same as that implemented in our 3D U-Net, consisting of up-sampling layers followed by a sequence of convolutional layers, batch normalization, and ReLU activation. Besides, four skip connections are built between the encoder and decoder, wherein feature maps before each pooling layer in the encoder are concatenated with same-scale feature maps in the decoder. All of the layers in the model are trainable during transfer learning. Adam method~\citep{kinga2015method} with a learning rate of $1e-4$ is used for optimization.

\textit{MedicalNet:} We download MedicalNet models~\citep{chen2019med3d} that have been pre-trained on eight publicly available 3D segmentation datasets. ResNet-50 and ResNet-101 backbones are chosen because they are reported by~\citet{chen2019med3d} as the most compelling backbones for target segmentation and classification tasks, respectively. Like I3D, we append a decoder at the end of the pre-trained encoder, randomly initialize its weights, and link the encoder with the decoder using skip connections. Owing to the 3D ResNet backbones, the resultant segmentation network for MedicalNet is much heavier than our 3D U-Net. To be consistent with the original programming platform of MedicalNet, we re-implement all five target tasks in PyTorch, using the same data separation and augmentation. We report the highest results achieved by any of the two backbones in \tablename~\ref{ch5:tab:top_existing_models}.


\subsection{Parabolic (complex plane)}


\documentclass{article}
\usepackage{ltcadiz}
\begin{document}

\section{Introduction}

The prelude is a Z section.
It is an implicit parent of every other section.
It assists in defining the meaning of number literal expressions
(see 12.2.6.9),
and the list arguments of operators
(see 12.2.12).
via syntactic transformation rules later in this International Standard.
The prelude is presented here using the mathematical lexis.

\section{Formal definition of prelude}

\begin{zsection}
\SECTION prelude
\end{zsection}

The section is called $prelude$ and has no parents.

% Mark-up directives for core language symbols are as follows.
%
%%Zchar	\{ U+007B
%%Zchar	\} U+007D
%%Zinchar \where U+007C
%%Zchar \Delta U+0394
%%Zchar \Xi U+039E
%%Zprechar \theta U+03B8
%%Zprechar \lambda U+03BB
%%Zprechar \mu U+03BC
%%Zchar \ldata U+27EA
%%Zchar \rdata U+27EB
%%Zchar \lblot U+2989
%%Zchar \rblot U+298A
%%Zchar \vdash U+22A2
%%Zinchar \land U+2227
%%Zinchar \lor U+2228
%%Zinchar \implies U+21D2
%%Zinchar \iff U+21D4
%%Zprechar \lnot U+00AC
%%Zprechar \forall U+2200
%%Zprechar \exists U+2203
%%Zinchar \in U+2208
%%Zinchar \spot U+2981
%%Zinchar \hide U+29F9
%%Zinchar \project U+2A21
%%Zinchar \semi U+2A1F
%%Zinchar \pipe U+2A20
%%Zpreword \IF if
%%Zinword \THEN then
%%Zinword \ELSE else
%%Zpreword \LET let
%%Zpreword \pre pre
%%Zinword \function function
%%Zinword \generic generic
%%Zinword \relation relation
%%Zinword \leftassoc leftassoc
%%Zinword \rightassoc rightassoc
%%Zinword \listarg {,}{,}
%%Zinword \varg \_
%%Zprechar \power U+2119
%%Zinchar \cross U+00D7
%%Zchar \arithmos U-0001D538
%%Zchar \nat U+2115

% Mark-up directives for the rest of the Greek alphabet are as follows.
%
%%Zchar \alpha U+03B1
%%Zchar \beta U+03B2
%%Zchar \gamma U+03B3
%%Zchar \delta U+03B4
%%Zchar \epsilon U+03B5
%%Zchar \zeta U+03B6
%%Zchar \eta U+03B7
%%Zchar \iota U+03B9
%%Zchar \kappa U+03BA
%%Zchar \nu U+03BD
%%Zchar \xi U+03BE
%%Zchar \pi U+03C0
%%Zchar \rho U+03C1
%%Zchar \sigma U+03C3
%%Zchar \tau U+03C4
%%Zchar \upsilon U+03C5
%%Zchar \phi U+03C6
%%Zchar \chi U+03C7
%%Zchar \psi U+03C8
%%Zchar \omega U+03C9
%%Zchar \Gamma U+0393
%%Zchar \Theta U+0398
%%Zchar \Lambda U+039B
%%Zchar \Pi U+03A0
%%Zchar \Sigma U+03A3
%%Zchar \Upsilon U+03A5
%%Zchar \Phi U+03A6
%%Zchar \Psi U+03A8
%%Zchar \Omega U+03A9

% Extra mark-up directives for vertical spacing.
%
%%Zword \Also \\
%%Zword \ALSO \\
%%Zword \zBreak \\
%%Zword \ZBREAK \\

% Extra mark-up for type colon for typechecked terms
%
%%Zinword \typecolon U+FF1A

NOTE 1
\begin{zed}
\generic (\power \_)
\end{zed}
$\power$ has already been introduced in this International Standard
(see 8.3),
so this operator template is not necessary.
However, it may be a convenient way of introducing to a tool
the association of $\power$ with the appropriate token and precedence,
especially in preparation for the toolkit's $\power_1$ (see B.3.6).
A tool may introduce $\cross$ here similarly,
that being the only other Z core notation whose precedence
lies amongst those of user-defined operators.

%\begin{zed}
%\generic 8 \rightassoc (\_ \cross \_)
%\end{zed}

\begin{zed}
[\arithmos]
\end{zed}

The given type $\arithmos$, pronounced ``arithmos'', provides
a supply of values for use in specifying number systems.

\begin{axdef}
\nat : \power \arithmos
\end{axdef}

The set of natural numbers, $\nat$, is a subset of $\arithmos$.

%the original definition doesn't seem right... should be an axdef?
%Changed by Tim Miller
%\begin{zed}
%number_literal_0: \nat
%number_literal_1: \nat
%\end{zed}

\begin{axdef}
  number\_literal\_0 : \nat\\
  number\_literal\_1 : \nat
\end{axdef}

$0$ and $1$ are natural numbers, all uses of which are transformed
to references to these declarations (see 12.2.6.9).

\begin{zed}
\function 30 \leftassoc (\_ + \_)
\end{zed}

\begin{axdef}
\_ + \_ : \power ((\arithmos \cross \arithmos) \cross \arithmos)
\where
\forall m, n: \nat @ \exists_1 p: ( \_ + \_ ) @ p.1 = (m, n)\\
\forall m, n: \nat @ m + n \in \nat\\
\forall m, n: \nat | m + 1 = n + 1 @ m = n\\
\forall n: \nat @ \lnot n + 1 = 0\\
\forall w: \power \nat | 0 \in w \land (\forall y: w @ y + 1 \in w) @ w = \nat\\
\forall m: \nat @ m + 0 = m\\
\forall m, n: \nat @ m + (n + 1) = (m + n) + 1
\end{axdef}

Addition is defined here for natural numbers.
(It is extended to integers in the mathematical toolkit,
annex~B.)
Addition is a function.
The sum of two natural numbers is a natural number.
The operation of adding 1 is an injection on natural numbers,
and produces a result different from 0.
There is an induction constraint that all natural numbers are
either 0 or are formed by adding 1 to another natural number.
0 is an identity of addition.
Addition is associative.

%NOTE 2~~The definition of addition is equivalent to the following definition,
%which is written using notation from the mathematical toolkit
%(and so is unsuitable as the normative definition here).
%\begin{axdef}
%\_~+~\_ : \arithmos \cross \arithmos \rel \arithmos
%\where
%(\nat \cross \nat) \dres (\_~+~\_) \in (\nat \cross \nat) \fun \nat\\
%\lambda n : \nat @ n + 1 \in \nat \inj \nat\\
%disjoint \langle\{0\}, \{n : \nat @ n+1\}\rangle\\
%\forall w : \power \nat | \{0\} \cup \{n:w @ n+1\} \subseteq w @ w = \nat\\
%\forall m : \nat @ m + 0 = m\\
%\forall m, n : \nat @ m + (n + 1) = m + n + 1\\
%\end{axdef}

\end{document}


% !TEX root = ./bursty_transcription.tex
\section{Finding the ``right'' model: Bayesian parameter inference}
\label{section_04_bayesian_inference}

In this section of the paper, we continue our program of providing one complete
description of the entire broad sweep of studies that have been made in the
context of the repressor-operator model, dating all the way back to the original
work of Jacob and Monod and including the visionary quantitative work of
M\"{u}ller-Hill and collaborators~\cite{Oehler1990} and up to more recent
studies \cite{Garcia2011a}. In addition, the aim is to reconcile the
thermodynamic and kinetic perspectives that have been brought to bear on this
problem. From Section~\ref{section_02_means}, this reconciliation depends on a
key quantitative question as codified by Eq.~\ref{eq:deltaFR_eq_noneq_equiv}:
does the free energy of repressor binding, as described in the thermodynamic
models and indirectly inferred from gene expression measurements, agree with the
corresponding values of repressor binding and unbinding rates in the kinetic
picture, measured or inferred more directly? In this section we tackle the
statistical inference problem of inferring these repressor rates from
single-cell mRNA counts data. But before we can turn to the full case of simple
repression, we must choose an appropriate model of the constitutive promoter and
infer the parameter values in that model. This is the problem we address first.

\subsection{Parameter inference for constitutive promoters}

From consideration of Fano factors in the previous section, we suspect that
model 5 in Figure~\ref{fig2:constit_cartoons}(A), a one-state bursty model of
constitutive promoters, achieves the right balance of complexity and simplicity,
by allowing both Fano factor $\nu>1$, but also by remedying, by design, the
problems of parameter degeneracy that model 4 in
Figure~\ref{fig2:constit_cartoons}(A) suffered~\cite{Razo-Mejia2020}. Does this
stand up to closer scrutiny, namely, comparison to full mRNA distributions
rather than simply their moments? We will test this thoroughly on single-cell
mRNA counts for different unregulated promoters from Jones et.\
al.~\cite{Jones2014}.

It will be instructive, however, to first consider the Poisson promoter, model 1
in Figure~\ref{fig2:constit_cartoons}. As we alluded to earlier, since the
Poisson distribution has a Fano factor $\nu$ strictly equal to 1, and all of the
observed data in Figure~\ref{fig2:constit_cartoons}(B) has Fano factor $\nu>1$,
we might already suspect that this model is incapable of fitting the data. We
will verify that this is in fact the case. Using the same argument we can
immediately rule out models 2 and 3 from Figure~\ref{fig2:constit_cartoons}(A).
These models have Fano factors $\nu\le 1$ meaning they are underdispersed
relative to the Poisson distribution. We will also not explicitly consider model
4 from~\fig{fig2:constit_cartoons}(A) since it was already thoroughly analyzed
in~\cite{Razo-Mejia2020}, and since model 5 can be viewed as a special case of
it.

Our objective for this section will then be to assess whether or not model 5 is
quantitatively able to reproduce experimental data. In other words, if our claim
is that the level of coarse graining in this model is capable of capturing the
relevant features of the data, then we should be able to find values for the
model parameters that can match theoretical predictions with single-molecule
mRNA count distributions. A natural language for this parameter inference
problem is that of Bayesian probability. We will then build a Bayesian inference
pipeline to fit the model parameters to data. We will use the full dataset of
single-cell mRNA counts from~\cite{Jones2014} used in
Figure~\ref{fig2:constit_cartoons}(B). To gain intuition on how this analysis is
done we analyzed the ``wrong'' model 1 in Figure~\ref{fig2:constit_cartoons}(A).
For this analysis, we direct the reader to Appendix~\ref{sec:bayesian} as an
intuitive introduction to the Bayesian framework with a simpler model.

\subsubsection{Model 5 - Bursty promoter}

Let us now consider the problem of parameter inference for model five
from~\fig{fig1:means_cartoons}(C). As derived in
Appendix~\ref{sec:gen_fcn_appdx}, the steady-state mRNA distribution in this
model is a negative binomial distribution~\cite{Kumar2015}, given by
\begin{equation}
p(m) = \frac{\Gamma(m+k_i)}{\Gamma(m+1)\Gamma(k_i)}
        \left(\frac{1}{1+b}\right)^{k_i}
        \left(\frac{b}{1+b}\right)^m,
\label{eq:neg_bionom}
\end{equation}
where $b$ is the mean burst size and $k_i$ is the burst rate in units of the
mRNA degradation rate $\gamma$. As sketched earlier, to think of the negative
binomial distribution in terms of an intuitive ``story,'' in the precise
meaning of~\cite{Blitzstein2015}, we imagine the arrival of
bursts as a Poisson process with rate $k_i$, where each burst has a
geometrically-distributed size with mean size $b$.

As for the Poisson promoter model, this expression for the steady-state mRNA
distribution is exactly the likelihood we want to use when stating Bayes
theorem. Denoting the single-cell mRNA count data as $D=\{m_1, m_2,\dots,
m_N\}$, Bayes' theorem takes the form
\begin{equation}
p(k_i, b \mid D) \propto p(D\mid k_i,b)p(k_i, b).
\end{equation}
We already have our likelihood --the product of $N$ negative binomials as
Eq.~\ref{eq:neg_bionom}-- so we only need to choose priors on $k_i$ and $b$. For
the datasets from~\cite{Jones2014} that we are analyzing, as for the Poisson
promoter model above we are still data-rich so the prior's influence remains
weak, but not nearly as weak because the dimensionality of our model has
increased from one parameter to two. Details on the arguments behind our prior
distribution selection are left for Appendix~\ref{sec:bayesian}. We state here
that the natural scale to explore these parameters is logarithmic. This is
commonly the case for parameters for which our previous knowledge based on our
domain expertise spans several orders of magnitude. For this we chose log-normal
distributions for both $k_i$ and $b$. Details on the mean and variance of these
distributions can be found in Appendix~\ref{sec:bayesian}.

We carried out Markov-Chain Monte Carlo (MCMC) sampling on the posterior of this
model, starting with the constitutive \textit{lacUV5} dataset
from~\cite{Jones2014}. The resulting MCMC samples are shown in
Figure~\ref{fig:constit_post_full}(A). In contrast to the active/inactive
constitutive model considered in~\cite{Razo-Mejia2020} (kinetic model 4 in
Figure~\ref{fig2:constit_cartoons}(A)), this model is well-identified with both
parameters determined to a fractional uncertainty of 5-10\%. The strong
correlation reflects the fact that their product sets the mean of the mRNA
distribution, which is tightly constrained by the data, but there is weak
``sloppiness''~\cite{Transtrum2015} along a set of values with a similar
product.

Having found the model's posterior to be well-identified as with the Poisson
promoter (Appendix~\ref{sec:bayesian}), the next step is to compare both models
with experimental data. To do this for the case of the bursty promoter, for each
of the parameter samples shown in Figure~\ref{fig:constit_post_full}(A) we
generated negative bionomial-distributed mRNA counts. As MCMC samples parameter
space proportionally to the posterior distribution, this set of random samples
span the range of possible values that we would expect given the correspondence
between our theoretical model and the experimental data. A similar procedure can
be applied to the Poisson promoter. To compare so many samples with the actual
observed data, we can use empirical cumulative distribution functions (ECDF) of
the distribution quantiles. This representation is shown in
Figure~\ref{fig:constit_post_full}(B). In this example, the median for each
possible mRNA count for the Poisson distribution is shown as a dark green line,
while the lighter green contains 95\% of the randomly generated samples. This
way of representing the fit of the model to the data gives us a sense of the
range of data we might consider plausible, under the assumption that the model
is true. For this case, as we expected given our premise of the Poisson promoter
being wrong, it is quite obvious that the observed data, plotted in black is not
consistent with the Poisson promoter model. An equivalent plot for the bursty
promoter model is shown in blue. Again the darker tone shows the median, while
the lighter color encompasses 95\% of the randomly generated samples. Unlike the
Poisson promoter model, the experimental ECDF closely tracks the posterior
predictive ECDF, indicating this model is actually able to generate the observed
data and increasing our confidence that this model is sufficient to parametrize
the physical reality of the system.

The commonly used promoter sequence \textit{lacUV5} is our primary
target here, since it forms the core of all the
simple repression constructs of~\cite{Jones2014} that we consider in
Section~\ref{sec:rep_kinetics_inference}. Nevertheless, we thought it wise to
apply our bursty promoter model to the other 17 unregulated promoters available
in the single-cell mRNA count dataset from~\cite{Jones2014} as a test that the
model is capturing the essential phenomenology. If the model fit well to all the
different promoters, this would increase our confidence that it would serve well
as a foundation for inferring repressor kinetics later in
Section~\ref{sec:rep_kinetics_inference}. Conversely, were the model to fail on
more than a couple of the other promoters, it would give us pause.

\begin{figure}%[h!]
\centering
\includegraphics[width=\textwidth]{../../figures/main/fig03.pdf}
\caption{\textbf{Constitutive promoter posterior inference and model
comparison.} (A) The joint posterior density of model 5, the bursty promoter
with negative binomially-distributed steady state, is plotted with MCMC samples.
1D marginal probability densities are plotted as flanking histograms. The model
was fit on \textit{lacUV5} data from~\cite{Jones2014}. (B) The empirical
cumulative distribution function (ECDF) of the observed population distribution
of mRNA transcripts under the control of a constitutive \textit{lacUV5} promoter
is shown in black. The median posterior predictive ECDFs for models (1),
Poisson, and (5), negative binomial, are plotted in dark green and dark blue,
respectively. Lighter green and blue regions enclose 95\% of all posterior
predictive samples from their respective models. Model (1) is in obvious
contradiction with the data while model (5) is not. Single-cell mRNA count data
is again from~\cite{Jones2014}. (C) Joint posterior distributions for burst rate
$k_i$ and mean burst size $b$ for 18 unregulated promoters
from~\cite{Jones2014}. Each contour indicates the 95\% highest posterior
probability density region for a particular promoter. Note that the vertical
axis is shared with (D). (D) Plots of the burst rate $k_i$ vs.\ the binding
energy for each promoter as predicted in~\cite{Brewster2012}. The dotted line
shows the predicted slope according to Eq.~\ref{eq:bursty_equil_corresp1},
described in text. Each individual promoter is labeled with a unique number in
both (C) and (D) for cross comparison and for comparison with
Figure~\ref{fig2:constit_cartoons}(B).}
\label{fig:constit_post_full}
\end{figure}

\fig{fig:constit_post_full}(C) shows the results, plotting the posterior
distribution from individually MCMC sampling all 18 constitutive promoter
datasets from~\cite{Jones2014}. To aid visualization, rather than plotting
samples for each promoter's posterior as in \fig{fig:constit_post_full}(A), for
each posterior we find and plot the curve that surrounds the 95\% highest
probability density region. What this means is that each contour
encloses approximately 95\% of the samples, and thus 95\% of the probability
mass, of its posterior distribution. Theory-experiment comparisons,
shown in Figure~\ref{figS:ppc_unreg} in Appendix~\ref{sec:bayesian},
display a similar level of agreement between data and predictive samples as for
the bursty model with \textit{lacUV5} in \fig{fig:constit_post_full}(B).

One interesting feature from \fig{fig:constit_post_full}(C) is that burst rate
varies far more widely, over a range of $\sim10^2$, than burst size, confined to
a range of $\lesssim10^1$ (and with the exception of promoter 6, just a span of
3 to 5-fold). This suggests that $k_i$, not $b$, is the key dynamic variable that
promoter sequence tunes.

\subsubsection{Connecting inferred parameters to prior work}
It is interesting to connect these inferences on $k_i$ and $b$ to the work
of~\cite{Brewster2012}, where these same 18 promoters were considered through
the lens of the three-state thermodynamic model (model 2 in
Figure~\ref{fig1:means_cartoons}(B)) and binding energies $\Delta\varepsilon_P$
were predicted from an energy matrix model derived from~\cite{Kinney2010}. As
previously discussed the thermodynamic models of gene regulation can only make
statements about the mean gene expression. This implies that we can draw the
connection between both frameworks by equating the mean mRNA $\left\langle m
\right\rangle$. This results in
\begin{equation}
\langle m \rangle = \frac{k_i b}{\gamma}
        = \frac{r}{\gamma}
        \frac{\frac{P}{N_{NS}}\exp(-\beta\Delta\varepsilon_P)}
                {1+\frac{P}{N_{NS}}\exp(-\beta\Delta\varepsilon_P)}.
\end{equation}
By taking the weak promoter approximation for the thermodynamic model ($P/N_{NS}
\exp(-\beta\Delta\varepsilon_r) \ll 1$) results in~\cite{Brewster2012}
\begin{equation}
\langle m \rangle = \frac{k_i b}{\gamma}
        = \frac{r}{\gamma} \frac{P}{N_{NS}}\exp(-\beta\Delta\varepsilon_P),
\end{equation}
valid for all the binding energies considered here.

Given this result, how are the two coarse-grainings related? A quick estimate
can shed some light. Consider for instance the \textit{lacUV5} promoter, which
we see from Figure~\ref{fig:constit_post_full}(A) has $k_i/\gamma \sim b \sim
\text{few}$, from Figure~\ref{fig:constit_post_full}(B) has $\langle m \rangle
\sim 20$, and from~\cite{Brewster2012} has $\beta\Delta\varepsilon_P \sim -
6.5$. Further we generally assume $P/N_{NS} \sim 10^{-3}$ since
$N_{NS}\approx4.6\times10^6$ and $P\sim10^3$. After some guess-and-check with
these values, one finds the only association that makes dimensional sense and
produces the correct order-of-magnitude for the known parameters is to take
\begin{equation}
\frac{k_i}{\gamma} = \frac{P}{N_{NS}} \exp(-\beta\Delta\varepsilon_P)
\label{eq:bursty_equil_corresp1}
\end{equation}
and
\begin{equation}
b = \frac{r}{\gamma}.
\label{eq:bursty_equil_corresp2}
\end{equation}
Figure~\ref{fig:constit_post_full}(D) shows that this linear scaling between
$\ln k_i$ and $-\beta\Delta\varepsilon_P$ is approximately true for all 18
constitutive promoters considered. The plotted line is simply
Eq.~\ref{eq:bursty_equil_corresp1} and assumes $P\approx 5000$.

While the associations represented by Eq.~\ref{eq:bursty_equil_corresp1} and
Eq.~\ref{eq:bursty_equil_corresp2} appear to be borne out by the data in
Figure~\ref{fig:constit_post_full}, we do not find the association of parameters
they imply to be intuitive. We are also cautious to ascribe too much physical
reality to the parameters. Indeed, part of our point in comparing the various
constitutive promoter models is to demonstrate that these models each provide an
internally self-consistent framework that adequately describes the data, but
attempting to translate between models reveals the dubious physical
interpretation of their parameters.

Our remaining task in this work is a determination of the physical reality of
thermodynamic binding energies in Figure~\ref{fig1:means_cartoons}, as codified
by the thermodynamic-kinetic equivalence of Eq.~\ref{eq:deltaFR_eq_noneq_equiv}.
For our phenomenological needs here model 5 in
Figure~\ref{fig2:constit_cartoons}(A) is more than adequate: the posterior
distributions in Figure~\ref{fig:constit_post_full}(C) are cleanly identifiable
and the predictive checks in Figure~\ref{figS:ppc_unreg} indicate no
discrepancies between the model and the mRNA single-molecule count data
of~\cite{Jones2014}. Of the models we have considered it is unique in satisfying
both these requirements. As a result we will use it as a foundation to build
upon in the next section when we add regulation.

\subsection{Transcription factor kinetics can be inferred from single-cell mRNA
distribution measurements}\label{sec:rep_kinetics_inference}
\subsubsection{Building the model and performing parameter inference}

Now that we have a satisfactory model in hand for constitutive promoters, we
would like to return to the main thread: can we reconcile the thermodynamic and
kinetic models by putting to the test Eq.~\ref{eq:deltaFR_eq_noneq_equiv}, the
correspondence between indirectly inferred thermodynamic binding energies and
kinetic rates? To make this comparison, is it possible to infer repressor
binding and unbinding rates from mRNA distributions over a population of cells
as measured by single-molecule Fluorescence \textit{in situ} Hybridization
in~\cite{Jones2014}? If so, how do these inferred rates compare to direct
single-molecule measurements such as from~\cite{Hammar2014} and to binding
energies such as from~\cite{Garcia2011a} and~\cite{Razo-Mejia2018}, which were
inferred under the assumptions of the thermodynamic models in
Figure~\ref{fig1:means_cartoons}(B)? And can this comparison shed light on the
unreasonable effectiveness of the thermodynamic models, for instance, in their
application in~\cite{Chure2019, Chure2019a}?

\begin{figure}%[h!]
\centering
\includegraphics[width=\textwidth]{../../figures/main/fig04.pdf}
\caption{\textbf{Simple repression parameter inference and comparison.}
(A) Contours which enclose 50\% and 95\% of the posterior probability mass are
shown for each of several 2D slices of the 9D posterior distribution. The model
assumes one unbinding rate for each operator (Oid, O1, O2) and one binding rate
for each aTc induction concentration (corresponding to an unknown mean repressor
copy number). (B, upper) Ratios of our inferred unbinding rates are compared
with operator binding energy differences measured by Garcia and
Phillips~\cite{Garcia2011a} (triangles) and Razo-Mejia et.\
al.~\cite{Razo-Mejia2018} (circles). Blue glyphs compare O2-O1, while orange
compare O1-Oid. Points with perfect agreement would lie on the dotted line. (B,
lower) Unbinding rates for O1 (green) and Oid (red) inferred in this work are
compared with single-molecule measurements from Hammar et.\
al.~\cite{Hammar2014}. We plot the comparison assuming illustrative mRNA
lifetimes of $\gamma^{-1}=3$~min (triangles) or $\gamma^{-1}=5$~min (squares).
Dotted line is as in upper panel. (C) Theory-experiment comparison are shown for
each of the datasets used in the inference of the model in (A). Observed
single-molecule mRNA counts data from~\cite{Jones2014} are plotted as black
lines. The median of the randomly generated samples for each condition is
plotted as a dark colored line. Lighter colored bands enclose 95\% of all
samples for a given operator/repressor copy number pair. The unregulated
promoter, \textit{lacUV5}, is shown with each as a reference.}
\label{fig4:repressed_post_full}
\end{figure}

As we found in Section~\ref{sec:beyond_means}, for our purposes the ``right''
model of a constitutive promoter is the bursty picture, model five in
Figure~\ref{fig2:constit_cartoons}(A). Therefore our starting point here is the
analogous model with repressor added, model 5 in
Figure~\ref{fig1:means_cartoons}(C). For a given repressor binding site and copy
number, this model has four rate parameters to be inferred: the repressor
binding and unbinding rates $k_R^+$, and $k_R^-$, the initiation rate of bursts,
$k_i$, and the mean burst size $b$ (we nondimensionalize all of these by the
mRNA degradation rate $\gamma$).

Before showing the mathematical formulation of our statistical inference model,
we would like to sketch the intuitive structure. The dataset
from~\cite{Jones2014} we consider consists of single-cell mRNA counts data of
nine different conditions, spanning several combinations of three unique
repressor binding sites (the so-called Oid, O1, and O2 operators) and four
unique repressor copy numbers. We assume that the values of $k_i$ and $b$ are
known, since we have already cleanly inferred them from constitutive promoter
data, and further we assume that these values are the same across datasets with
different repressor binding sites and copy numbers. In other words, we assume
that the regulation of the transcription factor does not affect the mean burst
size nor the burst initiation rate. The regulation occurs as the promoter is
taken away from the transcriptionally active state when the promoter is bound by
repressor. We assume that there is one unbinding rate parameter for each
repressor binding site, and likewise one binding rate for each unique repressor
copy number. This makes our model seven dimensional, or nine if one counts $k_i$
and $b$ as well. Note that we use only a subset of the datasets from Jones et.\
al.~\cite{Jones2014}, as discussed more in Appendix~\ref{sec:bayesian}.

Formally now, denote the set of seven repressor rates to be inferred as
\begin{equation}
\vect{k} =\{k_{Oid}^-, k_{O1}^-, k_{O2}^-,
k_{0.5}^+, k_{1}^+, k_{2}^+, k_{10}^+\},
\end{equation}
where subscripts for dissociation rates $k^-$  indicate the different repressor
binding sites, and subscripts for association rates $k^+$ indicate the
concentration of the small-molecule that controlled the expression of the LacI
repressor (see Appendix~\ref{sec:bayesian}). This is because for this particular
dataset the repressor copy numbers were not measured directly, but it is safe to
assume that a given concentration of the inducer resulted in a specific mean
repressor copy number~\cite{Chure2019a}. Also note that the authors
of~\cite{Jones2014} report estimates of LacI copy number per cell rather than
direct measurements. However, these estimates were made assuming the validity of
the thermodynamic models in Figure~\ref{fig1:means_cartoons}, and since testing
these models is our present goal, it would be circular logic if we were to make
the same assumption. Therefore we will make no assumptions about the LacI copy
number for a given inducer concentration.

Having stated the problem, Bayes' theorem reads
\begin{equation}
p(\vect{k}, k_i, b \mid D)
\propto
p(D \mid\vect{k}, k_i, b) p(\vect{k}, k_i, b),
\end{equation}
where $D$ is again the set of all $N$ observed single-cell mRNA counts
across the various conditions. We assume that individual single-cell
measurements are independent so that the likelihood factorizes as
\begin{equation}
p(D \mid\vect{k}, k_i, b)
= \prod_{j=1}^N p(m\mid \vect{k}, k_i, b)
= \prod_{j=1}^N p(m\mid k_j^+, k_j^-, k_i, b)
\end{equation}
where $k_j^\pm$ represent the appropriate binding and unbinding
rates out of $\vect{k}$ for the $j$-th measured cell. The probability
$p(m\mid k_j^+, k_j^-, k_i, b)$ appearing in the last expression
is exactly Eq.~\ref{eq:p_m_bursty+rep_appdx}, the steady-state
distribution for our bursty model with repression derived in
Section~\ref{sec:gen_fcn_appdx}, which for completeness we reproduce here as
\begin{equation}
\begin{split}
p(m \mid k_R^+, k_R^-, k_i, b) = & ~\frac{
        \Gamma(\alpha + m)\Gamma(\beta + m)\Gamma(k_R^+ + k_R^-)
        }
        {
        \Gamma(\alpha)\Gamma(\beta)\Gamma(k_R^+ + k_R^- + m)
        }
\frac{b^m}{m!}
\\
&\times {_2F_1}(\alpha+m, \beta+m, k_R^++k_R^-+m; -b).
\end{split}
\label{eq:p_m_bursty+rep}
\end{equation}
where $_2F_1$ is the confluent hypergeometric function of the second kind and
$\alpha$ and $\beta$, defined for notational convenience, are
\begin{equation}
\begin{split}
\alpha &= \frac{1}{2}
\left(k_i+k_R^-+k_R^+ + \sqrt{(k_i+k_R^-+k_R^+)^2 - 4k_i k_R^-}\right)
\\
\beta &= \frac{1}{2}
\left(k_i+k_R^-+k_R^+ - \sqrt{(k_i+k_R^-+k_R^+)^2 - 4k_i k_R^-}\right).
\end{split}
\end{equation}

This likelihood is rather inscrutable. We did not find any of the known
analytical approximations for ${_2F_1}$ useful in gaining intuition, so we
instead resorted to numerics. One insight we found was that for very strong or
very weak repression, the distribution in Eq.~\ref{eq:p_m_bursty+rep} is well
approximated by a negative binomial with burst size $b$ and burst rate $k_i$
equal to their constitutive \textit{lacUV5} values, except with $k_i$ multiplied
by the fold-change $\left(1+k_R^+/k_R^-\right)^{-1}$. In other words, once again
only the ratio $k_R^+/k_R^-$ was detectable. But for intermediate repression,
the distribution was visibly broadened with Fano factor greater than $1+b$, the
value for the corresponding constitutive case. This indicates that the repressor
rates had left an imprint on the distribution, and perhaps intuitively, this
intermediate regime occurs for values of $k_R^\pm$ comparable to the burst rate
$k_i$. Put another way, if the repressor rates are much faster or much slower
than $k_i$, then there is a timescale separation and effectively only one
timescale remains, $k_i\left(1+k_R^+/k_R^-\right)^{-1}$. Only when all three
rates in the problem are comparable does the mRNA distribution retain detectable
information about them.

Next we specify priors. As for the constitutive model, weakly informative
log-normal priors are a natural choice for all our rates. We found that if the
priors were too weak, our MCMC sampler would often become stuck in regions of
parameter space with very low probability density, unable to move. We struck a
balance in choosing our prior widths between helping the sampler run while
simultaneously verifying that the marginal posteriors for each parameter were
not artificially constrained or distorted by the presence of the prior.
% The only
% exception to this is the highly informative priors we placed on $k_i$ and $b$,
% since we have strong knowledge of them from our inference of constitutive
% promoters above.
All details for our prior distributions are listed in
Appendix~\ref{sec:bayesian}.

We ran MCMC sampling on the full nine dimensional posterior specified by this
model. To attempt to visualize this object, in
Figure~\ref{fig4:repressed_post_full}(A) we plot several two-dimensional slices
as contour plots, analogous to Figure~\ref{fig:constit_post_full}(C). Each of
these nine slices corresponds to the $(k_R^+, k_R^-)$ pair of rates for one of
the conditions from the dataset used to fit the model and gives a
sense of the uncertainty and correlations in the posterior.
We note that the 95\% uncertainties of all the rates span about $\sim0.3$
log units, or about a factor of two, with the exception of $k_{0.5}^+$, the
association rate for the lowest repressor copy number which is somewhat larger.

\subsubsection{Comparison with prior measurements of repressor binding energies}
Our primary goal in this work is to reconcile the kinetic and thermodynamic
pictures of simple repression. Towards this end we would like to compare the
repressor kinetic rates we have inferred with the repressor binding energies
inferred through multiple methods in~\cite{Garcia2011a}
and~\cite{Razo-Mejia2018}. If the agreement is close, then it suggests that the
thermodynamic models are not wrong and the repressor binding energies they
contain correspond to physically real free energies, not mere fit parameters.

Figure~\ref{fig4:repressed_post_full}(B) shows both comparisons, with the top
panel comparing to thermodynamic binding energies and the bottom panel comparing
to single-molecule measurements. First consider the top panel and its comparison
between repressor kinetic rates and binding energies. As described in
section~\ref{section_02_means}, if the equilibrium binding energies
from~\cite{Garcia2011a} and~\cite{Razo-Mejia2018} indeed are the physically real
binding energies we believe them to be, then they should be related to the
repressor kinetic rates via Eq.~\ref{eq:deltaFR_eq_noneq_equiv}, which we
restate here,
\begin{equation}
\Delta F_R = \beta\Delta\varepsilon_R - \log(R/N_{NS})
        = - \log(k_R^+/k_R^-).
\label{eq:deltaFR_eq_noneq_equiv_repeat}
\end{equation}
Assuming mass action kinetics implies that $k_R^+$ is proportional to repressor
copy number $R$, or more precisely, it can be thought of as repressor copy
number times some intrinsic per molecule association rate. But since $R$ is not
directly known for our data from~\cite{Jones2014}, we cannot use this equation
directly. Instead we can consider two different repressor binding sites and
compute the \textit{difference} in binding energy between them, since this
difference depends only on the unbinding rates and not on the binding rates.
This can be seen by evaluating Eq.~\ref{eq:deltaFR_eq_noneq_equiv_repeat} for
two different repressor binding sites, labeled (1) and (2), but with the same
repressor copy number $R$, and taking the difference to find
\begin{equation}
\Delta F_R^{(1)} - \Delta F_R^{(2)}
= \beta\Delta\varepsilon_1 - \beta\Delta\varepsilon_2
= - \log(k_R^+/k_1^-) + \log(k_R^+/k_2^-),
\end{equation}
or simply
\begin{equation}
\beta\Delta\varepsilon_1 - \beta\Delta\varepsilon_2
= \log(k_2^-/k_1^-).
\end{equation}
The left and right hand sides of this equation are exactly the horizontal and
vertical axes of the top panel of Figure~\ref{fig4:repressed_post_full}. Since
we inferred rates for three repressor binding sites (O1, O2, and Oid), there are
only two independent differences that can be constructed, and we arbitrarily
chose to plot O2-O1 and O1-Oid in Figure~\ref{fig4:repressed_post_full}(B).
Numerically, we compute values of $k_{O1}^- / k_{Oid}^-$ and $k_{O2}^- /
k_{O1}^-$ directly from our full posterior samples, which conveniently provides
uncertainties as well, as detailed in Appendix~\ref{sec:bayesian}.
% \mmnote{Explain in appendix how marginalization with sampling makes this
% ``trivial'' to compute from samples of the joint posterior, b/c sampling makes
% marginalization trivial.}
We then compare these log ratios of rates to the
binding energy differences $\Delta\varepsilon_{O1} - \Delta\varepsilon_{Oid}$
and from $\Delta\varepsilon_{O2} - \Delta\varepsilon_{O1}$ as computed from the
values from both~\cite{Garcia2011a} and~\cite{Razo-Mejia2018}. Three of the four
values are within $\sim0.5~k_BT$ of the diagonal representing perfect agreement,
which is comparable to the $\sim\text{few}\times0.1~k_BT$ variability between
the independent determinations of the same quantities between~\cite{Garcia2011a}
and~\cite{Razo-Mejia2018}. The only outlier involves Oid measurements
from~\cite{Razo-Mejia2018}, and as the authors of~\cite{Razo-Mejia2018} note,
this is a difficult measurement of low fluorescence signal against high
background since Oid represses so strongly. We are therefore inclined to regard
the failure of this point to fall near the diagonal as a testament to the
difficulty of the measurement and not as a failure of our theory.

On the whole then, we find these to be results very suggestive. Although not
conclusively, our results point at the possible interpretation of the
phenomenological free energy of transcription factor binding as the log of a
ratio of transition rates. This did not need to be the case; the free energy was
inferred from bulk measurements of mean gene expression relative to an
unregulated strain --the fold-change-- while the rates were extracted from
fitting single-molecule counts to the full mRNA steady state distribution.
Further single molecule experiments along the lines of~\cite{Hammar2014} could
shed further light into this idea.
% On the whole then, we regard this as striking confirmation of the validity of
% the thermodynamic models. Their lynchpin parameter is a phenomenological free
% energy of repressor binding that has previously only been inferred indirectly.
% Our result shows that the microscopic interpretation of this free energy, as the
% log of a ratio of transition rates, does indeed hold true to within the inherent
% uncertainties that remain in the entire theory-experiment dialogue.

\subsubsection{Comparison with prior measurements of repressor kinetics}
In the previous section we established the equivalence between the equilibrium
models' binding energies and the repressor kinetics we infer from mRNA
population distributions. But one might worry that the repressor rates we infer
from mRNA distributions are \textit{themselves} merely fitting parameters and
that they do not actually correspond to the binding and unbinding rates of the
repressor in vivo. To verify that this is not the case, we next compare our
kinetic rates with a different measurement of the same rates using a radically
different method: single molecule measurements as performed in Hammar et.\
al.~\cite{Hammar2014}. This is plotted in the lower panel of
Figure~\ref{fig4:repressed_post_full}(B).

Since we do not have access to repressor copy number for either the
single-cell mRNA data from~\cite{Jones2014} or the single-molecule data
from~\cite{Hammar2014}, we cannot make an apples-to-apples comparison of
association rates $k_R^+$. Further, while Hammar et.\ al.\ directly measure the
dissociation rates $k_R^-$, our inference procedure returns $k_R^-/\gamma$,
i.e., the repressor dissociation rate nondimensionalized by the mRNA degradation
rate $\gamma$. So to make the comparison, we must make an assumption for the
value of $\gamma$ since it was not directly measured. For most mRNAs in
\textit{E.\ coli}, quoted values for the typical mRNA lifetime $\gamma^{-1}$
range between about 2.5~min~\cite{Chen2015} to 8~min. We chose $\gamma^{-1} =
3$~min and $\gamma^{-1} = 5$~min as representative values and plot a comparison
of $k_{O1}^-$ and $k_{Oid}^-$ from our inference with corresponding values
reported in~\cite{Hammar2014} for both these choices of $\gamma$.

The degree of quantitative agreement in the lower panel of
Figure~\ref{fig4:repressed_post_full}(B) clearly depends on the precise choice
of $\gamma$. Nevertheless we find this comparison very satisfying, when two
wildly different approaches to a measurement of the same quantity yield broadly
compatible results. We emphasize the agreement between our rates and the rates
reported in~\cite{Hammar2014} for any reasonable $\gamma$: values differ by at
most a factor of 2 and possibly agree to within our uncertainties of 10-20\%.
From this we feel confident asserting that the parameters we have inferred from
Jones et.\ al.'s single-cell mRNA counts data do in fact correspond to repressor
binding and unbinding rates, and therefore our conclusions on the agreement of
these rates with binding energies from~\cite{Garcia2011a}
and~\cite{Razo-Mejia2018} are valid.

\subsubsection{Model checking}
In Figure~\ref{fig:constit_post_full}(B) we saw that the simple Poisson model of
a constitutive promoter, despite having a well behaved posterior, was clearly
insufficient to describe the data. It behooves us to carry out a similar check
for our model of simple repression, codified by Eq.~\ref{eq:p_m_bursty+rep} for
the steady-state mRNA copy number distribution. As derived in
Sections~\ref{section_02_means} and~\ref{sec:beyond_means}, we have compelling
theoretical reasons to believe it is a good model, but if it nevertheless turned
out to be badly contradicted by the data we should like to know.

The details are deferred to Appendix~\ref{sec:bayesian}, and here we only
attempt to summarize the intuitive ideas, as detailed at greater length by
Jaynes~\cite{Jaynes2003} as well as Gelman and
coauthors~\cite{Gelman2013,Gelman2013a}. From our samples of the posterior
distribution, plotted in Figure~\ref{fig4:repressed_post_full}(A), we generate
many replicate data using a random number generator. In
Figure~\ref{fig4:repressed_post_full}(C), we plot empirical cumulative
distribution functions of the middle 95\% quantiles of these replicate data with
the actual experimental data from Jones et.\ al.~\cite{Jones2014} overlaid,
covering all ten experimental conditions spanning repressor binding sites and
copy numbers (as well as the constitutive baseline UV5).

The purpose of Figure~\ref{fig4:repressed_post_full}(C) is simply a graphical,
qualitative assessment of the model: do the experimental data systematically
disagree with the simulated data, which would suggest that our model is missing
important features? A further question is not just whether there is a detectable
difference between simulated and experimental data, but whether this difference
is likely to materially affect the conclusions we draw from the posterior in
Figure~\ref{fig4:repressed_post_full}(A). More rigorous and quantitative
statistical tests are possible~\cite{Gelman2013}, but their quantitativeness
does not necessarily make them more useful. As stated in~\cite{Gelman2013a}, we
often find this graphical comparison more enlightening because it better engages
our intuition for the model, not merely telling \textit{if} the model is wrong
but suggesting \textit{how} the model may be incomplete.

Our broad brush takeaway from Figure~\ref{fig4:repressed_post_full}(C) is
overall of good agreement. There some oddities, in particular the long tails in
the data for Oid, 1~ng/mL, and O2, 0.5~ng/mL. The latter is especially odd since
it extends beyond the tail of the unregulated UV5 distribution. This is a
relatively small number of cells, however, so whether this is a peculiarity of
the experimental data, a statistical fluke of small numbers, or a real
biological effect is unclear. It is conceivable that there is some very slow
timescale switching dynamics that could cause this bimodality, although it is
unclear why it would only appear for specific repressor copy numbers. There is
also a small offset between experiment and simulation for O2 at the higher
repressor copy numbers, especially at 2 and 10~ng/mL. From the estimate of
repressor copy numbers from~\cite{Jones2014}, it is possible that the repressor
copy numbers here are becoming large enough to partially invalidate our
assumption of a separation of timescales between burst duration and repressor
association rate. Another possibility is that the very large number of zero mRNA
counts for Oid, 2~ng/mL is skewing its partner datasets through the shared
association rate. None of these fairly minor potential caveats cause us to
seriously doubt the overall correctness of our model, which further validates
its use to compare the thermodynamic models' binding energies to the kinetic
models' repressor binding and unbinding rates, as we originally set out to do.


%#!platex Naruse_Esurf_2020.tex

\section{Results}

\subsection{Properties of artificial data sets of turbidites}
Here, we describe the properties of turbidite artificial data generated for training and testing the inverse model. Several artificial datasets of turbidites are produced using a 1-D shallow water equation model. Figure \ref{fig:training_examples_artificial_turbidites} exhibits examples of the calculated spatial distribution in bed thickness and grain size of turbidites deposited in the region of the basin plain. Most beds exhibit the typical ``top-hat'' or ``core and drape'' shape of turbidites \cite{Hirayama1977,Talling2012,Pantopoulos2013}, where turbidite beds become thicker in the upstream part of the basin and then thin rapidly from their peak of thickness. Thereafter, beds continue over a long distance, gradually decreasing in thickness (Fig. \ref{fig:training_examples_artificial_turbidites}). At the same time, the grain size gradually becomes finer downstream. The maximum thickness of beds is 1.27 m on average (standard deviation $\sigma = 1.65$ m), and the mean value of the area where sediments with a thickness greater than 1 cm are distributed is 42.0 km ($\sigma=15.7$ km). Each bed is composed of four grain size classes. All distributions of the volume-per-unit-area of the grain size classes are still ``top hat'' shaped (Fig. \ref{fig:training_examples_artificial_turbidites}b), but the depositional center and the amounts of deposition are different for each classes depending on their size.

\begin{figure}[t]
  \includegraphics[width=8cm]{fig04.eps}
  \caption{Examples of turbidites calculated by the forward model. \textsf{a}. Spatial distributions of bed thickness. Four beds (Bed 1--4) were plotted as examples. \textsf{b}. Spatial distribution of the volume-per-unit-area for each grain size class in Bed 1 (Fig. 4a).}
  \label{fig:training_examples_artificial_turbidites}
\end{figure}

\subsection{Results of training}
We trained the NN inverse model with various numbers of artificial data and lengths of the sampling window, and the best result in terms of the value of the loss function for the validation sets and the practical usage of the model can be obtained with 3500 training data sets and 10 km-long sampling window  (Fig. \ref{fig:training_different_number_length}). Results with less than 2000 training data sets produce a discrepancy in the loss function between the training and the validation sets, indicating overlearning of the NN. Conversely, when the number of data sets exceeds 2000, the loss function of the validation set is slightly less than the value of the training set. As the number of training data increases, the resultant values of the loss function improve. However, when the number of data exceeds 2500, the improvement of values of the loss function became not so rapid. Regarding the distance of the sampling window, the training results are not stable when the sampling window is shorter than 5 km (Fig. \ref{fig:training_different_number_length}). On the other hand, the training results are stable when the window length is longer than 10 km, and the results gradually improve as the window length increases. However, extending the window length from 10 km to 30 km results in little improvement of the loss function. We do not fully understand why the results are not stable for sampling windows shorter than 5 km, but it probably indicates that the training results fall into a local optimum solution depending on the initial values of the weight coefficients of the neural network (given by random numbers) due to incomplete information. In any case, the loss function is very good (less than 0.01), so that even turbidites that can be tracked for less than 5 km are likely to give good results if the outcrop spacing is sufficiently narrow and detailed observation of beds is possible.

\begin{figure}[t]
  \includegraphics[width=8cm]{fig05.eps}
  \caption{Results of training of the NN with different numbers of training data sets and lengths of the sampling window.}
  \label{fig:training_different_number_length}
\end{figure}

Hereafter, we further investigate the performance of the inverse model trained on a 3500 dataset with a 10 km-long sampling window. The history of training indicates that the values of the loss function improved significantly in the first 1000 epochs, and the results are improved up to 15,000 epochs (Fig. \ref{fig:training_history}). Eventually, saturation is reached at approximately 20,000 epochs. The resultant loss function (i.e., the MSE of prediction) is $3.78 \times 10^{-3}$ for training sets and is $1.03 \times 10^{-3}$ for validation sets.

\begin{figure}[t]
  \includegraphics[width=8cm]{fig06.eps}
  \caption{Training History of the NN. 3500 datasets and 10 km-long sampling window were used for this training.}
  \label{fig:training_history}
\end{figure}

\subsection{Precision and accuracy of inverse analysis}
Using 300 test data sets, the performance of the inverse model trained with 3500 data sets and 10 km-long sampling window is evaluated. The estimated parameters are matched well with slight deviations (Figs. \ref{fig:test_scatter_plot}, \ref{fig:test_histogram_deviation}; Table \ref{table:bias_errors}). $R^2$ values are beyond 0.98 for all parameters. Particularly good agreement is obtained for the estimates of the initial height and the length of the suspended sediment cloud. Values of the normalized RMSE and MAE for these parameters are less than 9 \% and 6 \%, respectively. The sediment concentration is also precisely estimated. the normalized RMSE for the sediment concentration ranges from 12 to 16 \%, which corresponds to only 0.02--0.03 volumetric \%. The prediction for the basin slope shows relatively large errors (RMSE is close to 20 \% and MAE is 11.7 \%), but these errors correspond to only 0.03 \% of slope. Focusing on the bias of the estimates, all estimated values except for the basin slope tend to be slightly smaller, whereas the predicted values of the basin slope tend to be  larger (Fig. \ref{fig:test_histogram_deviation}). The values of the bias, however, range only from 2 to 12\% of the original value.

\begin{figure*}[t]
  \includegraphics[width=12cm]{fig07.eps}
  \caption{Result of the inverse analysis compared with the true parameters. The x-axis represents the value of the true parameter and the y-axis represents the value of the estimated parameter. The orange lines show that the two values are in a 1:1 relationship, and thus the plots on this line indicate that the prediction is perfectly consistent with the true value.}
  \label{fig:test_scatter_plot}
\end{figure*}

\begin{figure*}[t]
  \includegraphics[width=12cm]{fig08.eps}
  \caption{Histograms indicating the deviation of the predicted values from the true values.}
  \label{fig:test_histogram_deviation}
\end{figure*}

\begin{table*}[t]
  \caption{Errors and bias of the predicted parameters. Prediction errors are exhibited by the root mean squared error (RMSE) and the mean absolute error (MAE), and the mean bias is also described. Normalized values of RMSE, MAE and mean bias by true values are also shown.}
  \begin{tabular}{lrrrrrrr}
    \hline
    {} &  $R^2$ &  RMSE &  \begin{tabular}{c} RMSE \\ (normalized) \end{tabular} &   MAE &  \begin{tabular}{c} MAE \\ (normalized) \end{tabular} &  Mean bias & \begin{tabular}{c} Mean bias \\(normalized) \end{tabular} \\
    \hline
    Initial height & 0.99 & 18.97 m &               8.55 \% & 14.81 m &              5.96 \%&     -12.93 m&                   -5.18 \%\\
Initial length & 0.99 & 15.82 m&               7.53 \%& 12.09 m&              4.92 \%&      -2.33 m&                   -2.06 \%\\
$C_1$            & 0.99 &  0.02 \%&              12.91 \%&  0.02 \%&              6.00 \%&      -0.01 \%&                   -4.44 \%\\
$C_2$            & 0.99 &  0.02 \%&              15.57 \%&  0.02 \%&              7.67 \%&      -0.01 \%&                   -4.29 \%\\
$C_3$            & 0.99 &  0.02 \%&              13.03 \%&  0.02 \%&              6.39 \%&      -0.00 \%&                   -2.49 \%\\
$C_4$            & 0.99 &  0.03 \%&              13.71 \%&  0.02 \%&              6.67 \%&      -0.01 \%&                   -4.21 \%\\
$S_l$            & 0.98 &  0.03 \%&              19.56 \%&  0.03 \%&             11.67 \%&       0.03 \%&                   11.45 \%\\
  \hline
    \end{tabular}
  \label{table:bias_errors}
\end{table*}

The forward model is calculated again using the reconstructed values to examine the influence of the estimation error of the model input parameters on the predicted flow behavior (Fig. \ref{fig:test_example_time_evolution}). The chosen test values deviate from the true conditions as indicated by the RMSE value 0.27 (Table \ref{table:example_time_evolution}), but the time evolution of the flow characteristics agree very well with those calculated from the true values (Fig. \ref{fig:test_example_time_evolution}). When comparing the velocity and concentration of the flow at 10 km from the upstream end, the discrepancy between calculation results using reconstructed and original parameters is less than 5\% for both parameters.

\begin{table*}[t]
  \caption{The predicted and the true parameters used for an example of calculation for time evolution of the flow characteristics.}
  \begin{tabular}{lrrrrrrr} \hline
        {} &  Initial height (m) &  Initial length (m) &      $C_1$ &      $C_2$ &      $C_3$ &      $C_4$ &      $S_l$ \\
        \hline
        True input parameters &              484.41 &              318.18 &     0.17 &     0.05 &     0.95 &     0.74 &     0.23 \\
Estimated parameters  &              454.67 &              301.73 &     0.18 &     0.02 &     0.93 &     0.73 &     0.25 \\
    \hline
        \end{tabular}
  \label{table:example_time_evolution}
\end{table*}

\begin{figure}[t]
  \includegraphics[width=8cm]{fig09.eps}
  \caption{Example of forward model calculation with reconstructed and true parameters. The solid line indicates the calculation result using the predicted parameters, and the dashed line exhibits the results using the true parameters. \textsf{a}. Velocity distribution at 2000, 3500 and 5000 seconds after the flow initiation. \textsf{b}. Total sediment concentration at 2000, 3500 and 5000 seconds after the flow initiation. \textsf{c}. Spatial distribution of bed thickness at 25,000 seconds after the flow initiation. }
  \label{fig:test_example_time_evolution}
\end{figure}

\subsection{Tests for robustness against noise and subsampling on input data}
The test data with various amounts of normal random values are analyzed to verify the robustness of the inverse model. Consequently, even when the standard deviation of the normal random numbers given as measurement errors was set to approximately 200\% of the value of the original data, only a small effect is observed in the normalized RMS of the results of the inverse analysis (Fig. \ref{fig:test_noise}). The RMS values gradually increase when the standard deviation of errors exceeds 50\%, but there is no rapid increase in the RMSE of the results at any particular threshold.

\begin{figure}[t]
  \includegraphics[width=8cm]{fig10.eps}
  \caption{Result of inverse analysis of the test data sets with artificial noise. The values of RMSE are averaged over 20 times iterations. Error bars indicate standard errors of RMSE values.}
 \label{fig:test_noise}
\end{figure}

Similarly, using subsampling data obtained by extracting some of the spatial grids from the original data, we conducted an inverse analysis of the test datasets. The results show that there is little influence on the RMSE values of the inverse analysis of the test datasets when the sampling rate of grids is greater than 1 \% (Fig. \ref{fig:test_subsampling}). The RMSE values gradually increased when the sampling rate falls below 1 \%, and RMSE becomes extremely high when the rate drops below 0.4\%.

\begin{figure}[t]
  \includegraphics[width=8cm]{fig11.eps}
  \caption{Results of the inverse analysis for the subsampled test data sets. The values of RMSE are averaged over 20 times iterations. Error bars indicate standard errors of RMSE values.}
 \label{fig:test_subsampling}
\end{figure}

\subsection{Tests for influence of length of the upstream slope}
Here, a turbidite deposited in a different topographic setting was analyzed to determine the influence of the topographic assumptions to inversion results. The slope of 10 km instead of 5 km was set at the upstream end of the calculation domain. The initial conditions for this test assuming a 10 km slope were a suspended sediment cloud of 359 m high and 227 m long, with concentrations of 0.13 \%, 0.15 \%, 0.38 \%, and 0.65 \% for the four grain size classes, respectively. The gradient of the downstream slope was set to be 0.69 \%.

As a result, the initial conditions estimated by the inverse model trained on the assumption of 5 km upstream slope were a suspended sediment cloud of 117 m high and 587 m long, with concentrations of 0.33 \%, 0.38 \%, 0.48 \%, and 0.53 \% for each grain size class, and a downstream slope was estimated to be 0.96 \%. Then, these initial conditions were given to the forward model to calculate their time development, and the obtained parameters were compared on a basin plain where the turbidite was deposited (Fig. \ref{fig:test_slope_length}).

The results exhibited that the model with a 5 km slope predicted relatively close values to the original results for the flow velocity (Fig. \ref{fig:test_slope_length}). Both the model with a 5 and 10 km slope calculated velocities that were approximately 3 m/s in maximum over the basin plain and gradually decelerated downstream. However, because the slope length is different, the time to reach each point on the basin plain differs greatly.

In contrast, the concentration of the turbidity current was significantly overestimated in the model reconstruction assuming a 5 km slope (Fig. \ref{fig:test_slope_length}). When the flow reaches the downstream gentle slope at about 10 km, the original turbidity concentration was about 0.2 \% at maximum, while the restored value was closer to 0.5 \%. As a result, the thickness of turbidite estimated from the reconstructed initial values was also thicker than the original values.

\begin{figure}[t]
  \includegraphics[width=8cm]{fig09_slope.eps}
  \caption{Influence of the length of upstream slope to the result of inverse analysis. 10 km long slope was used to produce a turbidite, and the bed was analyzed by the inverse model trained with 5 km long upstream slope. Solid lines are values for currents produced the bed, and the dashed lines are reconstructed values.  a. Time development of flow velocity. b. Total sediment concentration. c. Distribution of bed thickness.}
 \label{fig:test_slope_length}
\end{figure}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                                                                 %
%  GEANT manual in LaTeX form                                     %
%                                                                 %
%  Michel Goossens (for translation into LaTeX)                   %
%  Version 1.00                                                   %
%  Last Mod. Jan 24 1991  1300   MG + IB                          %
%                                                                 %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Authors {F.Carminati} \Origin{N.Van Eijndhoven}
\Version{Geant 3.16}\Routid{PHYS337}
\Submitted {26.07.93}\Revised{16.12.93}
\Makehead {Birks' saturation law}
\section{Subroutines}
\Shubr{GBIRK}{(EDEP)}

\begin{DLtt}{MMMMMMMM}
\item[EDEP] ({\tt REAL}) and on output the energy
equivalent to the calorimeter response in the current step.
\end{DLtt}

Organic scintillators are usually calibrated with particles whose
energy is near the minimum ionisation ($\gamma \beta \approx 4$).
However, the energy response of such scintillators
for large local energy deposit is
attenuated. \Rind{GBIRK} returns the energy {\it detected} by
the scintillator in the current step.

\section{Method}

The phenomenological description of the response attenuation of
organic scintillators~\cite{bib-BIRK} is known as Birks' law:
\begin{equation}
R = \frac{\Delta E}{1 + C_1 \delta + C_2 \delta^2}
\hspace{4cm}
\delta = \frac{1}{\rho} \frac{dE}{dx} \hspace{0.5cm}
\mbox{MeV g$^{-1}$ cm$^{2}$}
\end{equation}
The values quoted in~\cite{bib-BIR1} for the parameters $C_1, C_2$
are:
\[
C_1 = 0.013 \: \mbox{g MeV$^{-1}$ cm$^{-2}$ \hspace{1.5cm} and \hspace{1.5cm}}
C_2 = 9.6 \times 10^{-6} \: \mbox{g$^{2}$ MeV$^{-2}$ cm$^{-4}$}
\]
These values have been measured for various scintillators.
If the charge of the particle is greater than one, a better description
can be obtained by correcting $C_1$:
\begin{equation}
\label{eq:phys337-1}
C_1' = \frac{7.2}{12.6} C_1 \approx 0.5714 C_1
\end{equation}

The values of the parameters of Birks' formula (if defined) are in the
{\tt ZEBRA} bank next to the material bank:
\begin{DLtt}{MMMMMMMMMMMMMMMMMMMM}
\item[JTM = LQ(JMATE-NTMED)] pointer to the current tracking medium bank;
\item[JTMN = LQ(JTM)] pointer to the {\it next} bank to the current
material one, where the specific tracking medium parameters are stored;
\item[MODEL = Q(JTMN+27)] flag controlling the correction (\ref{eq:phys337-1})
for multiply charged particles:
\begin{DLtt}{MMMM}
\item[$\neq$1] correction is not applied;
\item[$=$1] correction is applied;
\end{DLtt}
\item[C1 = Q(JTMN+28)] first parameter of the Birks' formula;
\item[C2 = Q(JTMN+29)] second parameter of the Birks' formula.
\end{DLtt}
These parameters are set via the \Rind{GSTPAR} routine, with
names {\tt BIRK1}, {\tt BIRK2} and {\tt BIRK3}, respectively. For instance,
to define the standard Birks' parameters for tracking medium {\tt ITM}
with correction for multiply charged particles, one would have to insert
the following piece of code, after the definition of tracking media
but before the call to \Rind{GPHYSI}
\begin{verbatim}
      CALL GSTPAR(ITM,'BIRK1',1.)
      CALL GSTPAR(ITM,'BIRK2',0.013)
      CALL GSTPAR(ITM,'BIRK3',9.6E-6)
\end{verbatim}