\section{Deep Reinforcement Learning}
	In this section the content of the two developed notebooks is discussed, mainly the theory behind \textit{DRL}\footnote{Deep Reinforcement Learning}. The reader gets to know the basic concept of software agents and RL, the deep learning aspect with the help of an exemplary neural net and work out the necessary formulas in order to translate the theory into code. Further the environment is introduced, which defines the problem, that the algorithm needs to solve and also holds all rules, which constrain the algorithm.  All the theory, explanations and code can be found in full length in the enclosed notebooks:
	\begin{enumerate}
		\item Deep Reinforcement Learning Beginners Tutorial (1) - Theory
		\item Deep Reinforcement Learning Beginners Tutorial (2) - Practice
	\end{enumerate}

\subsection{Software Agent}
	Before the topic of RL can be explained, some basics need to be addressed in order to clarify the following explanation.
	For this, the term of \textit{software agent} is introduced, which, as part of artificial intelligence, is basically a program, that acts self-sufficient to solve a task.
	There are three important aspects an agent should fulfill.
	First is autonomous action, which means that an agent needs to make decisions without external help.
	Further an agent needs to execute multiple actions to complete its task.
	This property is called proactive.
	Last but not least, an agent has to be reactive, which means it reacts on changes in the environment it is in.
	An optional ability, for an agent like this, is self-sufficient improvement. For this the agent builds up knowledge after repetitively doing its task and improves itself this way.\\
	Basically, the part of the agent, which controls its actions, can be filled with different algorithms. In this case, this will be an implementation of RL, which fits all requirements of an agent.

\subsection{Reinforcement Learning}
	Reinforcement Learning is considered one of the three machine learning paradigms, alongside supervised learning and unsupervised learning.
	The main goal of RL is to create an agent, which can navigate by actions in an environment in a way to maximize a representation of a cumulative reward.
	The environment is the space that contains the agent.
	The environment is discussed in detail later on.
	In order to maximize the reward, the agent has to learn by trial and error which actions most likely lead to a reward and which lead to a penalty.
	After some training, the agent will use its knowledge to avoid previous mistakes.
	It still has to explore the environment, because otherwise it can not be sure if it actually finds the global maximum or is just stuck at a local one and if there might be a better chain of actions.
	If this is compared to the real world, this method of learning is pretty close the human learning.
	If a person gets hurt for example, the person is more likely to try to avoid the situation in the future.
	Still curiosity or necessity leads humans to exploring and thus maybe getting them into danger, but in the best case, will lead them to some kind of positive reward, like food for example.

	\subsubsection{The Concept of RL}
		An agent is contained by an environment.
		A momentary snapshot of the environment is called state.
		A state contains all information at a certain point in time for example the position of the agent or enemies.
		There is a permanent exchange of information between the agent and its environment.
		The agent receives the actual state and has to choose an action based on its logic. Everything the agent can use to alter its environment is called an action.
		Executing an action changes the environment based on a certain set of rules.
		After executing an action the agents receives the new state and a reward, which helps to decide whether its decision leads to a positive result.
		A reward can also be negative.
		The main goal for the agent is to learn the best action for each state, so that it can maximize the reward.
		Basically, an environment is just a set of states.
		The agent can move between states by executing actions.
		A good example for an environment is the universe.
		Humans and other living beings are agents, moving in the world.
		All the time they have to look at their surroundings and choose an action like moving across the street or waiting until it is free.
		The laws of physics restrain them, take gravity for example.
		All living things receive rewards like getting hurt or feeling satisfied, that is how they evaluate the actions.
		The environment is constantly changing, so they have to reevaluate their decisions and choose an action again.
		The following picture shows the whole procedure:\\

		\begin{figure}[h!]
			\begin{center}
				\includegraphics[width=0.7\linewidth]{img/eoeSq.png}
				\caption{Concept of Reinforcement Learning}
				\label{concept}
			\end{center}
		\end{figure}

	\subsubsection{Deep Learning Aspect}
		The agent has to recognize and differentiate elements in the environment, to be able to act accordingly and choose a proper action.
		In the best case, the input is already reduced to some important numbers, which describe the actual state.
		In real applications this may not be possible or feasible with the given knowledge.
		Most of the time there is a picture available, for example the image of a camera or a game screen, but the input the agent receives can basically be any sensory data.\\
		Furthermore the agent must associate the occurrence of individual elements on screen with its own actions and the subsequent reward or penalty, which may occur only after several steps in the future.
		Depending on the situation it may be also necessary to estimate what the next state of the game could be.
		There are two possible ways to use the \textit{deep learning} aspect.
		\begin{enumerate}
			\item Image processing with a CNN\footnote{Convolutional Neural Network}, in order to process the image information and send this to the agent.
			\item A DQN\footnote{Deep Q-Learning Network}, in order to model the Q-function, which is discussed below.
		\end{enumerate}
		Following is an exemplary presentation of a neural network, which is designed according to the tasks just mentioned.
		\begin{figure}[h!]
			\begin{center}
				\includegraphics[width=0.7\linewidth]{img/v2-67ef75bb7f5e67b2a42645aa821894bf_hd.png}
				\caption{DQN with Convolutional Layers}
				\label{cnndqn}
			\end{center}
		\end{figure}\\
		There are two ways to learn in deep reinforcement learning.
		Either the agent can use Value Learning or Policy Learning.
		With Value Learning, it assigns values to each state-action pair, which correspond to the probabilities of the pair being the best option.
		In Policy Learning, it will learn a strategy instead, which gives it directly the best estimated action for a given state s.
		In the notebook, the focus will be on Value Learning.

	\subsubsection{Q-Learning}
		The task of the agent is the maximization of a cumulative future reward.
		An example of this can be the score in a game.
		In an environment, there is no guarantee for an immediate reward after executing an action.
		In many cases, only specific chains of actions lead to a positive reward, so the agent needs to learn multiple moves in succession in order to fulfill the given task.
		For example, if the agent needs to collect an object in order to get a higher score, it may be necessary to take multiple moves to reach it.
		The moves in between may not increase the reward significantly or even reduce it, however they are needed to complete the task in order to be successful.
		So the agent may have to learn to sacrifice some rewards, to earn an overall higher reward in the end.
		A simple way to implement RL is Q-Learning. A numerical value gets assigned to each action which can executed per state.
		This value is called Q-value.
		It represents an estimate of which action will result in the highest reward for the actual state.
		This is the knowledge of the agent.
		In the beginning all these values are initialized by 0.
		Each step the agent has to choose an action for the actual state.
		This is either random or knowledge based, but all the actions taken, provide data the agent is trained with.
		The agent uses random actions to explore its environment at first.
		If it uses its knowledge instead, the next action is chosen by exploitation, so the action with the highest Q-value for the given state is used.
		The next step is the associated function.
		The Q-function is the name giving aspect of a Deep Q-Learning Network, while the "Q" stands for "Quality".
		The higher the reward, the higher the estimated Q-value and associated quality.
		That means this function calculates a Q-value for the actual state, which represents the best action to take.
		Therefore taking the highest Q-value associated with the biggest estimated future total reward $Q(s', a')$, the last gained reward $r_t$ and a discount factor $\gamma$ into account is needed, in order to make rewards in the near future more decisive for the agent.
		The main formula to model such a behaviour is the so called \textit{Bellman equation}.

		\begin{figure}[h!]
			\begin{center}
				$Q(s_t, a_t) = r_t + \gamma * \max\limits_{a'} Q(s', a')$
			\end{center}
		\end{figure}


\section{Partion-based graph abstraction (PAGA)}

The original method is described here: \href{https://doi.org/10.1186/s13059-019-1663-x}{PAGA}

\begin{figure}[H]
\includegraphics[width=1.0\textwidth,height=0.9\textheight,keepaspectratio]{{{\pagaDir/paga}}}
\caption{PAGA graph}
\end{figure}

\subsection{UMAP}

\begin{figure}[H]
\includegraphics[width=1.0\textwidth,height=0.9\textheight,keepaspectratio]{{{\pagaDir/umap.paga.initialised}}}
\caption{PAGA initialised UMAP}
\end{figure}

\subsection{Force directed graphs}

\begin{figure}[H]
\includegraphics[width=1.0\textwidth,height=0.9\textheight,keepaspectratio]{{{\pagaDir/draw_graph_fa}}}
\caption{Force atlas 2 (FA2) force directed graph}
\end{figure}

\begin{figure}[H]
\includegraphics[width=1.0\textwidth,height=0.9\textheight,keepaspectratio]{{{\pagaDir/draw_graph_fa.paga.initialised}}}
\caption{PAGA initialised FA2 force directed graph}
\end{figure}

\clearpage


% !TEX root = main.tex
% !TEX spellcheck = en-US
\section{Definitions and lemmas for multi-message SRS-based protocols}
\label{sec:se_definitions}

\ourpar{Signatures of Knowledge from SE-SNARKs.}  The work of Groth and Maller
\cite{C:GroMal17} shows how to construct a signature of knowledge for messages in
$\{0, 1\}^*$ from target collision-resistant hash-function and simulation-extractable
SNARKs. This construction can be found in \Cref{sec:SoKconstruction}.
These schemes inherit the same properties as the underlying SNARK schemes and
may rely on a structured reference string, which can be updatable or universal, and they are
succinct.  In the following, our goal will be to show that recent efficient SNARKs are
simulation extractable, and therefore, they are a perfect candidate to build better
signatures of knowledge.

\ourpar{Simulation-extractability for multi-message protocols.}  Most recent SNARK schemes
are following the same blueprint: First, an interactive (multi-message)
information-theoretic proof system is considered.
%However, this initial system is idealized and also inefficient.
Second, this initial proof system is compiled into an efficient, non-interactive and
computationally sound scheme using cryptographic tools such as polynomial commitments
and the Fiat--Shamir transformation.

We remark that existing results on the Fiat--Shamir transform regarding existential
unforgeability (for signatures) and simulation extraction (for proof systems and
signatures of knowledge) work for $3$-message protocols without reference string that
require two transcripts for standard model extraction, e.g.,
\cite{JC:PoiSte00,INDOCRYPT:FKMV12,C:RotSeg21}.  In this section we prepare our
analysis for multi-message protocols with a universal or updatable SRS.  In order to
prove simulation-extractability for such protocols, we require more than just two
transcripts for extraction.  Moreover, in the updatable setting we consider protocols
that rely on an SRS where the adversary gets to contribute to the SRS. This makes our
analysis more challenging.  We first give a definition of simulation-extractability
which we base on~\cite{INDOCRYPT:FKMV12} adapted to the updatable SRS setting. Then,
to support multi-message SRS-based protocols compiled using the Fiat--Shamir transform
we generalize the unique response property, replace honest-verifier zero-knowledge
property with trapdoor-less simulation and special soundness with a forking special
soundness property that links up with our generalized forking lemma.

\subsection{Simulation extractability}
We note that the zero-knowledge property is only guaranteed for statements in the
language; for simulator $\simulator = (\simOH, \simOP)$, $\simOP$
answers with simulated proofs only for true statements.  This is, however, not sufficient
for \emph{simulation extractability} where the simulator that the adversary has access to
should be able to provide simulated proofs for false statements as well\footnote{Note,
  that simulation extractability property where the simulator is required to give
  simulated proofs for true statements only is called \emph{true simulation
    extractability.}}. Therefore, we introduce a wrapper oracle around the simulator
called $\simOP'$ that on input $(\srs, \inp)$ always returns the first output of
$\simulator (2, st, \srs, \inp)$, regardless of whether $\inp$ is in the language. We
define \emph{simulation-extractability} with respect to oracle $\simOP'$; that is,
simulation-extractability is with respect to a simulator
$\simulator' = (\simOH, \simOP')$.
%
\begin{definition}[Simulation-extractable NIZK]
	\label{def:updsimext}
  \label{def:simext}
	Let $\ps = (\kgen, \prover, \verifier, \simulator)$ be a NIZK proof system. We say that
  $\ps$ is \emph{updatable simulation-extractable} with respect to
  $\simulator' = (\simOH, \simOP')$ with \emph{extraction error} $\nu$ if for
  any $\ppt$ adversary $\adv$ that is given oracle access to an updatable SRS setup
  $\initU$ and simulator $\simulator$ and that produces an accepting
  proof for $\ps$ with probability $\accProb$, where
	\[
	\accProb = \condprob{
	\begin{matrix}
	  \verifier(\srs, \inp_{\advse}, \zkproof_{\advse}) = 1  \\
	  \wedge
	(\inp_{\advse}, \zkproof_{\advse}) \not\in Q
	\end{matrix}
}{
	\begin{matrix}
	  r \sample \RND{\advse}\\
	(\inp_{\advse}, \zkproof_{\advse}) \gets \advse^{\initU, \simOH, \simOP'
		} (1^\secpar; r)
	\end{matrix}
}
	\]
	there exists an extractor $\extse$ such that
	\[
	\extProb = \condprob{
	\begin{matrix}
  \verifier(\srs, \inp_{\advse}, \zkproof_{\advse}) = 1 \\
 \wedge  ~(\inp_{\advse}, \zkproof_{\advse}) \not\in Q   \\
	 \wedge  ~\REL(\inp_{\advse}, \wit_{\advse}) = 1
	\end{matrix}
}{
	\begin{aligned}
	& r \sample \RND{\advse},
	(\inp_{\advse}, \zkproof_{\advse}) \gets \advse^{\initU, \simOH, \simOP'
		} (1^\secpar; r) \\
	& \wit_{\advse} \gets \ext_\se (\srs, \advse, r,
	Q, Q_\ro, Q_\srs)
	\end{aligned}
}
	\]
	is at at least
	\[
	\extProb \geq \frac{1}{\poly} (\accProb - \nu)^d - \eps(\secpar)\,,
	\]
	for some polynomial $\poly$, constant $d$ and negligible $\eps(\secpar)$ whenever
	$\accProb \geq \nu$.
	Here, $\srs$ is the finalized SRS, list $Q$ contains all $(\inp, \zkproof)$ pairs where
	$\inp$ is an instance queried to $\simOP'$ by the adversary and
	$\zkproof$ is the simulator's answer. List $Q_\ro$ contains all $\advse$'s
	queries to $\simOH$ and the (simulated) random oracle's answers. \hamid{8.2}{And list $Q_\srs$ contains all $(\srs, \rho)$ of update SRSs and their proofs.}
  % \hamid{17.10}{shouldn't be "List $Q_\ro$ contains all $\advse$'s queries to $\simOH$"? Also, I think we need to remove $\ro$ from the statement of the definition!}
\end{definition}

% \chaya{13.10}{later, we should show $\simOP'$ for plonk/sonic/marlin. I think
%   it will follow from TLZK. But we should emphasize there how to define the above
%   oracles so it is clear that $\simO$ can produce accepting proofs for false
%   statements.}
% \michals{19.10}{Re showing that Plonk's et al simulators can produce acceptable
%   proofs for false statements, I am not convinced that's what we need -- we define SE
%   wrt concrete simulators, hence if someone defines a proof system with a simulator
%   that outputs a trapdoor when queried on a false statement, then such a proof
%   systems is simply not SE. For our main theorem we use trapdoor-less simulators
%   which cannot do harm, however I think we should discuss how to show that
%   possibility of RO programming doesn't do harm as well. }

\subsection{Unique-response protocols}
A technical hurdle identified by Faust et al.~\cite{INDOCRYPT:FKMV12} for proving
simulation extraction via the Fiat--Shamir transformation is that the transformed
proof system satisfies a unique response property. The original Fischlin's
formulation, although suitable for applications presented in
\cite{C:Fischlin05,INDOCRYPT:FKMV12}, does not suffice in our case. First, the
property assumes that the protocol has three messages, with the second being the
challenge from the verifier. That is not the case we consider here. Second, it is not
entirely clear how to generalize the property. Should one require that after the
first challenge from the verifier, the prover's responses are fixed?  That does not
work since the prover needs to answer differently on different verifier's challenges,
as otherwise the protocol could have fewer messages.  Another problem is that the
protocol could have a message, beyond the first prover's message, which is
randomized. Unique response cannot hold in this case. Finally, the protocols we
consider here are not in the standard model, but use an SRS.

We work around these obstacles by providing a generalised notion of the unique
response property. More precisely, we say that a $(2\mu + 1)$-message protocol
has \emph{unique responses from $i$}, and call it a $\ur{i}$-protocol, if it
follows the definition below:

\begin{definition}[$\ur{i}$-protocol in the updatable setting]
	\label{def:wiuru}
	Let $\proofsystem$ be a $(2\mu + 1)$-message public coin proof system
  $\ps = (\kgen, \prover, \verifier, \simulator)$. Let
  $\proofsystem_\fs = (\kgen_\fs, \prover_\fs, \verifier_\fs, \simulator_\fs)$ be
  $\proofsystem$ after the Fiat--Shamir transform and $\ro$ the random oracle. Denote
  by $\zkproof_1, \ldots, \zkproof_{\mu}, \zkproof_{\mu + 1}$ protocol messages
  output by the prover and by $\zkproof_i.\ch$ random oracle responses on a partial
  transcript
  $(\inp, \zkproof_1, \zkproof_1.\ch, \ldots, \zkproof_i)$,
  where $\inp$ is the proven statement. We say that $\proofsystem$ has \emph{unique
    responses from $i$} with security $\epsur(\secpar)$ if for any $\ppt$ adversary $\adv$:
  \[
	\Pr\left[
	\begin{aligned}
	& \zkproof \neq \zkproof', (\zkproof_1, \ldots, \zkproof_{i}) = (\zkproof'_1,
	\ldots, \zkproof'_{i}), \\
	& \verifier_\fs (\srs, \inp, \zkproof) =
	\verifier_\fs(\srs, \inp, \zkproof') = 1  \\
	\end{aligned}
	\,\left|\,
	\begin{aligned}
	& \inp, \zkproof, \zkproof'  \gets \adv^{\ro, \initU}(1^\secpar) \\
& \zkproof = (\zkproof_1, \zkproof_1.\ch, \ldots, \zkproof_{\mu}.\ch, \zkproof_{\mu + 1}), \zkproof' = (\zkproof'_1, \ldots,
	\zkproof'_{\mu}.\ch, \zkproof'_{\mu + 1})
	\end{aligned}
	\right.\right]
	\]
	is upper-bounded by some function $\epsur(\secpar)$. If $\epsur(\secpar)$ is negligible, we simply say $\proofsystem$ is $\ur{i}$.
\end{definition}
%
Note that in the above definition, $\srs$ is the SRS that $\advse$ finalised using
the update oracle $\initU$, defined in~\cref{fig:upd}.

Intuitively, a protocol is $\ur{i}$ if it is infeasible for a $\ppt$ adversary to
produce a pair of acceptable and different proofs $\zkproof$, $\zkproof'$ that are
the same on the first $i$ prover messages.  We note that the definition above is also
meaningful for protocols without an SRS. Intuitively in that case $\srs$ is the empty
string.

 \subsection{Trapdoor-Less Simulation}
 Security reductions in this paper sometimes need to produce simulated NIZK proofs without
 knowing the trapdoor, just by programming the random oracle. We call protocols which
 allow for such a simulation \emph{trapdoor-less zero-knowledge} (TLZK).

\begin{definition}[($k$-programmable) Trapdoor-Less Zero-Knowledge Proof System]
  \label{def:TLZK}
  Let $\ps$ be a $(2\mu + 1)$-message ZK proof system, and
  $\ps_\fs= (\kgen_{\fs}, \prover_{\fs}, \verifier_{\fs}, \simulator_{\fs})$ be its
  Fiat--Shamir variant, where $\simulator_{\fs} = (\simOH, \simOP)$ as defined in~\cref{prelim:nizk}. Let $\ro$ be the random oracle.
  %$\simulator_{\fs}$ takes as input $\srs$ and instance $\inp$, programs $\ro$, and outputs a proof $\zkproof_\simulator$.
  We call $\psfs$ \emph{trapdoor-less zero-knowledge} with security $\epszk$ if for any
  adversary $\adv$, $\abs{\eps_0(\secpar) - \eps_1(\secpar)} \leq \epszk(\secpar)$, where
  \begin{align*}
    \eps_0 (\secpar) = \Pr\left[ \adv^{\initU, \ro, \prover_{\fs}} (\secparam) \right],\,
    \eps_1 (\secpar)=  \Pr \left[\adv^{\initU, \simOH, \simOP} (\secparam) \right].
  \end{align*}

  If $\epszk(\secpar)$ is negligible, we say $\ps_{\fs}$ is trapdoor-less zero-knowledge. Additionally, we say that $\ps_{\fs}$ is $k$-programmable, if $\simulator_\fs$
  programs the random oracle \emph{only} after $k$-th prover's message.
  \end{definition}

\begin{remark}[TLZK vs HVZK]
  We note that TLZK notion is closely related to honest-verifier zero knowledge in the
  standard model. That is, if we consider an interactive proof system $\proofsystem$
  that is HVZK in the standard model then its Fiat--Shamir compiled version
  $\proofsystem_\fs$ is TLZK. This comes as the simulator $\simulator$ in
  $\proofsystem$ produces a valid simulated proof by picking verifier's challenges
  according to a predefined distribution and $\proofsystem_\fs$'s simulator
  $\simulator_\fs$ produces its proofs similarly by picking the challenges and
  additionally programming the random oracle to return the picked
  challenges. Importantly, in both $\proofsystem$ and $\proofsystem_\fs$ success of
  the simulator does not depend on access to an SRS trapdoor.
\end{remark}

We note that $\plonk$ is $2$-programmable TLZK, $\sonic$ is $1$-programmable TLZK,
and $\marlin$ is $1$-programmable TLZK. This follows directly from the proofs of
their standard model zero-knowledge property in
\cref{lem:plonk_hvzk,lem:sonic_hvzk,lem:marlin_hvzk}.

\subsection{Generalized forking lemma and forking special soundness}
Although dubbed ``general'', the forking lemma of~\cite{CCS:BelNev06} is not general
enough for our purpose as it is useful only for protocols where a witness can be
extracted from just two transcripts. To be able to extract a witness from, say, an
execution of $\plonkprot$ we need at least $(3 \numberofconstrains + 1)$ valid proofs
(where $\numberofconstrains$ is the number of constrains),
$(\noofc + 1)$ for $\sonicprot$
% (where $\multconstr$ and
% $\linconstr$ are numbers of multiplicative and linear constrains respectively)
, and
$\abs{\HHH} + 4$ for $\marlinprot$ (where $\abs{\HHH}$ is an upper bound for the size
of the witness, i.e.~number of gates). Here we use Attema et
al.~\cite{EPRINT:AttFehKlo21short} generalization of the forking lemma\footnote{The
  earlier versions of this paper used another generalization of the forking lemma
  provided by the paper authors. Change to Attema's et al.~result was due to better
  extraction success probability bound.}  which shows security of the Fiat--Shamir
transformation for multi-message protocols. The lemma, given probability of producing
an accepting transcript, $\accProb$, lower-bounds the probability of generating a
\emph{tree of accepting transcripts} $\tree$, which allows to extract a witness.

\begin{definition}[Tree of accepting transcripts, cf.~{\cite{EC:BCCGP16}}]
	\label{def:tree_of_accepting_transcripts}
	Consider a $(2\mu + 1)$-message proof system. A $(n_1,
  \ldots, n_\mu)$-tree of accepting transcripts is a tree where each node on
  depth $i$, for $i \in \range{1}{\mu + 1}$, is an $i$-th prover's message in an
  accepting transcript; edges between the nodes are labeled with verifier's
  challenges, such that no two edges on the same depth have the same
  label; and each node on depth $i$ has $n_{i} - 1$ siblings and $n_{i +
    1}$ children. The tree consists of $N = \prod_{i = 1}^\mu n_i$
  branches, where $N$ is the number of accepting transcripts. We require $N = \poly$.
\end{definition}

\iffalse

	\begin{figure}[t]
		\centering
		\fbox{
		\procedure{$\genforking_{\zdv}^{m, m'} (y,h_1^{1}, \ldots, h_{q}^{1})$}
		{
		\rho \sample \RND{\zdv}\\
		(i, s_1) \gets \zdv(y, h_1^{1}, \ldots, h_{q}^{1}; \rho)\\
    i_1 \gets i\\
		% \pcif i = 0\ \pcreturn (0, \bot)\\
		\pcfor j \in \range{2}{m'}\\
		\pcind h_{1}^{j}, \ldots, h_{i - 1}^{j} \gets h_{1}^{j - 1}, \ldots,
		h_{i - 1}^{j - 1}\\
		\pcind h_{i}^{j}, \ldots, h_{q}^{j} \sample H\\
		\pcind (i_j, s_j) \gets \zdv(y, h_1^{j}, \ldots, h_{i - 1}^{j}, h_{i}^{j},
		\ldots, h_{q}^{j}; \rho)\\
		%\pcind \pcif i_j = 0 \lor i_j \neq i\ \pcreturn (0, \bot)\\
   % \pcif \exists (j, j') \in \range{1}{m}^2, j \neq j' : (h_{i}^{j} = h_{i}^{j'})\
    \pcif \exists (j_1, \ldots, j_m) \in \range{1}{m'}^m, \text{ s.t. }\\
    \pcind j_k \neq j_{k'} \text{ for } k \neq k' \land \\
    \pcind i_{j_k} = i_{j_k'} \land \\
    \pcind
		\pcelse \pcreturn (0, \bot)
	}}
\caption{Generalised forking algorithm $\genforking_{\zdv}^{m, m'}$
  \michals{5.11}{This forking lemma version is not as general as in Attema et al to
    be exact -- they allow tree of acceptable transcripts to branch at multiple
    places. however, that is not necessary in our case. Later, we should allow ``full
    generality'' but that would also require modification of the forking soundness
    definition (which now also relies on a fact that the tree branches at a single
    point)}}
	\label{fig:genforking_lemma}
\end{figure}

% \begin{figure}[t]
%   \centering
%   \fbox{
%     \procedure{$\genforking_{\zdv}^{k_1, \ldots, k_\mu} (y, h_1^{1}, \ldots,
%       h_{q}^{1})$}
%     {
%       \zdv_{m + 1} (x)
%     }
%   }
% \end{figure}

\fi

\begin{figure}[t]
  \centering
  \fbox{\parbox{\textwidth}{
    \begin{enumerate}
    \item Run $\zdv_{m + 1}(\inp)$ as follows to obtain $(\vec{I}, y_1, v)$: relay the
      $q + \mu$ queries to the random oracle and record all query-response pairs. Set
      $i \gets I_m$, and let $c_i$ be the response to query $i$.
    \item If $v = 0$, abort with output $v = 0$.
    \item Else, repeat
      \begin{itemize}
      \item sample $c'_i \in C \setminus \smallset{c_i}$ (without replacement);
      \item run $\zdv_{m + 1} (\inp)$ as follows to obtain $(\vec{I'},y',v')$, aborting right after
        the initial run of $\adv (\inp)$ if $I'_m \neq I_m$: answer the query to $i$ with
        $c'_i$, while answering all other queries consistently if the query was performed
        by $\zdv_{m + 1} (\inp)$ already on a previous run and with a fresh random value in $C$
        otherwise;
      \end{itemize}
      until either $k_m - 1$ additional challenges $c'_i$ with $v' = 1$ and $I'_m = I_m$ have
      been found or until all challenges $c'_i \in C$ have been tried.
    \item In the former case, output $\vec{I}$, the $k_m$ accepting
      $(1, \ldots, 1, k_{m + 1}, \ldots, k_\mu)$-trees $y_1, \ldots ,y_{k_m}$, and
      $v \gets 1$; in the latter case, output $v \gets 0$.
    \end{enumerate}
    }}
    \caption{Extractor $\zdv_m (\inp)$ from Attema et al. Here $\vec{I}$ is an index vector
      that contains all random oracle queries made by the prover for the output proof;
      that is, for prover's messages $\vec{a} = a_1, \ldots, a_{\mu + 1}$,
      $I_1 = (\inp, a_1), \ldots, I_{\mu} = (\inp, a_1, \ldots, a_\mu)$.  $\vec{y}$ is a proof
      transcript that contains both prover's messages and random oracle responses, that is
      $\vec{y} = (a_1, (\inp, a_1), a_2, (\inp, a_1, a_2), \ldots)$. Finally, $v$ is the
      verification bit, that is $v = \verifier (\srs, \inp, \zkproof) = \verifier(\inp, \vec{y})$. }
  \label{fig:Attema-ext}
\end{figure}

\begin{lemma}[Generalized forking lemma]\label{lem:generalised_forking_lemma}\hamid{8.2}{should we refer to the above fig.3 and Attema paper here?}\hamid{1.3}{We should move this lemma after the Definition of forking special soundness!}
  \michals{19.1}{Here I am focused on a case where the tree branches at single point, hope
    we could generalize it} %
  Consider algorithm $\genforking$ that internally runs $\zdv_1 (x = (\srs, \inp))$ \hamid{1.3}{We are in the adaptive and updatable setting; So maybe $\zdv_1$ should be run without input $x = (\srs, \inp)$?}and
  makes at most $n + q \cdot (n - 1)$ queries to an adversary $\adv$ that outputs an
  acceptable proof with probability $\accProb$ and makes up to $q$ random oracle
  queries. Let proof system $\proofsystem$ be $(1, \ldots, n, \ldots, 1)$-forking special sound\hamid{1.3}{Why we need $(1, \ldots, n, \ldots, 1)$-forking special soundness here?} and
  $\epsk (\secpar)$ knowledge sound. \hamid{1.3}{This sentence is not clear:}Then for every probability that $\genforking$ outputs
  $(1, \ldots, n, \ldots, 1)$ tree of acceptable transcripts is at least
  \[
    \frac{\accProb - (q + 1) \cdot \epsk (\secpar)}{1 - \epsk (\secpar)}.
  \]
\end{lemma}

\oursubsub{Forking special soundness}
%\hamid{8.2}{I am a bit confused here! Forking special soundness should be defined for interactive arguments, no? Why we define it in the ROM and talk about FS in the definition below? To me, this definition should be something like $k$-special soundness property in Attema paper, based on which one can argue extractability of the witness from the tree of accepting transcripts (generated by the forking lemma).}
%\hamid{8.2}{In fact, I am thinking about adaptation of Attema result to our theorem in this way:
%\begin{itemize}
%	\item We are given a protocol that is $(k,n)$-special sound; that is, given a $(1, \ldots, 1, n, 1, \ldots, 1)$-tree of accepting transcripts outputs a witness for the proven statement. We also know that $(k,n)$-special soundness tightly implies knowledge soundness.
%	\item We use the Attema et al. extractor in Fig. 3 to extract a $(k,n):=(1, \ldots, 1, n, 1, \ldots, 1)$-tree of accepting transcripts. To this end, we let $\adv$ be an adversarial prover that makes up to $Q$ random oracle queries and outputs an acceptable proof with probability at least
%	$\accProb$. We consider $\adv$ as the recursion base which outputs a $(1, \ldots, 1)$-tree and then run the subextractor $\zdv_k (\inp)$ which outputs a $(k,n)$-tree of accepting transcript. The run time and success probability of $\zdv_k (\inp)$ is given in proposition 2 of Attema paper: $\zdv_k (\inp)$ makes an expected number of at most $n+Q(n-1)$ queries to $\adv$ and outputs a (k,n)-tree with probability at least $\frac{\accProb - (Q+1) \epsk(\secpar)}{1-\epsk(\secpar)}$, where $\epsk(\secpar) = \frac{n-1}{N}$, and $N$ is the cardinality of the challenge set.
%\end{itemize}
%}
Note that the special soundness property (as usually defined) holds for
all---even computationally unbounded---adversaries. Unfortunately, since a
simulation trapdoor for $\plonkprot$, $\sonicprot$ and $\marlinprot$ exists, the protocols
cannot be special sound in that regard. This is because an unbounded adversary
can recover the trapdoor and build an arbitrary number of simulated proofs for a fake
statement. Hence, we provide a weaker, yet sufficient, definition of
\emph{forking special soundness}. More precisely, we state that an
adversary that is able to answer correctly multiple challenges either knows the
witness or can be used to break some computational assumption.
% \chaya{a notion of rewinding-based knowledge soundness has been used to mean exactly
% the above in prior works, like BBF19. we should clarify if forking soundness
% different from rewinding-based knowledge soundness? seems to me like the difference is
% just that here it is tailored for the NI version.}  \chaya{I now see that the diff
% is the access to the simulator that the adversary gets. might be helpful to clarify
% this, either here or in a tech overview section. I am not sure why this needs to be
% defined this way by giving access to the simulator.}
However, differently from the standard definition of special soundness, we do
not require from the extractor to be able to extract the witness from \emph{any}
tree of acceptable transcripts. We require that the tree be produced honestly,
that is, all challenges are picked randomly --- exactly as an honest verifier would pick.
Intuitively, the tree is as it would be generated by a $\genforking$
algorithm from the generalized forking lemma.

\begin{definition}[$(k,n)$-forking special soundness in the updatable setting]
	Let $\proofsystem = (\kgen, \prover, \verifier, \simulator)$ be an
  $(2 \mu + 1)$-message proof system for a relation $\REL$.  Let $\proofsystem_\fs$
  be $\proofsystem$ after the Fiat--Shamir transform.

	For any $\ppt$ adversary $\advse^{\ro, \initU} (1^\secpar; r)$ with access to
  oracles $\initU$ defined in~\cref{fig:upd}, and random oracle $\ro$, we consider
  the procedure $\zdv$ that provided the transcript $(\srs, \adv, r)$ and
  $h_1, \ldots, h_q$ runs $\adv$ by providing it with random oracle queries and
  update oracle queries.
	%
	$\zdv$ returns the index $i$ of the
	random oracle query made for challenge $k$ and the proof $\adv$ returns.

	Consider the algorithm $\genforking_{\zdv}^{n}$
	that rewinds $\zdv$ to produce a $(1,\dots, n, \dots, 1)$-tree of
	accepting transcripts $\tree$.\hamid{1.3}{Not clear if this always happens with probability 1, or is such that over which the below probability is defined? Maybe we should say $\genforking_{\zdv}^{n}$ produce a tree or $\bot$?}

	We say that $\psfs$ is $(k,n)$-forking special sound with security loss $\epss(\secpar)$, if
	for any PPT adversary $\advse$\hamid{28.2}{How should we quantify over $\extt(\tree)$? Is it $\exists \ \extt(\tree) \ s.t \ \forall \advse ...$?}
	\begin{align*}
	\Pr\left[
	\REL(\inp, \wit) = 0
	\,\Biggl|\,
	\begin{aligned}
	& r \sample \RND{\advse},
	(\inp_{\advse}, \zkproof_{\advse}) \gets \advse^{\ro, \initU} (1^\secpar; r), \\
	&    (1, \tree) \gets \genforking_{\zdv}^{n}((\srs,\adv,r),Q_{H}),
	\wit \gets \extt(\tree)
	\end{aligned}
	\right] \leq \epss(\secpar).
	\end{align*}
	Here, $\srs$ is the SRS that $\advse$ finalized using the update oracle $\initU$.
	List $Q_\ro$ contains all $\advse$'s
	queries to $\ro$ and the random oracle's answers.
\end{definition}

\hamid{1.3}{I think we can replace lemma 2 with the following lemma which includes the adaptation of Attema result to the updatable setting.}
	\begin{lemma}[Forking special soundness]\label{lem:forking_ss}
		Let $\proofsystem$ be a proof system that is knowledge sound
		with knowledge error $\epsk$. Assume adversary $\advse$ with access to
		oracles $\initU$ defined in~\cref{fig:upd}, and random oracle $\ro$, that makes up to $q$ random
		oracle queries and outputs an acceptable proof with probability at least
		$\accProb$. Consider algorithm $\genforking$ that internally runs $\zdv_1$ in~\cref{fig:Attema-ext} and
		makes at most $n + q \cdot (n - 1)$ queries to $\advse$.
		\chb{If $\proofsystem$ is $(k,n)$-special sound (I am probably missing something, but I think we need this property! How we can prove this lemma without SS property?)}, then $\proofsystem_\fs$ is $(k, n)$-forking special sound with security loss $\frac{\accProb - (q + 1) \epsk (\secpar)}{1 - \epsk (\secpar)} + \epsdlog$.
	\end{lemma}
\hamid{3.3}{discuss tomorrow: by having both probabilities of getting a good tree T from GF and extracting a valid witness from T in the definition of forking special soundness, how can we later argue that a failure here leads to breaking dlog (i.e., the case of arguments)? I think we need to define forking special soundness without having $\wit \gets \extt(\tree)$ in the probability equation, no?}

\hamid{3.3}{Discuss this. Here is how I think we should proceed:
\begin{itemize}
	\item define $(k,n)$-special soundness.
	\item restate Attema forking lemma in the updatable setting --- call this lemma X.
	\item in the statement of main theorem, we assume the protocol is $(k,n)$-special sound with knowledge error $\epsk$. Then from Game 1 to Game 2, we use lemma X to upper-bound the probability of building the tree $\tree$ as $\frac{\accProb - (q + 1) \epsk (\secpar)}{1 - \epsk (\secpar)}$.
	\item If we cannot extract the witness (is possible because of dealing with arguments) from $\tree$, we make a reduction to dlog problem.
\end{itemize}}

\paragraph{Importance of the general forking lemma.}
To highlight the importance of the generalised forking lemma, we outline how it is
used in our simulation-extractability proof.  Let $\psfs$ be a forking special sound
proof system where for an instance $\inp$ the corresponding witness can be extracted
from a $(1, \ldots, 1, n_k, 1, \ldots, 1)$-tree of accepting transcripts.  Let $\adv$
be the simulation-extractability adversary that outputs an accepting proof with
probability at least $\accProb$. From $\adv$ we build an adversary $\bdv$ that runs
$\simulator$ internally. Although we use the same $\accProb$ to denote probability of
$\zdv$ outputting a non-zero $i$ and the probability of $\adv$ (and $\bdv$)
outputting an accepting proof, we claim that these probabilities are exactly the same
by the way we define $\zdv$ and $\bdv$. In the following we give the intuition of the
proof without the additional indirection via $\bdv$ whose goal is primarily to
simplify the forking special soundness definition.

Let $\adv$ output an accepting instance-proof pair $(\inp_{\advse},\zkproof_{\advse})$;
$r$ be $\adv$'s randomness; and $Q_\ro$ be the list of all random oracle queries made by
$\adv$.  All of these are given to the extractor $\ext$ that internally runs the forking
algorithm $\genforking_\zdv^{n_k}$.  Algorithm $\zdv$ takes
$(\srs, \advse, r)$ as input $y$ and $Q_\ro$ as input $h_1^1, \ldots, h_q^1$.
(For the sake of completeness, we allow $\genforking_\zdv^{n_k}$ to pick
$h^1_{l + 1}, \ldots, h^1_q$ responses if $Q_\ro$ has only $l < q$ elements.)

Next, $\zdv$ internally runs $\adv(\srs; r)$ %\hamid{$\\advse(1^\secpar; r)$}
and responds to its random oracle queries using $Q_\ro$. Note that $\adv$ makes the same
queries as it did before it output $(\inp_{\advse}, \zkproof_{\advse})$ as it is run on
the same random tape and with the same answers from the simulator and random oracle. Once
$\adv$ outputs $\zkproof_{\advse}$, algorithm $\zdv$ outputs $(i, \zkproof_{\advse})$,
where $i$ is the index of a random oracle query submitted by $\advse$ to receive the
challenge after the $k$-th message from the prover -- a message after which the tree of
transcripts branches.  Then, after the first run of $\advse$ is done, the extractor runs
$\zdv$ again, but this time it provides fresh random oracle responses
$h^2_i, \ldots, h^2_q$. Note that this is equivalent to rewinding $\advse$ to a point just
before $\advse$ is about to ask its $i$-th random oracle query. The probability that the
adversary produces an accepting transcript with the fresh random oracle responses is at
least $\accProb$. This continues until the required number of transcripts is obtained.

We note that in the original forking lemma, the forking algorithm $\forking$,
cf.~\cite{CCS:BelNev06}, gets as input only $y$ while elements $h^1_1, \ldots,
h^1_q$ are randomly picked from $H$ internally by $\forking$. However, assuming
that $h^1_1, \ldots, h^1_q$ are random oracle responses, and thus random, makes
the change only notational.

% We also note that the general forking lemma from
% \cref{lem:generalised_forking_lemma} works for protocols with an extractor that can obtain the
% witness from a $(1, \ldots, 1, n_k, 1, \ldots, 1)$-tree of accepting
% transcripts. This limitation however does not affect the main result of this
% paper, i.e.~showing that $\plonk$, $\sonic$, and $\marlin$ are simulation extractable.

\changedm{
  \subsubsection{From knowledge soundness to tree of acceptable transcripts}
  Let $\proofsystem$ be a proof system that is knowledge sound
  with knowledge error $\epsk$. Assume adversary $\adv$ that makes up to $q$ random
  oracle queries and outputs an acceptable proof with probability at least
  $\accProb$. Attema et al.~\cite{EPRINT:AttFehKlo21short} shown that for such a proof
  system probability that a tree building algorithm $\tdv$ succeeds in building an
  $(k, n)$-tree of accepting transcripts in expected
  running time $n + q (n - 1)$ is at least
  \[
    \frac{\accProb - (q + 1) \epsk (\secpar)}{1 - \epsk (\secpar)}.
  \]
}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End:


\section{Discussion}
\label{sec:discussion}

\subsection{All Features}

The variables prepared for the machine learning model were not correlated with each other. No substantial correlation was observed between the percentage of duplicate lines and the time until the first job was found. Unnoteworthy although reassuring connection, which has the highest correlation among those all variables, between the experience duration and \emph{WorkFindTime} is present.

Taking into account the results of the most common classifiers (\emph{ACC} and \emph{MMCE}) we conclude that the \emph{Random Forest} and \emph{SVM} algorithms coped best with the classification task. The \emph{SVM} had the best results, which is not a surprise - as in theory - it should produce solid results even with a small amount of training data. That also reassures the validity of this algorithm as it is ,,field-tested'' in other research papers with similar data size with success. \emph{KNN} which by definition is the simplest, performs the worst because it requires features in a uniform scale to classify with its best efficiency while our data has features of quite different scales.

However, one should have doubts about the quality of these indicators (the \emph{ACC} and \emph{MCEE}) because the data that we feed into the algorithm is not perfectly balanced (the $\%$ share of classes in the set is about $62\%$ to $38\%$). The \emph{F1} measure assessed the quality of all models on comparable level. On the other hand, the \emph{MCC}, which is often considered a key indicator, showed that the \emph{SVM} model was by far the best in classifying tasks.

The \emph{Kappa} coefficient for the \emph{Random Forest} and \emph{SVM} algorithms is in the range of $0.21 - 0.40$, therefore the \emph{Level of Agreement} can be describe as \emph{,,fair agreement''} (which is about an average result). For \emph{KNN} the compliance can be defined as \emph{,,slight agreement''}.

To sum up, the most effective model during the classification was the one using the \emph{SVM} algorithm. It was the best according to almost all indicators.

\subsection{GitHub Data Only}

Comparing the results of the classification using different features, we observed that a different number of features influenced each model in a different way. Each of the measures in the model that used \emph{SVM} algorithm, has worse value (besides the \emph{F1} indicator, which is better for each algorithm). This indicates that the algorithm performs better when it has access to the higher variety of features.

The \emph{KNN} algorithm performance improved as the number of features decreased. This may be due to the fact - as we previously mentioned - that it should perform best with the features of similar scale. As for \emph{Random Forest} - lower number of features made the \emph{ACC} indicator significantly worse while improving \emph{MCC}. It is difficult to explain the reason why such a difference in the results was obtained in this particular case.

\subsection{Run-down}

Summarizing, in most cases, the results of classification based on fewer features are worse, but the difference is not dramatic. Therefore, it can be concluded that the data obtained from the repository using our method of combining \emph{GitHub API} along with \emph{Mega Linter} tool and our parse scripts is to some extent sufficient to assess the potential of the GitHub user in terms of his potential for an employee.


\begin{figure}
    \begin{center}
        \includegraphics[width=0.66\textwidth]{figs/slab_model.pdf}
        \caption{\label{fig:av_dist}
        The $A_V$ distributions, $p(A_V)$, generated from the slab model (Eq.~\ref{eq:slab};
        black dash) compared to $p(A_V)$ of star-forming galaxies our SDSS
        sample (blue; Section~\ref{sec:obs}) and of $M_* > 10^{10}M_\odot$
        star-forming and quiescent galaxies in the GSWLC2 sample (orange).
        The $A_V$ values for both observations are derived using SED
        fitting but with different bands and methodologies~\citep{brinchmann2004, salim2018}.
        For the slab model, we generate $A_V$ values for each galaxy
        in the SDSS and GSWLC2 samples using Eq.~\ref{eq:slab} with its
        measured $M_*$ and $\ssfr$ and randomly sampled $i$.
        Despite the significant differences between the $p(A_V)$ of SDSS and
        GSWLC2, the slab model is able to generate $p(A_V)$ in good agreement
        with both observations using parameter values within the
        Table~\ref{tab:free_param} prior range.
        Therefore, the slab model provides a sufficiently flexible prescription
        for our \eda.
        }
    \end{center}
\end{figure}

\section{The Slab Model Based EDA}  \label{sec:slab}
In our \eda~prescription, we use the slab model to determine $A_V$, the
amplitude of attenuation, as a function of a randomly sampled inclination,
$i$, and $\tau_V$ (see Eq.~\ref{eq:slab} in Section~\ref{sec:dem}).
The slab model is based on the assumption that dust in galaxies have
slab-like geometry and are illuminated by the stellar radiation
source~\citep{somerville1999}.
For a given $\tau_V$, the attenuation depends solely on the orientation of the galaxy.
While this simplification reproduces the correlation between $A_V$ and $i$
found in observed star-forming galaxies~\citep[\eg][]{conroy2010, wild2011,
battisti2017, salim2020}, it ignores the detailed star-to-dust geometry that
impacts the attenuation curve.
It also does not provide a physically-motivated prescription for quiescent
galaxies, which typically have elliptical morphologies.
Despite its limitations, the slab model provides a robust empirical
prescription that allows us to produce realistic distributions of $A_V$.

In Figure~\ref{fig:av_dist}, we compare the $A_V$ distributions, $p(A_V)$,
of star-forming galaxies in SDSS (blue) and galaxies in the
\cite{salim2018} GSWLC2 sample (orange) to $p(A_V)$ generated from the
slab model (black dashed).
The $A_V$ values of the SDSS are derived using SED fitting from the
\cite{brinchmann2004} MPA-JHU catalog.
The GSWLC2 $A_V$ values are also derived from SED fitting UV and optical
photometry from GALEX and SDSS observations as well as mid-IR photometry from WISE.
The GSWLC2 $p(A_V)$ includes all galaxies, including quiescent ones, above
$M_* > 10^{10}M_\odot$.
We generate two $p(A_V)$ with the slab model for the SDSS and GSWLC2
samples separately.
For each SDSS/GSWLC2 galaxy, we determine $A_V$ by uniformly sampling
$\cos i$ from 0 to 1 and derive $\tau_V$ (Eq.~\ref{eq:tauv}) with the
galaxy's measured $M_*$ and $\ssfr$.
We pick $\mtaum, \mtaus, c_\tau$ values within the prior range
(Table~\ref{tab:free_param}) by hand to roughly reproduce the SDSS and GSWLC2 $p(A_V)$
distributions.

Galaxies in SDSS and GSWLC2 have substantially different $p(A_V)$.
While the galaxy populations only partially overlap, this difference is
{\em mostly} due to inconsistencies in the $A_V$ measurements of MPA-JHU and GSWLC2 --- even
for the same galaxy.
%In fact for the ${\sim}8000$ galaxies that are in both samples, the $A_V$ distributions appear similar to those of the corresponding complete samples.}
This difference in $p(A_V)$ illustrates the challenges in directly measuring
dust attenuation.
Despite the dramatic differences between the two, the slab model can
produce $p(A_V)$ in good agreement with both observed distributions.
We therefore conclude that the slab model provides a sufficiently flexible
prescription to sample a realistic distribution of $A_V$.

\begin{figure}
\begin{center}
    \includegraphics[width=0.8\textwidth]{figs/gswlc_dep.pdf}
    \caption{\label{fig:dep}
    Dependence of $A_V$ on $M_*$ (left) and $\ssfr$ (right) for the
    \cite{salim2018} GSWLC2 sample.
    In the left panel, we divide the GSWLC2 sample into bins of $\ssfr$:
    $\ssfr < 10^{-11}yr^{-1}$ (purple),
    $10^{-11} < \ssfr < 10^{-10}yr^{-1}$ (red),
    and  $10^{-10} < \ssfr$ (orange).
    In each of the $\ssfr$ bins, we find significant $M_*$ dependence.
    In the right panel, we divide the sample into bins of $M_*$:
    $10^{9.5} < M_* < 10^{10.5}M_\odot$ (blue) and $10^{10.5} M_\odot < M_*$ (green).
    In the $M_* > 10^{10.5}M_\odot$ bin, which roughly corresponds to our
    SDSS sample, we find significant $\ssfr$ dependence.
    The $M_*$ and $\ssfr$ dependence in $A_V$ we find in GSWLC2 is
    consistent with previous works and provides further motivation for our
    \eda~prescription.
    }
\end{center}
\end{figure}

In addition to the slab model, in the \eda, we also use a linear
dependence on $M_*$ and $\ssfr$ in the $V$ band optical depth,
$\tau_V$ (see Eq.~\ref{eq:tauv}).
This parameterization is motivated by observations that find significant
correlation between $A_V$ and $M_*$ and $\ssfr$~\citep[\eg~][]{garn2010, battisti2016, salim2020}.
We take a closer look at this correlation using the GWSLC2 sample in
Figure~\ref{fig:dep}.
We present the dependence of $A_V$ on $M_*$ (left panel) and $\ssfr$ (right
panel).
In the left panel, we divide the GSWLC2 galaxies by $\ssfr$:
$\ssfr < 10^{-11}yr^{-1}$ (purple), $10^{-11} < \ssfr < 10^{-10}yr^{-1}$
(red), and  $10^{-10} < \ssfr$ (orange).
For each of the $\ssfr$ bins, we find significant $M_*$ dependence in
$A_V$: more massive galaxies have higher $A_V$.
In the right panel, we divide the galaxies by $M_*$:
$10^{9.5} < M_* < 10^{10.5}M_\odot$ (blue) and $10^{10.5} M_\odot < M_*$
(green).
In both $M_*$ bins, galaxies with higher $\ssfr$ have higher $A_V$.
The dependence is stronger stronger for galaxies with $M_* > 10^{10.5}M_\odot$,
which roughly corresponds $M_*$ limit of our forward model (see
Figure~\ref{fig:avmsfr}).
Overall, the $M_*$ and $\ssfr$ dependence we find in $A_V$ from the GSWLC2 sample is
consistent with previous observations and further motivates our
\eda~prescription.

\begin{figure}
\begin{center}
    \includegraphics[width=\textwidth]{figs/av_mssfr_obs.pdf}
    \caption{\label{fig:av_obs}
    $M_*$ and $\ssfr$ dependence of dust attenuation at $1500 \AA$
    ($A_{1500}$; top) and at $5500\AA$ ($A_{V}$; bottom) of SDSS galaxies.
    The sample includes 2361 galaxies that pass our selection cut
    (Section~\ref{sec:obs}) and are also part of the GSWLC2 and MPA-JHU
    samples.
    In the left panels, we use $A_{1500}$ and $A_V$ from GSWLC2.
    In the center panel, we use $A_V$ from MPA-JHU.
    In the right panel, we use $A_V$ from the NSA.
    The $M_*$ and $\ssfr$ in each panel are from the respective samples.
    Same as Figure~\ref{fig:avmsfr}, the colormap in each hexbin represents the
    median attenuation for all galaxies in the bin (color bar).
    We only include bins with more than 5 galaxies.
    The bottom panels illustrate that $A_V$ measurements from GSWLC2, MPA-JHU,
    and NSA differ {\em significantly} even for the same galaxies.
    We, therefore, do not directly compare our \eda~predictions to observations.
    }
\end{center}
\end{figure}

In Figure~\ref{fig:av_obs}, we present the $M_*$ and $\ssfr$ dependence of dust
attenuation in SDSS galaxies, which contains 2361 galaxies that pass our
selection cut and are also in the GSWLC2 and MPA-JHU samples.
In the top panel, we present $A_{1500}$ from GSWLC2 as a function of $M_*$ and
$\ssfr$.
In the bottom panels, we present $A_V$ from GSWLC2 (left), MPA-JHU (center),
and NSA (right).
The NSA $A_V$ measurements are derived assuming intrinsic Balmer decrement of
2.85, $R_V=3.1$ and \cite{odonnell1994} extinction.
The colormap in each hexbin represents the median attenuation for galaxies in
the bin, same  as in Figure~\ref{fig:avmsfr}.
Bins with less than 5 galaxies are omitted.
For each observational sample (column), we use $M_*$ and $\ssfr$ from the
respective samples for consistency.
We find the same $M_*$ and $\ssfr$ dependence of $A_V$ as Figure~\ref{fig:dep}
even after our selection cut (bottom left).
The bottom panels highlight that evven for the same galaxies, $A_V$ from
GSWLC2, MPA-JHU, and NSA have significant differently amplitudes.
$M_*$ and $\ssfr$ are also significantly different across the samples.
Since observations have large discrepancies among dust attenuation measurements
and a detailed comparison is beyond the scope of this work, we refrain from
comparing our \eda~predicted dust attenuation (Section~\ref{sec:results}) to
observations.


\subsection{Latent class analysis}


\chapter{Related Work}\label{sec:RelatedWork}
Realistic RL scenarios often involve multiple agents solving problems together, for example robots working in warehouses and factories. Such multiagent environments come with many difficulties. On the one hand, in a scenario where agents work independently, it is very probable that they get in each other's way. They aim to achieve the highest score or finish their individual tasks, preventing the overall goal to be achieved.

In cooperative environments, on the other hand, agents share the reward and therefore cannot tell who contributed useful actions. The difficulties of agents working independently is discussed in chapter \ref{market}, whereas the cooperation challenge is the focus of the next chapter.

\section{Credit Assignment Problem}\label{CAP}
Sutton and Barto \cite{suba18} define a RL environment as cooperative, when agents execute their actions collectively each time step but receive one overall reward in return. In this case, individual learning is difficult or even impossible. Collective actions may contain bad choices that would be rewarded or, in case of a penalty, good actions that would be punished. Deciding which agent deserves more of the common reward is referred to as the CAP \cite{mi61}.

The CAP originated in a one-agent environment that only returned reward once the goal is reached or the terminating condition applies. A popular example of this is a chess game. In 1961, Minsky \cite{mi61} elaborated, that a player wins or loses the game, but cannot retrace which decision got him there. Later on, Sutton decomposed the CAP into subproblems, namely the structural and temporal CAP \cite{su84}. He suggests, that the temporal problem lies in assigning credit to each chess move by determining when the position improves or worsens, rewarding or penalizing that action. On the contrary, the structural CAP is assigning credit to the internal decision that leads to each particular action.

Transferring the single-agent CAP into a multiagent environment, Agogino and Tumer \cite{agtu04} show that the problem shifts from being a temporal to a structural type. They explain that while a single agent faces the temporal CAP due to many steps taken within an extended time period, in the multiagent case it becomes a structural CAP, because of multiple executed actions in a single-time-step. Since the actions are executed all at once, the challenge is now evaluating the decision that lies underneath.

Over the years, many solutions and theories emerged in order to solve CAPs in multiagent environments \cite{rabe09}, \cite{zhli20}, \cite{agtu04}. A popular example for a simple approach to solve the problem is the difference reward (DR) \cite{ngku18}, \cite{yltu14}, \cite{agtu04}. The idea is to calculate the overall reward with the joint multiagent actions as usual. Then, another reward is calculated for one agent if that agent would not have submitted its action. The difference between those two values indicates how profitable the action of the analyzing agent is. A high DR indicates a lucrative action, since excluding it leads to a small subtrahend value. With this method, each agent has the opportunity to learn how their actions contribute to the global reward, enabling individual learning.

Formally, in order to find the DR $D_i(z)$ for an agent $i$ and the joint action set $z$, the following function applies \cite{agtu04}:
\begin{equation}\label{eq:dr}
    D_i(z) = G(z) - G(z_{-i})
\end{equation}
The first part of the equation $G(z)$ represents the overall reward of the action set of one time step. The second part $G(z_{-i})$ demonstrates the result of the same actions and time step, excluding only the action of agent $i$. Another approach to find the DR is to select a default action in $G(z_{-i})$ for the analyzing agent $i$, instead of dropping its action completely \cite{vega96}. Calculating the equation \eqref{eq:dr} results in $D_i(z)$ of one agent $i$.

An example for this approach contains two agents, $i$ and $j$, that act in an environment. In a time step $t$ agent $i$ executes a good action and $j$ a bad action. For this scenario, good actions produce 0.6 as reward and bad choices result in -0.5 as punishment. In this example, the cooperation reward is the sum of the rewards, which is 0.1. Therefore, agent $j$ is rewarded for selecting a bad action through the positive reward of 0.1 and ends up learning this strategy. However, using the DR function, a reward of $D_j(z) = 0.1 - 0.6 = -0.5$ is calculated for agent $j$ and $D_i(z) = 0.1 - (-0.5) = 0.6$ for agent $i$. Thus, the credit is assigned to each agent depending on their contribution. Since there are only two actors, their DRs match the actual environment rewards of the actions.

As a downside, this approach is often inefficient or even infeasible \cite{ngku18}. Nguyen et al. \cite{ngku18} imply that large domains could make this calculation impossible. Sometimes, environment states need to be stored and replayed for every agent in order to calculate $G(z_{-i})$. Agogino and Tumer \cite{agtu04} claim that this simple function is not always applicable, since it can get difficult to exclude agents' actions. Nevertheless, this formula is often coupled with the CAP in research papers and builds a base for many advanced solutions of this topic.

\section{Markets}\label{market}
As described before, agents that share an environment and act independently can often hinder each other from reaching the common or individual goal. Sutton and Barto defined a game to be competitive, when agents receive varying reward signals \cite{suba18}. In most cases, agents follow a mixed-motive, meaning that their individual rewards could sometimes align and sometimes be in conflict. An environment is purely competitive, when the increase in reward of one agent leads to reward decrease of the others \cite{scbe21}.

In ``Stochastic Market Games'' \cite{scbe21} Schmid et al. introduced concepts that add incentives when agents act cooperatively in mixed-motive settings, to improve the overall rewards for all participants. The idea of a Stochastic Market Game (SMG) is to enable dominant cooperative strategies through a global and impartial trading market. According to the researchers, a stochastic game becomes a SMG if two conditions are met: First, the environment actions of agents are extended with market actions. Second, the reward function adjusts the calculated rewards based on agreements met in the market executions. Furthermore, Schmid et al. defined two types of markets: unconditional and conditional markets.

\subsection{Shareholder Market}
They compare the concept of unconditional markets to companies and shareholders, since shareholders do not need to fulfill any conditions to receive the dividends. In unconditional SMGs, both companies and shareholders are agents that buy and sell shares as market actions. Figure \ref{fig:sm} shows such a SM. During each time step, every agent has the possibility to put their share on the market or to announce a buying offer directed to another agent.

\begin{figure}[hpbt]
    \centering
    \includegraphics[width=0.5\textwidth]{pictures/SMG_sm}\\
    \caption[Shareholder Market]{Shareholder Market as presented in ``Stochastic Market Games'' \cite{scbe21}}\label{fig:sm}
\end{figure}

If the buying offer coincide with a share that is up for sale in the same step, a market transaction is registered. Now, the shareholder participates in the reward of the current step of the transaction agent by a fixed dividend $d$. Schmid et al. mention that an optional price $p$ can be defined as an amount a seller receives from the buyer upon each share purchase. However, they claim that agents with high rewards are very likely to gift their shares in order to align the goals of the other agents with their own. Shareholders profit from the success of the selling party through the dividends.

An example is an agent $i$ who wants to sell a share and agent $j$ that wants to buy a share specifically from agent $i$ with both being in time step $t$. In this scenario, the offers coincide and a share of $i$ is now claimed by $j$. If $i$ now receives a reward of one and the dividend is 20\%, agent $j$ obtains a reward of 0.2 in addition to its own reward. However, in a scenario where agent $i$ decided not to sell, agent $j$ is not be able to claim the share and does not profit from the reward of agent $i$. After each step, the shares are resolved and the matching starts again.

\subsection{Action Market}
On the contrary, the authors define conditional markets similar to purchase contracts, where buyers pay a fixed price $p$ to sellers when they in turn meet the buyers' demand. A proposed conditional SMG is the AM. In this case, actions are extended with a buying offer, containing one expected action from one specific agent, see Figure \ref{fig:am}.

\begin{figure}[hpbt]
    \centering
    \includegraphics[width=0.5\textwidth]{pictures/SMG_am}\\
    \caption[Action Market]{Illustrated Action Market}\label{fig:am}
\end{figure}

In an example with two agents, $i$ and $j$, $i$ needs a resource that is currently occupied by $j$. Hence, agent $i$ would profit from $j$ giving up the resource and therefore offers to buy from $j$ the action ``release''. Formally, agent $i$ executes action $\overrightarrow{a}_{i,t}$, containing an environment action $a^{\text{env}}_{i,t}$ and the offer to agent $j$ $a^{\text{offer}}_{i,j,t}$ \cite{scbe21}. The offer proposal shows that agent $i$ is willing to buy from $j$ at time step $t$ a specific action $a^{\text{env}}_{j,t}$. If agent $j$ fulfills the conditions and releases the resource in the same step $t$, it would receive a fixed price from agent $i$ in form of a reward. However, if that is not the case, the market conditions do not apply and $j$ is not paid by agent $i$.

In both market settings, a purchase is established if the specified agent happens to execute an action that matches with a buyer. It is important to emphasize that the matching is performed during a time step, leaving it to chance, whether purchases take place. For instance, in an AM, agents do not know in advance if and what action another agent is buying from them. Despite this uncertainty, the researchers show that both market implementations yield promising results. An increase of the overall rewards of participating agents in mixed-motive games is observed.


\subsubsection{CPI Swap}

A CPI swap is set up as a swap, with one leg of type {\tt CPI}. Listing \ref{lst:cpiswap} shows an example. The
CPI leg contains an additional {\tt CPILegData} block. See \ref{ss:cpilegdata} for details on the CPI leg specification.

\begin{listing}[H]
%\hrule\medskip
\begin{minted}[fontsize=\footnotesize]{xml}
    <SwapData>
      <LegData>
        <LegType>Floating</LegType>
        <Payer>true</Payer>
        ...
      </LegData>
      <LegData>
        <LegType>CPI</LegType>
        <Payer>false</Payer>
        ...
        <CPILegData>
        ...
        </CPILegData>
      </LegData>
    </SwapData>
\end{minted}
\caption{CPI Swap Data}
\label{lst:cpiswap}
\end{listing}


\section{Evaluation}
\label{sec:conclusion:evaluation}

This section will evaluate the research carried out in this thesis and answer the main research question presented in \cref{sec:introduction:research_question}. In order to answer the main research question, the subquestions need to be evaluated and answered. The answers to the subquestions are the following:

\begin{enumerate}
    \item ``What is a suitable formalisation of Ecore models and what Ecore models are valid within this formalisation?''

    For this thesis, a suitable formalisation of Ecore was presented in \cref{sec:formalisations:ecore_formalisation}. The formalisation is considered suitable since it can express almost all concepts of Ecore. Therefore, it is also able to express all the models used throughout this thesis, which are type models and instance models.

    The set of valid models within this formalisation depend on the model level. For type models, the set of consistent models is constrained by \cref{defin:formalisations:ecore_formalisation:type_models:type_model_consistency}. For instance models, the set of valid models is constrained by \cref{defin:formalisations:ecore_formalisation:instance_models:model_validity}. These definitions answer the question which models are valid within the formalisation, and the examples shown throughout this thesis show the existence of these models.

    The answer to this question is validated using the existing theory available on Ecore. Also, each part of the formalisation can be traced back to elements within the Ecore metamodel. Together, the answer to the first question is considered validated.

    \item ``What is a suitable formalisation of GROOVE grammars and what GROOVE grammars are valid within this formalisation?''

    Just like the previous question, a suitable formalisation was presented within this thesis. The suitable formalisation for GROOVE graphs can be found in \cref{sec:formalisations:groove_formalisation}. The formalisation is considered suitable since it can express the relevant GROOVE graphs used throughout this thesis, which are type models and instance models.

    The set of valid grammars within the formalisation depend on the set of graphs defined for a GROOVE grammar. In order to answer the main research question, it is only relevant to know when type graphs and instance graphs are valid. For type graphs, the set of consistent graphs is constrained by \cref{defin:formalisations:groove_formalisation:type_graphs:type_graph_validity}. For instance graphs, the set of valid graphs is constrained by \cref{defin:formalisations:groove_formalisation:instance_graphs:instance_graph_validity}. These definitions answer the question of which graphs are valid within the formalisation, and the examples shown throughout this thesis show the existence of these graphs. Since the main research question only needs to show the validity of the presented graph types, the answer to the question is sufficient.

    Once more, the answer to this question is validated using the existing theory available. Existing theory on GROOVE and graph theory has been used to validate the formalisation, and therefore the answer to this question is considered validated.

    \item ``What is a suitable formalisation for the model transformations between Ecore and GROOVE?''

    A formalisation for model transformations between Ecore and GROOVE was given as part of the presented transformation framework in \cref{chapter:transformation_framework}. \cref{defin:transformation_framework:type_models_and_type_graphs:combining_transformation_functions:transformation_function_type_model_type_graph}, \cref{defin:transformation_framework:type_models_and_type_graphs:combining_transformation_functions:transformation_function_type_graph_type_model}, \cref{defin:transformation_framework:instance_models_and_instance_graphs:combining_transformation_functions:transformation_function_instance_model_instance_graph} and \cref{defin:transformation_framework:instance_models_and_instance_graphs:combining_transformation_functions:transformation_function_instance_graph_instance_model} are the relevant definitions that specify the properties of (composable) transformation functions between different models and graphs.

    The formalisation is considered suitable since it gives rise to a significant set of possible transformations, which allow for building large complex models. The existence and correctness of this set are validated using the library of transformations in \cref{chapter:library_of_transformations}, which defines and proves a small set of possible transformations within this formalisation. These transformations show the existence of model transformations between Ecore and GROOVE that fit within the presented formalisation. Furthermore, the transformations can map between the formalisations of Ecore and GROOVE by using these formalisations as input and output, giving confidence in the fact that multiple transformation functions can cover all elements of Ecore and GROOVE. Therefore, the formalisation of the model transformations themselves can be considered suitable.

    \item ``What model transformations are correct within the formalisation?''

    \cref{defin:transformation_framework:type_models_and_type_graphs:combining_transformation_functions:transformation_function_type_model_type_graph}, \cref{defin:transformation_framework:type_models_and_type_graphs:combining_transformation_functions:transformation_function_type_graph_type_model}, \cref{defin:transformation_framework:instance_models_and_instance_graphs:combining_transformation_functions:transformation_function_instance_model_instance_graph} and \cref{defin:transformation_framework:instance_models_and_instance_graphs:combining_transformation_functions:transformation_function_instance_graph_instance_model} constrain the transformation functions that are considered syntactically valid within the formalisation. Furthermore, the library of transformations in \cref{chapter:library_of_transformations} has shown some examples of transformations that are proven to be valid with respect to these definitions, and each of these transformations is also reversible. Therefore, multiple transformations have been presented which are each syntactically correct within the formalisation. Furthermore, the formalisation allows us the combine these transformation functions (as presented in \cref{chapter:transformation_framework}), while maintaining the correctness of the functions. This has also been proven as part of this thesis.

    The answer to this question is validated by validating all the correctness proofs. For this thesis, all the proofs are validated using the Isabelle theorem prover, as presented by \cref{sec:background:theorem_proving_using_isabelle}. Therefore, it is validated that the proofs are correct, meaning that by following the definitions, it is clear which model transformations are syntactically valid within the formalisation. Therefore, the question is considered answered.

    \item ``How can correct model transformations between Ecore and GROOVE be composed?''

    As explained earlier in \cref{sec:introduction:validation}, answering this question consists of two parts. First \cref{chapter:transformation_framework} has shown on a formal level how model transformations are combined. This formalisation has been done for transformations at both levels. Secondly, it has been shown how to compose these transformations in practice, which can be found in \cref{chapter:application}.

    \cref{chapter:transformation_framework} describes the composability of model transformations by explaining the necessary definitions to combine models, graphs and transformation functions. Using \cref{defin:transformation_framework:type_models_and_type_graphs:combining_transformation_functions:tg_combine_mapping_function_correct}, \cref{defin:transformation_framework:type_models_and_type_graphs:combining_transformation_functions:tmod_combine_mapping_function_correct}, \cref{defin:transformation_framework:type_models_and_type_graphs:combining_transformation_functions:tmod_combine_mapping_function_correct} and \cref{defin:transformation_framework:instance_models_and_instance_graphs:combining_transformation_functions:imod_combine_mapping_function_correct} it is possible to show that these definitions indeed give rise to a new transformation function, which is then proven to be syntactically correct.

    \cref{chapter:application} shows how to apply these definitions by composing a large transformation function out of the small transformation functions presented in the library of transformations (see \cref{chapter:library_of_transformations}). This example shows how the definitions from \cref{chapter:transformation_framework} can be applied in practice, answering the practical part of the question.

    As explained before, all proofs within this thesis have been validated using the Isabelle theorem prover, as presented by \cref{sec:background:theorem_proving_using_isabelle}. That means that the proofs presented to prove the composition of transformation functions are correct. The correctness of these proofs validates that the method of composing transformation functions is syntactically correct. Furthermore, the example of \cref{chapter:application} shows that it is possible to apply these definitions in practice, which validates that there is actual use for the composability definitions. Together this shows that all parts of the answer are validated, so the question is considered validated.
\end{enumerate}

With the answers to the subquestions given, it is possible to answer the main research question:

``What is a suitable formalisation for composable model transformations between Ecore and GROOVE that gives rise to correct model transformations between Ecore and GROOVE?''

In order to give a formalisation for the model transformations, formalisations for Ecore and GROOVE were needed. Subquestions 1 and 2 have shown suitable transformations for both languages. The formalisation for composable model transformations is given as part of \cref{chapter:transformation_framework}. Subquestions 3 explains that the formalisation is suitable, whereas subquestion 5 shows that the formalisation indeed allows for composable model transformations. The correctness of these transformations is proven, which is explained in subquestions 4 and 5. Finally, \cref{chapter:library_of_transformations} and \cref{chapter:application} have shown that there indeed exist such model transformations in practice. The existence of these model transformations is also explained in subquestions 3 and 5. Altogether, these answer the main research question, as the thesis has shown that there exists a suitable formalisation for composable model transformations between Ecore and GROOVE. It has also shown that this formalisation gives rise to syntactically correct model transformations between Ecore and GROOVE.