\section{Formalism}\label{sec:check}

Next we formalize Bounded Refinement Types by defining
a core calculus \boundedcorelan and showing how it can
be reduced to \corelan, the core language of Abstract
Refinement Types~\ref{chapter:abstractrefinements}.
%
We start by defining the syntax
and semantics of \corelan~(\S~\ref{sec:syntax-corelan})
and the syntax of \boundedcorelan~(\S~\ref{sec:syntax-boundedcorelan}).
%
Next, we provide a translation from \boundedcorelan to
\corelan ~(\S~\ref{sec:translation}).
%
Then, we prove soundness by showing that our translation
is semantics preserving~(\S~\ref{sec:soundness}).
%
Finally, we describe how type inference remains
decidable in the presence of bounded refinements~(\S~\ref{sec:infer}).

\subsection{Syntax of \corelan}\label{sec:syntax-corelan}

\input{text/boundedrefinements/syntax}

We build our core language on top of \corelan, the language
of Abstract Refinement Types\ref{chapter:abstractrefinements}.
%
Figure~\ref{fig:boundedrefinements:syntax} summarizes the syntax of \corelan,
a polymorphic $\lambda$-calculus extended with abstract
refinements.
%
For an easier transition to the syntax of Bounded Refinement Types,
we rewrite the syntax of \corelan as initially presented in Figure~\ref{fig:abstractrefinements:syntax}
by stratifying type schemata to include bounded types, that for now,
are plain types.

\mypara{The Expressions} of \corelan include the usual variables $x$,
primitive constants $c$, $\lambda$-abstraction $\efunt{x}{\rtyp}{e}$,
application $\eapp{e}{e}$,
let bindings $\elet{x}{e}{e}{\rtyp}$,
type abstraction $\etabs{\alpha}{e}$,
and type application $\etapp{e}{\rtyp}$.
(We add let-binders to \corelan
from Figure~\ref{fig:abstractrefinements:syntax} as they can be reduced to $\lambda$-abstractions
in the usual way).
%
The parameter $\rtyp$ in the type application is a \emph{refinement
type}, as described shortly.  Finally, \corelan includes refinement
abstraction $\epabs{\rvar}{\rtyp}{e}$, which introduces a refinement
variable $\rvar$ (with its type $\rtyp$), which
can appear in refinements inside $e$, and the corresponding refinement
application $\epapp{e}{\constraint}$ that substitutes an abstract refinement
with the parametric refinement $\constraint$, \ie
refinements $\reft$ closed under lambda abstractions.
%%%\nv{Suggestion: replace $\epapp{e}{e}$ with $\epapp{e}{\phi}$ in the syntax}
%%%\nv{If abstract refinements can be instantiated with arbitrary expressions,}
%%%\nv{then all the careful syntax that ensures that refinements r come from}
%%%\nv{decidable theory is useless.}
%%%\nv{In Abstract refinement paper we had}
%%%\nv{arbitrary expressions as refinements thus we did not have this problem}
%%%\nv{It seems that bounds $\phi$, ie x-parametric refinements, is a good}
%%%\nv{candidate to substitute abstract refinements, ie replace $\epapp{e}{e}$}
%%%\nv{with $\epapp{e}{\phi}$}
%%%\nv{Question: is this a coincidence?}

\mypara{The Primitive Constants} of \corelan include
\etrue, \efalse, @0@, @1@, @-1@, \etc. In addition, we include a
special untypable constant \ecrash that models ``going wrong''.
Primitive operations return a crash when invoked with inputs
outside their domain, \eg when @/@ is invoked with @0@ as the
divisor or when an @assert@ is applied to \efalse.

\mypara{Atomic Refinements} $\areft$ are either concrete or abstract refinements.
%
A \emph{concrete refinement} \creft is a boolean valued expression
(such as a constant, negation, equality, \etc)
drawn from a \emph{strict subset} of the language of expressions
which includes only terms that
%
(a)~neither diverge nor crash and
%
(b)~can be embedded into an SMT decidable logic including
%
the quantifier free theory of linear arithmetic and uninterpreted
functions~\cite{Vazou14}.
%
An \emph{abstract refinement} $\rvapp{\pi}{x}$ is an application of
a refinement variable $\pi$ to a sequence of program variables.
%
A \emph{refinement} \reft is either a conjunction or
implication of atomic refinements.
%
To enable inference, we only allow implications to appear within
bounds $\constraint$ (\S~\ref{sec:infer}).

\mypara{The Types of} \corelan, written $\rtyp$, include basic types,
dependent functions and schemata quantified over type and refinement
variables $\tvar$ and $\rvar$ respectively.
%
A basic type is one of $\tbint$, $\tbbool$, or a type
variable $\alpha$.
%
A refined type $\rtyp$ is either a refined basic type $\tref{b}{\reft}$,
or a dependent function type $\trfun{x}{\rtyp}{\rtyp}{\reft}$ where
the parameter $x$ can appear in the refinements of the output type.
%
(We include refinements for functions, as refined type variables can be
replaced by function types. However, typechecking ensures these refinements
are trivially true).
%
In \corelan bounded types $\bt$ are just a synonym for types $\rtyp$.
%
Finally, schemata quantify bounded types over type
and refinement variables.

\subsection{Syntax of \boundedcorelan}\label{sec:syntax-boundedcorelan}

\begin{figure}[t!]
\centering
\captionsetup{justification=centering}
$$
\begin{array}{rrcl}
\centering

\emphbf{Bounded Types} \quad
  & \bt         & ::= & \rtyp
                        \spmid \tconstraint{\constraint}{\bt} \\[0.03in]

\emphbf{Expressions} \quad
  & e           & ::= & \dots
                        \spmid \econstraint{\constraint}{e}
                        \spmid \econstantconstraint{e}{\constraint} \\
\end{array}
$$
\caption{Extending Syntax of \corelan to \boundedcorelan.}
\label{fig:boundedsyntax}
\end{figure}
\ra{1.0}

Figure~\ref{fig:boundedsyntax} shows how we obtain the syntax for
\boundedcorelan by extending the syntax of \corelan with
\emph{bounded} types.

\mypara{The Types} of \boundedcorelan extend those of \corelan with
bounded types $\bt$, which are the types $\rtyp$ guarded by bounds
$\constraint$.

%% {HERE: Colin:How does $\phi$ influence or  interact with $\rho$?  It’s not clear here.}
\mypara{The Expressions} of \boundedcorelan extend those of \corelan
with \emph{abstraction} over bounds $\econstraint{\constraint}{e}$ and
\emph{application} of bounds $\econstantconstraint{e}{\constraint}$.
%
Intuitively, if an expression $e$ has some type $\bt$
then $\econstraint{\constraint}{e}$ has the type
$\tconstraint{\constraint}{\bt}$.
%
We include an explicit bound application form
$\econstantconstraint{e}{\constraint}$ to simplify
the formalization; these applied bounds are automatically
synthesized from the type of $e$, and are of the form
$\overline{\efunt{x}{\bt}{}}{\etrue}$.

\mypara{Notation.}
We write
$b$,
$\tpp{b}{\rvapp{\pi}{x}}$, and
$\tpref{b}{\rvapp{\pi}{x}}{\reft}$
to abbreviate
$\tref{b}{\etrue}$,
$\tref{b}{\rvapp{\pi}{x}\ \vref}$, and
$\tref{b}{\reft \wedge \rvapp{\pi}{x}\ \vref}$
respectively.
We say a type or schema is \emph{non-refined} if all the
refinements in it are $\true$.
%
We get the \textit{shape} of a type $\rtyp$ (\ie the System-F type)
by the function $\toshape{\rtyp}$ defined:
%
\begin{align*}
\toshape{\tref{b}{\reft}} \defeq &\ b \\
\toshape{\trfun{x}{\rtyp_1}{\rtyp_2}{\reft}} \defeq &\ \toshape{\rtyp_1} \rightarrow \toshape{\rtyp_2}
\end{align*}

\subsection{Translation from \boundedcorelan to \corelan}
\label{sec:translation}

Next, we show how to translate a term from \boundedcorelan to
one in \corelan. We assume, without loss of generality that the
terms in \boundedcorelan are in Administrative Normal Form
(\ie all applications are to variables).

\mypara{Bounds Correspond To Functions} that explicitly
``witness'' the fact that the bound constraint holds at a
given set of ``input'' values.
%
That is we can think of each bound as a universally quantified
relationship between various (abstract) refinements; by ``calling''
the function on a set of input values $x_1,\ldots,x_n$, we get
to \emph{instantiate} the constraint for that particular set
of values.

\mypara{Bound Environments} \benv are used by our translation
to track the set of
bound-functions (names) that are in scope at each program point.
%
These names are distinct from the regular program variables that
will be stored in Variable Environments \cenv.
%
We give bound functions distinct names so that they cannot appear
in the regular source, only in the places where calls are inserted
by our translation.
%
The translation ignores refinements entirely; both environments
map their names to their non-refined types.

\mypara{The Translation is formalized} in
Figure~\ref{fig:translation} via a
relation $\txexpr{\cenv}{\benv}{e}{e'}$
that translates the expression
$e$ in $\boundedcorelan$ into
$e'$ in $\corelan$.
%
Most of the rules in figure~\ref{fig:translation}
recursively translate the sub-expressions.
%
Types that appear inside expressions are syntactically restricted to
not contain bounds,
thus types inside expressions do not require translation.
%
Here we focus on the three interesting rules:

\begin{enumerate}
%
\item \emphbf{At bound abstractions} $\econstraint{\constraint}{e}$
 we convert the bound $\constraint$ into a bound-function
 parameter of a suitable type,
%
\item \emphbf{At variable binding sites} \ie $\lambda$- or let-bindings,
 we \emph{use} the bound functions to \emph{materialize} the
 bound constraints for all the variables in scope after the binding,
%
\item \emphbf{At bound applications} $\econstantconstraint{e}{\constraint}$
 we \emph{provide} regular functions that witness that the bound constraints hold.
%
\end{enumerate}

\input{text/boundedrefinements/translation}

\mypara{1.  Rule~\txCAbs} translates bound abstractions
$\econstraint{\phi}{e}$ into a plain $\lambda$-abstraction.
%
In the translated expression $\efunt{f}{\txbound{\constraint}}{e'}$
the bound becomes a function named $f$ with type
$\txbound{\constraint}$ defined:
%
\begin{align*}
\txbound{\efunt{x}{b}{\constraint}} \defeq & \tfun{x}{b}{\txbound{\constraint}}\\
\txbound{r} \defeq & \tref{\tbbool}{r}
\end{align*}
%
That is, $\txbound{\constraint}$ is a function type whose
final output carries the refinement corresponding to
the constraint in $\constraint$.
%
Note that the translation generates a fresh name $f$ for
the bound function (ensuring that it cannot be used in
the regular code) and saves it in the bound environment
$\benv$ to let us materialize the bound constraint when
translating the body $e$ of the abstraction.

\mypara{2. Rules~\txFun and~\txLet} materialize bound
constraints at variable binding sites ($\lambda$-abstractions
and let-bindings respectively).
%
%Even though application  of the axiomatic functions is required at specific places
%
If we view the bounds as universally quantified constraints
over the (abstract) refinements, then our translation exhaustively
and eagerly \emph{instantiates} the constraints at each point that
a new binder is introduced into the variable environment, over all
the possible candidate sets of variables in scope at that point.
%
The instantiation is performed by $\closure{\cenv}{\benv}{e}{x:\rtyp}$
%
%% \begin{figure}[t]
$$\begin{array}{rcl}
\closure{\cenv}{\benv}{e}{\tbind{x}{\utyp}}
  & \defeq & {\wraplet{e}{\cands{\cenv}{\benv}}} \\[0.05in]
\wraplet{e}{\set{e_1,\ldots,e_n}}
  & \defeq & {\elett{t_1}{e_1}{\ldots} \elett{t_n}{e_n}{e}} \\
  &        & {\mbox{where $t_i$ are fresh \tbbool binders}} \\[0.05in]
\cands{\cenv}{\benv}
  & \defeq & \{ \  f\ \overline{x}  \ | \ \tbind{f}{\utyp} \leftarrow \benv
                  , \ \overline{\tbind{x}{\_}} \leftarrow \cenv,\
    \EXT{\cenv}{f}{\utyp} \vdash_B \tbind{ f\ \overline{x}}{\tbbool}\ \} \
\end{array}$$
%% \caption{\textbf{Materializing Bound Functions}}
%% \label{fig:closure}
%% \end{figure}
% described in Figure~\ref{fig:closure} that
The function takes the environments
$\cenv$ and $\benv$, an expression $e$ and a variable $x$ of type
$\rtyp$ and uses let-bindings to materialize all the bound
functions in $\benv$ that accept the variable $x$.
%
Here, $\cenv \vdash_B \tbind{e}{\utyp}$ is the standard typing
derivation judgment for the non-refined System F and so
we elide it for brevity.
%%%\nv{Note that with the above definitions,}
%%%\nv{$\closure{\cenv}{\benv}{e}{\tbind{x}{\utyp}}$}
%%%\nv{will instantiate the bounds to all combinations}
%%%\nv{of variables, not only the ones that contain $x$}
%%%\nv{so you get much much repetition of instantiations}
%%%\nv{which is not unsound, but redundant}

\mypara{3. Rule~\txCApp} translates bound applications
$\econstantconstraint{e}{\constraint}$ into plain $\lambda$
abstractions that witness that the bound constraints
hold.
%
That is, as within $e$, bounds are translated to a bound
function (parameter) of type $\txbound{\constraint}$, we
translate $\constraint$ into a $\lambda$-term that, via
subtyping must have the required type $\txbound{\constraint}$.
%
We construct such a function via $\ctofun{\constraint}$
that depends only on the \emph{shape} of the bound,
\ie the non-refined types of its parameters (and not
the actual constraint itself).
%%
\begin{align*}
\ctofun{\reft} \defeq & \true \\
\ctofun{\efunt{x}{b}{\constraint}} \defeq &  \efunt{x}{b}{\ctofun{\constraint}}
\end{align*}
%
This seems odd: it is simply a constant function, how
can it possibly serve as a bound? The answer is that
subtyping in the translated \corelan term will verify
that in the context in which the above constant function
is created, the singleton $\true$ will indeed carry
the refinement corresponding to the bound constraint,
making this synthesized constant function a valid
realization of the bound function.
%
Recall that in the example @ex2@ of the overview (\S~\ref{sec:overview:implementation})
the subtyping constraint that decides if the constant $\true$
is a valid bound reduces to the equation \ref{vc:bounded:ex2}
that is a tautology.
%% \rj{ex2 EXAMPLE HERE?} \ref{vc:ex2}

\subsection{Soundness}\label{sec:soundness}

\begin{comment}
Definition of \erase{\dot}:

\begin{align*}
\erase{x}                      &= x \\
\erase{c}                      &= c\\
\erase{\eapp{e_1}{e_2}}        &= \eapp{\erase{e_1}}{\erase{e_2}}\\
\erase{\etabs{\alpha}{e}}      &= \etabs{\alpha}{\erase{e}}\\
\erase{\etapp{e}{\tau}}        &= \etapp{\erase{e}}{\erase{\tau}}\\
\erase{\epabs{\rvar}{\tau}{e}} &= \epabs{\rvar}{\erase{\tau}}{\erase{e}}\\
\erase{\epapp{e_1}{e_2}}           &= \epapp{\erase{e_1}}{\erase{e_2}}\\
%
\erase{\elet{x}{e_x}{e}{\tau}} &= \elet{x}{\erase{e_x}}{\erase{e}}{\erase{\tau}}\\
%
\erase{\econstraint{\phi}{e}}  &= \erase{e}\\
\erase{\econstantconstraint{e}{\phi}} &= \erase{e}
\end{align*}

\begin{definition}% [Operational]
$ e\boundedgoestostar{c} \Leftrightarrow {\erase{e}}\goestostar{c} $.
\end{definition}

The below does not hold:
\begin{lemma*}[Operational]%[Equivalence of Operational]
\label{theorem:operational}
If
   $\vdash_{B} e : \tau$ and
   $\txexpr{\emptyset}{\emptyset}{e}{e'}$
then
   $\erase{e} \goestostar{c} \Leftrightarrow {e'} \goestostar{c} $.
\end{lemma*}

Counterexample e = \{φ} =>  0

\end{comment}

\mypara{The Small-Step Operational Semantics} of \boundedcorelan
are defined by extending a similar semantics for \corelan
which is a standard call-by-value calculus where abstract
refinements are boolean valued functions.
%
Let $\stepcore$ denote the transition relation defining
the operational semantics of \corelan and \tclos{\stepcore}
denote the reflexive transitive closure of $\stepcore$.
%
We obtain the transition relation $\stepboundedcore$:
%
\begin{align*}
\econstantconstraint{(\econstraint{\constraint}{e})}{\constraint} &\boundedgoesto{e} &
e  & \stepboundedcore e', \text{if}\ {e \stepcore e'}
\end{align*}
%%%$$
%%%\inference{
%%%}{
%%%\econstantconstraint{(\econstraint{\constraint}{e})}{\constraint} \boundedgoesto{e}
%%%}[\rtobound]
%%%\qquad \qquad
%%%\inference{e \stepcore e'}
%%%          {e \stepboundedcore e'}[\rulename{O-Oth}]
%%%$$
%
Let $\boundedgoestostar{}$ denote the reflexive transitive
closure of $\stepboundedcore$.

\mypara{The Translation is Semantics Preserving} in the sense that
if a source term $e$ of $\boundedcorelan$ reduces to a constant
then the translated variant of $e'$ also reduces to the same
constant (as show in~\citep{vazou15techrep}):

\begin{lemma*}[Semantics Preservation]\ifextended{[Semantics Preservation]}{}
\label{theorem:operational}
If $\txexpr{\emptyset}{\emptyset}{e}{e'}$ and
   $e \boundedgoestostar{c}$
then $e' \goestostar{c}$.
\end{lemma*}

%%\rj{what about the extra let-bindings? techrep.}
%%
\ifextended{
\begin{proof}
By assumption, there exists a sequence
$e \equiv e_1 \boundedgoesto{e_2} \boundedgoesto{} \dots
\boundedgoesto{e_n\equiv c} $.
%
Let $i$ be the largest index in which rule \rtobound was applied.
%
Then, for some $\phi$ and $\phi'$,
$e_i$ contains a sub-expression of the form
$\econstantconstraint{(\econstraint{\phi}{e_i^0})}{\phi'}$.
%
Let $e_i^1$ be the expression we get if we replace
$\econstantconstraint{(\econstraint{\phi}{e_i^0})}{\phi'}$
with $e_i^0$ in $e_i$.
%
By the way we choose $i$, there exist a sequence
$e_i^1 \goestostar{c}$.

Let $e_i^2$
be the expression we get if we replace
$\econstantconstraint{(\econstraint{\phi}{e_i^0})}{\phi'}$
with $\eapp{(\efunt{f}{\txbound{\phi}}{e_i^0})}{(\ctofun{\phi'})}$ in $e_i$.
%
Then, since $f$ does not appear in $e_i^0$,
$e_i^2 \goestostar{c}$.
%
Finally,
let $g \defeq \ctofun{\phi'}$, then
by the definition of
$\ctofun{\cdot}$
we have that  $\forall e_1 \dots e_n$
if
there exists a type $\tau$ such that
$\emptyset \vdash g \ e_1 \dots e_n : \tau $,
then $g \ e_1 \dots e_n \goestostar{true}$.
%
Thus, for any expression,
if $e \goestostar{c}$, then $\elett{t}{f \ e_1 \dots e_n}{e}\goestostar{c}$

From the above, by the way we choose $i$ we have that
there exists a sequence
%
$\txex{e_i} \hookrightarrow \dots \hookrightarrow {c}$.

Since $n$ is finite, we iteratively apply the above procedure to
$e \equiv e_1\boundedgoesto{} \dots \boundedgoesto{} \txex{e_i}\hookrightarrow \dots \hookrightarrow {c}$.
until we get the sequence $ {\txex{e}}\goestostar{c}$.
\end{proof}}{}

\mypara{The Soundness of \boundedcorelan} follows by combining
the above Semantics Preservation
Lemma % ~\ref{theorem:operational}
with the soundness of \corelan.

\mypara{To Typecheck a \boundedcorelan program} $e$ we translate it
into a \corelan program $e'$ and then type check $e'$ using the rules of Figure~\ref{fig:rules};
if the latter check is safe, then we are guaranteed that the source term $e$ will
not crash:

\begin{theorem*}[Soundness]
\label{theorem:bounded}
If $\txexpr{\emptyset}{\emptyset}{e}{e'}$ and
   $\hastype{\emptyset}{e'}{\sigma}$
then $e \not \boundedgoestostar{\ecrash}$.
\end{theorem*}

\subsection{Inference}\label{sec:infer}

A critical feature of bounded refinements is that we can
automatically synthesize instantiations of the abstract
refinements by:
%
(1)~generating templates corresponding to the unknown types
    where fresh variables $\kvar$ denote the unknown refinements
    that an abstract refinement parameter $\rvar$ is instantiated
    with,
(2)~generating subtyping constraints over the resulting templates,
    and
(3)~solving the constraints via abstract interpretation.

\mypara{Inference Requires Monotonic Constraints.}
Abstract interpretation only works if the constraints
are \emph{monotonic}~\citep{cousotcousot77}, which in this case
means that the $\kvar$ variables, and correspondingly,
the abstract refinements $\rvar$ must only appear in
\emph{positive} positions within refinements (\ie not
under logical negations).
%
The syntax of refinements shown in Figure~\ref{fig:boundedrefinements:syntax}
violates this requirement via refinements of the
form $\rvapp{\rvar}{x} \Rightarrow \reft$.
%
%% (Note that the syntax otherwise only allows for positive
%% occurrences as $\neg (\rvapp{\rvar}{x})$ is disallowed.)

\mypara{We restrict implications to bounds} \ie prohibit
them from appearing elsewhere in type specifications.
%
Consequently, the implications only appear in the
\emph{output} type of the (first order) ``ghost''
functions that bounds are translated to.
%
The resulting subtyping constraints only have
\emph{implications inside super-types}, \ie as:
$$
\isSubType{\Gamma}{\stref{b}{\areft}}{\stref{b}{\areft_1 \Rightarrow \dots \Rightarrow \areft_{n} \Rightarrow\areft_q}}
$$
%
By taking into account the semantics of subtyping, we can
push the antecedents into the environment, \ie transform
the above into an equivalent constraint in the form:
$$
\isSubType{
\EXTT{
 \EXTT{\Gamma}{x_1}{\sxref{b_1}{\areft_1'}{x_1}},\dots
}{x_n}{\sxref{b_n}{\areft_n'}{x_n}}
}
{\stref{b}{\areft'}}
{\stref{b}{\areft_q'}}
$$
where all the abstract refinements variables $\rvar$
(and hence instance variables $\kvar$) appear positively,
ensuring that the constraints are monotonic, hence permitting
inference via Liquid Typing~\citep{LiquidPLDI08}.

\begin{comment}

\subsection{Semantics of \corelan}\label{sec:semantics-corelan}

\input{text/boundedrefinements/corerules}

Figure \ref{fig:rules} summarizes the static semantics of \corelan
as described in ~\citep{vazou13}.
Unlike~\citep{vazou13} that syntactically separates concrete ($p$)
from abstract ($\rvapp{\rvar}{x}$) refinements,
here, for simplicity, we merge both concrete and abstract refinements to
atomic refinements $\areft$.

\mypara{A type environment} $\Gamma$ is a sequence of type bindings $x:\sigma$.
We use environments to define three kinds of judgments:

\begin{itemize}
\item{\emphbf{Well-formedness judgments} (\isWellFormed{\Gamma}{\sigma})}
state that a type schema $\sigma$ is well-formed under environment
$\Gamma$. That is, the judgment states that the refinements in $\sigma$
are boolean expressions in the environment $\Gamma$.

\item{\emphbf{Subtyping judgments} (\isSubType{\Gamma}{\sigma_1}{\sigma_2})}
state that the type schema $\sigma_1$ is a subtype of the type schema
$\sigma_2$ under environment $\Gamma$. That is, the judgment states that
when the free variables of $\sigma_1$ and $\sigma_2$ are bound to values
described by $\Gamma$, the values described by $\sigma_1$ are a subset
of those described by $\sigma_2$.

\item{\emphbf{Typing judgments} (\hastype{\Gamma}{e}{\sigma})} state that
the expression $e$ has the type schema $\sigma$ under environment $\Gamma$.
That is, the judgment states that when the free variables in $e$ are bound
to values described by $\Gamma$, the expression $e$ will evaluate to a value
described by $\sigma$.
\end{itemize}

\mypara{The Well-formedness rules}
check that the concrete and abstract refinements are indeed $\tbbool$-valued
expressions in the appropriate environment. The key rule is \wtBase, which
checks that the refinement $\reft$ is boolean.

\mypara{The Subtyping rules}
stipulate when the set of values described by schema $\sigma_1$ is subsumed
by (\ie contained within) the values described by $\sigma_2$.
The rules are standard except for \tsubBase, which reduces subtyping of
basic types to validity of logical implications, by translating the
refinements $r$ and the environment $\Gamma$ into logical formulas:
%
\begin{align*}
\inter{r}      & \defeq r &
\inter{\Gamma} & \defeq \bigwedge \{ \SUBST{r}{\vref}{x}\ |\ (x, \tref{b}{r}) \in \Gamma \}
\end{align*}
%
Recall that we ensure that the refinements $r$
belong to a decidable logic so that validity
checking can be performed by an off-the-self
SMT solver.

\mypara{Type Checking Rules}
are standard except for \tpgen and \tpinst, which
pertain to abstraction and instantiation of abstract refinements.
%
The rule \tpgen is the same as \tfunction: we simply check the body
$e$ in the environment extended with a binding for the refinement
variable $\rvar$.
%
The rule \tpinst checks that the concrete refinement is of the appropriate
(unrefined) type $\tau$, and then replaces all (abstract) applications of
$\rvar$ inside $\sigma$ with the appropriate (concrete) refinement $\reft'$
with the parameters $\overline{x}$ replaced with arguments at that application.
%
In~\cite{vazou13} we prove the following soundness result for \corelan
which states that well-typed programs cannot crash:

\begin{lemma*}[Soundness of \corelan~\cite{vazou13}]
\label{theorem:core}
If   $\hastype{\emptyset}{e}{\sigma}$
then $e \not \goestostar{\ecrash}$.
\end{lemma*}

\end{comment}


\label{chapter:unit_tests}

In addition to the \maestro\ science problems, which use the full
capabilities of \maestro, there are a number of unit tests that
exercise only specific components of the \maestro\ solvers.  These
tests have their own drivers (a custom \code{varden.f90}) that
initialize only the data needed for the specific test and call
specific \maestro\ routines directly.

\section {\tt test\_advect}

  This test initializes a Gaussian density field (no other scalar
  quantities are used) and a uniform velocity field in any one of the
  coordinate directions.  The Gaussian profile is advected through the
  period domain exactly once and the error in the density profile (L2
  norm) is computed.  The driver for this problem does this for every
  dimension twice (once with a positive velocity and once with a
  negative velocity), and loops over all advection methods
  (\runparam{ppm\_type} {\tt = 0,1,2} and \runparam{bds\_type} {\tt = 1}).  After
  all coordinate directions are tested, the norms are compared to
  ensure that the error does not show any directional bias.

  Note: the BDS advection method does not guarantee that the error be
  independent of the advection direction---small differences can
  arise.  What's happening here is that within each cell BDS is trying
  to define a tri-linear profile for rho subject to the constrants in
  the BDS paper~\cite{bds3d} (constraints 1 and 2 on p.\ 2044 after eq. 3.4).  We
  do not solve the L2 minimization problem exactly so we iterate up to
  3 times using a simple heuristic algorithm designed to work toward
  the constraint.  The iteration will give different results depending
  on orientation since we work through the corners in arbitrary order.

\begin{figure}[t]
\centering
\includegraphics[height=1.9in]{\unitfigpath/dens_2d_orig_density}
\includegraphics[height=1.9in]{\unitfigpath/dens_2d_ppm1_xp_final_density}
\includegraphics[height=1.9in]{\unitfigpath/dens_2d_ppm1_xp_final_abserror}
%
\caption[Results of the {\tt test\_advect} unit test]{\label{fig:unit:advtest}
  Initial Gaussian density profile (left); density profile after
  advecting to the right for one period (center); absolute error
  between the final and initial density fields, showing the error in
  the advection scheme (right).}
\end{figure}

\section {\tt test\_average}

  This test initializes a 1D radial base state quantity with a
  Gaussian distribution, maps it into the 3D domain (assuming a
  spherical geometry) using the routines provided by
  the \code{fill\_3d\_module} module, and then calls \code{average} to
  put it back onto a 1D radial array.  This way we test the accuracy
  of our procedure to map between the 1D radial and 3D Cartesian
  states.  The output from this test was described in detail
  in \cite{multilevel}.

\section {\tt test\_basestate}

  This test initializes the base state to contain a hydrostatic
  model and then evolves the state with heating to watch the
  hydrostatic adjustment of the atmosphere.  In particular,
  the base state velocity, $w_0$, is computed in response to
  the heating and this is used to advect the base state density
  and compute the new pressure, $p_0$.  An early version of
  this routine was used for the plane-parallel expansion test
  in \cite{lowMach2}.  This version of the test was also shown
  for a spherical, self-gravitating star in \cite{multilevel}.

\section {\tt test\_diffusion}

  This test initializes a Gaussian temperature profile and calls
  the thermal diffusion routines in \maestro\ to evolve the state
  considering only diffusion.  The driver estimates a timestep
  based on the explicit thermal diffusion timescale and loops
  over calls to the thermal diffusion solver.  A Gaussian remains
  Gaussian when diffusing, so an explicit error can be computed
  by comparing to the analytic solution.  This test is
  described in \cite{xrb}.

\section {\tt test\_eos}

  This test sets up a 3-d cube with $\rho$ varying on one axis, $T$ on
  another, and the composition on the third.  The EOS is then called
  in every zone, doing $(\rho, T) \rightarrow  p, h, s, e$  and stores those
  quantities.  Then it does each of the different EOS types to recover
  either $T$ or $\rho$ (depending on the type), and stores the new $T$ (or
  $\rho$) and the relative error with the original value.  A plotfile is
  stored holding the results and errors.  This allows us to determine
  whether the EOS inversion routines are working right.

\section {\tt test\_particles}

  This test exercises the particle advection routine.  A simple
  circular velocity field, with the magnitude increasing with radius
  from the center is initialized.  A number of particles are then
  initialized at various radii from the center and they are advected
  for one period.  The particle paths should be perfect circles, and
  the final particle position should overlap with the initial
  position.

  Particle data is stored separately from the fluid data.  Instead
  of being part of the plotfiles, the particle data is outputted
  each timestep into files named {\tt timestamp\_*}, where
  the number indicates which processor did the writing.  These
  particle files can be processed and the particle data plotted
  using the python routines in {\tt data\_processing/python/}.

  The output from this test can be visualized with the script {\tt
  plot.py} in the test directory.  The output shows the particle
  paths (see figure~\ref{fig:unit:particles}).

\begin{figure}[t]
\centering
\includegraphics[width=4in]{\unitfigpath/particle_paths}
%
\caption[Particle paths for the {\tt test\_particles} problem]{\label{fig:unit:particles}
  Particle paths for the {\tt test\_particles} problem.  The initial
  position of the particles is marked with an $\times$.}
\end{figure}

\section {\tt test\_projection}

  This tests the projection routines in 2- and 3-d---either the hgprojection
  ({\tt project\_type = 1}) or the MAC projection ({\tt project\_type =
  2}).  A divergence-free velocity field is initialized and then
  ``polluted'' by adding the gradient of a scalar.  The form of the
  scalar differs depending on the boundary conditions (wall and
  periodic are supported currently).  Finally, the hgproject routine
  is called to recover the initial divergence-free field.
  Figure~\ref{fig:unit:projtest} shows the initial field, polluted
  field, and result of the projection for the hgproject case.

\begin{figure}[t]
\centering
\includegraphics[height=1.9in]{\unitfigpath/wall_u_init_x-velocity}
\includegraphics[height=1.9in]{\unitfigpath/wall_u_plus_grad_phi_x-velocity}
\includegraphics[height=1.9in]{\unitfigpath/wall_u_new_x-velocity}
%
\caption[Results of the 2-d {\tt test\_projection} unit test]{\label{fig:unit:projtest}
  Initial divergence free velocity field (x-component; left); Velocity
  field plus gradient of a scalar (x-component; center); and resulting
  velocity after projecting out the non-divergence free portion
  (x-component; right).  This is with slipwall boundary conditions on
  all sides, a 2-level grid with the left half refined and right half
  coarse, and the hgprojection tested.}
\end{figure}

\begin{figure}
\centering
\includegraphics[height=8in]{\unitfigpath/test_project_3d}
%
\caption[Results of the 3-d {\tt test\_projection} unit test]{\label{fig:unit:projtest3d}
  Projection test in 3-d showing the x-velocity (left), y-velocity
  (middle), and z-velocity (right) initially (top row), after the
  gradient of a scalar is added (center row), and the resulting
  velocity after the projection.  This is with slipwall boundary conditions
  on all sides, a 2-level grid with an octant refined, and the hgprojection.}
\end{figure}

\section{\tt test\_react}

  This simply tests the reaction network by calling
  the \maestro\ \code{react\_state} routine directly.  The network is
  selected in the {\tt GNUmakefile} by setting the {\tt NETWORK\_DIR}
  variable.  A 3d cube is setup with density varying on one axis,
  temperature varying on another, and the composition varying on the
  third.  The density and temperature ranges are set in the inputs
  file.  The composition is read in via an input file.

  A good use of this test is to test whether a burner is threadsafe.
  This is accomplished by compiling with OpenMP (setting {\tt OMP=t})
  and the running with 1 thread and multiple threads (this can be done
  by setting the environment variable {\tt OMP\_NUM\_THREADS} to the
  desired number of threads).  Since each zone is independent of the
  others, the results should be identical regardless of the number
  of threads.  This can be confirmed using the {\tt fcompare} tool
  in {\tt amrex/Tools/Postprocessing/F\_Src/}.


%!TEX program = xelatex
%!TEX root = ./thesis.tex
\appendix

\chapter{Miscellaneous IAQI Standards}\label{chapter:IAQI_standards}

This appendix section lists a complete data preprocessing results

It uses the same way to map concentration data into IAQI, and finally calculates AQI, the only different is the thresholds. Both USA and China use the maximum IAQI to represent final AQI.

According to Technical Regulation on Ambient Air Quality Index (on trial) \cite{regulation2012} release by former Ministry of Environmental Protection of the People's Republic of China, IAQI (individual air quality index) and AQI (air quality index) should be calculated by the following equations:

\begin{equation}
    IAQI_p = \frac{IAQI_{Hi}-IAQI_{Lo}}{BP_{Hi}-BP_{Lo}}(C_p-BP_{Lo})+IAQI_{Lo}
\end{equation}

or equivalently:

\begin{equation}
    \label{formula:IAQI_appendix1}
    IAQI_p = \frac{C_p-BP_{Lo}}{BP_{Hi}-BP_{Lo}}(IAQI_{Hi}-IAQI_{Lo})+IAQI_{Lo}
\end{equation}

In formula \ref{formula:IAQI_appendix1}, $C_p$ is the concentration value for certain category like $PM_{2.5}$; $BP_{Hi}$ and $BP_{Lo}$ are high threshold/low threshold near $C_p$, respectively; $IAQI_{Hi}$ and $IAQI_{Lo}$ are high threshold/low threshold $C_p$, respectively.

Then calculate AQI by making use of the maximum of these individual IAQI values \ref{appendix:formula:AQI}:

\begin{equation}
    \label{appendix:formula:AQI}
    AQI = \max\{IAQI_1,IAQI_2,IAQI_3,...,IAQI_n\}
\end{equation}

For each pollutant category, concentration value will firstly all be mapped into the IAQI value according to its corresponding thresholds in the thresholds table \ref{table:IAQI-thresholds}.

The regulation \cite{regulation2012} determines concentration value thresholds for pollutant categories such as $SO_2$, $NO_2$, $CO$, $PM_{2.5}$, $PM_{10}$ and $O_3$. For one-hour average value and 24-hour average value, thresholds are determined differently, for $SO_2$, $NO_2$, $CO$. Note that the thresholds for $PM_{2.5}$ and $PM_{10}$ concentration value are defined by their 24-hour average, this means the method of calculating IAQI for a single day. However, it doesn't matter when we uses these thresholds for realtime (1 Hz) $PM_{2.5}$ and $PM_{10}$ concentration data. Table \ref{table:IAQI-thresholds} shows the $PM_{2.5}$ and $PM_{10}$ data-IAQI mapping relationship. Followed by data we could collect in our AIoT system, we only consider $PM_{2.5}$ and $PM_{10}$ data.

\begin{table}[!htbp]
    \centering
    \caption{Concentration thresholds of IAQI with regard to pollutant categories, China}
    \label{table:IAQI-thresholds}
    \begin{tabular}{|l|l|l|}
    \hline
    IAQI & $PM_{2.5}$ ($\mu g/m^3$) & $PM_{10}$ ($\mu g/m^3$) \\ \hline
    0    & 0   & 0   \\ \hline
    50   & 35  & 0   \\ \hline
    100  & 75  & 150 \\ \hline
    150  & 115 & 250 \\ \hline
    200  & 150 & 350 \\ \hline
    300  & 250 & 420 \\ \hline
    400  & 350 & 500 \\ \hline
    500  & 500 & 600 \\ \hline
    \end{tabular}
\end{table}

Use the formula and table above, we can plot in Figure \ref{fig:iaqi} the value-IAQI mapping curve for both $PM_{2.5}$ and $PM_{10}$. This is a visualization for the air pollutant concentration and its IAQI. For each category, data vary from 0 to 800, linearly.

\begin{figure}[!htbp]
    \centering
    \includegraphics[width=0.7\linewidth]{fig/iqai.pdf}
    \caption{IAQI curve of $PM_{2.5}$ (orange line) and $PM_{10}$ (green line) data.}
    \label{fig:iaqi}
\end{figure}

For data we collected, following the the IAQI thresholds table, each data of a pollutant category are mapped into IAQI values and have six levels, level 1 to 6. In USA's Technical Assistance Document \cite{daq2018} level 1 to 6 of the air quality index is represented by color \colorbox{green}{green}, \colorbox{yellow}{yellow}, \colorbox{orange}{orange}, \colorbox{red}{red}, \colorbox{purple}{purple} and \colorbox{maroon}{maroon}, respectively. We mapped our $PM_{2.5}$ and $PM_{10}$ data into IAQI respectively, as Figure \ref{fig/pm25_all_iaqi} and Figure \ref{fig/pm10_all_iaqi} shows.

\begin{figure}[!htbp]
    \begin{center}
    \includegraphics[width=\linewidth]{fig/iaqi/pm25_all_iaqi.png}
    \end{center}
    \caption{All $PM_{2.5}$ IAQI data.}
    \label{fig/pm25_all_iaqi}
\end{figure}

\begin{figure}[!htbp]
    \begin{center}
    \includegraphics[width=\linewidth]{fig/iaqi/pm10_all_iaqi.png}
    \end{center}
    \caption{All $PM_{10}$ IAQI data.}
    \label{fig/pm10_all_iaqi}
\end{figure}

\section{Other IAQI Calculation Method}
The air pollutant concentration thresholds are different in USA's Technical Assistance Document \cite{daq2018}, see Table \ref{table:IAQI-thresholds-USA}.

\begin{table}[!htbp]
    \centering
    \caption{Concentration thresholds of IAQI with regard to pollutant categories, USA}
    \label{table:IAQI-thresholds-USA}
    \begin{tabular}{|l|l|l|}
    \hline
    IAQI & $PM_{2.5}$ ($\mu g/m^3$) & $PM_{10}$ ($\mu g/m^3$) \\ \hline
    0    & 0     & 0   \\ \hline
    50   & 12.1  & 55  \\ \hline
    100  & 35.5  & 155 \\ \hline
    150  & 55.5  & 255 \\ \hline
    200  & 150.5 & 355 \\ \hline
    300  & 250.5 & 425 \\ \hline
    500  & 500.4 & 604 \\ \hline
    \end{tabular}
\end{table}

It uses the same way to map concentration data into IAQI, and finally calculates AQI, the only different is the thresholds. Both USA and China use the maximum IAQI to represent final AQI. As for Hong Kong's method for IAQI and AQI calculation, they use integer AQHI(Air Quality Health Index) and map it into five health risk levels. AQHI uses the \textbf{sum} of the sum of the percentage added health risk (\%AR) with regard to the 3-hour moving average concentrations belonging to four criteria of air pollutants: ozone ($O_3$), nitrogen dioxide ($NO_2$), sulphur dioxide ($SO_2$), and particulate matter (PM) \cite{aqhi2018}.

The calculation uses weighting and exponent, for example:

\begin{equation}
    \%AR(PM_{2.5}) = (e^{(\beta(PM_{2.5})\times C(PM_{2.5}))} – 1) \times 100\%
\end{equation}

where $\beta(PM_{2.5})$ is the added health risk factor for $PM_{2.5}$ and $\beta(PM_{2.5})=0.0002180567$. It is a regression coefficient technically.

\chapter{Other Data Preprocessing Results}\label{chapter:other_data_preprocessing_results}

This appendix section lists the complete data preprocessing results.

Figure \ref{fig:appendix:all_pm25_iaqi_level} shows

\begin{figure}[!htbp]
    \begin{center}
        \includegraphics[width=\linewidth]{fig/iaqi_level/pm25_0_iaqi_level_dpi1200.png}
    \end{center}
    \begin{center}
        \includegraphics[width=\linewidth]{fig/iaqi_level/pm25_1_iaqi_level_dpi1200.png}
    \end{center}
    \begin{center}
        \includegraphics[width=\linewidth]{fig/iaqi_level/pm25_2_iaqi_level_dpi1200.png}
    \end{center}
    \begin{center}
        \includegraphics[width=\linewidth]{fig/iaqi_level/pm25_3_iaqi_level_dpi1200.png}
    \end{center}
    \caption{All $PM_{2.5}$ IAQI level.}
    \label{fig:appendix:all_pm25_iaqi_level}
\end{figure}

Figure \ref{fig:appendix:all_pm25_iaqi_level_and_labeled} shows the origin IAQI level and labeled IAQI level in same figures. Origin IAQI level and labeled IAQI level are plotted with different linewidth. For each subfigure, the bolder line is the labeled IAQI level, and is wrapping the orgin IAQI curve, with contrast to the origin IAQI level.

\begin{figure}[!htbp]
    \begin{center}
        \includegraphics[width=\linewidth]{fig/labeled_iaqi_level/origin_and_labeled/pm25_0.png}
    \end{center}
    \begin{center}
        \includegraphics[width=\linewidth]{fig/labeled_iaqi_level/origin_and_labeled/pm25_1.png}
    \end{center}
    \begin{center}
        \includegraphics[width=\linewidth]{fig/labeled_iaqi_level/origin_and_labeled/pm25_2.png}
    \end{center}
    \begin{center}
        \includegraphics[width=\linewidth]{fig/labeled_iaqi_level/origin_and_labeled/pm25_3.png}
    \end{center}
    \caption{All labeled $PM_{2.5}$ IAQI level.}
    \label{fig:appendix:all_pm25_iaqi_level_and_labeled}
\end{figure}

Figure \ref{fig:appendix:all_pm25_polygonalized_iaqi_level} shows the results of yielding all polygonalized IAQI level lines from the labeled IAQI level.

\begin{figure}[!htbp]
    \begin{center}
        \includegraphics[width=\linewidth]{fig/labeled_iaqi_level/pm25_0_labeled_iaqi_level.png}
    \end{center}
    \begin{center}
        \includegraphics[width=\linewidth]{fig/labeled_iaqi_level/pm25_1_labeled_iaqi_level.png}
    \end{center}
    \begin{center}
        \includegraphics[width=\linewidth]{fig/labeled_iaqi_level/pm25_2_labeled_iaqi_level.png}
    \end{center}
    \begin{center}
        \includegraphics[width=\linewidth]{fig/labeled_iaqi_level/pm25_3_labeled_iaqi_level.png}
    \end{center}
    \caption{All polygonalized $PM_{2.5}$ IAQI level.}
    \label{fig:appendix:all_pm25_polygonalized_iaqi_level}
\end{figure}

% 多余的
% \chapter{Other Model Training Data}\label{chapter:other_model_training_data}


\subsection{The function \stackfull}
\Label{sec:stackfull}

The function \stackfull is the runtime version of the predicate \logicref{StackFull}.

\input{Listings/stack_full.h.tex}

As in the definition of the predicate \StackFull the implementation of
\stackfull in the next listing simply checks whether the
size of the stack equals its capacity.

\input{Listings/stack_full.c.tex}

Note that with our definition of stack equality (\S\ref{sec:stack-equality})
there can be equal stack with different capacities.
As a consequence, there can are equal stacks where one is full while the other is not.
In other words, \stackfull is not well-defined!


\documentclass[journal=jpcbfk,manuscript=article,layout=singlecolumn,articletitle=true]{achemso}
\setkeys{acs}{articletitle = true}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Place any additional packages needed here.  Only include packages
%% which are essential, to avoid problems later.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{graphicx,bm,lineno,mathpazo}
\usepackage{amssymb,amsfonts,amsmath,mathtools}
\usepackage{algorithm,algpseudocode,booktabs,dcolumn}
\captionsetup{labelfont=bf}
\usepackage[input-decimal-markers={.},separate-uncertainty=true,table-auto-round,scientific-notation = false,zero-decimal-to-integer=true]{siunitx}
\DeclareMathOperator*{\tr}{tr}
\DeclareMathOperator*{\argmax}{arg\,max}
\usepackage[caption=false,font=normalsize,position=top]{subfig}
\usepackage[kerning,spacing,tracking]{microtype}
\microtypecontext{spacing=nonfrench}
\usepackage{indentfirst}
\overfullrule=5pt
\usepackage[normalem]{ulem}
\usepackage{color}
\usepackage{empheq}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% New command
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newcommand{\boxedeq}[2]{\begin{empheq}[box={\fboxsep=6pt\fbox}]{align}\label{#1}#2\end{empheq}}

\title{From $\bm{H}^0$ to $\bm{H}$}
\author{Yi-Tsao Chen}
\date{December 2019}

\begin{document}

\maketitle

\section{Determination of Eigenbasis from $\bm{H}^0$ and $\bm{H}$}
\begin{align}
\label{eq:FPH0}
\frac{\partial \rho(x,t)}{\partial t} &= -\bm{H}^0 \rho(x,t) \\
\bm{H}^0 &= -D\nabla^2  + V_{\rm eff}(x)
\end{align}

We used spectral finite element method(SEM) to find eigenbasis of $\bm{H}^0$, $\{| \psi_i^0 \rangle\}$, and the corresponding eigenvalues $\lambda^0_i$
\begin{align}
\bm{H}^0 | \psi_i^0 \rangle  = \lambda^0_i| \psi_i^0 \rangle
\end{align}
After that, the eigenbasis of $\bm{H}^0$ is then used to solve the eigenvalue problem that involves the dark operator required for the smFRET likelihood calculation:
\begin{align}
\label{eq:PsiBasis}
\left ( \bm{H}^0 + \bm{H}^I \right ) | \psi_i \rangle = \lambda_i | \psi_i \rangle
\end{align}
Here, $\bm{H}^I=I(x)$, and the new eigenvector is then constructed as a linear combination of $|\psi^0_i \rangle$
\begin{align}
| \psi_i \rangle = \sum_j c_{ij} | \psi_j^0(x) \rangle
\end{align}
so the question now is to find $c_{ij}$ and get a correct linear combination.
\clearpage

\section{Find $c_{ij}$}
First, assume $i=1~\textrm{to}~n$,
\begin{equation}
\label{psi0psi}
\begin{split}
| \psi_1 \rangle = c_{11}| \psi_1^0 \rangle + c_{12}| \psi_2^0 \rangle +  c_{13}| \psi_3^0 \rangle+\cdots + c_{1n}| \psi_n^0 \rangle\\
| \psi_2 \rangle = c_{21}| \psi_1^0 \rangle + c_{22}| \psi_2^0 \rangle +  c_{23}| \psi_3^0 \rangle+\cdots + c_{2n}| \psi_n^0 \rangle\\
| \psi_3 \rangle = c_{31}| \psi_1^0 \rangle + c_{32}| \psi_2^0 \rangle +  c_{33}| \psi_3^0 \rangle+\cdots + c_{3n}| \psi_n^0 \rangle\\
\vdots \\
| \psi_n \rangle = c_{n1}| \psi_1^0 \rangle + c_{n2}| \psi_2^0 \rangle +  c_{n3}| \psi_3^0 \rangle+\cdots + c_{nn}| \psi_n^0 \rangle
\end{split}
\end{equation}
According to equation (\ref{eq:PsiBasis}), and let $i=1$
\begin{equation}
\label{eg1}
\bm{H} | \psi_1 \rangle = \lambda_1 | \psi_1 \rangle
\end{equation}
Here $\bm{H}=\bm{H}^0 + \bm{H}^I$. Then, substitute equation (\ref{psi0psi}) into equation (\ref{eg1})
\begin{equation}
\label{egextend}
\begin{split}
c_{11}\bm{H}| \psi_1^0 \rangle + c_{12}\bm{H}| \psi_2^0\rangle +  c_{13}\bm{H}| \psi_3^0\rangle + \cdots +  c_{1n}\bm{H}| \psi_n^0\rangle =\\ \lambda_1 \left( c_{11}| \psi_1^0 \rangle + c_{12}| \psi_2^0 \rangle +  c_{13    }| \psi_3^0 \rangle+\cdots + c_{1n}| \psi_n^0 \rangle\right)
\end{split}
\end{equation}
And we multiply $\langle\psi_1^0|$ on the left-hand side of equation (\ref{eg1})
\begin{equation}
\label{eg1}
\langle\psi_1^0|\bm{H} | \psi_1 \rangle = \langle\psi_1^0|\lambda_1 | \psi_1 \rangle
\end{equation}
Equation (\ref{egextend}) becomes
\begin{equation}
\begin{split}
c_{11}\langle\psi_1^0|\bm{H}| \psi_1^0 \rangle + c_{12}\langle\psi_1^0|\bm{H}| \psi_2^0\rangle +  c_{13}\langle\psi_1^0|\bm{H}| \psi_3^0\rangle + \cdots +  c_{1n}\langle\psi_1^0|\bm{H}| \psi_n^0\rangle =\\ \lambda_1 \left( c_{11}\langle\psi_1^0| \psi_1^0 \rangle + c_{12}\langle\psi_1^0| \psi_2^0 \rangle +  c_{13}\langle\psi_1^0| \psi_3^0 \rangle+\cdots + c_{1n}\langle\psi_1^0| \psi_n^0 \rangle\right)
\end{split}
\end{equation}
The eigenbasis $\{|\psi_i^0 \rangle\}$ is orthonormal, so
\begin{equation}
\label{eg1lastform}
\begin{split}
c_{11}\langle\psi_1^0|\bm{H}| \psi_1^0 \rangle + c_{12}\langle\psi_1^0|\bm{H}| \psi_2^0\rangle +  c_{13}\langle\psi_1^0|\bm{H}| \psi_3^0\rangle + \cdots +  c_{1n}\langle\psi_1^0|\bm{H}| \psi_n^0\rangle =\lambda_1 c_{11}
\end{split}
\end{equation}
We also can generate more equations by
\begin{equation}
\label{eg1gen}
\begin{split}
\langle\psi_2^0|\bm{H} | \psi_1 \rangle &= \langle\psi_2^0|\lambda_1 | \psi_1 \rangle\\
\langle\psi_3^0|\bm{H} | \psi_1 \rangle &= \langle\psi_3^0|\lambda_1 | \psi_1 \rangle\\
&\vdots \\
\langle\psi_n^0|\bm{H} | \psi_1 \rangle &= \langle\psi_n^0|\lambda_1 | \psi_1 \rangle
\end{split}
\end{equation}
By collecting equation (\ref{eg1}) and (\ref{eg1gen}), and express as equation (\ref{eg1lastform}), we can get the following matrix expression
\begin{equation}
\begin{bmatrix}
\langle\psi_1^0|\bm{H}| \psi_1^0 \rangle & \langle\psi_1^0|\bm{H}| \psi_2^0 \rangle &\langle\psi_1^0|\bm{H}| \psi_3^0 \rangle & \cdots & \langle\psi_1^0|\bm{H}| \psi_n^0 \rangle \\
\langle\psi_2^0|\bm{H}| \psi_1^0 \rangle & \langle\psi_2^0|\bm{H}| \psi_2^0 \rangle &\langle\psi_2^0|\bm{H}| \psi_3^0 \rangle & \cdots & \langle\psi_2^0|\bm{H}| \psi_n^0 \rangle \\
\vdots \\
\langle\psi_n^0|\bm{H}| \psi_1^0 \rangle & \langle\psi_n^0|\bm{H}| \psi_2^0 \rangle &\langle\psi_n^0|\bm{H}| \psi_3^0 \rangle & \cdots & \langle\psi_n^0|\bm{H}| \psi_n^0 \rangle
\end{bmatrix}
\begin{bmatrix}
c_{11} \\ c_{12} \\ \vdots \\ c_{1n}
\end{bmatrix}
= \lambda_1
\begin{bmatrix}
c_{11} \\ c_{12} \\ \vdots \\ c_{1n}
\end{bmatrix}
\end{equation}
Define
\begin{equation}
K=
\begin{bmatrix}
\langle\psi_1^0|\bm{H}| \psi_1^0 \rangle & \langle\psi_1^0|\bm{H}| \psi_2^0 \rangle &\langle\psi_1^0|\bm{H}| \psi_3^0 \rangle & \cdots & \langle\psi_1^0|\bm{H}| \psi_n^0 \rangle \\
\langle\psi_2^0|\bm{H}| \psi_1^0 \rangle & \langle\psi_2^0|\bm{H}| \psi_2^0 \rangle &\langle\psi_2^0|\bm{H}| \psi_3^0 \rangle & \cdots & \langle\psi_2^0|\bm{H}| \psi_n^0 \rangle \\
\vdots \\
\langle\psi_n^0|\bm{H}| \psi_1^0 \rangle & \langle\psi_n^0|\bm{H}| \psi_2^0 \rangle &\langle\psi_n^0|\bm{H}| \psi_3^0 \rangle & \cdots & \langle\psi_n^0|\bm{H}| \psi_n^0 \rangle
\end{bmatrix}
\end{equation}
To generalize
\begin{equation}
K\textbf{c}_{i} =
\begin{bmatrix}
\langle\psi_1^0|\bm{H}| \psi_1^0 \rangle & \langle\psi_1^0|\bm{H}| \psi_2^0 \rangle &\langle\psi_1^0|\bm{H}| \psi_3^0 \rangle & \cdots & \langle\psi_1^0|\bm{H}| \psi_n^0 \rangle \\
\langle\psi_2^0|\bm{H}| \psi_1^0 \rangle & \langle\psi_2^0|\bm{H}| \psi_2^0 \rangle &\langle\psi_2^0|\bm{H}| \psi_3^0 \rangle & \cdots & \langle\psi_2^0|\bm{H}| \psi_n^0 \rangle \\
\vdots \\
\langle\psi_n^0|\bm{H}| \psi_1^0 \rangle & \langle\psi_n^0|\bm{H}| \psi_2^0 \rangle &\langle\psi_n^0|\bm{H}| \psi_3^0 \rangle & \cdots & \langle\psi_n^0|\bm{H}| \psi_n^0 \rangle
\end{bmatrix}
\begin{bmatrix}
c_{i1} \\ c_{i2} \\ \vdots \\ c_{in}
\end{bmatrix}
= \lambda_{i}
\begin{bmatrix}
c_{i1} \\ c_{i2} \\ \vdots \\ c_{in}
\end{bmatrix}
\end{equation}
Simply written as
\begin{equation}
\label{eigprob}
K\textbf{c}_{i} = \lambda_{i}\textbf{c}_{i}
\end{equation}
Therefore, to find $c_{ij}$, which can assemble $\{| \psi_i\rangle\}$ by $\{| \psi_i^0\rangle\}$, turns out to be a eigenvalue problem as shown in equation (\ref{eigprob}).
At last, the matrix elements of $K$
\begin{align}
K_{ij} = \langle \psi_i^0 | \bm{H}^0 + \bm{H}^I | \psi^0_j(x) \rangle = \lambda_i^0 \delta_{ij} + \langle \psi_i^0 | \bm{H}^I |\psi^0_j \rangle
\end{align}

\end{document}


% $Id$

\subsubsection{ESMF\_MESHELEMTYPE}
\label{const:meshelemtype}

 {\sf DESCRIPTION:\\}
 An ESMF Mesh can be constructed from a combination of different elements. The type of elements that can
be used in a Mesh depends on the Mesh's parameteric dimension, which is set during Mesh creation. The
following are the valid Mesh element types for each valid Mesh parametric dimension (2D or 3D) .

\medskip

\begin{verbatim}

                     3                          4 ---------- 3
                    / \                         |            |
                   /   \                        |            |
                  /     \                       |            |
                 /       \                      |            |
                /         \                     |            |
               1 --------- 2                    1 ---------- 2

           ESMF_MESHELEMTYPE_TRI            ESMF_MESHELEMTYPE_QUAD

     2D element types (numbers are the order for elementConn during Mesh create)

\end{verbatim}

For a Mesh with parametric dimension of 2 ESMF supports two native element types (illustrated above),
but also supports polygons with more sides. Internally these polygons are represented as a set of
triangles, but to the user should behave like other elements.
To specify the non-native polygons in the {\tt elementType} argument use the number of corners
of the polygon (e.g. for a pentagon use 5). The connectivity for a polygon should be specified in counterclockwise order.
The following table summarizes this information:

\smallskip

\begin{tabular}{|l|c|l|}
\hline
Element Type &  Number of Nodes  & Description \\
\hline
ESMF\_MESHELEMTYPE\_TRI  & 3 & A triangle \\
ESMF\_MESHELEMTYPE\_QUAD & 4 & A quadrilateral (e.g. a rectangle) \\
 N & N & An N-gon  (e.g. if N=5 a pentagon) \\
\hline
\end{tabular}

\medskip
\medskip

\begin{verbatim}

                 3                               8---------------7
                /|\                             /|              /|
               / | \                           / |             / |
              /  |  \                         /  |            /  |
             /   |   \                       /   |           /   |
            /    |    \                     5---------------6    |
           4-----|-----2                    |    |          |    |
            \    |    /                     |    4----------|----3
             \   |   /                      |   /           |   /
              \  |  /                       |  /            |  /
               \ | /                        | /             | /
                \|/                         |/              |/
                 1                          1---------------2

       ESMF_MESHELEMTYPE_TETRA             ESMF_MESHELEMTYPE_HEX

  3D element types (numbers are the order for elementConn during Mesh create)

\end{verbatim}

For a Mesh with parametric dimension of 3 the valid element types (illustrated above) are:

\smallskip

\begin{tabular}{|l|c|l|}
\hline
Element Type & Number of Nodes & Description \\
\hline
ESMF\_MESHELEMTYPE\_TETRA & 4 & A tetrahedron (NOT VALID IN BILINEAR OR PATCH REGRID)\\
ESMF\_MESHELEMTYPE\_HEX  & 8 & A hexahedron (e.g. a cube) \\
\hline
\end{tabular}


%In this section we provide details of the three datasets that we evaluate in this paper, as well as the methodology used.
%\subsection{Dataset description}

%\subsubsection{Twitter example (Detection of natural disasters)}
In this section, we first describe the dataset used for our experiments, then we describe the baselines we used for comparison.
\subsection{Dataset description}
The scenario we have chosen to evaluate our algorithms is related to the detection of natural disasters discussed in a collection of tweets.  We started with a corpus of approximately 1 billion tweets crawled from the Twitter streaming API during 2013 and 2014 with the following restrictions:
%the Twitter data described in~\cite{Iman2017}:
%, which was curated by four students as follows:
(1) the dataset was restricted to users located within the US, (2) non-English tweets were filtered out, (3) only the tweets related to 12 natural disasters were kept -- tweets related to other natural disasters were removed. These natural disasters are temporally, and geographically disjoint -- a storm, a hurricane,  a drought, two floods, two earthquakes, two tornadoes,  and three blizzards. Finally, (4) false positive tweets (i.e., tweets mentioning natural disasters specific-keywords but are not related to a particular natural disaster) from the dataset were intentionally included, e.g., \textit{\textquotedblleft I'm flooded with questions from followers\textquotedblright{}}. The final dataset contains 39,486 tweets with 5,075 marked as relevant tweets (in this context, a relevant tweet is a tweet related to a natural disaster).

\subsection{Baselines description}
We used the following two baselines for comparison:

\subfour{Optimal solution:} To benchmark the performance of the greedy algorithms we proposed on moderately sized datasets, we use an
exact Mixed Integer Linear Progamming (MILP) optimization-based formulation to maximize EF1. In brief, the formulation of EF1 in Equation \ref{eq:EF1} can be transformed into a fractional MILP formulation with constraints corresponding to each of the cluster attribute selection criteria (space, time, and content).  The parameters of these constraints are then chosen to optimize the EF1 objective.

Because there are no direct solvers for fractional MILPs, we can transform the problem into a pure MILP formulation for which we have efficient and optimal solvers using the Charnes-Cooper method \cite{Charnes1962} and Glover linearization method \cite{Glover1975} with big-M constraints.
%Lastly, we add three constraints to the optimization to select elements through three clustering parameters, i.e., time, position and keywords. %We refer the reader to [ADD ANONYMOUS CITATION?] for more details o this method.

\subfour{K-Means clustering:} In addition, we also propose to use X-Means \cite{Pelleg2000} as a baseline method for clustering. X-Means is a simple extension of K-Means~\cite{kmeans_original} which tries to  automatically determine the number of clusters. Starting with only one cluster, the X-Means wrapper applies after each run of K-Means, making local decisions about which subset of the current centroids should split themselves in order to better fit the data.  In order to provide X-means with spatial, temporal, and content coherence, the distance metric we have used for X-Means is a linear combination of the following: (i) the Euclidean distance of time, (ii) the Euclidean distance of location, and (iii) the cosine distance of the textual content. This distance metric is formally defined as follows:
\begin{equation}
\begin{array}{ll}
d(i,j)= & \alpha\times\textrm{[time distance]} +\\
& \beta\times\textrm{[location distance]}+\\
 & \gamma\times\textrm{[text cosine distance]}
\end{array}
\end{equation}
\noindent where $\alpha$, $\beta$, and $\gamma$ are weights that sum to 1, set all to 1 in the off-line evaluation and set respectively to 0.1, 0.8,  and 0.1 in the user study -- values manually tuned in extensive pilot user trials to maximize cluster quality.  Once clusters are extracted by X-means, we use the EF1 metric to extract the top clusters as required by our interface.

%We assume having an agent monitoring tweets, a topical tweets classifier (e.g., \cite{Iman2017}), and a display showing the locations of the tweets (See Figure \ref{Fig:TwitterData}).  We used the  2.5 TB of Twitter data described in  \cite{Iman2017}, for which we restricted our analysis to the 9M tweets of January 2014.

%\begin{figure}[t]
%\begin{centering}
%\subfigure[Twitter Networks.]{\includegraphics[width=2.8cm]{imgs/twitter_example_3}\label{Fig:TwitterData}}\subfigure[Enron Networks.]{\includegraphics[width=2.8cm]{imgs/enron_net_4}\label{Fig:EnronData}}\subfigure[Reddit Networks.]{\includegraphics[width=2.8cm]{imgs/srforum3}\label{Fig:RedditData}}
%\par\end{centering}
%\caption{Layouts used for the different datasets.}
%\end{figure}


% !TeX spellcheck = en_US
\section{METHOD}
\label{sec:method}

For tackling the proposed problem, a YOLO detector has been used at first. The YOLO network usually relies on Darknet \cite{darknet2018}, an open source neural network framework written in C and CUDA. However, we initially relied on a popular python translation of Darknet, called Darkflow \cite{thtrieu2016}, to set up the learning pipeline. Since Darkflow does not support YOLO v3 yet, we used YOLO v2 as the model to be fine-tuned for an initial testing. Anyway, this approach did not yield any good result, with the network not being able to properly converge. A variety of setups of parameters were tried, but the loss and moving average loss values never decreased under the approximate threshold of 30 (starting from about 130). In particular, the learning rate has ranged from $1 \cdot 10^{-3}$ to $5 \cdot 10^{-7}$ with a step of a 10 factor and fixed decay equal to $5 \cdot 10^{-4}$, batch size has ranged from 1 to 64 with step 8, and both Adam and Sgd (Standard Gradient Descent) optimizers have been tested. Moving to YOLO v3 using an open source Keras implementation \cite{experiencor2018} resulted approximately in the same bad performances with the same setups. The training was performed both using and not using different weights pretrained on COCO dataset, namely those provided by Joseph Redmon via the official website of the YOLO project \cite{yolo2018}. In light of these results, a different approach has been preferred. Instead of using an end-to-end model to detect bounding boxes and translate each character, we pipelined two different models: 1) a model to detect the characters and 2) a model to predict the specific Japanese character from each detected object. In particular, we implemented a CenterNet architecture for detection and a quite simple CNN for classification. The name CenterNet is used because the architecture is taken by the paper \citetitle{Zhou2019-sd} \cite{Zhou2019-sd} which uses this same name.

\subsection{Character detection using CenterNet}
\label{ssec:charactercenternet}

In the referenced paper \cite{Zhou2019-sd}, the authors propose a deep neural network for object detection and classification tasks called CenterNet. The difference with other object detection networks is that CenterNet predicts the position of a variable number of keypoints in the image, leaving the task of inferring bounding boxes to a post-processing operation. The aim of this network is hereby described reporting some information and notations from the official paper. Let $I \in R^{W \times H \times 3}$ be an image of height $H$, width $W$ and 3 channels. CenterNet will predict a heatmap  $\widehat{Y} \in [0,1]^{\frac{W}{R} \times \frac{H}{R} \times C}$, where $R$ is the output stride of the network and $C$ is the number of keypoint types, which is the number of categories to be predicted. $R$, the output stride, is the scale factor of the network (in our case, since we worked with squared images, measured as $\frac{input\_width}{output\_width}$). With respect to the predicted heatmap:

\begin{itemize}
	\item A prediction $\widehat{Y}_{x,y,c}=1$ means that a keypoint of type $c \in C$ has been detected at position $(x\cdot R, y\cdot R)$ of the input image.
	\item A prediction $\widehat{Y}_{x,y,c}=0$, corresponds to a background point (i.e. non-keypoint).
\end{itemize}

An example of accurate detection is shown in figure \ref{fig:detection-example}.

\begin{figure}
	\centering
	\includegraphics[width=0.9\columnwidth]{detection/std.png}
	\caption{Example of detection and corresponding heatmap on a single image.}
	\label{fig:detection-example}
\end{figure}

\subsubsection{Implementation}
\label{sssec:implementationdet}

To implement the detection model we followed both the implementation described in the paper and an existing implementation created by a Kaggle user specifically for the purpose of the competition (see \cite{k-mat2019}).\\ We call this model the \textit{detector model}, since its purpose is detecting the center of each character to be recognized in the image. Note that in our implementation there is just one type of keypoint that should be recognized. However, as reported in the paper, CenterNet allows to use more keypoint types (e.g. it has been used in human pose recognition task, where each keypoint type represented a different human joint). Theoretically it may be possible to use this same architecture also for character classification, using a different keypoint type for each Kuzushiji character. However, the number of classes (in subset $C$) would be very high (i.e. $>$ 4200) and consequently the output of the model would be huge and impractical to manipulate, additionally resulting in a higher training and predict time. Thus, we preferred the two steps approach, using different models for detection and classification.

\subsubsubsection{Description of the input samples}
\label{ssssec:inputsamples}

Firstly, each training image is randomly cropped and resized to the input dimension, which was set to $512 \times 512$. Then, the output of the training sample is generated using the annotations created during the pre-processing step. For each keypoint $p$, a downsampled point is computed as $\hat{p}=\left\lfloor \dfrac{p}{R} \right\rfloor$, which represents the point in the output space.\\
The heatmap is generated with a gaussian function applied to each pixel of the image as in equation \ref{eq:gaussian}.

\begin{equation}\label{eq:gaussian}
	Y_{x,y,c}=\exp\left(\dfrac{1}{2} \cdot \left(\frac{x-p_x}{\sigma}\right)^2 \cdot \left(\frac{y-p_y}{\gamma}\right)^2 \right)
\end{equation}

In the equation we set $\sigma = \dfrac{w}{10}$ and $\gamma = \dfrac{h}{10}$, where $w$ and $h$ are the image width and height respectively.\\
For the purpose of our project, the heatmap slightly differs from the one described in the original paper. The implementation we opted for has been used in the aforementioned Kaggle notebook, and allows the creation of heatmaps such that the keypoint heat can form elliptic shapes and not only perfect circles. In addition, to recover from the discretization error caused by the output stride, we also add a local pixel offset $\widehat{O} \in R^{\frac{W}{R} \times \frac{H}{R} \times 2}$ (as in the original paper). For example, if a keypoint location in the image is $(12.32, 122.18)$ the offset will be of $(0.32, 0.18)$ with the values divided by the output stride.
Finally to predict the bounding box for each object we add information about the object size $\widehat{S} \in R^{\frac{W}{R} \times \frac{H}{R} \times 2}$. This size is computed as $(xmax-xmin, ymax-ymin)$, divided by the output size, which in our case is 128. Each training example is then fed to the network. Summing up, the shape of each input image is (512, 512, 3), while the target output $Y_{x,y,c}$ for each example has shape (128, 128, 6), where:

\begin{itemize}
	\item $Y_{x,y,0}$ contains the gaussian heatmap;
	\item $Y_{x,y,2}$ is the width offset;
	\item $Y_{x,y,3}$ is the height offset;
	\item $Y_{x,y,4}$ and $Y_{x,y,5}$ are respectively the width and height estimation for the bounding box.
\end{itemize}

We followed the implementation on Kaggle adding in $Y_{x,y,1}$ the “ground truth map” with 1 in correspondence of centers and 0 for all other points. This map is used to compute the loss function as described in the next paragraph.

\subsubsubsection{Loss function}
\label{ssssec:lossfunctiondet}

We used the loss functions provided in the Kaggle implementation, which follows the description of the paper. The main loss function is computed from three separate losses $L_h$, $L_o$, $L_s$, that measure the error on the heatmap, offset and size respectively. The heatmap loss is defined as (\ref{eq:heatloss}):
\begin{gather}
\begin{aligned}
L_h=-\dfrac{1}{N}\sum_{x,y}
\begin{cases}
(1-\widehat{Y}_{x,y,0})^{\alpha} \\~~~~\cdot \log(\widehat{Y}_{x,y,0} + \epsilon)~~~~~~~~~\text{if}~Y_{x,y,0} = 1 \vspace*{10pt} \\
(1-Y_{x,y,0})^{\beta} \cdot (\widehat{Y}_{x,y,0})^\alpha \\ ~~~~\cdot \log(1-\widehat{Y}_{x,y,0} + \epsilon) ~~~~\text{if}~Y_{x,y,0}=0
\end{cases}
\end{aligned}
\label{eq:heatloss}
\raisetag{37pt}
\end{gather}
where $\alpha=2$ and $\beta=4$ as suggested in the paper, while $\epsilon=10^{-6}$ is a small constant added to prevent the $\log$ argument from becoming $0$. Note that in the original paper the sum was done over all $x$, $y$, $c$, but in our implementation we want to predict one keypoint type only, so in the definition we replaced the $c$ variable with $0$.\\
Offset and size loss are defined in equations \ref{eq:offloss} and \ref{eq:sizeloss} as:
\begin{equation}\label{eq:offloss}
L_o = \dfrac{1}{N}\sum_{k=1}^{2}\sum_{x,y}\left|Y_{x,y,k+1} - \widehat{Y}_{x,y,k}\right|
\end{equation}
\begin{equation}\label{eq:sizeloss}
	L_s = \dfrac{1}{N}\sum_{k=3}^{4}\sum_{x,y}\left|Y_{x,y,k+1} - \widehat{Y}_{x,y,k}\right|
\end{equation}
In both equations, the first sum refers to the fact that both height and width offset and size values are considered in the computation.

\noindent Hence, the final loss is obtained as:
\begin{equation}
	L=L_h+ \lambda_1 \cdot L_o+ \lambda_2 \cdot L_s
\end{equation}
with $\lambda_1=1$ and $\lambda_2=5$, the same values used in the paper.

\subsubsubsection{Network architecture}
\label{ssssec:networkarchitecturedet}

The network follows the encoder-decoder architecture similar to the one used in U-Net \cite{Ronneberger2015-tf}. As an encoder we used both ResNet34 \cite{deep-residual} and ResNet50, removing the last average pooling and fully connected layers. On top of the encoder we added the decoder, as a stack of three up-convolutional layers. This architecture is the simplest among those described in the CenterNet research paper. In addition, following the U-Net architecture we added some cross connections from the encoder to the decoder.\\
Note that in the following description we will not specify the 2D or 3D nature of image transformations (like convolution), since all the operations we performed are bidimensional.\\
The full architecture diagram using ResNet34 is shown in figure \ref{fig:detector}. For ResNet50 we used the pre-trained model available in Keras, initialized with the weights used for ImageNet classification.\\

\begin{figure}[h]
	\caption{Architecture of the detector model.}
	\centering
	\includegraphics[width=0.45\textwidth]{architectures/detector.png}
	\label{fig:detector}
\end{figure}

\noindent During the encoding phase, the input image of dimension $512 \times 512$ is shrunk through 5 convolutional blocks.\\
The first block uses a convolution of size 7 and stride 2 and a max pooling operation to downsample the image by a factor of 4.\\
The second block is built of 3 standard residual blocks with identity shortcut. Such a block can be visualized in figure \ref{fig:resblock-id}. These blocks do not change the image dimension and they just apply two convolutions to add more feature maps, with the addition block at the end.\\
The third block halves the image size using a residual block with projection shortcut and three residual blocks with identity shortcut. Projection shortcut block, shown in figure \ref{fig:resblock-proj}, does the same operations of identity shortcut, but adds a $1\times 1$ convolution to match the dimensions of the feature maps to be summed up.\\
Fourth and fifth blocks just repeat this steps with a different number of identity shortcut blocks. Note that each residual block double the number of channels in the image. For what concerns the decoder, the encoded input of size $(16, 16, 512)$ is passed to the decoder that uses three up-convolutional blocks to upsample it to the desired output dimension of $(128, 128, 5)$. Here, we made two major changes with respect to the standard CenterNet implementation of the decoder:

\begin{enumerate}
	\item we added cross connections in U-Net fashion, and
	\item we replaced the deformable convolution layer with a simple convolution.
\end{enumerate}

\begin{figure}
	\caption{Residual block with identity shortcut.}
	\centering
	\includegraphics[width=0.35\textwidth]{architectures/resblock_id.png}
	\label{fig:resblock-id}
\end{figure}

\begin{figure}
	\caption{Residual block with projection shortcut.}
	\centering
	\includegraphics[width=0.45\textwidth]{architectures/resblock_proj.png}
	\label{fig:resblock-proj}
\end{figure}

The difference between a standard convolution and deformable convolution is that using the former the receptive field and the sampling locations are fixed all over the top feature map, while using the latter they are adaptively adjusted according to the scale and shape of the objects in deformable convolution. This is implemented by augmenting the standard convolutional grid with a matrix of parameters called offsets, which must be learned in addition to the standard convolution weights \cite{DeformableConv2017}. The authors of the original paper reported an improvement over standard convolution for object detection tasks, in particular in the detection of irregular objects. On the other hand, the characters to be recognised in the Kuzushiji documents are arranged in adjacent columns, in a grid-like pattern. For this reason, and considering the additional computational time required to train a deformable convolutional network, we decided to replace them with their standard counterparts. \\

The decoder is composed of three identical blocks, each featuring the following layers:

\begin{itemize}
	\item a convolution to reduce the number of features;
	\item a transposed convolution to upsample the image by a factor of 2;
	\item a concatenate operation that concatenates the output of a block of the encoder with the output of the transposed convolution;
	\item a batch normalization to reduce variances of values;
	\item a leaky ReLu activation.
\end{itemize}

The last block is composed of a $1 \times 1$ convolution to reduce the number of features to 5 and a sigmoid activation. The final output is an array with 3 dimensions shaped as $(128, 128, 5)$. This output is the predicted heatmap $\widehat{Y}$. Each of the five layers in the outer dimension has the same meaning explained above, with the difference that the “ground truth map” is not predicted, since this information is provided inside the heatmap itself. This is the reason why the output has 5 instead of 6 layers for each image.

\subsubsection{Pre-processing}
\label{sssec:preprocessingdet}

The input samples are firstly resized to $512 \times 512 \times 3$ and normalized, as well as random cropped for data augmentation. A maximum crop factor is determined considering the average character size in the page. This is determined separately for width and height as shown in equations \ref{eq:stretchw} and \ref{eq:stretchh}, where $w$ and $h$ are the dimensions of the original image and $char\_area$ is the average bounding box area within the page.

\begin{equation}\label{eq:stretchw}
	c_w=\sqrt{\frac{w \cdot h}{char\_area}}
\end{equation}
\begin{equation}
\label{eq:stretchh}
	c_h=\frac{h}{w}\sqrt{\frac{w \cdot h}{char\_area}}
\end{equation}

This crop procedure allows to train the network on smaller fragment of pages with better recognition rate of small characters. In addition, this step ensures good recognition performances when paired with the post-processing passage described in the next section.\\
\noindent Finally, all output classes, expressed as unicode characters, are mapped to integer values.

\subsubsection{Post-processing: from centers to bounding boxes}
\label{sssec:preprocessingandtraining}

The previously described network predicts the location of the center of each character within the page. From that we infer the coordinates of the bounding box location using the size information in the predicted heatmap ($Y_{x,y,3}$ and $Y_{x,y,4}$) and summing the offsets ($Y_{x,y,1}$ and $Y_{x,y,2}$). To further improve the accuracy of the bounding boxes, some post-processing has been made as well. Firstly, all heatmap elements with score prediction below 0.7 were filtered out, since they probably represent noise. The remaining ones are the predicted centers. The predicted offsets are then added to the coordinates of these centers in the heatmap and the definitive coordinates of the bounding boxes are computed as:
\begin{eqnarray*}
	xmin=x_c - \frac{width}{2}\\
	ymin= y_c - \frac{height}{2}\\
	xmax=x_c + \frac{width}{2}\\
	ymax=y_c + \frac{height}{2}
\end{eqnarray*}
Then, out-of-picture boxes and the ones with an area too different from the average area in the page (too big or too small) are removed. Finally overlap-based non maximum suppression is performed comparing the intersection area between the boxes. Each lower score bounding box sharing more than 40\% of the area of higher score boxes, or having more than 60\% of its area inside other higher score bounding boxes is removed.\\

\subsubsection{Splitting the images}
\label{sssec:tiling}
\noindent Despite the non-maximum-suppression, we noticed that some small characters were not detected by the network. This happens because the feature map is, in some cases, too small with respect to the original image, and may lose some information \cite{Ozge-Unel2019-jl}, leading to lower IoU score with the ground truth bounding boxes.
Our solution was to divide each image into a number $n$ of smaller images (that we call \textit{tiles}) and use the model to predict bounding boxes in each one of the smaller images. A 25\% overlap between the tiles is used to preserve the objects along the tile boundaries and prevent any miss due to image partitioning. An example of partitioned image is shown in figure \ref{fig:tiles4}.

\begin{figure}
	\centering
	\includegraphics[width=0.47\columnwidth]{detection/tiles_4.png}
	\caption{An image divided in 4 tiles with 25\% overlap.}
	\label{fig:tiles4}
\end{figure}

This procedure resulted in more precise bounding box predictions and an higher IoU score on the test set. A comparison can be seen in figure \ref{fig:comparisontiling}. \\
We empirically determined the optimal number of tiles by measuring the IoU score over the same first 100 examples of the test set. Results gave a 0.7731 score using a $2 \times 2$ grid and 0.7850 using a $3 \times 3$ grid.
Since the difference in the IoU score was negligible we chose to use a $2 \times 2$ grid, with the benefit of halving the time required to generate predictions for a single image.\\
It is worth to say that the same model trained without the random crop described in §\ref{sssec:preprocessingdet} did not reach such scores, but rather got an average IoU score of 0.53. That is because the crop procedure enhances the ability of the model to detect small features, particularly useful when predicting over portions of images.

\begin{figure*}[!htb]
	\centering
	\begin{subfigure}{\columnwidth}
		\centering
		\includegraphics[width=0.55\columnwidth]{detection/tiled_prediction.png}
		\caption{Prediction with $2\times2$ tiling.}
		\label{fig:tiledpred}
	\end{subfigure}
	\begin{subfigure}{\columnwidth}
		\centering
		\includegraphics[width=0.56\columnwidth]{detection/standard_prediction.png}
		\caption{Prediction with no tiling.}
		\label{fig:standardpred}
	\end{subfigure}
	\caption{Comparison of prediction on the same page with and without tiling. Image (\ref{fig:tiledpred}) got an IoU score of 0.83, while (\ref{fig:standardpred}) scored 0.59.}
	\label{fig:comparisontiling}
\end{figure*}

\subsection{Classification of characters using a standard CNN}
\label{ssec:classificationcnn}

While the model described in the previous section was used to detect characters, the goal of the classifier is to classify each detected character in one of the 4212 current Japanese characters, hopefully the one corresponding to its translation. The number of classes in our model is the number of different Kuzushiji characters found in the training set. In the official dataset for the competition, a dictionary of 4787 Kuzushiji characters and their translation was provided, but not all of them are present in the training set, while other characters are extremely rare. This may lead to worse classification performances for characters the model has seen fewer times.

\subsubsection{Implementation}
\label{sssec:implementationclass}

\begin{figure}[!hbt]
	\caption{Preactivated residual block.}
	\centering
	\includegraphics[width=0.35\textwidth]{architectures/resblock_pre.png}
	\label{fig:preactresblock}
\end{figure}
\begin{figure}[!hbt]
	\caption{Architecture of the classifier.}
	\centering
	\includegraphics[width=0.35\textwidth]{architectures/classifier.png}
	\label{fig:classifier}
\end{figure}

\subsubsubsection{Network architecture}
\label{ssssec:networkarchitectureclass}

The classifier is a small convolutional model based on the one proposed in the Kaggle implementation. The architecture diagram is shown in figure \ref{fig:classifier}. We use a variant of residual block that apply batch normalization and ReLu activation before the convolution. This block is called preactivated residual block and can be visualized in figure \ref{fig:preactresblock}. The three repeated blocks are followed by an average pooling layer with kernel size equal to the input dimension (called global pooling), which produce a single value for each feature map. This value is then fed into the final dense layer with softmax activation and number of units equal to the number of categories. The architecture diagram is shown in figure \ref{fig:classifier}.

We tested some simple variations using different types of residual blocks, and they worked equally well, so we did not spend much time experimenting with other architectures. An example of accurate classification is shown in figure \ref{fig:classification-example}.
\begin{figure}
	\centering
	\includegraphics[width=0.9\columnwidth]{classification/std.png}
	\caption{Example of classification after detection on a single image.}
	\label{fig:classification-example}
\end{figure}

\subsubsubsection{Loss function}
\label{ssssec:lossfunctionclass}

Since the character classes are codified as integers, we used sparse categorical crossentropy to evaluate the error of the model.

\subsubsection{Pre-processing: data augmentation}
\label{sssec:preprocessingclass}

In order to deal with the main challenging features of cursive handwritten text, some data augmentation has been made on the $32 \times 32$ cropped and resized images of the detected characters. In particular, the objective was emulating the imprecision, variability and deformation inherent the writing of the characters. For this purpose, the dataset has been augmented with copies of the original images slightly altered with respect to brightness, rotation, width and height shifts, and zoom. The augmentation has been performed using the \textit{ImageDataGenerator} utility provided by Keras.


\ourpar{Signatures of Knowledge from SE-SNARKs.}  \markulf{22.04}{wrong place} The work of Groth and Maller
\cite{C:GroMal17} shows how to construct a signature of knowledge for messages in
$\{0, 1\}^*$ from target collision-resistant hash-function and simulation-extractable
SNARKs. This construction can be found in \Cref{sec:SoKconstruction}.
These schemes inherit the same properties as the underlying SNARK schemes and
may rely on a structured reference string, which can be updatable or universal, and they are
succinct.  In the following, our goal will be to show that recent efficient SNARKs are
simulation extractable, and therefore, they are a perfect candidate to build better
signatures of knowledge.

\section{SNARKY Signature Definition}

%\subsection{Signatures of Knowledge}
\markulf{22.04}{Moved here from preliminaries. Intro and maybe a final short section on SoK, definition could go to appendix..}

Signatures of Knowledge (SoKs) \cite{C:ChaLys06} are closely related to
simulation-extractable SNARKs: A signer can generate a valid signature for a message
only if she has a valid witness for the statement.
%
The notion of SoKs mimics digital signatures with strong existential unforgeability:
even if the adversary has seen many signatures on arbitrary messages under arbitrary
statement, it cannot create a new signature (not seen before) without knowing the
witness for the statement in question.
%
To define Signatures of Knowledge we use a recent version \cite{C:GroMal17} that
implicitly considers only compact (succinct) signatures. \COMMENT{We omit the
  description of SoK's algorithms and security properties and defer them to
  \cref{sec:sok}.} Intuitively, a SoK behaves like a non-interactive proof system
that ``binds'' the message being signed to the proof.

%\subsection{Signatures of Knowledge}
\label{sec:sok}
A $\SoK$ for an efficiently decidable binary relation $\REL$ is defined as a tuple of PPT algorithms $\SoK = (\signsetup,  \sign, \allowbreak \verify,  \simsetup, \simsign)$:

\begin{description}
    \item[$\signsetup(1^\secpar, \REL) \rightarrow  \param$:]
	The setup algorithm takes a security parameter $\secpar$ and a binary relation $\REL$
	and returns public parameters $ \param$.  The input $ \param$ is implicit to all subsequent algorithms.

    \item[$\sign(\mesage, \inp, \wit)  \rightarrow \signature$:]
The signing algorithm takes as input a message $ \mesage \in \{0,1\}^{*}$,
a statement $\inp$, and a witness $\wit$.
	Outputs a signature $\signature$.

    \item[$\verify(\mesage, \inp, \signature) \rightarrow 1/ 0$:]
The verification algorithm takes as input
 a message $\mesage$,  a statement $\inp$
 together with a signature $\signature$,
	outputs $1$ if the the signature is valid, $0$ otherwise.

    \item[$\simsetup(\REL) \rightarrow (\param, \td)$:]
    	A simulated setup algorithm which takes as input a relation $\REL$ and returns public parameters $\param$ and a trapdoor $\td$.

   \item[$\simsign(\td, \mesage, \inp) \rightarrow \signature'$:]
   	A simulated signing  that takes as input  a trapdoor $\td$, a message $\mesage$ and a statement $\inp$ and returns a simulated signature $\signature'$.
\end{description}

A SoK scheme should satisfy correctness, extractability and simulatability:

\begin{description}
\item[Perfect Correctness.] This guarantees that a signer with a valid witness can
  always produce a signature that will convince the verifier. For all
  $\secpar\in \NN$, for all efficiently decidable binary relation $\REL$, for all
  $(\inp, \wit) \in \REL$, and for all $ \mesage \in \{0,1\}^{*}$:
   \[
  \condprob{
	  \begin{matrix}
~\verify(\mesage, \inp, \signature) = 1
	 \end{matrix}
}{
	  \begin{matrix}
~(\param, \td) \gets \signsetup(1^\secpar, \REL)\\
~ \signature \gets  \sign(\mesage, \inp, \wit)
 \end{matrix} }  =1.
\]

%

\markulf{22.04}{Because of the trapdoor these definitions are not very suitable for us.}
\item[Simulation Extractability:] This guarantees that an adversary is not able to issue a new signature
unless it knows a witness. This should hold even if the adversary gets to see signatures on
arbitrary messages under arbitrary statements. We model this notion in a strong sense, by
letting the adversary see simulated signatures for arbitrary messages and statements, which
potentially include false statements. Even under this strong attack model, we require that
whenever the adversary outputs a valid signature not queried before, it is possible to extract a
witness for the signature. More formally,  for any PPT adversary $\adv$ there exists a PPT extractor $\ext_\adv$ such that:
   \[
  \condprob{
	  \begin{matrix}
~ \verify(\mesage, \inp, \signature) = 1   \\
\land (\inp, \wit) \notin \REL \\
\land (\mesage, \inp, \signature) \notin Q_{\simsign_\td}
	 \end{matrix}
}{
	  \begin{matrix}
~(\param, \td) \gets \simsetup(1^\secpar, \REL),\\
~ (\mesage, \inp, \signature) \gets \adv^{\simsign_\td} (\param; r),\\
\wit \gets \ext_\adv(\param, Q_{\simsign_\td}, r)
 \end{matrix}
} \leq \negl
\]
where the adversary has access to simulated signatures via the oracle
$\simsign_\td(\inp_i, \mesage_i) \gets \simsign(\td, \inp_i, \mesage_i)$ and the
extractor $\ext_\adv$ takes as input public parameters $\param$, list
$Q_{\simsign_\td}$ of all queries made by $\adv$, and $\adv$'s randomness $r$.

%\vspace{4pt}
%
\item[Perfect Simulatability:] The verifier should not learn anything about the
  witness from the signature.  The secrecy of the witness is modelled by the ability
  to simulate signatures without the witness. More precisely, we say the signatures
  of knowledge are simulatable if there is a simulator that can create public
  parameters together with an associated trapdoor that enables producing signatures
  without a witness that are indistinguishable from real ones.  More formally, for
  any number of queried $\mesage_i\in \{0,1\}^{*}$ and $(\inp_i, \wit_i) \in \REL$:
	\begin{equation}
	\begin{split}
\left|	\condprob{
  \begin{matrix}
		\adv(\param) = 1
  \end{matrix}
		}{
		\begin{matrix}
      \param \gets  \signsetup(1^\secpar, \REL),\\
      ~\signature_i \gets \sign(\mesage_i, \inp_i, \wit_i)
		\end{matrix}
		}\right. \\
	 - \left.\condprob{
		\begin{matrix}
		\adv (\param) = 1
		\end{matrix}
}{
		\begin{matrix}
      ~(\param,\td) \gets  \simsetup(1^\secpar, \REL), \\
      ~\signature_i \gets \simsign(\td,\mesage_i, \inp_i)
		\end{matrix}} \right| \leq \negl.
\end{split}
\end{equation}
\end{description}

\section{SNARKY Signatures Construction}\label{sec:SoKconstruction}

We will recall the construction of
a signature of knowledge scheme for messages in $\{0, 1\}^*$
from an SE-SNARK as shown in \cite{C:GroMal17}.

Consider a target collision-resistant hash function also known as universal one-way hash function $H:\{0, 1\}^{\ell_k} \times \{0, 1\}^* \to \{0, 1\}^{\ell_h}$ with $\ell_K, \ell_h$ polynomials in the security parameter $\secpar$.

The target collision-resistance of $H$ requires that for any $\ppt$ adversary $\adv$ :

	\[  \condprob{
	m_0 \gets \adv(1^\secpar); K \gets \{0, 1\}^{\ell_K}; m_1 \gets \adv(K)
}{
	m_0 \neq m_1 ~\land  ~H_K(m_0)=H_K(m_1)
} \leq \negl.
	\]

For any given relation $\REL'$, we consider the  following relation $\REL$:
 	\[\REL= \{((K,h,\inp),\wit) : K\in  \{0, 1\}^{\ell_K} ~\land ~h \in  \{0, 1\}^{\ell_h} ~\land ~(\inp, \wit) \in \REL'\}.
 	\]

Let $\RELGEN(\secparam)$ be the relation generator that runs $\REL' \gets \RELGEN'(\secparam)$ and returns $\REL$ defined as above.
Let $H$ be a target collision-resistant hash function and let $\ps = (\SRScer, \prover, \verifier, \simulator)$ be a SE-NIZK argument for $\RELGEN$.
We build a signature of knowledge  $\SoK$ for $\RELGEN'$ that works as follows:

\begin{description}
    \item[$\signsetup(1^\secpar, \REL') \rightarrow  \param$:]
 Run $(\srs,\rho) \leftarrow \kgen(\REL)$.
 Return $ \param = \srs$.

    \item[$\sign(\mesage, \inp, \wit)  \rightarrow \signature$:]
Sample $K \gets \{0, 1\}^{\ell_K}$.  Compute
$\zkproof \gets  \prover(\srs, (K, H_K(\mesage), \inp), \wit) $.
Return $\signature = (K, \zkproof)$.

    \item[$\verify(\mesage, \inp, \signature) \rightarrow b$:]
Parse $\signature = (K, \zkproof)$.
Return $b \gets \verifier(\srs, (K, H_K(\mesage), \inp), \zkproof)$.

    \item[$\simsetup(\REL') \rightarrow (\param, \td)$:]
  Run   $(\srs,\rho) \leftarrow \kgen(\REL)$.
 Return $( \param = \srs,  \td = \rho)$.\hamid{29.4}{Not clear how the correctness proof $\rho$ is used as a trapdoor?!}

   \item[$\simsign(\td, \mesage, \inp) \rightarrow \signature'$:]
   Sample $K \gets \{0, 1\}^{\ell_K}$.  Compute
$\zkproof \gets \simulator(\srs, \rho, (K, H_K(\mesage), \inp)) $.
Return $\signature' = (K, \zkproof)$.

\end{description}

\begin{theorem}[Signature of Knowledge]
  \label{thm:SoK}
If $H:\{0, 1\}^{\ell_k} \times \{0, 1\}^* \to \{0, 1\}^{\ell_h}$ is a target collision-resistant hash function
and $\ps = (\SRScer, \prover, \verifier, \simulator)$ be is a SE-NIZK for $\RELGEN(\secparam)$, then the scheme
 $\SoK = (\signsetup,  \sign, \allowbreak \verify,  \simsetup, \simsign)$ above is a signature of knowledge for $\RELGEN'(\secparam)$ for  the message space $\{0, 1\}^*$.
\end{theorem}
 The proof can be found in
 \cite{C:GroMal17}.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End: