\vssub
\subsection{~Output parameters} \label{sub:outpars}
\vssub

The wave model can output parameters on the geographical grid of the model that can be integrated parameters or
parameters as a function of frequency, with one grid for each frequency. All parameters that are a function of frequency (e.g.  \textbf{EF} or \textbf{USF}) require the setting
of specific namelist parameters in the {\F OUTS} namelist defined in ww3\_grid.inp or  ww3\_grid.nml . This is to reduce the memory use
if these parameters are not needed.

Below, a brief definition of output field parameters is provided. A table
with definitions may be found in the sample {\code ww3\_shel.inp} file,
in \para\ref{sec:ww3shel}. That input file also provides a list of flags
indicating if output parameters are available in different field
output file types (ASCII, grib, igrads, NetCDF).
For any details on how these parameters are computed, the user may read the code of the  {\code w3iogo} routine, in the {\code w3iogomd.ftn} module.

Selection of field outputs in  {\code ww3\_shel.inp} is most easily performed by providing a list of the
requested parameters, for example, {\textbf HS DIR SPR}  will request the calculation of significant wave height, mean direction and directional spread. These will thus be stored in the {\code out\_grd.XX} file and can be post-processed, for example in NetCDF using   {\code ww3\_ouf}. Examples are given in \para\ref{sec:ww3multi} and
\para\ref{sec:ww3ounf}. The names for these namelists are the bold names below, for
example \textbf{HS}.

All parameters listed below are available in ASCII and NetCDF output files. If selected output file types are grads or grib, some parameters may not be available. Availability (or not) is identified
in the first two columns in the field output parameter table within the example input file in
\para\ref{sec:ww3shel}. That table also identifies, for all parameters,
the internal WAVEWATCH III code tags, the output tags (names used is ASCII
file extensions, NetCDF variable names and namelist-based selection (see
also \para\ref{sec:ww3ounf}), and the long parameter name/definition.

Finally we note that in all definitions the frequency is the \emph{relative} frequency. Thus, in the presence of currents, these can only be compared to drifting measurement data or data obtained in wavenumber and converted to frequency. Comparison to fixed instrument data requires the use of the full spectrum and proper conversion to the fixed reference frame.

\begin{list}{\Roman{outgrps})\hfill}
            {\usecounter{outgrps} \leftmargin 7mm \labelwidth 5mm
             \rightmargin 0mm \itemsep \baselineskip \parsep 0mm}

\item {Forcing fields}

\begin{list}{\arabic{outpars})\hfill}
            {\usecounter{outpars} \leftmargin 8mm \labelwidth 7mm
             \rightmargin 0mm \itemsep 0mm \parsep 0mm}
\item \textbf{DPT} The mean water depth (m). This includes varying water levels.
\item \textbf{CUR} The mean current velocity (vector, m/s).
\item \textbf{WND} The mean wind speed (vector, m/s). This wind speed is always the
      speed as input to the model, i.e., is not corrected for the current
      speed.
\item \textbf{AST} The air-sea temperature difference ($^\circ$C).
\item \textbf{WLV} Water level.
\item \textbf{ICE} Ice concentration.
\item \textbf{IBG} Wave attenuation due to icebergs: this parameter is the inverse of the
  e-folding scale associated to the loss of wave energy in a field of small
  icebergs \citep{art:Aea11}.
\item \textbf{D50} Sediment median grain size ($D_{50}$).
\item \textbf{IC1} Ice thickness.
\item \textbf{IC5} Maximum ice flow diameter, $D_{\max}$.
\end{list}

\item{Standard mean wave parameters}

\begin{list}{\arabic{outpars})\hfill}
            {\usecounter{outpars} \leftmargin 8mm \labelwidth 7mm
             \rightmargin 0mm \itemsep 0mm \parsep 0mm}
\item \textbf{HS} Significant wave height (m) [see Eq.~(\ref{eq:etot})]
      \begin{equation} H_s = 4 \sqrt{E} \: . \label{eq:Hs} \end{equation}
\item \textbf{LM} Mean wave length (m) [see Eq.~(\ref{eq:zbar})]
      \begin{equation} L_m = 2\pi \overline{k^{-1}}
      \: . \label{eq:Lm} \end{equation}
\item \textbf{T02} Mean wave period (s) [see Eq.~(\ref{eq:zbar})]
      \begin{equation} T_{m02} = 2\pi /\sqrt{\overline{\sigma^{2}}}
      \: . \label{eq:Tm02} \end{equation}
\item \textbf{T0M1} Mean wave period (s) [see Eq.~(\ref{eq:zbar})]
      \begin{equation} T_{m0,-1} = 2\pi \overline{\sigma^{-1}}
      \: . \label{eq:Tm0m1} \end{equation}
\item \textbf{T01} Mean wave period (s) [see Eq.~(\ref{eq:zbar})]
      \begin{equation} T_{m0,1} = 2\pi /\overline{\sigma}
      \: . \label{eq:Tm} \end{equation}
\item \textbf{FP} Peak frequency (Hz), calculated from the one-dimensional frequency
      spectrum using a parabolic fit around the discrete peak.
\item \textbf{DIR} Mean wave direction (degr., meteorological convention)
      \begin{equation} \theta_m = \mbox{atan} \left ( \frac{b}{a} \right )
      \: , \label{eq:theta_m} \end{equation} \begin{equation}
      a = \int_0^{2\pi} \int_0^\infty \cos(\theta) F(\sigma,\theta) \:
      d\sigma \: d\theta \: , \end{equation} \begin{equation}
      b = \int_0^{2\pi} \int_0^\infty \sin(\theta) F(\sigma,\theta) \:
      d\sigma \: d\theta \: . \end{equation}
\item \textbf{SPR} Mean directional spread \citep[degr.;][]{art:KVH88}
      \begin{equation} \sigma_\theta = \left [ 2 \left \{ 1 - \left (
      \frac{a^2+b^2}{E^2} \right )^{1/2} \right \} \right ]^{1/2}
      \: , \label{eq:sig_th} \end{equation}
\item \textbf{DP} Peak direction (degr.), defined like the mean direction, using the
      frequency/wavenumber bin containing of the spectrum $F(k)$ that
      contains the peak frequency only.
\item \textbf{HIG} Infragravity height.
\item \textbf{MXE}   Max surface elev (Space-time extreme, STE)
\item \textbf{MXES}  St Dev of max surface elev (STE)
\item \textbf{MXH}  Max wave height (STE)
\item \textbf{MXHC}  Max wave height from crest (STE)
\item \textbf{SDMH}  St Dev of MXC (STE)
\item \textbf{SDMHC} St Dev of MXHC (STE)
\item \textbf{WBT}   Dominant wave breaking probability $b_T$ (\ref{eq:bt})
\item \textbf{TP} Peak wave period, derived from reciprocal of peak freq
(\textbf{FP}).
\end{list}

\item{Spectral parameters (first 5 moments and wavenumbers).

All these parameters are a function
of frequency and thus these are three-dimensional arrays. Because of the large memory use, the
computation of these parameters requires the activation of switches in the {\F OUTS} namelist.}

\begin{list}{\arabic{outpars})\hfill}
            {\usecounter{outpars} \leftmargin 8mm \labelwidth 7mm
             \rightmargin 0mm \itemsep 0mm \parsep 0mm}

\item \textbf{EF} Wave frequency spectrum (m$^2$/Hz)
      \begin{equation} E(f) = 2 \pi  \int F(\sigma,\theta) \mathrm{d} \theta
      \: . \label{eq:Ef} \end{equation}
\item \textbf{TH1M} Mean direction for each frequency  \citep[degr.;][]{art:KVH88}
       \begin{equation} \theta_1 (f)= \mbox{atan} \left ( \frac{b_1(f)}{a_1(f)} \right )
      \: , \label{eq:theta_b} \end{equation} \begin{equation}
      a_1(f) = 2 \pi \int_0^{2\pi} \int_0^\infty \cos(\theta) F(\sigma,\theta) \:
       \mathrm{d}\theta \: , \end{equation} \begin{equation}
      b_1(f) = 2 \pi \int_0^{2\pi} \int_0^\infty \sin(\theta) F(\sigma,\theta) \:
       \mathrm{d}\theta \: . \end{equation}
\item \textbf{STH1M} First directional spread for each frequency  (degr.; )
      \begin{equation} \sigma_1(f) = \left [ 2 \left \{ 1 - \left (
      \frac{a_1(f)^2+b_1(f)^2}{E(f)^2} \right )^{1/2} \right \} \right ]^{1/2}
      \: , \label{eq:sig_th1} \end{equation}
\item \textbf{TH2M} Mean direction from $a_2$ and $b_2$ (degr.)
       \begin{equation} \theta_2 (f)= \mbox{atan} \left ( \frac{b_2(f)}{a_2(f)} \right )
      \: , \label{eq:theta_2} \end{equation} \begin{equation}
      a_2(f) = 2 \pi \int_0^{2\pi} \int_0^\infty \cos(2 \theta) F(\sigma,\theta) \:
       \mathrm{d}\theta \: , \end{equation} \begin{equation}
      b_2(f) = 2 \pi \int_0^{2\pi} \int_0^\infty \sin(2 \theta) F(\sigma,\theta) \:
       \mathrm{d}\theta \: . \end{equation}
\item \textbf{STH2M} Directional spreading from $a_2$ and $b_2$ (degr.)
      \begin{equation} \sigma_2(f) = \left [ 0.5 \left \{ 1 - \left (
      \frac{a_2(f)^2+b_2(f)^2}{E(f)^2} \right )^{1/2} \right \} \right ]^{1/2}
      \: , \label{eq:sig_th1} \end{equation}
\item \textbf{WN}  Wavenumbers $k(\sigma)$ (rad/m)
      \begin{equation} \sigma^2 = g k \tanh (k D)
      \: , \label{eq:k} \end{equation}
\end{list}

\item{Spectral partition parameters} \label{out:spec_part}

These output parameters are based on
partitioning of the spectrum into individual wave fields.

\begin{list}{\arabic{outpars})\hfill}
            {\usecounter{outpars} \leftmargin 8mm \labelwidth 7mm
             \rightmargin 5mm \itemsep 0mm \parsep 0mm}

\item \textbf{PHS}  Wave heights $H_s$ of partitions of the spectrum (see
      below). \label{out:first_part}
\item \textbf{PTP} Peak (relative) periods of partitions of the spectrum (parabolic fit).
\item \textbf{PLP} Peak wave lengths of partitions of the spectrum (from peak period).
\item \textbf{PSP} Mean direction of partitions of the spectrum.
\item Directional spread of partition of the spectrum
      Cf. Eq.~(\ref{eq:sig_th}).
\item \textbf{PWS} Wind sea fraction of partition of the spectrum.  The method of
\cite{art:HP01} is used, implemented as described in \cite{tol:Oahu07a}. With
this, a `wind sea fraction' $W$ is introduced
\begin{equation}
W = E^{-1}  \: E |_{U_p > c}  \:\:\: , \label{eq:wsf}
\end{equation}
where $E$ is the total spectral energy, and $E |_{U_p > c}$ is the energy in
the spectrum for which the projected wind speed $U_p$ is larger than the local
wave phase velocity $c = \sigma / k$. The latter defines an area in the
spectrum under the direct influence of the wind. To allow for nonlinear
interactions to shift this boundary to lower frequencies, and subsequently to
have fully grown wind seas inside this are, $U_p$ includes a multiplier
$C_{mult}$
\begin{equation}
U_p = C_{mult} U_{10} \cos ( \theta - \theta_w )  \:\:\: . \label{eq:Up}
\end{equation}
The multiplier can be set by the user. The default value is $C_{mult} = 1.7$.
\item \textbf{PDP} Peak direction of partitions of the spectrum.
\item \textbf{PQP} Goda peakedness of partitions of the spectrum.
\item \textbf{PPE} JONSWAP peak enhancement factor of partitions
      of the spectrum. It is a measure of amplification
      of the spectral peak to the peak of the corresponding
      \citeauthor{art:PM64} spectrum.
\item \textbf{PGW} Gaussian frequency width (Hz) of partitions of the spectrum.
      Least-square fit of a Gaussian spectrum to the one-dimensional spectral partition.
\item \textbf{PSW} Spectral width of partitions of the spectrum \citep{art:LH84}
\item \textbf{PTE} Wave energy period of partitions of the spectrum [see Eq.~(\ref{eq:Tm0m1})]
\item \textbf{PT1} Mean wave period of partitions of the spectrum [see Eq.~(\ref{eq:Tm})]
\item \textbf{PT2} Wave zero-upcrossing period of partitions of the spectrum [see Eq.~(\ref{eq:Tm02})]
\item \textbf{PEP} Wave peak spectral density of partitions
\item \textbf{TWS} Wind sea fraction of the entire spectrum.
\item \textbf{PNR} Number of partitions found in the spectrum. \label{out:last_part}
\end{list}

\item{Atmosphere-waves layer}
\begin{list}{\arabic{outpars})\hfill}
            {\usecounter{outpars} \leftmargin 8mm \labelwidth 7mm
             \rightmargin 0mm \itemsep 0mm \parsep 0mm}
\item \textbf{UST}  The friction velocity $u_\ast$ (scalar). Definition depends on
      selected source term parameterization (m/s). An alternative vector version
      of the stresses is available for research (requires user intervention in
      the code).
\item \textbf{CHA}  Charnock parameter for air-sea friction (without dimensions)
\item \textbf{CGE} Energy flux (W/m)
      \begin{equation} C_g E =  \rho_w g \overline{C_g} E
      \: . \label{eq:CgE} \end{equation}
\item \textbf{FAW} Wind to wave energy flux
\item \textbf{TAW} Net wave-supported stress (wind to wave momentum flux)
\item \textbf{TWA} Negative part of the wave-supported stress
\item \textbf{WCC} Whitecap coverage (without dimensions)
\item \textbf{WCF} Wave to wind momentum flux
\item \textbf{WCH} Whitecap mean thickness (m)
\item \textbf{WCM} Mean breaking wave height (m) (NOT AVAILABLE YET)
%\item \textbf{PWS} Moment of whitecap distribution (?) (NOT PLUGGED YET)
\end{list}

\item{Wave-ocean layer}

\begin{list}{\arabic{outpars})\hfill}
            {\usecounter{outpars} \leftmargin 8mm \labelwidth 7mm
             \rightmargin 0mm \itemsep 0mm \parsep 0mm}
\item \textbf{SXY} Radiation stresses
      \begin{equation} S_{xx} = \rho_w g \int \!\!\!\! \int \left
        ( n - 0.5 + n \cos^2 \theta \right )  \: F(k,\theta) \: dk d\theta
      \: , \label{eq:Sxx} \end{equation}
      \begin{equation} S_{xy} =\rho_w g \int \!\!\!\! \int
        n \sin \theta \cos \theta  \: F(k,\theta) \: dk d\theta
      \: , \label{eq:Syy} \end{equation}
      \begin{equation} S_{yy} =\rho_w g \int \!\!\!\! \int \left
        ( n - 0.5 + n \sin^2 \theta \right )  \: F(k,\theta) \: dk d\theta
      \: , \label{eq:Sxy} \end{equation}
      where
      \begin{equation} n = \frac{1}{2} + \frac{kd}{\sinh 2kd}
      \: . \label{eq:n} \end{equation}
\item \textbf{TWO} Wave to ocean momentum flux
\item \textbf{BHD} Bernoulli head (m$^2$/s$^2$)
      \begin{equation} J =  g \int \!\!\!\! \int  \frac{k}
        {\sinh 2kd}  \: F(k,\theta) \: dk d\theta
      \: , \label{eq:BHD} \end{equation}
\item \textbf{FOC} Wave to ocean energy flux (W/m$^2$)
\item \textbf{TUS} Stokes volume transport (m$^2$/s)
     \begin{equation} (M^w_x,M^w_y) =  g \int \!\!\!\! \int  \frac{(k \cos(\theta),k \sin(\theta))}
        {\sigma}  \: F(k,\theta) \: dk d\theta
      \: , \label{eq:Mw} \end{equation}
\item \textbf{USS} Stokes drift at the sea surface (m/s)
     \begin{equation} (U_{ssx},U_{ssy}) =  \int \!\!\!\! \int  \sigma \cosh 2kd \frac{(k \cos(\theta),k \sin(\theta))}
        {\sinh^2 kd}  \: F(k,\theta) \: dk d\theta
      \: , \label{eq:Uss} \end{equation}
\item \textbf{P2S} Second order pressure variance (m$^2$) and peak period of this pressure (s) which contributes to acoustic and seismic noise,
      \begin{equation} F_{p2D}(k=0) = \int_0^{\infty} \frac{4 \sigma}{C_g} \int_0^{\pi}  F(k,\theta) \:  F(k,\theta+\pi) \:  d\theta dk
      \: , \label{eq:sop} \end{equation}
\item \textbf{USF} Frequency spectrum of Stokes drift at the sea surface (m/s/Hz)
     \begin{equation} (U_{ssx}(f),U_{ssy}(f)) =  \int \sigma \cosh 2kd \frac{(k \cos(\theta),k \sin(\theta))}
        {\sinh^2 kd}  \: F(k,\theta) \frac{2 \pi}{C_g} d\theta
      \: , \label{eq:Ussf} \end{equation}
Note that US3D=1 must be set in the ww3\_grid.inp OUTS namelist in order to activate the 3D arrays needed for USF output.
\item \textbf{P2L} Frequency spectrum of the second order pressure (m$^2$s)  which contributes to acoustic and seismic noise,
      \begin{equation} F_{p2D}(k=0,f) =  \frac{2 \sigma}{\pi} \int_0^{\pi}  \frac{4 \pi^2}{C_g^2} F(k,\theta) \:  F(k,\theta+\pi) \:  d\theta
      \: . \label{eq:sopf} \end{equation}
\item \textbf{TWI}   Wave to sea ice stress
\item \textbf{FIC}   Wave to sea ice energy flux
\item \textbf{USP}   Surface Stokes drift partitioned into run-time defined frequency components (m/s),
      \begin{equation} (U_{ssp},V_{ssp})(n) =  \int \!\!\!\! \int_{k_n}  \sigma \cosh 2kd \frac{(k\
      \cos(\theta),k \sin(\theta))}
        {\sinh^2 kd}  \: F(k,\theta) \: dk d\theta
      \: . \label{eq:Ussp} \end{equation}
The number $n$ and the central wavenumbers for the partitions $k_n$ are `run-time' in the sense that they are defined in the model definition file generated from ww3\_grid.inp (see the sample input file for details on setting these quantities).  The wavenumber integral bounds for each $n$ partition divide the model wave spectra to accomodate the prescribed values of $k_n$.
\end{list}

\item{Wave-bottom layer}

\begin{list}{\arabic{outpars})\hfill}
            {\usecounter{outpars} \leftmargin 8mm \labelwidth 7mm
             \rightmargin 0mm \itemsep 0mm \parsep 0mm}

\item \textbf{ABR} Near-bottom rms excursion amplitude
      \begin{equation} a_{b,rms} = \left [ 2 \int \!\!\!\! \int
      \frac{1}{\sinh^2 kd} \: F(k,\theta) \: dk d\theta \right ] ^{1/2}
      \: . \label{eq:ab_rms} \end{equation}
\item \textbf{UBR} Near-bottom rms orbital velocity
      \begin{equation} u_{b,rms} = \left [ 2 \int \!\!\!\! \int
      \frac{\sigma^2}{\sinh^2 kd} \: F(k,\theta) \: dk d\theta \right ] ^{1/2}
      \: . \label{eq:ub_rms} \end{equation}
\item \textbf{BED} Bedform parameters: ripple height and directions (NOT TESTED YET)
\item \textbf{FBB} Energy dissipation in WBBL
\item \textbf{TBB} Momentum loss in WBBL
\end{list}

\item{Spectrum parameters}

\begin{list}{\arabic{outpars})\hfill}
            {\usecounter{outpars} \leftmargin 8mm \labelwidth 7mm
             \rightmargin 0mm \itemsep 0mm \parsep 0mm}

\item \textbf{MSS} Mean square slopes in $u$ and $c$ directions (down-wave and cross-wave components of slopes variances).
\item \textbf{MSC} Spectral tail level (without dimensions)
\item \textbf{MSD} Direction of the maximum slope variance mss$_u$
\item \textbf{MCD} Spectral tail direction
\item \textbf{QP}  Peakedness parameter \citep{art:G70}
      \begin{equation} Q_p = \frac{2}{E^2} \int_0^{2\pi} \int_0^\infty
      \sigma\:F(\sigma,\theta)^2\:d\sigma\:d\theta \: \label{eq:qp}
      \end{equation}
\end{list}

\item{Numerical diagnostics }

\begin{list}{\arabic{outpars})\hfill}
            {\usecounter{outpars} \leftmargin 8mm \labelwidth 7mm
             \rightmargin 0mm \itemsep 0mm \parsep 0mm}
\item \textbf{DTD} Average time step in the source term integration (s).
\item \textbf{FC} Cut-off frequency $f_c$ (Hz, depends on parameterization of
      input and dissipation).
\item \textbf{CFX} Maximum CFL number for spatial advection
\item \textbf{CFD} Maximum CFL number for angular advection
\item \textbf{CFK} Maximum CFL number for wavenumber advection
\end{list}

\item{User defined }

\begin{list}{\arabic{outpars})\hfill}
            {\usecounter{outpars} \leftmargin 8mm \labelwidth 7mm
             \rightmargin 0mm \itemsep 0mm \parsep 0mm}

\item \textbf{U1} Slot for user defined parameter (requires modification of code).
\item \textbf{U2} Idem.
\end{list}

\end{list}

%\vspace{\baselineskip}
%\centerline{\ldots NEEDS CLEAN UP BELOW HERE \ldots}
%\vspace{\baselineskip}

%THESE TWO PARAMETERS WILL BE PUT BACK.
%Old output types \ref{out:old_fpw} and \ref{out:old_thw} have become obsolete
%with the introduction of the partitioned output types \ref{out:first_part}
%through \ref{out:last_part}. However, they are retained in the present model
%release to provide required downward compatibility with model version 1.18 at
%NCEP. These output types are likely to be retired in upcoming versions of \ws.


\chapter{Tweet usage overview}
List of previously observed uses for tweets.
\begin{itemize}
    \item Time series
    \item Stock index time series analysis
    \item Message volume
    \item Message polarity, Bullishness/bearishness
    \item Predicting the stock market
    \item Predict next day-trading volume
    \item Daily one-day-ahead predictions
    \item Topic based prediction
    \item Vector regression
    \item Dirichlet Process mixture
    \item Label propagation
    \item Constrained and supervised learning.
\end{itemize}


% The ICIP2016 paper according to the given template
%          spconf.sty  - ICASSP/ICIP LaTeX style file, and
%          IEEEbib.bst - IEEE bibliography style file.
% --------------------------------------------------------------------------
\documentclass{article}
\usepackage{spconf,amsmath,graphicx}
\usepackage{enumitem}
\usepackage{verbatim}

% Definitions.
% --------------------
\def\B{{\mathbf B}}
\def\M{{\mathbf M}}
\def\I{{\mathbf I}}
\def\mcT{{\mathcal{T}}}
\def\mcD{{\mathcal{D}}}
\def\mcA{{\mathcal{A}}}
\def\mcL{{\mathcal{L}}}
\def\mcV{{\mathcal{V}}}
\def\p{{\mathbf p}}
\def\q{{\mathbf q}}
\def\r{{\mathbf r}}
\def\s{{\mathbf s}}
\def\S{{\mathbf S}}
\def\mcN{{\mathcal{N}}}
\DeclareMathOperator*{\argmax}{arg\,max}
% Title.
% ------
\title{A Data-driven Region Detector for Structured Image Scenes}
%
% Single address.
% ---------------
\name{Elena Ranguelova}
\address{Netherlands eScience Center\\ Amsterdam, The Netherlands}
%\\ e-mail: E.Ranguelova@esciencecenter.nl}

\begin{document}

\maketitle

\begin{abstract}
Finding correspondences between two images of the same scene, taken from different viewpoints, with semantic features is a challenging problem.
This paper proposes a Data-driven Morphology Salient Regions (DMSR) approach for detecting interest regions repeatedly.
A binarization algorithm creates a compact image representation that is then analyzed for saliency using morphology.
DMSR has comparable performance to the renowned Maximally Stable Extremal Regions (MSER) detector on structured scenes
and better invariance to lighting, blur and on a high-resolution benchmark.
This is achieved via significantly fewer detected regions, leading to better scalability. %- a much desired property in the big data era.
DMSR is shown to be a better choice than MSER for analysis of scientific imagery in the big data era, e.g.,
it detects precisely meaningful regions in images used for wild-life biometrics.
The paper also introduces OxFrei, a dataset for transformation-independent detection evaluation.
\end{abstract}

%\begin{keywords}
%region detection, data-driven, structured scenes, morphology, scientific visual analytics
%\end{keywords}

\section{Introduction}
\label{sec:intro}
The first fundamental step in numerous computer vision applications (wide baseline stereo matching, image retrieval, visual mining, etc.\,) is to reliably and repeatedly find  the correspondence between a pair of different images of the same scene \cite{Matas2002BMVC, Rodriguez2013, EscaleraCVPR07}. One class of methods- {\em region detectors} find distinct (salient) regions, which correspond to the same image patches, detected independently in each image. The detectors must be {\em covariant} (often called {\em invariant}) to, usually, {\em affine} transformations and various photometric distortions.

A decade ago, a performance evaluation paper by the Visual Geometry Group in Oxford compared existing region detectors \cite{Mikolajczyk:2005}.
A clear conclusion of the comparison was that {\em  Maximally Stable Extremal Regions (MSER)} is the best performing detector for {\em structured} scenes, e.g., those containing homogeneous regions with distinctive boundaries \cite{Matas2002BMVC}. MSER has become the de-facto standard in the field, e.g., it is in the MATLAB Computer Vision Systems Toolbox and OpenCV. Despite its success, the detector has several drawbacks: it is sensitive to image blur; it produces nested and redundant regions and its performance degrades with the increase of image resolution \cite{CorRos2013}. Analysis in geometric scale-space have showed that the formulation of the region stability criterion makes MSER prefer regular shapes \cite{Kimmel11}.

\begin{figure}[htb]

\begin{minipage}[b]{.48\linewidth}
  \centering
  \centerline{\includegraphics[width=4cm]{./Figs/mserTailA}}
\end{minipage}
\begin{minipage}[b]{0.51\linewidth}
  \centering
  \centerline{\includegraphics[width=4cm]{./Figs/mserTailB}}
\end{minipage}
\hfill
\begin{minipage}[b]{.48\linewidth}
  \centering
  \centerline{\includegraphics[width=4cm]{./Figs/dmsrTailA}}
\end{minipage}
\begin{minipage}[b]{0.51\linewidth}
  \centering
  \centerline{\includegraphics[width=4cm]{./Figs/dmsrTailB}}
\end{minipage}
\hfill
\vspace{-0.4cm}
\caption{Region detection on two images of the tail of the same humpback whale.
Top row: MSER, bottom row: DMSR(All).}
\label{fig:tails}
\vspace{-0.3cm}
\end{figure}

%----------------------------------------------------------------
\begin{figure}[htb]

\begin{minipage}[b]{.24\linewidth}
  \centering
  \centerline{\includegraphics[width=1.8cm]{./Figs/mserLeatherbackA}}
   \centerline{(a)}\medskip
\end{minipage}
\hfill
\begin{minipage}[b]{0.24\linewidth}
  \centering
  \centerline{\includegraphics[width=1.8cm]{./Figs/mserLeatherbackB}}
\centerline{(b)}\medskip
\end{minipage}
\hfill
\begin{minipage}[b]{.24\linewidth}
  \centering
  \centerline{\includegraphics[width=1.8cm]{./Figs/dmsrLeatherbackA}}
\centerline{(c)}\medskip
\end{minipage}
\hfill
\begin{minipage}[b]{0.24\linewidth}
  \centering
  \centerline{\includegraphics[width=1.8cm]{./Figs/dmsrLeatherbackB}}
 \centerline{(d)}\medskip
\end{minipage}
 \vspace{-0.2cm}
\caption{Region detection on two images of the pineal spot of the same leatherback turtle.
(a),(b): MSER, (c),(d): DMSR.}
\label{fig:turtle}
 \vspace{-0.2cm}
\end{figure}

Since then, many researchers have proposed improvements to MSER with no drastic increase of performance. An MSER color extension, {\em Maximally Stable Color Region}, outperforms both an MSER-per-color-channel combination and a color blob detector \cite{Forssen07}. Improving the MSER region distinctiveness by morphological dilation on the detected Canny edges is proposed in \cite{Wang14}. The improved detector shows better performance in classification application, but evaluation of repeatability is not reported. MSER has been extended to {\em Maximally Stable Volumes} for successful segmentation of 3D medical images and paper fiber networks \cite{DonoserB06}.

While most research has focused on generic applications, the emerging fields of {\em animal and plant biometrics} are attracting more attention \cite{Kuehl2013, leafsnap_eccv2012}. Computer vision is becoming a vital technology enabling the wild-life preservation efforts of ecologists in the big data era. Along with the individual or species photo-ID, the scientists wish to obtain reliable measurements of meaningful structures from images. The generic region detectors do not satisfy this need: Fig.\,\ref{fig:tails}, top row shows over-abundant or not semantic regions and Fig.\,\ref{fig:turtle}, (a), (b) shows missed detection. On the contrary, our {\em Data-driven Morphology Salient Regions (DMSR) detector} finds the semantic regions (Fig.\,\ref{fig:tails}, bottom row and Fig.\,\ref{fig:turtle}, (c), (d)).

\begin{comment}
%----------------------------------------------------------------------
\begin{figure}[htb]
\vspace{-0.5cm}
\begin{minipage}[b]{.49\linewidth}
  \centering
  \centerline{\includegraphics[width=5cm]{./Figs/binary_marks}}

\end{minipage}
%\hfill
\begin{minipage}[b]{0.49\linewidth}
  \centering
  \centerline{\includegraphics[width=5cm]{./Figs/binary_marks_clean_color_coded}}

\end{minipage}
\vspace{-0.5cm}
\caption{Binary salient regions detection.
Color coding: holes-blue, islands- yellow,
indentations - green, protrusions- red. }
\label{fig:binary_sal}
\end{figure}
%----------------------------------------------------------------------
\end{comment}

Although crucial for the development of detectors, there is a shortage of evaluation benchmarks, especially for performance analysis independently of the image content. The standard {\em Oxford dataset} is very small: eight test sequences containing six (one base and five transformed) images of the same scene each. Every pair (base, transformed) is related via a given transformation matrix (homography) \cite{Mikolajczyk:2005}.  The {\em Freiburg dataset} contains $416$ higher resolution images, generated by transforming $16$ base images in order to de-tangle transformations from content \cite{FischerDB14}.
The {\em TNT dataset} contains versions of the same viewpoint sequences with increasing resolution from $1.5$ to $8$~MPixel per image. Highly accurate image pair homographies are given. It is suitable for evaluating robustness to resolution rather than to transformations \cite{CorRos2013}.

This paper contributes to solving the identified problems. We propose a new regions detector, DMSR, and made the software available as open source \cite{elena_ranguelova_2016_45156}. It is related to the {\em Morphology-based Stable Salient Regions (MSSR)} detector that we developed in the context of humpback whale identification \cite{RangMSSR06, RangHumpb06}. DMRS includes a binarization, robust-to-lighting-and-blur, that yields a much smaller number of regions and is more stable across transformations. It has similar or higher (lighting, blur and increased resolution) repeatability compared to MSER, while detecting non-redundant perceptually salient regions (Fig.~\ref{fig:tails}). Also, we compose and share an openly available dataset, OxFrei, combining the natural homographies of the Oxford and the higher resolution images of the Freiburg datasets \cite{elena_ranguelova_2016_45156}.

\begin{comment}
\section{Data-driven Morphology Salient Regions Detection}
\label{sec:DMSR}

MSER and MSSR decompose a gray-scale image into binary cross-sections and evaluate the stability of the connected components (CCs) or accumulate saliency masks. DMSR starts with a data-driven binarization, producing one binary image, thus transforming the problem into binary saliency.

%----------------------------------------------------------------------
\begin{table}[hbt]
\begin{minipage}[b]{0.99\linewidth}\begin{tabular}{|l l|}
\hline
{\bf ISS} & A CC $S^i_{fb} = \{\p \in \mcD, \forall \p=foreground,$\\&$\forall \q \in \partial S^i_{fb}, \q=background, \q \notin \partial \B $ \},\\
$2$ types & $S^i_{10}$ (islands), $S^i_{01}$  (holes); $\S^i = S_{01}^i \cup S_{10}^i$\\
{\bf BSS} &  $S_{fb}^b: \{\p \in S_{fb}^b \subset{\cal B}^f, \forall \p = foreground,$\\&$ \q \in \partial S_{fb}^b \subset {\partial \cal B}^f,\forall \q = background \}$, \\
& $|\partial {\cal B}^f| - |\partial ({\cal B}^f \backslash S_{fb}^b)| < 2 \pi r$\\
$2$ types & $S^b_{10}$ (protr.), $S^b_{01}$ (indent.); $\S^b = S_{01}^b \cup S_{10}^b$\\
{\bf Regions} &  $\S = \S^i$ (DMSR); $\S = \S^i \cup \S^b$ (DMSRA)  \\
\hline
\end{tabular}
\hfill
\centering
\caption{Binary saliency definitions used in Section \ref{ssec:binary}.}\label{table:binary_sal}
\end{minipage}
\vspace*{-0.4cm}
\end{table}

\subsection{Binary Salient Regions Detection}
\label{ssec:binary}
We claim that the perceptual saliency in a binary image of a structured scene
 $\B: \mcD \subset \mathcal{Z}^2 \rightarrow \{0,1\}$ (1-white, 0-black)
is only due to the spatial layout of the image regions \cite{RangHumpb06}.
There are  $4$ types of salient regions. The $2$ types of {\em inner salient structures (ISS)} are (1) {\em holes} -- set of connected black pixels entirely surrounded by white pixels, and (2) {\em islands} -- set of connected white pixels surrounded by black ones, i.e.~inverse of holes. A significant connected component ${\cal B}^1$ is defined as a CC with area proportional to the image area by $\Lambda$. The radius of the morphological structuring element is $r$ and  the area opening parameter for  noise filtering is $\lambda$. The $2$ {\em boundary salient structures (BSS)} are (3) {\em protrusions}- set of white pixels on the border of a significant CC, which if pinched off from the CC, will increase its boundary with no more than $2\pi r$, and (4) the {\em indentations}- protrusions inverse.

The types are also valid for the MSSR detector. The regions are obtained from $\B$ by morphological operations: hole filling, top hat and area opening, for details see \cite{RangMSSR06, RangHumpb06}. The ISS are similar to the definition of the MSER+ and MSER- regions \cite{Matas2002BMVC}. In this paper, detectors using only ISS, i.e., directly comparable to MSER, are denoted by DMSR/MSSR, while DMSRA/MSSRA are detectors using all region types. These definitions are summarized in Table~\ref{table:binary_sal} and the exact shaped regions from a synthetic $100 \times 100$ binary image with parameters $\Lambda=100$, $r=5$ and $\lambda = 10$ are shown on Fig.~\ref{fig:binary_sal}.

\subsection{Data-driven binarization}
\label{ssec:binarize}
Any gray-scale image  $\I:~\mcD~\subset~\mathcal{Z}^2~\rightarrow ~\mcT $, where $\mcT=\{0,1,\ldots, t_{max}\}$ and $t_{max}~=~2^n-1~=~255$ is the maximum gray value encoded by $n~=~8$ bits, can be decomposed into cross-sections at
every possible level $t$:  $\I = \sum_{t \in \mcT}CS_t(\I)$. Obtaining a section at level $t$ is equivalent to thresholding the image at threshold $t$: $CS_t(\I)= 1.(\I>t) + 0.(\I<t)$ is a binary image.
Three sets of connected components in $CS_t(\I)$ are defined: $\mcA_t$- {\em all}, $\mcL_t$- the {\em large} and $\mcV_t$- the {\em very large} CCs.  The size of each CC category is defined by$\Lambda_{\mcL}$ and $\Lambda_{\mcV}$ fraction of the image area $A_{\I}$. Let us denote the normalized number of elements in a set ($|\cdot|$) by $\Vert \cdot \Vert = |\cdot| / \max_{t \in \mcT}|\cdot|$.
Finding the optimal threshold $t_{opt}$ is then defined as:
\begin{equation*}
\vspace{-0.05cm}
t_{opt} = \argmax_{t \in \mcT}( w^{\mcA} \Vert \mcA_t \Vert + w^{\mcL} \Vert \mcL_t \Vert + w^{\mcV} \Vert \mcV_t \Vert ),
\vspace{-0.05cm}
\end{equation*}
where $w^{\cdot}$ are the weights per set of CCs.

The standard Otsu thresholding does not select a single stable $CS$, while choosing $t_{opt}$ ensures stable number of regions across transformations (see Figs.~\ref{fig:binary_hist} and \ref{fig:leuven_bin} for lighting).
%------------------------------------------------------------------------
\vspace{-0.8cm}
\begin{figure}[htb]

\begin{minipage}[b]{0.49\linewidth}
  \centering
  \centerline{\includegraphics[width=4.65cm]{./Figs/hist_otsu_leuven_1_4}}
  \centerline{(a) Otsu}\medskip
\end{minipage}
\hfill
%\vspace{-0.5cm}
\begin{minipage}[b]{0.49\linewidth}
  \centering
  \centerline{\includegraphics[width=4.8cm]{./Figs/hist_numcc_leuven_1_4}}
\centerline{(b) Max number CC}\medskip
\end{minipage}
\hfill
\vspace{-0.5cm}
\caption{Finding the optimal threshold for two images from the 'Leuven' sequence
(Oxford dataset, lighting): the base image- solid black line, the forth image - dotted blue line.}
\label{fig:binary_hist}
%
\end{figure}
%------------------------------------------------------------------------
%\vspace{-0.5cm}

\begin{figure}[htb]

\begin{minipage}[b]{.3\linewidth}
  \centering
  \centerline{\includegraphics[width=2.8cm]{./Figs/leuven1}}
%  \vspace{1.5cm}
%   \centerline{(a)}\medskip
\end{minipage}
\hfill
\begin{minipage}[b]{0.3\linewidth}
  \centering
  \centerline{\includegraphics[width=2.7cm]{./Figs/leuven1_otsu}}
%  \vspace{1.5cm}
  % \centerline{(b)}\medskip
\end{minipage}
\hfill
\begin{minipage}[b]{0.3\linewidth}
  \centering
  \centerline{\includegraphics[width=2.7cm]{./Figs/leuven1_numcc}}
%  \vspace{1.5cm}
  % \centerline{(c)}\medskip
\end{minipage}

\begin{minipage}[b]{.3\linewidth}
  \centering
  \centerline{\includegraphics[width=2.7cm]{./Figs/leuven4}}
%  \vspace{1.5cm}
   \centerline{(a)}\medskip
\end{minipage}
\hfill
\begin{minipage}[b]{0.3\linewidth}
  \centering
  \centerline{\includegraphics[width=2.7cm]{./Figs/leuven4_otsu}}
%  \vspace{1.5cm}
   \centerline{(b)}\medskip
\end{minipage}
\hfill
\begin{minipage}[b]{0.3\linewidth}
  \centering
  \centerline{\includegraphics[width=2.7cm]{./Figs/leuven4_numcc}}
%  \vspace{1.5cm}
   \centerline{(c)}\medskip
\end{minipage}

 \vspace{-0.5cm}
\caption{Binarization of two images of the 'Leuven' sequence (lighting). Top row- base image, bottom row- forth image;   (a) gray scale; (b) Otsu binarization, (c) proposed binarization.}
\label{fig:leuven_bin}
%
\end{figure}
%------------------------------------------------------------------------

After the data-driven binarization, the DMSR detector finds the set of affine-covariant regions $\S$ from the single binary image $CS_{t_{opt}}$ as described in Section \ref{ssec:binary} and \cite{RangMSSR06, RangHumpb06}. DMSR produces fewer non-overlapping and perceptually salient regions (visualized by their equivalent ellipses, not exact shapes) compared to MSER (see Figures~\ref{fig:det_graffiti} and \ref{fig:wood}).

\section{Performance  Evaluation}
\label{sec:perf}
The {\em repeatability score} ($R$) and the {\em number of correspondences} ($N_C$) are the main performance evaluation measures \cite{Mikolajczyk:2005}. The maximum overlap error between matching regions is $40\%$. The $R$ score between a pair of base and transformed image, $(\I_B,\I_T)$, is the ratio between $N_C$ in the common image part and the smaller number of regions in the pair. The structured scenes from each dataset are considered and five detectors are evaluated: MSER, MSSR(A) and DMSR(A). The MSER software is used with its default settings and the (D)MSSR(A) parameters are: $r = 0.02*\sqrt{A_{\I} / \pi}$, $\lambda=3r$, $\Lambda_{\mcL}=0.001$, $\Lambda_{\mcV}=0.01$ and  $w^{\cdot}=0.33$.
All performance plots and detected regions on all data are available online \cite{elena_ranguelova_2016_45156}.

%------------------------------------------------------------------------
\begin{figure}[htb]

\begin{minipage}[b]{.49\linewidth}
  \centering
  \centerline{\includegraphics[width=4.0cm]{./Figs/mserGraffiti1}}
 % \vspace{0.2cm}
 % \centerline{(a) MSER}\medskip
\end{minipage}
\hfill
\begin{minipage}[b]{0.49\linewidth}
  \centering
  \centerline{\includegraphics[width=4.0cm]{./Figs/dmsrGraffiti1}}
 % \vspace{0.2cm}
 % \centerline{(b) DMSR}\medskip
\end{minipage}
\vspace{-0.25cm}
\caption{Region detectors on the base image of the 'Graffiti' sequence, Oxford dataset. Left: MSER, right: DMSR}
\label{fig:det_graffiti}
%
\end{figure}
%------------------------------------------------------------------------
\begin{figure}[htb]

\begin{minipage}[b]{.49\linewidth}
  \centering
  \centerline{\includegraphics[width=4.0cm]{./Figs/mserWood}}
 % \vspace{0.2cm}
 % \centerline{(a) MSER}\medskip
\end{minipage}
\hfill
\begin{minipage}[b]{0.49\linewidth}
  \centering
  \centerline{\includegraphics[width=4.0cm]{./Figs/dmsrWood}}
 % \vspace{0.2cm}
 % \centerline{(b) DMSR}\medskip
\end{minipage}
%\vspace{-0.25cm}
\caption{Salient region detectors on microscopy wood images. Left: MSER (every second region is shown), right: DMSR}
\label{fig:wood}
%
\end{figure}
%\vspace{-0.5cm}
%------------------------------------------------------------------------
\begin{figure}[htb]
\centering
\begin{minipage}[b]{.99\linewidth}
  \centering
  \centerline{\includegraphics[width=8.5cm]{./Figs/repeatability_all_affine_bikes_blur}}
 % \vspace{0.2cm}
 % \centerline{(a) MSER}\medskip
\end{minipage}
\hfill
%\vspace{-0.25cm}
\caption{Region detection on 'Bikes', Oxford dataset.}
\label{fig:det_bikes}
%
\end{figure}
%------------------------------------------------------------------------
%\vspace{-0.5cm}
\begin{figure}[htb]
\centering
\begin{minipage}[b]{.99\linewidth}
  \centering
 \centerline{\includegraphics[width=8.5cm]{./Figs/repeatability_all_combined_01_graffiti_ligthing_good}}
 % \vspace{0.2cm}
 % \centerline{(a) MSER}\medskip
\end{minipage}
\hfill
%\vspace{-0.25cm}
\caption{Region detectors on '01\_graffiti', OxFrei dataset.}
\label{fig:det_frei}
%
\end{figure}
%------------------------------------------------------------------------

\begin{figure}[htb]

\begin{minipage}[b]{.9\linewidth}
  \centering
  \centerline{\includegraphics[width=8.5cm]{./Figs/blur_some_combined}}
 % \vspace{0.2cm}
 % \centerline{(a) MSER}\medskip
\end{minipage}
\hfill
%\vspace{-0.25cm}
\caption{Robustness of region detectors to blur on five sequences of the OxFrei dataset.}
\label{fig:blur}
%
\end{figure}
%------------------------------------------------------------------------
%------------------------------------------------------------------------
\subsection{Oxford dataset}
\label{ssec:oxford}
Each image sequence of the Oxford dataset consists of one base and five increasingly distorted images \cite{Mikolajczyk:2005}. They are obtained independently of each other and the homographies between each pair $(\I_B,\I_T)$ are the provided  ground truth. Each sequence can be used to test only one transformation $T$.
For the viewpoint ('Graffiti'), the best $R$ is achieved by DMSR with up to $72\%$, which is $10\%$ more than the second performing MSER for a $40^{\circ}$ change. DMSR is performing worse on the scale ('Boat'), but as good as or better than MSER on the blur ('Bikes', Fig.~\ref{fig:det_bikes}) and lighting ('Leuven') sequences.

\subsection{OxFrei dataset}
\label{ssec:combined}
The creators of the Freiburg dataset separated $T$ from the image content by applying a few transformations (alas not fully documented) to different base images \cite{FischerDB14}. To address the shortage of evaluation datasets, we release OxFrei - a combination of the strong features of the Oxford and Freiburg datasets \cite{elena_ranguelova_2016_45156}. We transform the Freiburg base images with all homographies of the Oxford dataset. In this way, we create $54$ images in $9$ structured scenes each under realistic blur, lighting, scaling and viewpoint transformations.

The dataset allows a transformation independent robustness study by comparing performance on all data subject to the same realistic $T$. Figures~\ref{fig:det_frei} and \ref{fig:blur} show $R$ for lighting for one and blur for a few sequences. The standard plots (like on Fig.~\ref{fig:det_frei}) are cross-sections along the data dimension of the 3D plots (like on Fig.~\ref{fig:blur}) for one sequence. The 2D plots for all OxFrei experiments are online \cite{elena_ranguelova_2016_45156}. We conclude that MSER is better for zoom and viewpoint (the latter contradicting with the result on the single Oxford 'Graffiti' sequence), while DMSR is robust to lighting and blur (Fig. \ref{fig:blur}).

\subsection{TNT hi-res benchmark}
\label{ssec:tnt}

The $R$ score of all detectors from the Oxford study drops down on hi-res images \cite{CorRos2013}. On the 'underground' sequence from the TNT set, MSER loses up to $25\%$ between $1.5$~Mpx ($R_1$) and $8$Mpx ($R_4$) resolutions. On the contrary, DMSR increases the $R$ score as resolution increases on 'underground' and 'posters' sequences with up to $75\%$, which is a $15\%$ increase from $R_1$ to $R_4$ for a $40^{\circ}$ viewpoint change (Figure \ref{fig:tnt}).

\begin{figure}[htb]

\begin{minipage}[b]{.9\linewidth}
  \centering
  \centerline{\includegraphics[width=8.5cm]{./Figs/posters_rep}}
 % \vspace{0.2cm}
 % \centerline{(a) MSER}\medskip
\end{minipage}
\hfill
\caption{Robustness of region detectors to image resolution and viewpoint. 'Posters', TNT dataset.}
\label{fig:tnt}
\vspace{-0.25cm}
\end{figure}
%------------------------------------------------------------------------
\begin{figure}[htb]

\begin{minipage}[b]{0.9\linewidth}
  \centering
  \centerline{\includegraphics[width=8.5cm]{./Figs/posters_numreg}}
 % \vspace{0.2cm}
 % \centerline{(b) DMSR}\medskip
\end{minipage}
%\vspace{-0.25cm}
\caption{Number of region correspondences versus image resolution and viewpoint. 'Posters', TNT dataset.}
\label{fig:tnt_numreg}
%
\end{figure}
%------------------------------------------------------------------------

\subsection{Animal and plant biometrics}
\label{ssec:bio}
The DMSR detector has been compared to MSER on several small animal individual photo-ID datasets (humpback whales, leatherback turtles, newts) and on a wood species identification dataset \cite{RangHumpb06, PauwelsZB08, wood}. In all cases DMSR produces fewer and perceptually more acurate salient regions as illustrated by Figures~\ref{fig:tails}, \ref{fig:turtle} and \ref{fig:wood}. For the wood microscopy images, it is not possible to obtain accurate statistics on the cell properties with MSER, while the detected DMSR regions enable such wood anatomy research.

On all datasets, the $N_C$ plane of DMSR has the lowest values and the least slope of all detectors (Figs.~\ref{fig:det_bikes}, \ref{fig:det_frei}, right and Fig.~\ref{fig:tnt_numreg}) \cite{elena_ranguelova_2016_45156}.
The number of detected DMSR regions is up to an order of magnitude lower compared to MSER - crucial for the efficiency of the following matching step on large-scale image data.
Using all $4$ types of regions does not improve performance over using only $ISS$ (holes and islands) with the exception of detecting markings on humpback whale tails.

\section{CONCLUSIONS}
Combining data-driven binarization with morphological operations yeilds a region detector with comparable to superior performance to MSER on various datasets. DMSR produces a much smaller number of regions- a very desired property in large scale processing. It copes better with blur, lighting and increased resolution and detects perceptually salient regions, which makes it a good choice for scientific imagery analytics.
For detection evaluation, high-resolution transformation-independent datasets should become the standard.
\label{sec:concl}

\end{comment}
% To start a new column (but not a new page) and help balance the last-page
% column length use \vfill\pagebreak.
% -------------------------------------------------------------------------
%\vfill
%\pagebreak

%\section{REFERENCES}
%\label{sec:ref}

% References should be produced using the bibtex program from suitable
% BiBTeX files (here: strings, refs, manuals). The IEEEbib.bst bibliography
% style file from IEEE produces unsorted bibliography list.
% -------------------------------------------------------------------------

\bibliographystyle{IEEEbib}
\bibliography{icip2016}

\end{document}


\section{Capabilities}

% \emph{Describe organizational experience in relevant subject area(s), existing intellectual property, or specialized facilities. Discuss any work in closely related research areas and previous accomplishments.}

% \clearpage
% \subsection{Background}

\subsection[ Open Connectome Project]{Open Connectome Project} \label{sss:ocp}

We have previously built a petascale data-intensive computing infrastructure designed around serial electron microscopy images \cite{Burns13a}. As we will explain in greater detail below, in this proposal, we will extend existing capabilities in a number of ways.  First, we will greatly expand the different modalities and scales of data.  This will enable us to ingest and query data from worms, flies, mice, monkeys, and humans, all using the exact same API.  Second, we will greatly exapnd visualization capabilities, including both two- and three-dimensional  (2D and 3D) local and remote capabilities, to further lower the barrier to entry, enabling citizen scientists to interact with the data as we do.  Third, we will significantly enhance support for adding annotations of a wide variety of types to each data product.  This includes image annotations (for example, different atlases or local statistics), text annotations (such as names, relavant citations, etc.), and connectivity annotations.  Fourth, we will add significant analytical capabilities to the users.

Collectively, these modifications will enable workflows such as the following: (i) search for all connections to hippocampus in any brain of any animal, (ii) visualize each brain (only those parts connected to hippocampus) in 3D, (iii) grade each in terms of quality, (iv) compute an average neighborhood of hippocampus, weighted by quality score, (v) push such annotation back to the server so that others may now query against it.

\subsection[ Graph Analytics]{Graph Analytics} \label{sss:graph}

We and others have begun foundational statistical theory for the analysis of single graphs \cite{Athreya2013, Sussman2012, rukhin2012on, Lee13a, Lee14a, Tang2013a, shuffled, Borges2011, Priebe2009a, Qin2013, lee2011random, Mhembere2013a, Wang2013a, signal-subgraph, Rukhin2011, ceyhan2007a, Tang11a, Grothendieck2010, Priebe2012, Sussman13, lee2011a, PCP10, Park2012a, Mhembere2013b, Sussman2012a, solka2002a, pilla2003adaptive, poston1997a, Naiman2001, ceyhan2006relative, Priebe2011c, Robinson2012, supervenience, RVC13, BVN, Abrams2002, Fishkind2012b, marchette2008predicting, priebe2003class}.  Similarly, we have worked on computing distances between graphs \cite{Tang14a}, as well as matching  a pair of graphs \cite{Vogelstein11a, Fishkind2012a, Lyzinski13a, Fiori13a, Lyzinski14a}, including weighted graphs, graphs with different numbers of vertices, etc. \cite{Lyzinski14b}.  However, theory for jointly embedding populations of graphs, foundational for testing as well as both supervised and unsupervised learning remains completely absent.  This includes both matched graphs (where we need not align the vertices), and unmatched graphs (where we must either align or otherwise be invariant to alignment).

\subsection[ FlashGraph]{FlashGraph} \label{sss:flash}

FlashGraph was developed to enable efficient processing of massive individual graphs on commodity machines \cite{Zheng14a}.  Specifically, it was designed for graph traversal algorithms, such as breadth-first-search and connected components.
We will extend it to support semi-external memory implementations of linear algebra subroutines (including sparse matrix operations), as well as operate on populations of graphs

\subsection[ GRAPHS vs. SIMPLEX]{GRAPHS vs. SIMPLEX}

With respect to the above background subsections, our GRAPHS proposal does not address \S \ref{sss:ocp} at all.  In GRAPHS we do develop a few statistical methods on graphs, but we do not build a foundational theory of richly attributed graphs upon which a great variety of further applications may be developed.  In terms of FlashGraph, our GRAPHS proposal only addresses graph traversal style algorithms for single simple graphs, versus here where we propose to extend it to include populations of graphs and a whole suite of dense and sparse matrix operations on graphs. Thus, this proposal is entirely complementary to our SIMPLEX proposal, extending the deliverables significantly. Specifically, this proposal will enables us to fuse previously totally disparate methods and code bases under a single coherent roof to enable previously unconceptualized capabilities, both internally and for the community.

\subsection[ Specialized Facilities]{Specialized Facilities}

This proposal is heavily computational, and hence we describe the computational resources available to us at this time.  Dr.~Vogelstein and Dr.~Priebe, as core faculty in the Center for Imaging Science (CIS), have access to the CIS cluster; Dr.~Vogelstein, as core faculty also in the Institute for Computational Medicine (ICM), as access to the ICM cluster; and Dr. Vogelstein and Dr. Burns, as members of the Institute for Data Intensive Engineering and Sciences (IDIES), have access to those facilities.  Moreover, the three faculty are utilizing resources from Dr.~Vogeslstein's startup package to build a new dedicated ``Bruster'' for doing brain computations.  Finally, Dr.~Priebe and Dr.~Vogelstein, as members of CIS, have an allocation of one million compute hours on XSEDE\footnote{\url{https://www.xsede.org/}}, a nationally supported compute cluster.  The faculty and personnel on this proposal have been utilizing these resources  in preliminary work for this proposal, as well as the funded CRCNS grant (CRCNS-1208044; co-PIs Vogelstein, Burns, et al.) and Big Data grant (BIGDATA-1251208; co-PIs Vogelstein, Burns, et al.),  a CRCNS grant on computational infrastructures upon which the methods developed herein can operate at accelerated rates (NSF Proposal ID-1311505), and several others.  Below we briefly describe current resources.

\para{{The GrayWulf Cluster}} % (fold) \label{par:paragraph_name}
GrayWulf is a distributed database cluster at JHU consisting of 50 database nodes with 22TB and an 8-core server each, for a total of 1.1PB. The cluster was purchased on funds from the Gordon and Betty Moore Foundation, the Pan-STARRS project and Microsoft Research. The cluster already hosts several large datasets (Pan-STARRS, turbulence, SDSS, various Virtual Observatory catalogs and services, environ- mental sensor data, computer security datasets, network traffic analysis data, etc). Currently about 800TB is already utilized. The cluster has an IO performance exceeding many supercomputers: the aggregate sequential read speed is more than 70 Gbytes/sec. The internal connectivity has recently been upgraded to 10Gbps Ethernet.

\para{{The HHPC Cluster}} % (fold) \label{par:paragraph_name}
%
The same computer room hosts a 1400 core BeoWulf cluster, a computational facility shared among several JHU faculty.  The HHPC and the GrayWulf share a common 288-port DDR Infiniband switch for an extremely high-speed interconnect. There is an MPI interface under development that will enable very fast peer-to-peer data transfers between the compute nodes and the database nodes. The Deans of the JHU Schools provide funds to cover the management and operational costs of the two connected clusters. The second generation of the cluster is about to be purchased, adding another 2,400 cores to the system.

\para{{NSF-MRI NVIDIA cluster}} % (fold)\label{par:paragraph_name}
%
JHU has received an NSF MRI grant (CMMI-0923018), to purchase a large GPU cluster. We have 100 Fermi C2050's in production. Burns is a co-PI on this grant. There is a natural cohesion between the Data-Scope and the GPU cluster, and coordination between the principal architects of the two systems.

% paragraph paragraph_name (end)

\para{{Bloomberg 156 Data Center}} % (fold) \label{par:paragraph_name}
%
The NSF has awarded a \$1,337,272 grant for ``Advanced CyberInfrastructure for High Performance Data Intensive Computing'' (OCI-0963185). This infrastructure project renovated room 156 in Bloomberg Hall to create a flexible, stable environment for a high density of computing equipment that supports research and research training on the Homewood Campus. The 3100 square foot room is covered with a raised floor fed with cold air from seven Liebert air conditioners, and a dedicated chilled water line is available for water-cooled racks. Bloomberg 156 supports a steady load of at least 450kVA, with potential expansion to 750kVA. To ensure a stable environment for data repositories, 150kVA of power has both battery and generator backup. The grant also upgraded the network infrastructure supporting the space from 1GigE to 10GigE to insure that users throughout campus can access the data center effectively and that data can be streamed to and from the outside world through Internet2.  This network infrastructure includes a Cisco Nexus 7000 chassis that accommodate the 100GigE uplink to Internet 2. The room has collocated the GPU cluster, HHPC, and Data-Scope. This has created a tightly coupled, heterogeneous clusters with compute, data, and GPU components, allowing computational science to be done in new ways. The center serves as a focal point for interdisciplinary activities in computational science and engineering.

% paragraph paragraph_name (end)

\para{{Data-Scope}} % (fold)\label{par:paragraph_name}
% paragraph paragraph_name (end)
The NSF has recently awarded a \$2M MRI grant (OCI MRI-1040114, co-PI Burns) to build a 6.5PB cluster for data-intensive computations. The system components have arrived and are being commissioned. The system will have an aggregate sequential throughput in excess of 500GBps, and will also contain 90 GPU cards, providing a substantial floating point processing capability. The Data-Scope will be connected entirely through 10Gbps Ethernet. There is an ongoing NSF-funded effort to bring 100G connectivity to JHU, through a collaboration with the Mid-Atlantic Crossroads (MAX). The system components are currently evaluated and tested, and deployment is expected to happen in June.

The driving goal behind the Data-Scope design is to maximize stream processing throughput over TB- size datasets while using commodity components to keep acquisition and maintenance costs low. Performing the first pass over the data directly on the servers' PCIe backplane is significantly faster than serving the data from a shared network file server to multiple compute servers. This first pass commonly reduces the data significantly, allowing one to share the results over the network without losing performance. Furthermore, providing substantial GPU capabilities on the same server enables us to avoid moving too much data across the network as it would be done if the GPUs were in a separate cluster.

\begin{figure}[h!]
\centering
\includegraphics[width=1.0\linewidth]{../JHU-Proposal/JHU-Figs/datascope.pdf}
\caption[Datascope.]{Schematic illustration of our DataScope compute cluster.}
\label{fig:orgchat}
\end{figure}

\para{{Open Connectome Project Mini-Cluster}} % (fold)\label{par:paragraph_name}
%
% \jovo{replace this with bruster description from greg}
By virtue of a Dean's graph to Professor Burns, Professor Burns and Dr.~Vogelstein have established an Open Connectome Project Mini-Cluster.  This cluster contains four ``braincrunch'' machines, each 4-core development boxes, along with two ``braingraph'' machines, each currently serving about 20TB of brain imaging data, and finally one ``awesome'' machine, which has 500GB of RAM and 16 128GB solid state hard drives.  These machines are all actively used already for this project by PI Vogelstein and co-I Burns.

% subsection computing (end)

\para{Center for Imaging Science Computing Resources}
%
The Center for Imaging Science (CIS) offers extensive computing resources. The major computational machine within CIS is the Intel Itanium2 cluster, which is a 32 processor cluster that is part of the TeraGrid, which includes over 20 teraflops of computing power distributed at 9 sites, facilities capable of managing and storing over 1 PB of data and high-resolution visualization environments. The CIS infrastructure also includes two 32GB/8cpu and one 128GB/16cpu computational servers; 120TB of data storage; three tape libraries with a backup capacity of over 210TB; and over 35 visualization workstations.

\para{Institute for Computational Medicine Computing Resources}
%
The Institute for Computational Medicine (ICM) offers additional computing resources, including 250 node, 2000 core, IBM iDataPlex cluster attached to a 1 Petabyte Storage Area Network, and an IBM TotalStorage 3584 UltraScalable tape library with 4 LTO4 drives and 250 slots.


\subsection{ARIMAX}


\section{Likelihood functions}


\section{VIPS packages}
\mylabel{sec:packages}

\subsection{Arithmetic}

See \fref{fg:arithmetic}.

Arithmetic functions work on images as if each band element were a separate
number. All operations are point-to-point --- each output element depends
exactly upon the corresponding input element. All (except in a few cases
noted in the manual pages) will work with images of any type (or any mixture
of types), of any size and of any number of bands.

Arithmetic operations try to preserve precision by increasing the number of
bits in the output image when necessary. Generally, this follows the ANSI C
conventions for type promotion --- so multiplying two \verb+IM_BANDFMT_UCHAR+
images together, for example, produces a \verb+IM_BANDFMT_USHORT+ image, and
taking the \verb+im_costra()+ of a \verb+IM_BANDFMT_USHORT+ image produces
a \verb+IM_BANDFMT_FLOAT+ image. The details of the type conversions are
in the manual pages.

\begin{fig2}
\begin{verbatim}
$ vips --list arithmetic
im_abs               - absolute value
im_acostra           - acos of image (result in degrees)
im_add               - add two images
im_asintra           - asin of image (result in degrees)
im_atantra           - atan of image (result in degrees)
im_avg               - average value of image
im_point_bilinear    - interpolate value at single point, linearly
im_bandmean          - average image bands
im_ceil              - round to smallest integal value not less than
im_cmulnorm          - multiply two complex images, normalising output
im_costra            - cos of image (angles in degrees)
im_cross_phase       - phase of cross power spectrum of two complex images
im_deviate           - standard deviation of image
im_divide            - divide two images
im_exp10tra          - 10^pel of image
im_expntra           - x^pel of image
im_expntra_vec       - [x,y,z]^pel of image
im_exptra            - e^pel of image
im_fav4              - average of 4 images
im_floor             - round to largest integal value not greater than
im_gadd              - calculate a*in1 + b*in2 + c = outfile
im_invert            - photographic negative
im_lintra            - calculate a*in + b = outfile
im_linreg            - pixelwise linear regression
im_lintra_vec        - calculate a*in + b -> out, a and b vectors
im_litecor           - calculate max(white)*factor*(in/white), if clip == 1
im_log10tra          - log10 of image
im_logtra            - ln of image
im_max               - maximum value of image
im_maxpos            - position of maximum value of image
im_maxpos_avg        - position of maximum value of image, averaging in case of draw
im_maxpos_vec        - position and value of n maxima of image
im_measure           - measure averages of a grid of patches
im_min               - minimum value of image
im_minpos            - position of minimum value of image
im_minpos_vec        - position and value of n minima of image
im_multiply          - multiply two images
im_powtra            - pel^x ofbuildimage
im_powtra_vec        - pel^[x,y,z] of image
im_remainder         - remainder after integer division
im_remainderconst    - remainder after integer division by a constant
im_remainderconst_vec - remainder after integer division by a vector of constants
im_rint              - round to nearest integal value
im_sign              - unit vector in direction of value
im_sintra            - sin of image (angles in degrees)
im_stats             - many image statistics in one pass
im_subtract          - subtract two images
im_tantra            - tan of image (angles in degrees)
\end{verbatim}
\caption{Arithmetic functions}
\label{fg:arithmetic}
\end{fig2}

\subsection{Relational}

See \fref{fg:relational}.

Relational functions compare images to other images or to constants. They
accept any image or pair of images (provided they are the same size and
have the same number of bands --- their types may differ) and produce a
\verb+IM_BANDFMT_UCHAR+ image with the same number of bands as the input
image, with 255 in every band element for which the condition is true and
0 elsewhere.

They may be combined with the boolean functions to form complex relational
conditions. Use \verb+im_max()+ (or \verb+im_min()+) to find out if a
condition is true (or false) for a whole image.

\begin{fig2}
\begin{verbatim}
$ vips --list relational
im_blend             - use cond image to blend between images in1 and in2
im_equal             - two images equal in value
im_equal_vec         - image equals doublevec
im_equalconst        - image equals const
im_ifthenelse        - use cond image to choose pels from image in1 or in2
im_less              - in1 less than in2 in value
im_less_vec          - in less than doublevec
im_lessconst         - in less than const
im_lesseq            - in1 less than or equal to in2 in value
im_lesseq_vec        - in less than or equal to doublevec
im_lesseqconst       - in less than or equal to const
im_more              - in1 more than in2 in value
im_more_vec          - in more than doublevec
im_moreconst         - in more than const
im_moreeq            - in1 more than or equal to in2 in value
im_moreeq_vec        - in more than or equal to doublevec
im_moreeqconst       - in more than or equal to const
im_notequal          - two images not equal in value
im_notequal_vec      - image does not equal doublevec
im_notequalconst     - image does not equal const
\end{verbatim}
\caption{Relational functions}
\label{fg:relational}
\end{fig2}

\subsection{Boolean}

See \fref{fg:boolean}.

The boolean functions perform boolean arithmetic on pairs of
\verb+IM_BANDFMT_UCHAR+ images. They are useful for combining the results of
the relational and morphological functions. You can use
\verb+im_eorconst()+ with 255 as \verb+im_not()+.

\begin{fig2}
\begin{verbatim}
$ vips --list boolean
im_andimage          - bitwise and of two images
im_andimageconst     - bitwise and of an image with a constant
im_andimage_vec      - bitwise and of an image with a vector constant
im_orimage           - bitwise or of two images
im_orimageconst      - bitwise or of an image with a constant
im_orimage_vec       - bitwise or of an image with a vector constant
im_eorimage          - bitwise eor of two images
im_eorimageconst     - bitwise eor of an image with a constant
im_eorimage_vec      - bitwise eor of an image with a vector constant
im_shiftleft         - shift integer image n bits to left
im_shiftright        - shift integer image n bits to right
\end{verbatim}
\caption{Boolean functions}
\label{fg:boolean}
\end{fig2}

\subsection{Colour}
\label{sec:colour}

See \fref{fg:colour}.

The colour functions can be divided into two main types. First, functions to
transform images between the different colour spaces supported by VIPS:
\verb+RGB+ (also referred to as \verb+disp+), \verb+sRGB+,  \verb+XYZ+,
\verb+Yxy+, \verb+Lab+, \verb+LabQ+, \verb+LabS+, \verb+LCh+ and
\verb+UCS+), and second, functions for calculating colour difference
metrics. Figure~\ref{fg:convert} shows how the VIPS colour spaces
interconvert.

\begin{fig2}
\figw{5in}{interconvert.png}
\caption{VIPS colour space conversion}
\label{fg:convert}
\end{fig2}

The colour spaces supported by VIPS are:

\begin{description}

\item[\texttt{LabQ}]
This is the principal VIPS colorimetric storage format. See the
man page for \verb+im_LabQ2Lab()+ for an explanation. You cannot perform
calculations on \verb+LabQ+ images. They are for storage only. Also refered
to as \verb+LABPACK+.

\item[\texttt{LabS}]
This format represents coordinates in \cielab{} space as a three-
band \verb+IM_BANDFMT_SHORT+ image, scaled to fit the full range of bits. It is
the best format for computation, being relatively compact, quick, and
accurate. Colour values expressed in this way are hard to visualise.

\item[\texttt{Lab}]
\verb+Lab+ colourspace represents \cielab{} colour values with a three-band
\verb+IM_BANDFMT_FLOAT+ image. This is the simplest format for general work: adding the
constant 50 to the L channel, for example, has the expected result.

\item[\texttt{XYZ}]
\ciexyz{} colour space represented as a three-band \verb+IM_BANDFMT_FLOAT+
image.

\item[\texttt{XYZ}]
\cieyxy{} colour space represented as a three-band \verb+IM_BANDFMT_FLOAT+
image.

\item[\texttt{RGB}]
(also refered to as \verb+disp+) This format is similar to the RGB colour
systems used in other packages. If you want to export your image to a PC,
for example, convert your colorimetric image to \verb+RGB+, then turn it
to TIFF with \verb+im_vips2tiff()+. You need to supply a structure which
characterises your display. See the manual page for \verb+im_col_XYZ2rgb()+
for hints on these guys.

VIPS also supports \verb+sRGB+. This is a version of RGB with a carefully
defined and standard conversion from XYZ. See:

\begin{verbatim}
http://www.color.org/
\end{verbatim}

\item[\texttt{LCh}]
Like \verb+Lab+, but rectangular $ab$ coordinates are replaced with polar $Ch$
(Chroma and hue) coordinates. Hue angles are expressed in degrees.

\item[\texttt{UCS}]
A colour space based on the CMC(1:1) colour difference measurement. This
is a highly uniform colour space, much better than \cielab{} for expressing
small differences. Conversions to and from \verb+UCS+ are extremely slow.

\end{description}

All VIPS colourspaces assume a D65 illuminant.

The colour-difference functions calculate either $\Delta{}E$ \cielab{} (1976
or 2000) or $\Delta{}E$ CMC(1:1) on two images in \verb+Lab+, \verb+XYZ+
or \verb+disp+ colour space.

\begin{fig2}
\begin{verbatim}
$ vips --list colour
im_LCh2Lab           - convert LCh to Lab
im_LCh2UCS           - convert LCh to UCS
im_Lab2LCh           - convert Lab to LCh
im_Lab2LabQ          - convert Lab to LabQ
im_Lab2LabS          - convert Lab to LabS
im_Lab2UCS           - convert Lab to UCS
im_Lab2XYZ           - convert D65 Lab to XYZ
im_Lab2XYZ_temp      - convert Lab to XYZ, with a specified colour temperature
im_Lab2disp          - convert Lab to displayable
im_LabQ2LabS         - convert LabQ to LabS
im_LabQ2Lab          - convert LabQ to Lab
im_LabQ2XYZ          - convert LabQ to XYZ
im_LabQ2disp         - convert LabQ to displayable
im_LabS2LabQ         - convert LabS to LabQ
im_LabS2Lab          - convert LabS to Lab
im_UCS2LCh           - convert UCS to LCh
im_UCS2Lab           - convert UCS to Lab
im_UCS2XYZ           - convert UCS to XYZ
im_XYZ2Lab           - convert D65 XYZ to Lab
im_XYZ2Lab_temp      - convert XYZ to Lab, with a specified colour temperature
im_XYZ2UCS           - convert XYZ to UCS
im_XYZ2Yxy           - convert XYZ to Yxy
im_XYZ2disp          - convert XYZ to displayble
im_XYZ2sRGB          - convert XYZ to sRGB
im_Yxy2XYZ           - convert Yxy to XYZ
im_dE00_fromLab      - calculate delta-E CIE2000 for two Lab images
im_dECMC_fromLab     - calculate delta-E CMC(1:1) for two Lab images
im_dECMC_fromdisp    - calculate delta-E CMC(1:1) for two displayable images
im_dE_fromLab        - calculate delta-E for two Lab images
im_dE_fromXYZ        - calculate delta-E for two XYZ images
im_dE_fromdisp       - calculate delta-E for two displayable images
im_disp2Lab          - convert displayable to Lab
im_disp2XYZ          - convert displayable to XYZ
im_float2rad         - convert float to Radiance packed
im_icc_ac2rc         - convert LAB from AC to RC using an ICC profile
im_icc_export        - convert a float LAB to an 8-bit device image with an ICC profile
im_icc_export_depth  - convert a float LAB to device space with an ICC profile
im_icc_import        - convert a device image to float LAB with an ICC profile
im_icc_import_embedded - convert a device image to float LAB using the embedded profile
im_icc_present       - test for presence of ICC library
im_icc_transform     - convert between two device images with a pair of ICC profiles
im_lab_morph         - morph colourspace of a LAB image
im_rad2float         - convert Radiance packed to float
im_sRGB2XYZ          - convert sRGB to XYZ
\end{verbatim}
\caption{Colour functions}
\label{fg:colour}
\end{fig2}

\subsection{Conversion}

See \fref{fg:conversion}.

These functions may be split into three broad groups: functions which convert
between the VIPS numeric formats (\verb+im_clip2fmt()+, for example, converts
an image of any type to the specified \verb+IM_BANDFMT+), functions
supporting complex arithmetic (\verb+im_c2amph()+, for example,  converts
a complex image from rectangular to polar co ordinates) and functions
which perform some simple geometric conversion (\verb+im_extract()+ forms
a sub-image).

\verb+gbandjoin+ and the C function \verb+im_gbandjoin()+ will do a bandwise
join of many images at the same time. See the manual pages.

\begin{fig2}
\begin{verbatim}
$ vips --list conversion
im_bandjoin          - bandwise join of two images
im_bernd             - extract from pyramid as jpeg
im_black             - generate black image
im_c2amph            - convert real and imaginary to phase and amplitude
im_c2imag            - extract imaginary part of complex image
im_c2ps              - find power spectrum of complex image
im_c2real            - extract real part of complex image
im_c2rect            - convert phase and amplitude to real and imaginary
im_clip2c            - convert to signed 8-bit integer
im_clip2cm           - convert to complex
im_clip2d            - convert to double-precision float
im_clip2dcm          - convert to double complex
im_clip2f            - convert to single-precision float
im_clip2fmt          - convert image format to ofmt
im_clip2i            - convert to signed 32-bit integer
im_clip2s            - convert to signed 16-bit integer
im_clip2ui           - convert to unsigned 32-bit integer
im_clip2us           - convert to unsigned 16-bit integer
im_clip              - convert to unsigned 8-bit integer
im_copy              - copy image
im_copy_morph        - copy image, setting pixel layout
im_copy_swap         - copy image, swapping byte order
im_copy_set          - copy image, setting informational fields
im_copy_set_meta     - copy image, setting a meta field
im_extract_area      - extract area
im_extract_areabands - extract area and bands
im_extract_band      - extract band
im_extract_bands     - extract several bands
im_extract           - extract area/band
im_falsecolour       - turn luminance changes into chrominance changes
im_fliphor           - flip image left-right
im_flipver           - flip image top-bottom
im_gbandjoin         - bandwise join of many images
im_grid              - chop a tall thin image into a grid of images
im_insert            - insert sub-image into main image at position
im_insert_noexpand   - insert sub-image into main image at position, no expansion
im_lrjoin            - join two images left-right
im_mask2vips         - convert DOUBLEMASK to VIPS image
im_msb               - convert to uchar by discarding bits
im_msb_band          - convert to single band uchar by discarding bits
im_print             - print string to stdout
im_recomb            - linear recombination with mask
im_replicate         - replicate an image horizontally and vertically
im_ri2c              - join two non-complex images to form complex
\end{verbatim}
\caption{Conversion functions}
\label{fg:conversion}
\end{fig2}

\begin{fig2}
\begin{verbatim}
im_rot180            - rotate image 180 degrees
im_rot270            - rotate image 270 degrees clockwise
im_rot90             - rotate image 90 degrees clockwise
im_scale             - scale image linearly to fit range 0-255
im_scaleps           - logarithmic scale of image to fit range 0-255
im_rightshift_size   - decrease size by a power-of-two factor
im_slice             - slice an image using two thresholds
im_subsample         - subsample image by integer factors
im_system            - run command on image
im_tbjoin            - join two images top-bottom
im_text              - generate text image
im_thresh            - slice an image at a threshold
im_vips2mask         - convert VIPS image to DOUBLEMASK
im_wrap              - shift image origin, wrapping at sides
im_zoom              - simple zoom of an image by integer factors
\end{verbatim}
\caption{Conversion functions (cont.)}
\end{fig2}

\subsection{Matricies}

See \fref{fg:matricies}.

VIPS uses matricies for morphological operations, for convolutions, and
for some colour-space conversions. There are two types of matrix: integer
(\verb+INTMASK+) and double precision floating point (\verb+DOUBLEMASK+).

For convenience, both types are stored in files as ASCII. The first
line of the file should start with the matrix dimensions, width first,
then on the same line an optional scale and offset. The two size fields
should be integers; the scale and offset may be floats. Subsequent lines
should contain the matrix elements, one row per line.  The scale and
offset are the conventional ones used to represent non-integer values in
convolution masks --- in other words:

\[
result = {value \over scale} + offset
\]

If the scale and offset are missing, they default to 1.0 and 0.0.  See the
sections on convolution for more on the use of these fields. So as an example,
a 4 by 4 identity matrix would be stored as:

\begin{verbatim}
4 4
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
\end{verbatim}

And a 3 by 3 mask for block averaging with convolution might be stored as:

\begin{verbatim}
3 3 9 0
1 1 1
1 1 1
1 1 1
\end{verbatim}

\noindent
(in other words, sum all the pels in every 3 by 3 area, and divide by 9).

This matrix contains only integer elements and so could be used as an
argument to functions expecting both \verb+INTMASK+ and \verb+DOUBLEMASK+
matricies. However, masks containing floating-point values (such as the
output of \verb+im_matinv()+) can only be used as arguments to functions
expecting \verb+DOUBLEMASK+s.

A set of functions for mask input and output are also available for
C-programmers --- see the manual pages for \verb+im_read_dmask()+. For
other matrix functions, see also the convolution sections and the arithmetic
sections.

\begin{fig2}
\begin{verbatim}
$ vips --list matrix
im_matcat          - append matrix in2 to the end of matrix in1
im_matinv          - invert matrix
im_matmul          - multiply matrix in1 by matrix in2
im_mattrn          - transpose matrix
\end{verbatim}
\caption{Matrix functions}
\label{fg:matricies}
\end{fig2}

\subsection{Convolution}

See \fref{fg:convolution}.

The functions available in the convolution package can be split into five
main groups.

First, are the convolution functions. The most useful function is
\verb+im_conv()+ which will convolve any non-complex type with an
\verb+INTMASK+ matrix. The output image will have the same size, type, and
number of bands as the input image. Of the other \verb+im_conv()+ functions,
functions whose name ends in \verb+_raw+ do not add a black border around the
output image, functions ending in \verb+f+ use a \verb+DOUBLEMASK+ matrix
and write float (or double) output, and functions containing \verb+sep+
are for seperable convolutions.  \verb+im_compass()+, \verb+im_lindetect()+
and \verb+im_gradient()+ convolve with rotating masks.  \verb+im_embed()+
is used by the convolution functions to add the border to the output.

Next, are the build functions. \verb+im_gauss_*mask()+ and its ilk
generate gaussian masks, \verb+im_log_*mask()+ generate logs of Laplacians.
\verb+im_addgnoise()+ and \verb+im_gaussnoise()+ create or add gaussian
noise to an image.

Two functions do correlation: \verb+im_fastcor()+ does a quick and dirty
correlation, \verb+im_spcor()+ calculates true spatial correlation, and is
rather slow.

Some functions are provided for analysing images: \verb+im_zerox()+ counts
zero-crossing points in an image, \verb+im_mpercent()+ finds a threshold
that will isolate a percentage of points in an image.

Finally, \verb+im_resize_linear()+ and \verb+im_shrink()+ do as you would
expect.

\begin{fig2}
\begin{verbatim}
$ vips --list convolution
im_addgnoise         - add gaussian noise with mean 0 and std. dev. sigma
im_compass           - convolve with 8-way rotating integer mask
im_contrast_surface  - find high-contrast points in an image
im_contrast_surface_raw - find high-contrast points in an image
im_conv              - convolve
im_conv_raw          - convolve, no border
im_convf             - convolve, with DOUBLEMASK
im_convf_raw         - convolve, with DOUBLEMASK, no border
im_convsep           - seperable convolution
im_convsep_raw       - seperable convolution, no border
im_convsepf          - seperable convolution, with DOUBLEMASK
im_convsepf_raw      - seperable convolution, with DOUBLEMASK, no border
im_convsub           - convolve uchar to uchar, sub-sampling by xskip, yskip
im_dmask_xsize       - horizontal size of a doublemask
im_dmask_ysize       - vertical size of a doublemask
im_embed             - embed in within a set of borders
im_fastcor           - fast correlate in2 within in1
im_fastcor_raw       - fast correlate in2 within in1, no border
im_gauss_dmask       - generate gaussian DOUBLEMASK
im_gauss_imask       - generate gaussian INTMASK
im_gauss_imask_sep   - generate separable gaussian INTMASK
im_gaussnoise        - generate image of gaussian noise with specified statistics
im_grad_x            - horizontal difference image
im_grad_y            - vertical difference image
im_gradcor           - non-normalised correlation of gradient of in2 within in1
im_gradcor_raw       - non-normalised correlation of gradient of in2 within in1, no padding
im_gradient          - convolve with 2-way rotating mask
im_imask_xsize       - horizontal size of an intmask
im_imask_ysize       - vertical size of an intmask
im_rank_image        - point-wise pixel rank
im_lindetect         - convolve with 4-way rotating mask
im_log_dmask         - generate laplacian of gaussian DOUBLEMASK
im_log_imask         - generate laplacian of gaussian INTMASK
im_maxvalue          - point-wise maximum value
im_mpercent          - find threshold above which there are percent values
im_phasecor_fft      - non-normalised correlation of gradient of in2 within in1
im_rank              - rank filter nth element of xsize/ysize window
im_rank_raw          - rank filter nth element of xsize/ysize window, no border
im_read_dmask        - read matrix of double from file
im_resize_linear     - resize to X by Y pixels with linear interpolation
im_rotate_dmask45    - rotate DOUBLEMASK clockwise by 45 degrees
im_rotate_dmask90    - rotate DOUBLEMASK clockwise by 90 degrees
im_rotate_imask45    - rotate INTMASK clockwise by 45 degrees
im_rotate_imask90    - rotate INTMASK clockwise by 90 degrees
im_sharpen           - sharpen high frequencies of L channel of LabQ
im_shrink            - shrink image by xfac, yfac times
im_spcor             - normalised correlation of in2 within in1
im_spcor_raw         - normalised correlation of in2 within in1, no black padding
im_stretch3          - stretch 3%, sub-pixel displace by xdisp/ydisp
im_zerox             - find +ve or -ve zero crossings in image
\end{verbatim}
\caption{Convolution functions}
\label{fg:convolution}
\end{fig2}

\subsection{In-place operations}
\label{sec:inplace}

See \fref{fg:inplace}.

A few of the in-place operations are available from the command-line. Most are
not.

\begin{fig2}
\begin{verbatim}
$ vips --list inplace
im_circle          - plot circle on image
im_flood_blob_copy - flood while pixel == start pixel
im_insertplace     - draw image sub inside image main at position (x,y)
im_line            - draw line between points (x1,y1) and (x2,y2)
im_lineset         - draw line between points (x1,y1) and (x2,y2)
\end{verbatim}
\caption{In-place operations}
\label{fg:inplace}
\end{fig2}

\subsection{Frequency filtering}

See \fref{fg:freq}.

The basic Fourier functions are \verb+im_fwfft()+ and
\verb+im_invfft()+, which calculate the fast-fourier transform and inverse
transform of an image. Also \verb+im_invfftr()+, which just returns the real
part of the inverse transform.
The Fourier image has its origin at pel (0,0) ---
for viewing, use \verb+im_rotquad()+ to move the origin to the centre of
the image.

Once an image is in the frequency domain, it can be filtered by multiplying
it with a mask image. The VIPS mask generator is \verb+im_create_fmask()+
see the manual page for details of the arguments, but it will create low
pass, high pass, ring pass and band pass filters, which may each be ideal,
Gaussian or Butterworth. There is also a fractal mask option.

The other functions in the package build on these base
facilities. \verb+im_freqflt()+ transforms an input image to
Fourier space, multiplies it by a mask image, and transforms it back
again. \verb+im_flt_image_freq()+ will create a mask image of the correct
size for you, and call \verb+im_freqflt()+.  \verb+im_disp_ps()+ will call
the right combinations of functions to make a displayable power spectrum
for an image.

\begin{fig2}
\begin{verbatim}
$ vips --list freq_filt
im_create_fmask    - create frequency domain filter mask
im_disp_ps         - make displayable power spectrum
im_flt_image_freq  - frequency domain filter image
im_fractsurf       - generate a fractal surface of given dimension
im_freqflt         - frequency-domain filter of in with mask
im_fwfft           - forward fast-fourier transform
im_rotquad         - rotate image quadrants to move origin to centre
im_invfft          - inverse fast-fourier transform
im_invfftr         - real part of inverse fast-fourier transform
\end{verbatim}
\caption{Fourier functions}
\label{fg:freq}
\end{fig2}

\subsection{Histograms and LUTs}

See \fref{fg:hist}.

VIPS represents histograms and look-up tables in the same way --- as images.

They should have either \verb+Xsize+ or \verb+Ysize+ set to 1, and the
other dimension set to the number of elements in the table. The table can be
of any size, have any band format, and have any number of bands.

Use \verb+im_histgr()+ to find the histogram of an image.  Use
\verb+im_histnD()+ to find the n-dimensional histogram of an n-band
image.  Perform operations on histograms with \verb+im_histcum()+,
\verb+im_histnorm()+, \verb+im_histspec()+, \verb+im_invertlut()+. Visualise
histograms with \verb+im_histplot()+. Use a histogram (or LUT) to transform
an image with \verb+im_maplut()+. Build a histogram from scratch with
\verb+im_identity()+ or \verb+im_identity_ushort()+.

Use \verb+im_lhist*()+ for local histogram equalisation, and
\verb+im_stdif*()+ for statisticaol differencing. The \verb+im_tone_*()+
functions are for operations on the L channel of a LAB image. Other
functions are useful combinations of these basic operations.

\begin{fig2}
\begin{verbatim}
$ vips --list histograms_lut
im_gammacorrect      - gamma-correct image
im_heq               - histogram-equalise image
im_hist              - find and graph histogram of image
im_histcum           - turn histogram to cumulative histogram
im_histeq            - form histogram equalistion LUT
im_histgr            - find histogram of image
im_histnD            - find 1D, 2D or 3D histogram of image
im_histnorm          - form normalised histogram
im_histplot          - plot graph of histogram
im_histspec          - find histogram which will make pdf of in match ref
im_hsp               - match stats of in to stats of ref
im_identity          - generate identity histogram
im_identity_ushort   - generate ushort identity histogram
im_ismonotonic       - test LUT for monotonicity
im_lhisteq           - local histogram equalisation
im_lhisteq_raw       - local histogram equalisation, no border
im_invertlut         - generate correction table from set of measures
im_buildlut          - generate LUT table from set of x/y positions
im_maplut            - map image through LUT
im_project           - find horizontal and vertical projections of an image
im_stdif             - statistical differencing
im_stdif_raw         - statistical differencing, no border
im_tone_analyse      - analyse in and create LUT for tone adjustment
im_tone_build        - create LUT for tone adjustment of LabS images
im_tone_build_range  - create LUT for tone adjustment
im_tone_map          - map L channel of LabS or LabQ image through LUT
\end{verbatim}
\caption{Histogram/LUT functions}
\label{fg:hist}
\end{fig2}

\subsection{Morphology}

See \fref{fg:morph}.

The morphological functions are used on one-band \verb+IM_BANDFMT_UCHAR+ binary
images (images containing only zero and not-zero). They search images
for particular patterns of pixels (specified with the mask argument),
either adding or removing pixels when they find a match. They are useful
for cleaning up images --- for example, you might threshold an image, and
then use one of the morphological functions to remove all single isolated
pixels from the result.

If you combine the morphological operators with the mask rotators
(\verb+im_rotate_imask45()+, for example) and apply them repeatedly, you
can achieve very complicated effects: you can thin, prune, fill, open edges,
close gaps, and many others. For example, see `Fundamentals  of  Digital
Image Processing' by A.  Jain, pp 384-388, Prentice-Hall, 1989 for more ideas.

Beware that VIPS reverses the usual image processing convention, by assuming
white objects on a black background.

The mask you give to the morphological functions should contain only the
values 0 (for background), 128 (for don't care) and 255 (for object). The
mask must have odd length sides --- the origin of the mask is taken to be
the centre value. For example, the mask:

\begin{verbatim}
3 3
128 255 128
255 0   255
128 255 128
\end{verbatim}

\noindent
applied to an image with \verb+im_erode()+, will find all black pixels
4-way connected with white pixels. Essentially, \verb+im_dilate()+
sets  pixels  in the output  if  any part of the mask matches, whereas
\verb+im_erode()+ sets pixels only if all of the mask matches.

The \verb+_raw()+ version of the functions do not add a black border to the
output. \verb+im_cntlines()+ and \verb+im_profile+ are occasionally useful for
analysing results.

See the boolean operations \verb+im_and()+, \verb+im_or()+  and
\verb+im_eor()+ for  analogues of  the  usual  set difference and set
union operations.

\begin{fig2}
\begin{verbatim}
$ vips --list morphology
im_cntlines        - count horizontal or vertical lines
im_dilate          - dilate image with mask, adding a black border
im_dilate_raw      - dilate image with mask
im_erode           - erode image with mask, adding a black border
im_erode_raw       - erode image with mask
im_profile         - find first horizontal/vertical edge
\end{verbatim}
\caption{Morphological functions}
\label{fg:morph}
\end{fig2}

\subsection{Mosaicing}

See \fref{fg:mosaicing}.

These functions are useful for joining many small images together to make one
large image. They can cope with unstable contrast, and arbitary sub-image
layout, but will not do any geometric correction. The mosaicing functions
can be grouped into layers:

The lowest level functions are \verb+im_correl()+. and \verb+im_affine()+.
\verb+im_correl()+  searches a large image for a small sub-image, returning
the position of the best sub-image match.  \verb+im_affine()+ performs
a general affine transform on an image: that is, any transform in which
parallel lines remain parallel.

Next, \verb+im_lrmerge()+ and \verb+im_tbmerge()+ blend two images together
left-right or up-down.

Next up are  \verb+im_lrmosaic()+ and \verb+im_tbmosaic()+. These use the
two low-level merge operations to join two images given just an approximate
overlap as a start point. Optional extra parameters let you do 'balancing'
too: if your images have come from a source where there is no precise
control over the exposure (for example, images from a tube camera, or a
set of images scanned from photographic sources), \verb+im_lrmosaic()+
and \verb+im_tbmosaic()+ will adjust the contrast of the left image to
match the right, the right to the left, or both to some middle value.

The functions \verb+im_lrmosaic1()+ and \verb+im_tbmosaic1()+ are first-order
analogues of the basic mosaic functions: they take two tie-points and use
them to rotate and scale the right-hand or bottom image before starting to join.

Finally, \verb+im_global_balance()+ can be used to re-balance a mosaic
which has been assembled with these functions. It will generally do a
better job than the low-level balancer built into \verb+im_lrmosaic()+
and \verb+im_tbmosaic()+. See the man page. \verb+im_remosaic()+ uses the same
techniques, but will reassemble the image from a different set of source
images.

\begin{fig2}
\begin{verbatim}
$ vips --list mosaicing
im_align_bands       - align the bands of an image
im_correl            - search area around sec for match for area around ref
im__find_lroverlap   - search for left-right overlap of ref and sec
im__find_tboverlap   - search for top-bottom overlap of ref and sec
im_global_balance    - automatically rebuild mosaic with balancing
im_global_balancef   - automatically rebuild mosaic with balancing, float output
im_lrmerge           - left-right merge of in1 and in2
im_lrmerge1          - first-order left-right merge of ref and sec
im_lrmosaic          - left-right mosaic of ref and sec
im_lrmosaic1         - first-order left-right mosaic of ref and sec
im_match_linear      - resample ref so that tie-points match
im_match_linear_search - search sec, then resample so that tie-points match
im_maxpos_subpel     - subpixel position of maximum of (phase correlation) image
im_remosaic          - automatically rebuild mosaic with new files
im_tbmerge           - top-bottom merge of in1 and in2
im_tbmerge1          - first-order top-bottom merge of in1 and in2
im_tbmosaic          - top-bottom mosaic of in1 and in2
im_tbmosaic1         - first-order top-bottom mosaic of ref and sec
\end{verbatim}
\caption{Mosaic functions}
\label{fg:mosaicing}
\end{fig2}

\subsection{CImg functions}

See \fref{fg:cimg}.

These operations wrap the anisotropic blur function from the CImg library.
They are useful for removing noise from images.

\begin{fig2}
\begin{verbatim}
$ vips --list cimg
im_greyc           - noise-removing filter
im_greyc_mask      - noise-removing filter, with a mask
\end{verbatim}
\caption{CImg functions}
\label{fg:cimg}
\end{fig2}

\subsection{Other}

See \fref{fg:other}.

These functions generate various test images. You can combine them with
the arithmetic and rotate functions to build more complicated images.

The \verb+im_benchmark*()+ operations are for testing the VIPS SMP system.

\begin{fig2}
\begin{verbatim}
$ vips --list other
im_benchmark       - do something complicated for testing
im_benchmark2      - do something complicated for testing
im_benchmarkn      - do something complicated for testing
im_eye             - generate IM_BANDFMT_UCHAR [0,255] frequency/amplitude image
im_grey            - generate IM_BANDFMT_UCHAR [0,255] grey scale image
im_feye            - generate IM_BANDFMT_FLOAT [-1,1] frequency/amplitude image
im_fgrey           - generate IM_BANDFMT_FLOAT [0,1] grey scale image
im_fzone           - generate IM_BANDFMT_FLOAT [-1,1] zone plate image
im_make_xy         - generate image with pixel value equal to coordinate
im_zone            - generate IM_BANDFMT_UCHAR [0,255] zone plate image
\end{verbatim}
\caption{Other functions}
\label{fg:other}
\end{fig2}

\subsection{IO functions}

See \fref{fg:io}.

These functions are related to the image IO system.

\begin{fig2}
\begin{verbatim}
$ vips --list iofuncs
im_binfile         - open a headerless binary file
im_cache           - cache results of an operation
im_guess_prefix    - guess install area
im_guess_libdir    - guess library area
im_header_get_type - return field type
im_header_int      - extract int fields from header
im_header_double   - extract double fields from header
im_header_string   - extract string fields from header
im_version         - VIPS version number
im_version_string  - VIPS version string
\end{verbatim}
\caption{IO functions}
\label{fg:io}
\end{fig2}

\subsection{Format functions}

See \fref{fg:format}.

These functions convert to and from various image formats. See
\pref{sec:format} for a nice API over these. VIPS can read more than these
formats, see the man page for \verb+VipsFormat+.

\begin{fig2}
\begin{verbatim}
$ vips --list format
im_csv2vips          - read a file in csv format
im_jpeg2vips         - convert from jpeg
im_magick2vips       - load file with libMagick
im_png2vips          - convert PNG file to VIPS image
im_exr2vips          - convert an OpenEXR file to VIPS
im_ppm2vips          - read a file in pbm/pgm/ppm format
im_analyze2vips      - read a file in analyze format
im_tiff2vips         - convert TIFF file to VIPS image
im_vips2csv          - write an image in csv format
im_vips2jpeg         - convert to jpeg
im_vips2mimejpeg     - convert to jpeg as mime type on stdout
im_vips2png          - convert VIPS image to PNG file
im_vips2ppm          - write a file in pbm/pgm/ppm format
im_vips2tiff         - convert VIPS image to TIFF file
\end{verbatim}
\caption{Format functions}
\label{fg:format}
\end{fig2}

\subsection{Resample functions}

See \fref{fg:resample}.

These functions resample images with various interpolators.

\begin{fig2}
\begin{verbatim}
$ vips --list resample
im_affine            - affine transform
im_affinei           - affine transform
im_affinei_all       - affine transform of whole image
im_similarity_area   - output area xywh of similarity transformation
im_similarity        - similarity transformation
\end{verbatim}
\caption{Resample functions}
\label{fg:resample}
\end{fig2}