\documentclass{article}

%%% Automatic generation of tutorial examples
%%% TODO: cd web ; ../doc/ToulBar2.sh > ! ../doc/ToulBar2.tex

\usepackage[utf8]{inputenc}
\usepackage{hyperref}
\usepackage{doxygen}
%%\usepackage{underscore}

\def\ci{\perp\!\!\!\perp}

\def\toulbar2{\textsc{toulbar2}}
\def\toolbar{\textsc{toolbar}}
\title{\toulbar2 User documentation}
\author{The \toulbar2 developer team}
\date{\today}

\begin{document}
\maketitle
\section{What is \toulbar2}

\toulbar2 is an exact black box discrete optimization solver targeted
at solving cost function networks (CFN), thus solving the so-called
``weighted Constraint Satisfaction Problem'' or WCSP. Cost function
networks can be simply described by a set of discrete variables each
having a specific finite domain and a set of integer cost functions,
each involving some of the variables. The WCSP is to find an
assignment of all variables such that the sum of all cost functions is
minimum and lest than a given upper bound often denoted as $k$ or
$\top$. Functions can be typically specified by sparse or full tables
but also more concisely as specific functions called ``global cost
functions''~\cite{Schiex16a}.

Using on the fly translation, \toulbar2 can also directly solve
optimization problems on other graphical models such as Maximum
probability Explanation (MPE) on Bayesian networks~\cite{koller2009}, and
Maximum A Posteriori (MAP) on Markov random field~\cite{koller2009}. It can also
read partial weighted MaxSAT problems, Quadratic Pseudo Boolean
problems (MAXCUT) as well as Linkage \texttt{.pre} pedigree files for
genotyping error detection and correction.

\toulbar2 is exact. It will only report an optimal solution when it
has both identified the solution and proved its optimality. Because it
relies only on integer operations, addition and subtraction, it does
not suffer from rounding errors. In the general case, the WCSP,
MPE/BN, MAP/MRF, PWMaxSAT, QPBO or MAXCUT being all NP-hard problems
and thus \toulbar2 may take exponential time to prove optimality. This
is however a worst-case behavior and \toulbar2 has been shown to be
able to solve to optimality problems with half a million non Boolean
variables defining a search space as large as $2^{829,440}$. It may
also fail to solve in reasonable time problems with a search space
smaller than $2^{264}$.

\toulbar2 provides and uses by default an ``anytime''
algorithm~\cite{Katsirelos15a} that tries to quickly provide good solutions together
with an upper bound on the gap between the cost of each solution and
the (unknown) optimal cost. Thus, even if it is unable to prove
optimality, it will bound the quality of the solution provided.
It can also apply a variable neighborhood search algorithm exploiting a problem decomposition~\cite{Ouali17}.
This algorithm is complete (if enough CPU-time is given) and it can be run in parallel using OpenMPI.

Beyond the service of providing optimal solutions, \toulbar2 can also find a greedy sequence of diverse solutions~\cite{Ruffini19a} or
exhaustively enumerate solutions below a cost threshold and
perform guaranteed approximate weighted counting of solutions. For
stochastic graphical models, this means that \toulbar2 will compute
the partition function (or the normalizing constant $Z$). These
problems being \#P-complete, \toulbar2 runtimes can quickly increase
on such problems.

By exploiting the new \toulbar2 python interface, with incremental solving capabilities, it is possible to learn a CFN from data and to combine it with mandatory constraints~\cite{Schiex20b}.
See examples at \url{https://forgemia.inra.fr/thomas.schiex/cfn-learn}.

\section{How do I install it ?}

\toulbar2 is an open source solver distributed under the MIT license as a set of C++ sources managed with git at
\url{http://github.com/toulbar2/toulbar2}. If you want
to use a released version, then you can download there source archives of a specific release
%%binary archives as a shell archive, an rpm or a Debian package
that should be easy to
compile on most Linux systems.
%% as well as an auto-installing executable for Windows.

If you want to compile the latest sources yourself, you will need a modern C++
compiler, CMake, Gnu MP Bignum library, a recent version of boost
libraries and optionally the jemalloc memory management and OpenMPI libraries. You
can then clone \toulbar2 on your machine and compile it by executing:

\begin{verbatim}
git clone https://github.com/toulbar2/toulbar2.git
cd toulbar2
mkdir build
cd build
# ccmake ..
cmake ..
make
\end{verbatim}

Finally, \toulbar2 is available in the debian-science
section of the unstable/sid Debian version. It should therefore be
directly installable using:

\begin{verbatim}
sudo apt-get install toulbar2
\end{verbatim}

If you want to try \toulbar2 on crafted, random, or real problems,
please look for benchmarks in the
\href{http://costfunction.org/en/benchmark}{Cost
  Function benchmark Section}. Other benchmarks coming from various discrete optimization languages are available at \href{http://genoweb.toulouse.inra.fr/~degivry/evalgm}{Genotoul EvalGM}~\cite{Hurley16b}.

\section{How do I test it ?}

Some problem examples are available in the directory {\sf toulbar2/validation}. After compilation with cmake, it is possible to run a series of tests using:
\begin{verbatim}
make test
\end{verbatim}

For debugging \toulbar2 (compile with flag \verb!CMAKE_BUILD_TYPE="Debug"!), more test examples are available at \href{https://forgemia.inra.fr/thomas.schiex/cost-function-library}{Cost Function Library}.
The following commands run \toulbar2 (executable must be found on your system path) on every problems with a 1-hour time limit and compare their optimum with known optima (in .ub files).
\begin{verbatim}
cd toulbar2
git clone https://forgemia.inra.fr/thomas.schiex/cost-function-library.git
./misc/script/runall.sh ./cost-function-library/trunk/validation
\end{verbatim}

Other tests on randomly generated problems can be done where optimal solutions are verified by using an older solver \href{https://forgemia.inra.fr/thomas.schiex/toolbar}{\toolbar} (executable must be found on your system path).
\begin{verbatim}
cd toulbar2
git clone https://forgemia.inra.fr/thomas.schiex/toolbar.git
cd toolbar/toolbar
make toolbar
cd ../..
./misc/script/rungenerate.sh
\end{verbatim}

\section{Using it as a black box}

Using \toulbar2 is just a matter of having a properly formatted input
file describing the cost function network, graphical model, PWMaxSAT,
PBO or Linkage \texttt{.pre} file and executing:

\begin{verbatim}
toulbar2 [option parameters] <file>
\end{verbatim}

and \toulbar2 will start solving the optimization problem described in
its file argument. By default, the extension of the file (either
\texttt{.cfn}, \texttt{.cfn.gz}, \texttt{.cfn.xz}, \texttt{.wcsp}, \texttt{.wcsp.gz}, \texttt{.wcsp.xz},
\texttt{.wcnf}, \texttt{.wcnf.gz}, \texttt{.wcnf.xz}, \texttt{.cnf}, \texttt{.cnf.gz}, \texttt{.cnf.xz}, \texttt{.qpbo}, \texttt{.qpbo.gz}, \texttt{.qpbo.xz}, \texttt{.opb}, \texttt{.opb.gz}, \texttt{.opb.xz},
\texttt{.uai}, \texttt{.uai.gz}, \texttt{.uai.xz}, \texttt{.LG}, \texttt{.LG.gz}, \texttt{.LG.xz}, \texttt{.pre} or \texttt{.bep}) is used
to determine the nature of the file (see
section~\ref{file-formats}). There is no specific order for the
options or problem file. \toulbar2 comes with decently optimized
default option parameters. It is however often possible to set it up
for different target than pure optimization or tune it for faster
action using specific command line options.

\section{Quick start}
\label{quickstart}

\input{ToulBar2.tex}

\section{Command line options}

If you just execute:

\begin{verbatim}
toulbar2
\end{verbatim}

\toulbar2 will give you its (long) list of optional parameter which we
now describe in more detail.

%If you don't known much about Constraint
%and Cost Function Programming, section~\ref{how-work} describes some
%of the inner working of \toulbar2 to help you tune it to your
%requirements.

To deactivate a default command line option, just use the
command-line option followed by ``\texttt{:}''. For example:

\begin{verbatim}
toulbar2 -dee: <file>
\end{verbatim}

will disable the default Dead End Elimination~\cite{Givry13a} (aka Soft Neighborhood
Substitutability) preprocessing.

\subsection{General control}

\begin{description}
\item[{-agap=[decimal]}] : stops search if the absolute optimality gap reduces below the given value (provides guaranteed approximation) (default value is 0)
\item[{-rgap=[double]}] : stops search if the relative optimality gap reduces below the given value (provides guaranteed approximation) (default value is 0)
\item[{-a=[integer]}] finds at most a given number of solutions with a cost strictly lower than the initial upper bound and stops,
or if no integer is given, finds all solutions (or counts the number of zero-cost satisfiable solutions in conjunction with BTD)
\item[{-D}] approximate satisfiable solution count with BTD
\item[{-logz}] computes log of probability of evidence (i.e. log
  partition function or log(Z) or PR task) for graphical models only
  (problem file extension .uai)
\item[{-timer=[integer]}] gives a CPU time limit in seconds. \toulbar2
  will stop after the specified CPU time has been consumed. The time
  limit is a CPU user time limit, not wall clock time limit.
\item[{-bt=[integer]}] gives a limit on the number of backtracks (9223372036854775807 by default)
\item[{-seed=[integer]}] random seed non-negative value or use current time if a negative value is given (default value is 1)
\end{description}

\subsection{Preprocessing}

\begin{description}
\item[{-nopre}] deactivates all preprocessing options (equivalent to -e:
  -p: -t: -f: -dec: -n: -mst: -dee: -trws:)
\item[{-p=[integer]}]preprocessing only: general variable elimination
  of degree less than or equal to the given value (default value is
  -1)
\item[{-t=[integer]}] preprocessing only: simulates restricted path
  consistency by adding ternary cost functions on triangles of binary
  cost functions within a given maximum space limit (in MB)
\item[{-f=[integer]}] preprocessing only: variable elimination of
  functional (f=1) (resp. bijective (f=2)) variables (default value is
  1)
\item[{-dec}] preprocessing only: pairwise decomposition~\cite{Favier11a} of cost
  functions with arity $>=3$ into smaller arity cost functions (default
  option)
\item[{-n=[integer]}] preprocessing only: projects n-ary cost functions
  on all binary cost functions if n is lower than the given value
  (default value is 10). See~\cite{Favier11a}.
\item[{-mst}] find a maximum spanning tree ordering for DAC
\item[{-S}] preprocessing only: performs singleton consistency (only in conjunction with option -A)
\item[{-M=[integer]}] preprocessing only:
  apply the Min Sum Diffusion algorithm (default is inactivated, with
  a number of iterations of 0). See~\cite{Cooper10a}.
\item[{-trws=[float]}]
  preprocessing only: enforces TRW-S until a given precision is reached (default value is 0.001). See Kolmogorov 2006.
\item[{--trws-order}] replaces DAC order by Kolmogorov's TRW-S order.
\item[{--trws-n-iters=[integer]}] enforce at most N iterations of TRW-S (default value is 1000).
\item[{--trws-n-iters-no-change=[integer]}] stop TRW-S when N iterations did not change the lower bound up the given precision (default value is 5, -1=never).
\item[{--trws-n-iters-compute-ub=[integer]}] compute a basic upper bound every N steps during TRW-S (default value is 100)
 \end{description}

\subsection{Initial upper bounding}

\begin{description}
\item[{-l=[integer]}] limited discrepancy search~\cite{Ginsberg95}, use a negative value to stop the search after the given absolute number of discrepancies has been explored (discrepancy bound = 4 by default)
\item[{-L=[integer]}] randomized (quasi-random variable ordering) search with restart (maximum number of nodes/VNS restarts = 10000 by default)
\item[{-i=["string"]}] initial upper bound found by INCOP local search
  solver~\cite{idwalk:cp04}. The string parameter is optional, using ``0 1 3 idwa 100000
  cv v 0 200 1 0 0'' by default with the following meaning:
  {\em stoppinglowerbound randomseed nbiterations method nbmoves
  neighborhoodchoice neighborhoodchoice2 minnbneighbors maxnbneighbors
  neighborhoodchoice3 autotuning tracemode}.
\item[{-x=[(,i[$=\#<>$]a)*]}] performs an elementary operation ('$=$':assign, '$\#$':remove, '$<$':decrease, '$>$':increase) with value a on variable of index i (multiple operations are separated by a comma and no space) (without any
  argument, a complete assignment -- used as initial upper bound and
  as a value heuristic -- read from default file "sol" taken as a certificate or given directly as an additional input
  filename with ".sol" extension and without {\bf -x})
\item[{-ub=[decimal]}] gives an initial upper bound
\item[{-rasps=[integer]}] : VAC-based upper bound probing heuristic (0: disable, >0: max. nb. of backtracks, 1000 if no integer given) (default value is 0)
\item[{-raspslds=[integer]}] : VAC-based upper bound probing heuristic using LDS instead of DFS (0: DFS, >0: max. discrepancy) (default value is 0)
\item[{-raspsdeg=[integer]}] : automatic threshold cost value selection for probing heuristic (default value is 10 degrees)
\item[{-raspsini}] : reset weighted degree variable ordering heuristic after doing upper bound probing
\end{description}

\subsection{Tree search algorithms and tree decomposition selection}

\begin{description}
\item[{-hbfs=[integer]}] hybrid best-first search~\cite{Katsirelos15a}, restarting from the
  root after a given number of backtracks (default value is 10000)
\item[{-open=[integer]}] hybrid best-first search limit on the number
  of stored open nodes (default value is -1)
\item[{-B=[integer]}] (0) DFBB, (1) BTD~\cite{Schiex06a}, (2) RDS-BTD~\cite{Sanchez09a}, (3) RDS-BTD with
  path decomposition instead of tree decomposition~\cite{Sanchez09a} (default value is
  0)
\item[{-O=[filename]}] reads either a reverse variable elimination order (given by a list of variable indexes) from a file
  in order to build a tree decomposition (if BTD-like and/or variable
  elimination methods are used) or reads a valid tree decomposition directly (given by a list of clusters in topological order of a rooted forest, each line contains a cluster number, followed by a cluster parent number with -1 for the first/root(s) cluster(s), followed by a list of variable indexes). It is also used as a DAC ordering.
\item[{-O=[negative integer]}] build a tree decomposition (if BTD-like
  and/or variable elimination methods are used) and also a compatible
  DAC ordering using
  \begin{itemize}
    \item (-1) maximum cardinality search ordering,
    \item (-2) minimum degree ordering,
    \item (-3) minimum fill-in ordering,
    \item (-4) maximum spanning tree ordering (see -mst),
    \item (-5) reverse Cuthill-Mckee ordering,
    \item (-6) approximate minimum degree ordering,
    \item (-7) default file ordering
    \end{itemize}
    If not specified, then use the variable order in which variables appear in the problem file.
\item[{-j=[integer]}] splits large clusters into a chain of smaller embedded clusters with a number of proper variables less than this number (use options "-B=3 -j=1 -svo -k=1" for pure RDS, use value 0 for no splitting) (default value is 0).
\item[{-r=[integer]}] limit on the maximum cluster separator size (merge cluster with its father otherwise, use a negative value for no limit) (default value is -1)
\item[{-X=[integer]}] limit on the minimum number of proper variables in a cluster (merge cluster with its father otherwise, use a zero for no limit) (default value is 0)
\item[{-E=[float]}] merges leaf clusters with their fathers if small local treewidth (in conjunction with option "-e" and positive threshold value) or ratio of number of separator variables by number of cluster variables above a given threshold (in conjunction with option -vns) (default value is 0)
\item[{-R=[integer]}] choice for a specific root cluster number
\item[{-I=[integer]}] choice for solving only a particular rooted cluster subtree (with RDS-BTD only)
 \end{description}

\subsection{Variable neighborhood search algorithms}

\begin{description}
\item[{-vns}] unified decomposition guided variable neighborhood search~\cite{Ouali17} (UDGVNS). A problem decomposition into clusters can be given as *.dec, *.cov, or *.order input files or using tree decomposition options such as -O. For a parallel version (UPDGVNS), use "mpirun -n [NbOfProcess] toulbar2 -vns problem.wcsp".
\item[{-vnsini=[integer]}] initial solution for VNS-like methods found: (-1) at random, (-2) min domain values, (-3) max domain values, (-4) first solution found by a complete method, (k=0 or more) tree search with k discrepancy max (-4 by default)
\item[{-ldsmin=[integer]}] minimum discrepancy for VNS-like methods (1 by default)
\item[{-ldsmax=[integer]}] maximum discrepancy for VNS-like methods (number of problem variables multiplied by maximum domain size -1 by default)
\item[{-ldsinc=[integer]}] discrepancy increment strategy for VNS-like methods using (1) Add1, (2) Mult2, (3) Luby operator (2 by default)
\item[{-kmin=[integer]}] minimum neighborhood size for VNS-like methods (4 by default)
\item[{-kmax=[integer]}] maximum neighborhood size for VNS-like methods (number of problem variables by default)
\item[{-kinc=[integer]}] neighborhood size increment strategy for VNS-like methods using: (1) Add1, (2) Mult2, (3) Luby operator (4) Add1/Jump (4 by default)
\item[{-best=[integer]}] stop VNS-like methods if a better solution is found (default value is 0)
\end{description}

\subsection{Node processing \& bounding options}

\begin{description}
\item[{-e=[integer]}] performs ``on the fly'' variable elimination of variable with small
  degree (less than or equal to a specified value, default is 3 creating a
  maximum of ternary cost functions). See~\cite{Larrosa00}.
\item[{-k=[integer]}] soft local consistency level (NC~\cite{Larrosa2002} with Strong NIC for global cost functions=0~\cite{LL2009}, (G)AC=1~\cite{Schiex00b,Larrosa2002}, D(G)AC=2~\cite{CooperFCSP}, FD(G)AC=3~\cite{Larrosa2003}, (weak) ED(G)AC=4~\cite{Heras05,LL2010}) (default value is 4). See also~\cite{Cooper10a,LL2012asa}.
\item[{-A=[integer]}] enforces VAC~\cite{Cooper08} at each search node with a search depth less than a given value (default value is 0)
\item[{-V}] VAC-based value ordering heuristic (default option)
\item[{-T=[decimal]}] : threshold cost value for VAC (default value is 1)
\item[{-P=[decimal]}] : threshold cost value for VAC during the preprocessing phase only (default value is 1)
\item[{-C=[float]}] : multiplies all costs internally by this number when loading the problem (cannot be done with cfn format and probabilistic graphical models in uai/LG formats) (default value is 1)
\item[-vacthr] automatic threshold cost value selection for VAC during search (must be combined with option -A)
\item[{-dee=[integer]}] restricted dead-end elimination~\cite{Givry13a} (value pruning by dominance rule from EAC value (dee$>=1$ and dee$<=3$)) and soft neighborhood substitutability (in preprocessing (dee=2 or dee=4) or during search (dee=3)) (default value is 1)
\item[-o] ensures an optimal worst-case time complexity of DAC and EAC (can be slower in practice)
\end{description}

\subsection{Branching, variable and value ordering}

\begin{description}
\item[-svo] searches using a static variable ordering heuristic. The
  variable order value used will be the same order as the DAC order.
\item[-b] searches using binary branching (by default) instead of n-ary branching.
  Uses binary branching for interval domains and small domains
  and dichotomic branching for large enumerated domains (see option -d).
\item[-c] searches using binary branching with last conflict
  backjumping variable ordering heuristic~\cite{Lecoutre09}.
\item[{-q=[integer]}] use
  weighted degree variable ordering heuristic~\cite{boussemart2004} if the number of cost
  functions is less than the given value (default value is 1000000).
\item[{-var=[integer]}]
  searches by branching only on the first [given value]
  decision variables, assuming the remaining variables are
  intermediate variables that will be completely assigned by the
  decision variables (use a zero if all variables are decision
  variables, default value is 0)
\item[{-m=[integer]}]
  use a variable ordering heuristic that selects first variables such
  that the sum of the mean (m=1) or median (m=2) cost of all incident
  cost functions is maximum~\cite{Schiex14a} (in conjunction with weighted degree
  heuristic -q) (default value is 0: unused).
\item[{-d=[integer]}]
  searches using dichotomic branching. The default d=1 splits domains
  in the middle of domain range while d=2 splits domains in the middle
  of the sorted domain based on unary costs.
\item[-sortd] sorts domains in preprocessing based on increasing unary costs (works only for binary WCSPs).
\item[-sortc] sorts constraints in preprocessing based on lexicographic ordering (1), decreasing DAC ordering (2 - default option), decreasing constraint tightness (3), DAC then tightness (4), tightness then DAC (5), randomly (6) or the opposite order if using a negative value.
\item[-solr] solution-based phase saving (reuse last found solution as preferred value assignment in the value ordering heuristic) (default option).
\item[-vacint] VAC-integrality/Full-EAC variable ordering heuristic (can be combined with option -A)
\end{description}

\subsection{Diverse solutions}

\toulbar2 can search for a greedy sequence of diverse solutions with guaranteed local optimality and minimum pairwise Hamming distance~\cite{Ruffini19a}.

\begin{description}
\item[{-div=[integer]}] : minimum Hamming distance between diverse solutions (use in conjunction with -a=integer with a limit of 1000 solutions) (default value is 0)
\item[{-divm=[integer]}] : diversity encoding method: 0:Dual 1:Hidden 2:Ternary (default value is 0)
\item[{-mdd=[integer]}] : maximum relaxed MDD width for diverse solution global constraint (default value is 0)
\item[{-mddh=[integer]}] : MDD relaxation heuristic: 0: random, 1: high div, 2: small div, 3: high unary costs (default value is 0)
\end{description}

\subsection{Console output}

\begin{description}
\item[-help] shows the default help message that \toulbar2 prints when
  it gets no argument.
\item[{-v=[integer]}] sets the verbosity level (default 0).
\item[{-Z=[integer]}] debug mode (save problem at each node if verbosity
  option -v=num $>= 1$ and -Z=num $>=3$)
\item[{-s=[integer]}] shows each solution found during search. The solution is
  printed on one line, giving by default (-s=1) the value (integer) of each variable
  successively in increasing file order. For -s=2, the value name is used instead, and for -s=3, variable name=value name is printed instead.
\end{description}

\subsection{File output}

\begin{description}
\item[{-w=[filename]}] writes last/all solutions found in the specified
  filename (or "sol" if no parameter is given). The current directory
  is used as a relative path.
\item[{-w=[integer]}] 1: writes value numbers, 2: writes value names, 3: writes also variable names (default value is 1, this option can be used in combination with -w=filename).
\item[{-z=[filename]}] saves problem in wcsp or cfn format in filename (or
  "problem.wcsp"/"problem.cfn" if no parameter is given) writes also the graphviz
  dot file and the degree distribution of the input problem
\item[{-z=[integer]}] 1 or 3: saves original instance in 1-wcsp or 3-cfn format (1 by default), 2 or 4: saves
  after preprocessing in 2-wcsp or 4-cfn format (this option can be used in combination with -z=filename)
\item[{-x=[(,i[$=\#<>$]a)*]}] performs an elementary operation ('$=$':assign, '$\#$':remove, '$<$':decrease, '$>$':increase) with value a on variable of index i (multiple
  operations are separated by a comma and no space) (without any
  argument, a complete assignment -- used as initial upper bound and
  as value heuristic -- read from default file "sol" or given as input
  filename with ".sol" extension)
\end{description}

\subsection{Probability representation and numerical control}

\begin{description}
\item[{-precision=[integer]}] probability/real precision is a conversion
  factor (a power of ten) for representing fixed point numbers
  (default value is 7)
\item[{-epsilon=[float]}] approximation factor for computing the partition
  function (greater than 1, default value is infinity)
\end{description}

\subsection{Random problem generation}

\begin{description}
\item[{-random=[bench profile]}]
  bench profile must be specified as follows.
  \begin{itemize}
  \item n and d are respectively the number of variable and the
    maximum domain size of the random problem.

    bin-{n}-{d}-{t1}-{p2}-{seed}
    \begin{itemize}
    \item t1 is the tightness in percentage \% of random binary cost functions
    \item p2 is the number of binary cost functions to include
    \item the seed parameter is optional
    \end{itemize}

    binsub-{n}-{d}-{t1}-{p2}-{p3}-{seed} binary random \& submodular cost functions
    \begin{itemize}
      \item t1 is the tightness in percentage \% of random cost functions
      \item  p2 is the number of binary cost functions to include
      \item p3 is the percentage \% of submodular cost functions among p2 cost functions
        (plus 10 permutations of two randomly-chosen values for each domain)
      \end{itemize}

       tern-{n}-{d}-{t1}-{p2}-{p3}-{seed}
       \begin{itemize}
       \item p3 is the number of ternary cost functions
       \end{itemize}

      nary-{n}-{d}-{t1}-{p2}-{p3}...-{pn}-{seed}
       \begin{itemize}
       \item pn is the number of n-ary cost functions
       \end{itemize}

       salldiff-{n}-{d}-{t1}-{p2}-{p3}...-{pn}-{seed}
       \begin{itemize}
       \item pn is the number of salldiff global cost functions (p2 and
         p3 still being used for the number of random binary and
         ternary cost functions). {\em salldiff} can be replaced by {\em gcc} or {\em regular} keywords with three possible forms ({\em e.g., sgcc, sgccdp, wgcc}) and by {\em knapsack}.
       \end{itemize}
     \end{itemize}
   \end{description}

\section{Input File formats}
\label{file-formats}
Notice that by default \toulbar2 distinguishes file formats based on their extension.
It is possible to read a file from a unix pipe using option {\tt -stdin=[format]}~; {\em e.g.,} {\tt cat example.wcsp | toulbar2 --stdin=wcsp}\\

It is also possible to read and combine multiple problem files (warning, they must be all in the same format, either wcsp, cfn, or xml).
Variables with the same name are merged (domains must be identical), otherwise the merge is based on variable indexes (wcsp format).

\subsection{cfn format (.cfn, .cfn.gz, and .cfn.xz file extension)}
With this JSON compatible format, it is possible:
\begin{itemize}
\item to give a name to variables and functions.
\item to associate a local label to every value that is accessible inside toulbar2 (among others for heuristics design purposes).
\item to use decimal and possibly negative costs.
\item to solve both minimization and maximization problems.
\item to debug  your .cfn files: the parser gives a cause and line number when it fails.
\item to use gzip'd or xz compressed files directly as input (.cfn.gz and .cfn.xz).
\item to use dense descriptions for dense cost tables.
\end{itemize}

See a full description in file document CFNformat.pdf in the doc repository on GitHub or directly on the toulbar2 Web site.

\subsection{wcsp format (.wcsp file extension)}

It is a text format composed of a list of numerical and string terms separated by spaces. Instead of using names for making reference to variables, variable indexes are employed. The same for domain values. All indexes start at zero.

Cost functions can be defined in intention (see below) or in extension, by their list of tuples. A default cost value is defined per function in order to reduce the size of the list. Only tuples with a different cost value should be given (not mandatory). All the cost values must be positive. The arity of a cost function in extension may be equal to zero. In this case, there is no tuples and the default cost value is added to the cost of any solution. This can be used to represent a global lower bound constant of the problem.

The wcsp file format is composed of three parts\-: a problem header, the list of variable domain sizes, and the list of cost functions.

\begin{DoxyItemize}
\item Header definition for a given problem\-: \begin{DoxyVerb}<Problem name>
<Number of variables (N)>
<Maximum domain size>
<Number of cost functions>
<Initial global upper bound of the problem (UB)>
\end{DoxyVerb}
 The goal is to find an assignment of all the variables with minimum total cost, strictly lower than U\-B. Tuples with a cost greater than or equal to U\-B are forbidden (hard constraint).
\item Definition of domain sizes \begin{DoxyVerb}<Domain size of variable with index 0>
...
<Domain size of variable with index N - 1>
\end{DoxyVerb}
 \begin{DoxyNote}{Note}
domain values range from zero to {\itshape size-\/1}

a negative domain size is interpreted as a variable with an interval domain in $[0,-size-1]$
\end{DoxyNote}
\begin{DoxyWarning}{Warning}
variables with interval domains are restricted to arithmetic and disjunctive cost functions in intention (see below)
\end{DoxyWarning}

\item General definition of cost functions
\begin{DoxyItemize}
\item Definition of a cost function in extension \begin{DoxyVerb}<Arity of the cost function>
<Index of the first variable in the scope of the cost function>
...
<Index of the last variable in the scope of the cost function>
<Default cost value>
<Number of tuples with a cost different than the default cost>
\end{DoxyVerb}
 followed by for every tuple with a cost different than the default cost\-: \begin{DoxyVerb}<Index of the value assigned to the first variable in the scope>
...
<Index of the value assigned to the last variable in the scope>
<Cost of the tuple>
\end{DoxyVerb}
 \begin{DoxyNote}{Note}
Shared cost function\-: A cost function in extension can be shared by several cost functions with the same arity (and same domain sizes) but different scopes. In order to do that, the cost function to be shared must start by a negative scope size. Each shared cost function implicitly receives an occurrence number starting from 1 and incremented at each new shared definition. New cost functions in extension can reuse some previously defined shared cost functions in extension by using a negative number of tuples representing the occurrence number of the desired shared cost function. Note that default costs should be the same in the shared and new cost functions. Here is an example of 4 variables with domain size 4 and one All\-Different hard constraint decomposed into 6 binary constraints.
\end{DoxyNote}

\item Shared C\-F used inside a small example in wcsp format\-:
\begin{DoxyCode}
AllDifferentDecomposedIntoBinaryConstraints 4 4 6 1
4 4 4 4
-2 0 1 0 4
0 0 1
1 1 1
2 2 1
3 3 1
2 0 2 0 -1
2 0 3 0 -1
2 1 2 0 -1
2 1 3 0 -1
2 2 3 0 -1
\end{DoxyCode}

\item Definition of a cost function in intension by replacing the default cost value by -\/1 and by giving its keyword name and its K parameters \begin{DoxyVerb}<Arity of the cost function>
<Index of the first variable in the scope of the cost function>
...
<Index of the last variable in the scope of the cost function>
-1
<keyword>
<parameter1>
...
<parameterK>
\end{DoxyVerb}

\end{DoxyItemize}
\end{DoxyItemize}Possible keywords of cost functions defined in intension followed by their specific parameters\-:
\begin{DoxyItemize}
\item $>$= {\itshape cst} {\itshape delta} to express soft binary constraint $x \geq y + cst$ with associated cost function $max( (y + cst - x \leq delta)?(y + cst - x):UB , 0 )$
\item $>$ {\itshape cst} {\itshape delta} to express soft binary constraint $x > y + cst$ with associated cost function $max( (y + cst + 1 - x \leq delta)?(y + cst + 1 - x):UB , 0 )$
\item $<$= {\itshape cst} {\itshape delta} to express soft binary constraint $x \leq y + cst$ with associated cost function $max( (x - cst - y \leq delta)?(x - cst - y):UB , 0 )$
\item $<$ {\itshape cst} {\itshape delta} to express soft binary constraint $x < y + cst$ with associated cost function $max( (x - cst + 1 - y \leq delta)?(x - cst + 1 - y):UB , 0 )$
\item = {\itshape cst} {\itshape delta} to express soft binary constraint $x = y + cst$ with associated cost function $(|y + cst - x| \leq delta)?|y + cst - x|:UB$
\item disj {\itshape cstx} {\itshape csty} {\itshape penalty} to express soft binary disjunctive constraint $x \geq y + csty \vee y \geq x + cstx$ with associated cost function $(x \geq y + csty \vee y \geq x + cstx)?0:penalty$
\item sdisj {\itshape cstx} {\itshape csty} {\itshape xinfty} {\itshape yinfty} {\itshape costx} {\itshape costy} to express a special disjunctive constraint with three implicit hard constraints $x \leq xinfty$ and $y \leq yinfty$ and $x < xinfty \wedge y < yinfty \Rightarrow (x \geq y + csty \vee y \geq x + cstx)$ and an additional cost function $((x = xinfty)?costx:0) + ((y= yinfty)?costy:0)$
\item Global cost functions using a dedicated propagator\-:
\begin{DoxyItemize}
\item clique {\itshape 1} ({\itshape nb\-\_\-values} ({\itshape value})$\ast$)$\ast$ to express a hard clique cut to restrict the number of variables taking their value into a given set of values (per variable) to at most {\itshape 1} occurrence for all the variables (warning! it assumes also a clique of binary constraints already exists to forbid any two variables using both the restricted values)
\item knapsack {\itshape capacity} ({\itshape weight})$\ast$ to express a reverse knapsack hard constraint with minimum capacity and a list of weights associated to the variables in the scope of the constraint (warning! it assumes Boolean 0/1 variables and is equivalent to a linear constraint with >= operator, use negative numbers to express the <= operator)
\end{DoxyItemize}
\item Global cost functions using a flow-\/based propagator\-:
\begin{DoxyItemize}
\item salldiff var$\vert$dec$\vert$decbi {\itshape cost} to express a soft alldifferent constraint with either variable-\/based ({\itshape var} keyword) or decomposition-\/based ({\itshape dec} and {\itshape decbi} keywords) cost semantic with a given {\itshape cost} per violation ({\itshape decbi} decomposes into a binary cost function complete network)
\item sgcc var$\vert$dec$\vert$wdec {\itshape cost} {\itshape nb\-\_\-values} ({\itshape value} {\itshape lower\-\_\-bound} {\itshape upper\-\_\-bound} ({\itshape shortage\-\_\-weight} {\itshape excess\-\_\-weight})?)$\ast$ to express a soft global cardinality constraint with either variable-\/based ({\itshape var} keyword) or decomposition-\/based ({\itshape dec} keyword) cost semantic with a given {\itshape cost} per violation and for each value its lower and upper bound (if {\itshape wdec} then violation cost depends on each value shortage or excess weights)
\item ssame {\itshape cost} {\itshape list\-\_\-size1} {\itshape list\-\_\-size2} ({\itshape variable\-\_\-index})$\ast$ ({\itshape variable\-\_\-index})$\ast$ to express a permutation constraint on two lists of variables of equal size (implicit variable-\/based cost semantic)
\item sregular var$\vert$edit {\itshape cost} {\itshape nb\-\_\-states} {\itshape nb\-\_\-initial\-\_\-states} ({\itshape state})$\ast$ {\itshape nb\-\_\-final\-\_\-states} ({\itshape state})$\ast$ {\itshape nb\-\_\-transitions} ({\itshape start\-\_\-state} {\itshape symbol\-\_\-value} {\itshape end\-\_\-state})$\ast$ to express a soft regular constraint with either variable-\/based ({\itshape var} keyword) or edit distance-\/based ({\itshape edit} keyword) cost semantic with a given {\itshape cost} per violation followed by the definition of a deterministic finite automaton with number of states, list of initial and final states, and list of state transitions where symbols are domain values
\end{DoxyItemize}
\item Global cost functions using a dynamic programming D\-A\-G-\/based propagator\-:
\begin{DoxyItemize}
\item sregulardp var {\itshape cost} {\itshape nb\-\_\-states} {\itshape nb\-\_\-initial\-\_\-states} ({\itshape state})$\ast$ {\itshape nb\-\_\-final\-\_\-states} ({\itshape state})$\ast$ {\itshape nb\-\_\-transitions} ({\itshape start\-\_\-state} {\itshape symbol\-\_\-value} {\itshape end\-\_\-state})$\ast$ to express a soft regular constraint with a variable-\/based ({\itshape var} keyword) cost semantic with a given {\itshape cost} per violation followed by the definition of a deterministic finite automaton with number of states, list of initial and final states, and list of state transitions where symbols are domain values
\item sgrammar$\vert$sgrammardp var$\vert$weight {\itshape cost} {\itshape nb\-\_\-symbols} {\itshape nb\-\_\-values} {\itshape start\-\_\-symbol} {\itshape nb\-\_\-rules} ((0 {\itshape terminal\-\_\-symbol} {\itshape value})$\vert$(1 {\itshape nonterminal\-\_\-in} {\itshape nonterminal\-\_\-out\-\_\-left} {\itshape nonterminal\-\_\-out\-\_\-right})$\vert$(2 {\itshape terminal\-\_\-symbol} {\itshape value} {\itshape weight})$\vert$(3 {\itshape nonterminal\-\_\-in} {\itshape nonterminal\-\_\-out\-\_\-left} {\itshape nonterminal\-\_\-out\-\_\-right} {\itshape weight}))$\ast$ to express a soft/weighted grammar in Chomsky normal form
\item samong$\vert$samongdp var {\itshape cost} {\itshape lower\-\_\-bound} {\itshape upper\-\_\-bound} {\itshape nb\-\_\-values} ({\itshape value})$\ast$ to express a soft among constraint to restrict the number of variables taking their value into a given set of values
\item salldiffdp var {\itshape cost} to express a soft alldifferent constraint with variable-\/based ({\itshape var} keyword) cost semantic with a given {\itshape cost} per violation (decomposes into samongdp cost functions)
\item sgccdp var {\itshape cost} {\itshape nb\-\_\-values} ({\itshape value} {\itshape lower\-\_\-bound} {\itshape upper\-\_\-bound})$\ast$ to express a soft global cardinality constraint with variable-\/based ({\itshape var} keyword) cost semantic with a given {\itshape cost} per violation and for each value its lower and upper bound (decomposes into samongdp cost functions)
\item max$\vert$smaxdp {\itshape def\-Cost} {\itshape nbtuples} ({\itshape variable} {\itshape value} {\itshape cost})$\ast$ to express a weighted max cost function to find the maximum cost over a set of unary cost functions associated to a set of variables (by default, {\itshape def\-Cost} if unspecified)
\item M\-S\-T$\vert$smstdp to express a spanning tree hard constraint where each variable is assigned to its parent variable index in order to build a spanning tree (the root being assigned to itself)
\end{DoxyItemize}
\item Global cost functions using a cost function network-\/based propagator\-~\cite{Ficolofo2012}:
\begin{DoxyItemize}
\item wregular {\itshape nb\-\_\-states} {\itshape nb\-\_\-initial\-\_\-states} ({\itshape state} and cost)$\ast$ {\itshape nb\-\_\-final\-\_\-states} ({\itshape state} and cost)$\ast$ {\itshape nb\-\_\-transitions} ({\itshape start\-\_\-state} {\itshape symbol\-\_\-value} {\itshape end\-\_\-state} {\itshape cost})$\ast$ to express a weighted regular constraint with weights on initial states, final states, and transitions, followed by the definition of a deterministic finite automaton with number of states, list of initial and final states with their costs, and list of weighted state transitions where symbols are domain values
\item walldiff hard$\vert$lin$\vert$quad {\itshape cost} to express a soft alldifferent constraint as a set of wamong hard constraint ({\itshape hard} keyword) or decomposition-\/based ({\itshape lin} and {\itshape quad} keywords) cost semantic with a given {\itshape cost} per violation
\item wgcc hard$\vert$lin$\vert$quad {\itshape cost} {\itshape nb\-\_\-values} ({\itshape value} {\itshape lower\-\_\-bound} {\itshape upper\-\_\-bound})$\ast$ to express a soft global cardinality constraint as either a hard constraint ({\itshape hard} keyword) or with decomposition-\/based ({\itshape lin} and {\itshape quad} keyword) cost semantic with a given {\itshape cost} per violation and for each value its lower and upper bound
\item wsame hard$\vert$lin$\vert$quad {\itshape cost} to express a permutation constraint on two lists of variables of equal size (implicitly concatenated in the scope) using implicit decomposition-\/based cost semantic
\item wsamegcc hard$\vert$lin$\vert$quad {\itshape cost} {\itshape nb\-\_\-values} ({\itshape value} {\itshape lower\-\_\-bound} {\itshape upper\-\_\-bound})$\ast$ to express the combination of a soft global cardinality constraint and a permutation constraint
\item wamong hard$\vert$lin$\vert$quad {\itshape cost} {\itshape nb\-\_\-values} ({\itshape value})$\ast$ {\itshape lower\-\_\-bound} {\itshape upper\-\_\-bound} to express a soft among constraint to restrict the number of variables taking their value into a given set of values
\item wvaramong hard {\itshape cost} {\itshape nb\-\_\-values} ({\itshape value})$\ast$ to express a hard among constraint to restrict the number of variables taking their value into a given set of values to be equal to the last variable in the scope
\item woverlap hard$\vert$lin$\vert$quad {\itshape cost} {\itshape comparator} {\itshape righthandside} overlaps between two sequences of variables X, Y (i.\-e. set the fact that Xi and Yi take the same value (not equal to zero))
\item wsum hard$\vert$lin$\vert$quad {\itshape cost} {\itshape comparator} {\itshape righthandside} to express a soft sum constraint with unit coefficients to test if the sum of a set of variables matches with a given comparator and right-\/hand-\/side value
\item wvarsum hard {\itshape cost} {\itshape comparator} to express a hard sum constraint to restrict the sum to be {\itshape comparator} to the value of the last variable in the scope

Let us note $<$$>$ the comparator, K the right-\/hand-\/side value associated to the comparator, and Sum the result of the sum over the variables. For each comparator, the gap is defined according to the distance as follows\-:
\begin{DoxyItemize}
\item if $<$$>$ is == \-: gap = abs(K -\/ Sum)
\item if $<$$>$ is $<$= \-: gap = max(0,Sum -\/ K)
\item if $<$$>$ is $<$ \-: gap = max(0,Sum -\/ K -\/ 1)
\item if $<$$>$ is != \-: gap = 1 if Sum != K and gap = 0 otherwise
\item if $<$$>$ is $>$ \-: gap = max(0,K -\/ Sum + 1);
\item if $<$$>$ is $>$= \-: gap = max(0,K -\/ Sum);
\end{DoxyItemize}
\end{DoxyItemize}
\end{DoxyItemize}\begin{DoxyWarning}{Warning}
The decomposition of wsum and wvarsum may use an exponential size (sum of domain sizes).

{\itshape list\-\_\-size1} and {\itshape list\-\_\-size2} must be equal in {\itshape ssame}.

Cost functions defined in intention cannot be shared.
\end{DoxyWarning}
\begin{DoxyNote}{Note}
More about network-\/based global cost functions can be found here \href{https://metivier.users.greyc.fr/decomposable/}{\tt https\-://metivier.\-users.\-greyc.\-fr/decomposable/}
\end{DoxyNote}
Examples\-:
\begin{DoxyItemize}
\item quadratic cost function $x0 * x1$ in extension with variable domains $\{0,1\}$ (equivalent to a soft clause $\neg x0 \vee \neg x1$)\-:
\begin{DoxyCode}
2 0 1 0 1 1 1 1
\end{DoxyCode}

\item simple arithmetic hard constraint $x1 < x2$\-:
\begin{DoxyCode}
2 1 2 -1 < 0 0
\end{DoxyCode}

\item hard temporal disjunction $x1 \geq x2 + 2 \vee x2 \geq x1 + 1$\-:
\begin{DoxyCode}
2 1 2 -1 disj 1 2 1000
\end{DoxyCode}

\item clique cut (\{x0,x1,x2,x3\}) on Boolean variables such that value 1 is used at most once (warning! must add a clique of binary constraints too)\-:
\begin{DoxyCode}
4 0 1 2 3 -1 clique 1 1 1 1 1 1 1 1 1
2 0 1 0 1 1 1 1000
2 0 2 0 1 1 1 1000
2 0 3 0 1 1 1 1000
2 1 2 0 1 1 1 1000
2 1 3 0 1 1 1 1000
2 2 3 0 1 1 1 1000
\end{DoxyCode}

\item knapsack constraint (\{x0,x1,x2,x3\}) on Boolean variables such that $2*x0 + 3*x1 + 4*x2 + 5*x3 >= 10$\-:
 \begin{DoxyCode}
4 0 1 2 3 -1 knapsack 10 2 3 4 5
\end{DoxyCode}

\item soft\-\_\-alldifferent(\{x0,x1,x2,x3\})\-:
\begin{DoxyCode}
4 0 1 2 3 -1 salldiff var 1
\end{DoxyCode}

\item soft\-\_\-gcc(\{x1,x2,x3,x4\}) with each value {\itshape v} from 1 to 4 only appearing at least v-\/1 and at most v+1 times\-:
\begin{DoxyCode}
4 1 2 3 4 -1 sgcc var 1 4 1 0 2 2 1 3 3 2 4 4 3 5
\end{DoxyCode}

\item soft\-\_\-same(\{x0,x1,x2,x3\},\{x4,x5,x6,x7\})\-:
\begin{DoxyCode}
8 0 1 2 3 4 5 6 7 -1 ssame 1 4 4 0 1 2 3 4 5 6 7
\end{DoxyCode}

\item soft\-\_\-regular(\{x1,x2,x3,x4\}) with D\-F\-A (3$\ast$)+(4$\ast$)\-:
\begin{DoxyCode}
4 1 2 3 4 -1 sregular var 1 2 1 0 2 0 1 3 0 3 0 0 4 1 1 4 1
\end{DoxyCode}

\item soft\-\_\-grammar(\{x0,x1,x2,x3\}) with hard cost (1000) producing well-\/formed parenthesis expressions\-:
\begin{DoxyCode}
4 0 1 2 3 -1 sgrammardp var 1000 4 2 0 6 1 0 0 0 1 0 1 2 1 0 1 3 1 2 0 3 0 1 0 0 3 1
\end{DoxyCode}

\item soft\-\_\-among(\{x1,x2,x3,x4\}) with hard cost (1000) if $\sum_{i=1}^4(x_i \in \{1,2\}) < 1$ or $\sum_{i=1}^4(x_i \in \{1,2\}) > 3$\-:
\begin{DoxyCode}
4 1 2 3 4 -1 samongdp var 1000 1 3 2 1 2
\end{DoxyCode}

\item soft max(\{x0,x1,x2,x3\}) with cost equal to $\max_{i=0}^3((x_i!=i)?1000:(4-i))$\-:
\begin{DoxyCode}
4 0 1 2 3 -1 smaxdp 1000 4 0 0 4 1 1 3 2 2 2 3 3 1
\end{DoxyCode}

\item wregular(\{x0,x1,x2,x3\}) with D\-F\-A (0(10)$\ast$2$\ast$)\-:
\begin{DoxyCode}
4 0 1 2 3 -1 wregular 3 1 0 0 1 2 0 9 0 0 1 0 0 1 1 1 0 2 1 1 1 1 0 0 1 0 0 1 1 2 0 1 1 2 2 0 1 0 2 1 1 1 2
       1
\end{DoxyCode}

\item wamong (\{x1,x2,x3,x4\}) with hard cost (1000) if $\sum_{i=1}^4(x_i \in \{1,2\}) < 1$ or $\sum_{i=1}^4(x_i \in \{1,2\}) > 3$\-:
\begin{DoxyCode}
4 1 2 3 4 -1 wamong hard 1000 2 1 2 1 3
\end{DoxyCode}

\item wvaramong (\{x1,x2,x3,x4\}) with hard cost (1000) if $\sum_{i=1}^3(x_i \in \{1,2\}) \neq x_4$\-:
\begin{DoxyCode}
4 1 2 3 4 -1 wvaramong hard 1000 2 1 2
\end{DoxyCode}

\item woverlap(\{x1,x2,x3,x4\}) with hard cost (1000) if $\sum_{i=1}^2(x_i = x_{i+2}) \geq 1$\-:
\begin{DoxyCode}
4 1 2 3 4 -1 woverlap hard 1000 < 1
\end{DoxyCode}

\item wsum (\{x1,x2,x3,x4\}) with hard cost (1000) if $\sum_{i=1}^4(x_i) \neq 4$\-:
\begin{DoxyCode}
4 1 2 3 4 -1 wsum hard 1000 == 4
\end{DoxyCode}

\item wvarsum (\{x1,x2,x3,x4\}) with hard cost (1000) if $\sum_{i=1}^3(x_i) \neq x_4$\-:
\begin{DoxyCode}
4 1 2 3 4 -1 wvarsum hard 1000 ==
\end{DoxyCode}

\end{DoxyItemize}Latin Square 4 x 4 crisp C\-S\-P example in wcsp format\-:
\begin{DoxyCode}
latin4 16 4 8 1
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 0 1 2 3 -1 salldiff var 1
4 4 5 6 7 -1 salldiff var 1
4 8 9 10 11 -1 salldiff var 1
4 12 13 14 15 -1 salldiff var 1
4 0 4 8 12 -1 salldiff var 1
4 1 5 9 13 -1 salldiff var 1
4 2 6 10 14 -1 salldiff var 1
4 3 7 11 15 -1 salldiff var 1
\end{DoxyCode}

4-\/queens binary weighted C\-S\-P example with random unary costs in wcsp format\-:
\begin{DoxyCode}
4-WQUEENS 4 4 10 5
4 4 4 4
2 0 1 0 10
0 0 5
0 1 5
1 0 5
1 1 5
1 2 5
2 1 5
2 2 5
2 3 5
3 2 5
3 3 5
2 0 2 0 8
0 0 5
0 2 5
1 1 5
1 3 5
2 0 5
2 2 5
3 1 5
3 3 5
2 0 3 0 6
0 0 5
0 3 5
1 1 5
2 2 5
3 0 5
3 3 5
2 1 2 0 10
0 0 5
0 1 5
1 0 5
1 1 5
1 2 5
2 1 5
2 2 5
2 3 5
3 2 5
3 3 5
2 1 3 0 8
0 0 5
0 2 5
1 1 5
1 3 5
2 0 5
2 2 5
3 1 5
3 3 5
2 2 3 0 10
0 0 5
0 1 5
1 0 5
1 1 5
1 2 5
2 1 5
2 2 5
2 3 5
3 2 5
3 3 5
1 0 0 2
1 1
3 1
1 1 0 2
1 1
2 1
1 2 0 2
1 1
2 1
1 3 0 2
0 1
2 1
\end{DoxyCode}

%\subsubsection{CPD final stanza}
%TODO

\subsection{UAI and LG formats (.uai, .LG)}

It is a simple text file format specified below to describe probabilistic graphical model instances. The format is a generalization of the Ergo file format initially developed by Noetic Systems Inc. for their Ergo software.

\begin{itemize}
\item {\bf Structure}~\\

A file in the UAI format consists of the following two parts, in that order:
\begin{verbatim}
<Preamble>

<Function tables>
\end{verbatim}

The contents of each section (denoted $<...>$ above) are described in the following:
\item {\bf Preamble}~\\

The preamble starts with one line denoting the type of network. This will be either BAYES (if the network is a Bayesian network) or MARKOV (in case of a Markov network). This is followed by a line containing the number of variables. The next line specifies each variable's domain size, one at a time, separated by whitespace (note that this implies an order on the variables which will be used throughout the file).

The fourth line contains only one integer, denoting the number of functions in the problem (conditional probability tables for Bayesian networks, general factors for Markov networks). Then, one function per line, the scope of each function is given as follows: The first integer in each line specifies the size of the function's scope, followed by the actual indexes of the variables in the scope. The order of this list is not restricted, except when specifying a conditional probability table (CPT) in a Bayesian network, where the child variable has to come last. Also note that variables are indexed starting with 0.

For instance, a general function over variables 0, 5 and 11 would have this entry:
\begin{verbatim}
3 0 5 11
\end{verbatim}

A simple Markov network preamble with three variables and two functions might for instance look like this:

\begin{DoxyCode}
MARKOV
3
2 2 3
2
2 0 1
3 0 1 2
\end{DoxyCode}

The first line denotes the Markov network, the second line tells us the problem consists of three variables, let's refer to them as X, Y, and Z. Their domain size is 2, 2, and 3 respectively (from the third line). Line four specifies that there are 2 functions. The scope of the first function is X,Y, while the second function is defined over X,Y,Z.

An example preamble for a Belief network over three variables (and therefore with three functions) might be:

\begin{DoxyCode}
BAYES
3
2 2 3
3
1 0
2 0 1
2 1 2
\end{DoxyCode}

The first line signals a Bayesian network. This example has three variables, let's call them X, Y, and Z, with domain size 2, 2, and 3, respectively (from lines two and three). Line four says that there are 3 functions (CPTs in this case). The scope of the first function is given in line five as just X (the probability P(X)), the second one is defined over X and Y (this is (Y $|$ X)). The third function, from line seven, is the CPT P(Z $|$ Y). We can therefore deduce that the joint probability for this problem factors as P(X,Y,Z) = P(X).P(Y $|$ X).P(Z $|$ Y).

\item {\bf Function tables}~\\

In this section each function is specified by giving its full table (i.e, specifying the function value for each tuple). The order of the functions is identical to the one in which they were introduced in the preamble.

For each function table, first the number of entries is given (this should be equal to the product of the domain sizes of the variables in the scope). Then, one by one, separated by whitespace, the values for each assignment to the variables in the function's scope are enumerated. Tuples are implicitly assumed in ascending order, with the last variable in the scope as the 'least significant'.

To illustrate, we continue with our Bayesian network example from above, let's assume the following conditional probability tables:

\begin{eqnarray*}
X & P(X) &\\
0 & 0.436 &\\
1 & 0.564 &\\
&&\\
X & Y & P(Y | X)\\
0 & 0 & 0.128\\
0 & 1 & 0.872\\
1 & 0 & 0.920\\
1 & 1 & 0.080\\
&&\\
Y & Z & P(Z | Y)\\
0 & 0 & 0.210\\
0 & 1 & 0.333\\
0 & 2 & 0.457\\
1 & 0 & 0.811\\
1 & 1 & 0.000\\
1 &	2 &	0.189\\
\end{eqnarray*}

The corresponding function tables in the file would then look like this:

\begin{DoxyCode}
2
 0.436 0.564

4
 0.128 0.872
 0.920 0.080

6
 0.210 0.333 0.457
 0.811 0.000 0.189
\end{DoxyCode}

(Note that line breaks and empty lines are effectively just whitespace, exactly like plain spaces " ". They are used here to improve readability.)

In the LG format, probabilities are replaced by their logarithm.

\item {\bf Summary}~\\

To sum up, a problem file consists of 2 sections: the preamble and the full the function tables, the names and the labels.

For our Markov network example above, the full file could be:

\begin{DoxyCode}
MARKOV
3
2 2 3
2
2 0 1
3 0 1 2

4
 4.000 2.400
 1.000 0.000

12
 2.2500 3.2500 3.7500
 0.0000 0.0000 10.0000
 1.8750 4.0000 3.3330
 2.0000 2.0000 3.4000
\end{DoxyCode}

Here is the full Bayesian network example from above:

\begin{DoxyCode}
BAYES
3
2 2 3
3
1 0
2 0 1
2 1 2

2
 0.436 0.564

4
 0.128 0.872
 0.920 0.080

6
 0.210 0.333 0.457
 0.811 0.000 0.189
\end{DoxyCode}

\item {\bf Expressing evidence}~\\

Evidence is specified in a separate file. This file has the same name as the original problems file but an added .evid extension at the end. For instance, problem.uai will have evidence in problem.uai.evid.

The file simply starts with a line specifying the number of evidence variables. This is followed by the pairs of variable and value indexes for each observed variable, one pair per line. The indexes correspond to the ones implied by the original problem file.

If, for our above example, we want to specify that variable Y has been observed as having its first value and Z with its second value, the file example.uai.evid would contain the following:

\begin{DoxyCode}
2
 1 0
 2 1
\end{DoxyCode}
\end{itemize}

\subsection{Partial Weighted MaxSAT format}

\paragraph{Max-SAT input format (.cnf)}~\\

The input file format for Max-SAT will be in DIMACS format:

\begin{DoxyCode}
c
c comments Max-SAT
c
p cnf 3 4
1 -2 0
-1 2 -3 0
-3 2 0
1 3 0
\end{DoxyCode}

\begin{itemize}
\item The file can start with comments, that is lines beginning with the character 'c'.
\item Right after the comments, there is the line "p cnf nbvar nbclauses" indicating that the instance is in CNF format; nbvar is the number of variables appearing in the file; nbclauses is the exact number of clauses contained in the file.
\item Then the clauses follow. Each clause is a sequence of distinct non-null numbers between -nbvar and nbvar ending with 0 on the same line. Positive numbers denote the corresponding variables. Negative numbers denote the negations of the corresponding variables.
\end{itemize}

\paragraph{Weighted Max-SAT input format (.wcnf)}~\\

In Weighted Max-SAT, the parameters line is "p wcnf nbvar nbclauses". The weights of each clause will be identified by the first integer in each clause line. The weight of each clause is an integer greater than or equal to 1.

Example of Weighted Max-SAT formula:

\begin{DoxyCode}
c
c comments Weighted Max-SAT
c
p wcnf 3 4
10 1 -2 0
3 -1 2 -3 0
8 -3 2 0
5 1 3 0
\end{DoxyCode}

\paragraph{Partial Max-SAT input format (.wcnf)}~\\

In Partial Max-SAT, the parameters line is "p wcnf nbvar nbclauses top". We associate a weight with each clause, which is the first integer in the clause. Weights must be greater than or equal to 1. Hard clauses have weight top and soft clauses have weight 1. We assume that top is a weight always greater than the sum of the weights of violated soft clauses.

Example of Partial Max-SAT formula:

\begin{DoxyCode}
c
c comments Partial Max-SAT
c
p wcnf 4 5 15
15 1 -2 4 0
15 -1 -2 3 0
1 -2 -4 0
1 -3 2 0
1 1 3 0
\end{DoxyCode}

\paragraph{Weighted Partial Max-SAT input format (.wcnf)}~\\

In Weighted Partial Max-SAT, the parameters line is "p wcnf nbvar nbclauses top". We associate a weight with each clause, which is the first integer in the clause. Weights must be greater than or equal to 1. Hard clauses have weight top and soft clauses have a weight smaller than top. We assume that top is a weight always greater than the sum of the weights of violated soft clauses.

Example of Weighted Partial Max-SAT formula:

\begin{DoxyCode}
c
c comments Weighted Partial Max-SAT
c
p wcnf 4 5 16
16 1 -2 4 0
16 -1 -2 3 0
8 -2 -4 0
4 -3 2 0
3 1 3 0
\end{DoxyCode}

\subsection{QPBO format (.qpbo)}

In the quadratic pseudo-Boolean optimization (unconstrained quadratic programming) format,
the goal is to minimize or maximize the quadratic function:

$$X' * W * X = \sum_{i=1}^N \sum_{j=1}^N  W_{ij} * X_i * X_j$$

where $W$ is a symmetric squared $N \times N$ matrix expressed by all its non-zero half ($i \leq j$) squared matrix coefficients, $X$ is a vector of $N$ binary variables with domain values in $\{0,1\}$ or $\{1,-1\}$, and $X'$ is the transposed vector of $X$.

Note that for two indices $i \neq j$, coefficient $W_{ij} = W_{ji}$ (symmetric matrix) and it appears twice in the previous sum.
It can be controled by the option {\tt -qpmult=[double]} which defines a coefficient multiplier for quadratic terms (default value is 2).

Note also that coefficients can be positive or negative and are real float numbers. They are converted to fixed-point real numbers by multiplying them by $10^{precision}$ (see option {\em -precision} to modify it, default value is 7).  Infinite coefficients are forbidden.

Notice that depending on the sign of the number of variables in the first text line, the domain of all variables is either $\{0,1\}$ or $\{1,-1\}$.

Warning! The encoding in Weighted CSP of variable domain $\{1,-1\}$ associates for each variable value the following index: value 1 has index 0 and value -1 has index 1 in the solutions found by toulbar2.
The encoding  of variable domain $\{0,1\}$ is direct.

Qpbo is a file text format:
\begin{itemize}
\item First line contains the number of variables $N$ and the number of non-zero coefficients $M$.

If $N$ is negative then domain values are in $\{1, -1\}$, otherwise $\{0, 1\}$.
If $M$ is negative then it will maximize the quadratic function, otherwise it will minimize it.

\item Followed by $|M|$ lines where each text line contains three values separated by spaces:
position index $i$ (integer belonging to $[1,|N|]$),
position index $j$ (integer belonging to $[1,|N|]$),
coefficient $W_{ij}$ (float number)
such that $i \leq j$ and $W_{ij} \neq 0$
\end{itemize}

\subsection{OPB format (.opb)}

The OPB file format is used to express pseudo-Boolean satisfaction and optimization models.
These models may only contain $0/1$ Boolean variables. The format is defined by an optional objective function followed by a set of linear constraints.
Variables may be multiplied together in the objective function, but currently not in the constraints due to some restriction in the reader.
The objective function must start with the \verb!min:! or \verb!max:! keyword followed by \verb!coef_1 varname_1_1 varname_1_2 ... coef2 varname\_2\_1 ...! and end with a \verb!;!.
Linear constraints are composed in the same way, ended by a comparison operator (\verb!<=!, \verb!>=!, or \verb!=!) followed by the right-hand side coefficient and \verb!;!.
Each coefficient must be an integer beginning with its sign (\verb!+! or \verb!-! with no extra space).
Comment lines start with a \verb!*!.

An example with a quadratic objective and 7 linear constraints is:
\begin{DoxyCode}
max: +1 x1 x2 +2 x3 x4;
+1 x2 +1 x1 >= 1;
+1 x3 +1 x1 >= 1;
+1 x4 +1 x1 >= 1;
+1 x3 +1 x2 >= 1;
+1 x4 +1 x2 >= 1;
+1 x4 +1 x3 >= 1;
+2 x1 +2 x2 +2 x3 +2 x4 <= 7;
\end{DoxyCode}

Internally, all integer costs are multiplied by a power of ten depending on the -precision option.
For problems with big integers, try to reduce the precision ({\em e.g.,} use option -precision 0).

\subsection{XCSP2.1 format (.xml)}

CSP and weighted CSP in XML format XCSP 2.1, with constraints in extension only, can be read. See a description of this deprecated format here \url{http://www.cril.univ-artois.fr/CPAI08/XCSP2_1.pdf}.

Warning, \toulbar2 must be compiled with a specific option XML in the cmake.

\subsection{Linkage format (.pre)}

See \texttt{mendelsoft} companion software at \url{http://miat.inrae.fr/MendelSoft} for pedigree correction. See also \url{https://carlit.toulouse.inra.fr/cgi-bin/awki.cgi/HaplotypeInference} for haplotype inference in half-sib families.

%\subsection{BEP format}

\section{Using it as a library}

See \toulbar2 reference manual which describes the libtb2.so C++ library API.

\section{Using it from Python}

A Python interface is now available. Compile \toulbar2 with cmake option PYTB2 (and without MPI options) to generate a Python module {\sc pytoulbar2}
 (in lib directory). See examples in {\tt src/pytoulbar2.cpp} file and {\tt web/TUTORIALS} directory.

An older verion of \toulbar2 was integrated inside Numberjack. See  \url{https://github.com/eomahony/Numberjack}.

\bibliographystyle{plain}
\bibliography{publications.bib}

\end{document}


\documentclass[10pt,conference,compsocconf]{IEEEtran}

\usepackage{hyperref}
\usepackage{graphicx}	% For figure environment
\usepackage{authblk}
\usepackage{tabularx}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{float}

\title{Project 2 on Machine Learning\\Text classification\\Team Yoor}

\author[1]{Sergei Volodin}
\author[1]{Omar Mehio}
\author[1]{Baran Nama}
\affil[1]{EPFL}
\affil[ ]{\textit {\{sergei.volodin,omar.mehio,baran.nama\}@epfl.ch}}

\begin{document}

\maketitle

\begin{abstract}
A classification dataset consisting of Tweets is being studied. First, the data is thoroughly explored using visual aids. Several basic Natural Language Processing methods are applied. Results are evaluated using cross-validation. Model overview is given and the best model is chosen.
\end{abstract}

\section{Introduction}
This paper investigates into improving the quality of sentiment analysis on Tweets dataset \cite{kaggle}.
It consists of $N_1=N_2=1250000$ positive/negative tweets, each of them representing a message in English and numerical alphabet $\Sigma$ with no longer than 140 characters.
This way, each of $N=N_1+N_2=2500000$ tweets is assigned to one of the classes $\mathcal{C}=\{\mbox{:(},\,\mbox{:)}\}$.
The task is to minimize the classification error.
In other words, if $\mathcal{D}=\{(x_n, y_n)\}_{n=1}^N$ is the dataset with tweets $x_i\in\Sigma^*$ being messages and $y_n\in \mathcal{C}$ being class labels, the goal is to train a classifier $f\colon\, \Sigma^*\to\mathcal{C}$ which minimizes the loss function $l(y,\hat{y})=[y\neq \hat{y}]$.

The task of sentiment analysis of tweets was thoroughly studied \cite{sota1, sota2, sota3, sota4}. These solutions usually give accuracy around 80\%.
Several techniques were applied, mostly consisting of two steps.
First, the words are converted to dense vectors using Glove, word2vec, cbow or skip-gram models.
After that, the resulting word vectors are used to construct features for the whole tweet.
At the end, the vector is feeded into a classifier, such as SVM or Logistic Regression.
Two latter steps might be replaced with a neural network accepting variable-length input such as CNN.
Moreover, the embeddings themselves might be trained using backpropagation while training the classifier.

Claim: it is possible to find a model which fits the data better than the current state-of-the-art using expert knowledge on the Tweets dataset.

Next sections describe in details our approaches and compare them to various baselines.
\section{Models and Methods}
\subsection{Data cleaning}
\newcolumntype{L}[1]{>{\hsize=#1\hsize\raggedright\arraybackslash}X}%
\newcolumntype{R}[1]{>{\hsize=#1\hsize\raggedleft\arraybackslash}X}%
\newcolumntype{C}[1]{>{\hsize=#1\hsize\centering\arraybackslash}X}%

\begin{table*}[htbp]
	\centering
	\small
	\begin{tabularx}{\textwidth}{| L{0.1} | L{0.05} | L{0.7} | L{0.2} | }
		\hline
		Class&Row&Message&Comment\\
		\hline
		:( & 171 & {\tt <user> 5k i could cry i'm so unfit !} & User tag \\\hline
		:( & 99804 & {\tt if i feel like this tomorrow i'm going to the er } & Contraction, missed comma\\\hline
		:) & 524 & {\tt its the weekend ! ! <user> s coming home anddd <user> s baby shower , exciteddd } & Letter repetitions\\\hline
		:) & 99615 & {\tt <user> aren't you just an adorable granny <url> } & URL tag\\\hline
		:) & 443 & {\tt <user> :d :d :d :d :d wish jocelyn had a twitter . \#kudos for her too . } & Hashtag, emoticons \\\hline
		:) & 468 & {\tt retweet if you was born in the 90 ' s ! \#90's babies } & Grammar\\\hline
		:( & 14 & {\tt <user> i'm white . \#aw } & Appear neutral \\\hline
		:( & 99594 & {\tt <user> <user> <user> they're there tonight ! ! ! } & Appear positive \\\hline
	\end{tabularx}
	\caption{Examples of tweets in the small dataset}
	\label{tab:tweet-sexample}
\end{table*}

This paragraph describes the data being studied.
Tweets are short messages no longer than 140 characters long \cite{twitter}.
The table \ref{tab:tweet-sexample} represents a few examples from the small dataset.
Being considered an informal way of communication, tweets often contain misspelled words and letter repetitions, grammatical and other writing mistakes.
Besides plain text with punctuation, tweets also contain hashtags, two types of tags, {\em user} and {\em url}, which are tokens for replaced user mentions and URL links, respectively.
In addition, tweets sometimes contain emoticons.
Despite the fact that objects in the training dataset mostly comply with its classes, some tweets cannot be determined as positive/negative even by a human (appear neutral).
Moreover, some of the tweets are clearly mislabeled in the training data.
Dataset contains repeating tweets.

\begin{figure}[!htb]
	\centering \includegraphics[width=0.5\textwidth]{../plots/types.png}
	\caption{Distribution of Tweet properties}
	\label{prop}
\end{figure}

The first step taken was exploring special properties of tweets in the datasets. These properties include hashtags, punctuation and are present in huge percentages (Fig \ref{prop}). Through preprocessing the aim was to maximize the knowledge that can be extracted from the datasets in order to build word representations, which is achieved through correction of misspelled words and stemming techniques.

Usually tweeters repeat a letter in a word to stress on a certain emotion, as an example: "i am soooooo angryyyy". This problem was tackled by two different approaches. The first approach is stemming, which is defined as the process by which every word is replaced by it's origin. Linguistically any word can be changed to different forms by either adding affixes or prefixes. Through stemming, the aim is to minimize the presence of these different forms and replacing them with the original stem. The stemming algorithm presented in \ref{Stemming} solves the mentioned problem.

\begin{algorithm}
\caption{Stemming(token)}
\label{Stemming}
\begin{algorithmic}
\IF{token in dictionary}
\STATE return token
\ELSE
\STATE $X \leftarrow $ token with repeating character replaced by two instances of that character
\IF  {X == token}
\STATE return token
\ELSE
\STATE return Stemming(X)
\ENDIF
\ENDIF
\end{algorithmic}
\end{algorithm}

The second approach adopted is misspelling correction. The process goes as follows: a token in the tweet is proposed to an English dictionary, the dictionary responds by either indicating that the word is valid or not. This dictionary provides the most relevant suggestions to correct a wrong token. The levenshtein distance was used to determine whether a word should be converted to the suggestion or not. Levenshtein distance is defined as: "the distance between two strings is the minimal number of insertions, deletions, and substitutions of one character for another that will transform one string into the other". Thus to explore how many misspelled words are in the dataset, we plotted the distribution in Fig. \ref{levDistA}. Most of the data can be covered by setting our levensteihn threshold to 4.
\begin{figure}[!htb]
	\centering \includegraphics[width = 0.5\textwidth]{../plots/distributionA.png}
	\caption{Levensteihn distance distribution on raw tweets}
	\label{levDistA}
\end{figure}

\begin{figure}[!htb]
	\centering \includegraphics[width=220px]{../plots/raw.png}
	\centering \includegraphics[width=220px]{../plots/redundant.png}
	\caption{Logistic Regression accuracy on correct and stemmed tweets}
	\label{raw}
\end{figure}

The metric used to compare both methods was prediction accuracy of logistic regression. As a first step the combined dataset of both the positive and negative tweets was exposed to stemming and correction. Then for each method a vocabulary of words is constructed. Finally, the occurrence matrix is built from both the tweets and the dictionary. K-fold is used to split the matrix and then logistic regression is run using the occurrence matrix as the data matrix, labels are split into 0 denoting positive sentiment and 1 denoting a negative sentiment.  As observed in Fig. \ref{raw}, the spelling correction method is superficial to the stemming method with reaching a maximum of 78.8 \% prediction accuracy.

Another property of the dataset is the presence of redundant tweets. To view their impact, repeated instances of a tweet are eliminated and the newly formed unique dataset is exposed to the previous methods. The stemming technique overpowers the misspelling correction technique. Although with very small improvement in accuracy prediction (79\%).

\begin{figure}[!htb]
	\centering \includegraphics[width=0.5\textwidth]{../plots/distributionB.png}
	\caption{Levensteihn distance distribution on tweets with non-alphabetic starting character stripped}
	\label{levDistB}
\end{figure}

Tokens in our datasets starting with non alphabetic characters were eliminated from previous experiments as all dictionaries can't recognize non alphanumeric words. Thus a new set of tweets was constructed having all these tokens stripped of the initial non-alphabetic character or characters. As before, the distribution of levensteihn distances (Fig. \ref{levDistB}) for suggested words was plotted, which results in a more uniform distribution.

\begin{figure}[!htb]
	\centering \includegraphics[width=0.5\textwidth]{../plots/filtered_text.png}
	\caption{Logistic Regression accuracy on tweets with non-alphabetic starting character stripped }
	\label{filtered}
\end{figure}

The newly constructed dataset is subject to previous methods. Elimination of redundant tweets,stemming and misspelling correction. Accuracy prediction then improves to 80.2\%, the best score reached so far.

\begin{figure}[!htb]
	\centering \includegraphics[width=100px]{../plots/commonElem.png}
	\centering \includegraphics[width=100px]{../plots/nonfrequent.png}
	\caption{Elimination of common frequent terms vs separate non-frequent terms}
	\label{freq}
\end{figure}
Now with the optimal model having initial non-alphabetic characters removed, redundant tweets eliminated and misspelling correction applied with levensteihn threshold set to 5. The dataset is tackled from a different angle, the effect of the frequency of which one token appears in the document is explored. Thus two approaches are considered: removing most frequent tokens that are common in both datasets and removing non-frequent words from the combined vocabulary set. The results of the first experiment came as a surprise, since we suspected that elimination of common words would make it easier for the classifier to distinguish between a positive sentiment tweet and a negative one. A reasoning contributed for such results is the limited number of characters provided for users to express themselves and thus most of the tweet content gets distributed among tagging other users, using punctuation and stop words. The second approach was of more relevance, where elimination of non frequent tokens from the vocabulary has helped improving accuracy to 78\%, as observed in Fig. \ref{freq}.

\subsection{Prediction models}
\subsubsection{GloVe embeddings}
Several models were considered.
First, the baseline was the GloVe \cite{glove} model, which considers training embeddings on the co-occurence matrix. The pre-trained embeddings (Twitter, Stanford NLP group) with dimensions of 100 and 50 were used. Higher dimension embeddings did not fit to the server memory.

\subsubsection{Word count model}
Secondly, a word count model was considered as a baseline solution because of its simplicity. This model uses high. A tweet is represented by a high-dimension sparse vector, each item corresponds to a number of occurrences of a word in the tweet.
This vector was further classified using several techniques, such as SVM and Neural Network.
This method gave accuracy which placed the team to the 47/66 place in the public Kaggle leaderboard and allowed for tweaking its parameters with relatively small training time (a Google Cloud instance with 8 CPU was used).
This model was trained both on raw and preprocessed data. Raw data gave higher score, when using preprocessed data resulted in overfitting. A reduction of the vocabulary size was also considered by taking only words occurring most frequently.
\subsubsection{Convolutional neural networks (CNN)}

Representation of tweets were in a 1d vectors, where convolutional layers are summed in sequential manner. Due to small size of the embeddings, pooling layers were not used between convolutional layers. Dropout layers were performed before output layer and between convolutional layers \cite{cnn5} in order to prevent overfitting. For hidden layers, ReLU is used, for first two elements of output layer tanh activation function is used and for the last element of output layer, softmax activation function is used in favor of common practice \cite{cnn6}.
Note that, the number of filters for convolutional layers, the size of the filters, activation function types regarding the type of the layer, dropout amount and the number of convolutional layers are just a subset of complete hyper parameters of this model that were not regulated.

\section{Results}
\begin{table}[ht]
	\centering
	\small
	\begin{tabular}{|c | c |  c | c | c |}
		\hline
		Model & Dataset & Train acc. & Val. acc. & L2 \\
		\hline
		GloVe FC & Full & 0.62 & 0.62 & $10^{-6}$\\
		\hline
		WC LR & Partial & 0.86 & 0.81 & 1000, not tuned\\
		\hline
		WC LR & Full & 0.86 & 0.83 & 1000, not tuned\\
		\hline
		WC FC 100x50 & Full & {\bf 0.87} & {\bf 0.86} & 1000, not tuned\\
		\hline
		WC FC 100x50 & Full, clean & 0.82 & 0.83 & 1\\ [1ex]
		\hline
	\end{tabular}
	\caption{Comparison of different methods (WC=Word Count, GloVe~-- pretrained GloVe vectors, LR~-- Logistic Regression, FC~-- fully-connected, WC~-- word count)}
	\label{tab:results}
\end{table}

The following models were considered: CNN, Logistic Regression, Fully Connected Neural Network. Moreover, a simple word count model was considered. The best model was the word count model on raw data. However, proper hyperparameter tuning was not performed, this, it is not possible to claim that it is better than other models. Our contribution consists of cleaning tweets and proving that it's possible to classify them using a simple word count model.
\section{Discussion}
Despite performing data cleaning, we were not able to run all of the models on cleaned data. Moreover, the hyperparameter tuning was not performed in a proper way.
\section{Summary}
We have considered tweet classification task. Each tweet, consisting of no more than 140 alphanumeric characters, must be assigned to one of the classes: positive or negative sentiment. Data was cleaned by correcting spelling mistakes. Several models were considered: CNN, FC NN on pretrained GloVe embeddings, and simple word count model with FC NN. In this project, the word count model on raw data proved to be the most efficient. However, hyperparameters were not properly tuned for the models considered.

\begin{thebibliography}{99}

	\bibitem{twitter} \href{http://twitter.com}{Twitter}
	\bibitem{sota1} Go, Alec, Lei Huang, and Richa Bhayani. "Twitter sentiment analysis." Entropy 17 (2009): 252.
	\bibitem{sota2} Kouloumpis, Efthymios, Theresa Wilson, and Johanna D. Moore. "Twitter sentiment analysis: The good the bad and the omg!." Icwsm 11.538-541 (2011): 164.
	\bibitem{sota3} Tapan Sahni, Chinmay Chandak, Naveen Reddy, Manish Singh. "Efficient Twitter Sentiment Classification using Subjective Distant Supervision"
	\bibitem{sota4} Alec Go, Richa Bhayani,	Lei Huang. "Twitter Sentiment Classification using Distant Supervision"
	\bibitem{glove} Jeffrey Pennington, Richard Socher, Christopher D. Manning. "GloVe: Global Vectors for Word Representation"
	\bibitem{cnn5} Park, S., and Kwak, N. (2016, November). "Analysis on the Dropout Effect in Convolutional Neural Networks." In Asian Conference on Computer Vision (pp. 189-204). Springer, Cham.
	\bibitem{cnn6} http://cs231n.github.io/neural-networks-1/{\#}actfun
\end{thebibliography}
\end{document}


    \filetitle{prior}{Add new prior to system priors object}{systempriors/prior}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
S = prior(S,Expr,PriorFunc,...)
S = prior(S,Expr,[],...)
\end{verbatim}

\paragraph{Input arguments}\label{input-arguments}

\begin{itemize}
\item
  \texttt{S} {[} systempriors {]} - System priors object.
\item
  \texttt{Expr} {[} char {]} - Expression that defines a value for which
  a prior density will be defined; see Description for system properties
  that can be referred to in the expression.
\item
  \texttt{PriorFunc} {[} function\_handle \textbar{} empty {]} -
  Function handle returning the log of prior density; empty prior
  function, \texttt{{[}{]}}, means a uniform prior.
\end{itemize}

\paragraph{Output arguments}\label{output-arguments}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \texttt{S} {[} systempriors {]} - The system priors object with the
  new prior added.
\end{itemize}

\paragraph{Options}\label{options}

\begin{itemize}
\item
  \texttt{'lowerBound='} {[} numeric \textbar{} \emph{\texttt{-Inf}} {]}
  - Lower bound for the prior.
\item
  \texttt{'upperBound='} {[} numeric \textbar{} \emph{\texttt{Inf}} {]}
  - Upper bound for the prior.
\end{itemize}

\paragraph{Description}\label{description}

\subparagraph{System properties that can be used in
\texttt{Expr}}\label{system-properties-that-can-be-used-in-expr}

\begin{itemize}
\item
  \texttt{srf{[}VarName,ShockName,T{]}} - Plain shock response function
  of variables \texttt{VarName} to shock \texttt{ShockName} in period
  \texttt{T}. Mind the square brackets.
\item
  \texttt{ffrf{[}VarName,MVarName,Freq{]}} - Filter frequency response
  function of transition variables \texttt{TVarName} to measurement
  variable \texttt{MVarName} at frequency \texttt{Freq}. Mind the square
  brackets.
\item
  \texttt{corr{[}VarName1,VarName2,Lag{]}} - Correlation between
  variable \texttt{VarName1} and variables \texttt{VarName2} lagged by
  \texttt{Lag} periods.
\item
  \texttt{spd{[}VarName1,VarName2,Freq{]}} - Spectral density between
  variables \texttt{VarName1} and \texttt{VarName2} at frequency
  \texttt{Freq}.
\end{itemize}

If a variable is declared as a
\href{modellang/logvariables}{\texttt{log-variable}}, it must be
referred to as \texttt{log(VarName)} in the above expressions, and the
log of that variables is returned, e.g.
\texttt{srf{[}log(VarName),ShockName,T{]}}. or
\texttt{ffrf{[}log(TVarName),MVarName,T{]}}.

\subparagraph{Expressions involving combinations or functions of
parameters}\label{expressions-involving-combinations-or-functions-of-parameters}

Model parameter names can be referred to in \texttt{Expr} preceded by a
dot (period), e.g. \texttt{.alpha\^{}2 + .beta\^{}2} defines a prior on
the sum of squares of the two parameters (\texttt{alpha} and
\texttt{beta}).

\paragraph{Example}\label{example}

Create a new empty systemprios object based on an existing model.

\begin{verbatim}
s = systempriors(m);
\end{verbatim}

Add a prior on minus the shock response function of variable
\texttt{ygap} to shock \texttt{eps\_pie} in period 4. The prior density
is lognormal with mean 0.3 and std deviation 0.05;

\begin{verbatim}
s = prior(s,'-srf[ygap,eps_pie,4]',logdist.lognormal(0.3,0.05));
\end{verbatim}

Add a prior on the gain of the frequency response function of transition
variable \texttt{ygap} to measurement variable `y' at frequency
\texttt{2*pi/40}. The prior density is normal with mean 0.5 and std
deviation 0.01. This prior says that we wish to keep the cut-off
periodicity for trend-cycle decomposition close to 40 periods.

\begin{verbatim}
s = prior(s,'abs(ffrf[ygap,y,2*pi/40])',logdist.normal(0.5,0.01));
\end{verbatim}

Add a prior on the sum of parameters \texttt{alpha1} and
\texttt{alpha2}. The prior is normal with mean 0.9 and std deviation
0.1, but the sum is forced to be between 0 and 1 by imposing lower and
upper bounds.

\begin{verbatim}
s = prior(s,'alpha1+alpha2',logdist.normal(0.9,0.1), ...
    'lowerBound=',0,'upperBound=',1);
\end{verbatim}