\chapter{Ensembles of cost-sensitive decision trees}\label{ch:9}

\begin{remark}{Outline}
In this chapter, we propose a framework of ensembles of example-dependent cost-sensitive
decision-trees. The algorithms are based in training example-dependent cost-sensitive decision
trees using four different random inducer methods and then blending them using three different
combination approaches.
First, in Section \ref{sec:9:ensemble}, we give the background behind ensemble learning. Then, in
Section \ref{sec:9:ecsdt}, we present the ensembles of cost-sensitive decision-trees framework.
Moreover, in Section \ref{sec:9:theoretical}, we prove theoretically that combining individual
cost-sensitive classifiers achieves better  results in the sense of higher financial savings.
Finally, in Section \ref{sec:9:experiments}, we compare the results of the proposed algorithm,
against state-of-the-art methods, using the five real-world cost-sensitive databases.
\end{remark}

\section{Introduction}
\label{sec:9:intro}

Ensemble learning is a widely studied topic in the machine learning community. The main idea behind
the ensemble methodology is to combine several individual base classifiers in   order to have a
classifier that outperforms each of them \citep{Rokach2009}.

In the context of cost-sensitive classification, some authors have proposed methods for using
ensemble techniques. In \citep{Masnadi-shirazi2011}, the authors proposed a framework for
cost-sensitive boosting that is expected to minimized the losses by using optimal cost-sensitive
decision rules. In \citep{Street2008}, a bagging algorithm with adaptive costs was proposed. In
his doctoral thesis, Nesbitt \citep{Nesbitt2010}, proposed a method for cost-sensitive
tree-stacking. In this method different decision trees are learned, and then combined in a way
that a cost function is minimize. Lastly in \citep{Lomax2013}, a survey of application of
cost-sensitive learning with decision trees is shown, in particular including other methods that
create cost-sensitive ensembles. However, in all these methods, the  misclassification costs only
dependent on the class, therefore, assuming a constant cost across  examples. As a consequence,
these methods are not  well suited for example-dependent cost-sensitive  problems.

In this chapter, we propose a framework of ensembles of example-dependent cost-sensitive
decision-trees. The algorithms are based in training example-dependent cost-sensitive decision
trees using four different random inducer methods and then blending them using three different
combination approaches. Moreover, we propose two new cost-sensitive combination approaches,
cost-sensitive weighted voting and cost-sensitive stacking. The latter being an extension of our
previously proposed cost-sensitive logistic regression. We evaluate the proposed framework using
five different databases from four real-world problems. In particular, credit card fraud detection,
churn modeling, credit scoring and direct marketing. The results show that the proposed method
outperforms state-of-the-art example-dependent cost-sensitive methods in three databases, and have
a similar result in the other two.

\section{Ensemble methods}
\label{sec:9:ensemble}

Nowadays,   ensemble methods are  one
of the most popular and well studied machine learning techniques   \citep{Zhou2012}, and it can be
noted that since 2009 all the first-place and   second-place winners of the KDD-Cup
competition\footnote{\url{https://www.sigkdd.org/kddcup/}}   used  ensemble methods. The core
principle in ensemble learning, is to induce random perturbations into  the learning procedure in
order to produce several different base classifiers from a single  training set, then combining the
base classifiers in order to make the final prediction.  In order to induce the random permutations
and therefore create the different base classifiers,   several methods have been proposed, in
particular: bagging \citep{Breiman1996},   pasting~\citep{Breiman1999}, random forests
\citep{Breiman2001} and random patches   \citep{Louppe2012}. Finally, after  the base   classifiers
are trained, they are typically   combined using either   majority voting,  weighted  voting    or
stacking~\citep{Zhou2012}.

\begin{figure}[t!]
\centering
\includegraphics[scale=.5]{ch9_fig1}
\caption{Main reasons regarding why ensemble methods perform better than
  single models: statistical, computational and representational \citep{Dietterich2000a}.}
\label{fig:9:1}
\end{figure}

As shown in \figurename{~\ref{fig:9:1}, there are three main reasons regarding why ensemble
methods perform better than single models: statistical, computational and representational
\citep{Dietterich2000a}. First, from a statistical point of view, when the learning set is too
small, an algorithm can find several good models within the search space, that arise to the same
performance on the training set $\mathcal{S}$. Nevertheless, without a validation set, there is
a risk of choosing the wrong model. The second reason is computational; in general, algorithms
rely on some local search optimization and may get stuck in a local optima. Then, an ensemble may
solve this by focusing different algorithms to different spaces across the training set. The last
reason is representational. In most cases, for a learning set of finite size, the  true function
$f$ cannot be represented by any of the candidate models. By combining several  models in an
ensemble, it may be possible to obtain a model with a larger coverage across the  space of
representable functions.

  The most typical form of an ensemble is made by combining $T$ different base classifiers.
  Each  base classifier $M(\mathcal{S}_j)$ is trained by applying algorithm $M$ to a random subset
  $\mathcal{S}_j$ of the training set $\mathcal{S}$.  %  $\mathcal{S_j} = RI(\mathcal{S})$
  For simplicity we define $M_j \equiv  M(\mathcal{S}_j)$ for $j=1,\dots,T$, and
  $\mathcal{M}=\{M_j\}_{j=1}^{T}$ a set of base classifiers.
  Then, these models are combined using majority voting to create the ensemble $H$ as follows
  \begin{align}\label{eq:9:majority-vote}
    f_{mv}(\mathcal{S},\mathcal{M}) = \argmax_{c \in \{0,1\}} \sum_{j=1}^T
    \mathbf{1}_c(M_j(\mathcal{S})).
  \end{align}

%   \begin{remark}{Theoretical performance of an ensemble}
%   Moreover, i
\subsection{Theoretical performance of an ensemble}
  If we assume that each one of the $T$ base classifiers has a probability $\rho$ of
  being correct, the probability of an ensemble making the correct decision, assuming independence,
  denoted by $P_c$, can be calculated using the binomial \mbox{distribution \citep{Hansen1990}}
  \begin{equation}\label{eq:9:prob}
    P_c = \sum_{j>T/2}^{T} {{T}\choose{j}} \rho^j(1-\rho)^{T-j}.
  \end{equation}
  Furthermore, as shown in \cite{Lam1997}, if $T\ge3$ then:
  \begin{equation}\label{eq:9:Pc}
  \lim_{T \to  \infty} P_c= \begin{cases}
            1  &\mbox{if } \rho>0.5 \\
            0  &\mbox{if } \rho<0.5 \\
            0.5  &\mbox{if } \rho=0.5 ,
            \end{cases}
  \end{equation}
	leading to the conclusion that
	\begin{equation}\label{eq:9:Pc2}
  \rho \ge 0.5 \quad \text{and} \quad T\ge3 \quad \Rightarrow \quad P_c\ge \rho.
  \end{equation}
%   \end{remark}

\section{Ensembles of cost-sensitive decision trees}
\label{sec:9:ecsdt}

In this section we present our framework of ensembles of  example-dependent
cost-sensitive  decision-trees. The algorithms are based in training example-dependent
cost-sensitive decision trees, see Chapter~\ref{ch:8}, using four different  random inducer methods
and then blending them using three different combination approaches. Moreover, we propose two new
cost-sensitive combination approaches, cost-sensitive weighted  voting and cost-sensitive stacking.
The latter being an extension of our proposed cost-sensitive logistic regression, see
Chapter~\ref{ch:7}.

The remainder of the section is organized as follows: First, we introduce the different random
inducers used to create the base classifiers. Then, we present the combination methods. Finally, we
define our proposed algorithms.

\subsection{Random inducers}

With the objective of creating an ensemble of example-dependent cost-sensitive decision trees, we
first create $T$ different random subsamples $\mathcal{S}_j$ for $j=1,\dots,T$, of the training
set
$\mathcal{S}$, and train a $CSDT$ algorithm on each one. In particular we create the different
subsets using four different methods: bagging \citep{Breiman1996}, pasting \citep{Breiman1999},
random forests \citep{Breiman2001} and random patches \citep{Louppe2012}.

In bagging \citep{Breiman1996}, base classifiers are built on randomly drawn bootstrap subsets of
the original data, hence producing different base classifiers. Similarly, in pasting
\citep{Breiman1999}, the base classifiers are built on random  samples without replacement from
the training set. In random forests \citep{Breiman2001}, using decision trees as the base learner,
bagging   is extended and   combined  with a  randomization of the input features that  are used
when  considering candidates  to split    internal nodes. In particular, instead of looking for
the best  split among all   features, the   algorithm selects, at each node, a random subset of
features  and then determines   the best split only over  these features. In the random patches
algorithm \citep{Louppe2012}, base classifiers are created by randomly     drawn bootstrap subsets
of both examples and features. To further clarify the difference between the random inducer
methods, in \figurename{~\ref{fig:9:2}}, we show a visual representation of the random inducers
algorithms.

\begin{figure}[t!]
\centering
\includegraphics[scale=.55]{ch9_fig2}
\caption{Visual representation of the random inducers algorithms.}
\label{fig:9:2}
\end{figure}

\subsection{Combination methods}

Lastly, the base classifiers are combined using either majority voting, cost-sensitive weighted
voting and cost-sensitive stacking. Majority voting consists in collecting the predictions of
each base classifier and selecting the decision with the highest number of votes, see
(\ref{eq:9:majority-vote}).

\subsubsection{Cost-sensitive weighted voting}

This method is an extension of weighted voting. First, in the traditional approach, a
similar comparison of the votes of the base classifiers is made, but giving a weight $\alpha_j$
to each classifier $M_j$ during the voting phase \citep{Zhou2012}
\begin{align} \label{eq:9:weighted-majority-vote}
  f_{wv}(\mathcal{S},\mathcal{M}, \alpha)
  =\argmax_{c \in \{0,1\}} \sum_{j=1}^T \alpha_j \mathbf{1}_c(M_j(\mathcal{S})),
\end{align}
where $\alpha=\{\alpha_j\}_{j=1}^T$.
The calculation of $\alpha_j$ is related to the performance of each classifier $M_j$.
It is usually defined as the normalized misclassification error   $\epsilon$ of the base
classifier $M_j$  in the out of bag set   $\mathcal{S}_j^{oob}=\mathcal{S}-\mathcal{S}_j$
\begin{equation}
  \alpha_j=\frac{1-\epsilon(M_j(\mathcal{S}_j^{oob}))}{\sum_{j_1=1}^T
  1-\epsilon(M_{j_1}(\mathcal{S}_{j_1}^{oob}))}.
\end{equation}

\noindent However, as previously discussed, the misclassification measure is not suitable in  many
real-world classification problems. We  herein propose a method to calculate the weights $\alpha_j$
 taking into account the actual savings of the classifiers. Therefore using (\ref{eq:3:savings}), we
define
\begin{equation}
  \alpha_j=\frac{Savings(M_j(\mathcal{S}_j^{oob}))}
  {\sum_{j_1=1}^T Savings(M_{j_1}(\mathcal{S}_j^{oob}))}.
\end{equation}
This method guaranties that the base classifiers that contribute to a higher increase in savings
have more importance in the ensemble.

\subsubsection{Cost-sensitive stacking}
\label{sec:9:staking}

The staking method consists in combining the different base classifiers by learning a
second level algorithm on top of them \citep{Wolpert1992}. In this framework, once the base
classifiers are constructed using the training set  $\mathcal{S}$, a new set is constructed
where the output of the base classifiers  are now considered as the features while keeping the
class labels.

Even though there is no restriction on which algorithm can be used as a second level learner,
it is common to use a linear model \citep{Zhou2012}, such as
\begin{align}
  f_s(\mathcal{S},\mathcal{M},\beta) =
  g \left( \sum_{j=1}^T \beta_j M_j(\mathcal{S}) \right),
\end{align}
where $\beta=\{\beta_j\}_{j=1}^T$, and $g(\cdot)$ is the sign function
\mbox{$g(z)=sign(z)$} in the case of a linear regression or the sigmoid function, defined
as \mbox{$g(z)=1/(1+e^{-z})$}, in the case of a logistic regression.

Moreover, following the logic used in \citep{Nesbitt2010}, we propose learning the set of
parameters $\beta$  using our cost-sensitive logistic regression ($CSLR$), see Chapter~\ref{ch:7}.
The $CSLR$ algorithm consists in introducing example-dependent costs into a logistic regression, by
changing the objective function of the model to one that is cost-sensitive. For the specific case of
cost-sensitive stacking, we define the cost function as:
\begin{align}
  &J(\mathcal{S},\mathcal{M},\beta)=
  \sum_{i=1}^{N} \bigg[ y_i\bigg(
  f_s(\mathbf{x}_i,\mathcal{M},\beta) \cdot(C_{TP_i} - C_{FN_i}) + C_{FN_i} \bigg)
  \nonumber \\
  & + (1-y_i)\bigg( f_s(\mathbf{x}_i,\mathcal{M},\beta) \cdot(C_{FP_i} - C_{TN_i})
    +C_{TN_i} \bigg) \bigg].
\end{align}
Then, the parameters $\beta$ that minimize the logistic cost function are used in order to
combine the different base classifiers. However, as discussed in Chapter~\ref{ch:7},
this cost function is not convex for all possible cost matrices, therefore, we use genetic
algorithms to minimize it.

Similarly to cost-sensitive weighting, this method guarantees that the base classifiers that
contribute to a higher increase in savings have more importance in the ensemble. Furthermore,
by learning an additional second level cost-sensitive method, the combination is made such that
the overall   savings measure is maximized.

\subsection{Algorithms}

Finally, Algorithm \ref{alg:9:algorithms} summarizes the $ECSDT$ methods. In total, we
evaluate 12 different algorithms, as four different random inducers (bagging, pasting, random
forest and random patches) and three different combinators (majority voting, cost-sensitive
weighted voting and cost-sensitive stacking) can be selected in order to construct the
cost-sensitive ensemble.

\begin{algorithm}\label{alg:9:algorithms}
The proposed $ECSDT$ algorithms.
\textnormal{
\begin{algorithmic}[1]
\Require $CSDT$ (an example-dependent cost-sensitive decision tree algorithm), $T$ the number of
iterations, $\mathcal{S}$ the training set, $inducer$, $N_e$ number of
examples for each base classifier, $N_f$ number of examples for each base classifier,
$combinator$.
\Function{ECSDT}{$\mathcal{S},T,inducer,N_e,N_f,combinator$}
  \State \textbf{Step 1:} Create the set of base classifiers
  \For{$j  \leftarrow 1$ to $T$}
    \State \textbf{switch}($inducer$)
    \State \textbf{case} Bagging:
      \State $\mathcal{S}_j \leftarrow $ Sample $N_e$ examples from $\mathcal{S}$ with replacement.
    \State \textbf{case} Pasting:
      \State $\mathcal{S}_j \leftarrow $ Sample $N_e$ examples from $\mathcal{S}$ without
      replacement.
    \State \textbf{case} Random forests:
      \State $\mathcal{S}_j \leftarrow $ Sample $N_e$ examples from $\mathcal{S}$ with replacement.
    \State \textbf{case} Random patches:
      \State $\mathcal{S}_j \leftarrow $ Sample $N_e$ examples and $N_f$ features from
      $\mathcal{S}$ with replacement.
    \State \textbf{end switch}
    \State $M_j \leftarrow CSDT(\mathcal{S}_j)$
    \State $\mathcal{S}_j^{oob} \leftarrow \mathcal{S}-\mathcal{S}_j$
    \State $\alpha_j \leftarrow Savings( M_j(\mathcal{S}_j^{oob}))$
  \EndFor
  \State \textbf{Step 2:} Combine the different base classifiers
  \State \textbf{switch}($combinator$)
  \State \textbf{case} Majority voting:
  \State  $H \leftarrow f_{mv}(\mathcal{S}, \mathcal{M})$
  \State \textbf{case} Cost-sensitive weighted voting:
  \State $H \leftarrow f_{wv}(\mathcal{S}, \mathcal{M}, \alpha)$
  \State \textbf{case} Cost-sensitive stacking:
  \State $\beta \leftarrow \argmin_{\beta \in \mathbb{R}^T}
        J(\mathcal{S},\mathcal{M},\beta)$
  \State  $H \leftarrow f_{s}(\mathcal{S}, \mathcal{M},\beta)$
  \State \textbf{end switch}
  \State \Return $H$
\EndFunction
\end{algorithmic}
}
\end{algorithm}

\section{Theoretical analysis of the ECSDT}
\label{sec:9:theoretical}

  Although the above proposed algorithm is simple, there is %little
  no work that has formally
  investigated ensemble performance in terms other than accuracy. In this section, our aim is to
  prove theoretically that combining individual cost-sensitive classifiers achieves better results
  in the sense of higher savings.

  We denote $\mathcal{S}_a$ $a\in \{0,1\}$, as the subset of $\mathcal{S}$
  where the examples belong to the class $a$:
  \begin{equation}\label{eq:9:S_a}
    \mathcal{S}_a = \{\mathbf{x}_i^* \vert y_i = a, i \in 1,\dots,N\},
  \end{equation}
  where $\mathcal{S}=\mathcal{S}_0 \cup \mathcal{S}_1$, $\mathcal{S}_0 \cap \mathcal{S}_1 =
  \varnothing$, and $N_a=\vert \mathcal{S}_a \vert$. Also, we define the average cost of the base
  classifiers as:
  \begin{align}\label{eq:9:avg_cost}
    \overline{Cost} (\mathcal{M}(\mathcal{S}))= \frac{1}{T} \sum_{j=1}^{T} Cost(M_j(\mathcal{S})).
  \end{align}

  \noindent Firstly, we prove the following lemma that states the cost of an ensemble $H$ on the
  subset $\mathcal{S}_a$ is lower than the average cost of the base classifiers on the same set for
  $a  \in \{0,1\}$.

  \begin{lemma}\label{lemma1}
  Let $H$ be an ensemble of $T\ge3$ classifiers $\mathcal{M}=\{M_1, M_2,\dots,M_T\}$, and
  $\mathcal{S}$ a testing set of size $N$. If each one of the base classifiers has a probability
  of being correct higher or equal than one half, $\rho \ge \frac{1}{2}$, and the
  \textit{reasonableness conditions} of the cost matrix are satisfied, then the following holds true
  \begin{align}\label{eq:9:lemma}
    Cost(H(\mathcal{S}_a)) &\le \overline{Cost} (\mathcal{M}(\mathcal{S}_a)) , \quad a  \in\{0,1\},
 \end{align}
  \end{lemma}

  \noindent\begin{proof}
  First, we decompose the total cost of the ensemble by applying equations (\ref{eq:3:cost_total})
  and (\ref{eq:3:cost}). Additionally, we separate the analysis for $a=0$ and $a=1$:

  \textbullet\ $a=0:$
  \begin{align}
  Cost(H(\mathcal{S}_0)) =& \sum_{i=1}^{N_0} y_i(c_i C_{TP_i} + (1-c_i)C_{FN_i})+ \nonumber \\
    & (1-y_i)(c_i C_{FP_i} + (1-c_i)C_{TN_i}) .
  \end{align}

  \noindent Moreover, we know from (\ref{eq:9:prob}) that the probability of an ensemble
  making the right decision, i.e., $y_i=c_i$, for any given  example, is equal to $P_c$.
  Therefore, we can use this probability to estimate the expected savings of an ensemble:
  \begin{align}\label{eq:21}
    Cost(H(\mathcal{S}_0)) &= \sum_{i=1}^{N_0} P_c C_{TN_i} +(1-P_c)C_{FP_i}.
  \end{align}

  \textbullet\ $a=1:$

  \noindent In the case of $\mathcal{S}_1$, and following the same logic as when $a=0$, the cost of
  an  ensemble is:
  \begin{align}\label{eq:22}
    Cost(H(\mathcal{S}_1)) &= \sum_{i=1}^{N_1} P_c C_{TP_i} + (1-P_c)C_{FN_i}.
  \end{align}

  \noindent The second part of the proof consists in analyzing the right hand side of
  (\ref{eq:9:lemma}),   specifically, the average cost of the base classifiers on set
  $\mathcal{S}_a$. To do that, with the help of (\ref{eq:3:cost_total}) and (\ref{eq:9:avg_cost}),
  we may express the average cost of the base classifiers as:
  \begin{align}
    \overline{Cost} (\mathcal{M}(\mathcal{S}_a)) = \frac{1}{T} \sum_{j=1}^{T} \sum_{i=1}^{N_a}
    Cost(M_j(\mathbf{x}_i^*)).
  \end{align}

  \noindent We define the set of base classifiers that make a negative prediction as
  \begin{align}
    \mathcal{T}_{i0}=\{M_j(\mathbf{x}_i^*) \vert M_j(\mathbf{x}_i^*) = 0, j \in 1,\dots,T\},
  \end{align}
  similarly, the set of classifiers that make a positive prediction as
  \begin{align}
    \mathcal{T}_{i1}=\{M_j(\mathbf{x}_i^*) \vert M_j(\mathbf{x}_i^*) = 1, j \in 1,\dots,T\}.
  \end{align}
  Then, by taking the cost of negative and positive predictions from (\ref{eq:3:f_a}), the
  average cost of the base learners becomes:
  \begin{align}
    \overline{Cost} (\mathcal{M}(\mathcal{S}_a)) =&
    \frac{1}{T} \sum_{i=1}^{N_a} \bigg( \vert \mathcal{T}_{i0} \vert \cdot
    Cost(f_0(\mathbf{x}_i^*)) \nonumber \\
    & + \vert \mathcal{T}_{i1} \vert \cdot Cost(f_1(\mathbf{x}_i^*)) \bigg).
  \end{align}

  \noindent We separate the analysis for $a=0$ and $a=1$:

  \textbullet\ $a=0:$

  \begin{align}
    \overline{Cost} (\mathcal{M}(\mathcal{S}_0)) =& \sum_{i=1}^{N_0} \bigg(
    \frac{\vert \mathcal{T}_{i0} \vert}{T} \cdot C_{TN_i}
    + \frac{\vert \mathcal{T}_{i1} \vert}{T} \cdot C_{FP_i}\bigg).
  \end{align}
  Furthermore, we know from (\ref{eq:9:prob}) that an average base classifier will have a correct
  classification probability of $\rho$, then $\frac{\vert \mathcal{T}_{i0} \vert}{T}=\rho$, leading
  to:
  \begin{align}\label{eq:26}
    \overline{Cost} (\mathcal{M}(\mathcal{S}_0)) =& \sum_{i=1}^{N_0}
    \bigg( \rho \cdot C_{TN_i} + (1-\rho) \cdot C_{FP_i} \bigg).
  \end{align}

  \textbullet\ $a=1:$

 \noindent Similarly, for the set $\mathcal{S}_1$, the average classifier will have a correct
  classification probability of $\rho$, then $\frac{\vert \mathcal{T}_{i1} \vert}{T}=\rho$.

  \noindent Therefore,
  \begin{align}\label{eq:27}
    \overline{Cost} (\mathcal{M}(\mathcal{S}_1)) =& \sum_{i=1}^{N_1}
    \bigg( \rho \cdot C_{TP_i} + (1-\rho) \cdot C_{FN_i}\bigg) .
  \end{align}

	\noindent Finally, by replacing in (\ref{eq:9:lemma}) the expected savings of an  ensemble with
  (\ref{eq:21}) for $a=0$ and (\ref{eq:22}) for $a=1$, and the average cost of the base learners
  with  (\ref{eq:26}) for $a=0$ and (\ref{eq:27}) for $a=1$, (\ref{eq:9:lemma}) is rewritten as:

  \noindent for $a=0$:
  \begin{align}\label{eq:9:lemma1}
    \sum_{i=1}^{N_0} \bigg(P_c C_{TN_i} +(1-P_c)C_{FP_i}\bigg) \le
    \sum_{i=1}^{N_0} \bigg(\rho \cdot C_{TN_i} + (1-\rho) \cdot C_{FP_i}\bigg),
  \end{align}
  for $a=1$:
  \begin{align}\label{eq:9:lemma2}
    \sum_{i=1}^{N_1} \bigg(P_c C_{TP_i} + (1-P_c)C_{FN_i} \bigg)\le
    \sum_{i=1}^{N_1}  \bigg(\rho \cdot C_{TP_i} + (1-\rho) \cdot C_{FN_i}\bigg).
  \end{align}

  \noindent Since $\rho \ge \frac{1}{2}$ , then $P_c\ge\rho$ from (\ref{eq:9:Pc2}), and using the
  \textit{reasonableness conditions} described in Section \ref{sec:3:example-dependent}, i.e,
  $C_{FP_i} > C_{TN_i}$ and   $C_{FN_i} > C_{TP_i}$, we find that (\ref{eq:9:lemma1}) and
  (\ref{eq:9:lemma2}) are  True.
  \end{proof}

  Lemma \ref{lemma1} separates the costs on sets $\mathcal{S}_0$ and  $\mathcal{S}_1$. We are
  interested in analyzing the overall savings of an ensemble. In this direction, we demonstrate
  in the following theorem, that the expected savings of an ensemble of classifiers are higher
  than the expected average savings of the base learners.

  \begin{theorem}\label{theorem1}
  Let $H$ be an ensemble of $T\ge3$ classifiers $\mathcal{M}=\{M_1,\dots,M_T\}$, and $\mathcal{S}$ a
  testing set of size $ N $, then the expected savings of using $H$ in
  $\mathcal{S}$ are lower than the average expected savings of the base classifiers, in other words,
  \begin{equation}\label{eq:9:theorem}
    Savings(H(\mathcal{S})) \ge \overline{Savings}(\mathcal{M}(\mathcal{S})).
  \end{equation}
  \end{theorem}

  \begin{proof}
  Given (\ref{eq:3:savings}), (\ref{eq:9:theorem}) is equivalent to
  \begin{equation}\label{eq:9:theorem2}
    Cost(H(\mathcal{S})) \le \overline{Cost} (\mathcal{M}(\mathcal{S})).
  \end{equation}

  \noindent Afterwards, by applying the cost definition of (\ref{eq:3:cost}), and grouping the
  sets of negative and positive examples using (\ref{eq:9:S_a}), (\ref{eq:9:theorem2}) becomes
  \begin{equation}
    \sum_{a\in \{0,1\}} Cost(H(\mathcal{S}_a)) \le \sum_{a\in \{0,1\}} \overline{Cost} (\mathcal{M}
    (\mathcal{S}_a)),
  \end{equation}

  \noindent which can be easily proved using Lemma \ref{lemma1}, since, if the cost of an ensemble
  $H$ is  lower than the average cost of the base classifiers on both $\mathcal{S}_0$ and
  $\mathcal{S}_1$,  implies that it is also lower on the sum of the cost on both sets,
  therefore, proving Theorem~\ref{theorem1}.
  \end{proof}

\section{Experiments}
\label{sec:9:experiments}

For the experiments we use five datasets from four different real world example-dependent
cost-sensitive problems: Credit card fraud detection (see Section~\ref{sec:4:fraud}), credit
scoring (see Section~\ref{sec:4:creditscoring}), churn modeling (see Section~\ref{sec:5:churn}) and
direct marketing (see Section~\ref{sec:5:directmarketing}). The different datasets are summarized
in \tablename{~\ref{tab:4:databases}} and \tablename{~\ref{tab:5:databases}}.

\begin{table}[!t]
    \centering
    \footnotesize
    \rotatebox{270}{
    \begin{tabular}{l l r@{\hskip 0in}c@{\hskip 0in}l r@{\hskip 0in}c@{\hskip 0in}l r@{\hskip
    0in}c@{\hskip 0in}l  r@{\hskip 0in}c@{\hskip 0in}l r@{\hskip 0in}c@{\hskip 0in}l} %sum 7.7
    \hline
    \bf{Family} & \bf{Algorithm} & \multicolumn{3}{c}{\bf{Fraud}} &
    \multicolumn{3}{c}{\bf{Churn}} & \multicolumn{3}{c}{\bf{Credit 1}}
    &  \multicolumn{3}{c}{\bf{Credit 2}} & \multicolumn{3}{c}{\bf{Marketing}} \\
    \hline
CI&DT-t & 0.3176 &$\pm$& 0.0357 & -0.0018 &$\pm$& 0.0194 & 0.1931 &$\pm$& 0.0087& -0.0616 &$\pm$& 0.0229 & -0.2342 &$\pm$& 0.0609 \\
&LR-t & 0.0092 &$\pm$& 0.0002 & -0.0001 &$\pm$& 0.0002 & 0.0177 &$\pm$& 0.0126 & 0.0039 &$\pm$& 0.0012 & -0.2931 &$\pm$& 0.0602\\
&RF-t & 0.3342 &$\pm$& 0.0156 & -0.0026 &$\pm$& 0.0079 & 0.1471 &$\pm$& 0.0071& 0.0303 &$\pm$& 0.0040 & -0.2569 &$\pm$& 0.0637\\
&DT-u & 0.5239 &$\pm$& 0.0118 & -0.0389 &$\pm$& 0.0583 & 0.3287 &$\pm$& 0.0125 & -0.1893 &$\pm$& 0.0314 & -0.0278 &$\pm$& 0.0475\\
&LR-u & 0.1243 &$\pm$& 0.0387 & 0.0039 &$\pm$& 0.0492 & 0.4118 &$\pm$& 0.0313 &  0.1850 &$\pm$& 0.0231 & 0.2200 &$\pm$& 0.0376 \\
&RF-u & 0.5684 &$\pm$& 0.0097 & 0.0433 &$\pm$& 0.0533 & 0.4981 &$\pm$& 0.0079 & 0.1237 &$\pm$& 0.0228 & 0.1227 &$\pm$& 0.0443\\
\hline
CPS&DT-r & 0.3439 &$\pm$& 0.0453 & 0.0054 &$\pm$& 0.0568 & 0.3310 &$\pm$& 0.0126& 0.0724 &$\pm$&0.0212 & 0.1960 &$\pm$& 0.0527 \\
&LR-r & 0.3077 &$\pm$& 0.0301 & 0.0484 &$\pm$& 0.0375 & 0.3965 &$\pm$& 0.0263&  0.2650 &$\pm$& 0.0115 & 0.4210 &$\pm$& 0.0267 \\
&RF-r & 0.3812 &$\pm$& 0.0264 & 0.1056 &$\pm$& 0.0412 & 0.4989 &$\pm$& 0.0080  & 0.3055 &$\pm$& 0.0106 & 0.3840 &$\pm$& 0.0360\\
&DT-o & 0.3172 &$\pm$& 0.0274 & 0.0251 &$\pm$& 0.0195 & 0.1738 &$\pm$& 0.0092 &  0.0918 &$\pm$&0.0225 & -0.2598 &$\pm$& 0.0559\\
&LR-o & 0.2793 &$\pm$& 0.0185 & 0.0316 &$\pm$& 0.0228 & 0.3301 &$\pm$& 0.0109  &  0.2554 &$\pm$&0.0090 & 0.3129 &$\pm$& 0.0277\\
&RF-o & 0.3612 &$\pm$& 0.0295 & 0.0205 &$\pm$& 0.0156 & 0.2128 &$\pm$& 0.0081  &  0.2242 &$\pm$&0.0070 & -0.1782 &$\pm$& 0.0618\\
\hline
BMR&DT-t-BMR & 0.6045 &$\pm$& 0.0386 & 0.0298 &$\pm$& 0.0145 & 0.1054 &$\pm$& 0.0358& 0.2740 &$\pm$& 0.0067 & 0.4598 &$\pm$& 0.0089 \\
&LR-t-BMR & 0.4552 &$\pm$& 0.0203 & 0.1082 &$\pm$& 0.0316 & 0.2189 &$\pm$& 0.0541 & \bf{0.3148} &\bf{$\pm$}& \bf{0.0094} & \bf{0.4973} &\bf{$\pm$}& \bf{0.0084}\\
&RF-t-BMR & 0.6414 &$\pm$& 0.0154 & 0.0856 &$\pm$& 0.0354 & 0.4924 &$\pm$& 0.0087 & 0.3133 &$\pm$& 0.0094 & 0.4807 &$\pm$& 0.0093\\
  \hline
  \multicolumn{17}{c}{(Models with the highest savings are marked in bold)}
  \end{tabular}
  }
    \caption{Results of the CI and BMR algorithms measured by savings}
    \label{tab:9:results_savings}
  \end{table}

\begin{table}[!t]
    \centering
    \footnotesize
    \rotatebox{90}{
    \begin{tabular}{l l r@{\hskip 0in}c@{\hskip 0in}l r@{\hskip 0in}c@{\hskip 0in}l r@{\hskip
    0in}c@{\hskip 0in}l  r@{\hskip 0in}c@{\hskip 0in}l r@{\hskip 0in}c@{\hskip 0in}l} %sum 7.7
    \hline
    \bf{Family} & \bf{Algorithm} & \multicolumn{3}{c}{\bf{Fraud}} &
    \multicolumn{3}{c}{\bf{Churn}} & \multicolumn{3}{c}{\bf{Credit 1}}
    &  \multicolumn{3}{c}{\bf{Credit 2}} & \multicolumn{3}{c}{\bf{Marketing}} \\
    \hline
CST&CSLR-t & 0.6113 &$\pm$& 0.0262 & 0.1118 &$\pm$& 0.0484 & 0.4554 &$\pm$& 0.1039  & 0.2748 &$\pm$&0.0069 & 0.4484 &$\pm$& 0.0072\\
&CSDT-t & 0.7116 &$\pm$& 0.2557 & 0.1115 &$\pm$& 0.0378 & 0.4829 &$\pm$& 0.0098 &  0.2835 &$\pm$& 0.0078 & 0.4741 &$\pm$& 0.0063\\
\hline
ECSDT&CSB-mv-t & 0.7124 &$\pm$& 0.0162 & 0.1237 &$\pm$& 0.0368 & 0.4862 &$\pm$& 0.0102 &  0.2945 &$\pm$& 0.0105 & 0.4837 &$\pm$& 0.0078\\
&CSB-wv-t & 0.7276 &$\pm$& 0.0116 & 0.1539 &$\pm$& 0.0255 & 0.4862 &$\pm$& 0.0102  & 0.2948 &$\pm$&0.0106 & 0.4838 &$\pm$& 0.0079\\
&CSB-s-t & 0.7181 &$\pm$& 0.0109 & 0.1441 &$\pm$& 0.0364 & 0.4847 &$\pm$& 0.0096 &  0.2856 &$\pm$& 0.0088 & 0.4769 &$\pm$& 0.0078\\
&CSP-mv-t & 0.7106 &$\pm$& 0.0113 & 0.1227 &$\pm$& 0.0399 & 0.4853 &$\pm$& 0.0104  & 0.2919 &$\pm$& 0.0097 & 0.4831 &$\pm$& 0.0081\\
&CSP-wv-t & 0.7244 &$\pm$& 0.0202 & 0.1501 &$\pm$& 0.0302 & 0.4854 &$\pm$& 0.0105 & 0.2921 &$\pm$& 0.0098 & 0.4832 &$\pm$& 0.0082\\
&CSP-s-t & 0.7212 &$\pm$& 0.0067 & 0.1488 &$\pm$& 0.0272 & 0.4848 &$\pm$& 0.0084  &  0.2870 &$\pm$& 0.0084 & 0.4752 &$\pm$& 0.0089\\
&CSRF-mv-t & 0.6498 &$\pm$& 0.0598 & 0.0300 &$\pm$& 0.0488 & 0.4980 &$\pm$& 0.0120 &  0.2274 &$\pm$& 0.0520 & 0.3929 &$\pm$& 0.0655\\
&CSRF-wv-t & 0.7249 &$\pm$& 0.0742 & 0.0624 &$\pm$& 0.0477 & 0.4979 &$\pm$& 0.0124 &  0.2948 &$\pm$& 0.0079 & 0.4728 &$\pm$& 0.0125\\
&CSRF-s-t & 0.6731 &$\pm$& 0.0931 & 0.0586 &$\pm$& 0.0507 & 0.4839 &$\pm$& 0.0160  &  0.2518 &$\pm$& 0.0281 & 0.3854 &$\pm$& 0.0899\\
&CSRP-mv-t & 0.7220 &$\pm$& 0.0082 & 0.1321 &$\pm$& 0.0280 & \bf{0.5154} &\bf{$\pm$}& \bf{0.0077}& 0.3053 &$\pm$& 0.0087 & 0.4960 &$\pm$& 0.0075 \\
&CSRP-wv-t & \bf{0.7348} &\bf{$\pm$}& \bf{0.0131} & 0.1615 &$\pm$& 0.0252 & 0.5152 &$\pm$& 0.0083 & 0.3015 &$\pm$& 0.0086 & 0.4885 &$\pm$& 0.0076\\
&CSRP-s-t & 0.7336 &$\pm$& 0.0108 & \bf{0.1652} &\bf{$\pm$}& \bf{0.0264} & 0.4989 &$\pm$& 0.0088& 0.2956 &$\pm$& 0.0078 & 0.4878 &$\pm$& 0.0080 \\
  \hline
  \multicolumn{17}{c}{(Models with the highest savings are marked in bold)}
  \end{tabular}
  }
    \caption{Results of the CST and ECSDT algorithms measured by savings}
    \label{tab:9:results_savings2}
  \end{table}

  We first use three classification algorithms, decision tree ($DT$), logistic
  regression ($LR$) and random forest ($RF$). Using the implementation of \textit{Scikit-learn}
  \citep{Pedregosa2011}, each algorithm is trained using the different training sets: training
  ($t$), under-sampling ($u$), cost-proportionate rejection-sampling  ($r$) \citep{Zadrozny2003}
  and   cost-proportionate over-sampling ($o$) \citep{Elkan2001}. Afterwards,  we evaluate the
  results of  the algorithms using $BMR$, see Chapter~\ref{ch:6}. Then, the cost-sensitive
  logistic  regression ($CSLR$), see Chapter\ref{ch:7}, and cost-sensitive decision tree ($CSDT$),
  see Chapter~\ref{ch:8}, are also evaluated. Lastly, we calculate the propose ensembles of
  cost-sensitive decision trees algorithms. In particular, using each of the random inducer
  methods, bagging ($CSB$), pasting ($CSP$), random forests ($CSRF$) and random patches ($CSRP$),
  and then blending the base classifiers using each one of the combination methods; majority
  voting ($mv$), cost-sensitive weighted voting ($wv$) and cost-sensitive stacking ($s$). Unless
  otherwise stated, the random selection of the training set was repeated 50 times, and in each
  time the models were trained and results collected, this allows us to measure the stability of
  the results. The implementation of the cost-sensitive algorithms is done using
  the \textit{CostCla} library, see Appendix~\ref{ch:A}.

    The results are shown in \tablename{ \ref{tab:9:results_savings}} and \tablename{ \ref{tab:9:results_savings2}}. First, when observing  the
  results of the cost-insensitive methods ($CI$), that is, $DT$, $LR$ and $RF$ algorithms trained
  on the $t$ and $u$ sets, the $RF$ algorithm produces the best result by savings in three out of
  the five sets, followed by the $LR-u$. It is also clear that the results on the $t$ dataset are
  not as good as the ones on the $u$, this is highly related to the unbalanced distribution of the
  positives and negatives in all the databases.

    \begin{table}[!t]
     \centering
    \footnotesize
    \begin{tabular}{ll c c  } %sum 7.7
      \hline
      \bf{Family} & \bf{Algorithm} & \bf{F-Rank} & \bf{perBest}\\
      \hline
        ECSDT&CSRP-wv-t&2.6&98.35\\
        ECSDT&CSRP-s-t&3.4&97.72\\
        ECSDT&CSRP-mv-t&4.0&94.99\\
        ECSDT&CSB-wv-t&5.6&95.49\\
        ECSDT&CSP-wv-t&7.4&94.72\\
        ECSDT&CSB-mv-t&8.2&91.39\\
        ECSDT&CSRF-wv-t&9.4&84.35\\
        BMR&RF-t-BMR&9.4&86.16\\
        ECSDT&CSP-s-t&9.6&93.80\\
        ECSDT&CSP-mv-t&10.2&91.00\\
        ECSDT&CSB-s-t&10.2&93.12\\
        BMR&LR-t-BMR&11.2&73.98\\
        CPS&RF-r&11.6&77.37\\
        CST&CSDT-t&12.6&88.69\\
        CST&CSLR-t&14.4&83.34\\
        ECSDT&CSRF-mv-t&15.2&70.88\\
        ECSDT&CSRF-s-t&16.0&75.68\\
        CI&RF-u&17.2&52.83\\
        CPS&LR-r&19.0&63.39\\
        BMR&DT-t-BMR&19.0&60.05\\
        CPS&LR-o&21.0&53.05\\
        CPS&DT-r&22.6&35.33\\
        CI&LR-u&22.8&40.43\\
        CPS&RF-o&22.8&34.81\\
        CI&DT-u&24.4&27.01\\
        CPS&DT-o&25.0&24.25\\
        CI&DT-t&26.0&16.14\\
        CI&RF-t&26.2&16.73\\
        CI&LR-t&28.0&1.19\\
      \hline
    \end{tabular}
    \caption{Savings Friedman ranking and average percentage of best result}
    \label{tab:9:results_ranking}
  \end{table}
  \begin{table}[t]
    \centering
    \footnotesize
    \begin{tabular}{l c c c c c  } %sum 7.7
      \hline
      \bf{Algorithm} & \bf{Fraud} & \bf{Churn} &\bf{Credit1} & \bf{Credit2} & \bf{Marketing} \\
      \hline
      RF-u&17&18&5&23&23\\
      RF-r&20&13&3&3&19\\
      RF-t-BMR&14&14&8&2&9\\
      CSDT-t&10&11&16&14&12\\
      CSRP-wv-t&1&2&2&5&3\\
      \hline
    \end{tabular}
    \caption{Savings ranks of best algorithm of each family by database}
    \label{tab:9:results_best}
  \end{table}

  \begin{figure}[t]
    \centering
    \includegraphics{ch9_fig3}
    \caption{\textbf{Comparison of the savings of the algorithms versus the highest savings in
    each database.} The $CSRP-wt$ is very close to the best result in all the databases.
    Additionally, even though the $LR-BMR$ is the best algorithm in two databases, the performance
    in the other three is very poor.}
    \label{fig:9:comparison_best}
  \end{figure}

  In the case of cost-proportionate sampling methods ($CPS$), specifically the
  cost-proportionate rejection sampling ($r$) and cost-proportionate over
  sampling ($o$). It is observed than in four cases the savings increases quite
  significantly. It is on the fraud detection database where these methods do not outperform the
  algorithms trained on the under-sampled set. This may be related to the fact that in this
  database the initial percentage of positives is 1.5\% which is similar to the percentage in the
  $r$ and   $o$ sets. However it is 50.42\% in the $u$ set, which may help explain why this method
  performs much better as measured by savings.

  Afterwards, in the case of the $BMR$ algorithms, the results show that this method outperforms
  the previous ones in four cases and has almost the same result in the other set. In the fraud
  detection  set, the results are quite better, since the savings of the three classification
  algorithms increase when using this methodology. The next family of algorithms is the
  cost-sensitive training, which includes the $CSLR$ and $CSDT$ techniques. In this case, only in
  two databases the results are improved. Lastly, we evaluate the proposed $ECSDT$ algorithms. The
  results show that these methods arise to the best overall results in three sets, while being
  quite competitive in the others.

  Subsequently, in order to statistically sort the classifiers we computed the Friedman ranking
  (F-Rank)  statistic \citep{Demsar2006}. This rank increases with the cost of the algorithms.
  We also calculate the average savings of each algorithm compared with the highest savings in
each set (perBest), as a  measure of how close are the savings of an algorithm to the best result.
  In \tablename{~\ref{tab:9:results_ranking}}, the results are shown. It is observed that  the
  first six algorithms, according to the F-Rank, belong to the $ECSDT$ family. In particular, the
  best three classifiers is the ensemble of cost-sensitive decision trees using the random patches
  approach. Giving the best result the one that blends the base classifiers using weighted voting
  method. Moreover as shown in  \tablename{~\ref{tab:9:results_best}}, this method ranks on each
  dataset \nth{1}, \nth{2}, \nth{2}, \nth{5} and \nth{3}, respectively. For comparison the best
  method from an other family is the $RF$ with $BMR$, which ranks \nth{14}, \nth{14}, \nth{8},
  \nth{2} and \nth{9}.

\begin{figure}[!t]
  \centering
  \subfloat[Comparison of the Friedman ranking.]{\includegraphics{ch9_fig4}%
    \label{fig:9:family_rank}}
  \hfil
  \subfloat[Comparison of the average savings of the algorithms versus the
    highest savings.]{\includegraphics{ch9_fig5}%
    \label{fig:9:family_avgsav}}
  \caption{\textbf{Comparison of the results by family of classifiers.} The $ECSDT$ family has the
  best performance measured either by Friedman ranking or average percentage of best model.}
  \label{fig:9:comparison_family}
\end{figure}

   \begin{figure}[!t]
  \centering
  \subfloat[Comparison by inducer methodology.]{\includegraphics{ch9_fig6}%
    \label{fig:9:rank_ecsdt1}}
  \hfil
  \subfloat[Comparison by combination of base classifiers approach.]{\includegraphics{ch9_fig7}%
    \label{fig:9:rank_ecsdt2}}
  \caption{\textbf{Comparison of the Friedman ranking within the $ECSDT$ family.} Overall, the
  random inducer method that provides the best results is the $CSRP$. Moreover, the best
  combination method compared by Friedman ranking is the cost-sensitive weighted voting.}
  \label{fig:9:rank_ecsdt}
\end{figure}

  Moreover, when analyzing the perBest statistic, it is observed that it follows almost the same
  order as the F-Rank. Notwithstanding, there are cases in which algorithms ranks are different in
  the two statistics, for example the $CSDT-t$ algorithm has a lower F-Rank than the $RF-BMR$,
  but the perBest if better. This happens because, the F-Rank does not take into account the
  difference in savings within algorithms. This can be further investigated in \figurename{
  \ref{fig:9:comparison_best}}. Even that ranks of the $BMR$ models are better than the $CSDT$, the
  latter is on average closer to the best performance method in each set. Moreover, it is confirmed
  that the $CSRP-wt$ is very close to the best result in all cases. Lastly, it is shown why the
  F-Rank of the $LR-BMR$ is high, given the fact that is the best model in two databases. The
  reason  for that, is because the performance on the other sets is very poor.

  Furthermore \figurename{~\ref{fig:9:family_rank}}, shows the Friedman ranking of each family of
  classifiers. The $ECSDT$ methods are overall better, followed by the $BMR$ and the $CST$ methods.
  As expected, the $CI$ family is the one that performs the worst. Nevertheless, there is
  a significant variance within the ranks in the $ECSDT$ family, as the best one has a Friedman
  ranking of 2.6 and the worst 16. Similar results are found when observing the perBest shown in
  \figurename{~\ref{fig:9:family_avgsav}}. However, in the case of the perBest, the $CST$ methods
  perform better than the $BMR$. It is important, in both cases it is confirmed that the $ECSDT$
  family of methods is the one that arise to the best results as measured by savings.

  \begin{figure}[t!]
  \centering
  \includegraphics{ch9_fig8}
  \caption{\textbf{Comparison of the Friedman ranking of the savings and F1Score sorted by F1Score
  ranking.} The best two algorithms according to their Friedman rank of F1Score are indeed the
  best ones measured by the Friedman rank of the savings. However, this relation does not
  consistently hold for the other algorithms as the correlation between the rankings is just
  65.10\%.}
  \label{fig:9:friedman_vs_fscore}
  \end{figure}

  We further investigate the different methods that compose the $ECSDT$ family, first by inducer
  methods and by the combination approach. In \figurename{ \ref{fig:9:rank_ecsdt1}}, the Friedman
  ranking of the $ECSDT$ methods grouped by inducer algorithm are shown. It is observed that the
  worst method is the random forest methodology. This may be related to the fact that within the
  random inducer methods, this is the only one that also modified the learner algorithm in the
  sense that it randomly select features for each step during the decision tree growing. Moreover,
  as expected the bagging and pasting methods perform quite similar, after all the only difference
  is that in bagging the sampling is done with replacement, while it is not the case in pasting.
  In  general the best methodology is random patches. Additionally, in \figurename{
  \ref{fig:9:rank_ecsdt2}}, a similar analysis is made taking into account the combination of base
  classifiers approach. In this case, the best combination method is weighted voting, while
  majority voting and staking have a similar performance.

\begin{table}[!t]
    \centering
    \footnotesize
    \rotatebox{90}{
    \begin{tabular}{l l r@{\hskip 0in}c@{\hskip 0in}l r@{\hskip 0in}c@{\hskip 0in}l r@{\hskip
    0in}c@{\hskip 0in}l  r@{\hskip 0in}c@{\hskip 0in}l r@{\hskip 0in}c@{\hskip 0in}l} %sum 7.7
    \hline
    \bf{Family} & \bf{Algorithm} & \multicolumn{3}{c}{\bf{Fraud}} &
    \multicolumn{3}{c}{\bf{Churn}} & \multicolumn{3}{c}{\bf{Credit 1}}
    &  \multicolumn{3}{c}{\bf{Credit 2}} & \multicolumn{3}{c}{\bf{Marketing}} \\
    \hline
CI&DT-t & 0.4458 &$\pm$& 0.0133 & 0.0733 &$\pm$& 0.0198 & 0.2593 &$\pm$& 0.0068&  0.2614 &$\pm$& 0.0083 & 0.2647 &$\pm$& 0.0079 \\
&LR-t & 0.1531 &$\pm$& 0.0045 & 0.0000 &$\pm$& 0.0000 & 0.0494 &$\pm$& 0.0277 & 0.0155 &$\pm$& 0.0037 & 0.2702 &$\pm$& 0.0125\\
&RF-t & 0.2061 &$\pm$& 0.0041 & 0.0249 &$\pm$& 0.0146 & 0.2668 &$\pm$& 0.0085  &  0.0887 &$\pm$&0.0061 & 0.2884 &$\pm$& 0.0116\\
&DT-u & 0.1502 &$\pm$& 0.0066 & 0.1175 &$\pm$& 0.0103 & 0.2276 &$\pm$& 0.0044&  0.3235 &$\pm$&0.0055 & 0.2659 &$\pm$& 0.0061 \\
&LR-u & 0.0241 &$\pm$& 0.0163 & 0.1222 &$\pm$& 0.0098 & 0.3160 &$\pm$& 0.0314  &  \bf{0.3890} &\bf{$\pm$}& \bf{0.0053} & 0.3440 &$\pm$& 0.0083\\
&RF-u & 0.0359 &$\pm$& 0.0065 & 0.1346 &$\pm$& 0.0112 & 0.3193 &$\pm$& 0.0053  & 0.3815 &$\pm$&0.0051 & 0.3088 &$\pm$& 0.0065\\
\hline
CPS&DT-r & 0.4321 &$\pm$& 0.0086 & 0.1206 &$\pm$& 0.0132 & 0.2310 &$\pm$& 0.0049 &0.3409 &$\pm$&0.0046 & 0.2739 &$\pm$& 0.0076 \\
&LR-r & 0.1846 &$\pm$& 0.0123 & 0.1258 &$\pm$& 0.0111 & 0.3597 &$\pm$& 0.0156  &  0.3793 &$\pm$& 0.0049 & 0.3374 &$\pm$& 0.0101\\
&RF-r & 0.2171 &$\pm$& 0.0100 & 0.1450 &$\pm$& 0.0131 & 0.3361 &$\pm$& 0.0067 & 0.3570 &$\pm$& 0.0048 & 0.3103 &$\pm$& 0.0075\\
&DT-o & \bf{0.4495} &\bf{$\pm$}& \bf{0.0063} & 0.1022 &$\pm$& 0.0180 & 0.2459 &$\pm$& 0.0081 & 0.3258 &$\pm$& 0.0056 & 0.2634 &$\pm$& 0.0089\\
&LR-o & 0.1776 &$\pm$& 0.0117 & 0.1085 &$\pm$& 0.0203 & 0.3769 &$\pm$& 0.0067 & 0.3804 &$\pm$& 0.0044 & \bf{0.3568} &\bf{$\pm$}& \bf{0.0102}\\
&RF-o & 0.2129 &$\pm$& 0.0080 & 0.0841 &$\pm$& 0.0201 & 0.3281 &$\pm$& 0.0078  & 0.3212 &$\pm$& 0.0054 & 0.3083 &$\pm$& 0.0093\\
\hline
BMR&DT-t-BMR & 0.2139 &$\pm$& 0.0215 & 0.0941 &$\pm$& 0.0157 & 0.1514 &$\pm$& 0.0390 & 0.3338 &$\pm$& 0.0052 & 0.2433 &$\pm$& 0.0071\\
&LR-t-BMR & 0.1384 &$\pm$& 0.0044 & 0.1370 &$\pm$& 0.0150 & 0.1915 &$\pm$& 0.0340  &  0.3572 &$\pm$&0.0045 & 0.2954 &$\pm$& 0.0079\\
&RF-t-BMR & 0.2052 &$\pm$& 0.0183 & 0.1264 &$\pm$& 0.0156 & 0.3186 &$\pm$& 0.0072&  0.3551 &$\pm$&0.0053 & 0.2744 &$\pm$& 0.0070\\
      \hline
      \multicolumn{17}{c}{(Models with the highest $F_1Score$ are marked in bold)}
    \end{tabular}
    }
\caption{Results of the CI and BMR algorithms measured by $F_1Score$}
\label{tab:9:results_f1score}
  \end{table}

\begin{table}[!t]
    \centering
    \footnotesize
    \rotatebox{270}{
    \begin{tabular}{l l r@{\hskip 0in}c@{\hskip 0in}l r@{\hskip 0in}c@{\hskip 0in}l r@{\hskip
    0in}c@{\hskip 0in}l  r@{\hskip 0in}c@{\hskip 0in}l r@{\hskip 0in}c@{\hskip 0in}l} %sum 7.7
    \hline
    \bf{Family} & \bf{Algorithm} & \multicolumn{3}{c}{\bf{Fraud}} &
    \multicolumn{3}{c}{\bf{Churn}} & \multicolumn{3}{c}{\bf{Credit 1}}
    &  \multicolumn{3}{c}{\bf{Credit 2}} & \multicolumn{3}{c}{\bf{Marketing}} \\
    \hline
CST&CSLR-t & 0.2031 &$\pm$& 0.0065 & 0.1134 &$\pm$& 0.0151 & 0.1454 &$\pm$& 0.0517  & 0.3363 &$\pm$&0.0045 & 0.2339 &$\pm$& 0.0051\\
&CSDT-t & 0.2522 &$\pm$& 0.0980 & 0.1288 &$\pm$& 0.0194 & 0.2754 &$\pm$& 0.0059 & 0.3483 &$\pm$& 0.0046 & 0.2680 &$\pm$& 0.0060\\
\hline
ECSDT&CSB-mv-t & 0.2112 &$\pm$& 0.0125 & 0.1481 &$\pm$& 0.0122 & 0.2927 &$\pm$& 0.0108 & 0.3503 &$\pm$& 0.0046 & 0.2758 &$\pm$& 0.0072\\
&CSB-wv-t & 0.2112 &$\pm$& 0.0091 & 0.1686 &$\pm$& 0.0125 & 0.2926 &$\pm$& 0.0108  &  0.3503 &$\pm$&0.0046 & 0.2757 &$\pm$& 0.0072\\
&CSB-s-t & 0.2072 &$\pm$& 0.0103 & 0.1554 &$\pm$& 0.0121 & 0.2818 &$\pm$& 0.0075  &  0.3533 &$\pm$& 0.0046 & 0.2799 &$\pm$& 0.0074\\
&CSP-mv-t & 0.2098 &$\pm$& 0.0126 & 0.1480 &$\pm$& 0.0136 & 0.2903 &$\pm$& 0.0108&0.3499 &$\pm$& 0.0044 & 0.2749 &$\pm$& 0.0064 \\
&CSP-wv-t & 0.2099 &$\pm$& 0.0054 & 0.1651 &$\pm$& 0.0120 & 0.2905 &$\pm$& 0.0110 & 0.3498 &$\pm$&0.0045 & 0.2749 &$\pm$& 0.0064\\
&CSP-s-t & 0.2064 &$\pm$& 0.0069 & 0.1590 &$\pm$& 0.0099 & 0.2809 &$\pm$& 0.0062&  0.3524 &$\pm$&0.0046 & 0.2778 &$\pm$& 0.0104 \\
&CSRF-mv-t & 0.2208 &$\pm$& 0.0022 & 0.1081 &$\pm$& 0.0224 & 0.2994 &$\pm$& 0.0226& 0.3560 &$\pm$&0.0118 & 0.2780 &$\pm$& 0.0106 \\
&CSRF-wv-t & 0.2175 &$\pm$& 0.0019 & 0.1220 &$\pm$& 0.0234 & 0.2992 &$\pm$& 0.0236  & 0.3549 &$\pm$&0.0069 & 0.2552 &$\pm$& 0.0187\\
&CSRF-s-t & 0.2169 &$\pm$& 0.0045 & 0.1304 &$\pm$& 0.0162 & 0.2916 &$\pm$& 0.0236  &  0.3540 &$\pm$&0.0071 & 0.2578 &$\pm$& 0.0186\\
&CSRP-mv-t & 0.2691 &$\pm$& 0.0054 & 0.1511 &$\pm$& 0.0110 & 0.4031 &$\pm$& 0.0079  &0.3743 &$\pm$& 0.0050 & 0.2916 &$\pm$& 0.0062\\
&CSRP-wv-t & 0.2780 &$\pm$& 0.0041 & \bf{0.1710} &\bf{$\pm$}& \bf{0.0140} & \bf{0.4049}
&\bf{$\pm$}& \bf{0.0066} & 0.3721 &$\pm$& 0.0051 & 0.2781 &$\pm$& 0.0063\\
&CSRP-s-t & 0.2735 &$\pm$& 0.0148 & 0.1622 &$\pm$& 0.0103 & 0.3953 &$\pm$& 0.0141  & 0.3720 &$\pm$& 0.0049 & 0.2922 &$\pm$& 0.0137\\
      \hline
      \multicolumn{17}{c}{(Models with the highest $F_1Score$ are marked in bold)}
    \end{tabular}
    }
\caption{Results of the CST and ECSDT algorithms measured by $F_1Score$}
\label{tab:9:results_f1score2}
  \end{table}

  Finally, in \tablename{~\ref{tab:9:results_f1score}} and \tablename{~\ref{tab:9:results_f1score2}} the results of the algorithms measured by
  F1Score are shown. It is observed that the model with the highest savings is not the same as the
  one with the highest F1Score in all of the databases as selecting a method by a traditional
  statistic does not give the same result as selecting it using a business oriented measure such
  as financial savings. This can be  further examined in
  \mbox{\figurename{~\ref{fig:9:friedman_vs_fscore}}}, where the ranking of
  the F1Score and savings are compared. It is observed that the best two algorithms according to
  their Friedman rank of F1Score are indeed the best ones measured by the Friedman rank of the
  savings. However, this relation does not consistently hold for the other algorithms as the
  correlation between the rankings is just 65.10\%.


\documentclass[12pt,a4paper]{article}
\usepackage{researchpack}

\usepackage{xcolor}
\newcommand{\andrea}[1]{{\bf \textcolor{blue}{{Andrea: #1}}}}
\newcommand{\stefano}[1]{{\bf \textcolor{violet}{{Stefano: #1}}}}
\newcommand{\bereket}[1]{{\bf \textcolor{red}{{Bereket: #1}}}}

\title{Social Elicitation}

\begin{document}
\maketitle

\paragraph{Setting.} We have:

\begin{itemize}

    \item A set of configurations $\calX$ implicitly defined by hard constraints.

    \item A configuration $\vx \in \calX$ is a vector of $N$ 0-1 attributes.

    \item A set of users $u \in \{1, \ldots, M\}$.

    \item Users have true preferences $\vw^u_* \in \bbR^N_+$.

    \item Users have learned preferences $\vw^u \in \bbR^N_+$.  These may be
        learned using multi-user setmargin.

    \item Let $W \in \bbR^{N \times M}_+$ be the matrix obtained by stacking
        the preference vectors side-by-side. $W_*$ is the analogue for the
        true preferences.

    \item We assume that the weights $W$ are observed.

    \item The aggregate preference of the users \emph{as a group} are
        defined as a function $f$ of $W$.

        Initially, we will focus on linear combinations:
        %
        $$ f(\vx;W) := \sum_{u=1}^M \omega_u \langle \vw^u, \vx \rangle $$

        Linear combinations can capture the contribution of different users
        to the overall utility of configurations.

    \item Our goal is to learn the parameters $\vomega \in \bbR^M$ from
        collective, group-wise feedback.

\end{itemize}

\paragraph{Collective feedback.} The overall idea is:

\begin{itemize}

    \item Feedback is collective: when queried with a set of $S$ configurations
        $\vx^1, \ldots, \vx^S$, the group indicates the overall most preferred
        configuration, say $\vx^i$.

        (Let's just use $S = 2$ for starters.)

    \item We update the user contributions $\vomega$ to $f$ using the
        feedback: users whose $\vw^u$ ranks $\vx^i$ high are most important,
        and users who rank $\vx^i$ low are less important.

    \item Feedback: choosing $\vx^i$ out of the $S$ query configurations
        amounts to a set of $S - 1$ ranking constraints of the form:
        %
        \begin{align*}
            \forall j \in [S], j \ne i
                & \quad f(\vx^i) \succcurlyeq f(\vx^j)    \\
                & \quad \Longleftrightarrow \langle \sum_{u=1}^M \omega_u \vw^u, \vx^i \rangle \ge \langle \sum_{u=1}^M \omega_u \vw^u, \vx^j \rangle \\
                & \quad \Longleftrightarrow  \sum_{u=1}^M \omega_u \langle \vw^u, \vx^i \rangle \ge \sum_{u=1}^M \omega_u \langle \vw^u, \vx^j \rangle \\
                & \quad \Longleftrightarrow  \sum_{u=1}^M \omega_u \langle \vw^u, \vx^i - \vx^j \rangle \ge 0
        \end{align*}
        %
        The gradient of this expression w.r.t. $\omega_u$ for a single pair
        $i, j$ is:
        %
        $$ \frac{\partial}{\partial \omega_u} \sum_u \omega_u \langle \vw^u, \vx^i - \vx^j \rangle = \langle \vw^u, \vx^i - \vx^j \rangle $$

        This leads to a Perceptron update of the form:
        %
        $$ \omega^{t+1}_u \gets \omega^t_u + \eta \sum_{j \ne i} \langle \vw^u, \vx^i - \vx^j \rangle $$
        %
        or, more coincisely:
        %
        $$ \vomega^{t+1} \gets \vomega^t + \eta W^\top \sum_{j \ne i} (\vx^i - \vx^j) $$
        %
        where $\eta$ is the step size (just $\frac{2}{S(S - 1)}$ for starters).

    \item The (initial) algorithm is as follows:

        \begin{itemize}

            \item Initialize $\vomega^1$ to zero (or at random, e.g. from a
                standard Normal distribution).

            \item Select a set of $S$ objects with maximal utility and
                diversity, for instance by solving:
                %
                \begin{align*}
                    \max_{\vx^1, \ldots, \vx^2}
                        & \;\; \lambda \sum_{i=1}^S f(\vx^i) + (1 - \lambda) \sum_{i < j} \|\vx^i - \vx^j\|_1 \\
                    \text{s.t.}
                        & \;\; f(\vx^i) \ne f(\vx^j)
                            & \forall i,j \in [S], i < j
                \end{align*}
                %
                where $f$ is computed using the current value for $\vomega$.
                Here $\lambda \in [0, 1]$ is a user-provided hyper-parameter,
                and $\|\cdot\|_1$ is the $\ell_1$ norm, i.e.:
                %
                $$ \|\vx\|_1 := \sum_{z=1}^N |x_z| $$
                %
                Note that:
                %
                $$ \|\vx^i - \vx^j\|_1 = \sum_{z=1}^N |x^i_z - x^j_z| = \sum_{z=1}^N \max \{x^i_z - x^j_z, x^j_z - x^i_z\} $$

                Encoding the above as a MILP problem is a bit tricky; here
                is a derivation for the $S=2$ case (the others are similar):
                %
                \begin{align*}
                    \max_{\vx^1, \vx^2}
                        & \;\; \lambda (f(\vx^1) + f(\vx^2)) + (1 - \lambda) \|\vx^1 - \vx^2\|_1 \\
                    \text{s.t.}
                        & \;\; f(\vx^1) \ne f(\vx^2) \\
                    = \max_{\vx^1, \vx^2}
                        & \;\; \lambda (f(\vx^1) + f(\vx^2)) + (1 - \lambda) \sum_{z=1}^N |x^1_z - x^2_z| \\
                    \text{s.t.}
                        & \;\; f(\vx^1) \ne f(\vx^2) \\
                    = \max_{\vx^1, \vx^2, \vepsilon}
                        & \;\; \lambda (f(\vx^1) + f(\vx^2)) + (1 - \lambda) \sum_{z=1}^N \varepsilon_z \\
                    \text{s.t.}
                        & \;\; \varepsilon_z \le |x^1_z - x^2_z|                            & \forall z = 1, \ldots, N \\
                        & \;\; f(\vx^1) \ne f(\vx^2) \\
                    = \max_{\vx^1, \vx^2, \vepsilon}
                        & \;\; \lambda (f(\vx^1) + f(\vx^2)) + (1 - \lambda) \sum_{z=1}^N \varepsilon_z \\
                    \text{s.t.}
                        & \;\; \varepsilon_z \le \max \{ x^1_z - x^2_z, x^2_z - x^1_z \}    & \forall z = 1, \ldots, N \\
                        & \;\; f(\vx^1) \ne f(\vx^2) \\
                    = \max_{\vx^1, \vx^2, \vepsilon, \vb^1, \vb^2}
                        & \;\; \lambda (f(\vx^1) + f(\vx^2)) + (1 - \lambda) \sum_{z=1}^N \varepsilon_z \\
                    \text{s.t.}
                        & \;\; \varepsilon_z \le (x^1_z - x^2_z) + b^1_z \cdot M            & \forall z = 1, \ldots, N \\
                        & \;\; \varepsilon_z \le (x^2_z - x^1_z) + b^2_z \cdot M            & \forall z = 1, \ldots, N \\
                        & \;\; b^1_z + b^2_z = 1                                            & \forall z = 1, \ldots, N \\
                        & \;\; f(\vx^1) \ne f(\vx^2)
                \end{align*}
                %
                Here I introduced the variables $\vepsilon$ and $\vb$ to
                linearize the problem; $M$ is supposed to be a large positive
                constant, e.g. $M = 10^6$. This is called the ``big-$M$''
                trick.

                \stefano{@Andrea: maximizing the $\ell_1$ norm over a
                categorical representation is equivalent to requiring that the
                configurations $\vx^1, \ldots, \vx^S$ have a maximal number of
                different attributes; e.g. if $\vx^1$ and $\vx^2$ differ in
                all attributes, regardless of the actual values, they achieve
                maximal $\ell_1$ distance.}

                This can be done \emph{incrementally}: select $\vx^1$ with
                maximal aggregate utility $f$; then, for all $i = 2, \ldots,
                S$, select $\vx^i$ with maximal aggregate utility $f$ and
                diversity from $\vx^1, \ldots, \vx^{i-1}$. This should be
                much faster.

                \stefano{@Andrea, @Bereket: improvements are welcome.}

            \item Use the Perceptron update above to obtain $\vomega^{t+1}$.

            \item Repeat until convergence or a maximum number of iterations
                $T$ elapsed.

        \end{itemize}

\end{itemize}

\paragraph{Competitors.}

WRITEME

\paragraph{Experiments.}

\begin{itemize}

    \item The group is simulated using a Plackett-Luce user response model,
        see the paper.

        \stefano{This may be sub-optimal for groupwise-feedback; suggestions
        are welcome.}

\end{itemize}

\paragraph{Related work.}

\begin{itemize}

    \item \stefano{@Bereket: please add here the  relevant literature that
        you found, I'll do the same as time permits.}
    \item \bereket{\cite{boutilier2011preference}}
\end{itemize}

\bibliographystyle{unsrt}
\bibliography{social-choice}
\end{document}


\section{Introduction}
\label{section:intro}

Age is the most difficult stellar property to measure, and the difficulty of
age-dating is particularly acute for low mass (GKM) stars on the main sequence
(MS).
Using conventional dating methods, uncertainties on the ages of these stars
can be as large as the age of the Universe.
GKM dwarfs are difficult to age-date because most of their physical and
observable properties do not change rapidly.
This is represented in the spacing of isochrones on a Hertzsprung-Russell
diagram (HRD) or color-magnitude diagram (CMD).
On the MS, isochrones are tightly spaced and, even with very precise
measurements of effective temperature and luminosity, the position of a MS
star on the HRD may be consistent with range of isochrones spanning several
billion years.
At the main-sequence turnoff however, isochrones are spread further apart, so
that sufficiently precisely measured temperatures and luminosities can yield
ages that are extremely precise (with minimum statistical uncertainties on the
order of 5-10\%) \citep[\eg][]{pont2004}.
% Typical age uncertainties of dwarfs.
The classical method for measuring stellar ages is isochrone placement, or
isochrone fitting, where surface gravity changes resulting from fusion in the
core (usually observed via luminosity, $L$, and effective temperature, \teff,
or absolute magnitude and colour) are compared with a set of models that trace
stellar evolution across the HRD, or CMD.
CMD/HRD position has been thoroughly mapped with physical models, and can be
used to calculate relatively accurate (but not necessarily precise) ages,
barring some systematic variations between different models,
\citep[\eg][]{yi2001, dotter2008, dotter2016}.
On the MS itself, there is little differentiation between stars of different
ages in the $L$ and \teff\ plane, so ages tend to be very imprecise.
The method of inferring a star's age from its rotation period, called
`gyrochronology', is much better suited for measuring ages on the MS because
MS stars spin down relatively rapidly.

Magnetic braking in MS stars was first observed by \citet{skumanich1972} who,
studying young clusters and the Sun, found that the rotation periods of
Solar-type stars decay with the square-root of time.
It has since been established that the rotation period of a MS star depends,
to first order, only on its age and effective temperature or color
\citep[\eg][]{barnes2003}.
The convenient characteristic of stars that allows their ages to be inferred
from their {\it current} rotation periods and independently of their
primordial ones, comes from the steep dependence of spin-down rate on rotation
period \citep{kawaler1989}.
Stars spinning with high angular velocity will experience a much greater
angular momentum loss rate than slowly spinning stars and for this reason, no
matter the initial rotation period, Solar type stars will have the same
rotation period after around the age of the Hyades, 500-700 million years
\citep{irwin2009, gallet2015}.
After this time, the age of a star can be inferred, to first order, from its
dust-corrected color (\eg\ B-V or \gcolor) and {\it current} rotation period
alone \citep[See][for an analysis of how initial conditions effect gyrochronal
ages]{epstein2014}.

The relation between age, rotation period and mass has been studied in detail,
and several different models have been developed to capture the rotational
evolution of Sun-like stars.
Some of these models are theoretical and based on physical processes; modeling
angular momentum loss as a function of stellar properties as well as the
properties of the magnetic field and stellar wind \citep[\eg][]{kawaler1988,
kawaler1989, pinsonneault1989, vansaders2013, matt2015, vansaders2016}.
Other models are empirical and capture the behavior of stars from a purely
observational standpoint, using simple functional forms that can reproduce the
data \citep[\eg][]{barnes2003, barnes2007, mamajek2008, angus2015}.
Both types of model, theoretical and empirical, must be calibrated using
observations.
Old calibrators are especially important because new evidence suggests that
rotational evolution goes through a transition at old age or, more
specifically, at a large Rossby number, $Ro$ (the ratio of rotation period to
the convective overturn timescale).
For example, stars shown to be old from \kepler\ asteroseismic data rotate
more rapidly than expected given their age \citep{angus2015, vansaders2016}.
A new physically motivated gyrochronology model, capable of reproducing these
data, was recently introduced \citep{vansaders2016}.
It relaxes magnetic braking at a critical Rossby number of around 2,
approximately the Solar value.
This model predicts that, after stellar rotation periods lengthen enough to
move stars across this $Ro$ threshold, stars conserve angular momentum and
maintain a nearly constant rotation period from then until they evolve off the
MS.
% As demonstrated in section \ref{section:results} however, this does not mean
% that the ages of stars with Ro $>$ 2 cannot be measured from their rotation
% periods, especially if their rotation periods are combined with HRD or CMD
% placement.

The gyrochronology models that capture post $Ro$-threshold, rotational
evolution \citep{vansaders2016} are the current state-of-the-art in rotation
dating.
These models can be computed over a grid of stellar parameters, and
interpolated-over to predict the age of a star.
The process of measuring the age of a field star with these models is similar
to inferring an age using any set of isochrones, with the difference being
that rotation period is an additional observable dimension.
Ages calculated using these models are therefore likely to be much more
precise than using rotation-free isochrones since rotation period provides an
additional anchor-point for the age of a star.
We present here a complementary method that combines isochrones with an {\it
empirical} gyrochronology model.
The methodology is related to the \citet{vansaders2016} model in that both use
a combination of rotation periods and other observable properties that track
stellar evolution on the HRD in concert.
The main difference is that the gyrochronology model used here is an entirely
empirically calibrated one, as opposed to a physically derived one.

One major advantage of using a physically motivated gyrochronology model is
the ability to rely on physics to interpolate or extrapolate over parts of
parameter space with sparse data coverage.
However, rotational spin-down is a complex process that is not yet fully
understood and currently no physical model can accurately reproduce all the
data available.
For this reason, even physically motivated gyrochronology models cannot always
be used to reliably extrapolate into unexplored parameter space.
Physical models, when calibrated to data, can provide insight into the physics
of stars, however, if accurate and precise {\it prediction} of stellar
properties is desired, empirical models can have advantages over physical
ones.
For example, the data may reveal complex trends that cannot be reproduced with
our current understanding of the physical processes involved, but may be
captured by more flexible data-driven models.
In addition, it is relatively straightforward to build an element of
stochasticity into empirical models, \ie\ to allow for and incorporate
outliers or noisy trends.
This is particularly important for stellar spin down because rotation periods
can be affected by additional confounding variables which are not always
observed (having a binary or planetary companion, for example).
A further advantage of empirical models is that inference is more tractable:
it can be extremely fast to fit them to data.

In this work we calibrated a new empirical gyrochronology relation, fit to the
Praesepe open cluster and the Sun, in \gaia\ \gcolor\ color.
\Gaia\ $G$, $G_{BP}$ and $G_{RP}$ apparent magnitudes are now the most
abundant photometric measurements, available for more than a billion stars
\citep{brown2018}.
In fact, an important point of context for this work is the new
availability of data relevant to gyrochronology and stellar ages.
\Gaia\ now provides broad-band photometry and parallaxes for over a billion
stars, and \kepler, \ktwo\ and \tess\ are providing rotation periods for
hundreds of thousands of stars.
Gyrochronology is becoming one of the most readily available age-dating
methods, so continuing to improve gyrochronology relations and methods is
important.
Like any other gyrochronology relation, this new Praesepe-based model does not
perfectly reproduce all the observed data and some simple modifications could
make significant improvements, for example, by including a mixture model to
account for outliers and binaries, and by removing the period-age,
period-color separability to account for different period-color shapes seen in
clusters of different ages.
We leave these improvements for a future project and, for now, test this
new Praesepe-based model, which is built into an open source {\it Python}
package for gyrochronology called \sd.
\sd\ provides the framework for simultaneous gyrochronology and isochrone
fitting, and, because \sd\ is modular, it would be straightforward to update
the gyrochronology relation in the future.

This paper is laid out as follows.
In section \ref{section:method} we describe our new age-dating model and its
implementation, in section \ref{section:results} we test this model on
simulated stars and cluster stars, and in section \ref{section:conclusion} we
discuss the implications of these tests and future pathways for development.
Throughout this paper we use the term {\it `observables'} to refer to the
following observed properties of a star: \teff, \logg, observed bulk
metallicity ($[\mathrm{Fe/H}]$), parallax ($\bar{\pi}$), photometric
colors in different passbands (${\bf m_x} = [m_J, m_H, m_K, m_B, m_V, m_G,
m_{GBP}, m_{GRP}...]$, etc) and rotation period ($P_{\mathrm{rot}}$).
The term `parameters' refers to the physical properties of that star: age
($t$), equivalent evolutionary phase (EEP), model metallicity
($\mathrm{M/H}$), distance ($D$) and V-band extinction ($A_V$).
These are the properties that generate the observables.


\documentclass[12pt]{article}

% text
\newcommand{\documentname}{\textsl{Article}}
\newcommand{\sectionname}{Section}

% math
\newcommand{\given}{\,|\,}
\newcommand{\trans}[1]{{#1}^{\!\mathsf{T}}}

\begin{document}\sloppy\sloppypar\raggedbottom\frenchspacing

\section{Introduction}

Types of supernovae are few in number.

Types of nucleosynthetic process are few in number.

Open clusters show great homogeneity, so intra-cloud mixing must be good.

Therefore, stars cannot span a huge space in chemical abundances.

Indeed, this conclusion has been arrived at previously in theory, and
the dimensionality has been measured before in data. The measurements to
date suffer from significant issues: precision, methodology.

Here we test this with the best possible data set at the present day:
All the reasons these twins are good.

\section{Generalities}

This \documentname\ wants to ask the question:
\emph{What is the observed dimensionality of chemical-abundance
  space?}
Unfortunately---and like all important questions---this question is
ill posed.
Our problem in this \sectionname\ is to write down well-posed questions
that go some way towards answering the ill-posed question.

\section{Data}

Bedell!

Show (some of) the 900 plots here; they justify the precision claims.
I would say all 32 against feh and then all 32 against something that
clearly demonstrates the precision.

\section{Experiments and results}

PCA first; it is highly suggestive. Take then the opportunity to cricize PCA.

Something like HMF next. State the model and assumptions. Make it such that
there is an excess variance parameter that is being estimated. Etc.
Something like:
\begin{eqnarray}
y_n &=& \mu + W\cdot x_n + \delta_n + \epsilon_n
\\
p(\delta_n\given q^2) &=& N(\delta\given 0, q^2\,I)
\\
p(\epsilon_n) &=& N(\epsilon\given 0, C_n)
\\
p(x_n) &=& N(x_n\given 0, I)
\\
p(y_n\given \mu,W,q^2) &=& N(y_n\given \mu, W\cdot\trans{W}+q^2\,I+C_n)
\quad ,
\end{eqnarray}
where the data point $y_n$ is a $D$-vector,
$W$ is a $D\times K$ rectangular matrix,
$x_n$ is a $K$-vector,
$\delta_n$ is a deviation due to intrinsic variance,
$\epsilon_n$ is a deviation due to measurement noise,
$q^2$ is an intrinsic variance,
$I$ is the $D\times D$ identity tensor,
$C_n$ is the noise variance tensor for data point $y_n$,
and we have marginalized out all but $\mu,W,q^2$ at the end.

Is there any sense in which the eigen-directions look like SNe yields?
Or nucleosynthetic yields from processes?

\section{Discussion}

\appendix
\section{Linear vs log space}

Almost all work on element abundances works in log (base 10), and most is
relative to Solar.
If we assume that there are a finite number $K$ of nucleosynthetic pathways,
and that each of those pathways produces elements in nearly fixed
ratios of atoms or mass, then the abundances will lie in a $K$-dimensional
subspace (or lower).
This subspace will only appear as a strictly linear subspace (that is, strictly
spanned by $K$ vectors) in linear ratios of elements.
These linear ratios can be mass ratios or atomic number ratios, and they can be ratioed
to other elements or Solar abundances, but without taking logarithms.

If the dimensionality of the element abundance linear subspace is $K$, and the
subspace is searched for with a linear algorithm but in the logarithmic element
abundances, then in principle more than $K$ significant dimensions or vectors
will be required to span the space of the data.
This is because a linear subspace in the linear data will obtain some curvature
when the logs are taken, and that curvature will be in directions that are orthogonal
to the subspace (in general).

For these reasons, in this paper, when we perform the dimensionality reduction,
we take the (log) abundances back to linear, perform the dimensionality reduction
in the linear space, and then transform the output so that it can be interpreted
logarithmically.
This latter transformation---of eigenvectors in the linear space to equivalents
in the log space---works as follows:

[Give equation and derivaton here.]

The amusing thing about all this, in the context of the project reported in this paper,
is that the spread in chemical abundances is very small in the sample.
In the limit of infinitesimal element-abundance spreads, there will be no differences
between the properly transformed linear eigenvectors and logarithmic space eigenvectors.
We demonstrate this in Figure~XXX.

\end{document}


\chapter{Introduction}
\label{chapter:intro}

\section{Motivation}
\label{sec:motivation}
Human beings have always felt the need to explore and find a way to a better life. Starting with stone tools (2.6 million years ago) and continuing with personal computers, one can clearly see widespread proofs of human evolution. In the latter years, humanity has discovered Machine Learning which has the purpose to allow machines learn to do different tasks.

Deep learning is a subfield of the machine learning area, drawing its inspiration from the human brain's functionality. Through deep learning, we are trying to solve some of the most pressuring human problems such as cancer classification (benign or malignant tumor)\cite{mitosis}, self-driving cars based on pedestrian detection\cite{pedestrian} or recognizing digits form photos taken with Google Street View\cite{svhn}, all using unsupervised learning of features.

This paper's proposes an in-depth look on solving reinforcement learning problems, using convolutional neural networks. More exactly, we want to design an agent which is capable of learning to play a game\cite{atari} seeing only the pixels from the frames and being rewarded after finishing the game.

This idea was brought to attention in 2013, when Alex Graves et al. from DeepMind\footnote{\url{http://deepmind.com/}} came with the idea of creating a convolutional neural network capable of playing different Atari 2600 games, being almost as good as a game tester. We will see later what almost means. The motivation behind this experiment is not only connected with the fact that we have a machine capable of playing games, but the more important problem is that they achieved the generalization of a machine that could learn several types of games, which in fact is the starting of a new revolution in machine learning. If we are capable of making one unique algorithm that can solve multiple tasks, we are capable of simulating the real human brain and capable of reducing complex problems to others, much simpler thus we can reduce the programming burden and concentrate on solving the more urgent problems such as cancer classification.

Moreover, there are also other types of applications that can reveal the power of convolutional neural networks and one of them is the neural algorithm for creating paintings\cite{paintings} from ordinary pictures. The inputs are represented by two pictures, one which is the painting (\textbf{e.g} \textit{The Starry Night} of Vincent van Gogh) and the other one which is a normal photo, like a picture of a house. The output of the network is the house painted in van Gogh's technique.

\section{Project description}
\label{sec:proj-description}
This paper presents two different algorithms aiding the creation of a good agent which can learn to play games. It is worth mentioning that the agent does not know the rules and can not infer most of the rules at the beginning of the game. The whole topic has been split into three parts: running Q-Learning on Hanoi Towers game, learning the values predicted by Q-Learning using a convolutional neural network and binding the first two parts together.

First of them is the classical Q-Learning, which searches for the optimal policy for taking actions in each state of the game. We will discuss about the exploration vs exploitation problem, how to determine the learning rate suitable for our purposes, how many episodes we have to play until we determine a policy close to the optimal one.

The second topic we approach in this paper is the possibility of predicting a continuous output from an image. Practically, we take the dataset from Q-Learning which contains images as input and a table of four values corresponding to the four possible actions in the Hanoi Tower games: up, down, left and right.
Here, we can determine whether we have a good or a bad convolutional neural network. In other words, we will test the capacity of learning samples, the capacity for generalization.

One must pay attention to the data pre-processing, this being an important step in data selection. For example, when we learn to classify things, we may not need the color information. Converting data from RGB to YUV gives us the possibility of separating luminance from chrominance.

After that we need to look closer at the model. We need a model that can be proven to converge at the optimal solution and also a model that can learn very fast. Datasets are very big and can contain millions of pictures.

It is also important that one knows when to stop training to avoid overfitting, our primary target being that of minimizing the test error. We have to take care of splitting the dataset in training, testing or validation, how to determine when to stop training the network and how to formalize the results afterwards.

Another important step is called the loss function, the moment when we determine how far we are from our target. Here, we can find the border between underfitting and overfitting, called the optimum state. If we have high bias, it means that we have underfitting and our classifier is not able to predict data well and if we have high variance, it means that we have solid training samples for learning, but new samples cannot be predicted.

The last part is the one where we combine the first two modules. We will try to get a closer look on what those from DeepMind have done with the Atari paper\cite{atari}. The most important part is the connection between the Q-Learning algorithm and the convolutional neural network. We will see how weights are updated according to rewards received during the game.

The main problem is that of creating an architecture capable of playing any game without adapting initial meta parameters. All the above-mentioned problems will be discussed in this aper and formalized using experimental results, plots or anything that can serve as a proof.
\newpage

\section{Technologies}
\label{sec:technologies}
All algorithms described in this paper have been implemented using Torch7\footnote{\url{http://torch.ch/}}, a deep learning framework written in Lua\footnote{\url{http://www.lua.org/about.html}}, a scripting language based on a C API. For creating the game, generating frames or modifying images the LOVE platform\footnote{\url{https://love2d.org/wiki/Main_Page}} has been used, this also being written in Lua.  For interactive computing(image/filter visualization) iTorch\footnote{\url{https://github.com/facebook/iTorch}} is preferred. For illustrating different functions Octave\footnote{\url{https://www.gnu.org/software/octave/}} is the best open-source choice.

Why Lua and Torch? They provide a fast environment as opposed to others\cite{torch7}, multiple modules with functions already implemented such as transfer functions(tanh, sigmoid), loss functions(Mean Squared Error, Negative Log Likelihood) or convolutional layers(SpatialMaxPooling, SpatialSubSampling)

\section{Structure of this paper}
\label{sec:paper_structure}

\labelindexref{Chapter}{chapter:state} \textbf{State of the art} presents the entire architecture of a network based on a complex research in the machine learning, pattern recognition and reinforcement learning fields.

\labelindexref{Chapter}{chapter:related} \textbf{Related work} presents an in-depth look into the ``Human-level control through deep reinforcement learning'' paper is provided. This paper has been chosen for the fact that it is the primary resource which proves that the combination of deep learning with reinforcement learning is possible.

\labelindexref{Chapter}{chapter:system-design} \textbf{System design and implementation} describes several architectures using convolutional and pooling layers which may lead to learning. Every step in choosing the architecture is well documented, either with mathematical reasoning or with empirical reasoning.

\labelindexref{Chapter}{chapter:results} \textbf{Results} presents error plots provided by training and testing convolutional neural networks.

\labelindexref{Chapter}{chapter:conclusions} \textbf{Conclusions} is dedicated to the personal views of the paper's author regarding this topic and also presents the systematic investigation in gathering information.

\labelindexref{Chapter}{chapter:future-work} \textbf{Future work} describes future tunings of the architectures and also further investigation on one of the following fields: machine learning, pattern recognition or reinforcement learning.


\section{Splines}


\documentclass[12pt]{article}
\usepackage{fullpage}
\usepackage{url}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{color}
\usepackage{listings}
\usepackage{verbatim}
\usepackage{enumitem}
\usepackage[parfill]{parskip}

\newcommand{\xb}{\mathbf{x}}
\newcommand{\yb}{\mathbf{y}}
\newcommand{\wb}{\mathbf{w}}
\newcommand{\Xb}{\mathbf{X}}
\newcommand{\Yb}{\mathbf{Y}}
\newcommand{\tr}{^T}
\newcommand{\hb}{\mathbf{h}}
\newcommand{\Hb}{\mathbf{H}}

\newcommand{\cmt}[1]{{\footnotesize\textcolor{red}{#1}}}
\newcommand{\todo}[1]{\cmt{TO-DO: #1}}

\title{CS294-112 Deep Reinforcement Learning HW3}

\author{ Heri Zhao
}

\date{Fall 2018}

\usepackage{courier}

\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.95,0.95,0.92}

\lstdefinestyle{mystyle}{
    backgroundcolor=\color{backcolour},
    commentstyle=\color{codegreen},
    keywordstyle=\color{magenta},
    numberstyle=\tiny\color{codegray},
    stringstyle=\color{codepurple},
    basicstyle=\footnotesize\ttfamily,
    breakatwhitespace=false,
    breaklines=true,
    captionpos=b,
    keepspaces=true,
    %numbers=left,
    numbersep=5pt,
    showspaces=false,
    showstringspaces=false,
    showtabs=false,
    tabsize=2
}

\lstset{style=mystyle}

\begin{document}

\maketitle

\section*{Part1 Q-learning}

\subsection*{Question 1: basic Q-learning performance}
\begin{figure}[H]
  \centering
  \includegraphics[height=3.4in]{part1_q1.png}
  \caption{Learning curves for basic Q-learning on game Pong}
\end{figure}
\begin{lstlisting}[language=bash]
python run_dqn_atari.py
\end{lstlisting}

\subsection*{Question 2: double Q-learning}
\begin{figure}[H]
  \centering
  \includegraphics[height=3.4in]{part1_q2.png}
  \caption{Learning curves for vanilla vs double Q-learning on game Pong}
\end{figure}
\begin{lstlisting}[language=bash]
# manually change line 87 in run_dqn_atari.py to pass double_q=True
python run_dqn_atari.py
\end{lstlisting}
We can see that both vanilla and double Q-learning reach to rewards 15+ after 4M timesteps. The double Q-learning is slightly better than vanilla at 4M timesteps. This matches the Figure 4 of \href{https://arxiv.org/pdf/1509.06461.pdf}{this paper}. Also noticed that the learning curve is little slow for double Q-learning, however this probably because of the random seed.

\subsection*{Question 3: experimenting with hyperparameters}
\begin{figure}[H]
  \centering
  \includegraphics[height=3.4in]{part1_q3.png}
  \caption{Learning curves of \textit{learning rate multiplier=0.1, 0.5, 1, 10} on game Pong with double Q-learning}
\end{figure}
\begin{lstlisting}[language=bash]
# manually change line 87 in run_dqn_atari.py to pass double_q=True
python3 run_dqn_atari.py 0.1
python3 run_dqn_atari.py 0.5
python3 run_dqn_atari.py 10
\end{lstlisting}
Here we choose the \textit{learning rate multiplier} to experiment with the hyper-parameters.
We can see that with learning rate very large the Q-learning basically can not learn anything, the rewards of lr=10 did not increase after 2M timesteps. The curve of lr=0.1 is actually increasing, but slowly, it probably will get the high rewards, but will take more time. The performance of lr=0.5 is very much similar to lr=1.

\newpage
\section*{Part2 Actor-Critic}

\subsection*{Question 1: Sanity check with Cartpole}
\begin{figure}[H]
  \centering
  \includegraphics[height=3.4in]{part2_q1.png}
  \caption{Learning curves of different number of target updates and number of gradient combinations on CartPole with Actor-Critic. We can see that (ntu=10, ngsptu=10) is best parameter combination, as it converges to the optimal rewards fast and more stable than (1, 100).}
\end{figure}
\textbf{Explanation}: Since we are using neural network to fit value function, (ntu=1, ngsptu=100) means we only update target once, while taking many gradient descent steps. This will probably bring network too close to the current target, instead of fitting to get a more generalized value function. The bias will be larger compared with (ntu=10, ngsptu=10). For (ntu=100, ngsptu=1), we only take one gradient step, so it is likely that we will not have chance to fit value function, and finally get larger bias.

\newpage
\subsection*{Question 2: Run actor-critic with more difficult tasks}
We can see that the performance is similar to what we got in HW2, but slightly worse.
\begin{figure}[H]
  \centering
  \includegraphics[height=3.4in]{part2_q2_2.png}
  \caption{Learning curves of InvertedPendulum with ntu=10 and ngsptu=10}
\end{figure}
\begin{lstlisting}[language=bash]
python train_ac_f18.py InvertedPendulum-v2 -ep 1000 --discount 0.95 -n 100 -e 3 -l 2 -s 64 -b 5000 -lr 0.01 -clr 0.01 --exp_name 10_10 -ntu 10 -ngsptu 10
\end{lstlisting}
\begin{figure}[H]
  \centering
  \includegraphics[height=3.4in]{part2_q2_1.png}
  \caption{Learning curves of HalfCheetah with ntu=10 and ngsptu=10}
\end{figure}
\begin{lstlisting}[language=bash]
python train_ac_f18.py HalfCheetah-v2 -ep 150 --discount 0.90 -n 100 -e 3 -l 2 -s 32 -b 30000 -lr 0.02 -clr 0.02 --exp_name 10_10 -ntu 10 -ngsptu 10
\end{lstlisting}

\newpage
\subsection*{Bonus}
Here I am trying to tune actor-critic such that it could perform better.
\begin{enumerate}
\item Add 2 more hidden layers for critic network.
\item Use separate smaller learning rate to train critic network (0.01, 0.001).
\item Increase the number of target updates to 20.
\end{enumerate}
From the Figure 7, we can see that after tunning, the performance is getting better, and is very similar to what we got in HW2. The best one has critic\_learning\_rate=0.01, ntu=20, ngsptu=10, and this also has lower variance.
\begin{figure}[H]
  \centering
  \includegraphics[height=3.4in]{bonus.png}
  \caption{Learning curves of HalfCheetah. \textit{clr} is representing critic-learning-rate.}
\end{figure}
\begin{lstlisting}[language=bash]
# please change line 263 in train_ac_f18.py to use "n_layers=self.n_layers + 2" before running
python train_ac_f18.py HalfCheetah-v2 -ep 150 --discount 0.90 -n 100 -e 3 -l 2 -s 32 -b 30000 -lr 0.02 -clr 0.01 --exp_name clr0.01_ntu10_ngsptu10 -ntu 10 -ngsptu 10
python train_ac_f18.py HalfCheetah-v2 -ep 150 --discount 0.90 -n 100 -e 3 -l 2 -s 32 -b 30000 -lr 0.02 -clr 0.001 --exp_name clr0.001_ntu10_ngsptu10 -ntu 10 -ngsptu 10
python train_ac_f18.py HalfCheetah-v2 -ep 150 --discount 0.90 -n 100 -e 3 -l 2 -s 32 -b 30000 -lr 0.02 -clr 0.01 --exp_name clr0.01_ntu20_ngsptu10 -ntu 20 -ngsptu 10
\end{lstlisting}

\end{document}


\chapter{Conclusion and Future Work}
\label{chap:conclusion}

%In this chapter, we conclude the whole thesis and propose several possible directions for future pursuit.

%-------------------------------------------------------------------------------
\section{Conclusion}
%\label{Conclusion:con}

As a new research topic in the machine learning area, hash function learning has attracted plenty of interest in recent years. Different from locality sensitive hashing methods which design data-independent hash functions for specific similarity measures, hash function learning methods learn data-dependent hash functions directly from the data.

In the present thesis, we explore two research issues in hash function learning, namely, active hashing and multimodal hashing. Active hashing aims to actively select the data from which to learn hash functions, whereas multimodal hashing focuses on learning the hash functions for data involving multiple modalities. The contributions of this thesis are summarized as follows:
\begin{description}
\item[Active Hashing] To eliminate the data selection bias and reduce the labeling cost of existing supervised or semi-supervised hash function learning methods, we propose the active hashing framework. Under this framework, the hash functions are learned from the data which are carefully selected before learning. To evaluate the data informativeness, a simple yet effective criterion which is based on uncertainty is proposed. We also present a batch mode algorithm for the data selection procedure. Through extensive experiments, we demonstrate that active hashing outperforms passive hashing by a large margin.

\item[Spectral Multimodal Hashing] To learn hash functions for multimodal data, we propose the spectral multimodal hashing method (\mbox{SMH}). Given that the data in different modalities are aligned, that is, every object has a representation in each modality, \mbox{SMH} learns the hash functions through spectral analysis of modality correlation. Due to its resemblance to canonical correlation analysis in statistics, SMH is easy to extend to support kernel similarity and more than two modalities. We also propose a novel method to learn the thresholds for the hash functions. Experimental results show that our \mbox{SMH} model outperforms the state-of-the-art methods. %However, \mbox{SMH} has an apparent limitation, that is, it is only for the aligned data which may not be available in some applications. In the next chapter, we propose a new multimodal hashing model for graph data which is more general than aligned data.

\item[Multimodal Latent Binary Embedding] For the multimodal data which are organized in similarity graphs, we propose a novel probabilistic model called multimodal latent binary embedding (MLBE) to learn the hash functions. As a latent factor model, MLBE regards the binary latent factors as hash codes and hence maps data points from multiple modalities to a common Hamming space in a principled probabilistic manner. Although finding exact posterior distributions of the latent factors (hash codes) is intractable, we devise an efficient alternating learning algorithm based on \mbox{MAP} estimation. Experimental results show that MLBE compares favorably with state-of-the-art multimodal hashing models. %For our future research, we will go beyond the point estimation approach presented in this paper to explore a more Bayesian treatment based on variational inference for enhanced robustness and efficiency. We would also like to extend \mbox{MLBE} to determine the code length $K$ automatically from data.  This is an important yet largely unaddressed issue in existing methods. Besides, we also plan to apply \mbox{MLBE} to other tasks such as multimodal medical image registration.

\item[Co-regularized Hashing] For the general multimodal data, we propose a new method based on two popular supervised learning techniques: boosting and co-regularization, and name it co-regularized hashing (CRH). We no longer assume that the data are aligned or organized in graphs in CRH. To learn each bit of the hash codes, the objective function is in the form of a difference of convex functions, and we develop an efficient learning algorithm based on the convex-concave procedure (\mbox{CCCP}) and a stochastic gradient method. Hash functions for different bits are learned using a standard boosting method. Comparative studies based on benchmark data sets show that \mbox{CRH} outperforms two state-of-the-art multimodal hashing methods.
\end{description}
%\begin{itemize}
%%\item[Active Hashing] We first present the active hashing framework, which provides a general approach to select labeled data from which existing semi-supervised hash function learning methods can learn.
%
%\item We first study a simple case in which multimodal data are aligned, and propose one method based on spectral analysis.
%
%\item Then for graph data, which is more general than aligned data, we give a probabilistic model based on latent feature models.
%
%\item Finally, we move to the general case and present co-regularized hashing, which takes advantage of boosting and optimization in learning multimodal hash functions. Complete comparative experiments validate the effectiveness of our models.
%\end{itemize}

%Multimodal hashing is more general and challenging than most existing hashing methods which are for uni-modal data.

%-------------------------------------------------------------------------------
\section{Future Work}

%Though we have tried to solved some basic problems of active hashing and multimodal hashing, there are still several open questions needing to be answered.

To take our work further, we wish to study the following topics in the near future:

\begin{description}
\item[Challenging issues in Multimodal Hashing] In multimodal hash function learning, there are several challenging issues worth further studying. Among other things, it is the most important to learn an appropriate length of hash codes automatically from data. It is well known that the hash codes generated by HFL methods are always much shorter than those generated by LSH methods, however, there is no benchmark result on learning the code length. %Moreover, we would like to study some other applications of multimodal hash function learning, including medical image alignment and image or video de-duplication.

\item[Hierarchical Hash Function Learning] Most existing \mbox{HFL} methods assume that different hash bits are equally important, in other words, there is no structure in the hash codes. This assumption can be easily violated in applications where similarity search should be performed in different resolutions or levels. Although several researchers have made some preliminary attempts on generating multiple bits based on one hash function~\cite{liu2011icml}, to the best of our knowledge, no result has been reported on hash functions with structures.

We propose to develop hierarchical \mbox{HFL} methods. In addition to supporting different levels of search, the structural information of hash codes can be used to reduce the search space and hence further improve the scalability. For example, one can use coarse-level hash codes to find a candidate space and then fine-level hash codes for nearest neighbors. Moreover, hierarchical \mbox{HFL} can be used to combine different hashing models. For example, given that some hash functions are sensitive to recall while others are sensitive to precision, we can organize different kinds of hash codes in such a hierarchy that the new model can take advantages of different models.

\item[Sublinear Hash Function Learning]
Many \mbox{HFL} algorithms rely on expensive matrix operations such as eigen-decompostion and inversion. Obviously, they are not suitable for very high-dimensional data, such as text and image, unless the data are properly preprocessed. Although recent progress shows that linear time complexity can be expected~\cite{lin2010cvpr}, complicated approximation techniques are required in these methods. Furthermore, even linear time algorithms are sometimes not feasible in real large-scale applications.

Recently, a family of sublinear algorithms has emerged in the machine learning area~\cite{hazan2011nips}. These algorithms have been applied to learning support vector machines (\mbox{SVM}) for classification problems. We believe that it is also possible to develop sublinear \mbox{HFL} algorithm, because some \mbox{HFL} algorithms such as CRH are similar to \mbox{SVM} training algorithms in some sense.

\item[Collaborative Filtering and Hash Function Learning]
Almost all \mbox{HFL} methods focus on similarity search applications. Nevertheless, other large-scale data mining tasks can also benefit from \mbox{HFL}. One outstanding example is collaborative filtering (\mbox{CF}), which is a very important technique for data mining and, more specifically, the recommender systems. So far as we know, several non-learning based hashing methods have been used to improve collaborative filtering~\cite{shi2009aistats,weinberger2009icml}, and we believe that \mbox{HFL} can make further improvement.

One possible approach may be to formulate \mbox{CF} as a multimodal hashing problem. To recommend movies to users, we just find nearest neighbor movies of the users in a common Hamming space. The learning problem needs to take into consideration the collaborative rating matrix as well as other side information, and the whole learning procedure should be conducted in sublinear time.

%\item[Applications of Hash Function Learning]
%To further explore the topic of hash function learning, in the near future, we would like to explore the idea of learning hash functions for problems other than similarity search, such as classification, regression, ranking and collaborative filtering.

\end{description}

%\subsection{Active Hashing}
%For active hashing, we plan to explore other criteria of data informativeness. For example, we may select the points which maximize information gain or minimize the expected error on test data. Moreover, although we have demonstrated that it is more effective to select and label points rather than pairwise constraints, it would be interesting to explore the informativeness of pairwise constraints directly for some supervised \mbox{HFL} methods. Last but not least, another possible research direction is to explore other batch mode algorithms for active hashing.

%\subsubsection{Unified Framework for Hash Function Learning }
%
%%\paragraph{What makes good codes?}
%%Yes, it depends on applications. But we should be clear when and why and how.
%There are different categorizations of \mbox{HFL} methods, such as supervised/unsupervised algorithms. However, little research effort has been put on summarize the assumptions in the unifying framework.
%
%We would like to propose a general framework, such as most current methods can be put into proper positions.