\documentclass{article}
\usepackage{hyperref}

\begin{document}

\title{Etherpad and EasySync Technical Manual}
\author{AppJet, Inc., with modifications by the Etherpad Foundation}
\date{\today}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\tableofcontents
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Documents}
\begin{itemize}
\item A document is a list of characters, or a string.
\item A document can also be represented as a list of \emph{changesets}.
\end{itemize}

\section{Changesets}

\begin{itemize}
\item A changeset represents a change to a document.
\item A changeset can be applied to a document to produce a new document.
\item When a document is represented as a list of changesets, it is assumed that the first changeset applies to the empty document, [].
\end{itemize}

\section{Changeset representation} \label{representation}

$$(\ell \rightarrow \ell')[c_1,c_2,c_3,...]$$

where

\begin{itemize}
\item[] $\ell$ is the length of the document before the change,
\item[] $\ell'$ is the length of the document after the change,
\item[] $[c_1,c_2,c_3,...]$ is an array of $\ell'$ characters that described the document after the change.
\end{itemize}

Note that $\forall c_i : 0 \leq i \leq \ell'$ is either an integer or a character.

\begin{itemize}
\item Integers represent retained characters in the original document.
\item Characters represent insertions.
\end{itemize}

\section{Constraints on Changesets}

\begin{itemize}
\item Changesets are canonical and therefor comparable.  When represented in computer memory, we always use the same representation for the same changeset.  If the memory representation of two changesets differ, they must be different changesets.
\item Changesets are compact.  Thus, if there are two ways to represent a changeset in computer memory, then we always use the representation that takes up the fewest bytes.
\end{itemize}

Later we will discuss optimizations to changeset
representation (using ``strips'' and other such
techniques).  The two constraints must apply to any
representation of changesets.

\section{Notation}

\begin{itemize}
\item We use the algebraic multiplication notation to represent changeset application.
\item While changesets are defined as operations on documents, documents themselves are represented as a list of changesets, initially applying to the empty document.
\end{itemize}

\paragraph{Example}
$A=(0\rightarrow 5)[``hello"]$
$B=(5\rightarrow 11)[0-4, ``\ world"]$

We can write the document ``hello world'' as $A\cdot B$ or
just $AB$.  Note that the ``initial document'' can be made
into the changeset $(0\rightarrow
N)[``<\mathit{the\ document\ text}>"]$.

When $A$ and $B$ are changesets, we can also refer to $(AB)$ as ``the composition'' of $A$ and $B$.  Changesets are closed under composition.

\section{Composition of Changesets}

For any two changesets $A$, $B$ such that

\begin{itemize}
\item[] $A=(n_1\rightarrow n_2)[\cdots]$
\item[] $B=(n_2\rightarrow n_3)[\cdots]$
\end{itemize}
it is clear that there is a third changeset $C=(n_1\rightarrow n_3)[\cdots]$ such that applying $C$ to a document $X$ yields the same resulting document as does applying $A$ and then $B$.  In this case, we write $AB=C$.

Given the representation from Section \ref{representation}, it is straightforward to compute the composition of two changesets.

\section{Changeset Merging}

Now we come to realtime document editing.  Suppose two different users make two different changes to the same document at the same time.  It is impossible to compose these changes.    For example, if we have the document $X$ of length $n$, we may have $A=(n\rightarrow n_a)[\ldots n_a \mathrm{characters}]$,  $B=(n\rightarrow n_b)[\ldots n_b \mathrm{characters}]$ where $n\neq n_a\neq n_b$.

It is impossible to compute $(XA)B$ because $B$ can only be applied to a document of length $n$, and $(XA)$ has length $n_a$.  Similarly, $A$ cannot be applied to $(XB)$ because $(XB)$ has length $n_b$.

This is where \emph{merging} comes in.  Merging takes two changesets that apply to the same initial document (and that cannot be composed), and computes a single new changeset that preserves the intent of both changes.  The merge of $A$ and $B$ is written as $m(A,B)$.  For the Etherpad system to work, we require that $m(A,B)=m(B,A)$.

Aside from what we have said so far about merging, there are many different implementations that will lead to a workable system.  We have created one implementation for text that has the following constraints.

\section{Follows} \label{follows}

When users $A$ and $B$ have the same document $X$ on their screen, and they proceed to make respective changesets $A$ and $B$, it is no use to compute $m(A,B)$, because $m(A,B)$ applies to document $X$, but the users are already looking at document $XA$ and $XB$.  What we really want is to compute $B'$ and $A'$ such that
$$XAB' = XBA' = Xm(A,B)$$

``Following'' computes these $B'$ and $A'$ changesets.  The definition of the ``follow'' function $f$ is such that $Af(A,B)=Bf(B,A)=m(A,B)=m(B,A)$.  When we compute $f(A,B)$
\begin{itemize}
\item Insertions in $A$ become retained characters in $f(A,B)$
\item Insertions in $B$ become insertions in $f(A,B)$
\item Retain whatever characters are retained in \emph{both} $A$ and $B$
\end{itemize}

\paragraph{Example}

Suppose we have the initial document $X=(0\rightarrow 8)[``\mathit{baseball}"]$ and user $A$ changes it to ``basil'' with changeset $A$, and user $B$ changes it to ``below'' with changeset $B$.

We have
$X=(0\rightarrow 8)[``\mathit{baseball}"]$ \\
$A=(8\rightarrow 5)[0-1, ``\mathit{si}", 7]$ \\
$B=(8\rightarrow 5)[0, ``\mathit{e}", 6, ``\mathit{ow}"]$ \\

First we compute the merge $m(A,B)=m(B,A)$ according to the constraints

$$m(A,B)=(8\rightarrow 6)[0, "e", "si", "ow"] = (8\rightarrow 6)[0, ``\mathit{esiow}"]$$

Then we need to compute the follows $B'=f(A,B)$ and $A'=f(B,A)$.

$$B'=f(A,B)=(5\rightarrow 6)[0,``\mathit{e}",2,3,``\mathit{ow}"]$$

Note that the numbers $0$, $2$, and $3$ are indices into $A=(8\rightarrow 5)[0,1,``\mathit{si}",7]$

\begin{tabular}{ccccc}
0 & 1 & 2 & 3 & 4 \\
0 & 1 & s & i & 7
\end{tabular}

$A'=f(B,A)=(5\rightarrow 6)[0,1,"si",3,4]$

We can now double check that $AB'=BA'=m(A,B)=(8\rightarrow 6)[0,``\mathit{esiow}"]$.

Now that we have made the mathematical meaning of the
preceding pages complete, we can build a client/server
system to support realtime editing by multiple users.

\section{System Overview}

There is a server that holds the current state of a
document.  Clients (users) can connect to the server from
their web browsers.  The clients and server maintain state
and can send messages to one another in real-time, but
because we are in a web browser scenario, clients cannot
send each other messages directly, and must go through the
server always.  (This may distinguish from prior art?)

The other critical design feature of the system is that
\emph{A client must always be able to edit their local
  copy of the document, so the user is never blocked from
  typing because of waiting to send or receive data.}

\section{Client State}

At any moment in time, a client maintains its state in the
form of 3 changesets.  The client document looks like
$A\cdot X \cdot Y$, where

$A$ is the latest server version, the composition of all
changesets committed to the server, from this client or
from others, that the server has informed this client
about.  Initially $A=(0\rightarrow N)[<\mathit{initial\ document\ text}>]$.

$X$ is the composition of all changesets this client has
submitted to the server but has not heard back about yet.
Initially $X=(N\rightarrow N)[0,1,2,\ldots, N-1]$, in
other words, the identity, henceforth denoted $I_N$.

$Y$ is the composition of all changesets this client has
made but has not yet submitted to the server yet.
Initially $Y=(N\rightarrow N)[0,1,2,\ldots, N-1]$.

\section{Client Operations}

A client can do 5 things.

\begin{enumerate}
\item Incorporate new typing into local state
\item Submit a changeset to the server
\item Hear back acknowledgement of a submitted changeset
\item Hear from the server about other clients' changesets
\item Connect to the server and request the initial document
\end{enumerate}

As these 5 events happen, the client updates its
representation $A\cdot X \cdot Y$ according to the
relations that follow.  Changes ``move left'' as time goes
by: into $Y$ when the user types, into $X$ when change
sets are submitted to the server, and into $A$ when the
server acknowledges changesets.

\subsection{New local typing}

When a user makes an edit $E$ to the document, the client
computes the composition $(Y\cdot E)$ and updates its local
state, i.e. $Y \leftarrow Y\cdot E$.  I.e., if $Y$ is the
variable holding local unsubmitted changes, it will be
assigned the new value $(Y\cdot E)$.

\subsection{Submitting changesets to server}

When a client submit its local changes to the server, it
transmits a copy of $Y$ and then assigns $Y$ to $X$, and
assigns the identity to $Y$.  I.e.,

\begin{enumerate}
\item Send $Y$ to server,
\item $X \leftarrow Y$
\item $Y \leftarrow I_N$
  (the identity).
\end{enumerate}

This happens every 500ms as long as it receives an
acknowledgement.  Must receive ACK before submitting
again.  Note that $X$ is always equal to the identity
before the second step occurs, so no information is lost.

\subsection{Hear ACK from server}

When the client hears ACK from server,

$A \leftarrow A\cdot X$ \\
$X \leftarrow I_N$

\subsection{Hear about another client's changeset}

When a client hears about another client's changeset $B$,
it computes a new $A$, $X$, and $Y$, which we will call
$A'$, $X'$, and $Y'$ respectively.  It also computes a
changeset $D$ which is applied to the current text view on
the client, $V$.  Because $AXY$ must always equal the
current view, $AXY=V$ before the client hears about $B$,
and $A'X'Y'=VD$ after the computation is performed.

The steps are:

\begin{enumerate}
\item Compute $A' = AB$
\item Compute $X' = f(B,X)$
\item Compute $Y' = f(f(X,B), Y)$
\item Compute $D=f(Y,f(X,B))$
\item Assign $A \leftarrow A'$, $X \leftarrow X'$, $Y \leftarrow Y'$.
\item Apply $D$ to the current view of the document
  displayed on the user's screen.
\end{enumerate}

In steps 2,3, and 4, $f$ is the follow operation described
in Section \ref{follows}.

\paragraph{Proof that $\mathbf{AXY=V \Rightarrow A'X'Y'=VD}$.}
Substituting $A'X'Y'=(AB)(f(B,X))(f(f(X,B),Y))$, we
recall that merges are commutative.  So for any two
changesets $P$ and $Q$,
$$m(P,Q)=m(Q,P)=Qf(Q,P)=Pf(P,Q)$$

Applying this to the relation above, we see
\begin{eqnarray*}
A'X'Y'&=& AB f(B,X) f(f(X,B),Y) \\
      &=&AX f(X,B) f(f(X,B),Y) \\
      &=&A X Y f(Y, f(X,B)) \\
      &=&A X Y D \\
      &=&V D
\end{eqnarray*}
As claimed.

\subsection{Connect to server}

When a client connects to the server for the first time,
it first generates a random unique ID and sends this to
the server.  The client remembers this ID and sends it
with each changeset to the server.

The client receives the latest version of the document
from the server, called HEADTEXT.  The client then sets

\begin{itemize}
\item[] $A \leftarrow \mathrm{HEADTEXT}$
\item[] $X \leftarrow I_N$
\item[] $Y \leftarrow I_N$
\end{itemize}

And finally, the client displays HEADTEXT on the screen.

\section{Server Overview}

Like the client(s), the server has state and performs
operations.  Operations are only performed in response to
messages from clients.

\section{Server State}

The server maintains a document as an ordered list of
\emph{revision records}.  A revision record is a data
structure that contains a changeset and authorship
information.

\begin{verbatim}
RevisionRecord = {
  ChangeSet,
  Source (unique ID),
  Revision Number (consecutive order, starting at 0)
}
\end{verbatim}

For efficiency, the server may also store a variable
called HEADTEXT, which is the composition of all
changesets in the list of revision records.  This is an
optimization, because clearly this can be computed from
the set of revision records.

\section{Server Operations Overview}

The server does two things in addition to maintaining
state representing the set of connected clients and
remembering what revision number each client is up to date
with:

\begin{enumerate}
\item Respond to a client's connection requesting the initial document.
\item Respond to a client's submission of a new changeset.
\end{enumerate}

\subsection{Respond to client connect}
When a server receives a connection request from a client,
it receives the client's unique ID and stores that in the
server's set of connected clients.  It then sends the
client the contents of HEADTEXT, and the corresponding
revision number.  Finally the server notes that this
client is up to date with that revision number.

\subsection{Respond to client changeset}

When the server receives information from a client about
the client's changeset $C$, it does five things:

\begin{enumerate}
\item Notes that this change applies to revision number
  $r_c$ (the client's latest revision).
\item Creates a new changeset $C'$ that is relative to the
  server's most recent revision number, which we call
  $r_H$ ($H$ for HEAD).  $C'$ can be computed using
  follows (Section \ref{follows}).  Remember that the server has a series of
  changesets,
$$S_0\rightarrow S_1\rightarrow \ldots S_{r_c}\rightarrow S_{r_c+1} \rightarrow \ldots \rightarrow S_{r_H} $$
$C$ is relative to $S_{r_c}$, but we need to compute $C'$ relative to $S_{r_H}$.
We can compute a new $C$ relative to $S_{r_c+1}$ by computing $f(S_{r_c+1},C)$.  Similarly we can repeat for
$S_{r_c+2}$ and so forth until we have $C'$ represented relative to $S_{r_H}$.
\item Send $C'$ to all other clients
\item Send ACK back to original client
\item Add $C'$ to the server's list of revision records by creating a new revision record out of this and the client's ID.

\appendix

\section*{Additional topics}
\begin{enumerate}
\item Optimizations (strips, more caching, etc.)
\item Pseudocode for composition, merge, and follow
\item How authorship information is used to color-code the document based on who typed what
\item How persistent connections are maintained between client and server
\end{enumerate}
\end{enumerate}

\end{document}


\chapter{Python Relevant Code}
\label{chp:pythoncode}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Rserve Python Client} \label{sec:rconnect}

The class \lstinline{r_connect.py} is a client for R package Rserve. Makes
requests to ECoL R package and parses data.

The import statements have been made redundant in this snippet and the rest of
the snippets in this appendix.

\begin{lstlisting}[language=Python, caption={Connection code to requests R ECoL
functions}, label={lst:r-connect}]
class r_connect:

    __metrics = None

    def __init__ (self):
        self.__connection = self.__connect()

    def __connect (self):
        return pyRserve.connect()

    def get_metrics (self, X=None, Y=None):
        if X is None and Y is None :
            if self.__metrics is not None:
                return self.__metrics
            else :
                # No data and no parameters: finish execution
                error_message = '...'
                raise Exception(error_message)
                sys.exit (400)
        else :
            # Stores connection to R's RPC.
            connect = self.__connection

            # Sends the input matrix and the output vector to R.
            connect.r.X = X
            connect.r.y = Y

            # Library to use in R.
            connect.r('df_X <- as.data.frame(X)')
            connect.r('df_y <- as.data.frame(y)')
            connect.r('library("ECoL")')

            ## Metrics, uses a dictionary to provide a faster access to its
            # contents.
            metrics = {}

            # Balance: C1, C2
            balance = self.safe_connect('balance(df_X, df_y)')
            balance_dic_entry = { 'balance' :  balance }
            metrics.update (balance_dic_entry)

            # Correlation: C1, C2, C3, C4
            correlation = self.safe_connect('correlation(df_X, df_y, summary=c("mean"))')
            correlation_dic_entry = { 'correlation' : correlation }
            metrics.update (correlation_dic_entry)

            # Dimensionality: T2, T3, T4
            dimensionality = self.safe_connect('dimensionality(df_X, df_y, summary=c("mean"))')
            dimensionality_dic_entry = { 'dimensionality' : dimensionality }
            metrics.update (dimensionality_dic_entry)

            # Linearity: L1, L2, L3
            linearity = self.safe_connect('linearity(df_X, df_y, summary=c("mean"))')
            linearity_dic_entry = { 'linearity' : linearity }
            metrics.update (linearity_dic_entry)

            # Neighborhood: N1, N2, N3, N4, T1, LSC
            neighborhood = self.safe_connect('neighborhood(df_X, df_y, summary=c("mean"))')
            neighborhood_dic_entry = { 'neighborhood' : neighborhood }
            metrics.update (neighborhood_dic_entry)

            # Network: Density, ClsCoef, Hubs
            network = self.safe_connect('network(df_X, df_y, summary=c("mean"))')
            network_dic_entry = { 'network' : network }
            metrics.update (network_dic_entry)

            # Overlap: F1, F1v, F2, F3, F4
            overlap = self.safe_connect('overlapping(df_X, df_y, summary=c("mean"))')
            overlap_dic_entry = { 'overlap' : overlap }
            metrics.update (overlap_dic_entry)

            # Smoothness: S1, S2, S3, S4
            smoothness = self.safe_connect('smoothness(df_X, df_y, summary=c("mean"))')
            smoothness_dic_entry = { 'smoothness' : smoothness }
            metrics.update (smoothness_dic_entry)

            self.__metrics = metrics

            return metrics

    def print_metrics (self, metrics=None) :
        print ('\n\n=== Printing metrics ===', end='\n\n')

        ...
        ...

    def get_print_metrics(self, X, Y):
        self.get_metrics (X, Y)
        self.print_metrics (self.__metrics)

        return self.__metrics

    def safe_connect(self, operation) :
        connection = self.__connection

        return connection.r(operation)

\end{lstlisting}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Datasets and Operations on Data}

The following code contains the logic to read and parse datasets requested
through the function \lstinline{getDataset(...)}. The datasets can be shuffled
if specified through parameter, but in this situation no experiments used
that option (as the results should be replicated, the input that should always
be the same).

The class also uses \textit{sklearn} library to calculate some metrics. The
functions in this code simply parse the results so that they are easier to read
afterwards (no need to have more than 4 digits of precision in float numbers).

Not all the code has been copied, as some parts repeat.

\begin{lstlisting}[language=Python, caption={Load datasets, calculate metrics,
export results to CSV files, etc.}, label={lst:data}]
...
...

def __load_arff(path):
    '''
        Loads the dataset content of an .arff file into the workspace.
        Input:
            - path: Location of the file.
        Output:
            - dataset: Data from the file.
    '''

    dataset = []

    with open(path, 'r') as data:
        dataset = arff.load(data)['data']

    return dataset

def __load_csv(path):
    '''
        Loads the data content of an .csv file into the workspace.
        Input:
            - path: Location of the file.
        Output:
            - dataset: Data from the file.
    '''

    dataset = []

    with open (path, 'r') as csv_file:
        dataset = pd.read_csv(
            csv_file,
            sep=','
        )

    return dataset

def __parte_dataset(dataset, input_select, target_select):
    '''
        Takes a dataset and parses it to only take what is necessary.
        Inputs:
            - dataset: Input raw data as a Pandas Data Framework.
            - start: Where columns start to be useful.
        Output:
            - input: Input arrays of the dataset.
            - target: Target column of the dataset.
    '''

    num_rows    = len(dataset)
    last_column = target_select if (target_select < 0) else None

    # Input
    input_columns = dataset.columns[input_select:last_column]
    input = dataset[input_columns].head(num_rows).astype(float).to_numpy()

    # Target
    target_column = dataset.columns[target_select]
    target = dataset[target_column].head(num_rows)

    # Parse output
    sample = target[0]
    if not isinstance(sample, str):
        target = numpy.where(target > 0, 1, 0)
    else:
        has_faults = target == 'yes'
        target = numpy.where(has_faults , 1, 0)

    return input, target

def __data_preparation(path, input_select, target_select, type='arff', shuffle=False):
    '''
        Loads an .arff/.csv file and parses its content to input/output valid
        for a Machine Learning application.
        Input:
            - path: Location of the file.
            - start: Where columns start to be useful (string columns are not
                necessary).
            - type: It can be:
                - arff
                - csv
        Output:
            - input: Input arrays of the dataset.
            - target: Target column of the dataset.
    '''

    # Load data
    dataset = []
    if   type == 'arff':
        dataset = __load_arff(path)
    elif type == 'csv':
        dataset = __load_csv(path)
    else:
        raise Exception('Not a valid file type. Try with arff or csv!')
        sys.exit(404)
    dataset = pd.DataFrame(dataset)

    if shuffle:
        dataset = dataset.sample(frac=1)

    # Parse data
    data = (dataset, input_select, target_select)
    input, target = __parte_dataset(*data)

    return input, target

def __data_preparation_iris_dataset(path):
    # Load data
    dataset = __load_csv(path)
    dataset = pd.DataFrame(dataset)

    # Parse data
    start    = 0
    last     = -1
    num_rows = len(dataset)

    # Input
    input_columns = dataset.columns[start : last]
    input = dataset[input_columns].head(num_rows).astype(float).to_numpy()

    # Target
    target_column = dataset.columns[last]
    target = dataset[target_column].head(num_rows)

    # Parse target
    values = {'setosa': 1, 'versicolor': 2, 'virginica': 3}
    target = numpy.array( [
        values[type] for type in target
    ] )

    return input, target

def get_dataset(name, shuffle=False):
    '''
        Retrieves the data of a given dataset, separated into input information
        and target output - .arff or .csv files only.
        Input:
            - name: Dataset name.
                - ant
                - apache
                - camel
                - iris
                - ivy
                - jedit
                - log4j
                - poi
                - synapse
                - xalan
                - xerces
        Output:
            - input: Input arrays of the dataset.
            - target: Target column of the dataset.
    '''

    path       = ''     # Location of file
    start      = 0      # From which columns to use
    output_col = -2     # Position of output column
    type       = 'arff' # Type of file to be read. Default .arff

    if name == 'ant':
        # Retrieve data
        path = './dataset/ant-1.7.arff'
        start = 3

    elif name == 'apache':
        # Retrieve data
        path = './dataset/Apache.csv'
        start = 3
        output_col = -1
        type = 'csv'

    elif name == 'camel':
        # Retrieve data
        path = './dataset/camel-1.6.arff'
        start = 3

    elif name == 'iris': # Special case. Non-binary output. Needs different
                         # treatment.
        # Retrieve data
        path = './dataset/iris.csv'
        input, target = __data_preparation_iris_dataset(path, shuffle)

        return input, target

    elif name == 'ivy':
        # Retrieve data
        path = './dataset/ivy-2.0.arff'
        start = 3

    ...
    ...
    elif name == 'xerces':
        # Retrieve data
        path = './dataset/xerces-1.4.arff'
        start = 3

    elif name == 'hadoop-1':
        # Retrieve data
        path = './dataset/hadoop-proc-0.1.csv'
        start = 2
        output_col = 1
        type = 'csv'

    ...
    ...
    elif name == 'hadoop-8':
        # Retrieve data
        path = './dataset/hadoop-proc-0.8.csv'
        start = 2
        output_col = 1
        type = 'csv'

    else:
        raise Exception('The given dataset name is not valid.')
        sys.exit(404)

    # Return results
    input, target = __data_preparation(path, start, output_col, type, shuffle)

    return input, target

def confusion_matrix(x_test: list, y_test: list, classifier):
    '''
        Obtains the confusion matrix for a given testing dataset, with binary
        output.
        Input:
            - x_test: paremeters for testing dataset.
            - y_test: target/desired output for testing dataset.
            - classifier: trained classifier.
        Output:
            (
                true_positive: successfully predicted positives
                true_negative: successfully predicted negatives
                false_positive: unsuccessfully predicted positives
                false_negative: unsuccessfully predicted positives
            )
    '''
    # Confusion Matrix Cells
    true_positive:float  = 0
    true_negative:float  = 0
    false_positive:float = 0
    false_negative:float = 0

    # Calculate number of repetitions of each classification.
    for (input, target) in zip(x_test, y_test):
        prediction:int = classifier.predict([input])[0]

        if prediction == target:    # Success
            if prediction: true_positive += 1
            else:          true_negative += 1
        else:                       # Wrong
            if prediction: false_positive += 1
            else:          false_negative += 1

    # Transform absolute to relative values
    num_samples = len(y_test)
    true_positive  = true_positive  / num_samples
    true_negative  = true_negative  / num_samples
    false_positive = false_positive / num_samples
    false_negative = false_negative / num_samples

    return (true_positive, true_negative, false_positive, false_negative)

def recall(targets, predictions) -> float:
    '''
        Sensitivity, recall, hit rate or True Positive Rate (RPR)
        https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html
               TP       TP
        TPR = ---- = --------- = 1 - FNR
               P      TP + FN
    '''
    return round(metrics.recall_score(targets, predictions), 4)

def fall_out(targets, predictions) -> float:
    '''
        Fallout, or False Positive Rate (FPR)
               FP       FP
        FPR = ---- = --------- = 1 - TNR
               N      FP + TN
    '''
    # Calculate TN and FP rates
    num_items:int = len(targets)
    false_positive:float = 0
    true_negative: float = 0
    # Count
    for (t, o) in zip(targets, predictions):
        if (o == t and o == 0):
            true_negative  += 1
        if (o != t and o == 1):
            false_positive += 1
    try:
        # Rates
        true_negative  /= num_items
        false_positive /= num_items

        # False Positive Rate
        fdr:float = false_positive / (false_positive + true_negative)
        return round(fdr, 4)
    except:
        return 0

def precision(targets, predictions) -> float:
    '''
        Precision or positive predictive value (PPV).
        https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html
                 TP
        PPV = --------- = 1 - FDR
               TP + FP
    '''
    return round(metrics.precision_score(targets, predictions), 4)

def balanced(targets, predictions) -> float:
    '''
        Balanced Accuracy (BA)
        https://scikit-learn.org/stable/modules/generated/sklearn.metrics.balanced_accuracy_score.html#sklearn.metrics.balanced_accuracy_score
              TPR + TNR
        BA = -----------
                  2
    '''

    return round(metrics.balanced_accuracy_score(targets, predictions), 4)

def f1(targets, predictions) -> float:
    '''
        F1 Score. Is the harmonic mean of precision and sensitivity.
        https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html
                PPV x TPR         2 TP
        F1 = 2 ----------- = ----------------
                PPV + TPR     2 TP + FP + FN
    '''
    return round(metrics.f1_score(targets, predictions), 4)

def mcc(targets, predictions) -> float:
    '''
        Matthews Correlation Coefficient (MCC).
        https://scikit-learn.org/stable/modules/generated/sklearn.metrics.matthews_corrcoef.html
                          TP x TN - FP x FN
        MCC = --------------------------------------------
               sqrt((TP + FP)(TP + FN)(TN + FP)(TN + FN))
    '''
    return round(metrics.matthews_corrcoef(targets, predictions), 4)

def auc(targets, predictions) -> float:
    '''
        Area Under Receiver Operating Characteristic Curve,
        Area Under ROC Curve or AUC.
        https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html
        https://scikit-learn.org/stable/modules/generated/sklearn.metrics.auc.html
               1 + TPR - FPR
        AUC = ---------------
                    2
    '''
    fpr, tpr, thresholds = metrics.roc_curve(targets, predictions)

    return round(metrics.auc(fpr, tpr), 4)

def store_results(filename:str, metrics:list):
    with open('./code/results/' + filename + '.csv', mode='a', newline='') as csvfile:
        writer = csv.writer(csvfile, delimiter=',', quoting=csv.QUOTE_MINIMAL)
        writer.writerow(metrics)

def calculate_results(targets:list, predictions:list) -> list:
    # Metrics
    ppv:float = precision(targets, predictions)
    tpr:float = recall   (targets, predictions)
    fpr:float = fall_out (targets, predictions)
    ba :float = balanced (targets, predictions)
    fm :float = f1       (targets, predictions)
    m  :float = mcc      (targets, predictions)
    a  :float = auc      (targets, predictions)
    metrics:list  = [ppv, tpr, fpr, ba, fm, m, a]

    return metrics

def train_predict(clf, input_train, target_train, test):
    # Train
    clf.fit(input_train, target_train)
    # Test
    return clf.predict(test)
\end{lstlisting}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiment 1. Complexity Metrics Comparison}
\label{sec:exp-compl-metrics-code}

The objective is to see if there is any dependency between some complexity
metrics (balance, dimensionality, overlapping, etc.).

This programs obtains the complexity metrics (more in section \ref{sec:ecol})
from different datasets, and stores the results on a CSV file for their
comparison. The dataset is loaded, metrics are retrieved for every dataset and
those metrics are plotted and stored.

\begin{lstlisting}[language=Python, caption={Compare complexity metrics of
different datasets}, label={lst:metrics-comparison}]
...
...
from r_connect import r_connect
import data as DATASETS

def add_metrics(storage, dataset_metrics):
    '''
        Includes a new set of calculated metrics into a variable.
        It creates an array for each metric's measure.
        Input:
            - storage(dict) : Data structure to store the list of measures from
                the metrics of datasets.
            - dataset_metrics (dict): Metrics computed from a dataset.
        Output:
            - storage (dict): Metrics added to the already stored ones.
    '''
    # Iterate through the metrics
    for m in storage:
        # Metric might not have been calculated for dataset
        if m in dataset_metrics:
            metric = dataset_metrics[m]

            # Iterate through the measures of the metric
            for measure in storage[m]:
                try:
                    # Measure might not have been calculated for dataset
                    if measure in storage[m]:
                        storage[m][measure].append(float(metric[measure]))
                except Exception:
                    pass
        else:
            print('Something went wrong')

    return storage

def plot_metrics_comparison(metrics):
    '''
        Input:
            - metrics:
        Output:
    '''
    i = 0
    j = 0
    for metric in metrics:
        rows = 2
        cols = int(math.ceil(len(metrics[metric]) / rows))
        fig, axs = plt.subplots(rows, cols)
        for measure in metrics[metric]:
            DATASETS.store_results('metrics-hadoop', [metric, measure] + [round(i, 4) for i in metrics[metric][measure]])
            if cols == 1:
                if metrics[metric][measure]:
                    axs[i].bar(datasets, metrics[metric][measure])

                else:
                    axs[i].text(0.5, 0.5, 'No data')

                title = metric + ': ' + measure
                axs[i].set_title(title)
            else:
                if metrics[metric][measure]:
                    axs[i, j].bar(datasets, metrics[metric][measure])

                else:
                    axs[i, j].text(0.5, 0.5, 'No data')

                title = metric + ': ' + measure
                axs[i, j].set_title(title)

            if j == cols - 1:
                i += 1
                j =  0
            else:
                j += 1

        i = 0
        j = 0
        plt.show()

if __name__ == '__main__':
    '''
        Main code - to be executed
    '''

    # Datasets to analyze
    '''
    datasets = [
        'ant',
        'apache',
        'camel',
        'ivy',
        'jedit',
        'log4j',
        'synapse',
        'xalan',
        'xerces'
    ]
    '''
    datasets = [
        'hadoop-1',
        'hadoop-2',
        'hadoop-3',
        'hadoop-4',
        'hadoop-5',
        'hadoop-6',
        'hadoop-7',
        'hadoop-8',
    ]

    # Store metrics computed from datasets (datatype: dict)
    results = {
        'balance': {
            'C1': [],
            'C2': []
        },
        'correlation': {
            'C1': [],
            'C2': [],
            'C3': [],
            'C4': []
        },
        'dimensionality': {
            'T1': [],
            'T2': [],
            'T3': []
        },
        'linearity': {
            'L1': [],
            'L2': [],
            'L3': []
        },
        'neighborhood': {
            'N1':  [],
            'N2':  [],
            'N3':  [],
            'N4':  [],
            'T1':  [],
            'LSC': [],
        },
        'network': {
            'Density': [],
            'ClsCoef': [],
            'Hubs':    []
        },
        'overlap': {
            'F1':  [],
            'F1v': [],
            'F2':  [],
            'F3':  [],
            'F4':  []
        },
        'smoothness': {
            'S1': [],
            'S2': [],
            'S3': [],
            'S4': []
        }
    }

    # Connect to R
    connector = r_connect()

    # Get Metrics
    for set in datasets:
        print(set)
        inputs, targets = DATASETS.get_dataset(set)
        metrics = connector.get_metrics(inputs, targets)

        results = add_metrics(results, metrics)

    # Print Metrics
    print(results)
    plot_metrics_comparison(results)
\end{lstlisting}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiment 2. Compare Metrics on K-fold Cross Validation}
\label{sec:exp-kfold-code}

The function calls to the \textit{data} (referenced as \textit{DATASETS} in the
code) object are unfolded in the snippet \ref{lst:data}.

The objective is to look for a relation between confusion matrix metrics
(recall, fallout, etc.) between the folds generated out of a K-fold Cross
Validation. See if there is a certain linearity between folds and what might
cause certain behaviors.

In order do create this experiment, a dataset is loaded, and then it is folded
into k-equally-sized parts. For each part, three different classification
algorithms are trained and tested to compare the aforementioned metrics
(more about the metrics in \ref{sec:ev-metrics}). This same process is repeated
for every dataset.

\begin{lstlisting}[language=Python, caption={Calculate Metrics from Confusion
Matrix to compare in-fold results}, label={lst:metrics-kfold}]
...
...
import data as DATASETS

if __name__ == '__main__':
    # Data
    datasets = [
        'ant',
        'apache',
        'camel',
        'ivy',
        'jedit',
        'log4j',
        'poi',
        'synapse',
        'xalan',
        'xerces',
        'hadoop-1',
        'hadoop-2',
        'hadoop-3',
        'hadoop-4',
        'hadoop-5',
        'hadoop-6',
        'hadoop-7',
        'hadoop-8'
    ]

    for d in datasets:
        print('----------' + d + '----------')
        inputs, target = DATASETS.get_dataset(d)

        # K-fold Parameters
        k = 5
        kf = KFold(n_splits=k)
        splits = kf.split(inputs)
        #                   Precision Recall Fallout Balanced F1 MCC AUC
        mean_naive = np.array([0,       0,      0,      0,    0,  0,  0])
        mean_tree  = np.array([0,       0,      0,      0,    0,  0,  0])
        mean_knn   = np.array([0,       0,      0,      0,    0,  0,  0])

        # Get measurements using K-fold
        for train_index, test_index in splits:
            # Data partition
            x_train, x_test = inputs[train_index], inputs[test_index]
            y_train, y_test = target[train_index], target[test_index]

            filename = d + '-k' + str(k)

            # Train NN
            print('---> Naive Bayes')
            clf   = GaussianNB()
            predictions   = DATASETS.train_predict(clf, x_train, y_train, x_test)
            naive_metrics = DATASETS.calculate_results(y_test, predictions)
            mean_naive = np.add(mean_naive, naive_metrics)
            naive_metrics = ['Naive Bayes'] + naive_metrics
            DATASETS.store_results(filename, naive_metrics)

            print('---> Decision Tree')
            clf   = DecisionTreeClassifier()
            predictions  = DATASETS.train_predict(clf, x_train, y_train, x_test)
            tree_metrics = DATASETS.calculate_results(y_test, predictions)
            mean_tree = np.add(mean_tree, tree_metrics)
            tree_metrics = ['Decision Tree'] + tree_metrics
            DATASETS.store_results(filename, tree_metrics)

            print('---> Nearest Centroid')
            clf   = NearestCentroid()
            predictions = DATASETS.train_predict(clf, x_train, y_train, x_test)
            knn_metrics = DATASETS.calculate_results(y_test, predictions)
            mean_knn = np.add(mean_knn, knn_metrics)
            knn_metrics = ['Nearest Centroid'] + knn_metrics
            DATASETS.store_results(filename, knn_metrics)

            print('------------------------------')

        # K-fold Mean
        mean_naive = [round(i, 4) for i in (mean_naive / k)]
        DATASETS.store_results(filename, ['Naive Bayes Mean'] + mean_naive)
        mean_tree = [round(i, 4) for i in (mean_tree / k)]
        DATASETS.store_results(filename, ['Decision Tree Mean'] + mean_tree)
        mean_knn = [round(i, 4) for i in (mean_knn / k)]
        DATASETS.store_results(filename, ['Nearest Centroid Mean'] + mean_knn)

        print([mean_naive, mean_tree, mean_knn])
\end{lstlisting}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiment 3. Compare Metrics with Under-sampling and K-fold Cross
Validation}\label{sec:exp3-under}

The function calls to the \textit{data} (referenced as \textit{DATASETS} in the
code) object, can be observed in the snippet \ref{lst:data}. Also, the
experiment is the same as in \ref{lst:metrics-kfold}, although an
under-sampling filter is applied to each fold of the dataset before using any
classification algorithm.

This experiment has to be executed as many times as datasets want to be
analyzed, as a bulk execution of all the datasets would probably end in error.
The reason for this is that if the sampling strategy is too low, then
there should not be enough data to properly train the classification. Also,
it depends on the amount of samples of the minority class, that is why not a
low enough value can be set for all datasets, and a single-dataset execution
must be performed.

The values commented at the right of each dataset indicate the recommended
sampling strategy for that dataset.

\begin{lstlisting}[language=Python, caption={Calculate Metrics from Confusion
Matrix to compare in-fold results after applying under-sampling filter},
label={lst:metrics-kfold-undersampling}]
...
...
import data as DATASETS

if __name__ == '__main__':
    # Data
    # In this case a loop cannot be used to make all experiments, as the value
    # of the undersampling may vary on each dataset.
    dataset ='ant'         # 50
    #dataset = 'apache'    # 50
    #dataset = 'camel'     # 50
    #dataset = 'ivy'       # 30
    #dataset = 'jedit'     # 7
    #dataset = 'log4j'     # 11
    #dataset = 'synapse'   # 50
    #dataset = 'xalan'     # 6
    #dataset = 'xerces'    # 50
    #dataset = 'hadoop-1'  # 38
    #dataset = 'hadoop-2'  # 31
    #dataset = 'hadoop-3'  # 35
    #dataset = 'hadoop-4'  # 32
    #dataset = 'hadoop-5'  # 26
    #dataset = 'hadoop-6'  # 22
    #dataset = 'hadoop-7'  # 34
    #dataset = 'hadoop-8'  # 10

    inputs, target = DATASETS.get_dataset(dataset)

    # K-fold Parameters
    k = 5
    kf = KFold(n_splits=k, shuffle=True, random_state=1)
    splits = kf.split(inputs)
    #                  Precision  Recall Fallout Balanced F1 MCC AUC
    mean_naive = np.array([0,       0,      0,      0,    0,  0,  0])
    mean_tree  = np.array([0,       0,      0,      0,    0,  0,  0])
    mean_knn   = np.array([0,       0,      0,      0,    0,  0,  0])

    RANDOM_STATE = 42

    # Get measurements using K-fold
    for train_index, test_index in splits:
        # Data partition
        x_train, x_test = inputs[train_index], inputs[test_index]
        y_train, y_test = target[train_index], target[test_index]

        X, Y = make_imbalance(
            x_train, y_train,
            sampling_strategy={0:50, 1:50},
            random_state=RANDOM_STATE)

        filename = dataset + '-k' + str(k) + '-under'

        # Train NN
        print('---> Naive Bayes')
        clf   = GaussianNB()
        pipeline = make_pipeline(
            NearMiss(version=2),
            clf)
        predictions   = DATASETS.train_predict(pipeline, X, Y, x_test)
        naive_metrics = DATASETS.calculate_results(y_test, predictions)
        mean_naive = np.add(mean_naive, naive_metrics)
        naive_metrics = ['Naive Bayes'] + naive_metrics
        DATASETS.store_results(filename, naive_metrics)

        print('---> Decision Tree')
        clf   = DecisionTreeClassifier()
        pipeline = make_pipeline(
            NearMiss(version=2),
            clf)
        predictions  = DATASETS.train_predict(pipeline, X, Y, x_test)
        tree_metrics = DATASETS.calculate_results(y_test, predictions)
        mean_tree = np.add(mean_tree, tree_metrics)
        tree_metrics = ['Decision Tree'] + tree_metrics
        DATASETS.store_results(filename, tree_metrics)

        print('---> Nearest Centroid')
        clf   = NearestCentroid()
        pipeline = make_pipeline(
            NearMiss(version=2),
            clf)
        predictions = DATASETS.train_predict(pipeline, X, Y, x_test)
        knn_metrics = DATASETS.calculate_results(y_test, predictions)
        mean_knn = np.add(mean_knn, knn_metrics)
        knn_metrics = ['Nearest Centroid'] + knn_metrics
        DATASETS.store_results(filename, knn_metrics)

        print('------------------------------')

    # K-fold Mean
    mean_naive = [round(i, 4) for i in (mean_naive / k)]
    DATASETS.store_results(filename, ['Naive Bayes Mean'] + mean_naive)
    mean_tree = [round(i, 4) for i in (mean_tree / k)]
    DATASETS.store_results(filename, ['Decision Tree Mean'] + mean_tree)
    mean_knn = [round(i, 4) for i in (mean_knn / k)]
    DATASETS.store_results(filename, ['Nearest Centroid Mean'] + mean_knn)

    print([mean_naive, mean_tree, mean_knn])
\end{lstlisting}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiment 4. Compare Metrics with Over-sampling and K-fold Cross
Validation}

The function uses \textit{data} (referenced as \textit{DATASETS} in the code)
object - which is unfolded in the snippet \ref{lst:data}. Also, the experiment
is the same as in \ref{lst:metrics-kfold} and
\ref{lst:metrics-kfold-undersampling}, but in this situation an over-sampling
filter is applied to each fold of the dataset before starting the classification
process.

Once again, a bulk execution with all the datasets is performed - unlike in
Section~\ref{sec:exp3-under} there is minimum number to meet, so all the
datasets can be treated generically.

The oversampling algorithm selected for this experiment has been SMOTE
(see Section~\ref{smote} for more information). Also, the same three classifiers
used for the previous experiments are the ones used to compare results:
(1) Naive Bayes - Gaussian; (2) Decission Tree; and (3) kNN Nearest Centroid.

\begin{lstlisting}[language=Python, caption={Calculate Metrics from Confusion
Matrix to compare in-fold results after applying over-sampling filter},
label={lst:metrics-kfold-oversampling}]
...
...
import data as DATASETS

if __name__ == '__main__':
    # Data
    datasets = [
        'ant',
        'apache',
        'camel',
        'ivy',
        'jedit',
        'log4j',
        'poi',
        'synapse',
        'xalan',
        'xerces',
        'hadoop-1',
        'hadoop-2',
        'hadoop-3',
        'hadoop-4',
        'hadoop-5',
        'hadoop-6',
        'hadoop-7',
        'hadoop-8'
    ]

    for d in datasets:
        print('----------' + d + '----------')
        inputs, target = DATASETS.get_dataset(d)

        # K-fold Parameters
        k = 5
        kf = KFold(n_splits=k)
        splits = kf.split(inputs)
        #                   Precision Recall Fallout Balanced F1 MCC AUC
        mean_naive = np.array([0,       0,      0,      0,    0,  0,  0])
        mean_tree  = np.array([0,       0,      0,      0,    0,  0,  0])
        mean_knn   = np.array([0,       0,      0,      0,    0,  0,  0])

        # Get measurements using K-fold
        for train_index, test_index in splits:
            # Data partition
            x_train, x_test = inputs[train_index], inputs[test_index]
            y_train, y_test = target[train_index], target[test_index]

            filename = d + '-k' + str(k) + '-over'

            # Train NN
            # Pipeline to NN
            print('---> Naive Bayes')
            clf = make_pipeline(
                SMOTE(),
                GaussianNB())
            predictions   = DATASETS.train_predict(clf, x_train, y_train, x_test)
            naive_metrics = DATASETS.calculate_results(y_test, predictions)
            mean_naive = np.add(mean_naive, naive_metrics)
            naive_metrics = ['Naive Bayes'] + naive_metrics
            DATASETS.store_results(filename, naive_metrics)

            print('---> Decision Tree')
            clf = make_pipeline(
                SMOTE(),
                DecisionTreeClassifier())
            predictions   = DATASETS.train_predict(clf, x_train, y_train, x_test)
            tree_metrics = DATASETS.calculate_results(y_test, predictions)
            mean_tree = np.add(mean_tree, tree_metrics)
            tree_metrics = ['Decision Tree'] + tree_metrics
            DATASETS.store_results(filename, tree_metrics)

            print('---> Nearest Centroid')
            clf = make_pipeline(
                SMOTE(),
                NearestCentroid())
            predictions   = DATASETS.train_predict(clf, x_train, y_train, x_test)
            knn_metrics = DATASETS.calculate_results(y_test, predictions)
            mean_knn = np.add(mean_knn, knn_metrics)
            knn_metrics = ['Nearest Centroid'] + knn_metrics
            DATASETS.store_results(filename, knn_metrics)

            print('------------------------------')

        # K-fold Mean
        mean_naive = [round(i, 4) for i in (mean_naive / k)]
        DATASETS.store_results(filename, ['Naive Bayes Mean'] + mean_naive)
        mean_tree = [round(i, 4) for i in (mean_tree / k)]
        DATASETS.store_results(filename, ['Decision Tree Mean'] + mean_tree)
        mean_knn = [round(i, 4) for i in (mean_knn / k)]
        DATASETS.store_results(filename, ['Nearest Centroid Mean'] + mean_knn)

        print([mean_naive, mean_tree, mean_knn])

\end{lstlisting}

\cleardoublepage


\documentclass[modern]{aastex63}
\usepackage{amsmath}

% typography
\usepackage[T1]{fontenc}
\setlength{\parindent}{1.\baselineskip}
\newcommand{\acronym}[1]{{\small{#1}}}
\newcommand{\package}[1]{\textsl{#1}}
\newcommand{\code}[1]{\texttt{#1}}
\newcommand{\changes}[1]{{\textbf{#1}}}
\renewcommand{\twocolumngrid}{} % This is evil Hogg stuff right here.

% math
\newcommand{\nupeak}{\nu_\mathrm{peak}}
\newcommand{\numax}{\nu_\mathrm{max}}
\newcommand{\hquad}{~~}
\newcommand{\given}{\,|\,}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\T}{^{\!\mathsf{T}\!}}
\newcommand{\inv}{^{-1}}
\newcommand{\scalar}[1]{#1}
\renewcommand{\vector}[1]{\boldsymbol{#1}}
\newcommand{\vy}{\vector{y}}
\newcommand{\valpha}{\vector{\alpha}}
\newcommand{\veta}{\vector{\eta}}
\newcommand{\vtheta}{\vector{\theta}}
\renewcommand{\tensor}[1]{\mathbf{#1}}
\newcommand{\tB}{\tensor{B}}
\newcommand{\tC}{\tensor{C}}
\newcommand{\tK}{\tensor{K}}
\DeclareMathOperator{\diag}{diag}
\newcommand{\tLambda}{\tensor{\Lambda}}
\newcommand{\mat}[1]{\mathsf{#1}}
\newcommand{\mM}{\mat{M}}
\newcommand{\normal}{\mathcal{N}\!\,}

% aastex parameters
% \received{not yet; THIS IS A DRAFT}
%\revised{not yet}
%\accepted{not yet}
% % Adds "Submitted to " the argument.
% \submitjournal{ApJ}
\shorttitle{}
\shortauthors{bonaca \& hogg}

%@arxiver{}
\usepackage{amsmath}

\begin{document}\sloppy\sloppypar\raggedbottom\frenchspacing % trust me

\title{Likelihood for the forest of asteroseismic modes}

\correspondingauthor{Ana~Bonaca}
\email{ana.bonaca@cfa.harvard.edu}

\author[0000-0002-7846-9787]{Ana~Bonaca}
\affil{Center for Astrophysics | Harvard \& Smithsonian, 60 Garden Street, Cambridge, MA 02138, USA}

\author[0000-0003-2866-9403]{David~W.~Hogg}
\affiliation{Center for Cosmology and Particle Physics, Department of Physics, New York University}
\affiliation{Center for Data Science, New York University}
\affiliation{Max-Planck-Institut f\"ur Astronomie, Heidelberg}
\affiliation{Center for Computational Astrophysics, Flatiron Institute, 162 Fifth Avenue, NY 10010, USA}

\begin{abstract}\noindent % trust me
Foo and bar.
\end{abstract}

\section{Introduction}
\label{sec:intro}

- successes of asteroseismology
-- make sure to review the literature on red giants

- possible to do bc kepler and tess come close to ideal dataset for asteroseismology: sampling good, timebaseline sufficient to resolve modes
%- no data exactly that, but kepler and tess come close, therefore resolving modes
- typical workflow: fft + peak bagging

- dream: do large-scale asteroseismic missions
- while asteroseismic missions getting larger themselves, e.g., tess, plato, a wealth of time domain data at different cadence
- some work done (cite atlas/asassn paper), but much more possible if able to deal w gaia / lsst cadence
-> need generative model

- sth about why generative model hard / not done so far in asteroseismology (physical model hard / don't have a good theory of amplitudes? so doing fft bc can get frequencies alone easily)
- or work in other new approaches (e.g., machine learning - blancato paper, but won't be generalized to the ground based data), from the ground (Lomb-Scargle, regular sampling)
- examples of contexts where generative model worked, why it might be a good idea to try here
- benefits of being able to do inference:
-- can use even suboptimally sampled data, just will get noisier constraints
-- test properties of stars

- luminous red giants valuable for mapping the galaxy
- high amplitude, coherent models -- good asteroseismic targets too, especially for less precise photometry
- model treating modes as coherent in data stream, but they are not in the tess window for sun-like stars, geared towards giants where coherence times are long (would need to use a proper gp or cut data into pieces for sunlike stars)
- not solving all problems for all people, but can do this specific regime
- can take more data than before, maybe map the whole galaxy
- be modest on what we solve
- part of a big project that astronomy should solve

- paper plan:
2) develop likelihood
3) test on aguirre sample: identify where works well and where not
4) blind search: here they are + hopefully make sense
5) discussion

figures:
sec 2: pedagogical -- labeled powerspectrum w nupeak, dnu, bell params
sec 2: row 1: schematic of different models (just the comb, comb + nupeak marginalization, comb + bell); row 2: likelihood surfaces of the same lightcurve under these models
sec 3: compare likelihood + powerspectra on the bottom for different regimes
sec 4: HRD color-coded by inferre dnu, numax, etc

- beginning of the methods section: two likelihoods we could use
1) comprehensive to include all things we know about stars
2) quick, can search computationally, but doesn't have all the inputs (e.g., no odd/even l, or taking care of them in a weird way to avoid having a parameter, filtering, hard to apply to unevenly sampled streams, like gaia -- new kind of data might require new kinds of hacks if we want something computationally fast)

discussion:
-- talk about all kinds of data regimes: tess (very good data, short timebaseline, slightly gappy), ground-based (very good sampling, rv spec | sampling horrible, like lsst), space-based (precision great, sampling bad, gaia) -- regimes where fft troublesome (nobody has a plan for gaia, even thinks its possible)
-- regimes of timescales (min dt between observations, max T, baseline)
-- in asteroseismic spectrum relevant timescales: 1/numax, 1/dnu, coherence time (1/true width of the modes) -- very separated, so a lot of datasets are in between (giants probably not even resolved in kepler, coherence time longer than 4 yrs -- spinoff undergrad paper for kepler: can we measure coherence time for giant modes)

\section{Data}\label{sec:data}

... Words here ...

\section{Likelihood and marginalized likelihood}\label{sec:lhood}

There are many ways to construct a likelihood function for an
asteroseismic-like frequency comb that generates a light curve.
We will build up in stages of increasing sophistication.
The simplest idea is that the light curve is generated by a set (``comb'') of
$K$ equally spaced frequencies.

Presume that there are $N$ data points (normalized flux measurements)
$y_n$, normalized and then unit-subtracted (or high-pass filtered??)
to have a mean very near zero, and a quantitative interpretation in terms
of fractional variation of the star.
The data points are taken at (precisely known) times $t_n$.
Each of these flux measurements has some kind of uncertainty estimate
$\sigma_n$ associated with it and inverse variance $\sigma_n^{-2}$.
The simplest $K$-frequency model is that these flux measurements are
generated by a set of $K$ frequencies $\nu_k$ (where $K$ is odd) as follows:
\begin{align}
  y_n &= f(t_n) + \mbox{noise}
  \\
  f(t_n) &= a_0 + \sum_{k=1}^K \left[a_k\,\cos(2\pi\,\nu_k\,t_n) + b_k\,\sin(2\pi\,\nu_k\,t_n)\right]
  \\ \label{eq:nuk}
  \nu_k &= \nupeak + \left(k - \frac{K + 1}{2}\right)\,\Delta\nu
  \quad \mbox{for $1\leq k\leq K$, $K$ odd,}
\end{align}
where $\nupeak$ is the position of one line (preferably a central line)
and $\Delta\nu$ is the asteroseismic ``large'' frequency difference.
In this formulation, $a_0$ and the $a_k,b_k$ comprise a vector of
$2\,K+1$ linear parameters and $\nupeak, \Delta\nu$ comprise a
blob of 2 nonlinear parameters.
We assemble the linear and nonlinear parameters into column vectors $\vtheta,\veta$
\begin{align}
  \vtheta &\equiv \begin{bmatrix} a_0 & a_1 & b_1 & a_2 & b_2 & \hdots & a_K & b_K \end{bmatrix}\T
  \\
  \veta &\equiv \begin{bmatrix} \Delta\nu & \nupeak & \hdots & K \end{bmatrix}\T
  \quad ,
\end{align}
where the ellipsis indicates that we will be augmenting the
nonlinear-parameter vector $\veta$ in a moment.

Under the assumption of Gaussian noise, the basic log likelihood $\ln L_0$ looks like a
chi-squared:
\begin{align}\label{eq:like}
  \ln L_0(\vtheta,\veta)
  &= -\frac{1}{2}\,\sum_{n=1}^N \frac{[y_n - f(t_n)]^2}{\sigma_n^2}
     -\frac{1}{2}\,\sum_{n=1}^N \ln(2\pi\,\sigma_n^2)
  \quad,
\end{align}
where we have included the $\ln\sigma^2$ terms because they will reappear later.
The basic likelihood is a function of both the linear parameters $\vtheta$ and
the nonlinear parameters $\veta$ because the $f(t)$ functions depend on these
parameters, and nothing has been marginalized out (yet).
The optimization of the basic log likelihood $\ln L_0$
over the linear parameters $a_0$ and the $a_k, b_k$, and the nonlinear
parameters $K, \nupeak, \Delta\nu$, would by itself provide a
classical-statistics estimate of the large frequency difference
$\Delta\nu$.

Acting on real data, this likelihood function is a mind-bogglingly complex
function of the nonlinear parameters.
For this reason, the only safe optimization method in $\Delta\nu, \nupeak$ is
brute-force search!
We perform brute-force search for example light curves  and show the results in XXX
and YYY.
For signal-processing reasons, the brute-force grid in grid search must be
substantially finer than a frequency resolution of $\delta\nu = 1/(K\,T)$ where
$K$ is the number of modes in the comb, and $T$ is the full time span of the
light curve data ($T = t_N - t_1$).
That hurts.

If we define a few linear-algebra operators, we can write the likelihood in a more
generalizable form.
We construct a N-vector (column vector) $\vy$, a diagonal square noise
variance tensor $\tC$, and a rectangular design matrix $\mM$:
\begin{align}
  \vy &\equiv \begin{bmatrix} y_1 & y_2 & y_3 & \hdots & y_N \end{bmatrix}\T
  \\
  \tC &\equiv \diag(\begin{bmatrix} \sigma_1^2 & \sigma_2^2 & \sigma_3^2 & \hdots & \sigma_N^2 \end{bmatrix})
  \\
  \mM &\equiv \begin{bmatrix}
    1 & \cos(2\pi\nu_1 t_1) & \sin(2\pi\nu_1 t_1) & \cos(2\pi\nu_2 t_1) & \hdots & \sin(2\pi\nu_K t_1) \\
    1 & \cos(2\pi\nu_1 t_2) & \sin(2\pi\nu_1 t_2) & \cos(2\pi\nu_2 t_2) & \hdots & \sin(2\pi\nu_K t_2) \\
    1 & \cos(2\pi\nu_1 t_3) & \sin(2\pi\nu_1 t_3) & \cos(2\pi\nu_2 t_3) & \hdots & \sin(2\pi\nu_K t_3) \\
    \vdots & \vdots         & \vdots              & \vdots              &        & \vdots \\
    1 & \cos(2\pi\nu_1 t_N) & \sin(2\pi\nu_1 t_N) & \cos(2\pi\nu_2 t_N) & \hdots & \sin(2\pi\nu_K t_N) \\
  \end{bmatrix}
  \quad ,
\end{align}
where the $\diag()$ operator makes a diagonal matrix with the inputs down
the diagonal.
With these definitions, the basic log likelihood becomes
\begin{align}
  \ln L_0(\vtheta,\veta) &= -\frac{1}{2}\,[\vy - \mM\cdot\vtheta]\T\cdot\tC\inv\cdot[\vy - \mM\cdot\vtheta] - \frac{1}{2}\ln\det(2\pi\,\tC)
  \quad ,
\end{align}
where the $\vtheta$ dependence is explicit, and the $\veta$ dependence
is implicit through the design matrix $\mM$.
For actual implementation of this log likelihood, we never construct
the exceedingly sparse tensor $\tC$ or $\tC\inv$ explicitly, and we
never call any kind of linear-algebra \code{inv()} or \code{logdet()} function:
The inverse of $\tC$ is just the $\diag()$ of the inverses, and the
log determinant is just the sum of the logs.

But we know more than just what's encoded in the basic log-likelihood function $\ln L_0$.
We want to use our knowledge.
One thing we know is that the amplitudes $\sqrt{a_k^2+b_k^2}$
are expected to be higher for the
modes $\nu_k$ closer to the star's characteristic frequecy $\numax$.
One model for this, inspired by the idea that they might be driven
by an effectively Gaussian noise internal to the star, is that the mode
amplitudes $a_k, b_k$ are drawn independently from a Gaussian with zero
mean and variance $H_k$ that depends on mode frequency $\nu_k$ as
\begin{equation}\label{eq:bell}
  H_k = H\,\exp(- \frac{[\nu_k - \numax]^2}{\Gamma^2})
  \quad ,
\end{equation}
where parameters $H, \numax, \Gamma$ set the height, center, and width of
the envelope of mode variances, and the frequencies $\nu_k$ are given
above (\ref{eq:nuk}).
We can pile these into another diagonal variance tensor $\tLambda$---this time
a prior variance not a noise variance
\begin{align}
  \tLambda &\equiv \diag(\begin{bmatrix} \Lambda_0 & H_1 & H_1 & H_2 & H_2 & \hdots & H_K & H_K \end{bmatrix})
  \quad ,
\end{align}
where $\Lambda_0$ is a (small) prior variance for the constant term $a_0$.
Recall that the flux data were normalized to unity and then had unity
(or a filter??) subtracted, to give them mean zero.

Once we have set our prior beliefs about the mode amplitudes using $H,
\numax, \Gamma$, we don't care about the amplitudes' particular values.
Following \citet{gaussianproduct}, we can marginalize out these amplitudes
with a closed-form expression to make the
linear-marginalized log-likelihood function $\ln L_1$:
\begin{align}
  \ln L_1(\veta) &= -\frac{1}{2}\,\vy\T\cdot\tB\inv\cdot\vy - \frac{1}{2}\ln\det(2\pi\,\tB)
  \\
  \tB &= \tC + \mM\cdot\tLambda\cdot\mM\T
  \\
  \veta &\equiv \begin{bmatrix} \Delta\nu & \numax & \nupeak & H & \Gamma & \Lambda_0 & K \end{bmatrix}\T
  \quad ,
\end{align}
where $\ln L_1(\veta)$ depends only on the nonlinear parameters $\veta$ because
the linear parameters $\vtheta$ have been marginalized out,
$\tB$ is a composite variance tensor taking account both the noise
and the prior variance, and the nonlinear parameter list $\veta$ just got
longer.
Once again, implementation of the expression for $\ln L_1$ is not straightforward:
We use the matrix inversion lemma and matrix determinant lemma to simplify each
term to a smaller linear-algebra operation than what's implied by any naive
implementation.

Still in the vector $\veta$ there are too many nonlinear parameters for brute-force
search, so we reduce dimensionality
using known regularities of stars:
Mode amplitudes are a function of evolutionary
state, where evolutionary state can be represented by the characteristic
frequency $\numax$, or the position of
the star on the color-magnitude diagram or in the space of temperature
and surface gravity.
...
\begin{align}
  H &\leftarrow \mbox{some function of $\numax$ here}
  \\
  \Gamma &\leftarrow \mbox{some function of $\numax$ here}
  \quad ,
\end{align}
... and the log likelihood becomes very insensitive to $K$ once $K$ is large enough;
we set $K=XX$ in all that follows; this is large enough.

... HOGG thinks that we should also set something like
\begin{align}
  \Lambda_0 &\leftarrow \frac{H}{(\numax\,T)^2} \quad\mbox{or}\quad
                        \frac{H\,\Gamma}{\Delta\nu\,(\numax\,T)^2}
  \quad ,
\end{align}
for signal-processing reasons?

... Make $\ln L_2$, fixing $H, \Gamma$ to be deterministic functions of $\numax$
and marginalized over $\nupeak$  ...
\begin{align}
  \ln L_2(\numax,\Delta\nu) = \ln \left[\frac{1}{\Delta\nu}\,\int_{\numax-\frac{1}{2}\,\Delta\nu}^{\numax+\frac{1}{2}\,\Delta\nu} \exp\left(\ln L_1(\veta)\right)\,\dd\nupeak\right]
  \quad ,
\end{align}
where ...
... implementation notes ... in addition to all the above, we have to be
very careful how we construct the $\nupeak$ grid for the numerical marginalization...

One thing missing in all of the above is that the asteroseismic comb of
frequencies is not a single comb, as described, but rather two combs, one
for the even-$\ell$ and one for the odd-$\ell$ modes, with these two combs
offset from one another and with different expected amplitudes.
Right now we are simply ignoring this fact, because we don't see how to
implement both combs without introducing another finely gridded nonlinear
parameter (the small frequency difference $\delta$) and maybe another
coarse nonlinear parameter (a ratio of amplitude variances ($H_{\mathrm{odd}}/H$).
However, since the second set of modes brings significant power,
the method would almost certainly be improved if we could afford the additional
parameters. We will return to this below (will we?).

Finally, if we wanted to include a Gaussian-Process noise model to
take care of the red noise in the spectrum, we would ``just'' add a kernel
matrix $\tK$ into the marginalized variance tensor $\tB$:
\begin{align}
  \tB &= \tC + \tK + \mM\cdot\tLambda\cdot\mM\T
  \\
  [\tK]_{nn'} &= q(|t_n - t_n'|; \valpha)
  \quad ,
\end{align}
where $[\tK]_{nn'}$ denotes the $(n, n')$ entry of the matrix $\tK$,
$q()$ is a kernel function, and $\valpha$ is a vector of hyper-parameters
controlling the height and shape of the kernel.
Conceptually this is simple, but in practice it slows things down.
That said, DFM (CITE) has very fast methods; maybe we should do this?

\section{Experiments and results}

foo and bar

\section{Discussion}

Hello World!

\acknowledgments
It is a pleasure to thank
Andy Casey (Monash),
Stephen Feeney (UCL),
and
Dan Foreman-Mackey (Flatiron)
for valuable work and discussions related to this project.

\software{
\package{Astropy} \citep{astropy, astropy:2018},
\package{IPython} \citep{ipython},
\package{matplotlib} \citep{mpl},
\package{numpy} \citep{numpy}
}

\bibliographystyle{aasjournal}
\bibliography{tess_seismology}

\end{document}


\section{Conclusions}
\label{sec:conclusions}

\lang is an ML-like language extended with sound handling of linear and affine resources. Its main novel feature is the combination of full type inference and a practically useful notion of shared and exclusive borrowing of linear and affine resources.
Although the inferred types are much richer internally than plain ML types, most of that complexity can be hidden from user-level programmers. On the other hand, programmers of libraries dealing with resources have sufficient expressiveness at their fingertips to express many resource management schemes.

The main restriction of the current system is that the lifetime of borrows is determined by lexical scoping. Overcoming this restriction is subject of future work and will probably require extending the type system by some notion of effect, which is currently discussed in the OCaml community.
Moreover, other systems rely on existential types for extra expressiveness. We chose not to include existentials to preserve complete type inference, but our design can be extended in this direction.
Finally, our matching construct is very simplistic.
Our implementation supports full algebraic data types and we believe it
can be further extended to support manipulating borrows of data-structures and
internal mutability.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End: