%Chapter 2

\renewcommand{\thechapter}{2}

\chapter{Comparing and Evaluating Metagenomic Assemblies}

\section{Abstract}
%\boldmath
The ever-decreasing costs of sequencing technology has led to a sharp increase in metagenomics projects over the past decade, allowing us to better understand the diversity and function of microbial communities found in the world around us.
The first step in these analyses is to perform
genome assembly to piece together the DNA fragments into complete, or
near complete, genomes.
%The first step in these analyses involves the use of tools called assemblers that piece together the DNA fragments into complete or near complete genome sequences.
Metagenomic assemblers inherit all the difficulties of traditional single genome assembly, but with the additional complexity of trying to resolve assemblies of closely related species with drastically varying abundances.
Assessing and comparing the quality of single genome assembly still relies on the availability of independently determined standards, such as manually curated genomic sequences.
These standards are often not possible in metagenomic projects, where a large portion of the organisms and strains are novel.
Thus, we must rely on \emph{de novo} methods for assessing and comparing assembly qualities.
Here we describe an extension to our \emph{de novo} LAP framework to evaluate metagenomic assemblies.
We will show that by modifying our likelihood calculation to take into account abundances of assembled sequences, we can accurately and efficiently compare metagenomic assemblies.
%We evaluate our extended framework on results generated from the Human Microbiome Project (HMP) and
We find that our extended LAP framework is able to reproduce results on data from the Human Microbiome Project (HMP) that closely match the reference-based evaluation metrics and outperforms other \emph{de novo} metrics traditionally used to measure assembly quality.
Finally, we have integrated our LAP framework into the metagenomic analysis pipeline MetAMOS, allowing any user to reproduce quality assembly evaluations with relative ease.
%Even without knowledge of the true reference sequences, our \emph{de novo} metric
%Here we introduce an extension to our LAP  \emph{de novo} probabilistic measure of assembly quality, which allows for an objective comparison of multiple assemblies generated from the same set of reads.
%We define the quality of a sequence produced by an assembler, as the conditional probability of observing the sequenced reads from the assembled sequence.

\section{Introduction}
The genome sequence of an organism is a vital resource for biologists trying to better understand its function and evolution.
Generating this sequence is not an easy task as modern sequencing technologies can only ``read'' small pieces of the genome.
These sequences, known as \emph{reads}, have to be pieced together by tools called assemblers using a collection of different heuristics since in almost all practical cases, assemblers cannot fully and accurately reconstruct the genome~\cite{myers1995,medvedev2007computability}.

Practical implementations of assembly algorithms (such as ABySS~\cite{ABySS}, SOAPdenovo~\cite{SOAPdenovo}, Velvet~\cite{Velvet}, etc.) return just an approximate solution that is often fragmented and contains numerous errors.
Ideally, further experiments would be performed manually to correct the hundreds to thousands of errors~\cite{salzberg2005misassemblies}, and fill in the gaps between the chunks of assembled sequences (called contigs)~\cite{nagarajan2010finishing}.
However, the additional cost and effort necessary to finish a genome is only justifiable for a few high-priority organisms (typically model organisms).
Recent technology advances have demonstrated automatically assembled finish-grade quality bacterial genomes~\cite{chin2013nonhybrid, koren2013reducing, ribeiro2012finished}. However, the majority of genomes sequences available today are considered to be in a ``draft'' state, with no clear indication of their respective quality, possibly impacting the conclusions and experiments done on their sequences.

Despite the unresolved challenges of clonal genome assembly, the decreasing costs of sequencing technology has led to a sharp increase in metagenomics projects over the past decade.
% have been sharply on the rise over the past decade,
These projects allow us to better understand the diversity and function of microbial communities found in the environment, including the ocean\cite{rusch2007sorcerer,wu2011stalking,yooseph2007sorcerer}, Arctic regions \cite{varin2012metagenomic}, other living organisms\cite{he2013comparative} and the human body\cite{gill2006metagenomic,peterson2009nih}.
Traditional \emph{de novo} genome assemblers have trouble assembling these datasets due to the presence of closely related species and and the need to distinguish between true polymorphisms and errors arising from the sequencing technology.
Metagenomic assemblers often use heuristics based on sequencing (Meta-IDBA \cite{peng2011meta} and MetaVelvet\cite{namiki2012metavelvet}) and k-mer (Ray Meta \cite{boisvert2012ray}) coverage to split the assembly graph into subcomponents that represent different organisms, then apply traditional assembly algorithms on the individual organisms.

As the number of metagenomic assemblers available to researchers continues to increase, the development of approaches for validating and comparing the output of these tools is of critical importance.
Despite the incremental improvements in performance, none of the assembler tools available today outperforms the rest in all cases (as highlighted by recent assembly
bake-offs GAGE\cite{salzberg2011gage} and Assemblathons 1~\cite{earl2011assemblathon} and 2~\cite{bradnam2013assemblathon}).
Different assemblers ``win'' depending on the specific downstream analyses, structure of the genome, and sequencing technology used.
These competitions highlight the inherent difficulty of assessing assembly quality -- where do you set the line between increased contiguity and decreasing accuracy of the resulting sequence?
Evaluating the trade-off between increased contiguity and errors is difficult even when there is a gold standard reference genome to compare to, which is not available in most practical assembly cases.
Thus, we are forced to heavily rely on \emph{de novo} approaches based on sequence data alone.
% (including global ``sanity checks,'' such as gene density, which is expected to be high in bacterial genomes).

One objective \emph{de novo} metric, that has been used to evaluate and compare assembly quality, is based on the likelihood that the observed reads are generated from the given assembly, which can be accurately estimated by modeling the sequencing process.
This metric was proposed by Gene Myers in his pioneering work in the 1990's, where he suggested that the correct assembly must be consistent with the statistical properties of the data generation process.
This idea was extended and used by recent assembly evaluation frameworks: ALE ~\cite{clark2013ale}, CGAL~\cite{rahman2013cgal}, and LAP~\cite{LAP}.

%In pioneering work in the 1990's, Gene Myers suggested that the correct assembly given the set of reads must be consistent with the data generation process.
%This idea was extended and used by assemblers and assembly evaluation software: the arrival-rate statistic (A-statistic) in Celera
%assembler~\cite{CeleraAssembler} to identify collapsed repeats, and
%as an objective function in quasi-species (ShoRAH~\cite{SHORAH},
%ViSpA~\cite{VISPA}), metagenomic (Genovo~\cite{genovo2011}),
%general-purpose assemblers~\cite{medvedev2009maximum}, and recent assembly
%evaluation frameworks.

Most of the previous \emph{de novo} and reference-based validation methods have been designed for single genome assembly.
Currently, there are no universally-accepted reference-based metrics for evaluating metagenomic assemblies.
Despite reference sequences being available for a small fraction of organisms found in metagenomic environments \cite{angly2006marine,dinsdale2008functional}, it is not clear how to distinguish errors from genomic variants found within a population.
%In addition to chimeric contigs within a single organism, there are now potential chimeras that span multiple organisms.
Furthermore, it is not clear how to weigh errors occurring in more abundant organisms.
Likelihood-based frameworks, such as ALE ~\cite{clark2013ale}, CGAL~\cite{rahman2013cgal}, and LAP~\cite{LAP}, rely on the assumption that the sequencing process is approximately uniform across the genome; however, the sequencing depth across genomes in metagenomic samples can vary greatly~\cite{carrigg2007dna,krsek1999comparison,morgan2010metagenomic,temperton2009bias}.

%Assessing the quality of the sequence output by an assembler is of critical importance for downstream analyses and allows researchers to choose from a collection of genome assembler.

In our paper, we describe an extension to our LAP framework to evaluate metagenomic assemblies.
We will show that by modifying our likelihood calculation to take into account contig abundances, we can accurately and efficiently evaluate metagenomic assemblies.
We evaluate our extended framework on data from the Human Microbiome Project (HMP).
Finally, we show how our LAP framework can be used automatically by the metagenomic assembly pipeline, MetAMOS\cite{treangen2013metamos}, to select the best assembler for a specific dataset, and to provide users with a measure of assembly quality.
The software implementing our approach is made available, open-source and free of charge, at: \url{http://assembly-eval.sourceforge.net/} and with the MetAMOS package: \url{https://github.com/treangen/MetAMOS}.

%\hfill mds

%\hfill July 17, 2013

%\newpage
\begin{figure}
\begin{center}
\includegraphics[width=4.572in]{metagenome}
\end{center}
\renewcommand{\baselinestretch}{1}
\small\normalsize
\begin{quote}
\caption[The metagenome of an environment]{The metagenome of an environment can be viewed as the concatenation of the organisms found in the environment whose multiplicity is determined by their abundance. \label{fig:metagenome}}
\end{quote}
\end{figure}
\renewcommand{\baselinestretch}{2}
\small\normalsize
\newpage

% \begin{figure}[!t]
% \centering
% \includegraphics[width=3in]{metagenome}
% % where an .eps filename suffix will be assumed under latex,
% % and a .pdf suffix will be assumed for pdflatex; or what has been declared
% % via \DeclareGraphicsExtensions.
% \caption{The metagenome of an environment can be viewed as the concatenation of the organisms found in the environment whose multiplicity is determined by their abundance.}
% \label{fig:metagenome}
% \end{figure}

\section{Methods}
\subsection{Likelihood of an assembly}

Our LAP framework measures the quality of an assembly as the probability that the observed reads, $R$, are generated from the given assembly, $A$: $\Pr[R|A]]$ \cite{LAP}.
Assuming that the event of observing each read is independent, then the probability $\Pr[R|A]$ of the read set $R$ being produced from the assembly $A$, is the product of the individual read probabilities, $p_r$.  That is,
\begin{equation}
  \label{probability_reads_given_assembly}
  \Pr[R \vert A]=\prod_{r \in R} p_r
\end{equation}

By modeling the data generation process, we can calculate the probability of each read, $p_r$.
Assuming uniform coverage, where each position in the genome is covered by roughly the same amount of reads as any other position, a read may be sequenced starting from any position of the genome with equal probability.
In the basic error-free model, if a read matches to one position in the assembly, and its reverse-complement does not match anywhere, then the probability of the read being produced from the assembly is $p_r=\frac{1}{2L}$, where $L$ is the length of the assembly.
The length of the assembly is doubled due to the double-stranded molecules of DNA that make up the genome.
Thus, if a read
%and its reverse-complement
matches at $n_r$ positions on the assembly, then
% and its reverse-complement
\begin{equation}
  \label{error_free_probability}
  p_r = \frac{n_r}{2L}
\end{equation}

Ghodsi et. al.~\cite{LAP} detail how to modify the calculation of $p_r$ to handle practical constraints, e.g., sequencing errors and mate pairs.
They also show that the true genome maximizes $\Pr[R|A]$.

%The metric used in accessing the assembly probability is the logarithm of the geometric mean of the read probabilities, referred to as the log average probability (LAP).
Calculating $\Pr[R|A]$ can be expensive for dataset sizes commonly encountered in sequencing projects (tens to hundreds of millions of reads).
Thus, we can approximate the likelihood of the assembly by using a random subset of the reads ($R^\prime$).
To counteract the effect of sample size on the probability, we define the assembly quality as the geometric mean of the read probabilities:
%
\begin{align}
\label{average_log_probability}
  \operatorname{AP}(R^\prime) =
  \left(\prod_{r \in R^\prime} p_r\right)^{\frac{1}{\left|R^\prime\right|}} \nonumber  \\
  \operatorname{LAP}(R^\prime) = \log_{10} \left( \operatorname{AP}(R^\prime) \right) = \frac{\sum_{r \in R^\prime} \log_{10} p_r}{\left|R^\prime\right|}
\end{align}
%
%For the remainder of this paper, we define the assembly quality as the average log likelihood (LAP) of the reads given the assembly.
%This formulation allows us to estimate the
The mean of the read probabilities over the sample is expected to be equal to the mean over all reads, but if the sample size is too small, then the accuracy of the estimation will be poor.

\begin{figure}
\begin{center}
%\begin{figure*}[tbhp!]
%\centering

\subfigure[\emph{B. cereus} (1 copy, 5.2MB) and \emph{A. baumannii} (4 copies, 4.0MB)]{\includegraphics[width=4in]{ref_abun_1} \label{fig:ref_abun_1}}
\hfil
\subfigure[\emph{B. cereus} (4 copies, 5.2MB) and \emph{A. odontolyticus} (7 copies, 2.4MB)]{\includegraphics[width=4in]{ref_abun_2} \label{fig:ref_abun_2}}
\end{center}
\renewcommand{\baselinestretch}{1}
\small\normalsize
\begin{quote}
\caption[LAP scores for simulated metagenomic communities.] {LAP scores for simulated metagenomic communities. Each cell (x,y) represents the LAP score for a mixture of \emph{x} copies of the x-label bacteria and \emph{y} copies of the y-label bacteria.  In both groups, the true abundance ratios maximize the LAP score (indicated by a black rectangle in respective plots). \label{fig:ref_abun}}
\end{quote}
\end{figure}
\renewcommand{\baselinestretch}{2}
\small\normalsize
\newpage

\subsection{Extending LAP to metagenomic assemblies}

An important simplifying assumption of our framework is that the sequencing process is uniform in coverage.
In metagenomics, however, the relative abundances of organisms are rarely uniform~\cite{carrigg2007dna,krsek1999comparison,morgan2010metagenomic,temperton2009bias}, reflecting the difference in abundance between the different organisms within a community.
Here we show that taking this abundance information into account allows us to extend the LAP framework to metagenomic data.
%This feature makes using the LAP score incorrect out of the box.
%Later steps in the pipelines often deal with abundance estimation and phylogenetic classification.
We now assume that while the abundances of each organism may vary dramatically, the sequencing process still has uniform coverage across the \emph{entire} community.
%It is important to note while the abundances of each organism may vary dramatically, the sequencing process still produces a uniform coverage of the complete environment.
For example, consider a simple community containing two organisms (A and B), one which is twice as abundant as the other.
This community, thus, comprises twice as much of A's DNA than that of B.
Assume, for simplicity, that the community contains exactly three chromosomes (two of A and one of B).
A random sequencing process would sample each of these equally, and an ideal metagenomic assembler would produce two contigs, one covered twice as deep as the other.
%the case where we have three bacteria in an environment (1 copy of strain \emph{A}, 2 copies of strain \emph{B}).
%Now, lets assume we performed enough sequencing to produce a 1x coverage of each individual bacterium.
%An assembler would only output both bacteria strains, despite the difference in coverage.

In essence, we view the collection of individual genomes and their relative abundances as a single \emph{metagenome} where each genome is duplicated based on their abundance (\figurename \ref{fig:metagenome}).
%Thus, we now assume we have uniform coverage across the entire metagenome.
This setting is similar to that of repeats in single genome assembly, where a repetitive element can now include an entire genome.
Like in the case of single genomes, the assembler that correctly estimates these repeat counts maximizes the LAP score.
In other words, in order to accurately evaluate the metagenomic assemblies using our LAP framework, the abundance (or copy number) of each contig is needed.
%The assembler that figures out the correct copy number of repeats (in this case, contigs) will have the highest LAP score.
As most metagenomic assemblers do not report this information,
here we use the average coverage of the contig (provided by the MetAMOS pipeline) to represent the copy number.

%In order to accurately calculate the LAP of a metagenomic assembly, in addition to the actual assemblies, we need the abundances.
In the error-free model, we compute the probability of a read, $p_r$, given the assembled sequence and abundance as:
%
\begin{equation}
  \label{meta_read_probability}
  p_r = \frac{\sum_{c \in \text{Contigs}}\text{abun}(c)*n_{rc}}{2\hat{L}}
\end{equation}
%
\begin{equation}
  \label{meta_read_length}
  \hat{L} = \sum_{c \in \text{Contigs}}\text{abun}(c)*L_{c}
\end{equation}
%
where $\text{abun}(c)$ is the abundance of contig c, $n_{rc}$ is the number of times read $r$ occurs in contig $c$, and $\hat{L}$ is the adjusted total assembly length.  In the case where the abundance of each contig is 1, calculating $p_r$ is identical to the original LAP (single genome) formulation.  A similar modification can be done to handle sequencing errors outlined in \cite{LAP}.

Our prior work has shown we can approximate the probabilities using fast and memory efficient search alignment programs (e.g., Bowtie2~\cite{langmead2012fast}) when it is impractical to calculate the exact probabilities for large read sets.
We can apply the metagenomics modification above to the alignment tool-based method:
%
\begin{equation}
\label{}
p_{r} = \frac{\sum_{j \in S_r} \text{abun}(j_{\text{contig}})*p_{r,j_{\text{subs}}}}{2\hat{L}}
\end{equation}
where $S_r$ is the set of alignments in the SAM file for the read $r$ and the probability of alignment, $p_{r,j_{\text{subs}}}$, is approximated by $\epsilon^{subs}(1 - \epsilon)^{l - subs}$ where $\epsilon$ is the probability of an error (a mismatch, an insertion, or a deletion).

%allow the assembler to tell us what copy number it expects for each contig - that is the average coverage for the contig (provided in our analysis by the MetAMOS pipeline).
%Our LAP framework uses the average per basepair coverage of assembled contigs provided by MetAMOS.

An important factor in any likelihood-based assembly evaluation framework is the handling of reads that do not align well to the given assembly.
In practice, unalignable reads are often the result of sequencing errors and contaminants.
If these reads are given a probability close to 0, then the best assembler would be the one that incorporates the most reads.
In our original LAP framework, a read that does not align well does not decrease the overall assembly probability more than the probability of an assembly that contains the appended read as an independent contig.
This does not change when we handle metagenomic data, since the average coverage of the ``new'' contig is one.

%However, commonly used metagenomic abundance tools are often targeted at taxonomic classification~\cite{segata2012metagenomic,brady2009phymm,liu2010metaphyler,huson2007megan}.
%One strategy to estimate contig abundances is to first perform taxonomic classification on the reads and then align the contigs with the taxonomic database, transitively applying the taxonomic abundances to the contigs.
%In lieu of taxonomic abundances, our LAP framework uses the average per basepair coverage of assembled contigs provided by MetAMOS.
%We use modified version of Sailfish\cite{sailfish}, a tool to quickly calculate transcript abundances using RNA-seq data, to estimate contig abundances.
%One of the underlying assumptions of Sailfish is that the coverage

% \begin{landscape}
% \renewcommand{\baselinestretch}{1}
% \small\normalsize
%
% \renewcommand{\baselinestretch}{2}
% \small\normalsize
% \end{landscape}

\begin{landscape}
\renewcommand{\baselinestretch}{1}
\small\normalsize

\begin{table}[tbp]
\renewcommand{\arraystretch}{1.2}
\centering
\scriptsize
\begin{tabular}{{l}{l}{c}{c}{c}{c}{c}{c}{c}{c}{c}{c}{c}}
\hline
\bfseries Dataset & \bfseries Assembler & \bfseries LAP & \bfseries \#ctgs &  \bfseries Good ctgs & \bfseries Total aln & \bfseries Slt & \bfseries Hvy & \bfseries Ch & \bfseries Size @ 10 Mbp &\bfseries Max ctg size & \bfseries Err per Mbp & \bfseries Aligned reads \\
\hline\hline

mockE & SOAPdenovo & \textbf{-27.031} & 63107 & \textbf{99.3\%} & \textbf{51} & \textbf{166} & 131          & \textbf{1} & 28,208          & 249,819          & \textbf{5.8}  & \textbf{85.75\%} \\
mockE & Velvet     & -28.537          & 12,830 & 96.2\%          & 41          & 256          & \textbf{100} & 2          & 42,269          & 179,673          & 8.7  & 83.30\%        \\
mockE & MetaVelvet & -27.102          & 22,772 & 96.8\%          & 49          & 462          & 156          & 4          & \textbf{62,138} & \textbf{367,458} & 12.7    & 85.65\%      \\
mockE & Meta-IDBA  & -31.166          & 22,032 & 95.4\%          & 47          & 362          & 151          & 3          & 26,141          & 249,069          & 11  & 81.81\%\\
\\
mockS & SOAPdenovo & -60.161          & 44,928 & \textbf{98.8\%} & \textbf{28} & 135          & 98           & \textbf{0} & 5,672           & 186,064          & \textbf{8.3} & 69.78\%  \\
mockS & Velvet     & -60.711          & 21,050 & 95.8\%          & \textbf{28} & 485          & 115          & 1          & 6,060           & 119,120          & 21.5          & 67.26\% \\
mockS & MetaVelvet & -60.442          & 20,551 & 95.3\%          & \textbf{28} & 517          & 143          & 3          & 6,685           & \textbf{217,330} & 20.1         & 67.72\% \\
mockS & Meta-IDBA  & \textbf{-58.851} & 4,559  & 92.5\%          & 18          & \textbf{101} & \textbf{83}  & \textbf{0} & \textbf{13,150} & 119,604          & 10.2  & \textbf{70.67\%}\\
\\
Tongue  & SOAPdenovo & \textbf{-13.844} & 35,230 & \textbf{89.10\%} & \textbf{11} & 1,138 & 2,618 & \textbf{0} & 11,359 & \textbf{238,051} & \textbf{341.5} & \textbf{88.14\%}\\
Tongue  & Meta-IDBA & -21.368 & 25,698 & 88.70\% & 7 & \textbf{710} & \textbf{2,087} & \textbf{0} & 4,215 & 59,188 & 399.6 & 58.89\% \\

\hline
\end{tabular}
\caption[Comparison of assembly statistics for HMP mock Even and mock Staggered datasets]{Comparison of assembly statistics for HMP mock Even and mock Staggered datasets.Numbers in bold represent the best value for the specific dataset.}
\label{tab:hmp}
\end{table}

\renewcommand{\baselinestretch}{2}
\small\normalsize
\end{landscape}

\subsection{Integration into MetAMOS}

In addition to being a standalone framework, the software implementing our metagenomic LAP approach comes packaged with the MetAMOS pipeline.
This allows users the option to run MetAMOS with different assemblers and have our framework automatically select the assembly with the highest LAP score without any prior knowledge from the user.
The first step of the MetAMOS pipeline is to \verb!Preprocess! the reads, optionally filtering out low quality reads.
Those reads are used by the next step \verb!Assemble!.
Users specify the desired assembler using the \verb!-a! parameter of \verb!runPipeline!.
We modified MetAMOS so users can now specify multiple assemblers (comma-separated) after the \verb!-a! parameter, and \verb!runPipeline! will run all assemblers and select the assembly yielding the highest LAP score to be used in downstream analyses.
%The software implementing our approach is made available, open-source and free of charge, at: \url{http://assembly-eval.sourceforge.net/} and with the MetAMOS package: \url{https://github.com/treangen/MetAMOS}.

\section{Results}
\subsection{Likelihood score maximized using correct abundances}
A key property of our framework is that the correct copy numbers (abundances) and assemblies maximizes our LAP score.
To illustrate this property, we simulated two metagenomic communities and calculated the LAP of the reference genomes with a combination of abundances.
The first simulated community consisted of \emph{Bacillus cereus} and \emph{Acinetobacter baumannii} at a ratio of 1:4.
We generated 200bp reads at 20x coverage of the metagenome (20x of \emph{B. cereus} and 80x of \emph{A baumannii}).
We calculated the LAP scores of the error-free reference genomes for all combinations of abundances (ranging from 1 copy to 8 copies) for each bacteria.
%We calculated the LAP score on error-free reference assemblies with all combinations of abundances from 1 copy to 8 copies for each bacteria.
The second simulated community consisted of \emph{Bacillus cereus} and \emph{Actinomyces odontolyticus} at a ratio of 4:7.
We generated 200bp reads at 20x coverage of the metagenome (80x of \emph{B. cereus} and 140x of \emph{A odontolyticus}).
We calculated the LAP scores of the error-free reference genomes for all combinations of abundances (ranging from 1 copy to 13 copies) for each bacteria.

%\begin{figure}[!t]
%\centering
%\includegraphics[width=3in]{ref_abun_2}
% where an .eps filename suffix will be assumed under latex,
% and a .pdf suffix will be assumed for pdflatex; or what has been declared
% via \DeclareGraphicsExtensions.
%\caption{LAP scores for simulated B. cereus (4x, 5.2MB) and A. odontolyticus (7x, 2.4MB)}
%\label{fig:ref_abun_2}
%\end{figure}

We expect the highest LAP scores for the assemblies that contain the correct abundance ratios (1:4 or 2:8 in the first community, and 4:7 in the second community).
%1:4 , 2:8 boxes to be highest in figure A, and 4:7, 8:14, etc. in panel 2.  Y
As seen in Figs. \ref{fig:ref_abun_1} and \ref{fig:ref_abun_2}, our LAP score is able to accurately reflect the varying organism abundance ratios present in the sample.
The LAP score increases as the estimates approach the true abundance ratios, with the true ratio yielding the highest LAP scores in both communities.

\begin{figure}[htb!]
\begin{center}
\includegraphics[width=.8\textwidth]{errors}
\end{center}
\renewcommand{\baselinestretch}{1}
\small\normalsize
\begin{quote}
\caption[Synthetic errors in simulated \emph{E. coli} and \emph{B. cereus} (1 copy, 5.2Mbp) community]{Synthetic errors in simulated \emph{E. coli} (5 copies, 4.9Mbp) \emph{B. cereus} (1 copy, 5.2Mbp) community.}
\label{fig:errors}
\end{quote}
\end{figure}
\renewcommand{\baselinestretch}{2}
\small\normalsize

\subsection{Impact of errors on synthetic metagenomes}

One of the often overlooked aspects of metagenomic assembly evaluation is the weighing of errors that occur in contigs with different abundances.
%In single genome assembly, errors are often not weighted since the genome has uniform coverage.
%When detecting errors with the aid of a reference, all substitution errors occurring within a genome are weighted the same (ignoring additional information, such as whether it occurs in a genic or regulatory element).
%When organism abundances are uniform, the weight of an assembly error is also uniform.
In metagenomic samples the relative organism abundances can vary by orders of magnitudes.
A typical reference-based evaluation would equally weight the errors irrespective of the abundance of the erroneous contigs.
The proposed metagenomic LAP score, however, automatically handles this situation and appropriately weighs errors according to genome abundance.
%Thus, errors in highly abundant organisms will have more of an impact on the LAP score than errors in organism of lower abundance.
To illustrate this, we simulated a small metagenomic community consisting of \emph{Escherichia coli} and \emph{Bacillus cereus} at a 5:1 ratio.
We introduced an increasing number of common assembly errors (single-base substitutions, insertions, deletions, and inversions) into the two organisms assemblies and observed the resulting LAP score.

As shown in \figurename \ref{fig:errors}, the higher the number of synthetic errors, the lower the LAP score.
Insertions/deletions were more deleterious to the LAP score than substitutions, since in addition to causing a mismatch, an insertion/deletion changes the overall genome size.
Although inversions did not change the overall genome size (and would therefore not be detected by simplistic measures such as N50), these errors had the greatest impact on the LAP score because they prevented the alignment of reads across the boundaries of the inversions.
%With substitutions and indels, reads were still able to align across the error and only incurred the additional mismatch penalty.
%Conversely, read alignment tools will fail to detect read alignments that span the borders of the inversions.

%Whereas substitutions, insertions, and deletions still allowed reads to align albeit with a lower probability, reads that inversions are essentially treated as complete

As expected, errors introduced into the more abundant organism, \emph{E. coli}, had a greater affect on the LAP score than those inserted into \emph{B. cereus}.
Our LAP score was able to accurately weigh the errors by the abundance of each organism.

\subsection{Likelihood scores correlate with reference-based metrics}
%c|c|c|c|c|c

With real metagenomic samples, it is difficult to make evaluations given the lack of high quality references.
Using purely simulated data has the issue of not accurately capturing the error and bias introduced by sequencing technology.
Thus, to evaluate our LAP score, we use two `mock' communities (Even and Staggered) provided by the Human Microbiome Project (HMP) consortium\cite{mitreva2012structure,methe2012framework}.
These communities were created using specific DNA sequences from organisms with known reference genomes (consisting of over 20 bacterial genomes and a few eukaryotes) and abundances.
The mock Even community consisted of 100,000 16S copies per organism per aliquot, while the mock Staggered community consisted of 1,000 to 1,000,000 16S copies per organism per aliquot.
Data used from the HMP mock communities are available at \url{http://www.ncbi.nlm.nih.gov/bioproject/48475}.
We calculated the LAP score on assemblies produced by MetAMOS~\cite{treangen2013metamos} using several assemblers: SOAPdenovo~\cite{SOAPdenovo}, Metavelvet~\cite{namiki2012metavelvet}, Velvet~\cite{Velvet}, and Meta-IDBA~\cite{peng2011meta}.
The additional \emph{de novo} and reference-based metrics for the assemblies were taken from MetAMOS~\cite{treangen2013metamos}.  These metrics include:
   \begin{itemize}
   \item number contigs (\# ctgs) -- total number of contigs/scaffolds in the assembly
   \item good contigs (Good Ctgs) -- fraction of contigs that mapped without errors to reference genomes
   \item total aligned (Total aln) -- total amount of sequence (in Mbp) that can be aligned to the reference genomes
   \item slight mis-assemblies (Slt) -- alignments that cover 80\% or more of the aligned contig in a single match (Slt)
   \item heavy misassemblies (Hvy) -- alignments that cover less than 80\% of the aligned contig in a single match or have two or more matches to a single reference
   \item chimeras (Ch) -- contigs with matches to two distinct reference genomes
   \item size at 10 megabases (Size @ 10 Mbp) - the size of the largest contig $c$ such that the sum of all contigs larger than $c$ is more than 10 Mbp (similar to the commonly used N50 size)
   \item max contig size (Max ctg size) -- size of the largest contig in the assembly
   \item errors per megabase (Err per Mbp) -- average number of errors per Mbp in the assembly
   \end{itemize}

Generally, the \emph{de novo} LAP scores agree with the referenced-based metrics (Table \ref{tab:hmp}).
In the mock Even dataset, SOAPdenovo has the greatest LAP score, the highest fraction of contigs that can align to a reference genome without error, total amount of sequence that can be aligned to a reference genome, while also having the lowest amount of misassemblies (including chimeric) and errors per Mbp.
It is important to note that if user selected an assembly based on the best contiguity at 10Mbp, they would select the MetaVelvet assembly, which contains double the error rate per Mbp as the SOAPdenovo assembly while aligning 2Mbp less to the references.

\begin{figure}[tb!]
\begin{center}
\includegraphics[width=.8\textwidth]{coverage}
\end{center}
\renewcommand{\baselinestretch}{1}
\small\normalsize
\begin{quote}
\caption[Frequency of contig abundances for assemblies of the HMP mock Staggered dataset]{Frequency of contig abundances for assemblies of the HMP mock Staggered dataset.}
\label{fig:coverage}
\end{quote}
\end{figure}
\renewcommand{\baselinestretch}{2}
\small\normalsize

Since the abundances of each organism in the mock Even dataset are fairly similar, the mock Staggered abundance distribution creates a more realistic scenario encountered in metagenomic environments.
Here, the Meta-IDBA assembly has the greatest LAP score, but aligns roughly a third less sequences to the reference genomes than SOAPdenovo.
The Meta-IDBA assembly contains approximately a tenth of the amount of contigs (4,559 vs. 44,928) as SOAPdenovo.
The SOAPdenovo assembly contains a greater number of contigs at a very low abundance (\figurename \ref{fig:coverage}).
On large contigs Meta-IDBA performs better than SOAPdenovo and has a lower error rate (see \figurename 4 in \cite{treangen2013metamos}).
However, Meta-IDBA assembles a smaller fraction of the low-abundance genomes than SOAPdenovo, leading fewer sequences to align.
The LAP score penalizes misassemblies within abundant contigs in the SOAPdenovo results.

%Given how the mock Staggered community was constructed (each ratio of 16S copies per aliquot occurred roughly the same number of times), the frequency of contig abundances should be approximately uniform for the community.
%The SOAPdenovo assembly was able to align more reads, but the reads were localized in these low coverage regions.
%The distribution of contig abundances in the Meta-IDBA assembly was closer to the expected distribution than the SOAPdenovo assembly.

%The Meta-IDBA assembly a far more uniform frequency of contig abundances compared to the SOAPdenovo assembly.

Next, we applied our framework on real data where we did not know the actual genomes comprising the sample (HMP tongue dorsum female sample, SRS077736) (Table \ref{tab:hmp}).
Although we do not know for certain which organisms are present in the sample, the HMP identified a reference genome set with high similarity to the sequences within the sample (HMP Shotgun Community profiling SRS077736).
The reference-based error metrics only consider chimeric errors due to the possibility of structural difference between an organism and its version in the reference database.
We calculated the LAP of the assemblies using a single library consisting of 42,013,917 reads.
The SOAPdenovo assembly had a far greater LAP score than the Meta-IDBA assembly.
The higher score is due to the SOAPdenovo assembly recruiting more reads (88.14\% to 58.89\%) in relation to its genome size (46Mbp to 37Mbp) than the Meta-IDBA assembly.
Furthermore, the MetAMOS metrics show that the SOAPdenovo assembly contained approximately 60 less errors per Mbp than the Meta-IDBA assembly.

%These datasets have the advantage over simulated data because it captures the error and bias introduced from the sequence technology.

\renewcommand{\baselinestretch}{1}
\small\normalsize

\begin{table}[tb!]
\label{tab:metamos_lap}
\centering
\begin{tabular}{{l}{c}{c}{c}{c}}
\hline
\bfseries Assembler & \bfseries Contigs & \bfseries LAP & \bfseries N50 (Kbp) & \bfseries  Errors \\
\hline \hline
newbler  & \bf{1} & \bf{-13.064} & \bf{156} & 1 \\
SOAPdenovo & 23 & -14.238 & 9 & \bf{3} \\
Velvet & 3 & -13.157 & 92 & \bf{0} \\
MetaVelvet & 3 & -13.157 & 92 & \bf{0} \\
\hline
\end{tabular}
\caption[Self-tuning MetAMOS using \emph{C. ruddii} test dataset]{Self-tuning MetAMOS using \emph{C. ruddii} test dataset.}
\end{table}

\renewcommand{\baselinestretch}{2}
\small\normalsize

\subsection{Tuning assembly parameters for MetAMOS}
Assemblathon1~\cite{earl2011assemblathon} has shown that assembly experts can often get drastically different assemblies using the same assemblers, highlighting the difficulty of choosing the \emph{right} parameters for a given assembler.
Our metagenomic LAP framework comes packaged with the MetAMOS pipeline, allowing users the option to run MetAMOS with different assemblers and automatically select the assembly with the highest LAP score.
This step occurs without any prior knowledge from the user.

We showcase the ease of use of the automated assembler selection within MetAMOS using the \emph{Carsonella ruddii} (156Kbp) dataset packaged with MetAMOS (Table ~\ref{tab:metamos_lap}).
Errors were found using DNADIFF~\cite{phillippy2008genome} and MUMmer~\cite{delcher2003using}.
The newbler assembly produced one contig containing the complete \emph{C. ruddii} genome.
The SOAPdenovo assembly produced a severely fragmented assembly with the most number of errors.
The MetaVelvet and Velvet produced identical assemblies, containing 3 contigs of sizes 92Kbp, 65Kbp, and 1.7Kbp, but contained an additional 158bp compared to the \emph{C. ruddii} genome.
Upon closer inspection, there were overlaps between the contigs ranging from 38bp to 73bp.
This is not surprising given MetaVelvet's and Velvet's de bruijn graph-based approach could not resolve repetitive regions between the contigs.
Newbler, on the other hand, contained only a single insertion error.
The LAP score of the Newbler assembly was greater due to more reads being able to align across the regions that were broken apart in the MetaVelvet and Velvet assemblies.
Additionally, the Newbler assembly did not contain the duplicated sequence found in the other assemblies.
MetAMOS was able to select the most likely assembly without requiring any additional input from the user.

\section{Discussion}

In this paper, we have proposed an extension to our LAP framework to perform \emph{de novo} comparisons of metagenomic assemblies.
Unlike traditional \emph{de novo} metrics used for measuring assembly quality, our extended LAP score correlates well with reference-based measures of metagenomic assembly quality.
However, in this study, we have realized that there is a lack of reference-based metrics when evaluating metagenomic assemblies.
Misjoins betweens organisms may be more deleterious than misjoins within a single genome.
Furthermore, current reference-based metrics do not take into account the relative abundances of the organisms when evaluating metagenomic assemblies.
The metrics provided by MetAMOS do not factor in the contig abundances when examining assembly errors.
This made it difficult to compare our LAP score to their reference-based metrics because, intuitively, an error in a highly abundant organism should be \emph{worse} than an error in a rare, low coverage organism.
Our LAP score implicitly weighs the errors in abundant contigs more than those in lesser abundant contigs.
In our results, we have proposed one such reference-based metric that scales the errors by the relative abundance of the contig it occurs within.

It is important to note that we have only focused on the complete reconstruction of the metagenome from the set of reads.
Assembly algorithms are designed with specific biological applications in mind, such as, the conservative reconstruction of the genic regions.
Studies focusing on the genic regions may tolerate large-scale rearrangements as long as the genic regions were correctly assembled.
Conversely, other studies may want to focus on the reconstruction and detection of rare pathogenic bacteria in an environment.
These application specific assembly algorithms all attempt to optimize their formulation of the assembly problem.

Our metagenomic LAP extension relies heavily on the idea that the sequencing process of the metagenome is roughly uniform, and that the reads are independently sampled from the genome.
Biases exist in all steps of the sequencing process, from the extraction of DNA from organisms with different cell membranes/walls~\cite{carrigg2007dna,krsek1999comparison} to the sequencing protocol used~\cite{morgan2010metagenomic,temperton2009bias}.
In the future, we would like to implement a more specific model that better captures the sequencing process.

%De novo metric, more misleading in assemblies -> chimera

%We would like to stress that \emph{de novo} measures of assembly quality, such as ours, are critically needed by researchers attempting to assemble of yet unknown genomes.

Results from GAGE~\cite{salzberg2011gage} and Assemblathon~\cite{earl2011assemblathon,bradnam2013assemblathon} have shown that the specific characteristics of the data being assembled has a great impact on the performance of the assembler.
This problem is magnified in metagenomic assembly.
By integrating our LAP framework in MetAMOS, we have allowed researchers to accurately and effortlessly run and evaluate assemblies without any prior knowledge on evaluating assembly quality.

In our framework, we use the average coverage of the contig (provided by MetAMOS) to estimate abundance.
There are issues with this measure as it is possible that mis-assembled repeats within a contig will affect our estimate of depth of coverage and could impact our underlying statistics.
A better approach is to use something more robust than the mean coverage, such as the median coverage, to avoid being influenced by such regions.
While the user can supply the median coverage to our standalone framework, future work includes building this feature into MetAMOS.
Another approach involves breaking contigs at regions of differing coverage (using tools such as AMOSvalidate\cite{phillippy2008genome}), so there will be less deviation in the average coverage within the contig.

It should be noted that in some cases it may not be tractable to run the complete collection of assemblers with MetAMOS.
In such cases, we should first employ heuristics (such as \cite{chikhi2013informed}) to aid in selecting potential assemblers (and parameters) to run.
For the assembler selection process, we can use the LAP framework's sampling procedure in combination with calculating read probabilities in parallel to reduce runtime.

Our goal was to provide a global measure of how good a metagenomic assembly may be, not to detect assembly errors.
Other likelihood-based frameworks, such as ALE, use frequencies of certain sequences to aid in detection of possible chimeric contigs.
We are able to apply similar modifications to our LAP framework to find regions of possible misassembly.
Finally, we plan to extend our framework to give a more detailed breakdown of the LAP scores of segments assembled using the same subset of reads across different assemblies.
The goal would be to take high-scoring assembled segments from individual assemblies to recreate an assembly with overall greater likelihood.
This approach will be of great benefit to the field of metagenomic assembly since assemblers are often designed with different constraints and goals in mind, e.g., low memory footprint, assembling high/low coverage organisms, or tolerating population polymorphisms.
For example, on the mock Staggered dataset, Meta-IDBA best assembled the most abundant genomes while SOAPdenovo had a better representation of the low abundance organisms.
Providing a systematic way of combining assembler approaches using our LAP score will produce better assemblies for downstream analyses.

% An example of a floating figure using the graphicx package.
% Note that \label must occur AFTER (or within) \caption.
% For figures, \caption should occur after the \includegraphics.
% Note that IEEEtran v1.7 and later has special internal code that
% is designed to preserve the operation of \label within \caption
% even when the captionsoff option is in effect. However, because
% of issues like this, it may be the safest practice to put all your
% \label just after \caption rather than within \caption{}.
%
% Reminder: the "draftcls" or "draftclsnofoot", not "draft", class
% option should be used if it is desired that the figures are to be
% displayed while in draft mode.
%
%\begin{figure}[!t]
%\centering
%\includegraphics[width=2.5in]{myfigure}
% where an .eps filename suffix will be assumed under latex,
% and a .pdf suffix will be assumed for pdflatex; or what has been declared
% via \DeclareGraphicsExtensions.
%\caption{Simulation Results}
%\label{fig_sim}
%\end{figure}

% Note that IEEE typically puts floats only at the top, even when this
% results in a large percentage of a column being occupied by floats.

% An example of a double column floating figure using two subfigures.
% (The subfig.sty package must be loaded for this to work.)
% The subfigure \label commands are set within each subfloat command, the
% \label for the overall figure must come after \caption.
% \hfil must be used as a separator to get equal spacing.
% The subfigure.sty package works much the same way, except \subfigure is
% used instead of \subfloat.
%
%\begin{figure*}[!t]
%\centerline{\subfloat[Case I]\includegraphics[width=2.5in]{subfigcase1}%
%\label{fig_first_case}}
%\hfil
%\subfloat[Case II]{\includegraphics[width=2.5in]{subfigcase2}%
%\label{fig_second_case}}}
%\caption{Simulation results}
%\label{fig_sim}
%\end{figure*}
%
% Note that often IEEE papers with subfigures do not employ subfigure
% captions (using the optional argument to \subfloat), but instead will
% reference/describe all of them (a), (b), etc., within the main caption.

% An example of a floating table. Note that, for IEEE style tables, the
% \caption command should come BEFORE the table. Table text will default to
% \footnotesize as IEEE normally uses this smaller font for tables.
% The \label must come after \caption as always.
%
%\begin{table}[!t]
%% increase table row spacing, adjust to taste
%\renewcommand{\arraystretch}{1.3}
% if using array.sty, it might be a good idea to tweak the value of
% \extrarowheight as needed to properly center the text within the cells
%\caption{An Example of a Table}
%\label{table_example}
%\centering
%% Some packages, such as MDW tools, offer better commands for making tables
%% than the plain LaTeX2e tabular which is used here.
%\begin{tabular}{|c||c|}
%\hline
%One & Two\\
%\hline
%Three & Four\\
%\hline
%\end{tabular}
%\end{table}

% Note that IEEE does not put floats in the very first column - or typically
% anywhere on the first page for that matter. Also, in-text middle ("here")
% positioning is not used. Most IEEE journals/conferences use top floats
% exclusively. Note that, LaTeX2e, unlike IEEE journals/conferences, places
% footnotes above bottom floats. This can be corrected via the \fnbelowfloat
% command of the stfloats package.

\section{Conclusion}
In this paper we have described an extension to our \emph{de novo} assembly evaluation framework (LAP) for comparing metagenomic assemblies.
We showed that the true metagenome and correct relative abundances maximizes our extended LAP score.
Furthermore, we have integrated our framework into the metagenomic assembly pipeline MetAMOS, showing that any user is able to reproduce quality evaluations of metagenomic assemblies with relative ease.


\section{Using PySCF to generate integrals for AFQMC}
\label{sec:pyscf}
These notes describe useful knowledge and tips in the use of pyscf.
The main purpose of pyscf in the group is in the generation of 2-electron integrals for correlated calculations in solids.

For a detailed description of pyscf, please see the documentation by the authors.

Examples of how to generate integrals:
\begin{itemize}
\item Run a standard pyscf calculation, e.g., an HF or DFT calculation. Make sure you preserve the chkfile and make sure you store the core hamiltonian on the chkfile. An example of how to do this for a single k-point calculation follows.
\begin{lstlisting}[style=Python,caption=The following is an example PySCF input file for single k-point calculations.]
import numpy
import h5py
from mpi4py import MPI
from pyscf.pbc import gto, scf, dft, cc

cell = gto.Cell()
cell.a = '''
3.5668 0 0
0 3.5668 0
0 0 3.5668'''
cell.atom = '''C 0. 0. 0.
C 0.8917 0.8917 0.8917
C 1.7834 1.7834 0.
C 2.6751 2.6751 0.8917
C 1.7834 0. 1.7834
C 2.6751 0.8917 2.6751
C 0. 1.7834 1.7834
C 0.8917 2.6751 2.6751'''
cell.basis = 'gth-szv'
cell.pseudo = 'gth-pade'
cell.gs = [10]*3 # 10 grids on positive x direction, => 21^3 grids in total
cell.verbose = 4
cell.build()

kpt = cell.get_abs_kpts([0.25, 0.25, 0.25])

mf = scf.RHF(cell,kpt,exxdiv=0)
mf.chkfile = "scf.dump"                         # store checkpoint file in scf.dump
ehf = mf.kernel()
print("HF energy (per unit cell) = %.17g" % ehf)

hcore = mf.get_hcore(kpt=kpt)            # obtain and store core hamiltonian
with h5py.File(mf.chkfile) as fh5:
  fh5['scf/hcore'] = hcore
\end{lstlisting}

\item {For a calculation with k-points:
Run a standard pyscf calculation, e.g., an HF or DFT calculation. Make sure you preserve the chkfile and make sure you store the core hamiltonian on the chkfile. An example of how to do this for a single k-point calculation follows.}

\begin{lstlisting}[style=Python,caption=The following is an example PySCF input file for calculations with k-points.]
import numpy
import h5py
from mpi4py import MPI
from pyscf.pbc import gto, scf, dft, cc

cell = gto.Cell()
cell.a = '''
3.5668 0 0
0 3.5668 0
0 0 3.5668'''
cell.atom = '''C 0. 0. 0.
C 0.8917 0.8917 0.8917
C 1.7834 1.7834 0.
C 2.6751 2.6751 0.8917
C 1.7834 0. 1.7834
C 2.6751 0.8917 2.6751
C 0. 1.7834 1.7834
C 0.8917 2.6751 2.6751'''
cell.basis = 'gth-szv'
cell.pseudo = 'gth-pade'
cell.gs = [10]*3 # 10 grids on positive x direction, => 21^3 grids in total
cell.verbose = 4
cell.build()

nk = [2,2,2]
kpts = cell.make_kpts(nk)

mf = scf.KRHF(cell,kpts,exxdiv=0)
mf.chkfile = "scf.dump"                         # store checkpoint file in scf.dump
ehf = mf.kernel()
print("HF energy (per unit cell) = %.17g" % ehf)

hcore = mf.get_hcore(kpts=kpts)          # obtain and store core hamiltonian
fock = (hcore + mf.get_veff(kpts=kpts))  # store fock matrix (required with orthoAO=True)
with h5py.File(mf.chkfile) as fh5:
  fh5['scf/hcore'] = hcore
  fh5['scf/fock'] = fock
\end{lstlisting}
\end{itemize}

Once the checkpoint file has been created, it is possible to generate the integral file. The recommended approach follows:

\begin{lstlisting}[style=Python,caption=The following is an example input file for calculating the integrals.]
from mpi4py import MPI
from qmctools import integrals_from_chkfile

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
nproc = comm.Get_size()

integrals_from_chkfile.eri_to_h5("fcidump", rank, nproc, "scf.dump")

comm.Barrier()

if rank==0:
    integrals_from_chkfile.combine_eri_h5("fcidump", nproc)
\end{lstlisting}

It is also possible to generated the Cholesky decomposed integrals in pyscf directly. This is typically faster and more appropriate. (The phfmol code expects integrals directly, but these can be generated from QMCPACK with the Cholesky decomposed ones.) To generate them for a run with k-points, for example, the recommended approach follows: (NOTE: for some reason, mpi4py must be imported first!!!)

For calculations with kpoints (those generated with K...), use integrals\_from\_chkfile.eri\_to\_h5\_kpts(...).
Additional arguments to eri\_to\_h5 are:

\begin{itemize}
\item \textbf{Cholesky}. Determines whether 2-electron integrals or their Cholesky factorization is calculated.
  Default: False
\item \textbf{orthoAO}. If True, generates the integrals in the orthogonalized AO basis. If False, generates the integrals in the MO basis found on the checkpoint file. For UHF calculations, only orthoAO=True is allowed. If set to False, the fock matrix must be stored in the scf dump file.
  Default: False
\item \textbf{LINDEP\_CUTOFF}.  Cutoff used to define linearly dependent basis functions.
  Default: 1e-9
\item \textbf{gtol}. Cutoff applied during writing for 2-electron integrals. If Cholesky=True, then this is the cutoff used in the iterative Cholesky factorization. In this case, the resulting factorized Hamiltonian will have a residual error smaller than the requested cutoff.
  Default: 1e-6
\item \textbf{wfnName}. Name of the file with the wavefunction. This is generated only  when orthoAO=True.
  Default: ``wfn.dat''
\item \textbf{wfnPHF}. Name of file with initial guess for phfmol code.
  Default: None (no file is generated)
\item \textbf{MaxIntgs}. Maximum number of integrals  (or terms in Cholesky matrix) per block in the hdf5 file. This controls the size of the hdf5 datasets.
  Default: 2000000
\item \textbf{maxvecs}. Represents the maximum number of Cholesky vectors allowed. The actual maximum number of Cholesky vectors in the calculation is set to maxvecsnmo. So a value of 10 leads to an actual cutoff in the number of vectors of 10*nmo, where nmo is the total number of MOs in the calculation. The calculation will stop at the requested number of vectors even if the tolerance is not reached.
  Default: 20
\end{itemize}


\chapter{Introduction}
\Atlas is an ECMWF software library for parallel flexible
data-structures supporting structured/unstructured grids,
structured/unstructured meshes, various function spaces
and utilities.
The main aim of \Atlas is to investigate and develop more
scalable dynamical core options for numerical weather prediction
(NWP). \Atlas is also intended to create modern interpolation
and product generation software.

\Atlas is predominantly written in C++, with the main features
available to Fortran through an F2003 interface. To be used
effectively, it requires some knowledge of Unix (such as Linux).
It is known to run on a number of systems, some of which are
directly supported by ECMWF.

\Atlas includes the following macro data objects.
%
\begin{itemize}
\item Grid: a list of coordinates (i.e. points) without connectivity rules;
\item Mesh: a collection of elements linked by precise connectivity rules;
\item Field: a physical quantity such as wind velocity or pressure;
\item FieldSet: a collection of Fields;
\item FunctionSpace: a given spatial discretization space (e.g. spectral,
finite element, etc.).
\end{itemize}
%
From these objects it is possible to construct new algorithms
to be tested within the context of numerical weather prediction
(NWP), to generate and manipulate grids for production
cases, etc. The overall structure of the library is depicted
in \fig{fig:intro-schematics}.

From this figure, we note that there is the additional object
called Metadata and related to the Field object. Metadata
contains a description of a given Field (e.g. units, etc.).
We also note that the Mesh object is formed by the Nodes
and HybridElements objects, with the last being composed
by Elements. These additional items represents the bricks
to ultimately build the mesh object.

The structure in Fig. \ref{fig:intro-schematics} will be
further explained in chapter \ref{chap:structure}.
%
\begin{figure}[htb!]
\centering
\includegraphics[width=0.95\textwidth]{imgs/schematics.pdf}
\caption{Schematics of the \Atlas library.}
\label{fig:intro-schematics}
\end{figure}
%


\documentclass{article}
    \usepackage{subcaption}
    \usepackage{amsmath, amssymb}
    \usepackage{graphicx, float}
    \usepackage[hidelinks]{hyperref}
    \usepackage[bottom]{footmisc}
    \usepackage[margin=.8in, tmargin=.8in]{geometry}
    \usepackage{esint} % for double line integrals
    \usepackage{xcolor} % for color macros
    \usepackage{tikz}
    \usepackage{algorithm, algpseudocode}
    \newcommand{\citep}[2][]{(\cite[#1]{#2})}

    \usetikzlibrary{arrows,decorations.markings}
    \usepackage[
        backend=biber,
        % style=alphabetic,
        citestyle=authoryear,
        sorting=ynt,
        bibencoding=utf8
    ]{biblatex}

    \bibliography{bibliography}

    \renewcommand{\baselinestretch}{1.2}
    \setlength\parindent{0pt}
    \setlength{\parskip}{\baselineskip}

    \newcommand{\eps}{\epsilon}
    \newcommand{\der}{\partial}
    \newcommand{\del}{\nabla}
    \newcommand{\bm}[1]{\mathbf{#1}}
    % \newcommand{\vf}[1]{\vec{\mathbf{#1}}}
    \newcommand{\vf}[1]{\mathbf{#1}}
    \newcommand{\norm}{\hat{\bm{n}}}
    \newcommand{\normal}{\mathcal{N}}
    \newcommand{\var}{\text{var}}
    \newcommand{\bx}{\vf{x}}
    \newcommand{\by}{\vf{y}}
    \newcommand{\bz}{\vf{z}}
    \newcommand{\bw}{\vf{w}}
    \newcommand{\bfu}{\vf{f}}
    \newcommand{\giv}{\ |\ }
    \newcommand{\bmu}{\pmb{\mu}}
    \newcommand{\gauss}{\mathcal{N}}
    \newcommand{\data}{\mathcal{D}}
    \newcommand{\model}{\mathcal{M}}
    \DeclareMathOperator*{\argmax}{arg\,max}
    \DeclareMathOperator*{\argmin}{arg\,min}

    \newcommand{\id}{\mathbb{I}}
    \newcommand{\de}{\text{d}}
    \newcommand{\tran}{\text{T}}
    \newcommand{\expect}{\mathbb{E}}
    \newcommand{\kl}{D_{KL}}
    \newcommand{\post}{P(\bw \giv \data)}

    \newcommand{\red}[1]{\textcolor{red}{#1}}
    \newcommand{\blue}[1]{\textcolor{blue}{#1}}

    \setlength\parindent{0pt}
    \captionsetup{justification=centering}

    \title{Reinforcement Learning}
    \date{\today}
    \author{Traiko Dinev \textless traiko.dinev@gmail.com\textgreater}

\begin{document}

\maketitle
\textit{NOTE: This partially follows Reinforcement Learning, a masters level course at the University of Edinburgh. Credit also goes to \url{https://github.com/FrancescoSaverioZuppichini/Reinforcement-Learning-Cheat-Sheet} for some great ideas. The result graphs were taken from a courseswork exercise I did for Machine Learning Practical (MLP).}

\textit{NOTE: Note this "summary" is NOT a reproduction of the course materials nor is it copied from the corresponding courses. It was entirely written and typeset from scratch.}

\textit{License: Creative Commons public license; See README.md of repository}

\section{Markov Decision Processes}
\begin{figure}[ht]
    \centering
    \begin{subfigure}{0.4\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/agent_environment.png}
        \label{fig:agent_environment}
        \caption{Agent-Environment Interaction (from Sutton \& Barto)}
    \end{subfigure}
    %
    \centering
    \begin{subfigure}{0.25\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/Markovkate_01.png}
        \label{fig:mdp}
        \caption{An example Markov Decision Process (from wikipedia)}
    \end{subfigure}
\end{figure}

The standard reinforcement learning problem is defined as follows \citep{sutton_reinforcement_1998}; An agent is placed in an environment. At each timestep $t$ the environment is in a state $S_t = s \in \mathcal{S}$; the agent takes an action $A_t = a \in \mathcal{A}$ and receives a real-valued reward $R_{t+1} = r \in \mathbb{R}$. $S_t, A_t$ and $R_t$ are random variables. After taking an action, the state changes probabilistically, as defined in the transition function for the environment $P(S_{t+1} = s' \giv S_t = s, A_t = a)$. Our goal is to learn a policy (in most cases deterministic) $\pi \colon \mathcal{S} \to \mathcal{A}$ that for each state maps to the action that the agent would take. We define the return as:

\begin{equation}
    G_t = \sum_{k=0}^\infty \gamma^k R_{t + k + 1}
\end{equation}

where $\gamma$ is the discount parameter. The agent's goal is to maximize the expected return $\mathbb{E}[G_t \giv S_0 = s_0]$. We define the state-action value function for a policy $\pi$ as the expected return for a state and action under that policy:

\begin{equation}
    q_\pi(S_t, A_t)\ =\ \mathbb{E}_\pi\ \big[G_{t}\ |\ S_t = s, A_t = a \big]
\end{equation}

We can define an optimal policy $\pi^*$ as the policy that for each state $s \in \mathcal{S}$ and action $a \in \mathcal{A}$ satisfies $q_{\pi^*}(s,a) = q_*(s,a) > q_\pi(s, a)$ for every other policy $\pi$. The Q-function for each optimal policy is the same. We define the Bellman optimal equation \citep{sutton_reinforcement_1998}:

\begin{equation}
    q_*(s_t, a_t)\ =\ \mathbb{E}_{\pi^*}\ \big[ R_{t + 1} + \gamma\ \text{max}_{a'} q_*(s_{t+1}, a')\ |\ S_t = s_t, A_t = a_t\big]
\end{equation}

We can also define a value function $V(s_t) = \mathbb{E}\ \big[G_t\ |\ S_t = s_t \big]$ and an optimal value function in a corresponding way as the optimal Q function. The value functions also satisfy a similar Bellman equation \citep{sutton_reinforcement_1998}

In the classical reinforcement learning scenario, the optimal policy can be learned by instead learning an optimal Q function (or value function) and then computing a greedy policy:

\begin{equation}
    \pi^*(a\ |\ s) = \text{argmax}_a\ Q^*(s, a)
\end{equation}

\subsection{Q-learning and SARSA}
\textbf{Q-learning} is one of the simplest algorithms \citep{sutton_learning_1988}. The Q values are stored in a table and continuous states are discretized into buckets. After each step, the Q value for the state-action pair is updated according to the following rule:

\begin{equation}
    \begin{split}
        q(S_t, \ A_t) &= (1\ -\ \alpha)\ q(S_t,\ A_t)\ +\\
            &\alpha\ (R_{t + 1} + \gamma\ \text{max}_a\ q(S_{t+1},\ A_{t + 1}))
    \end{split}
\end{equation}

Here $\alpha$ is the learning rate, starting at $1.0$, which is exponentially decrease (anneal) by some factor (e.g. $0.995$) after each episode. The max operator approximates the Bellman optimalitiy equation.

\textbf{SARSA} \citep{a._rummery_-line_1994} is similar to Q-learning but the max operator is replaced by the action that would be taken by the current policy. The update rule is as follows:

\begin{equation}
    \begin{split}
        q(S_t,\ A_t) &= (1\ -\ \alpha)\ q(S_t, A_t)\ + \\
            &\alpha\ (R_{t+1} + \gamma q(S_{t+1}, A_{t+1}))
    \end{split}
\end{equation}

where the learning rate $\alpha$ is similarly annealed. SARSA is an example of an on-policy method, where the Q updates are computed by sampling the next action from the current policy; thus $S_{t+1}, A_{t+1}$ are computed before the update is made.

\subsection{Deep Q Learning}
Deep Q Networks (DQN) were introduced by DeepMind in their seminal paper \citep{mnih_human-level_2015}. Instead of using tables to store and update the Q values, neural networks are used. This eliminates the problem that  tables grow exponentially large with the complexity of the state. At each step, the transitions $(s_t, a_t, r_{t+1}, s_{t+1})$ are stored in what is called the \textit{experience replay}. The replay memory is also buffered with a first-in-first-out policy so as not to learn from earlier, less informative transitions. Periodically, the neural networks are trained via minibatch gradient descent. The inputs to the network are the states $\{ s_i \}_{i = 1}^N$ and the targets $y^i$ are computed as follows:

\begin{equation}
    y^i =
    \begin{cases}
        r_i\quad & \text{for final}\ s_{i+1} \\
        r_i + \gamma\ \text{max}_a Q(s_{i+1}, a; \theta) & \text{otherwise}
    \end{cases}
\end{equation}

where $\theta$ are the neural network parameters. Effectively we update the Q-value of the taken action to reflect the best possible value of Q under the current policy.

Because of the max operation in equation 6, the value for Q is often overestimated. Since our predictions for Q is inherently noisy, we pick the "most overestimated" value of Q. \textcite{DBLP:journals/corr/HasseltGS15} proposed a simple solution. Instead of using one network, we now have two identical deep networks. The second network is updated to match the first every $\tau$ steps. We use the second network to compute the value updates instead of using the max operator, thus reducing the overestimation of Q values.

\subsection{Double Deep Q Networks (DDQN)}:
\begin{equation}
    y^i =
    \begin{cases}
        r_i\quad & \text{for final}\ s_{i+1} \\
        r_i + \gamma\ Q(s_{i+1}, \text{argmax}_{a'} Q(s_{i+1}, a'; \theta); \theta^-) & \text{otherwise}
    \end{cases}
\end{equation}

Where $\theta^-$ are the target network's weights. In both cases the networks are trained not after every action, but each $T$ steps taken, where the counter carries over episodes. The networks are not immediately trained, but we wait a number of iterations to let the agent gather some data by randomly exploring first.

For both algorithms, we usually suspnd training if an episode is solved, as defined in each environment. This way once we think we have reached an optimal policy $\pi^*$, we stop training until the policy fails. We do not want to "un-learn" $\pi^*$ by continuing training afterwards.

\subsection{$\epsilon$ - greedy}
The $\epsilon$-greedy exploration policy is what enables the agent to randomly \textit{explore} rather than \textit{exploit} the best found strategy so far. Instead of taking the action according to the greedy policy, as described above, with a small probability $\epsilon$ the agent takes a random action each step. $\epsilon$ is annealed similarly to $\alpha$, the learning rate, in our case by a factor of $0.995$ each episode.

See algorithm \ref{alg:dqn} for the DQN training algorithm.

\begin{algorithm}[ht]
    \begin{algorithmic}
        \State Initialize neural network with parameters $\mathbf{\theta}$
        \State counter $\gets$ 0
        \State s $\gets$ initial environment state
        \Repeat
            \State a $\gets$ $\text{argmax}_a$ Q(s, a)
            \State a $\gets$ random action with probability $\epsilon$

            \State Perform action and observe tuple $(s', r, \text{done})$
            \State Store tuple $(s, a, s', r, \text{done})$ in replay memory

            \If{last environment was not solved}
                \If{counter $\geq$ start\_training}
                    \If{counter \% train\_frequency == 0}
                        \State Sample random minibatch of size $B$
                        \State Train network according to equation 6
                    \EndIf
                \EndIf
            \EndIf

            \State s $\gets\ s'$

            \If{done}
                \State s $\gets$ initial environment state
                \State Reset environment
            \EndIf
            \State counter $\gets$ counter + 1
        \Until{maxEpisodes is reached}
    \end{algorithmic}
    \caption{DQN Training with $\epsilon$-greedy}
    \label{alg:dqn}
\end{algorithm}

\section{Policy Iteration}
These are a family of dynamic programming algorithms. The general Policy Iteration algorithm first evaluates a policy (Policy Evaluation) after which it updates it, taking a maximum over the values calculated in the evaluation step.

\textbf{Policy Evaluation}. Repeat until the chanages in $V(s)$, $\Delta V(s)$ are still large.
\begin{align*}
    \forall& s \in S \\
    & V(s) = \sum_a \pi(a \giv s) \sum_{s', r} P(s', r \giv s, a) \big[ r + \gamma V(s') \big]
\end{align*}

\textbf{Policy Improvement}. Repeat until the policy is unstable.
\begin{align*}
    \forall& s \in S \\
    & \pi(s) = \argmax_a \sum_{s', r} P(s', r \giv s, a) [r + \gamma V(s')]
\end{align*}

\section{Value Iteration}
This combines the above two in the same update cycle:
\begin{align*}
    \forall& s \in S \\
    & V(s) = \underbrace{\max_a \sum_{s', r}}_{\blue{\text{Note max}}} P(s', r \giv s, a) \big[ r + \gamma V(s') \big]
\end{align*}

Note the $max$ operator. This is used instead of summing over the policy $\pi(a \giv s)$. Thus we don't just evaluate the policy, but also improve it (by using the max operator). At the end we return a policy:

\begin{equation*}
    \pi(s) = \argmax_a \sum_{s', r} P(s', r \giv s, a) [r + \gamma V(s)]
\end{equation*}

\section{Monte Carlo}
Monte Carlo methods average returns over random exploratory episodes.

\begin{align*}
    &\text{Generate an episode at}\ s_0\ \text{following} \pi \\
    &\forall s, a \\
    &\quad G \gets \text{return following first occurrence of}\ s, a \\
    &\quad \text{append G to Returns(s, a)} \\
    &\quad Q(s, a) \gets \text{average(Returns(s, a))}
\end{align*}

\section{Eligibility Traces}
    \textit{TODO}

\section{Function Approximation}
This is the general case of approximating Q-functions (or V-functions) with a function, such as a deep network above. The general update rule is just a gradient descent rule. Using an MSE error, we get:

\begin{align*}
    \bm{\theta} &\gets \bm{\theta} + \alpha(V^\pi(s_t) - V_t(s_t)) \ \nabla_\bm{\theta} V_t(s_t) \\
    &= \bm{\theta} + \alpha(V^\pi(s_t) - V_t(s_t)) \ \nabla_\bm{\theta} V_t(s_t)
\end{align*}

where:

\begin{equation*}
    V_t(s_t) = f(\bm{\theta})
\end{equation*}

and $V^\pi(s_t)$ is the state value as computed in the current episode-.

\subsection{Policy Gradients Methods}
    \textit{For an excellent overview, see the tutorial by L. Weng (\cite{policy_gradient_tutorial}).}

    While SARSA tries to maximize the Q-function by using TD-errors, it still does not handle continuous actions or states very well. Indeed there are extensions of the method that better represent continuous states. Most notably, Deep Q-Networks (DQN, \cite{mnih_playing_2013}) store the Q-function as a neural network. The inputs are the states $s \in \mathcal{S}$ and the outputs -- the Q-value of the states for each action $a$. This representation, however, still relies on the environment having a discrete set of actions $\mathcal{A}$. Deep Q-Networks are similarly updated using a TD-error and the policy is chosen again using an arg max over actions.

    For continuous action spaces, such as in the swing-up problem, Deep Q-Networks suffer from the same problem as SARSA -- we need to discretize the action space, which leads to inaccuracies if there are too few buckets or high computational times if there are too many.

    We can instead look at the value of the start state $s_0$ -- $v_\pi(s_0)$ -- and try to maximize it. The policy gradient theorem states that:

    \begin{equation} \label{eq:policy_grad}
        \nabla v_\pi(s) = \mathbb{E}_\pi \big[
            G_t \nabla_\theta \ln \pi(a \giv s; \theta)
        \big]
    \end{equation}

    This assumes we have a parametric policy $\pi(a \giv s; \theta)$. We can now maximize $v_\pi(s_0)$ by performing a gradient \textbf{ascent} on the parameters of our policy $\theta$.

    We give a proof of the policy gradient theorem in the appendix. We combine the proof of \cite[Chapter~13]{sutton_learning_1988} with the proof of \cite{policy_gradient_tutorial} with some additional clarifications.

    \subsubsection{REINFORCE}

    % \begin{figure}[t]
    %     \centering
    %     \includegraphics[width=0.4\textwidth]{figures/reinforce_trajectory.png}
    %     \caption{Trajectory using REINFORCE with a neural network policy estimator.}
    %     \label{fig:reinforce_trajectory}
    % \end{figure}

    \autoref{eq:policy_grad} can be approximated by a single sample of $G_t$ over one episode. This is the idea behind REINFORCE:

    \begin{equation*}
        \nabla v_\pi(s) = \mathbb{E}_\pi \big[
            G_t \nabla_\theta \ln \pi(a \giv s; \theta)
        \big] \approx
            G_t \nabla_\theta \ln \pi(a \giv s; \theta)
    \end{equation*}

    We can thus perform a simple gradient \textbf{ascent} that maximizes $v_\pi(s_0)$:

    \begin{equation*}
        \theta \gets \theta + \alpha G_t \nabla_\theta \ln \pi(A_t \giv S_t)
    \end{equation*}

    In order to apply this in practice, we average returns over the entire episode -- $\frac{1}{N} \sum_i G_i$ and additionally normalize them. We also average the gradient of the policy function in the same way. This is akin to batch gradient descent, where averaging over many samples provides a more stable gradient.

    The last key ingredient is the policy. Instead of using an $\epsilon$-greedy exploration, we use a "normal" policy:

    \begin{equation*}
        \pi(a \giv s) = \mathcal{N}(f(s; \theta), \sigma)
    \end{equation*}

    where $f(s; \theta)$ is a parametric predictor of the mean of the normal distribution.

    \subsubsection{Deep Deterministic Policy Gradient (DDPG)}
    \begin{figure}[t]
        \centering
        \includegraphics[width=0.4\textwidth]{figures/ddpg_trajectory.png}
        \caption{Trajectory using DDPG.}
        \label{fig:ddpg_trajectory}
    \end{figure}

    \begin{figure*}[t!]
        \vskip 5mm
        \begin{center}
        \begin{subfigure}[b]{0.4\textwidth}
        \centerline{\includegraphics[width=\columnwidth]{figures/actor_loss.png}}
        % \caption{}
        % \label{fig:mountaincar_game}
        \end{subfigure}
        %
        \begin{subfigure}[b]{0.4\textwidth}
        \centerline{\includegraphics[width=\columnwidth]{figures/critic_loss.png}}
        % \caption{LunarLander-v2}
        % \label{fig:lunarlander_game}
        \end{subfigure}

        \caption{Actor and critic losses when using DDPG. The critic learns quickly, while the critic loss oscillates. We found that often the critic did not converge well and got stuck in local minima, such as immediately accelerating to the left or right. We show a fitted linear regression line for the actor loss.}
        \label{fig:ddpg_losses}
        \end{center}
        \vskip -5mm
    \end{figure*}

    In REINFORCE, we used the episodic return $G_t$ in the policy gradient. We could instead take the DQN approach (\cite{mnih_playing_2013}) and use a \textbf{critic} neural network (with parameters $\theta^Q$) to predict the $q_\pi(s, a)$ value. Additionally, we could use a deterministic policy, where $a = \pi(s; \theta^\pi)$. Again this is a neural network, called the \textbf{actor}. This slightly changes the policy gradient theorem, as proven in (\cite{pmlr-v32-silver14}). Additionally, DDPG uses the double deep-Q-learning trick (\cite{van_hasselt_deep_2015}), where the TD targets for the neural network use a second predictor network in order to reduce the variance of updates. Combining these steps, we arrive at Algorithm \ref{alg:ddpg}. We next break down the steps of the algorithm.

    We utilize an experience replay with stored transitions \\
    $(s_t, a_t, s_{t + 1}, r_t)$. At each step of the algorithm, we train over a minibatch of transitions samples from the replay. \textbf{Firstly}, we update the Q-network using a one-step TD target, similar to SARSA's update rule:

    \begin{equation} \label{eq:td_dqn}
        y_i = r_i + q(s_{i + 1}, \pi(s_{i + 1}; \theta^{\pi'}); \theta^{Q'})
    \end{equation}

    We use the target Q-value network $\theta^{Q'}$ and a one-step on-policy TD error using the target policy network $\theta^{\pi'}$. We then update the neural network using mean-squared error to match the target in \autoref{eq:td_dqn}. Thus the Q-network learns to predict the Q-values (returns) of the deterministic policy $\pi$. The policy is again a neural network, parameterized by $\theta^{\pi'}$.

    Next, we update the actor network using the deterministic policy gradient theorem (\cite{pmlr-v32-silver14}):

    \begin{equation*}
        \nabla \theta^\pi =
            \frac{1}{N} \sum_i \nabla_a Q(s, a; \theta^Q) \big|_{s = s_i, a = \pi(s_i)} \nabla_{\theta^\pi} \pi(s; \theta^\pi) \big|_{s_i}
    \end{equation*}

    This is the gradient of the Q-function:

    \begin{equation*}
        \nabla \theta^\pi =
            \frac{1}{N} \sum_i \nabla_{\theta^\pi} Q(s_i, \pi(s_i; \theta^\pi); \theta^Q)
    \end{equation*}

    Indeed this is an analog to the policy gradient theorem for deterministic policies, again using monte-carlo gradient estimation. The deterministic policy gradient theorem was introduces in \cite{pmlr-v32-silver14} and the proof is similar to that of the policy gradient theorem.

    Finally, we update the target networks parameterized by $\theta^{Q'}$ and $\theta^{\pi'}$ using an exponential moving average. This ensures that our target network is more stable:

    \begin{align*}
        \theta^{Q'} &\gets \tau \theta^Q + (1 - \tau) \theta^{Q'} \\
        \theta^{\pi'} &\gets \tau \theta^\pi + (1 - \tau) \theta^{\pi'} \\
    \end{align*}

    We applied DDPG to the inverted cart-pole system.

    We used two neural networks with one layer with $100$ neurons. Both the actor and the critic networks were trained using the Adam rule and the same $\sigma$ and $\gamma$ decay as previously. See \autoref{fig:ddpg_trajectory} for a trajectory learned by DDPG. See \autoref{fig:ddpg_losses} for an example plot of the loss functions for the actor and critic.

    We found that DDPG was by far the most difficult to converge. While the critic learned to predict Q-values well (see an example in \autoref{fig:ddpg_losses}), the actor oscillated a lot, sometimes getting stuck. Fitting a linear regression for the actor loss reveals that the overall trend is for the loss to decrease, albeit at a lower rate.

\clearpage
\printbibliography

\clearpage
\clearpage

\newcommand{\prob}{\text{P}}
\renewcommand{\vec}[1]{\mathbf{#1}}
\newcommand{\dotvec}[1]{\mathbf{\dot{#1}}}
\renewcommand{\d}[1]{\mathrm{d#1}}

\section{Appendix}
Here we provide short proofs of Lagrange's equations and the policy gradient theorem.
\subsection{Lagrangian Dynamics}
Here we give a short proof of the following theorem, known as Lagrange's equations:
\begin{equation*}
    \frac{\d{}}{\d{t}} \bigg( \frac{\partial L}{\partial \dot{\vec{q}_i}} \bigg) - \frac{\partial L}{\partial \vec{q}_i} = Q_i
\end{equation*}

The proof follows that of \cite[Section~1.5]{zak2003systems} and \cite{koskie}.

By Newton, work is defined as the force times the distance ($W = Fd$). In two (and similarly in three) dimensions, we define a generic parametric curve $C$ as:
\begin{align*}
    x &= x(t), \quad y = y(t) \\
    \vec{q} &= \langle x, y \rangle, \quad \vec{v} = \langle \dot{x}, \dot{y} \rangle \\
    0 &< t < 1
\end{align*}

If placed in a vector field $\vec{F} = \langle F_x, F_y \rangle$, the work exerted can be expressed as:

\begin{equation} \label{eq:work}
    \lim_{\Delta \vec{q} \to 0} \sum_i \vec{F} \cdot \Delta \vec{q} =
        \int_C \vec{F} \cdot \d{\vec{q}} =
        \int_C F_x \d{x} + \int_C F_y \d{y}
\end{equation}

% By Theorem 1 this is:

% \begin{equation} \label{eq:work}
%     W = \int_C F_x \frac{\d{x}}{\d{t}} \d{t} + \int_C F_y \frac{\d{y}}{\d{t}} \d{t} = \int_C F_x \dot{x} + F_y \dot{y}\ \d{t}
% \end{equation}

From Newton's second law, we know that:
\begin{align*}
    &\vec{F} = m \vec{a} = m \frac{\d{\vec{v}}}{\d{t}} \\
    F_x = m \ddot{x} = &m \frac{\dot{x}}{\d{t}} \quad\quad F_y = m \ddot{y} = m \frac{\dot{y}}{\d{t}}
\end{align*}

We also know that:
\begin{equation}
    \ddot{x} dx = \frac{\d{x}}{\d{t}} dx = \dot{x} \d{\dot{x}}
\end{equation}

Substituting in \autoref{eq:work}, we obtain:
\begin{equation}
    W_{AB} = \int_A^B m \dot{x} \d{\dot{x}} + \int_A^B m \dot{y} \d{\dot{y}} = m \bigg[ \frac{\dot{x}^2}{2} + \frac{\dot{y}^2}{2}  \bigg] \bigg|_A^B = \frac{m}{2} ( \norm{\dot{\vec{q}_B}}^2 - \norm{\dot{\vec{q}_A}}^2 )
\end{equation}

which is the definition of kinetic energy. Hence the work done is the difference in kinetic energies $K_A$ and $K_B$ at $A$ and $B$. This is the work-energy theorem:

\begin{equation}
    W_{AB} = K_B - K_A
\end{equation}

By the law of conservation of energy, the sum of potential and kinetic energies must remain constant. Equivalently, if the kinetic energy changes, the potential energy changes in the opposite direction.

\begin{align*}
    \Delta& K + \Delta U = 0 \\
    W &= \Delta K = - \Delta U
\end{align*}

The fundamental theorem of vector calculus states that if $\vec{U}$ is a conservative (solenoid) field, i.e. $\vec{U} = \nabla f$, then:

\begin{equation}
    \int_a^b \vec{U} \cdot \d{\vec{q}} = f(b) - f(a)
\end{equation}

We know that:
\begin{equation*}
    \Delta U = - \int_A^B \vec{F} \cdot \d{\vec{q}}
\end{equation*}

We can thus conclude that:
\begin{equation} \label{eq:fuckme}
    \vec{F}(\vec{q}) = -\nabla U (\vec{q})
\end{equation}

We can define the kinetic energy in multiple dimensions as :
\begin{equation} \label{eq:partialK}
    K = \frac{m \dot{\vec{q}}^\text{T} \dot{\vec{q}}}{2} \quad
        \frac{\partial K}{\partial\dot{\vec{q}}_i} = m \dot{\vec{q}}_i
\end{equation}

Combining \autoref{eq:partialK} with \autoref{eq:fuckme}, we obtain:
\begin{equation}
    \vec{F}_i = m \vec{a}_i = \frac{\d{}}{\d{t}} \bigg( \frac{\partial K}{\partial \dot{\vec{q}}_i} \bigg) = - \frac{\partial U}{\partial \vec{q}_i}
\end{equation}

Define the \textbf{Lagrangian} as $L = K - U$. Rearranging, we obtain:
\begin{equation*}
    \frac{\d{}}{\d{t}} \bigg( \frac{\partial L}{\partial \dot{\vec{q}}_i} \bigg) - \frac{\partial L}{\partial \vec{q}_i} = 0
\end{equation*}

where we note that:
\begin{equation*}
    \frac{\partial L}{\partial \dot{\vec{q}_i}} = \frac{\partial K}{\partial \dot{\vec{q}}_i}, \quad \quad \frac{\partial L}{\partial \vec{q}_i} = \frac{\partial U}{\partial \vec{q}_i}
\end{equation*}

An important case is when we add some energy to the system. As that adds overall energy, we can write:
\begin{equation*}
    \frac{\d{}}{\d{t}} \bigg( \frac{\partial L}{\partial \dot{\vec{q}_i}} \bigg) - \frac{\partial L}{\partial \vec{q}_i} = Q_i
\end{equation*}

where $Q_i$ is the generalized force contribution in the direction of $\vec{q}_i$.

\subsection{Policy Gradient Theorem}
Here we prove the policy gradient theorem. This proof combines that of \cite{sutton_learning_1988} with \cite{policy_gradient_tutorial} with some additional clarifications I felt were needed. We have a policy $\pi(a \giv s; \theta)$, which is parameterized by some vector $\theta$. For instance, we could use a neural network or a linear regression. Then:

\begin{equation*}
    \nabla v_\pi(s) = \mathbb{E}_\pi \big[
        G_t \nabla_\theta \ln \pi(a \giv s; \theta)
    \big]
\end{equation*}

where $\mu(s)$ is the stationary distribution of the markov chain. To define this, first we define the transition probability as a matrix. Let:

\begin{equation*}
    M_\pi[s_{t}, s_{t - 1}] =
        P(S_t = s_{t} \giv S_{t - 1} = s_{t}, \pi)
\end{equation*}

The transition probability depends both on the transition function and the current policy $\pi$. Then assume we start at the state $s_0$. This makes the distribution of states a vector with $0$ entries everywhere and $1$ at $s_0$. The distribution over next states would be:

\begin{align*}
    p(S_0) &= [0, 0, \dots, \underbrace{0}_{\text{at} s_0}, \dots, 0]^T \\
    p(S_1) &= M p(S_0) \\
    p(S_t) &= M^t p(S_0)
\end{align*}

A stationary distribution is then the probability distribution over states as $t \to \infty$. With a little abuse of notation:

\begin{equation}
    \mu(S_\infty) = \lim_{t \to \infty} M^t p(S_0)
\end{equation}

Firstly, we need to prove the following \textbf{Lemma 1}:
\begin{align*}
    q_\pi(s, a) &= \mathbb{E}_\pi[G_t \giv S_t = s, A_t = a] =
        \sum_{s', r} \mathbb{E}_\pi [
            G_t \giv S_t = s, A_t = a, R_{t + 1} = r, S_{t + 1} = s'
        ] P(s', r \giv s, a) \\
    &= \sum_{s', r} P(s', r \giv s, a)\ \mathbb{E}_\pi \big[
        R_{t + 1} + G_{t + 1} \giv s, a, r, s'
    \big] = \sum_{s', r} P(s', r \giv s, a) \big[r + v_\pi(s')\big]
\end{align*}

Then we expand the definition of the value function:

\begin{align*}
    \nabla v_\pi(s) &= \nabla\mathbb{E}_\pi[G_t \giv S_t = s] =
        \nabla \bigg[\sum_a \mathbb{E}[G_t \giv S_t = s, A_t = s]\ \pi(a \giv s) \bigg]
    = \nabla \bigg[\sum_a q_\pi(s, a)\ \pi(a \giv s) \bigg] \\
    &= \sum_a \bigg[ \nabla \pi(a \giv s) q_\pi (s, a) +
        \pi(a \giv s) \nabla q_\pi(s, a) \bigg] \\
    &= \sum_a \bigg[ \nabla \pi(a \giv s) q_\pi (s, a) +
        \pi(a \giv s) \nabla \sum_{s', r} P(s', r \giv s, a) (r + v_\pi(s')) \bigg] \\
    &= \sum_a \bigg[ \nabla \pi(a \giv s) q_\pi (s, a) +
        \pi(a \giv s) \sum_{s'} P(s' \giv s, a)\ \nabla v_\pi(s')) \bigg] \\
    &= \phi(s) + \sum_{s'} \sum_a \pi(a \giv s) P(s' \giv s, a) \nabla v_\pi(s')
        = \phi(s) + \sum_{s'} P(s \to s', 1) \nabla v_\pi(s') \\
\end{align*}

where $P(s \to s', 1)$ is the transition probability of getting to state $s'$ from state $s$ in one step. $\phi(s) = \sum_a \bigg[ \nabla \pi(a \giv s) q_\pi (s, a) \bigg]$. We continue to unroll the recursion, eventually arriving at the stationary distribution:

\begin{align*}
    \nabla v_\pi(s) &= \phi(s) + \sum_{s'} P(s \to s', 1) \nabla v_\pi(s') \\
    &= \phi(s) + \sum_{s'} P(s \to s', 1) \bigg[
        \phi(s') + \sum_{s''} \big[
            P(s' \to s'', 1) \nabla v_\pi(s'')
        \bigg]
    \bigg] \\
    &= \phi(s) + \sum_{s'} P(s \to s', 1)
        \phi(s') + \sum_{s''}
            P(s \to s'', 2) \nabla v_\pi(s'')\\
    &= \dots \\
    &=\sum_x \sum_{k = 0}^\infty P(s \to x, k) \phi(x) \\
    &= \sum_s \eta(x) \sum_a  q_\pi(s, a)
        \nabla_\theta \pi(a \giv s) \\
    &\propto \sum_s \frac{\eta(x)}{\sum_{\bar{x}} \eta(\bar{x})} \sum_a  q_\pi(s, a)
    \nabla_\theta \pi(a \giv s) \\
    &\propto \sum_s \mu(x) \sum_a  q_\pi(s, a)
    \nabla_\theta \pi(a \giv s) \\
    &\propto \sum_s \mu(x) \sum_a \pi(a \giv s) q_\pi(s, a) \frac{
        \nabla_\theta \pi(a \giv s)
    }{
        \pi(a \giv s)
    } \\
    &= \mathbb{E}_\pi \big[
        q_\pi(s, a) \nabla_\theta \ln \pi(a \giv s; \theta)
    \big] \\
    &= \mathbb{E}_\pi \big[
        G_t \nabla_\theta \ln \pi(a \giv s; \theta)
    \big]
\end{align*}

where on the last line we used the chain rule and the fact that $\nabla \ln x = \frac{1}{x} \nabla x$. The last step uses the following property of conditional expectations:
\begin{equation*}
    \mathbb{E}\big[ \mathbb{E} [X \giv Y] \big] =
       \sum_y \mathbb{E} [X \giv Y = y] P(Y = y) = \mathbb{E}[X]
\end{equation*}

\end{document}


This section discusses and concludes the results from section \ref{sec:evaluation} given by this paper, in sections \ref{sec:results}-\ref{sec:temp}. We also present future research and improvements in section \ref{sec:future}.

\subsection{Energy Disaggregation results}
\label{sec:results}
The disaggregation algorithm has shown to be implemented correctly but used with caution with regards to data, parameters and computational power. The algorithm has an overall accuracy of around 25\% by the equation \ref{eq:acc}. This would however say that we have a good accuracy compared to what the energy profiles show in the figures in section \ref{sec:evaluation}. Most of the predicted energy profiles for larger datasets are heavily overestimated in their energy consumption. Moreover, the results for a week and two weeks have shown not to provide accurate results given that we use lesser basis functions than the models suggest. They might be improved by providing the algorithm with computational power and thereby use sufficient basis functions as shown in the results in the beginning of section \ref{sec:evaluation}. Here we used roughly 10 times the amount of basis functions compared to the dimensions of the data, which provided more accurate results. The algorithm can surely be improved with computational power by providing the algorithm with more samples (houses), in the results provided in this thesis we had to even cut a good amount of good data to be able to run the algorithm, the dataset that Pecan Street \cite{pecan} really provided, was an extensive amount of data, we discuss more on that in section \ref{sec:dataset}.

The algorithm shown to be good at producing complex structures, such as a dishwasher being "on" and "off" during a certain time period, as shown in the figures for the basis functions \ref{fig:basis_functions}, it can also be shown in the prediction for the air-condition in figure \ref{fig:subset}. However, it was not accurate at which of the appliances that were being used, the algorithm seemed to infer that one appliance stood for most of the energy consumption, which was false for most of the data. It also highly overestimated the power consumption of all of the appliances. This could be a fault in the DDSC algorithm implemented as shown in the evaluation of the norms during the training. The norm of the activations seemed to react heavily for the whole home energy usage and might have provided the algorithm with an overestimation of the energy consumption for all of the appliances.

\subsection{Algorithm}

The algorithm provided in Kotler et.al in \cite{DDSC} was cumbersome to implement. They provide a section with model implementation but state that they have precluded the details due to space constraints. Thankfully Scikit-learn \cite{scikit} has been a source to go to for help regarding implementing such an algorithm, such as the DDSC. The algorithm also has some parameter choices, which was fitted using grid-search, this was however not implemented and could provide the algorithm with better results. They did not state explicitly which parameters they used for disaggregation which would have made the implementation easier, see section \ref{sec:params} below for more thought on the parameters for the algorithm. The overestimation of the energy consumption might have been avoided by implementing the extension presented below in section \ref{sec:extensions}.

\subsubsection{Extensions}
\label{sec:extensions}
Here we present the extension proposed by Kotler et.al. \cite{DDSC} to modify the standard Sparse Coding formulation. This could be implemented into the model, which was said to increase accuracy by 1.3\% when using both of the extensions, more detail explanations are in their paper \cite{DDSC} page five.

\textbf{Total energy priors.} Kotler et. al. mention that the Sparse Coding model presented does not take into consideration the different power consumptions that the appliances might have. They could take similar shapes such as dishwasher and refrigerator into the same category while they might have totally different power consumption while operating. In summary, this extension penalizes the deviation between a device and its mean total energy. \cite{DDSC}
\vspace{0.1in}
\[
F_{TEP}(\bar{\mathbf{X}},\mathbf{B}_{1:k},\mathbf{A}_{1:k}) = F(\bar{\mathbf{X}},\mathbf{B}_{1:k} \mathbf{A}_{1:k}) + \lambda_{TEP}\sum_{i=1}^k \norm{\mu_i\mathbf{1}^T-\mathbf{1}^T\mathbf{B}_i\mathbf{A}_i}^2_2
\]
where $\mathbf{1}$ denotes a vector of ones of the appropriate size, and $\mu_i = \frac{1}{m} \mathbf{1}^T \mathbf{X}_i$ denotes the average total energy of device class $i$.
\vspace{0.1in}
\\
\textbf{Group Lasso.} Since energy consumption exhibit some sparsity at the device level (zero energy consumption, or not being monitored in the home), Kotler et.al. encourage a grouping effect to the activations. This could have prevented the algorithm for prioritizing one appliance across all of the other appliances. To achieve extension, one can employ the group Lasso algorithm \cite{grouplasso},
\vspace{0.1in}
\[
F_{GL}(\bar{\mathbf{X}},\mathbf{B}_{1:k},\mathbf{A}_{1:k}) = F(\bar{\mathbf{X}},\mathbf{B}_{1:k} \mathbf{A}_{1:k}) + \lambda_{GL}\sum_{i=1}^k \sum_{j=1}^m \norm{\mathbf{a}_i^{(j)}}_2
\]
They also present \textbf{Shift Invariant Sparse Coding}, which they say could not capture the information wanted. \cite{DDSC}

\subsection{Dataset}
\label{sec:dataset}

The dataset needed a lot of preparation to be able to even come remotely close to being a full dataset. Kotler et.al. did not address if they spent time on data pre-processing any of the data which seems almost unreasonable for their amount of data. Furthermore, the assumptions made in the data pre-processing for this thesis, presented in detail in section \ref{sec:prep} have made an impact on the results and Kotler et.al. do not present any of these assumptions that must have been made to be able to work with that amount of data. The dataset used in this thesis has not been validated via a cross-validation, which could improve the algorithm slightly.
In this thesis data from the Pecan Street was shown to represent a Weibull distribution which could be used for a generalization of the energy consumption in the area around Pecan Street or used for disaggregation based on distributional disaggregation, such as Semi-Markov models \cite{kotler2}.

%\pagebreak[3]
\subsection{Temporal difference}
\label{sec:temp}
This thesis has shown that trying to train the algorithm by exploiting temporal difference has not been proven useful. The conclusion drawn from this is that one should use, as much data as available as Sparse Coding needs enough data for it to provide a good representation of the profiles. Training data for other appliances other than that of Pecan Street are scarce and hard to come by, which indicates that we need to use the data that is available.

\subsection{Future research}
\label{sec:future}
Here we present future research that might come to help with DDSC algorithm or provide insight into the field of energy disaggregation as a whole.
\subsubsection{Hyper-parameter Optimization}
\label{sec:params}

The type of hyper-parameter controls the capacity of a model, i.e., how flexible the model is, how many degrees of freedom it has in fitting the data. Proper control of model capacity can prevent overfitting, which happens when the model is too flexible, and the training process adapts too much to the training data, thereby losing predictive accuracy on new test data. So a proper setting of the hyper-parameters is important \cite{hyper}.

There exists algorithms for defining the hyper-parameters of the model, one being that of Sequential Model-based Global Optimization (SMBO). These algorithms have been used in applications where evaluation of the fitness function is expensive. In an application where the true fitness function $f : X \rightarrow R$ is costly to evaluate, model-based algorithms approximate $f$ with a surrogate
that is cheaper to evaluate \cite{hyper}. There also exits "The Gaussian Process Approach", Tree-structured Parzen Estimator Approach (TPE), Random Search for Hyper-Parameter Optimization in DBNs (deep-belief-networks) and Sequential Search for Hyper-Parameter Optimization in DBNs. The latter of the two could prove to be valuable for methods just like the DDSC algorithm for providing the algorithm with the correct hyper-parameters for the model \cite{hyper}.

\subsubsection{Autoencoders}

Most Deep Learning systems heavily use unlabeled as well as labeled data. Large amounts of unlabeled data (Millions of pictures, gigabytes of text, tons of hours of voice) are used for feature learning mainly through deep autoencoders. The output of this phase is a high level abstraction of the data. The recent development with using autoencoders by Google in 2012, where even Andrew Yg. contributed to the work. There an unsupervised deep learning approach was used, trained it with Millions of YouTube images and the final neurons could recognize faces, cars, and cats. So for this network you just need to map the neurons to the labels you like to have, e.g. this is a face, a car, a cat. \cite{google} The large amount of unlabeled data that makes up the energy sector makes deep learning approaches so strong.

\subsubsection{Block Coordinate Update}
\label{sec:bcd}
Regularized block multiconvex optimization presented in 2013 by Yangyang Xu and Wotao Yin \cite{block2}, is an interesting approach, where the feasible set and objective function are generally non-convex but convex in each block of variables. It also accepts non-convex blocks and requires these blocks to be updated by proximal minimization. Compared to the existing state-of-the-art algorithms, the proposed algorithm demonstrate superior performance in both speed and solution quality. This work could pose to be the next approach to energy disaggregation.

\subsubsection{Dropout}
Dropout, by Hinton et al. \cite{dropout}, in 2014, is perhaps the biggest invention in the field of neural networks in recent years. It addresses the main problem in machine learning that is overfitting. It does so by “dropping out” some unit activations in a given layer that is setting them to zero. Thus it prevents co-adaptation of units and can also be seen as a method of assembling many networks sharing the same weights. For each training example a different set of units to drop is randomly chosen. The dropout procedure can also be applied to the input layer by randomly deleting some of the input-vector components typically an input component is deleted with a smaller probability. Dropout has been reported to yield remarkable improvements on several difficult problems, for instance in speech and image recognition and hopefully could provide a means to remove overfitting in energy disaggregation as well. \cite{dropout}

\subsection{Final words, Open questions}
One interesting take that was discussed during this thesis was the precision of the algorithms for energy disaggregation. If we would be able to disaggregate with an accuracy of over 90\%, how would the public react with privacy issues? Would people want to provide their energy usage and if so, how much can be attained by the utility companies?