% =========================================================================
% SciPost LaTeX template
% Version 1e (2017-10-31)
%
% Submissions to SciPost Journals should make use of this template.
%
% INSTRUCTIONS: simply look for the `TODO:' tokens and adapt your file.
%
% - please enable line numbers (package: lineno)
% - you should run LaTeX twice in order for the line numbers to appear
% =========================================================================

% TODO: uncomment ONE of the class declarations below
% If you are submitting a paper to SciPost Physics: uncomment next line
\documentclass[submission, Phys]{SciPost}
% If you are submitting a paper to SciPost Physics Lecture Notes: uncomment next line
%\documentclass[submission, LectureNotes]{SciPost}
% If you are submitting a paper to SciPost Physics Proceedings: uncomment next line
%\documentclass[submission, Proceedings]{SciPost}

\usepackage{graphicx}% Include figure files
\usepackage{subcaption}
\usepackage[labelformat=parens,labelsep=quad, skip=3pt]{caption}
\usepackage{dcolumn}% Align table columns on decimal point
\usepackage{bm}% bold math
\usepackage{xcolor}
\usepackage{hyperref}
\usepackage{afterpage}
\usepackage{bbold}
\usepackage{amssymb}
% below, packages for all kinds of unicode symbols
%\usepackage{amsmath}
%\usepackage{amsxtra}
%\usepackage{bbold}
%\usepackage{isomath}
%\usepackage{mathdots}
%\usepackage{stmaryrd}
\usepackage{wasysym}
%\usepackage{natbib}

\newcommand{\todo}[1]{\textcolor{red}{[#1]}}

% numerical scales
%\newcommand{\stiny}{\scalebox{0.5}{x}}
%\newcommand{\ssmall}{\scalebox{0.9}{x}}
%\newcommand{\smedium}{\scalebox{1.3}{x}}
%\newcommand{\slarge}{\scalebox{1.7}{x}}

%\newcommand{\stiny}{\scalebox{0.5}{s}}
%\newcommand{\ssmall}{\scalebox{0.9}{s}}
%\newcommand{\smedium}{\scalebox{1.3}{s}}
%\newcommand{\slarge}{\scalebox{1.7}{s}}

\newcommand{\stiny}{\scalebox{0.5}{s}}
\newcommand{\ssmall}{\scalebox{0.9}{s}}
\newcommand{\smedium}{\scalebox{1.3}{s}}
\newcommand{\slarge}{\scalebox{1.7}{s}}

%\newcommand{\stiny}{\scalebox{0.5}{\CIRCLE}}
%\newcommand{\ssmall}{\scalebox{0.9}{\CIRCLE}}
%\newcommand{\smedium}{\scalebox{1.3}{\CIRCLE}}
%\newcommand{\slarge}{\scalebox{1.7}{\CIRCLE}}

\newcommand{\U}{\left[
	\begin{array}{cccc}
		\ssmall & \ssmall & \ssmall & \ssmall \\
		\ssmall & \ssmall & \ssmall & \ssmall \\
		\ssmall & \ssmall & \ssmall & \ssmall \\
		\ssmall & \ssmall & \ssmall & \ssmall \\
	\end{array}
	\right]}

\newcommand{\Dp}{\left[
	\begin{array}{cccc}
		\slarge &  &  &  \\
		& \smedium &  &  \\
		&  & \ssmall &  \\
		&  &  & \ssmall \\
	\end{array}
	\right]}

\newcommand{\Dm}{\left[
	\begin{array}{cccc}
		\ssmall &  &  &  \\
		& \ssmall &  &  \\
		&  & \ssmall &  \\
		&  &  & \stiny \\
	\end{array}
	\right]}

\newcommand{\D}{\left[
	\begin{array}{cccc}
		\slarge &  &  &  \\
		& \smedium &  &  \\
		&  & \ssmall &  \\
		&  &  & \stiny \\
	\end{array}
	\right]}

\begin{document}

% TODO: write your article's title here.
% The article title is centered, Large boldface, and should fit in two lines
\begin{center}{\Large \textbf{
Fast and stable determinant quantum Monte Carlo
}}\end{center}

% TODO: write the author list here. Use initials + surname format.
% Separate subsequent authors by a comma, omit comma at the end of the list.
% Mark the corresponding author with a superscript *.
\begin{center}
Carsten Bauer\textsuperscript{1*}
\end{center}

% TODO: write all affiliations here.
% Format: institute, city, country
\begin{center}
{\bf 1} Institute for Theoretical Physics, University of Cologne, 50937 Cologne, Germany
\\
% TODO: provide email address of corresponding author
* bauer@thp.uni-koeln.de
\end{center}

\begin{center}
\today
\end{center}

% For convenience during refereeing: line numbers
%\linenumbers

\section*{Abstract}
{\bf
% TODO: write your abstract here.
We assess numerical stabilization methods employed in fermion many-body quantum Monte Carlo simulations. In particular, we empirically compare various matrix decomposition and inversion schemes to gain control over numerical instabilities arising in the computation of equal-time and time-displaced Green's functions within the determinant quantum Monte Carlo (DQMC) framework. Based on this comparison, we identify a procedure based on pivoted QR decompositions which is both efficient and accurate to machine precision. The Julia programming language is used for the assessment and implementations of all discussed algorithms are provided in the open-source software library \href{http://github.com/crstnbr/StableDQMC.jl}{\texttt{StableDQMC.jl}}.
}

% TODO: include a table of contents (optional)
% Guideline: if your paper is longer that 6 pages, include a TOC
% To remove the TOC, simply cut the following block
\vspace{10pt}
\noindent\rule{\textwidth}{1pt}
\tableofcontents\thispagestyle{fancy}
\noindent\rule{\textwidth}{1pt}
\vspace{10pt}

\section{Introduction}

Many-fermion systems play an important role in condensed matter physics. Due to their intrinsic correlations they feature rich phase diagrams which can not be captured by purely classical nor non-interacting theories. Especially at the lowest temperatures, quantum mechanical fluctuations driven by Heisenberg's uncertainty principle become relevant and lead to novel phases of matter like superconductivity and states beyond the Fermi liquid paradigm \cite{Bauer2020, Schattner2016}. Because of the presence of interactions, predicting microscopic and thermodynamic properties of fermion many-body systems is inherently difficult. Analytical approaches are typically doomed to fail in cases where one can not rely on the smallness of an expansion parameter \cite{Berg2019}.

Fortunately, the determinant quantum Monte Carlo (DQMC) method \cite{Blankenbecler1981, Loh2005, Scalapino1993, Santos2003, Assaad2002a} overcomes this limitation. The key feature of DQMC is that it is numerically exact - given sufficient computation time the systematical error is arbitrarily small. Provided the absence of the famous sign-problem\cite{Loh1990, Troyer2005}, it allows for an efficient exploration of the relevant region of the exponentially large configuration space in polynomial time. It is an important unbiased technique for obtaining reliable insights into the physics of many-fermion systems which, among others, has been applied to the attractive and repulsive Hubbard model \cite{Hirsch1985, White1989, Moreo1991}, the Kane-Mele-Hubbard \cite{Hohenadler2012}, and metallic quantum criticality, including studies of antiferromagnetic\cite{Bauer2020, Schattner2016, Gerlach2017, Berg2019}, Ising-nematic \cite{SchattnerLederer2016}, and deconfined quantum critical points \cite{Gazit2017} where fermionic matter fields are coupled to bosonic order parameters.

Although conceptually straightforward, care has to be taken in the implementation of DQMC because of inherent numerical instabilities arising from ill-conditioned matrix exponentials. Over time, stabilization schemes \cite{Loh1989, Loh2005, Bai2011, Sorella1989, Assaad2002a} based on various matrix factorizations, such as singular value decomposition (SVD), modified Gram-Schmidt, and QR decomposition, have been proposed for lifting these numerical issues. It is the purpose of this manuscript to review a subset of these techniques and to compare them with respect to accuracy and speed. Particular emphasis is placed on concreteness and reproducibility: we provide implementations of all discussed algorithms as well as the code to recreate all visualizations in this manuscript in form of the software library \href{http://github.com/crstnbr/StableDQMC.jl}{\texttt{StableDQMC.jl}}. We choose the open-source, high-level programming language Julia\cite{Bezanson2017, JuliaNature2019} for our assessment which has proven \cite{diffeqbench, Luo2019} to be capable of reaching a performance comparable to established low-level languages in the field of numerical computing. Readers are invited to open issues and pull requests at the library repository to discuss, improve, and extend the list of stabilization routines. Beyond reproducibility, the software library will also serve as an important abstraction layer allowing users to focus on physical simulation instead of numerical implementation details.

Specifically, the structure of the manuscript is as follows. We start by providing a brief introduction into the DQMC method in Sec.~\ref{sec:QMC}. In Sec.~\ref{sec:instabilities} we illustrate numerical instabilities in the DQMC and discuss their origin. Following this, we demonstrate (Sec.~\ref{sec:stabilization}) how matrix factorizations can be utilized to remedy these numerical artifacts in chains of matrix products. In Sec.~\ref{sec:etgf} we present and benchmark different schemes for stabilizing the computation of the equal-times Green's function, the fundamental building block in DQMC. Lastly, we turn to the calculation of time-displaced Green's functions in Sec.~\ref{sec:TDGF} before concluding and summarizing in Sec.~\ref{sec:discussion}.

\section{Determinant Quantum Monte Carlo}\label{sec:QMC}

We begin by reviewing the essentials of the determinant quantum Monte Carlo method \cite{Blankenbecler1981, Loh2005, Scalapino1993, Santos2003, Assaad2002a}. We assume a generic quantum field theory that can be split into a purely bosonic part $S_B$ and a contribution $S_F$ from itinerant fermions. The latter comprises both fermion kinetics, $T$, and boson-fermion interactions, $V$. A famous example is given by the Hubbard model after a decoupling of the on-site
interaction $U n_{i, \uparrow} n_{i, \downarrow}$ by means of a continuous Hubbard-Stratonovich or a discrete Hirsch transformation \cite{Hirsch1983}\footnote{Depending on the decomposition channel, the bosonic field $\phi$ represents either spin or charge fluctuations in this case.}. The quantum statistical partition function is given by
%
\begin{equation}
\mathcal{Z} = \int D\left( \psi, \psi^\dagger, \phi \right) e^{-S_B[\phi] - S_F[\psi, \psi^\dagger, \phi]} \,.
\end{equation}
%
The first step in DQMC is to apply the quantum-classical mapping \cite{Sachdev2011} and switch from the $d$ dimensional quantum theory above to a $D = d + 1$ dimensional classical theory. Here, the extra finite dimension of the classical theory is given by imaginary time $\tau$ and has an extent proportional to inverse temperature $\beta = 1/T$. Discretizing imaginary time into $M$ slices, $\beta = M \Delta \tau$, and applying a Trotter-Suzuki decomposition \cite{Trotter1959, Suzuki1986} one obtains
%
\begin{align}
	\mathcal{Z} &= \int D\phi \ e^{-S_B} \mathrm{Tr}{\left[\exp{\left( -\Delta\tau \sum_{l=1}^M \psi^\dagger \left[T + V_\phi\right] \psi \right)}\right]} \label{eq:discretizedpi} \,.
\end{align}
%
A separation of the matrix exponential then leads to a systematic error of the order $\mathcal{O}\left(\Delta\tau^2\right)$ in the partition function,
\begin{align}
	e^{A + B} &\approx e^A e^B, \quad \nonumber\\
	e^{-\Delta\tau (T + V)} &\approx e^{- \frac{\Delta\tau}{2}T} e^{-\Delta\tau V} e^{- \frac{\Delta\tau}{2}T} + \mathcal{O}\left(\Delta\tau^3\right), \nonumber\\
	\mathcal{Z} &= \int D\phi \ e^{-S_B} \mathrm{Tr}{\left[ \prod_{l=1}^{m} B_l \right]} + \mathcal{O}\left(\Delta\tau^2\right). \label{eq:Ztr}
\end{align}
%
Here, $B_l = e^{- \frac{\Delta\tau}{2}\psi^\dagger T \psi} e^{-\Delta\tau \psi^\dagger V_\phi \psi} e^{- \frac{\Delta\tau}{2}\psi^\dagger T \psi}$ are imaginary time slice propagators. Note that the contribution $e^{-\Delta\tau \psi^\dagger V_\phi \psi}$ depends on the bosonic field $\phi$ due to a potential fermion-boson coupling in $V$. Rewriting the trace in \eqref{eq:Ztr} as a determinant\cite{Assaad2002a} yields the fundamental form
%
\begin{align}
	\mathcal{Z} &= \int D\phi \ e^{-S_B} \det{G_\phi^{-1}} + \mathcal{O}\left(\Delta\tau^2\right), \label{eq:DQMC}
\end{align}
%
where
\begin{align}
	G = \left[ \mathbb{1} + B_M B_{M-1} \cdots B_1 \right]^{-1} \label{etgf}
\end{align}
is the equal-time Green's function \cite{Sachdev2011}. Accordingly, the Metropolis probability weight is given by
\begin{align}
p = \min \left\{ 1 , e^{-\Delta S_\phi}  \dfrac{\det G}{\det G'} \right\}. \label{eq:metropolis}
\end{align}
This implies that, considering a generic, global update one needs to compute the Green's function $G$ and it's determinant in each DQMC step\footnote{For local updates one can typically avoid those explicit calculations and compute the ratio of determinants in Eq.~\eqref{eq:metropolis} directly\cite{Schattner2016}.}.

Importantly, it is only under specific circumstances, such as the presence of a symmetry, that the integral kernel of the partition function can be safely interpreted as a probability weight since $G_\phi$ and its determinant are generally complex valued. This is the famous sign problem \cite{Li2019}.

\section{Numerical instabilities}\label{sec:instabilities}

To showcase the typical numerical instabilities arising in the DQMC framework we consider the Hubbard model in one dimension at half filling,
\begin{align}
	H = -t\sum_{\langle i,j \rangle} c_i^\dagger c_j + U \sum_i \left( n_{i\uparrow} - \frac{1}{2} \right) \left( n_{i\downarrow} - \frac{1}{2} \right) \label{eq:model},
\end{align}
which is free of the sign-problem\cite{Li2019}. We set the hopping amplitude to unity, $t=1$\footnote{We will consider the canonical discrete decoupling \cite{Hirsch1983} in the spin channel due to Hirsch in our analysis.}.
%\begin{align}
%H = \left[
%\begin{array}{cccccccc}
%-0.1 & 1.0 &  &  &  &  &  &  \\
%1.0 & -0.1 & 1.0 &  &  &  &  &  \\
% & 1.0 & -0.1 & 1.0 &  &  &  &  \\
% &  & \ddots & \ddots & \ddots &  &  &  \\
%  &  &  & 1.0 & -0.1 & 1.0 &  \\
%  &  &  &  & 1.0 & -0.1 & 1.0 \\
%  &  &  &  &  & 1.0 & -0.1 \\
%\end{array}
%\right].
%\end{align}

As seen from Eq.~\eqref{etgf}, the building block of the equal-time Green's function is a matrix chain multiplication of imaginary time slice matrices. To simplify our purely numerical analysis below we assume that these slice matrices $B_i$ are independent of imaginary time,
\begin{align}
	B(\beta, 0) \equiv B_M B_{M-1} \cdots B_1 = \underbrace{B B \cdots B}_{M \textrm{ factors}}, \label{eq:Bchain}
\end{align}
which, physically, amounts to assuming a constant bosonic field $\phi = \textit{const}$.

First, we consider the non-interacting system, $U=0$. As apparent from Fig.~\ref{fig:naive_vs_stable}, a naive computation of Eq.~\ref{eq:Bchain} fails for $\beta \geq \beta_c \approx 10$. Leaving a discussion of the stabilization of the computation for the next section, let us highlight the origin of this instability. The eigenvalues of the non-interacting system are readily given by
\begin{align}
	&\epsilon_k = -2t\cos(k),
\end{align}
such that energy values are bounded by $-2t \leq \epsilon_k \leq 2t$. A single positive definite slice matrix $B = e^{-\Delta \tau T}$ therefore has a condition number of the order of $\kappa \approx e^{4|t|\Delta \tau}$ and, consequently, $B(\tau, 0)$ has $\kappa \approx e^{4|t|M\Delta \tau} = e^{4|t|\beta}$. This implies that the scales present in $B(\tau, 0)$ broaden exponentially at low temperatures $T=1/\beta$ leading to inevitable roundoff errors due to finite machine precision which spoil the result.

We can estimate the expected inverse temperature of this breakdown for the data type \texttt{Float64}, that is double floating-point precision according to the IEEE 754 standard \cite{Goldberg1991}, by solving $\kappa(\beta) \sim 10^{-17}$ for $\beta_c$. One finds $\beta_c \approx 10$ in good agreement with what is observed in Fig.~\ref{fig:naive_vs_stable_float64}. Switching to the data type \texttt{Float128}\footnote{The datatype \texttt{Float128} is provided by the Julia package \href{https://github.com/JuliaMath/Quadmath.jl}{\texttt{Quadmath.jl}}.} (quadruple precision) with $\beta_c \approx 20$ in Fig.~\ref{fig:naive_vs_stable_float128}, the onset of roundoff errors is shifted to lower temperatures in accordance with expectations.

\begin{figure}
	\centering
	\begin{subfigure}{0.48\textwidth}
		\includegraphics[width=\textwidth]{figures/naive_vs_stable.pdf}
		\caption{64-bit floating point (\texttt{Float64}) \label{fig:naive_vs_stable_float64}}
	\end{subfigure}%
	\hspace{10pt}
	\begin{subfigure}{0.48\textwidth}
		\includegraphics[width=\textwidth]{figures/naive_vs_stable_float128.pdf}
		\caption{128-bit floating point (\texttt{Float128}) \label{fig:naive_vs_stable_float128}}
	\end{subfigure}
	\vspace{5pt}
	\caption[MyCaption]{\textbf{Numerical instabilities} due to finite machine precision (green) arising in the calculation of the time slice matrix chain product $B_M B_{M-1} \cdots B_1$ for model~\eqref{eq:model}. The dashed line (grey) indicates the expected floating point precision\protect\footnotemark. The arbitrary precision result (orange) is shown for comparison.}
	\label{fig:naive_vs_stable}
\end{figure}

\footnotetext{We estimate the precision as $p = \log_{10}(2^{\textrm{fraction}})$, where $\textrm{fraction}$ is the mantissa of a given binary floating point format. This gives $p\sim 16$ for \texttt{Float64} and $p\sim 34$ for \texttt{Float128}.}

\section{\label{sec:stabilization}Stabilization: time slice matrix multiplications}

\subsection{Stabilization scheme}
A trivial solution to the issue outlined above is to perform all numerical operations with arbitrary precision. In Julia, this can be realized by means of the \texttt{BigFloat} data type\footnote{Technically, \texttt{BigFloat} has a finite, arbitrarily high precision.}. However, this comes at the expense of (unpractical) slow performance due to algorithmic overhead and lack of hardware support. Arbitrary precision numerics is nevertheless a valuable tool and we will use it to benchmark the accuracy of stabilization methods below\footnote{For our non-interacting model system one can alternatively simply diagonalize the Hamiltonian and calculate the Green's function exactly.}.

How can we get a handle on the numerical instabilities in a floating point precision computation? The strategy is to keep the broadly different scales in the matrix exponentials separated throughout the computation (as much as possible) and only mix them in a final step, if necessary. A useful tool for extracting the scale information is a matrix decomposition,
\begin{align}
	B = UDX. \label{eq:decomp}
\end{align}
Here, $U$ and $X$ are matrices of order unity and $D$ is a real diagonal matrix hosting the exponentially spread scales of $B$. We will refer to the values in $D$ as singular values independent of the particular decomposition. Using Eq.~\eqref{eq:decomp}, we can stabilize the matrix multiplication of slice matrices $B_1$ and $B_2$ in Eq.~\eqref{eq:Bchain} as follows, (\texttt{fact\_mult} in \texttt{StableDQMC.jl})

\begin{align}
	B_2 B_1 &= \underbrace{U_2 D_2 X_2}_{B_2}\underbrace{U_1 D_1 X_1}_{B_1} \nonumber\\
	&= U_2 \underbrace{(D_2 ((X_2 U_1) D_1))}_{U' D' X'} X_1)\label{eq:stabmult}\\
	&= U_r D_r X_r. \nonumber
\end{align}
Here, $U_r = U_2 U'$, $D_r = D'$, $X_r = X' X_1$, and $U'D'X'$ indicates an intermediate matrix decomposition. If we follow this scheme, in which parentheses indicate the order of operations, largely different scales present in the diagonal matrices won't be additively mixed throughout the computation. Specifically, note that the multiplication of the well-conditioned, combined, unit-scale matrix $U = X_2 U_1$ with $D_1$ and $D_2$ does preserve the scale information: the diagonal matrices merely rescale the columns and rows of $U$,
\begin{align}
D_2  U D_1 &= \D  \underbrace{\U}_{\textrm{unit scale}} \D  \\
&= \D \left[
\begin{array}{cccc}
\ssmall\slarge & \ssmall\smedium & \ssmall^2 & \ssmall\stiny \\
\ssmall\slarge & \ssmall\smedium & \ssmall^2 & \ssmall\stiny \\
\ssmall\slarge & \ssmall\smedium & \ssmall^2 & \ssmall\stiny \\
\ssmall\slarge & \ssmall\smedium & \ssmall^2 & \ssmall\stiny \\
\end{array}
\right]\\
&= \left[
\begin{array}{cccc}
	\slarge^2\ssmall & \slarge\smedium\ssmall & \slarge\ssmall^2 & \slarge\ssmall\stiny \\
	\slarge\smedium\ssmall & \smedium^2\ssmall & \smedium\ssmall^2 & \smedium\ssmall\stiny \\
	\slarge\ssmall^2 & \smedium\ssmall^2 & \ssmall^3 & \ssmall^2\stiny \\
	\slarge\ssmall\stiny & \smedium\ssmall\stiny & \ssmall^2\stiny & \ssmall\stiny^2 \\
\end{array}
\right].
\end{align}

Repeating the procedure \eqref{eq:stabmult}, we obtain a numerically accurate $UDX$ decomposition of the full time slice matrix chain $B(\tau, 0)$, which preserves the scale information as indicated in Fig.~\ref{fig:naive_vs_stable}.\footnote{Note that we do not discuss the faster way to calculate $B^M$ as $U D^M X$. This is intentional since most real systems will involve fermion-boson interactions and the slice matrices will depend on $\phi(\tau)$.} We note in passing that in practice it is often unnecessary to stabilize every individual matrix product. Instead one typically performs a mixture of naive and stabilized products for the sake of speed while still retaining numerical accuracy \cite{Assaad2002a}.

\subsection{Matrix decompositions}

There are a various matrix decompositions that one could employ to obtain the factorization ${B = UDX}$, Eq~\eqref{eq:decomp}. In the following we will consider the two most popular choices in DQMC codes \cite{Assaad2002a, Loh1989, Santos2003}.

\subsubsection{SVD ($UDV^\dagger$)}
The singular value decomposition (SVD) is given by
\begin{align}
	B = USV^\dagger,
\end{align}
where $U$ and $V^\dagger$ are unitary and $S$ is real and diagonal.

For computing the SVD of a matrix of regular floating point precision (\texttt{Matrix\{Float64\}}), Julia utilizes the heavily optimized routines provided by LAPACK\footnote{We will report on results obtained with the LAPACK implementation OpenBLAS that ships with Julia. Qualitatively similar results have been found in an independent test based on Intel's Math Kernel Library (MKL).} \cite{LAPACK}. Concretely, there exist three different implementations of SVD algorithms:\footnote{Note that the names of LAPACK functions typically encode properties of the input matrix such as realness or symmetry. In Julia multiple-dispatch takes care of routing different matrix types to different \textit{methods}. The Julia function \texttt{gesdd} works for both real and complex matrices, that is there is no (need for) \texttt{cgesdd}.}
\begin{itemize}
	\item \texttt{gesdd} (default): Divide-and-conquer (D\&C)
	\item \texttt{gesvd}: Conventional
	\item \texttt{gesvj}: Jacobi algorithm (through \href{https://github.com/RalphAS/JacobiSVD.jl}{JacobiSVD.jl})
\end{itemize}
To simplify the manual access to these algorithms we export convenience wrappers of the same name in \texttt{StableDQMC.jl}. We will compare all three variants below and benchmark them against an arbitrary precision computation using \texttt{BigFloat}. Since LAPACK doesn't support special number types, we will utilize the native-Julia SVD implementation provided by \href{https://github.com/JuliaLinearAlgebra/GenericSVD.jl/}{GenericSVD.jl} in this case.

\subsubsection{QR ($UDT$)}
A QR decomposition reads
\begin{align}
	B = QR = UDT,
\end{align}
where $Q$ is unitary, $R$ is upper triangular, and we have split $R$ into a diagonal part $D$ and an upper triangular part $T$ in the second step. Specifically, $U = Q$ is unitary, $D = \textrm{diag}(R)$ is real and diagonal, and $T$ is upper triangular.

In Julia, one can obtain the $QR$ factored form of a matrix using \texttt{qr} from the standard library \texttt{LinearAlgebra}. We will consider the pivoted QR, \texttt{qr(M, Val\{true\})}, based on LAPACK's \texttt{geqp3} in our analysis. A factorization into $UDT$ form is provided by functions \texttt{udt} and \texttt{udt!} in \texttt{StableDQMC.jl}.

\subsection{Benchmarks}

\subsubsection{Accuracy}
\begin{figure}
	\centering
	\includegraphics[width=0.8\textwidth]{figures/decomp_comparison_simple.pdf}
	\caption{\textbf{Comparison of matrix decompositions} to heal the numerical instabilities in the calculation of the time slice matrix chain product $B_M B_{M-1} \cdots B_1$ for model~\eqref{eq:model}. The QR and Jacobi SVD singular values seem to lie on top of the exact ones whereas regular SVD and divide-and-conquer SVD show large deviations at low temperatures $\beta \gtrsim 25$ ($\Delta \tau = 0.1$).}
	\label{fig:decomp_comparison_simple}
\end{figure}

Supplementing our general considerations above, we test the correctness of the matrix product stabilization procedure with respect to varying the concrete SVD and QR factorization algorithms. Fig.~\ref{fig:decomp_comparison_simple} shows the logarithmic singular values of the time slice matrix chain $B(\beta, 0)$ as a function of inverse temperature $\beta$ obtained from employing different matrix decompositions. Clearly, the accuracy of the computed singular values shows a strong dependence on the chosen factorization algorithm. While the results for the QR decomposition and Jacobi SVD seem to fall on top of the exact result, we observe large deviations for the conventional and D\&C SVD algorithms. This effect is particularly pronounced at low temperatues, $\beta \gtrsim 25$. The fact that small scales are lost while large scales remain relatively accurate can be understood from LAPACK's SVD errorbounds \cite{errorbounds}.

\subsubsection{Efficiency}
Turning to computational efficiency, we illustrate runtime cost measurements for all considered SVD variants relative to the QR decomposition in Fig.~\ref{fig:benchmark_decomps}. We find that both the conventional SVD and Jacobi SVD are an order of magnitude slower than the QR decomposition while only the divide-and-conquer algorithm shows comparable speed. Among the SVD variants, the Jacobi SVD is the most costly by a large margin, having about twice the runtime of the conventional SVD for small system sizes.

\begin{figure}
	\centering
	\includegraphics[width=0.8\textwidth]{figures/benchmark_decomps.pdf}
	\caption{\textbf{Computational efficiency of matrix decompositions}. Shown is the runtime cost of the factorization of a complex matrix of size $N \times N$ by means of various SVD algorithms relative to the QR decomposition. \label{fig:benchmark_decomps}}
\end{figure}

\section{Stabilization: equal-time Green's function} \label{sec:etgf}

Similar to the considerations above, a naive computation of the Green's function according to Eq.~\eqref{etgf} is potentially unstable because of numerical roundoff errors due to finite machine precision. In particular, adding the identity to the ill-conditioned slice matrix chain $B(\tau, 0)$ will generally wash out small singular values and will lead to a non-invertible result such that the subsequent inversion in Eq.~\eqref{etgf} is ill-defined. This clearly prohibits a safe calculation of the equal-time Green’s function and asks for numerical stabilization techniques.

\subsection{Inversion schemes}
As for the time slice matrix products in Eq.~\eqref{eq:stabmult}, the strategy will be to keep exponentially spread scales as separated as possible. A straightforward scheme \cite{Santos2003, Assaad2002a} (\texttt{inv\_one\_plus}) to add the unit matrix and perform the inversion of $\mathbb{1} + B(\tau, 0)$ in a stabilized manner is given by

\begin{align}
	G &= [\mathbb{1} + UDX]^{-1} \nonumber \\
	&= [U\underbrace{(U^\dagger X^{-1} + D)}_{udx}X]^{-1} \nonumber\\
	&= [(Uu)d(xX)]^{-1} \label{eq:inversion}\\
	&= U_r D_r X_r \nonumber,
\end{align}
where $U_r = (xX)^{-1}$, $D_r = d^{-1}$, and $X_r = (Uu)^{-1}$. Here, the intermediate addition (parentheses in the second line of \eqref{eq:inversion}) of unit scales and singular values is separated from the unitary rotations such that $U^\dagger X^{-1}$ only acts as a clean cutoff,
\begin{align}
U^\dagger X^{-1} + D = \U + \D = \left[
\begin{array}{cccc}
\slarge & \ssmall & \ssmall & \ssmall \\
\ssmall & \smedium & \ssmall & \ssmall \\
\ssmall & \ssmall & \ssmall & \ssmall \\
\ssmall & \ssmall & \ssmall & \ssmall \\
\end{array}
\right].
\end{align}

As we will demonstrate for the time-displaced Green's function in Sec.~\ref{sec:TDGF}, a procedure like Eq.~\eqref{eq:inversion} based on a single intermediate decomposition will still fail to give accurate results for some of the matrix decompositions. For this reason, we consider another stabilization procedure put forward by Loh \textit{et al.} \cite{Loh2005, Loh1989} (\texttt{inv\_one\_plus\_loh}), in which one initially separates the scales of the diagonal matrix $D$ into two factors $D_p = \max(D, 1)$ and $D_m = \min(D, 1)$,
\begin{align}
D_p = \Dp, \quad D_m = \Dm,
\end{align}
and performs two intermediate decompositions,
\begin{align}
	G &= [\mathbb{1} + UDX]^{-1} \nonumber\\
	&= [\mathbb{1} + UD_mD_pX]^{-1} \nonumber\\
	&= [(X^{-1} D_p^{-1} + U D_m) D_p X]^{-1} \label{eq:inversion_loh} \\
	&= X^{-1} \underbrace{[D_p^{-1} (\underbrace{X^{-1} D_p^{-1} + UD_m}_{udx})^{-1}]}_{udx} \nonumber \\
	&= U_r D_r X_r, \nonumber
\end{align}
where $U_r = X^{-1}u$, $D_r = d$, and $X_r = x$.
%\begin{align}
%	X^{-1} D_p^{-1} + UD_m &\approx \left[
%	\begin{array}{cccc}
%	\ssmall + \ssmall/\slarge & \ssmall + \ssmall/\smedium & \ssmall& \ssmall\stiny + \ssmall \\
%	\ssmall + \ssmall/\slarge & \ssmall + \ssmall/\smedium & \ssmall& \ssmall\stiny + \ssmall \\
%	\ssmall + \ssmall/\slarge & \ssmall + \ssmall/\smedium & \ssmall& \ssmall\stiny + \ssmall \\
%	\ssmall + \ssmall/\slarge & \ssmall + \ssmall/\smedium & \ssmall& \ssmall\stiny + \ssmall \\
%	\end{array}
%	\right] \approx \left[
%	\begin{array}{cccc}
%	\ssmall + \ssmall/\slarge & \ssmall + \ssmall\stiny & \ssmall& \ssmall\stiny + \ssmall \\
%	\ssmall + \ssmall/\slarge & \ssmall + \ssmall\stiny & \ssmall& \ssmall\stiny + \ssmall \\
%	\ssmall + \ssmall/\slarge & \ssmall + \ssmall\stiny & \ssmall& \ssmall\stiny + \ssmall \\
%	\ssmall + \ssmall/\slarge & \ssmall + \ssmall\stiny & \ssmall& \ssmall\stiny + \ssmall \\
%	\end{array}
%	\right] \\
%D_p^{-1} (udx)^{-1} &\approx \left[
%\begin{array}{cccc}
%\ssmall/\slarge & \ssmall/\slarge & \ssmall/\slarge & \ssmall/\slarge \\
%\ssmall/\smedium & \ssmall/\smedium & \ssmall/\smedium & \ssmall/\smedium \\
%\ssmall & \ssmall & \ssmall & \ssmall \\
%\ssmall & \ssmall & \ssmall & \ssmall \\
%\end{array}
%\right].
%\end{align}

\subsection{Benchmarks}\label{sec:benchmarks}

We assess how different matrix decomposition algorithms perform in stabilized computations of $B(\beta, 0)$, the Green's function $G$ both with respect to accuracy and speed. All results are for the Hubbard model, Eq.~\ref{eq:model}, with the interaction strength set to $U=0$ and $U=1$ (alpha transparent in all plots)

\subsubsection{Accuracy}

\begin{figure}
	\centering
	\begin{subfigure}{0.98\textwidth}
		\centering
		\includegraphics[width=.65\textwidth]{figures/legend.pdf}
		\vspace{5pt}
	\end{subfigure}
	\begin{subfigure}{0.48\textwidth}
	\includegraphics[width=\textwidth]{figures/accuracy_greens_svd_regularinv.pdf}
\caption{\texttt{inv\_one\_plus}, Eq.~\eqref{eq:inversion} \label{fig:greens_accuracy_regularinv}}
	\end{subfigure}%
	\hspace{10pt}
	\begin{subfigure}{0.48\textwidth}
	\includegraphics[width=\textwidth]{figures/accuracy_greens_svd_loh.pdf}
\caption{\texttt{inv\_one\_plus\_loh}, Eq.~\ref{eq:inversion_loh} \label{fig:greens_accuracy_loh}}
	\end{subfigure}
	\vspace{5pt}
	\caption{\textbf{Accuracy of the Green's function} obtained from stabilized computations using the listed matrix decompositions and inversion schemes. Shown is ${\Delta G = \log(\textrm{max}(\textrm{abs}(G - G_{\textrm{exact}})))}$ for $U=0$ (solid) and $U=1$ (alpha transparent).}
\end{figure}

Starting from a stabilized computation of $B(\beta,0)$, Sec.~\ref{sec:stabilization}, we calculate the equal-time Green's function by performing the inversion according to the schemes outlined above and varying the applied matrix factorization. In Fig.~\ref{fig:greens_accuracy_regularinv} we show our findings for \texttt{inv\_one\_plus}, Eq.~\ref{eq:inversion}, where we have taken the maximum absolute difference between the computed and the exact Green's function as an accuracy measure. At high temperatures and for $U=0$, we observe that all decompositions lead to a good approximation of $G_{\textrm{exact}}$ with an accuracy close to floating point precision. However, when turning to lower temperatures the situations changes dramatically. We find that only the QR decomposition and the Jacobi SVD deliver the Green's function reliably. Compared to the other SVD variants, which fall behind by a large margin and fail to reproduce the exact result, they consistently show about optimal accuracy even in the presence of interactions. As displayed in Fig.~\ref{fig:greens_accuracy_loh}, switching to the inversion scheme \texttt{inv\_one\_plus\_loh}, Eq.~\ref{eq:inversion_loh}, does generally improve the accuracy but deviations of the regular SVD and D\&C SVD remain of the order of unity at the lowest temperatures.

These findings suggest that only the QR decomposition and the Jacobi SVD, irrespective of the inversion procedure, are suited for computing the equal time Green's function in DQMC reliably.

%\begin{figure}
%	\centering
%	\begin{subfigure}{0.48\textwidth}
%		\includegraphics[width=\textwidth]{figures/accuracy_det_svd_regularinv.pdf}
%		\caption{\texttt{inv\_one\_plus}, Eq.~\ref{eq:inversion}. \label{fig:det_accuracy_regularinv}}
%	\end{subfigure}%
%	\hspace{10pt}
%	\begin{subfigure}{0.48\textwidth}
%		\includegraphics[width=\textwidth]{figures/accuracy_det_svd_loh.pdf}
%		\caption{\texttt{inv\_one\_plus\_loh}, Eq.~\ref{eq:inversion_loh} \label{fig:det_accuracy_loh}}
%	\end{subfigure}
%	\vspace{5pt}
%	\caption{\textbf{Accuracy of the determinant} of the equal-time Green's function obtained from stabilized computations using the listed matrix decompositions and the inversion schemes. Shown are results for $U=0$ (solid) and $U=1$ (alpha transparent).\label{fig:det_accuracy}}
%\end{figure}
%
%In Fig.~\ref{fig:det_accuracy} we show the logarithm of the relative error of the Green's function determinant, relevant in the Metropolis acceptance\footnote{For local updates on can generally avoid full calculations of Green's function determinants by exploiting locality and performing a Laplace expansion since only ratios of determinants appear in Eq.~\ref{eq:metropolis}. In fact, in an optimal implementation the computation of the acceptance rate is $O(1)$ rather than $O(N^3)$.}, obtained for all combinations of matrix decompositions and inversion schemes. Both the QR decomposition and the Jacobi SVD lead to accurate results for all accessed temperatures, irrespective of the employed inversion scheme. The other two SVD based methods on the other hand show large relative deviations for both \texttt{inv\_one\_plus} and \texttt{inv\_one\_plus\_loh}.

%\begin{table}[b]
%	\bgroup
%	\def\arraystretch{1.25}%  1 is the default, change whatever you need
%	\begin{tabular}{ccc}
%		$\beta = 5$ & Method & $\textrm{max}(\textrm{abs}(G - G_{\textrm{exact}}))$ \\
%		\hline
%		& QR & 6.1e-16 \\
%
%		& SVD, simple & 8.1e-14 \\
%
%		& SVD, D\&C & 6.9e-14 \\
%
%		& SVD, Jacobi & 6.1e-14 \\
%		\hline
%		& \\
%		$\beta = 40$ & Method & $\textrm{max}(\textrm{abs}(G - G_{\textrm{exact}}))$ \\
%		\hline
%
%		& QR & 1.2e-15 \\
%
%		& SVD, simple & 4.7e-1 \\
%
%		& SVD, D\&C & 4.9e-1 \\
%
%		& SVD, Jacobi & 3.1e-4 \\
%		\hline
%	\end{tabular}
%	\egroup
%	\caption{\textbf{Accuracy of the Green's function} obtained from stabilized computations using the listed matrix decompositions at high ($\beta = 5$) and low temperatures ($\beta = 40$). \label{tbl:greens_accuracy}}
%\end{table}

%\begin{table}
%	\bgroup
%	\def\arraystretch{1.25}%  1 is the default, change whatever you need
%	\begin{tabular}{cccc}
%		$\beta = 5$ & Method & $\Delta_\textrm{abs} (\textrm{det} G)$ & $\Delta_\textrm{rel}(\textrm{det} G)$ \\
%		\hline
%		& QR & 2.3e-27 & 1.1e-15 \\
%
%		& SVD, simple & 3.2e-26 & 1.5e-14 \\
%
%		& SVD, D\&C & 1.8e-25 & 8.5e-14 \\
%
%		& SVD, Jacobi & 1.2e-25 & 5.7e-14 \\
%		\hline
%		& \\
%		$\beta = 40$ & Method & $\Delta_\textrm{abs}(\textrm{det} G)$ & $\Delta_\textrm{rel}(\textrm{det} G)$ \\
%		\hline
%
%		& QR & 4.1e-107 & 7.7-15 \\
%
%		& SVD, simple & 4.5e-90 & 1.0e-0 \\
%
%		& SVD, D\&C & 2.6e-93 & 1.0e-0 \\
%
%		& SVD, Jacobi & 8.8e-97 & 1.7e-4 \\
%		\hline
%	\end{tabular}
%	\egroup
%	\caption{\textbf{Accuracy of the determinant} of the equal-time Green's function obtained from stabilized computations using the listed matrix decompositions at high ($\beta = 5$) and low temperatures ($\beta = 40$). \label{tbl:det_accuracy}}
%\end{table}

\subsubsection{Efficiency}

Independent of the employed inversion scheme, matrix decompositions are expected to be the performance bottleneck in the Green's function computation. We hence expect the speed differences apparent in Fig.~\ref{fig:benchmark_decomps} to dominate benchmarks of the full Green's function calculation as well. This anticipation is qualitatively confirmed in Fig.~\ref{fig:greens_benchmark}, which shows the runtime cost of the Green's function computation for both inversion schemes and all matrix decompositions relative to the QR. While the divide-and-conquer SVD is in the same ballpark as the QR decomposition the other SVD algorithms fall behind by a large margin (an order of magnitude) for both inversion procedures. Importantly, this apparent runtime difference is increasing with system size. The observation that the relative slowdown factor is larger for the inversion scheme \texttt{inv\_one\_plus\_loh} can be understood from the fact that it requires one additional intermediate matrix decomposition.

Combined with the accuracy results these findings suggest that among the QR decomposition and the Jacobi SVD, which are found to be reliable in both inversion schemes, the QR decomposition has a significantly lower runtime cost and is therefore to be preferred in DQMC.

\begin{figure}
	\centering
	\begin{subfigure}{0.98\textwidth}
		\centering
		\includegraphics[width=.65\textwidth]{figures/legend_dashed.pdf}
		\vspace{5pt}
	\end{subfigure}
	\begin{subfigure}{0.48\textwidth}
			\includegraphics[width=\textwidth]{figures/benchmark_fact_inv_one_plus.pdf}
		\caption{\texttt{inv\_one\_plus}, Eq.~\eqref{eq:inversion}. \label{fig:greens_benchmark_regularinv}}
	\end{subfigure}%
	\hspace{10pt}
	\begin{subfigure}{0.48\textwidth}
		\includegraphics[width=\textwidth]{figures/benchmark_fact_inv_one_plus_loh.pdf}
		\caption{\texttt{inv\_one\_plus\_loh}, Eq.~\eqref{eq:inversion_loh}. \label{fig:greens_benchmark_loh}}
	\end{subfigure}
	\vspace{5pt}
	\caption{\textbf{Efficiency of the stabilized Green's function calculation} using the listed matrix decompositions and inversion schemes. Shown are results for $U=0$.\label{fig:greens_benchmark}}
\end{figure}

%\afterpage{\clearpage}
\section{Stabilization: time-displaced Green's function}\label{sec:TDGF}

In this section, we turn to the stabilization of time-displaced Green's functions. While these are not required in the basic DQMC, that is for generating a representative Markov chain of configurations, they are central to the measurement of time-displaced correlation functions such as pairing correlations and the superfluid density\cite{Santos2003, Scalapino1993}.

First, we generalize our definition of the Green's function, Eq.~\ref{etgf}, to include imaginary time $\tau = l \Delta \tau$,
\begin{align}
	G(\tau) = \langle c_{i} c_{j}^\dagger \rangle_{\phi_l} = \left[ \mathbb{1} + B_{l-1}\dots B_1 B_M \dots B_l \right]^{-1}. \label{eq:etgf_tau}
\end{align}
Note that $ G \equiv G_1 = G_{M+1} = \left[ \mathbb{1} + B_M \dots B_l \right]^{-1} $ due to fermionic boundary conditions. The time displaced Green's function can now be defined in terms of the time ordering operator $T$ as \cite{Santos2003, Assaad2002a}
\begin{align*}
	G_{l_1,l_2} &\equiv G(\tau_1, \tau_2) \equiv \langle T c_i(\tau_1) c_j^\dagger(\tau_2) \rangle_\varphi.
\end{align*}
More explicitly, this reads
\begin{align}
	G(\tau_1, \tau_2) = \begin{cases}
		B_{l_1} \cdots B_{l_2 + 1} G_{l_2 + 1}, &\tau_1 > \tau_2,\\
		- \left( 1 - G_{l_1 + 1} \right) \left( B_{l_2} \cdots B_{l_1 + 1}\right)^{-1}, &\tau_2 > \tau_1.
	\end{cases} \label{eq:tdgf_propagate}
\end{align}
In principle, this gives us a prescription for how to calculate $G(\tau_1, \tau_2)$ from the equal time Green's function discussed in Sec.~\ref{sec:etgf}. However, when $|\tau_1 - \tau_2|$ is large a naive calculation of the matrix products in Eq.~\ref{eq:tdgf_propagate} would be numerically unstable, as seen in Sec.~\ref{sec:stabilization}. More importantly, by first calculating the equal-time Green's function one already mixes (and looses) scale information in the last recombination step, ${G = UDX}$. It is therefore  advantageous to compute the time-displaced Green's function directly as (assuming $\tau_1>\tau_2$ for simplicity)
\begin{align}
	G(\tau_1, \tau_2) &= B_{l_1} \cdots B_{l_2 + 1} G_{l_2 + 1}\\
	&= B_{l_1} \cdots B_{l_2 + 1} \left[ \mathbb{1} + B_{l_2}\dots B_1 B_M \dots B_{l2+1} \right]^{-1}\\
	&= \left[\underbrace{B_{l_2+1}^{-1} \cdots B_{l_1}^{-1}}_{U_LD_LX_L} + \underbrace{B_{l_2}\dots B_1 B_M \dots B_{l1+1}}_{U_RD_RX_R}\right]^{-1} \\
	&= \left[U_L D_L X_L + U_R D_R X_R\right]^{-1}.
\end{align}
%\begin{align}
%	G(\tau_1, \tau_2) = (U_L D_L X_L + U_R D_R X_R)^{-1}.
%\end{align}

\subsection{Inversion schemes}
As for the equal time Green's function (Sec.~\ref{sec:etgf}), one must be careful to keep the scales in $D_L$ and $D_R$ separated when performing the summation and inversion to avoid unnecessary floating point roundoff errors.
As a first explicit procedure, we consider a simple generalization of Eq.~\ref{eq:inversion} (\texttt{inv\_sum}),
\begin{align}
	G(\tau_1, \tau_2) &= [U_L D_L X_L + U_R D_R X_R]^{-1} \nonumber\\
	&= [U_L \underbrace{(D_L X_L X_R^{-1} + U_L^\dagger U_R D_R)}_{udx} X_R ]^{-1} \nonumber\\
	&= [(U_L u) d^{-1} (x X_R)]^{-1} \label{eq:inversion_tdgf}\\
	&= U_r D_r X_r, \nonumber
\end{align}
where $U_r = (x X_R)^{-1}$, $D_r = d^{-1}$, and $X_r = (U_L u)^{-1}$.

Analogously, we can generalize the scheme by Loh \textit{et al.}\cite{Loh2005}, Eq.~\ref{eq:inversion_loh}, in which we split the scales into matrix factors $D_m = \min(D, 1)$, $D_p = \max(D, 1)$, (\texttt{inv\_sum\_loh})
\begin{align}
	G(\tau_1, \tau_2) &= [U_L D_L X_L + U_R D_R X_R]^{-1} \nonumber\\
	&= [U_L D_{Lm} D_{Lp} X_L + U_R D_{Rm} D_{Rp} X_R]^{-1} \nonumber\\
	&= \left[U_L D_{Lp} \underbrace{\left( \dfrac{D_{Lm}}{D_{Rp}} X_L X_R^{-1} + U_L^\dagger U_R \dfrac{D_{Rm}}{D_{Lp}} \right)}_{udx} X_R D_{Rp} \right]^{-1} \nonumber\\
	&= X_R^{-1} \underbrace{\dfrac{1}{D_{Rp}} [udx]^{-1} \dfrac{1}{D_{Lp}}}_{udx} U_L^\dagger \label{eq:inversion_tdgf_loh} \\
	&= U_r D_r X_r, \nonumber
\end{align}
where $U_r = X_R^{-1} u$, $D_r = d$, and $X_r = x U_L^\dagger$.

We note that Hirsch\cite{Hirsch1988, Santos2003} has proposed an alternative method for computing the time-displaced Green's function based on a space-time matrix formulation of the problem. Although this technique has been successfully deployed in many-fermion simulations we won't discuss it here because of its subpar computational scaling: for a system composed of $N$ lattice sites, fermion flavors $f$, and imaginary time extent $M$ one has to invert (naively a $\mathcal{O}(x^3)$ operation) a matrix which takes up $\mathcal{O}((NMf)^2)$ memory. Similarly, Assaad \textit{et al.}\cite{Assaad2002a} have described an approach to compute both equal time and time-displaced Green's functions in one step. However, this requires to work with extended matrices of doubled linear dimension compared to the regular Green's functions.

\begin{figure}
	\centering
	\begin{subfigure}{0.98\textwidth}
		\centering
		\includegraphics[width=.65\textwidth]{figures/legend.pdf}
		\vspace{5pt}
	\end{subfigure}
	\begin{subfigure}{0.48\textwidth}
		\includegraphics[width=\textwidth]{figures/accuracy_tdgf_U0.pdf}
		\caption{\texttt{inv\_sum}, Eq.~\eqref{eq:inversion_tdgf}. \label{fig:tdgf_accuracy_regularinv}}
	\end{subfigure}%
	\hspace{10pt}
	\begin{subfigure}{0.48\textwidth}
		\includegraphics[width=\textwidth]{figures/accuracy_tdgf_loh_U0.pdf}
		\caption{\texttt{inv\_sum\_loh}, Eq.~\eqref{eq:inversion_tdgf_loh}. \label{fig:tdgf_accuracy_loh}}
	\end{subfigure}
	\vspace{5pt}
	\caption{\textbf{Accuracy of the time-displaced Green's function} obtained from stabilized computations using the listed matrix decompositions and inversion schemes. Shown is $\Delta G = \log(\textrm{max}(\textrm{abs}(G(\tau, 0) - G_{\textrm{exact}}(\tau, 0))))$ for $\beta=40$. \label{fig:tdgf_accuracy}}
\end{figure}

\subsection{Benchmarks}
\subsubsection{Accuracy}

In Fig.~\ref{fig:tdgf_accuracy}, we show the logarithmic, maximal, absolute deviation of the time-displaced Green's function from the arbitrary precision result as a function of time-displacement $\tau$ at inverse temperature $\beta=40$. Focusing on the inversion scheme \texttt{inv\_sum} first, Fig.~\ref{fig:tdgf_accuracy_regularinv}, both regular and D\&C SVD clearly fail to capture the intrinsic scales sufficiently and errors much beyond floating point precision are visible. Although the QR decomposition systematically leads to equally or more accurate results for all considered imaginary times, it fails to be reliable at long times $\tau\sim\beta/2$ (the Green's function is anti-periodic in $\tau$). Only the Jacobi-method based SVD leads to an accurate Green's function at all imaginary times.

Switching to the inversion scheme \texttt{inv\_sum\_loh}, the situation changes, as illustrated in Fig.~\ref{fig:tdgf_accuracy_loh}. While the non-Jacobi SVDs still have insufficient accuracy, the result for the QR decomposition improves dramatically compared to \texttt{inv\_sum} and leads to stable Green's function estimates up to floating point precision along the entire imaginary time axis.

Similar to our findings for the equal-time Green's function, this suggests that only the Jacobi SVD and the QR decomposition are reliable for DQMC. The latter, however, must be paired with the \texttt{inv\_sum\_loh} inversion scheme to be reliable. To the best of our knowledge, this finding has not yet been mentioned in the literature.

\subsubsection{Efficiency}\label{sec:tdgfbenchmark}

Finally, we compare the computational runtime cost associated with both stable approaches: the Jacobi SVD combined with the regular inversion and the QR decomposition paired with \texttt{inv\_sum\_loh}. As shown in Fig.~\ref{fig:tdgf_benchmark}, we find that the latter is consistently faster for all considered system sizes. In relative terms, the SVD based approach falls behind by at least a factor of two and seems to display inferior scaling with the chain length $N$.
This indicates that QR decompositions should be preferred over singular value decompositions when computing time-displaced Green's functions, in spite of the need to use an inversion scheme of higher complexity.

\begin{figure}
	\centering
	\includegraphics[width=0.8\textwidth]{figures/benchmark_jacobi_vs_qr_tdgf.pdf}
	\caption{\textbf{Efficiency of the time-displaced Green's function} obtained from stabilized computations using the QR decomposition in combination with the inversion scheme \texttt{inv\_sum\_loh}, Eq.~\eqref{eq:inversion_tdgf_loh} and the Jacobi SVD paired up with the regular inversion scheme \texttt{inv\_sum}, Eq.~\eqref{eq:inversion_tdgf}. Measurements are taken over multiple runs at $\tau = \beta/2 = 20$. The inset show the slowdown of the Jacobi SVD relative to the QR based approach. \label{fig:tdgf_benchmark}}
\end{figure}

\section{Discussion}\label{sec:discussion}

Numerical instabilities arise naturally in finite machine-precision quantum Monte Carlo simulations of many-fermion systems. Different schemes based on matrix factorizations have been proposed to handle the intrinsic exponential scales underlying these instabilities in a stable manner. As we have shown in this manuscript, these techniques can have vastly different accuracy and efficiency rendering them more or less suited for determinant quantum Monte Carlo simulations.

For our test system, the one-dimensional Hubbard model, we find that conventional and divide-and-conquer based singular value decompositions consistently fail to produce accurate equal time and time-displaced Green's functions, in particular at the lowest considered temperatures, $\beta \sim 40$. Only the QR decomposition and the Jacobi-method based SVD are able to stabilize the computation and produce reliable results. Importantly, we observe that in case of the time-displaced Green's function, the QR must be paired with an inversion scheme put forward by Loh \textit{et al.}\cite{Loh1989}, an observation that, to the best of our knowledge, has not been mentioned in the literature before. No such qualitative dependence on the inversion procedure is observed for the Jacobi SVD.

In terms of efficiency, we find that the QR decomposition outperforms the Jacobi SVD by a large margin when utilized in stable Green's function computations. While expected from the fact that QR decompositions are computationally cheaper than SVDs, this difference is even apparent when the QR factorization is employed in a inversion scheme of higher complexity involving two (rather than one) matrix decompositions.

In summary, our empirical assessment demonstrates that, among the considered matrix factorizations and algorithms, the QR decomposition paired with the appropriate inversion schemes is the optimal stabilization method for Green's function calculations in DQMC as it is both fast and stable.

Finally, let us remark that the performance of any stabilization scheme is affected by the condition number of the involved imaginary time slice matrices and is therefore, in principle, model (parameter) dependent. We have not investigated this dependence in this manuscript, but have found qualitatively similar results for a spin-fermion model for antiferromagnetic metallic quantum criticality in Ref.~\cite{Bauer2020}. We therefore believe that our major conclusions bear some universality and can serve as a useful guide.

\section*{Acknowledgements}
We thank Peter Bröcker, Yoni Schattner, Snir Gazit, and Simon Trebst for useful discussions and Frederick Freyer for identifying a few typos in an early version of this manuscript.

\begin{appendix}

\section{Inversion schemes for time slice matrix stacks}

In practical DQMC implementations one typically stores intermediate decomposed time slice matrix products $B(\tau_1, \tau_2)$ in a stack for reuse in future equal time Green's function calculations \cite{Assaad2002a, Santos2003, Broecker2016}. In this case, the inversion schemes in Eq.~\eqref{eq:inversion} needs to be prefixed by a stable procedure to combine two elements $U_L, D_L, X_L$ and $U_R, D_R, X_R$ from the stack, corresponding to $B_{l-1}\dots B_1$ and $B_M \dots B_l$ in Eq.~\ref{eq:etgf_tau}. Below we describe the latter for both matrix decompositions considered in the main text\footnote{In \texttt{StableDQMC.jl}, Julia's multiple dispatch will automatically select the correct method based on the number of provided UDT factorizations.}.

\subsection{QR/UDT}

(\texttt{StableDQMC.jl}: \texttt{inv\_one\_plus(::UDT, ::UDT)})
\begin{align}
	G &= \left[\mathbb{1} + U_L D_L T_L \left( U_R D_R T_R \right)^\dagger \right]^{-1}\nonumber \\
	&= \left[\mathbb{1} + U_L \underbrace{\left( D_L \left( T_L T_R^\dagger \right) D_R \right)}_{udt} U_R^\dagger \right]^{-1} \\
	&= \left[\mathbb{1} + U D T \right]^{-1},\nonumber
\end{align}
with $U=U_Lu$, $D=d$, and $T=tU_R^\dagger$. This $UDT$ factorization may then be substituted into Eq.~\eqref{eq:inversion}.

\subsection{SVD}

(\texttt{StableDQMC.jl}: \texttt{inv\_one\_plus(::SVD, ::SVD)})
\begin{align}
G &= \left[\mathbb{1} + U_L D_L V_L^\dagger U_R D_R V_R^\dagger \right]^{-1} \nonumber \\
&= \left[\mathbb{1} + U_L \underbrace{\left( D_L \left( V_L^\dagger U_R \right) D_R \right)}_{udv^\dagger} V_R^\dagger \right]^{-1} \\
&= \left[\mathbb{1} + U D V^\dagger \right]^{-1},\nonumber
\end{align}
with $U=U_Lu$, $D=d$, and $V=v V_R$. This SVD factorization may then be substituted into Eq.~\eqref{eq:inversion}.

\end{appendix}

%\section{About references}
%Your references should start with the comma-separated author list (initials + last name), the publication title in italics, the journal reference with volume in bold, start page number, publication year in parenthesis, completed by the DOI link (linking must be implemented before publication). If using BiBTeX, please use the style files provided  on \url{https://scipost.org/submissions/author_guidelines}.
%in order to simplify the production of your paper.

% TODO:
% Provide your bibliography here. You have two options:

% FIRST OPTION - write your entries here directly, following the example below, including Author(s), Title, Journal Ref. with year in parentheses at the end, followed by the DOI number.
%\begin{thebibliography}{99}
%\bibitem{1931_Bethe_ZP_71} H. A. Bethe, {\it Zur Theorie der Metalle. i. Eigenwerte und Eigenfunktionen der linearen Atomkette}, Zeit. f{\"u}r Phys. {\bf 71}, 205 (1931), \doi{10.1007\%2FBF01341708}.
%\bibitem{arXiv:1108.2700} P. Ginsparg, {\it It was twenty years ago today... }, \url{http://arxiv.org/abs/1108.2700}.
%\end{thebibliography}

% SECOND OPTION:
% Use your bibtex library
% \bibliographystyle{SciPost_bibstyle} % Include this style file here only if you are not using our template
\bibliography{references}

\nolinenumbers

\end{document}


\section{Updates: 03/16/2020}%
\label{sec:updates_2020_03_16}
% \subsection{Current \& Future Items}
\begin{itemize}
  \item \textbf{\uline{Symplectic}}
    \begin{itemize}
      \item With $\alpha_{S_{i}} = 0$ for \(i = x, v\), (only \(T_{i}, Q_{i}\)
        terms), this should be trivial.
      % \item With only \(T_{i}, Q_{i}\) terms, this should be trivial.
      \item Even with \(S\) terms, the formulas are pretty simple, and should
        be easy to verify once we can be sure \(S = 0\) is working properly.
      \item For now it is probably still enough to only consider \(T_{x}\)
        since we mainly want to understand the source of the bias.
      \item Once we understand that we can add more terms and work on tuning it
        better.
    \end{itemize}
  \item \textbf{\uline{Reversibility}}
    \begin{itemize}
      \item We can check that the trained sampler is reversible using the
        following procedure:
        \begin{enumerate}
          \item Randomly choose the direction \(d\) to create the initial state
            \(\xi = (x, v, d)\).
          \item Run the dynamics and flip the direction:
            \begin{equation}
              \mathbf{FL} \xi = \mathbf{F}\xi^{\prime} = (x^{\prime}, v^{\prime}, - d)
            \end{equation}
            % \item Flip the direction and run the dynamics in the reverse direction:
          \item Run the dynamics and flip the direction again:
            \begin{equation}
              \mathbf{FL} \xi^{\prime} = \mathbf{F} \xi^{\prime\prime} =
              (x^{\prime\prime}, v^{\prime\prime}, d)
              % \mathbf{F}\xi^{\prime} = (x^{\prime}, v^{\prime}, -d) \rightarrow
              % \xi^{\prime\prime} = (x^{\prime\prime}, v^{\prime\prime}, -d)
            \end{equation}
          \item Check the difference:
            \begin{align}
              \delta x &= x - x^{\prime\prime} \\
              \delta v &= v - v^{\prime\prime}
            \end{align}
        \end{enumerate}
%
      % \item Run the dynamics according to the following two procedures.
      %   \begin{enumerate}
      %     \item Run the dynamics backwards then forwards to get
      %       \(\xi^{\mathrm{fb}}\).
      %       \begin{equation}
      %         \xi^{\mathrm{fb}} = \mathbf{FL}^{\mathrm{f}}\mathbf{FL}^{\mathrm{b}}\xi
      %       \end{equation}
      %     \item Run the dynamics forwards then backwards to get
      %       \(\xi^{\mathrm{bf}}\).
      %       \begin{equation}
      %         \xi^{\mathrm{bf}} = \mathbf{FL}^{\mathrm{b}}\mathbf{FL}^{\mathrm{f}}\xi
      %       \end{equation}
      %   \end{enumerate}
      % \textbf{\item \color{red}{(AI1):}} Look for outliers in the reversibility
      %   using the \(\max\) of the differences, \(\xi^{fb} - \xi\) and
      %   \(\xi^{bf} - \xi\).
      \item \textbf{\color{red}{(AI2):}} The violations should get worse as the volume
        increases, so it is probably best to formulate the network in terms of
        the group variables \((\cos\phi_{\mu}(i),\sin\phi_{\mu}(i)) =
        e^{i\phi_{\mu}(i)}\) instead of the algebra \(\phi_{\mu}(i)\).
        This would easily apply to higher groups as well. For \(U(1)\), this
        doubles the size of the inputs, but should work better overall.
      \item Having the network treat \(0\) and \(2\pi\) differently is a
        potential source of reversibility violation, though it may be small in
        practice.  Continue looking for evidence of this.
    % \begin{todolist}
      \color{red}{\item Using the above criterion it was observed that the sampler
        does indeed violate reversibility, as shown in
      Fig~\ref{fig:reverse_diffs}.}\color{black}
    \item In order to identify the root cause of the reversibility violation, I
      am currently working on stepping through the sub-updates of the dynamics
      code and checking reversibility at each step.
      \begin{figure}[htpb!]
        \centering
        \includegraphics[width=\textwidth]{updates_2020_03_16/reverse_diffs_traceplot.pdf}
        \caption{Traceplot of average differences \(\langle \delta x\rangle,
          \langle \delta v\rangle\) demonstrating the reversibility
        violation.}%
        \label{fig:reverse_diffs}
      \end{figure}
      % \color{blue}{\item[\done] \textbf{{(AI1, AI2):}}} Having looked through
      %   existing inference data (specifically, those models for which \(\delta \phi_{P}
      %   > 0\)), there don't appear to be any violations in reversibility.
    % \end{todolist}
  \end{itemize}
  \item \textbf{\uline{Ergodicity}}
    \begin{itemize}
      \item Technically, this may be an autocorrelation problem, but in
        practice it is difficult to distinguish them.
        \begin{itemize}
          \item This may be the main problem. L2HMC seems to only learn certain
            types of updates, but fails at general mixing.
        \end{itemize}
      \item \textbf{\color{red}{(AI3)}} In principle, there should be some initial conditions that
        give a negative bias, and some positive. Mapping out the bias distribution
        for different seeds may help confirm this, though the distribution may not
        be symmetric, and could have a large tail on one side.
      \item \textbf{\color{red}{(AI4)}} Alternating HMC with L2HMC is an idea to fix this.
      \item If L2HMC mixes poorly on its own, then we may need to run mostly HMC.\@
      \item Perhaps, running \(N\) updates of HMC and \(1\) L2HMC, for varying \(N\),
        will avoid the L2HMC bias and show an improvement over either alone.
      \item We can periodically switch between L2HMC and generic HMC during
        inference to see if there is any benefit. An example of this procedure
        is shown in Fig~\ref{fig:mixed_samplers}.
    \end{itemize}
    \begin{figure}
      \centering
      \includegraphics[width=0.95\textwidth]{figures/updates_2020_03_16/mix_samplers.pdf}
      \caption{Results obtained by periodically mixing between L2HMC and HMC
      during inference.}%
      \label{fig:mixed_samplers}
    \end{figure}
    \begin{todolist}
      \color{blue}{\item[\done] \textbf{{(AI3, AI4):}}} No evidence in recent
      data showing a negative bias, although it may be that the distribution is
      \emph{very} one sided. Continuing to look for counter-examples.
    \end{todolist}
  \item \textbf{\uline{Training}}
    \begin{itemize}
      \item \textbf{\color{red}{(AI5)}} Run more tests with current code at \(T_{x} = 1\) to
        further map this distribution and see how the average distance,
        \(\delta x_{\mathrm{out}}\) and acceptance, \(A(\xi^{\prime}|\xi)\)
        % (\texttt{accept\(_\)prob})
        after training correlate with the bias.
      \item \textbf{\color{red}{(AI6)}} The initial tests of alternate loss functions seemed
        promising. Continue to explore alternate loss functions.
        \begin{itemize}
          \item Maybe \(|\delta x| * A^{2}(\xi^{\prime}|\xi)\)
          \item Or others that avoid anything we can hopefully determine from
            \textbf{\color{red}{(AI5)}} (or other tests), that correlate with increased bias.
        \end{itemize}
      \item Try running on \(O(2)\) model in 1D compare against Yannicks
        results.
    \end{itemize}
  \item \textbf{\uline{Code/scaling}}
    \begin{itemize}
      \item Longer-term, we want to write L2HMC in QEX for Aurora. This would
        potentially be faster and easier to scale up.
        \begin{itemize}
          \item However, the full dense layer won't scale well, and would need
            to be replaced eventually.
        \end{itemize}
      \item \textbf{\color{red}{(AI7)}} One option is to replace the dense
        weight matrix with a low-rank approximation. This could be investigated
        by taking the SVD of the weight matrix and looking at how the singular
        values fall off.
        \begin{itemize}
          \item Could also replace the weight matrix after training (on the
            dense matrix), then run inference on a low-rank approximation to
            see how it compares.
        \end{itemize}
      \item \textbf{\color{red}{(AI8)}} We could experiment with a few other
        variants that would be easy to implement and scale well, like a local
        connection (stencil) in combination with a low-rank fully connected
        layer. This would be relatively easy to implement in QEX.\@
      \item In the 2D case, the singular value decomposition (SVD) of a weight
        matrix \(W\) in the network can be written as:
        \begin{equation}
          W = USV^{H}
        \end{equation}
        where \(S = \mathrm{s}\) contains the singular values of \(W\) and \(U,
        V^{H}\) are unitary. The rows of \(V^{H}\) are the eigenvectors of
        \(W^{H}W\) and the columns of \(U\) are the eigenvectors of \(WW^{H}\).
        In both cases the corresponding (possibly non-zero) eigenvalues are
        given by \(s^{2}\).
      \item The amount of overall variance explained by the \(i^{th}\) pair of
        SVD vectors is given by \(s_{i}^2 / \sum_{j} s_{j}^{2}\), where
        \(s_{j}\) are the singular values (diagonal of \(S\)).
    \end{itemize}
    % \begin{todolist}
    %   \color{blue}{\item[\done] \textbf{{(AI7, AI8):}}} We can look at the
    % \end{todolist}
    \begin{figure}[htpb!]
      \centering
      \begin{subfigure}[b]{0.5\textwidth}
        \centering
        \includegraphics[width=\textwidth]{updates_2020_03_16/xnet_x_layer_svd.pdf}
      \end{subfigure}%
      ~
      \begin{subfigure}[b]{0.5\textwidth}
        \centering
        \includegraphics[width=\textwidth]{updates_2020_03_16/vnet_v_layer_svd.pdf}
      \end{subfigure}
      \caption{Plots showing the percent of the total variance explained by the
      \(i^{th}\) singular value for two layers in our neural network.}
    \end{figure}
\end{itemize}

\subsection{Simplifying the \texorpdfstring{$x$}{x}-update}% {{{
\label{subsec:simplify_x_update}%
% To determine which part of the algorithm is responsible for biasing the average
% plaquette we have been trying to simplify the network/algorithm as much as
% possible by removing individual items and
One technique for determining the source of the bias in the plaquette is to
remove/simplify individual components in the network, and see if any of these
changes fixes the issue.

One possible simplification that we have begun to explore is to combine the two
\(x\) sub-updates into a single update by explicitly setting the ``site''
masks in Eq.~\ref{eq:forward_update}~\ref{eq:backward_update} be
%
\begin{align}
  m^{t} &= [1, 1, \ldots, 1]\\
  \bar{m}^{t} &= [0, 0, \ldots, 0]
\end{align}
%
for \(t = 1, \ldots, N_{\mathrm{LF}}\).

In this case, the forward update (\(d = 1\)), becomes:
%
\begin{align}
  x^{\prime} &= x\odot\exp{\left(\varepsilon S_{x}(\zeta_2)\right)}
    + \varepsilon\left(v^{\prime}\odot\exp{\left(
      \varepsilon Q_{x}(\zeta_{2})\right)} + T_{x}(\zeta_{2})
    \right)\\
  x^{\prime\prime} &= x^{\prime}\\
                   &= x\odot\exp{\left(\varepsilon S_{x}(\zeta_2)\right)} +
                   \varepsilon\left(v^{\prime}\odot\exp{\left( \varepsilon
                   Q_{x}(\zeta_{2})\right)} + T_{x}(\zeta_{2}) \right)\\
\end{align}
%
And similarly for the backwards (d=-1) update:
%
\begin{align}
  x^{\prime} &= x\\
  x^{\prime\prime} &= {\left[x^{\prime}
      - v^{\prime}\odot\varepsilon\left(\exp(\varepsilon Q_{x}(\zeta_{3}))
      + T_{x}(\zeta_{3})\right)\right]\odot\exp\left({
          -\varepsilon S_{x}(\zeta_{3})
      }\right)}
\end{align}
% }}}

\begin{figure}[htpb]
  \centering
  \includegraphics[width=\textwidth]{zero_masks/original_masks}
  \caption{Using the original (randomly) assigned site masks, we see the bias
  is present.}
\end{figure}
%
\begin{figure}[htpb]
  \centering
  \includegraphics[width=\textwidth]{zero_masks/zero_masks2}
  \caption{Using the combined \(x\) sub-updates with \(m^{t} = [1, 1, \ldots,
    1]\) and \(\bar{m}^{t} = [0, 0, \ldots, 0]\). We see that the bias has
    slightly improved, although both the acceptance rate and tunneling rate
  appear to suffer dramatically.}
\end{figure}

%
% \clearpage
% \section{TODO:}
% \begin{todolist}
% \item[\done] Export saved weights from tensorflow as arrays to make the inference run
%   portable and library independent.
% \item Try with $O(2)$ model in $D = 1$.
% \item Calculate the relative probabilities for the topological sectors using
%   Eq.~\ref{eq:rel_prob}.
% % \item Get a reasonable estimate of the \textbf{integrated autocorrelation}
% %   time.
% % \item Write unit tests for dynamics engine.
% \end{todolist}
% From Yannick's calculation, the relative probabilities for the topological
% sectors is given by
% \begin{equation}
%   P = \exp\left[-\frac{\beta}{2}{\frac{{(2\pi)}^2 n^2}{N_{s} N_{t}}}\right]


\subsubsection{\tt ipape}
\label{benchmarkipape}

{\tt ipape} is a synthetic AKS benchmark that performs
a character-character matrix product({\tt \qM\qand\qdot\qeq\qtran\0M}),
on arrays of shape {\tt [ 2000, 2000]}.
We expect \wlf and \awlf to perform equally well on this
benchmark.


\documentclass[]{report}

\usepackage{fixltx2e,fix-cm}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{url}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{courier}
\usepackage{color}
\usepackage{listings}

\usepackage{tikz} % for diagram in first Fig.
\usetikzlibrary{shapes,arrows}
\tikzstyle{block} = [draw, rectangle, fill=white, minimum height=2em, minimum width=4em]
\tikzstyle{sum} = [draw, circle, node distance=2em]
\tikzstyle{input} = []
\tikzstyle{node} = [coordinate, node distance=2em]

\definecolor{matgreen}{rgb}{0,0.6,0}
\definecolor{matpurple}{rgb}{0.625,0.125,0.9375}
\definecolor{gray}{rgb}{0.5,0.5,0.5}

\lstset{language=Matlab,
   keywords={break,case,catch,classdef,continue,else,elseif,end,for,function,global,if,otherwise,persistent,properties,return,switch,try,while},
   basicstyle=\ttfamily\footnotesize,
   keywordstyle=\color{blue},
   commentstyle=\color{matgreen},
   stringstyle=\color{matpurple},
   numbers=left,
   numberstyle=\color{gray},
   stepnumber=1,
   numbersep=10pt,
   backgroundcolor=\tiny\color{white},
   tabsize=4,
   showspaces=false,
   showstringspaces=false}
\lstnewenvironment{shortlisting}{
\lstset{language=Matlab,
   keywords={break,case,catch,classdef,continue,else,elseif,end,for,function,global,if,otherwise,persistent,properties,return,switch,try,while},
   basicstyle=\ttfamily,
   keywordstyle=\color{blue},
   commentstyle=\color{matgreen},
   stringstyle=\color{matpurple},
   numbers=none,
   numberstyle=\tiny\color{gray},
   stepnumber=1,
   numbersep=10pt,
   backgroundcolor=\color{white},
   tabsize=4,
   showspaces=false,
   showstringspaces=false}
}{}

\def\d{{\mathbf d}}
\def\x{{\mathbf x}}
\def\K{{\mathbf K}}

\begin{document}

\title{Kernel Adaptive Filtering Toolbox for Matlab}
\author{Steven Van Vaerenbergh}
\maketitle

\begin{abstract}
The Kernel Adaptive Filtering Toolbox is a benchmarking toolbox to evaluate and compare kernel adaptive filtering algorithms in Matlab. Kernel adaptive filtering algorithms are online techniques suitable for nonlinear filtering, prediction, tracking and regression. This toolbox contains implementations of all major kernel adaptive filtering algorithms, and tools to measure and compare the performance, memory usage, speed and other variables.
\end{abstract}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\tableofcontents

\chapter{Introduction}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

This document is a work in progress.

\section{Goals of the toolbox}
The goals of this toolbox are twofold:
\begin{enumerate}
\item To provide a repository for Matlab implementations of kernel adaptive filtering algorithms;
\item To provide tools that allow to compare all aspects of the different algorithms
\end{enumerate}
A list of the included algorithms and tools can be found in Chapter \ref{chap:algos}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{First use}

\section{Installation}

\begin{enumerate}
\item Download the toolbox and run the \verb"installation.m" file in the root folder.
\item Type \verb"savepath" to save the changes to the path.
\end{enumerate}

\section{Directories included in the toolbox}

\begin{itemize}
\item \verb"data/" contains data sets.
\item \verb"demo/" contains demos and test files.
\item \verb"lib/" contains the algorithm libraries and utilities.
\end{itemize}

\section{Octave / Matlab pre-2008a}

This toolbox uses the \verb"classdef" command which is not supported in Matlab pre-2008a and not yet in Octave. The older 0.x versions of this toolbox do not use classdef and can therefore be used with all versions of Matlab and Octave. \verb"http://sourceforge.net/projects/kafbox/files/"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Kernel adaptive filtering}

\section{Framework for online kernel methods}
% summary: growing functional representation

Online learning methods update their solution iteratively. In the standard online learning framework, each iteration consists of several trials \cite{wiki:online_machine_learning}. During the $n$-th iteration:
\begin{enumerate}
\item The algorithm first receives an input datum, $\x_n$;
\item Then, it calculates the estimated output $\hat y_n$ corresponding to this datum.
\item The true outcome $y_n$ is made available shortly thereafter, which enables the algorithm to calculate the loss $L(\cdot)$ incurred on the data pair $(\x_n,y_n)$;
\item Finally, the solution is updated.
\end{enumerate}

 A typical setup for online system identification with a kernel-based method is depicted in Fig. \ref{ork:fig:diagram}. It represents an unknown nonlinear system, whose input data $\x_n$ and response $y_n$ (including additive noise $r_n$) can be measured at different time steps, and an adaptive kernel-based algorithm, which is used to identify the system's response.

\begin{figure}[h]
\centering
\begin{tikzpicture}[auto, node distance=2cm,>=latex]
    \node [input, name=input] {$\mathbf{x}_n$};
    \node [node, right of=input] (node1) {};
    \draw (input) -- (node1);
    \node [block, right of=node1] (system) {Nonlinear system};
    \draw [->] (node1) -- (system);
    \node [sum, right of=system, xshift=7em] (plus) {};
    \draw [->] (system) -- node[pos=0.8, below] {$+$} node {$f(\mathbf{x}_n)$} (plus);
    \node [input, name=noise, above of=plus, node distance=3em] {$r_n$};
    \draw [->] (noise) -- node[pos=0.8, right] {$+$}(plus);

    \node [block, below of=system, yshift=1em] (transformation) {$\mathcal{X}\rightarrow \mathcal{H}$};
    \draw [->] (node1) |- (transformation);
    \node [node, right of=transformation, xshift=3em] (node2) {};
    \draw (transformation) -- (node2);
    \node [block, right of=node2] (filter) {$\boldsymbol{\mathbf{\alpha}}_n$};
    \draw [->] (transformation) -- node {$\kappa(\mathbf{x}_n,\cdot)$} (filter);
    \node [sum, right of=filter, xshift=4em] (minus) {};
    \draw [->] (filter) -- node[pos=0.9, below] {$-$} node {$\hat y_n$} (minus);
    \draw [->] (plus) -| node[pos=0.9, right] {$+$} node {$y_n$} (minus);
    \node [input, right of=minus, xshift=-2em] (output) {$e_n$};
    \draw [->] (minus) -- (output);

    \node [block, below of=filter, yshift=1em] (algorithm) {Adaptive algorithm};
    \draw [->] (node2) |- (algorithm);
    \draw [->] (minus) |- node {$e_n$} (algorithm);
    \node [node, above of=algorithm, yshift=1em] (node3) {};
    \draw (algorithm) -- (node3);
    \node [node, above of=node3, xshift=1.5em, yshift=1.5em] (node4) {};
    \draw [->] (node3) -- (node4);

    \node [block, right of=node2] (filter2) {$\boldsymbol{\mathbf{\alpha}}_n$}; % repeat to fix z-index
\end{tikzpicture}
%\centerline{\includegraphics[height=150pt]{chapters/chapter_ork/fig/diagram.eps}}
\caption{Kernel-based adaptive system identification. Figure adapted from \cite{parreira2012stochastic}.}
\label{ork:fig:diagram}
\end{figure}

Online algorithms should be capable of operating during extended periods of time, processing large amounts of data. Kernel methods rely on a functional representation that grows as the amount of observations increases. A na\"ive implementation of an online kernel method will therefore require growing computational resources during operation, leading to performance issues once the available memory is insufficient to store the training data or once the computations for one update take more time than the interval between incoming data \cite{kivinen2004online}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Kernel adaptive filtering algorithms}

Kernel adaptive filtering algorithms are online regression algorithms. While early kernel adaptive filtering algorithms had a clear connection to classical adaptive filters, such as LMS, RLS and KAPA filters, later implementations explored Gaussian Process and projections based algorithms.

The different kernel adaptive filtering algorithms proposed in the literature vary in several aspects:
\begin{itemize}
\item The type of filter they relate to: typically LMS, RLS or KAPA;
\item The computational complexity w.r.t. the amount of elements in memory, $M$: typically $\mathcal{O}(M)$ or $\mathcal{O}(M^2)$;
\item The approach to building a sparse dictionary and to possible prune data from it afterwards;
\item The convergence speed;
\item The ability to employ multiple kernels and learn their respective weights.
\end{itemize}
The algorithm profiler described in Chapter \ref{sec:profiler} allows to compare these different aspects empirically.

\section{Algorithm code structure}

Since all algorithms share the same structure, they are coded in a common framework. In particular:
\begin{itemize}
\item Each algorithm is contained in a single file in the \verb"/lib" folder.
\item Each algorithm is implemented as an object in matlab, using the \verb"classdef" syntax.
\item Each algorithm has two basic public methods: \verb"evaluate" and \verb"train".
The object code contains only one iteration of the algorithm. The for-loop over the time index that governs the online operation should go in an external script.
\end{itemize}

Furthermore, efforts are made to make the code
\begin{enumerate}
\item As human-readable as possible, in the first place;
\item Short and structured, in the second place.
\end{enumerate}

As far as possible, algorithm structure should follow the pseudocode from the corresponding publication. And, finally, the following best practices for scientific computing are taken into account \cite{wilson2014best}:
\begin{itemize}
\item Variable naming should correspond to the nomenclature used in the corresponding publication whenever possible;
\item Comments should be used sparingly. Document the design and purpose of the code rather than its mechanics.
\end{itemize}

\section{Template for kernel adaptive filtering algorithms}

\lstinputlisting{../lib/kafbox_template.m}

\section{Performing online learning}

Each kernel adaptive filtering algorithm is implemented as a Matlab class. To perform online learning with an algorithm, first define its options:
\begin{shortlisting}
options = struct('nu',1E-4,'kerneltype','gauss','kernelpar',32);
\end{shortlisting}
Next, create an instance of the filter. E.g., for an instance of the KRLS algorithm that uses the ALD criterion run:
\begin{shortlisting}
kaf = aldkrls(options);
\end{shortlisting}
One iteration of training is performed by feeding one input-output data pair to the filter:
\begin{shortlisting}
kaf = kaf.train(x,y);
\end{shortlisting}
The outputs for one or more test inputs are evaluated as follows:
\begin{shortlisting}
Y_test = kaf.evaluate(X_test);
\end{shortlisting}

\section{Example learning script: time-series prediction}

\lstinputlisting{../demo/demo_prediction.m}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Algorithm profiler}
\label{sec:profiler}

The algorithm profiler is a tool that allows to compare the trade-off between cost and performance between several algorithm configurations and/or several algorithms.

A demo configuration script is given in the file\\ \verb"demo/demo_profiler_prediction_lorenz.m".

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{List of included algorithms}
\label{chap:algos}

Matlab implementations of the following algorithms are included in the toolbox:

\begin{itemize}
\item Approximate Linear Dependency Kernel Recursive Least-Squares (ALD-KRLS) \cite{engel2004kernel}.
\item Sliding-Window Kernel Recursive Least-Squares (SW-KRLS) \cite{vanvaerenbergh2006sliding}.
\item Naive Online Regularized Risk Minimization Algorithm (NORMA) \cite{kivinen2004online}.
\item Kernel Least-Mean-Square (KLMS) \cite{liu2008kernel}.
\item Fixed-Budget Kernel Recursive Least-Squares (FB-KRLS) \cite{vanvaerenbergh2010fixed}.
\item Kernel Recursive Least-Squares Tracker (KRLS-T) \cite{vanvaerenbergh2012kernel}.
\item Quantized Kernel Least Mean Squares (QKLMS) \cite{chen2012quantized}.
\item Random Fourier Feature Kernel Least Mean Square (RFF-KLMS) algorithm \cite{singh2012random}.
\item Extended Kernel Recursive Least Squares (EX-KRLS) \cite{liu2009extended}.
\item Gaussian-Process based estimation of the parameters of KRLS-T \cite{vanvaerenbergh2012estimation}.
\item Kernel Affine Projection (KAP) algorithm with Coherence Criterion \cite{richard2009online}.
\item Kernel Normalized Least-Mean-Square (KNLMS) algorithm with Coherence Criterion \cite{richard2009online}.
\item Recursive Least-Squares algorithm with exponential weighting (RLS) \cite{sayed2003fundamentals}.
\item Multikernel Normalized Least Mean Square algorithm with Coherence-based Sparsification (MKNLMS-CS) \cite{yukawa2012multikernel}.
\item Parallel HYperslab Projection along Affine SubSpace (PHYPASS) algorithm \cite{takizawa2013efficient}.
\item Fixed-budget kernel least mean squares (FB-KLMS) algorithm \cite{rzepka2012fixed}.
\item Leaky Kernel Affine Projection Algorithm (LKAPA, including KAPA-1 and KAPA-3) and Normalized Leaky Kernel Affine Projection Algorithm (NLKAPA, including KAPA-2 and KAPA-4) \cite{liu2008kernel}.
\item Kernel Affine Projection Subgradient Method (KAPSM) \cite{slavakis2008online}.
\item Kernel Least Mean Squares algorithm with Coherence-Sparsification criterion and L1-norm regularization (KLMS-CSL1) and with active L1-norm regularization (KLMS-CSAL1) \cite{gao2013kernel}.
\item  Mixture Kernel Least Mean Square (MxKLMS) algorithm \cite{pokharel2013mixture}.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{About}
\begin{itemize}
\item {\bf Name:} Kernel Adaptive Filtering Toolbox
\item {\bf Abbreviation:} KAFBOX
\item {\bf Punchline:} a Matlab benchmarking toolbox for kernel adaptive filtering
\item {\bf Author:} Steven Van Vaerenbergh
  \begin{itemize}
  \item Homepage: \url{http://gtas.unican.es/people/steven/}
  \item Email: \url{steven@gtas.dicom.unican.es}
  \item Affiliation: Advanced Signal Processing Group, Department of Communications Engineering, University of Cantabria, Spain.
  \end{itemize}
\item {\bf Collaborators:} Miguel Lazaro-Gredilla, Sohan Seth, Masahiro Yukawa, Masa-aki Takizawa, Osamu Toda, Dominik Rzepka, Pantelis Bouboulis
\item {\bf URL:} \url{https://github.com/steven2358/kafbox}
\item {\bf Citing:} Published reports of research using this code (or a modified version) should cite \cite{vanvaerenbergh2013comparative}.
\item {\bf Environments:} Matlab\footnote{\url{http://www.mathworks.com/products/matlab/}}.
\item {\bf Origin:} KAFBOX is a fork of Kernel Methods Toolbox (KMBOX) v0.6 (\url{http://sourceforge.net/p/kmbox/})
\item {\bf Extending the code:} Template files are provided to encourage external authors to include their own code into the toolbox. Contributions can be made through Github's fork and pull system.
\item {\bf License:} FreeBSD
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\bibliographystyle{alpha}
\bibliography{kaf,extra}

\end{document}


\documentclass{article}
\usepackage{graphicx}
\usepackage{epsfig}
\usepackage{amssymb,amsmath}
\usepackage{array}
\graphicspath{ {Desktop} }
\setlength{\parindent}{0pt}
\title{CTA200 2020 Project Summary}
\author{SURP Student Ethan Sun}
\date{May 19th, 2020}

\begin{document}

\maketitle
\section*{1. Introuction}
The purpose of this project is to rederive the self similar solutions for a shock wave propagating through a medium with varying densities and reproduce the results found by Akira Sakurai. The problem can be separated into two parts. The first part considers the stage before the shock wave arrives at the boundary, while the second one deals with the subsequent expansion as the gas expands into vacuum. I will describe the steps I took while deriving the solutions and present the results from numerical integrations in the following sections.

\section*{2. Derivation}
The problem can be simplified into an one-dimensional case since we are studying the phenomena close to the gas boundary.
\\
\centerline{\includegraphics[scale=0.6]{fig1.png}}
\\
\centerline{Figure 1}
\\
\bigbreak
\newpage
Figure 1 illustrates the proble in its first stage. The \textit{x}-axis points into the gas and is perpendicular to the boundary, \textit{x} = 0, where the initial density $\rho_0(\textit{x})$ = 0. The location of the shock wave is \textit{x} = $\textit{X}(\textit{t})$ and the shock wave travels in the negative \textit{x} direction with velocity \textit{U}:
\begin{equation}
U = \frac{dX}{dt}
\end{equation}
The velocity \textit{u}, pressure \textit{p}, and desity $\rho$ satisfy the following equations derived from one-dimensional continuity equation, Newton's First Law, and polytropic relation:
\begin{equation}
\frac{D\rho}{Dt} = -\rho\frac{\partial u}{\partial x}
\end{equation}
\begin{equation}
\frac{Du}{Dt} = -\frac{1}{\rho}\frac{\partial p}{\partial x}+F_0
\end{equation}
\begin{equation}
\frac{Dp\rho^{-\gamma}}{Dt} = 0
\end{equation}
where $F_0$ is body force per unit mass and $\gamma$ is the polytropic index. By applying the chain rule and substituting equation (2) into equation (4), equation (4) can be simplified into:
\begin{equation}
\frac{Dp}{Dt} = -\gamma p\frac{\partial u}{\partial x}
\end{equation}
Since our area of interest is close to the gas boundary, it is reasonable to assume the density decreases as a power of \textit {x}:
\begin{equation}
\rho_0(x) = k_1x^\alpha
\end{equation}
When $F_0$ is zero, the solutions to equations (2), (3), (5) together with equation (6) can be found as a similarity solution under the category of progressing wave described by Courant and Friedrichs:
\begin{equation}
    \begin{split}
        u &= U(x)f(\eta) \\
        p &= \rho_0(x)U^2(x)g(\eta) \\
        \rho &= \rho_0(x)h(\eta)
    \end{split}
\end{equation}
with
\begin{equation}
\eta = (\frac{X}{x})^{\lambda+1}, (0\leqslant \eta \leqslant 1)
\end{equation}
and
\begin{equation}
U(X) = k_2X^{-\lambda}
\end{equation}
\newpage
where we assumed this tyoe of similarity solution in order to fit the singularity at $X = 0$. We can then obtain the following ODEs for \textit{f}, \textit{g}, \textit{h} by substituting equations (7), (8), (9) into equations (2), (3), (5):
\begin{equation}
    \begin{split}
        (1-\eta f)f^{'}-\eta \frac{g^{'}}{h} &= \frac{\lambda}{1+\lambda}f^{2}-\frac{\alpha-2\lambda}{1+\lambda}\frac{g}{h}\\
        (1-\eta f)\frac{h^{'}}{h}-\eta f^{'} &= -\frac{\alpha-\lambda}{1+\lambda}f\\
        (1-\eta f)\frac{g^{'}}{g}-\lambda\eta f^{'} &= -\frac{\alpha-(\gamma+2)\lambda}{1+\lambda}f
    \end{split}
\end{equation}
$\lambda$ could be proved to be positive and therefore \textit{U} becomes large as \textit{X} becomes small. Since density decreases, the speed of sound $\textit c_0$ is also small around $\textit x=0$. This means the Mach number is large at this location and a strog shock wave shall present. According to the Rankine-Huginot conditions, the following equations could be written at $\textit x=X$:
\begin{equation}
    \begin{split}
        (u)_{x=X} &= \frac{2}{\gamma+1}U(X) \\
        (p)_{x=X} &= \frac{2}{\gamma+1}\rho_{0}(X)U^{2}(X) \\
        (\rho)_{x=X} &= \rho_{0}(X)\frac{\gamma+1}{\gamma-1}
    \end{split}
\end{equation}
We could then obtain the following boundary conditions for \textit{f}, \textit{g}, \textit{h} at $\eta=1$ by comparing equation (7) with (11):
\begin{equation}
    \begin{split}
        f(1) &= \frac{2}{\gamma+1} \\
        g(1) &= \frac{2}{\gamma+1} \\
        h(1) &= \frac{\gamma+1}{\gamma-1}
    \end{split}
\end{equation}
However, a solution cannot be found as it always breaks down at a certain point $\eta_0$, except for one special $\lambda$ value. A set of approximated values of $\lambda$ for certain $\alpha$ and $\gamma$ values were determined using Whiham's method:
\\
\centerline{\includegraphics[scale=0.75]{fig2.png}}
\\
\centerline{Table 1}
\\
\bigbreak
\newpage
We could then numerically integrate equation (10) from $\eta=1$ to $\eta=0$ as shown in Section 3. The results obtained from Sakurai's work are as follows:
\\
\centerline{\includegraphics[scale=0.9]{fig3.png}}
\\
\centerline{Table 2}
\\
\bigbreak
Finally, we could obtain the distributions of velocity, pressure, and density $\textit{u}_1$, $\textit{p}_1$, $\rho_1$ behind the shock wave when $\textit{t}=0$ by setting $\eta=0$ in equation (7) and substituting the $\textit{f}(0)$, $\textit{g}(0)$, $\textit{h}(0)$ terms with the values we just found:
\begin{equation}
    \begin{split}
        u_{1} &= U(x)f(0) = f(0)k_{2}x^{-\lambda} \\
        p_{1} &= \rho_{0}(x)U^{2}(x)g(0) = g(0)k_{1}k_{2}^{2}x^{\alpha-2\lambda} \\
        \rho_{1} &= \rho_{0}(x)h(0) = h(0)k_{1}x^{\alpha}
    \end{split}
\end{equation}
The next step would be applying the similarity solution method again to the region where $\textit{x}\leqslant0$, as gas expands into the vacuum. We could also use the results we found in equation (13) as a initial condition for the new ODEs. However, I was not able to go through the derivation for this part due to the limited time.

\section*{3. Numerical Integration}
The objective is to reproduce the integrations Sakurai performed in his work. I first simplified the ODEs in equation (10) into expressions of $\textit{f}^{'}$, $\textit{g}^{'}$ and $\textit{h}^{'}$. Ideally, the $scipy.integrate.solve\_bvp$ function should be able to solve this problem since we have all the boundary conditions at $\eta=1$.
\bigbreak
However, among the multiple sets of initial guess I experimented, none of them could yield a valid solution curve similar to the ones presented in Sakurai's work. It is very likely that I wasn't applying this solving function correctly. I then decided to integrate the ODEs as an initial value problem instead. At this point what I was trying to do was simply reproducing the integrated results, rather than finding the $\textit{f}(0)$, $\textit{g}(0)$ and $\textit{h}(0)$ values as indicated before. Therefore, I relied on the values from Table 2 as my initial conditions and applied the solving function $scipy.integrate.odeint$. The resulting plots are as follows.
\newpage
\centerline{\includegraphics[scale=0.8]{gamma 1.67.pdf}}
\centerline{Figure 2.1: $\gamma$=$\frac{5}{3}$, reproduced}
\centerline{\includegraphics[scale=0.9]{gamma 1.67.png}}
\centerline{Figure 2.2: $\gamma$=$\frac{5}{3}$, Sakurai}

\newpage
\centerline{\includegraphics[scale=0.7]{gamma 1.4.pdf}}
\centerline{Figure 3.1: $\gamma$=$\frac{7}{5}$, reproduced}
\centerline{\includegraphics[scale=0.8]{gamma 1.4.png}}
\centerline{Figure 3.2: $\gamma$=$\frac{7}{5}$, Sakurai}

\newpage
\centerline{\includegraphics[scale=0.7]{gamma 1.2.pdf}}
\centerline{Figure 4.1: $\gamma$=$\frac{6}{5}$, reproduced}
\centerline{\includegraphics[scale=0.7]{gamma 1.2.png}}
\centerline{Figure 4.2: $\gamma$=$\frac{6}{5}$, Sakurai}

\newpage
\section*{4. Comments}
As shown in the plots above, the reproduced results generally match Sakurai's findings. The resulting values at $\eta=1$ also agree with the boundary conditions we obtained earlier with  relatively small errors.
\bigbreak
Similar plots could be obtained for the second part of the problem, where the initial conditions are supposed to be found from the plots above when $\eta=0$.
\bigbreak
I was only able to partially finish this project, as further understanding of the theories used in Sakurai's work is much needed. On the other hand, I also need more coding practice in order to find the correct solutions in a more efficient manner. The current script I wrote need to be optimized in many ways, including finding a working method to solve the boundary value problem.
\bigbreak
This project would be a nice start for the rest of my summer research at CITA as the study of shock wave behaviors is a primary focus.

\end{document}