\chapter{Gas Flows and Turbulence}
\label{ch:turbulence}

\marginnote{\textbf{Suggested background reading:}
\begin{itemize}
\item \href{http://adsabs.harvard.edu/abs/2014arXiv1402.0867K}{Krumholz, M.~R. 2014, Phys.~Rep., 539, 49}, section 3.3 \nocite{krumholz14c}
\end{itemize}
\textbf{Suggested literature:}
\begin{itemize}
\item \href{http://adsabs.harvard.edu/abs/2013MNRAS.436.1245F}{Federrath, C. 2013, MNRAS, 436, 1245} \nocite{federrath13b}
\end{itemize}
}

This chapter covers the physics of turbulence in the cold interstellar medium. This will be something of a whirlwind tour, since turbulence is an entire research discipline unto itself. Our goal is to understand the basic statistical techniques used to describe and model interstellar turbulence, so that we will be prepared to apply them in the context of star formation.

\section{Characteristic Numbers for Fluid Flow}

\subsection{The Conservation Equations}

To understand the origins of turbulence, both in the ISM and more generally, we start by examining the equations of fluid dynamics and the characteristic numbers that they define. Although the ISM is magnetized, we will first start with the simpler case of an unmagnetized fluid. Fluids are governed by a series of conservation laws. The most basic one is conservation of mass:
\begin{equation}
\frac{\partial}{\partial t} \rho = -\nabla \cdot (\rho\mathbf{v}).
\end{equation}
This equation asserts that the change in mass density at a fixed point is equal to minus the divergence of density times velocity at that point. Physically, this is very intuitive: density at a point changes at a rate that is simply equal to the rate at which mass flows into or out of an infinitesimal volume around that point.

We can write a similar equation for conservation of momentum:
\begin{equation}
\label{eq:momentum}
\frac{\partial}{\partial t}(\rho \mathbf{v}) = -\nabla \cdot(\rho \mathbf{v v}) - \nabla P + \rho \nu \nabla^2 \mathbf{v}.
\end{equation}
Note that the term $\mathbf{vv}$ here is a tensor product. This is perhaps more clear if we write things out in index notation:
\begin{equation}
\frac{\partial}{\partial t}(\rho v_i) = -\frac{\partial}{\partial x_j} (\rho v_i v_j) - \frac{\partial}{\partial x_i} P
+ \rho \nu \frac{\partial}{\partial x_j}\left(\frac{\partial}{\partial x_j} v_i\right)
\end{equation}
The intuitive meaning of this equation can be understood by examining the terms one by one. The term $\rho\mathbf{v}$ is the density of momentum at a point. The term $\nabla \cdot(\rho \mathbf{v v})$ is, in analogy to the equivalent term in the conservation of mass equation, the rate at which momentum is advected into or out of that point by the flow. The term $\nabla P$ is the rate at which pressure forces acting on the fluid change its momentum. Finally, the last term, $\rho \nu \nabla^2 \mathbf{v}$, is the rate at which viscosity redistributes momentum; the quantity $\nu$ is called the kinematic viscosity.

The last term, the viscosity one, requires a bit more discussion. All the other terms in the momentum equation are completely analogous to Newton's second law for single particles. The viscous term, on the other hand, is unique to fluids, and does not have an analog for single particles. It describes the change in fluid momentum due to the diffusion of momentum from adjacent fluid elements. We can understand this intuitively: a fluid is composed of particles moving with random velocities in addition to their overall coherent velocity. If we pick a particular fluid element to follow, we will notice that these random velocities cause some of the particles that make it up to diffuse across its boundary to the neighboring element, and some particles from the neighboring element to diffuse into the one we are following. The particles that wander across the boundaries of our fluid element carry momentum with them, and this changes the momentum of the element we are following. The result is that momentum diffuses across the fluid, and this momentum diffusion is called viscosity.

Viscosity is interesting and important because it's the only term in the equation that converts coherent, bulk motion into random, disordered motion. That is to say, the viscosity term is the only one that is dissipative, or that causes the fluid entropy to change.

\subsection{The Reynolds Number and the Mach Number}
\label{ssec:reynolds_mach}

To understand the relative importance of terms in the momentum equation, it is helpful to make order of magnitude estimates of their sizes. Let us consider a system of characteristic size $L$ and characteristic velocity $V$; for a molecular cloud, we might have $L\sim 10$ pc and $V\sim 5$ km s$^{-1}$. The natural time scale for flows in the system is $L/V$, so we expect time derivative terms to be of order the thing being differentiated divided by $L/V$. Similarly, the natural length scale for spatial derivatives is $L$, so we expect spatial derivative terms to be order the quantity being differentiated divided by $L$. If we apply these scalings to the momentum equation, we expect the various terms to scale as follows:
\begin{equation}
\frac{\rho V^2}{L} \sim \frac{\rho V^2}{L} + \frac{\rho c_s^2}{L} + \rho \nu \frac{V}{L^2},
\end{equation}
where $c_s$ is the gas sound speed, and we have written the pressure as $P = \rho c_s^2$. Canceling the common factors, we get
\begin{equation}
1 \sim 1 + \frac{c_s^2}{V^2} + \frac{\nu}{VL}.
\end{equation}

From this exercise, we can derive two dimensionless numbers that are going to control the behavior of the equation. We define the Mach number and the Reynolds number as
\begin{eqnarray}
\mathcal{M} & \sim & \frac{V}{c_s} \\
\mathrm{Re} & \sim & \frac{LV}{\nu}.
\end{eqnarray}
The meanings of these dimensionless numbers are fairly clear from the equations. If $\mathcal{M} \ll 1$, then $c_s^2/V^2 \gg 1$, and this means that the pressure term is important in determining how the fluid evolves. In contrast, if $\mathcal{M} \gg 1$, then the pressure term is unimportant for the behavior of the fluid. In a molecular cloud,
\begin{equation}
c_s  =\sqrt{\frac{k_B T}{\mu m_H}} = 0.18 \left(\frac{T}{10\,\mathrm{K}}\right)^{1/2}\mbox{ km s}^{-1},
\end{equation}
where $\mu =2.33$ is the mean mass per particle in a gas composed of molecular hydrogen and helium in the usual cosmic abundance ratio of 1 He per 10 H atoms. Thus $\mathcal{M} = V/c_s \sim 20$, and we learn that pressure forces are unimportant.

The Reynolds number is a measure of how important viscous forces are. Viscous forces are significant for $\mathrm{Re} \sim 1$ or less, and are unimportant of $\mathrm{Re} \gg 1$. We can think of the Reynolds number as describing a characteristic length scale $L\sim \nu/V$ in the flow. This is the length scale on which diffusion causes the flow to dissipate energy. Larger scale motions are effectively dissipationless, while smaller scales ones are damped out by viscosity.

To estimate the Reynolds number in the molecular ISM, we must know its viscosity. For an ideal gas, the kinematic viscosity is $\nu=2\overline{u}\lambda$, where $\overline{u}$ is the RMS molecular speed (which is of order $c_s$) and $\lambda$ is the particle mean free-path. The mean free path is of order the inverse of cross-section times density, $\lambda \sim 1/(\sigma n) \sim [(1\mbox{ nm})^2 (100\mbox { cm}^{-3})]^{-1}\sim 10^{12}$ cm. Plugging this in then gives
$\nu \sim 10^{16}$ cm$^2$ s$^{-1}$ and $\mathrm{Re} \sim 10^9$. Viscous forces are clearly unimportant in molecular clouds.

The extremely large value of the Reynolds number immediately yields a critical conclusion: molecular clouds must be highly turbulent, because flows with $\mathrm{Re}$ of more than $\sim 10^3-10^4$ invariably are. Figure \ref{fig:reynoldsnum_nsf} illustrates this graphically from laboratory experiments.

\begin{figure}
\includegraphics[width=\linewidth]{reynoldsnum_nsf}
\caption[Comparison of flows at varying Reynolds numbers]{
\label{fig:reynoldsnum_nsf}
Flows at varying Reynolds number Re. In each panel, a fluid that has been dyed red is injected from the top into the clear fluid on the bottom. The fluids are a glycerin-water mixture, for which the viscosity can be changed by altering the glycerin to water ratio. By changing the viscosity and the injection speed, it is possible to alter the Reynolds number of the injected flow. The frames show how the flow develops as the Reynolds number is varied. This image is a still from the \href{https://www.youtube.com/playlist?list=PL0EC6527BE871ABA3}{National Committee for Fluid Mechanics Film Series}  \citep{taylor64a}, which, once you get past the distinctly 1960s production values, are a wonderful resource for everything related to fluids.
}
\end{figure}

\section{Modeling Turbulence}

We have remarkably little understanding of how turbulence actually works. However, we have developed some simple models and tools to describe it, and we will next explore those.

\subsection{Velocity Statistics}

One quantity of interest in a turbulent medium is the structure of the velocity field. How does the velocity change from point to point?  In a turbulent medium velocity fluctuates in time and space, and so the best way to proceed is to study those fluctuations statistically. Many statistical tools exist to characterize turbulent motions, and many are used in astrophysics, but we will stick to a few of the simpler ones. We will also make two simplifying assumptions. First we assume that the turbulence is homogenous, in the sense that the turbulent motions vary randomly but not systematically, with position in the fluid. Second, we assume that it is isotropic, so that turbulent motions do not have a preferred directions. Neither of these are likely to be strictly true in a molecular cloud, particularly the second, since large-scale magnetic fields provide a preferred direction, but we will start with these assumptions and relax them later.

Let $\mathbf{v}(\mathbf{x})$ be the velocity at position $\mathbf{x}$ within some volume of interest $V$. To characterize how this varies with position, we define the autocorrelation function
\begin{equation}
A(\mathbf{r})\equiv \frac{1}{V} \int \mathbf{v}(\mathbf{x})\cdot \mathbf{v}(\mathbf{x}+\mathbf{r}) \, d\mathbf{x} \equiv \langle \vecv(\vecx) \cdot \vecv(\vecx+\vecr)\rangle,
\end{equation}
where the angle brackets indicate an average over all positions $\mathbf{x}$. Here, $A(0) = \langle|\vecv|^2\rangle$ is just the mean square velocity in the fluid. If the velocity field is isotropic, then clearly $A(\mathbf{r})$ cannot depend on the direction, and thus must depend only on $r=|\mathbf{r}|$. Thus $A(r)$ tells us how similar or different the velocities are at points separated by some distance $r$.

It is often more convenient to think about this in Fourier space than in real space, so rather than the autocorrelation function we often instead think about its Fourier transform. We define the Fourier transform of the velocity field in the usual way, i.e.,
\begin{equation}
\tilde{\vecv}(\veck) = \frac{1}{(2\pi)^{3/2}}\int \vecv(\vecx) e^{-i\veck\cdot\vecx} \,d\vecx.
\end{equation}
We then define the power spectrum
\begin{equation}
\Psi(\veck) \equiv | \tilde{\vecv}(\veck)|^2.
\end{equation}

Again, for isotropic turbulence, the power spectrum depends only on the magnitude of the wave number, $k=|\veck|$, not its direction, so it is more common to talk about the power per unit radius in $k$-space,
\begin{equation}
P(k) = 4\pi k^2 \Psi(k).
\end{equation}
This is just the total power integrated over some shell from $k$ to $k+dk$ in $k$-space. Note that Parseval's theorem tells us that
\begin{equation}
\int P(k)\, dk = \int | \tilde{\vecv}(\veck)|^2 \, d\veck = \int \vecv(\vecx)^2 \, d\vecx,
\end{equation}
i.e., the integral of the power spectral density over all wavenumbers is equal to the integral of the square of the velocity over all space, so for a flow with constant density (an incompressible flow) the integral of the power spectrum just tells us how much kinetic energy per unit mass there is in the flow. The Wiener-Khinchin theorem also tells us that $P(\veck)$ is just the Fourier transform of the autocorrelation function,
\begin{equation}
\Psi(\veck) = \frac{1}{(2\pi)^{3/2}} \int A(\vecr) e^{-i\veck\cdot\vecr} \,d\vecr.
\end{equation}

The power spectrum at a wavenumber $k$ then just tells us what fraction of the total power is in motions at that wavenumber, i.e., on that characteristic length scale. The power spectrum is another way of looking at the spatial scaling of turbulence. It tells us how much power there is in turbulent motions as a function of wavenumber $k=2\pi/\lambda$. A power spectrum that peaks at low $k$ means that most of the turbulent power is in large-scale motions, since small $k$ corresponds to large $\lambda$. Conversely, a power spectrum that peaks at high $k$ means that most of the power is in small-scale motions.

The power spectrum also tells us about the how the velocity dispersion will vary when it is measured over a region of some characteristic size. Suppose we consider a volume of size $\ell$, and measure the velocity dispersion $\sigma_v(\ell)$ within it. Further suppose that the power spectrum is described by a power law $P(k)\propto k^{-n}$. The total kinetic energy per unit mass within the region is, up to factors of order unity,
\begin{equation}
\mathrm{KE} \sim \sigma_v(\ell)^2,
\end{equation}
but we can also write the kinetic energy per unit mass in terms of the power spectrum, integrating over those modes that are small enough to fit within the volume under consideration:
\begin{equation}
\mathrm{KE} \sim \int_{2\pi/\ell}^\infty P(k) \, dk \propto \ell^{n-1}.
\end{equation}
It therefore follows immediately that
\begin{equation}
\label{eq:vdisp}
\sigma_v = c_s \left(\frac{\ell}{\ell_s}\right)^{(n-1)/2},
\end{equation}
where we have normalized the relationship by defining the sonic scale $\ell_s$ as the size of a region within which the velocity dispersion is equal to the thermal sound speed of the gas.

\subsection{The Kolmogorov Model and Turbulent Cascades}

The closest thing we have to a model of turbulence is in the case of subsonic, hydrodynamic turbulence; the basic theory for that goes back to \citet{kolmogorov41a}.\footnote{An English translation of \citet{kolmogorov41a} (which is in Russian) can be found in \citet{kolmogorov91a}.} Real interstellar clouds are neither subsonic nor hydrodynamic (since they are strongly magnetized, as we discuss in Chapter \ref{ch:magnetic}), but this theory is still useful for understanding how turbulence works. Kolmogorov's theory of turbulence begins with the realization that turbulence is a phenomenon that occurs when Re is large, so that there is a large range of scales where dissipation is unimportant. It is possible to show by Fourier transforming Equation (\ref{eq:momentum}) that for incompressible motion transfer of energy can only occur between adjacent wavenumbers. Energy at a length scale $k$ cannot be transferred directly to some scale $k' \ll k$. Instead, it must cascade through intermediate scales between $k$ and $k'$.

This gives a simple picture of how energy dissipates in fluids. Energy is injected into a system on some large scale that is dissipationless, and it cascades down to smaller scales until it reaches a small enough scale that $\mbox{Re}\sim 1$, at which point dissipation becomes significant. In this picture, if the turbulence is in statistical equilibrium, such that is neither getting stronger or weaker, the energy at some scale $k$ should depend only on $k$ and on the rate of injection or dissipation (which are equal) $\psi$.

This allows us to make the following clever dimensional argument: $k$ has units of $1/L$, i.e., one over length. The power spectrum $P(k)$ has units of energy per unit mass per unit radius in $k$-space. The energy per unit mass is like a velocity squared, so it has units $L^2/T^2$, and this is divided by $k$, so $P(k)$ has units of $L^3/T^2$. The injection and dissipation rates $\psi$ have units of energy per unit mass per unit time, which is a velocity squared divided by a time, or $L^2/T^3$.

Since $P(k)$ is a function only of $k$ and $\psi$, we can write $P(k) = C k^\alpha \psi^\beta$ for some dimensionless constant $C$. Then by dimensional analysis we have
\begin{eqnarray}
\frac{L^3}{T^2} & \sim & L^{-\alpha} \left(\frac{L^2}{T^3}\right)^\beta \\
L^3 & \sim & L^{-\alpha+2\beta} \\
T^{-2} & \sim & T^{-3\beta}\\
\beta & = & \frac{2}{3} \\
\alpha & = & 2\beta - 3 = -\frac{5}{3}
\end{eqnarray}

This immediately tell us three critical things. First, the power in the flow varies with energy injection rate to the $2/3$ power. Second, this power is distributed such that the power at a given wavenumber $k$ varies as $k^{-5/3}$. This means that most of the power is in the largest scale motions, since power diminishes as $k$ increases. Third, if we now take this spectral slope and use it to derive the scale-dependent velocity dispersion from equation (\ref{eq:vdisp}), we find that $\sigma_v \propto \ell^{1/3}$, i.e., velocity dispersion increase with size scale as size to the $1/3$ power. This is an example of what is known in observational astronomy as a linewidth-size relation -- linewidth because the observational diagnostic we use to characterize velocity dispersion is the width of a line. This relationship tells us that larger regions should have larger linewidths, with the linewidth scaling as the 1/3 power of size in the subsonic regime.

The subsonic regime can be tested experimentally on Earth, and Kolmogorov's model provides an excellent fit to observations. Figure \ref{fig:kolmogorov} shows one example.

\begin{figure}
\includegraphics[width=\linewidth]{kolmogorov_champagne78}
\caption[Experimental power spectra for Kolmogorov turbulence]{
\label{fig:kolmogorov}
An experimentally-measured power spectrum for turbulence generated by an air jet. The $x$ axis is the wavenumber, and the open and filled points show the velocity power spectrum for the velocity components parallel and transverse to the stream, respectively. Credit: \citeauthor{champagne78a}, J.~Fluid.~Mech., 86, 67-108, 1978, reproduced with permission.
}
\end{figure}

\section{Supersonic Turbulence}

\subsection{Velocity Statistics}

We have seen that real interstellar clouds not only have $\mathrm{Re} \gg 1$, they also have $\mathcal{M} \gg 1$, and so the flows within them are supersonic. This means that pressure is unimportant on size scales $L \gg \ell_s$. Since viscosity is also unimportant on large scales (since $\mbox{Re} \gg 1$), this means that gas tends to move ballistically on large scales. On small scales this will produce very sharp gradients in velocity, since fast-moving volumes of fluid will simply overtake slower ones. Since the viscosity term gets more important on smaller scales, the viscosity term will eventually stop the fluid from moving ballistically. In practice this means the formation of shocks -- regions where the flow velocity changes very rapidly, on a size scale determined by the viscosity.\footnote{In real interstellar clouds the relevant viscosity is the magnetic one, as we shall see in Chapter \ref{ch:magnetic}.}

We expect that the velocity field that results in this case will look like a series of step functions. The power spectrum of a step function is a power law $P(k)\propto k^{-2}$. One can establish this easily from direct calculation. Let us zoom in on the region around a shock, so that the change in velocity on either side of the shock is small. The Fourier transform of $v$ in 1D is
\begin{equation}
\tilde{v}(k) = \frac{1}{\sqrt{2\pi}} \int v(x) e^{-i kx}\, dx
\end{equation}
The integral of the periodic function $e^{-ikx}$ vanishes for all periods in the regions where $v$ is constant. It is non-zero only in the period that includes the shock. The amplitude of $\int v(x) e^{-i kx}\, dx$ during that period is simply proportional to the length of the period, i.e., to $1/k$. Thus, $\tilde{v}(k)\propto 1/k$. It then follows that $P(k)\propto k^{-2}$ for a single shock. An isotropic system of overlapping shocks should therefore also look approximately like a power law of similar slope. This gives a velocity dispersion versus size scale $\sigma_v\propto \ell^{1/2}$, and this is exactly what is observed. Figure \ref{fig:polaris} shows an example.

\begin{marginfigure}
\includegraphics[width=\linewidth]{polaris_ossenkopf02}
\caption[Linewidth versus size in the Polaris Flare cloud]{
\label{fig:polaris}
Linewidth versus size in the Polaris Flare Cloud obtained from CO observations. Diamonds show the total measured velocity width within apertures of the size indicated on the $x$ axis, while triangles show the dispersion obtained by taking the centroid velocity in each pixel and measuring the dispersion of centroids. The three sets of points joined by lines represent measurements from three separate telescopes. Credit: \citeauthor{ossenkopf02a}, A\&A, 390, 307, 2002, reproduced by permission \copyright\,ESO.
}
\end{marginfigure}

Note that, although the power spectrum is only slightly different than that of subsonic turbulence ($-2$ versus about $-5/3$), there is really an important fundamental difference between the two regimes. Most basically, in Kolmogorov turbulence decay of energy happens via a cascade from large to small scales, until a dissipative scale is reached. In the supersonic case, on the other hand, the decay of energy is via the formation of shocks, and as we have just seen a single shock generates a power spectrum $\propto k^{-2}$, i.e., it non-locally couples many scales. Thus, in supersonic turbulence there is no locality in $k$-space. All scales are coupled at shocks.

\subsection{Density Statistics}

In subsonic flows the pressure force is dominant. Thus if the gas is isothermal, then the density stays nearly constant -- any density inhomogeneities are ironed out immediately by the strong pressure forces. In supersonic turbulence, on the other hand, the flow is highly compressible. It is therefore of great interest to ask about the statistics of the density field.

Numerical experiments and empirical arguments (but not fully rigorous proofs) indicate that the density field for a supersonically-turbulent, isothermal medium is well-described by a lognormal distribution,
\begin{equation}
\label{eq:denpdf}
p(s) = \frac{1}{\sqrt{2\pi \sigma_s^2}} \exp\left[-\frac{(s-s_0)^2}{2\sigma_s^2}\right],
\end{equation}
where $s=\ln(\rho/\overline{\rho})$ is the log of the density normalized to the mean density $\overline{\rho}$. This distribution describes the probability that the density at a randomly chosen point will be such that $\ln(\rho/\overline{\rho})$ is in the range from $s$ to $s+ds$. The median of the distribution $s_0$ and the dispersion $\sigma_s$ must be related to one another, because we require that
\begin{equation}
\overline{\rho} = \int p(s) \rho \, ds.
\end{equation}
With a bit of algebra, one can show that this equation is satisfied if and only if
\begin{equation}
s_0 = -\sigma_s^2/2.
\end{equation}

Instead of computing the probability that a randomly chosen point in space will have a particular density, we can also compute the probability that a randomly chosen mass element will have a particular density. This more or less amounts to a simple change of variables. Consider some volume of interest $V$, and examine all the material with density such that $\ln(\rho/\overline{\rho})$ is in the range from $s$ to $s+ds$. This material occupies a volume $dV = p(s) V$, and thus must have a mass
\begin{eqnarray}
dM & = & \rho p(s) \, dV \\
& = & \overline{\rho} e^s \cdot \frac{1}{\sqrt{2\pi \sigma_s^2}} \exp\left[-\frac{(s-s_0)^2}{2\sigma_s^2}\right]\,  dV \\
& = & \overline{\rho} \frac{1}{\sqrt{2\pi \sigma_s^2}} \exp\left[-\frac{(s+s_0)^2}{2\sigma_s^2}\right]\,  dV
\end{eqnarray}
It immediately follows that the mass PDF is simply
\begin{equation}
p_M(s) = \frac{1}{\sqrt{2\pi \sigma_s^2}} \exp\left[-\frac{(s+s_0)^2}{2\sigma_s^2}\right],
\end{equation}
i.e., exactly the same as the volume PDF but with the peak moved from $-s_0$ to $+s_0$. Physically, the meaning of these shifts is that the typical volume element in a supersonic turbulent field is at a density lower than the mean, because much of the mass is collected into shocks. The typical mass element lives in one of these shocked regions, and thus is at higher-than-average density. Figure \ref{fig:turbrender} shows an example of the density distribution produced in a numerical simulation of supersonic turbulence.
\begin{marginfigure}
\includegraphics[width=\linewidth]{turbrender_padoan99}
\caption[Volume rendering of the density field for supersonic turbulence]{
\label{fig:turbrender}
Volume rendering of the density field in a simulation of supersonic turbulence. The surfaces shown are isosurfaces of density. Credit: \citet{padoan99a}, \copyright\,AAS. Reproduced with permission.
}
\end{marginfigure}

The lognormal functional form is not too surprising, given the central limit theorem. Supersonic turbulence consists of an alternating series of shocks, which cause the density to be multiplied by some factor, and supersonic rarefactions, which cause it to drop by some factor. The result of multiplying a lot of positive density increases by a lot of negative density drops at random tends to produce a normal distribution in the multiplicative factor, and thus a lognormal distribution in the density.

This argument does not, however, tell us about the dispersion of densities, which must be determined empirically from numerical simulations. The general result of these simulations \citep[e.g.,][]{federrath13b} is that
\begin{equation}
\sigma_s^2 \approx \ln \left(1 + b^2 \mathcal{M}^2 \frac{\beta_0}{\beta_0+1}\right),
\end{equation}
where the factor $b$ is a number in the range $1/3-1$ that depends on how compressive versus solenoidal the velocity field is, and $\beta_0$ is the ratio of thermal to magnetic pressure at the mean density and magnetic field strength, something we will discuss further in Chapter \ref{ch:magnetic}.

In addition to the density PDF, there are higher order statistics describing correlations of the density field from point to point. We will defer a discussion of these until we get to models of the IMF in Chapter \ref{ch:imf_th}, where they play a major role.


In this chapter we are going to explain and show the full functionality of \suaca. For each available analysis we will show an example run and analyze the results. As we want to compare the different runs with each other we will use the following example code for each of them:

\begin{mdframed}[backgroundcolor=light-gray, roundcorner=10pt,leftmargin=1, rightmargin=1, innerleftmargin=15, innertopmargin=1,innerbottommargin=1, outerlinewidth=1, linecolor=light-gray]
\begin{lstlisting}[language={myLang}, basicstyle=\small, numbers=left]
   mov rax, 1
   cmp rcx, 0
   jne else
   add rbx, rax
   jmp end
else:
   add rbx, rax
end:
   add rbx, rbx
\end{lstlisting}
\end{mdframed}

For the sake of simplicity this code is only designed to be an example which contains two branches which both have a dependency on the previous and the following instruction.

\section{Throughput Analysis}
\label{sec:plain}

As mentioned above, the major use case of \iaca\ is analyzing an innermost loop. While \iaca\ will therefore always assume a loop and somehow determine its number of iterations, \suaca\ will give the user the option to choose them. For the following example \suaca\ considered $200$ loop iterations and the Sandy Bridge microarchitecture. We will do so for all future examples except stated otherwise. Now consider the following output which will demonstrate the basic values \suaca\ computes.

\begin{example}
    Block throughput: 2.34 cycles
    Block throughput with perfect front-end: 2.34 cycles
    Block throughput with infinitely usable ports: 2.00 cycles
    Block throughput without dependencies: 2.34 cycles
    Microops per cycle: 2.99

    Analysis for architecture: SNB

     Line  ||   Num   ||   had   || caused  ||            Used Ports
           ||   Uops  || to wait || to wait ||   0   ||   1   ||   2   ||   3   ||   4   ||   5   ||
    ------------------------------------------------------------------------------------------------
       0   ||    1    ||  15.1   ||  43.2   ||  0.0  ||  1.0  ||       ||       ||       ||       || mov rax, 0x1
       1   ||    1    ||  15.8   ||  32.3   ||  0.3  ||  0.3  ||       ||       ||       ||  0.3  || cmp rcx, 0x0
       2   ||    1    ||  16.5   ||  20.8   ||       ||       ||       ||       ||       ||  1.0  || jnz 0x7
       3   ||    1    ||  15.5   ||  28.9   ||  0.7  ||  0.3  ||       ||       ||       ||       || add rbx, rax
       4   ||    1    ||  16.8   ||  20.4   ||       ||       ||       ||       ||       ||  1.0  || jmp 0x5
       5   ||    1    ||  15.2   ||  28.9   ||  0.3  ||  0.7  ||       ||       ||       ||       || add rbx, rax
       6   ||    1    ||  15.8   ||  43.9   ||  1.0  ||       ||       ||       ||       ||       || add rbx, rbx
    Total number of Uops: 7
\end{example}

At the beginning, the following values are noticed:
\begin{itemize}
    \item \textbf{Block throughput} is the average number of cycles needed to execute the program ($\frac{Total\ number\ of\ cycles}{Number\ of\ iterations}$).
    \item \textbf{Block throughput with perfect front-end} can be used to see if the front-end of the processor was the bottleneck of the execution. To compute this value \suaca\ will perform a full analysis of the program. However, it will assume that $number\ of\ \mu ops\ loaded\ per\ cycle = capacity\ of\ reservation\ station$. If the runtime experiences a speedup, we conclude that the front-end was indeed the bottleneck. Note that this does not ignore the maximum capacity of the scheduler.
    \item \textbf{Block throughput with infinitely usable ports} is computed similarly. It will perform a full analysis, but every port can be used arbitrarily in each cycle. So several \microops\ can use the same port simultaneously. Should the runtime improve, we can come to the conclusion that one of the ports has to be the bottleneck.
    \item \textbf{Block throughput without dependencies} is as well akin to the aforementioned values. This time the dependencies are ignored during the simulation. An improved throughput indicates that the dependencies between instructions are the bottleneck.
\end{itemize}

When looking at the block throughput values here we can observe that our example program runs quite significantly faster with infinitely usable ports. This makes sense as only ports $0, 1$ and $5$ can be used by the instructions we are using. The two jump instructions are the biggest offender since they can exclusively use port $5$.\\

In some corner cases it might be possible that both the front-end and the ports are responsible for a decreased runtime. This can occur if every loaded instruction is directly computed (front-end bottleneck), but if the front-end was faster there would be no other port to run the additional loaded instructions on. In this case, none of the first values would differ from the normal \emph{Block throughput}, although they are actually both part of the bottleneck.\\
Similar behavior can be observed with multiple combinations of the above mentioned versions of our simulation. For this reason \suaca\ gives the user the option to perform these special simulations in all possible combinations.\\

In the table we can observe the following columns:
\begin{itemize}
    \item The \textbf{had to wait} column describes the average number of cycles the instruction experienced a delay from either blocked ports or register dependencies.
    \item The \textbf{caused to wait} column describes the average number of cycles the instruction caused a delay similar to the \textbf{had to wait} value. However, it will not track transitive dependencies. So consider a program that has a dependency chain of $A \rightarrow B \rightarrow C$ (where $A$, $B$ and $C$ are instructions of your program) and $A$ is not fully computed. $A$ will cause $B$ to be delayed, resulting in an increased \textbf{caused to wait} value of $A$. $B$ will then cause $C$ to be delayed, resulting in an increased \textbf{caused to wait} value of $B$.
    \item The \textbf{Used Ports} columns describe how many cycles the respective port has been used on average. Due to the port pipelining this is equal to the number of \microops\ that were assigned to this port in all cases except the divider pipe (see \autoref{sec:dividerpipe}). If possible, \suaca\ will always assign a \microop\ to the port that has been used the least during the analysis (out of the ports that this particular \microop\ is able to use) in order to achieve an even distribution of the ports. A detailed description of how those are computed can be found in \autoref{sec:chooseport}.
\end{itemize}

In \suaca's output the values of the \emph{caused to wait} column exceed those of the \emph{had to wait} column quite significantly this is due to the fact that multiple instructions can cause a delay for a single other instruction. We will further explain this in \autoref{sec:loop}.\\
In the \emph{Used Ports} column we can observe that line $0$ has used port $0$ $0.0$ times. This just means that this instruction has used port $0$, but to such a small amount that the rounding resulted in the $0.0$ value.

\section{Latency Analysis}
\label{sec:loop}

The latency of a program is the number of cycles needed to execute it once. \suaca\ can be used to compute the latency by running its analysis with a single iteration.

\begin{example}
    Block throughput: 3.00 cycles
    Block throughput with perfect front-end: 3.00 cycles
    Block throughput with infinitely usable ports: 3.00 cycles
    Block throughput without dependencies: 3.00 cycles
    Microops per cycle: 2.33

    Analysis for architecture: SNB

     Line  ||   Num   ||   had   || caused  ||            Used Ports
           ||   Uops  || to wait || to wait ||   0   ||   1   ||   2   ||   3   ||   4   ||   5   ||
    ------------------------------------------------------------------------------------------------
       0   ||    1    ||         ||   1.0   ||  1.0  ||       ||       ||       ||       ||       || mov rax, 0x1
       1   ||    1    ||         ||   1.0   ||       ||  1.0  ||       ||       ||       ||       || cmp rcx, 0x0
       2   ||    1    ||   1.0   ||   1.0   ||       ||       ||       ||       ||       ||  1.0  || jnz 0x7
       3   ||    1    ||   1.0   ||   1.0   ||  1.0  ||       ||       ||       ||       ||       || add rbx, rax
       4   ||    1    ||   1.0   ||         ||       ||       ||       ||       ||       ||  1.0  || jmp 0x5
       5   ||    1    ||         ||   1.0   ||       ||  1.0  ||       ||       ||       ||       || add rbx, rax
       6   ||    1    ||   1.0   ||         ||  1.0  ||       ||       ||       ||       ||       || add rbx, rbx
    Total number of Uops: 7
\end{example}

A closer look at the concrete values of this particular run reveals that the sum of the \emph{caused to wait} column is $5$, whereas the \emph{had to wait} column only sums up to $4$. This is due to the fact that both line $3$ and $5$ are responsible for the delay of line $6$ due to the dependency via the $rbx$ register. The exact dependencies can be found in \autoref{fig:wloop}. The same behavior arises when an instruction cannot be executed, because all ports are blocked. When three different instructions $A, B$ and $C$ block the ports another instruction $D$ would like to use the \emph{caused to wait} values of $A, B$ and $C$ are increased while only $D$'s \emph{had to wait} value will increase.\\
We can also see that the instruction in line $0$ has actually used port $0$. This explains the $0.0$ average usage value of the throughput analysis.

\section{Control Flow Graph}
The control flow graph is mainly used to compute the correct dependency graph. The $CFG$ of our example program can be seen in ~\autoref{fig:cfg}. The red edge only appears if the analysis runs in a loop as it represents the ``back jump'' to the start of the program that will not appear in a single iteration. For the sake of readability we dashed the red edge and will do so in future graphs.

\begin{figure}
    \centering
    \resizebox{!}{8cm}{
    \begin{tikzpicture}
    \node[mynode] (A) at (0, 0)     {0: mov rax, 0x1};
    \node[mynode] (B) at (0, -2)    {1: cmp rcx, 0x0};
    \node[mynode] (C) at (0, -4)    {2: jnz 0x7};
    \node[mynode] (D) at (-3, -6)   {3: add rbx, rax};
    \node[mynode] (E) at (-3, -8)   {4: jmp 0x5};
    \node[mynode] (F) at (1, -7)    {5: add rbx, rax};
    \node[mynode] (G) at (0, -10)   {6: add rbx, rbx};

    \Edges[](A,B);
    \Edges[](B,C);
    \Edges[](C,D);
    \Edges[](C,F);
    \Edges[](D,E);
    \Edges[](E,G);
    \Edges[](F,G);
    \Edges[style={->, bend right = 90, dashed}, color=red](G,A);

    \end{tikzpicture}}
    \caption{Control flow graph}
    \label{fig:cfg}
\end{figure}

\FloatBarrier

\section{Dependency Graph}
The dependency graph describes all register dependencies that occur in the program. An edge from node $A$ to node $B$ indicates that the instruction represented by $B$ depends on the instruction represented by $A$. \suaca\ will only track read-after-write dependencies, as those are the ones that can actually cause an instruction to be delayed. Whenever an instruction uses a memory address, \suaca\ will try to extract all used registers. Because we use the \emph{XED} library \cite{xed}, we can also consider the suppressed operands that cannot be seen in the code. One example for those suppressed operands is the \emph{RFLAGS} register, but there are several examples where an instruction has to access a register that does not appear in the code itself. \suaca\ will not keep track of the stack as this would often require runtime specific information. The detailed algorithm that is used to generate this graph can be found in \autoref{sec:depanalysis}. First consider the graph shown in \autoref{fig:woloop} which will be generated in the ``single loop case''.\\

\setlength{\abovecaptionskip}{-5pt}
\begin{figure}
\centering
\resizebox{\textwidth}{!}{
\begin{tikzpicture}
    \node[mynode] (A) at (-4, 0)  {0: mov rax, 0x1};
    \node[mynode] (B) at (-7, -2) {3: add rbx, rax};
    \node[mynode] (C) at (-1, -2)  {5: add rbx, rax};
    \node[mynode] (D) at (-4, -4) {6: add rbx, rbx};

    \Edges[label=$RAX$, style=bright](A,B);
    \Edges[label=$RAX$, style=bleft](A,C);
    \Edges[label=$RBX$, style={->, bend right = 10}](B,D);
    \Edges[label=$RBX$, style={->, bend left = 10}](C,D);

    \node[mynode] (E)  at (3, 0)  {1: cmp rcx, 0x0};
    \node[mynode] (F)  at (3, -2.5) {2: jnz 0x7};

    \Edges[label=$RFLAGS - zf$](E,F);

    \node[mynode] (G)  at (3, -4) {4: jmp 0x5};

\end{tikzpicture}
}
\caption{Dependency graph without loop dependencies}
\label{fig:woloop}
\end{figure}

This graph was generated with the $CFG$ in mind since there is no edge from node $3$ to node $5$. Additionally, we can observe that \suaca\ does differentiate between the different flags contained in the $RFLAGS$ register as the dependence is only reasoned with the $zf$ flag.\\
When \suaca\ is called with at least $2$ iterations it will also track all ``loop dependencies''. \autoref{fig:wloop} shows the graph with those in consideration. We can see that our program has two loop dependencies.

\begin{figure}
    \centering
    \resizebox{\textwidth}{!}{
    \begin{tikzpicture}
    \node (a) at (0, 1) {};
    \node[mynode] (A) at (-4, 0)  {0: mov rax, 0x1};
    \node[mynode] (B) at (-7, -2) {3: add rbx, rax};
    \node[mynode] (C) at (-1, -2)  {5: add rbx, rax};
    \node[mynode] (D) at (-4, -4) {6: add rbx, rbx};

    \Edges[label=$RAX$, style=bright](A,B);
    \Edges[label=$RAX$, style=bleft](A,C);
    \Edges[label=$RBX$, style=bleft](B,D);
    \Edges[label=$RBX$, color = red, style={bleft, dashed}](D, B);
    \Edges[label=$RBX$, style=bright](C,D);
    \Edges[label=$RBX$, color = red, style={bright, dashed}](D, C);

    \node[mynode] (E)  at (3, 0)  {1: cmp rcx, 0x0};
    \node[mynode] (F)  at (3, -2.5) {2: jnz 0x7};

    \Edges[label=$RFLAGS - zf$](E,F);

    \node[mynode] (G)  at (3, -4) {4: jmp 0x5};

    \end{tikzpicture}
}
    \caption{Dependency graph with loop dependencies}
    \label{fig:wloop}
\end{figure}

\FloatBarrier

\section{Architecture Selection}

As previously discussed, one of the big advantages of \suaca\ is that one can easily add new architectures on which the analysis can be based on.\\
\suaca\ gives the user the ability to choose a specific microarchitecture. For the next example we will use Intel's Coffee Lake microarchitecture instead of the Sandy Bridge microarchitecture we used previously (the first two columns are left out to improve readability):\\

\begin{example}
Block throughput: 2.00 cycles
Block throughput with perfect front-end: 2.00 cycles
Block throughput with infinitely usable ports: 2.00 cycles
Block throughput without dependencies: 1.75 cycles
Microops per cycle: 3.49

Analysis for architecture: CFL

   had   || caused  ||            Used Ports
 to wait || to wait ||   0   ||   1   ||   2   ||   3   ||   4   ||   5   ||   6   ||   7   ||
------------------------------------------------------------------------------------------------------------------
   4.7   ||  15.9   ||  0.1  ||  0.8  ||       ||       ||       ||  0.1  ||  0.1  ||       || mov rax, 0x1
   5.4   ||  13.6   ||  0.2  ||  0.5  ||       ||       ||       ||  0.1  ||  0.2  ||       || cmp rcx, 0x0
   6.2   ||   9.8   ||  0.4  ||       ||       ||       ||       ||       ||  0.6  ||       || jnz 0x7
  43.2   ||  48.9   ||  0.1  ||  0.2  ||       ||       ||       ||  0.7  ||  0.0  ||       || add rbx, rax
   5.1   ||   9.1   ||  0.2  ||       ||       ||       ||       ||       ||  0.8  ||       || jmp 0x5
  42.8   ||  51.6   ||  0.7  ||  0.1  ||       ||       ||       ||  0.2  ||  0.0  ||       || add rbx, rax
  43.6   ||  90.8   ||  0.1  ||  0.2  ||       ||       ||       ||  0.7  ||  0.0  ||       || add rbx, rbx
Total number of Uops: 7
\end{example}

The most significant change marks the infinitely usable ports analysis which no longer experiences an improvement in runtime. This is due to the larger number of ports in the Coffee Lake architecture. Instead, we can now observe that the three $add$ instructions, or more so their dependencies on each other, are responsible for most of the delays. We can conclude this from the \emph{had to wait} and \emph{caused to wait} values of these instructions and also the improved throughput with ignored dependencies.

\section{Detailed Information}
\label{sec:detail}

\suaca\ can also deliver some detailed information about one particular line. This can be useful to determine how a specific instruction causes and experiences a delay. The following table shows the result of a run on our example program with $200$ iterations and details for line $0$. We are using the Sandy Bridge architecture again.

\begin{Example}
Detailed delay information for instruction: mov rax, 0x1 in line 0

                Maximum latency: 1

                Latencies for dependencies:

                Line || 0 -> Line || Line -> 0
                ----------------------------------
                 3   ||     1     ||     0
                 5   ||     1     ||     0

                Delay caused by dependencies:

                Line || was delayed || has delayed
                ----------------------------------
                 3   ||    15.2     ||     0.0
                 5   ||    14.5     ||     0.0

                Delay caused by blocked ports:

                Port || was delayed || has delayed
                ----------------------------------
                 0   ||     0.0     ||    15.1
                 1   ||    13.5     ||    15.1
                 5   ||     0.0     ||    15.1
\end{Example}

In order to get a better understanding of those values, we will split the output and explain them step by step.

\begin{Example}
Maximum latency: 1

Latencies for dependencies:

Line || 0 -> Line || Line -> 0
----------------------------------
 3   ||     1     ||     0
 5   ||     1     ||     0
\end{Example}

First we can see that this instruction has a maximum latency of one cycle. This value can differ from those of the table below as we have seen in ~\autoref{sec:measurements}.\\
The table itself shows the latencies \suaca\ used for the dependencies. The second column shows the delay from the analyzed line to the line given in the first column and the third row shows the delay in the other direction. In our case, lines $3$ and $5$ depend on line $0$ (see ~\autoref{fig:wloop}) and we can see that those lines actually have to wait one cycle for line $0$ to be finished, and line $0$ itself is independent of the other two.\\

\begin{Example}
Delay caused by dependencies:

Line || was delayed || has delayed
----------------------------------
 3   ||    15.2     ||     0.0
 5   ||    14.5     ||     0.0
\end{Example}

This table now shows that lines $3$ and $5$ actually are delayed by our analyzed line. On average, line $3$ has to wait $15.2$ cycles for line $0$ to be finished while line $5$ has to wait $14.5$ cycles. Of course, both do not cause any delay on line~$0$ as there is no dependence.\\

\begin{Example}
Delay caused by blocked ports:

Port || was delayed || has delayed
----------------------------------
 0   ||     0.0     ||    15.1
 1   ||    13.5     ||    15.1
 5   ||     0.0     ||    15.1
\end{Example}

Finally \suaca\ outputs how much delay was caused by the ports. First consider that the $mov$ instruction in line $0$ can, in theory, use ports $0, 1$ and $5$. In our case it causes a delay of $13.5$ cycles per iteration on another instruction, because it uses port $1$. It does not cause any delay on the other two ports that it might use simply because it always uses port $1$ in our particular case (see ~\autoref{sec:loop}). The second column exhibits that line $0$ experiences a delay of $15.1$ cycles per iteration because all three usable ports were blocked. The $mov$ instruction we are considering only consists of a single \microop\ which leads to all three of those values being identical.\\
More precisely, the $mov$ instruction can only be delayed by blocked ports if all three of its usable ports are blocked, otherwise it would just select the free one. So all three of those blocked ports are responsible hence the three identical values.\\
However, this is not always the case since an instruction might consist of more than one \microop. We will discuss this further in \autoref{sec:chooseport}.

\section{Branch Analysis}

Finally, \suaca\ is able to analyze different branches. As we have seen in \autoref{fig:cfg} and \autoref{fig:woloop}, the ``normal'' analysis already considers branches for its dependencies. However, as the effect of the instructions will be completely ignored in the simulation, the branches will not have any other effect.\\
The actual branch analysis will perform two simulations, one for each branch. This will always consider the first $jump$ instruction and this one only. It will not acknowledge every single possible path through a program with multiple branches. We will now consider the branch analysis of our example program, with the Sandy Bridge architecture and $200$ iterations:

\begin{example}
Left branch analysis:

Block throughput: 2.00 cycles
Block throughput with perfect front-end: 2.00 cycles
Block throughput with infinitely usable ports: 2.00 cycles
Block throughput without dependencies: 2.00 cycles
Microops per cycle: 2.99

Analysis for architecture: SNB

 Line  ||   Num   ||   had   || caused  ||            Used Ports
       ||   Uops  || to wait || to wait ||   0   ||   1   ||   2   ||   3   ||   4   ||   5   ||
 ------------------------------------------------------------------------------------------------
   0   ||    1    ||  14.6   ||  30.1   ||  0.0  ||  1.0  ||       ||       ||       ||       || mov rax, 0x1
   1   ||    1    ||  15.6   ||  30.3   ||       ||  1.0  ||       ||       ||       ||       || cmp rcx, 0x0
   2   ||    1    ||  15.6   ||  23.8   ||       ||       ||       ||       ||       ||  1.0  || jnz 0x7
   3   ||    1    ||  15.6   ||  31.2   ||  1.0  ||       ||       ||       ||       ||       || add rbx, rax
   4   ||    1    ||  16.6   ||  22.9   ||       ||       ||       ||       ||       ||  1.0  || jmp 0x5
   6   ||    1    ||  15.7   ||  30.3   ||  1.0  ||  0.0  ||       ||       ||       ||       || add rbx, rbx
Total number of Uops: 6

Right branch analysis:

Block throughput: 2.00 cycles
Block throughput with perfect front-end: 2.00 cycles
Block throughput with infinitely usable ports: 2.00 cycles
Block throughput without dependencies: 1.67 cycles
Microops per cycle: 2.49

Analysis for architecture: SNB

 Line  ||   Num   ||   had   || caused  ||            Used Ports
       ||   Uops  || to wait || to wait ||   0   ||   1   ||   2   ||   3   ||   4   ||   5   ||
 ------------------------------------------------------------------------------------------------
   0   ||    1    ||   1.0   ||   2.8   ||  0.5  ||  0.5  ||       ||       ||       ||       || mov rax, 0x1
   1   ||    1    ||   1.5   ||   3.4   ||  0.2  ||  0.7  ||       ||       ||       ||  0.1  || cmp rcx, 0x0
   2   ||    1    ||   6.8   ||   4.7   ||       ||       ||       ||       ||       ||  1.0  || jnz 0x7
   5   ||    1    ||  40.0   ||  42.2   ||  0.5  ||  0.3  ||       ||       ||       ||  0.2  || add rbx, rax
   6   ||    1    ||  40.7   ||  42.9   ||  0.4  ||  0.2  ||       ||       ||       ||  0.4  || add rbx, rbx
Total number of Uops: 5
\end{example}

This kind of information can be useful to determine which of the two branches is more favorable for the execution. It can as well be used to detect if there are any significant differences between the two at all.\\
One can also use this in combination with the detailed analysis as \suaca\ will tell you the original line of all instructions in both parts of the branching analysis.

\newpage
\section{The Command Line Interface (CLI)}
The CLI of \suaca\ works as followed:\\
\[
suaca\ [option]\ path\_to\_file
\]
where $[option]$ is one or several of the following:
\begin{itemize}
    \item \emph{-pf} triggers the perfect front-end analysis.
    \item \emph{-ip} triggers the infinite port analysis.
    \item \emph{-nd} triggers the dependency free analysis.
    \item \emph{-cfg} will print the control flow graph into a file called $controlflow.dot$. The format will be graphviz readable.
    \item \emph{-dg} will print the dependency graph into a file called $dependency.dot$. The format will also be graphviz readable.
    \item \emph{-p} triggers the ``performance mode''. More specifically, this will prevent the three extra analyses that are needed to generate the three additional block throughput values. In most cases however, \suaca's performance bottleneck will be the parser for the measurement files which cannot be deactivated.
    \item \emph{-b} triggers the branch analysis.
    \item \emph{-{}-iform} is used to print the iforms of all instructions.
    \item \emph{-{}-arch x} will consider $x$ as the underlaying microarchitecture of the analysis. At the time of writing the available options are NHM (Nehalem), SNB (Sandy Bridge), IVB (Ivy Bridge), HSW (Haswell), BDW (Broadwell), SKL (Skylake), KBL (Kaby Lake) and CFL (Coffee Lake). The default value is SNB.
    \item \emph{-{}-loop x} will trigger the loop analysis. The default value of $x$ is $1$.
    \item \emph{-{}-detail x} will print detailed information about line $x$.
     \item \emph{-{}-setup x y} sets the default values for the architecture ($x$) and the number of iterations ($y$). Note that one always has to use both values.
    \item \emph{-{}-print-default} prints the default values for architecture and number of iterations.
\end{itemize}


%% Chapter 3: The real time alghoritm
%  - The edge detector as identifier
%  - The MOG as movement detector: a switch for the identifier

\section{Real Time Application}

	\subsection{Edge and MOG: the Rude Approach}

	The real time application follow a different approach, that has some advantages
	and some drawbacks. Two algorithms runs in parallel, to understand the status
	of the parking. The first algorithm, that is a mixture of gaussians, is used to
	identify a movement on the single parking spot. When a movement is detected, it
	enables the execution of the second algorithm, that is a Canny edge detector.
	The result of the edge detector (gray-scale image, due to linear perspective
	transformation) is summed up on the area of the single park spot, and if it
	reach a certain threshold (in configuration file the first element of
	\verb+param+).

	While there is movement on a parking spot, the system set its state to
	\emph{wait}, to make the user understand that it cannot derive conclusions on
	its state.

	The MOG system is calibrated with a really low learning rate ($\alpha = 0.1$)
	and for each pixel 3 gaussians model are generated. The Canny edge detection
	has an high threshold, and uses a Sobel kernel of dimension $3\times3$.

	The threshold value was derived analyzing a set of collected data, over time.
	The separation is almost obvious.

	% TODO inserire figura che spiega come funziona algoritmo a sinistra e a destra
	% la calibrazione dell'algoritmo (devo scaricare il file!)
	\begin{figure}[H] \label{fig:mog}
		\centering
			\includegraphics[keepaspectratio, scale=0.55]{img/mogandedge.pdf}
			\includegraphics[keepaspectratio, scale=0.2]{img/img2.pdf}
		\caption{The combination of MOG and Canny Edge algorithms}
	\end{figure}

	\subsection{Drawbacks}

	This implementation seems to work quite well, but for sure it is not an optimal
	solution. For each frame, a mixture of gaussians and a Canny algorithm, that
	have both an high computational load, run on the whole set of pixels. So we can
	count:
	\begin{itemize}
	  \item extremely high computational load.
	  \item this system is not robust against illumination: sometimes the shadows
	  could bring to a formation of fake edges that could be bring the system to
	  interpret the parking spot as busy. The good point
	\end{itemize}

\section{Conclusions}

	There are a lot of improvements that could be added to make this code better,
	but the first should be the optimization of the code. There are several point
	that need to be revised:
	\begin{itemize}
	  \item On each frame, for each pixel, both edge detection and MOG runs on the
	  whole image area. The code needs to be modified for an extraction
	  and execution of the algorithm on the littler area. This means add a support
	  for relative coordinates.
	  \item Elimination of unused variables. Some of the code was written for
	  support standard \verb+Rect+ instead of a polygonal area. Those part should
	  be removed from the code to make it lighter.
	  \item Add a stronger use of pointer for \verb+Mat+ in functions. Actually a
	  lot of functions receives as variables the whole \verb+Mat+ instead of a more
	  lighter pointer.
	\end{itemize}
	From the algorithmic point of view, the two methods must be joined to work
	together: the edge detection, launched by the MOG movement detector, should
	have more than one threshold, one higher that represents the highest
	probability that the place is busy, one lower that represents the lowest
	probability that the place has a car parked in. While the edge get a result in
	between this two thresholds, the histograms classifier will be invoked to make
	a last classification. In this way we exploit the illumination weaknesses of
	the edge detector and the training weakness of the classifier.

	\begin{figure}[H] \label{fig:final}
	\centering
		\includegraphics[keepaspectratio, scale=0.38]{img/ICbuffer.png}
	\caption{Image of the final result}
	\end{figure}


% Define document class
\documentclass[modern]{aastex631}

% Bibliography stuff
\bibliographystyle{aasjournal}
\setlength\LTcapwidth{\textwidth}

% Begin!
\begin{document}

% Title
\title{
    \vspace{-3em}
    \textbf{
        What can we learn about exoplanet maps in emitted light from photometric observations of secondary eclipses using JWST?
    }
}
% Author list
\author[0000-0001-8630-9794]{Fran Bartoli\'c}
\email{fb90@st-andrews.ac.uk}
\affil{Centre~for~Exoplanet~Science, SUPA, School~of~Physics~and~Astronomy, University~of~St.~Andrews, St.~Andrews, UK}
\affil{Center~for~Computational~Astrophysics, Flatiron~Institute, New~York, NY, USA}
\author[0000-0002-0296-3826]{Rodrigo Luger}
\author[0000-0002-9328-5652]{Daniel Foreman-Mackey}
\affil{Center~for~Computational~Astrophysics, Flatiron~Institute, New~York, NY, USA}
%

\begin{abstract}
We investigate the degeneracies involved in the process of reconstructing spatial maps of
exoplanets from secondary eclipse light curves in emitted light, and test the feasibility of
spatially resolving localized features on the daysides of Hot Jupiters using JWST.
We find that even with noiseless light curves and assuming fixed orbital parameters, there are
interesting degeneracies in the structure of inferred maps.
With realistic light curves, differentiating between a homogenous spot and finer spatial
structure (due to, for instance, the presence of large storms and clouds) is highly dependant
on the intensity contrast between the feature and the background.
There is a signal-to-noise dependent threshold which needs to be reached so that the ingress
and egress parts of the light curve start constraining higher order modes of the surface map.
Without reaching this threshold there is no hope of constraining finer structure on the dayside
of the planet and the information content about the surface is dominated by the phase curve
signal and the eclipse depth.
We find that even for the brightest Hot Jupiters, the signal-to-noise
ratio of the simulated JWST light curves isn't high enough to provide much information about
the surface beyond excluding the presence of very high contrast features.
\end{abstract}

\section{Introduction}
\label{sec:introduction}
One of the most notable advances in exoplanetary science over the past decade has been the ability
to reconstruct coarse two-dimensional spatial maps of exoplanets using high precision phase
curves and secondary eclipse observations.
\cite{knutson2007}, \cite{majeau2012b} and \cite{dewit2012} used Spitzer mid-infrared observations
of secondary eclipses of the Hot Jupiter HD189733b and found that surface emission is best
described by the presence of a large hot spot on the dayside of the planet which is
longitudinally offset from the substellar point.
Similarly, \cite{stevenson2014} produced temperature maps of the Hot Jupiter WASP-43b,
\cite{demory2013} mapped the Hot Jupiter Kepler-7b in reflected light and \cite{demory2016a}
mapped the thermal emission from the Super Earth 55 Cancri e, although all of these only
captured longitudinal variations in intensity.

Real exoplanet atmospheres of are certain to have three-dimensional spatial
inhomogeneities in emission more complex than a single hot spot due to the presence of clouds, zonal
jets, storms, waves etc. \citep{showman2020}.
Recent high resolution simulations of Hot Jupiter atmospheres by \cite{cho2021} which were able
to capture smaller scale turbulence show the presence of storms at a range of scales, including
planetary scales, with quasi-periodic time variability and multiple equilibrium cycles.
These atmospheric phenomena should result in spatial variation in the intensity of emitted
and reflected light across the visible disc of the planet which is potentially observable.

There have been significant advances in statistical modeling of phase curves and eclipse light
curves in recent years.
Most notably, \cite{luger2019a} introduced the \textsf{starry} algorithm which enables analytic
computation of phase curves and occultation light curves for bodies with arbitrary emission maps
expressed in a spherical harmonic basis and\cite{luger2021d} expanded the algorithm for the (considerably more complicated) case of
reflected light.

of bodies whose emission surface is
expanded in a basis of spherical harmonics.

In this paper, we investigate under which conditions it possible to recover more complex
spatial features than large hot spots.
We focus on emitted light from tidally locked planets and ignore uncertainties in the
orbital elements.
We start by describing the degeneracies inherent in the eclipse mapping problem even at
infinite signal-to-noise and then investigate the relationship between noise and information
content of phase curves and eclipse light curves.

\section{Methods}
\label{sec:methods}

\subsection{Model}
\label{ssec:model}
We use the \textsf{starry} mapping framework (CITE) to compute eclipse light curves of a simulated planet with thermal surface emission.
\textsf{starry} enables fast computation of occultation (eclipse) light curves in both emitted and reflected light by analytically integrating the flux over the planet and the stellar disc
Most importantly, \textsf{starry}

To compute eclipse light curves
\section{Results}
\label{sec:results}

\subsection{Degeneracies}
\label{ssec:degeneracies}

\begin{figure}[t!]
    \begin{centering}
    \includegraphics[width=\linewidth]{figures/preimages.pdf}
    \caption{}
    \label{fig:preimages}
    \end{centering}
\end{figure}

\section{Discussion}
\label{sec:discussion}

% Bibliography
\bibliography{bib}

\appendix
\section{Appendix 1}
\clearpage

\end{document}


\documentclass{article}

\usepackage{adjustbox}
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{amsmath}
\usepackage{batter_pitcher_2vec}
\usepackage[doi=false,
            eprint=false,
            isbn=false,
            maxbibnames=99,
            sorting=none]{biblatex}        % bibliography
\usepackage{booktabs}       % professional-quality tables
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
\usepackage{hyperref}       % hyperlinks
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[margin=1.0in]{geometry}
\usepackage{mathtools}
\usepackage{microtype}      % microtypography
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{subcaption}
\usepackage{tikz}
\usepackage{url}            % simple URL typesetting

\addbibresource{batter_pitcher_2vec.bib}

\title{\texttt{(batter|pitcher)2vec}: Statistic-Free Talent Modeling With Neural Player Embeddings}
\date{}

\author{
    Michael A. Alcorn\thanks
    {Personal website: \url{https://sites.google.com/view/michaelaalcorn}\newline
    \hspace*{1.8em}Code: \url{https://github.com/airalcorn2/batter-pitcher-2vec}} \\
    Red Hat, Inc.\\
    Chicago, IL 60615 \\
    \texttt{michaelaalcorn@gmail.com} \\
}

\begin{document}

\maketitle

\begin{abstract}

This paper introduces \texttt{(batter|pitcher)2vec}, a neural network algorithm inspired by \texttt{word2vec} that learns distributed representations of Major League Baseball players. The representations are discovered through a supervised learning task that attempts to predict the outcome of an at-bat (e.g., strike out, home run) given the context of a specific batter and pitcher. The learned representations qualitatively appear to better reflect baseball intuition than traditional baseball statistics, for example, by grouping together pitchers who rely primarily on pitches with dramatic movement. Further, like \texttt{word2vec}, the representations possess intriguing algebraic properties, for example, capturing the fact that Bryce Harper might be considered Mike Trout's left-handed doppelgänger. Lastly, \texttt{(batter|pitcher)2vec} is significantly more accurate at modeling future at-bat outcomes for previously unseen matchups than simpler approaches.

\end{abstract}

\section{Introduction}

Baseball is notorious for its culture of statistical bookkeeping. The depth and breadth of baseball statistics allows fans and professionals alike to compare players and forecast outcomes with varying degrees of accuracy. Although many traditional baseball statistics can be quite informative and useful, they can also be somewhat arbitrary, and thus may not accurately reflect the true talent of any given player.

The field of Sabermetrics was developed in an effort to address some of the inherent limitations of standard baseball statistics. For example, Wins Above Replacement (WAR) ``offers an estimate to answer the question, `If this player got injured and their team had to replace them with a freely available minor leaguer or a AAAA player from their bench, how much value would the team be losing?'\thinspace'' \parencite{WAR}. However, the WAR formula (\ref{eqn:war}) is, itself, somewhat ad hoc, reflecting the intuition of the statistic's designer(s):

\begin{equation}
\label{eqn:war}
WAR = \frac{BR + BRR + FR + PA + LA + RR}{RPW}
\end{equation}

where $BR$ is batting runs, $BRR$ is base running runs, $FR$ is fielding runs, $PA$ is a positional adjustment, $LA$ is a league adjustment, $RR$ is replacement runs, and $RPW$ is runs per win.

Whereas the WAR statistic uses a combination of conventional baseball statistics to quantify an individual player's impact on his team's overall record, the Player Empirical Comparison and Optimization Test Algorithm (PECOTA) forecasts player performance by identifying a neighborhood of players with historically similar statistics (both traditional and of the Sabermetrics variety) and then performing an age correction \parencite{PECOTA}. However, the neighborhoods produced by PECOTA are proprietary, which precludes deeper investigation by the community. The proprietary Deserved Run Average (DRA) incorporates contextual information to predict player value, but is limited in its expressiveness as a strictly linear model \parencite{Judge2015, Turkenkopf2015}.

When scouts assess the talent of a player, they do not simply count the number of times the player made it to first base or struck out. They consider the player's tendencies in a number of different contexts. For example, a scout might notice that a certain left-handed batter tends to ground out to third base when facing left-handed curveball pitchers. Or that a certain right-handed fastball pitcher tends to struggle against short right-handed batters. An algorithm capable of learning these subtleties would be a valuable tool for assessing player talent.

The task of extracting informative measures of talent for Major League Baseball (MLB) players has a surprising parallel in the field of natural language processing --- the task of constructing useful word embeddings. Words, like MLB players, can be considered distinct elements in a set, and one common way to represent such categorical data in machine learning algorithms is as one-hot encodings. A one-hot encoding is an $N$-dimensional vector (where $N$ is the number of elements in the set) consisting entirely of zeros except for a single one at the location corresponding to the element's index. For example, the one-hot encodings for a vocabulary consisting of the words ``dog'', ``cat'', and ``constitutional'' might be $\begin{bsmallmatrix} 1 & 0 & 0 \end{bsmallmatrix}$, $\begin{bsmallmatrix} 0 & 1 & 0 \end{bsmallmatrix}$, and $\begin{bsmallmatrix} 0 & 0 & 1 \end{bsmallmatrix}$, respectively.

One drawback of one-hot encodings is that each vector in the set is orthogonal to and equidistant from every other vector in the set; i.e., every element in the set is equally similar (or dissimilar) to every other element in the set. Words, however, \emph{do} exhibit varying degrees of semantic similarity. For example, the words ``dog'' and ``cat'' are clearly more semantically similar than the words ``dog'' and ``constitutional''. Word embedding algorithms learn to mathematically encode such similarities as geometric relationships between vectors (e.g., cosine similarity or Euclidean distance).

Perhaps the most famous word embedding algorithm is \texttt{word2vec} \parencite{Mikolov2013}, a neural network that learns distributed representations of words in a supervised setting. The \texttt{word2vec} algorithm can take two different forms: \textit{\textbf{(1)}} the continuous bag-of-words (CBOW) model or \textit{\textbf{(2)}} the skip-gram model (see Figure~\ref{fig:word2vec}). The CBOW model attempts to predict a word ($w_t$) given some surrounding words (e.g., $w_{t-1}$ and $w_{t+1}$, although the context can be larger). In contrast, the skip-gram model attempts to predict the surrounding words of a given central word. Incorporating pre-trained word embeddings into models built for other natural language processing tasks often leads to improved performance (an example of ``transfer learning'') \parencite{Bengio2003}.

\begin{figure}[h]
\captionsetup[subfigure]{labelformat=empty}
\centering

    \begin{subfigure}[t]{0.45\textwidth}
    \centering
    \includegraphics[width=5cm]{cbow_fig.png}
    \caption{}
    \end{subfigure}%
    \begin{subfigure}[t]{0.45\textwidth}
    \centering
    \includegraphics[width=5cm]{skip_gram.png}
    \caption{}
    \end{subfigure}

\caption{The CBOW (left) and skip-gram (right) model architectures for a window of three words. A one-hot vector is depicted as a black square among an array of white squares while an embedding is depicted as an array of gray squares.}
\label{fig:word2vec}
\end{figure}

This paper introduces \texttt{(batter|pitcher)2vec}, a neural network algorithm that adapts representation learning concepts (like those found in \texttt{word2vec}) to a baseball setting, modeling player talent by learning to predict the outcome of an at-bat given the context of a specific batter and pitcher. Unlike many Sabermetrics statistics, \texttt{(batter|pitcher)2vec} learns from ``raw'' baseball events as opposed to aggregate statistics, which allows it to incorporate additional contextual information when modeling player talent.

Previous work modeling player ``style'' in other sports has ignored the context in which events occur, and has relied on aggregate statistics for data. \parencite{Hackett2017} applied latent Dirichlet allocation to aggregate statistics of mixed martial arts finishing moves to discover factors describing fighter style. \parencite{Wei2016} used dictionary learning to discover tennis shot prototypes, and then aggregated these prototypes to describe player styles. Finally, \parencite{Schulte2017} utilized aggregate location data to characterize hockey player style. Both \parencite{Oh2015} and \parencite{Kuehn2016} described models that predict the probability of different events occurring during basketball games given the players on the court, but the models are strictly linear in the player parameters, which limits their expressiveness. \texttt{(batter|pitcher)2vec} marries the expressiveness of latent variable models with the predictive power of contextual models to improve upon each.

\section{Data}
\label{data}

\begin{table}[h]
\caption{The possible at-bat outcomes.}
\centering
\begin{adjustbox}{max width=\textwidth}
    \begin{tabular}{ | l | l | l | l | }
    \hline
    Symbol & Description & Symbol & Description \\
    \hline\hline
    1-9 & Out first handled by fielder \# & E1-E9 & Error first handled by fielder \# \\
    \hline
    S1-S9 & Single first handled by fielder \# & K & Strike out \\
    \hline
    D1-D9 & Double first handled by fielder \# & W & Walk \\
    \hline
    DG & Ground rule double & IW & Intentional walk \\
    \hline
    T1-T9 & Triple first handled by fielder \# & BK & Balk \\
    \hline
    HR & Home run & HP & Hit by a pitch \\
    \hline
    \end{tabular}
\end{adjustbox}
\label{table:at_bats}
\end{table}

Play-by-play data for each game from the 2013, 2014, 2015, and 2016 seasons were obtained from the Retrosheet website \parencite{Retro}. Each play description was converted into a tuple consisting of the batter, pitcher, and at-bat outcome; for example, (Mike Trout, Anthony Bass, HR), where HR is the symbol denoting a home run. There are 52 potential at-bat outcomes (see Table~\ref{table:at_bats}), but the model only considers 49 different outcomes because there were no observed instances of doubles first fielded by the catcher, triples first fielded by the catcher, or triples first fielded by the third baseman. The raw training data set consisted of 557,436 at-bats from the 2013, 2014, and 2015 seasons representing 1,634 different batters and 1,226 different pitchers. To ensure an adequate amount of data was available for learning player representations, only the most frequent batters and pitchers involved in 90\% of at-bats were included in the analysis. The final data set consisted of 461,231 at-bats with 524 batters and 565 pitchers.

\section{Model}
\label{model}

\begin{figure}[h]
\centering
\includegraphics[width=0.75\textwidth,height=\textheight,keepaspectratio]{batter_pitcher_model.png}
\caption{The \texttt{(batter|pitcher)2vec} model architecture.}
\label{fig:batter_pitcher}
\end{figure}

The \texttt{(batter|pitcher)2vec} model architecture can be seen in Figure~\ref{fig:batter_pitcher}, and the similarities between \texttt{(batter|pitcher)2vec} and \texttt{word2vec} should be readily apparent. The model takes one-hot encodings of a batter and pitcher as input and then selects the corresponding player weights from the the batter (\ref{eqn:batter_embedding}) and pitcher (\ref{eqn:pitcher_embedding}) weight matrices, respectively. The player weight vectors are then passed through a ``sigmoid''/logistic activation function.

\begin{equation}
\label{eqn:batter_embedding}
w_b = \sigma(W_B \cdot h_b)
\end{equation}

\begin{equation}
\label{eqn:pitcher_embedding}
w_p = \sigma(W_P \cdot h_p)
\end{equation}

Here, $h_b$ is the $N_B$-dimensional one-hot vector (where $N_B$ is the number of batters) for the batter indexed by $b$, $W_B$ is the batter embedding matrix, $\sigma$ is the logistic activation function (i.e., $\frac{1}{1 + e^{-x}}$) and $w_b$ is the batter's embedding. Likewise, $h_p$ is the $N_P$-dimensional one-hot vector for the pitcher indexed by $p$, $W_P$ is the pitcher embedding matrix, and $w_p$ is the pitcher's embedding.

The batter and pitcher embeddings are then concatenated together (\ref{eqn:concat}) and fed into a standard softmax layer (\ref{eqn:output} and \ref{eqn:prob}), which outputs a probability distribution over at-bat outcomes.

\begin{equation}
\label{eqn:concat}
w_{b \oplus p} = w_b \oplus w_p
\end{equation}

\begin{equation}
\label{eqn:output}
z = W_o \cdot w_{b \oplus p} + b_o
\end{equation}

\begin{equation}
\label{eqn:prob}
p(o_i | h_b, h_p) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}
\end{equation}

Here, $o_i$ is the at-bat outcome indexed by $i$ and $K$ is the number of possible at-bat outcomes.

Maximizing the likelihood of this model is equivalent to minimizing the model's average cross entropy (\ref{eqn:entropy}), which is the standard loss function used in machine learning classification tasks:

\begin{equation}
\label{eqn:entropy}
\mathcal{L}(D) = \frac{1}{N}\sum_{i=1}^{N}H(p_i,q_i) = -\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{K} p_i(o_j)log(q_i(o_j))
\end{equation}

where $D$ is the training data, $N$ is the number of training samples, $H(p_i,q_i)$ is the cross entropy between the probability distributions $p_i$ and $q_i$, $p_i$ is the true at-bat outcome distribution for training sample $i$, and $q_i$ is the predicted outcome distribution for training sample $i$.

The model was implemented in Keras 2.0 \parencite{Keras2015} and trained on a laptop with 16 gigabytes of RAM and an Intel i7 CPU. The model was trained for $100$ epochs using Nesterov accelerated mini-batch ($100$ samples/mini-batch) gradient descent with the learning rate, momentum, and decay hyperparameters set at $0.01$, $0.9$, and $10^{-6}$, respectively. Both batters and pitchers were embedded into a nine-dimensional space. The code to generate the results described in this paper can be found at \url{https://github.com/airalcorn2/batter-pitcher-2vec}.

\section{Results}
\label{results}

\subsection{Visual inspection of the player embeddings}

Visually inspecting low-dimensional approximations of neural network representations can often provide some intuition for what the model learned. Several plots of the first two principal components of the batter embeddings can be seen in Figure~\ref{fig:batter_traits}. A number of trends are readily apparent; for example, left-handed hitters are clearly distinguishable from right-handed hitters, and batters with high single rates are antipodal to batters with low single rates (a similar pattern is visible for home run rates). At the least, \texttt{(batter|pitcher)2vec} appears to be capable of capturing the information contained in standard baseball statistics.

\begin{figure}[h]
\captionsetup[subfigure]{labelformat=empty}
\centering
    \begin{subfigure}{0.5\linewidth}
    \centering
    \includegraphics[width=1\linewidth]{batter_single.png}
    \caption{}
    \end{subfigure}%
    \begin{subfigure}{0.5\linewidth}
    \centering
    \includegraphics[width=1\linewidth]{batter_hr.png}
    \caption{}
    \end{subfigure}\\[1ex]

    \begin{subfigure}{0.5\linewidth}
    \centering
    \includegraphics[width=1\linewidth]{batter_hand.png}
    \caption{}
    \end{subfigure}
\caption{Plots of the first two principal components of the batter embeddings colored by various batter qualities.}
\label{fig:batter_traits}
\end{figure}

\subsection{Nearest neighbors}

Probing the neighborhoods of individual embeddings can also yield deeper insights about the model. The t-SNE algorithm \parencite{VanderMaaten2008} was used to map the batter and pitcher embeddings into two dimensions so that they could be visualized (Figure~\ref{fig:tsne}). Intriguing player clusterings are readily apparent, with close pairs including: Mike Trout/Paul Goldschmidt, Dee Gordon/Ichiro Suzuki, and Aroldis Chapman/Dellin Betances.

\begin{figure}[h]
\captionsetup[subfigure]{labelformat=empty}
\centering

    \begin{subfigure}[b]{0.75\textwidth}
    \includegraphics[width=1\linewidth]{batter_tsne.png}
    \caption{}
    \end{subfigure}

    \begin{subfigure}[b]{0.75\textwidth}
    \includegraphics[width=1\linewidth]{pitcher_tsne.png}
    \caption{}
    \end{subfigure}

\caption{Two-dimensional t-SNE map of the learned batter (top) and pitcher (bottom) representations.}
\label{fig:tsne}
\end{figure}

When calculating nearest neighbors in the learned embedding space, Paul Goldschmidt is indeed Mike Trout's nearest neighbor; an unsurprising result considering how each athlete is known for his rare blend of speed and power \parencite{Kory2015}. Similarly, Ichiro Suzuki is Dee Gordon's nearest neighbor, which is to be expected as both have a reputation for being able to get on base \parencite{Sullivan2015}. Notably, when clustering players on common MLB stats (e.g., HRs, RBIs), Paul Goldschmidt is not among Mike Trout's ten nearest neighbors, nor is Ichiro Suzuki among Dee Gordon's ten nearest neighbors.

For pitchers, Craig Kimbrel is Aroldis Chapman's nearest neighbor, which is unsurprising considering both are known as elite closers with overpowering fastballs \parencite{Mirsky2016}. Similarly, Félix Hernández, much like his two nearest neighbors, Jean Machi and Carlos Carrasco, is known for having a pitch with incredible movement in his repertoire \parencite{Buchanan2015, Romano2015, Berg2016}. Other nearest neighbors are not as immediately obvious, for example, Clayton Kershaw and Craig Stammen (although, Zack Greinke is Kershaw's second nearest neighbor), but a method like \texttt{(batter|pitcher)2vec} could potentially reveal surprising similarities between players who are not considered similar by human standards.

\subsection{Opposite-handed doppelgängers}

One of the most fascinating properties of effective word embeddings is their analogy capabilities \parencite{Mikolov2013a}. For example, when using \texttt{word2vec} word embeddings, subtracting the vector for ``France'' from the vector for ``Paris'' and then adding the vector for ``Italy'' produces a vector that is very close to the vector for ``Rome'', which corresponds to the analogy Paris : France :: Rome : Italy \parencite{Mikolov2013a}.

To investigate the algebraic properties of the \texttt{(batter|pitcher)2vec} representations, the average batter vector was calculated for both left-handed and right-handeded hitters and then subtracted from select players in an attempt to generate opposite-handed doppelgängers. For example, subtracting the average left-handed batter vector from the vector for Mike Trout (a right-handed batter) produces a vector with Chris Davis, David Ortiz, and Bryce Harper (all powerful, left-handed batters) as the three nearest neighbors, which is suggestive of a valid batter algebra (see \parencite{Spector2016} for a discussion of the similarities between Mike Trout and Bryce Harper). Similarly, subtracting the average left-handed batter vector from the vector for Dee Gordon yields a vector that is very close to the vector for Tyler Saladino, a fitting candidate for Gordon's opposite-handed doppelgänger \parencite{Chamberlain2017}.

\subsection{Modeling previously unseen at-bat matchups}

\begin{table}[h]
\caption{Average cross entropy for a naïve strategy, multinomial logistic regression, and \texttt{(batter|pitcher)2vec} on previously unseen batter/pitcher matchups.}
\centering
\begin{adjustbox}{max width=\textwidth}
    \begin{tabular}{ | l | l | }
    \hline
    Model & Average Cross Entropy \\
    \hline\hline
    Naïve & $2.8113$ \\
    \hline
	Multinomial Logistic Regression & $2.8118$ \\
    \hline
    \texttt{(batter|pitcher)2vec} & $2.7848$ \\
    \hline
    \end{tabular}
\end{adjustbox}
\label{table:results}
\end{table}

The representations learned by neural networks are theoretically interesting because they suggest the models are discovering causal processes when the models are able to generalize (or transfer) well \parencite{RepresentationLearning}. In the case of \texttt{(batter|pitcher)2vec}, the ability to accurately model at-bat outcome probability distributions for previously unseen batter/pitcher pairs would indicate the neural network was extracting important aspects of baseball talent during learning. To test this hypothesis, at-bat outcomes were collected from the 2016 season for previously unseen matchups that included batters and pitchers from the training set. In all, there were 21,479 previously unseen matchups corresponding to 51,580 at-bats.

A naïve strategy was used as a baseline for performance comparisons. For any given batter, the probability that an at-bat would result in a specific outcome was defined as:

\begin{equation}
\label{eqn:batter_naïve}
p(o_i|b_j)=\frac{c_{i,j} + r_i}{\sum_{k=1}^{K} c_{j,k} + 1}
\end{equation}

where $o_i$ denotes the outcome indexed by $i$, $b_j$ represents the batter indexed by $j$, $c_{i,j}$ is the number of times the batter indexed by $j$ had an at-bat resulting in the outcome indexed by $i$ in the training data, $r_i$ is the proportion of all at-bats that resulted in the outcome indexed by $i$ in the training data, and $K$ is the number of outcomes. Essentially, the procedure adds one at-bat to each batter's data, but distributes the probability mass of that single at-bat across all possible outcomes based on data from all batters. $r_i$ can thus be considered a type of ``prior'' or smoothing factor. $p(o_i|t_j)$ was similarly defined for pitchers. The naïve expected outcome distribution for a given batter ($b_j$) and pitcher ($t_k$) matchup is thus defined as:

\begin{equation}
\label{eqn:naïve}
p(o_i|b_j,t_k) = \frac{p(o_i|b_j) + p(o_i|t_k)}{2}
\end{equation}

The cross entropy was calculated for each at-bat in the test set using both the naïve approach and \texttt{(batter|pitcher)2vec}. The naïve approach produced an average cross entropy of $2.8113$ on the test set while \texttt{(batter|pitcher)2vec} produced a significantly ($p < 0.001$) lower average cross entropy of $2.7848$, a $0.94\%$ improvement (Table~\ref{table:results}). For comparison, a multinomial logistic regression model (Figure~\ref{fig:log_reg}) trained and tested on identical data sets produced an average cross entropy of $2.8118$, which is slightly worse than the naïve approach (Table~\ref{table:results}). \texttt{(batter|pitcher)2vec} thus appears to be exceptional in its ability to model at-bat outcome distributions for previously unseen matchups.

\begin{figure}[h]
\centering
\includegraphics[width=0.75\textwidth,height=\textheight,keepaspectratio]{logistic_regression.png}
\caption{A graphical model depiction of multinomial logistic regression.}
\label{fig:log_reg}
\end{figure}

\section{Future directions}
\label{future}

These results prove neural embedding algorithms offer a principled means of modeling talent from ``raw data'', i.e., without resorting to ad hoc statistics. Just as pre-trained word embeddings can be used to improve the performance of models in various natural language processing tasks, player embeddings could be used to better inform baseball strategy. For example, by swapping the embeddings of players in a proposed trade and ``back simulating'' games from earlier in the season, teams would be able to assess how many more wins (or losses) they would have obtained with the candidate player(s) on the roster (effectively establishing a counterfactual). Likewise, after first applying \texttt{(batter|pitcher)2vec} to minor league baseball players, a second model could be trained that learns to map a player's minor league representation to his MLB representation. Such a model would allow teams to scout prospects by surveying their neighboring MLB players in the mapped space (this framework is conceptually similar to the multimodal model described in \parencite{Ngiam2011}, which learns a map between audio and video representations).

\begin{figure}[h]
\centering
\includegraphics[width=0.75\textwidth,height=\textheight,keepaspectratio]{image_embedding.png}
\caption{Image-like representations of plays could incorporate both spatial and talent information, with the $(x, y)$ coordinates of the ``image'' encoding athlete location and the ``channels''/embeddings encoding player talent. Here, the red and blue squares (which extend in the $z$ direction) depict two different player embeddings.}
\label{fig:image_embed}
\end{figure}

Because the neural embedding template is so flexible, it can easily be adapted to suit a variety of inputs and outputs in different sports settings. Modifying \texttt{(batter|pitcher)2vec} to predict PITCHf/x measurements as opposed to discrete outcomes could be fruitful as PITCHf/x data would likely convey more information about a player's physical characteristics. \texttt{(batter|pitcher)2vec} could also be extended to include the pitcher's supporting defense as one of the model's inputs, thereby providing additional context when predicting at-bat outcomes (mirroring the approach taken in \parencite{Le2014} with \texttt{doc2vec}). A similar architecture was proposed by \parencite{Alcorn2016} to model National Football League offenses and defenses and was used in \parencite{Le2017} to capture soccer team ``styles''. Player embeddings could even make powerful additions to models with more complex spatiotemporal components. For example, \parencite{Wang2016} represented National Basketball Association plays as a sequence of images where each player was colored according to his position, and a player was assigned to a position based on his nearest neighbors in a latent feature space acquired from an autoencoder trained on aggregate statistics. Rather than using hard position assignments to color players, the ``color channels'' of the image could contain player embeddings (Figure~\ref{fig:image_embed}), which would enable much richer modeling of play dynamics.

\subsubsection*{Acknowledgments}

I would like to thank my colleague Erik Erlandson for his suggestion to investigate opposite-handed doppelgängers.

\printbibliography

\end{document}