% \documentclass[12pt, oneside]{book}

\chapter{Equation of State and Pressure Gradients}
\label{chap:EquationofState}

In this chapter we go back now and consider some fundamentals about water that are important to this course.  Seawater is a relatively complex substance.  As discussed before, lateral density differences can drive lateral circulation, hence it is important to understand and predict how those differences arise.  Further, the ocean is a huge reservoir of heat, and understanding its heat content is important for predicting weather and future climate.

\section{Temperature}

Temperature is a thermodynamic property of a substance, the value of which is proportional to the kinetic energy of the random motions of the molecules in the substance (these are called \Wikiref{Brownian motion}). The heat necessary to bring a substance to a given temperature is proportional to the \emph{\Wikiref{heat capacity}}.  In order to change a parcel of water's temperature by $\Delta T$, we need to supply energy proportional to the heat capacity of the water:
\begin{equation}
   \Delta E = c_p\ \Delta T
\end{equation}
where $c_p$ has units of $J\, C^{-1}\ kg^{-1}$, and $\Delta E$ has units of $J\ kg^{-1}$.  Values for $c_p$ are given in \fref{fig:HeatCapacity}, and note that the heat capacity is not constant, but depends on the temperature of the water, its salinity, and its pressure!  The dependence is non-linear, so we usually resort to using a computer to calculate the empirically-derived values.  However, for a litre of fresh water ($\approx 1\ kg$) at $20^oC$ we would need 4180 J of energy to raise the temperature by one degree.

\begin{figure}
    \includegraphics[width=3in]{./figs/HeatCapacity.pdf}
    \caption{Heat capacity of water a) for fresh water as a function of temperature, b) as a function of both salinity and temperature.  Both are presented at the sea surface.}
    \label{fig:HeatCapacity}
\end{figure}

The heat capacity of water is impressive compared to air, where typical values are $1000\ J\, C^{-1}\ kg^{-1}$.  The whole atmosphere has a weight of $\approx 10^{4} \ kg\ m^{-2}$, so heating the whole atmosphere one degree requires $\approx 10^7 J\ m^{-2}$.  Conversely, the ocean at 4000 m deep and with a density close to $1000 \ kg\,m^{-3}$ weighs approximately $4\times10^6\ kg\ m^{-2}$, so raising its temperature by one degree requires $16\times10^9 J\ m^{-2}$, more than three orders of magnitude more energy.  This helps explain why the ocean is such an important reservoir of heat in the climate system.

\subsection{Measurement}

\begin{figure}[hbtp]
  \begin{center}
    \includegraphics[width=1in]{figs/CTD.pdf}
    \includegraphics[width=3in]{figs/ThermistorConductity.pdf}
    \caption{a) CTD b) thermistor and conductivity probe on a CTD.}
    \label{fig:CTD}
  \end{center}
\end{figure}

Temperature is measured in the ocean by thermometers.  The typical technology is what is called a ``\Wikiref{thermistor}'', which is a metal in which the electrical resistance changes proportional to its temperature.  Since its relatively easy to design an electronic circuit to measure resistance, this has allowed rapid and automatic measurement of ocean temperatures since such thermistors were first developed in the 1970s.

Once calibrated, quality ocean thermistors are typically accurate to $0.001\ ^oC$, and are stable to similar levels. It takes time for the thermistor to respond to temperature changes in the ocean, so designing the sensors is a tradeoff between physical robustness and response time.  Typical response time is about 0.3--0.5 s, which is quite fast for ocean changes, but can be slow relative to a CTD being lowered in the ocean.

\subsection{In-situ and potential temperature}
\label{sec:potential-temperature}

Ocean temperatures are largely set at the sea surface either by absorbing sunlight or exchanging heat with the atmosphere, and subsequently modified by ocean mixing.  The temperature of the water is thus a powerful \emph{tracer} of where a water mass last saw the surface of the ocean, with warmer waters found near the equator, and colder near the poles.

However, the effect of pressure on the temperature makes it harder to trace where water came from.  The temperature of a water parcel that is moved from one depth to a deeper one will increase because the pressure has increased, even if no heat is exchanged with its surroundings during the move (we call this an \emph{\Wikiref{adiabatic process}}).  A thermistor will measure this change. If the parcel is \emph{adiabatically} brought back to the original depth, its temperature will drop again.

In order to remove this adiabatic heating effect, its useful to differentiate between the actual \emph{in-situ} temperature and instead report the \emph{potential temperature}.  This is simply the temperature the water would be if it was brought adiabatically to the surface.

The reason for doing this is dramatically illustrated by considering the temperature measured in the Kermadec Trench \citep{Warren73}.  The trench is much deeper than the surrounding water, so water in the trench has been there a long time.  However, the in-situ temperature increases with depth which might make us think either it is different water, or that there is a heating source from below (\fref{fig:Trench}, curve labeled with "t").  However, computing the \emph{potential temperature} $\theta$ shows that this water is the same potential temperature (\emph{isothermal}) deeper than 5000 m.

\begin{marginfigure}
\begin{center}
  \includegraphics[width=2in]{figs/Trench}
  \caption{CTD data from Kermadec Trench \citep{Warren73} \emph{in situ} temperature and potential temperature. (Figure from \href{http://oaktrust.library.tamu.edu/handle/1969.1/160216}{Stewart})}
  \label{fig:Trench}
  \end{center}
\end{marginfigure}

Outside of the deepest part of the ocean, the difference between potential and in-situ temperatures are not large (\fref{fig:P16Temp}).  In the deepest ocean, its clear that the cold tongue of water that originates in the Antarctic Circumpolar Current originates shallower in the water column than we would infer from just looking at the \emph{in-situ} temperature.  This improved ability to trace the source of water without accounting for pressure effects is the main motivation for using potential temperature, and most publications will use potential temperature exclusively.

\begin{figure}[hbt]
  \begin{center}
    \includegraphics[width=5in]{./figs/P16Temp.pdf}
    \caption{North-south sections in the Pacific ocean at a longitude of 150 W, plotted as the measured \emph{in-situ} temperature (upper panel) and the potential temperature, $\theta$ (lower panel).    }
    \label{fig:P16Temp}
  \end{center}
\end{figure}

\section{Salinity}

Salinity is the measure of dissolved solids in seawater, and is expressed as grams of solids per kilogram of seawater.  Because it is derived from solids in the seawater, salinity is hard to define, and quite hard to accurately measure in the ocean. Oceanographers are greatly aided by the fact that the residence time of a salt ion is very long compared to the mixing timescale of the ocean.  While we think of rivers as being composed of fresh water, the input of salt ions largely come from the trace amounts in these rivers. In total, there is $\approx 5 \times  10^{19}\ kg$ of salt in the ocean, but inputs are $\approx 2.5 \times  10^{12}\ kg/year$, so the \emph{\Wikiref{residence time}} of salt in the ocean is around 20 million years.  The overturning time scale of the ocean is 3000 y (at the most), so salt in the ocean is ``well-mixed''. This long residence time means that the ratio of salt constituents in the oceans only changes on geological time scales, and thus in the open ocean the ratio of various ions can be considered a constant (closer to river sources the ion ratios can be quite different).

Note, however, that the ratio of the solids that make up the salt being constant does not mean that \emph{salinity} itself is constant.  The concentration of salt in a given volume of water depends on how much river water or rain water that parcel has been in contact with, or if it has been subject to evaporation.

\subsection{Measurement}

Chemical oceanographers spent many decades studying the chemical makeup of salt water.  The standard way of measuring salinity was to titrate the chloride solids and then use the fact that the ratio of salt ions in the ocean is relatively constant to infer the total salinity.  This is done more rarely now, and instead oceanographers measure the \emph{conductivity} of  seawater using a conductivity cell. This uses the fact that salt water conducts electricity better than fresh water, and thus the amount of electricity thats able to get from an electrode to an anode through a known quantity of seawater is proportional to the conductivity.

Electric circuits to measure conductivity are relatively straight forward, but the most stable measurements require a good-sized sensing sensing element over which to take the measurement.  The conductivity cell in the CTD shown in \fref{fig:CTD}a is about 15 cm long to allow a stable and accurate estimate.  This means that there is a limitation to how fine the vertical resolution of the conductivity estimate is.

\begin{marginfigure}
\begin{center}
    \includegraphics[width=2.2in]{./figs/Conductivity}
    \caption{Conductivity as a function of temperature and salinity}
    \label{fig:Conductivity}
\end{center}
\end{marginfigure}

The largest challenge in measuring salinity using conductivity is that conductivity is also a strong function of temperature as well as salinity (\fref{fig:Conductivity}).  That means that we need to have a well-matched temperature measurement at the same time and place as the conductivity measurement in order to back out the salinity.  Mismatched sensors lead to a phenomena known as "salinity spiking".  It is easily understood if we consider measuring through a temperature interface in water that  has a uniform salinity (\fref{fig:SalinitySpike}).  If the temperature sensor is lagged compared to the the conductivity sensor, the conductivity will go down, but the temperature measured by the temperature sensor will not until a deeper depth.  When the salinity is calculated from these two measurements, there will be a sharp discontinuity in inferred salinity at the interface because the sensors are not matched up. This is a very hard problem to get rid of, and involves making sure the two sensors are not lagged from one another, and making sure that their signals are smoothed enough that the mismatch in how the sensors respond to changes are smoothed over.

\begin{figure}[hbt]
  \begin{center}
    \includegraphics[width=4in]{./figs/SalinitySpike.pdf}
    \caption{Salinity spiking due to conductivity and temperature sensors not being aligned; here the temperature sensor lags the conductivity sensor, so the cold change with depth is registered in the conductivitsy sensor before the temperature. Note the conductivity change is completely due to temperature.}
    \label{fig:SalinitySpike}
  \end{center}
\end{figure}

Regardless of measurement issues, the measurement of salinity using conductivity and temperature is very wide spread.  In order to differentiate such measurements from fundamental chemically based estimates, oceanographers often use the units of "practical salinity units" or "psu".  When these units are used, the measurement is almost always from a CTD.  More recently, there is a new salinity unit used as part of an update to the equation of state (\href{www.teos-10.org}{TEOS10}), where salinity is denoted as "absolute salinity" (AS), and the units are g/kg.  For this class we will use the older ``psu''a.

\section{Pressure}
\label{sec:pressure}

Pressure is the force per unit area that a parcel of fluid exerts on its neighbours.  Remember that in a fluid (gas or liquid), the molecules are in free motion.  As discussed above, the more kinetic energy in this motion, the higher the temperature of the fluid.  That same kinetic energy exerts itself as a force on any container that holds the molecules. Consider a ballon filled with air.  The air inside the balloon is pushing out against the ballon with a certain force per area.  This happens because the molecules of air are moving in random motion due to their heat, and some of that momentum is exerted against the balloon's surface.  Force always has a direction, but pressure forces are omni-directional or \emph{isotropic}.  So in the case of the balloon, the force exerted by the pressure is always perpendicular to the surface of the balloon.

\begin{figure}[h]
    \includegraphics[width=2.8in]{figs/Balloon.png}
    \caption{Sketch of how molecules in a fluid exert pressure on a balloon and on one another. First blowup is a schematic of air molecules hitting the balloon, second is of molecules on one side of a surface inside the air hitting molecules from the other side and the two sets of molecules therefore exerting a force on one another.  Note that in order to get a force we must specify an area element, which we've called $\delta A$ here.  }
    \label{fig:Ballon}
\end{figure}

Inside the balloon, the molecules  exert a pressure force on one another. If we insert an area element somewhere in the balloon, the molecules on the left side of the barrier exert a force to the right, and the molecules on the right side exert a force to the left.  If the pressure is the same on both sides of the area element, then the forces are equal and opposite and the fluid will be stationary.  If there are \emph{pressure gradients} inside the balloon, then one side may push harder than the other and the fluid will move.  We discuss pressure gradients a great deal in this class and will come back to them below (\fref{sec:pressure-gradients}).

\subsection{Measuring pressure}

This is usually relatively straight forward using a \href{https://en.wikipedia.org/wiki/Strain_gauge}{strain gauge}.  The CTD is equipped with a pressure gauge that is typically sensitive to better than .0025\% of their range.  Hence a deep-ocean pressure sensor capable of measuring to 7000 m will have an accuracy of 0.14 dbar.

The units of pressure are ``officially'' Pascals, with $1\ Pa = 1\ N\,m^{-2}$.  However, oceanographers almost always  report in the units of ``decibars'', where $1\ Pa = 10^{-4}\ dbar$.  The reasoning behind this is straight forward: pressure at 1-m depth is $g \rho_0 \Delta z = 9.8 \times 1023 \times 1 = 10025\ Pa = 1.0025\ dbar$.  So the pressure at a depth of 1-m is very close to 1 dbar.

\section{Calculating density}

Density is the mass per volume of a substance, and expressed in $\mathrm{kg\ m^{-3}}$.  The density of seawater varies from approximately $1000\ \mathrm{kg\ m^{-3}}$ for freshwater near the surface at 4 degrees C, to $1050\ \mathrm{kg\ m^{-3}}$ in the very deepest ocean, at salinities of 35 psu, temperature $-1\ \mathrm{^oC}$ and at a depth of 5000 m.  Notable is the fact that density of seawater only varies by about 5\% over the world's oceans.

Density differences in seawater give rise to pressure differences, and hence help drive currents in the ocean; less dense water wants to float on top of more-dense water (for example in \fref{fig:lock-exchange}) .  Unfortunately, there is no direct way to measure density \emph{in-situ}, and scooping volumes of water out of the ocean to weigh is highly impractical.

\begin{figure}[htbp]
    \includegraphics[width=3in]{./figs/ShinEtAl04Fig2.pdf}
    \caption{Lock exchange with dense water on left and light water on the right. \citep{Shinetal04}}
     \label{fig:lock-exchange}
\end{figure}

Instead, we rely on \emph{empirical} relationships between density and the ``state'' of the seawater.  The state variables that give the density are  temperature, salinity, and pressure.  Increasing the temperature of water tends to decrease the density (except near the freezing point), while increasing the salinity and pressure tend to increase the density.

\subsection{Density relation}

\begin{marginfigure}
    \includegraphics[width=2.7in]{./figs/Density.pdf}
    \caption{Density as a function of T and S at the sea-surface.  Contours are the density, the red line is the freezing point at the sea surface.  }
    \label{fig:Density}
\end{marginfigure}

The equation for density is a non-linear function of temperature, salinity, and pressure, and is determined empirically.  Various equations of state have been published over the years, but the typically used ones are EOS80 and TEOS-10.  See the \href{http://www.teos-10.org}{TEOS-10} website for details and references.  The differences between these are subtle, and for our purposes we will use the older EOS80.

The non-linearity of the equation of state is clear in \fref{fig:Density}.  For a given salinity, the density depends on the temperature in a non-linear fashion, with density changing more quickly with temperature at high temperatures and more slowly at low temperatures, and indeed changing sign at the lowest temperatures, with density dropping again as temperatures get lower.  Practically, this means that we must compute the density of seawater on a computer; there are specialized software packages to do this (\texttt{seawater} and/or \texttt{gsw} routines in Matlab, python, or R).

Often, however, it is useful to \emph{linearize} the equation of state if T, S, and P do not vary much.  In that case we can write:

\begin{eqnarray*}
    \rho & \approx & \rho_0 \left[ 1 - \alpha \left(T - T_0\right) + \beta \left(S - S_0\right) + \gamma \left(P - P_0\right)\right]\\
    \Delta \rho & \approx & \rho_0\left[- \alpha \Delta T + \beta \Delta S + \gamma \Delta P\right]
\end{eqnarray*}

$\alpha$ is called the ``thermal expansion'' co-efficient, and for values  $T_0, S_0, P_0 = 15, 30, 0$ is $\alpha \approx 2\times 10^{-4}\ \mathrm{[^oC^{-1}]}$; so for a change of temperature of $+1\ ^oC$ the density decreases $0.2 \ \mathrm{kg\, m^{-3}}$.  At the same values, $\beta \approx 7\times10^{-4}\ \mathrm{psu^{-1}}$, so a change of 1 psu increases the density by $0.7 \ \mathrm{kg\, m^{-3}}$.  For the same values the value of $\gamma \approx 4.5\times10^{-6} \ \mathrm{dbar^{-1}}$, so moving a parcel 1 dbar higher pressure will increase its density by $+4.5\ \times 10^{-3} \ \mathrm{kg\, m^{-3}}$.  Note that $\gamma$ is relatively linear, so moving to 1000 dbar (i.e. 1000-m deep) increases the density by $+4.42\ \mathrm{kg\, m^{-3}}$.

\subsection{Potential density}
\label{sec:potential-density}

\begin{figure}[hbt]
  \begin{center}
    \includegraphics[width=4.5in]{figs/P16Pden.pdf}
    \caption{Potential density along P16 hydrographic section (north-south along approximately 150 W) relative to the sea surface (top plot) and relative to 4000 dbar (bottom plot).  Both are contoured with the same-spaced contour intervals, though the absolute values are different.  Note some near-surface contours have been omitted for clarity.}
    \label{fig:P16Pden}
  \end{center}
\end{figure}

The concept of potential density is similar to that of potential temperature (\fref{sec:potential-temperature}), where we would like to remove the pressure effect from the density in order to see where a parcel of water originated.  Unfortunately, the non-linearity of the equation of state of seawater makes it impossible to do this cleanly, in that if we change the pressure at one depth, the change of density is not the same if we change it at another depth $\Delta \rho(S, T, \Delta P, P=0) \neq \Delta \rho(S, T, \Delta P, P=4000)$.  For $S=30\ \mathrm{psu}$, $T=15\ \mathrm{^oC}$, $\Delta P = 10\ \mathrm{dbar}$ we have $0.0447\ \mathrm{kg\ m^{-3}}$ at 0 dbar, and  $0.0412\ \mathrm{kg\ m^{-3}}$  at 4000 dbar.  To deal with this, we calculate the ``potential density'' relative to a given pressure.

The most commonly used potential density is calculated relative to  the surface pressure and is signified as $\sigma_{\theta} = \rho(S, T, 0) - 1000\ \mathrm{kg\ m^{-3}}$.  Note subtracting 1000 from the density is because all ocean densities are near 1000.  Often seen are potential densities relative to 1000-m depth increments in the ocean, signified at $\sigma_1, \sigma_2, \sigma_3, \sigma_4$.  Each gives a  different view of the density along P16 (\fref{fig:P16Pden}).  If we adiabatically bring the deep water to the surface it all appears almost the same density, whereas if we bring it to 4000 m there is much more apparent ``stratification'' (more contour lines per depth).  If we are interested in the dynamics of the deep sea, these adjustments are crucial.

The contours of potential density in \fref{fig:P16Pden} are called ``isopycnals''.

As a vivid example of how the equation of state depends on both temperature and salinity, again consider the section along P16 in the Pacific (\fref{fig:P16PdenwTS}).  The potential temperature and salinity is contoured with potential density contours.  It is clear that water at a given density is colder and fresher as you move north, and hence both temperature and salinity are necessary to understand the density of the ocean.

\begin{figure}[hbt]
\includegraphics[width=4.5in]{./figs/P16PdenwTS.pdf}
\caption{Potential temperature and salinity contoured with isopycnals anlong section P16 in the Pacific Ocean}
\label{fig:P16PdenwTS}
\end{figure}

Another way of looking at the same data that we will use a lot in this course is to plot the data in $\theta-S$ space, where each dot is a separate measurement of temperature and salinity (\fref{fig:P16TSzoom}).  The red data dots are in the south, and blue in the north, and along most isopycnals, the water becomes fresher and colder as profiles from progressively further north are looked at.

\begin{figure}[hbt]
    \includegraphics[width=3.0in]{./figs/P16TSzoom}
    \caption{Potential temperature as a function of Salinity along P16. Red data points were collected in the south, and blue data points in the north.  Isopycnals (relative to atmospheric pressure) are contoured.  Note only water below 1000-m is shown here.}
    \label{fig:P16TSzoom}
\end{figure}

\subsection{Cabbeling}

\begin{marginfigure}
    \includegraphics[width=2.5in]{./figs/Cabelling.pdf}
    \caption{Density as a function of T and S at the sea-surface.  Contours are the density, the red line is the freezing point at the sea surface.  }
    \label{fig:Cabelling}
\end{marginfigure}

A final note about the equation of state of seawater is the curious effect known as \Wikiref{cabbeling}. As we will discuss in this course, mixing waters of different densities requires energy to overcome the density differences between the water masses.  Therefore water tends to be relatively sorted with depth, with dense water under light.  However, there exist substantial temperature-salinity (T/S) differences in the ocean, and as we can see from \fref{fig:Density} there is a whole range of T/S properties for any one density contour. It requires very little energy to mix two water parcels on the same \emph{isopycnal}.

However, if we mix two water masses from the same isopycnal, throughout most of the ocean it will lead to a new, slightly denser water mass, as illustrated \fref{fig:Cabelling}.  If equal volumes of water masses with T/S properties signified by the red dots along the $\sigma_{\theta} = 24\ \mathrm{kg\,m^{-3}}$ isopycnal are mixed, their T/S properties fall exactly half way between the parents (green dot). But a water mass with those T/S properties has a density of $\sigma_{\theta} = 24.022\ \mathrm{kg\,m^{-3}}$, which is a bit denser than the parent water masses.  This curious effect is believed to cause circulation in the ocean, particularly near sharp fronts and at high latitudes where the effect is most pronounced.

\section{Other seawater properties}

The properties described above are the main ones dealt with in this course, but there are a few others that are important.

\subsection{Sound}

Sound speed in the water depends on its state variables (S, T, and P), and again can be looked up in empirical fits, typically ranging between 1450 and 1550 m/s (\fref{fig:SoundSpeed}), with higher speeds for warmer and saltier water.

 \begin{marginfigure}
 \begin{center}
    \includegraphics[width=2.5in]{./figs/SoundSpeed.pdf}
    \caption{Speed of sound in waters as a function of T and S at the sea-surface. }
    \label{fig:SoundSpeed}
 \end{center}
\end{marginfigure}

Sound decays in seawater more slowly than in air, with higher frequency sounds decaying over shorter distances.  Sound also tends to spread spherically, so its energy falls off with the cube of the distance from the source (i.e. $r^{-3}$).  However, a peculiarity of the sound profile in the ocean means that there is a minimum in the sound speed approximately 500-1000 m deep in the ocean.  This occurs because sounds speed drops with temperature in the upper kilometer of the ocean, but increases with pressure in the bottom kilometer, where the temperature doesn't change very quickly \fref{fig:sofar}.  This minimum in the sound speed creates a (sound) waveguide called the \Wikiref{SOFAR} channel (Sound Fixing and Ranging).  Waves always refract towards a slower medium, so a sound emitted in the SOFAR channel will start to move upwards and downwards in the water column, but the waves (and hence sound energy) will be refracted back towards the SOFAR channel.  This means that instead of sound falling off as $r^{-3}$, it falls off as $r^{-2}$, which is much slower.  This effect makes it possible for submarines and whales to communicate over long distances using low-frequency sound.

\begin{figure}[hbtp]
  \begin{center}
    \includegraphics[width=4in]{./figs/sofar-color-big.png}
    \caption{Schematic of the SOFAR channel, left is sound speed profile, and on right are sound rays sketched from a source in the SOFAR channel (from \href{https://dosits.org/science/movement/sofar-channel/}{Sound in the Sea}) }
    \label{fig:sofar}
  \end{center}
\end{figure}

\subsection{Light transmission}

While sound can propagate long distances in the ocean, light cannot.  In clear water, less than 10\% of incoming light penetrates deeper than 50 m (\fref{fig:Light-Intensity}), with blue light (short wavelengths) getting deeper than red light (long wavelengths). Note that the longer-wavelength red light decays the fastest, whereas longer wavelength sound waves decay more slowly.  Thats because the decay of light is due to absorption by the water molecules.

\begin{marginfigure}
    \includegraphics[width=2.0in]{figs/Light-Intensity.png}
    \caption{Profiles of light penetration in the ocean for different colors.  \href{https://manoa.hawaii.edu/exploringourfluidearth/media_colorbox/2788/media_original/en}{Inouye; Exploring Fluid Earth, U.\ Hawaii}}
    \label{fig:Light-Intensity}
\end{marginfigure}

The fact that light does not transmit very far in the ocean is an important difference with the atmosphere, where light is relatively transparent to air.  It means that radiative heat that comes in from the sun is mostly deposited in the upper few meters of the ocean, and we can think of it as deposited at the surface.  It also has profound effect on biological productivity of the ocean, since it means photosynthetic plant growth has to take place in the upper 10s of meters.

\subsection{Ice}

The freezing point of water is indicated in \fref{fig:Density}, and is zero degrees for fresh water and decreases to less than zero for salty open-ocean water.  The effect of freezing ice on the ocean circulation is profound. Of course, locally it has the effect of insulating the ocean from the atmosphere.  Once ice forms on the surface, the loss of heat to the cold atmosphere drops precipitously, as any heat exchange has to propagate through the ice.  Further, the effects of wind on the ocean drop, as the wind has to move the more-solid ice floes around before it can have an effect on the ocean below.  Of course the ice floes do move, but the effect on the upper ocean is greatly dampened.  The wind also generates waves that can propagate under water (\emph{internal waves}), and these are greatly damped by ice cover.  Finally, ocean surface waves are radically damped, leading to a calmer sea state, and reduced \Wikiref{fetch (geography)} over which the waves can develop.

The other very important effect of ice on the ocean is for deep- and bottom-water formation.  The densest waters in the ocean are formed near ice edges for two reasons.  The first is that these regions tend to be cold, and cold water is dense.  The second is that when ice freezes the salt crystals are not able to stay in the ice, a process known as \Wikiref{brine rejection}. The salt sinks away from the forming ice and is dissolved in the water underneath the ice-formation region leading to very cold and salty water.  This water sinks, and though it mixes as it goes, it remains dense enough to sink to the bottom of the ocean.

\begin{figure}
       \includegraphics[width=3.2in]{figs/Antarctic_shelf_ice_hg}
       \caption{Schematic of ice processes off the Antarctic continental shelf.  Catabatic winds blow off the continent and push the ice offshore.  New ice is formed, dense water produced by brine rejection and cooling, and the dense water sinks, bringing up new warm, fresher water from mid-depth. (From \href{https://en.wikipedia.org/wiki/Polynya}{Hannes Grobe, Alfred Wegener Institute for Polar and Marine Research, Bremerhaven, Germany})}
       \label{fig:AntarcticShelf}
\end{figure}

Deep-water formation tends to occur in ``marginal seas'' like the Wedell or Ross Seas off Antarctica, and the Greenland/Iceland seas in the north.  Dense water is produced in these regions at the ice edge, or in breaks in the ice forced by wind or by tidal mixing called \Wikiref{polynya}s. Polynyas tend to dominate the air-sea exchange of heat and brine production because they are open water but exposed to very cold and often windy conditions.  The wind forces the polynya to stay open by pushing ice downwind, but also drives dense water formation due to both brine rejection, sensible, and latent heat loss (\fref{fig:AntarcticShelf}).  This deep water forms \Wikiref{Antarctic bottom water} and \Wikiref{North Atlantic deep water} masses that can be traced throughout the world's oceans as anomalously cold and salty water;  the closer the water is observed to the source region, the colder and saltier it is.  This water can be clearly seen in T/S sections on the major ocean basins (\fref{fig:P16TempSalt}). We will discuss this substantially more in the next few weeks.

\begin{figure}[hbt]
  \begin{center}
    \includegraphics[width=4.5in]{./figs/P16TempSalt.pdf}
    \caption{North-south sections in the Pacific ocean at a longitude of 150 W of potential temperature, $\theta$ and salinity. Note the Antarctic bottom water starting south of -60 Lat, and pouring northwards into the Pacific.}
    \label{fig:P16TempSalt}
  \end{center}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Pressure Gradients}

Pressure is a very important quantity to the dynamics of the oceans,
and fluids in general.  Pressure \emph{gradients} give rise to net
accelerations that cause water to move, often in surprising ways.  There was some introduction to this qualitatively in the estuaries discussion.  Here we solidify those concepts quantitatively because we need to understand pressure gradients to make progress on understanding how the ocean moves.

\begin{figure}[htbp]
    \includegraphics[width=3.5in]{./figs/PGFig1.pdf}
    \caption{A tank with the water surface tilted from equilibrium.  The surface wants to flatten, so water must be moved from area A to area B.  What is the action of a dye streak through the water column?}
    \label{fig:PGFig1}
\end{figure}

To motivate ourselves, recall the demo of sloshing water in a tank (\fref{fig:PGFig1}).  The surface interface is tilted such that area A is elevated, and area B is depressed.  Our intuition might say that the water needs to run downhill, and we may expect a flow, confined to the water surface, of water from area A to area B. Under this hypothesis, what do we expect the dye streak to do?  What does the dye stream actually do (approximately)?

To understand what is happening, we must understand pressure in the fluid.  The basic definition was given above (\fref{sec:pressure})

\subsection{Hydrostatic Pressure}

``Dynamic'' pressure happens for fast flows, like those over an
airplane wing.  Fortunately, in the ocean things are usually slow
enough that we can ignore the ``dynamic'' part, and focus on the
``hydrostatic'' part.

\begin{figure}[htbp]
\includegraphics[width=3.5in]{figs/OneDens.png}
\caption{The pressure in the small orange box is  the weight per area of the water above.  $\delta y $ is in into the page, so $\delta x\ \delta y$ is the area of the top of the fluid parcel.  If the green column of water is static, then the two forces are equal and opposite.  }
\label{fig:OneDens}
\end{figure}

\subsection{Single density fluid}
``Hydrostatic pressure'' is the pressure water parcels exert on one another when the water is not moving.  Consider a
bucket of water, and think about the forces on a little ``cube''  of
water inside the bucket (\fref{fig:OneDens}, orange cube).  Suppose our cube is 0.1 m deep in the
bucket. There is a column of water above our little cube that is not
moving.  The force of gravity wants to pull this column of water
down (\fref{fig:OneDens}, green arrow).  The force is simply $F_{w}=mg$, where $g=9.8\, \mathrm{m\,s^{-2}}$, the mass $m=\rho V$, where
$\rho = 1000\ \mathrm{kg\,m^{-3}}$ is the density of water, and $V$ is the
volume of the column of water $V = (0.1 \ \mathrm{m})\,\delta x\,\delta y$.  So, the water column
is pushing down with force
\begin{eqnarray*}
F&=&\delta x\,\delta y\,(0.1\,\mathrm{m})\,
(9.8\ \mathrm{m\,s^{-2}})\,(1000\ \mathrm{kg\,m^{-3}})\\
&\approx &\delta x\,\delta y\ 980\ \mathrm{N\,m^{-2}}\\
F/A &\approx & 0.098\ \mathrm{dbar},
\end{eqnarray*}
where we have called $\delta x\, \delta y$ our area element (i.e. $\delta A$ from \fref{fig:Ballon}).
What provides the force that holds this water up, against gravity?  Its the water in our small cube. This water provides a force from directly below $F_{p}=P\,\delta x\,\delta y$ (\fref{fig:OneDens}, orange arrow).  If the water is not moving, $F_{p}=-F_{w}$, so that the sum of forces in the vertical is zero.
We can use this deduction to calculate the pressure at a depth $h$:
\begin{equation}
P(z=-h) = \rho\,g\,h.
\end{equation}
In other words, the pressure is the weight of the water above, per unit area.

It is important to remember that the squishing from the top also causes squishing to the sides, so the water pressurized by the water above also exerts forces on the water in the horizontal direction.  If the water beside our small cube has the same pressure forces acting on it, then the water won't move.

\subsection{Two-density fluid}

\begin{figure}[htbp]
\includegraphics[width=2.7in]{figs/TwoDens.png}
\caption{Pressure in a two-layer fluid.  Imagine the first layer has a different density than the second, and that the densities are (approximately) constant in each layer.}
\label{fig:TwoDens}
\end{figure}

If the density of the water varies due to changes in temperature or salinity, then the weight needs to be calculated by summing up the different densities with depth.  i.e. imagine that the first meter, the density is $1000\ \mathrm{kg\,m^{-3}}$, and for the second meter, the density is $1010\ \mathrm{kg\,m^{-3}}$ (\fref{fig:TwoDens}).  Then the pressure at 2 m is:
\begin{eqnarray}
  P(z)&=&\int \rho(z)\,g\ \mathrm{d}z\\
  &\approx&\sum_{j=1}^{n} \rho_{j}\, g\ \Delta z_{j}\\
  & =& (1000\times1\times9.8\ +\ 1010\times1\times9.8) \ \ \mathrm{N\,m^{-2}}\\
  &=& 19698\ \mathrm{N\,m^{-2}}
\end{eqnarray}
Adding more layers to the water column, or making the layers different thicknesses is just handled by adding more terms to the sum.  Often it is useful to use a computer for this kind of calculation!

\section{Pressure gradients}
\subsection{Surface pressure gradients}
\label{sec:pressure-gradients}

We wouldn't care too much about pressure if it did not cause water to move.  Consider a sloshing bathtub, mid-slosh (\fref{fig:Bathtub}).  In this situation it is intuitively obvious that the water wants to move from left to right, but what force is pushing it?  First, where is pressure the highest at point A or point B?   These points are both a depth $h$ below the \emph{resting} depth of the surface of the fluid.  Above point A, there is slightly more water due to the ``slosh'', whereas above point B there is slightly less.  Therefore the pressure at point A is greater than at point B.  ($P_{A}(z=-h) = \rho g (h+\eta_{A})$ and $P_{B}(z=-h) = \rho g (h+\eta_{B})$, where $\eta$ is the height of the water above its resting depth. $\eta_{A}>0$, and $\eta_{B} <0$, therefore $P_{A}>P_{B}$).

It should also be obvious that \emph{everywhere} in the fluid the pressure is decreasing to the right along any given depth i.e. $dP/dx<0$.  In fact the \emph{horizontal pressure gradient} can be calculated from
\begin{equation}
P(z=-h) = \rho g (h+\eta)
\end{equation}
and taking the derivative to get the gradient:
\begin{equation}
\frac{\mathrm{d}P}{\mathrm{d}x}(z=-h) = \rho g \frac{\mathrm{d}\eta}{\mathrm{d}x}.
\end{equation}
or, the change in the pressure is caused by the slope of the sea surface, $\mathrm{d}\eta / \mathrm{d}x$.

\begin{figure}[htbp]
\begin{center}
\includegraphics[width=4in]{figs/Bathtub.png}
\caption{Pressure in a sloshing tub.  For pressure gradients, we always compare the pressure at the same distance from the surface of the \emph{if it was flat} (i.e. the geoid), which here we have denoted $z=-h$.  The sea surface is described as the anomaly from this flat surface, $\eta >0 $ means the surface is above the flat level, and $\eta<0$ means that it is below.}
\label{fig:Bathtub}
\end{center}
\end{figure}

How does this move the water?  Lets consider the force diagram on a block of water inside the tub (\fref{fig:ForceDia})  If the pressure is greater on the left hand side, then the force into the block from the left is larger than the force into the block on the right, and the net force $F_{A}-F_{B}>0$, and the block tends to move to the right.  Note that in this case $\mathrm{d}P/\mathrm{d}x<0$.

Recall Newton's second law of motion:
\begin{equation}
    \sum_{i=1}^{N} F_i = m \frac{du}{dt}
\end{equation}
where $du/dt$ is the acceleration (change of velocity with time).  For our block, $m = \rho\ \delta x\ \delta y\ \delta z$,  and $F_A = P_A \ \delta y\ \delta z$, and $F_B = - P_B \ \delta y\ \delta z$, so:
\begin{equation}
    P_A - P_B = \rho\ \delta x\ \frac{du}{dt}
\end{equation}
or, as $\delta x$ is made very small:
\begin{equation}
\frac{\mathrm{d}u}{\mathrm{d}t} = -\frac{1}{\rho}\frac{\mathrm{d}P}{\mathrm{d}x} + (\mathrm{other\ Forces\ in\ x})
\end{equation}
A question for the reader is to check that this gives the correct sign of the acceleration compared to their intuition.

\begin{figure}[htbp]
    \includegraphics[width=3in]{figs/ForceDia.png}
    \caption{Lateral forces across a small hypothetical block.}
    \label{fig:ForceDia}
\end{figure}

For the example of the bathtub given above, it should also hopefully be clear that the pressure gradient, and thus the acceleration, does not depend on the depth below the surface.  The pressure gradient, $dP/dx$, only depends on the sea surface tilt, not on $z$.

\subsection{Internal pressure gradients}

If the fluid has varying density, then the pressure field can be more complicated, and the motions harder to predict.  For the simplest case, consider a two-layer fluid, with a flat upper surface, and a tilted interface between two layers with densities $\rho_{1}$ and $\rho_{2}$ (\fref{fig:TwoLayer}) with $\rho_{2}>\rho_{1}$.  Again, the pressure is lower on the right than left, so water in the deeper layer has a pressure gradient force from high pressure to low pressure, i.e. to the right.
\begin{figure}[htbp]
\begin{center}
\includegraphics[width=4in]{figs/TwoLayer.png}
\caption{How to calculate the pressure gradient in a two-layer fluid.}
\label{fig:TwoLayer}
\end{center}
\end{figure}

Mathematically, $P_{A}(z=-h) = g(\rho_{1} h_{1}(x_{A}) + \rho_{2} (h-h_{1}(x_{A})))$, and
$P_{B}(z=-h) = g(\rho_{1} h_{1}(x_{B}) + \rho_{2} (h-h_{1}(x_{B})))$.  We can estimate the pressure gradient as
\begin{equation}
\frac{\mathrm{d}P}{\mathrm{d}x} \approx \frac{P_{B}-P_{A}}{x_{B}-x_{A}} = g(\rho_{1}-\rho_{2})\frac{h_{1}(x_{B})-h_{1}(x_{A})}{x_{B}-x_{A}} = -g(\rho_{2}-\rho_{1})\frac{\mathrm{d}h_1}{\mathrm{d}x}
\end{equation}
So again, the pressure gradient depends on the slope of the interface, except this time, the interface is between the two densities.

Of course, the cases can be combined (\fref{fig:MultipleLayers}).  There can be a surface tilt, and many interface tilts.  While care needs to be taken to keep track of all the layer thicknesses, and the algebra looks messy on the page, the fundamental idea is to simply calculate the weight of the water about the point of interest, including the effect of the local sea-surface elevation.

\begin{figure}[htbp]
\begin{center}
\includegraphics[width=4in]{figs/MultipleLayers.png}
\caption{Many-layer fluid.}
\label{fig:MultipleLayers}
\end{center}
\end{figure}

\clearpage
\section{Exercise}

A cruise goes out and makes temperature measurements at two locations 25 km apart as follows:

  \begin{tabular}{l|cc}
    $z$& $T_A$& $T_B$\\
    \hline
    10 & 10.4& 10.1 \\
    30 & 8.7 & 8.4\\
    50 & 4.7 & 5.2\\
    70 & 4.1 & 4.8\\
  \end{tabular}

The density of fresh water at $10\mathrm{^oC}$ is $1008\ \mathrm{kg\,m^{-3}}$.  What is the \emph{approximate} density at each location where a temperature measurement was taken?

What direction would you expect the water to be flowing at each depth?

Approximate the strength and sign of the horizontal hydrostatic pressure \emph{gradient} at 20, 40, 60 and 80 m, assuming a flat sea surface. Make sure you write out the equation so I can check your math. Does your answer make sense with the answer above? Hint: discretize the water column by assuming that it consists of 4 20-m layers of water, each with a constant temperature. Also, don't round off your density values too much!

What must the difference in the sea surface height be at the two stations for your estimated horizontal pressure gradient to be zero at 40 m?

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:


\chapter{Vision Workbench Type System}

The Vision Workbench is an example of what is often called a
``multi-paradigm'' C++ library.  That is, different components of the
library adopt different C++ programming models, such as the generic
programming model or the object-oriented programming model, often in
combination.  At the core, however, is a set of data types and related
tools that fall largely within the template-based generic programming
paradigm.  The purpose of this chapter is to describe this core type
system in some detail.  If your intention is simply to {\it use} the
Vision Workbench for image processing tasks you can probably afford
to skim or even skip this material.  The primary intended audience is
programmers who wish to extend the Vision Workbench's core
capabilities in one way or another.

Data types in the Vision Workbench can be broadly divided into three
categories: the fundamental data types, including the simple numeric
types; the compound types, such as RGB pixel types and vectors; and
the container types, such as images.  We shall discuss each of these
in turn.  (Other C++ types, such as functors, do make an appearance in
the Vision Workbench, but these are not {\it data} types {\it per se}.)

\section{The Scalar Types}

\begin{table}[t]\begin{centering}
\begin{tabular}{|c|l|l|} \hline
Type & Description & Notes \\ \hline \hline
\verb#int8# & Signed 8-bit integer & \\ \hline
\verb#uint8# & Unsigned 8-bit integer & Most common for low-dynamic-range imaging \\ \hline
\verb#int16# & Signed 16-bit integer & \\ \hline
\verb#uint16# & Unsigned 16-bit integer & \\ \hline
\verb#int32# & Signed 32-bit integer & \\ \hline
\verb#uint32# & Unsigned 32-bit integer & \\ \hline
\verb#int64# & Signed 64-bit integer & \\ \hline
\verb#uint64# & Unsigned 64-bit integer & \\ \hline
\verb#float32# & 32-bit floating point & Most common for high-dynamic-range imaging \\ \hline
\verb#float64# & 64-bit floating point & \\ \hline
\end{tabular}
\caption{The core Vision Workbench scalar {\tt typedef}s, defined in {\tt <vw/FundamentalTypes.h>}.}
\label{tbl:scalar-types}
\end{centering}\end{table}

The most fundamental data types of all are the built-in C++ integral
and floating-point numeric types.  In scientific programming contexts
like image processing and machine vision it is generally important to
be able to specify the exact nature of the numeric data types that you
are working with.  Unfortunately the C++ language makes few promises
about the sizes of data types such as \verb#int# or even \verb#char#.
To work around this limitation, the Vision Workbench provides a number
of portable \verb#typedef#s that you are encouraged to use instead.
These are listed in Table~\ref{tbl:scalar-types}.  (In fact most of
these data types are simple wrappers around similar types provided
by the Boost \verb#cstdint# library.)

The Vision Workbench uses standard C++ complex numbers, as defined
in the standard header file \verb#<complex>#.  The \verb#std::complex<>#
class takes a single template parameter, the underlying numeric type
to be used for the real and imaginary components, which should usually
be one of the scalar types listed in Table~\ref{tbl:scalar-types}.
In particular, it is almost always best to use one of the floating-point
types for complex numbers.  For example, \verb#std::complex<float64>#
is a good choice for use in frequency-domain image processing
(discussed in Section~\ref{sec:advanced.frequency}).

There is a special type trait template, \verb#IsScalar<>#, that you
can use in template code to determine whether or not a type is a
simple numeric type like we have described in this section.  It
inherits from either \verb#boost::true_type# or \verb#boost::false_type#
accordingly.  Its primary use is to prevent template functions such
as scalar multiplication from being too general:
\begin{verbatim}
  template <class ScalarT>
  typename boost::enable_if< IsScalar<ScalarT>, MyClass >::type
  operator*( MyClass const& m, ScalarT s ) {
    /* compute result */
  }
\end{verbatim}
In this example we use the Boost \verb#enable_if# library to restrict
the definition of the \verb#*# operator to cases where the second
argument really is a scalar.  Without this restriction this would
have been an overly-general function definition and would likely
have caused problems if we had attempted to defined any other
product for the \verb#MyClass# class later on.  If you decide to
extend the Vision Workbench to support additional scalar types,
such as bigints, you should specialize \verb#IsScalar<># accordingly
to ensure proper behavior.

\section{Type Deduction}

\begin{table}[t]\begin{centering}
\begin{tabular}{|c|l|} \hline
Class & Description \\ \hline \hline
\verb#SumType<T1,T2># & Result type of a sum operation \\ \hline
\verb#DifferenceType<T1,T2># & Result type of a difference operation \\ \hline
\verb#ProductType<T1,T2># & Result type of a product operation \\ \hline
\verb#QuotientType<T1,T2># & Result type of a quotient operation \\ \hline
\end{tabular}
\caption{The Vision Workbench type deduction classes, defined in {\tt <vw/TypeDeduction.h>}.}
\label{tbl:type-deduction}
\end{centering}\end{table}

The C++ language has many intricate rules for type promotion and
deduction in complex mathematical expressions.  Unfortunately it
provides no built-in mechanism to extend this automatic type deduction
system or query its behavior.  Consider adding two images with
compatible but different pixel types: what should the resulting
pixel type be?  The Vision Workbench provides a standard set of type
deduction traits classes, defined in \verb#<vw/TypeDeduction.h># and
listed in Table~\ref{tbl:type-deduction}, that allow you to both query
and specialize the type deduction behavior of Vision Workbench types.
Like all Vision Workbench type computation classes, they ``return''
their result types in a member type named \verb#type#.

As a trivial example, imagine writing a template function that simply
computes the sum of its two arguments.  What should its return type
be?  We can use \verb#SumType<># to compute it:
\begin{verbatim}
  template <class T1, class T2>
  inline typename SumType<T1,T2>::type sum( T1 a, T2 b ) {
    return a + b;
  }
\end{verbatim}
Remember that the C++ language does not allow this to be fully
automatic.  If you define a new type with an unusual addition
operator, you will need to manually specialize \verb#SumType<># at the
same time.  However, the most common default behaviors are provided.
For example, any built-in type is assumed to be promoted to any
user-defined type, and any user-defined type operating with itself is
assumed to return itself.  These type deduction classes do also
replicate the standard C++ promotion behavior when used with the
built-in numeric types.

\section{The Pixel Types}

It is possible to use any of the fundamental scalar types described in
the previous section as an \verb#ImageView#'s pixel type.  However in
most circumstances a {\it compound} pixel type, consisting of one or
more channels with associated semantics, is more appropriate.  The
Vision Workbench provides several compound pixel types corresponding
to the most common color spaces used in image processing.  These are
listed in Table~\ref{tbl:color-pixel-types}.  Each is a template
class taking one template parameter, the scalar type used to store
the channels.  For example, the native pixel type of a standard JPEG
image is represented by \verb#PixelRGB<uint8>#.

\begin{table}[t]\begin{centering}
\begin{tabular}{|c|l|l|} \hline
Type & Description & Channels \\ \hline \hline
\verb#PixelGray<T># & Grayscale & Grayscale value (\verb#v#) \\ \hline
\verb#PixelGrayA<T># & Grayscale w/ alpha & Grayscale value (\verb#v#), alpha (\verb#a#) \\ \hline
\verb#PixelRGB<T># & RGB & Red (\verb#r#), green (\verb#g#), blue (\verb#b#) \\ \hline
\verb#PixelRGBA<T># & RGB w/ alpha &  Red (\verb#r#), green (\verb#g#), blue (\verb#b#), alpha (\verb#a#) \\ \hline
\verb#PixelHSV<T># & HSV & Hue (\verb#h#), saturation (\verb#s#), value (\verb#v#) \\ \hline
\end{tabular}
\caption{The Vision Workbench color-space pixel types, defined in {\tt <vw/PixelTypes.h>}.}
\label{tbl:color-pixel-types}
\end{centering}\end{table}

\begin{table}[t]\begin{centering}
\begin{tabular}{|c|l|l|} \hline
Syntax & Description \\ \hline \hline
\verb#PixelChannelType<PixT>::type# & The pixel type's underlying channel type \\ \hline
\verb#PixelNumChannels<PixT>::value# & The number of channels in the pixel type \\ \hline
\verb#PixelChannelCast<PixT,ChT>::type# & The same pixel type with a new channel type \\ \hline
\hline
\verb#PixelIsCompound<PixT># & Is the pixel type a compound type? \\ \hline
\verb#PixelMakeReal<Pixt>::type# & Converts to the corresponding real channel type \\ \hline
\verb#PixelMakeComplex<Pixt>::type# & Converts to the corresponding complex channel type \\ \hline
%\verb#apply_per_pixel_channel(func,pixel)# & The result of applying \verb#func(ch)# to each channel \\ \hline
%\verb#apply_per_pixel_channel(func,pixel1,pixel2)# & The result of applying \verb#func(ch1,ch2)# to each channel \\ \hline
\end{tabular}
\caption{The pixel traits and pixel type computation classes.}
\label{tbl:pixel-traits}
\end{centering}\end{table}

Several type trait classes are provided for use in writing generic
pixel manipulation code and are listed in Table~\ref{tbl:pixel-traits}.
The first section of the table lists the classes that you must specialize
when you write a new compound pixel type.  The second section of
Table~\ref{tbl:pixel-traits} lists convenience types that are defined
in terms of the other, specialized types. For example, here is how
the types are specialized for the RGB pixel type:
\begin{verbatim}
  template <class ChannelT>
  struct PixelChannelType<PixelRGB<ChannelT> > {
    typedef ChannelT type;
  };

  template <class ChannelT>
  struct PixelNumChannels<PixelRGB<ChannelT> > {
    static const unsigned value = 3;
  };

  template <class OldChT, class NewChT>
  struct PixelChannelCast<PixelRGB<OldChT>, NewChT> {
    typedef PixelRGB<NewChT> type;
  };
\end{verbatim}
When you define a new pixel type, you will usually want to define a
provide a similar set of template specializations.  To simplify
the process, a macro is provided that you can use to automatically
specialize the templates in the usual manner:
\begin{verbatim}
  VW_DECLARE_PIXEL_TYPE( PixelRGB, 3 );
\end{verbatim}
This macro expands to the same set of template specializations shown
above, describing a pixel type named \verb#PixelRGB# with three
channels.

Note that it is {\it not} necessary to declare a type using this macro,
or even to provide specializations for the traits templates described
above, just to use that type as the pixel type for an image.  These
specializations are only necessary if you want to declare a type with
multi-channel semantics.  For any other type, the default Vision
Workbench behavior is to treat the type as a single-channel pixel type
whose channel type is equal to the type itself.

Several convenience functions are also provided to simplify working
with pixels in generic template functions.  The first is the
\verb#pixel_channel_cast<>()# function, which casts a pixel to a pixel
of the corresponding pixel type but with the specified channel type.
The syntax mirrors the built-in C++ casting functions, except the
template parameter is the new channel type instead of the new type as
a whole.  In the following example we explicitly down-cast the channel
type of a pixel from \verb#float64# to \verb#float32# in order to pass
it to a function that happens to take a \verb#PixelRGB<float32>#
argument:
\begin{verbatim}
  PixelRGB<float64> pixel;
  some_function( pixel_channel_cast<float32>( pixel ) );
\end{verbatim}
Note that this is not needed in the more common case that the
function that you wish to call is itself generic and can accept
any channel type.

Sometimes it is desirable to apply a function to each channel
of a pixel, or to corresponding channels from two pixels.  Re-scaling a
pixel or adding two pixels on a per-channel basis can be cast into
this form, for example.  You can use the generic function
\verb#apply_per_pixel_channel()# to do this.  Here is a trivial
example that demonstrates re-scaling:
\begin{verbatim}
  float32 triple(float32 v) { return 3*v; }
  // Later, in some other function...
  PixelRGB<float> pixel(.1,.2,.3);
  PixelRGB<float> result = apply_per_pixel_channel(&triple,pixel);
\end{verbatim}
In this case it would have been simpler to multiply the pixel
by $3$ directly, but the point is that we could have performed
any arbitrarily complex operation on each channel instead.
The binary form simply takes an extra pixel argument:
\begin{verbatim}
  float32 sum(float32 a, float32 b) { return a+b; }
  // Later...
  PixelRGB<float> pixel1(.1,.2,.3), pixel2(.2,.1,.4);
  PixelRGB<float> result = apply_per_pixel_channel(&sum,pixel1,pixel2);
\end{verbatim}

%% \subsection{Mathematical Operators}
%% Virtually all image processing operations depend on pixel
%% arithmetic in one way or another.


\documentclass[main.tex]{subfiles}
\begin{document}

\marginpar{Tuesday \\ 2019-11-26}

% (Based on few Giorgio notes-BOFGN hereafter)

\paragraph{Pulsation assisted winds}

The density \(\rho_{c}\) at the condensation radius \(r_C\) is critical: it determines the mass loss rate \(\dot{M}\).
If it is too low no wind can be generated.
Therefore, there is a need for a mechanism to increase the density scale height: these winds are \emph{pulsation assisted}, the driving mechanism are shocks occurring in pulsating stars.
So, \(\dot{M}\) is very sensitive to these pulsations, which ``launch'' the gas to a height sufficient for it to form dust grains.

The wind starts to be accelerated at the critical radius \(r_C\), and its acceleration is so fast that it is reasonable to assume that this critical radius is the same as the sonic radius at which the gas reaches \(v=a\).
This is the dust formation radius and also the condensation radius.

The momentum equation of the dust driven wind reads:
%
\begin{align}
v \dv{v}{r} + \frac{GM}{r^2} + \frac{1}{\rho }\dv{P}{r} = g _{\text{rad}} = \frac{GM}{r^2} \Gamma_{d}
\,,
\end{align}
%
with the \(\Gamma_{d}\) defined in \eqref{eq:definition-gamma-dust}, proportional to the opacity \(\kappa_{rp}\).

\subsection{Maximum mass loss rate: the single scattering limit}

In this case we also have a maximum mass loss rate given by
%
\begin{align}
\dot{M} _{\text{max}} v_{ \infty } = \frac{L}{c}
\,,
\end{align}
%
which is rather low since the asymptotic velocity is low.

As was the case with line driven winds, multiple scattering can enhance this maximum mass loss rate by a factor depending on the optical depth \(\tau_{w} = \int_{r_s}^{ \infty } \kappa \rho \dd{r}\), however now we also have a dependence on \(\Gamma_{d}\):
%
\begin{align}
\dot{M} _{\text{max}} = \frac{L}{v_{ \infty } c } \qty(\frac{\Gamma_{d} -1}{\Gamma_{d}}) \tau_{w}
\,.
\end{align}

\begin{bluebox}
Here is the derivation of this expression, from Lamers:
we integrate the momentum equation multiplied by \(4 \pi r^2 \rho \) from the star's radius \(R_{*}\) to infinity, but we split certain integrals into the contributions from \(R_{*}\) to \(r_s\) and from \(r_s\) to infinity. We get:
%
\begin{subequations}
\begin{align}
0 &= \int_{R_*}^{ \infty } \qty[ 4 \pi r^2 \rho  v \dv{v}{r} + 4 \pi GM \rho
+ 4 \pi r^2 \dv{P}{r} - 4 \pi GM \Gamma_{d}] \dd{r}  \\
\begin{split}
0 &= \int_{R_*}^{ \infty } 4 \pi r^2 \rho  v \dv{v}{r} \dd{r}
+ \int_{R_*}^{r_s}  \qty[\frac{1}{\rho } \dv{P}{r} + \frac{GM}{r^2} ] 4 \pi r^2 \rho \dd{r} + \\
&\phantom{=}\
+ \int_{r_s}^{ \infty } \frac{1}{\rho } \dv{P}{r} 4 \pi r^2 \rho \dd{r}
+ \int_{r_s}^{ \infty } \frac{GM}{r^2} (1 - \Gamma_{d})
\rho 4 \pi r^2 \dd{r}
\,,
\end{split}
\end{align}
\end{subequations}
%
but \(4 \pi r^2 \rho \dd{r} = \dd{m}\), so the second integral can be written as
%
\begin{align}
\int_{R_*}^{r_s} \frac{1}{\rho } \dv{P}{r}  + \frac{GM}{r^2}
 \dd{m}
\,,
\end{align}
%
which is precisely 0 if we have hydrostatic balance, and we do under \(r_s\). So this term can be neglected.
Similarly, the third integral is
%
\begin{align}
\int_{r_s}^{ \infty } \frac{1}{\rho } \dv{P}{r} \dd{m}
\,,
\end{align}
%
and it is also negligible since the pressure gradient is very low beyond the sonic point.

The first integral can be written as
%
\begin{align}
\int_{R_*}^{ \infty } v \dv{v}{r} \dd{m}
\,,
\end{align}
%
and we can recognize the expression of the convective derivative: \(\dot{v}  = v \dv*{v}{r}\) if the wind is stationary. The contribution to this integral at \(R_*\) is negligible: the term which remains is \(\dot{M} v_{ \infty }\). So, we are left with
%
\begin{align}
\dot{M} v_{ \infty } \approx - 4 \pi  GM (1- \Gamma_{d}) \int_{r_s}^{ \infty }
\rho \dd{r}
\,,
\end{align}
%
and we define \(\tau_{w} = \int_{r_s}^{ \infty } \rho \kappa_{rp} \dd{r}\) we can express this as
%
\begin{align}
\dot{M} v_{ \infty } \approx 4 \pi GM \frac{\Gamma_{d} -1 }{\kappa_{rf}} \tau_{w}
\,,
\end{align}
%
and now we can insert the expression for \(\kappa_{rf}\) which is found by inverting the definition of \(\Gamma_{d}\): we get
%
\begin{align}
\kappa_{rp} = \frac{4 \pi c GM }{L} \Gamma_{d}
\,,
\end{align}
%
so
%
\begin{subequations}
\begin{align}
\dot{M} v_{ \infty } &\approx 4 \pi  GM \frac{L}{4 \pi c GM} \frac{\Gamma_{d} - 1}{\Gamma_{d}} \tau_{w}  \\
&\approx \frac{L}{c} \frac{\Gamma_{d} - 1}{\Gamma_{d}} \tau_{w}
\,,
\end{align}
\end{subequations}
%
and far from the star \(\Gamma_{d} \gg 1\), so the expression simplifies to
%
\begin{align}
\dot{M} \approx \frac{L}{c v_{ \infty }}\tau_{w}
\,.
\end{align}

It would seem that \(\tau \) can be arbitrarily large, but this is not the case: by conservation of energy, we can see that
%
\begin{subequations}
\begin{align}
\frac{1}{2} \dot{M} v_{ \infty }^2 &< L  \\
\dot{M} &< \frac{2L}{v_{ \infty }^2}
\,,
\end{align}
\end{subequations}
%
so, comparing the formulas, we see that \(\tau_{w} < 2 c / v_{ \infty }\).
More precise considerations allow us to drop the factor \(2\).
\end{bluebox}

\subsection{Dust grain opacities}

In this lecture the attention was focused on the importance of C/O ratio in the formation of dust grains in dust driven winds.

In AGB stars the core, mainly made of heavier elements such as carbon and oxygen, is sourrounded by a helium envelope and, more externally, a hydrogen envelope.
Between these areas there exist some convective flows that can bring oxygen and carbon from the core to the more external parts of the star.
This process is called the \emph{third dredge-up}.

Reaching thanks to convection these external layers, which lay at a lower temperature, the heavy elements can form molecular bounds.
In particular the strongest bounded molecule that can be formed is carbon monoxyde (\ce{CO}).
This implies that, if the star has a C/O ratio $>1$ (i.e.\ it has more abundance of carbon over oxygen) almost all the oxygen is used to form carbon monoxyde, while the rest of the carbon can form other molecules: for this reason, in these stars we observe carbon grains.

Viceversa, in oxygen-rich stars we have with the specular mechanism, which ends in the formation of silicate grains.
At this point we started a discussion that will be continued in the following lecture about dust grains' opacity, focusing first on cross section.

We must distinguish two different kinds of opacities:
\begin{enumerate}
  \item the absorptive grain opacity \(\kappa_{d}\): this accounts for pure absorption \emph{without} scattering, it determines the pressure, so it defines the condensation radius;
  \item the radiation pressure mean opacity \(k_{rp}\):\footnote{Do note that this is a latin letter \(k\), while the other opacity is a greek \(\kappa \).} this accounts for both absorption and scattering. Scattering does not affect the temperature of the gas, but it \emph{does} affect the radiative acceleration.
\end{enumerate}

\subsection{Dust grain cross section}

Let $a$ be the radius of the grain, and $Q$ the efficency of the cross section, which depends both on \(a\) and on the wavelength \(\lambda \) of the light.
The distribution of the grain radii is as such:
%
\begin{align}
n(a) \dd{a} \approx K a^{-3.5} [a _{\text{min}} < a < a _{\text{max}} ] \dd{a}
\,,
\end{align}
%
where \(a _{\text{min}} \approx \SI{5}{nm}\) while \(a _{\text{max}} \approx \SI{250}{nm}\), while \(k\) is a normalization factor.

In this case we have a cross section $c=\pi a^2$ and, considering both absorption and scattering cross sections $c_a$, $c_s$, we have
%
\begin{equation}
c_{\text{tot}}
=c_a+c_s
=\pi a^2(Q_a(a,\lambda)+Q_c(a,\lambda))
\end{equation}
%
where $Q_a \ll Q_s$ for IR wavelengths: typically the absorptive efficiency \(Q_a\) can be modelled by \(Q_a \propto a \lambda^{-p}\), where the exponent \(p\) is around 1 in the infrared --- it then approaches 2 for longer wavelengths;\footnote{Actually the exponent is very much dependent on the chemical composition of the dust, this is an average.} on the other hand the scattering efficiency is in the Rayleigh regime, \(Q_s \propto \lambda^{-4}\).

Now we can define the global absorption opacity
%
\begin{equation}
k_\lambda=\frac{1}{\rho } \int_{a_{\text{min}}}^{a_{\text{max}}}Q_a\pi a^2 n \dd{a}
\,,
\end{equation}
%
the scattering opacity
%
\begin{align}
\sigma_{\lambda } = \frac{1}{\rho } \int_{a_{\text{min}}}^{a_{\text{max}}} Q_s \pi a^2 n \dd{a}
\,,
\end{align}
%
and the mean opacity
%
\begin{equation}
k=\int \dd{\lambda} k_\lambda
\end{equation}
%
which is the relevant quantity for the momentum equation.
\end{document}


\chapter{SLAM with Object Landmarks} \label{chap:object-slam}

In this chapter I present a SLAM formulation with object landmarks, which is scalable to medium sized indoor scenes and is robust to data association errors through the use of semantic information. Further I also present a compositional rendering method to render an accurate model frame at any given time during operation to propagate updated information to the front end in turn providing better initialization for the back end. This work is similar in spirit to a line of work started by \cite{salas-morenoSLAMSimultaneousLocalisation2013}. In this thesis however, the focus is on accurate trajectory estimation as well as object reconstruction. In essence, our work can be regarded as a bridge between feature based SLAM methods and dense SLAM methods.

\section{Related Literature}

In this section we review relevant literature in two aspects: classical geometry-based SLAM, and the application of deep semantic object detection in SLAM.
% Typically, SLAM methods incorporate semantics in two ways: (1) \textit{Loosely coupled}, where semantics are used post a completely independent map generation step. For instance, \cite{mccormac2017semanticfusion} generates a semantic map by fusing 2D segmentations in a post-processing step through bayesian updates, and \cite{grinvald2019volumetric} proposes a mapping scheme given camera trajectory to fuse geometric segmentation with Convolutional Neural Network (CNN) segmentations (2) \textit{Tightly coupled} where semantics are used directly to jointly optimize the map and camera trajectory. This tightly coupled treatment is called \textit{Object level SLAM}.

\subsection{Geometry-based SLAM}
\subsubsection{Problem formulation and pose optimization}
Modern geometry-based SLAM systems can be generally classified into \textit{feature-based} and \textit{direct} methods.
%
Feature-based SLAM systems ~\cite{mur-artalORBSLAM2OpenSourceSLAM2017, kleinParallelTrackingMapping2007} usually maintain a collection of sparse 3D \textit{point landmarks} corresponding to hand-crafted feature \textit{keypoints} detected in 2D images. In order to correct accumulated \textit{pose} error, \textit{i.e.,} drift, these methods resort to bundle adjustment \cite{triggsBundleAdjustmentModern2000} that jointly minimizes reprojection error between \textit{landmarks} and 2D \textit{keypoints} via \textit{pose} optimization. While being accurate in estimating trajectory, these approaches only come with sparse 3D maps that are less interpretable for visualization and recognition.
%
Direct SLAM~\cite{engelLSDSLAMLargeScaleDirect, engelDirectSparseOdometry2018} on the other hand, relies on pixel-wise projective data association between frames for odometry. Given relative poses between certain keyframes, pose graphs~\cite{dellaertFactorGraphsRobot2017} are formulated and optimized to obtain globally consistent poses without landmark constraints.

Our approach can be regarded as a bridge between the two approaches. We replace point landmarks with objects in feature-based SLAM. As a result, since pose constraints attached to objects can naturally replace reprojection error, we may directly convert such a landmark-pose constraint optimization to pose graph optimization (PGO).

\subsubsection{Map representation}
For \textit{dense} scene reconstruction, as introduced in Chapter \ref{chap:background} the volumetric Truncated Signed Distance Function (TSDF)~\cite{curlessVolumetricMethodBuilding1996} representation has been adopted and improved in several \textit{direct SLAM} frameworks. KinectFusion~\cite{newcombeKinectFusionRealtimeDense2011} introduced a plain $512^3$ grid for small scenes and objects. VoxelHashing~\cite{niessnerRealtime3DReconstruction2013} designed spatial hashing to scale this data structure to larger scenes. Similar implementations are available in CPU/GPU in the modern Open3D framework~\cite{zhouOpen3DModernLibrary2018, dongGPUAcceleratedRobust2019}, and we adopt this representation due to its ease of use.

\subsection{Object Instance Segmentation and Object based SLAM}
\subsubsection{Object instance segmentation}
In recent years, \textit{Region proposal} based Convolutional Neural Networks (R-CNN) \cite{renFasterRCNNRealTime2016} have established themselves as de-facto standards for object \textit{instance segmentation} from images. Amongst the literature, \textit{Mask-RCNN}~\cite{heMaskRCNN2018} and \textit{PointRend}~\cite{kirillovPointRendImageSegmentation2020} are the best off-the-shelf solutions.
In this work, we use \textit{PointRend} \cite{kirillovPointRendImageSegmentation2020}, which shows significant improvement over \cite{heMaskRCNN2018} by reformulating the mask generation as a rendering problem. In essence, this formulation is consistent with our tracking via compositional rendering module.

\subsubsection{Object based SLAM}
Applying aforementioned DNNs on 2D images, several works for RGBD and monocular SLAM have attempted to incorporate object instance detection. CubeSLAM~\cite{yangMonocularObjectPlane2019} and QuadricSLAM~\cite{nicholsonQuadricSLAMDualQuadrics2019} fit cuboids and quadrics, respectively, to detected objects to generate parameterized object landmarks. While improving the localization accuracy compared to baselines, these methods fail to densely map objects. \textit{MaskFusion}~\cite{runzMaskFusionRealTimeRecognition2018} adds labels to oriented point clouds and supports dense object visualization, but does not maintain persistent objects globally in a graph. \textit{Fusion++}~\cite{mccormacFusionVolumetricObjectLevel2018}, on the other hand, supports persistent dense reconstruction from fixed size $64^3$ voxel grids, yet is sensitive to voxel size tuning and may fail to adapt to objects at varying scales.
Our system utilizes scalable voxel grids that do not require much tuning to adjust to object scales. With a seamless CPU to GPU memory transfer implementation, larger environments can also be handled on-the-go.

% In this paper, we demonstrate an object graph formulation for simultaneous dense object reconstruction and accurate camera tracking, leveraging significant improvements in Deep Learning towards Object Detection and Object instance segmentation.

%and \citet{runz_maskfusion_2018} that propose object based representations for a scene. However, \cite{fusionPP} is based on unscalable discrete voxel grids, and uses lower resolution objects of $64^3$ to run feasibly, while \cite{runz_maskfusion_2018} is a surfel based method to track multiple objects and forgoes a factor-graph formulation and .
%More recently, \citet{xu2019mid}, proposes an octree datastructure, and utilizes a joint weighted photometric and geometric dense tracking similar to ours. This paper, however, focuses on tracking dynamic objects, and does not utilize a \textit{factor-graph} to jointly optimize camera, and \textit{correct} object poses.
% R-CNNs extract multiple region proposals, and individually classify and score the predictions to obtain labels and detection scores respectively. More recently, \citet{he2017mask} adds a parallel \textit{mask head} to generate object mask using Fully convolutional networks (FCN). Another line of work \cite{redmon2016you} predicts object bounding boxes directly by dividing an image into blocks and predicting probabilities.

% For directly segmenting \textit{3D data}, sparse convolution~\cite{4Dspatiotempro} and Key point convolution~\cite{kpconv} has started to gain attention in the community for its success on non-uniform point cloud data.

%In our work, we choose object detectors for images rather than point cloud, since we take RGBD data as raw input, and data association for objects from 2D is easier to obtain and update given depth maps and poses.

%\subsection{Object-based SLAM}
%\textbf{Sparse Feature based methods}. These approaches use object detection as opposed to object instance mask segmentation, to instantiate oriented 3D objects and fit cuboids \cite{yang2019cubeslam} or Quadrics \cite{quadric_slam}. They handle data association either by grouping detected keypoints into semantic labels, or assume known associations and define an Intersection-over-Union (IoU) overlap based error for optimization respectively. Although, these methods show improvement over baselines, they do not focus on object reconstruction fidelity. In contrast, our work uses a dense Hash-table based scalable volume data structure, which allows for accurate object reconstructions.

% \textbf{Direct dense methods}. SLAM++ %\cite{salas-moreno_slam_2013} was an early attempt towards SLAM with 3D object landmarks, that used a dense object representation. This method employed a dictionary of object descriptors generated through only geometric cues of each object model for data-association, and attempted camera trajectory optimization through multiple Frame to model ICP refinement between the live frame and \textit{each} detected object model. Further, a major limitation was that, it required pre-processed high quality mesh models of the expected objects in the scene prior to operation.

% The problem of high-fidelity 3D reconstruction has received substantial attention over the years. Key to high-quality 3D reconstruction is the choice of underlying representation for fusing sensor measurements of different modalities. Thus, most of the research on achieving globally consistent 3D models at scale uses the RGBD input that encapsulates the semantic as well as geometric (depth) information simultaneously. ~\citet{choi2015robust} proposes a framework that uses RGBD input to generate a very high-fidelity reconstruction of indoor scenes. They provide globally consistent models by optimizing across the entire pose trajectory. However, their approach requires offline processing and access to all the input frames resulting in hours of processing time, meaning real-time viability of refinement of reconstructed areas is almost impossible. Such joint global registration of multiple partially overlapping surfaces has also been considered by other works~\cite{huber2003fully,theiler2015globally}. To bridge this gap, this work proposes an object-centric pipeline for data association that reduce the expensive computational cost in registration.
% ~\citet{kinectfusion}'s seminal work KinectFusion introduced a new method and made RGBD reconstructions feasible. The key insight was to incorporate frame-to-model tracking as opposed to frame-to-frame tracking, which improved reconstruction accuracy significantly. Although it achieves impressive results, this approach still accumulates drift in the map estimate over long trajectories, since it does not retrospectively correct erroneous tracking. \citet{salas-moreno_slam_2013} extended KinectFusion and introduced an efficient pose graph formulation, that could be continually refined. However, this system required a detailed geometric model of the object a-priori. Further, ~\citet{fusionPP} propose an online object-level SLAM system wherein the 6DoF pose graph consists of only the reconstructed object instances. Using RGBD data of indoor scenes as their input, they leverage Mask-RCNN~\cite{Detectron2018} to generate 2D instance mask predictions of those objects. Consequently, these mask predictions are fused online into the per-object Truncated Signed Distance Function (TSDF) along with a 3D voxel mask to fuse the instance foreground. Although they demonstrate high quality object reconstruction within globally consistent loop-closed object SLAM maps, the memory usage scales cubically with the size of a TSDF. To combat the memory overflow, they propose integrating octree or voxel hashing into their pipeline to further boost run-time performance. Moreover, the system proposed by them is still susceptible to spurious detections resulting in a growing clutter of partial object reconstructions. As an improvement to this problem, they propose a learning-based mechanism for filtering and reconstructing objects.

\section{Compositional and Scalable Object SLAM} \label{sec: methodology}

 \subsection{System Overview}
Our pipeline can be divided into typical SLAM components and a deep perception module, connected by an object-based semantic map. Figure~\ref{fig:overview} provides an overview.

%
Our pipeline consists of 5 modules each running in a separate thread: semantic segmentation, frame-to-model odometry, object data association and map update, PGO, and compositional rendering.
%
Incoming RGBD frames are initially processed through  \textit{semantic segmentation} (\S\ref{subsec: segmentation})  to obtain instance masks, labels, and semantic descriptors, from DNNs for keyframes.
%
Then, odometry between the incoming live frame and the \textit{compositional render} from the map (\S\ref{subsec: rendering}) is estimated via \textit{frame-to-model odometry} (\S\ref{subsec: tracking}) to obtain relative poses.
%
Maintained objects visible in the frame are rendered given the estimated camera pose, and objects are associated with 2D instance detections to either integrate or initialize new objects in the global map (\S\ref{subsec: segmentation}).
%
Separately, a global factor-graph is updated to optimize the camera trajectory and object poses (\S\ref{subsec: optimization}). The optimized object poses are rendered to generate a compositional model of the scene for subsequent tracking (\S\ref{subsec: rendering}).

Before we discuss these modules in detail from \S\ref{subsec: tracking} to \S\ref{subsec: rendering}, we introduce core concepts and notations in \S\ref{subsec: notation}.

 \begin{figure}[htbp]
 	\centering
    \includegraphics[width=\linewidth]{figs/icra2021-compressed.pdf}
    \caption{\label{fig:overview} System overview: Top shows the deep object segmentation pipeline that runs asynchronously, Masked keyframes from the segmentation pipeline are used in Data association and Map update (shown with red lines). Bottom shows the major stages of the reconstruction system, specifically object models are used in tracking via compositional raycasting (shown with green lines).}
 \end{figure}
\subsection{Core concepts and notations} \label{subsec: notation}
A background volume $V_B$ is a spatially-hashed voxel grid on GPU, where small $16^3$ subvolumes are allocated around observed 3D points. It is created and updated as a \textit{temporary} instance for stable tracking. An object volume $V_{O_i}$ is akin to the background volume, but persistently maintains the object label, ID, and corresponding object descriptors.

A 3D volume $V$'s properties, including surface vertex positions, normals, and colors can be mapped to 2D images given a camera pose $\mathbf{T} \in SE(3)$ and camera intrinsics defined as $K$ with ray-casting. We denote such \textit{rendered images} by \( \langle \mathcal{N}, \mathcal{V}, \mathcal{C} \rangle \)  for normal, vertex, and color maps respectively. They can be associated with input RGBD images $\langle \mathcal{I}, \mathcal{D}\rangle $ that consist of color ($\mathcal{I}$) and depth ($\mathcal{D}$) images via projective closest points.

We use subscripts and superscripts to indicate multiple coordinate frames used in our pipeline, including $C_i$ for $i$th camera, $O_j$ for $j$th object, and $W$ for background or world coordinate frame. For instance \(\langle \mathcal{N}_{C_s}, \mathcal{V}_{C_s}, \mathcal{C}_{C_s} \rangle \) represents 2D maps rendered from the volumes in the $C_s$ camera coordinate frame.
$\mathbf{T}_{O_j}^W \in SE(3)$ encodes a rigid transformation from object $j$ to world. Finally, we denote the respective measurements between nodes with variable $\mathbf{Z}$.

\subsection{Hybrid Frame to Model Odometry} \label{subsec: tracking}

In RGBD camera tracking, we seek to estimate the relative camera pose \(\mathbf{T}^{C_t}_{C_s}\) given an incoming RGBD target frame \( \langle \mathcal{I}_{C_t}, \mathcal{D}_{C_t}\rangle \) and a source model \( \langle \mathcal{N}_{C_{s}}, \mathcal{V}_{C_{s}}, \mathcal{C}_{C_{s}}\rangle \) of the scene rendered by placing a virtual camera at the previous camera frame ${C_s}$.

We accomplish this by minimizing the joint weighted dense geometric error residual $r_D$ and the photometric error residual $r_I$. The general energy function is formulated as in \cite{parkColoredPointCloud2017} by accumulating residual at every point $p \in \mathbb{Z}^2$ with a valid data association:
\begin{align}
    E(\mathbf{T}^{C_t}_{C_{s}}) = \sum_{p} (1 - \sigma) r_I^2(\mathbf{T}^{C_t}_{C_s}, p) + \sigma r_D^2(\mathbf{T}^{C_t}_{C_s}, p) \label{eq:1}
\end{align}
Here, we adapt the geometric ICP residual as the point-to-plane distance between the incoming depth map $\mathcal{D}$ and the rendered vertex and normal map $(\mathcal{V}_{C_s}, \mathcal{N}_{C_s})$ as follows, using the formulation in \cite{newcombeKinectFusionRealtimeDense2011}:
\begin{align}
    r_D(T^{C_t}_{C_s}, p) = \bigg(T^{C_t}_{C_{s}} \mathcal{V}_{C_s}( \hat{p} ) - \mathcal{V}_{C_{t}}(p )\bigg) \cdot \mathcal{N}_{C_{t}}( p ) \label{eq:2}
\end{align}
where $\mathcal{V}_{C_t}$ is the vertex map from unprojecting the input depth image $\mathcal{D}_{C_t}$.
Additionally, we use a photometric error residual to improve tracking robustness, which is defined as:
\begin{align}
    r_I(T^{C_t}_{C_s}, p) = \mathcal{C}_{C_s}(\hat{p}) - \mathcal{I}_{C_t}(p) \label{eq:3}
\end{align}
In equations (\ref{eq:2}) and (\ref{eq:3}), $\hat{p}$ is the correspondence of $p$ in the source frame, and is computed via \textit{warping}:
\begin{align}
    \hat{p} = K {\mathbf{T}^{C_t}_{C_s}}^{-1}\mathcal{D}_{C_t}(p)K^{-1}[p^\top, 1]^\top \label{eq:4}
\end{align}
It must be noted that the points $p$ are a subset of pixels with valid object-level data associations detailed in \S\ref{subsec: rendering}.

The energy function in equation \ref{eq:1} is minimized using the Gauss-Newton algorithm. We implement the minimization in a coarse to fine scheme using an image pyramid, on the GPU in parallel since each pixel acts independently in the energy function using \textit{reduction} with appropriate thread conflict handling as described in \cite{dongGPUAcceleratedRobust2019}.

\subsection{Object Instance Segmentation and Association} \label{subsec: segmentation}

\textbf{2D instance segmentation:} Object detection and instance masks are generated every $n^{th}$ frame (we choose $n=10$) in a separate thread from the \textit{PointRend} backend. \textit{PointRend} uses a Resnet-50-FPN backbone network to generate a convolutional feature map. In particular, after an empirical evaluation, we found that \textit{PointRend} provided better masks over \textit{Mask-RCNN}.

The semantic segmentation module maps incoming \textit{RGB} frame \(\mathcal{I}\) into a set of object labels $[l_1, \dots l_k]$, a set of binary object masks $M_n^i$ defined over $l \in \mathcal{L} \triangleq \{0, \dots, L_{max}-1\}$ object classes ($L_{max}=80$ in the MS-COCO dataset), bounding boxes $b \in \mathbb{N}^4$, and a probability distribution $p(l_i \mid \mathcal{I})$. We also extract the object feature map for the accepted object proposals, from the penultimate fully connected layer of the R-CNN from the object classifier head. We observe that these feature maps provide us with robust data association in ambiguous situations.
%
To obtain instance segmentation for frames not sent to the DNN, we warp the binary mask images from the most recent frame with a segmentation and fill the holes in the warped masks using the \textit{flood fill} algorithm.

Once the current camera pose and the semantic segmentation information are available, instance detections are associated with existing objects. Unmatched instance detections are used to initialize new object volumes.

\textbf{3D instance generation:} When an unmatched object is to be instantiated, the masked depth frame at $C_i$ is unprojected and transformed into the world frame to obtain the object point cloud:
\begin{align}
X_{W} = \mathbf{T}^W_{C_i} K^{-1} D_{C_i}(p) [p^\top, 1]^\top.
\end{align}
To obtain relatively high fidelity reconstruction, we adaptively calculate a conservative voxel length of
\begin{align}
    l = \gamma \| \max(X_W) - \min(X_W) \|_{\infty},
\end{align}
where $\min, \max$ operators are applied to all dimensions of $X \in \mathbb{R}^3$ simultaneously. We empirically use $\gamma = 1 / 64\sqrt{2}$, but due to the scalability of the volume our model is less sensitive to $\gamma$.
Finally, the object pose is simply chained by
\begin{align}
    \mathbf{T}_{O}^W = \mathbf{T}_{C_i}^{W} \bigg({\mathbf{T}_{C_i}^O}\bigg)^{-1},
\end{align}
where $\mathbf{T}_{C_i}^O = [I \mid t_{C_i}^O]$ with $t_{C_i}^O = \min(X_W) - t_{C_i}^W $. Each new object is also initialized with the object feature map from its corresponding instance mask.

\textbf{2D--3D semantic data association:}
To associate existing object volumes to 2D instances, visible objects are rendered (in \S\ref{subsec: rendering}) in the current frame. The rendered color map $\mathcal{C}$ is thresholded to obtain a virtual binary mask. An intersection over union (IoU) between the virtual binary mask $\hat{\mathcal{M}}$ and the instance masks $\mathcal{M}_i$ in the current frame is used as a scoring metric as defined in \cite{mccormacFusionVolumetricObjectLevel2018}.

As opposed to computing the $\argmax_{i}{\text{IoU}(\mathcal{M}_i, \hat{\mathcal{M})}}$, we associate objects as given below:
\begin{align}
    i = \argmin_{i \in \mathcal{S}} (\| f_i - \hat{f} \|_1),
\end{align}
where $\hat{f}$ and $f_i$ denote feature map of the object render (identical to the object in question), and the instance masks respectively and $\mathcal{S} \triangleq \{i : \text{IoU}(\mathcal{M}_i, \hat{M}) > 0.2\}$.
Associating object renders to instance masks in this manner prevents incorrectly fusing object instances between nearby similar objects, in cases where there is large accumulated drift.

% In practice, we reuse the object render generated after pose-graph optimization from the previous frame, for improved performance, as \textit{ray-casting} is the largest bottleneck on the system.

For subsequent fusion of a 2D instance detection to its associated 3D object, the instance mask---containing the object foreground---and the bounding box mask---containing both the foreground and background are used. Similar to \cite{mccormacFusionVolumetricObjectLevel2018} we integrate the object in both the foreground and background through a weighted average of TSDF, color, and additionally maintain binomial foreground-background count variables for each voxel. This smoothes out artifacts from integration of 2D instances with spurious masks (see illustration in Figure~\ref{fig:object-integration}).

\begin{figure}[htpb]
    \centering
    \includegraphics[width=0.8\linewidth]{figs/object-integration.png}
    \caption{Each pixel is a binomial trial given a latent foreground probability of the associated voxel. By maintaining foreground and background counts, we estimate the expected latent probability required during rendering.}%
    \label{fig:object-integration}
\end{figure}

Finally, we update the object feature map by a gated weight average:
\begin{align}
    f_t = \frac{w_{t-1} \cdot f_{t-1} + \mathcal{H}(f_{t-1}, f_{in}) \cdot f_{in}}{w_{t-1} + \mathcal{H}(f_{t-1}, f_{in})}\\
    \mathcal{H}(f_{t-1}, f_{in}) = \frac{\textrm{sgn}(\lambda - ||f_{t-1} - f_{in}||_1) + 1}{2}
\end{align}
where $\mathcal{H}$ is the Heaviside step function that hard-filters outlier input feature map $f_{in}$ compared to the maintained object feature map $f_{t-1}$ with weight $w_{t-1}$ controlled by the threshold $\lambda$.

\subsection{Factor Graph Optimization} \label{subsec: optimization}

As we have mentioned before, a background volume is maintained for stable tracking and to handle \textit{objectless} frames. The background volume additionally maintains the ratio ($r$) of visible volume units in the current camera frustum to the total number of allocated volume units in the volume. A low ratio implies that the camera may have moved away from a particular part of the scene. Pose graph optimization is conditionally triggered when the background volume is reset owing to low ratio of visible units ($r < 0.2$) and when there are new objects added into the graph.

Our object factor graph formulation is similar to \cite{salas-morenoSLAMSimultaneousLocalisation2013, mccormacFusionVolumetricObjectLevel2018}. The variable nodes $\mathcal{X} = \{\mathbf{x}_1, \dots \mathbf{x}_N\}$ are partitioned into camera pose variables $\mathbf{T}^{W}_{C_i} \in SE(3)$ and object pose variables $\mathbf{T}^W_{O_j} \in SE(3)$. The first camera pose is initialized as the world frame $W$.

Assuming a Gaussian noise model, the \textit{MAP} inference problem with the above variable nodes reduces to solving the following non-linear least squares optimization:
\begin{align}
    \mathcal{X}^* = \argmin_{\mathcal{X}} \Big( \sum_{k \in |C|} \|  \mathbf{Z}^{C_k}_{C_{k-1}} \ominus \mathbf{T}^{C_k}_{C_{k-1}} \|_{\Sigma_{k,k-1}}^2 + \sum_{j \in |\mathcal{O}|, k \in |\mathcal{C}|} \| \mathbf{Z}^{O_j}_{C_k} \ominus \mathbf{T}^{O_j}_{C_k} \|_{\Sigma_{o_j, k}}^2 \Big)
\end{align}
where the operator $ \mathcal{Y} \ominus \mathcal{X} = \text{Log}(\mathcal{X}^{-1} \mathcal{Y})$ expresses the relative error in the local tangent vector space \cite{solaMicroLieTheory2020}. $\Sigma_{k, k-1}$ denotes the covariance between relative camera pose measurements,  $\Sigma_{o_j, k}$ is the covariance in the camera to object measurement. They can be approximated by information matrices computed from \textit{odometry}, however, empirically we found that a constant information matrix can achieve reasonable results. We obtain the relative camera measurements $\mathbf{Z}^{C_k}_{C_{k-1}}$ from \textit{frame to model odometry} (\S\ref{subsec: tracking}), and obtain frame to object measurements $\mathbf{Z}^{O_j}_{C_k}$ by performing an additional Gauss Newton iteration with only the object pixels. Finally, the expected relative camera pose $\mathbf{T}^{C_k}_{C_{k-1}}$ and expected camera object pose $\mathbf{T}^{O_j}_{C_k}$ used in the factors are calculated as:
\begin{align}
    \mathbf{T}^{C_k}_{C_{k-1}} &= \bigg({\mathbf{T}^{W}_{C_k}}\bigg)^{-1} \mathbf{T}^{W}_{C_{k-1}}, \\
    \mathbf{T}^{O_j}_{C_k} &= \bigg({\mathbf{T}^{W}_{O_j}}\bigg)^{-1} \mathbf{T}^{W}_{C_k}.
\end{align}

We solve the optimization in GTSAM \cite{FactorGraphsGTSAM2019} using Levenberg Marquardt. Since the entire object volume is transformed as a rigid body, the object volumes remain unchanged in memory after optimization. We note that this circumvents the time-consuming re-integration that usually takes place in volumetric methods after PGO \cite{whelanElasticFusionDenseSLAM2015}.

\subsection{Compositional Rendering} \label{subsec: rendering}
Compositional rendering is a serialized operation that generates normal, vertex, and color maps by ray-casting 3D objects in the viewing frustum into 2D object instances, and is illustrated in Figure~\ref{fig:comp-render}.

\begin{figure}[htpb]
    \centering
    \includegraphics[width=0.8\linewidth]{figs/comp-render.png}
    \caption{An illustration of compositional rendering.}%
    \label{fig:comp-render}
\end{figure}

\(\langle \mathcal{N}_{C_i}, \mathcal{V}_{C_i}, \mathcal{C}_{C_i} \rangle \) is in fact an aggregation of separate renderings from object volumes \(\langle \mathcal{N}_{C_i}^{V_{{O}_j}}, \mathcal{V}_{C_i}^{V_{{O}_j}}, \mathcal{C}_{C_i}^{V_{\mathcal{O}_j}} \rangle \) and the background volume \(\langle \mathcal{N}_{C_i}^{V_{B}}, \mathcal{V}_{C_i}^{V_{B}}, \mathcal{C}_{C_i}^{V_{{B}}}  \rangle \), depending on the masks. In particular, we render the background volume, based on a background mask that is constructed from the union of existing virtual object masks in the current frame, and associated instance masks.

Then, the composed per-pixel map model render can be obtained as follows:
\begin{align}
    &\hat{k} = \argmin_k \mathcal{V}_k(p)[z], ~k \in \{O_1, \cdots, O_n, B\} \\
    &\langle \mathcal{N}^*(p), \mathcal{V}^*(p), \mathcal{C}^*(p) \rangle = \langle  \mathcal{N}_{\hat{k}}(p), \mathcal{V}_{\hat{k}}(p), \mathcal{C}_{\hat{k}}(p) \rangle,
\end{align}
where $\hat{k}$ is the volume index corresponding to the minimum distance to camera center for pixel $p$.

\begin{figure}[htpb]
    \centering
    \includegraphics[width=\linewidth]{figs/compositional-render.pdf}
    \caption{Compositional rendering during reconstruction on TUM \textit{fr3\_long\_office\_household} dataset. Left shows the background render, and right shows the composed render. Compositionally rendered objects are shown in bounding boxes.}
    \label{fig:compositional_render}
\end{figure}

Object volumes not currently visible are downloaded from GPU into CPU memory. Note that downloading the object volume does not affect the optimization problem, since the object volumes are required only for integration and raycasting.


\subsection{RAM}

Simple and fast memory. You can store your data there, you can also run program
from there.

\subsubsection{Function}

Internal RAM is organized as 32bit wide words and is implemented with internal
SRAM block memory in FPGA. Thus is really fast. It is fastest memory what you
can use (of course, except registers).

Usage is really straightforward, simply write data with LD instruction to address
that belong to RAM.

\subsubsection{Hacking}

SoC can have multiple instances of RAM, as much as you want and there is
enough FPGA SRAM (M9K for Cyclone IV) block free. Formula for estimate usage
of M9K is: $ blocks = \frac{2^{ADDRESS\_WIDE}}{256} $.

You can change size of internal RAM by parameter ADDRESS\_WIDE, value for this
parameter have following meaning: $ size [words] = 2^{ADDRESS\_WIDE}$.

\begin{lstlisting}[language=VHDL, frame=single]
entity ram is
    generic(
        BASE_ADDRESS: unsigned(23 downto 0) := x"000000";
        ADDRESS_WIDE: natural := 8
    );
    port(
        clk: in std_logic;
        address: in unsigned(23 downto 0);
        data_mosi: in unsigned(31 downto 0);
        data_miso: out unsigned(31 downto 0);
        WR: in std_logic;
        RD: in std_logic;
        ack: out std_logic
    );
end entity ram;
\end{lstlisting}

As you can see, there is only these one parameter for changing features of RAM.
Another ports are used for connection onto system BUS. When you want add more
instances of RAM, you have to also specify base address.