% ========== Chapter on Macroscopic Resistivity

\chapter {The Macroscopic Resistivity of the Human Brain}
\label{chap:macroscopicResistivity}

Current flow in the brain is controlled both by the anatomy, and by the associated electrical resistivity of its three components: cerebrospinal fluid (csf), gray matter, and white matter. This is called volume conduction. Volume conduction applies both to the current flow due to neural activity and to the current flow from electrical brain stimulation. Understanding it is essential for source modeling and inversion of ECoG, EEG and MEG. Understanding it is also essential for modeling the direct electrical stimulation (DES) of the brain using internal and external (transcranial) electrodes and using pulsed magnetic fields. Our interest came from our DES studies using clinical ECoG electrodes. We report here measurements of the propagation of bipolar current pulses made using clinical electrocorticography (ECoG) grids. We stimulate between a pair of electrodes in an ECoG grid and record the resulting voltage waveforms across the entire grid. We find our measurements are described qualitatively very well by two simple analytical models for a volume conductor with an apparent resistivity. As far as we know, our measurements are the first direct macroscopic measurements that show the brain can be qualitatively modeled as a simple volume conductor without considering the detailed anatomy.

We compare our measurements with two simple models and find qualitative agreement with both. The first model is a flat semi-infinite homogeneous isotropic resistivity half space. The second is a homogeneous isotropic resistivity sphere.Of course, the brain is neither. The first model neglects finite size and shape corrections. The second model includes them.

Because the resistivity values of the three components are different, the value of the resistivity that you measure will depend on the electrode locations. For example, if the electrodes are very small and very close together and inside a single component of the brain, then you will measure the resistivity of that component. However, we have been unable find any published measurements that satisfy these conditions. This means that all such measurements will report an apparent resistivity which is a weighted average over the regions where the stimulation current flows. For 2-point measurements, the apparent resistivity is dominated by the spatial region near the stimulation electrodes. Such 2-point measurements have been called spreading resistance, constriction resistance, and contact resistance. Our 2-point measurements yield a single resistivity value for each pair of stimulation electrodes. For 4-point measurements, the apparent resistivity contains contributions from the spatial regions near all four electrodes and from the regions joining them where current flows. Our 4-point measurements yield a distribution of resistivity values that depend on the locations of the stimulation electrodes and the recording electrodes.

\section{Introduction}

How does current flow in the brain? There are contributions from both the anatomy, and the associated electrical resistivity of its three components: cerebrospinal fluid (csf), gray matter, and white matter. This is called volume conduction. Volume conduction applies both to the current flow due to neural activity and to the current flow from electrical brain stimulation. Understanding it is essential for source modeling and inversion of ECoG, EEG and MEG. Understanding it is also essential for modeling the direct electrical stimulation (DES) of the brain using internal and external (transcranial) electrodes and using pulsed magnetic fields. Our interest came from our DES studies using clinical ECoG electrodes. We report here measurements of the propagation of bipolar current pulses made using clinical electrocorticography (ECoG) grids. We stimulate between a pair of electrodes in an ECoG grid and record the resulting voltage waveforms across the entire grid. We find our measurements are described qualitatively very well by two simple analytical models for a volume conductor with an apparent resistivity. As far as we know, our measurements are the first direct macroscopic measurements that show the brain can be qualitatively modeled as a simple volume conductor without considering the detailed anatomy.

We compare our measurements with two simple models and find qualitative agreement with both. The first model is a flat semi-infinite homogeneous isotropic resistivity half space. The second is a homogeneous isotropic resistivity sphere.Of course, the brain is neither. The first model neglects finite size and shape corrections. The second model includes them.

Because the resistivity values of the three components are different, the value of the resistivity that you measure will depend on the electrode locations. For example, if the electrodes are very small and very close together and inside a single component of the brain, then you will measure the resistivity of that component. However, we have been unable find any published measurements that satisfy these conditions. This means that all such measurements will report an apparent resistivity which is a weighted average over the regions where the stimulation current flows.

We present our analysis of the data that we have collected on the direct electrical stimulation (DES) of human brains in vivo with the goal of providing insight into the apparent resistivity on the macroscopic scale. The volume of resistive material that is probed depends on the geometry of the electrodes used. As our minimum electrode spacing is one centimeter, we probe the resistivity on the centimeter and larger length scale. Due to the size of our electrode arrays compared to the size of the brain, our measurements probe the apparent resistivity rather than the true resistivity. This is because the brain is not a simple homogeneous medium. Because of this, the apparent resistivity that we measure, and that all previous in vivo studies have measured, depends on the anatomy and on  the resistivity of the three components (cerebrospinal fluid (CSF), gray matter, and white matter). We here focus on analyses considering the brain as a medium with an apparent resistivity (Figure \ref{fig:mrModelingOverview})).

\begin{figure}[ht]
	\centering
	\includegraphics[width=1\textwidth]{figures/macroscopicResistivity/resistivity_schematics_part1}
	\caption[What is the resistance of the human brain?]{a) We here consider the simple case of where the brain is a resistive medium, with current injected at two electrode locations. Electrical stimulation of cortex results in charge injection which flows between a source and a sink electrode. b) In the case of bipolar electrodes, the majority of current flows between the two electrodes. The equipotential lines represent curves along which the electric potential is the same. Using other electrodes in the array, we are able to measure the electric potentials relative to a distant reference electrode.  The stimulation currents pass between the two electrodes, and depending on their separation and the topography of the cortex, a varying amount of current may pass through different regions of cortex. }
	\label{fig:mrModelingOverview}
\end{figure}

Since Penfield, intraoperative awake DES has been used extensively to map the function of different brain regions \cite{Penfield1937}. The Montreal procedure developed by Penfield and his coworkers has been called the gold standard for mapping brain function \cite{Borchers2012}. In modern times, awake DES is routinely used to prevent resection of crucial brain areas during epilepsy surgery to remove the epileptic focus and analogously during brain surgery to remove tumors. Not surprisingly, there have been and still are, many attempts to improve on awake DES [???]. Nevertheless, clinically, it is still the gold standard.

In epilepsy surgery, an electrocortiography (ECoG) grid is placed on the cortical surface and the patient is taken off of their anti-seizure medication. The location of the seizure focus is determined by analyzing the pattern of the seizure activity across the grid. Once the seizure focus has been located, the patients are put back on their anti-seizure medication. It is from these patients that we acquire our data.

Future applications of DES and DBS will include neuroprosthetic devices and brain-computer interfaces to enable users to interact with the physical and cyber world solely via brain signals. Most of the work towards brain-computer interfaces up until now has focused on reading signals from the brain to drive prosthetics or move a computer cursor.  In the future, precise control of prosthetics, and a richer user environment, will also require input into the brain from a prosthetic device. Direct electrical stimulation of the brain is just starting to be used to provide haptic feedback in humans \cite{Cronin2016a,Collins2016}, and previous work has demonstrated the efficacy of both cortical and peripheral stimulation to provide enhancements in brain computer interface performance \cite{Klaes2014,Dadarlat2015}.

\section{Theory}

This section summarizes the theory underlying resistivity measurements made using bulk samples. It contains the equations that we used to calculate our 2-point resistivity values and our 4-point resistivity distributions from our measured voltages.

\subsection{Resistance Measurements Made on Bulk Samples}

Resistance Measurements Made on Bulk Samples

In resistance measurements on bulk samples, a voltage is applied to two contacts on the sample and the resulting current is measured, or vice versa. To convert the measured resistance into the resistivity of a homogeneous isotropic resistive sample, or into the apparent resistivity of an inhomogeneous isotropic resistive sample, requires the detailed spatial structure of the sample and the location of the electrodes. In the next sections, we first discuss the apparent resistivity, as well as the necessary equations for converting resistance measurements made on semi-infinite bulk samples and on spherical samples using 2-point measurements and 4-point measurements.

\section{The apparent resistivity}
For a half space composed of any number of uniform thickness and uniform resistivity layers, the equation for the voltage on the surface has the same form as that for the same electrode configuration on a uniform resistivity substrate with the value of the resistivity $\rho$ for the uniform resistivity substrate replaced by an apparent resistivity $\rho_a$. The apparent resistivity depends on the paths that the current takes, which depends on both the geometry and the resistivities of the various layers. It is the appropriate average over all paths.

Apparent resistivity is the basis for geological resistivity prospecting \cite{W.M.TelfordL.PGeldart1990}. For example, for a two-layer earth, when the electrodes are very close together, the current penetration is very shallow and the apparent resistivity is equal to the resistivity of the top layer. When the electrodes are very far apart, the current penetration is very deep and the apparent resistivity is equal to the resistivity of the bottom layer. In between, it is the appropriate average over the paths. The rule-of-thumb for a uniform resistivity substrate is that the $ 50 \%$ penetration depth is about one half of the spacing between the two stimulation electrodes. For geological resistivity prospecting, the separation between the current electrodes is varied and the apparent resistivity is measured. From these measurements, the composition versus depth can be determined.

Because the brain is layered, any measurements of the resistivity of an intact living brain will measure an apparent resistivity unless the electrodes are very close together. In humans, the thickness of the gray matter is about 3.5 mm, so to measure the resistivity of the gray matter without seeing the white matter using a four-point probe would require the entire electrode array to be sub-millimeter in size. Our clinical ECoG grids have an electrode spacing of 1 cm. Consequently, the focus of this paper is on determining the apparent resistivity of the human brain on the macroscopic scale. In our measurements, there is an additional issue  due to the presence of CSF with unknown thickness between the clinical grid and the surface of the cortex during our measurements.

\subsection{Two-point measurements and electrode resistance}

For a constant-voltage ohmic electrode on the surface of half-space of uniform resistivity material (with infinity held at zero potential), the steady-state voltage and current are related by Ohm’ss law $V=IR$ and the electrode resistance is defined to be $R=V/I$. For circular electrodes with radius $a$, the resistance is related to the resistivity of the material by $R=(rho/4a)$. For two identical well separated electrodes, the resistance between them is twice that for a single electrode $R_2=(rho/2a)$. These equations are routinely used to measure the resistivity of bulk semiconductors.

However, because of the capacitance of metallic electrodes on the surface of the brain, the above relation does not apply under steady state conditions due to the time-dependent charging of the double-layer capacitance. However, it does apply under constant current pulse conditions before the electrodes start to charge. {ref 1} The measured time-dependent applied voltages that were used to produce constant current stimulation are shown in Figure 1. We determined the electrode resistances by extracting the voltage jump at the onset of the stimulation and dividing it by the applied current \cite{Huang2008a}.

What happens as the two electrodes are brought together? Do the resistances still add? It turns out that in general they do not. This is because superposition does not apply to closely spaced constant voltage electrodes. The analytic solution to Laplace’s equation for multiple constant voltage electrodes on the surface of a half-space of uniform resistivity material is extremely complicated. The boundary conditions are mixed: constant voltage boundary conditions apply under the two electrodes and constant normal derivative (zero current) boundary conditions apply everywhere else on the surface. A pair of circular constant voltage electrodes separated by a center-to-center distance $d$ that is large compared to their radius $a$ can be well approximated by the standard formula of $R_2=(rho/2a)$, with an error of less than 0.1\% for our application. \cite{Kristiansson2005}.

We calculated the apparent resistivity values for each subject using our measured electrode resistances by solving for $\rho$

$$\rho = {2 a R \over A} $$

\subsection{Two-Point Measurements Made on Homogeneous Spherical Samples}

We were unable to find any analytic or numerical results for circular electrodes on a sphere, so we calculated the scale factor relative to the flat semi-infinite case using BEM [makarov] For 2.3 mm diameter platinum electrodes on a 6 cm radius uniform resistivity sphere we found the relationship between the resistivity and the measured resistance is given by

rho=XaR.

\subsection{Four-Point Measurements Made on Homogeneous Flat Bulk Samples}

The four-electrode method is the gold standard for making resistivity measurements. For 4-point measurements on homogeneous isotropic flat bulk samples, the resistivity is related to the measured voltage difference between the two sense electrodes in terms of the current imposed between the two source electrodes by \cite{Miccoli2015}

It's main advantage is that it is insensitive to inevitable variations in the contacts. For example, since our surfaces are not flat, the effective contact area can be smaller than the actual area of the
contact. When this happens or the region under the contact is inhomogeneous, the contact resistivity will not be equal to $2aR$. However, by using constant current stimulation and measuring away from the stimulation electrodes, the voltages on the recording electrodes will be correct. They do not depend on the contact area. This is why the point electrode model works for our measurements.

$ V = (I rho / 2 pi) [ (1/r_1 - 1/r_2) - (1/r_3 - 1/r_4) ] $

where $r_1$ is the distance from the first source electrode to the first sense electrode,$ r_2 $is the distance from the second source electrode to the first sense electrode,$ r_3 $is the distance from the first source electrode to the second sense electrode,$ r_4 $is the distance from the second source electrode to the second sense electrode,

I is the current, and rho is the resistivity.

The most often used electrode configurations are linear and square. For finite size samples, there are correction factors that depend on the electrode configuration, spacing, and location and on the size and shape of the sample. In addition to the bulk equation above, there are well known results in the literature for parallelepipeds, disks, and cylinders that include the size and shape factors.

\subsection{Four-Point Measurements Made on Homogeneous Spherical Samples}

The relationship between the voltage measured between two sense electrodes on the surface of a sphere and the current imposed between two source electrodes on the surface of the sphere was first worked out in a remarkable, now classic paper by Helmholtz in 1859 \cite{Wilson1950}. It is given by \cite{Frank1952}

$$ V = (I rho / 4 pi) [ 2/rb -2/ra + (1/R) ln {ra+R-a cos(beta) / rb+R-b cos(theta)} $$ Frank’s eqn 12

where blah, blah, blah

you should probably include a drawing of the geometry—doesn’t that sound fun?

Two-Point Measurements Made on Inhomogeneous Flat Bulk Samples

For the special case of 4-point measurements on inhomogeneous isotropic flat bulk samples composed of parallel layers with constant resistivity values, the apparent resistivity that will be measured can be calculated in terms of the resistivity values, the thicknesses, and the arrangement of the layers. This relationship is the basis of vertical electrical sounding used to determine the geological layering underneath the surface of the earth [Geological Survey Professional Paper 499] and of the 2-point and 4-point characterization of the layering of semiconductor materials [NBS Special Publications 400-10 and 400-48 and Ehrstein]. The apparent resistivity is the ratio of the measured resistivity divided by the resistivity that would be measured for the same structure if it were homogeneous and isotropic with a resistivity equal to unity. However, the structure of the brain under our clinical ECoG grids does not consist of parallel layers with constant resistivity values so we simply present the distribution of apparent resistivity values that we measure.

\subsection{Four-Point Measurements Made on Inhomogeneous Spherical Samples}

There have been many models of the brain made using concentric spheres with constant resistivity values. However, the structure of the brain under our clinical ECoG grids does not consist of concentric spheres with constant resistivity values

so we simply present the distribution of apparent resistivity values that we measure.

The Ehrstein chapter is in $ https://link.springer.com/chapter/10.1007/978-1-4757-1352-7_1 $
%
%\section{Theory for the statics and dynamics of ECoG electrodes}

%How should the static and dynamic behavior of the ECoG electrodes be modeled?
%Three primary static models exist and have been used extensively in the literature: point electrodes, finite-size constant voltage electrodes, and finite-size constant current electrodes.  Point electrodes have been widely used in georesistivity studies and are natural there because the electrode separations are much greater than the electrode dimensions. Both constant voltage and constant current stimulation is used clinically. At short times, ECoG electrodes are predicted to act like constant voltage sources; after a few time constants, they are predicted to act like constant current sources [???]. In the constant voltage regime, the current density under the electrode is sharply peaked at the edge of the electrode \cite{Wiley1982}. In the constant current regime, the current density under the electrode is uniform.
%
%There are exact closed-form solutions for the voltage on the surface of a homogeneous isotropic resistive half-space in the literature for all three models.
%All three models are often represented using Fourier-Bessel expansions (See appendix ???).The calculated voltage distributions are shown in figure ???. Note that at the distance of even our nearest recording electrodes relative to the stimulation electrodes, they all look like point sources. Asymptotically (beyond one cm) all of the curves decay in a matter similar $1/r$, with the precise fall off depending on the particulars of problem geometry. Physically, this is because the electrode looks like a point once you are far enough away from it. This explains why the point source model fits our data so well.
%
%For point electrodes on the surface of a half space with uniform resistivity, the voltage everywhere on and inside the sample is given by:
%
%$$V(r) = \bigg( {\rho \; i_0  \over 2 \pi} \bigg) \ {1\over r}$$
%
%\noindent where $i_0$ is the current that is injected at the electrode and removed at infinity, $\rho$ is the resistivity of the half space, and $r$
%is the distance between the current source and the point where the voltage is measured. This is the voltage measured relative to infinity which is defined to be at zero potential.
%
%For a pair of constant current point electrodes on the surface of a half space with uniform resistivity, the voltage everywhere on the surface is given by superposition
%
%$$V(r) = \bigg( {\rho \; i_0  \over 2 \pi} \bigg) \ \bigg[ {1\over r_+} -  {1\over r_-} \bigg]$$
%
%\noindent here $r_+$ is the distance between the positive current source and the point where the voltage is measured, and
%$r_-$ is the distance between the negative current source and the point where the voltage is measured. Again this voltage is measured relative to infinity. We use this equation to analyze our data. For point electrodes on the surface of a half space with uniform resistivity, the apparent resistivity $\rho_a$ is equal to the true resistivity $\rho$. For multilayer systems, the apparent resistivity $\rho_a$ is not equal to the true resistivity $\rho$. This is used to obtain information about the
%resistivity of the layers.
\section{Methods}

\subsection{Patient Characteristics}

Our patients were implanted with electrode grids for Our  patient population is summarized in Table \ref{table:mrSubjs}. The locations of their stimulation and recording electrodes are summarized in Figure \ref{fig:mrCompositeBrain}.

\begin{figure}[ht]
	\centering
	\includegraphics[width=0.8\textwidth]{figures/macroscopicResistivity/compositeAllOnLeft_1_28_2019}
	\caption[Stimulation electrode locations]{Shown are the locations of the stimulation electrodes projected in common MNI coordinates onto the left hemisphere. Due to both post-operative brain shift and reconstruction error, the electrode locations are not exact.}
	\label{fig:mrCompositeBrain}
\end{figure}
%
%\renewcommand{\tabcolsep}{1pt}
%\renewcommand{\arraystretch}{0.7}
%\begin{table}[ht]
%	\scriptsize
%	\begin{tabularx}{\textwidth}{@{}lXXXXXX@{}}
%		\toprule
%Subject \#  & class & Types of electrodes & Stimulation Amplitudes (mA) & Stimulation electrode \newline separation (cm) & Pairs stimulated \newline (-/+) & total \# stimuli \\
%		\midrule
%1 & ECoG & 1x64 ECoG grid & 1.75 & 1 & 22/30 & 3001 \\
%2 & ECoG & 1x64 ECoG grid & 0.75 & 1 & 13/14 & 10000 \\
%3 & ECoG & 1x64 ECoG grid & 3.5 & 1 & 11/12 & 10005 \\
%4 & ECoG & 1x64 ECoG grid & 0.75 & 1 & 59/60 & 7014 \\
%5 & ECoG & 1x64 ECoG grid & 3 & 1 & 56/55 & 10590 \\
%6 & ECoG & 1x64 ECoG grid & 2.5 & 1 & 54/62 & 7000 \\
%7 & ECoG & 1x64 ECoG grid & 1.75 & 1 & 64/56 & 10000\\
%		\bottomrule
%	\end{tabularx}
%	\caption[Subject characteristics]{Table showing the subject number, type of electrodes used, the stimulation current for the constant current stimulation, the distance between the stimulation electrodes, which were the pairs of electrodes stimulated (-, cathodal first, + anodal first), and how many total stimuli were delivered}
%	\label{table:mrSubjs}
%\end{table}

\renewcommand{\tabcolsep}{1pt}
\renewcommand{\arraystretch}{0.7}
\begin{table}[ht]

	\begin{tabularx}{\textwidth}{@{}lXXX@{}}
		\toprule
		Subject \# & Stimulation Amplitudes (mA) & Pairs stimulated \newline (-/+) & total \# stimuli \\
		\midrule
		1  & 1.75 & 22/30 & 3001 \\
		2  & 0.75 & 13/14 & 10000 \\
		3  & 3.5 & 11/12 & 10005 \\
		4 & 0.75 & 59/60 & 7014 \\
		5 & 3 & 56/55 & 10590 \\
		6  & 2.5  & 54/62 & 7000 \\
		7  & 1.75  & 64/56 & 10000\\
		\bottomrule
	\end{tabularx}
	\caption[Subject characteristics]{Table showing the subject number, the stimulation current, which were the pairs of electrodes stimulated (-, cathodal first, + anodal first), and how many total stimuli were delivered}
	\label{table:mrSubjs}
\end{table}

\section{Data Acquisition}

Our data was acquired at 12 kHz using a TDT System 3 with a RZ5D and PZ5 Neurodigitizer (Tucker Davis Technologies, Alachua, Florida, USA). We delivered biphasic, bipolar, constant current stimulation stimulation pulses using a TDT IZ2H-16 stimulator and a LZ48-400 battery pack. The length of each phase was 1.2 msec. The voltage delivered to the stimulation electrodes during constant current stimulation was also measured using the TDT.

During each stimulation pulse, the stimulation artifact voltages were measured for all of the electrodes that were not stimulated. These "artifacts" are our signals. For example, Figure 2 shows what they look like for subject 3. We remove any local, time varying DC offset by subtracting the average time period from 100 ms to 5 ms before the stimulation pulse begins.

To calculate the resistivity values, we developed an algorithm to extract the quasi-static voltages. Briefly, we average all of the recorded pulses, and then use z-score thresholding to detect the onset and offset of each stimulus pulse. We then detect the transition point as the stimulus waveform switches polarity, and subsequently shift our start and end indices by a fixed number of samples to ensure we are inside the stimulation waveform regime. We then use these start and end indices on a single trial basis to extract the standard deviations at every time point in the quasi-static regime, as well as the standard deviation across samples(Figure \ref{fig:mrStimExtraction})

\begin{figure}[ht]
	\centering
	\includegraphics[width=0.5\textwidth]{figures/macroscopicResistivity/stimulation_extraction}
	\caption[Data extraction]{The magnitude of the stimulation artifact for each recording electrode was determined by fitting the flat regions during the positive and negative phases of the bipolar biphasic stimulation pulses. This minimized the edge transients.}
	\label{fig:mrStimExtraction}
\end{figure}

The means and standard deviations for the recorded voltages for our subjects are highly reproducible (Figure \ref{fig:mrMeansStandards})

\begin{figure}[ht]
	\centering
	\includegraphics[width=0.95\textwidth]{figures/macroscopicResistivity/meansAndStds_first7}
	\caption[Data quality]{The means and the standard deviations of the magnitudes of the stimulation artifacts for the first seven subjects plotted versus the electrode number. The blue and green lines indicate the stimulation electrodes. The stimulation artifact was not measured at the stimulation electrodes. Note the reproducibility of the measurements.}
	\label{fig:mrMeansStandards}
\end{figure}

%Electrode Capacitance
%
%The easiest way to measure the electrode capacitance under our constant current pulsed stimulation conditions is to reduce the stimulation current so that no significant electrochemistry occurs. Figure 5 shows the measured stimulation voltage waveforms for a clinical ECoG strip in saline for two stimulation currents 50 and 500 microamps. Both show the voltage jump just after the stimulation starts.
%
%For 50 microamps, the measured voltage increase after the initial voltage jump is linear in time. From the slope of this rise, we determined the electrode capacitance using $Q=CV$.
%Because the simulation current is constant,
%
%$$Q = (5 10^{-6} amps)(1.2 10^{-3} sec) = C (0.15 \; volts)$$
%
%Solving for $C$, we find $C$=0.4 microfarads. Because there are two electrodes in series, the capacitance of each electrode is 0.8 microfarads.
%This is in reasonable agreement with the capacitance calculated using a literature value for the double-layer capacitance per unit area
%
%$$C = (Co)(area)$$
%$$C= (18 \; microfarads/cm^2)( \pi r^2)$$
%$$C= (18 \; microfarads/cm^2)(3.14)(0.115)^2$$
%$$C= 0.75 \; microfarads$$
%
%For 500 microamps, the measured voltage increase after the initial voltage jump is not linear in time. This is due to the electrochemistry. Platinum is being reduced at the cathode and oxidized at the anode. This produces the curvature in the waveform.

\section{Experimental results for the statics and dynamics of ECoG electrodes}

%\subsection{Validity of Measurements}
%
%A common issue in recordings of electrical activity is cross talk between electrodes as well as the wires leading to the recording system \cite{Nelson2017}. We believe that our results are faithful measurements of the propagating electrical waves due to the lack of transients that would indicate capacitive coupling during the periods of rapid current changes (see figure \ref{fig:mrStimExtraction} with extracted waveform). Similarly, if we observed significant cross talk between channels, we would not be able to recapitulate the solution resistivities accurately as demonstrated above.

\subsection{Two-point measurements}

We use constant current bipolar biphasic stimulation. At low currents, the voltage required to produce pulsed constant current stimulation depends on the resistance and the double-layer capacitance. At intermediate currents, it depends on the pseudo-capacitance associated with the reversable oxidation and reduction of the electrodes (Fig. \ref{fig:mrStimSchem}). At high currents, it depends on the electrolysis of the water [???].

We extracted all time windows around when the stimulator was set to deliver a bipolar, biphasic pulse, and discarded any pulses where the stimulator incorrectly delivered stimulation, which we defined as being greater than 1.5 times, or 0.5 times the median maximum peak voltage delivered (We excluded 534/10590 trials for subject 5, and 139/7000 trials for subject 6). We consider the first phase of stimulation, in order to avoid analyzing where the stimulator failed to deliver a proper biphasic waveform during the second phase of stimulation at high currents.

All of measured stimulation voltage pulses required to produce the experimental current levels are shown in Figure \ref{fig:mrStimMonInd}. The average of these curves is shown in figure \ref{fig:mrStimMonAverage}.

	\begin{figure}[ht]
		\centering
		\includegraphics[width=0.95\textwidth]{figures/macroscopicResistivity/stimMonitorInd}
		\caption[Individual stimulation monitor output]{Individual trials of the first phase of the monitored voltage on the stimulator output. N represents the number of trials visualized, as well as used for further analyses}
		\label{fig:mrStimMonInd}
	\end{figure}

\begin{figure}[ht]
	\centering
	\includegraphics[width=0.95\textwidth]{figures/macroscopicResistivity/stimMonitorAverage}
	\caption[Average stimulation monitor output]{Individual trials of the first phase of the monitored voltage on the stimulator output. N represents the number of trials visualized, as well as used for further analyses}
	\label{fig:mrStimMonAverage}
\end{figure}

In order to extract the two-point resistivity as described above, we first have to calculate the jump voltage, which is the first part of the curve in Figure \ref{fig:mrStimSchem} b)

\begin{figure}[ht]
	\centering
	\includegraphics[width=0.8\textwidth]{figures/macroscopicResistivity/resistivity_schematics_part2}
	\caption[Constant current stimulation]{a) Schematic of our constant current stimulation pulses. b) Different parts of the recorded voltage required to drive the constant current stimulation include an immediate electrode jump resistance, charging of the double layer capacitance, and Faradaic reactions}
	\label{fig:mrStimSchem}
\end{figure}s

The time dependence of the voltage curves is shown in Figure \ref{fig:jumpvoltages}. It is clear from these plots that there is a non-linear relationship between each individual trial and jump voltage, suggesting that two-point measurements in our stimulation setup are noisy.

\begin{figure}[ht]
	\centering
	\includegraphics[width=0.95\linewidth]{figures/macroscopicResistivity/JumpVoltages}
	\caption[Jump voltages over time all subjects]{The initial jump voltages for all subjects to illustrate the time dependence and non-linearities}
	\label{fig:jumpvoltages}
\end{figure}

The same data is shown again normalized by dividing by the stimulation current in figure \ref{fig:twoPointFigures}. In this form the data has units of ohms and is the dynamic impedance of the sample in response to our applied stimulation. Properly engineered stimulators must be able to drive these dynamic impedances. We determined the electrode resistances by extracting the voltage jump at the onset of the stimulation and dividing it by the applied current.
This gives us total resistance for each subject.

\begin{figure}[ht]
	\centering
	\includegraphics[width=0.95\linewidth]{figures/macroscopicResistivity/twoPointFigures}
	\caption[Two-point results]{The mean voltage (+/- standard deviation) is plotted for each subject. This is subsequently divided by the applied current, and appropriately converted into resistivity values}
	\label{fig:twoPointFigures}
\end{figure}

The values are shown in Table ??? (Should we do one table with all of the resistivity values?) .

\section{Flat vs. Spherical Models}
The brain can be considered to be a spherical object, but on the scale of an array of ECoG electrodes, what is the local topography? We visualize our reconstructed electrode grids, and calculate curvatures. We demonstrate that the electrode array can locally be flat, or in other places well represented by a sphere. Motivated by this, we consider both analytic flat and spherical apparent resistivity models.

FIGURE ABOUT CURVATURE

\subsection{Voltage over time}
To assess the time dependent factor of resistivity in the brain, we calculated the average voltage during the first phase of stimulation for each individual trial for every channel, and plotted this as a function of time (Figure \ref{fig:mrVoltageOverTime}). As shown, the resistivity of the brain over time stays relatively constant with no systematic variation, indicating that averaging across measurements is appropriate.

\begin{figure}[ht]
	\centering
	\includegraphics[width=0.95\textwidth]{figures/macroscopicResistivity/subject_4_variationTrial}
	\caption[Recorded voltage over time]{Plot of mean trial voltage across time for the first phase for all non-stimulation channels for subject 4}
	\label{fig:mrVoltageOverTime}
\end{figure}

\subsection{Saline Phantom Validation}

Using our same data extraction and analysis strategy as outlined in the methods, we compared our results with ECoG grids to a commercial conductivity meter using a sodium chloride saline phantom model to serve as validation. We created two solutions of 0.511 and 3.44 ohm-m NaCl solutions. By using the TDT, we determined the solution resistivities to be 0.514 ohm-m and 3.44 ohm-m respectively (Figure \ref{fig:mrSalineValidation})

\begin{figure}[ht]
	\centering
	\includegraphics[width=0.95\linewidth]{figures/macroscopicResistivity/salineValidation}
	\caption[Saline Validation]{The initial jump voltages for all subjects are concatenated to illustrate the time dependence and non-linearities}
	\label{fig:mrSalineValidation}
\end{figure}

\section{Four-electrode measurements - flat model}

There are many (62 choose 2=1891) choices for the recording electrode pairs, and subsequently the following histograms have these number of calculated values.

%Figure 13 shows the 4-point histograms.
%Two of the four electrodes are stimulation electrodes and two are recording electrodes. Our subjects had 64 electrode ECoG grids. So there are many (62 choose 2=1891) choices for the recording electrode pairs The eighth histogram in figure 13 is the sum of the seven individual histograms. Note that the path of the current flow that produces the recorded voltages is different for every pair of recording electrodes.

\section{Four-electrode measurements - spherical theory}

\section{Discussion}

There has been considerable interest in, and controversy about, whether neural signals propagate without  frequency-dependent absorption or dispersion\cite{Miceli2017}. This was not a primary objective of our study, but we can say a little about it. To zeroth order, the square wave like current pulses we inject at the stimulation electrodes propagate without any obvious absorption or dispersion: the voltages we measure using the recording electrodes exhibit little dispersion over the frequencies in the pulses (they remain square-wave like out to the most distant recording electrodes). They also do not exhibit any excess attenuation (they have the 1/r decay expected for a resistive medium).

If we had recording electrodes very close to the stimulation electrodes, we might be able to study the cross over from constant-voltage-like
to constant-current-like behavior. Presumably, we do not see it because our recording electrodes are so far away that all three types of
electrode behavior are in their asymptotic $1/r$ regime.

As shown in the section on 2-point measurements, the voltage exhibits non-linear drifts over time, which affects. The recorded voltages on the recording electrodes are more uniform, illustrating the benefits of the 3 and 4-point techniques over the 2-point technique when possible.

%Figure 14 shows false color plots of the measured V/I values.
%These plots show the spatial structure of the surface voltage distributions well, but do not show the actual values quantitatively.
%Linear plots versus the electrode number display the values quantitatively, but do not display the spatial structure well.
%Figures 15 and 16 show linear plots of V/I parallel and perpendicular to the stimulation dipole axis.
%
%These plots clearly show the range of the apparent resistivity values for these subjects.
%These plots also show the relative contributions to the average curve of V/I values.
%
%
%In geoelectric prospecting and semiconductor characterization, the resistivity versus depth is determined by making measurements as the recording electrode spacing is increased [???].
%Larger recording electrode separations probe deeper. This motivated us to extract the apparent resistivity versus electrode separation.
%
%We divided the electrode separations into bins. Each bin was 1 cm wide. The centers of the bins were at 1.5, 2.5, 3.5, 4.5, and 5.5 cm. We then fit all of the electrodes within each bin
%to determine the apparent resistivity. The distance dependence of the apparent resistivity is shown in figure 17.
%
%Note that the apparent resistivity values tend to increase as the recording electrode separation increases. This is the expected behavior if the resistivity of the white matter is larger than that of the gray matter because larger recording electrode separations include larger contributions from the white matter.
%
%White matter tracts in humans demonstrate complex morphologies, and depending on the orientation, have different resistivity values []. We expect that as we increase the electrode separation, there are contributions from both parallel (lower/equal resistivity to gray matter), and perpendicular white matter (higher resistivity than gray matter) tracts, which in aggregate results in an increased resistivity at greater depths.
%
%To see if we could reduce the effects of the anatomy, we aligned the locations of the stimulation electrodes to create an average brain.
%Figure 14 shows the spatial distributions of the measured responses to stimulation. The measurements are shown in ohms, the measured voltages
%have been divided by the stimulation current for each subject.
%
%If all of the brains were lamellar, all of the spatial distributions would look the same. There are clear differences in the spatial distributions.
%To see if we could reduce these differences, we first averaged over the subjects. This reduced but did not eliminate the differences.
%To reduce the differences further, we used the symmetry expected for the distributions. They should be left-right symmetric and up-down antisymmetric.
%Figures 18 and 19 show the result of averaging and symmetrizing.
%
%The next step is to see how well the dependence of the apparent resistivity on the separation of the recording electrodes can be determined from the averaged and symmetrized values.
%This is shown in figures 20 and 21.
%
%From the fit, the apparent resistivity vs the distance from the center of the stimulation dipole is given in ohm-m by 2.24 + 0.41*(distance in cm).
%It ranges from 3 ohm-m at a distance of 2 cm to 5 ohm-m at a distance of 7 cm.

\section{Future Directions}

This research points to the applicability of simple analytic models to accurately depict the voltages measured on the surface of the cortex. Future directions will explore both measurements and stimulation in implanted depth electrodes to explore the extensibility of this model to three dimensions.

Comparison and integration with FEM models will allow for enhanced understanding of current spread and cortical activation through tissue. The analytic models may serve as a simple benchmark for FEM models to be compared against. These analytic models do not replace FEM models, as on the smaller (less than electrode radius) scale the errors increase for the analytic model without correction factors or more complicated structuring. For studies with Utah arrays and $\mu$ECoG arrays, FEM models would be of tremendous benefit. Similarly, we cannot comment on the differences in voltages at the top or bottom of sulci and gyri purely from our model, and similarly, the electric field vectors at any component would depend heavily on the geometry of the interface between materials of different resistivities in cortex.

\section{Conclusions}

\section{Code Availability}
The code to recreate our analyses and extensive figures for all 21 subjects is available from the following repository

https://github.com/davidjuliancaldwell/ElectrodeModeling

\section{Data Availability}
The curated data for our subjects is also available publicly

https://drive.google.com/open?id=1j\_J7ncKy4aAgdwGUmLxid6NIPJ8\_Eytz


% !TeX root = ../../main.tex
% Add the above to each chapter to make compiling the PDF easier in some editors.

\chapter{Background and Foundations}\label{chapter:background_and_foundations}

This chapter represents the basic concepts used throughout our work. First, a brief introduction to the reinforcement learning field. Second, Markov Decision Processes (MDPs), the standard mathematical formalism framework for reinforcement learning will be introduced. Then, We will discuss the value functions and policy gradient methods. Next, we will discuss the methods used to achieve the RL goal and differentiate between them. In addition, we conclude the use of deep learning in reinforcement learning with some of Deep Reinforcement Learning (DRL) Algorithms. Finally, we discuss related work in distributed reinforcement learning.

\section{Reinforcement Learning}

\textbf{RL} (\textit{The science of decision making}) is a machine learning approach to teach agent how to solve tasks through trial and error interaction with a dynamic, unknown environment. It formalizes the idea of \textit{\textbf{rewarding or punishing}} an agent for its behavior. This makes the agent more likely to repeat or forego that behavior in the future. In contrast with other machine learning methods, the agent is not told the proper actions to take. Instead, the agent interacts with its environment, and upon observing the consequences of its actions, it can learn to alter its behavior in response to rewards received. The goal of the agent is to maximize the expected cumulative reward. The main components of RL are the \textbf{Agent} and the \textbf{Environment}. The typical interaction loop between agent and environment is illustrated in Figure~\ref{fig:agent_env} below.

\begin{figure}[!htb]
	\centering
	\includegraphics[width=.5\linewidth]{figures/Agent-Env.png}
	\caption{Reinforcement Learning interaction loop. The agent takes an action at a\textsubscript{t} state s\textsubscript{t}. The environment then responds with the corresponding reward r\textsubscript{t+1} and the new state s\textsubscript{t+1}, which are fed back to the agent~\parencite{arulkumaran2017brief}}
	\label{fig:agent_env}
\end{figure}

\subsubsection{Agent}\label{Agent}

An agent is considered the brain (e.g. for a robot, the agent is typically not the whole robot, but the specific program running on the robot's CPU that makes the decision on the action). The agent decides to takes proper actions and it is where the learning process happen and improve over time. For the learning process, the agent observes the state of the world and based on that takes an action. The environment changes according to the agent's actions and it might also change on its own. Examples of the agents are a drone making a delivery, or a robot learning to walk.
%or super mario navigating a video game.

\subsubsection{Environment}\label{Environment}

An Environment constitutes a world for the agent to act and learn from. To describe the environment to an agent we have a \textbf{state} \(s\) which is a complete description of the state of the world. Sometimes an \textbf{observation} \(o\) is a \textit{partial} description of a state, which may omit information. Environments could be the whole surrounding 3D space or 2D images from cameras for a real-world task like a robotic arm, or it could represent an entire virtual world or games from an emulator like OpenAI Gym~\parencite{brockman2016openai}. When the agent can observe the complete state of the environment, we say that the environment is \textbf{fully observed} (e.g. Atari games). When the agent can only see a partial observation, we say that the environment is \textbf{partially observed} (e.g. robotic camera in a navigation task).
Some of the environments can be found in this figure~\ref{fig:envs_examples}

\input{chapters/02/existing_envs.tex}

\subsection{Markov Decision Process}

Formally, reinforcement learning can be described as a Markov decision process, An MDP is a 5-tuple, $ \left\langle S, A, \mathcal{T}, R, \gamma \right\rangle $ which consists of:

\begin{itemize}
	\item A set of all states \(S\), plus a distribution of starting states \(p(s0)\).
	\item A set of valid actions \(A\).
	\item Transition dynamics $ \mathcal{T}\left(\mathbf{s}_{t+1} | \mathbf{s}_{t}, \mathbf{a}_{t}\right) $ that map a state-action pair at time \(t\) onto a distribution of states at time \(t+1\).
	\item An immediate $ \mathcal{R}\left(\mathbf{s}_{t}, \mathbf{a}_{t}, \mathbf{s}_{t+1}\right) $ reward function.
	\item A discount factor \(\gamma \in(0,1)\), where lower values place more emphasis on immediate rewards.
\end{itemize}

Extra objects can be defined depending on problem setting:
\begin{itemize}
	\item $\rho_0$: Initial state distribution
\end{itemize}

Markov Decision Process refers to the fact that the system obeys the \textbf{Markov property}: which indicates that transitions only depend on the most recent state and action, and no prior history, in other words, the future is conditionally independent of the past given the present state. $ p\left(s_{t+1} | s_{1}, a_{1}, \ldots, s_{t}, a_{t}\right)=p\left(s_{t+1} | s_{t}, a_{t}\right) $

\subsection{Rewards and Return}

The reward function (\(R\)) is critically important in RL. It depends on the current state of the world, the action just taken, and the next state of the world:

\begin{center}
	\begin{equation}
		r_{t}=R\left(s_{t}, a_{t}, s_{t+1}\right).
	\end{equation}
\end{center}

The goal of the agent is to maximize some notion of cumulative reward over a trajectory. There are two kinds of return, \textbf{the finite-horizon undiscounted return}, which is just the sum of rewards obtained in a fixed window of steps:

\begin{center}
	\begin{equation} \label{eq:1}
		R(\tau)=\sum_{t=0}^{T} r_{t}.
	\end{equation}
\end{center}

Another kind of return is \textbf{the infinite-horizon discounted return}, which is the sum of all rewards ever obtained by the agent, but \textit{discounted} by how far off in the future they’re obtained. This formulation of reward includes a discount factor \(\gamma \in(0,1)\):

\begin{center}
	\begin{equation} \label{eq:2}
		R(\tau)=\sum_{t=0}^{\infty} \gamma^{t} r_{t}.
	\end{equation}
\end{center}

The use of a discount factor is crucial as \textbf{mathematically} an infinite-horizon sum of rewards may not converge to a finite value, and is hard to deal with in equations. But with a discount factor and under reasonable conditions, the infinite sum converges.

\subsection{Policies}

A policy ($\pi$) is a rule used by an agent to decide what actions to take. It's considered as the strategy that the agent employs to determine the next action based on the current state. It maps states to actions $ \pi : \mathcal{S} \rightarrow p(\mathcal{A}=\mathbf{a} | \mathcal{S}) $, the actions that promise the highest reward.

The policy can be deterministic where it is usually denoted by $ \mu: a_{t}=\mu\left(s_{t}\right) $ or stochastic denoted by $ \pi:  a_{t} \sim \pi\left(\cdot | s_{t}\right) $
with the two most common kinds of stochastic policies categorical policies (discrete action spaces) and diagonal Gaussian policies (continuous action spaces).

A very important two key computations for training stochastic policies are:
\begin{itemize}
	\item Sampling actions from the policy, and
	\item Computing log likelihoods of particular actions, $ \log \pi_{\theta}(a|s) $.
\end{itemize}

\subsubsection{Reinforcement Learning Goal}

The goal of RL is to select a policy which maximizes the \textbf{expected return} ($ J(\pi) $) when the agent acts according to it, where:
\begin{center}
	\begin{equation} \label{eq:expected_return}
		J(\pi)=\int_{\tau} P(\tau | \pi) R(\tau)=\underset{\tau \sim \pi}{\mathrm{E}}[R(\tau)],
	\end{equation}
\end{center}

with $ P(\tau | \pi) $ as the probability distributions over \textit{T-step} trajectories.
\begin{center}
	\begin{equation}
		P(\tau | \pi)=\rho_{0}\left(s_{0}\right) \prod_{t=0}^{T-1} P\left(s_{t+1} | s_{t}, a_{t}\right) \pi\left(a_{t} | s_{t}\right).
	\end{equation}
\end{center}

So the central optimization problem of reinforcement learning can be described as:
\begin{center}
	\begin{equation}
		\pi^{*}=\arg \max _{\pi} J(\pi),
	\end{equation}
\end{center}
with $\pi^*$ being the optimal policy.

\subsection{Learning Optimal Policies}

There are two main approaches to solve reinforcement learning problems: methods based on \textit{value functions} (Critic-only), and methods based on \textit{policy search} (Actor-only). There is also a hybrid, \textit{actor-critic approach}, which combines both value functions and policy search, where the actor and critic are both represented explicitly and learned separately. We will now explain these approaches and other useful concepts for solving RL problems.

\subsubsection{Critic-only: Learning based on value functions}
Critic only methods are based on the idea to first find the optimal value function and then to derive an optimal policy from this value function.  Value function methods are based on estimating the value (expected return) of being in a given state. The state-value function $V^{\pi}(\mathbf{s})$ is the expected return when starting in state $\mathbf{s}$ and following $\pi$ is defined as:
\begin{center}
	        \begin{equation}
	                V^{\pi}(\mathbf{s})=\mathbb{E}[R | \mathbf{s}, \pi],
	        \end{equation}
\end{center}

and the optimal state-value function defined as:
\begin{center}
	        \begin{equation}
	                V^{*}(\mathbf{s})=\max _{\pi} V^{\pi}(\mathbf{s}) \quad \forall \mathbf{s} \in \mathcal{S}.
	        \end{equation}
\end{center}

The optimal policy, $\pi^{*}$, has a corresponding state-value function $V^{*}(\mathbf{s})$ where it could be retrieved by choosing among all actions available at $\mathbf{s_t}$ and picking the action $\mathbf{a}$ that maximizes expected optimal value for the succeeding state.

In practice, the MDP is often unknown and the transition dynamics $\mathcal{T}$ are unavailable so the only way to get information about it is by interacting with the environment and observing rewards.
Hence, construct another function, the \textit{state-action value or quality function} $Q^{\pi}(\mathbf{s}, \mathbf{a})$ estimates the value function and derives an optimal policy.
It is similar to $V^{\pi}$, except that the initial action $\mathbf{a}$ is provided, and $\pi$ is only followed from the succeeding state onwards:
\begin{center}
	        \begin{equation}
	                Q^{\pi}(\mathbf{s}, \mathbf{a})=\mathbb{E}[R | \mathbf{s}, \mathbf{a}, \pi].
	        \end{equation}
\end{center}

A selection of these approaches \textit{dynamic programming, temporal difference learning, eligibility trace} is used in order to learn these functions.

\subsubsection{Actor-only: Policy search}
Policy search methods do not need to maintain a value function model but directly search for an optimal policy $\pi^{*}$. This is only possible if the search space is restricted. Typically, a class of policies is parametrized by a real-valued parameters vector $\theta$, these parameters are updated to maximize the expected return $\mathbb{E}[R | \theta]$ using either gradient-based or gradient-free optimization~\parencite{deisenroth2013survey}.

Policy based reinforcement learning is an \textit{optimization} problem, we need to find $\theta$ that maximize the expected return~\eqref{eq:expected_return}.
Some approaches do not use gradient-like \textit{Hill climbing, Genetic algorithms}, but the common use is \textit{policy gradients}.

\textbf{Policy Gradients}: Gradients provide a strong learning signal to improve a parameterized policy. In order to compute the expected return \eqref{eq:expected_return}, averaging trajectories generated by the current policy parameterization is needed. This averaging requires either deterministic approximations or stochastic approximations via sampling.
Due to gradients cannot pass through these samples of a stochastic function,
Hence we use an estimator of the gradient, known in RL as the \textbf{REINFORCE rule}~\parencite{williams1992simple}, \textit{score function}~\parencite{fu2006gradient} or \textit{likelihood-ratio estimator}~\parencite{glynn1990likelihood}.
Gradient ascent using the estimator, which is similar to the practice of optimizing the log-likelihood in supervised learning, increases the log probability of the sampled action, weighted by the return.

\subsubsection{Actor-Critic Method:}
Actor-Critic method is the combination of both \textit{value functions} and an explicit representation of \textit{the policy} as shown in Figure ~\ref{fig:actor_critic}, where the \textit{actor} (policy) learns by using feedback from the \textit{critic} (value function). These methods trade-off variance reduction of policy gradients with bias introduction from value function methods.

\begin{figure}[!htb]
	        \centering
	        \includegraphics[width=.3\linewidth]{figures/actor_critic.png}
	        \caption{Actor-critic set-up. The actor (policy) receives a state from the environment and chooses an action to perform. At the same time, the critic (value function) receives the state and reward resulting from the previous interaction.~\parencite{arulkumaran2017brief}}
	        \label{fig:actor_critic}
\end{figure}

% \clearpage

\section{Deep Reinforcement Learning}

A recent breakthrough in deep learning fields relied on efficiently training a deep neural network on large training datasets, which improves many fields including \textit{computer vision and speech recognition}. These models are trained directly from raw inputs using stochastic gradient descent to update the networks' weights. These successes lead its way to reinforcement learning in which it connects RL algorithms with deep neural networks to operate directly on raw images and efficiently generate training data using stochastic gradient updates.

% \subsection{State of the Art}
In the following, we briefly discuss four state-of-the-art algorithms, \textit{Deep Q Network}, two deep policy search methods: \textit{Deep Deterministic Policy Gradient} and \textit{Proximal Policy Optimization}, also an asynchronous deep actor-critic method called \textit{Asynchronous Advanced Actor-Critic}.
These methods are currently the most popular and effective algorithms, proposed by DeepMind\footnote{\url{https://deepmind.com}} and OpenAI\footnote{\url{https://openai.com}}. In this work, some of these methods are used for experiments.

\subsubsection{Deep Q Network}
Deep Q Network (\textbf{DQN}) was first proposed by~\parencite{mnih2013playing}, it presents the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning.

Q-learning algorithm was used to make the decision, using stochastic gradient descent to update the weights. Since deep learning handles only with independent data samples, the \textit{experience replay} mechanism was used to break correlations along with \textit{Fixed Q-target}. DQN algorithm replaces the tabular representation for Q-value function with the deep neural network.

\subsubsection{Deep Deterministic Policy Gradient}
Since the rise of deep neural network function approximations for learning value or action-value function, deep deterministic policy gradient method have been proposed by~\parencite{lillicrap2015continuous}. It used an \textit{actor-critic} approach based on the \textbf{Deterministic Policy Gradient (DPG) algorithm}~\parencite{silver2014deterministic}, combined with \textit{experience replay and fixed Q-target} techniques which inspired by \textbf{DQN} to use such function approximation in a stable and robust way. In addition, a robust strategy in deep learning called \textit{batch normalization}~\parencite{ioffe2015batch} is adopted to scale the range of input vector observations in order to make the network capable of finding hyper-parameters which generalize across environments with different scales of state values.
The problem of exploration in off-policy algorithms like Deep Deterministic Policy Gradient (DDPG) can be addressed in a very easy way and independently from the learning algorithm. Exploration policy is then constructed by adding noise sampled from a noise process \textit{N} to the actor policy.

\subsubsection{Proximal Policy Optimization}
One of the main issues in policy gradient methods is defining the step size (\textit{learning rate $\alpha$}). Hence, the new robust policy gradient methods, which we call Proximal Policy Optimization (PPO)~\parencite{schulman2017proximal, heess2017emergence} was proposed to solve this problem. It also uses some of the benefits from trust region policy optimization~\parencite{schulman2015trust}. these methods bound parameter updates to a trust region to ensure stability. This algorithm is similar to natural policy gradient methods and it is also considered as a variant of Trust Region Policy Optimization (TRPO), it directly uses the first-order optimization methods to optimize the objective.

\subsubsection{Asynchronous Advanced Actor-Critic}
Asynchronous Advanced Actor Critic (A3C)~\parencite{mnih2016asynchronous} is an asynchronous method using Advanced Actor-Critic (A2C). \textit{\textbf{Asynchronous}} means asynchronously execute multiple agents in parallel, on multiple instances of the environment and all using a replica of the   (NN) (\textit{asynchronous data parallelism}). It often works in a multi-core CPU or GPU. The idea behind it is that there is a \textit{global network} and \textit{multiple actor-learners} which have their own set of network parameters. A thread is dedicated for each agent, and each thread interacts with its own copy of the environment.
Giving each thread a different exploration policy also improves robustness, since the overall experience available for training becomes more diverse. Moreover, in A3C just one deep neural network is used both for estimation of policy $\pi(s)$ and value function $V_{\pi}(s)$; because we optimize both of these goals together, we learn much faster and effectively. We also don’t need to consider the data correlation and oscillations issues because different agent gets different transitions when playing in the same environments.

\input{chapters/02/related_work.tex}


% !TEX TS?program = pdflatexmk
\documentclass{article}
\usepackage{geometry}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{bold-extra}
\usepackage{authblk}
\usepackage[backend=biber, citestyle=authoryear]{biblatex}
\usepackage{graphicx}
\usepackage{epstopdf}
\usepackage{placeins}
\usepackage[width=.8\linewidth, labelfont=bf]{caption}
\usepackage[
	hidelinks,
	colorlinks=true,
	allcolors=blue
]{hyperref}

\addbibresource{bibliography.bib}
\graphicspath{{.}}

\setlength{\parindent}{0em}
\setlength{\parskip}{0.25cm}
\geometry{
	a4paper,
	left=1in,
	right=1in,
	top=0.75in,
	bottom=0.75in
}

\title{\sc COVID-19 Age-Structured SEIR Model\\\Huge{\bfseries Technical Report}} %FIXME
\author[1]{Pranav Minasandra}
\author[2]{Vishwesha Guttal}
%\author[3]{Prateek Sharma}
%\author[4]{Shivkumar Jolad}
%
\affil[1]{Centre for Ecological Sciences \& UG Programme,  Indian Institute of Science, Bengaluru}
\affil[2]{Centre for Ecological Sciences,  Indian Institute of Science, Bengaluru}
%\affil[3]{Department of Physics, Indian Institute of Science}
%\affil[4]{Public Policy, School of Liberal Education, FLAME University, Pune}
\date{}

\begin{document}
\maketitle\hrule

\section*{Disclaimer}
This model is simple, ignores many features of real disease dynamics and demographic features of Indian population. Hence, it is to be used only for understanding concepts but NOT for making quantitative forecasts about the pandemic.
%
%\section{What is this model?}
%This is an age-structured SEIR compartmental model of the epidemiology of the coronavirus that causes COVID-19.
%
%Here, we assume, with respect to the disease, that all individuals can be categorised as one of the following:
%\begin{itemize}
%\item Susceptible ($S$)
%\item Exposed ($E$) (to virus but not sick yet)
%\item Infectious ($I$) (who in turn are categorised into asymptomatic, symptomatic (mild-moderate), hospitalised and critically ill).
%\item Recovered  ($R$)  (and hence immune)
%\item Dead ($D$)
%\end{itemize}
%
%\emph{Age-structured} means is that we account for the age structure of a particular demographic. In our specific case, we pay attention to what proportion of the population is senior citizens.
%
%This model is parametrised for the states of Maharashtra, Karnataka, Tamil Nadu, and the country of India. The parameters were derived from intelligent guesswork from multiple data sources (see the About section, but also Caveats).
%
%\section{What is the use of this model?}
%This model can be used to understand how the epidemic spreads without and with interventions. Specifically, we incorporate two types of interventions -- one where the vulnerable population is isolated and the other is the complete lockdown. Therefore, we can investigate how different types of physical distancing and their effectiveness affects disease spread.
%
%On Tab 2 (Total Infections), one can view the total number of individuals who have been affected by the virus. This can be seen for two different intervention conditions and a baseline sceario.
%
%The model illustrates the role of incubation time, which provides the disease a sort of \emph{inertia}.
%
%On Tab 3 (Hospitalisations), we show the number of hospitalised individuals - based on our model analysis. There is a gap between when a person gets infected and when she/he is brought to a hospital; therefore, we see that this number reacts slowly to a lockdown. This effect could be seen in reality as well.
%
%\section{Can I plan according to the results of this model?}
%No. This model is for illustrative purposes only, and should not be used for quantitative forecasting.
%%Most models in this time will not be reliable, for the simple reason that not much is known about COVID-19 precisely.
%%Please wait for better parametrised more elaborate models that will surely appear at some point in the future.
%
%\section{Can I play with the code for this model?}
%Yes, please. The code for this model is available at our github repository:
%If you have any questions regarding the modelling approaches used, please send a well-drafted email to
%\texttt{pranavm@iisc.ac.in} .
%
%\section{Why don't you show actually observed data anywhere?}
%Empirical data is subject to the dynamics of societal awareness of the disease, testing rates, stochasticity, and other numbers.
%Numerous questions have been raised about the data we have.
%Therefore, showing data in our plots could have been misleading.
%Moveover, there are many online sources where real data are beautifully shown.
%We therefore refrain from showing this data.
%
%%\section{Why don't you talk about the number of deaths?}
%%Our model incorporates this data, but we believe discussing this variable with models like these is irresponsible at a time like this.
%
%\section{Can you tell me more about your model?}
\section{Age-structured mathematical model for COVID-19}
Our population model is based on the classic SEIR models, which have been appropriately modified to include
various epidemiological features of COVID-19.
Our model closely follows the work of \cite{weitz2020} who has applied it to the US state Georgia, and was inspired by a simpler version by \cite{hill2020}.

We assume that the population consists of susceptible ($S$), exposed ($E$), infected but asymptotic ($I_a$),  infected with mild symptoms ($I_s$),  infected requiring hospitalisation ($I_h$), infected and critical ($I_c$), recovered ($R$) and dead ($D$) individuals. The total population is given by $N=S+E+I_a+I_s+I_h+I_c+R+D$.

We incorporate some key empirical features of covid-19: (i) asymptomatic individuals can infect the susceptible (\cite{li2020science,park2020})
(ii) a person can recover directly from any of the infected stages, or progress sequentially towards higher severity and eventually death.
(iii) Crucially, it is now well established that COVID-19 mortality rates are highly age-dependent, with the population of senior citizens who are
vulnerable (aged more than 60 years) at substantially higher risk than the the younger population (those less than 60 years). Therefore, we incorporate the age-structure available for each of the (three)Indian states that we apply our model to. We consider two age-structures; we denote them as $S_1$ and $S_2$, representing younger and older populations, respectively.

For each age-structure $S_i \,\, i \in \{1,2\}$, the dynamics of epidemic is given by:

\begin{align}
\dot{S_i} & = - \sum_j \frac{(\beta_{a,ji} I_{a,j} + \beta_{s,ji} I_{s,j} +  \beta_{h,ji} I_{h,j} + \beta_{c,ji} I_{c,j})}{N} S_i \\
\dot{E_i} & = \sum_j \frac{(\beta_{a,ji} I_{a,ji} + \beta_{s,ji} I_{s,j} +  \beta_{h,ji} I_{h,j} + \beta_{c,j} I_{c,i})}{N} S_i - \alpha_{e,i} E_i \\
\dot{I_{a,i}} & = (1-p_i) \alpha_{e_i} E - \gamma_{a,i} I_{a,i} \\
\dot{I_{s,i}} & =  p_i \alpha_{e,i} E   - \gamma_{s,i} I_{s,i} -  \alpha_{s,i} I_{s,i} \\
\dot{I_{h,i}} & =  \alpha_{s,i}  I_{s,i} -  \gamma_{h,i} I_{h,i}  -\alpha_{h,i} I_{h,i}  \\
\dot{I_{c,i}} & =  \alpha_{h,i} I_{h,i} - \gamma_{c,i} I_{c,i}  - \alpha_{c,i} I_{c,i}\\
\dot{R_i} & = \gamma_{a,i} I_{a,i} +  \gamma_{s,i} I_{s,i} + \gamma_{h,i} I_{h,i} + \gamma_{c,i} I_{c,i} \\
\dot{D_i} & =  \alpha_{c,i} l_{c,i}
\end{align}

\noindent where $\beta_{x,ji}$'s represent transmission rate of infection from $j$ age-class to $i$ age-class from different infected groups $x$ (with $x \in \{a, s, h, c\}$),  $\gamma$'s represent the rate of recovery for different infected stages and $\alpha$'s represent the rate of progression from one stage of disease to the next.

The dynamics of this model are shown in Fig.\ \ref{fig:modeldynamics}.
\begin{figure}
\centering
\includegraphics[width=.8\linewidth]{modeldynamics.png}
\caption{The typical dynamics of the COVID-19 pandemic, as expected by our model and parametrisation, are shown here.}
\label{fig:modeldynamics}
\end{figure}

\subsection{Model parameters}
We parametrise the model based on findings from recent studies (\cite{wu2020lancet,li2020science,park2020,hill2020}).  In Table 1, we list various epidemiological parameters from the literature. In general, we assume that older age-class is more likely to pass on to severe stages of the infection; we assume that this rate is twice that of the younger age group. We assume a high-mortality rate of the critically infected. See Table 2 for model parameters and Table 3 for relationship between model parameters and empirical values.

We incorporate the total population size and age-structure data for different states based on a model projection of population pyramid for 2021 which uses data from census-2011 of the government of India. See Tables 4.

We consider three scenarios:

(A) {\it Baseline scenario} - which assumes that the trends of the data from the first case in the state $24^{th}$ March 2020 will continue without a full lockdown. Therefore, this baseline includes various measures in place in India upto $24^{th}$ March 20120, such as contact tracing, home quarantine of the suspected, hospital quarantine of the positive cases, etc. Here, we assume that rate of infection of the susceptible population does not depend on age structure; this is motivated by the recent analyses by \cite{singh2020} which shows that house-holds in India often have three-generational interactions.

(B) {\it Lockdown}: We then consider the impact of lockdown which was initiated on $25^{th}$ March. Here, since the rate of effectiveness of such a lockdown is difficult to quantify, especially in light of the migrant population movement it triggered and an in-general denser population profile of India,  we consider two scenarios -- a moderate reduction of the transmission rates (in particular $\beta_a$ and $\beta_s$) corresponding to 75\% or a partial success of $50\%$ only.

(C) {\it Protect the vulnerable}: Here, in addition to the restrictions in baseline scenario, the priority given to protect the vulnerable -- who are typically aged and/or have comordibities like diabetes, hypertension, cardiac or respiratory problems. Therefore, we set only some $\beta$'s to low values. Note that this measure will protect transmission of disease from asymptomatic younger population to the old, but not among younger populations. Specifically, we assume the following transmission rates for asymptomatic individuals:$\beta_{a,12} = 0.25 \beta_a$, $\beta_{a,22}=0.25 \beta_a$, $\beta_{a,11} = \beta_{a,21} = \beta_a$. For the symptomatic, we assume $\beta_{s,12} = 0.1 \beta_s$, $\beta_{s,22}=0.1 \beta_s$, $\beta_{s,11} = \beta_{s,21} = \beta_s$.

Various intervention measures can reduce the damage caused by the pandemic to varying extents.
Figs.\ \ref{fig:infs}, \ref{fig:hosps}, and \ref{fig:deaths}  show how much each intervention measure can affect the dynamic if applied indefinitely.
It is clear that the lockdown is most likely a very strong measure to control the disease,
but (a) it is not possible to know the strength of the lockdown accurately, and (b) an indefinite
lockdown is not feasible for various reasons. In the app, you can control the duration and strength of the lockdown, and
examine qualitatively the effects it can have on pandemic spread.

\begin{figure}[htp]
\centering
\includegraphics[width=.8\linewidth]{infsvsint.png}
\caption{Infections in the COVID-19 pandemic as functions of time, as expected by our model and parametrisation, are shown here.}
\label{fig:infs}
\end{figure}
\begin{figure}[htp]
\centering
\includegraphics[width=.8\linewidth]{hospvsint.png}
\caption{The numbers of hospitalised individuals, as expected by our model and parametrisation, are shown here.}
\label{fig:hosps}
\end{figure}
\begin{figure}[htp]
\centering
\includegraphics[width=.8\linewidth]{deathsvsints.png}
\caption{The number of deaths predicted by our model and parametrisation, are shown here.}
\label{fig:deaths}
\end{figure}

Age-structuring of a population can have impact on the deaths caused by this pandemic.
Fig. \ref{fig:age} shows how hypothetical populations with different numbers of senior citizens
can react very differently to a pandemic. We very cautiously predict that India might have a lower
overall COVID-19 death rate than other countries owing to its relatively younger population.
However, due to large population sizes in India, the absolute number of deaths might be higher.

\begin{figure}
\centering
\includegraphics[width=.8\linewidth]{effectsofage.png}
\caption{Here, 3 hypothetical populations with 1000 individuals each is attacked by the COVID-19 pandemic.
	Death rates as functions of age-structure are shown.}
\label{fig:age}
\end{figure}

\FloatBarrier
\begin{table}[htp]
\caption{Empirical parameters}
\begin{center}
\begin{tabular}{|c|c|c|}
\hline
Term & Description & Value  						 \\  \hline
$\tau_I$ & Average incubation period & 5 days 				 \\
$\tau_a$ & Average duration of asymptomatic infections & 7 days 	 \\
$\tau_s$ & Average duration of mild symptomatic infections & 7 days 	 \\
$\tau_h$ & Average duration of hospitalisations & 4 days 		 \\
$\tau_c$ & Average duration of ICU admissions & 10 days 		 \\
$p_a$ & Fraction of asymptomatic cases & 0.3 				 \\
$p_s$ & Fraction of mild symptomatic cases & 0.55 			 \\
$p_h$ & Fraction of hospitalisations & 0.15 				 \\
$p_c$ & Fraction of severe cases & 0.05 				 \\
$f_c$ & Case fatality rate & 0.01 					 \\
\hline
\end{tabular}
\end{center}
\label{basic-seir:params}
\end{table}%

\begin{table}[htp]
\caption{Model parameter definitions and their values}
\begin{center}
\begin{tabular}{|c|c|c|}
\hline
Parameter & Description & Value \\ \hline
$\beta_{a,ji} = \beta_a$ & Transmission rate from asymptomatic individuals to susceptible  & 0.3 per day \\
$\beta_{s,ji} = \beta_s$ & Transmission rate from symptomatic individuals & 0.6 per day \\
$\beta_{h,ji} = \beta_h$ & Transmission rate from hospitalised individuals & 0.001 per day \\
$\beta_{c,ji} = \beta_c$ & Transmission rate from critical individuals & 0.001 per day \\
$p$ & Fraction of exposed who become symptomatic & 0.75 \\
$\gamma_{e,i} = \gamma_e$ & rate at which exposed individuals turn infected & $(\tau_I)^{-1}$ \\
$\gamma_{a,i} = \gamma_a$ & rate at which asymptomatic individuals recover & $(\tau_a)^{-1}$   \\
$\gamma_{s,i} = \gamma_s$ & rate at which symptomatic individuals recover & $p_s / \tau_s$ \\
$\alpha_{s,1}$ & rate at which young symptomatic individuals need hospitalisation &  $1/\tau_s - \gamma_s$ \\
$\alpha_{s,2}$ & rate at which old symptomatic individuals need hospitalisation &  $2\alpha_{s,1}$ \\
$\alpha_{h,1}$ & rate at which young hospitalised individuals become critical & $(\tau_h)^{-1} \, p_h /(p_h + p_c)$  \\
$\alpha_{h,2}$ & rate at which old hospitalised individuals become critical & $ 2 \alpha_{h,1}$ \\
$\gamma_{h,i}$ & rate at which hospitalised individuals recover &  $1/\tau_h - \alpha_{h,i}$ \\
$\alpha_{c,1}$ & rate at which young critical individuals die & $(\tau_c)^{-1} f_c / p_c $ \\
$\alpha_{c,2}$ & rate at which old critical individuals die & $2 \alpha_{c,1}$ \\
$\gamma_{c,i}$ & rate at which critical individuals recover &  $ 1/\tau_c - \alpha_{c,i}$  \\
\hline
\end{tabular}
\end{center}
\label{basic-seir:params}
\end{table}%

\begin{table}[htp]
\caption{Relationship between empirical and model parameters}
\begin{center}
\begin{tabular}{|c|c|c|}
\hline
Average incubation time & $\tau_a = 1/\gamma_e$  & \\ \hline
Duration of mild symptomatic cases & $\tau_s = 1/(\gamma_s+\alpha_s)$  &  $\alpha_s = 1/\tau_s - \gamma_s$ \\
Fraction of mild symptomatic cases & $ p_s=\gamma_s / (\gamma_s + \alpha_s) = \gamma_s \tau_s $ & $\gamma_s = p_s / \tau_s$  \\  \hline
Duration of hospitalisation & $\tau_h = 1/(\gamma_h + \alpha_h)$  &  $\gamma_h = 1/\tau_h - \alpha_h$  \\
Fraction of hospitalisations \& critical cases &  $ p_c /(p_h + p_c) = \alpha_h / (\gamma_h + \alpha_h)  = \alpha_h \tau_h $  &  $\alpha_h = (\tau_h)^{-1} \, p_h /(p_h + p_c)$  \\  \hline
Duration of critical cases & $\tau_c = 1/(\gamma_c + \alpha_c)$ &   $\gamma_c = 1/\tau_c - \alpha_c$ \\
Fraction of severe cases &  $ f_c / p_c =  \alpha_c / (\alpha_c + \gamma_c) = \alpha_c \tau_c $  & $\alpha_c = (\tau_c)^{-1} f_c / p_c $\\
\hline
\end{tabular}
\end{center}
\label{basic-seir:params}
\end{table}%

\begin{table}[htp]
\caption{Age-structure data of different states based on model projections for 2021. The fractions have been rounded off to nearest second decimal place}
\begin{center}
\begin{tabular}{|c|c|c|c|}
\hline
State & Total population & Upto 60 years  & Above 60 years\\  \hline
Maharashtra & 11.9 crores  & 0.89 & 0.11\\
Kerala & 3.45 crores & 0.86 & 0.14 \\
Karnataka  & 6.56 crores & 0.9 & 0.1\\
Tamil Nadu & 7.52 crores & 0.88 & 0.12 \\ \hline
All-of-India & 133 crores & 0.91 & 0.09 \\
\hline
\end{tabular}
\end{center}
\label{age-str}
\end{table}

\FloatBarrier\printbibliography
\end{document}


\section{Verification}
\label{sec:prototype-semantics:verification}

For small state spaces, like the one in Figure~\ref{fig:prototype-semantics:statespace}, it is easy to verify properties manually by inspecting the graph.
In case of larger state spaces, reduction techniques can be applied to first reduce the labeled transition system that represents the state space before verifying properties manually.
The tool \ltsconvert, which is part of the \mCRLTwo toolset~\cite{mcrl2}, takes a labeled transition system in various formats as input and converts it to an equivalent labeled transition system in another format.
One of the formats that \ltsconvert is able to process is the \DOT format.
The tool is also capable of reducing labeled transition systems by means of an equivalence relation.
It supports several of these equivalence relations.

\begin{figure}[hbt]
  \centering
  \includegraphics[scale=.4]{prototype-semantics/figs/statespace_reduced}
  \caption{A reduced LTS visualized using \DOT}
  \label{fig:prototype-semantics:statespace_reduced}
\end{figure}

Figure~\ref{fig:prototype-semantics:statespace_reduced} shows the state space obtained after reduction has been applied to the state space in Figure~\ref{fig:prototype-semantics:statespace}.
In this case, branching bisimilarity~\cite{GlabWeijBisim96} was used as equivalence relation.
The labeled transition system has been reduced by, first, turning all labels except \SLCOSignal{S}{`a'} and \SLCOSignal{T}{`a'} to internal, unobservable labels and then removing all redundant states and transitions using \ltsconvert.
A similar procedure can be applied for the other reductions that are supported by this tool.

A number of the transformations described in Section~\ref{sec:slco:endogenous} transform \SLCO models into other \SLCO models with equivalent observable behavior.
By producing a labeled transition system representing the state space of the input and output model of such a transformation and reducing both labeled transition systems using the technique described above, the correctness of this transformation for the given input model can be verified by comparing the reduced labeled transition systems.
If both reductions lead to the same labeled transition system, the transformation has indeed preserved the observable~behavior.
We used this approach to assess the correctness of the transformations described in Sections~\ref{subsubsec:slco:sync2async}, \ref{subsubsec:slco:ll}, and~\ref{subsubsec:slco:merge} by applying these transformations to a number of models and comparing the state spaces of these models before and after transformation.

When labeled transition systems get too large for reduction and manual inspection, other tools can be used for verification.
One approach is converting labeled transition systems to the \BCG and \AUT file formats that are used by the \CADP toolset~\cite{DBLP:conf/tacas/GaravelLMS11} to represent labeled transition systems.
The \CADP toolset offers tools that take a labeled transition system and a temporal logic property as input and perform on-the-fly verification of the property on the labeled transition system. Alternatively, the previously mentioned \mCRLTwo toolset can be used for verification too.
This toolset includes a tool that can transform labeled transition systems to the proprietary format of the toolset as well as tools that can be used to analyze, simulate, manipulate, and visualize models described using this format.
These two example toolsets clearly show the added benefit of producing labeled transition systems from \SLCO models.
Transforming models to this common description format makes it possible to verify properties of models using existing tools, without additional effort.


\subsection{The strong duality theorem}


% Generated by GrindEQ Word-to-LaTeX 2008
% ========== UNREGISTERED! ========== Please register! ==========
% LaTeX/AMS-LaTeX

\documentclass[a4paper]{article}
\usepackage{anysize}
\marginsize{1cm}{1cm}{1cm}{1cm}

\usepackage{amssymb}
\usepackage{amsmath}
\usepackage[pdftex]{graphicx}
\usepackage{epsfig}
\usepackage{subfigure}
\usepackage{listings}
\lstset{language=haskell}
\lstset{commentstyle=\textit}
\lstset{mathescape=true}
%\lstset{labelstep=1}
%\lstset{backgroundcolor=,framerulecolor=}
\lstset{backgroundcolor=,rulecolor=}
\linespread{2.0}

\begin{document}

%%% remove comment delimiter ('%') and select language if required
%\selectlanguage{spanish}

\noindent
\section{Mutable Records}

\noindent Up until now we have defined a very general form of a heap with mutable references and we have given the first concrete implementation that allows us to write and run programs that use these constructs; thanks to monads and some syntactic sugar we even obtain code that looks and feels very stateful and imperative. We now move to studying a way to define and manipulate records (which are no more than heterogeneous lists plus some definition of labels instead of Church Numeral based access) within our system.

\noindent We start by giving a new predicate to characterize record types$ r$:

\begin{lstlisting}
Heap h ref $\Rightarrow$ Record ref h r
\end{lstlisting}

A record has a label type which takes as input an additional type parameter (the type of the selected field):

\begin{lstlisting}
Label ref h r : *$\to$ *
\end{lstlisting}

From a reference to a record and a label that selects a field of type $\alpha$ we can select a reference to a value of type $\alpha$ to a field inside the record:

\begin{lstlisting}
$\Leftarrow$  : ref h r $\to$ Label ref h r $\alpha$ $\to$ ref h $\alpha$
\end{lstlisting}

We can now say that our heterogeneous lists are records, and labels are a pair of functions similar to how references store a get and a set function:

\begin{lstlisting}
Label r $\alpha$ =Label (r$\to$ $\alpha$)  (r$\to$ $\alpha$ $\to$ r)
\end{lstlisting}

We also need a function to convert a Church Numeral into a label:

\begin{lstlisting}
CNum n $\wedge$ HList h $\Rightarrow$ labelAt : n $\to$ HAt h n
labelAt _ = Label ($\lambda$h.hRead h) (_:n($\lambda$h.$\lambda$v.hWrite v h (_:n)))
\end{lstlisting}

We can now instance the $Record$ predicate:

\begin{lstlisting}
Heap h ref $\wedge$ HList r $\Rightarrow$ Record ref h r
    Label ref h r $\alpha$ = Label r $\alpha$
    (Reference get set) $\Leftarrow$ (Label read write) =
            Reference (do r $\leftarrow$ get
                               return (read r))
                             ($\lambda$ v.do r $\leftarrow$ get
                                     set (write r v))
\end{lstlisting}

The selection operator simply does some packing and unpacking of the label into a reference.

\noindent At this point we could consider a simple example that shows how we can use records mutably:

\begin{lstlisting}
Person = String:String:Int:Nil
name : Label Person String
name = labelAt Z
surname : Label Person String
surname = labelAt (S Z)
age : Label Person Int
age = labelAt (S S Z)

$ex_{3}$ = do ``John'':''Smith'':27:Nil >>+ ($\lambda$p.
                 do (p$\Leftarrow$age) := 25
                    let n = p$\Leftarrow$surname
                    in n*=(++'' Jr''))
runST Nil $ex_3$
\end{lstlisting}

\noindent In the example above we declare a record type ($Person$) and we name its fields with labels. At this point we dynamically allocate an instance of our record, we manipulate it by reassigning the age to $25$ and adding the $"Jr."$ String to the surname.

\noindent

\end{document}

% == UNREGISTERED! == GrindEQ Word-to-LaTeX 2008 ==


\section{Goals and Objectives}
\label{sec:performance-modeling-goals-and-objectives}
\textit{Performance modeling} is the art of studying the behavior of a system in terms of performance metrics in order to maximize the return on tech investments.
To this aim, the \textit{simulation} is the most effective and scalable way to gather system insights and prove performance conjectures without even requiring the existence of the real system.

In this paper, we consider the system architecture in Figure~\ref{fig:system-architecture} and use a simulator to study its behavior with the goal of minimizing the response time.

The system is made of two following layers and accepts classed end-user workloads:

\begin{itemize}
		\item \textbf{Cloudlet:} upfront layer made of one-hop finite resources, having the ability to off-load tasks to the Cloud server, accordingly to an \textit{off-loading policy} based on the occupancy state of the Cloudlet. In particular, the Cloudlet may \textit{forward} incoming tasks to Cloud or \textit{restart} preempted tasks in Cloud with some \textit{overhead}.

		\item \textbf{Cloud:} backfront layer made of a remote Cloud server with virtually unlimited resources.
\end{itemize}

We assume the Cloudlet  providing tasks with higher service rates than the Cloud because, since the former is placed on the network edge, it has lower transmission time and it is less exposed to network latencies.

We propose to evaluate the system behavior with two alternative off-loading policies:

\begin{itemize}
	\item \textbf{Off-Loading Policy 1 (OP1):} the Cloudlet makes no distinction between tasks; it forwards incoming tasks to the Cloud when it has no available resources. Actually, OP1 represents the absence of an off-loading policy because in this case no off-loading is actually performed.

	\item \textbf{Off-Loading Policy 2 (OP2):} the Cloudlet gives higher priority to $1^{st}$ class tasks; it accepts only a dynamically thresholded occupancy of $2^{nd}$ class tasks, interrupts $2^{nd}$ class tasks to free up resources for $1^{st}$ class tasks and forwards incoming tasks to the Cloud when it has no available resources.
\end{itemize}

We propose to evaluate the convergence of the system to a steady state and the following performance metrics in the steady state with a $95\%$ level of confidence:

\begin{itemize}
	\item \textit{System/Cloudlet/Cloud response time} both global and per-class;

	\item \textit{System/Cloudlet/Cloud throughput} both global and per-class;

	\item \textit{System/Cloudlet/Cloud mean population} both global and per-class;

	\item \textit{Cloudlet interruption percentage} of the $2^{nd}$ class tasks;

	\item \textit{System response time of interrupted tasks}.
\end{itemize}

\begin{figure}
	\includegraphics[width=\columnwidth]{fig/architecture}
	\caption{System architecture.}
	\label{fig:system-architecture}
\end{figure}

It is worth studying such a system because it represents the simplest form of a Fog Computing infrastructure, where the end-user workload is  served by Edge compute nodes,  possibly off-loading it to a Cloud provider. The goal of such a study is then to asses the impact of distinct off-loading policies given the trade-off between (i) Edge compute nodes, which are barely exposed to network latencies, but hardly scale and (ii) Cloud compute nodes,  which are virtually infinite, but more exposed to network latencies.


\subsection{The Euclidian group}


    \filetitle{dat2dec}{Convert dates to their decimal representations}{dates/dat2dec}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
Dec = dat2dec(Dat)
Dec = dat2dec(Dat,Pos)
\end{verbatim}

\paragraph{Input arguments}\label{input-arguments}

\begin{itemize}
\item
  \texttt{Dat} {[} numeric {]} - IRIS serial date number.
\item
  \texttt{Pos} {[} \emph{\texttt{'start'}} \textbar{} \texttt{'centre'}
  \textbar{} \texttt{'end'} {]} - Point within the period that will
  represent the date; if omitted, \texttt{Pos} is set to
  \texttt{'start'}.
\end{itemize}

\paragraph{Output arguments}\label{output-arguments}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \texttt{Dec} {[} numeric {]} - Decimal number representing the input
  dates, computed as \texttt{year + (per-1)/freq}.
\end{itemize}

\paragraph{Description}\label{description}

\paragraph{Example}\label{example}