% JuliaCon proceedings template
\documentclass{juliacon}
\setcounter{page}{1}

\usepackage{hyperref}
% \usepackage[linesnumbered,ruled,vlined]{algorithm2e}
\usepackage{graphicx}
\usepackage[font=small,labelfont=bf]{caption}
\usepackage{subcaption}
% \usepackage{todonotes}
\usepackage{amsmath}
\usepackage{etoolbox}
% \usepackage{comment}
% \usepackage{minted}
\usepackage{listings}

% \setminted[Julia]{
%     fontfamily=fdm,
%     fontsize=\footnotesize,
%     escapeinside=,mathescape=true
%     }

\lstset{language=Julia,mathescape=true}

\newtoggle{anonymous}
% \toggletrue{anonymous}
\newcommand{\Zygote}{\iftoggle{anonymous}{$\partial$P.jl}{Zygote}}
\newcommand{\Zygoteplain}{\iftoggle{anonymous}{$\partial$P.jl}{Zygote}}

\setlength{\parindent}{2em}
\setlength{\parskip}{1em}

% Change link color
\hypersetup{colorlinks=true}

\begin{document}

\input{header}

\maketitle

\begin{abstract}
We describe a Differentiable Programming ($\partial P$) system that is able to take gradients of Julia programs making Automatic Differentiation a first class language feature. Our system supports almost all language constructs (control flow, recursion, mutation, etc.) and compiles high-performance code without requiring any user intervention or refactoring to stage computations. This enables an expressive programming model for deep learning and, more importantly, it enables users to utilize the existing Julia ecosystem of scientific computing packages in deep learning models.

\headingtable

\end{abstract}

\section{Introduction}

    At first glance, a casual practitioner might think that scientific computing and machine learning are different scientific disciplines. Modern machine learning has made its mark through breakthroughs in neural networks. Their applicability towards solving a large class of difficult problems in computer science has led to the design of new hardware and software to process extreme amounts of labelled training data, while simultaneously deploying trained models in devices. Scientific computing, in contrast, a discipline that is perhaps as old as computing itself, tends to use a broader set of modelling techniques arising out of the underlying physical phenomena. Compared to the typical machine learning researcher, many computational scientists works with smaller volumes of data but with more computational complexity and range. As we look closer, many similarities emerge. Both disciplines have a preference for using dynamic languages such as Python, R and Julia. Often, performance critical sections in Python and R are written in C++ and Fortran, less so in Julia. The core computational routines are grounded in numerical linear algebra, and fundamentally, hardware has been designed to accelerate this computational core.

    There are deep historical and intellectual links between machine learning and scientific computing.  On the surface, deep learning appears to be data driven and scientific computing is about very technical numerically intensive differential equations that mirror physical processes. It is our viewpoint that the fields are closer than is often realized with opportunities for machine learning and scientific computing to benefit from stronger interactions.

    Specifically, machine learning has strongly benefited from {\bf Numerical Linear Algebra} software. The optimized numerical linear algebra stack was first developed in the context of scientific computing - BLAS (Levels 1 through 3) kernels~\cite{blas}, LAPACK routines for computing matrix factorizations~\cite{lapack}, and MPI for message passing in parallel computing~\cite{mpi}. The early CUDA libraries were developed by Nvidia for accelerating scientific kernels on GPUs.

    This trend is continuing with more recent techniques such as {\bf Neural ODEs}. A neural ODE \cite{chen2018neural} is a neural network layer which compacts the $L$ layers of a ResNet into a single ODE definition solved in $N<L$ steps of an adaptive ODE solver. By changing to this form, the memory and computational costs for the residual network model are decreased.  Additionally, the continuous ODE formulation does not restrict the model to a grid like the traditional RNN, naturally allowing for fitting time series with irregular observations. Existing ODE solver software can then be employed for efficient solution of the system.

    This is a two-way street. The 2018 Gordon Bell prize~\cite{gordon-bell-2018-a}, awarded for the largest high-performance scientific computing application, applied deep learning to climate analytics at exascale. Scientific computing also has much to benefit from advances in machine learning:

    \begin{enumerate}
    \item{\bf Surrogate modeling:}  Scientific simulations are often expensive to run as they evaluate a system using first principles. These simulations can be accelerated by having machine learning models approximate the input-output relation. Neural networks or other surrogate models can be trained on expensive simulations once and then used repeatedly in place of the simulations, making it possible to explore the parameter space, propagate uncertainties, and fit the data in ways that have previously been impossible.

    \item{\bf Adjoint sensitivity analysis:} Calculating the adjoint of an ordinary differential equation system $\frac{du}{dt} = f(u,p,t)$ requires solving the reverse ODE $\frac{d\lambda^{*}}{dt} = \lambda^{*} \frac{df}{du} + \frac{df}{dp}$. The term $\lambda^{*} \frac{df}{du}$ is the primitive of backpropogation, and thus applying machine learning AD tooling to the ODE function $f$ accelerates the scientific computing adjoint calculations.

    \item{\bf Inverse problems:}  For many parameterized scientific simulations, one can ask "what parameters would make my model best fit the data?" This inverse problem is pervasive yet difficult because it requires the ability to efficiently compute the gradient of a large existing simulation. One can train a model on a simulator, which can then be used to quickly solve inverse problems, but this currently requires generating massive amounts of simulation data for training, which is slow and computationally expensive. By being able to differentiate through simulators, we can learn much more quickly and efficiently.

    \item{\bf Probabilistic Programming:} Inference on statistical models is a crucial tool in the sciences. Probabilistic programming
    enables more complex models and scaling to huge data sets by combining statistical methods with the generality of programming language constructs. Automatic differentiation is the backbone of many probabilistic programming tools, but domain specific languages lack access to an existing ecosystem of tools and packages. $\partial P$ in a general purpose language has the benefit of higher composability, access to better abstractions, and enabling richer and more accurate models. The Turing.jl~\cite{Turing.jl-2018} and Gen.jl~\cite{Gen.jl-2019} packages are excellent examples of these capabilities.

\end{enumerate}

    Differentiable Programming ($\partial P$) has the potential to be the lingua franca that can further unite the worlds of scientific computing and machine learning. The choice of a language to implement this system is an important one. Supporting multiple languages within a single $\partial P$ system causes an explosion in complexity, vastly increasing the developer effort required. Our $\partial P$ system extends the Julia programming language~\cite{Julia-2017-a} with differentiable programming capabilities. We chose the Julia language because of the abundance of pure-Julia packages for both machine learning and scientific computing allowing us to test our ideas on fairly large real-world applications.

    Our system can be directly used on existing Julia packages, handling  user-defined types, state-based control flow, and plentiful scalar operations through source-to-source AD. In this paper we briefly describe how we achieve our goals for a $\partial P$ system and showcase its ability to solve problems which mix machine learning and pre-existing scientific simulation packages.

\subsection{A simple {\tt sin} example: Differentiate Programs not Formulas}

We start out with a very simple example to differentiate $\sin(x)$ written as a program through its Taylor series:
$$\sin x =
x - \frac{x^3}{3!}
+\frac{x^5}{5!}-\ldots .$$

One feature of our example is that the number of terms will not be fixed, but will depend on x through a numerical convergence criterion.

To run, install Julia v1.1 or higher, and install the Zygote.jl and ForwardDiff.jl packages with:

\begin{lstlisting}
using Pkg
Pkg.add("Zygote")
Pkg.add("ForwardDiff")
using Zygote, ForwardDiff

function s(x)
    t = 0.0
    sign = -1.0
    for i in 1:19
        if isodd(i)
            newterm = x^i/factorial(i)
            abs(newterm)<1e-8 && return t
            println("i=",i)
            sign = -sign
            t += sign * newterm
        end
    end
    return t
end
\end{lstlisting}

While the Taylor series for sine could have been written more compactly in Julia, for purposes of illustrating more complex programs, we purposefully used a loop, a conditional, a print statement, and function calls to \verb+isodd+ and \verb+factorial+, which are native Julia implementations. AD just works, and that is the powerful part of the Julia approach. Let’s compute the gradient at x = 1.0 and check whether it matches \verb+cos(1.0)+:

\begin{lstlisting}{Julia}
julia> # Forward Mode AD

julia> ForwardDiff.derivative(s, 1.0)
i=1
i=3
i=5
i=7
i=9
i=11
0.540302303791887

julia> Zygote.gradient(s, 1.0) # Reverse Mode AD
i=1
i=3
i=5
i=7
i=9
i=11
(0.5403023037918872,)

julia> cos(1.0)
0.5403023058681398
\end{lstlisting}

\section{Implementation}
\label{Impl}

    Recent progress in tooling for automatic differentiation (AD) has been driven primarily by the machine learning community. Many state of the art reverse-mode AD tools such as Tracker.jl~\cite{Flux.jl-2018,2019sbc}, PyTorch~\cite{pytorch}, JAX~\cite{jax}, and TensorFlow~\cite{tensorflow} (in the recent Eager version) employ tracing methods to extract simplified program representations that are more easily amenable to AD transforms. These traces evaluate derivatives only at specific points in the program space. Unfortunately, this generally unrolls all control flow and requires compilation and optimization for every new input value.

    This choice has been driven largely by the fact that, as the JAX authors put it, ``ML workloads often consist of large, accelerable, pure-and-statically-composed (PSC) operations''~\cite{jax}. Indeed, for many ML models the per-executed-operation overhead (in both time and memory) incurred by tracing-based AD systems is immaterial, because these execution time and memory requirements of the operations dwarf any AD overhead.

    However, this assumption does not hold for many scientific inverse problems, or even the cutting edge of ML research. Instead, these problems require a $\partial P$ system capable of: (1) low overhead, independent of the size of the executed operation (2) Efficient support for control flow (3) Complete, efficient support for user defined data types (4) Customizability (5) Composability with existing code unaware of $\partial P$, and (6) Dynamism.

    Particularly, scientific programs tend to have adaptive algorithms, whose control flow depends on error estimates and thus the current state of the simulation, numerous scalar operations, define large nonlinear models using every term individually or implementing specialized numerical linear algebra routines, and pervasive use of user-defined data structures to describe model components, which require efficient memory handling (stack-allocation) in order for the problem to be computationally feasible.

    To take these kinds of problems, \Zygote\ does not utilize the standard methodology and instead generates a derivative function directly from the original source which is able to handle all input values. This is  called a source-to-source transformation, a methodology with a long history~\cite{baydin2018automatic} going back at least to the ADIFOR source-to-source AD program for FORTRAN 77~\cite{adifor}. Using this source-to-source formulation, \Zygote\ can then be compile, heavily optimize, and re-use a single gradient definition for all input values. Significantly, this transformation keeps control flow in tact: not unrolling loops to allow for all possible branches in a memory-efficient form. However, where prior source-to-source AD work has often focused on static languages, \Zygote\ expands upon this idea by supporting a full high level language, dynamic, Julia, in a way that allows for its existing scientific and machine learning package ecosystem to benefit from this tool.

\subsection{Generality, Flexibility, and Composability}

    One of the primary design decisions of a $\partial P$ system is how these capabilities should be exposed to the user. One convenient way to do so is using a differential operator $\mathcal{J}$ that operates on first class functions and once again returns a first class function (by returning a function we automatically obtain higher order derivatives, through repeated application of $\mathcal{J}$). There are several valid choices for this differential operator, but a convenient choice is

\[
    \mathcal{J}(f) := x \to (f(x), J_f(x)z),
\]

i.e. $\mathcal{J}(f)(x)$ returns the value of $f$ at x, as well as a function which evaluates the jacobian-vector product between $J_f(x)$ and some vector of sensitivities $z$. From this primitive we can define the gradient of a scalar function $g: \mathbb{R}^n \to \mathbb{R}$ which is written as:

\[
    \nabla g(x) := \left[\mathcal{J}(g)(x)\right]_2(1)
\]

    ($[]_2$ selects the second value of the tuple, $1 = \partial z / \partial z$ is the initial sensitivity).

    This choice of differential operator is convenient for several reasons: (1) The computation of the forward pass often computes values that can be re-used for the computation of the backwards pass. By combining the two operations, it is easy to re-use this work. (2) It can be used to recursively implement the chain rule (see figure \ref{chain}).

\begin{figure}
    \begin{minipage}{0.5\textwidth}
    \begin{lstlisting}[
    framesep=2mm,
    ]{Julia}
    function $\mathcal{J}$($f \circ g$)(x)
        $a$, $\underline{a}$ = $\mathcal{J}(f)(x)$
        $b$, $\underline{b}$ = $\mathcal{J}(g)(a)$
        $b$, $z$ $\to$ begin
            $\underline{a}$($\underline{b}$(z))
        end
    end
    \end{lstlisting}
    % \begin{verbatim}
    % function J(f . g)(x)
    %     a, da = J(f)(x)
    %     b, db = J(g)(a)
    %     b, z -> da(db(z))
    % end
    % \end{verbatim}
    \caption{The differential operator $\mathcal{J}$ is able to implement the chain rule through a local, syntactic recursive transformation.}\label{chain}
    \end{minipage}
    \hspace{0.1in}
    \begin{minipage}{0.5\textwidth}
    % \begin{lstlisting}
    % [
    % framesep=2mm,
    % fontsize=\footnotesize,
    % fontfamily=fdm,
    % xleftmargin=6pt,
    % escapeinside=,mathescape=true
    % ]{Julia}
    \begin{lstlisting}
    julia> f(x) = x^2 + 3x + 1
    julia> gradient(f, 1/3)
    (3.6666666666666665,)

    julia> using Measurements;
    julia> gradient(f, 1/3 +- 0.01)
    (3.6666666666666665 +- 0.02,)
    \end{lstlisting}
    \caption{With two minimal definitions, \Zygote\ is able to obtain derivatives of any function that only requires those definitions, even through custom data types (such as \textit{Measurement}) and many layers of abstraction.}\label{generalize}
    \end{minipage}
\end{figure}

    This second property also suggests the implementation strategy: hard code the operation of $\mathcal{J}$ on a set of primitive $f$'s and let the AD system generate the rest by repeated application of the chain rule transform. This same general approach has been implemented in many systems \cite{pearlmutter2008reverse,wang2018demystifying} and a  detailed description of how to perform this on Julia's SSA form IR is available in earlier work \cite{Zygote.jl-2018}.

    However, to achieve our extensibility and composability goals, we implement a slight twist on this scheme. We define a fully user extensible function $\partial$ that provides a default fallback as follows
\[
\partial(f)(args...) = \mathcal{J}(f)(args...),
\]
    where the implementation that is generated automatically by $\mathcal{J}$ recurses to $\partial$ rather than $\mathcal{J}$ and can thus easily be intercepted using Julia's standard multiple dispatch system at any level of the stack. For example, we might make the following definitions:

\[
\partial(f)(::\text{typeof}(+))(a::\text{IntOrFloat}, b::\text{IntOrFloat}) = a + b, z \to (z, z) \]\[
\partial(f)(::\text{typeof}(*))(a::\text{IntOrFloat}, b::\text{IntOrFloat}) = a * b, z \to (z * b, a * z)
\]

    i.e. declaring how to compute the partial derivative of $+$ and $*$ for two integer or float-valued numbers, but simultaneously leaving unconstrained the same for other functions or other types of  values (which will thus fall back to applying the AD transform). With these two definitions, any program that is ultimately just a composition of `+`, and `*` operations of real numbers will work. We show a simple example in figure \ref{generalize}. Here, we used the user-defined \textit{Measurement} type from the \textit{Measurements.jl} package~\cite{Measurements.jl-2016}. We did not have to define how to differentiate the $\wedge$ function or how to differentiate $+$ and $*$ on a \textit{Measurement}, nor did the \textit{Measurements.jl} package have to be aware of the AD system in order to be differentiated. This extra, user-extensible layer of indirection has a number of important consequences:

    \begin{itemize}
        \item \textbf{The AD system does not depend on, nor require any knowledge of primitives on new types.} By default we provide implementations of the differentiable operator for many common scalar mathematical and linear algebra operations, written with a scalar LLVM backend and BLAS-like linear algebra operations. This means that even when Julia builds an array type to target TPUs \cite{XLA.jl-2018}, its XLA IR primitives are able to be used and differentiated without fundamental modifications to our system.
    \item \textbf{Custom gradients become trivial.} Since all operations indirect through $\partial$, there is no difference between user-defined custom gradients and those provided by the system. They are written using the same mechanism, are co-optimized by the compiler and can be finely targeted using Julia's multiple dispatch mechanism.
\end{itemize}

Since Julia solves the two language problem, its Base, standard library, and package ecosystem are almost entirely pure Julia. Thus, since our $\partial P$ system does not require primitives to handle new types, this means that almost all functions and types defined throughout the language are automatically supported by \Zygote, and users can easily accelerate specific functions as they deem necessary.

\section{$\partial P$ in Practice}

A more extensive code listing for these examples is available at the following URL: \iftoggle{anonymous}{Hidden for review - an anonymized version is included as supplemental material}{https://github.com/MikeInnes/zygote-paper}.

\subsection{Deep Learning}
\label{dl}

\Zygote\ is a flexible backend for calculating gradients of deep learning models. A typical example is shown here, where a recurrent architecture using LSTMs \cite{Hochreiter:1997:lstm} is used to learn Shakespeare.
The code sample below demonstrates many powerful elements of \Zygote, making use of several convenient Julia features in the process.  First, the defined model has no special data types within it to enable AD; the models are defined for forward-pass calculation only, with backwards-pass definitions only for basic building blocks such as BLAS operations and basic array manipulation. \Zygote\ is used to wrap the loss computation, explicitly denoting the bounds of the computation that should be differentiated to calculate the gradients imposed upon the model, but all other pieces of code (including the LSTM layer definition itself) are written without automatic differentiation in mind. This model executes on the CPU, GPU~\cite{Flux.jl-2018} and Google TPU architecture~\cite{XLA.jl-2018}, with little to no change.

% \begin{verbatim}
\begin{lstlisting}{Julia}
alphabet, Xs, Ys = load_data("shakespeare_input.txt")

model = Chain(
LSTM(length(alphabet), 128),
LSTM(128, 128),
Dense(128, length(alphabet)),
softmax)

opt = ADAM(0.01)

# Run through our entire dataset a few times
for epoch_idx in 1:10,
  (x_batch, y_batch) in zip(Xs, Ys)

  # Calculate gradients upon the model for this
  # batch of data, summing crossentropy loss
  # across each time index in this batch.
  grads = \Zygoteplain.gradient(model) do model
    return sum(crossentropy.(model.(x_batch), y_batch))
  end

  # Update the model, then reset its internal
  # LSTM states
  model = update!(opt, model, grads)
  Flux.reset!(model)
end
\end{lstlisting}
% \end{verbatim}

\Zygote\ provides an extremely low-overhead AD interface. By performing source-to-source transformation, there is little runtime overhead from the AD transformation, beyond the actual cost of computing the backwards pass\footnote{In practice \Zygote\ may currently generate code that pessimizes certain compiler assumption and thus leads to slower execution. We are working on improving the compiler for these case to achieve true zero overhead.}. In addition, it has been shown to perform at the same level as TensorFlow for ResNet on a TPU pod~\cite{xla.jl}.

In order to measure this, we have benchmarked the backwards pass of a stacked LSTM network, measuring the runtime as batch size trends toward zero. We hypothesize that overall runtime should be linear in batch size, and so by linearly extrapolating our readings to a batch size of zero, we will be able to estimate the fixed overhead induced by each operation within the AD system, where an operation is a single adjoint definition.  We furthermore verify that this overhead is linear in the number of ops by measuring this for a variable number of stacked LSTMs and recording the number of ops per model.

Our benchmarks were run on an Intel(R) Core(TM) i5-6600 CPU @ 3.30GHz processor, running Linux with Julia version 1.3.  In all tests, multithreading was disabled both in the AD framework and in the underlying BLAS library, so as to avoid performance cliffs around multithreading thresholds. Our results are shown in Table \ref{table:zygote_overhead}, displaying an average overhead of \textbf{568.8 $ns$} per adjoint definition. This is a highly competitive as compared to other frameworks such as PyTorch, where $op$ overhead is at least 1 $\mu s$ \cite{PyTorchYear}.

\begin{table}[]
\centering
\begin{tabular}{cccc}
Layers & Total Overhead & Number of Ops & Overhead/op \\ \hline
1      & 147.0 us & 255 & 576.3 ns \\
2      & 280.5 us & 491 & 571.3 ns \\
3      & 406.1 us & 727 & 558.6 ns \\ \hline
\end{tabular}
\caption{\Zygote\ per-op overhead estimated across varying numbers of stacked LSTM layers.}
\label{table:zygote_overhead}
\end{table}

This vanishing overhead lowers the bar for AD systems to be integrated more thoroughly into programming languages at an extremely fine scale without worrying about performance issues. The lower the AD overhead, the smaller the minimum feasible kernel size for an AD system that is restricted by backwards pass efficiency.

\subsection{Differentiating a Trebuchet}

\begin{figure}[!htb]
  \centering\includegraphics[width=3.5in]{images/trebuchet-flow.png}
  \caption{{\bf Using a neural network surrogate to solve inverse problems}}
  \label{fig:trebuchet}
\end{figure}

Model-based reinforcement learning has advantages over model-agnostic methods, given that an effective agent must approximate the dynamics of its environment \cite{atkeson1997comparison}. However, model-based approaches have been hindered by the inability to incorporate realistic environmental models into deep learning models. Previous work has had success re-implementing physics engines using machine learning frameworks \cite{Degrave_2019,de2018end}, but this effort has a large engineering cost, has limitations compared to existing engines, and has limited applicability to other domains such as biology or meteorology.

 \Zygote\ can be used for  control problems, incorporating the model into backpropagation with one call to \texttt{gradient}. We pick trebuchet dynamics as a motivating example. Instead of aiming at a single target, we optimize a neural network that can aim it given any target. The neural net takes two inputs, the target distance in metres and the current wind speed. The network outputs trebuchet settings (the mass of the counterweight and the angle of release) that get fed into a simulator which solves an ODE and calculates the achieved distance. We then compare to our target, and backpropagate through the entire chain to adjust the weights of the network. Our dataset is a randomly chosen set of targets and wind speeds. An agent that aims a trebuchet to a given target can thus be trained in a few minutes on a laptop CPU, resulting in a network which solves the inverse problem with constant-time aiming that is $100\times$ faster than optimizing the trebuchet system directly (Figure~\ref{fig:trebuchet}). We present the code for this and other common reinforcement learning examples such as the cartpole and inverted pendulum~\cite{rlvsdp}.

\subsection{Computer Vision}

An emerging direction for computer vision is to see the problem as `inverse graphics': where vision models take pixel arrays to scene parameters, renderers take scene parameters to pixel arrays. Many high-fidelity, photo-realistic renderers are available which contain a vast amount of implicit knowledge about this mapping, and differentiation allows renderers and models to be used in an autoencoder-like setup to quickly bootstrap a full vision model.

As in other cases, the difficulty lies in getting efficient derivatives from a production-ready renderer, typically written in a performance language like C++. We repeat the going themes of this paper: ML framework-based reimplementations are typically limited compared to existing renderers (both by the available framework operations and by the years of work from rendering experts that has gone into production renderers), and workarounds such as Monte Carlo sampling \cite{li2018differentiable} impose a high efficiency cost (around $40\times$ slower than a single render) compared to AD (at most around ~$5\times$ due to division, in principle). The Julia community's approach is to build a renderer suitable, first and foremost, for traditional production rendering tasks, using a general and high-performance numerical language, and then differentiate it. Of course, this approach does not preclude the use of domain-specific methods such as Monte Carlo sampling where appropriate (section \ref{diffeq}).

In our examples we have used our prototype renderer~\cite{diffrender} to demonstrate optimization of the position of a point light source, given the desired final rendered image. We define a loss function that accepts a point light source as input, renders the scene, and compares it to a reference image. As usual, gradients are then trivial and we can begin to update the position of the point source.

\begin{lstlisting}{Julia}
guess = PointLight(Vec3(1.0),
                   20000.0,
                   Vec3(1.0, 2.0, -7.0))

function loss_function(light)
  rendered_color =
    raytrace(origin, direction, scene, light, eye_pos)
  rendered_img =
    process_image(rendered_color, screen_size.w,
                  screen_size.h)
  return mean((rendered_img .- reference_img).^2)
end

gs = gradient(x -> loss_function(x, image), guess)
\end{lstlisting}

\begin{figure}[!htb]
\minipage{0.32\textwidth}
  \includegraphics[width=\linewidth]{images/triangle_initial_guess.jpg}
  \caption{Initial Guess}\label{fig:raytracer1}
\endminipage\hfill
\minipage{0.32\textwidth}
  \includegraphics[width=\linewidth]{images/triangle_100.jpg}
  \caption{After 100 iterations}\label{fig:raytracer2}
\endminipage\hfill
\minipage{0.32\textwidth}%
  \includegraphics[width=\linewidth]{images/triangle_original.jpg}
  \caption{Target Image}\label{fig:raytracer3}
\endminipage
\end{figure}

\subsection{Financial Derivatives}
% NOTE: have to be careful with the word "derivative" when talking about finance, since it can have a very different meaning

Much of finance is concerned with how the value of a security will change according to changes in the market factors, such as interest rates, market indices, or spot prices of underlying assets. This is naturally expressed via the derivatives of security prices with respect to various conditions. Previous work has noted that financial contracts are compositional, and form a kind of programming language~\cite{jones2000composing,jones2003write}. Work in Julia has generalized this to a differentiable contract language, Miletus~\cite{Miletus.jl-2019}, where greeks can be derived for arbitrary contracts. Contract valuation can be mixed with free-form market simulations as well as other $\partial P$ tools like neural networks, enabling differentiable programming for finance.

In working with fixed-income securities such as bonds, the primary factors are interest rate curves. These are typically quoted as the \emph{par rate} at a set of times in the future (e.g. 1 month, 3 month, 6 month, 1 year, etc.), which are referred to as "key rates". Using this to determine the price of a bond requires (1) interpolating a yield curve from the par rates: this is a process known as \emph{bootstrapping}, and requires solving a sequence of implicit problems, typically via Newton iterations; and (2) computing the bond price from the yield curve by discounting cash flows.

The derivatives of a bond price with respect to the key rates are known as \emph{key rate durations}: computing these via AD requires higher-order derivatives (in order to differentiate through the Newton solver). However, this is entirely invisible to the programmer; several unrelated uses of differentiation, and even entirely different ADs, simply compose together and do the right thing.

\subsection{Quantum Machine Learning}
\label{yao}

On the one hand, a promising potential application and research direction for Noisy Intermediate-Scale Quantum (NISQ) technology \cite{preskill2018quantum} is \textit{variational quantum circuits} \cite{benedetti2019parameterized}, where a quantum circuit is parameterized by classical parameters in quantum gates, which may have less requirements on the hardware. Here, in many cases, the classical parameters are optimized with classical gradient-based algorithms. On the other hand, designing quantum circuits is hard, however, it is potentially an interesting direction to explore using gradient-based method to search  circuit architecture for certain task with AD support.

One such state of the art simulator is the Yao.jl~\cite{yao.jl} quantum simulator project. Yao.jl is implemented in Julia and thus composable with our AD technology. There are a number of interesting applications of this combination \cite{mitarai2018quantum, liu2018differentiable, liu2019variational}.

A subtle application is to perform traditional AD of the quantum simulator itself. As a simple example of this capability, we consider a Variational Quantum Eigensolver (VQE) \cite{kandala2017hardware}. A variational quantum eigensolver is used to compute the eigenvalue of some matrix $H$ (generally the Hamiltonian of some quantum system for which the eigenvalue problem is hard to solve classically, but that is easily encoded into quantum hardware). This is done by using a variational quantum circuit $\Phi(\theta)$ to prepare some quantum state $|\Psi\rangle = \Phi(\theta)|0\rangle$, measuring the expectation value $\langle \Psi|H|\Psi 0 \rangle$ and then using a classical optimizer to adjust $\theta$ to minimize the measured value. In our example, we will use a 4-site toy Hamiltonian corresponding to an anti-ferromagnetic Heisenberg chain:

\[
H = \frac{1}{4} \left[ \sum_{\langle i, j \rangle} \sigma_i^x \sigma_j^x + \sigma_i^y \sigma_j^y +  \sigma_i^z \sigma_j^z \right]
\]

We use a standard differentiable variational quantum circuit composed of layers (2 in our example) of (parameterized) rotations and CNOT entanglers with randomly initialized rotation angles. The corresponding code is showing in figure \ref{fig:yao_loss}  \footnote{For space efficiency, some details are factored into utility function omitted from this code listing. However, the code is very similar to the optimization loop from the deep learning example above. A full, expanded version of this example can be found with the full code listing in the supplemental material.}. The resulting plot can be seen in figure \ref{fig:yao_loss}.

\begin{figure}
    \begin{minipage}{0.48\textwidth}
    \begin{lstlisting}{Julia}
    using Yao, Zygote
    const nsites = 4
    let H = heisenberg(nsites),
        v0 = statevec(zero_state(nsites))
        energy(circuit) =
           ($\Psi$ = circuit*v0;
            real($\Psi'$ * H * $\Psi$))
    end
    circuit_init =
        random_diff_circuit(nsites, 2)
    optimize_plot_loss(
        energy, circuit_init, ADAM(0.1))
    \end{lstlisting}
  \caption{An ADAM optimizer is used to tune parameters of a variational quantum circuit to find the ground state of a 4-site anti-ferromagnetic Heisenberg chain Hamiltonian. The necessary gradients are obtained by automatic differentiation of a Yao.jl quantum simulator.}
\end{minipage}
\minipage{0.48\textwidth}
  \includegraphics[width=\linewidth]{images/yao_loss_plot.png}
  \caption{Optimization progress over steps of the classical optimizer to ground state.\label{fig:yao_loss}}
\endminipage
\end{figure}

\subsection{Neural Differential Equations with applications in finance}
\label{diffeq}

    Neural latent differential equations \cite{chen2018neural,pmlr-v5-alvarez09a,latent,DBLP:journals/corr/abs-1902-02376} incorporate a neural network into the ODE derivative function. Recent results have shown that many deep learning architectures can be compacted and generalized through neural ODE descriptions \cite{chen2018neural,He2016DeepRL,2019arXiv190401681D,DBLP:journals/corr/abs-1810-01367}. Latent differential equations have also seen use in time series extrapolation \cite{10.1093/bioinformatics/btn278} and model reduction \cite{ROMEROUGALDE2013170,8062736,2018arXiv180804930B,doi:10.1243/09544062JMES683}.

    Here we demonstrate financial time series forecasting with a neural stochastic differential equation (SDE). Typically, financial SDE models follow the form:

\[
    dX_t = f(X_t)dt + g(X_t)dW_t,
\]
    where
    $f:R^n \rightarrow R^n$ and $g:R^{n\times m} \rightarrow R^n$ with
    $W_t$ as the $m$-dimensional Wiener process. For example, the infamous Black-Scholes equation

\[
    \frac{\partial V}{\partial t} + \frac{1}{2}\sigma^2 S^2 \frac{\partial^2 V}{\partial S^2} + rS\frac{\partial V}{\partial S} - rV = 0
\]

    is related through the Feynman-Kac Theorem to a probability measure driven by a geometric Brownian motion stock price $dS_t = r S_t dt + \sigma S_t dW_t$, where $S$ is the stock price, $r$ is the fixed interest rate of an option, and $\sigma$ is the volatility. This signifies that the true value of an option contract can then be achieved by hedging such that the following conditional expectation holds:

\[
    V(S,t) = \mathbb{E} \left[\int_t^T e^{-\int_t^T rd\tau}f(S_\nu)d\nu + e^{-\int_t^T rd\tau}\psi(S_T)d\nu    \right]
\]
    where $\psi(S_\nu)$ is the value of the contract at the terminal time $T$ given a stock price $S$, showing that the PDE's solution is given by an average over the SDE solutions.

    To generalize the model, we replacing the fixed interest rate $r$ with a neural network train against financial time series data. Our financial simulator utilizes a high strong order adaptive integration provided by DifferentialEquations.jl \cite{DifferentialEquations.jl-2017,Rackauckas2017ADAPTIVEMF}.  Figure \ref{fig:neuralsde} depicts a two-dimensional neural SDE trained using the $l_2$ normed distance between the solution and the data points. Included is a forecast of the neural SDE solution beyond the data to a final time of 1.2, showcasing a potential use case.

\begin{figure}[!htb]
  \centering\includegraphics[width=3.5in]{images/neural_sde}
  \caption{{\bf Neural SDE Training}. For the SDE solution $X(t)$, the blue line shows $X_1(t)$ while the orange line shows $X_2(t)$. The green points shows the fitting data for $X_1$ while the purple points show the fitting data for $X_2$. The ribbons show the 95 percentile bounds of the stochastic solutions.}\label{fig:neuralsde}
\end{figure}

The analytical formula for the adjoint of the strong solution of a SDE is difficult to efficiently calculate due to the lack of classical differentiability of the solution\footnote{The the strong solution of stochastic differential equations are non-differentiable except on a measure zero set with probability 1. This means that standard Newtonian calculus cannot hold, and thus the derivation must take place in a the setting of a stochastic functional calculus known as the Malliavin Calculus \cite{doi:10.1137/S0363012902419059,huang2000malliavin,zhang_2004}.}. However, \Zygote\ still manages to calculate a useful derivative for optimization with respect to single solutions by treating the Brownian process as fixed and applying forward-mode automatic differentiation, showcasing \Zygote's ability to efficiently optimize its AD through mixed-mode approaches \cite{2018arXiv181201892R}. Common numerical techniques require computing the gradient with respect to a difference over thousands of trajectories to receive an average cost, while our numerical experiments suggest that it is sufficient with \Zygote\ to perform gradient decent on a neural SDE using single trajectories, reducing the overall computational cost by this thousands. This methodological advance combined with GPU-accelerated high order adaptive SDE integrators in DifferentialEquations.jl makes a whole new field of study accessible.

% \begin{comment}

% \subsection{Combining Probabilistic Programming with Photo-Realistic Computer Vision}
% \label{comp-vis}

% Here we show how to construct custom \texttt{UniformRGB} and \texttt{NormalRGB} distributions and their log PDFs. We combine this with the Turing probabilistic programming language, defining a generative model of an image where $\mu \sim \textnormal{UniformRGB}()$ and each $\textnormal{pixel} \sim \textnormal{NormalRGB}(\mu, \sigma)$.

% Aside from the correctness and usability benefits of having neural networks operate using types, rather than huge N-dimensional number-bags, there is an opportunity to take advantage of existing color science rather than re-learning from scratch. For example, work in natural language processing and colours typically assume a naive Euclidean loss in the RGB space; they can now use accurate perceptual losses or work in color spaces representing the various forms of color blindness. This is the essence of differentiable programming.

% %This methodology scales to full photo-realistic renderers.
% An emerging direction for computer vision is the notion of  `inverse graphics': where vision models take pixel arrays to scene parameters, renderers take scene parameters to pixel arrays. Many high-fidelity, photo-realistic renderers are available which contain a vast amount of implicit knowledge. Differentiation allows renderers and models to be used in an autoencoder-like setup to quickly bootstrap a full vision model.

% As in other cases, the difficulty lies in getting efficient derivatives from a production-ready renderer, typically written in a performance language like C++. We repeat the going themes of this paper: ML framework-based reimplementations are typically limited compared to existing renderers (both by the available framework operations and by the years of work from rendering experts that has gone into production renderers), and workarounds such as Monte Carlo sampling impose a high efficiency cost (around $40\times$ slower than a single render) compared to AD (at most around ~$5\times$ due to division, in principle). The Julia community's approach is to build a renderer suitable for traditional production rendering tasks, using a general and high-performance numerical language, and then differentiate it.

% \begin{lstlisting}[
%     frame=lines,
%     framesep=2mm,
%     fontsize=\footnotesize,
%     fontfamily=fdm,
%     xleftmargin=6pt,
%     escapeinside=,mathescape=true
%     ]{Julia}
% julia> guess = PointLight(Vec3(1.0), 20000.0, Vec3(1.0, 2.0, -7.0))

% julia> function loss_function(light)
%           rendered_color = raytrace(origin, direction, scene, light, eye_pos)
%           rendered_img = process_image(rendered_color, screen_size.w,
%                                         screen_size.h)
%           return mean((rendered_img .- reference_img).^2)
%       end

% julia> gs = gradient(x -> loss_function(x, image), guess)
% \end{lstlisting}

% In our examples we have used our prototype renderer to demonstrate optimization of the position of a point light source, given the desired final rendered image. We define a loss function that accepts a point light source as input, renders the scene, and compares it to a reference image. As usual, gradients are then trivial and we can begin to update the position of the point source.

% \begin{figure}[!htb]
% \minipage{0.32\textwidth}
%   \includegraphics[width=\linewidth]{images/triangle_initial_guess.jpg}
%   \caption{Initial Guess}\label{fig:raytracer1}
% \endminipage\hfill
% \minipage{0.32\textwidth}
%   \includegraphics[width=\linewidth]{images/triangle_100.jpg}
%   \caption{After 100 iterations}\label{fig:raytracer2}
% \endminipage\hfill
% \minipage{0.32\textwidth}%
%   \includegraphics[width=\linewidth]{images/triangle_original.jpg}
%   \caption{Target Image}\label{fig:raytracer3}
% \endminipage
% \end{figure}

% \end{comment}

\section{Conclusion}

The disciplines of machine learning and scientific computing have much to share. We presented a $\partial P$ system that can serve as the basis of a common shared infrastructure in both disciplines. We demonstrated how we  can compose ideas in machine learning and scientific computing to allow for new applications that transcend these domains, by using the same technology to differentiate programs in both domains. On the ML side, we show the same performance as existing ML frameworks for deep learning models (on CPUs, GPUs, and TPUs) and in reinforcement learning. In the case of scientific computing, we show neural SDEs and quantum machine learning. The system is open source, and we invite the reader to try their own examples.

\subsubsection*{Acknowledgments}

This work would not have been possible without the work of many people. We are particularly indebted to our users that contributed the examples in this paper.Thanks to Kai Xu for the Bayesian colours example, Avik Pal for building our differentiable ray tracer, Tejan Karmali and Neethu Mariya Joy for the model-based RL experiments, Roger Luo and JinGuo Liu for quantum experiments, and Simon Byrne for the finance example. Thanks also to Zenna Tavares, Jesse Bettencourt and Lyndon White for being early adopters of \Zygote\ and its underlying technology, and shaping its development. Much credit is also due to the core Julia language and compiler team for supporting \Zygote's development, including Jameson Nash, Jeff Bezanson and others. Thanks also to James Bradbury for helpful discussions on this paper and many other things.

\bibliographystyle{juliacon}
\bibliography{ref}

\end{document}

% Inspired by the International Journal of Computer Applications template


\section{Non-equilibrium Green function}

\begin{framenologo}
  \frametitle{Non-equilibrium Green function}
  \tableofcontents[currentsection]
\end{framenologo}

\subsection{Variables}

\begin{frame}
  \frametitle{Non-equilibrium Green function}
  \framesubtitle{Variables}

  \begin{block}<+->{Important variables in Green function techniques}

    \begin{tabular}[c]{>{$}l<{$}|l}
      \idxE & electrode index
      \\
      \HH_\kk & Hamiltonian
      \\
      \SO_\kk & Overlap, for orthogonal basis sets equals $\mathbf I$
      \\
      \DM_\kk & Density matrix
      \\
      \SE_\kk(\E)& Self-energy (not necessarily an electrode!)
      \\
      \Scat_{\idxE,\kk}(\E) & Scattering matrix from $\idxE$
      \\
      \G_\kk(\E) & Green function
      \\
      \Spec_{\idxE,\kk}(\E) & Spectral function originating from $\idxE$
      \\
      \T_{\idxE\mto\idxE'}(\E) & Transmission function from $\idxE$ to $\idxE'$
      \\
      \T_\idxE(\E) & Total transmission function out of $\idxE$
    \end{tabular}

  \end{block}

  \begin{block}<+->{Basic equations for Green function techniques}
    \vskip-2ex
\input{../equations.tex}
  \end{block}

\end{frame}

\subsection{Density of states}

\begin{frame}[label=DOS]
  \frametitle{Non-equilibrium Green function}
  \framesubtitle{Density of states}

  \begin{itemize}
    \item Density of states over all orbitals
    \begin{align*}
      \mathrm{DOS}(\E) &= \Tr[\DM(\E) \SO]
      \\
      \mathrm{DOS}(\E) &= -\frac1\pi\Im\Tr[\G(\E) \SO]
      \\
      \mathrm{ADOS}(\E) &= \frac1{2\pi}\Re\Tr[\Spec_\idxE(\E) \SO]
      \\
      \mathrm{DOS}(\E) &=\sum_\idxE \mathrm{ADOS}(\E) + \text{bound states}
    \end{align*}

    \vspace{-2ex}

    \item Local density of states on orbital $\nu$
    \begin{align*}
      \mathrm{DOS}_\nu(\E) &= [\DM(\E) \SO]_{\nu,\nu}
      \\
      \mathrm{DOS}_\nu(\E) &= -\frac1\pi\Im[\G(\E) \SO]_{\nu,\nu}
      \\
      \mathrm{ADOS}_\nu(\E) &= \frac1{2\pi}\Re[\Spec_\idxE(\E) \SO]_{\nu,\nu}
      \\
      \mathrm{DOS}_\nu(\E) &=\sum_\idxE \mathrm{ADOS}_\nu(\E) + \text{bound states}_\nu
    \end{align*}

    \vspace{-2ex}

    \item<2->%
    The overlap matrix is extremely important when calculating the density of states!

    \item<3->%
    $\SE$ broadens the DOS similarly to a large $\eta$ value, for states
    coupling to the electrodes

  \end{itemize}

  \only<2>{\hfill \hyperlink{integration<3>}{\beamergotobutton{Go to Rules of
              integration}}}

\end{frame}

\subsection{Transmission}

\begin{frame}
  \frametitle{Non-equilibrium Green function}
  \framesubtitle{Transmission}

  \begin{block}<+->{Transmission}
    \begin{itemize}
      \item Transmission between any two electrodes $\T_{\idxE\mto\idxE'}$

      \item Total transmission out of any electrode $\T_\idxE$

      \item Reflection into electrode $\RE_\idxE$, note $\mathbf 1$ is the \emph{bulk}
      transmission (open channels)

    \end{itemize}
    \vskip -2ex
    \begin{align*}
      \T_{\idxE\mto\idxE'}(\E) &=
      \Tr\big[\Scat_{\idxE'}(\E)\G(\E)\Scat_{\idxE}(\E)\G^\dagger(\E)\big]\quad\text{, for $\idxE\neq\idxE'$}
      \\
      \T_{\idxE}(\E) &\equiv\sum_{\idxE'\neq\idxE}\T_{\idxE\mto\idxE'} (\E)
      \\
      \RE_\idxE(\E) &=\mathbf 1 -\T_\idxE (\E)
      = \mathbf 1 -\Big\{
      \im \Tr\big[(\G(\E)-\G^\dagger(\E))\Scat_\idxE(\E)\big]
      -\Tr[\Scat_\idxE(\E) \G(\E)\Scat_\idxE(\E)\G^\dagger(\E)]
      \Big\}
    \end{align*}

  \end{block}

  \begin{block}<+->{Symmetries}
    When time-reversal symmetry applies we have:
    \begin{equation*}
      \T_{\idxE\mto\idxE',\kk} = \T_{\idxE'\mto\idxE,\kk} = \T_{\idxE\mto\idxE',-\kk}
    \end{equation*}
  \end{block}

\end{frame}

\begin{frame}
  \frametitle{Non-equilibrium Green function}
  \framesubtitle{Local currents/bond-currents}

  \begin{block}<+->{Orbital/Bond currents}
    Bond currents are \emph{local} currents flowing between two orbitals/bonds and are
    associated with \emph{out-going} states from electrode $\idxE$
    \begin{equation*}
      \JJ_{\idxE,\nu\mu} = \frac e h \Im\big[
      \Spec_{\idxE,\nu\mu}(\HH_{\mu\nu} - \E\SO_{\mu\nu})
      -
      \Spec_{\idxE,\mu\nu}(\HH_{\nu\mu} - \E\SO_{\nu\mu})\big]
    \end{equation*}

    The sum of bond-currents crossing \emph{any} device cross-section is equal to the
    current between the leads. Without the prefactor $e/h$ this also applies for the
    bond-transmissions\footnote{TBtrans calculates bond-\emph{transmissions}, although
        they are named \emph{currents}.}.

    The bond-currents obey Kirchoff's laws.

  \end{block}

  \begin{block}<+->{Symmetries}
    \emph{NOTE}, even when time-reversal symmetry is present
    \begin{align*}
      \JJ_{\idxE,\kk,\nu\mu} &= - \JJ_{\idxE,\kk,\mu\nu}
      \\
      \JJ_{\idxE,\kk,\nu\mu} &\neq \JJ_{\idxE,-\kk,\nu\mu}
    \end{align*}
    hence when calculating bond-currents for periodic structures it is imperative to
    average the \emph{entire} Brillouin zone. This may be understood from \emph{momentum}
    considerations.
  \end{block}

\end{frame}

\begin{frame}
  \frametitle{Bond-current in periodic calculations}
  \framesubtitle{Importance of Brillouin-zone averaging}

  \vskip -6em
  \begin{center}
    \begin{itemize}
      \item Square lattice
      \item Periodicity along $x$ direction
      \item Transport along $y$ direction
      \item Bond-currents $k$-averaged
    \end{itemize}
  \end{center}

  \begin{columns}

    \column{.5\linewidth}
    \incg[width=.99\linewidth]{trs}

    \begin{itemize}
      \item Sampled $k_x\in \big[0;\frac12\big]$
    \end{itemize}

    \column{.5\linewidth}
    \incg[width=.99\linewidth]{no_trs}

    \begin{itemize}
      \item Sampled $k_x\in \big]-\frac12;\frac12\big]$
    \end{itemize}

  \end{columns}

\end{frame}

\begin{frame}
  \frametitle{Non-equilibrium Green function}
  \footnotesize

  \vskip -2ex
  \begin{columns}

    \column{.5\textwidth}
    \begin{block}{Basic equations}
      \vskip-2ex
\input{../equations.tex}
    \end{block}

    \column{.5\textwidth}
    \begin{block}{Local Density of States}
      \vskip-2ex
      \begin{align*}
        \rho_{\nu}(\E) &= -\frac1\pi\Im[\G(\E) \SO]_{\nu} = \sum_\idxE\rho_\nu^\idxE(\E) +
        \text{bound states}
        \\
        \rho_{\nu}^{\idxE}(\E) &= \frac1{2\pi}\Re[\Spec_\idxE(\E) \SO]_{\nu}
      \end{align*}
    \end{block}

  \end{columns}

  \begin{block}{Transmission}
    \vskip-2ex

    \begin{align*}
      \T_{\idxE\mto\idxE'}(\E) &=
      \Tr\big[\Scat_{\idxE'}(\E)\G(\E)\Scat_{\idxE}(\E)\G^\dagger(\E)\big]\quad\text{, for $\idxE\neq\idxE'$}
      \\
      \T_{\idxE}(\E) &\equiv\sum_{\idxE'\neq\idxE}\T_{\idxE\mto\idxE'} (\E)
      \\
      \RE_\idxE(\E) & %=\mathbf 1 -\T_\idxE (\E)
      = \mathbf 1 -\Big\{
      \im \Tr\big[(\G(\E)-\G^\dagger(\E))\Scat_\idxE(\E)\big]
      -\Tr[\Scat_\idxE(\E) \G(\E)\Scat_\idxE(\E)\G^\dagger(\E)]
      \Big\}
      \\
      I_{\idxE\mto\idxE'} &= \frac{e^2}{h}\iint\cd\E\dd\kk\, \T_{\idxE\mto\idxE'}(\E)[n_{F,\idxE}(\E) - n_{F,\idxE'}(\E)].
    \end{align*}

  \end{block}

  \begin{block}{Orbital current}

    \begin{equation*}
      \JJ_{\idxE,\nu\mu} = \frac e h \Im\big[
      \Spec_{\idxE,\nu\mu}(\HH_{\mu\nu} - \E\SO_{\mu\nu})
      -
      \Spec_{\idxE,\mu\nu}(\HH_{\nu\mu} - \E\SO_{\nu\mu})\big]
    \end{equation*}

  \end{block}

  \begin{center}
    Quantities depend on $\int_\BZ$, i.e. required $k$ points are typically \emph{very}
    high for correct energy resolved physical quantities.

    Related to the band-structure, a fine $k$-grid is necessary to correctly reproduce the
    band-structure.
  \end{center}

\end{frame}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "talk"
%%% End:


\sekshun{Ranges}
\label{Ranges}
\index{ranges}

A \emph{range} is a first-class, constant-space representation of a
regular sequence of integer indices.
Ranges support iteration over the sequences they represent
and are the basis for defining domains (\rsec{Domains}).

Ranges are presented as follows:
\begin{itemize}
\item definition of the key range concepts \rsec{Range_Concepts}
\item range types \rsec{Range_Types}
\item range values \rsec{Range_Values}
\item range assignment \rsec{Range_Assignment}
\item operators on ranges \rsec{Range_Operators}
\item predefined functions on ranges \rsec{Predefined_Range_Functions}
\end{itemize}

\section{Range Concepts}
\label{Range_Concepts}
\index{ranges!concepts}

\index{ranges!represented sequence}
\index{ranges!sequence}
A range has four primary properties. Together they define the sequence
of indices that the range represents, or the \emph{represented sequence},
as follows.

\begin{itemize}

\index{ranges!low bound}
\item The \emph{low bound} is either an integer or -$\infty$.

\index{ranges!high bound}
\item The \emph{high bound} is either an integer or +$\infty$.
  The low and high bounds determine the span of the represented sequence.
%
  Chapel does not represent $\infty$ explicitly. Instead, infinite
  bound(s) are represented implicitly in the range's type
  (\rsec{Range_Types}).
  When the low and/or high bound is $\infty$, the represented sequence
  is unbounded in the corresponding direction(s).

\index{ranges!stride}
\item The \emph{stride} is a non-zero integer.
  It defines the distance between any two adjacent members of the
  represented sequence.
  The sign of the stride indicates the direction of the sequence:
  \begin{itemize}
  \item[$\bullet$] $stride > 0$ indicates an increasing sequence,
  \item[$\bullet$] $stride < 0$ indicates a decreasing sequence.
  \end{itemize}

\index{ranges!alignment}
\index{ranges!alignment!ambiguous}
\item The \emph{alignment} is either an integer or is \emph{ambiguous}.
  It defines how the represented sequence's members are aligned relative to 0.
%REVIEW: vass: please put a better definition of "alignment" above
% if you can make one.
% (BTW see also the definition of "aligned integer" below.)
  For a range with a stride other than 1 or -1,
  ambiguous alignment means that the represented sequence is undefined.
  In such a case, certain operations discussed later result in an error.

\begin{openissue}
We consider disallowing ambiguous alignment for ranges whose both bounds
are integers (not $\infty$), in order to enable more efficient
implementation.
\end{openissue}

\end{itemize}

More formally, the represented sequence for the range
$(low, high, stride, alignmt)$
contains all indices $ix$ such that:

\begin{tabular}{ll}

$low \leq ix \leq high$ and $ix \equiv alignmt \pmod{|stride|}$ &
 if $alignmt$ is not ambiguous \\
$low \leq ix \leq high$ &
 if $stride = 1$ or $stride = -1$ \\
the represented sequence is undefined &
 otherwise

\end{tabular}

\index{ranges!represented sequence!increasing}
\index{ranges!represented sequence!decreasing}
The sequence, if defined, is increasing if $stride > 0$ and decreasing if $stride < 0$.

\index{ranges!empty}
If the represented sequence is defined but
there are no indices satisfying the applicable equation(s) above,
the range and its represented sequence are \emph{empty}.

\index{ranges!aligned integer}
We will say that an integer $ix$ is \emph{aligned}
w.r.t. the range $(low, high, stride, alignmt)$
if:
\begin{itemize}
\item $alignmt$ is not ambiguous and $ix \equiv alignmt \pmod{|stride|}$, or
\item $stride$ is 1 or -1.
\end{itemize}
\noindent Furthermore, $\infty$ is never aligned.

Ranges have the following additional properties.
\begin{itemize}

\index{ranges!alignment!ambiguous}
\item A range is \emph{ambiguously aligned} if
  \begin{itemize}
  \item its alignment is ambiguous, and
  \item its stride is neither 1 nor -1.
  \end{itemize}

\index{ranges!first index}
\item The \emph{first index} is the first member of the represented sequence.

  A range \emph{has no} first index when the first member is undefined,
  that is, in the following cases:
  \begin{itemize}
  \item the range is ambiguously aligned,
  \item the represented sequence is empty,
  \item the represented sequence is increasing and the low bound is -$\infty$,
  \item the represented sequence is decreasing and the high bound is +$\infty$.
  \end{itemize}

\index{ranges!last index}
\item The \emph{last index} is the last member of the represented sequence.

  A range \emph{has no} last index when the last member is undefined,
  that is, in the following cases:
  \begin{itemize}
  \item it is ambiguously aligned,
  \item the represented sequence is empty,
  \item the represented sequence is increasing and the high bound is +$\infty$,
  \item the represented sequence is decreasing and the low bound is -$\infty$.
  \end{itemize}

\index{ranges!aligned low bound}
\item The \emph{aligned low bound} is the smallest integer that is
  greater than or equal to the low bound and is aligned w.r.t. the range,
  if such an integer exists.
% For now we are allowing an empty range to have an aligned low bound.

  The aligned low bound equals the smallest member of the represented
  sequence, when both exist.

\index{ranges!aligned high bound}
\item The \emph{aligned high bound} is the largest integer that is
  less than or equal to the high bound and is aligned w.r.t. the range,
  if such an integer exists.
% For now we are allowing an empty range to have an aligned high bound.

  The aligned high bound equals the largest member of the represented
  sequence, when both exist.

% Can re-introduce this if needed, but need to discuss the definition.
% The intention is that a range is "naturally aligned" IFF
% its alignment does not need to be printed out.
%
%A range is \emph{naturally aligned} if it has a positive stride and its
%alignment coincides with its low bound, or it has a negative stride and its
%alignment coincides with its high bound.
%Specifically, a range with \chpl{low}, \chpl{high}, \chpl{stride}
%and \chpl{alignment} values of $l$, $h$, $s$ and $a$ (respectively) is naturally aligned if:
%\begin{itemize}
%\item It is \chpl{bounded}, its stride is positive, and $a = l ({\rm modulo} s)$,
%\item It is \chpl{bounded}, its stride is negative, and $a = h ({\rm modulo} s)$,
%\item It is \chpl{boundedLow} and $a = l ({\rm modulo} s)$,
%\item It is \chpl{boundedHigh} and $a = h ({\rm modulo} s)$ or
%\item It is $boundedNone$ and $a = 0$.
%\end{itemize}
%\noindent The \chpl{isNaturallyAligned} predicate returns \chpl{true} when the operand
%range is naturally aligned.  When a range is formatted for output, the alignment
%is printed only if the range is not naturally aligned.
%\index{ranges!alignment!natural}

\index{ranges!iterable}
\item The range is \emph{iterable}, that is, it is legal to iterate over it,
  if is has the first index.

\end{itemize}

%REVIEW: any other properties to include in the above?

\section{Range Types}
\label{Range_Types}
\index{ranges!types}
\index{types!range}

The type of a range is characterized by three parameters:
\begin{itemize}

\index{ranges!idxType}
\index{ranges!stride type}
\item \chpl{idxType} is the type of the indices of the range's
  represented sequence. However, when the range's low and/or high
  bound is $\infty$, the represented sequence also contains indices
  that are not representable by \chpl{idxType}.

  \chpl{idxType} must be an integral type and is \chpl{int} by default.
%
  The range's low bound and high bound (when they are not $\infty$)
  and alignment are of the type \chpl{idxType}. The range's stride
  is of the signed integer type that has the same bit size as \chpl{idxType}.

\index{ranges!boundedType}
\item \chpl{boundedType} indicates which of the range's bounds are not $\infty$.
  \chpl{boundedType} is an enumeration constant of the type
  \chpl{BoundedRangeType}. It is discussed further below.

\index{ranges!stridable}
\item \chpl{stridable} is a boolean that determines whether the range's stride
  can take on values other than 1.
  \chpl{stridable} is \chpl{false} by default.
  A range is called \emph{stridable}
  if its type's \chpl{stridable} is \chpl{true}.

\end{itemize}

\chpl{boundedType} is one of the constants of the following type:

\begin{chapel}
enum BoundedRangeType { bounded, boundedLow, boundedHigh, boundedNone };
\end{chapel}

The value of \chpl{boundedType} determines which bounds of the range
are integers (making the range ``bounded'', as opposed to infinite,
in the corresponding direction(s))
as follows:

\begin{itemize}

\index{ranges!bounded}
\item \chpl{bounded}:
  both bounds are integers.

\index{ranges!boundedLow}
\item \chpl{boundedLow}:
  the low bound is an integer (the high bound is +$\infty$).

\index{ranges!boundedHigh}
\item \chpl{boundedHigh}:
  the high bound is an integer (the low bound is -$\infty$).

\index{ranges!boundedNone}
\item \chpl{boundedNone}:
  neither bound is an integer (both bounds are $\infty$).

\end{itemize}

\noindent \chpl{boundedType} is \chpl{BoundedRangeType.bounded} by default.

The parameters \chpl{idxType}, \chpl{boundedType} and \chpl{stridable}
affect all values of the corresponding range type.
For example, the range's low bound is -$\infty$ if and only if
the \chpl{boundedType} of that range's type is either \chpl{boundedHigh}
or \chpl{boundedNone}.

\begin{rationale}
Providing \chpl{boundedType} and \chpl{stridable} in a range's type
allows the compiler to identify the more common cases
where the range is \chpl{bounded} and/or its stride is 1.
The compiler can also detect user and library code that is
specialized to these cases.
As a result, the compiler has the opportunity to optimize these
cases and the specialized code more aggressively.
\end{rationale}

%REVIEW: vass: we need to come up with a common way to define
% generic types prior to the generics chapter.

% gotta specify syntax for 'range-type' because it's referenced in Types.tex
A range type has the following syntax:
\begin{syntax}
range-type:
  `range' ( named-expression-list )
\end{syntax}

That is, a range type is obtained as if by invoking the range type constructor
(\rsec{Type_Constructors}) that has the following header:

\begin{chapel}
  proc range(type idxType = int,
             param boundedType = BoundedRangeType.bounded,
             param stridable = false) type
\end{chapel}

As a special case, the keyword \chpl{range} without a parenthesized
argument list refers to the range type with the default values
of all its parameters, i.e.,
\chpl{range(int, BoundedRangeType.bounded, false)}.

\begin{chapelexample}{rangeVariable.chpl}
The following declaration declares a variable \chpl{r}
that can represent ranges of 32-bit integers,
with both high and low bounds specified,
and the ability to have a stride other than 1.
\begin{chapel}
var r: range(int(32), BoundedRangeType.bounded, stridable=true);
\end{chapel}
\begin{chapelpost}
writeln(r);
var i32: int(32) = 3;
r = i32..13 by 3 align 1;
writeln(r);
\end{chapelpost}
\begin{chapeloutput}
1..0
3..13 by 3 align 1
\end{chapeloutput}
\end{chapelexample}

\section{Range Values}
\label{Range_Values}
\index{ranges!values}

A range value consists of the range's four primary properties
(\rsec{Range_Concepts}):
low bound, high bound, stride and alignment.

\subsection{Range Literals}
\label{Range_Literals}
\index{ranges!literals}

Range literals are specified with the following syntax.

\begin{syntax}
range-literal:
  expression .. expression
  expression ..
  .. expression
  ..
\end{syntax}

The expressions to the left and to the right of \chpl{..}, when given,
are called the low bound and the high bound expression, respectively.

The type of a range literal is a range with the following parameters:

\begin{itemize}

\item \chpl{idxType} is determined as follows:
  \begin{itemize}

  \item If both the low bound and the high bound expressions are given
        and have the same integral type, then \chpl{idxType} is that type.

%REVIEW: not precise. E.g. it does not allow one bound to be converted
% to another bound's type. Also, should be the most specific such type.
  \item If both the low bound and the high bound expressions are given
        and an implicit conversion is allowed from each expression's type
        to the same integral type, then \chpl{idxType} is that
        integral type.

%REVIEW: should be the most specific integral type.
  \item If only one bound expression is given and it has an integral type
        or an implicit conversion is allowed from that expression's type
        to an integral type, then \chpl{idxType} is that integral type.

  \item If neither bound expression is given, then \chpl{idxType} is
        \chpl{int}.

  \item Otherwise, the range literal is not legal.
  \end{itemize}

\item \chpl{boundedType} is a value of the type \chpl{BoundedRangeType}
that is determined as follows:
  \begin{itemize}

  \item \chpl{bounded}, if both the low bound and the high bound expressions
         are given,

  \item \chpl{boundedLow}, if only the high bound expression is given,

  \item \chpl{boundedHigh}, if only the low bound expression is given,

  \item \chpl{boundedNone}, if neither bound expression is given.
  \end{itemize}

\item \chpl{stridable} is \chpl{false}.

\end{itemize}

The value of a range literal is as follows:

\begin{itemize}

\item The low bound is given by the low bound expression,
if present, and is -$\infty$ otherwise.

\item The high bound is given by the upper bound expression,
if present, and is +$\infty$ otherwise.

\item The stride is 1.

\item The alignment is ambiguous.

\end{itemize}

\subsection{Default Values}
\label{Range_Default_Values}
\index{ranges!default values}

The default value for a range type depends on the type's
\chpl{boundedType} parameter as follows:

\begin{itemize}

\item \chpl{1..0} (an empty range) if \chpl{boundedType} is \chpl{bounded}

\item \chpl{1..} if \chpl{boundedType} is \chpl{boundedLow}

\item \chpl{..0} if \chpl{boundedType} is \chpl{boundedHigh}

\item \chpl{..} if \chpl{boundedType} is \chpl{boundedNone}

\end{itemize}

\begin{rationale}
We use 0 and 1 to represent an empty range because these values
are available for any \chpl{idxType}.

We have not found the natural choice of the default value for
\chpl{boundedLow} and \chpl{boundedHigh} ranges.
The values indicated above are distinguished by the following property.
Slicing the default value for a \chpl{boundedLow} range with
the default value for a \chpl{boundedHigh} range (or visa versa)
produces an empty range, matching the default value for a
\chpl{bounded} range
\end{rationale}

\section{Common Operations}
\label{Ranges_Common_Operations}
\index{ranges!operations}

%REVIEW: vass: this should really be implicit conversion
% and apply uniformly, like e.g. numeric implicit conversions.
%If needed, define also explicit range conversions.
\subsection{Range Assignment}
\label{Range_Assignment}
\index{ranges!assignment}

Assigning one range to another results in the target range
copying the low and high bounds, stride, and alignment
from the source range.

Range assignment is legal when:
\begin{itemize}
\item An implicit conversion is allowed from \chpl{idxType} of the source range
       to \chpl{idxType} of the destination range type,
\item the two range types have the same \chpl{boundedType}, and
\item either the destination range is stridable or the source range's
      stride is 1.
\end{itemize}

\subsection{Range Comparisons}
\label{Range_Comparisons}
\index{ranges!comparisons}

Ranges can be compared using equality and inequality.

\begin{protohead}
proc ==(r1: range(?), r2: range(?)): bool
\end{protohead}
\begin{protobody}
Returns \chpl{true} if the two ranges have the same represented sequence
or the same four primary properties,
and \chpl{false} otherwise.
%REVIEW: present examples of equal ranges whose low/high bounds differ
% or one is ambiguously aligned (with low>high) and the other is not.
\end{protobody}

\subsection{Iterating over Ranges}
\label{Iterating_over_Ranges}
\index{ranges!iteration}

A range can be used as an iterator expression in a loop. This is legal
only if the range is iterable. In this case the loop iterates over the
members of the range's represented sequence, in the order defined by
the sequence. If the range is empty, no iterations are executed.

\subsubsection{Iterating over Unbounded Ranges in Zippered Iterations}
\label{Iterating_over_Unbounded_Ranges_in_Zippered_Iterations}
\index{ranges!iteration!zippered}

When a range with the first index but without the last index is used
in a zippered iteration (~\rsec{Zipper_Iteration}),
it generates as many indices as needed
to match the other iterator(s).

\begin{chapelexample}{zipWithUnbounded.chpl}
The code
\begin{chapel}
for i in zip(1..5, 3..) do
  write(i, "; ");
\end{chapel}
\begin{chapelpost}
writeln();
\end{chapelpost}
produces the output
\begin{chapelprintoutput}{}
(1, 3); (2, 4); (3, 5); (4, 6); (5, 7);
\end{chapelprintoutput}
\end{chapelexample}

%REVIEW: perhaps define implicit/explicit conversions as well
% under "Common Operations"?

\section{Range Operators}
\label{Range_Operators}
\index{ranges!operators}

The following operators can be applied to range
expressions and are described in this section: stride (\chpl{by}),
alignment (\chpl{align}), count (\chpl{\#}) and slicing (\chpl{\(\)}
or \chpl{\[\]}).
Chapel also defines a set of functions that operate on ranges.
They are described in \rsec{Predefined_Range_Functions}.

\begin{syntax}
range-expression:
  expression
  strided-range-expression
  counted-range-expression
  aligned-range-expression
  sliced-range-expression
\end{syntax}

\subsection{By Operator}
\label{By_Operator_For_Ranges}
\index{ranges!strided}
\index{by (range)@\chpl{by} (range)}
\index{operators!by (range)@\chpl{by} (range)}
\index{ranges!by operator@\chpl{by} operator}

The \chpl{by} operator selects a subsequence of the range's represented sequence,
optionally reversing its direction.
The operator takes two arguments, a base range and an integral step.
It produces a new range whose represented sequence contains
each $|$step$|$-th element of the base range's represented sequence.
The operator reverses the direction of the represented sequence if step$<$0.
If the resulting sequence is increasing,
it starts at the base range's aligned low bound, if it exists.
If the resulting sequence is decreasing,
it starts at the base range's aligned high bound, if it exists.
%
Otherwise, the base range's alignment is used to determine
which members of the represented sequence to retain.
%
If the base range's represented sequence is undefined,
the resulting sequence is undefined, too.

The syntax of the \chpl{by} operator is:
\begin{syntax}
strided-range-expression:
  range-expression `by' step-expression

step-expression:
  expression
\end{syntax}

The type of the step must be a signed or unsigned integer of the same
bit size as the base range's \chpl{idxType}, or an implicit conversion must be allowed
to that type from the step's type.
It is an error for the step to be zero.

\begin{future}
We may consider allowing the step to be of any integer type,
for maximum flexibility.
\end{future}

The type of the result of the \chpl{by} operator is the type of the
base range, but with the \chpl{stridable} parameter set to \chpl{true}.

Formally, the result of the \chpl{by} operator is a range with the following
primary properties:

\begin{itemize}

\item The low and upper bounds are the same as those of the base range.

\item The stride is the product of the base range's stride
      and the step, cast to the base range's stride type before multiplying.

\item The alignment is:

  \begin{itemize}

  \item the aligned low bound of the base range, if such exists
        and the stride is positive;

  \item the aligned high bound of the base range, if such exists
        and the stride is negative;

  \item the same as that of the base range, otherwise.

  \end{itemize}

\end{itemize}

\begin{chapelexample}{rangeByOperator.chpl}
In the following declarations, range \chpl{r1} represents the odd integers
between 1 and 20. Range \chpl{r2} strides \chpl{r1} by two and represents
every other odd integer between 1 and 20: 1, 5, 9, 13 and 17.
\begin{chapel}
var r1 = 1..20 by 2;
var r2 = r1 by 2;
\end{chapel}
\begin{chapelpost}
writeln(r1);
writeln(r2);
\end{chapelpost}
\begin{chapeloutput}
1..20 by 2
1..20 by 4
\end{chapeloutput}
\end{chapelexample}

%REVIEW we were not happy with this rationale at 7/19/2011 spec review
\begin{rationale}
{\it Why isn't the high bound specified first if the stride is
negative?}  The reason for this choice is that the \chpl{by} operator
is binary, not ternary.  Given a range \chpl{R} initialized
to \chpl{1..3}, we want \chpl{R by -1} to contain the ordered sequence
$3,2,1$.  But then \chpl{R by -1} would be different from \chpl{3..1
by -1} even though it should be identical by substituting the value in
R into the expression.
\end{rationale}

\subsection{Align Operator}
\label{Align_Operator}
\index{ranges!align}
\index{align@\chpl{align}}
\index{operator!align@\chpl{align}}

The \chpl{align} operator can be applied to any range, and creates a new range
with the given alignment.

The syntax for the \chpl{align} operator is:
\begin{syntax}
aligned-range-expression:
  range-expression `align' expression
\end{syntax}
\noindent The type of the resulting range expression is the same as that of the
range appearing as the left operand.  An implicit conversion from
the type of the right operand to the index type of the operand range
must be allowed.
The resulting range has the
same low and high bounds and stride as the source range. The
alignment equals the \chpl{align} operator's right operand
and therefore is not ambiguous.

\begin{chapelexample}{alignedStride.chpl}
\begin{chapel}
var r1 = 0 .. 10 by 3 align 0;
for i in r1 do
  write(" ", i);			// Produces "0 3 6 9".
writeln();

var r2 = 0 .. 10 by 3 align 1;
for i in r2 do
  write(" ", i);			// Produces "1 4 7 10".
writeln();
\end{chapel}
\begin{chapeloutput}
 0 3 6 9
 1 4 7 10
\end{chapeloutput}
\end{chapelexample}

When the stride is negative, the same indices are printed in reverse:
\begin{chapelexample}{alignedNegStride.chpl}
\begin{chapel}
var r3 = 0 .. 10 by -3 align 0;
for i in r3 do
  write(" ", i);			// Produces "9 6 3 0".
writeln();

var r4 = 0 .. 10 by -3 align 1;
for i in r4 do
  write(" ", i);			// Produces "10 7 4 1".
writeln();
\end{chapel}
\begin{chapeloutput}
 9 6 3 0
 10 7 4 1
\end{chapeloutput}
\end{chapelexample}

To create a range aligned relative to its \chpl{first} index, use
the \chpl{offset} method (\rsec{Range_Offset_Method}).

\subsection{Count Operator}
\label{Count_Operator}
\index{ranges!count operator}
\index{ranges!#@\chpl{#}}
\index{operators!# (range)@\chpl{#} (range)}

The \chpl{#} operator takes a range and an integral count and creates a new
range containing the specified number of indices.  The low or high bound of the
left operand is preserved, and the other bound adjusted to provide the specified
number of indices.  If the count is positive, indices are taken from the start
of the range; if the count is negative, indices are taken from the end of the
range.  The count must be less than or equal to the \chpl{length} of the range.

\begin{syntax}
counted-range-expression:
  range-expression # expression
\end{syntax}

The type of the count expression must be a signed or unsigned integer
of the same bit size as the base range's \chpl{idxType}, or an
implicit conversion must be allowed to that type from the count's
type.

The type of the result of the \chpl{#} operator is the type of the
range argument.

Depending on the sign of the count and the stride, the high or low bound is
unchanged and the other bound is adjusted so that it is $c * stride - 1$ units
away.  Specifically:
\begin{itemize}
\item If the count times the stride is positive, the low bound is preserved
and the high bound is adjusted to be one less than the low bound plus that
product.
\item Iff the count times the stride is negative, the high bound is preserved
and the low bound is adjusted to be one greater than the high bound plus that
product.
\end{itemize}

\begin{rationale}
Following the principle of preserving as much information from the original
range as possible, we must still choose the other bound so that
exactly \emph{count} indices lie within the range.  Making the two bounds lie
$count * stride - 1$ apart will achieve this, regardless of the current
alignment of the range.

This choice also has the nice symmetry that the alignment can be adjusted
without knowing the bounds of the original range, and the same number of indices
will be produced:
\begin{chapel}
r # 4 align 0   // Contains four indices.
r # 4 align 1   // Contains four indices.
r # 4 align 2   // Contains four indices.
r # 4 align 3   // Contains four indices.
\end{chapel}
\end{rationale}

It is an error to apply the count operator with a positive count to a range that
has no first index.  It is also an error to apply the count operator
with a negative count to a range that has no last index.
It is an error to apply the count operator to a range that is ambiguously aligned.
%REVIEW: vass: does not have to be an error -
% just produce a range whose alignment is ambiguous.

\begin{chapelexample}{rangeCountOperator.chpl}
The following declarations result in equivalent ranges.
\begin{chapel}
var r1 = 1..10 by -2 # -3;
var r2 = ..6 by -2 # 3;
var r3 = -6..6 by -2 # 3;
var r4 = 1..#6 by -2;
\end{chapel}
\begin{chapelpost}
writeln(r1 == r2 \&\& r2 == r3 \&\& r3 == r4);
writeln((r1, r2, r3, r4));
\end{chapelpost}
\begin{chapeloutput}
true
(1..6 by -2, 1..6 by -2, 1..6 by -2, 1..6 by -2)
\end{chapeloutput}
Each of these ranges represents the ordered set of three indices: 6, 4, 2.
\end{chapelexample}

\subsection{Arithmetic Operators}
\label{Range_Arithmetic}
\index{ranges!arithmetic operators}
\index{operators!arithmetic!range}

The following arithmetic operators are defined on ranges and integral
types:

\begin{chapel}
proc +(r: range, s: integral): range
proc +(s: integral, r: range): range
proc -(r: range, s: integral): range
\end{chapel}

The \chpl{+} and \chpl{-} operators apply the scalar via the operator
to the range's low and high bounds, producing a shifted version of the
range.  If the operand range is unbounded above or below, the missing bounds
are ignored.
The index type of the resulting range is the type of the value
that would result from an addition between the scalar value and a value
with the range's index type.  The bounded and stridable parameters for
the result range are the same as for the input range.

The stride of the resulting range is the same as the stride of the
original. The alignment of the resulting range is shifted by the same amount as
the high and low bounds.  It is permissible to apply the shift operators to a
range that is ambiguously aligned.  In that case, the resulting range is also
ambiguously aligned.

\begin{chapelexample}{rangeAdd.chpl}
The following code creates a bounded, non-stridable range \chpl{r}
which has an index type of \chpl{int} representing the indices ${0, 1, 2, 3}$.
It then uses the \chpl{+} operator to create a second range \chpl{r2}
representing the indices ${1, 2, 3, 4}$.  The \chpl{r2} range is bounded,
non-stridable, and is represented by indices of type \chpl{int}.
% REVIEW: bradc: More interesting example?
\begin{chapel}
var r = 0..3;
var r2 = r + 1;    // 1..4
\end{chapel}
\begin{chapelpost}
writeln((r, r2));
\end{chapelpost}
\begin{chapeloutput}
(0..3, 1..4)
\end{chapeloutput}
\end{chapelexample}

\subsection{Range Slicing}
\label{Range_Slicing}
\index{ranges!slicing}

Ranges can be \emph{sliced} using other ranges to create new
sub-ranges.  The resulting range represents the intersection between
the two ranges' represented sequences.  The stride and alignment of the resulting range are adjusted as
needed to make this true.
\chpl{idxType} and the sign of the stride of the result are determined
by the first operand.

Range slicing is specified by the syntax:
\begin{syntax}
sliced-range-expression:
  range-expression ( range-expression )
  range-expression [ range-expression ]
\end{syntax}

If either of the operand ranges is ambiguously aligned, then the resulting range
is also ambiguously aligned.  In this case, the result is valid only if the
strides of the operand ranges are relatively prime.  Otherwise, an error is
generated at run time.

\begin{rationale}
If the strides of the two operand ranges are relatively prime, then they are
guaranteed to have some elements in their intersection, regardless whether their
relative alignment can be determined.  In that case, the bounds and stride in the resulting
range are valid with respect to the given inputs.
The alignment can be supplied later to create a valid range.

If the strides are not relatively prime, then the result of the slicing
operation would be completely ambiguous.  The only reasonable action for the
implementation is to generate an error.
\end{rationale}

If the resulting represented sequence cannot be expressed as a range of the
expected type, an error is generated. This can happen, for example, when
the operands represent all odd and all even numbers, or when the first
operand is an unbounded range with unsigned \chpl{idxType} and the second
operand represents only negative numbers.

\begin{chapelexample}{rangeSlicing.chpl}
In the following example, \chpl{r} represents the integers from 1 to
20 inclusive.  Ranges \chpl{r2} and \chpl{r3} are defined using range
slices and represent the indices from 3 to 20 and the odd integers
between 1 and 20 respectively. Range \chpl{r4} represents the odd
integers between 1 and 20 that are also divisible by 3.
%REVIEW: bradc: Give <l,h,s,a> values?
\begin{chapel}
var r = 1..20;
var r2 = r[3..];
var r3 = r[1.. by 2];
var r4 = r3[0.. by 3];
\end{chapel}
\begin{chapelpost}
writeln((r, r2, r3, r4));
\end{chapelpost}
\begin{chapeloutput}
(1..20, 3..20, 1..20 by 2, 1..20 by 6 align 3)
\end{chapeloutput}
\end{chapelexample}

\section{Predefined Functions on Ranges}
\label{Predefined_Range_Functions}
\index{ranges!predefined functions}

\subsection{Range Type Parameters}
\label{Range_Type_Accessors}
\index{ranges!type accessors}

\index{ranges!boundedType@\chpl{boundedType}}
\index{predefined functions!boundedType@\chpl{boundedType}}
\begin{protohead}
proc $range$.boundedType : BoundedRangeType
\end{protohead}
\begin{protobody}
Returns the \chpl{boundedType} parameter of the range's type.
\end{protobody}

\index{ranges!idxType@\chpl{idxType}}
\index{predefined functions!idxType@\chpl{idxType}}
\begin{protohead}
proc $range$.idxType : type
\end{protohead}
\begin{protobody}
Returns the \chpl{idxType} parameter of the range's type.
\end{protobody}

\index{ranges!stridable@\chpl{stridable}}
\index{predefined functions!stridable@\chpl{stridable}}
\begin{protohead}
proc $range$.stridable : bool
\end{protohead}
\begin{protobody}
Returns the \chpl{stridable} parameter of the range's type.
\end{protobody}

\subsection{Range Properties}
\label{Range_Properties}
\index{ranges!properties}

Most of the methods in this subsection report on
the range properties defined in \rsec{Range_Concepts}.
A range's represented sequence can be examined, for example,
by iterating over the range in a for loop \rsec{The_For_Loop}.

\begin{openissue}
The behavior of the methods that report properties that may be
undefined, $\infty$, or ambiguous, may change.
\end{openissue}

\index{ranges!aligned@\chpl{aligned}}
\index{predefined functions!aligned@\chpl{aligned}}
\begin{protohead}
proc $range$.aligned : bool
\end{protohead}
%REVIEW: better name?
\begin{protobody}
Reports whether the range's alignment is \emph{not} ambiguous.
\end{protobody}

\index{ranges!alignedHigh@\chpl{alignedHigh}}
\index{predefined functions!alignedHigh@\chpl{alignedHigh}}
\begin{protohead}
proc $range$.alignedHigh : idxType
\end{protohead}
\begin{protobody}
Returns the range's aligned high bound.
If the aligned high bound is undefined (does not exist),
the behavior is undefined.
\end{protobody}
\begin{chapelexample}{alignedHigh.chpl}
The following code:
\begin{chapel}
var r = 0..20 by 3;
writeln(r.alignedHigh);
\end{chapel}
produces the output
\begin{chapelprintoutput}{}
18
\end{chapelprintoutput}
\end{chapelexample}

\index{ranges!alignedLow@\chpl{alignedLow}}
\index{predefined functions!alignedLow@\chpl{alignedLow}}
\begin{protohead}
proc $range$.alignedLow : idxType
\end{protohead}
\begin{protobody}
Returns the range's aligned low bound.
If the aligned low bound is undefined (does not exist),
the behavior is undefined.
\end{protobody}

\index{ranges!alignment@\chpl{alignment}}
\index{predefined functions!alignment@\chpl{alignment}}
\begin{protohead}
proc $range$.alignment : idxType
\end{protohead}
\begin{protobody}
Returns the range's alignment.
If the alignment is ambiguous, the behavior is undefined.
See also \chpl{aligned}.
\end{protobody}

\index{ranges!first@\chpl{first}}
\index{predefined functions!first@\chpl{first}}
\begin{protohead}
proc $range$.first : idxType
\end{protohead}
\begin{protobody}
Returns the range's first index.
If the range has no first index, the behavior is undefined.
See also \chpl{hasFirst}.
\end{protobody}

\index{ranges!hasFirst@\chpl{hasFirst}}
\index{predefined functions!hasFirst@\chpl{hasFirst}}
\begin{protohead}
proc $range$.hasFirst(): bool
\end{protohead}
\begin{protobody}
Reports whether the range has the first index.
\end{protobody}

\index{ranges!hasHighBound@\chpl{hasHighBound}}
\index{predefined functions!hasHighBound@\chpl{hasHighBound}}
\begin{protohead}
proc $range$.hasHighBound() param: bool
\end{protohead}
\begin{protobody}
Reports whether the range's high bound is \emph{not} +$\infty$.
\end{protobody}

\index{ranges!hasLast@\chpl{hasLast}}
\index{predefined functions!hasLast@\chpl{hasLast}}
\begin{protohead}
proc $range$.hasLast(): bool
\end{protohead}
\begin{protobody}
Reports whether the range has the last index.
\end{protobody}

\index{ranges!hasLowBound@\chpl{hasLowBound}}
\index{predefined functions!hasLowBound@\chpl{hasLowBound}}
\begin{protohead}
proc $range$.hasLowBound() param: bool
\end{protohead}
\begin{protobody}
Reports whether the range's low bound is \emph{not} -$\infty$.
\end{protobody}

\index{ranges!high@\chpl{high}}
\index{predefined functions!high@\chpl{high}}
\begin{protohead}
proc $range$.high : idxType
\end{protohead}
\begin{protobody}
Returns the range's high bound.
If the high bound is +$\infty$, the behavior is undefined.
See also \chpl{hasHighBound}.
\end{protobody}

% Can re-introduce this if needed, but need to define natural alignment first.
%\begin{protohead}
%proc $range$.isNaturallyAligned(): bool
%\end{protohead}
%\begin{protobody}
%Reports whether the range is naturally aligned.
%\end{protobody}

\index{ranges!isAmbiguous@\chpl{isAmbiguous}}
\index{predefined functions!isAmbiguous@\chpl{isAmbiguous}}
\begin{protohead}
proc $range$.isAmbiguous(): bool
\end{protohead}
\begin{protobody}
Reports whether the range is ambiguously aligned.
\end{protobody}

\index{ranges!last@\chpl{last}}
\index{predefined functions!last@\chpl{last}}
\begin{protohead}
proc $range$.last : idxType
\end{protohead}
\begin{protobody}
Returns the range's last index.
If the range has no last index, the behavior is undefined.
See also \chpl{hasLast}.
\end{protobody}

\index{ranges!length@\chpl{length}}
\index{predefined functions!length@\chpl{length}}
\begin{protohead}
proc $range$.length : idxType
\end{protohead}
\begin{protobody}
Returns the number of indices in the range's represented sequence.
If the represented sequence is infinite or is undefined,
an error is generated.
\end{protobody}

\index{ranges!low@\chpl{low}}
\index{predefined functions!low@\chpl{low}}
\begin{protohead}
proc $range$.low : idxType
\end{protohead}
\begin{protobody}
Returns the range's low bound.
If the low bound is -$\infty$, the behavior is undefined.
See also \chpl{hasLowBound}.
\end{protobody}

\index{ranges!size@\chpl{size}}
\index{predefined functions!size@\chpl{size}}
\begin{protohead}
proc $range$.size : idxType
\end{protohead}
\begin{protobody}
Same as $range$.length.
\end{protobody}

\index{ranges!stride@\chpl{stride}}
\index{predefined functions!stride@\chpl{stride}}
\begin{protohead}
proc $range$.stride : int(numBits(idxType))
\end{protohead}
\begin{protobody}
Returns the range's stride. This will never return 0.
If the range is not stridable, this will always return 1.
\end{protobody}

\subsection{Other Queries}
\label{Range_Queries}
\index{ranges!other queries}

\index{ranges!boundsCheck@\chpl{boundsCheck}}
\index{predefined functions!boundsCheck@\chpl{boundsCheck}}
\begin{protohead}
proc $range$.boundsCheck(r2: range(?)): bool
\end{protohead}
\begin{protobody}
Returns \chpl{false} if either range is ambiguously aligned.
Returns \chpl{true} if range \chpl{r2} lies entirely within this range
and \chpl{false} otherwise.
\end{protobody}

\index{ranges!ident@\chpl{ident}}
\index{predefined functions!ident@\chpl{ident}}
\begin{protohead}
proc ident(r1: range(?), r2: range(?)): bool
\end{protohead}
\begin{protobody}
Returns \chpl{true} if the two ranges are the same in every respect: i.e. the
two ranges have the same \chpl{idxType},
\chpl{boundedType}, \chpl{stridable}, \chpl{low}, \chpl{high}, \chpl{stride} and
\chpl{alignment} values.
%REVIEW: bradc: Might be nice if only the field properties (and not the type params) were
%compared?  Types could be compared separately using '=='?
\end{protobody}

\index{ranges!indexOrder@\chpl{indexOrder}}
\index{predefined functions!indexOrder@\chpl{indexOrder}}
\begin{protohead}
proc $range$.indexOrder(i: idxType): idxType
\end{protohead}
\begin{protobody}
If \chpl{i} is a member of the range's represented sequence, returns an integer giving
the ordinal index of \chpl{i} within the sequence using 0-based indexing.
Otherwise, returns \chpl{(-1):idxType}.
It is an error to invoke \chpl{indexOrder} if the represented sequence
is not defined or the range does not have the first index.
\end{protobody}

\begin{example}
The following calls show the order of index 4 in each of the given
ranges:
\begin{chapel}
(0..10).indexOrder(4) == 4
(1..10).indexOrder(4) == 3
(3..5).indexOrder(4) == 1
(0..10 by 2).indexOrder(4) == 2
(3..5 by 2).indexOrder(4) == -1
\end{chapel}
\end{example}

\index{ranges!member@\chpl{member}}
\index{predefined functions!member@\chpl{member}}
\begin{protohead}
proc $range$.member(i: idxType): bool
\end{protohead}
\begin{protobody}
Returns \chpl{true} if the range's represented sequence
contains \chpl{i}, \chpl{false} otherwise.
It is an error to invoke \chpl{member} if the represented sequence
is not defined.
\end{protobody}

\index{ranges!member@\chpl{member}}
\index{predefined functions!member@\chpl{member}}
\begin{protohead}
proc $range$.member(other: range): bool
\end{protohead}
\begin{protobody}
Reports whether \chpl{other} is a subrange of the receiver. That is,
if the represented sequences of the receiver and \chpl{other}
are defined and the receiver's sequence contains all members of the
\chpl{other}'s sequence.
%REVIEW: bradc: How different than boundsCheck?
\end{protobody}

\subsection{Range Transformations}
\label{Range_Transformations}
\index{ranges!transformations}

\index{ranges!alignHigh@\chpl{alignHigh}}
\index{predefined functions!alignHigh (range)@\chpl{alignHigh} (range)}
\begin{protohead}
proc $range$.alignHigh()
\end{protohead}
\begin{protobody}
Sets the high bound of this range to its aligned high bound, if it is defined.
Generates an error otherwise.
\end{protobody}

\index{ranges!alignLow@\chpl{alignLow}}
\index{predefined functions!alignLow (range)@\chpl{alignLow} (range)}
\begin{protohead}
proc $range$.alignLow()
\end{protohead}
\begin{protobody}
Sets the low bound of this range to its aligned low bound, if it is defined.
Generates an error otherwise.
\end{protobody}

\index{ranges!expand@\chpl{expand}}
\index{predefined functions!expand (range)@\chpl{expand}}
\begin{protohead}
proc $range$.expand(i: idxType)
\end{protohead}
\begin{protobody}
Returns a new range whose bounds are extended by $i$ units on each end.  If $i <
0$ then the resulting range is contracted by its absolute value.  In symbols,
given that the operand range is represented by the tuple $(l,h,s,a)$, the result
is $(l-i,h+i,s,a)$.  The stride and alignment of the original range are preserved.
If the operand range is ambiguously aligned, then so is the resulting range.
\end{protobody}

\index{ranges!exterior@\chpl{exterior}}
\index{predefined functions!exterior (range)@\chpl{exterior} (range)}
\begin{protohead}
proc $range$.exterior(i: idxType)
\end{protohead}
\begin{protobody}
Returns a new range containing the indices just outside the low or high bound of
the range (low if $i < 0$ and high otherwise).  The stride and alignment of the
original range are preserved.  Let the operand range
be denoted by the tuple $(l,h,s,a)$.  Then:
\begin{itemize}
\item[] if $i < 0$, the result is $(l+i,l-1,s,a)$,
\item[] if $i > 0$, the result is $(h+1,h+i,s,a)$, and
\item[] if $i = 0$, the result is $(l,h,s,a)$.
\end{itemize}
If the operand range is ambiguously aligned, then so is the resulting range.
\end{protobody}

\index{ranges!interior@\chpl{interior}}
\index{predefined functions!interior (range)@\chpl{interior} (range)}
\begin{protohead}
proc $range$.interior(i: idxType)
\end{protohead}
\begin{protobody}
Returns a new range containing the indices just inside the low or high bound of
the range (low if $i < 0$ and high otherwise).  The stride and alignment of the
original range are preserved.  Let the operand range
be denoted by the tuple $(l,h,s,a)$.  Then:
\begin{itemize}
\item[] if $i < 0$, the result is $(l,l-(i-1),s,a)$,
\item[] if $i > 0$, the result is $(h-(i-1),h,s,a)$, and
\item[] if $i = 0$, the result is $(l,h,s,a)$.
\end{itemize}
This differs from the behavior of the count operator, in that \chpl{interior()}
preserves the alignment, and it uses the low and high bounds rather
than \chpl{first} and \chpl{last} to establish the bounds of the resulting range.
If the operand range is ambiguously aligned, then so is the resulting range.
\end{protobody}

\index{ranges!offset@\chpl{offset}}
\index{predefined functions!offset (range)@\chpl{offset} (range)}
\begin{protohead}
proc $range$.offset(n: idxType)
\end{protohead}
\label{Range_Offset_Method}
\begin{protobody}
Returns a new range whose alignment is this range's first index plus
\chpl{n}. The new alignment, therefore, is not ambiguous.
If the range has no first index, a run-time error is generated.
\end{protobody}

\index{ranges!translate@\chpl{translate}}
\index{predefined functions!translate (range)@\chpl{translate} (range)}
\begin{protohead}
proc $range$.translate(i: integral)
\end{protohead}
\begin{protobody}
Returns a new range with its \chpl{low}, \chpl{high} and \chpl{alignment} values
adjusted by $i$.  The \chpl{stride} value is preserved.
If the range's alignment is ambiguous, the behavior is undefined.
%REVIEW: vass: can define this to produce ambiguous alignment in such cases.
\end{protobody}