\chapter{Virtual Memory and Interprocess Communication}

\epigraph{Abbott: Now you've got it. \\
Costello: I throw the ball to Naturally. \\
Abbott: You don't! You throw it to Who! \\
Costello: Naturally. \\
Abbott: Well, that's it - say it that way. \\
Costello: That's what I said.}{Abbott and Costello on Effective Communication}

In very simple embedded systems and early computers, processes directly access memory -- ``Address 1234'' corresponds to a particular byte stored in a particular part of physical memory.
For example the IBM 709 had to read and write directly to a tape with no level of abstraction \cite[P. 65]{ibm709}.
Even in systems after that, it was hard to adopt virtual memory because virtual memory required the whole fetch cycle to be altered through hardware -- a change many manufacturers still thought was expensive.
In the PDP-10, a workaround was used by using different registers for each process and then virtual memory was added later \cite{ricm}.
In modern systems, this is no longer the case.
Instead each process is isolated, and there is a translation process between the address of a particular CPU instruction or piece of data of a process and the actual byte of physical memory (``RAM'').
Memory addresses no longer map to physical addresses
The process runs inside virtual memory.
Virtual memory not only keeps processes safe because one process cannot directly read or modify another process's memory.
Virtual memory also allows the system to efficiently allocate and re-allocate portions of memory to different processes.
The modern process of translating memory is as follows.

\begin{enumerate}
\item A process makes a memory request
\item The circuit first checks the Translation Lookaside Buffer (TLB) if the address page is cached into memory. It skips to the reading from/writing to phase if found otherwise the request goes to the MMU.
\item The Memory Management Unit (MMU) performs the address translation. If the translation succeeds, the page get pulled from RAM -- conceptually the entire page isn't loaded up. The result is cached in the TLB.
\item The CPU performs the operation by either reading from the physical address or writing to the address.
\end{enumerate}

\section{Translating Addresses}

The Memory Management Unit is part of the CPU, and it converts a virtual memory address into a physical address.
First we'll talk about what the virtual memory abstraction is and how to translate addresses

To illustrate, imagine you had a 32 bit machine, meaning pointers are 32 bits.
They can address $2^{32}$ different locations or 4GB of memory where one address is one byte.
Imagine we had a large table for every possible address where we will store the `real' i.e. ~physical address.
Each physical address will need 4 bytes -- to hold the 32 bits.
Naturally, This scheme would require 16 billion bytes to store all of entries.
It should be painfully obvious that our lookup scheme would consume all of the memory that we could possibly buy for our 4GB machine.
Our lookup table better be smaller than the memory we have otherwise we will have no space left for our actual programs and operating system data.
The solution is to chunk memory into small regions called `pages' and `frames' and use a lookup table for each page.

\subsection{Terminology}

A \textbf{page} is a block of virtual memory.
A typical block size on Linux operating system is 4KiB or $2^{12}$ addresses, though you can find examples of larger blocks.
So rather than talking about individual bytes we can talk about blocks of 4KiBs, each block is called a page.
We can also number our pages (``Page 0'' ``Page 1'' etc). Let's do a sample calculation of how many pages are there assume page size of 4KiB.

For a 32 bit machine,

\[
   2^{32} \text{address} / 2^{12} \text{(address/page)} = 2^{20} \text{pages}.
\]

For a 64 bit machine,

\[
   2^{64} \text{address} / 2^{12} \text{(address/page)} = 2^{52} \text{pages} \approx 10^{15} \text{pages}.
\]

We also call this a \textbf{frame} or sometimes called a `page frame' is a block of \emph{physical memory} or RAM -- Random Access Memory.
A frame is the same number of bytes as a virtual page or 4KiB on our machine.
A frame actually stores the bytes of interest.
To access a particular byte in a frame, you go from the start of the frame and add the offset -- discussed later.

A \textbf{page table} is a maps a number to a particular frame.
For example Page 1 might be mapped to frame 45, page 2 mapped to frame 30.
Other frames might be currently unused or assigned to other running processes, or used internally by the operating system.
You can imagine a page table just as that

\begin{figure}[H]
  \centering
  \includegraphics[width=.5\textwidth]{ipc/drawings/frame_table.png}
  \caption{Explicit Frame Table}
\end{figure}

In practice we will omit the first column because it will always be sequentially 0, 1, 2, etc and instead we'll use the offset from the start of the table as the entry number.

Now to go through the actual calculations.
We will assume that a 32 bit machine has 4KiB pages
Naturally to address all the possible entries, there are $2^{20}$ frames.
Since there are $2^{20}$ possible frames, we will need 20 bits to number all of the possible frames meaning \keyword{Frame Number} must be 2.5 bytes long.
In practice, we'll round that up to 4 bytes and do something interesting with the rest of the bits.
With 4 bytes per entry x $2^{20}$ entries = 4 MiB of physical memory are required to hold the entire page table for a process.

Remember our page table maps pages to frames, but each frame is a block of contiguous addresses.
How do we calculate which particular byte to use inside a particular frame?
The solution is to re-use the lowest bits of the virtual memory address directly.
For example, suppose our process is reading the following address- \keyword{VirtualAddress = 11110000111100001111000010101010 (binary)}

So to give an example say we have the virtual address above.
How would we split it up using a one page table to frame scheme?

\begin{figure}[H]
\centering
\includegraphics[width=.5\textwidth]{ipc/drawings/address_split.png}
\caption{Splitting Address}
\end{figure}

We can imagine the steps to dereference as one process.
In general, it looks like the following.

\begin{figure}[H]
  \centering
  \includegraphics[width=1.0\textwidth]{ipc/drawings/level_split.png}
  \caption{One level dereference}
\end{figure}

The way to read from a particular address above is visualized below.

\begin{figure}[H]
  \centering
  \includegraphics[width=1.0\textwidth]{ipc/drawings/level_split_filled.png}
  \caption{One level dereference example}
\end{figure}

And if we were reading from it, 'return' that value.
This sounds like a perfect solution.
Take each address and map it to a virtual address in sequential order.
The process will believe that the address look continuous, but the top 20 bits are used to figure out \keyword{page\_num},
which will allow us to find the frame number, find the frame, add the \textbf{offset} -- derived from the last 12 bits -- and do the read or write.

There are other ways to split it as well.
On a machine with page size 256 Bytes, then the lowest 8 bits (10101010) will be used as the offset.
The remaining upper bits will be the page number (111100001111000011110000). This offset is treated as a binary number and is added to the start of the frame when we get it.

We do a have a problem with 64 bit operating systems.
For a 64 bit machine with 4KiB pages, each entry needs 52 bits.
Meaning we need roughly
With $2^{52}$ entries thats $2^{55}$ bytes (roughly 40 petabytes).
So our page table is too large.
In 64 bit architectures memory addresses are sparse, so we need a mechanism to reduce the page table size, given that most of the entries will never be used.
We'll take about this below. There is one last piece of terminology that needs to be covered.

\subsection{Multi-level page tables}

Multi-level pages are one solution to the page table size issue for 64 bit architectures.
We'll look at the simplest implementation - a two level page table.
Each table is a list of pointers that point to the next level of tables, not all sub-tables need to exist.
An example, two level page table for a 32 bit architecture is shown below.

\begin{figure}[H]
\centering
\includegraphics[width=.5\textwidth]{ipc/drawings/three_address_split.png}
\caption{Three way Address Split}
\end{figure}

So what is the intuition for dereferencing an address?
First, you take the top level page table and find the \keyword{Index1}'th entry.
That will contain a number that will lead you to the appropriate sub-table
Then go to the \keyword{Index2}'th entry of that table.
That will contain a frame number.
This is the good old fashioned 4KiB RAM that we were talking about earlier.
Then you add the offset and do the read or write.

\subsubsection{Visualizing The Dereference}

In one diagram, the dereference looks like the following image.

\begin{figure}[H]
  \centering
  \includegraphics[width=1.0\textwidth]{ipc/drawings/multi_level_split.png}
  \caption{Full page table dereference}
\end{figure}

Following our example, here is what the dereference would look like.

\begin{figure}[H]
  \centering
  \includegraphics[width=1.0\textwidth]{ipc/drawings/multi_level_filled.png}
  \caption{Full page example dereference}
\end{figure}

\subsubsection{Calculating Size Concerns}

Now some calculations on size.
Each \keyword{page\_table\_num} index is 10 bits wide because there are only $2^{10}$ possible \keyword{sub-tables}, so we need 10 bits to store each directory index.
We'll round up to 2 bytes for the sake of reasoning.
If 2 bytes are used for each entry in the top level table and there are only $2^{10}$ entries, we only need 2KiB to store this entire first level page table.
Each subtable will point to physical frames, and each of their entries need to be the required 4 bytes to be able to address all the frames as mentioned earlier.
However, for processes with only tiny memory needs, we only need to specify entries for low memory address for the heap and program code and high memory addresses for the stack.

Thus the total memory overhead for our multi-level page table has shrunk from 4MiB for the single level implementation to 3 page tables of memory or 2KiB for the top level and 4KiB for the two intermediate levels or 10KiB.
Here's why.
We need at least one frame for the high level directory and two frames for just two sub-tables.
One sub-table is necessary for the low addresses -- program code, constants and possibly a tiny heap.
The other sub-table is for higher addresses used by the environment and stack.
In practice, real programs will likely need more sub-table entries, as each subtable can only reference 1024*4KiB = 4MiB of address space.
The main point still stands.
We have significantly reduced the memory overhead required to perform page table lookups.

\subsection{Page Table Disadvantages}

There are lots of problems with page tables -- one of the big problems is that they are slow.
For a single page table, our machine is now twice as slow!
Two memory accesses are required.
For a two-level page table, memory access is now three times as slow -- three memory accesses are required.

To overcome this overhead, the MMU includes an associative cache of recently-used virtual-page-to-frame lookups.
This cache is called the TLB (``translation lookaside buffer'').
Every time a virtual address needs to be translated into a physical memory location, the TLB is queried in parallel to the page table.
For most memory accesses of most programs, there is a significant chance that the TLB has cached the results.
However if a program does not have good cache coherence, the TLB will not have the result cache and now the MMU must use the much slower page table to determine the physical frame.

\subsection{MMU Algorithm}

There is a sort of pseudocode associated with the MMU.
We will assume that this is for a single level page table.

\begin{enumerate}
\item Receive address
\item Try to translate address according to the programmed scheme
\item If the translation fails, report an invalid address
\item Otherwise,
	\begin{enumerate}
  \item If the TLB contains the physical memory, get the physical frame from the TLB and perform the read and write.
	\item If the page exists in memory, check if the process has permissions
		to perform the operation on the page meaning the process has access
		to the page, and it is reading from the page/writing to a page
		that it has permissions to do so.
		\begin{enumerate}
		\item If so then do the dereference provide the address, cache the results in the TLB
		\item Otherwise trigger a hardware interrupt. The kernel
			will most likely send a SIGSEGV or a Segmentation Violation.
		\end{enumerate}
	\item If the page doesn't exist in memory, generate an Interrupt.
		\begin{enumerate}
		\item The kernel could realize that this page could either be not
			allocated or on disk.
      If it fits the mapping, allocate the page and try the operation again.
		\item Otherwise, this is an invalid access and the kernel will most likely send a SIGSEGV to the process.
		\end{enumerate}
	\end{enumerate}
\end{enumerate}

How would you alter this for a multi-level page table?

\subsection{Frames and Page Protections}

Frames be shared between processes, and this is where the heart of the chapter comes into play.
We can use these tables to communicate with processes.
In addition to storing the frame number, the page table can be used to store whether a process can write or only read a particular frame.
Read only frames can then be safely shared between multiple processes.
For example, the C-library instruction code can be shared between all processes that dynamically load the code into the process memory.
Each process can only read that memory.
Meaning that if you try to write to a read-only page in memory you will get a \keyword{SEGFAULT}.
That is why sometimes memory accesses SEGFAULT and sometimes they don't, it all depends on if your hardware says that you can access.

In addition, processes can share a page with a child process using the \keyword{mmap} system call.
\keyword{mmap} is an interesting call because instead of tying each virtual address to a physical frame, it ties it to something else. It is an important distinction that we are talking about mmap and not memory-mapped IO in general. The \keyword{mmap} system call can't reliably be used to do other memory mapped operations like communicate with GPUs and write pixels to the screen -- this is mainly hardware dependent.

\subsubsection{Bits on Pages}

This is \textit{heavily} dependent on the chipset.
We will include some bits that have historically been popular in chipsets.

\begin{enumerate}
\item The read-only bit marks the page as read-only.
Attempts to write to the page will cause a page fault.
The page fault will then be handled by the Kernel.
Two examples of the read-only page include sharing the c runtime library between multiple processes for security you wouldn't want to allow one process to modify the library and Copy-On-Write where the cost of duplicating a page can be delayed until the first write occurs.

\item The execution bit defines whether bytes in a page can be executed as CPU instructions.
Processors may will merge these bits into one and deem a page either writable or executable.
This bit is useful because it prevents stack overflow or code injection attacks when writing user data into the heap or the stack because those are not read only and thus not executable.
Further reading: \href{http://en.wikipedia.org/wiki/NX\_bit\#Hardware\_background}{background}

\item The dirty bit allows for a performance optimization.
A page on disk that is paged in to physical memory, then read from, and subsequently paged out again does not need to be written back to disk, since the page hasn't changed.
However, if the page was written to after it's paged in, its dirty bit will be set, indicating that the page must be written back to the backing store.
This strategy requires that the backing store retain a copy of the page after it is paged in to memory.
When a dirty bit is not used, the backing store need only be as large as the instantaneous total size of all paged-out pages at any moment.
When a dirty bit is used, at all times some pages will exist in both physical memory and the backing store.
\item There are plenty of other bits. Take a look at your favorite architecture and see what other bits are associated!
\end{enumerate}

\subsection{Page Faults}

A page fault is when a running program tries to access some virtual memory in its address space that is not mapped to physical memory.
Page faults will also occur in other situations.
There are three types of Page Faults

\begin{enumerate}
\item \textbf{Minor} If there is no mapping yet for the page, but it is a valid address.
  This could be memory asked for by \keyword{sbrk(2)} but not written to yet meaning that the operating system can wait for the first write before allocating space -- if it was read from, the operating system could short circuit the operation to read 0.
  The OS simply makes the page, loads it into memory, and moves on.

\item \textbf{Major} If the mapping to the page is not in memory but on disk.
  What this will do is swap the page into memory and swap another page out.
  If this happens frequently enough, your program is said to \emph{thrash} the MMU.

\item \textbf{Invalid} When you try to write to a non-writable memory address or read to a non-readable memory address.
  The MMU generates an invalid fault and the OS will usually generate a \keyword{SIGSEGV} meaning segmentation violation meaning that you wrote outside the segment that you could write to.

\end{enumerate}

\subsection{What does this have to do with IPC?}

What does this have to do with IPC?
Well internally before this point in your programming career you knew that processes had isolation.
But, you one didn't really know how that isolation mapped.
Two, you may have not know how you can break this isolation.
To break any memory level isolation you have two avenues.
One is to ask the kernel to provide some kind of interface.
The other is to ask the kernel to map two pages of memory to the same virtual memory area, and handle all the synchronization yourself.

\section{mmap}

\keyword{mmap} as mentioned earlier is a trick of virtual memory of instead of mapping a page to just a frame, that frame can be backed by a file on disk, or the frame can be shared among processes.
We can use that to read from a file on disk efficiently or sync changes to the file.
One of the big optimizations is that this does not mean that the file is malloc'ed to memory right away.
Take the following code for example.

\begin{lstlisting}[language=C]
int fd = open(...); //File is 2 Pages
char* addr = mmap(..fd..);
addr[0] = 'l';
\end{lstlisting}

The kernel may say, see that you want to mmap the file into memory, so it will reserve some space in your address space that is the length of the file.
That means when you write to \keyword{addr[0]} that you are actually writing to the first byte of the file.
The kernel can actually do some optimizations too.
Instead of loading the file into memory, it may only load pages at a time because if the file is 1024 pages.
You may only access 3 or 4 pages making loading the entire file a waste of time.
That is why page faults are so powerful.
They let the operating system take control of how much you use your files.

\subsection{mmap Definitions}

\keyword{mmap} does more than just take a file and map it to memory.
It is the general interface for creating shared memory among processes.
Currently it only supports regular files and POSIX \keyword{shmem} \cite{mmap_2018}.
Naturally, you can read all about it in the citation above, which references the current working group POSIX standard.
Some other options to note in the page will follow.

The first option is that the \keyword{flags} argument of mmap can take many options.

\begin{enumerate}
\item \keyword{PROT\_READ} This means the process can read the memory. This isn't the only flag that gives the process read permission however! The underlying file descriptor in this case must be opened with read privileges.
\item \keyword{PROT\_WRITE} This means the process can write to the memory. This has to be supplied for a process to write to a mapping. If this is supplied and \keyword{PROT\_NONE} is also supplied, the latter wins and no writes can be performed. The underlying file descriptor in this case must either be opened with write privileges or a private mapping must be supplied below
\item \keyword{PROT\_EXEC} This means the process can execute this piece of memory. Although this is not stated in POSIX documents, this shouldn't be supplied with WRITE or NONE because that would make this invalid under the NX bit or not being able to execute (respectively)
\item \keyword{PROT\_NONE} This means the process can't do anything with the mapping. This could be useful if you implement guard pages in terms of security. If you surround critical data with many more pages that can't be accessed, that decreases the chance of various attacks.
\item \keyword{MAP\_SHARED} This mapping will be synchronized to the underlying file object. The file descriptor must've been opened with write permissions in this case.
\item \keyword{MAP\_PRIVATE} This mapping will only be visible to the process itself. Useful to not thrash the operating system.
\end{enumerate}

Remember that once you are done \keyword{mmap}ping that you \keyword{munmap} to tell the operating system that you are no longer using the pages allocated, so the OS can write it back to disk and give you the addresses back in case another mmap needs to occur.
There are accompanying calls \keyword{msync} that take a piece of mmap'ed memory and sync the changes back to the filesystem though we won't cover that in depth. It is important because if you believe your operating system may crash or hang with the next few operations, you should create a hard backup of all the operations that have been executed.

The other parameters to mmap are described in the annotated walkthrough below.

\subsection{Annotated mmap Walkthrough}

Below is an annotated walk through of the example code in the man pages

\begin{lstlisting}[language=C]
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#define handle_error(msg) \
do { perror(msg); exit(EXIT_FAILURE); } while (0)

int
main(int argc, char *argv[])
{
  // Various variables that we'll talk about as the code goes through
  char *addr;
  int fd;
  struct stat sb;
  off_t offset, pa_offset;
  size_t length;
  ssize_t s;

  if (argc < 3 || argc > 4) {
    fprintf(stderr, "%s file offset [length]\n", argv[0]);
    exit(EXIT_FAILURE);
  }

  // Open the file and get the size of the file
  fd = open(argv[1], O_RDONLY);
  if (fd == -1)
  handle_error("open");

  if (fstat(fd, &sb) == -1)           /* To obtain file size */
  handle_error("fstat");

  // Grab where the user wants to start the file from
  offset = atoi(argv[2]);

  // Zeros off the bottom N bits of a page (usually 12)
  // mmap works based off of pages so we want to get
  // The first page the user wants
  pa_offset = offset & ~(sysconf(_SC_PAGE_SIZE) - 1);
  /* offset for mmap() must be page aligned */

  if (offset >= sb.st_size) {
    fprintf(stderr, "offset is past end of file\n");
    exit(EXIT_FAILURE);
  }

  if (argc == 4) {
    length = atoi(argv[3]);
    if (offset + length > sb.st_size)
    length = sb.st_size - offset;
    /* Can't display bytes past end of file */

  } else {    /* No length arg ==> display to end of file */
    length = sb.st_size - offset;
  }

  /*
  Here are the arguments for MMAP
  1. NULL, this tells mmap we don't need any particular address to start from
  2. length + offset - pa_offset, mmaps the ``rest'' of the file into memory (starting form offset)
  3. PROT_READ, we want to read the file
  4. MAP_PRIVATE, tell the OS, we don't want to share our mapping
  5. fd, object descriptor that we refer to
  6. pa_offset, the page aligned offset to start from
  */
  addr = mmap(NULL, length + offset - pa_offset, PROT_READ,
  MAP_PRIVATE, fd, pa_offset);
  if (addr == MAP_FAILED)
  handle_error("mmap");

  // Write the file as if a normal buffer
  s = write(STDOUT_FILENO, addr + offset - pa_offset, length);
  if (s != length) {
    if (s == -1)
    handle_error("write");

    fprintf(stderr, "partial write");
    exit(EXIT_FAILURE);
  }

  // This is important! You must mmap the memory out
  munmap(addr, length + offset - pa_offset);

  // This is also important, tells the OS that any fd level changes need to be addressed
  close(fd);

  exit(EXIT_SUCCESS);
}
\end{lstlisting}

\subsection{MMAP Communication}

So how would we use mmap to communicate across processes?
Conceptually, it would be the same as using threading.
Let's go through a broken down example.
First, we need to allocate some space.
We can do that with the \keyword{mmap} call.
We'll also allocate space for 100 integers

\begin{lstlisting}[language=C]
int size = 100 * sizeof(int);
void *addr = mmap(0, size, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
int *shared = addr;
\end{lstlisting}

Then, we need to fork and perform some communication.
Our parent will store some values, and our child will read those values.

\begin{lstlisting}[language=C]
pid_t mychild = fork();
if (mychild > 0) {
  shared[0] = 10;
  shared[1] = 20;
} else {
  sleep(1); // Check the synchronization chapter for a better way
  printf("%d\n", shared[1] + shared[0]);
}
\end{lstlisting}

Now, there is no assurance that the values will be communicated because the process used \keyword{sleep} not a mutex.
Most of the time this will work.

\begin{lstlisting}[language=C]
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h> /* mmap() is defined in this header */
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>

int main() {

  int size = 100 * sizeof(int);
  void *addr = mmap(0, size, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);

  printf("Mapped at %p\n", addr);

  int *shared = addr;
  pid_t mychild = fork();
  if (mychild > 0) {
    shared[0] = 10;
    shared[1] = 20;
  } else {
    sleep(1); // We will talk about synchronization later
    printf("%d\n", shared[1] + shared[0]);
  }

  munmap(addr,size);
  return 0;
}
\end{lstlisting}

This piece of code allocates space for a 100 integers and creates a piece of memory that is shared between all processes.
The code then forks.
The parent process writes two integers to the first two slots.
To avoid a data race, the child sleeps for a second and then prints out the stored values.
This is not a perfect way to protect against data races.
We could use a mutex across processes mentioned in the synchronization section.
But for this simple example, it works just fine.
Note that each process should call munmap when done using the piece of memory.

Sharing anonymous memory is a very efficient form of inter-process communication because there is no copying, system call, or disk-access overhead - the two processes literally share the same physical frame of main memory.
On the other hand, shared memory, like in a multithreading context, creates room for data races.
Processes that share writable memory might need to use synchronization primitives like mutexes to prevent these from happening.

\section{Pipes}

You've seen the virtual memory way of IPC, but there are more standard versions of IPC that are provided by the kernel.
One of the big utilities is UNIX, now POSIX, pipes. A pipe simply takes in a stream of bytes and spits out a sequence of bytes.

One of the big starting points of pipes was way back in the PDP-10 days. In those days, a write to the disk or even your terminal was very slow as it may have to be printed out.
The Unix programmers still wanted to create small, portable programs that did one thing very well and could be composed.
As such, pipes were invented to take the output of one program
and feed it to the input of another program though they have other uses today -- you can read more \href{https://en.wikipedia.org/wiki/Pipeline\_\%28Unix\%29}{At the wikipedia page}
Consider if you type the following into your terminal.

\begin{lstlisting}[language=bash]
$ ls -1 | cut -d'.' -f1 | sort | uniq | tee dirents
\end{lstlisting}

What does the following code do? Well it \keyword{ls}'s the current directory.
The -1 means that it outputs one entry per line.
The \keyword{cut} command then takes everything before the first period.
\keyword{sort} sorts all the input lines, \keyword{uniq} makes sure all the lines are uniq.
Finally \keyword{tee} outputs the contents to the file \keyword{dir\_contents} and the terminal for your perusal.
The important part is that bash creates \textbf{5 separate processes} and connects their standard outs/stdins with pipes the trail looks something like this.

\begin{figure}[H]
  \centering
  \includegraphics[width=.8\textwidth]{ipc/drawings/pipe_process.png}
  \caption{Pipe Process Filedescriptor redirection}
\end{figure}

The numbers in the pipes are the file descriptors for each process and the arrow represents the redirect or where the output of the pipe is going.
A POSIX pipe is almost like its real counterpart - you can stuff bytes down one end and they will appear at the other end in the same order.
Unlike real pipes however, the flow is always in the same direction, one file descriptor is used for reading and the other for writing.
The \keyword{pipe} system call is used to create a pipe.
These file descriptors can be used with \keyword{read} and with \keyword{write}. A common method of using pipes is to create the pipe before forking in order to communicate with a child process

\begin{lstlisting}[language=C]
int filedes[2];
pipe (filedes);
pid_t child = fork();
if (child > 0) { /* I must be the parent */
  char buffer[80];
  int bytesread = read(filedes[0], buffer, sizeof(buffer));
  // do something with the bytes read
} else {
  write(filedes[1], "done", 4);
}
\end{lstlisting}

There are two file descriptors that pipe creates.
\keyword{filedes[0]} contains the read end.
\keyword{filedes[1]} contains the write end.
How your friendly neighborhood TAs remember it is one \textit{reads before you can write, or reading comes before writing}.
You can groan all you want at it, but it is helpful to remember what is the read end and what is the write end.

One can use pipes inside of the same process, but there tends to be no added benefit.
Here's an example program that sends a message to itself.

\begin{lstlisting}[language=C]
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>

int main() {
  int fh[2];
  pipe(fh);
  FILE *reader = fdopen(fh[0], "r");
  FILE *writer = fdopen(fh[1], "w");
  // Hurrah now I can use printf
  printf("Writing...\n");
  fprintf(writer,"%d %d %d\n", 10, 20, 30);
  fflush(writer);

  printf("Reading...\n");
  int results[3];
  int ok = fscanf(reader,"%d %d %d", results, results + 1, results + 2);
  printf("%d values parsed: %d %d %d\n", ok, results[0], results[1], results[2]);

  return 0;
}
\end{lstlisting}

The problem with using a pipe in this fashion is that writing to a pipe can block meaning the pipe only has a limited buffering capacity.
The maximum size of the buffer is system dependent; typical values from 4KiB up to 128KiB though they can be changed.

\begin{lstlisting}[language=C]
int main() {
  int fh[2];
  pipe(fh);
  int b = 0;
  #define MESG "..............................."
  while(1) {
    printf("%d\n",b);
    write(fh[1], MESG, sizeof(MESG))
    b+=sizeof(MESG);
  }
  return 0;
}
\end{lstlisting}

\subsection{Pipe Gotchas}

Here's a complete example that doesn't work!
The child reads one byte at a time from the pipe and prints it out - but we never see the message!
Can you see why?

\begin{lstlisting}[language=C]
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <signal.h>

int main() {
  int fd[2];
  pipe(fd);
  //You must read from fd[0] and write from fd[1]
  printf("Reading from %d, writing to %d\n", fd[0], fd[1]);

  pid_t p = fork();
  if (p > 0) {
    /* I have a child therefore I am the parent*/
    write(fd[1],"Hi Child!",9);

    /*don't forget your child*/
    wait(NULL);
  } else {
    char buf;
    int bytesread;
    // read one byte at a time.
    while ((bytesread = read(fd[0], &buf, 1)) > 0) {
      putchar(buf);
    }
  }
  return 0;
}
\end{lstlisting}

The parent sends the bytes \keyword{H,i,(space),C...!} into the pipe.
The child starts reading the pipe one byte at a time.
In the above case, the child process will read and print each character.
However, it never leaves the while loop!
When there are no characters left to read it simply blocks and waits for more unless \textbf{All the writers close their ends}
Another solution could also exit the loop by checking for an end-of-message marker,

\begin{lstlisting}[language=C]
while ((bytesread = read(fd[0], &buf, 1)) > 0) {
  putchar(buf);
  if (buf == '!') break; /* End of message */
}
\end{lstlisting}

Okay so we know that when a process tries to read from a pipe where there are still writers, the process blocks.
If no pipe has no writers, read returns 0.
If a process tries to write with some readers the read goes through, or fails -- partially or completely -- if the pipe is full.
Why happens when a process tries to write when there are no readers left?

\begin{verbatim}
If all file descriptors referring to the read end of a pipe have been closed,
then a write(2) will cause a SIGPIPE signal to be generated for the calling process.
\end{verbatim}

Tip: Notice only the writer (not a reader) can use this signal. To
inform the reader that a writer is closing their end of the pipe, you
could write your own special byte (e.g.~0xff) or a message (
\keyword{"Bye!"})

Here's an example of catching this signal that does not work! Can you see why?

\begin{lstlisting}[language=C]
#include <stdio.h>
#include <stdio.h>
#include <unistd.h>
#include <signal.h>

void no_one_listening(int signal) {
  write(1, "No one is listening!\n", 21);
}

int main() {
  signal(SIGPIPE, no_one_listening);
  int filedes[2];

  pipe(filedes);
  pid_t child = fork();
  if (child > 0) {
    /* I must be the parent. Close the listening end of the pipe */
    /* I'm not listening anymore!*/
    close(filedes[0]);
  } else {
    /* Child writes messages to the pipe */
    write(filedes[1], "One", 3);
    sleep(2);
    // Will this write generate SIGPIPE ?
    write(filedes[1], "Two", 3);
    write(1, "Done\n", 5);
  }
  return 0;
}
\end{lstlisting}

The mistake in above code is that there is still a reader for the pipe!
The child still has the pipe's first file descriptor open and remember the specification? All readers must be closed

When forking, \emph{It is common practice} to close the unnecessary (unused) end of each pipe in the child and parent process. For example the parent might close the reading end and the child might close the writing end and vice versa if you have two pipes.

The last addendum is that you can actually set the file descriptor to return when there is no one listening instead of SIGPIPE because by default SIGPIPE terminates your program.
The reason that this is default behavior is it makes the pipe example above work.
Consider this useless use of cat

\begin{lstlisting}[language=bash]
$ cat /dev/urandom | head -n 20
\end{lstlisting}
Which grabs 20 lines of input from urandom. \keyword{head} will terminate after 20 newline characters have been read. Well what about \keyword{cat}?
\keyword{cat} needs to receive a SIGPIPE informing it that the program just tried to write to a pipe that no one is listening on.

\subsection{Other pipe facts}

A pipe gets filled up when the writer writes too much to the pipe without the reader reading any of it. When the pipes become full, all writes fail until a read occurs. Even then, a write may partial fail if the pipe has a little bit of space left but not enough for the entire message. To avoid this, usually two things are done. Either increase the size of the pipe. Or more commonly, fix your program design so that the pipe is constantly being read from.

As hinted at before, Pipe write are atomic up to the size of the pipe. Meaning that if two processes try to write to the same pipe, the kernel has internal mutexes with the pipe that it will lock, do the write, and return. The only gotcha is when the pipe is about to become full. If two processes are trying to write and the pipe can only satisfy a partial write, that pipe write is not atomic -- be careful about that!

Unnamed pipes live in memory and are a simple and efficient form of inter-process communication (IPC) that is useful for streaming data and simple messages. Once all processes have closed, the pipe resources are freed.

It is also common design for a pipe to be one way -- meaning one process should do the writing and one process do the reading. Otherwise the child would attempt to read its own data intended for the parent (and vice versa)!

\subsection{Pipes and Dup}

A lot of time, you'll want to use \keyword{pipe2} in combination with dup.
Take for example the simple program in the command line.

\begin{lstlisting}[language=bash]
$ ls -1 | cut -f1 -d.
\end{lstlisting}

This command take the output of \keyword{ls -1} which lists the content of the current directory on one line each and pipes it to cut.
Cut take a delimiter in this case a dot and a field position in our case 1 and outputs per line the nth field by each delimiter.
At a high level, this grabs the file names without the extension of our current directory.

Underneath the hood, this is how bash does it internally.

\begin{lstlisting}[language=C]
#define _GNU_SOURCE

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>

int main() {

  int pipe_fds[2];
  // Call with the O_CLOEXEC flag to prevent any commands from blocking
  pipe2(pipe_fds, O_CLOEXEC);

  // Remember for pipe_fds, you read then write (reading is 0 and writing is 1)

  if(!fork()) {
    // I am the child

    // Make the stdout of the process, the write end
    dup2(pipe_fds[1], 1);

    // Exec! Don't forget the cast
    execlp("ls", "ls", "-1", (char*)NULL);
    exit(-1);
  }

  // Same here, except the stdin of the process is the read end
  dup2(pipe_fds[0], 0);

  // Same deal here
  execlp("cut", "cut", "-f1", "-d.", (char*)NULL);
  exit(-1);

  return 0;
}
\end{lstlisting}

The results of the two programs should be the same.
Remember as you encounter more complicated examples of piping processes up, you need to close all unused ends of pipes otherwise you will deadlock waiting for your processes to finish.

\subsection{Pipe Conveniences}

If you already have a file descriptor, you can `wrap' it yourself into a FILE pointer using \keyword{fdopen}.

\begin{lstlisting}[language=C]
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

int main() {
  char *name="Fred";
  int score = 123;
  int filedes = open("mydata.txt", "w", O_CREAT, S_IWUSR | S_IRUSR);

  FILE *f = fdopen(filedes, "w");
  fprintf(f, "Name:%s Score:%d\n", name, score);
  fclose(f);
\end{lstlisting}

For writing to files this is unnecessary - just use \keyword{fopen} which does the same as \keyword{open} and \keyword{fdopen} However for pipes, we already have a file descriptor - so this is great time to use \keyword{fdopen}

Here's a complete example using pipes that almost works!
Can you spot the error?
Hint: The parent never prints anything!

\begin{lstlisting}[language=C]
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>

int main() {
  int fh[2];
  pipe(fh);
  FILE *reader = fdopen(fh[0], "r");
  FILE *writer = fdopen(fh[1], "w");
  pid_t p = fork();
  if (p > 0) {
    int score;
    fscanf(reader, "Score %d", &score);
    printf("The child says the score is %d\n", score);
  } else {
    fprintf(writer, "Score %d", 10 + 10);
    fflush(writer);
  }
  return 0;
}
\end{lstlisting}

Note the unnamed pipe resource will disappear once both the child and parent have exited.
In the above example, the child will send the bytes and the parent will receive the bytes from the pipe.
However, no end-of-line character is ever sent, so \keyword{fscanf} will continue to ask for bytes because it is waiting for the end of the line i.e.~it will wait forever! The fix is to ensure we send a newline character, so that \keyword{fscanf} will return.

\begin{lstlisting}[language=C]
change:   fprintf(writer, "Score %d", 10 + 10);
to:       fprintf(writer, "Score %d\n", 10 + 10);
\end{lstlisting}

If you want your bytes to be sent to the pipe immediately, you'll need to fflush!
At the beginning of this course we assumed that file streams are always \emph{line buffered} -- the C library will flush its buffer every time you send a newline character.
Actually this is only true for terminal streams - for other file streams the C library attempts to improve performance by only flushing when it's internal buffer is full or the file is closed.

\textbf{Even though we have a section on it, it is highly not recommended to fdopen() anything that doesn't point to an underlying file descriptor}. The reason being is that while we get conveniences we also get annoyances like the buffering example we mentioned, caching, etc etc.
The basic UNIX motto is that anything that you can \keyword{fseek} or move to an arbitrary position in, you should be able to \keyword{fdopen}. Files satisfy this behavior, shared memory also, terminals, etc.
When it comes to pipes, sockets, epoll objects, etc, don't do it.

\section{Named Pipes}

An alternative to \emph{unnamed} pipes is \emph{named} pipes created using \keyword{mkfifo}.
From the command line: \keyword{mkfifo} From C: \keyword{int\ mkfifo(const\ char\ *pathname,\ mode\_t\ mode);}

You give it the path name and the operation mode, it will be ready to go!
Named pipes take up virtual no space on disk.
This means the actual contents of the pipe aren't printed to the file and read from that same file.
What the operating system is essentially telling you when you have a named pipe is that it will create an unnamed pipe that refers to the named pipe, and that's it!
There is no additional magic.
This is just for programming convenience if processes are started without forking meaning that there would be no way to get the file descriptor to the child process for an unnamed pipe.

\subsection{Why is my named pipe hanging?}

A named pipe \keyword{mkfifo} is simply a pipe that you call \keyword{open(2)} on with read and/or write permissions.
This is useful if you want to have a pipe between two processes without one processing having to fork the other process.
There are some gotchas with named pipes.
There is more down below, but we'll introduce it here for a simple example.
Reads and writes hang on Named Pipes until there is at least one reader and one writer, take this.

\begin{lstlisting}[language=bash]
1$ mkfifo fifo
1$ echo Hello > fifo
# This will hang until I do this on another terminal or another process
2$ cat fifo
Hello
\end{lstlisting}

Any \keyword{open} is called on a named pipe the kernel blocks until another process calls the opposite open. Meaning, echo calls \keyword{open(..,\ O\_RDONLY)} but that blocks until cat calls \keyword{open(..,\ O\_WRONLY)}, then the programs are allowed to continue.

\subsection{Race condition with named pipes}

What is wrong with the following program?

\begin{lstlisting}[language=C]
//Program 1

int main(){
  int fd = open("fifo", O_RDWR | O_TRUNC);
  write(fd, "Hello!", 6);
  close(fd);
  return 0;
}

//Program 2
int main() {
  char buffer[7];
  int fd = open("fifo", O_RDONLY);
  read(fd, buffer, 6);
  buffer[6] = '\0';
  printf("%s\n", buffer);
  return 0;
}
\end{lstlisting}

This may never print hello because of a race condition.
Since you opened the pipe in the first process under both permissions, open won't wait for a reader because you told the operating system that you are a reader!
Sometimes it looks like it works because the execution of the code looks something like this.

\begin{tabular}{|c|c|c|}
  & Process 1 & Process 2 \\ \hline
  Time 1 & open(O\_RDWR) \& write() & \\
  Time 2 & & open(O\_RDONLY) \& read() \\
  Time 3 & close() \& exit() & \\
  Time 4 & print() \& exit() & \\
\end{tabular}

But here is an invalid series of operations that causes a race condition.

\begin{tabular}{|c|c|c|}
  & Process 1 & Process 2 \\ \hline
  Time 1 & open(O\_RDWR) \& write() & \\
  Time 2 & close() \& exit() & \\
  Time 3 & & open(O\_RDONLY) (Blocks Indefinitely) \\
\end{tabular}

\section{Files}

On Linux, there are two abstractions with files. The first is the Linux \keyword{fd} level abstraction.

 \begin{itemize}
\item \keyword{open} takes a path to a file and creates a file descriptor entry in the process table. If the file is not available to you, it errors out.
\item \keyword{read} takes a number of bytes that the kernel has received and reads them into a user space buffer. If the file is not open in read mode, this will break.
\item \keyword{write} outputs a number of bytes to a file descriptor. If the file is not open in write mode, this will break. This may be buffered internally.
\item \keyword{close} removes a file descriptor from a process' file descriptors. This always succeeds on a valid file descriptor.
\item \keyword{lseek} takes a file descriptor and moves it to a certain position. Can fail if the seek is out of bounds.
\item \keyword{fcntl} is the catch all function for file descriptors. You can do everything with this function. Set file locks, read, write, edit permissions, etc \ldots{}
 \end{itemize}

The Linux interface is very powerful and expressive, but sometimes we need portability for example if we are writing for a Macintosh or windows.
This is where C's abstraction comes into play.
On different operating systems, C uses the low level functions to create a wrapper around files you can use everywhere, meaning that C on Linux uses the above calls.

\begin{itemize}
\item \keyword{fopen} opens a file and returns an object. \keyword{null} is returned if you don't have permission for the file.
\item \keyword{fread} reads a certain number of bytes from a file. An error is returned if already at the end of file when which you must call \keyword{feof()} in order to check.
\item \keyword{fgetc/fgets} Get a char or a string from a file
\item \keyword{fscanf} Read a format string from the file
\item \keyword{fwrite} Write some objects to a file
\item \keyword{fprintf} Write a formatted string to a file
\item \keyword{fclose} Close a file handle
\item \keyword{fflush} Take any buffered changes and flush them to a file
\end{itemize}

But you don't get the expressiveness that Linux gives you with system calls you can convert back and forth between them with \keyword{int\ fileno(FILE*\ stream)} and \keyword{FILE*\ fdopen(int\ fd...)} Another important aspect to note is the C files are \textbf{buffered} meaning that their contents may not be written right away by default. You can can change that with C options.

\textbf{Danger} With portability you lose something important, the ability to tell an error. Take for example the advice that we gave you above. You can \keyword{fopen} a file descriptor and get a \keyword{FILE*} object but \textbf{it won't be the same as a file} meaning that certain calls will fail or act weirdly.
The C API reduces this weirdness, but for example you cannot \keyword{fseek} to a part of the file, or perform any operations with its buffering.
The problem is the API won't give a lot of warning because C needs to maintain compatibility with other operating systems.
To keep things simple for you, use the C API of files when you are dealing with a file on disk, which will work fine. Otherwise, be in for a rough ride for portability sake.

\subsection{How do I tell how large a file is?}

For files less than the size of a long, using fseek and ftell is a
simple way to accomplish this.
Move to the end of the file and find out the current position.

\begin{lstlisting}[language=C]
fseek(f, 0, SEEK_END);
long pos = ftell(f);
\end{lstlisting}

This tells us the current position in the file in bytes - i.e.~the
length of the file!

\keyword{fseek} can also be used to set the absolute position.

\begin{lstlisting}[language=C]
fseek(f, 0, SEEK_SET); // Move to the start of the file
fseek(f, posn, SEEK_SET);  // Move to 'posn' in the file.
\end{lstlisting}

All future reads and writes in the parent or child processes will honor this position.
Note writing or reading from the file will change the current position.
See the man pages for fseek and ftell for more information.

\subsection{But don't do this}

\textbf{Note: This is not recommended in the usual case because of a quirk with the C language}.
That quirk is that longs only need to be \textbf{4 Bytes big} meaning that the maximum size that \keyword{ftell} can return is a little under 2 Gibibytes which we know nowadays our files could be hundreds of gibibytes or even terabytes on a distributed file system.
What should we do instead? Use \keyword{stat}! We will cover stat in a later part but here is some code that will tell you the size of the file

\begin{lstlisting}[language=C]
struct stat buf;
if(stat(filename, &buf) == -1){
  return -1;
}
return (ssize_t)buf.st_size;
\end{lstlisting}

\keyword{buf.st\_size} is of type \keyword{off\_t} which is big enough for large files.

\subsection{Gotchas with files}

What happens when file streams are closed by two different processes?
Closing a file stream is unique to each process.
Other processes can continue to use their own file-handle.
Remember, everything is copied over when a child is created, even the relative positions of the files.
As you might have observed with using \keyword{fork}, there is a quirk of the implementation of files and their caches on Ubuntu that will rewind a file descriptor once a file has closed.
As such, make sure to close before forking or at least don't trigger a cache inconsistency which is much harder.

\section{So Many IPC Options}

Okay so now you have a list of tools in your toolbox in order to tackle communicating between processes, so what should you use?

There really is no hard answer, though this is the most interesting question.
Generally we have retained pipes for legacy reasons.
This means that we only really use them to redirect stdin, stdout, and stderr for the collection of logs and similar programs.
You may find processes trying to communicate with unnamed or named pipes as well.
Most of the time you won't be dealing with this interaction directly though.

Files are used almost all the time as a form of IPC.
Hadoop is a great example where processes will write to append only tables and then other processes will read from those tables.
We generally use files under a few cases.
One case is if we want to save the intermediate results of an operation to a file for future use.
Another case is if putting it in memory would cause an out of memory error.
On Linux, file operations are generally pretty cheap, so most programmers use it as a larger intermediate storage.

mmap is used for two scenarios.
One is a \keyword{linear} or \keyword{near-linear} read through of the file.
Meaning, you read the file front to back or back to front.
The key is that you don't jump around too much.
Jumping around too much causes thrashing and loses all the benefits of using mmap.
The other case of mmap is a direct memory inter-process communication.
This means that you can store structures in a piece of mmap'ed memory and share them between two processes.
Python and ruby use this mapping all the time to utilize copy on write semantics.

\section{Topics}

\begin{enumerate}
  \item Virtual Memory
  \item Page Table
  \item MMU/TLB
  \item Address Translation
  \item Page Faults
  \item Frames/Pages
  \item Single level vs multi level page table
  \item Calculating offsets for multi-level page table
  \item Pipes
  \item Pipe read write ends
  \item Writing to a zero reader pipe
  \item Reading from a zero writer pipe
  \item Named pipe and Unnamed Pipes
  \item Buffer Size/Atomicity
  \item Scheduling Algorithms
  \item Measures of Efficiency
\end{enumerate}

\section{Questions}

\begin{enumerate}
\item What is virtual memory?
\item What are the following and what is their purpose?
  \begin{enumerate}
    \item Translation Lookaside Buffer
    \item Physical Address
    \item Memory Management Unit. Multilevel page table. Frame number. Page number and page offset.
    \item The dirty bit
    \item The NX Bit
  \end{enumerate}
\item What is a page table? How about a physical frame? Does a page always need to point to a physical frame?
\item What is a page fault? What are the types? When does it result in a SEGFAULT?
\item What are the advantages to a single level page table? Disadvantages? How about a multi leveled table?
\item What does a multi leveled table look like in memory?
\item How do you determine how many bits are used in the page offset?
\item Given a 64 bit address space, 4kb pages and frames, and a 3 level page table, how many bits is the Virtual page number 1, VPN2, VPN3 and the offset?
\item What is a pipe? How do I create a pipe?
\item When is SIGPIPE delivered to a process?
\item Under what conditions will calling read() on a pipe block? Under what conditions will read() immediately return 0
\item What is the difference between a named pipe and an unnamed pipe?
\item Is a pipe thread safe?
\item Write a function that uses fseek and ftell to replace the middle character of a file with an 'X'
\item Write a function that create a pipe and uses write to send 5 bytes, "HELLO" to the pipe. Return the read file descriptor of the pipe.
\item What happens when you mmap a file?
\item Why is getting the file size with ftell not recommended? How should you do it instead?
\end{enumerate}

\bibliographystyle{plainnat}
\bibliography{ipc/ipc}


% !TEX root = ../../main.tex
\chapter{Object Reconstruction and Identification}
\label{ch:objreco}

Combining measurements from various ATLAS subdetectors allows for the identification of physics objects like electrons, muons, photons, and hadrons. The trajectory of a particle can be reconstructed by matching position measurements in the calorimeters and MS with track hits in the ID. Converging tracks extrapolated towards the IP identify vertices, and the one with the largest $\sum \pT^2$ for all associated tracks is labeled the primary vertex (PV).  Reconstructed objects are associated to the same interaction when their tracks extrapolate to the PV. In this analysis, the reconstructed objects of interest are: electrons, muons and jets. Additionally, missing transverse energy, $\MET$, is used to identify neutrinos which do not interact with the detector. This chapter describes the selections and algorithms used to identify these physics objects.

%
\section{Electrons}
\label{ch:objreco:el}
Electron objects are reconstructed by matching clusters of EM calorimeter energy deposits to reconstructed tracks in the ID. To build the EM cluster, a sliding window algorithm is used~\cite{sliding_window}. The window is a fixed-size grid of cells, $N_{\eta}\times N_{\phi}$=3\times5, in the middle layer of the LAr calorimeter where approximately $80\,\%$\,of the energy in an EM shower is deposited (\Sect{\ref{ch:atlas:particle_showers}}). For each cell, the energy is summed across all the longitudinal layers, forming a tower. If the transverse energy of the towers in the window is above 2.5\,\GeV\,and is a local maximum, a seed cluster is formed.

After loose shower shape requirements are applied to the seed clusters, a cone-shaped region of interest (ROI) of $\Delta R<0.3$ from the seed barycenter is defined~\cite{electron_efficiency, electron_efficiency_2016}.  Pattern recognition and track fitting algorithms are then used to match ID tracks to seed clusters. Track seeds from the ID with $\pT>1\,\GeV$\, are tested against pion and electron pattern recognition algorithms, and are required to reside in a cluster ROI. A track fit is performed twice to extrapolate the ID track to the calorimeter, using the track momentum or the cluster momentum, and is required to be near the cluster. If either succeeds, a final optimized track reconstruction is performed using a Gaussian Sum Fitter (GSF)~\cite{gsm_filter}.
%A summary of the track reconstruction and cluster matching parameters is shown in~\Tab{\ref{tab:el_cl_tr}}.

An electron candidate is formed if there is at least one ID track matched to the seed cluster. The EM cluster is then rebuilt by summing energy in a grid of $3\times 7$ cells in each layer, starting in the middle layer. After calibrations and corrections are applied, the four-momentum of the electron candidate is calculated using the energy measurement of the EM cluster and the $\eta$ and $\phi$ measurements from the track.

%\begin{table}[tb]
%\begin{center}
%\begin{tabular}{l|c|c|c|c}
%\hline\hline
%\textbf{Reconstruction} & \multicolumn{2}{c|}{$\Delta\phi_{\textrm{EM cluster}}$} & \multicolumn{2}{c}{$\Delta\eta_{\textrm{EM cluster}}$}  \\\cline{2-5}
%\textbf{Method}& Curve toward & Curve away & non-TRT-only & TRT-only \\\hline
%Track Momentum & $<0.2$ & $<0.05$ & $<0.05$ & --- \\\cline{2-5}
%Cluster Momentum & $<0.1$ & $<0.05$ & $<0.05$ & --- \\\hline
%\multicolumn{1}{l}{\textbf{Final}} &\multicolumn{4}{c}{\,} \\\hline
%$\quad$ non-TRT-only (GSF) & \multicolumn{2}{c|}{$<0.1$} & \multicolumn{2}{c}{---} \\\cline{2-5}
%$\quad$ TRT-only & $<0.03$ & $<0.02$ & --- & $<0.35$\\\hline\hline
%\end{tabular}
%\caption[Calorimeter cluster track matching requirements]{Requirements on the proximity of the extrapolated tracks to the calorimeter cluster seeds are shown for several reconstruction methods. The $\Delta\phi$ requirements depend on if the reconstructed track curves toward or away from the seed cluster. If extrapolation with either the track momentum or cluster momentum succeeds, a final track reconstruction is performed. For non-TRT-only tracks, the final reconstruction uses an optimized Gaussian Sum Filter (GSF) algorithm~\cite{gsm_filter}. }
%\label{tab:el_cl_tr}
%\end{center}
%\end{table}

Background objects like hadronic jets, non-prompt electrons from photon conversions, and semi-leptonic decays of hadrons with heavy quarks can ``fake'' an electron signature and be reconstructed at this stage as an electron candidate~\cite{electron_efficiency}. A multivariate analysis (MVA) technique takes into account several cluster and track variables to create a likelihood (LH) identification for each candidate as either signal or background. Different working points balancing signal efficiency with background rejection are provided as shown in~\Fig{\ref{fig:el_id_eff}}; in this analysis, the ``LooseLH'' and ``TightLH'' are considered, where TightLH is a subset of LooseLH. The LooseLH working point has very high signal efficiency and mostly offers light flavor jet discrimination. The TightLH working point has a lower signal efficiency, but additionally rejects photon conversions and heavy flavor jets.

In addition to the discrimination due to the LH identification, further suppression of background electrons is provided with isolation criteria~\cite{electron_efficiency_2016}. A real electron from, for example, a $W$ boson decay, will be produced in relative isolation while jets, converted photons, and other fake electrons will be accompanied by nearby energy deposits. Isolation is calculated both from the calorimeter EM cluster, and from the ID track. Calorimeter isolation, $E_{\rm T}^{\textrm{cone} 0.2}$, calculates the sum of transverse energy ($E_{\rm T}$) in a cone of $\Delta R<0.2$ around the
\begin{wrapfigure}{r}{.6\textwidth}
\begin{center}
\includegraphics[width=.59\textwidth]{figures/ObjectReconstruction/el_id_eff}
\end{center}
\caption[Electron reconstruction and identification efficiencies]{The reconstruction and identification efficiencies for signal electrons is presented as a function of transverse energy for several working points~\cite{el_id_eff}.}
\label{fig:el_id_eff}
\end{wrapfigure}
electron center, and subtracts the $5\times7$ cell window contribution from the electron. Track isolation, $\pT^{\textrm{varcone} 0.2}$, calculates the sum of transverse momentum for quality tracks in a variable cone of size $\Delta R< \textrm{min}(0.2, 10\,\GeV/E_{\rm T})$ centered around and excluding the electron track. Working points using calorimeter and/or track based isolation are provided for either 1) a specific efficiency with variable cuts or 2) fixed cut values. The ``LooseTrackOnly'' and ``FixedCutTight'' isolation working points are used in this analysis. The LooseTrackOnly working point only uses the ID track based isolation for a constant $99\,\%$\, efficiency. The FixedCutTight working point cuts at $E_{\rm T}^{\textrm{cone} 0.2}/E_T < 0.06$ and $\pT^{\textrm{varcone} 0.2}/E_T < 0.06$.

The identification and isolation working points are optimized for electrons coming from the PV. This analysis additionally requires the recommended cuts on longitudinal ($z_0$) and transverse ($d_0$) impact parameters to verify the reconstructed electron is associated with the PV. Electrons are required to have $|\eta|<2.47$, excluding the calorimeter transition region between $1.37<|\eta|<1.52$, where the uncertainties can be large due to poor resolution. Finally, a minimum transverse momentum is required to avoid trigger turn-on curves, as discussed in~\Ch{\ref{ch:event_selection}}. This analysis uses two types of electrons, ``signal'' and ``veto'', which are defined and summarized in~\Tab{\ref{tab:lep_def}}. Veto electrons are used to avoid overlap with similar analyses with distinct final states.

\begin{table}[htbp]
\begin{center}
\begin{tabular}{l|c|c|c|c}
\hline\hline
\multicolumn{1}{c|}{\textbf{Cut}} & \multicolumn{2}{c|}{\textbf{Electron Definition}} &  \multicolumn{2}{c}{\textbf{Muon Definition}} \\\cline{2-5}
& Veto & Signal & Veto & Signal \\\hline
\pt [\GeV] & $>7$ & $> 27$  & $>7$ & $> 27$ \\\hline
$|\eta|$ & \multicolumn{2}{c|}{$<2.47\notin[1.37, 1.52]$} & $<2.7$ & $<2.5$ \\\hline
Identification & LooseLH & TightLH & Loose & Medium  \\\hline
\vbox{\hbox{\strut Isolation}\hbox{\strut}} & \vbox{\hbox{\strut LooseTrack-}\hbox{\strut Only}} & \vbox{\hbox{\strut FixedCutTight}\hbox{\strut}} & \vbox{\hbox{\strut LooseTrack-}\hbox{\strut Only}} & \vbox{\hbox{\strut FixedCutTight-}\hbox{\strut TrackOnly}}\\\hline
$|d_0/\sigma(d_0)^{BL}|$ & \multicolumn{2}{c|}{$<5$} & \multicolumn{2}{c}{$<3$}\\\hline
$|z_0\sin\theta|$ [mm]& \multicolumn{4}{c}{$< 0.5$}  \\\hline\hline
\end{tabular}
\caption[Electron and muon object definitions]{The object definitions for electrons and muons used in this analysis are shown. Veto leptons are defined to reduce overlap with similar final state searches. }
\label{tab:lep_def}
\end{center}
\end{table}
%
\section{Muons}
Muon objects are reconstructed by matching track candidates independently created in the MS and ID~\cite{muon_eff}. In the MS, algorithms create segments by matching hits aligned in the bending plane between multiple layers. If multiple track segments match in different layers, a track candidate is formed.

Using combinations of information from the ID, calorimeters, and MS, four types of muon candidates are defined. In decreasing priority, they are:
\begin{itemize}
	\item \underline{Combined (CB)}: The full muon track is reconstructed, starting in the MS and extrapolating towards the ID.
	\item \underline{Segment Tagged (ST)}: The ID track is extrapolated towards the MS, where it must match at least one MDT/CSC track segment (useful for low-\pT muons).
	\item \underline{Calo Tagged (CT)}: The ID track is extrapolated to the calorimeter, where it matches an energy deposit (useful in the region $|\eta|<0.1$ where the MS has no coverage).
	\item \underline{Stand Alone (SA)}: A MS track not matched to an ID track, but extrapolated close to the IP, is used to recover muons in the range $2.5<|\eta|<2.7$ that has poor ID coverage.
\end{itemize}

Background objects, such as decaying pions and kaons, can be reconstructed as a ``fake'' muon candidate. Using the four types of muon candidates, and additional cuts based on track and calorimeter based variables, working points for muon identification are created to balance signal efficiency with background rejection. In this analysis, the ``Medium'' and ``Loose'' identification working points are used, corresponding to approximate efficiencies of 96.1\,\% and 98.1\,\%, respectively. The Medium working point only uses CB and SA candidates, while the Loose working point considers all four types.

Muon isolation is performed analogously to electron isolation: track-based and calorimeter-based isolation measurements are used to create working points with either 1) efficiency based variable cuts or 2) fixed cuts. The track-based muon isolation, $\pt^{\textrm{varcone} 0.3}$, uses a slightly larger cone, $\Delta R < 0.3$, with respect to the electron case.  In this analysis, ``LooseTrackOnly'' and ``FixedCutTightTrackOnly'' isolation working points are used. The LooseTrackOnly working point has a constant $99\,\%$\, efficiency. The FixedCutTightTrackOnly requires $\pt^{\textrm{varcone} 0.3}/\pT < 0.06$.

Finally, as with electrons, two types of muon objects are defined, ``signal'' and ``veto''. Recommended $\pT$, $\eta$, and PV association cuts are additionally placed on the muon objects, whose definitions are summarized in~\Tab{\ref{tab:lep_def}}.

%
\section{Jets}
Hadronization of quarks from the initial hard scatter event, or subsequent hadronic decays of unstable particles ($W\ra q\overline{q}'$, $Z\ra q\overline{q}$), creates showers of lower energy particles, called jets, in the detector. Jets are formed by clustering nearby energy deposits in the calorimeter and matching the cluster with ID tracks.

Topologically connected three dimensional cell clusters, or ``topo-clusters''~\cite{topo_cluster}, are used as constituents in the formation of a jet. Unlike the sliding-window calorimeter cluster seeds used for electrons, topo-clusters can have variable sizes. A topo-cluster is seeded if the energy of a cell is four standard deviations above the noise level ($E_{\textrm{cell}}>4 \sigma_{\textrm{cell}}^{\textrm{noise}}$), and connected cells are added to the cluster if $E_{\textrm{cell}}>2 \sigma_{\textrm{cell}}^{\textrm{noise}}$.

The anti-$k_{\rm T}$ jet reconstruction algorithm~\cite{anti_kt_algo} is a sequential combination algorithm used in this analysis to combine topo-clusters into jet objects. A distance parameter is defined in~\Eqn{\ref{eq:anti_kt}}.
\begin{eqnarray}
\label{eq:anti_kt}
d_{ij} &=& \textrm{min}\left(\frac{1}{p_{\rm T,i}^2},\frac{1}{p_{\rm T,j}^2}\right)\frac{\Delta R_{ij}^2}{R^2}  \\
d_{iB} &=& \frac{1}{p_{\rm T,i}^2}
\end{eqnarray}
The distance $d_{ij}$ measures the distance between two topo-clusters, while $d_{iB}$ measures the distance between a topo-cluster and the beam line. The distance parameter, $R$, roughly defines the size of the jet. The clustering procedure iteratively finds the smallest distance among all constituents. If the smallest distance is between two clusters, $d_{ij}$, the clusters are combined and the procedure continues. If it is between a cluster and the beam axis, $d_{iB}$, the cluster is defined as a jet and removed from the collection of remaining constituents. The distances are then recalculated and more jets are found until there are no remaining clusters. The anti-$k_{\rm T}$ algorithm tends to combine high-\pT constituents first and produces a roughly conical jet. ID tracks are matched with jets through a procedure known as ``ghost association''~\cite{ghost_assoc}. In effect, tracks are identified as particles with infinitesimal momentum and included in the clustering process. Their negligible momentum ensures the final jet clusters are not affected, but will include the tracks. In this analysis, ``small-R jets'' ($j$) and ``large-R jets'' ($J$), corresponding to $R=0.4$ and $R=1.0$, are used.

The energies of calorimeter jets are corrected for PU with a $\mu$-dependent subtraction. Additionally, a ``jet energy scale" (JES) response correction~\cite{jet_energy_measurement}  is applied to accurately calibrate both EM and hadronic showers. The JES aims to correct for the non-compensating nature of the calorimeters (hadrons have a lower detector response with respect to EM objects like electrons and photons), and energy loss in regions of the detector without measuring instruments. Small-R jets in this analysis use a simple JES scheme, called ``EM+JES'', in which the jets are measured at the EM scale\footnote{
	 The EM scale means the energy of an EM shower will be correctly measured.
} and a scale factor is applied based on the $\pT$ and $\eta$ measurements of the jet. For large-R jets, a local cluster weighting (LCW) procedure applies calibrations at a topo-cluster level, according to whether the cluster corresponds to a hadronic or EM energy deposit.  In this ``LCW+JES" scheme, jets are reconstructed from the locally calibrated clusters and a final JES scale factor is applied (smaller than in the EM+JES scheme).
%
\subsection{Small-R Jets}
\label{ch:objectReconstruction:smallr}
Small-R jets ($R=0.4$) are used in this analysis to identify both VBF jets, and jets from $b$-quark decays. In VBF candidate events, the initial quarks, which radiate vector bosons, have a small deflection and hadronize into two jets with a large separation in $\eta$. Jets with $b$-hadrons, called $b$-jets, are useful for identifying candidate events from top quark pair decays ($t\bar{t}$). Candidate $b$-jets (VBF jets) must have $\pT>20\,\GeV$\, (30\,\GeV) and $|\eta| < 2.5$ (4.5).

Although a uniform PU subtraction is applied, local deviations may create PU jets. These jets can originate from both QCD effects (from a single PU vetex) and stochastic effects (contributions from multiple vertices). To reject both types of PU jets, a jet vertex tagger (JVT)~\cite{jvt} is used to assign jets to the PV. The JVT is a two-dimensional likelihood discriminant which factors in tracking and vertex information. For this analysis, the 92\,\% efficiency working point is used for jets with $\pT<60\,\GeV$\,and $|\eta|<2.4$. A residual 2\,\% of PU jets remain at this efficiency.

A multivariate $b$-tagging algorithm called MV2~\cite{b_jet} is used to tag small-R jets that contain $b$-hadrons. MV2 takes as input several $b$-tagging algorithms based on the impact parameter, secondary vertex reconstruction, and full decay chain reconstruction. The MV2 algorithm uses a boosted decision tree with background composition from $c$-jets (light-jets) of 7\,\% (93\,\%). Light-jets refer to jets from gluons or $u, d,$ or $s$ quarks. For this analysis, the 85\,\%\,efficiency $b$-tagging working point is used. The inverse of the corresponding mis-tag rate, called the rejection ratio, is 3.1 (33) for $c$-jets (light-jets)~\cite{b_jet_opt}. VBF candidate jets are required to fail $b$-tagging. A summary of the small-R jets definitions is shown in~\Tab{\ref{tab:small_j_def}}.

\begin{table}[htbp]
\begin{center}
\begin{tabular}{l|c|c}
\hline\hline
\multicolumn{1}{c|}{\textbf{Cut}} & \multicolumn{2}{c}{\textbf{Small-R Jet Definition}} \\\cline{2-3}
& $b$-jet Candidate & VBF jet Candidate  \\\hline
Algorithm & \multicolumn{2}{c}{Anti-$k_{\rm T}$ $R=0.4$}  \\\hline
Energy Calibration & \multicolumn{2}{c}{EM+JES} \\\hline
\pt [\GeV] &$>20$&$>30$\\\hline
$|\eta|$ &$<2.5$&$<4.5$\\\hline
\vbox{\hbox{\strut Pileup Removal}\hbox{\strut (JVT 92\,\% efficiency)}} & \multicolumn{2}{c}{\vbox{\hbox{\strut if\, $\pT<60\,\GeV$}\hbox{\strut \&\& $|\eta|<2.4$}}}  \\\hline
$b-$tag (MV2c10 85\,\% efficiency)& pass & fail \\\hline\hline
\end{tabular}
\caption[Small-R jet object definition]{The object definitions for small-R jets used in this analysis are shown.}
\label{tab:small_j_def}
\end{center}
\end{table}

%
\subsection{Large-R Jets}
\label{ch:objectReconstruction:larger}
Large-R jets ($R=1.0$) are used in this analysis to identify boosted hadronically decaying vector bosons.  Jets from the two-body decays of boosted vector bosons have a substructure typically absent from decays of gluons and light quarks~\cite{boost_trim}. To elucidate the substructure differences, grooming techniques remove soft contributions to the jet. In particular, this analysis uses a grooming technique called trimming, depicted in~\Fig{\ref{fig:jet_trim}}. Jet trimming further aims to improve the jet mass resolution by removing contamination from initial state radiation (radiation from the incoming hadrons), multiple parton interactions (further interactions among the partons after the hadron collision), and PU interactions.
The trimming procedure in this analysis involves the following steps:
\begin{itemize}
	\item Re-cluster the large-R jet ($J$) constituents into ``sub-jets'' with the anti-$k_{\rm T}$ algorithm with distance parameter $R=0.2$.
	\item Remove sub-jets if $p_{\rm T}^{\textrm{sub-jet}}< 0.05\times p_{\rm T}^{J}$.
	\item Recombine the surviving sub-jets into a final trimmed jet.
\end{itemize}
\begin{figure}[tbp]
\begin{center}
\includegraphics[width=.8\textwidth]{figures/ObjectReconstruction/jet_trim}
\caption[Jet trimming illustration]{The jet trimming procedure is illustrated. In this analysis, sub-jets are created with the anti-$k_{\rm T}$ algorithm with distance parameter $R_{\textrm{sub}}=0.2$, and the fractional \pT threshold is set at $f_{\textrm{cut}}=0.05$~\cite{boost_trim}.}
\label{fig:jet_trim}
\end{center}
\end{figure}

Even with trimming, jet mass resolution can suffer in the high-\pT regime from a loss of angular information when multiple highly boosted decay products are reconstructed as a single topo-cluster. A track mass, $m_{\textrm{track}}$, based on ID charged particle tracks can be used to improve the mass resolution~\cite{jet_track_mass}. The track mass is calculated by ghost associating ID tracks with $\pT>0.4\,\GeV$ to the large-R jet, and then summing the masses associated with all matched tracks. To correct for the missing neutral particle contribution to $m_{\textrm{track}}$, a ratio of the calorimeter-based to track-based transverse momentum is applied to the track mass. The resulting quantity, called the track-assisted mass, is shown in~\Eqn{\ref{eq:m_ta}}.
\begin{equation}
m_{\textrm{TA}} \equiv m_{\textrm{track}}\times\frac{p_{\rm T}^{\textrm{calo}}}{p_{\rm T}^{\textrm{track}}}
\label{eq:m_ta}
\end{equation}

To take advantage of both the calorimeter and track-assisted mass, a weighted sum, minimizing the jet mass resolution and called the combined mass, is defined in~\Eqn{\ref{eq:m_comb}}~\cite{jet_comb_mass}.
\begin{equation}
m_{\textrm{comb}} \equiv w_{\textrm{calo}}\times m_{\textrm{calo}} + w_{\textrm{track}}\times m_{\textrm{TA}}
\label{eq:m_comb}
\end{equation}
In this analysis, $m_{\textrm{calo}}$ and $m_{\textrm{TA}}$ are taken to be uncorrelated (reflecting an approximate 10\,\%\,correlation for $\pT > 1\,\TeV$). The weights in~\Eqn{\ref{eq:m_comb}} can thus be expressed in terms of the estimated jet mass resolution, $\sigma_{\textrm{calo}}$ ($\sigma_{\textrm{TA}}$) for $m_{\textrm{calo}}$ ($m_{\textrm{TA}}$), as shown in~\Eqn{\ref{eq:m_comb_uncorr}}.
\begin{equation}
m_{\textrm{comb}} = \frac{\sigma_{\textrm{calo}}^{-2}m_{\textrm{calo}} +  \sigma_{\textrm{TA}}^{-2}m_{\textrm{TA}}}{\sigma_{\textrm{calo}}^{-2}+\sigma_{\textrm{TA}}^{-2}}
\label{eq:m_comb_uncorr}
\end{equation}
The estimated jet mass resolutions are calculated as a function of \pT and $\eta$. The improvement in jet mass resolution of the combined mass is shown in~\Fig{\ref{fig:m_comb_res}}. The jet transverse momentum is also re-scaled to be compatible with the combined mass, as shown in~\Eqn{\ref{eq:jet_pt_comb}}.
\begin{equation}
p_{\rm T}^{\textrm{comb}} \equiv p_{\rm T}^{\textrm{calo}}\times\frac{m_{\textrm{comb}}}{m_{\textrm{calo}}}
\label{eq:jet_pt_comb}
\end{equation}
In the rest of this thesis, the combined mass and combined transverse momentum of the large-R jet will be referred to as $m(J)$ and $\pT(J)$ respectively. To ensure a complementary overlap between calorimeter and tracking information, large-R jets are required to have $|\eta|<2.0$. This search focuses on boosted $W$ or $Z$ bosons; thus, the large-R jet is required to have $\pT(J)>200\,\GeV$ and $m(J)>50\GeV$.
\begin{figure}[tbp]
\begin{center}
\subfloat[]{\includegraphics[width=.49\textwidth]{figures/ObjectReconstruction/m_comb_res}\label{fig:m_comb_res:a}}
\subfloat[]{\includegraphics[width=.49\textwidth]{figures/ObjectReconstruction/med_reconst_w_mass}\label{fig:m_comb_res:b}}
\end{center}
\caption[Performance of combined mass for large-R jets]{Three jet mass definitions are used to compare \protect\subref{fig:m_comb_res:a} the fractional jet mass resolution and \protect\subref{fig:m_comb_res:b} the median reconstructed jet mass, for simulated boosted $W$ or $Z$ bosons, as a function of truth \pT. The combined mass improves jet mass resolution for $250\,\GeV<\pT<2.5\,\TeV$ and $|\eta|<2.0$~\cite{jet_track_mass}.}
\label{fig:m_comb_res}
\end{figure}

To further differentiate jets due to a two-body decay of boosted $W$ or $Z$ bosons from QCD jets due to single partons, a substructure variable ($D_2^{\beta=1}$), based on the ratio of generalized two and three point energy correlation functions, is introduced~\cite{d2_one, d2_two}, as shown in~\Eqnrange{\ref{eq:en_corr_func}}{\ref{eq:d2_def}}.
\begin{eqnarray}
e_2^{\beta} & = & \frac{1}{\pT^2(J)}\sum_{1\leq i<j\leq n_j}\pT^i\pT^j\Delta R_{ij}^{\beta}  \label{eq:en_corr_func} \\
e_3^{\beta} & = & \frac{1}{\pT^2(J)}\sum_{1\leq i<j<k\leq n_j}\pT^i\pT^j\pT^k\Delta R_{ij}^{\beta}\Delta R_{ik}^{\beta}\Delta R_{jk}^{\beta} \\
D_{2}^{\beta=1} & = & \frac{e_3^{\beta=1}}{\left(e_2^{\beta=1}\right)^3} \label{eq:d2_def}
\end{eqnarray}
For $n_j$ constituents in the jet, $\pT^i$ is the transverse momentum of the i$^{\textrm{th}}$ constituent, and $\Delta R_{ij}$ is the previously defined angular separation between the i$^{\textrm{th}}$ and j$^{\textrm{th}}$ constituents.

A boson tagging algorithm, called SmoothedWZTagger~\cite{smoothed_wz_tagger}, identifies large-R jets as either a $W$ or $Z$ boson decay, based on the large-R jet combined mass and $D_2^{\beta=1}$ substructure variable, at fixed signal efficiency working points of $50\,\%$\, and $80\,\%$. The selections are optimized in \pT bins and a smoothing function is fit to the resulting mass-window and $D_2^{\beta=1}$ threshold cuts, as depicted in~\Fig{\ref{fig:smooth_cuts}}. A summary of the large-R jet definition is presented in~\Tab{\ref{tab:large_jet_def}}.
\begin{figure}[htbp]
\begin{center}
\subfloat[]{\includegraphics[width=.5\textwidth]{figures/ObjectReconstruction/MassWindow_newWZTagger}\label{fig:smooth_cuts:a}}
\subfloat[]{\includegraphics[width=.5\textwidth]{figures/ObjectReconstruction/D2UpperCut_newWZTagger}\label{fig:smooth_cuts:b}}
\caption[Boson tagger mass and substructure cuts]{The SmoothedWZTagger selections cuts for \protect\subref{fig:smooth_cuts:a} the combined mass window and \protect\subref{fig:smooth_cuts:b} the upper-cut on $D_2^{\beta=1}$, are shown as a function of \pT for both the $50\,\%$\,and $80\,\%$ efficiency working points. The $50\,\%$\,($80\,\%$) efficiency working point corresponds to a background rejection factor of 45-75 (11-13) and 50-70 (9-13) for $W$ and $Z$ bosons respectively.}
\label{fig:smooth_cuts}
\end{center}
\end{figure}
\begin{table}[htbp]
\begin{center}
\begin{tabular}{l|c}
\hline\hline
\textbf{Cut} &\textbf{Large-R Jet Definition} \\\hline
Algorithm & Anti-$k_{\rm T}$ $R=1.0$  \\\hline
Grooming & Trimming with $R_{\textrm{sub-jet}}=0.2$, $f_{\textrm{cut}}=0.05$ \\\hline
Energy Calibration & LCW+JES \\\hline
\pt [\GeV] &$>200$\\\hline
$|\eta|$ &$<2.0$\\\hline
Mass [\GeV] & $>50$ \\\hline
Boson Tagging & SmoothedWZTagger \\\hline\hline
\end{tabular}
\caption[Large-R jet definition]{The object definition for large-R jets used in this analysis.}
\label{tab:large_jet_def}
\end{center}
\end{table}

%
\section{Missing Transverse Energy}

Neutrinos produced in collisions pass through ATLAS without leaving energy deposits in the calorimeters, or tracks in the ID and MS. Since the colliding protons have no initial momentum transverse to the beam line, momentum conservation implies that the vector sum of the transverse momentum of all the final state particles should also be zero. Deviations from zero can indicate the presence of a non-interacting particle, like a neutrino, in the final state.  In \Eqn{\ref{eq:met_eq}}, the calculation of \MET is summarized as the negative vector sum of all reconstructed objects, and additional ``soft terms'' corresponding to tracks from the PV that are not matched to reconstructed objects.
\begin{eqnarray}
E^{\textrm{miss}}_{x(y)} &=& - \sum\limits_{e\in\{\textrm{electrons}\}} p_{x(y)}^{e}  - \sum\limits_{\mu\in\{\textrm{muons}\}} p_{x(y)}^{\mu}  - \sum\limits_{j\in\{\textrm{jets}\}} p_{x(y)}^{j}   - \sum\limits_{s\in\{\textrm{soft terms}\}} p_{x(y)}^{s} \nonumber \\
\MET & = & \sqrt{\left(E_x^{\textrm{miss}}\right)^2+ \left(E_y^{\textrm{miss}}\right)^2}
\label{eq:met_eq}
\end{eqnarray}
The use of soft terms that have ID tracks consistent with the PV ignores possible contributions from neutral particles. A calorimeter-based soft term, using topologically connected clusters unassociated with reconstructed particles, can be used to measure these contributions. However, the track-based soft terms offer better \MET resolution and are less pile-up dependent~\cite{met_perf}. As a result, the track-based soft terms are used in this analysis to reconstruct \MET. Photons and hadronically decaying taus are not used in this analysis, and are reconstructed as jets in the \MET calculation.

In this analysis, the \MET is identified as the transverse momentum of the neutrino, $\pT^{\nu}$. To determine the $z$ component of the neutrino momentum, the $W$ boson from the $W\ra\ell\nu$ decay is constrained to be exactly on-shell with $m(W\ra\ell\nu)=80.4\,\GeV$. Using this constraint, a quadratic equation for the neutrino $p_z$ can be solved. If a solution is complex, the real part is taken, and if two unique solutions exist, the smallest in absolute value is taken. A more detailed discussion is presented in~\App{\ref{ch:neutrinopz}}.

%
\section{Overlap Removal}
As electrons, muons, and jets use a combination of tracking and calorimeter measurements, a single collection of energy deposits and tracks may be reconstructed independently as several physics objects. To remove any ambiguity, a procedure is defined to prioritize which reconstructed objects are kept and which are discarded.

\begin{itemize}
\item If there is a shared ID track between an electron candidate and a muon candidate, the electron is removed. In this case, the MS track from the muon candidate cannot be due to an electron.
\item Next, if there is a small-R jet and electron candidate with $\Delta R(j,e)<0.2$, the small-R jet is removed since the reconstructed jet does not differentiate between hadronic and EM showers. However, the electron is removed if $0.2<\Delta R(j,e)<\textrm{min}(0.4,0.04+10\,\GeV/\pT(e))$. Electrons reconstructed near the edge of a jet are most likely from non-prompt decays of jet constituents. The sliding cone, with maximum size $R=0.4$ at low $\pT(e)$, recovers boosted prompt electrons that are close to the edge of jets.
\item If there is a large-R jet and electron with $\Delta R(J,e)<1.0$, the large-R jet is removed.
\item If a muon and small-R jet satisfy $\Delta R(j,\mu)<0.2$, and either 1) the jet has fewer than two tracks or 2) $\pT(\mu)/\pT(j)>0.5$ and $\pT(\mu)/\sum\pT(\textrm{tracks})>0.7$, the jet is discarded. This indicates the jet is most likely from the calorimeter energy loss of a muon. If the jet is not removed, a sliding cone is used to remove the muon if $\Delta R(j,\mu)<\textrm{min}(0.4,0.04+10\,\GeV/\pT(\mu))$. Similar to the electron case, boosted muons sufficiently far from the jet center are kept, while lower \pT muons likely from non-prompt decays near the jet edge are removed.
\item There is no overlap removal between muons and large-R jets, as muons are unlikely to deposit enough calorimeter energy to be reconstructed as a jet with $\pT>200\,\GeV$.
\end{itemize}


\section{Pickpocket}
Pickpocketing is to steal a small item \Parentheses{interpretation up to the DM/GM} from an NPC's - or another player character's - inventory, while undetected by the target. As with all of the aforementioned non-combat abilities that involve a dice roll, pickpocketing is impossible for characters who do not have a feat that enables it. Like with the other dice-rolling abilities, when the character attmepts to pickpocket, they must roll a d20 dice \Parentheses{or the sum of four d5 dices, or five d4 dices} - or, in case of a video game, use a random number generator to generate a number between 1 and 20. When the received number is smaller than the character's dexterity, the pickpocket attempt is succesful. Otherwise, it is a fail, and the target is alerted.


\section{Space from behavior}

We now give the main theorem of the paper.

\begin{theorem}[main]
  The metric induced by the inner product coincides with the metric
  induced by bisimulation.
\end{theorem}

\paragraph{Proof sketch} The metric induced by bisimulation (when we
put in the definition) will be the quote of the smallest witness of
the smallest distinguishing formulae. The inner product quotes the
effect of calculating the difference (what's left after you whack $P$
against $Q$ and let them run). These two notions coincide.

To make this statement precise enough to prove, we have a number of
obligations to discharge. First of all, we have to give the metric
induced by bisimulation. To do this, we need to give the
Hennessy-Milner construction for the process calculus defined
above. This logic enjoys the usual property that processes are
bisimilar if and only if there is no distinguishing formula. We will
take the distance between processes to be the smallest distinguishing
formula. Thus, we have to give a measure of the size of the formula of
this logic, which we write $\#(\phi)$.

With these definitions in hand we show that the quotation of the
smallest witness of the smallest formulae is ``bisimilar'' to the name
computed by the inner product. Notice that up this point, the
calculation we are calling inner product has yet to be proved to
provide a metric. Establishing the correspondence to the HML-induced
metric is actually what gives us the right to think of the inner
product calculation as a metric.

Note that it is possible to give an alternative definition following
Caires' notion of a characteristic formula. If $\meaningof{\phi}$
denotes the set of processes satisfying $\phi$ and
$\meaningof{P}_{\phi}$ denotes the characteristic formula of a
process, then we can show that
$\meaningof{\meaningof{\innerprod{P}{Q}}_{\phi}} =
\meaningof{\quotep{D}_{\phi}(P,Q)}$ where

\begin{definition}[metric]
  \begin{mathpar}
    \Delta(P,Q) := \{ \phi \; : \; (P \models \phi \; \& \; Q \not\models \phi) \vee (P \not\models \phi \; \& \; Q \models \phi) \}
    \and
    D_{\phi}(P,Q) := \bigvee_{\phi \in \Delta(P,Q)} \phi
    \and
    \quotep{D}_{\phi}(P,Q) := \quotep{\bigvee_{\phi \in \Delta(P,Q)} \phi}
    \and
    D(P,Q) := min \{ \#(\phi) \; : \; \phi \in \Delta(P,Q) \}
  \end{mathpar}
\end{definition}

So, without further ado, we give you the HML construction for a
reflective calculus.

\subsection{Namespace logic: a Hennessy-Milner logic of reflection}
Namespace logic resides in the subfamily of Hennessy-Milner logics
discovered by Caires and Cardelli and known as spatial logics
\cite{DBLP:journals/tcs/CairesC04}. Thus, as is seen below, in
addition to the action modalities, we also find formulae for
\emph{separation}, corresponding, at the logical level, to the
structural content of the parallel operator at the level of the
calculus. Likewise, we have quantification over names.

In this connection, however, we find an interesting difference between
spatial logics investigated heretofore and this one. As in the
calculus, we find no need for an operator corresponding to the $\nu$
construction. However, revelation in spatial logic, is a structural
notion \cite{DBLP:journals/tcs/CairesC04}. It detects the
\emph{declaration} of a new name. No such information is available in
the reflective calculus or in namespace logic. The calculus and the
logic can arrange that names are used in a manner consistent with
their being declared as new in the {\pic}, but it cannot detect the
declaration itself. Seen from this perspective, revelation is a
somewhat remarkable observation, as it seems to be about detecting the
programmer's intent.

\begin{mathpar}
  \inferrule* [lab=boolean connectives] {} {{\phi,\psi} \bc \ptrue \; | \; \neg \phi \; |\; \phi \& \psi}
  \and
  \inferrule* [lab=spatial connectives] {} {| \; \pzero \; | \; \phi | \psi}
  \and
  \inferrule* [lab=nominal connectives] {} {| \; \pdropf{b} \; | \; \pquant{n}{\psi}{\phi}}
  \and
  \inferrule* [lab=behavioral connectives] {} {| \; \pprefix{a}{\vec{b}}{\phi} \; | \; \plift{a}{\vec{\phi}}}
  \and
  \inferrule* [lab=fixpt connectives] {} {| \; \pgfp{X}{\phi} \; | \; X}
  \and
  \inferrule* [lab=patterns] {} {{a} \bc \quotep{\phi} \; | \; b}
  \and
  \inferrule* [lab=literals] {} {{b} \bc \quotep{P} \; | \; n}
\end{mathpar}

We let $\PFormula$ denote the set of formulae generated by the
$\phi$-production, $\QFormula$ denote the set of formulae generated by
the $a$-production and $\PropVar$ denote the set of propositional
variables used in the $\textsf{rec}$ production.

Inspired by Caires' presentation of spatial logic
\cite{DBLP:conf/fossacs/Caires04}, we give the semantics in terms of
sets of processes (and names). We need the notion of a valuation $v :
\PropVar \to \wp(\Proc)$, and use the notation $v\substn{\mathcal{S}}{X}$ to mean

\begin{eqnarray}
  v\substn{\mathcal{S}}{X}(Y) & = &
  \left\{ \begin{array}{ccc}
      S & & Y = X \\
      v(Y) & & otherwise \\
    \end{array}
  \right.\nonumber
\end{eqnarray}

The meaning of formulae is given in terms of two mutually recursive functions,

\begin{eqnarray}
\pmeaningof{ - }( - ) : \PFormula \times [\PropVar \to \wp(\Proc)] \to \wp(\Proc) \nonumber\\
\nmeaningof{ - }( - ) : \QFormula \times [\PropVar \to \wp(\Proc)] \to \wp(\QProc) \nonumber
\end{eqnarray}

taking a formula of the appropriate type and a valuation, and
returning a set of processes or a set of names, respectively.

\begin{eqnarray}
  \pmeaningof{\ptrue}(v) & = & \Proc \nonumber \\
  \pmeaningof{\pzero}(v) & = & \{ P : P \scong \pzero \} \nonumber \\
  \pmeaningof{\neg \phi}(v) & = & \Proc / \pmeaningof{\phi}(v) \nonumber\\
  \pmeaningof{\phi \& \psi}(v) & = & \pmeaningof{\phi}(v) \cap \pmeaningof{\psi}(v) \nonumber\\
  \pmeaningof{\binpar{\phi}{\psi}}(v) & = &
  \{ P : \exists P_0, P_1.P \scong \binpar{P_0}{P_1}, \; P_0 \in \pmeaningof{\phi}(v), \;  P_1 \in \pmeaningof{\psi}(v) \} \nonumber\\
  \pmeaningof{\pdropf{b}}(v) & = & \{ P : \exists Q, P'.P \scong \binpar{Q}{\dropn{x}}, \; x \in \nmeaningof{b}(v) \} \nonumber\\
  \pmeaningof{\plift{a}{\phi}}(v) & = & \{ P : \exists x, P'.P \scong \lift{x}{P'},
                                           \; x \in \nmeaningof{a}(v),
                                           \; P' \in \pmeaningof{\phi}(v) \} \nonumber\\
  \pmeaningof{\pprefix{a}{b}{\phi}}(v) & = & \{ P : \exists x, P'.P \scong \prefix{x}{y}{P'}, x \in \nmeaningof{a}(v), \nonumber\\
                                   &   &            \; \; \; \forall c . \exists z . {P'}\substn{z}{y} \in \pmeaningof{{\phi}\substn{c}{b}}(v) \} \nonumber\\
  \pmeaningof{\pgfp{X}{\phi}}(v) & = & \cup \{ \mathcal{S} \subseteq \Proc : \mathcal{S} \subseteq \pmeaningof{\phi}(v\substn{\mathcal{S}}{X})\} \nonumber\\
  \pmeaningof{\pquant{n}{\psi}{\phi}}(v) & = & \cap_{x \in \nmeaningof{\quotep{\psi}}(v)} \pmeaningof{{\phi}\substn{x}{n}}(v) \nonumber\\
  \nmeaningof{\quotep{\phi}}(v) & = & \{ x : x \nameeq \quotep{P}, P \in \pmeaningof{\phi}(v) \} \nonumber\\
  \nmeaningof{\quotep{P}}(v) & = & \{ x : x \nameeq  \quotep{P} \} \nonumber
\end{eqnarray}

We say $P$ witnesses $\phi$ (resp., $x$ witnesses $\quotep{\phi}$),
written $P \models \phi$ (resp., $x \models \quotep{\phi}$) just when
$\forall v . P \in \meaningof{\phi}(v)$ (resp., $\forall v . x \in \meaningof{\quotep{\phi}}(v)$).

\begin{theorem}[Equivalence]
	$P \wbbisim Q \riff \forall \phi . P \models \phi \riff Q \models \phi .$
\end{theorem}

The proof employs an adaptation of the standard strategy. As noted in
the introduction, this theorem means that there is no algorithm
guaranteeing that a check for the witness relation will terminate.

\subsubsection{Syntactic sugar }

In the examples below, we freely employ the usual DeMorgan-based
syntactic sugar. For example,

\begin{eqnarray}
	\phi \Rightarrow \psi & \triangleq & \neg ( \phi \& \neg \psi ) \nonumber\\
	\phi \vee \psi & \triangleq & \neg ( \neg \phi \& \neg \psi ) \nonumber
\end{eqnarray}

Also, when quantification ranges over all of $\Proc$, as in
$\pquant{n}{\quotep{\ptrue}}{\phi}$, we omit the typing for the
quantification variable, writing $\pquantuntyped{n}{\phi}$.

\subsubsection{An example}

\paragraph{Controlling access to namespaces}

Suppose that $\quotep{\phi}$ describes some namespace, i.e. some
collection of names. We can insist that a process restrict its next
input to names in that namespace by insisting that it witness the formula

\begin{eqnarray}
  \pprefix{\quotep{\phi}}{b}{\ptrue} \& \neg \pprefix{\quotep{\neg \phi}}{b}{\ptrue} \nonumber
\end{eqnarray}

which simply says the the process is currently able to take input from
a name in the namespace $\quotep{\phi}$ and is not capable of input on
any name not in that namespace. In a similar manner, we can limit a
server to serving only inputs in $\quotep{\phi}$ throughout the
lifetime of its behavior \footnote{Of course, this formula also says
  the server never goes down, either -- or at least is always willing
  to take such input...;-)}

\begin{eqnarray}
  \pgfp{X}{\pprefix{\quotep{\phi}}{b}{X} \& \neg \pprefix{\quotep{\neg \phi}}{b}{\ptrue}} \nonumber
\end{eqnarray}

This formula is reminiscent of the functionality of a firewall, except
that it is a \emph{static} check. A process witnessing this formula
will behave as though it were behind a firewall admitting only access
to the ports in $\quotep{\phi}$ without the need for the additional
overhead of the watchdog machinery.

\subsection{The size of a formula}

We give a recursive definition of the size of a formula, written
$\#(\phi)$, in terms of a measure of the set of processes it
denotes.

% To find this formula we note that we have two mutually
% recursive equations to solve. If we parameterize the size in a
% measure, $\mu$, on the set of processes, then we have

% \begin{mathpar}
%   \#(\phi)(\mu) := \mu(\meaningof{\phi})
% \end{mathpar}

% while if we parameterize a measure on a notion of formula size,
% $\sigma$, we have

% \begin{mathpar}
%   \mu(S)(\sigma) := \Sigma_{P \in S} \sigma(\meaningof{P}_{\phi})
% \end{mathpar}

\begin{definition}[measure]
  For the time being we simply demand a Lebesgue measure, written
  $\mu_{\Proc}$ on the set $\Proc$, and define
  \begin{mathpar}
    \#(\phi) := \mu_{\Proc}(\meaningof{\phi})
  \end{mathpar}
\end{definition}

\begin{definition}[witness]
  TBD
\end{definition}