\chapter[A RL Approach for Data Delivery Optimization]{A Reinforcement Learning Approach for Data Delivery Optimization in Multi-Robot Network Patrolling}
\label{rl}
This chapter aims to illustrate the feasibility of a \emph{reinforcement learning} approach to optimizing the data delivery in MRP. The $\mathit{DynFloR}$ method showcased in \emph{Chapter 2} lacks one key asset in order to be fully reliable, that being \emph{learning} how to react in uncertain conditions and dynamic environments.

When thinking about the nature of learning, perhaps several examples of instances come up in our minds when we learn something simply by interacting with an environment, and receiving a positive or negative reward when making a decision. One issue encountered in \emph{RL} modelling of complex environments is the exponential pool of states in which the environment can be found, and the large number of actions that the robots can perform at any given time.

\par The study conducted in this chapter consists in a new approach, based on \emph{Q-learning}, under the assumptions defined in \emph{Chapter 2} (i.e. the environment is deterministic, the network of robots is centralized and offline, and the robots have periodic meetings). Experiments and simulations carried out on several synthetic and randomly generated examples prove that the \emph{RL} approach for the data delivery optimization in MRP is feasible.

\par The rest of the chapter is structured as follows. The \emph{reinforcement learning} approach for data delivery optimization in a deterministic setting of the MRP is introduced in Section 3.1. Section 3.2 is concerned with the advantages and limitations of the proposed model, while Section 3.3 contains the conclusions of the chapter and possible future developments of this work.
%RL MODEL
\pagebreak

\section{The proposed RL methodology}

This section introduces a centralized and offline \emph{RL} approach for data delivery optimization in a deterministic setting of the MRP, being applicable in a deterministic setting of DMRP, with minimal improvements.

\subsection{The DronemRL model}\label{rlmodel}

The problem approached in this chapter is the same problem defined in \emph{Chapter 2}, so we will focus solely on the \emph{reinforcement learning model}.
\subsubsection{The action model}

For this specific problem an action was defined to be a list $X=[x_{1}, x_{2},\dots,x_{k}]$, where $k=\frac{n(n-1)}{2}$. There are $\frac{n(n-1)}{2}$ elements in our list, because we have  $\binom{n}{2}$ ordered pairs of robots $(R_{i}, R_{j})$ with $i < j$.
That being said an element at any given position $p, 1 \leq p \leq k$ in our action list describes the quantity of data exchanged by robots $(R_{i}, R_{j})$, $i < j$, this quantity is bounded by robot's $MaxCapacity$ meaning that there cannot be transfers between $(R_{j}, R_{i})$ if $j > i$, so in order to address this problem the domain of values for $x_{i} \in X$ is $\{0, 1,\dots, 2\cdot MaxCapacity\}$.
Therefore $x_{i} \in X$ has the following meaning:

$$
x_{i} = \begin{cases}
    \text{transfer $x_i$ data from $R_i\rightarrow R_j$}, & 0\leq x \leq MaxCap \\
     \text{transfer $x_i - MaxCap$ data from $R_j\rightarrow R_i$}, & MaxCap < x\leq 2\cdot MaxCap\\
    \end{cases}
$$

\noindent Every action of the environment can be thought to be an element of the following cartesian product:
$$ A = \varprod_{i=1}^{\frac{n(n-1)}{2}} A_i$$
where $A_{i}=\{0, 1, 2,..., 2\cdot MaxCapacity\}$ meaning that there are  $(2\cdot Maxcapacity)^{\frac{n(n-1)}{2}}$ actions in total.
Thus it can already be seen that the action space is extremely large. For example if we have $10$ robots and \emph{MaxCapacity} $= 10$, we would have $20^{45}$ actions. This problem is addressed in \cite{dulacarnold2015deep}, by having an actor-critic model, where the actor applies actions and the critic learns to generate actions in some proximity of the current action using some defined mathematical norm. Even though this approach solves the problem of very large action spaces, there is still the problem that at any given time there is a small number of valid actions to choose from and most of the time the critic will generate invalid values, hence determining the model to apply only invalid actions.

\par In order to address the issue of large state and action spaces the classic Q-learning algorithm, from Section \ref{qLearningAlgo} was modified, to initialize and update the Q-table in a dynamic way (i.e. $states$ are added in the Q-table only when they are encountered, not at the beginning of the algorithm) and variation of the Q-learning algorithm will be called \emph{DronemRL}.

\subsubsection{The state model}
For the state model of the environment, it is enough to know the quantity of data each robot holds, the current time step and the position of each robot.
Therefore, a state is represented as a list $S=[s_1, s_2, \dots, s_{2n + 1}]$, where elements $[s_1, \dots, s_n]$ represent the quantity of data of each robot, $s_{n + 1}$ is the current time step and $[s_{n + 2}, \dots ,s_{2n + 1}]$ represent the current position of each robot.

\subsubsection{The reward function}
After trying several types of reward functions, the one which proved to be the best in practice is defined below:

\begin{itemize}
  \item If the action is invalid (i.e. $R_{j}$ cannot receive the data transferred by $R_{i}$ or $R_{i}$ does not have enough data in its memory), or there is an invalid meeting return a penalty reward of $invalidActionReward=-10^{7}$.
  \item If the chosen action is to do nothing when it is not the case (i.e. the robots can exchange data) reutrn $\frac{2}{3}\cdot invalidActionReward$.
  \item Else add  $dataTrasferred$ to the reward value for each transfer to $R_{0} \text{ (the sink)}$ and  $1$ for any other valid transfer.
\end{itemize}

The reward function is defined in such a way that if our model chooses an invalid action, it receives a large negative reward in order to adjust the $Q(s,a)$ value for that action in that particular state, thus next time when our model is in state $s$ this invalid action won't be chosen. Also the reward function encourages the model to select actions which facilitate transfer of large amount of data to our \emph{sink}. In \emph{RL} the reward function is the core element of $learning$ because it must be defined in a way that it emphasizes the environment characteristics. The best known reward function \cite{rsab} is the one which gives $-1$ reward for every action and $0$ for terminal actions, it is guaranteed that the model will learn to choose the best actions in every state, but the drawback is that the learning time required for such a reward function is tremendously large, therefor impractical for many complex environments such as ours.

\subsubsection{$DronemRL$ algorithm}\label{rlalgo}
After constructing the environment having the properties stated in Section \ref{rlmodel} we train the reinforcement learning model for a number of \emph{episodes}. For every episode of the training process we apply the update rule defined in Section \ref{qLearningAlgo}, based on \emph{Bellman's equation}. One important aspect to take into consideration about this algorithm is that, we do not initialize the entire Q-table (i.e. a matrix storing every possible $(state,action)$ pairs), instead we initialize the Q-table to be empty and only add new states and actions dynamically when we encounter them.

The pseudocode for the dynamic Q-table update is presented in Algorithm \ref{alg:qUpdate} while the DronemRL algorithm is given in Algorithm \ref{alg:dronemrl}.

\begin{algorithm}[!htb]
\caption{Function $UpdateQTable$}\label{alg:qUpdate}
\begin{algorithmic}
\Function{$UpdateQTable$}{$Q, state, action, reward, newState, \alpha, \gamma$}
\Ensure{Dynamically updates the Q-table based on \emph{Bellman's equation} }
\If{$newState \notin Q$}
    \State{//if the new state is not in the Q-table add it}
    \State{$addStateInQTable$($newState$)}
\EndIf
\State{//get the maximum Q-value of the new state}
\State{$maxQFuture \leftarrow max$($Q(newState)$)}
\State{//get the current Q-value}
\State{$currentQValue \leftarrow Q(state, action)$}
\State{//update the Q-table}
\State{$Q(state,action) \leftarrow Q(state, action) + \alpha \cdot (reward + \gamma \cdot maxQFuture)$}
\EndFunction

\end{algorithmic}
\end{algorithm}

\begin{algorithm}[!htb]
\caption{Algorithm $DronemRL$} \label{alg:dronemrl}
\begin{algorithmic}
\State{\textbf{Algorithm} DronemRL is:}
\State{//determines the optimal policy for the robots data delivery}
\Require{$env, nEpisodes, \epsilon, decay, maxNumOfSteps, \alpha, \gamma$ {\rmfamily - the input parameters for the $DronemRL$ algorithm}}
\Ensure{returns the Q-table containing Q(s,a) values as described in Section \ref{qLearningAlgo} }
\State {//initialize empty Q-table}
\State{$Q \leftarrow emptyQTable$()}
\State{//applies the Q-learning algorithm}
\For{$episode \gets 0, nEspisodes$}
\State{//sets the reward for current episode to 0}
\State{$episodeReward \leftarrow 0$}
\State{//the current episode is not done}
\State{$done \leftarrow false$}
\State{//sets the environment to its initial state}
\State{$state \leftarrow reset$($env$)}

\Repeat
\State{$action \leftarrow selectAction$($\epsilon$,$state$)}
\State{//applies action to the environment}
\State{$actionResult \leftarrow step$($action$)}
\State{//gets the new state, the reward of applying the action and if the new state is terminal }
\State{$newState \leftarrow getState$($actionResult$)}
\State{$reward \leftarrow getReward$($actionResult$)}
\State{$done \leftarrow getDone$($actionResult$)}

\State{$episodeReward \leftarrow episodeReward + reward$}

\State{//if the episode is not done (i.e. we are not in a terminal state, and we have not reached the maximum number
of steps per episode}
\If{$done \neq true$}
\State{//updates the Q-table}
\State{$UpdateQTable$($Q$, $state$, $action$, $reward$, $newState$, $\alpha$, $\gamma$)}
\Else
\State{$time \leftarrow getTime$($newState$)}
\If{$time = maxNumOfSteps$}
\State{//if we reached the end of the episode because we've used the maximum number of steps, penalize the action}
\State{$Q(state, action) \leftarrow -1$}
\Else
\State{$Q(state, action) \leftarrow reward$}
\EndIf
\EndIf
\State{$state \leftarrow newState$}
\Until{$done=true$}
\EndFor
\end{algorithmic}
\end{algorithm}

\newpage
\section{Results and discussion}
In this section the results obtained using the proposed algorithm are showcased. All operations involving the manipulation of Q-tables were performed using the Python Numpy \cite{numpy} numerical computation library.
\par The algorithm was tested against several randomly generated environments having 3 robots with the same $MaxCapacity=10$ for each robot. Figure \ref{minReward} and Figure \ref{maxReward} show the evolution of the minimum and maximum reward per $50$ episodes for a total number of $50000$ episodes.
\begin{figure*}[!htb]
\begin{minipage}{0.5\linewidth}
\centering
  \includegraphics[scale=0.5]{Figures/minsBigEnv.png}
  \caption{Results for $3$ robots and $InitialMemory=4$}
  \label{minReward}
\end{minipage}
\hfill
\begin{minipage}{0.5\linewidth}
\centering
  \includegraphics[scale=0.5]{Figures/maxBigEnv.png}
  \caption{Results for $3$ robots and $InitialMemory=4$}
  \label{maxReward}
\end{minipage}
\end{figure*}

\par It can be seen from the figures above that the algorithm reaches convergence well before 50000 episodes. The Q-learning model quickly identifies the optimal data transfer policy between robots. Given the fact that the initial memory of the robots is $4$ and the $MaxCapacity$ for all robots is $10$, one robot $R_i$ can accommodate all the memory initially contained in the memory of $R_j$, explaining the rapid convergence of our algorithm on this particular case.
\par However, tests were ran on more complex environments, Figure \ref{allRewards} depicts the minimum, maximum and the average reward per $50$ episodes on an environment having 3 robots, with $MaxCapacity = 9$, but having $initialMemory = 7$. In this case the transfer policy is more complex because when two robots meet for the first time they cannot exchange all their data, as they need to figure out that they can only send $2$ units of data between them.

\begin{figure*}[!htb]
\centering
\includegraphics[scale=0.7]{Figures/plotBigEnv.png}
\caption{Results for 3 robots and $MaxCapacity=9$, $InitialMemory=7$}
\label{allRewards}
\end{figure*}

\par It can be observed that the algorithm converged slowly on this more complex environment and the main factor which contributes to the convergence is the ratio between $MaxCapacity$ and $InitialMemoroy$. Therefore, the following question arises: Given $P=\frac{InitialMemory}{MaxCapacity}$ will the algorithm eventually converge? In order to provide an answer, several tests were made with different ratios and results can be seen in Figure \ref{convergenceFig}

\begin{figure*}[!htb]
\centering
\includegraphics[scale=0.7]{Figures/convergenceGraph1.png}
\caption{Convergence graph with respect to $P$}
\label{convergenceFig}
\end{figure*}

As expected, Figure \ref{convergenceFig} reveals that eventually our algorithm convergence independent of the ratio $P=\frac{InitialMemory}{MaxCapacity}$ with linear dependence on on some intervals.

\subsection{Comparison to related work}
As it was stated in Section \ref{compRelatedWordkDynFloR}, no approaches for optimizing data delivery in MRP were found, but techniques for similar problems will be discussed.

Because classical RL algorithms are limited due to the exponential number of states and actions, we will focus on approaches oriented on \emph{DeepQ-learning}. In Deep Q-learning instead of having a Q-table in memory for storing the Q-values for every pair $(s,a)$ where $s \in S$ - the set of total States and $a \in A$ - set of total actions, we have a neural network which will $predict$ our QValues as seen in Figure 3.1.

\begin{figure*}[!htb]
\centering
\includegraphics[scale=0.19]{Figures/QLearningvsDQN.jpg}
\caption{Classic Q-learning vs Deep Q-learning}
\label{fig:qlearningvsDQN}
\end{figure*}

\par One widely used Deep Q-learning (DQN) architecture is the one presented in \cite{dulacarnold2015deep}, also called Deep Determnistic Policy Gradient (DDPG), based on the actor critic model discussed earlier. This architecture is based on two Neural Networks, one is the actor and the other one is the critic. These networks compute action predictions for the current state and generate a temporal-difference error signal at each step. The input of the actor network is the current state, while the output is a single real value representing an action chosen from a \emph{continuous} or large discrete action space. The output of the critic is simply the Q-value of the current state and action supplied by the actor. The deterministic policy gradient theorem \cite{dpgTheorem} provides the update rule for the weights of the actor network while the critic network is updated from the gradients obtained from the temporal difference error signal.
While this approach gives good results it is often hard to find a suitable modelling for the MRP problem because of the high complexity of the environment. \\
\par Another way of tackling the training in environments with large number of actions and states is using Multi Agent approaches (every agent develops its own policy and learns how to cooperate with others) even though algorithms using this approach are notoriously unstable to train. However the OpenAI research team managed to come up with a feasible and robust approach \cite{lowe2017multiagent} by extending the actor-critic model. In the Multi-Agent DDPG (MaDDPG) OpenAI adapted the above DDPG algorithm for multi agent environments. It uses a decentralised actor, centralised critic training but during execution agents select actions relying only on their own state. This helps in easing the training as the environment becomes more stationary for each agent and the number of states and actions for each agent decreases exponentially. A diagram of the algorithm can be found in Figure \ref{fig:maddpg} \cite[Figure-1]{lowe2017multiagent}

\begin{figure*}[!htb]
\centering
\includegraphics[scale=0.4]{Figures/maddpg.jpg}
\caption{Multi-Agent DDPG.}
\label{fig:maddpg}
\end{figure*}

\section{Future work}
In this chapter we have investigated the \emph{reinforcement learning} approach for the data delivery optimization problem in a simplified setting for the \emph{multi-robot patrolling} problem in which the environment is deterministic. Also we have introduced our centralized and offline approach whose goal is to maximize the quantity of data which arrives to our station (\emph{sink}). The aim of the above study was to prove that a \emph{reinforcement learning approach} is feasible.
\par Future work will include more rigorous testing and increase in performance. We also aim to transform the centralized system into a multi-agent system by using the approach defined in \cite{zhang2019multiagent}, this would reduce the state and action space exponentially, while increasing the performance of the system. Another enhancement which can be added to our approach is integrating the DDPG algorithm discussed earlier, which would certainly increase the performance, but our ultimate goal is to use the MaDDPG approach as it is considered to be state of the art (SOTA) when it comes to multi agent reinforcement learning. Using the MaDDPG would be suitable for the Data Delivery Optimization problem in the MRP, having each robot learn its own policy while learning to cooperate with others in training, with minimal state and action spaces, further improving the work done by \emph{DynFloR}.


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{tikz}

\title{Module System}
\author{Fin}
\date{September 2021}

\begin{document}

\maketitle

\section{Introduction}
The module system I'll describe here has only one goal: Scalability.
I'll say scalability for a part of the codebase is achieved when the amount of context that a programmer needs to be aware of when working on that part is independent of the scale of the codebase.
A part of the codebase being scalable is defined as that part being in a comprehensible state, which is achieved by keeping the context around it to a minimum, there are possibly some other factors that might make it harder to comprehend the codebase, but only a too large amount of context can make it impossible.
It is important to note that some parts in a codebase might be scalable even when others aren't.
Scaling is defined as making changes to the codebase and not ending up with a codebase where the changed parts are left in an uncomprehensible state.

The terminology I'll use here is a mix of computer science jargon and graph theory jargon. There is a name collision for "Module". In this article a Module means a file. When represented in the graphs the module is a single node.

To get a clear view of the options for which modules import which, all programs will be visualized as directed graphs.
The vertices are modules.
The edges represent one module importing another.
The module at the end of the arrow is the imported module.
The module at the start of the arrow is the module which does the import.
The reason why the module system will be explained using graph theory is because graph theory is a good model to describe and visualize this particular kind of module system. Another reason is that we can also abstract away from the actual module system and just think about graphs when this is necessary. It comes in handy when you want to be sure every form of module system (under graph theory constraints) has been considered.

The method to get to the resulting module system is as follows:

\begin{enumerate}
    \item Take the graph for a module system where different nodes
(representing modules) have a rough constant amount of context unrelatable to the size of the graph. The amount of \emph{possible} edges to be drawn should be as small as possible.

    \item Look at the problems this poses in terms of actually constructing programs under those constraints. Then see how these problems can be solved while still keeping as many constraints on the edges as possible.

    \item Iterate on step 2, if no problem was found then we’ll stop iterating and take the module system as done.
\end{enumerate}

%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%

\section{Building Programs With Trees}

The most restrictive model would be a list. every module imports one other module (in the constraints of a tree).
I don't think this model is worth adding to the module system as the practical use for it seems minimal.
Next most restrictive option would be a binary tree. Every module can import 2 other modules (in the constraints of a tree). For this again, there seems to be no practical use.
We could go on and on checking for N children whether it'd be advantageous to add the model to the module system, but i don't think a specific number will ever give us a good constraint.

The next most restrictive model would be to not have a set number N, but to allow any number N of children. This gives us a regular tree, and I believe this could be useful in parts of a project that are formed by composition. This model is scalable, because no matter how deep a tree gets, each node only has to worry about the effect on its parent, and the only “tools” a node needs to know about are its children and the core programming language itself.

\begin{tikzpicture}

%SPACING
\node[](v0) at (0,8) {};
%SPACING

\node[circle,draw=black](v1) at (3,7) {File\_1\_0};
\node[circle,draw=black](v2) at (0,4) {File\_2\_0};
\node[circle,draw=black](v3) at (6,4) {File\_2\_1};
\node[circle,draw=black](v4) at (3,1) {File\_3\_0};
\node[circle,draw=black](v5) at (6,1) {File\_3\_1};
\node[circle,draw=black](v6) at (9,1) {File\_3\_1};

%SPACING
\node[](v7) at (0,0) {};
%SPACING

\draw[->,very thick] (v1) -- (v2);
\draw[->,very thick] (v1) -- (v3);

\draw[->,very thick] (v3) -- (v4);
\draw[->,very thick] (v3) -- (v5);
\draw[->,very thick] (v3) -- (v6);

\end{tikzpicture}

In this example, File\_2\_1 only needs to worry about its effect on File\_1\_0. In other words: When File\_2\_1 changes the implementation of a function for instance, it only needs to make sure that File\_1\_0 (the only module that can import that function) still behaves correctly. Of course it also needs to make sure the module itself (in this case File\_2\_1) isn’t affected badly, but since that is always a factor, no matter how the module system works, I won’t talk about that, just keep it in mind.

The only tools that File\_2\_1 needs to worry about is what the children modules export (functions, types, macros, ...). And of course the language itself is also a tool available. But then again, the (core) language itself will always be a tool, no matter what the module system looks like. So again I also won’t mention that factor anymore, just keep it in mind.

In theory this module system works great! No matter how large the program already is, the programmer will only need to keep in mind a certain amount of context which is independent of the size of the program. That context is:

\begin{enumerate}
    \item The influence of the developed module on the parent.
    \item The tools provided by the children of the developed module.
\end{enumerate}

%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%

\section{Adding Libraries}

There is something that bugs me about this way of structuring programs though. That is that there is no way of reusing code! In theory the programmer will always only have to worry about the context specified in Section 1, in reality that’s hard to achieve and wrong assumptions will be made. Let me illustrate this with an example:

"\emph{Say a programmer has a strcmp() function. This function takes two strings, and returns 1 if the former is bigger, 0 if they are both equal, and -1 if the former is smaller. Since there is no way to reuse code, the programmer will copy paste this function quite a lot. One day another programmer joins the team, and he too needs to compare two strings. He’s not aware of the already existing code for strcmp() in the codebase, nor should he! Anyways, the programmer wants to know if two strings are equal. If they are equal, he makes his strcmp() return 1, else it returns 0. Now a third programmer joins the team and scrolls through the codebase. He’ll find inexplicable behaviour because when he sees a strcmp in the code, it’ll sometimes mean one thing, and other times it’ll mean another thing. He can’t really see the “correct” implementation of strcmp(), because there isn’t one “correct” implementation, there are multiple!}"

Yes, in theory every programmer should get familiar with the correct context and only that context first before reading code in a module, but in reality assumptions will be made, and it will be a human source of error that could be avoided if there was some way of reusing code. Besides, even though it would theoretically be a “scalable” approach to software construction, it’s unnecessarily tedious and the lack of code reuse makes for incredibly slow progress.

That is why a separate folder, independent from the project structure, should be made. In that folder go all the libraries, and a library can be imported from any module. This way a programmer has two options:

\begin{enumerate}
    \item Add a module to the project. When writing/changing code in this module, the programmer knows what influence it has, namely the parent.
    \item Add a library to the Libraries folder. When writing/changing code in this library, all bets are off regarding the influence of it. Not only is it unknown how many modules are affected in the project, in the future new modules might also be needing those libraries.

\end{enumerate}

This separation makes programmers know what they’re getting into. Are they changing a module? Then the scope of the effect is known and limited, and can be checked for errors. Are they changing a library? Then they are aware that it can affect the entire program.

\begin{tikzpicture}

%SPACING
\node[](v0) at (0,8) {};
%SPACING

\node[circle,draw=black](lib) at (9,7) {Libraries};

\node[circle,draw=black](v1) at (3,7) {File\_1\_0};
\node[circle,draw=black](v2) at (0,4) {File\_2\_0};
\node[circle,draw=black](v3) at (6,4) {File\_2\_1};
\node[circle,draw=black](v4) at (3,1) {File\_3\_0};
\node[circle,draw=black](v5) at (6,1) {File\_3\_1};
\node[circle,draw=black](v6) at (9,1) {File\_3\_1};

%SPACING
\node[](v7) at (0,0) {};
%SPACING

\draw[->,very thick] (v1) -- (v2);
\draw[->,very thick] (v1) -- (v3);

\draw[->,very thick] (v3) -- (v4);
\draw[->,very thick] (v3) -- (v5);
\draw[->,very thick] (v3) -- (v6);

\draw[->,dashed,very thick] (lib) -- (v2);
\draw[->,dashed,very thick] (lib) -- (v5);
\draw[->,dashed,very thick] (lib) -- (v6);

\end{tikzpicture}

In the case of libraries, you also want to structure them a different way. The goal of libraries is to offer the programmer certain functionalities. The problem a programmer can have when adding libraries to the project is that it's hard to find the right library between all your other libraries. That is why libraries should be organized with a file system. The goal is to make it as easy as possible for the programmer to find the correct library. So there must be a way of searching the correct library efficiently. The way that is achieved is by letting folders represent certain areas of functionality. For instance, a "Math" folder represents the area of math functionalities. In those folders are subfolders, which represent more specific areas. Think about having a "Calculus" and a "Discrete Math" subfolder in the "Math" folder. With this heuristic programmers can retrieve the needed functionality as quickly as possible.

The goal of a library folder structure is different from the goal of a program structure.
The library structure is made with the goal that a programmer must find the needed library as quickly as possbile.
The program structure is made with the goal that it represents exactly the architecture of the program.
Without a module system like this one, projects often tend to look like a hybrid between the two. It kind of reflects the structure of the program, but at the same time similar modules are often grouped together to make it clear which modules have a lot in common.

%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%

\section{Adding Clusters}

Yet sometimes you know that you would like a module to be included in a finite (let’s say n) amount of other modules. In that case you’re in a pickle. You now have two options really:

\begin{enumerate}
    \item Rewrite the module n times and make it a child of those n other modules.
    \item Write the code once and put it in the Library folder.
\end{enumerate}

In the first case, you’re back to that same old problem of not reusing code where you really could and should. In the second case, you have made a module accessible to all modules, even those who don’t need access. If we allow the programmer to import other modules without any restriction, then pretty much every module is global, just like libraries, and then we have defeated the point of decoupling certain parts and modules of the program from each other. So we need a way of allowing us to break the restrictions of a tree, while also not ending up with every module being accessible by every other module.

\begin{tikzpicture}

%SPACING
\node[](v0) at (0,8) {};
%SPACING

\node[circle,draw=black](v1) at (3,7) {File\_1\_0};
\node[circle,draw=black](v2) at (0,4) {File\_2\_0};
\node[circle,draw=black](v3) at (6,4) {File\_2\_1};
\node[circle,draw=black](v4) at (3,1) {File\_3\_0};
\node[circle,draw=black](v5) at (6,1) {File\_3\_1};
\node[circle,draw=black](v6) at (9,1) {File\_3\_1};

%SPACING
\node[](v7) at (0,0) {};
%SPACING

\draw[->,very thick] (v1) -- (v2);
\draw[->,very thick] (v1) -- (v3);

\draw[->,very thick] (v3) -- (v4);
\draw[->,very thick] (v3) -- (v5);
\draw[->,very thick] (v3) -- (v6);
\draw[->,very thick] (v4) -- (v5);
\draw[->,very thick] (v5) -- (v2);
\draw[->,very thick] (v6) -- (v5);
\draw[->,very thick] (v1) -- (v4);
\draw[->,very thick] (v1) -- (v4);
\draw[->,very thick] (v1) -- (v5);
\draw[->,very thick] (v4) -- (v2);

\end{tikzpicture}

The solution is clusters. Basically, they are parts of the program where modules can import each other without forming a tree hierarchy. Those parts where that breaking of the rules happens are isolated, Basically there is one parent module acting as the “interface” of the cluster, and the children modules can access each other in this cluster. Children modules normally can’t access each other in a tree, but in a cluster they can. The “interfacing” parent module is a component of another structure, either a cluster or a tree. Note that in these clusters there is still the rule that the graph must be an acyclic directed graph, in order to avoid circular dependencies.

\begin{tikzpicture}

%SPACING
\node[](v0) at (0,8) {};
%SPACING

\node[circle,draw=black](v1) at (3,7) {File\_1\_0};
\node[circle,draw=black](v2) at (0,4) {File\_2\_0};
\node[circle,draw=black](v3) at (6,4) {File\_2\_1};
\node[circle,draw=black](v4) at (3,1) {File\_3\_0};
\node[circle,draw=black](v5) at (6,1) {File\_3\_1};
\node[circle,draw=black](v6) at (9,1) {File\_3\_1};

%SPACING
\node[](v7) at (0,0) {};
%SPACING

\draw[->,very thick] (v1) -- (v2);
\draw[->,very thick] (v1) -- (v3);

\draw[->,very thick] (v3) -- (v4);
\draw[->,very thick] (v3) -- (v5);
\draw[->,very thick] (v3) -- (v6);

\draw[->,very thick] (v4) -- (v5);
\draw[->,very thick] (v5) -- (v6);

\draw[dotted] (2,0) rectangle (10,5);
\draw[dashed] (5,3) rectangle (7,5);

\end{tikzpicture}

This picture shows how the “interfacing parent” is then again part of the tree structure. The cluster encapsulates modules that can use each other in random ways (as long as the result is an acyclic directed graph). The children (File4\_0, File4\_1 and File4\_2) can be an “interfacing” parent of another lower level cluster or a parent from children in another lower level tree structure.

When creating structures with clusters and subclusters there is again only a finite amount of context a programmer needs to know about when working on a module. For the interfacing parent that context is the same as for a parent in a tree. For the children of the interfacing parent that context is:

\begin{enumerate}
    \item The influence on the interfacing parent.
    \item All the other clusterchildren of the interfacing parent.
    \item Whatever subtrees/subclusters the current module is built from.
\end{enumerate}

%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%

\section{Adding Other Graph Restrictions}

So far we've looked at what kind of "subgraphs" we'd need to add. When there was a problem we faced, it became clear we needed to support less restrictive forms of graphs as well. We have covered some problems that were clear at first glance, but the question becomes: are there other problems we looked over that could benefit from another form of subgraph. We've missed many graphs, and we can orient which we missed as follows:
from most to least restrictive:

... - trees - ... - acyclic graphs - ... - graphs

Before trees we could place lists, or binary trees for instance. I see no reason why those structure enforcements would benefit a project. Perhaps there are use cases, but at the same time I feel like adding them to the module system would do more harm by increasing the complexity of the module system than it would benefit by posing more constraints than a tree in a few cases.

Between trees and acyclic graphs, there could be a form of graph such as a tree with the extra possibility of parents also being able to access grandchildren. Again, This kind of enforcement seems not like it would do more good than harm when adding it to the module system.

Another model could be that each node is imported by multiple other nodes, but can only import one node. That would give an inverted tree. This again has seemingly no practical use.

I've covered these examples just to show that there are more graph forms than we considered. I also showed why we didn't take some other graph forms. Even though I rejected the graph forms specified above, I have admittedly not covered all possible graph forms. Who knows there might be another graph form worth adding.

%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%

\section{Trivia}

\begin{enumerate}
    \item This idea of building clusters from clusters is called “Hierarchical Modularity” in graph theory. The model I described for the module system is actually a bit more restrictive, as it allows a cluster to only have one connection to a subcluster. That connected node from the subgraph is here called the "interfacing parent".

    \item Confusingly, modules in graph theory jargon are synonyms for clusters. Here however, the definition of a module is just a node in the graph (a file). The definition of a cluster here is a directed acyclic graph of nodes.

    \item It is not convenient to actually save a project as a graph. So one might start wondering how it could be done. My solution is to have for every node a directory, containing two things:

    \begin{enumerate}
        \item The actual source file with the code.

        \item a directory for the children. If the children are part of the tree structure, the directory is called “TreeChildren”. If the children form a cluster, the directory is called “ClusterChildren”. Either way, that directory contains a bunch of children module nodes. Which in turn are directories containing 2 things, ... You see the recursion here that builds projects...

    \end{enumerate}

    \item The difference in use of directories here compared to how other programming languages have module systems is that the other languages just groups similar modules together, while in this language modules are grouped under restrictions and these groupings make for the structure of the program. With that being said, in the libraries folder, similar modules should be grouped together since the libraries folder doesn’t contain implementations but instead contains only “interfaces” or “declarations”. Hence the libraries folder should aid a heuristic search for the desired library module.

\end{enumerate}

%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%
%---------------------------------------------------------------------------%

\section{Conclusion}

This module system “works” if it is possible to keep adding code with a roughly constant amount of context to keep in mind, where the amount of context is unrelated to the scale of the program. If a module system accomplishes that, then it “works” for the purpose stated here (scalability).

We can also say that every structure of a program can be made with the module system, since by definition the only structure that can't be made by acyclic graphs are cyclic graphs. In programming cyclic graphs introduce problems (circular dependencies), so we don't want that kind of structure for a program ever. So for all the structures a program might need, the current module system works.

Even if it works though, that doesn't mean it is perfect. For instance, if the module system hadn't trees, but only clusters built from subclusters and so on ..., then the amount of context one would need to know about would still be unrelated to the size of the program. However by adding trees we've made it so programmers need to worry about even less context in some areas of code since trees pose more constraints. So even though a module system of only clusters "works", a module system that has also trees besides clusters is better. Following the same rationale for other graph forms it is possible that there exist better module systems that use other graph forms too. We haven't found those graph forms, but we cannot say that it is impossible that they exist.

Besides, there might be value in looking for a module system which is more user friendly in some way. Although the priority here is scalability, there might be other kinds of module systems (perhaps Object Oriented if done correctly) which provide scalability along with other benefits.

Even though I don’t want to restrict the search for better module systems to be in the bounds of graph theory, I also believe that module systems that fit graph theory are an untapped area and so they shouldn’t be ignored.

\end{document}


\documentclass[../notes.tex]{subfiles}

\pagestyle{main}
\renewcommand{\chaptermark}[1]{\markboth{\chaptername\ \thechapter\ (#1)}{}}
\setcounter{chapter}{39}

\begin{document}

\chapter{Quantum Mechanics I: Wave Functions}
\section{The Wave Equation}
\begin{itemize}
    \item \marginnote{8/24:}\textbf{Quantum wave}: The wave-like nature of an electron.
    \item But waves must satisfy a wave equation.
    \begin{itemize}
        \item The classical one didn't work.
        \item In 1925, Schr\"{o}dinger determined that in one dimension, an electron moving in a potential $V$ (the nucleus-electron Coulombic attraction) satisfies
        \begin{equation*}
            -\frac{\hbar^2}{2m}\pdv[2]{\Psi(x,t)}{x}+V\Psi = i\hbar\pdv{\Psi}{t}
        \end{equation*}
        \begin{itemize}
            \item This wave equation wasn't derived in an analogous method to Figure \ref{fig:waveEquationDerivation}, but rather was constructed from conservation of energy.
        \end{itemize}
        \item This wave function has both real and imaginary parts with the inclusion of $i=\sqrt{-1}$.
    \end{itemize}
\end{itemize}

\section{Electrons in the Double Slit Experiment}
\begin{itemize}
    \item Electrons exhibit both diffraction and interference in the double slit experiment.
    \begin{itemize}
        \item $d\sin\theta=m\lambda$ applies when $\lambda$ is the deBroglie wavelength.
        \item Even with only one electron being emitted at a time, you get interference (further confirms wave-like nature of electrons): $\Psi_\text{electron}=\Psi_\text{slit 1}+\Psi_\text{slit 2}$.
        \item Observing which slit an electron goes through removes the interference pattern.
    \end{itemize}
    \item $P\propto\Psi^2$ is independent of time, so you have static charge distributions.
\end{itemize}

\section{Office Hours (Gazes)}
\begin{itemize}
    \item Quantifying optical roughness?
    \item What are $C_V$ and $C_p$? Are they specific to certain gasses?
    \item What is an adiabatic process? Is it a straight conversion from temperature (internal) energy to work? I find this highly unintuitive.
    \begin{itemize}
        \item Heat flow is held constant, so we have the process happening in an insulating container.
        \item Temperature, pressure, and volume are all changing, hence the isotherm-crossing curve in Figures \ref{fig:ottoCycle} and \ref{fig:carnotCycle}.
        \item Running a fire piston backwards does cause the interior to get colder.
        \item Releasing air from a compressed air can is also an adiabatic process (pressure decreases, volume effectively gets much bigger as it enters the room, and can gets cold).
    \end{itemize}
    \item Additive entropies derivation.
\end{itemize}

\end{document}