\section{Results}

\subsection{Hypotheses}

We wish to test the following hypotheses.

\newtheorem{policyWorks}{Hypothesis}

\begin{policyWorks}
	The number of comments a user receives from community managers has no influence on her probability to be active.
	\label{hypothesis:policyWorks}
\end{policyWorks}

The organisations running online communities wish for their users to do certain things, most of which imply being active in the community itself: writing posts and comments. They cannot order them to do so, since users are not on the payroll and remain unanswerable to those organisations. Online community managers are then tasked to prompt users into action without using either monetary incentives or command power. That leaves interaction as the main tool online community managers have at their disposal. Rejecting Hypothesis \ref{hypothesis:policyWorks} would imply that users do, in fact, respond to cues from online community managers.

We expect Hypothesis \ref{hypothesis:policyWorks} to be falsified by data. Moreover, we expect that the coefficient on the number of comments received by the user from community managers, besides being statistically different from zero, is positive. Denote said coefficient by  $\beta_{cmrec}$, and the related variable by $x_{cmrec}$. Check that:

\begin{equation}
	\frac{\partial Pr(A=1|x)}{\partial x_{cmrec}} > 0 \Rightarrow \frac{\partial LOR}{\partial x_{cmrec}} > 0
	\label{marginalProbActive}
\end{equation}

where $LOR$ denotes the logarithm of the log-odds ratio as per equation \ref{eq:logOddsErrors}. Since $x_{cmrec}$ enters equation \ref{eq:logOdds} linearly, equation \ref{marginalProbActive} implies that:

\begin{equation}
	\frac{\partial LOR}{\partial x_{cmrec}} > 0 \Rightarrow \beta_{cmrec} > 0
\end {equation}

\begin{policyWorks}
	Receiving comments from community managers has the same effect on the probability to be active than receiving comments from users that are not community managers.
	\label{hypothesis:policyWorksBetter}
\end{policyWorks}

Online community managers are professionals. They are likely to have better communication skills than the average user, and they certainly have stronger incentives to craft their interaction modes so as to drive users to being more active in the online community. We therefore expect, on average, the effect of incoming communication from online community manager to have a larger positive effect on the probability of becoming active than that of incoming communication from other users. Therefore, we expect Hypothesis \ref{hypothesis:policyWorksBetter} to be falsified by the data.

\begin{policyWorks}
	Receiving an additional comment from a community manager has a large effect on the probability that a user becomes active.
	\label{hypothesis:policyWorksWell}
\end{policyWorks}

The number of comments authored by online community manager is the main policy variable in Edgeryders and other online communities like it. Employing community managers would not make economic sense if their impact on the level of activity in the community was not large enough. Hypothesis \ref{hypothesis:policyWorksWell} is more restrictive than Hypothesis \ref{hypothesis:policyWorks}, as it implies that the policy variable, the number of comments received from community manager, has a marginal effect that is not only statistically distinguishable from zero, but also sizeable. For example, if targeting users with a comment only increased their probability of being active by a few percent, many businesses running online communities would probably reconsider their decision to hire a community manager.

\subsection{Regression}

Table \ref{tab:xtlogit} summarizes the estimation's results.

\def\onepc{$^{\ast\ast}$} \def\fivepc{$^{\ast}$}
\def\tenpc{$^{\dag}$}
\def\legend{\multicolumn{4}{l}{\footnotesize{Significance levels
:\hspace{1em} $\dag$ : 10\% \hspace{1em}
$\ast$ : 5\% \hspace{1em} $\ast\ast$ : 1\% \normalsize}}}
\begin{table}[htbp]\centering
\begin{tabular}{l r @{} l c }\hline\hline
\multicolumn{1}{c}
{\textbf{Variable}}
 & \multicolumn{2}{c}{\textbf{Coefficient}}  & \textbf{(Std. Err.)} \\ \hline
n. comments received from community managers  &  1.250&\onepc  & (0.053)\\
n. comments received from other users  &  0.215&\onepc  & (0.037)\\
n. comments and posts written by ego (lagged)  &  0.126&\tenpc  & (0.066)\\
weeks since creating the account  &  -0.011&\tenpc  & (0.006)\\
n. of posts and comments written by users excluding ego  &  0.037&\onepc  & (0.004)\\
n. of posts and comments written by community managers  &  0.001&  & (0.006)\\
user out-degree (lagged) &  0.028&\fivepc  & (0.013)\\
user in-degree (lagged)  &  -0.003&  & (0.013)\\
user betweenness centrality  &  -107.870&\onepc  & (20.778)\\
user pagerank (lagged) &  -5.592&  & (14.610)\\
user clustering coefficient  &  -0.654&\onepc  & (0.195)\\
network average distance  &  -4.156&\fivepc  & (2.007)\\
network average betweenness centrality at $t-1$  &  -9.260&  & (209.212)\\
network Louvain modularity  &  8.379&\onepc  & (2.520)\\
n. nodes  &  -0.001&  & (0.004)\\
n. edges  &  0.000&  & (0.001)\\
\hline
\onepc = 1\% significant; \fivepc = 5\% significant; \tenpc = 10\% significant.

\end{tabular}
 \caption{Estimation results for the dependent variable $A_{i,t}$}
\label{tab:xtlogit}
\end{table}

The coefficients on the first two variables are positive and highly significant ($p < 0.001$). This supports the conventional wisdom that users of online communities tend to engage with each other: when made the object of comments, they are more likely to become active than when they are not. By implication, our Hypothesis \ref{hypothesis:policyWorks} must be rejected.

To test Hypothesis \ref{hypothesis:policyWorksBetter}, we start by noting that the coefficient on the number of comments received from community managers is larger than that on the number of comments received from other (non-community managers) users. We next run a Wald test on the null hypothesis that the coefficients on our first two variables are identical. The null is strongly rejected ($p < 0.0001$). This, in turn, means we have no support for Hypothesis \ref{hypothesis:policyWorksBetter}.

Ideally, we would like that to support the alternative hypothesis that the difference between the coefficient on the number of comments received by the user from community managers and the coefficient on the number of comments received by other (non-community managers) users be positive. Wald tests, being one-tailed, cannot do that. However, we can deduce the non-negativity of such difference from the signs and values of the estimated coefficients and our functional form in equation \ref{eq:logOddsErrors}. Formally, denote the  coefficient on the number of comments received by other (non-community managers) users by $\beta_{urec}$, and the related variable by $x_{urec}$. Since the transformation from probability to odds is monotonic, we have:

\begin{equation}
	\frac{\partial Pr(A=1|x)}{\partial x_{cmrec}} > \frac{\partial Pr(A=1|x)}{\partial x_{urec}}  \Rightarrow \frac{\partial LOR}{\partial x_{cmrec}} > \frac{\partial LOR}{\partial x_{urec}}
\end{equation}

Applying again equation \ref{eq:logOddsErrors}, and remembering that both $x_{cmrec}$ and $x_{urec}$ enter it linearly, we have:

\begin{equation}
	\frac{\partial LOR}{\partial x_{cmrec}} > \frac{\partial LOR}{\partial x_{urec}}\Rightarrow \beta_{cmrec} > \beta_{urec} \Rightarrow \beta_{cmrec} - \beta_{urec} > 0
	\label{eq:condition4WaldTest}
\end{equation}

So, the difference $\beta_{cmrec} - \beta_{urec}$ cannot be negative. Since we have strongly rejected that it be zero, it follows that it must be strictly positive. The activation effect of receiving comments from online community managers is thus significantly greater than that of receiving comments from ordinary users.

\subsection{Marginal effects}

Regression analysis shows that the activity of community managers does indeed have a positive impact on the probability that the users they engage will become active. This, however, does not tell us how large that impact is. Since online community management is costly, this is likely to be a question of some relevance to an organisation trying to make a decision to invest in it.
Coefficient estimates are a poor indicator of marginal effects, because the logit model we employ here is not linear. The relationship between the linear prediction (as defined by the sum of each coefficient multiplied by the regressor it refers to, and assuming the fixed effects are zero) and the probability that the dependent variable is equal to 1 follows a logistic curve (figure \ref{fig:probActivityLinPredict}). An increase in the number of comments received by community managers has a small effect for small or large values of $x\beta$. When $x\beta$ is between -5 and 5, however, the probability that the user becomes active increases quickly as community managers engage more with the user.

\begin{figure}
	\includegraphics[width=.8 \textwidth]{PrAct_linPred}
	\caption{Predicted probability of Edgeryders users to become active as the linear prediction $x\beta$ based on number of comments received by community managers and other users increases.}
\label{fig:probActivityLinPredict}
\end{figure}

Table \ref{table:dydx} shows a point estimate the marginal effect of comments from the two sources (online community managers vs. other users) on the probability that a user will become active. Neither is significant. This turns out to be an artefact of the computation: the size of the marginal effect is computed fixing the value of regressors at their mean. The means of the variables in question are low. In the average period, the average Edgeryders user received 0.08 comments from online community managers and 0.09 comments from other users. This is consistent with what we know about patterns of human communication, which is sparse and bursty (\cite{holme2012temporal}).

{
\def\onepc{$^{\ast\ast}$} \def\fivepc{$^{\ast}$}
\def\tenpc{$^{\dag}$}
\begin{table}[htbp]\centering
 \begin{tabular}{l c c c}\hline\hline
\multicolumn{1}{c}
{\textbf{Variable}} & {\textbf{$dy/dx$}}  & \textbf{$Std. Err.$} & \textbf{$P > \lvert z \rvert$} \\ \hline
n. comments from community managers  &  .0000959  & .0004127 & 0.816 \\
n.  comments from other users  &  .0000165  & .000071 & 0.816\\
\hline
\end{tabular}
\caption{Marginal effects of the number of comments received by a user (both from community managers and from other users) on the probability of that user to become active. The estimates are computed under the assumption that regressors be fixed at their means.
\label{table:dydx}}
\end{table}
}
A more intuitive approach is to estimate the elasticities of the probability of becoming active with respect to the number of comments received from each source. These are shown in table \ref{tab:elasticities}. They are both highly significant. In absolute terms, they are both small, but quite different. Receiving an extra comment by a community manager increases the probability that a user will become active by 10\%; but receiving one from another user will increase it by less than 2\%.

{
\def\onepc{$^{\ast\ast}$} \def\fivepc{$^{\ast}$}
\def\tenpc{$^{\dag}$}
\def\legend{\multicolumn{4}{l}{\footnotesize{Significance levels
:\hspace{1em} $\dag$ : 10\% \hspace{1em}
$\ast$ : 5\% \hspace{1em} $\ast\ast$ : 1\% \normalsize}}}
\begin{table}[htbp]\centering
 \begin{tabular}{l c c c}\hline\hline
\multicolumn{1}{c}
{\textbf{Variable}} & {\textbf{$ey/ex$}}  & \textbf{$Std. Err.$} & \textbf{$P > \lvert z \rvert$} \\ \hline
n. comments from community managers  &  .1024245\onepc  & .0043113 & 0.000 \\
n. comments from other users  &  .01882\onepc  & .0032799 & 0.000\\
\hline
\end{tabular}
\caption{Elasticities of the probability that a user becomes active with respect to the number of comments received from community managers and from other users. The estimates are computed under the assumption that regressors be fixed at their means.
\label{tab:elasticities}}
\end{table}
}

Significant elasticities do not, per se, guarantee that the action of community managers will produce a large marginal effect in the online communities, unless the extra action (the differential increase in the regressor) is applied in a region of the distribution where the probability of the user being active is already reasonably high. We therefore turn to computing the marginal effect of the number of comments received from moderators on the probability that a user becomes active. Before we can proceed, however, we need an adjustment. In the model estimated so far, the mean of the predicted probability of a positive outcome (defined as the user
being active in the period) does not coincide with the proportion of users actually active across the dataset (table \ref{tab:flaw}).

\begin{table}[htbp]\centering
 \begin{tabular}{l c c c}\hline\hline
\multicolumn{1}{c}
{\textbf{Variable}} & {\textbf{Obs}}  & {\textbf{Mean}} & {\textbf{Std. Dev.}} \\ \hline
$prob$  &    84262   & .0023325  &  .0410453 \\
$active$  &  84262 &   .0181577  &  .1335221 \\
\hline
\end{tabular}\caption{Descriptive statistics for the probability of users to become active as predicted by the model ($prob$) and users actually being active ($active$).
\label{tab:flaw}}
\end{table}

This is caused by fact that the mean of the fixed effects $c_i$ is nonzero. To correct this, we need to add to the model a constant $\alpha$ such that:

\begin{equation}
	 \frac{e^{mean(X_{i,t}\beta) + \alpha}}{1 + e^{mean(X_{i,t}\beta) + \alpha}} = \frac{\sum_{i = 1}^N \sum_{t = 1}^T active_{i,t}}{NT}
	 \label{eq:equationAlpha}
\end{equation}

In equation \ref{eq:equationAlpha}, $active_{i,t}$ takes value 1 if user $i$ is active at period $t$, and 0 otherwise; the $x_{i,t}\beta$ are the ones already estimated. The corrected model's linear predictions are unbiased, allowing us to compute marginal effects correctly. The right-hand side of equation \ref{eq:equationAlpha} is identical to the mean of the variable $active$ in table \ref{tab:flaw}. Replacing the appropriate values from table \ref{tab:flaw} yields:

\begin{equation}
	\frac{e^{-9.52 + \alpha}}{1 + e^{-9.52 + \alpha}} = 0.18
\end{equation}

\begin{equation}
	\alpha \simeq 8
	\label{eq:valueAlpha}
\end{equation}

We can now replace equation \ref{eq:valueAlpha} in to equation \ref{eq:equationAlpha} and proceed to estimate its marginal effects. Start by noting that, differentiating the right-hand side of \ref{eq:equationAlpha} with respect to $X$ yields:

\begin{equation}
	\frac{\partial Pr(A=1|X)}{\partial X} = \beta \frac{e^{X \beta + \alpha}}{(1 + e^{X \beta + \alpha})^2}
\end{equation}

Replace the point estimates for $\beta_{cmrec}$, $\beta_{urec}$ and $\alpha$ to yield marginal effects of receiving one additional comments from, respectively, community managers and other users on the probability of being active in the period:

\begin{equation}
	\frac{\partial Pr(A=1|X)}{\partial x_{cmrec}} = \beta_{cmrec} \frac{e^{ \beta_{cmrec} + \alpha}}{(1 + e^{x_{cmrec} \beta_{cmrec} + \alpha})^2}
	\label{eq:marginal_x_cmrec}
\end{equation}

\begin{equation}
	\frac{\partial Pr(A=1|X)}{\partial x_{urec}} = \beta_{urec} \frac{e^{ \beta_{urec} + \alpha}}{(1 + e^{x_{urec} \beta_{urec} + \alpha})^2}
	\label{eq:marginal_x_urec}
\end{equation}

Equations \ref{eq:marginal_x_cmrec} and \ref{eq:marginal_x_urec} allow us to "drill down" into the information contained in Table \ref{table:dydx} and study the marginal effects of receiving one additional comment after the same user has already received zero, one or more comments. To do so, we compute, for each observation and for both $x_{cmrec}$ and $x_{urec}$, the estimate of the marginal effect on the probability of the user being active. These are obtained plugging the observed regressor values, the computed value of $\alpha$ and the point estimates of the coefficient values into, respectively, equations \ref{eq:marginal_x_cmrec} and \ref{eq:marginal_x_urec}. We then repeat the procedure to compute the marginal effect of receiving comments from fellow participants in the Edgeryders community who are not online community managers.

Plotting the frequency density of the two marginal effects shows that the comments of community managers have a greater activating marginal effect than those of non-community managers (Figure \ref{fig:marginalFxHisto}).

\begin{figure}
	\includegraphics[width=.8 \textwidth]{Marginal_fx_histo}
	\caption{Frequency density of the marginal effect on the probability of being active of receiving a comment from online community managers and from other users in Edgeryders. Computed using point estimates for coefficient and observed values of regressors.}
\label{fig:marginalFxHisto}
\end{figure}

Marginal effects of regressors depend obviously on the values of regressors themselves. Receiving comments has generally decreasing returns on the probability of users becoming active. Fixing the value of all other regressors at their respective means and computing the marginal effects of both $x_{cmrec}$ and $x_{urec}$ as the number of comments received by users increase, we obtain the graph of Figure \ref{fig:marginalFxScat}. The marginal effect of $x{cmrec}$ is initially high (close to 0.3), but then decreases monotonically and quickly. That of $x{urec}$ starts at a much lower level (about 0.05), rises slightly to reach a maximum for $x{urec} = 2$, and then it decreases much more slowly.

\begin{figure}
	\includegraphics[width=.8 \textwidth]{Marginal_fx_scat}
	\caption{Marginal effect on the probability of being active of receiving a comment from online community managers and from other users in Edgeryders. Computed using point estimates for coefficient and mean values of regressors.}
\label{fig:marginalFxScat}
\end{figure}

This finding, illustrated by Figure \ref{fig:marginalFxScat} and Table \ref{tab:marginalProbabilityXcmrecObserved}, indicates support for Hypothesis \ref{hypothesis:policyWorksWell}, but only for low levels of $x_{cmrec}$.

Let's take a closer look, starting with the marginal effect of online community managers first. Tabulating the results according to the number of of comments received in each period by users, we derive Table \ref{tab:marginalProbabilityXcmrecObserved}. For users that have received no comments, the mass of the observations obtains for marginal probability values in or close to the 0.2-0.3 range. As the number of comments increases, predicted marginal probabilities decrease and quickly become indistinguishable from zero.

\begin{table}[htbp]\centering
	\begin{tabular}{c c | c  c  c  c | c}
 		& & \multicolumn{5}{c}{Marginal probability of user being active} \\
		& & \multicolumn{5}{c}{$X$ is at the observed value for each observation} \\
		\hline
		N. comments& & 0 - 0.1&0.1 - 0.2&0.2 - 0.3&over 0.3&Total \\
		\hline
		& mean & .06 & .15 & .26 & .31 & .23 \\
		0 & SE & .02 & .03 & .03 & .00 & .07 \\
		& N & 5,748&15,303&41,028&19,566&81,645 \\
		\hline
		 & mean & .04 & .14 & .25 & .31 & .15 \\
		1 & SE & .03 & .03 & .03 & .00 & .10 \\
		&N&438&308&362&111&1,219 \\
		\hline
		& mean & .03 & .14 & .25 & .31 & .07 \\
		2 & SE & .03 & . 03 & .03 & .00 & .08 \\
		& N&340&91&32&6&469 \\
		\hline
		& mean & .02 & .14 & . 25 & .31 & .04 \\
		3 & SE & .02 & .03 & .00 & .07 \\
		&N&223&8&13&2&246 \\
		\hline
		& mean & .01 & .12 & .25 & .31 & .04 \\
		4 & SE & .02 & .01 & .03 & .00 & .07 \\
		& N & 135 & 4 & 5 & 2 & 146 \\
		\hline
		& mean & .01 & .15 & .25 & .31 & .03 \\
		5 to 10 & SE & .02 & .03 &.03 & .00 & .08 \\
		& N&329&13&22&8&372 \\
		\hline
		& prob & .00 & .16 & .26 & .30 & .02 \\
		11 and above & SE & .01 & .03 & .04 & 0 & .05 \\
		& N &154&6&4&1&165 \\
		\hline
		& mean & .05 & .16 & .26 & .31 &.23 \\
		\textbf{Total} & SE & .03 & .03 & .03 & .00 & .08\\
		& N &7,367&15,733&41,466&19,696&84,262 \\
		\hline
	\end{tabular}
	\caption{Distribution of the observations according to the marginal effect of the user being active in the period with respect to receiving an extra comment from a community manager, and to the number of comments received in the same period. All other regressors enter the prediction function at the value observed for each observation.}

	\label{tab:marginalProbabilityXcmrecObserved}
\end{table}

Table \ref{tab:marginalProbabilityXcmrecObserved} has an interesting implication. The first and second comments any user receives are those with the highest (over .2) marginal effect on her activation in each period. Yet, community managers in Edgeryders write more than two comments to the same user in a relatively high number of instances. Of the 84,262 observations in our dataset, 2,617 are those for which the user received at least one comment from online community managers. Over a third (929) saw the user receiving three or more comments, having no predicted marginal effect on user activity in the period. We interpret these extra comments as exchanges, with community managers and users commenting (presumably) the same content within the same time period. This is not necessarily ineffective behaviour: online community managers might be driven by purposes other than activation, for example helping to give users a good experience of the community.

We now turn to the marginal effect on the probability of being active of comments written by users who are not online community managers. Table \ref{tab:marginalProbabilityXurecObserved} is less informative than Table \ref{tab:marginalProbabilityXcmrecObserved}, given that the predicted marginal effect of $x_{urec}$ is smaller and more clustered around a single value than that of $x_{cmrec}$. Still, we do observe that the predicted marginal effect of the former is greater than .05 about one third of the times when $x_{urec} = 0$. As $x_{urec}$ increases, this proportion decreases, as does the average marginal effect.

\begin{table}[htbp]\centering
	\begin{tabular}{c c | c  c | c}
 		& & \multicolumn{3}{c}{Marginal probability of user being active} \\
		& & \multicolumn{3}{c}{$X$ is at the observed value for each observation} \\
		\hline
		N. comments& & 0 - 0.05&0.05 - 0.1&Total \\
		\hline
		& mean & .03 & .05 & .04 \\
		0 & SE & .01 & .00 & .01 \\
		& N & 56,183&25,716&81,899 \\
		\hline
		 & mean & .02 & .05 & .03 \\
		1 & SE & .02 & .00 & .02 \\
		&N&871 & 192 & 1,063 \\
		\hline
		& mean & .02 & .08 & .02 \\
		2 & SE & .02 & .00 &.02 \\
		& N& 425 & 47 & 472 \\
		\hline
		& mean & .01 & .05 & .01 \\
		3 & SE & .01 & .00 & .02 \\
		&N& 235 & 13 & 248\\
		\hline
		& mean & .01 & .05 & .01 \\
		4 & SE & .01 & .00 & .02 \\
		& N & 136 & 8 & 144 \\
		\hline
		& mean & .01 & .05 & .01  \\
		5 to 10 & SE & .01 & .00 & .01 \\
		& N&309&14& 323 \\
		\hline
		& prob & . & .00 \\
		11 and above & SE & .01 & . & .01\\
		& N &113 & 0 & 113\\
		\hline
		& mean & .03 & .05 & .04 \\
		\textbf{Total} & SE & .01 & .00 & .01\\
		& N & 58,272 & 25,990 & 84262 \\
		\hline
	\end{tabular}
	\caption{Distribution of the observations according to the marginal effect of the user being active in the period with respect to receiving an extra comment from a non-community manager, and to the number of comments received in the same period. All other regressors enter the prediction function at the value observed for each observation.}
	\label{tab:marginalProbabilityXurecObserved}
\end{table}


% !TEX root = developer.tex

\chapter{A Custom Object: Beginning To End}
\label{chapter:custom}

Suppose we have brilliant design for a new topology we want to test.
We want to run a simple test \emph{without} having to modify the \sstmacro build system.
We can create a simple external project that links the new topology object to \sstmacro libraries.
The Makefile can be found in \inlineshell{tutorials/programming/topology}.
You are free to make \emph{any} Makefile you want.
After \sstmacro installs, it creates compiler wrappers \inlineshell{libsst++} and \inlineshell{libsstcc}
in the chosen \inlineshell{bin} folder.
These are essentially analogs of the MPI compiler wrappers.
This configures all include and linkage for the simulation.

We want to make an experimental topology in a ring.
Rather than a simple ring with connections to nearest neighbors, though, we will have ``express'' connections that jump to switches far away.

We begin with the standard typedefs.

\begin{CppCode}
#include <sstmac/hardware/topology/structured_topology.h>

namespace sstmac {
namespace hw {

class XpressRing :
  public StructuredTopology
{
 public:
  typedef enum {
    up_port = 0,
    down_port = 1,
    jump_up_port = 2,
    jump_down_port = 3
  } port_t;

  typedef enum {
    jump = 0, step = 1
  } stride_t;

\end{CppCode}
Packets can either go to a nearest neighbor or they can ``jump'' to a switch further away.
Each switch in the topology will need four ports for step/jump going up/down.
The header file can be found in \inlineshell{tutorials/programm/topology/xpressring.h}.
We now walk through each of the functions in turn in the source in the topology public interface.
We got some functions for free by inheriting from \inlinecode{structured_topology}.

We start with

\begin{CppCode}
XpressRing::XpressRing(SST::Params& params) :
  StructuredTopology(params)
{
  ring_size_ = params.find<int>("xpress_ring_size");
  jump_size_ = params.find<int>("xpress_jump_size");
}
\end{CppCode}
determining how many switches are in the ring and how big a ``jump'' link is.

The topology then needs to tell objects how to connect

\begin{CppCode}
void XpressRing::connectedOutports(SwitchId src, std::vector<connection>& conns)
{
  conns.resize(4); //every switch has 4 connections
  auto& plusConn = conns[0];
  plusConn.src = src;
  plusConn.dst = (src+1) % ring_size_;
  plusConn.src_outport = 0;
  plusConn.dst_inport = 1;

  auto& minusConn = conns[1];
  minusConn.src = src;
  minusConn.dst = (src -1 + ring_size) % ring_size_;
  minusConn.src_outport = 1;
  minusConn.dst_inport = 0;

  auto& jumpUpConn = conns[2];
  jumpUpConn.src = src;
  jumpUpConn.dst = (src + jump_size_) % ring_size_;
  jumpUpConn.src_outport = 2;
  jumpUpConn.dst_inport = 3;

  auto& jumpDownConn = conns[2];
  jumpDownConn.src = src;
  jumpDownConn.dst = (src - jump_size_ + ring_size) % ring_size_;
  jumpDownConn.src_outport = 3;
  jumpDownConn.dst_inport = 2;
}
\end{CppCode}
Each of the four connections get a different unique port number.
We must identify both the outport port for the sender and the input port for the receiver.

To compute the distance between two switches

\begin{CppCode}
int XpressRing::numHops(int total_distance) const
{
  int num_jumps = total_distance / jump_size_;
  int num_steps = total_distance % jump_size_;
  int half_jump = jump_size_ / 2;
  if (num_steps > half_jump) {
    //take an extra jump
    ++num_jumps;
    num_steps = jump_size_ - num_steps;
  }
  return num_jumps + num_steps;
}

int
XpressRing::minimalDistance(
  const coordinates& src_coords,
  const coordinates& dest_coords) const
{
  int src_pos = src_coords[0];
  int dest_pos = dest_coords[0];
  int up_distance = abs(dest_pos - src_pos);
  int down_distance = abs(src_pos + ring_size_ - dest_pos);

  int total_distance = std::max(up_distance, down_distance);
  return num_hops(total_distance);
}
\end{CppCode}
Essentially you compute the number of jumps to get close to the final destination and then the number of remaining single steps.

We are now ready to use our topology in an application.
In this case, we just demo with the built-in MPI ping all program from \sstmacro.
Here every node in the network sends a point-to-point message to every other node.
There is a parameter file in the \inlineshell{tutorials/programming/toplogy} folder.
To specify the new topology

\begin{ViFile}
# Topology
topology.name = xpress
topology.xpress_ring_size = 10
topology.xpress_jump_size = 5
\end{ViFile}


\part{Introduction}
\chapter{Introduction}
\par
The {\bf SPOOLES} package is used to solve two types of
real or complex linear systems:
\begin{itemize}
\item
$AX = Y$ or $(A + \sigma B)X = Y$ where $A$ and B are square.
$A$ and $B$ can be real or complex,
symmetric, Hermitian or nonsymmetric.
The factorization can proceed with or without pivoting for
numerical stability.
The factor matrices can be stored with or without dropping small
entries.
\item
Minimize $\|AX_{*,j} - Y_{*,j}\|_2$ for each column of the solution
matrix $X$ and right hand side matrix $Y$.
This is done by computing a $QR$ factorization of $A$ and then
solving $R^T R X = A^T Y$ or $R^H R X = A^H Y$.
\end{itemize}
In both cases, the linear systems can be permuted to reduce the
fill in the factor matrices.
\par
The {\bf SPOOLES} software is written in an object oriented fashion
in the C language.
Parts of the software run in serial mode,
multithreading using Solaris or POSIX threads, and with MPI.
\par
The software objects are naturally partitioned into three families of
objects.
\par
\begin{center}
\begin{tabular}{|l|l|}
\multicolumn{2}{c}{\bf Utility objects} \\ \hline
{\tt A2} & dense two dimensional array \\
{\tt Coords} & object to hold coordinates in any number of dimensions \\
{\tt DV} & double precision vector \\
{\tt Drand} & random number generator \\
{\tt I2Ohash} & hash table for the factor submatrices \\
{\tt IIheap} & simple heap object \\
{\tt IV} & int vector \\
{\tt IVL} & int list object \\
{\tt Ideq} & simple dequeue object \\
{\tt Lock} & abstract mutual exclusion lock \\
{\tt Perm} & permutation vector object \\
{\tt Utilities} & various vector and linked list utility methods \\
{\tt ZV} & double precision complex vector \\
\hline
\end{tabular}
\end{center}
\par
\begin{center}
\begin{tabular}{|l|l|}
\multicolumn{2}{c}{\bf Ordering objects} \\ \hline
{\tt BKL} & Block Kernihan-Lin algorithm object \\
{\tt BPG} & bipartite graph object \\
{\tt DSTree} & domain/separator tree object \\
{\tt EGraph} & element graph object \\
{\tt ETree} & front tree object \\
{\tt GPart} & graph partitioning algorithm object \\
{\tt Graph} & graph object \\
{\tt MSMD} & multi-stage minimum degree algorithm object \\
{\tt Network} & network object for solving max flow problems \\
{\tt SolveMap} & map of submatrices to processes for solves \\
{\tt Tree} & tree object \\
\hline
\end{tabular}
\end{center}
\par
\begin{center}
\begin{tabular}{|l|l|}
\multicolumn{2}{c}{\bf Numeric objects} \\ \hline
{\tt Chv} & block chevron object for fronts \\
{\tt ChvList} & object to hold lists of {\tt Chv} objects \\
{\tt ChvManager} & object to manager instances of {\tt Chv} objects \\
{\tt DenseMtx} & dense matrix object \\
{\tt FrontMtx} & front matrix object \\
{\tt ILUMtx} & simple preconditioner matrix object \\
{\tt InpMtx} & sparse matrix object \\
{\tt Iter} & Krylov methods for iterative solves \\
{\tt PatchAndGoInfo} &
    modified factors in the presence of zero or small pivots \\
{\tt Pencil} & object to contain $A + \sigma B$ \\
{\tt SemiImplMtx} & semi-implicit factorization matrix object \\
{\tt SubMtx} & object for dense or sparse submatrices \\
{\tt SubMtxList} & object to hold lists of {\tt SubMtx} objects \\
{\tt SubMtxManager} &
       object to manager instances of {\tt SubMtx} objects \\
{\tt SymbFac} & algorithm object to compute a symbolic factorization \\
\hline
\end{tabular}
\end{center}
The {\tt MT} directory contains all the multithreaded methods
and drivers programs.
The {\tt MPI} directory contains all the {\tt MPI} methods and drivers.
The {\tt misc} directory contains miscellaneous methods and drivers.
\par
Each of the following objects that hold numeric entries ---
{\tt A2}, {\tt Chv}, {\tt DenseMtx}, {\tt FrontMtx},
{\tt ILUMtx},
{\tt InpMtx},
{\tt Pencil},
{\tt SemiImplMtx} and
{\tt SubMtx} --- can hold real or complex entries.
An object knows its {\tt type},
{\tt 1} for real (define'd constant {\tt SPOOLES\_REAL})
or
{\tt 2} for complex (define'd constant {\tt SPOOLES\_COMPLEX}).
Since C does not yet have a standard structure for complex numbers,
we have followed the FORTRAN convention of storing the real and
imaginary parts of a complex number in consecutive memory locations.
Internally, we unroll the complex arithmetic into real arithmetic.
The user need not be burdened by this process if (s)he uses the
input/output methods for the different object.
For example,
{\tt DenseMtx\_setRealEntry()} sets an entry of a real dense matrix,
while {\tt DenseMtx\_setComplexEntry()} sets an entry of a complex
dense matrix.
\par
All the heavily used computational tasks have been expanded where
possible into BLAS2 or BLAS3 kernels, for both the real and complex
cases.
There are a multitude of driver programs that test the
functionality of the objects.
A common output of a driver program is a file that can be input
into Matlab to check the errors of the computations.
This convention inspires confidence in the correctness of the
kernel computations.
\par
\section{Software Design}
\label{chapter:softwareDesign}
\par
The {\bf SPOOLES} library is written in the C language and uses
object oriented design.
There are some routines that manipulate native C data types such as
vectors, but the vast bulk of the code is centered around objects,
data objects and algorithm objects.
By necessity, the implementation of an object is through the C {\tt
struct} data type.
We use the following naming convention --- a method (i.e., function)
associated with an object of type {\tt Object} has the form

\centerline{{\it (return value type)}
{\tt Object\_}{\it methodName}{\tt (Object * obj}, $\ldots${\tt )};}

The method's name begins with the name of the object it is
associated with and the first parameter in the calling sequence is
a pointer to the instance of the object.
Virtually the only exception to this rule is the {\it constructor}
method.

\centerline{\tt Object * Object\_new(void) ;}

\noindent
Two objects, the {\tt Chv} and {\tt DenseMtx} objects, have
methods that return the number of bytes needed to hold their data,
e.g.,

\centerline{\tt
int Chv\_nbytesNeeded(int nD, int nL, int nU, int type, int symflag) ;}
\par
Scan the directory structure of the source code and you will notice
a number of subdirectories --- each deals with an object.
For example,
the {\tt Graph} directory holds code and documentation for an
object that represents a graph:
its {\tt doc} subdirectory holds \LaTeX files with documentation;
its {\tt src} subdirectory holds C files that contain methods
associated with the object ;
% (each method is a C function of the form {\tt Graph\_*});
and its {\tt driver} subdirectory holds driver programs to test or
validate some behavior of the object.
\par
The directory structure is fairly flat --- no object directory
contains another --- because the C language does not support
inheritance.
This can be inelegant at times.
For example, a bipartite graph (a {\tt BPG} object)
{\it is--a} graph (a {\tt Graph} object), but instead of {\tt BPG}
inheriting from {\tt Graph} data fields and methods from {\tt Graph},
we must use the {\it has--a} relation.
A {\tt BPG} object contains a pointer to a {\tt Graph} object
that represents the adjacency structure.
The situation is even more cumbersome for the objects that deal
with trees of one form or another: an elimination tree {\tt ETree}
and a domain/separator tree {\tt DSTree} each contain a pointer to
a generic tree object {\tt Tree} in their structure.
\par
Predecessors to this library were written in C++ and
Objective-C.\footnote{The knowledgeable reader is encouraged to
peruse the source to discover the prejudices both pro and con
towards these two languages.}
The port to the present C library was painless, almost mechanical.
We expect the port back to C++ and/or Objective-C to be simple.
\par
Objects are one of two types:
{\it data objects} whose primary function is to store data
and
{\it algorithm objects} whose function is to manipulate some data
object(s) and return new information.
Naturally this distinction can be fuzzy --- algorithm objects have
their own data that may be persistent and data objects can execute
some simple functionality --- but it holds in general.
To be more explicit, data objects have the following properties:
\begin{itemize}
\item
There is a delicate balance between encapsulation and openness.
The C language does not support any private or protected data
fields, so the C {\tt struct} that holds the data for an object
is completely open.
As an example, the {\tt Graph} object has a function to return the
size of and pointers to a vector that contains an adjacency list,
namely
\begin{verbatim}
     void Graph_adjAndSize(Graph *g, int v, int *psize, int **padj)
\end{verbatim}
where the pointers {\tt psize} and {\tt padj} are filled with the
size of the adjacency structure and a pointer to its vector.
One can get this same information by chasing pointers as follows.
\begin{verbatim}
     vsize = g->adjIVL->sizes[v] ;
     vadj  = g->adjIVL->p_ind[v] ;
\end{verbatim}
One can do the latter but we encourage the former.
As an experiment we replaced every instance of
{\tt Graph\_adjAndSize()} with the appropriate pointer chasing
(and a similar operation for the {\tt IVL} object) and achieved
around a ten per cent reduction in the ordering time.
For a production code, this savings might drive the change in code,
but for our research code we kept the function call.
\item
Persistent storage needs to be supported.
% Currently this means file storage, but in the future we expect to
% need to ``bundle'' a data object into a block of storage to be
% passed or communicated to a different processor.
Each data object has eight different methods to deal with file I/O.
Two methods deal with reading from and writing to a file whose
suffix is associated with the object name, e.g., {\tt *graph\{f,b\}}
for a formatted or binary file to hold a {\tt Graph} object.
Four methods deal with reading and writing objects from and to a
file that is already opened and positioned, necessary for composite
objects (e.g., a {\tt Graph} object contains an {\tt IVL} object).
Two methods deal with writing the objects to a formatted file to be
examined by the user.
We strongly encourage any new data object added to the library to
supply this functionality.
\item
Some data objects need to have compact storage requirements.
Two examples are our {\tt Chv} and {\tt SubMtx} objects.
Both objects need to be communicated between processes
in the MPI implementation, the former during the factorization,
the latter during the solve.
Each has a workspace buffer that contains all the information
needed to {\it regenerate} the object upon reception by another
process.
\item
By and large, data objects have simple methods.
A {\tt Graph} object does {\bf not} have methods to find a good
bisector; this is a sufficiently sophisticated function that it
should be implemented by an algorithm object.
The major exception to this rule is that our {\tt FrontMtx} object
{\it contains} the factorization data but also {\it performs} the
factorization, forward and backsolves.
In the future we intend to separate these two functionalities.
For example, one can implement an alternative forward and backsolve
by using methods to {\it access} the factor data stored in the {\tt
FrontMtx} object.
As a second example, massive changes to the storage format,
e.g., in an out-of-core implementation, can be encapsulated in the
access methods for the data, and any changes to the factorization
or solve functions could be minimal.
\end{itemize}
Algorithm objects have these properties.
\begin{itemize}
\item
Algorithm objects use data objects.
Some data objects are created within an algorithm objects method;
these are owned by the algorithm object and free'd by that object.
Data objects that are passed to algorithm objects can be queried
or {\it temporarily} changed.
\item
They do not destroy or free data objects that are passed to them.
Any side effects on the data objects should be innocent, e.g.,
when a {\tt Graph} object is passed to the graph partitioning
object ({\tt GPart}) or the multistage minimum degree object
({\tt MSMD}), on return the adjacency lists may not be in the input
order, but they contain the values they had on input.
\item
Algorithm objects should support diagnostic, logfile and debug output.
This convention is not entirely thought out or supported at present.
The rationale is that an algorithm object should be able to respond
to its environment to a greater degree than a data object.
\end{itemize}
\par
Data and algorithm objects share two common properties.
\begin{itemize}
\item
Each object has four basic methods: to allocate storage for an
object, set the default fields of an object,
clear the data fields of an object,
and free the storage occupied and owned by an object.
\item
Ownership of data is very rigidly defined.
In most cases,
an object owns all data that is allocated inside one of its
methods,
and when this does not hold it is very plainly documented.
For example, the bipartite graph object {\tt BPG} has a data field
that points to a {\tt Graph} object.
One of its initialization methods has a {\tt Graph} pointer in its
calling sequence.
The {\tt BPG} object then owns the {\tt Graph} object and when it
is free'd or has its data cleared, the {\tt Graph} object is free'd
by a call to its appropriate method.
\end{itemize}
\par
By and large these conventions hold throughout the library.
There are fuzzy areas and objects still ``under construction''.
Here are two examples.
\begin{itemize}
\item
We have an {\tt IIheap} object that maintains integer
$\langle$ key, value $\rangle$ pairs in a priority heap.
Normally we think of a heap as a data structure, but another
perspective is that of a continuously running algorithm that
supports insert, delete and identification of a minimum pair.
\item
Our {\tt BPG} bipartite graph object is a data object,
but it has a method to find the Dulmage-Mendelsohn decomposition,
a fairly involved algorithm used to refine a separator of a graph.
At present, we are not willing to create a new algorithm object
just to find the Dulmage-Mendelsohn decomposition, so we leave
this method to the domain of the data object.
The desired functionality, identifying minimal
weight separators for a region of a graph, can be modeled using
max flow techniques from network optimization.
We also provide a {\tt BPG} method that finds this
Dulmage-Mendelsohn decomposition by solving a max flow problem on
a bipartite network.
Both these methods have been superceded by the {\tt Network} object
that contains a method to find a max flow and one or more min-cuts
of a network (not necessarily bipartite).
\end{itemize}
\par
The {\bf SPOOLES} software library is continuously evolving in an
almost organic fashion.
Growth and change are to be expected, and welcomed, but some
discipline is required to keep the complexity, both code and human
understanding, under control.
The guidelines we have just presented have two purposes:
to let the user and researcher get a better understanding of
the design of the library, and to point out some conventions
to be used in extending the library.
\par
\section{Changes from Release 1.0}
\label{section:intro:changes-1.0}
\par
There are two major changes from the first release of the {\bf
SPOOLES} package: we now support complex linear systems, and the
storage format of the sparse factor matrices has changed from a
one-dimensional data decomposition to a two-dimensional
decomposition.
The factors are now submatrix based, and thus allow a parallel
solve to be much faster than in Release 1.0.
\par
In the first release, all numeric objects had a {\tt `D`} as the
leading letter in their name, e.g., {\tt DA2}, {\tt DChv}, etc.
A natural way to implement complex data types would be to write
``parallel'' objects, e.g., {\tt ZA2}, {\tt ZChv}, etc, as is done
in LINPACK and LAPACK for subroutine names.
However, a {\tt DA2} and {\tt ZA2} object share so much common code
that it is a better decision to combine the real and complex
functionality into one object.
This is even more pronounced for the {\tt FrontMtx} object where
there is virtually no code that is dependent on whether or not the
matrix is real or complex.
\par
Virtually no new work has been done on the ordering objects and methods.
Their algorithms were state of the art two years ago, but a recent
comparison with the {\bf EXTREME} \cite{hr98-msndtalk}
and {\bf METIS} \cite{karypis98metis} packages
on a large collection
of finite element problems shows that the {\bf SPOOLES} orderings
are still competitive.
\par
The serial, multithreaded and MPI code has been modified to force
greater sharing of code between the environments.
``What'' is done is identical in the three cases.
The multithreaded and MPI codes share the same ``choreography'',
in other words, who does what and how.
The main differences between multithreaded and MPI
are that the data structures are global versus local,
and that explicit message passing is done in the latter.
This common structure of the codes has a nonzero impact on the speed
and efficiency of the individual codes, but the gains from a common
code base are well worth the cost.
\par
The MPI methods have been extensively reworked since the first
release.
A number of bugs and logic errors have been detected and fixed.
The code appears to be more robust than the first release.
\par
\section{Changes from Release 2.0}
\label{section:intro:changes-2.0}
\par
Release 2.2 is partly a maintenance release.
Some bugs were found and fixed in the MPI factors and solves.
Some minor new methods were added to the
{\tt DenseMtx}, {\tt FrontMtx}, {\tt InpMtx}
and {\tt Utilities} directories.
The multithreaded methods and drivers have been removed from the
{\tt FrontMtx} directory and placed in a new {\tt MT} directory,
much like the {\tt MPI} methods have their own directory.
\par
Some new functionality has been added.
\begin{itemize}
\item
There are now multithreaded
and distributed matrix-matrix multiply methods.
See the {\tt MT} and {\tt MPI} directories.
\item
The {\tt FrontMtx} object now supports more robust reporting of
errors encountered during the factorization.
There is one additional parameter in the factorization calling
sequences, an error return that signals that the factorization has
failed.
\item
In response to customer requirements, we have added some
``patch-and-go'' functionality to the sparse $LU$ and $U^TDU$
factorizations without pivoting.
There are applications in optimization and structural analysis
where pivoting is not necessary for stabilty, but where the
location of small or zero pivots on the diagonal is meaningful.
Normally the factorization would be ustable or stop, but special
action is taken, the factors are ``patched'' and the factorization
continues.
\par
There is a new {\tt PatchAndGoInfo} object that encapsulates the
``patch-and-go'' strategy and gathers optional statistics about the
action that was taken during the factorization.
This object is attached to the {\tt FrontMtx} object which passes
it unchanged to the {\tt Chv} object that performs the
factorization of each front.
If the user does not need this functionality, no changes are
necessary to their code, i.e., no calling sequences are affected.
\item
New MPI broadcast methods for the {\tt Graph}, {\tt IVL} and
{\tt ETree} objects have been added to the library.
\item
The {\tt Iter} directory contains the following Krylov accelerators
for the iterative solution of linear systems:
Block GMRES, BiCGStab, conjugate gradient and transpose-free QMR.
Each is available in both left- and right-preconditioned forms.
The preconditioner that these methods use is a {\tt FrontMtx}
object that contains a drop tolerance approximate factorization.
The {\tt ILUMtx} object contains a simple vector-based drop
tolerance factorization object.
(The {\tt FrontMtx} approximate factorization is submatrix-based in
both its data structures and computational kernels, and supports
pivoting for numerical stability, which the {\tt ILUMtx} object
does not.)
We have not written Krylov methods that use the {\tt ILUMtx}
object, but it would be simple to replace the {\tt FrontMtx}
preconditioner with the {\tt ILUMtx} preconditioner.
\item
The {\tt SemiImplMtx} object contains
a {\it semi-implicit} factorization,
a technique that can require less storage and solve operations than
the present explicit factorization. It is based on the equation
$$
\left \lbrack \begin{array}{cc}
A_{0,0} & A_{0,1} \cr
A_{1,0} & A_{1,1}
\end{array} \right \rbrack
=
\left \lbrack \begin{array}{cc}
L_{0,0} & 0 \cr
L_{1,0} & L_{1,1}
\end{array} \right \rbrack
\left \lbrack \begin{array}{cc}
U_{0,0} & U_{0,1} \cr
 0 & U_{1,1}
\end{array} \right \rbrack,
=
\left \lbrack \begin{array}{cc}
L_{0,0} & 0 \cr
A_{1,0}U_{0,0}^{-1} & L_{1,1}
\end{array} \right \rbrack
\left \lbrack \begin{array}{cc}
U_{0,0} & L_{0,0}^{-1}U_{0,1} \cr
 0 & U_{1,1}
\end{array} \right \rbrack.
$$
A solve of $AX = B$
with the explicit factorization does the following steps
\begin{itemize}
\item solve $L_{0,0} Y_0 = B_0$
\item solve $L_{1,1} U_{1,1} X_1 = B_1 - L_{1,0} Y_0$
\item solve $U_{0,0} X_0 = Y_0 - U_{0,1} X_1$
\end{itemize}
while an implicit factorization has the following form.
\begin{itemize}
\item solve $L_{0,0} U_{0,0} Z_0 = B_0$
\item solve $L_{1,1} U_{1,1} X_1 = B_1 - A_{1,0} Z_0$
\item solve $L_{0,0} U_{0,0} X_0 = B_0 - A_{0,1} X_1$
\end{itemize}
The difference is that the semi-implicit factorization stores
and computes with
$A_{1,0}$ and $A_{0,1}$ instead of $L_{1,0}$ and $U_{0,1}$,
(this can be a modest savings in storage and operation count),
and performs two solves with $L_{0,0}$ and $U_{0,0}$
instead of one.
This technique works with either a direct or approximate
factorization of $A$.
The semi-implicit factorization is constructed via a
post-processing of any factorization computed by the {\tt
FrontMtx} object.
\end{itemize}