% To compile single chapters put a % symbol in front of "\comment" and "}%end of comment" below
%    and take off the % symbol from "\end{document" at the bottom line. Undo before compiling
%    the complete thesis
%%Note: You can only use \section command, you are not allowed, per TTU Graduate School, use
%%\subsection command for ghigher level subheadings. At most level 2 subheadings are allowed.

\chapter{Altering Attacks to be Normal}
\label{Altering Attacks to be Normal Chapter}

\section[DASP Image Pre-Processing]{DASP Image Pre-Processing}
\label{DASP Image Pre-Processing Section}

The suite of DASP algorithms presented in Chapter \ref{DASP Algorithm Development Chapter} provide methods for aligning and highlighting signal dimensions, such as modulations and harmonics, that are inherent to URE conducted from electronic equipment.  The resulting alignments can take a variety of shapes within the DASP images, from straight lines to curved lines to crossing radials to signal peaks; however, direct feature extraction from the raw DASP images can be problematic because of scaling, dynamic range, and confounding signal structures.

Multiple image processing and feature extraction techniques were utilized to prepare the DASP images for machine learning analysis.  Two machine learning techniques were explored, a feature vector based approach and an image classification learning method, with both requiring different DASP image pre-processing methods.  Both learning methods required the DASP images to be scaled to limit dynamic range and remove the background noise, as outlined in Section \ref{Image Scaling}.  To further highlight features of interest, such as lines and peaks, the raw DASP images were also processed with thresholding, edge, and line detection algorithms, as described in Sections \ref{Threshold Processing} and \ref{Edge and Line Detection}.  The image classification technique operated directly on the scaled and processed images with no further feature extraction required, whereas the DASP images were further processed through an image segmentation, matrix summing, and statistical measurement process to form feature vectors for the feature vector machine learning algorithms, as outlined in Sections \ref{Image Segmentation} and \ref{Statistical Feature Extraction}.

\subsection[Image Scaling]{Image Scaling}
\label{Image Scaling}

DASP images can suffer from dynamic range issues resulting from the large differences ($10$s of dB) in measured URE signal levels from a single device or across multiple devices, which can further be exacerbated by the bit growth associated with the continual multiplication and summing steps that comprise the DASP processes.  In addition, different parts of a given URE spectrum can vary widely because of broadband noise or narrowband frequency peaks unrelated to the device under test.  To normalize and scale DASP images before further image processing or feature extraction the image scaling algorithm described in Algorithm \ref{alg:imscalealg} was initially applied to all DASP generated images.

\begin{algorithm}
	\caption{Image Scaling Algorithm} \label{alg:imscalealg}
	\scriptsize
	\begin{algorithmic}[1]
		\Require~~
		\Statex $\mathbf{I}$ - Input Image
		\Statex $\mathbf{I_b}$ - Background Image
		\Ensure~~
		\Statex $\mathbf{I_s}$ - Scaled Image Output
		\Statex
		\State $\mathbf{I_s} \gets \mathbf{I} - \mathbf{I_b}$
		\State $\mathbf{I_s} \leq 0 \gets 0$
		\State $\mathbf{I_s}  \gets $ STANDARD DEVIATION FILTER of $\mathbf{I_s}$
		\State $\mathbf{I_s} = \log{\mathbf{I_s}}$
	\end{algorithmic}
\end{algorithm}

The URE collection process outlined in Chapter \ref{URE Data Collection Chapter} recorded $24000$ \textit{off} state captures of which $22800$ were randomly selected and removed from the test and training data set to generate a representative background for DASP device image normalization.  The removal of the extra \textit{None} state captures also balanced the number of samples per class and eliminated any concerns regarding class skew in the learning process.  The $22800$ \textit{off} state DASP images were averaged together to form a background image, $\bf{I_b}$, which was subtracted from the raw DASP images, $\bf{I}$, to generate a scaled image, $\bf{I_s}$.   A rectifying linear transformation was then applied to $\bf{I_s}$ followed by a standard deviation image filter resulting in a localized z-score across the entire image.  Finally, a logarithm was applied to each pixel of entire image to further reduce the dynamic range of the scaled image.

\subsection[Threshold Processing]{Threshold Processing}
\label{Threshold Processing}

The MASP algorithm was specifically developed to generate a peak at the intersection of a modulation frequency and carrier frequency.  These peaks are often obscured within the MASP image because of a large dynamic range or by being masked by unmodulated portions of a carrier frequency, as can be seen in Figure \ref{fig:masp_example}.  To highlight peaks within a MASP image a threshold processing algorithm was developed to generate a scatter plot with each scatter point representing a detected peak in the original image.  The algorithm is described in Algorithm \ref{alg:scatteralg} and operates on an input MASP image, $\mathbf{I}$, with the fill ratio, $p$, input parameter which determines the total number of scatter points to generate in the final scatter plot.

\begin{algorithm}
	\caption{Scatter Plot Peak Detection Algorithm} \label{alg:scatteralg}
	\scriptsize
	\begin{algorithmic}[1]
		\Require~~
		\Statex $\mathbf{I}$ - Input Image
		\Statex $p$ - Fill Ratio
		\Ensure~~
		\Statex $\mathbf{S}$ - Detected Scatter Points
		\Statex
		\Statex Initialize THRESHOLD
		\While {$1.1p < $ fill $ < 0.9p$}
			\State $\bf{I_t} \gets $ ZSCORE for each row of $\bf{I}$
			\State $\bf{I_t} \gets 0$ for all $\bf{I_t} \le$ THRESHOLD
			\State $\bf{I_t} \gets $ ZSCORE for each column of $\bf{I_t}$ for all $\bf{I_t} \ne 0$
			\State INITIALIZE $\bf{S} = 0$
			\For {elements of $\bf{I_t} \geq $ THRESHOLD}
				\State $\mathbf{S} \gets 1$
			\EndFor
			\State fill $ \gets \sum{\bf{S}}$
			\State UPDATE THRESHOLD
		\EndWhile
	\end{algorithmic}
\end{algorithm}

The scatter plot algorithm initializes a threshold which is iterated on until the number of scatter points is within $10\%$ of the fill ratio.  The algorithm separately scales and thresholds the rows and columns of an image to only generate scatter points that exceed a given threshold in both directions.  After thresholding, the total number of scatter points is calculated and the threshold value is updated to either increase or decrease the number of scatter points based upon the difference between the fill ratio and the total number of scatter points in the current iteration.

To illustrate the scatter point threshold algorithm and image output, the scatter plot peak detection algorithm was applied to the MASP image in Figure \ref{fig:masp_example} and is shown in Figure \ref{fig:maspl_scatter}.  The scatter plot shows groupings, rows, and columns of scatter points that identify modulation peaks that were not easily identifiable in the original MASP image.  For instance, the row of points at $f_m = 26$Hz indicate a significant number of higher frequency carriers being modulated by $26$Hz, which is not obvious in the original MASP image in Figure \ref{fig:masp_example}.

\begin{figure}[tb]
	\includegraphics[width=\textwidth]{./dasp_algorithm_results/masp_low_scatter_filenum_12001.eps}
	\centering
	\caption{Scatter plot resulting from the peak thresholding of the MASP image in Figure \ref{fig:masp_example}.  The scatter plot highlights peaks in the MASP image that would otherwise not be visible due to dynamic range of the image, such as the row of points located at the $26$Hz modulation frequency.  It should also be noted that the horizontal lines at $20$Hz and $60$Hz in Figure \ref{fig:masp_example} are no longer visible because their magnitudes did not exceed the threshold in both the vertical and horizontal dimensions.}
	\label{fig:maspl_scatter}
\end{figure}

In addition to the row of modulations at $26$Hz, its second harmonic at $52$Hz also showed a similar broadband modulation pattern.  Several modulation structures became evident upon further examination, such as the large clustering of modulations at the intersection of $80$Hz with low frequency carriers and the large number of clusters between the $100$kHz and $200$kHz carrier frequencies.  By performing the scatter point thresholding against the rows and columns of the MASP image, the resulting scatter plot can highlight modulation structures that are not evident in the raw images.

\subsection[Edge and Line Detection]{Edge and Line Detection}
\label{Edge and Line Detection}

Several of the DASP algorithms result in images that contain dimensional alignments that generate straight and curved lines, such as with HASP and CMASP, and much like the scatter plot peak detection algorithm for the MASP images, tailored transforms were required to highlight and exploit these features.  To highlight lines and edges within a DASP image a Laplacian of Gaussian (LoG) edge detector \cite{Marr1980} was applied to the DASP images using the MATLAB\textsuperscript \textregistered ~ \textit{edge} function\footnote{https://www.mathworks.com/help/images/ref/edge.html, June $26$, $2017$}.  Figure \ref{fig:cmasp_edge} shows the resulting image after application of the LoG edge detector to the image in Figure \ref{fig:cmasp_example} while using a LoG filter standard deviation value of $2$ and a filter size of $13 \times 13$.

\begin{figure}[tb]
	\includegraphics[width=\textwidth]{./dasp_algorithm_results/cmasp_edge_filenum_9601.eps}
	\centering
	\caption{CMASP image resulting from the application of a LoG edge detector to the image in Figure \ref{fig:cmasp_example}.  The edge detection algorithm removes variations in the background and highlights edges and lines for subsequent feature extraction or line detection processing.}
	\label{fig:cmasp_edge}
\end{figure}

Application of the LoG edge detector highlighted discontinuities in the processed image, resulting in an image with detection points forming lines and other shapes outlining the structures within the original image.  A radon transform \cite{Deans1981} was then applied to the CMASP edge detection image, Figure \ref{fig:cmasp_edge}, to detect radial lines traversing the image, as shown in Figure \ref{fig:cmasp_radon}.  The MATLAB\textsuperscript \textregistered ~ \textit{radon} function \footnote{https://www.mathworks.com/help/images/ref/radon.html, June $26$, $2017$} was utilized with projection angles between $0^{\circ}$ and $180^{\circ}$.  Figure \ref{fig:cmasp_radon} shows the radon output image along with detected peaks highlighted with circles, while Figure \ref{fig:cmasp_peak} provides a mesh plot of the same image to better illustrate image peaks.  Each peak in the radon image corresponds to a detected line at a given location ($x'$) and angle.

\begin{figure}[tb]
	\includegraphics[width=\textwidth]{./dasp_algorithm_results/cmasp_radon_filenum_9601.eps}
	\centering
	\caption{Radon transform of the CMASP image in Figure \ref{fig:cmasp_edge} with detected peaks highlighted with circles.  Each detected peak corresponds to a line in the original image.  The vertically aligned circles in the center of the image correspond to vertical lines in the original image and the circles in the left and right quadrants correspond to angled lines in the original image with the x-axis providing the angle and the y-axis the location.}
	\label{fig:cmasp_radon}
\end{figure}

\begin{figure}[tb]
	\includegraphics[width=\textwidth]{./dasp_algorithm_results/cmasp_radon_mesh_filenum_9601.eps}
	\centering
	\caption{Mesh plot of the radon transformed CMASP image in Figure \ref{fig:cmasp_edge} illustrating the detected peaks.}
	\label{fig:cmasp_peak}
\end{figure}

The LoG and radon functions provide a suite of tools for highlighting lines and edges within a DASP image.  As can be seen in Figure \ref{fig:cmasp_example}, the CMASP transform can generate many lines with no clear structure which could potentially confound and mask underlying discriminative features.  The application of the LoG edge detector can eliminate much of the underlying noise and highlight lines and edges of interest, while the subsequent application of the radon transform can detect highlighted lines and potentially provide a more compact representation of the CMASP image structure.  For instance, the large number of detected peaks in Figure \ref{fig:cmasp_radon} provided indications of a large number of distinct and left-to-right downward sloping lines, as evident in Figure \ref{fig:cmasp_edge}.

\subsection[Images Segmentation]{Image Segmentation}
\label{Image Segmentation}

Dimensional alignments or other features of interest relevant for the classification of URE generators occur within different regions of the DASP images.  To aid in feature extraction and to isolate features of interest, an image segmentation process was developed to generate sub-segments of an input DASP image.  The image segments are illustrated in Figure \ref{fig:haspd_segment}.  The segmentation of Figure \ref{fig:haspd_example} shows the isolation of aligned harmonics in segments $8$, $5$, and $2$.  In addition, the process also generated segments which did not contain features of interest for this particular device; however, other devices may generate features that do occur in these segments and therefore provide better discrimination for the purposes of classification.

\begin{figure}[tb]
	\includegraphics[width=\textwidth]{./dasp_algorithm_results/hasp_segment_filenum_12001.eps}
	\centering
	\caption{$3 \times 3$ image segmentation of the image in Figure \ref{fig:haspd_example} demonstrating the isolation of specific regions of interest.  By segmenting the image, features of interest can further be highlighted without confounders, such as in the center-bottom tile, segment $\#2$.}
	\label{fig:haspd_segment}
\end{figure}

\section[Statistical Feature Extraction]{Statistical Feature Extraction}
\label{Statistical Feature Extraction}

Extraction of features from the DASP images can be tailored to a specific device or class given \emph{a priori} knowledge of the URE characteristics, such as specific frequency peaks or modulation frequencies.  Given no prior knowledge of a device's URE characteristics and the low noise collection environment, no tailored feature extractors were utilized in the feature generation process.  Two machine learning classification techniques are explored in Chapter \ref{DASP Device Classification Chapter}. The first machine learning approach was based upon 1-D statistical feature vectors extracted from DASP images, while the second machine learning technique used was the CNN deep learning approach applied directly to the DASP images.

General statistical features based on variance ($\sigma^2$), skewness ($\gamma$), and kurtosis ($\kappa$), as defined by Equations \ref{eq:sigmaeq}, \ref{eq:gammaeq}, and \ref{eq:kappaeq}, respectively, were extracted from the DASP images to form a 1-D feature vector for machine learning, as also demonstrated by \cite{Zhenfei2009}, \cite{Klein2009}, \cite{Lukacs2016}, and \cite{Cobb2010}.  Samples over which to compute the statistics were taken from the summed rows and columns of each image, or image segment, as illustrated in Figure \ref{fig:haspd_summation}.  For whole image feature extraction, all of the pixels of the image were concatenated into a single vector and used as samples for statistical feature generation.

\begin{equation}
	\sigma^{2}_{\bf{x}} = \textbf{E}[(\textbf{x} - \mu)^2]
	\label{eq:sigmaeq}
\end{equation}

\begin{equation}
	\gamma_{\bf{x}} = \textbf{E}\left[\left(\frac{\textbf{x} - \mu}{\sigma}\right)^3\right]
	\label{eq:gammaeq}
\end{equation}

\begin{equation}
	\kappa_{\bf{x}} = \frac{\textbf{E}[(\textbf{x} - \mu)^4]}{(\textbf{E}[(\textbf{x} - \mu)^2])^2}
	\label{eq:kappaeq}
\end{equation}

where $\bf{x}$ are the samples taken from the summed rows, summed columns, and concatenated pixel vector of the DASP arrays shown in Figure \ref{fig:haspd_summation}.

\begin{figure}[ht]
	\includegraphics[width=\textwidth]{./dasp_algorithm_results/hasp_d_summing_filenum_9601.eps}
	\centering
	\caption{Column and row summing as applied to the HASP-D image shown in Figure \ref{fig:haspd_example}.  By aligning the independent carriers and their respective harmonics in the HASP-D image, the column summing provides a method for adding the ``coherent'' energy associated with the independent carriers, providing a significant response in the resulting summed vector.}
	\label{fig:haspd_summation}
\end{figure}

As illustrated in Figure \ref{fig:haspd_summation}, the HASP-D algorithm aligns harmonically related frequencies into a single column within the HASP-D image and the column summing allows all of the related harmonics to be added coherently into a single vector as demonstrated in the sum of columns plot.   Although not as pronounced in Figure \ref{fig:haspd_summation}, the row summing can provide insights into the harmonic structure of the underlying signal.  For instance, the summed row vector shows an alternating harmonic intensity structure between the $11$th and $17$th harmonics.

%\section[Image Structural Similarity]{Image Structural Similarity}
%
%\blindtext[1]
%
%\begin{equation}
	%SSIM(x,y) = \left[i(x,y)\right]^{\alpha}\left[c(x,y)\right]^{\beta}\left[S(x,y)\right]^{\gamma}
	%\label{eq:ssimeq}
%\end{equation}
%
%where
%
%\begin{equation}
	%i(x,y) = \left[i(x,y)\right]^{\alpha}\left[c(x,y)\right]^{\beta}\left[s(x,y)\right]^{\gamma}
	%\label{eq:ssimieq}
%\end{equation}
%
%\begin{equation}
	%c(x,y) = \frac{2\mu_{x}\mu_{y} + C_{1}}{\sigma^{2}_{x} +\sigma^{2}_{y} + C_{2}}
	%\label{eq:ssimceq}
%\end{equation}
%
%\begin{equation}
	%s(x,y) = \left[i(x,y)\right]^{\alpha}\left[c(x,y)\right]^{\beta}\left[S(x,y)\right]^{\gamma}
	%\label{eq:ssimseq}
%\end{equation}
%
%\begin{figure}[ht]
	%\includegraphics[width=\textwidth]{./dasp_algorithm_results/masp_refimage_ssim_device_6.eps}
	%\centering
	%\caption{Reference MASP image generated by averaging $20$ MASP images derived from URE collected from fluorescent lights and subtracting the background (i.e. the ``None'' state).}
	%\label{fig:masp_ref_image}
%\end{figure}
%
%\begin{figure}[ht]
	%\includegraphics[width=\textwidth]{./dasp_algorithm_results/masp_image_ssim_refcmp_device_6.eps}
	%\centering
	%\caption{Plot of structural similarity index values between MASP reference image of device $6$ and a random sampling of MASP images from all $20$ devices.  The peak at $6$ shows the highest similarity to device $6$, which confirms the applicability of comparing DASP images through SSIM values.}
	%\label{fig:masp_ssim_dev}
%\end{figure}
%
%\begin{figure}[ht]
	%\includegraphics[width=\textwidth]{./dasp_algorithm_results/masp_image_ssim_multrefcmp_device_6.eps}
	%\centering
	%\caption{Image of SSIM values using a comparison of the MASP reference image for device $6$ to captures with all combinations of two devices present.  The peaks at the rows and columns at $6$ show that even with confounders the SSIM still shows a peak when device $6$ is present and further shows that the SSIM to all other combinations of devices is low.}
	%\label{fig:masp_ssim_multi}
%\end{figure}

%\begin{figure}[ht]
	%\includegraphics[width=\textwidth]{./dasp_algorithm_results/masp_summing_filenum_9601.eps}
	%\centering
	%\caption{Column and row summing as applied to the MASP image shown in \ref{fig:masp_example}.  The row summing highlights the relative strengths and location of the modulating frequencies.}
	%\label{fig:masp_summation}
%\end{figure}

%\subsubsection[Clustering Analysis]{Cluster Analysis}
%
%\blindtext[1]
%
%\begin{figure}[ht]
	%\includegraphics[width=\textwidth]{./dasp_algorithm_results/masp_dbscan_noise_filenum_12001.eps}
	%\centering
	%\caption{MASP Scatter DBSCAN with Noise}
	%\label{fig:maspdbscannoise}
%\end{figure}
%
%\begin{figure}[ht]
	%\includegraphics[width=\textwidth]{./dasp_algorithm_results/masp_dbscan_nonoise_filenum_12001.eps}
	%\centering
	%\caption{MASP Scatter DBSCAN without Noise}
	%\label{fig:maspdbscanwithoutnoise}
%\end{figure}
%
%\begin{figure}[ht]
	%\includegraphics[width=\textwidth]{./dasp_algorithm_results/masp_blob_area_filenum_12001.eps}
	%\centering
	%\caption{MASP Blob analysis with detected corners}
	%\label{fig:maspblob}
%\end{figure}
%
%\begin{figure}[ht]
	%\includegraphics[width=\textwidth]{./dasp_algorithm_results/hasp_dbscan_nonoise_filenum_12001.eps}
	%\centering
	%\caption{HASP Scatter DBSCAN without Noise}
	%\label{fig:haspdbscanwithoutnoise}
%\end{figure}
%
%\begin{figure}[ht]
	%\includegraphics[width=\textwidth]{./dasp_algorithm_results/hasp_blob_area_filenum_12001.eps}
	%\centering
	%\caption{HASP Blob analysis with detected corners}
	%\label{fig:haspblob}
%\end{figure}
%\subsection[Histogram Based Features]{Histogram Based Features}
%
%\begin{figure}[ht]
	%\includegraphics[width=\textwidth]{./dasp_algorithm_results/masp_hist_filenum_12001.eps}
	%\centering
	%\caption{MASP Aligned Histogram}
	%\label{fig:masphist}
%\end{figure}
%
%\begin{figure}[ht]
	%\includegraphics[width=\textwidth]{./dasp_algorithm_results/masp_segment_filenum_12001.eps}
	%\centering
	%\caption{$3 \times 3$ image segmentation of the \ref{fig:masp_summation} image isolating specific regions of interest.  Tiling of the MASP image highlights specific locations of peaks associated with the intersection of carrier and modulation frequencies, such as illustrated in the top two rows of tiles.}
	%\label{fig:masp_segment}
%\end{figure}


%% !TEX root = manual.tex

\section{Spyplot Diagrams}
\label{sec:tutorials:spyplot}

Spyplots visualize communication matrices, showing either the number of messages or number of bytes sent between two network endpoints.
They are essentially contour diagrams, where instead of a continuous function $F(x,y)$ we are plotting the communication matrix $M(i,j)$.
An example spyplot is shown for a simple application that only executes an MPI\_Allreduce (Figure \ref{fig:spyplot}).
Larger amounts of data (red) are sent to nearest neighbors while decreasing amounts (blue) are sent to MPI ranks further away.

\begin{figure}[h]
\centering
\includegraphics[width=0.4\textwidth]{figures/spyplot/mpi_spyplot.png}
\caption{Spyplot of Bytes Transferred Between MPI Ranks for MPI\_Allreduce}
\label{fig:spyplot}
\end{figure}

Various spyplots can be activated.
The most commonly used are the MPI spyplots, for which you activate the spyplot as part of the MPI subcomponent.

\begin{ViFile}
node {
  app1 {
    mpi {
      spy_bytes {
        type = spyplot
        output = csv
        group = app1
      }
    }
  }
}
\end{ViFile}
After running, there will be a CSV containing the data sent by each component to another component.
The same statistic can be activated in both the \inlinecode{node.app1.mpi} namespaces and the \inlinecode{node.nic} namespaces.
The type of the statistic must be spyplot, but the output can be other formats (but just use csv).


\documentclass[12pt]{article}
\usepackage{geometry} % see geometry.pdf on how to lay out the page. There's lots.
\geometry{a4paper} % or letter or a5paper or ... etc
% \geometry{landscape} % rotated page geometry
\usepackage{graphicx}
% See the $$Article customise'' template for come common customisations

\title{Scaling Erasure codes with integrity audits}
\author{Viktor Trón}
\date{PhD thesis. draft of \today} % delete this line to display the current date
\pagestyle{plain}
\pagenumbering{arabic}
% \usepackage{color}\
\usepackage{amsmath}
\bibliographystyle{apalike}
%%% BEGIN DOCUMENT
\begin{document}

\maketitle
\tableofcontents

\begin{abstract}
This paper brings together two problem areas of computer infrastructure, coding theory and proof of custody to tackle the scaling of erasure codes in the context of a distributed storage system. We offer a double divide and conquer approach: we encode the data in a (merkle) tree and apply erasure code to each set of children, this spacial division trades off reliability for linear coding performance. On the other hand we refine the underlying loss model by arbitrary temporal resolution and conquer by showing repeated recursive audits when done frequently can counterbalance for the degree of reliability lost on the way. This technique shifts the quadratic complexity of the erasure codes to a background process and thereby significantly improving on the practical data size limitations inherent in erasure codes.
\end{abstract}

\section{Introduction}

Erasure codes and proofs of custody, though somewhat related, are not commonly mentioned in one sentence given they are typically used at very different application areas. Coding theory offers a solution to problems of data loss, improving data availability and reliability of data storage systems; while proof of custody is concerned with offering cryptographic contructs that allow custodians of data to prove to third parties that they have the data available.
Both constructions proved to be useful (and indeed complementary) in the process of designing and implementing swarm, a decentralised serverless hosting platform, a distributed storage network and content delivery service infrastructure \cite{ethersphere2016sw3}.

Swarm uses a content addressed storage over a network of nodes. Content addressed storage is crucial in organising retrieval and distribution of data chunks and solves the problem of eliminating unneccesary replication since data is stored at the address that is deterministically derived from the data itself. By choosing the address function to be irreversible and pseudorandom the system is capable of integrity protection and allows for efficient key-based routing protocols \cite{maymounkov2002kademlia}.

While eliminating unessessary replication is a desireable feature in a storage network, in order for such networks to be used in the real word, they must still be able to offer resilience in the face of various infrastructual failings. Most notably, some level of redundancy of data storage is essential to a zero-downtime, low-latency, fault-tolerant network. Erasure coding is a well known technique to implement redundant storage and therefore was a natural choice.

\section{Erasure codes}

Erasure codes are commonly used to ensure that data is available even if parts of the storage system are faulty. In particular they allow us to constuct storage schemes (data encoding and distribution) that solves this problem more efficiently than simple replication.  The problem is framed in the context of guaranteeing some probability that the data is retrievable given some model expressing the expected fault conditions in the storage subsystems.
Coding theory in practice is often concerned with RAID-s and computer hardware architecture synchronising disks for resilient datacentre storage.
Erasure codes in particular see the problem as how to encode data into shards distributed across $n$ disks so that the data is fully retrievable in the face of a particular probability that one disk is faulty.
Similarly in the context of a distributed chunkstore, the problem can be recouched as the question how to encode the data into chunks distributed across the nodes in the network so that the entire data is retrievable in the face of a particular probability that one chunk is missing\footnote{The distributed chunkstore model uses fixed-sized chunks which are either completely lost or completely undamaged. Since the swarm content storage uses the hashes of chunks as their addresses, and since the chunks are stored by custodian nodes that are randomly distributed in the same address space, we are safe to assume that the storage allocation is independent of all the other nodes and data. This assures that recovering a chunk at any point can practically be thought of as failing with equal and independent probability.}

The Cauchy-Reed-Solomon erasure code (henceforth CRS, \cite{lubyetal1995CRS}, \cite{plank2006optimizing}) is a \emph{systemic erasure code} which applied to a data blob of $m$ fixed-size pieces, produces $k$ extra pieces (so called \emph{parity pieces}) of the same size in such a way that any $m$ out of $n=m+k$ fix-sized pieces are to enough to reconstruct the original blob. The storage overhead is therefore given by $\frac{k}{m}$.%
%
\footnote{%
There are open source libraries that implement Reed Solomon or Cauchy-Reed-Solomon coding. See \cite{plank2009performance} for a thorough comparison.}

%
% formula for probability of loss.
%

Both the encoding and the decoding of CRS codes takes $O(mk)$ time, where $n$ is the number of data chunks, $k$ is the number of additional chunks as well as the maximum number of chunks that can be lost without losing decodeability. If $k$ is defined as a given fraction of $m$, which is necessary for guaranteeing a certain probability of retrievability under the condition of a fixed probability $p$ of losing a single chunk, the time complexity of CRS codes becomes $O(n^2)$, which is utterly unacceptable for large files. %large files = large n only when chunk size is fixed.

\section{The swarm chunk tree}

Swarm uses hierarchical merkle tree \cite{merkle1980protocols} to reorganise data into fixed sized chunks which are then sent off to the swarm nodes to store (by a node close to their address which makes it retrievable by the hash (content address).
Assuming $h$ the size of the hash output of the hash function used in bytes, $b$ is the branching factor. Each node represents the root hash of a subtree or, at the last level, the hash of a $b\cdot h$ long span (one chunk) of the file. Generically we may think of each chunk as consisting of $b$ hashes:

\begin{figure}[htbp]
   \centering
   \includegraphics{fig/chunk.pdf} % requires the graphicx package
   \caption{A swarm chunk consists of 4096 bytes of the file or a sequence of 128 subtree hashes.}
   \label{fig:chunk}
\end{figure}

while in the tree structure, the 32 bytes stored at the node represent the hash of the 128 children.

\begin{figure}[htbp]
   \centering
   \includegraphics{fig/treebasic.pdf} % requires the graphicx package
   \caption{ A generic node in the tree has 128 children.}
   \label{fig:treebasic}
\end{figure}

During normal swarm lookups, a swarm client performs a lookup for a hash value and receives a chunk in return. This chunk in turn constitutes another $b$ hashes to be looked up in return for another $b$ hashes and so on until the chunks received belong to the actual document. Here is a schematic: (See Figure \ref{fig:tree2}).

\begin{figure}[htbp]
   \centering
   \includegraphics{fig/tree2.pdf} % requires the graphicx package
   \caption{ The swarm tree is the data structure encoding how a document is split into chunks.}
   \label{fig:treebasic}
\end{figure}

\section{Erasure codes and the swarm chunk tree}

While off-the-shelf erasure coding could be used on documents uploaded in swarm, this solution has immediate problems. Apart from the quadratic complexity of encoding, chunking the CRS endoded datablob with the chunker would result in certain chunks to be more vulnarable, since their retrieval is dependent on the retrieval of the chunks that encode all their ancestor nodes in the chunktree.

This prompted us to try and align the notion of swarm chunk with the chunk used in the CRS scheme. This led us to encode redundancy directly into the swarm tree by applying the \emph{CRS scheme}  to each set of chunks that are children to a node in the chunkree.

The \emph{chunker} algorithm using CRS coding work the following way when splitting the document:

 1. Set input to the data blob.
 2. Read the input one chunk (say fixed 4096 bytes) at a time. Count the chunks by incrementing $i$. The last chunk read may be shorter.
 3. Repeat 2 until there's no more data or $i \equiv 0$ mod $m$
 4. use the CRS scheme on the last $i \mod\ m$ chunks to produce $k$ parity chunks resulting in a total of $n \leq m+k$ chunks.
 . Calculate the hashes of all the these chunks and concatenate then to result in the next chunk (of size $i\mod m$ of the next level. Record this chunk as the next
 6. If there is more data repeat 2. otherwise
 7. If the next level data blob has more than one chunk, set the input to this and  repeat from 2.
 8. Otherwise remember the blob as the root chunk.

Assume we fix the branching factor of the swarm hash (chunker) as $n=128$ and $h=32$ as the size of the \emph{SHA3 Keccak hash}. This gives us a chunk size of $4096$ bytes.

Let us now suppose that we start splitting out input document data into chunks, and after each $m$ chunks then add $k=n-m$ parity check pieces using a Reed-Solomon code so that now any $m\text{-out-of-}n$ chunks are
sufficient to reconstruct the document. On the next level up the chunks are composed of the hashes of the $m$  data chunks and the $k$ hashes of the parity chunks. Let's take the first $m$
of these and add an additional $k$ parity chunks to those such that any $m$ of the resulting $n$
chunks are sufficient to reconstruct the origial $m$ chunks. And so on and on every level. In terms of
availability, every subtree is equally important to every other subtree at this level. The resulting
data structure is not a balanced tree since on every level $i$ the last $k$ chunks are parity leaf
chunks while the first $m$ are branching nodes encoding a subtree of depth $i-1$ redundantly.
A typical piece of our tree would look like this (see Figure \ref{fig:tree-with-erasure}).

\begin{figure}[htbp]
   \centering
   \includegraphics{fig/tree-with-erasure.pdf} % requires the graphicx package
   \caption{The swarm tree with extra parity chunks using $m$ out of 128 CRS code. Chunks $p^{m+1}$ through $p^{128}$ are parity data for chunks $h^1_1 - h^1_{128}$ through $h^{m}_1  - h^{m}_{128}$.}
   \label{fig:tree-with-erasure}
\end{figure}

This pattern repeats itself all the way down the tree. Thus hashes $h^1_{m+1}$ through $h^1_{128}$ point to parity data for chunks pointed to by $h^1_1$ through $h^1_{m}$.Parity chunks $p^i$ do not have children and so the tree structure does not have uniform depth.

If the number of file chunks is not divisible by $m$, then we cannot proceed with the last batch in the same way as the others. We propose that we encode the remaining chunks with an erasure code that guarantees at least the same level of security as the others. Note that it is not as simple as choosing the same redundancy. For example a 50-of-100 encoding is much more secure against file loss than a 1-of-2 encoding even though the redundancy is 100\% in both cases. Overcompensating, we could say that there should always be the same number of parity chunks even when there are fewer than $m$ data chunks so that we always end up with $m\text{-out-of-}n$. We repeat this procedure in every row in the tree.

This leaves us with only one corner case: it is not possible to use our $m\text{-out-of-}n$ scheme on a single chunk ($m=1$) because it would amount to $k+1$ copies of the same chunk. The problem of course is that any number of copies of the same chunk all have the same hash and are therefore indistinguishable in the swarm. Thus when there is only a single chunk left over at some level of the tree, we'd have to apply some transformation to it to obtain a second (but different) copy before we could generate more parity chunks.

In particular this is always the case for the root chunk. To illustrate why this is critically important, consider the following. The root hash points to the root chunk. If this root chunk is lost, then the file is not retrievable from the swarm even if all other data is present. Thus we must find an additional method of securing and accessing the information stored in the root chunk.

Whenever a single chunk is left over ($m=1$) we propose to append an extra padding byte to the chunk not counting towards its size. In swarm, each 4096 byte chunk is actually stored together with 8 bytes of meta information - currently only storing the size of the subtree encoded by the chunk. Since the subtree size determines exactly what span of the chunk is substantive data, the padding differential byte is easily ignored when the document is assembled.%
%
\footnote{%
Note that the typical values for $k$ will be in the single digits so a single byte will allways suffice. Note that in the special cornercase when the singleton leftover chunk is a full chunk, we end up having an oversized chunk.
}

\section{Scalability issues}

This per-level $m\text{-out-of-}n$ Cauchy-Reed-Solomon erasure code once introduced into the swarm chunk tree does not only ensure file availability, but also offers further benefits of increased resilience and ways to speed up retrieval \cite{ethersphere2016sw3}.
The systemic code means that the pattern naturally lends itself to simple implementation of efficient retrieval and transparently preserves the random-access property of the chunk tree.

Note that by keeping $n$ and $k$ fixed we essentially limit the quadratic component of the encoding/decoding potentially to a fix constant $c$ even if parity chunks are needed and CRS encoding needs to be used.

While that solves the latency issues of retrieval, the reliability that the scheme provides in itself does not scale with data size. For such a scheme to scale (and keep the probability of failure under a threshold irrespective of the size of data to be stored) we would need to assume that for each subtree the proportion of lost chunks will be below some ratio. Which is a much weaker assumption than that the total loss of chunks will be below that ratio. While you could argue that there are practical limits of file size, but ultimately, it is not files, but collections accessible from the same swarm root manifest that we want to guard. And there is no practical limit for the size of such collections.

Here's a deliberately exagerrated example of how this approach is impractical:
Suppose you have two million chunks. You can add another million chunks of CRS code, giving you three million chunks, such that if you lose up to a million of those, you can still get your original two million back.
If you have just two chunks, you can XOR them and get a third chunk, so that if you lose any of the three, you can get your original two back. It's still a 50\% redundancy, like in the case of the two million chunks above. However, if you pair up your two million chunks and add a third one to each pair, which is their XOR, you do get an additional million chunks, just like in the first case, except that it won't protect you against the loss of any million chunks, because if you pick the lost million chunks at random, it is essentially certain, that there will be at least one triplet of which you lose at least two members.

Increasing chunk size while not decreasing $k$ results in quadratic time complexity. Not increasing $k$ results in a probability of successful decoding converging to zero.
While the scheme has desirable properties and solves decoding scalability under adverse conditions its applicability is practically in a limited size range.

In what follows we focus on the underlying loss model and  show how it offers an avenue to recover from the decreased reliability we  traded off for realtime decoding performance.

% In particular, we show

\section{Proof of custody}

We encountered proof of custody when designing and implementing a crypteconomic system that incentivises swarm nodes to behave in ways conducive to desired operation.
Accounting for bandwidth is relatively straightforward, therefore keeping popular chunks is never a problem. We soon realised however, that storage incentivisation ie., to make sure chunks that want to be kept and have funds to compensate other nodes are not deleted, is a lot more complex undertaking.

\cite{ethersphere2016smash} present a proof of custody formula inspired by the Wilkinson--Buterin proof of storage used by Storj \cite{wilkinsonetal2014storj}. The formula offers 3 different types of challenge that auditors can use in different stages. The authors specify an auditing and litigation scheme that has ideal properties to secure the swarm against chunk loss.

SMASH proofs offer integrity checking for chunks as well as for documents and document collections that

\begin{itemize}
\item  allows storers to initiate and control audits without storing any information other than the swarm hash of the chunk;
\item  allows owners to outsource auditing without a trusted third party;
\item  it provides a seed generation algorithm for securing large document collections with a single audit secret so it scales for both storage and bandwidth;
\item  the successive seeds contain error detection which makes it very efficient to find offending nodes without remembering the (masked) secret for each chunk;
\item  allows easy verification by third parties like smart contracts to serve as evidence  when it comes to litigation on the blockchain;
\item  works without ever writing anything to the blockchain which is only used for last-resort litigation;
\item  enables very small size proofs to optimize bandwidth use and prevent blockchain bloating
\item  offers guardians and storers ways to refute the challenge, including proof that auditors request is invalid.
\end{itemize}

We outlined an auditing and litigation protocol which

\begin{itemize}
\item  offers efficient ways to probe the swarm off-chain with recursive outsourceable collective audits;
\item  enables prompt incentivised escalation whereby an audit continues as litigation on the blockchain;
\item  helps nodes identify greedy peers that do not forward chunks;
\item  offer a way to repair improper syncronisation state.
\end{itemize}

In particular the CRASH (Collective Recursive Audit with Secret Hash) proof offers a way to audit large document collections of any and works the following way.
A swarm chunk encodes a subtree, in particular a non-leaf chunk consists of $b$ swarm hash segments ($b$ being the branching factor). These are the hashes of chunks on the lower level of the chunk tree, each in turn encoding their subtree. The CRASH output is a value defined on each subtree recursively and involves chaining audits on individual chunks where the input of one audit is the output of the previous one.
Since the chunks retrieved sit on an actual node's storage, this calculation can be outsourced to the nodes in a recursive auditing scheme where the intermediate (non leaf) nodes arz not only audited themselves, but they also serve as auditors of the subtrees encoded in their chunk. The audit requests used in CRASH are similar to retrieval requests except that in their response, custodians do not send back the chunk itself but instead they calculate the audit secret hash (ASH) and respond with that (see Figure \ref{fig:crash}).

\begin{figure}[htbp]
   \centering
   \includegraphics[scale=0.45]{../smash/fig/crash.pdf} % requires the graphicx package
   \caption{This figure zooms in on a chunk in a chunk tree of a document. The chunks represent their custodian nodes that act as auditorqs of the subtree their chunk encodes. The arrows represent the flow of information in the successive steps of calculating CRASH. The custodian of the non-leaf chunk receives a seed and iterates over the hashes of its chunk. It initiates a recursivqe audit challenge on their immediate child nodes. After receiving the response from the chunk's custodian, they perform validation against the error bits and calculate the next seed they then send on to the next child. In case the validation fails, the node backtracks and escalates the audit  to ASH proof challenge    (dashed lines). After piping the seed through the children's audits, it performs a self-challenge as if it was a leaf chunk and sends back the resulting audit secret to their parent auditor.
   }
   \label{fig:crash}
\end{figure}

If the entire audit were to be carried out by just one peer, then chunks for each intermediate node would need to be retrieved in order for the main auditor to initiate all the audit requests for subtrees. Collective auditing has the immediate benefit that no intermediate chunks ever need to be actually retrieved, because the audit of subtrees are carried out by peers that store the chunk. This means a successful audit require only one challenge-response exchange per node involved.
Discounting the one-off task of distributing metadata for the audit which is linear in the size of the collection, the network traffic for each subsequent audit
constitutes a single message roundtrip of a very small constant message.
In the simple context of a trusted system, this can be brought down to hashsize $h$ (32).

Let us assume that given the audit is successful, the probability that the chunk is retrieveably in its full integrity is $p$ and is independent for each chunk.
Calculating the secret itself is on a node is logarithmic in $m$.

Assuming that linear traffic overhead is realistic, the message roundtrip also translates to latency. Since audits are sent to nodes uniformly sampled, the latency is an independent sample. For simplicity we assume that the time used in the recursive network audits simply translates to a constant time factor. If the recursive audit is sequentially organised the time required by audits is linear in the number of nodes which is twice the number of chunks.

Note that partial verification is needed to check non-trusted nodes. In a trusted setup this is not needed and therefore the crash calculation can be massively parallelised on each level. Since the levels are still sequential the time complexity of such a calculation is $log_{b}(n)$.

\section{Integrity audits and erasure coded chunks}

When zooming in on the loss model, we realize that the assumption of a fix probability of a chunk not found is tacitly implying a time interval $d$. Given that the probability of chunks lost we can say the probability that a chunk loss will happen during that period of time. However, if we zoom in to this time period, we realise the individual loss events occur in time. If we somehow detect one and reconstruct it from the rest, it is as if it never happened. This is where we bring in the audits, if audits are carried out multiple times within a time period then it is possible to identify and repair chunks before the damage is too big.
Let us assume that CRASH audits are performed at regular intervals with frequency $f$ during the period $d$ in question. Assuming that the node uses $k$ parity chunks,  a chunk is now irrepairable if and only if all $k$ losses occur during one cycle of audit and repair. Let's derive formally how to set $f$ so that for any file size, the probability that a node is repairable is above a service quality threshold $q$

\begin{align*}
????
\end{align*}

In other words the frequency of audits is set according to the above, then service quality is guaranteed to be at least $q$ irrespective of $s$. The overall resource consumption of the audits are calculated as the following:
It terms of network traffic, it is for the duration of $d$, we send $f\cdot s$ fixed sized roundtrips, which is $O(s^2)$. In terms of time complexity, parallelisation in the trusted case requires it is $O(s\cdot log(s))$ in the trusted scenario and potentially
over the duration of $d$ as $a\cdot f\cdot s$ which is $O(s^2)$.
Of course in order to save the data, we also need to perform repairs on the chunks. Since repair involves fetching the relevant data and recode it,they take way more resources than audits themselves and effectively put a hard limit on the frequency of audits.
As $s$ increases, the speed of repairs won't be able to keep up, i.e., if the time it takes for one chunk to be repairs is longer than $d/f$, we need to resort to increasing $k$ (and $b$) or increasing the chunksize in order to keep the same reliability rate.
Surely this move increases the quadratic component in those cases when the decoding is used, i.e., is every time a retrieval coincides with a repair. Assuming in the worst case that repairs take the full length of the audit, the proportion of nodes needing CRS is the same as the proportion of repairs to audits. This ratio constant. Is it? why?

Note that we reproduced the quadratic complexity, however, we shifted it over to a background process with no real time concern beyond needing to keep up with repairs (on top of computational and network costs) and thereby pushing the range of applicability of erasure codes to include larger files when compared to the original coding used.

\section{Conclusion}

\bibliography{../refs.bib}
\end{document}%% LaTeX - Article customise


% The name of this file: cddlibman.tex
% written by by Komei Fukuda
% created March 15, 1999
% modified February 7, 2008
%
\documentclass[11pt]{article}
\usepackage{html,amsmath,amssymb}
\renewcommand{\baselinestretch}{1}
\renewcommand{\arraystretch}{1}
\setlength{\oddsidemargin}{0mm}
\setlength{\textwidth}{165mm}
\setlength{\topmargin}{-15mm}
\setlength{\textheight}{232mm}
%\setlength{\headsep}{0in}
%\setlength{\headheight}{0pt}
\pagestyle{plain}

\newcommand {\0} {{\bf 0}}
\newcommand{\R}{{\Bbb R}}

\begin{document}
\title{cddlib Reference Manual}
\author{Komei Fukuda\\
Institute for Operations Research\\
and Institute of Theoretical Computer Science\\
ETH Zentrum, CH-8092 Zurich, Switzerland\\
}
\date{ (cddlib ver. 0.94, manual ver. February 7, 2008)}

\maketitle

\tableofcontents

\begin{abstract}
This is a reference manual for cddlib-094.
The manual describes the library functions and data types implemented
in the cddlib C-library which is to perform fundamental polyhedral
computations such as representation conversions and linear programming
in both floating-point and GMP rational exact arithmetic.
Please read the accompanying README file and test programs to
complement the manual.

The new functions added in this version include {\tt dd\_MatrixCanonicalize}
to find a non-redundant proper H- or V-representation,
{\tt dd\_FindRelativeInterior} to find a relative interior point
of an H-polyhedron, and  {\tt dd\_ExistsRestrictedFace} (Farkas-type
alternative theorem verifier)
to check the existence of a point satisfying a specified system
of linear inequalities possibly including multiple strict inequalities.

The new functions are particularly important for the development of
related software packages MinkSum (by Ch. Weibel) and Gfan
(by Anders Jensen),

\end{abstract}

\section{Introduction} \label{INTRODUCTION}

The program  cddlib  is an efficient implementation \cite{fp-ddmr-96}  of
the double description Method~\cite{mrtt-ddm-53}
for generating  all vertices (i.e. extreme points)
and extreme rays of a general
convex polyhedron given by
a system of linear inequalities:
\[
   P = \{ x=(x_1, x_2, \ldots, x_d)^T \in R^{d}:  b - A  x  \ge 0 \}
\]
where $A$ is a given $m \times d$ real matrix and
$b$ is a given real $m$-vector.   In the mathematical
language, the computation is the transformation
of an {\em H-representation\/} of a convex polytope
to an {\em V-representation}.

cddlib is a C-library version of the previously released C-code cdd/cdd+.
In order to make this library version, a large part of the cdd source
(Version 0.61) has been rewritten.
This library version is more flexible since it can be called from other programs in C/C++.
Unlike cdd/cdd+, cddlib can handle any general input and is more general.
Furtthermore,  additional functions have been written to extend its functionality.

One useful feature of  cddlib/cdd/cdd+ is its capability
of handling the dual (reverse)  problem without any transformation
of data.  The dual transformation problem of a V-representation
to a minimal H-representation and is often called the
{\em (convex) hull problem\/}.  More explicitly,
is to obtain a linear inequality representation
of a convex polyhedron given as the Minkowski sum of
the convex hull of a finite set of points and the nonnegative
hull of a finite set of points in $R^{d}$:
\[
P = conv(v_1,\ldots,v_n) +  nonneg(r_{n+1},\ldots,r_{n+s}),
\]
where
 the {\em Minkowski sum of two subsets $S$ and $T$} of $R^{d}$ is defined
as
\[
S + T = \{ s + t \; |  s \in S \mbox{ and } t \in T \}.
\]
As we see in this manual, the computation can be done
in straightforward manner.  Unlike the earlier versions of
cdd/cdd+ that assume certain regularity conditions for input,
cddlib is designed to do a correct transformation for any general input.
The user must be aware of the fact that in certain cases the
transformation is not unique and there are polyhedra with
infinitely many representations.  For example, a line
segment (1-dimensional polytope) in $R^3$ has infinitely
many minimal H-representations, and a halfspace in the same space
has infinitely many minimal V-representations.  cddlib generates
merely one minimal representation.

cddlib comes with an LP code to solve the general
linear programming (LP) problem to maximize (or minimize) a linear
function over polyhedron $P$.   It is useful mainly for solving
dense LP's with large $m$ (say, up to few hundred thousands) and small $d$
(say, up to 100).  It implements a revised dual simplex method that
updates $(d+1)\times (d+1)$ matrix for a pivot operation.

The program cddlib has an I/O routines that read and write files in
{\em Polyhedra format\/} which was defined by David Avis and
the author in 1993, and has been updated in 1997 and 1999.
The program called lrs and lrslib \cite{a-lrshome-01} developed by David Avis is
a C-implementation of the reverse search algorithm~\cite{af-pachv-92}
for the same enumeration purpose, and it conforms to Polyhedra format as well.
Hopefully, this compatibility of the two programs
enables users to use both programs for the same input files
and to choose whichever is useful for their purposes.
From our experiences with relatively large problems,
the two methods are both useful and perhaps complementary
to each other.  In general, the program cddlib tends to be
efficient for highly degenerate inputs and the program rs
tends to be efficient for nondegenerate or slightly
degenerate problems.

Although the program can be used for nondegenerate inputs,
it might not be very efficient.  For nondegenerate inputs,
other available programs, such as the reverse search code lrs or
qhull (developed by the Geometry Center),
might be more efficient.  See Section~\ref{CODES}
for pointers to these codes.
The paper \cite{abs-hgach-97} contains many interesting results on polyhedral
computation and experimental results on cdd+, lrs, qhull and porta.

This program can be distributed freely under the GNU GENERAL PUBLIC LICENSE.
Please read the file COPYING carefully before using.

I will not take any responsibility of any problems you might have
with this program.  But I will be glad to receive bug reports or suggestions
at the e-mail addresses above. If cddlib turns out to be useful,
please kindly inform  me of  what purposes cdd has been used for.
I will be happy to include a list of applications in future
distribution  if I receive  enough replies.
The most powerful support for free software development
is user's appreciation and collaboration.

\section{Polyhedra H- and V-Formats (Version 1999)} \label{FORMAT}
\bigskip
Every convex polyhedron has two representations, one as
the intersection of finite halfspaces and the other
as Minkowski sum of the convex hull of finite points
and the nonnegative hull of finite directions.  These are
called H-representation and V-representation, respectively.

Naturally there are two basic Polyhedra formats,
H-format for  H-representation and V-format for
V-representation.    These two formats are designed
to be almost indistinguishable, and in fact, one can
almost pretend one for the other.   There is some asymmetry
arising from the asymmetry of two representations.

First we start with the H-representation.
Let $A$ be an $m \times d$ matrix, and let $b$ be a column $m$-vector.
The Polyhedra format  ({\em  H-format} )  of
the system  $\; b - A x \ge \0\;$ of $m$ inequalities in $d$ variables
$x =(x_1, x_2, \ldots, x_d)^T$ is

\begin{tabular}{ccl}
\\ \hline
\multicolumn{3}{l} {various comments}\\
\multicolumn{3}{l} {{\bf H-representation}}\\
\multicolumn{3}{l} {{\bf (linearity $t\;$ $i_1\;$  $i_2\;$  $\ldots$ $\;i_t$)}}\\
\multicolumn{3}{l} {{\bf begin}}\\
 $m$ & $d+1$ & numbertype\\
 $b$ & $-A$ \\
\multicolumn{3}{l} {{\bf end}}\\
\multicolumn{3}{l} {various options} \\ \hline
\end{tabular}

\bigskip
\noindent
where numbertype can be one of integer, rational or real.
When rational type is selected, each component
of $b$ and $A$ can be specified by the usual integer expression
or by the rational expression ``$p / q$''  or  ``$-p / q$'' where
$p$ and $q$ are arbitrary long positive integers (see the example
input file rational.ine).  In the 1997 format,
we introduced ``H-representation'' which must appear
before ``begin''.
There was one restriction in the old polyhedra format
(before 1997):  the last $d$ rows must determine
a vertex of $P$.  This is obsolete now.

In the new 1999 format, we added the possibility of specifying {\bf linearity\/}.
This means that
for H-representation, some of the input rows can be specified as  {\bf equalities}:
$b_{i_j} - A_{i_j} x = 0 \;$ for all $j=1,2, \ldots, t$.
The linearity line may be omitted if there are no equalities.

Option lines can be used to control computation of a specific program.
In particular both cdd and lrs use the option lines to represent
a linear objective function.  See the attached LP files, samplelp*.ine.

\bigskip
Next we define Polyhedra  {\em V-format}.  Let $P$ be
represented by $n$ gerating points and $s$ generating directions (rays) as
$P = conv(v_1,\ldots,v_n) +  nonneg(r_{n+1},\ldots,r_{n+s})$.
Then the Polyhedra V-format for $P$ is

\begin{tabular}{cll}
\\ \hline
\multicolumn{3}{l} {various comments}\\
\multicolumn{3}{l} {{\bf V-representation}}\\
\multicolumn{3}{l} {({\bf linearity $t\;$ $i_1\;$  $i_2\;$  $\ldots$ $\;i_t$ })}\\
\multicolumn{3}{l} {{\bf begin}}\\
 $n+s$ & $d+1$ & numbertype\\
 $1$ & $v_1$  & \\
 $\vdots$ & $\vdots$  & \\
 $1$ & $v_n$  & \\
 $0$ & $r_{n+1}$  & \\
 $\vdots$ & $\vdots$  & \\
 $0$ & $r_{n+s}$  & \\
\multicolumn{3}{l} {{\bf end}}\\
\multicolumn{3}{l} {various options} \\ \hline
\end{tabular}

\bigskip
\noindent
Here we do not require that
vertices and rays are listed
separately; they can appear mixed in arbitrary
order.

Linearity for V-representation specifies a subset of generators
whose coefficients are relaxed
to be {\bf free}:  for all $j=1,2, \ldots, t$, the $k=i_j$th generator ($v_{k}$ or $r_k$ whichever is the $i_j$th generator) is a free generator.
This means for each such a ray $r_k$,
the line generated by $r_k$ is in the polyhedron,
and for each such a vertex $v_k$, its coefficient is no longer nonnegative
but still the coefficients for all $v_i$'s must sum up to one.
It is highly unlikely that one needs to
use linearity for vertex generators, and it is defined mostly
for formality.

When the representation statement, either ``H-representation''
or ``V-representation'', is omitted, the former
``H-representation'' is assumed.

It is strongly suggested to use the following rule for naming
H-format files and V-format files:
\begin{description}
\item[(a)] use the filename  extension ``.ine'' for H-files (where ine stands for inequalities), and
\item[(b)]  use the filename  extension ``.ext'' for V-files (where ext stands for extreme points/rays).
\end{description}

\section{Basic Object Types (Structures) in cddlib}  \label{DATASTR}

Here are the types (defined in cddtypes.h) that are
important for the cddlib user.  The most important one,
{\tt dd\_MatrixType},
is to store a Polyhedra data in a straightforward manner.
Once the user sets up a (pointer to)  {\tt dd\_MatrixType} data,
he/she can load the data to an internal data type ({\tt dd\_PolyhedraType})
by using functions described in the next section, and apply
the double descrition method to get another representation.
As an option  {\tt dd\_MatrixType} can save a linear objective function
to be used by a linear programming solver.

The two dimensional array data in the structure {\tt dd\_MatrixType} is
{\tt dd\_Amatrix} whose components are of type {\tt mytype\/}.
The type mytype is set to be either the rational type {\tt mpq\_t} of
the GNU MP Library or the C double array of size $1$.
This abstract type allows us to write a single program that can
be compiled with the two or more different arithmetics, see example
programs such as simplecdd.c, testlp*.c and testcdd*.c
in the {\tt src} and {\tt src-gmp} subdirectories of the source
distribution.

There is another data type that is used very often, {\tt dd\_SetFamilyType}.
This is to store a family of subsets of a finite set.  Such a family
can represent the incidence relations between the set of extreme
points and the set of facets of a polyhedron.  Also, it can represent a
graph structure by listing the set of vertices adjacent to each vertex (i.e.
the adjacency list).   To implement  {\tt dd\_SetFamilyType},
we use  a separate set library called {\tt setoper}, that
handles the basic set operations,   This library is briefly introduced in
Section~\ref{SetFunctions}.

\begin{verbatim}

#define dd_FALSE 0
#define dd_TRUE 1

typedef long dd_rowrange;
typedef long dd_colrange;
typedef long dd_bigrange;

typedef set_type dd_rowset;   /* set_type defined in setoper.h */
typedef set_type dd_colset;
typedef long *dd_rowindex;
typedef int *dd_rowflag;
typedef long *dd_colindex;
typedef mytype **dd_Amatrix;  /* mytype is either GMP mpq_t or 1-dim double array. */
typedef mytype *dd_Arow;
typedef set_type *dd_SetVector;

typedef enum {
  dd_Real, dd_Rational, dd_Integer, dd_Unknown
} dd_NumberType;

typedef enum {
  dd_Inequality, dd_Generator, dd_Unspecified
} dd_RepresentationType;

typedef enum {
  dd_MaxIndex, dd_MinIndex, dd_MinCutoff, dd_MaxCutoff, dd_MixCutoff,
   dd_LexMin, dd_LexMax, dd_RandomRow
} dd_RowOrderType;

typedef enum {
  dd_InProgress, dd_AllFound, dd_RegionEmpty
} dd_CompStatusType;

typedef enum {
  dd_DimensionTooLarge, dd_ImproperInputFormat,
  dd_NegativeMatrixSize, dd_EmptyVrepresentation,
  dd_IFileNotFound, dd_OFileNotOpen, dd_NoLPObjective,
  dd_NoRealNumberSupport, dd_NoError
} dd_ErrorType;

typedef enum {
  dd_LPnone=0, dd_LPmax, dd_LPmin
} dd_LPObjectiveType;

typedef enum {
  dd_LPSundecided, dd_Optimal, dd_Inconsistent, dd_DualInconsistent,
  dd_StrucInconsistent, dd_StrucDualInconsistent,
  dd_Unbounded, dd_DualUnbounded
} dd_LPStatusType;

typedef struct matrixdata *dd_MatrixPtr;
typedef struct matrixdata {
  dd_rowrange rowsize;
  dd_rowset linset;
    /*  a subset of rows of linearity (ie, generators of
        linearity space for V-representation, and equations
        for H-representation. */
  dd_colrange colsize;
  dd_RepresentationType representation;
  dd_NumberType numbtype;
  dd_Amatrix matrix;
  dd_LPObjectiveType objective;
  dd_Arow rowvec;
}  dd_MatrixType;

typedef struct setfamily *dd_SetFamilyPtr;
typedef struct setfamily {
  dd_bigrange famsize;
  dd_bigrange setsize;
  dd_SetVector set;
} dd_SetFamilyType;

typedef struct lpsolution *dd_LPSolutionPtr;
typedef struct lpsolution {
  dd_DataFileType filename;
  dd_LPObjectiveType objective;
  dd_LPSolverType solver;
  dd_rowrange m;
  dd_colrange d;
  dd_NumberType numbtype;

  dd_LPStatusType LPS;  /* the current solution status */
  mytype optvalue;  /* optimal value */
  dd_Arow sol;   /* primal solution */
  dd_Arow dsol;  /* dual solution */
  dd_colindex nbindex;  /* current basis represented by nonbasic indices */
  dd_rowrange re;  /* row index as a certificate in the case of inconsistency */
  dd_colrange se;  /* col index as a certificate in the case of dual inconsistency */
  long pivots[5];
   /* pivots[0]=setup (to find a basis), pivots[1]=PhaseI or Criss-Cross,
      pivots[2]=Phase II, pivots[3]=Anticycling, pivots[4]=GMP postopt  */
  long total_pivots;
} dd_LPSolutionType;

\end{verbatim}

\section{Library Functions}  \label{LIBRARY}

Here we list some of the most important library functions/procedures.
We use the following convention:
{\tt poly} is of type {\tt dd\_PolyhedraPtr},
{\tt matrix}, {\tt matrix1} and {\tt matrix2} are of type {\tt dd\_MatrixPtr},
{\tt matrixP}, of type {\tt dd\_MatrixPtr*},
{\tt err} is of type {\tt dd\_ErrorType*},
{\tt ifile} and {\tt ofile} are of type {\tt char*},
{\tt A} is of type {\tt dd\_Amatrix},
{\tt point} and {\tt vector} are of type {\tt dd\_Arow},
{\tt d} is of type {\tt dd\_colrange},
{\tt m} and {\tt i} are of type {\tt dd\_rowrange},
{\tt x} is of type {\tt mytype},
{\tt a} is of type {\tt signed long integer},
{\tt b} is of type {\tt double},
{\tt set} is of type {\tt set\_type}.
  Also,
{\tt setfam} is of type {\tt dd\_SetFamilyPtr},
{\tt lp} is of type {\tt dd\_LPPtr},
{\tt lps} is of type {\tt dd\_LPSolutionPtr},
{\tt solver} is of type {\tt dd\_LPSolverType},
{\tt roworder} is of type {\tt dd\_RowOrderType}.

\subsection{Library Initialization}  \label{Initialization}

\begin{description}

\item[{\tt void dd\_set\_global\_constants(void)}]:\\
This is to set the global constants such as {\tt dd\_zero},
{\tt dd\_purezero} and
{\tt dd\_one} for sign recognition and basic arithmetic
operations.  {Every program to use cddlib must call this function}
before doing any computation.    Just call this once.
 See Section \ref{constants} for the definitions of
constants.

\item[{\tt void dd\_free\_global\_constants(void)}]:\\
This is to free the global constants. This should be called
when one does not use cddlib functions anymore.
\end{description}

\subsection{Core Functions}  \label{CoreLibrary}

There are two types of core functions in cddlib.  The first type
runs the double description (DD) algorithm and does a representation
conversion of a specified polyhedron.  The standard header
for this type is {\tt dd\_DD*}.  The second type solves
one or more linear programs with no special headers.
Both types of computations are nontrivial
and the users (especially for the DD algorithm) must
know that there is a serous limit in the sizes of problems
that can be practically solved.
Please check *.ext and *.ine files that come with cddlib to get
ideas of tractable problems.

In addition to previously defined objects, the symbol  {\tt roworder} is
of {\tt dd\_RowOrderType}. The symbol {\tt matrixP} is
a pointer to {\bf dd\_MatrixType}.
the arguments {\tt impl\_lin} and {\tt redset} are both a pointer
to {\tt dd\_rowset} type, and {\tt newpos} is a pointer to
{\tt dd\_rowindex} type.

\begin{description}
\item[{\tt dd\_PolyhedraPtr dd\_DDMatrix2Poly(matrix, err)}]:\\
Store the representation given by {\tt matrix} in a polyhedra data, and
generate the second representation of {\tt *poly}.  It returns
a pointer to the data. {\tt *err}
returns {\tt dd\_NoError} if the computation terminates normally.  Otherwise,
it returns a value according to the error occured.

\item[{\tt dd\_PolyhedraPtr dd\_DDMatrix2Poly2(matrix, roworder, err)}]:\\
This is the same function as  {\tt dd\_DDMatrix2Poly} except that the insertion
order is specified by the user.  The argument {\tt roworder} is of {\tt dd\_RowOrderType}
and takes one of the values:
  {\tt dd\_MaxIndex}, {\tt dd\_MinIndex}, {\tt dd\_MinCutoff}, {\tt dd\_MaxCutoff}, {\tt dd\_MixCutoff},
   {\tt dd\_LexMin}, {\tt dd\_LexMax}, {\tt dd\_RandomRow}.   In general, {\tt dd\_LexMin} is
the best choice which is in fact chosen in {\tt dd\_DDMatrix2Poly}.  If you know that
the input is already sorted in the order you like, use  {\tt dd\_MinIndex} or  {\tt dd\_MaxIndex}.
If the input contains many redundant rows (say more than $80\%$ redundant),
you might want to try {\tt dd\_MaxCutoff} which might result in much faster termination,
see \cite{abs-hgach-97,fp-ddmr-96}

\item[{\tt boolean dd\_DDInputAppend(poly, matrix, err)}]:\\
Modify the input representation in {\tt *poly}
by appending the matrix of {\tt *matrix}, and compute
the second representation.  The number of columns in
{\tt *matrix} must be equal to the input representation.

\item[{\tt boolean dd\_LPSolve(lp, solver, err)}]:\\
Solve {\tt lp} by the algorithm {\tt solver} and save
the solututions in {\tt *lp}.  Unlike the earlier versions
(dplex, cdd+), it can deal with equations and totally zero right
hand sides.   It is recommended that {\tt solver} is
{\tt dd\_DualSimplex}, the revised dual simplex method
that updates a $d\times d$ dual basis matrix in each pivot (where
$d$ is the column size of lp).

The revised dual simplex method is ideal for dense LPs in small number of variables
(i.e. small column size, typically less than $100$)
and many inequality constraints (i.e. large row size, can be a few ten thousands).
If your LP has many variables but only few constraints, solve the dual LP by
this function.

When it is compiled for GMP rational
arithmetic, it first tries to solve an LP with C  double
floating-point arithmetic and verifies whether the output
basis is correct with GMP.  If so, the correct solution is
computed with GMP.  Otherwise, it (re)solves the LP
from scratch with GMP.   This is newly implemented
in the version 093.  The original (non-crossover) version of
the same function is still  available as {\tt boolean dd\_LPSolve0}.

\item[{\tt dd\_boolean dd\_Redundant(matrix, i, point, err)}]:\\
Check whether $i$th data in {\tt matrix} is redundant for the representation.
If it is nonredundant, it returns a certificate.  For H-representation,
it is a {\tt point} in $R^d$ which satisfies
all inequalities except for the $i$th inequality.  If $i$ is a linearity,
it does nothing and always returns {\tt dd\_FALSE}.

\item[{\tt dd\_rowset dd\_RedundantRows(matrix, err)}]:\\
Returns a maximal set of row indices such that the associated rows
can be eliminated without changing the polyhedron.
The function works for both V- and H-representations.

\item[{\tt dd\_boolean dd\_SRedundant(matrix, i, point, err)}]:\\
Check whether $i$th data in {\tt matrix} is strongly redundant for the representation.
If $i$ is a linearity, it does nothing and always returns {\tt dd\_FALSE}.
Here,  $i$th inequality in H-representation is {\em strongly redundant\/} if it is redundant
and there is no point in the polyhedron satisfying the inequality with equality.
In V-representation,  $i$th point is {\em strongly redundant\/} if it is redundant
and it is in the relative interior of the polyhedron. If it is not strongly redundant, it returns a certificate.

\item[{\tt dd\_boolean dd\_ImplicitLinearity(matrix, i, err)}]:\\
Check whether $i$th row
in the input is forced to be linearity (equality
for H-representation).
If $i$ is linearity itself,
it does nothing and always returns {\tt dd\_FALSE}.

\item[{\tt dd\_rowset dd\_ImplicitLinearityRows(matrix, err)}]:\\
Returns the set of indices of rows that are
implicitly linearity.  It simply calls the library function
{\tt dd\_ImplicitLinearity} for each inequality and collects
the row indices for which the answer is {\tt dd\_TRUE}.

\item[{\tt dd\_boolean dd\_MatrixCanonicalize(matrixP, impl\_lin, redset, newpos, err)}]:\\
 The input is a pointer {\tt matrixP} to a matrix and the function
modifies the matrix by putting a maximally linear independent linearities (basis)
at the top of the matrix, and removing all redundant data.
All implicit linearities and all (removed) redundant rows
in the original matrix will be returned in the corresponding row sets.
The new positions of the original rows are returned by
the array {\tt newpos}.

The cardinality of the new linearity set {\tt  (*matrixP)->linset} is the codimension
of the polyhedron if it is H-polyhedron, and is the dimension of linearity space
if it is V-polyhedron.

Note that the present version should not be called a canonicalization
because it may generate two different representations of the same
polyhedron.  In the future, this function is expected to be correctly
implemented.

\item[{\tt dd\_boolean dd\_MatrixCanonicalizeLinearity(matrixP, impl\_linset, newpos. err)}]:\\
It does only the first half of {\tt dd\_boolean dd\_MatrixCanonicalize}, namely, it detects all
implicit linearities and puts a maximally independent linearities
at the top of the matrix.  For example, this function can be
used to detect the dimension of an H-polyhedron.

\item[{\tt dd\_boolean dd\_MatrixRedundancyRemove(matrixP, redset, newpos, err)}]:\\
It does essentially the second half of {\tt dd\_boolean dd\_MatrixCanonicalize},
namely, it detects all
redundancies.  This function should be used after {\tt dd\_MatrixCanonicalizeLinearity}
has been called.

\item[{\tt dd\_boolean dd\_FindRelativeInterior(matrix, impl\_lin, lin\_basis, lps, err)}]:\\
Computes a point in the relative interior of an H-polyhedron given by matrix, by solving
an LP. The point will be returned by {\tt lps}.
See the sample program allfaces.c that generates all nonempty faces of an H-polyhedron and
a relative interior point for each face.   The former returns all implicit linearity rows (implicit equations)
and the latter returns a basis of the union of linearity rows and implicit linearity rows.
This means that the cardinality of {\tt *lin\_basis} is the codimension of the polyhedron.

\item[{\tt dd\_boolean dd\_ExistsRestrictedFace(matrix, R, S, err)}]:\\
Returns the answer to the Farkas' type decision problem as to whether there is a point
in the polyhedron given by matrix satisfying all constraints in {\tt R} with
equality and all constraints in {\tt S} with strict inequality.  More precisely,
it is the linear feasibility problem:
\[
\begin{array}{llllll}
\exists\mbox{?} &x  &\mbox{ satisfying } & b_r - A_r x  &= 0, \; \forall r \in R\cup L \\
                &   &                    & b_s - A_s x  &> 0, \; \forall s \in S \\
                &   &                    & b_t - A_t x  &\ge 0, \; \forall t \in T,
\end{array}
\]
where $L$ is the set of linearity rows of {\tt matrix}, and $T$ represents
the set of rows that are not in $R\cup L \cup S$.
Both {\tt R} and {\tt S} are of {\tt dd\_rowset} type.  The set $S$ is
supposed to be disjoint from both $R$ and $L$.
If it is not the case, the set $S$ will be considered as $S \setminus (R \cup L)$.

This function ignores {\tt matrix->representation}, and thus even if it is
set to {\tt dd\_Generator} or {\tt dd\_Unspecified}, it treats the matrix
as if it were inequality representation.

\item[{\tt dd\_boolean dd\_ExistsRestrictedFace2(matrix, R, S, lps, err)}]:\\
It is the same as the function {\tt dd\_ExistsRestrictedFace} except that
it returns also a certificate for the answer.  The certificate is a solution
to the bounded LP:
\[
\begin{array}{lllllll}
\mbox{(P)} &\max  z  &\mbox{ subject to } & b_r - A_r x  &   & = 0, \; \forall r \in R\cup L \\
           &         &                    & b_s - A_s x  &-z &\ge 0, \;  \forall s \in S \\
          &         &                    & b_t - A_t x   &   &\ge 0, \; \forall t \in T \\
         &         &                    & 1              & -z&\ge 0,
\end{array}
\]
where $L$ is the set of linearity rows of {\tt matrix}, and $T$ represents
the set of rows that are not in $R\cup L \cup S$.
The answer for the decision problem is YES if and only if the LP attains
an optimal and the optimal value is positive.  The dual solution (either
an optimal solution or a dual unbounded direction) can be considered
as a certificate for the NO answer, if the answer is NO.

This function ignores {\tt matrix->representation}, and thus even if it is
set to {\tt dd\_Generator} or {\tt dd\_Unspecified}, it treats the matrix
as if it were inequality representation.

\item[{\tt dd\_SetFamilyPtr dd\_Matrix2Adjacency(matrix, err)}]:\\
Computes the adjacency list of input rows using
the LP solver and without running the representation conversion.  When
the input is H-representation, it gives the facet graph of the polyhedron.
For V-representation, it gives the (vertex) graph of the polyhedron.
It is required that the input matrix is a minimal representation.
Run redundancy removal functions before calling this function,
see the sample code adjacency.c.

\item[{\tt dd\_SetFamilyPtr dd\_Matrix2WeakAdjacency(matrix, err)}]:\\
Computes the weak adjacency list of input rows using
the LP solver and without running the representation conversion.  When
the input is H-representation, it gives the graph where its nodes are the facets
two nodes  are adjacent if and only if the associated facets have
some intersection.
For V-representation, it gives the graph where its nodes are the vertices
and two nodes are adjacent if and only if the associated vertices
are on a common facet.
It is required that the input matrix is a minimal representation.
Run redundancy removal functions before calling this function,
see the sample code adjacency.c.

\item[{\tt dd\_MatrixPtr dd\_FourierElimination(matrix, err)}]:\\
Eliminate the last variable from a system of linear inequalities
given by matrix by using the Fourier's Elimination.  If the
input matrix is V-representation, {\tt  *err} returns
{\tt dd\_NotAvailForV}.   This function does not
remove redundancy  and one might want to call
redundancy removal functions afterwards. See the sample code fourier.c.

\item[{\tt dd\_MatrixPtr dd\_BlockElimination(matrix, set, err)}]:\\
Eliminate a set of variables from a system of linear inequalities
given by matrix by using the extreme rays of the dual linear system.
See comments in the code cddproj.c for details.  This might be
a faster way to eliminate variables than the repeated FourierElimination when
the number of variables to eliminate is large.
If the input matrix is V-representation, {\tt  *err}  returns {\tt dd\_NotAvailForV}.
This function does not remove redundancy  and one might want to call
redundancy removal functions afterwards. See the sample code projection.c.

\item[{\tt dd\_rowrange dd\_RayShooting(matrix, point, vector)}]:\\
Finds the index of a halfspace first left by the ray starting from
{\tt point} toward the direction {\tt vector}.  It resolves
tie by a lexicographic perturbation.  Those inequalities violated
by {\tt point} will be simply ignored.

\end{description}

\subsection{Data Manipulations}  \label{DataLibrary}

\subsubsection{Number Assignments}
For number assignments, one cannot use such expressions as {\tt x=(mytype)a}.
This is because cddlib uses an abstract number type ({\tt mytype})
so that it can compute with various
number types such as C double and GMP rational.
User can easily add a new number type by redefining
arithmetic operations in cddmp.h and cddmp.c.

\begin{description}

\item[{\tt void dd\_init(x)}]:\\
This is to initialize a {\tt mytype} variable {\tt x} and to set it
to zero.    This initialization has to be called before
any {\tt mytype} variable to be used.

\item[{\tt void dd\_clear(x)}]:\\
This is to free the space allocated to a {\tt mytype} variable {\tt x}.

\item[{\tt void dd\_set\_si(x, a)}]:\\
This is to set a {\tt mytype} variable {\tt x} to the
value of signed long integer {\tt a}.

\item[{\tt void dd\_set\_si2(x, a, b)}]:\\
This is to set a {\tt mytype} variable {\tt x} to the
value of  the rational expression {\tt a/b}, where
{\tt a} is signed long and  {\tt b} is unsigned long
integers.

\item[{\tt void dd\_set\_d(x, b)}]:\\
This is to set a {\tt mytype} variable {\tt x} to the
value of double {\tt b}.  This is available only
when the library is compiled without {\tt -DGMPRATIONAL}
compiler option.

\end{description}

\subsubsection{Arithmetic Operations for {\tt mytype} Numbers}

Below  {\tt x}, {\tt y}, {\tt z}  are of type {\tt mytype}.

\begin{description}

\item[{\tt void dd\_add(x, y, z)}]:\\
Set {\tt x} to be the sum of  {\tt y} and  {\tt z}.

\item[{\tt void dd\_sub(x, y, z)}]:\\
Set {\tt x} to be the substraction of  {\tt z}  from  {\tt y}.

\item[{\tt void dd\_mul(x, y, z)}]:\\
Set {\tt x} to be the multiplication of  {\tt y}  and  {\tt z}.

\item[{\tt void dd\_div(x, y, z)}]:\\
Set {\tt x} to be the division of  {\tt y}  over  {\tt z}.

\item[{\tt void dd\_inv(x, y)}]:\\
Set {\tt x} to be the reciplocal of  {\tt y}.

\end{description}

\subsubsection{Predefined  Constants} \label{constants}

There are several {\tt mytype} constants defined when {\tt dd\_set\_global\_constants(void)} is called.
Some constants depend on the double constant {\tt dd\_almostzero} which is normally set to $10^{-7}$ in cdd.h.
This value can be modified depending on how numerically delicate your problems are but an extra
caution should be taken.

\begin{description}

\item[{\tt mytype dd\_purezero}]:\\
This represents the mathematical zero $0$.

\item[{\tt mytype dd\_zero}]:\\
This represents the largest positive number which should be considered to be zero.  In the GMPRATIONAL
mode, it is equal to {\tt dd\_purezero}.   In the C double mode, it is set to the value of {\tt dd\_almostzero}.

\item[{\tt mytype dd\_minuszero}]:\\
The negative of {\tt dd\_zero}.

\item[{\tt mytype dd\_one}]:\\
This represents the mathematical one $1$.

\end{description}

\subsubsection{Sign Evaluation and Comparison for {\tt mytype} Numbers}

Below {\tt x}, {\tt y}, {\tt z} are of type {\tt mytype}.

\begin{description}

\item[{\tt dd\_boolean dd\_Positive(x)}]:\\
Returns {\tt dd\_TRUE} if {\tt x} is considered positive,  and  {\tt dd\_FALSE} otherwise.
In the GMPRATIONAL mode, the positivity recognition is exact.  In the C double mode,
this means the value is strictly larger than  {\tt dd\_zero}.

{\tt dd\_boolean dd\_Negative(x)} works similarly.

\item[{\tt dd\_boolean dd\_Nonpositive(x)}]:\\
Returns the negation of {\tt dd\_Positive(x)}.   {\tt dd\_Nonnegative(x)} works similarly.

\item[{\tt dd\_boolean dd\_EqualToZero(x)}]:\\
Returns {\tt dd\_TRUE} if {\tt x} is considered zero,  and  {\tt dd\_FALSE} otherwise.
In the GMPRATIONAL mode, the zero recognition is exact.  In the C double mode,
this means the value is inbetween {\tt dd\_minuszero} and  {\tt dd\_zero}
inclusive.

\item[{\tt dd\_boolean dd\_Larger(x, y)}]:\\
Returns {\tt dd\_TRUE} if {\tt x} is strictly larger than {\tt y},  and  {\tt dd\_FALSE} otherwise.
This is implemented as {dd\_Positive(z)} where {\tt z} is the subtraction of {\tt y}
from {\tt x}.
{\tt dd\_Smaller(x, y)} works similarly.

\item[{\tt dd\_boolean dd\_Equal(x, y)}]:\\
Returns {\tt dd\_TRUE} if {\tt x} is considered equal to  {\tt y},  and  {\tt dd\_FALSE} otherwise.
This is implemented as {dd\_EqualToZero(z)} where {\tt z} is the subtraction of {\tt y}
from {\tt x}.
\end{description}

\subsubsection{Polyhedra Data Manipulation}
\begin{description}

\item[{\tt dd\_MatrixPtr dd\_PolyFile2Matrix (f, err)}]:\\
Read a Polyhedra data from stream {\tt f} and store it in {\tt matrixdata}
and return a pointer to the data.

\item[{\tt dd\_MatrixPtr dd\_CopyInequalities(poly)}]:\\
Copy the inequality representation pointed by poly to {\tt matrixdata}
and return {\tt dd\_MatrixPtr}.

\item[{\tt dd\_MatrixPtr dd\_CopyGenerators(poly)}]:\\
Copy the generator representation pointed by poly to {\tt matrixdata}
and return {\tt dd\_MatrixPtr}.

\item[{\tt dd\_SetFamilyPtr dd\_CopyIncidence(poly)}]:\\
Copy the incidence representation of the computed representation
pointed by poly to {\tt setfamily}
and return {\tt dd\_SetFamilyPtr}.  The computed representation is
{\tt Inequality} if the input is {\tt Generator}, and the vice visa.

\item[{\tt dd\_SetFamilyPtr dd\_CopyAdjacency(poly)}]:\\
Copy the adjacency representation of the computed representation
pointed by poly to {\tt setfamily}
and return {\tt dd\_SetFamilyPtr}.  The computed representation is
{\tt Inequality} if the input is {\tt Generator}, and the vice visa.

\item[{\tt dd\_SetFamilyPtr dd\_CopyInputIncidence(poly)}]:\\
Copy the incidence representation of the input representation
pointed by poly to {\tt setfamily}
and return {\tt d\_SetFamilyPtr}.

\item[{\tt dd\_SetFamilyPtr dd\_CopyInputAdjacency(poly)}]:\\
Copy the adjacency representation of the input representation
pointed by poly to {\tt setfamily}
and return {\tt d\_SetFamilyPtr}.

\item[{\tt void dd\_FreePolyhedra(poly)}]:\\
Free memory allocated to {\tt poly}.

\end{description}

\subsubsection{LP Data Manipulation}
\begin{description}

\item[{\tt dd\_LPPtr dd\_MakeLPforInteriorFinding(lp)}]:\\
Set up an LP to find an interior point of the feasible region of {\tt lp}
and return a pointer to the LP.  The new LP has one new variable
$x_{d+1}$ and one more constraint:
$\max x_{d+1}$ subject to $b - A x - x_{d+1} \ge 0$ and $x_{d+1} \le K$,
where $K$ is a positive constant.

\item[{\tt dd\_LPPtr dd\_Matrix2LP(matrix, err)}]:\\
Load {\tt matrix} to {\tt lpdata} and return a pointer to the data.

\item[{\tt dd\_LPSolutionPtr dd\_CopyLPSolution(lp)}]:\\
Load the solutions of {\tt lp} to {\tt lpsolution} and
return a pointer to the data.  This replaces the old name
{\tt dd\_LPSolutionLoad(lp)}.

\item[{\tt void dd\_FreeLPData(lp)}]:\\
Free memory allocated as an LP data pointed by {\tt lp}.

\item[{\tt void dd\_FreeLPSolution(lps)}]:\\
Free memory allocated as an LP solution data pointed by {\tt lps}.

\end{description}

\subsubsection{Matrix Manipulation}
\begin{description}

\item[{\tt dd\_MatrixPtr dd\_CopyMatrix(matrix)}]:\\
Make a copy of matrixdata pointed by {\tt matrix} and return
a pointer to the copy.

\item[{\tt dd\_MatrixPtr dd\_AppendMatrix(matrix1, matrix2)}]:\\
Make a matrixdata by copying {\tt *matrix1} and appending
the matrix in {\tt *matrix2} and return
a pointer to the data.  The colsize must be equal in
the two input matrices.  It returns a {\tt NULL} pointer
if the input matrices are not appropriate.
Its {\tt rowsize} is set to
the sum of the rowsizes of {\tt matrix1} and {\tt matrix2}.
 The new matrixdata inherits everything else
(i.e. numbertype, representation, etc)
from the first matrix.

\item[{\tt int dd\_MatrixAppendTo(\& matrix1, matrix2)}]:\\
Same as {\tt dd\_AppendMatrix} except that the first matrix
is modified to take the result.

\item[{\tt int dd\_MatrixRowRemove(\& matrix, i)}]:\\
Remove the $i$th row of {\tt matrix}.

\item[{\tt dd\_MatrixPtr dd\_MatrixSubmatrix(matrix, set)}]:\\
Generate the submatrix of {\tt matrix} by removing the
rows indexed by {\tt set} and return a matrixdata pointer.

\item[{\tt  dd\_SetFamilyPtr dd\_Matrix2Adjacency(matrix, err)}]:\\
Return the adjacency list of the representation given by {\tt matrix}.
The computation is done by the built-in LP solver.  The representation
should be free of redundancy when this function is called.
See the function  {\tt dd\_rowset dd\_RedundantRows}
and the example program adjacency.c.

\end{description}

\subsection{Input/Output Functions}  \label{IOLibrary}

\begin{description}

\item[{\tt dd\_MatrixPtr dd\_PolyFile2Matrix (f, err)}]:\\
Read a Polyhedra data from stream {\tt f} and store it in {\tt matrixdata}
and return a pointer to the data.

\item[{\tt boolean dd\_DDFile2File(ifile, ofile, err)}]:\\
Compute the representation conversion for a polyhedron given
by a Polyhedra file ifile, and write the other representation
in a Polyhedra file ofile.  {\tt *err}
returns {\tt dd\_NoError} if the computation terminates normally.  Otherwise,
it returns a value according to the error occured.

\item[{\tt void dd\_WriteMatrix(f, matrix)}]:\\
Write {\tt  matrix} to stream {\tt f}.

\item[{\tt void dd\_WriteNumber(f, x)}]:\\
Write {\tt x} to stream {\tt f}.  If {\tt x} is of GMP mpq\_t rational $p/q$,
the output is $p/q$.  If it is of C double, it is formated as a double float
with a decimal point.

\item[{\tt void dd\_WritePolyFile(f, poly)}]:\\
Write {tt poly} to stream {\tt f} in Polyhedra format.

\item[{\tt void dd\_WriteErrorMessages(f, err)}]:\\
Write error messages given by {\tt err} to stream {\tt f}.

\item[{\tt void dd\_WriteSetFamily(f, setfam)}]:\\
Write the set family pointed by {\tt setfam} to stream {\tt f}.
For each set, it outputs its index, its cardinality,
a colon ``:'' and a ordered list of its elements.

\item[{\tt void dd\_WriteSetFamilyCompressed(f, setfam)}]:\\
Write the set family pointed by {\tt setfam} to stream {\tt f}.
For each set, it outputs its index, its cardinality or the
negative of the cardinality, a colon ``:''
 and the elements in the set or its complements whichever is smaller.
Whenever it outputs the complements, the cardinality is negated
so that there is no ambiguity.
This will be considered standard for
outputing incidence (*.icd, *ecd) and adjacency
(*.iad, *.ead) data in cddlib.   But there is some minor incompatibility
with cdd/cdd+ standalone codes.

\item[{\tt void dd\_WriteProgramDescription(f)}]:\\
Write the cddlib version information to stream {\tt f}.

\item[{\tt void dd\_WriteDDTimes(f, poly)}]:\\
Write the representation conversion time information on {\tt poly}
 to stream {\tt f}.

\end{description}

\subsection{Obsolete Functions}  \label{ObsoleteFunctions}
\begin{description}
\item[{\tt boolean dd\_DoubleDescription(poly, err)}]:
(removed in Version 0.90c)\\
The new function
{\tt dd\_DDMatrix2Poly(matrix, err)} (see Section~\ref{CoreLibrary})
replaces (and actually combines) both this and
{\tt dd\_Matrix2Poly(matrix, err)}.

\item[{\tt dd\_PolyhedraPtr dd\_Matrix2Poly(matrix, err)}]:
(removed in Version 0.90c)\\
See above for the reason for removal.

\item[{\tt dd\_LPSolutionPtr dd\_LPSolutionLoad(lp)}]:
(renamed in Version 0.90c)\\
This function is now called {\tt dd\_CopyLPSolution(lp)}.

\end{description}

\subsection{Set Functions in {\tt setoper} library}  \label{SetFunctions}

The cddlib comes with a simple set operation library {\tt setoper}.  The key
type defined is {\tt set\_type}.   A set is represented by a fixed length
binary strings.  Thus, the maximum length of a set must be declared when
it is initialized.

Below the symbols {\tt a},   {\tt b},  {\tt c} are
of type  {\tt set\_type}.   The symbols {\tt aP} is a
pointer to type  {\tt set\_type}, and {\tt s}, {\tt t} are of type {\tt long}.
Here are some of the functions defined.  See {\tt setoper.h} for a
complete listing.

\begin{description}

\item[{\tt void set\_initialize(aP, s)}]:\\
Allocate a {\tt set\_type} space of maximum cardinality {\tt s}
 and make it pointed by {\tt aP}.  The set is initialized as empty set.

\item[{\tt void set\_free(a)}]:\\
Free the  {\tt set\_type} space allocated for {\tt a}.

\item[{\tt void set\_copy(a, b))}]:\\
Set {\tt a} to be {\tt b}.   The set {\tt a} must be pre-initialized
with the same maximum cardinality as that of {\tt b}.

\item[{\tt void set\_addelem(a, t))}]:\\
Add an element  {\tt t} to a set {\tt a}.    The set  {\tt a} stays unchanged
if it contains the element {\tt t}.

\item[{\tt long set\_card(a))}]:\\
Return the cardinality of set {\tt a}.

\item[{\tt int set\_member(t, a))}]:\\
Return $1$ if  {\tt t} is a member of set {\tt a}, and $0$ otherwise.

\item[{\tt void set\_write(a))}]:\\
Print out the elements of set {\tt a} to {\tt stdout}.  The function {\tt void set\_fwrite(f, a))} output
to stream {\tt f}.

\end{description}

\section{An Extension of the CDD Library in GMP mode}  \label{GMPLIB}

Starting from the version 093, the GMP version of cddlib, {\tt libcddgmp.a}, contains
all cdd library functions in two arithmetics.   All functions with the standard prefix {\tt dd\_}
are computed with the GMP rational arithmetic as before.  The same fuctions with
the new prefix {\tt ddf\_} are now added to the library  {\tt libcddgmp.a} that are based
on the C  double floating-point arithmetic.  Thus these functions are equivalent to
 {\tt libcdd.a} functions, except that all functions and  variable types are with prefix  {\tt ddf\_} and
the variable type {\tt mytype} is replaced by {\tt myfloat}.

In this sense,  {\tt libcdd.a} is a proper subset of  {\tt libcddgmp.a} and in principle one can
do everything with  {\tt libcddgmp.a}.   See how the new {\tt dd\_LPSolve} is written in
cddlp.c.

\section{Examples}  \label{EXAMPLES}

See example codes such as testcdd*.c , testlp*.c, redcheck.c, adjacency.c, allfaces,c
and simplecdd.c
in the {\tt src} and {\tt src-gmp} subdirectories of the source
distribution.

\section{Numerical Accuracy}  \label{accuracy}
 A little caution is in order.  Many people have observed
numerical problems of cddlib when the floating version of cddlib
is used.   As we all know, floating-point computation
might not give a correct answer, especially when an input
data is very sensitive to a small perturbation.  When
some strange behavior is observed, it is always wise
to create a rationalization of the input
(for example, one can replace 0.3333333 with 1/3)
and to compute it with cddlib compiled with gmp rational
to see what a correct behavior should be.  Whenever the time
is not important, it is safer to use gmp rational arithmetic.

If you need speedy computation with floating-point arithmetic,
you might want to ``play with'' the constant {\tt dd\_almostzero}
defined in cdd.h:

\begin{verbatim}
   #define dd_almostzero  1.0E-7
\end{verbatim}
\noindent
This number is used to recognize whether a number is zero:
a number whose absolute value is smaller
than {\tt dd\_almostzero} is considered zero, and nonzero otherwise.
You can change this to modify the behavior of cddlib.  One might
consider the default setting is rather large for double
precision arithmetic.  This is because cddlib is made
to deal with highly degenerate data and it works better
to treat a relatively large ``epsilon'' as zero.

Another thing one can do is scaling.  If the values in one column of
an input is of smaller magnitude than those in another column,
scale one so that they become comparable.

\section{Other Useful Codes}  \label{CODES}
There are several other useful codes available for vertex enumeration and/or
convex hull computation  such as lrs, qhull, porta and irisa-polylib.
The pointers to these codes are available at
\begin{enumerate}
\item lrs by D. Avis \cite{a-lrshome-01} (C implementation of the reverse search algorithm
\cite{af-pachv-92}).

\item qhull by C.B. Barber \cite{bdh-qach-03} (C implementation of
the beneath-beyond method, see \cite{e-acg-87,m-cg-94},
which is the dual of the dd method).

\item porta by T. Christof and A. L{\"o}bel \cite{cl-porta-97} (C implementation
of the Fourier-Motzkin elimination).

\item IRISA polyhedral library by D.K. Wilde
\cite{w-ldpo-93b} (C implementation
of a variation of the dd algorithm).

\item PPL: the Parma Polyhedra Library \cite{b-pplhome} by R. Bagnara (C++ implementation of
a variation of the dd algorithm).

\item {\tt pd} by A. Marzetta \cite{m-pdcip-97} (C implementation of the primal-dual algorithm
\cite{bfm-pdmvf-97}).

 \item Geometry Center Software List by N. Amenta \cite{a-dcg}.

 \item Computational Geometry Pages by J. Erickson \cite{e-cgp}.

 \item Linear Programming FAQ by R. Fourer and J. Gregory \cite{fg-lpfaq}.

 \item ZIB Berlin polyhedral software list:\\
 \htmladdnormallink{ftp://elib.zib-berlin.de/pub/mathprog/polyth/index.html}
{ftp://elib.zib-berlin.de/pub/mathprog/polyth/index.html}.

\item Polyhedral Computation FAQ \cite{f-pcfaq-98}.
\end{enumerate}

\section{Codes Using Cddlib}  \label{USERCODES}

There are quite a few nice programs using some functions of cddlib.
Here are some of them.

\begin{enumerate}

\item {\tt LattE} \cite{dhhhty-latte-05} computes the number of lattice points
in a convex polytope.

\item {\tt Minksum} \cite{w-msv-05} is a program to compute the V-representation
(i.e. the set of vertices) of the Minkowski addition of several convex polytopes
given by their V-representation in $\R^d$.  It is an implementation in C++ language
of the reverse search algorithm \cite{f-fzctmacp-04} whose time complexity is
polynomially bounded by the sizes of input and output.

\item {\tt Gfan} \cite{j-gvum-05} is a program to list all reduced Gr\"obner
bases of a general polynomial ideal given by a set of generating polynomials
in $n$-variables.   It is an implementation in C++ language
of the reverse search algorithm \cite{fjt-cgf-05}.

\item {\tt TOPCOM} \cite{r-topcom-05} computes the combinatorial structure
(the oriented matroid) of a point configuration and enumerates all triangulations
of a point set.   It detects the regularlity of a triangulation using cddlib.

\end{enumerate}

\section*{Acknowledgements.}
I am  grateful to Tom  Liebling who
provided me with an ideal opportunity to visit EPFL
for the academic year 1993-1994.  Later, Hans-Jakob L\"uthi (ETHZ) and
Emo Welzl  (ETHZ) joined to support the
the development of cdd codes (cdd, cdd+, cddlib).
Without their generous and continuing support, the present form of
this program would not have existed.

There are many other people who helped me to improve cdd, in particular,
I am indebted to  David Avis,
Alexander Bockmayr, David Bremner, Henry Crapo, Istvan Csabai,
Francois Margot, Marc Pfetsch, Alain Prodon, J\"org Rambau, Dima Pasechnik,
Shawn Rusaw, Matthew Saltzman, Masanori Sato, Anders Jensen,
Ruriko Yoshida, Charles Geyer, Michal Kvasnica, Sven Verdoolaege
 (listed in arbitrary order) and those listed
in the HISTORY file.

\bibliographystyle{plain}

\bibliography{fukuda1,fukuda2}

\end{document}