%\documentclass[aps,twocolumn,groupedaddress,floatfix]{revtex4}
\documentclass[twocolumn,groupedaddress,floatfix]{article}
\usepackage{amsmath}
%\usepackage{epsfig}
%\usepackage{amssymb}
%\usepackage{alltt}
\usepackage[latin1]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[english]{babel}
\usepackage[dvipsnames]{xcolor}

\newcommand{\Pred}{\mathtt{Predecessor}}
\newcommand{\Succ}{\mathtt{Successor}}
\newcommand{\Extra}{\mathtt{Extra}}
\newcommand{\Codes}{\mathtt{Codes}}

\newcommand{\paragraphs}[1]{\emph{#1}.---}
\newcommand{\todiscuss}[1]{{\color{Mahogany}*** #1***}}

\def \plotwi{8.7 cm}
\begin{document}

\title{Merks - a small learning world}
\author{Bruno Sciolla}

%\begin{abstract}
%\end{abstract}
\maketitle

\section{Current Merk}

The world of merks contains individuals called merks.
Each has a genome, a neural network, and a physical existence in the stage, that describes where the merk is, and what it does.
\subsection{Neurons}
A neuron with index $i \in [0, N]$ is connected to other neurons via links stored in a (signed) adjacency matrix $l_{ij}$. $l_{ij} = \{ -1, 0, 1\}$. A neuron has an activation value $a_i$. There are three types of neurons: {\bf sensor} neurons, {\bf action} neurons and normal neurons. Each neuron also has a binary label $L_i$.

At each time step the activation is updated according to:
\begin{align}
& a_i(t+1) = \sigma \left( \sum_j l_{ij} a_j(t) + s_i(t) \right)
\end{align}
where $\sigma(x) = \{-1,x,1\}$ on the intervals $[-\infty, -1]$, $[-1, 1]$, $[1, \infty]$ respectively.
The external sensor contribution $s_i(t)$ is added if the neuron is a sensor neuron, the value of which is fixed by the stage (can be any external impulse like bumping on a wall, the proximity to resources, or any 'physical sense' that the merk can have of the world).

\subsection{Actions}
If the neuron is an action neuron, its activation value $a_i(t)$ results in an action on the stage. For example, if the forward action neuron has activation value 1, the merk steps forward. Other possible actions are turning left or right, jumping, kicking, attempting to kill or reproduce, etc.

\subsection{Genome}
The genome is a binary sequence of fixed universal length.
$b = \{ 011000101110 \cdots \}$, typically 1000 binary numbers.

\subsection{Rules}
The genome is read by a sequence reader to produce rules of the form:
\begin{align}
&\Pred \rightarrow \Succ, \Extra, \Codes\\
&\textrm{Example}: 010 \rightarrow \{\}, 00, 0111
\end{align}
The predecessor, the successor and extra are three binary label. In the example, $010$, an empty label $\{ \}$, and $00$. The codes block is a succession of 4 bits.

\subsection{From genome to rules}
The genome is read in order, starting from the beginning of the sequence.

A valid rule starts with a $11$.
The next segment defines $\Pred$, the next segment defines $\Succ$, the next segment defines $\Extra$ and finally $4$ bits are used to define $\Codes$.
Because segments do not have a fixed length, a special rule is used to define a segment. The rule is that the odd rank bites (first, third, fifth, etc.) define the length of the segment: the segment extends as far as there are zeros in odd bites. The even bites define the actual value of the segment. So for example:
\begin{align}
&1 \Rightarrow \{\}\\
&001 \Rightarrow 0\\
&011 \Rightarrow 1\\
&0a0b1 \Rightarrow ab\\
&0a0b0c1 \Rightarrow abc\\
&0a0b0c0d1 \Rightarrow abcd \\
&\ldots  \notag
\end{align}

The genome is traversed to produce rules using non-overlapping segments.
Only some rules are kept: a valid rule must have a nonempty $\Pred$ and either $\Succ$ or $\Extra$ must be nonempty. A label longer than 6 bits is truncated to a length of 6 bits. Otherwise no limitations are applied.

\subsection{From rules to network}
The network is initialized with 3 neurons labelled $001, 000, 01$ and links $001 \xrightarrow[]{1} 000$, $000 \xrightarrow[]{1} 01$. The number above the arrow means that the link value is 1 in the adjacency matrix, e.g. $l_{01} = 1$.
Notice that several neurons can have the same label in the following.

The network is built, iterating over each neuron. For each neuron $\mathtt{Neuron}$, the $\Pred$ of the rule matching the label of the neuron best is found.
Basically matching means that every bit of the rule agrees from left to right with the corresponding bit of the label. Thus the rule can be at most as long as the label to match.
For example for a label $\mathtt{Neuron} = 001$:
\begin{align*}
&\textrm{rule with } \Pred \;0 \textrm{ matches} \\
&\textrm{rule with } \Pred \;1 \textrm{ does not match} \\
&\textrm{rule with } \Pred \;00 \textrm{ matches} \\
&\textrm{rule with } \Pred \;01 \textrm{ does not match} \\
&\textrm{rule with } \Pred \;001 \textrm{ matches} \\
&\textrm{rule with } \Pred \;0011 \textrm{ does not match}
\end{align*}

So for each neuron $\mathtt{Neuron}$, one rule is choosed, the one with longest $\Pred$ that matches the label. If there is no rule, we go to the next neuron.
If there is a rule, the network is modified as follows.
If label $\Extra$ is empty, the neuron is just labelled with a new name: $\Succ$.
If label $\Succ$ is empty, then let $\mathtt{Neighbors}$ be every neuron whose name matches $\Extra$.
Then if the bit $\Codes[1]$ is 1, then a link is made $\mathtt{Neuron} \xrightarrow[]{c} \mathtt{Neighbors}$ where $c = 1$ if $\Codes[2] = 0$ and $c = -1$ if $\Codes[2] = 1$.
Also, if the bit $\Codes[3]$ is 1, then a link is made $\mathtt{Neighbors} \xrightarrow[]{c} \mathtt{Neuron}$ where $c = 1$ if $\Codes[4] = 0$ and $c = -1$ if $\Codes[4] = 1$.
If both $\Succ$ and $\Extra$ are nonempty, a new neuron with label $\Extra$ is created. The new neuron is created with every link that $\mathtt{Neuron}$ had with other neurons. Afterwards, $\mathtt{Neuron}$ is renamed to $\Succ$.
Also, a link $\mathtt{Neuron} \xrightarrow[]{c_2} \mathtt{Extra}$ is created if $\Codes[1] = 1$ and a link $\mathtt{Extra} \xrightarrow[]{c_4} \mathtt{Neuron}$ if $\Codes[3] = 1$ with signs $c_2$ and $c_4$ fixed by the bits  $\Codes[2]$ and  $\Codes[4]$ respectively.

\end{document}


%!TEX root = gabarit-doctorat.tex
\section*{Issue, hypothesis and objectives}
\phantomsection\addcontentsline{toc}{section}{Issue, hypothesis and objectives}
\subsection{Issue}

Following recent decades, the public is becoming increasingly aware of the need to apply rigorous standards throughout the food chain.

In this project, we want to improve these tests by developing quantitative methods for identifying thermal insulation anomalies in refrigerated vehicles for the transport of food. We also want to develop an aging model of the panels constituting the refrigerated "box" of the vehicle to predict its remaining useful life. The implementation of improved tests will ensure food safety during the transportation of refrigerated food.
Indeed, the coefficient k of a vehicle with a tendency in time to decrease because of the aging of the insulating panels (migration of the insulating gases out of the foam panels, detachments of the walls following the mechanical stresses undergone by the vehicle, etc.) . Currently the standards only require testing at commissioning and afterwards at six years. This duration is probably too long and a quantitative model allowing a prediction of the evolution of the coefficient $K$ would add rigor to the current process. In addition, the imposed measurement tests of $K$ are essentially done on each of the faces, without taking into account the corner zones which are especially at risk. Infrared thermography would identify and quantify local thermal anomalies so as to establish an overall measure of $K$ more accurate while "sounding the alarm" in case of serious failure.

On the other hand, tradition active infrared thermography with heating stimulation device can be impossible to detect thermal insulation anomalies directly in refrigerated vehicle, since it would be too difficult to heat the entire vehicle ``box". Therefore, the idea of cold approach (compressed air, liquid nitrogen, etc.) comes up having its own advantages under this circumstance: it will be more economical than the heating approach, and more beneficial since there is less thermal noises.

\subsection{Hypothesis}
For the part of the application of infrared thermography in ``cold food chain", the heat exchange coefficients inside and outside of the insulated ``box" will be constant when the ATP standard test arrives at a stable condition. Moreover, the heat diffusion will be mainly one dimension.

For the part of the exploration of cold approaches in infrared thermography, heat propagation will have the same performance, since the material conductivity will not change whether in hot approach or the cold one.
\subsection{Objectives}
In this project, we are particularly interested in the ``cold chain" and more particularly the transport of food in refrigerated vehicles (ie trucks, trailers, containers). The transportation of dairy products, meat and frozen foods are especially at risk. In addition, the ever-increasing cost of energy makes it possible to limit refrigeration to a minimum. It is therefore essential to ensure an irreproachable thermal insulation during the inspection of refrigerated vehicles. There are thermal insulation compliance tests to ensure satisfactory food transportability \citep{Geneva1970}. During these tests, measurements of heat transfer coefficients are measured (coefficient $ K $). Nonetheless, if there are some local flaws or thermal bridges inside the panels, which could not be measured by ATP tests, then the insulation can no longer be guaranteed. Infrared thermography can be particularly helpful regarding these issues. %On the other hand this measure proves rather rough. In this project, we want to improve these tests.

To this end, three objectives will be pursued in this thesis:
\begin{itemize}
	\item \textbf{Objective 1$ ^\circ $ } - Development of quantitative methods for identifying thermal insulation anomalies in refrigerated vehicles for the transport of food.
	\item \textbf{Objective 2$ ^\circ $ } - Development of an aging model of panels constituting the refrigerated ``box" of the vehicle in order to predict its objective remaining useful lives.
	\item \textbf{Objective 3$ ^\circ $ } - Development of a model using a cold stimulation device to identify thermal insulation anomalies in insulated materials.
\end{itemize}

The achievement of these three objectives of this thesis requires the following specific tasks (which can be carried out in parallel):

\S 1.1 Establishment of a thermal image database on a maximum of refrigerated vehicles. §1.2 In parallel, laboratory tests on a refrigerated "box" of reduced size placed in a climatic chamber will allow the elaboration of a first model of thermal behavior. §1.3 First approaches to the treatment of infrared images (identification of anomalies - at the junctions of the panels).

\S 2.1 Stripping of images from the bank will be done and image processing approaches will be developed to identify areas of anomalies. The images will be segmented and the thermal parameters will be calculated since the images will have been acquired under specified conditions \citep{Ibarra-Castanedo2013Methods}. §2.2 The refrigerated "box" will be subjected to mechanical and thermal stresses in order to accelerate its aging which will allow a validation of the model of thermal behavior. Realistic artificial defects (based on the knowledge acquired in §1.1 will also be realized in the ``box"). §2.3 A test campaign for final validation (it should be possible to validate a certain aging by comparing the current results with previous measurements by re-testing some vehicles measured in advance. § 1.1). §2.4 Latest improvements to the thermal prediction model.

\S 3.1 Attempt of identifying defects in insulated material with infrared thermography by a cold stimulation (compressed air, liquid nitrogen, etc.) \S 3.2 Comparison with traditional heating approaches in infrared thermography for Non-Destructive Testing. \S 3.3 Comparison in modeling and simulation tests.

\section{Methodology}
Thought infrared thermography has the advantages such as fast inspection, ease of deployment, contactless, security and easy access to results with the imaging capabilities, raw infrared thermography results is difficult to handle and analyze in case of reflections and non-uniform external stimulation. To improve the inspection results, there are various post-processing techniques which have been developed. The recently popular data analyzing and processing methods are presented in detail in the following subsections.
\subsection{Fourier Transform (FT)}
Among all data processing methods, Fourier Transform (FT) is particularly interesting because it helps retrieve phase and amplitude data from raw results, since our principal results are images, which can be seen as signals with two dimensions.

It is well-known that any wave-form, periodic or not, can be approximated by the sum of purely harmonic waves oscillating at different frequencies. The Continuous Fourier Transform (CFT) is then given by:
\begin{equation}
F(\omega) = \int_{-\infty}^{\infty}f(t)e^{-j\omega t}dt = A(\omega)e^{i\phi(\omega)}
\end{equation}
where $\omega = 2 \pi f$. the FT technique serves to transform the perception of signal from a time-based domain to a frequency-based domain.

In case of discrete situation, the Discrete Fourier Transform (DFT) is possible to analyze the data in the frequency domain:
\begin{equation}
F_n = \Delta t \sum_{k=0}^{N-1}T(k\Delta t)e^{-\tfrac{i2\pi nk}{N}} = \Re(F_n) + i\Im(F_n)
\label{DFT}
\end{equation}
where $n$ designates the frequency increment ($n=0, 1, ..., N$), $\Delta t$ is the sampling interval, $N$ is the total number of infrared images, and $\Re$ and $\Im$ are the real and the imaginary parts of the transform, respectively.
DFT is often applied in Pulsed Phase Thermography (PPT), which analyzes phase data obtained from PT results. In addition, the Fast Fourier Transform (FFT) algorithm is often applied to reduce computation time.

\subsection{PPT}
From Eq.\ref{DFT}, amplitude $A_n$ and phase delay $\Phi_n$ are given by:
\begin{equation}
A_n = \sqrt{\Re(F_n)^2 + \Im(F_n)^2} \qquad \Phi_n = \tan^{-1}\frac{\Re(F_n)}{\Im(F_n)}
\end{equation}
It should be noted here that The processed sequence is less affected than the original data by undesired noise
sources such as environmental reflections, emissivity variations, non-uniform heating.

By selecting two pixels, the first in correspondence of a reference zone, the second in correspondence of a possible defect zone, and following both of them in time, after the pulse, the two profiles shown in Fig. \ref{T_profile_PPT} are obtained. By taking the FFT of the two signals, the two profiles of amplitude (Fig. \ref{FT_AM_profile_PPT}) and phase (Fig. \ref{FT_PH_profile_PPT})
as a function of frequency are obtained.
\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.35]{art/T_profile_PPT}
	\caption{Temperature profiles of a reference zone (blue) and a defect zone (red)}
	\label{T_profile_PPT}
\end{figure}

\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.35]{art/FT_AM_T_profile_PPT}
	\caption{Amplitude as a function of frequency: blue reference, red defect}
	\label{FT_AM_profile_PPT}
\end{figure}

\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.35]{art/FT_PH_T_profile_PPT}
	\caption{Phase as a function of frequency: blue reference, red defect}
	\label{FT_PH_profile_PPT}
\end{figure}

\subsection{Differential Absolute Contrast (DAC)}
Traditionally, once the temperature of a sound area (reference zone)$T_s(t) $ and that of a defect zone $T_{def}(t) $ are known, contrast methods can be applied by simply:
\begin{equation}
C_{ac}(t) = T_{def}(t) - T_s(t)
\label{AC_eq}
\end{equation}
which is known as the absolute contrast. However, this method becomes inconvenient when the sound area cannot be practical defined. In addition, the common case of non-uniform heating has a strong effect on the results.

To improve this, the Differential Absolute Contrast (DAC) has proven its amelioration for non-uniform heating situations\citep{Benitez2008, pilla2002new}.

DAC method starts from Eq. \ref{PT_eq}, the surface temperature increase based on one-dimensional model of the Fourier equation after an instantaneous Dirac heating pulse is applied:
\begin{equation}
\Delta T = T(0,t) - T_0  = \frac{Q}{e\sqrt{\pi t}}
\label{PT_eq_2}
\end{equation}
The temperature of the sound area at the surface ($z=0$) $T_s$ at time $t_1$ is given by:
\begin{equation}
\Delta T_s(t_1) = \frac{Q}{e\sqrt{\pi t_1}}
\end{equation}
Then at time $t$, the temperature can be written as:
\begin{equation}
\Delta T_s(t) = \frac{Q}{e\sqrt{\pi t_1}} = \sqrt{\frac{t_1}{t}}\cdot \Delta T(t_1)
\end{equation}
where $t_1$ is a time between the thermal pulse lasting and the time at which the first temperature spot of the subsurface defects appear.\\
The absolute temperature contrast (Eq. \ref{AC_eq}) can be rewritten as:
\begin{align}
C_{ac}(t) = & [T_{def}(t) -T_0] - [T_s(t) - T_0] \\
		  = & \Delta T_{def}(t) - \sqrt{\frac{t_1}{t}}\cdot \Delta T(t_1)
\end{align}
DAC Applications on different type of material samples can be found in \citep{pilla2002new}, which has proven this method has an effective improvement on the signal to noise ratio.

\subsection{Temperature Signal Reconstruction (TSR)}
The thermographic signal reconstruction (TSR) data processing technique is one of the most recent improvements which raise thermography to the level of the most established NDE techniques \citep{shepard2003reconstruction, Balageas2015}.

Same as DAC method, Eq. \ref{PT_eq_2} can be rewritten  in logarithmic way as:
\begin{equation}
\log (\Delta T) = \log (\frac{Q}{e}) - \frac{1}{2}\log (\pi t)
\end{equation}
In a general way, as: %with degree $n$:
\begin{equation}
\log (\Delta T) = a_0 + a_1\log (t) + a_2[\log (t)]^2 +...+ [a_n\log(t)]^n
\end{equation}
This technique usually consists in the fitting of the experimental log-log plot by a polynomial of degree $n$, and also in the computation of the 1$^{st}$ and 2$^{nd}$ derivatives of the thermograms.

In practice, logarithmic data may vary from ideal one-dimensional behavior for a variety of reasons (e.g. material inhomogeneities, nonlinear camera response or background radiation contributions). Nevertheless, the logarithmic behavior exhibits remarkable consistency, in that pixels representing defect free areas are nearly linear, and pixels corresponding to subsurface defects depart from the near-linear signature at a particular time that is correlated to the depth of the defect \citep{Shepard2002,Shepard2003}.

\subsection{Principal Component Thermography (PCT)}
Principal Component Analysis (PCA) is a statistical analysis tool used for identifying patterns in data and expressing the data in a way to highlight the similarities and differences in patterns. When data dimension is very high, the pattern matching of the data then becomes very difficult. Thus, PCA comes to extract features and reduce redundancy by projecting the data onto a system of orthogonal components, which is known as PCT \citep{Rajic2002,Rajic2002a}.

Singular Value Decomposition (SVD) is often used in PCA. Generally, a matrix $M$ of $m\times n$ can be decomposed in the form of:
\begin{equation}
A = U R V^T
\end{equation}
where $U$ is an $M\times N$ unitary matrix, $R$ is a diagonal $N \times N$ matrix with non-negative real numbers on the diagonal, $V$ is an $N\times N$ unitary matrix, and $V^T$ is the conjugate transpose of $V$.

The 3D thermogram matrix needs to be reorganized as a 2D $M\times N$ matrix $A$ with the dimension of time along the columns and space. The columns of $U$ represent a set of orthogonal statistical modes known as Empirical Orthogonal Functions (EOFs) which describe the data spatial variations, while the time variations are represented by the Principal Components (PCs) which are arranged row-wise in matrix $V_T$. Then applying the SVD to the 2D matrix, the resulting U matrix, providing the spatial information, can be rearranged as a 3D sequence \citep{Ibarra-Castanedo2006}. The whole procedure is illustrated in Fig. \ref{PCT_SVD}.
\begin{figure}[htbp]
	\centering
	\subfloat[Thermographic data rearrangement from a 3D sequence to a 2D matrix]
	{
		\includegraphics[scale=0.7]{art/PCT_SVD_1}
	}
	\\
	\subfloat[Rearrangement of a 2D matrix into a 3D matrix containing the EOFs]
	{
		\includegraphics[scale=0.7]{art/PCT_SVD_2}
	}
	\caption{PCT algorithm procedure}
	\label{PCT_SVD}
\end{figure}

Beyond reducing redundancy, PCT is also proposed as a contrast enhancement approach.
The entire results obtained in this thesis are mostly post-processed by PCT method.

\subsection{Receiver Operating 	Characteristic (ROC)}
The Receiver operating characteristic (ROC) curve is a technique in statistics which helps visualize, organize and select classifiers based on their performance. ROC graphs have long been used in different fields as signal detection \citep{swets2000better}, diagnostic systems\citep{swets1988measuring}, medical decision\citep{zou2002receiver}, and the earliest one in machine learning\citep{spackman1989signal}. While being frequently chosen a standard method in several scientific fields, ROC are rarely applied in the thermographic field.\citep{Bison2014a}.

\subsubsection{ROC concept}
Basically, a classification model (classifier or diagnosis) is a mapping of instances between certain classes/groups. For a two-class prediction problem (also known as binary classification), in which the results are marked either as positive ($p$) or negative ($n$). Then there are four results in a binary classification. Therefore, given an instance, it the instance is positive and it is classified as positive, it is called as a \textit{true positive}; if it is classified as negative, it is called as a \textit{false negative}. If the instance is negative and it is classified as negative, it is called as a \textit{true negative}; if it is classified as positive, it is called as a \textit{false positive}. When given a set of instances, a two-by-two confusion matrix (also called a contingency table) can be constructed representing the dispositions of the set of instances.
\begin{figure}[ht]
	\centering
	\includegraphics[scale=1.2]{art/ROCintro_1}
	\caption{Confusion matrix}
	\label{ROCintro_1}
\end{figure}

The parameters' rates in Fig. \ref{ROCintro_1} are calculated as following:\\
The \textbf{true positive rate} (also called \textit{recall}):
\begin{equation}
tp\; rate = \frac{\Sigma \; \text{True positives}}{\Sigma \; \text{Total positives (or Condition positives)}}
\end{equation}
The \textbf{false positive rate} (also called \textit{false alarm rate}):
\begin{equation}
fp\; rate = \frac{\Sigma \; \text{False positives}}{\Sigma \; \text{Total negatives (or Condition negatives)}}
\end{equation}

Additional rates associated are defined as:\\
The \textbf{true negative rate} (also called \textit{specificity}):
\begin{equation}
tn\; rate = \frac{\Sigma \; \text{True negatives}}{\Sigma \; \text{Total negatives (or Condition negatives)}}
\end{equation}
The \textbf{false negative rate}:
\begin{equation}
fn\; rate = \frac{\Sigma \; \text{False negatives}}{\Sigma \; \text{Total positives (or Condition positives)}}
\end{equation}
And:
\begin{equation*}
\text{sensitivity} = \text{recall}
\end{equation*}

\subsubsection{ROC space}
The corresponding ROC curve graph is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings \citep{Fawcett2006}.  Since TPR is equivalent to sensitivity and FPR is equal to 1 − specificity, the ROC graph is also called the sensitivity vs (1 − specificity) plot. In the ROC space, each prediction result or instance of a confusion matrix represents one point. Fig. \ref{ROC_space} shows one example of four prediction points.
\begin{figure}[ht]
	\centering
	\includegraphics[scale=0.56]{art/ROC_space-2}
	\caption{The ROC space}
	\label{ROC_space}
\end{figure}
There are several points in ROC space which are important to note. The lower left point $ (0,0) $ represents the strategy of never issuing a positive classification; such a classifier commits no false positive errors but also gains no true positives. The opposite strategy, of unconditionally issuing positive classifications, is represented by the upper right point $ (1,1) $. The point $ (0,1) $ represents perfect classification.  Informally, one point in ROC space is better than another if it is to the northwest (tp rate is higher, fp rate is lower, or both) of the first. In the figure, $C'$ has the best performance as shown.

The diagonal line (also called \textit{line of no-discrimination}) $y = x$ represents the strategy of randomly guessing a class. For example, if a classifier randomly guesses the positive class half the time, it can be expected to get half the positives and half the negatives correct; this yields the point $ (0.5, 0.5) $ in ROC space. The diagonal divides the ROC space. Points above the diagonal represent good classification results (better than random), points below the line represent poor results (worse than random). Note that the output of a consistently poor predictor could simply be inverted to obtain a good predictor. As $C$ and $C'$ shown in the figure.

In conclusion,  any classifier that produces a point in the lower right triangle can be negated to produce a point in the upper left triangle. Any classifier on the diagonal may be said to have no information about the class.

\subsubsection{Area under curve analysis}
In ROC analysis, a common method to compare different classifiers is to calculate the area under the ROC curve, known as AUC\citep{Fawcett2006}. In this way, a single scalar value will represent the expected performance of the classifiers. As in the ROC curve profile, the AUC is a part of the unit square are, therefore, its value will be between 0 and 1. Thus, random guessing classifier produces a diagonal line between (0,0) and (1,1), which has an area of 0.5. It can be then indicated that a classifier AUC value less than 0.5 is even worse than a random guessing.

An important statistical property of the AUC is that a classifier's AUC value is equivalent to the probability that the classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance.

\section{Modeling and simulation}
In this thesis, all modeling and simulation work are undertaken in the platform COMSOL Multiphysics{\textregistered}, which might be helpful when comparing with the experimental results.
%\section{Introduction to COMSOL Multiphysics}

COMSOL Multiphysics is a general-purpose software platform, based on advanced numerical methods, for modeling and simulating physics-based problems. It is a finite element analysis, solver and Simulation software / Finite Element Analysis (FEA) Software package for various physics and engineering applications, especially coupled phenomena, or multiphysics.

The advantages of using COMSOL Multiphysics for modeling:
\begin{itemize}
	\item COMSOL Multiphysics has an integrated modeling environment.
	\item COMSOL Multiphysics takes a semi-analytic approach: once questions specified, COMSOL symbolically assembles finite-element method matrices and organizes the bookkeeping.
	\item COMSOL Multiphysics is fully compatible with MATLAB, so user could define programming for the modeling,organizing the computation, or the post-processing has full functionality. COMSOL Script is a MATLAB-like integrated programming environment that can also provide these facilities.
	\item COMSOL Multiphysics provides pre-built templates as Application Modes and in the Model Library for common modeling applications.
	\item COMSOL Multiphysics provides multiphysics modeling--linking well known ``application modes" transparently.
	\item COMSOL Multiphysics innovated extended multiphysics--coupling between logically distinct domains and models that permits simultaneous solution.
\end{itemize}

One example of application of COMSOL simulation in infrared thermography for NDT \& E can be found in \citep{Cannas2012Modeling}. Where a small concrete wall with an inside cavity was heated by two halogen lamps to detect the defects position. A 3D model by finite element method under COMSOL platform has also been implemented to simulate the heating process. Experimental and numerical data have well matched, which allows parametric studies free from the experimental tests.

%All simulation parameters and conditions can be found in corresponding chapters.
%\bibliographystyle{plainnat}              % style de la bibliographie
%\bibliography{U:/Desktop/Bibliography/Biblio_th}


\section{Evaluation: Strengths \& Limitations}\label{sec:stringmatcher:evaluation}

Verification of Parallel String Matching is the first realistic
proof that uses (Liquid) Haskell
to prove properties \textit{about} program functions.
%
In this section we use the String Matching proof
to quantitatively and qualitatively evaluate theorem proving in Haskell.

\paragraph{Quantitative Evaluation.}
The Correctness of Parallel String Matching proof
can be found online~\cite{implementation}.
%
Verification time, that is the time Liquid Haskell needs to check the proof,
is 75 sec on a dual-core Intel Core i5-4278U processor.
%
The proof consists of \textit{1839} lines of code.
%
Out of those
\begin{itemize}
\item \textit{226} are Haskell ``runtime'' code,
\item \textit{112} are liquid comments on the ``runtime'' Haskell code,
\item \textit{1307} are Haskell proof terms, that is functions with @Proof@ result type, and
\item \textit{194}  are liquid comments to specify theorems.
\end{itemize}
Counting both liquid comments and Haskell proof terms as verification code,
we conclude that the proof requires 7x the lines of ``runtime'' code.
%
This ratio is high and takes us to 2006 Coq,
when Leroy~\cite{Leroy06formalcertification} verified
the initial CompCert C compiler with
the ratio of verification to compiler lines being 6x.
% 4,400 lines of compiler code and 28,000 lines devoted to verification.

\paragraph{Strengths.}
Though currently verbose,
deep verification using Liquid Haskell has many benefits.
%
First and foremost,
\textit{the target code is written in the general purpose Haskell}
and thus can use advanced Haskell features, including
type literals, deriving instances, inline annotations
and optimized library functions like @ByteString@.
Even diverging functions can coexist with the target code, as long
as they are not reflected into logic~\cite{Vazou14}.

Moreover, \textit{SMTs are used to automate the proofs}
over key theories like linear arithmetic and equality.
%
As an example, associativity of @(+)@
is assumed throughout the proofs while shifting indices.
%
Our proof could be further automated
by mapping refined strings to SMT strings and
using the automated SMT string theory.
%
We did not follow this approach because we want to show that
our techinique can be used to prove any (and not only domain specific)
program properties.

Finally, we get further automation via
\textit{Liquid Type Inference}~\cite{LiquidPLDI08}.
%
Properties about program functions,
expressed as type specifications with unit result,
often depend on program invariants,
expressed as vanilla refinement types, and vice versa.
%
For example, we need the invariant that all indices of
a string matcher are good indices
to prove associativity of @(mappend)@.
%%NV simplify the above.
%%%For example, we need the invariant that all indices of
%%%a string matcher are good indices
%%%to prove that
%%%when the target is smaller than the input
%%%then the indices field is an empty list,
%%%which is required to prove associativity of @(mappend)@
%
Even though Liquid Haskell cannot currently synthesize proof terms,
it performs really well at inferring and propagating program invariants (like good indices)
via the abstract interpretation framework of Liquid Types.

\paragraph{Limitations.}
There are severe limitations that should be addressed
to make theorem proving in Haskell a pleasant and usable technique.
%
As mentioned earlier \textit{the proofs are verbose}.
%
There are a few cases where the proofs require domain specific knowledge.
%
For example, to prove associativity of string matching
@x mappend (y mappend z) = (x mappend y) mappend z@
we need a theorem that performs case analysis on the relative length of
the input field of @y@ and the target string.
%
Unlike this case split though, most proofs
do not require domain specific knowledge and merely proceed
by term rewriting and structural inductuction
that should be automated
via Coq-like~\cite{coq-book} tactics or/and Dafny-like~\cite{dafny} heuristics.
%
For example, synquid~\cite{polikarpova16} could be used to automatically
synthesize proof terms.

Currently, we suffer from two engineering limitations.
%
First, all reflected function should exist in the same module,
as reflection needs access to the function implementation
that is unknown for imported functions.
%
This is the reason why we need to use a user defined,
instead of Haskell's built-in, list.
%
In our implementation we used @CPP@ as a current workaround
of the one module restriction.
%
Second, class methods
cannot be currently reflected.
%
Our current workaround is to define Haskell functions instead
of class instances.
For example (@append@, @nil@) and (@concatStr@, @emptyStr@)
define the monoid methods of List and Refined String respectively.

Overall, we believe that the strengths outweigh the limitations which
will be addressed in the near future,
rendering Haskell a powerful theorem prover.


\section{The Euclidian metric}


\documentclass[a4paper, 12pt]{article}

\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage [autostyle, english = american]{csquotes}
\MakeOuterQuote{"}
\usepackage{url}
\usepackage{import}
\usepackage{tabularx}
\usepackage{booktabs}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage[margin=1.25in]{geometry}
\usepackage{caption}
\usepackage{multirow}
\usepackage[table]{xcolor}
\usepackage{rotating}
\usepackage{mathtools}
\usepackage[multiple]{footmisc}
\usepackage{xr}
\usepackage{breakcites}
\usepackage{matlab-prettifier}
\usepackage[]{mcode}
\usepackage{listings}
\usepackage{color}
\usepackage{hyperref}
\usepackage{authblk}

\title {Profile Ranking Adaptive Choice-Based Conjoint Analysis: A Simple Standby to Utility-Based Analysis for Small Survey Populations}

 \author[1]{}
% \author[1]{Skyler Laney\thanks{skyler.laney@my.wheaton.edu}}

% \author[1]{Leo O'Malley\thanks{leo.omalley@my.wheaton.edu}}
% \author[1]{Cathy Shi\thanks{cathy.shi@my.wheaton.edu}}
% \author[1]{Danilo Diedrichs\thanks{danilo.diedrichs@wheaton.edu}}
% \author[1]{Nate Schatz\thanks{Nate.Schatz@my.wheaton.edu}
% \affil[1]{Department of Mathematics, Wheaton College}
\date{}

\usepackage[]{mcode}
\usepackage{matlab-prettifier}
\usepackage{listings} %For code in appendix
\usepackage{color} %red, green, blue, yellow, cyan, magenta, black, white
\definecolor{mygreen}{RGB}{28,172,0} % color values Red, Green, Blue
\definecolor{mylilas}{RGB}{170,55,241}
\usepackage{gensymb}
\usepackage{makeidx}
\makeindex
\pagestyle{empty}
\usepackage{endnotes}
\usepackage{lineno}
%\linenumbers
\begin{document}

%%%%For color MATLAB Scripts
\lstset
{ %Formatting for code in appendix
    language=Matlab,
    basicstyle=\scriptsize,
    numbers=left,
    stepnumber=1,
    showstringspaces=false,
    tabsize=1,
    breaklines=true,
    breakatwhitespace=false,
}
\maketitle
\hrulefill
\externaldocument{targettable}
\externaldocument{SimpleEquityModel}

 \vspace{.7in}

 \begin{abstract}
Serving as a standby to support Adaptive Choice-Based Conjoint (ACBC) surveys of very small populations, a new methodology called profile ranking based ACBC (PR-ACBC) is introduced as a simple way to overcome partworth utilities with possibly large variances. Without requiring any knowledge of partworth utilities, PR-ACBC begins with a simple computation using choice task data to obtain both individual and sample mean profile attribute rankings and their derived profile attribute level (PAL) rankings. The Maximum likelihood estimation of profile rankings for both a known population size $N$ (via multivariate hypergeometric distribution) and unknown $N$ (via Lagrange multiplier optimization) provide point estimates of population profile and PAL rankings.  The sample PAL rankings are also especially useful  for  multidimensional scaling (MDS) which offers a two-dimensional visual representation of similarities/dissimilarities in respondents.  Finally, PR-ACBC methodology is applied to a recent survey administered to  a small population of disaster relief organizations.  PAL rankings are compared with partworth utilities with respect to accuracy of choice task predictions and ranking of attribute importances.

\end{abstract}

%% THIS IS FOR A SHORT SCRIPT

% \begin{table}[!htpb]
%  \begin{tabular}{|l|}\toprule
%  {\bf MATLABScript.m}\\\hline
%  \parbox[b]{5.75in}{\lstinputlisting[style=Matlab-editor]{MATLABScript.m}}\\\hline\hline
%  \bottomrule
%  \end{tabular}
%  \end{table}

%% THIS IS FOR A LONG SCRIPT WHICH MUST BE SPLIT INTO
%% SHORTER BLOCK OF CODE YOU CAN SPECIFY THE
%% RANGE OF LINE NUMBERS DISPLAYED AND NUMBER
%% OF THE FIRST LINE

%  \begin{table}[!htpb]
% \centering
% \begin{tabular}{|l|}\hline
% MATLABScript.m. (p1 of 1)\\\hline
% \parbox[b]{5.8in}{\lstinputlisting[style=Matlab-editor,firstline=20, lastline=32, firstnumber=20]{MATLABScript.m}}\\\hline
% \end{tabular}
% \end{table}

 \vspace{1in}

\section{Introduction}

Adaptive Choice-Based Conjoint (ACBC) analysis surveys are a widely-utilized, well-developed, and highly effective  type of conjoint analysis (Orme and Chrzan, 2017).  In light of the fact that ACBC is not designed for use with very small populations ($N\le 100$),   we introduce Profile-Ranking (PR-) ACBC ss a new "value-added" methdod which applies ranking theory (Alvo and Yu 2014)) to data collected during the choice-task stage  to obtain attribute rankings and their associated profile attribute level (PAL) rankings.  PR-ACBC does not require any knowledge of partworth utilities, and thus serves as a simple type of validation of population inferences drawn from utility-based analysis. Unlike Maximum Difference (MaxDiff) scaling which is complementary to but not directly integrable into ACBC  (Sawtooth 2013), PR is easily integrated into ACBC choice task tournaments based on  the Method of Paired Comparisons (MPC) (Thurstone 1927).

A small population size $N$ for which PR-ACBC is designed  may be categorized based on a a rule of thumb for utility-based conjoint analysis (Orme 2014). In particular, letting $n$ be the number of survey respondents, $t$ the number of choice tasks, $a$ the number of alternatives per set, and $c$ the maximum number of levels in any attribute, we consider cases where $n < N << \frac{c}{at} \cdot 10^3$.

  Following the first application of ACBC to disaster relief (Gralla et al. 2014), we used Sawtooth's Lighthouse platform to structure our choice-task stage as a single-elimination tournament beginning with 16 profiles close to the respondent's \#1 profile revealed in the  ``Build Your Own'' (BYO) stage. The Lighthouse platform  offers  a sophisticated hierarchical-Bayesian Markov-Chain Monte-Carlo (HB MCMC) simulation (Rossi et. al. 2005) to estimate partworth utilities and their variances. Profile attribute rankings and their derived  profile attribute level (PALs) rankings on the other hand can easily be computed without knowledge of partworth utilities, and so serve as a simple validity check of utility-based  profile choice predictions and ranking of attribute importances for small populations.  PALs are also useful in  multi-dimensional scaling (MDS) (Alvo and Yu 2014) to give a 2-dimensional visual representation showing similarities in respondents and their ranking of attribute levels.

In Section 2, we introduce basic PR-ACBC methodology by means of a very simple generic (``toy'') survey with only 4 profiles constructed from 2 attributes each with 2 levels. We begin with a fundamental observation that the exact sample profile rankings and their derived PAL rankings directly obtainable from choice tournament data can not be obtained by multiple linear regression of attribute levels (part-worth utilities).  We therefore develop PR-ACBC using only survey tournament data without requiring any knowledge of partworth utilities.  Maximum likelihood estimate (MLE) population rankings for known population sizes are obtainable by a discrete multivariate hypergeometric distribution (Oberhofer and Kaufman 1987), and  for unknown population sizes by multivariable calculus optimization using Lagrange multipliers (Stewart 2016).   Population ranking intervals which must contain the unknown population profile rankings are easily computed from sample profile rankings.Similarly, the PAL-intervals which must contain the actual population attribute level rankings are easily obtained from sample data.   We then show how sample PALs are useful for multi-dimensional scaling (MDS) which show similarities in respondents and PAL rankings via 2-dimensional geometric representations. This is useful for comparing and contrasting sample respondent subgroups and clustering of attributes.  In Section 3 we illustrate PR methodology using a recent ACBC survey deployed to international disaster relief organizations with a headquarters in the U.S. or Canada.    This small survey population provided the original context motivating this methodological study, and is a sequel to a novel application of ACBC in disaster-response research (Gralla et. al. 2014).   Finally, in Section 4, we suggest a couple of major directions for further research in PR-ACBC, which is currently still in its initial stage of development.

\section{PR-ACBC Methodology}
In this section we introduce PR-ACBC methodology using a simple``toy'' survey.

\subsection{Simple Example}

Consider a generic ACBC survey with just 2 attributes each having 2 levels. We designate the 4 possible profiles $A=11, B=10, C=01, D=00$, where $X=x_1x_2$ designates that profile $X$ has level $x_1$ for the first attribute and level $x_2$ for the second attribute. Suppose we have obtained by anonymous survey choice tournament results  for $n=4$ respondents from a population of size $N>4$.  A sample tournament outcome is shown in Figure \ref{SimpleTourn}
\begin{figure}[!htpb]
\centering
\includegraphics[width=1.75in, height=1.5in]{SimpleTourn.png}
\caption{A respondent whose tournament data ranks profile A=11 first, C=01 second, B=10 is ranked third and D=00 fourth.  }
\label{SimpleTourn}
\end{figure}

{\flushleft This} respondent profile ranking is denoted ACBD, meaning profile A=11 is ranked 1 (tournament winner), profile C=01 is ranked 2 (runner-up), profile B=10 third, and D=00 ranked 4th. The rationale for the latter 2 rankings is that assuming transitivity in match outcomes, the highest B could be ranked if all profiles were paired is 2, while D could only be ranked as high as 3.

In our toy survey with just 4 possible profiles, there are already 4!=24 possible profile rankings. In the next section we consider an actual survey of disaster relief organizations (DROs) with 4 attributes containing 3 levels each, administered to a population of roughly $N=50$ faith based disaster relief organizations. This survey  has 81 profiles and 81!=5 797 126 020 747 367 985 879 734 231 578 109 105 412 357 244 731 625 958 745 865 049 716 390
 179 693 892 056 256 184 534 249 745 940 480 000 000 000 000 000 000 profile rankings,  For a sample size of $n=13$, and a choice task stage  which provides each respondent's ranking of 16 profiles, a meaningful ranking of  all the profiles is evidently impossible. As a result, our approach to ranking profiles, one more akin to using partworth utilities to determine profile choices, is to rank individual attribute levels rather than profiles as a whole.

Our toy survey has just four profile levels which we denote $(x_1,x_2)$ where $x_1$ is the attribute number (1 or 2) and $x_2$ the level number (0 or 1). Note that each level appears in exactly two profiles. For each attribute level, we compute a respondent's profile attribute level (PAL) ranking as the average of the two profile rankings in which the level appears.  For example since $(1,0)$ appears in profiles $C$ and $D$, if a respondent's profile ranking is ACBD, then the $(1,0)$ PAL ranking is (2+4)/2=3. (See Table \ref{PAL1})

\begin{table}[!htpb]

	\centering
	\begin{tabular}{c|cccc}
		Profile&\multicolumn{4}{c}{PAL Ranking}\\
		Ranking&(1,1)&(1,0)&(2,1),&(2,0)\\\hline
		ABCD& 1.5&3.5&2&3\\
		ACBD& 2&3&1.5&3.5\\
		BADC&1.5&3.5&3&2 \\
		CBAD& 2.5&2.5&2&3 \\\hline
		$m$&1.875&3.375&2.125&2.875\\
		$s$ &.4787&.25&.6292&.6292\\
	\end{tabular}
	\caption{Toy survey sample PAL rankings ($n=4$, $m$=mean, $s$=standard deviation, profiles A=11,B=10,C=01, and D=00.)}
	\label{PAL1}
\end{table}

In our DRO survey, there are  a total of 12 PALs $(x_1,x_2)$ with $x_1\in\{1,2,3,4\}$ and $x_2\in\{1,2,3\}$. Each PAL occurs in 27 out of the 81 profiles and has roughly a 99.85\% chance of appearing in a  in a single elimination round of 16 tournament (and hence have a ranking between 1 and 16). PALs not appearing in a tournament are assigned a ranking of 24.5 (the average of rankings 17 through 32 which would be assigned to the first round losers had one earlier round been included in the tournament.)  In short, PAL rankings are far more meaningful than profile rankings.

The main questions we will develop in the sequel are:
\begin{itemize}
\item POPULATION INFERENCES: \emph{what can we infer from sample PAL rankings about the population PAL rankings?}; and
\item APPLICATION TO REAL SURVEYS:  \emph{How well do PAL rankings predict  choice-tasks and attribute importances in the DRO survey?}
\end{itemize}

\subsection{A Fundamental Observation}
In this section we show that  least squares multiple  regression (LSRM) will not in general give exact sample profile rankings and their corresponding PAL rankings. In other words, part-worth utilities can only approximate sample PAL rankings.

Least squares multiple linear regression (LSRM) can be used to predict sample PAL rankings as we will now explain using our toy survey.  Let $U$ denote the level of attribute 1, $V$ the level of attribute 2, and $Y$ the ranking of a profilw with levels $U$ and $V$. Table \ref{Tab7} gives the dataset \{($U_i,V_i,Y_i$)\} ($i = 1,...,16$) where

\begin{eqnarray*}
	U_i&=& 1 \textup{ if attribute 1 has level 1, and 0 if it has level 2}\\
	V_i&=& 1 \textup{ if attribute 2 has level 1, and 0 if it has level 2}\\
	Y_i&=& \textup{ Respondent's ranking of a profile with $U=U_i$, $V=V_i$.}
\end{eqnarray*}

\begin{table}[!htpb]
	\centering
	\small
	\begin{tabular}{cc|ccccc}	\hline
		$U$ & $V$ & Respondent 1&  Respondent 2& Respondent 3& Respondent 4\\  \hline
		1 &1&$Y_1$&$Y_2$&$Y_3$&$Y_4$\\
		1 &0&$Y_5$&$Y_6$&$Y_7$&$Y_8$ \\
		0 &1&$Y_9$&$Y_{10}$&$Y_{11}$&$Y_{12}$ \\
		0 &0&$Y_{13}$&$Y_{14}$&$Y_{15}$&$Y_{16}$ \\\hline
	\end{tabular}
	\caption{{\small Sample profile rankings by respondent ($n=4$).}}
	\label{Tab7}
\end{table}

This dataset has certain properties:
\begin{itemize}
	\item
	Each column consists of a respondent's profile rankings and so contains the numbers 1,2,3 and 4.
	\item
	Table \ref{Tab7} can also be represented in the form of Table \ref{Tab8}, by which we see that $$\sum U_i = \sum V_i = \sum U_i^2 = \sum V_i^2= 8, $$ and $$ \sum U_iV_i = 4,$$ where the symbol $\sum $ represents $\displaystyle \sum_{n=1}^{16}$.
\end{itemize}

\begin{table}[!htpb]
	\centering
	\small
	\begin{tabular}{cc|c}
		$U$ & $V$ & Rank\\ \hline
		1&	1&	$Y_1$\\
		1&	1&	$Y_2$\\
		1&	1&	$Y_3$\\
		1&	1&	$Y_4$\\
		1&	0&	$Y_5$\\
		1&	0&	$Y_6$\\
		1&	0&	$Y_7$\\
		1&	0&	$Y_8$\\
		0&	1&	$Y_9$\\
		0&	1&	$Y_{10}$\\
		0&	1&	$Y_{11}$\\
		0&	1&	$Y_{12}$\\
		0&	0&	$Y_{13}$\\
		0&	0&	$Y_{14}$\\
		0&	0&	$Y_{15}$\\
		0&	0&	$Y_{16}$\\\hline
	\end{tabular}
	\caption{{\small Dataset's ranking structure with respondents combined.}}
	\label{Tab8}
\end{table}
Using least squares multiple linear regression (LSMR) on the dataset in Table \ref{Tab8}, we estimate each sample profile ranking $Y_i$ as $\hat{Y}_i$:
$$
\hat{Y}_i=c_0 + c_1 U_i + c_2 V_i,
$$
{\flushleft where} the regression coefficients $c_0,c_1,c_2$ are determined by minimizing the sum of squared residuals (SSR):
$$
SSR = \sum_{n=1}^{16}(Y_i-\hat{Y}_i)^2
=\sum_{n=1}^{16}(Y_i-(c_0 + c_1 U_i + c_2 V_i))^2.
$$
To minimize the SSR, we set the partial derivatives with respect to $c_0,c_1$ and $c_2$, equal to zero:
$$\frac{\partial SSR}{\partial c_0} = \frac{\partial SSR}{\partial c_1} = \frac{\partial SSR}{\partial c_2} = 0.$$
This yields the linear system:
$$\begin{cases}
nc_0 +  c_1\sum U_i + c_2\sum V_i = \sum Y_i\\
c_0\sum U_i + c_1\sum U_i^2 + c_2\sum U_iV_i = \sum U_iY_i\\
c_0\sum V_i + c_1\sum U_iV_i + c_2\sum V_i^2 = \sum V_iY_i
\end{cases},$$\\
which is equivalent to the matrix equation:

\[
\begin{bmatrix}
n& \sum U_i& \sum V_i \\
\sum U_i&\sum U_i^2&\sum U_iV_i\\
\sum V_i& \sum U_iV_i& \sum V_i^2\\

\end{bmatrix}
%
\begin{bmatrix}
c_0 \\
c_1\\
c_2
\end{bmatrix}
=
\begin{bmatrix}
\sum Y_i\\
\sum U_iY_i\\
\sum V_iY_i
\end{bmatrix}
.
\]

{\flushleft Simplifying} the sums and using Cramer's Rule  we obtain the regression coefficients $c_0,c_1$ and $c_2$:

$$
c_0 =
\frac{
	\begin{bmatrix}
	\sum Y_i & 8 & 8\\
	\sum U_iY_i& 8 & 4\\
	\sum V_iY_i & 4 & 8
	\end{bmatrix}
}{256} = \frac{
	\begin{bmatrix}
	\sum Y_i & 2 & 2\\
	\sum U_iY_i& 2 & 1\\
	\sum V_iY_i & 1 & 2
	\end{bmatrix}
}{16},
$$
$$
c_1 =
\frac{
	\begin{bmatrix}
	16 & \sum Y_i & 8 \\
	8 & \sum U_iY_i& 4\\
	8 & \sum V_iY_i & 8
	\end{bmatrix}
}{256} = \frac{
	\begin{bmatrix}
	4 & \sum Y_i & 2 \\
	2 & \sum U_iY_i& 1\\
	2 & \sum V_iY_i & 2
	\end{bmatrix}
}{16}, \textup{and}
$$
$$
c_2 =
\frac{
	\begin{bmatrix}
	16 &  8 & \sum Y_i\\
	8 & 8 & \sum U_iY_i \\
	8 & 4 & \sum V_iY_i
	\end{bmatrix}
}{256} = \frac{
	\begin{bmatrix}
	4 & 2 & \sum Y_i\\
	2 & 2 & \sum U_iY_i \\
	2 & 1 & \sum V_iY_i
	\end{bmatrix}
}{16}.
$$

The LSMR predicted profile rankings  are given by:
$$\hat{Y}_{A} = c_0 + c_1 + c_2 =
\frac{2\sum U_iY_i + 2\sum V_iY_i - \sum Y_i}{16},$$
$$\hat{Y}_{B} = c_0 + c_1 =
\frac{-6 \sum V_iY_i + \sum U_iY_i + \sum Y_i}{16},$$
$$\hat{Y}_{C} = c_0 + c_2 =
\frac{-2\sum U_iY_i + 2\sum V_iY_i + \sum Y_i}{16}, \textup{  and}$$
$$\hat{Y}_{D} = c_0 =
\frac{-2\sum U_iY_i - 2\sum V_iY_i +3 \sum Y_i}{16}.$$
The corresponding actual sample profile rankings obtained by averaging the respondent rankings are:
$$\bar{Y}_{A} = \frac{Y_1+Y_2+Y_3+Y_4}{4},$$
$$\bar{Y}_{B} = \frac{Y_5+Y_6+Y_7+Y_8}{4},$$
$$\bar{Y}_{C} = \frac{Y_9+Y_{10}+Y_{11}+Y_{12}}{4}, \textup{  and}$$
$$\bar{Y}_{D} = \frac{Y_{13}+Y_{14}+Y_{15}+Y_{16}}{4}.$$

Let $\bar{f}(X)$ denote the average sample ranking of PAL $X$. For example $f(U=1)=\frac{\bar{Y}_A+\bar{Y}_B}{2}=\frac{\sum_{i=1}^8 Y_i}{8}$.  On the other hand, the LSMR prediction for this PAL's sample ranking is $\hat{f}(X)=(\hat{Y}_A+\hat{Y}_B)/2=c_0+c_1+\frac{c_2}{2}.$ The relationship between the sample PAL rankings and LSMR predicted PAL rankings is shown in Table \ref{LSMR}).

\begin{table}[!htpb]
	\centering
	\scriptsize
	\begin{tabular}{c|c|c}
		PAL & $\bar{f}(X)$ & $ \hat{f}(X)$ \\ \hline
		& & \\
		U=1 & $\frac{\bar{Y}_A+\bar{Y}_B}{2}$  &$c_0+c_1+\frac{c_2}{2}$\\
			& & \\
		U=0 & $ \frac{\bar{Y}_C+\bar{Y}_D}{2}$  &$c_0+\frac{c_2}{2}$ \\
			& & \\
		V=1 &$\frac{\bar{Y}_A+\bar{Y}_C}{2}$  &$c_0+c_2+\frac{c_1}{2}$ \\
			& & \\
		V=0 & $\frac{\bar{Y}_B+\bar{Y}_D}{2}$  &$c_0+\frac{c_1}{2}$  \\
		& & \\\hline
	\end{tabular}
	\caption{{\small Predicted PAL rankings using LSMR coefficients.}}
	\label{LSMR}
\end{table}

Table \ref{Tab9} shows that the average sample profile rankings and LSRM predicted profile rankings are different for each of the four profiles. In this case, the regression coefficients are $c_0=3.75$, $c_1=-1.25$, $c_2=-1.25$.

\begin{table}[!htpb]
	\centering
	\scriptsize
	\begin{tabular}{cc|cccc|c|c|c}
		\multicolumn{2}{c}{} &\multicolumn{4}{c}{Respondents}\\\hline
		$U$ & $V$ & R 1&  R 2& R 3& R 4 &Sample Rank&Predicted Sample Rank& $\mid$Residual Error$|$\\  \hline
		1 &1&1&1&3&1&1.5&3.75-1.25(1)-1.25(1)=1.25&.25\\
		1 &0&2&3&1&3&2.25&3.75-1.25(1)-1.25(0)=2.5&.25 \\
		0 &1&3&2&2&2&2.25 &3.75-1.25(0)-1.25(1)=2.5&.25 \\
		0 &0&4&4&4&4&4 &3.75-1.25(0)-1.25(0)=3.75&.25\\\hline
	\end{tabular}
	\caption{{\small Predicted rankings for the sample outcomes in Table \ref{PAL1}.}}
	\label{Tab9}
\end{table}

Geometrically, if the four points representing the sample profile rankings are co-planar, there is no error; otherwise, the LSMR predicted profile rankings will have a residual error (Figure  \ref{sec4fig}).  Such a geometric interpretation  is not possible for surveys involving more than 2 attributes, in which case standard LSMR residual analysis indicates the error in sample  profile rankings using regression coefficients.

\begin{figure}[!htpb]
	\centering
	\includegraphics[width=2in,height=2in]{sec4fig.png}
	\caption{{\small The LSRM predicted sample profile rankings ($\hat{Y}=3.75-1.25U-1.25V$ (square vertices) have residual errors as the actual sample profile rankings (dots) do not belong to the plane of regression. }}
	\label{sec4fig}
\end{figure}

\newpage
 LSMR will not in general predict exact PAL rankings, as the latter   are computed directly from the sample profile rankings (Table \ref{LSMR2}).

\begin{table}[!htpb]
	\centering
	\scriptsize
	\begin{tabular}{c|cccc|c|c|c}
		&\multicolumn{4}{c|}{Respondent PAL Ranking}&&$c_0=3.75,c_1=-1.25,c_2=-1.25$&\\\hline
		PAL & R 1&  R 2& R 3& R 4 &Sample PAL Rank&Predicted Sample Rank&$|error|$ \\  \hline
		$U=1$ &1.5&2&2.5&2&1.875&$c_0+c_1+\frac{c_2}{2}=1.875$&0\\
		$U=0$&3.5&3&2.5&3&3&$c_0+\frac{c_2}{2}$=3.125&.125 \\
		$V=1$&2&1.5&1.5&1.5&1.625 &$c_0+c_2+\frac{c_1}{2}=1.875$&.2 \\
		$V=0$&3&3.5&3.5&3.375&2.875 &$c_0+\frac{c_1}{2}=3.125$&.25\\\hline
	\end{tabular}
	\caption{{\small Actual vrs. LSMR predicted PAL rankings for the sample outcomes in Table \ref{PAL1}.}}
	\label{LSMR2}
\end{table}

\subsection{Maximum Likelihood Estimation of PAL-ranking Intervals}
We will proceed to develop PR-ACBC methodology without assuming any knowledge of partworth utilitities.  Since sample PAL rankings are easily computed from sample profile rankings, we discuss how to obtain the population profile rankings most likely to have given the observed sample profile rankings.
\subsubsection{Known Population Size}
Let us assume our population has size $N=7$, and that our sample was a random selection of $n=4$ out of these 7. The number $N=7$ is for simplicity of illustrating the relevant cnmputations only, and could be any number $N>n=4$.  We will now use a multivariate hypergeometric distribution to obtain the maximum likelihood estimate (MLE) population profile rankings, meaning the population which was must likely to have yielded the observed sample profile rankings.

Suppose in our sample with $n=4$,  three different profile rankings $O_1,O_2,O_3$ are observed, with ranking $O_2$ ocurring twice in the sample. We create what we shall call a \emph{factor table}, whose $k$ largest factors $f_{ij}=1+(n_i/j)$ are used to determine the MLE population (see Table \ref{FT})
\begin{table}[!htpb]
	\scriptsize
	\centering
	\begin{tabular}{r|ccc|c}\hline
		\multicolumn{5}{c}{Choose the $k=3$ largest factors $f_{ij}$ for a population size $N=n+k=n+3$.}\\
		+3& $f_{13}=4/3$ & $f_{23}=5/3$&$f_{33}=4/3$&\\
		+2& $f_{12}=3/2$ & $f_{22}=2$ &$f_{32}=3/2 $&\\
		+1& $f_{11}=2$  &$f_{21}=3$&$f_{31}=2 $ &\\\hline
		Number observed in sample:&$n_1=1$& $n_2=2$ & $n_3=1$ & Sample size: $n=4$\\\hline
		Ranking:&1=$O_1$&2=$O_2$&3=$O_3$
	\end{tabular}
	\caption{Probability factor table. The $k=3$ largest factors $f_{ij}=1+(n_i/j)$ are used to determine the MLE population with size $N=n+k=4+3=7$. }
	\label{FT}
\end{table}
{\flushleft  Let} $N_j$ denote the number of rankings $O_j$ in the population.  The number of factors $a_j$ chosen from column $j$ in the factor table indicates that a  population with $N_j= n_j+a_j$ is a MLE.    In our case, the latter is not unique.  One choice of 3 largest factors is $f_{11}=2, f_{21}=3,f_{22}=2$ so that $a_1=1$, $a_2=2$ and $a_3=0$. A population $Y$ in which $N_1=n_1+a_1=2$ , $N_2=n_2+a_2=4$, and $N_3=n_3+a_3=1$ is a MLE. Another choice of 3 largest factors  is $f_{11}=2, f_{21}=3,f_{31}=1$ so that $a_1=1$, $a_2=1$ and $a_3=0$. Population $Z$ in which $N_1=n_1+a_1=2$ , $N_2=n_2+a_2=3$, and $N_3=n_3+a_3=2$ is also a MLE. We can verify this is so by computing the respective probabilities $p_Y$ and $p_Z$ that our observed sample arises from respective populations $Y$ and $Z$ :

\begin{equation}
p_Y=\frac{C(2,1)C(4,2)C(1,1)}{C(7,4)}=\frac{f_{11} \cdot f_{21}f_{22}}{C(7,4)}
\end{equation}
{\flushleft and }
\begin{equation}
p_Z=\frac{C(2,1)C(3,2)C(2,1)}{C(7,4)}=\frac{f_{11}\cdot f_{21}\cdot f_{31}}{C(7,4).}
\end{equation}
{\flushleft This} procedure generalizes to any number of rankings $m$ which appear in a sample of size $n$ ( Oberhofer and Kaufman (1987)). Let $n=\sum_{j=1}^m n_j$ where $n_j$ is the number of ranking $j$ appearing in the sample.  Form the probability factor table with $f_{ij}=1+\frac{n_i}{j}$ ($i=1,...,r$, $j=1,...,m$ and $N=n+r$.)  Choose the $r$ largest factors in the latter table and let $a_j$ be the number of factors chosen in column $j$.  Then a population with   $N_j=n_j+a_j$ ($j=1,...,m$) is a MLE.
\subsubsection{Unknown Population Size}

Let us assume now that rather than having a specified size, the population size is an unknown value $N$.  In this case, we wish to determine the probability $p_i$  that a member of the population has ranking $O_i$.  As before, we assume that our sample is drawn randomly from the population. The probability $p$ that the observed sample consisting of one $O_1$, two $O_3$'s and one $O_9$ is

\begin{equation}
p=f(p_1,p_3,p_9)=\frac{4!}{1!2!1!}[p_1p_3^2p_9],
\end{equation}
\label{eq:1}
{\flushleft where} $g(p_1,p_3,p_9)=p_1+p_3+p_9=1$.
The values of $p_1^*,p_3^*,$ and $p_9^*$ which maximize $H(p_1,p_3,p_9)=\ln (f(p_1,p_3,p_9))$ (and hence also maximizes $p=f(p_1,p_3,p_9)$)  are obtained using Lagrange multipliers:
\begin{eqnarray*}
	\nabla H(p_1^*,p_3^*,p_9^*) & = & \lambda \nabla g
	(p_1^*,p_3^*,p_9^*),
\end{eqnarray*}
{\flushleft and therefore}
\begin{eqnarray*}
	\frac{1}{p_1^*} & = & \lambda\\
	\frac{2}{p_23^*} & = & \lambda\\
	\frac{1}{p_9^*} & = & \lambda.
\end{eqnarray*}
{\flushleft (The scalar quantity $\lambda$ is called a Lagrange multiplier.) Using} $p_1^*+p_3^*+p_9^*=1$ gives $\frac{1}{\lambda} + \frac{2}{\lambda}+\frac{1}{\lambda}=1$ and so $ \lambda = 4$.  Hence, the values $p_1^*=\frac{1}{4}, p32^*=\frac{1}{2}$, $p_9^*=\frac{1}{4}$ maximize the probability of the observed sample outcomes.

In general, let $n_k$ be the number of sample outcomes $O_k$ ($k=1,2,...,K$) and let $p_k$ be the probability that a respondent in the population  has outcome $O_k$ $(k= 1, 2, ..., K)$.  The likelihood function $f(p_1, p_2, ..., p_K)$ giving the probability of observing the sample values $n_1, ..., n_{K}$ is given by

\begin{equation}
f(p_1, ...., p_K)= \frac{n!}{n_1!n_2!\cdot\cdot\cdot n_K!} \prod_{k=1}^K p_k^{n_k},
\end{equation}
\label{eq:4}

{\flushleft with} $\sum_{k=1}^{K}n_k=n$ and $\sum_{k=1}^{K}p_k=1$.
We seek to find the values $p_1^*, ..., p_{K}^*$ which maximize the likelihood function $f$, or equivalently, the log-likelihood function

\begin{equation}
H(p_1, ..., p_K)=\ln f = \ln(n!) - \sum_{k=1}^{K} n_k! +\sum_{k=1}^{K} n_k\ln(p_k),
\end{equation}
\label{eq:5}
{\flushleft subject} to the constraint $g(p_1, ..., p_{K})=p_1+p_2+...+p_K=1$.  Properties of gradients imply that the optimal values $p_i^*$ must satisfy

\begin{equation}
\nabla H(p_1^*, ..., p_K^*) = \lambda \nabla g(p_1^*, ..., p_{K}^*).
\end{equation}
\label{eq:6}
{\flushleft It} follows that for $k=1, ..., K$,

\begin{equation}
\frac{n_k}{p_k^*}=\lambda.
\end{equation}
\label{eq:7}
{\flushleft Hence,} $n=\sum_{k=1}^{K} n_k =  \lambda \sum_{k=1}^{K} p_k^* = \lambda$, and so the probabilities $p_k^* = \frac{n_k}{n}$ give the maximum likelihood of the observed sample outcomes $n_k$ ($k=1, 2, ..., K$).  For any sample of size $n$ and number $n_k$ of observed outcomes $O_k$ ($k=1, 2, ...K$), the maximum likelihood probabilities $p_k^*=\frac{n_k}{n}$
indicate that for a population of size $N$, the expected number $N_k$ of outcomes $O_k$ is given by $E(N_k)=p_k N.$  A maximum-likelihood population could be simulated by augmenting the observed $n$ sample outcomes, where the probability of outcome $O_k$ at each draw is given by $p_k$.   For a large number of such randomly constructed populations of size $N$, for each $k$ the average number of population outcomes $O_k$ is approximately $p_k N$.

\subsection{Profile Ranking Intervals}

Maximum likelihood  provides a point estimates into the population profile rankings and their corresponding PAL rankings.  It is easy to construct intervals guaranteed to include the actual population  profile and attribute level rankings.

First, suppose a profile X in a sample of size $n$ has mean profile ranking $\rho_n(X)$.  It is easy to construct an interval which contains the mean ranking $\rho_N(X)$ for any population size $N>n$:

\begin{equation}
1+\frac{n}{N}(\rho_n(X)-1)\le \rho_N(X)  \le 4-\frac{n}{N}(4-\rho_n(X))
\label{eq6}
\end{equation}

{\flushleft This} interval containing $\rho_N(X) $ is obtained by either (i) assigning the  rank 1 to $X$ for all $N-n$ members of the population not in the sample (lower bound for $\rho_N(X)$); or (ii) assigning the rank 4 to $X$  for all $N-n$ non-sample population members (upper bound for $\rho_k(X)$). Let  $\lambda=\frac{N-n}{N}$ be the fraction of the population not included in the sample.  It is easy to show that
\begin{equation}
\rho_n(X)-\lambda(\rho_n(X)-1)\le \rho_N(X) \le \rho_n(X) + \lambda(4-\rho_n(X)).
\label{PRI}
\end{equation}

{\flushleft We} call (\ref{PRI}) a \emph{profile ranking interval} for profile $X$.  The length of this interval is
  $3\lambda$, where the 3 arises algebraically as the difference between  the extreme rankings 1 and 4.

\begin{figure}[!htpb]
\centering
\includegraphics[width=6.5in, height=1.75in]{Confidence_Interval.png}
\caption{Given a profile $X$ and its sample ranking $\rho_n(X)$, the length of the population profile ranking interval  is determined by $\lambda=\frac{k-n}{k}$, the proportion of the population who have not taken the survey.}
\label{AL}
\end{figure}

The profile ranking interval can be used to quantify sample bias. Assume that, out of a total population $N=8$, two respondents have refused to take the survey, four have completed it, and the other two have not yet replied. In this case, the length of the PRI is  $\frac{3(8-4)}{8}=3/2$. If one of the non-respondents is convinced to participate, the interval length is reduced to $\frac{3(8-5)}{8}=\frac{9}{8}$, and if both non-respondents participate, then the interval is further reduced to $\frac{3(8-6)}{8}=\frac{3}{4}$. In other words, the two non-respondents  cause the length of the PRI interval to be twice as large, an important consideration in seeking to elicit survey response.

In a similar way to constructing profile ranking intervals, if PAL $\chi$ has a sample ranking mean $\rho_n(\chi)$, it is easy to form  a \emph{PAL ranking interval} for any population $N>n$. For our toy survey,

\begin{equation}
r_n(\chi)+2(N-n)r_n(\chi)\le r_N(\chi) \le r_n(\chi)+8(N-n)r_n(\chi),
\end{equation}

{\flushleft which} is equivalent to
\begin{equation}
r_n(X)-\lambda(r_n(X)-1)\le r_N(X) \le r_n(X) + \lambda(4-r_n(X)).
\label{ALRI}
\end{equation}
{\flushleft with} $\lambda=\frac{N-n}{N}$. Note that (\ref{PRI}) and (\ref{ALRI}) have the same form, so the length of both intervals is $3\lambda$.

\subsection{Multimensional Scaling}
One further type of analysis of PAL ranking data is a 2-dimensional geometric representation known as multidimensional scaling (MDS) (Alvo and Yu 2014).
Fundamental to MDS is use of a distance measure $d(\mu,\nu)$ in which the more similar (resp. dissimilar) are a pair of rankings $\mu$ and $\nu$, the smaller (resp. larger) is their distance.  A variety of distance measures have been used for MDS, where it is conventional to number the objects being ranked $1,2,...,t$ and represent a ranking as a permutation, $\mu:S\rightarrow S$ where $S=\{1,2,...,t\}$ and $\mu(i)$ is the rank of object $i$. For example, Hamming distance (from coding theory) is defined as

 \begin{equation}
 d_H{(\mu,\nu)}=t-\sum_{i=1}^t I(\mu(i)=\nu(i)).
 \end{equation}
 {\flushleft The} indicator function $I(\cdot)$ equals 1 if the statement inside parenthesis is true and 0 otherwise.  Hamming distance counts the number of positions where the permutations are different.

 Another example is Spearman distance, which is akin to usual Euclidean distance

\begin{equation}
d_S{(\mu,\nu)}=\frac{1}{2}\sum_{i=1}^t(\mu(i)-\nu(i))^2.
\end{equation}

{\flushleft Note} that Hamming distance formula satisfies the three required metric properties:

\begin{itemize}
	\item NON-NEGATIVITY $d_S(\mu,\nu)\ge 0$ for all $\mu$, $\nu$, with equality holding if and only if $\mu=\nu$;
	\item SYMMETRY $d_S(\mu,\nu)=d_S(\nu,\mu)$
	\item TRIANGLE INEQUALITY $d(\mu,\nu)+d(\nu,\sigma)\ge d(\mu,\sigma)$.
\end{itemize}
{\flushleft Spearman distance}, however, only satisfies the first two metric properties (Alvo and Yu ).

 In applying MDS to PR-ACBC, we let $(i,j)$ denote level $j$ of attribute $i$,  and $f(i,j)$ the average rank of level $(i,j)$ in an individual respondent's profile ranking. For example, for the profile ranking BACD, $f(1,1)=(1+2)/2=1.5$ since B=10 is ranked first and A=11 is ranked second; $f(20)=(2+4)/2=3$ since B=10 is ranked second and $D=00$ is ranked 4th.

 For our toy survey, each respondent $X$ will have four average profile level rankings $f_X(1,1), f_X(1,0), f_X(2,1), f_X(2,0)$.  Given 2 respondents $X$ and $Y$, we consider the squared Euclidean distance between their PAL rankings defined as
\begin{equation}
d_S(X,Y)=\sum_{i,j} [f_X(i,j)-f_Y(i,j)]^2.
\label{Spea}
\end{equation}

{\flushleft This} distance measure can be used for an ACBC survey with any number of attributes and levels.

Once a distance measure is defined, a 2-dimensional MDS is such that rankings are represented by points in an xy Cartesian coordinate system, and the Euclidean distance between these points reflects the relative distances between rankings.
Note that in this MDS, respondents R2 and R4 coincide since they have the same ranking (ACBD).

\begin{figure}[!htpb]
	\centering
	\includegraphics[width=6.5in, height=1.75in]{MDS1.png}
	\caption{Example of an MDS using the distance function defined in (\ref{Spea}) . In this case the sample profile rankings are ABCD, ACBD (twice), and BCAD}
	\label{AL}
\end{figure}

\section{ACBC Survey of Humanitarian Disaster Relief Organizations}

PR-based ACBC methodology is designed to analyze small populations and has many possible applications. One application involves  a study of a small population ($N=61$) of international  disaster relief organizations headquartered in the U.S. or Canada as shown in Table \ref{ORGS}.

\begin{table}[!htpb]
	\scriptsize
	\centering
	\begin{tabular}{c|c}\hline
		\multicolumn{2}{c}{Survey Population: International Disaster Relief Organizations Headuarted in the US/Canada}\\\hline
		Faith-based (Christian)($N_1=49$) & \multicolumn{1}{m{2.75in}}{Adventist Community Services, Adventist Development and Relief Agency,	AMG international,	Baptist World Aid, Billy Graham Evangelistic Association (rapid response team), Blessings international, Brethren Disaster Ministries,	Catholic Charities USA, Catholic Medical Mission Board,	Church World Service, Christian Disaster Response,
			Christian Aid Ministries, Convoy of Hope, Cooperative Baptist Fellowship, Covenant World Relief,
			Episcopal relief and Development, Food for the Hungry, Food for the Poor, Habitat for Humanity,	Hopeforce International,
			International Aid, Lutheran Disaster Response, Lutheran World Relief, MAP international, MedAir,
			Medical Teams International, Mennonite Central Committee, Mission Aviation Fellowship, Missionary expeditors,
			The National Baptist Convention - Office of Disaster Management, Nazarene Compassionate Ministries,	Operation Blessing International,
			Operation Safe, Presbytarian Church in America- Mission to North America, Presbytarian Disaster Assistance,
			Reach Global, S. Baptist N. Amer. Mission Board, Salvation Army World Service Organization, 	Samaritan's Purse,
			Service International, SOS Children --not disaster relief, Tearfund USA, United Church of Christ Disaster Ministries,
			United Methodist Committee on Relief (UMCOR), Water Mission, World Concern,	World Relief, World Renew, World Vision	}\\\hline
		Non faith-based 	($N_2=12$)&	\multicolumn{1}{m{2.75in}}{ All Hands and Hearts,Americares, Direct Relief, Headwaters Relief Organization,	Heart to Heart, Mercy Corps, Partnership with Native Americans, 	Save the Children, 	ShelterBox USA, 	SBP USA,	Team Rubicon.}\\\hline
	\end{tabular}
	\caption{Small populations comprising an ACBC disaster relief organization (DRO) survey.}
	\label{ORGS}
\end{table}

In disaster relief, effectiveness of a response may depend on the quality of collaboration between organizations with a broad diversity of religious and ideological perspectives. For effective coordination of relief, it is important that humanitarian organizations understand the unique traits and characteristics that shape their disaster response decisions. Through comparison of these factors, it is possible to design optimal partnerships and joint endeavors between organizations that may fulfill distinct, yet complimentary, humanitarian roles. Our research focuses on a few key attributes affecting our population group's decision whether or not to respond to an international humanitarian disaster.

To this end, we designed an ACBC survey that creates disaster profiles with attributes and levels for this survey are displayed in Figure \ref{AL}. Different disaster scenarios are paired off  in the choice task (single elimination tournament) stage, beginning with 16 profiles close to the Build-Your-Own (BYO) or ideal scenario. The tournament data is the basis for profile ranking. The survey was deployed and tournament data collected using Sawtooth's Lighthouse platform.

\begin{figure}[!htpb]
\centering
\includegraphics[width=5.75in, height=7in]{AttributeLevels.png}
\caption{{\small An ACBC survey with 4 attributes consisting of 3 levels each. }}
\label{AL}
\end{figure}

\subsection{Survey Data}

As shown in Figure 4, our humanitarian survey consists of four attributes, each with three levels. Thus, the number of possible profiles is $3^4=81$. These are identified by four digit numbers $X=x_1x_2x_3x_4$ where profile $X$ has level $x_1$ for the first attribute, level $x_2$ for the second attribute, level $x_3$ for the third attribute, and level $x_4$ for the fourth attribute. In the tournament stage of the competition, there are four rounds, in which sixteen profiles face off against each other in head to head match-ups, much like the FIFA World Cup Round of 16. The competing profiles are selected from the 81 possible profiles based on the respondent's BYO preferences. We assign a ranking of \# 1 to the tournament winner, \# 2 to the runner up, \# 3 to the semifinalist who lost to \#1, \# 4 to the quarterfinalist who lost to \#1, \# 5 to the quarter-finalist who lost to \#2 and so on. Profiles that do not appear in the tournament are ranked \# 24.5 (the average of \# 17 through \#32).

An Excel file keeps track of each responding organization's tournament and the PAL ranking of each survey.

\begin{figure}[!htpb]
	\centering
	\includegraphics[width=2.75in, height=2in]{PAL2.png}
		\includegraphics[width=2.75in, height=2in]{PAL1.png}
	\caption{{\small Respondent 1's tournament (left) and profile rankings for organizations 1 and 2 (right). The latter is processed by the MATLAB script in Table \ref{MAT1} to obtain the PAL rankings. }}
	\label{AL}
\end{figure}

\begin{table}[!htbp]
	\flushleft
	\begin{tabular}{|l|}\toprule
		\parbox[b]{6.25in}{\lstinputlisting[style=Matlab-editor]{PALRanking.m}}\\\bottomrule
	\end{tabular}
\caption{Main MATLAB script to compute PAL rankings.}
	\label{MAT1}
\end{table}

\begin{table}[!htpb]
	\flushleft
	\begin{tabular}{|l|}\toprule
		\parbox[b]{6.25in}{\lstinputlisting[style=Matlab-editor]{respondent.m}}\\\bottomrule
	\end{tabular}
	\caption{Respondent Class used by the main script in Table \ref{MAT1}}
		\label{MAT12}
	\end{table}

\newpage

\subsection{Survey Analysis}
\begin{table}[!htpb]
	\centering
	\scriptsize
	\begin{tabular}{c|ccc|ccc|ccc|ccc}
		\multicolumn{1}{c}{}	&\multicolumn{3}{c}{Funding}&\multicolumn{3}{c}{Scale}&\multicolumn{3}{c}{Need}&\multicolumn{3}{c}{Community Access Partner}\\
		\multicolumn{1}{c}{}	&$\ge 75\%$&$\sim 50\%$&$<25 \%$& ISAE3&ISAE2& ISAE1/&clear&optional&unknown&none&local&outside\\
		\multicolumn{1}{c}{}	& & & & & & undeclared& & & & & & \\
		ORG & L11&L12&L13&L21&L22&L23&L31&L32&L33&L41&L42&L43\\\hline
FBO1 &20.2&{\bf 18.6}*&22.7&20.8&{\bf 18.2}*&22.5&  21.2&{\bf 17.9}*&22.4&21.5&{\bf 18.4}*&21.6\\
FBO2 &21.2&21.5&{\bf 18.8}*&21.3&21.4&{\bf 18.8}*&  {\bf 17.8}*&21.5&22.2&22.1&{\bf 17.1}*&22.3\\
FBO3 &21.1&{\bf 18.1}*&22.5&20.3&{\bf 18.5}*&22.8&   20.8&{\bf 18.7}*&22.1&21.8&21.9&{\bf 17.9}*\\
FBO4 &19.8&
{\bf 17.2}*&19.6&19.8&{\bf 17.6}*&19.3&19.4&{\bf 17.3}*&
20.0&20.8&{\bf 15.0}*&20.8
\\
FBO5 &19.0&19.2&{\bf 16.8}*&18.9&{\bf 16.8}*&19.6&
{\bf 16.5}*&19.6&19.2&19.1&{\bf 17.1}*&19.0\\
FBO6 &18.0& 17.9&{\bf 15.8}*&17.6&{\bf 15.9}*&
18.2&{\bf 15.3}*&18.2&18.2&18.8&{\bf 14}*&18.\\
FBO7 &19.3&{\bf 17.4}*&20.0&19.5&{\bf 17.4}*&
19.7&18.9&{\bf 17.8}*&19.9&20.8&{\bf 15.4}*&20.4\\
FBO8 &19.2&{\bf 17.3}*&20.1&19.6&{\bf 17.4}*&
19.6&{\bf 16.6}*&19.8&20.3&20.2&{\bf 15.8}*&20.6\\
FBO9 & 20.9&21.0&{\bf 19.6}*&21.3&21.6&
{\bf 18.7}*&20.6&{\bf 18.8}*&22.2&22.8&{\bf 16.3}*&22.4\\
FBO10&20.6&{\bf 18.4}*&22.4&20.9&{\bf 18.8}*&
21.9&{\bf 18}*&21.4&22.1&22.6&{\bf 16.8}*&22.1\\
FBO11&19.9&19.7&{\bf 17}*&19.0&{\bf 17.5}*&20.1&
18.9&{\bf 17.2}*&20.6&20.5&{\bf 15.4}*&20.8\\
FBO12&18.8&19.2&{\bf 16.2}*&18.7&{\bf 16.4}*&
19.1&17.9&{\bf 16.7}*&19.5&19.4&{\bf 15.4}*&19.4\\
FBO13&20.9&21.8&{\bf 18.8}*&21.7&{\bf 18.1}*&21.6&
{\bf 18}*&21.9&21.6&22.4&{\bf 17.3}*&21.8\\\hline
mean&19.93&{\bf 19.04}&19.26&19.96&{\bf 18.12}&20.15&{\bf 18.45}&18.98&20.79&21.00&{\bf 16.62}&20.6\\\hline
partworth&{\bf 14.94}&4.08&-19.03&{\bf 17.84}&16.93&-34.77&{\bf 53.87}&-1.24&-52.62&-66.2&{\bf 97.84}&-31.64\\\hline\hline
NGO1 &20.8&21.5&{\bf 19.2}*&21.7& {\bf 18.7}*&21.0&   {\bf 17.3}*&21.6&22.6&22.4&{\bf 18.2}*&21.0\\
NGO2 &20.3&22  &{\bf 20.0}*&{\bf 18.4}*& 21.1&22.8&   {\bf 18.8}*&20.9&22.6&21.3&{\bf 19.4}*&21.7\\
NGO3 &{\bf 17.54}&20.80&21.07&{\bf 18.91}*& 19.56&20.94&   {\bf 16.67}&21.70&21.04&19.70*&{\bf 18.54}&21.17\\
NGO4 &{\bf 19.6}&20.3*&21.4&{\bf 18.0}*& 21.3&22.2&   21.2&{\bf 19.2}*&21.1&22.1&{\bf 17.9}*&21.6\\
NGO5&16.5&{\bf 15.1}*&17.8&16.9&{\bf 15.3}*&
17.3&{\bf 14.2}*&17.5&17.8&17.3&{\bf 16.7}*&15.4\\
NGO6 &20.6&21.5&{\bf 19.4}*&21.2&21.4&{\bf 18.8}*&   {\bf 16.6}*&22.3&22.7&{\bf 18.4}*&21.6&21.4\\\hline
mean&{\bf 19.27}&20.129&19.82&{\bf 19.2}&19.56&20.52&{\bf 17.45}&20.54&21.3&20.22&{\bf 18.72}&20.36\\\hline
partworth& {\bf 31.8}&-4.77&-27.03&{\bf 29.26}&.48&-29.74&{\bf 86.13}&-1.41&-84.72&-50.36&{\bf 44.72}&5.63\\\hline
	\end{tabular}
\caption{FBO/NGO Individual PAL rankings ($n=13$) and sample means compared with average zero-centered part-worth utilities. Top-ranked level in {\bf bold} (* denotes BYO level)}
\label{PAL}
\end{table}

\subsubsection{PAL ranking intervals}

\subsubsection{MDS}
\begin{figure}[!htpb]
	\centering
	\includegraphics[width=5.5in, height=3.75in]{RScript.pdf}
	\caption{R-script used for the MDS shown in Figure \ref{MDSout1}}
	\label{AL}
\end{figure}

\begin{figure}[!htpb]
	\centering
	\includegraphics[width=6.75in, height=5in]{Combined.pdf}
	\caption{Partworth utilities and profile level rankings }
	\label{Combined}
\end{figure}

\begin{figure}[!htpb]
	\centering
	\includegraphics[width=4.5in, height=3.75in]{FNMDS.png}
	\caption{Multidimensional scaling showing similarity of sample respondents and attribute levels.}
	\label{MDSout1}
\end{figure}

\newpage
\subsection{Validation of Part-worth Utilities}

\subsubsection{Attribute Importances}
Shows inadequacy of partworth utilities for determing attribute importance.

\subsubsection{Choice task prediction accuracy}

\section{Further Work}
Still in its infancy, PR-ACBC methodology has several major areas open to further research including analysis of  different ranking systems for various types of choice tournaments, determination of the population nd sample sizes for which PR-ACBC is most suitable, and application of PR-ACBC to other small population studies.

\section{Conclusion}
To complement well-established partworth-utility-based conjoint analysis methodology of survey data where the target populations are typically large, we have introduced a simple, intuitive approach to a small population's profile and level rankings based on choice-task sample data. A toy survey is used to introduce the basic concepts, and an actual survey administered to a small number of disaster relief organizations used as an example of PR-ACBC methodology.

\subsection*{Project Team}

Ming-Hsuan Chuang,
Daniel Daum, Michaela Flitsch, Erica Gralla, Jarrod Goenzel, Timotius Kartawijaya, Zoe Kallus, Courtney Linscott, Sara Magnuson, Mark Nussbaum, Zach Oslund, Matthew Rueger, Nick Varberg, Mike Veatch, Joyce Yan

\section*{References}
\begin{list}{}{\itemindent=-2em}
\small
\item Alvo, M., and Yu, P.L.H. 2014. \emph{Statistical Methods for Ranking Data}. Springer.

\item Gralla, E., Goentzel, J., and Fine G. 2014. Assessing trade-offs among multiple objectives for humanitarian aid delivery using expert preferences.
\emph{Production and Operations Management}, Springer-Verlag Berlin 23(6), 978-989.

\item Oberhofer, W. and Kaufman, H.1987.  Maximum Likelihood Estimation of a Multivariate Hypergeometric Distribution. \emph{Sankhya: The Indian Journal of Statistics, Series B (1960-2002)}, Indian Statistical Institute, 49(2), 188-191.

\item Orme, B.K. 2014. \emph{Getting Started with Conjoint Analysis.} Sawtooth Software.

\item Orme, B.K., and Chrzan, K. 2017. \emph{Becoming an Expert in Conjoint Analysis: Choice Modeling for Pros.} Sawtooth Software.

\item Rao, V. R. 2014. \emph{Applied Conjoint Analysis}. Springer.

\item  Stewart, J. 2016.  \emph{Calculus, Early Transcendentals (8E)}. Cengage Learning.

\item Rossi, P., Allenby, G. and McCulloch R. 2005. \emph{Baysian Statistics and Marketing.} John Wiley \& Sons, Ltd.

\item Sawtooth Software, 2013. The MaxDiff System Technical Paper. https://www.sawtoothsoftware.com/download/techpap/maxdifftech.pdf Accessed 11/27/2018.

\item Thurstone, L. L. 1927. A Law of Comparative Judgment, \emph{Psychological Review}, 4, 273-286.
\end{list}

\end{document}


\chapter{Analysis}
All results of the measurements are listed in \autoref{sec:dataTables}.
The complete set of all data is available at \url{https://github.com/TrackerSB/BachelorthesisData}.
The script used for generating the plots and calculating statistical variables is included in \autoref{lst:rscript}.

\section{Descriptive Statistics}
The violin plots in \autoref{fig:coverages} describe the distribution density of \dyncovs (left) and \dyncovp (right) over the projects.
The plots also contain marks for 25\% quantiles.
The higher these lines the more projects have a higher coverage observing the regions automatically optimizable by Polly.
\begin{figure}[!h]
    \caption[Distribution density of \dyncovs and \dyncovp]{
        Distribution density of \dyncovs (left) and \dyncovp (right) as violin plots.
        Each plot is divided in four quantiles with same size.
    }
    \includesvg[width=\textwidth]{compDyncovScopParent}
    \label{fig:coverages}
\end{figure}
In both plots the lowest quantile is the broadest one and has almost the same size.
Also in both plots the quantile between 25\% and 50\% is the thinnest one.
When looking at the left plot the quantile between 50\% and 75\% is broader than the upper one.
In the right plot it is the other way round.\\
As a result the theoretical speedup \(S_p=\speedupMaxRegions\) is only slightly higher than \(S_s=\speedupScops\).
The theoretical speedups are calculated with \(N=8\) cores.\\
\autoref{tab:statsMatrix} shows the mean (\(\mu\)), the standard deviation (s) and the variance (\(s^2\)) of either \dyncovp and \dyncovs and how much \dyncovp is bigger than \dyncovs (\(\Delta\)) in \%.
\begin{table}[!h]
    \myfloatalign
    \begin{tabularx}{\textwidth}{lCCC}
        \tableheadline{} & \tableheadline{SCoPs} & \tableheadline{\(\Delta\)} & \tableheadline{MaxRegions}\\\toprule
        \csvreader[head to column names]{csv/statsMatrix.csv}{}{\csvcoli&\csvcolii&\csvcoliii&\csvcoliv\\}
        \\\bottomrule
    \end{tabularx}
    \caption[Statistical evaluations of \dyncovp and \dyncovs]{
        This table contains the mean (\(\mu\)), standard deviation (s), the variance (\(s^2\)) of either \dyncovp and \dyncovs and how much \dyncovp is bigger than \dyncovs (\(\Delta\)) in \%.
    }
    \label{tab:statsMatrix}
\end{table}
The mean value is almost the same either looking at the \scops or the MaxRegions.
The variance and the standard deviation of \dyncovp are higher than of \dyncovs.\\
\autoref{fig:rejectReasonsGrouped} indicates that the most common reason for parents of \scops being rejected are that it is already the top level region, the loop bound could not be computed and calls to functions with (possible) side effects.
\begin{figure}[!h]
    \caption[Reasons for rejecting SCoPs]{
        Here the most common reasons for rejecting are listed.
        The larger the tile the more often it appears as reason for rejecting a parent as \scop.
        But a large reason does not automatically imply that it has also a big impact on the run time.
        (In this figure BB is an abbreviation for basic block)
    }
    \includesvg[width=\textwidth]{pieInvalidReasons}
    \label{fig:rejectReasonsGrouped}
\end{figure}
The concrete values can be found in \autoref{tab:rejectionReasons}.

\section{Hypothesis Testing}
To check the significance of the collected data first the independence of the two data sets \scops and MaxRegions has to be stated.
Furthermore a Shapiro-Wilk-Test \cite{shapiroWilkTest} reveals that neither of these data sets is distributed normally (\(p \ll 0.05\)).
A T-Test \cite{ttest} reveals that both datasets are significant (\(p \ll 0.05\)).\\
All p values are listed in \autoref{tab:pValues}.\\
\begin{table}[!h]
    \myfloatalign
    \begin{tabularx}{\textwidth}{Xcc}
        \tableheadline{Test} & \tableheadline{SCoPs} & \tableheadline{MaxRegions}\\\toprule
        \csvreader[head to column names]{csv/pValues.csv}{}{\(\csvcoli\)&\csvcolii&\csvcoliii\\}
        \\\bottomrule
    \end{tabularx}
    \caption[P values of statistical tests]{This table contains all p values necessary to be calculated for ensuring the correctness of the evaluation of the hypotheses.}
    \label{tab:pValues}
\end{table}
\(H_1\): Observing the given programs and test sets (\autoref{tab:subjectPrograms}) the most common reason for rejecting parents of \scops to be valid as well is truly that the parent already is the top level region.\\
\(H_2\): The mean of \dyncovs is significantly lower than \hTwoAbout as well as the mean of \dyncovp (see \autoref{tab:pValues}).\\
\(H_3\): A Mann-Whitney-U-Test \cite{utest} reveals that \dyncovp is not significantly higher than \dyncovs (p value: \utestPValue).\\