\documentclass{tufte-handout}
\usepackage{amsmath}
\usepackage[utf8]{inputenc}
\usepackage{mathpazo}
\usepackage{booktabs}
\usepackage{microtype}

\pagestyle{empty}

\title{Flow Report}
\author{Anatoli Cooper and Boris Marley}

\begin{document}
  \maketitle

  \section{Results}

  Our implementation successfully computes a flow of 163 on the input file, confirming the analysis of the American enemy.
  \sidenote{%
  Complete the report by filling in your correct names,
  filling in the parts marked $[\ldots]$,
  and changing other parts wherever necessary.
  Remove the sidenotes in your final hand-in.
  Insert `comrades!' and `for the motherland' wherever you see fit.
  }

  We have analysed the possibilities of descreasing the capacities near Minsk.
  Our analysis is summaries in the following table:\sidenote{%
    Complete and correct the table.
  }

\bigskip
  \begin{tabular}{rccc}\toprule
    Case & 4W--48 & 4W--49 & Effect on flow \\\midrule
    1& 30& 20 & no change \\
    2& 20 &30 & no change \\
    3&20 & 20& $-10$ \\
    $[\ldots]$ \\\bottomrule
  \end{tabular}
  \bigskip

  In case 4, the new bottleneck becomes
  \begin{quote}
      1--44, 3--2, $[\ldots]$
  \end{quote}
  The comrade from Minsk is advised to $[\ldots]$.

  \section{Implementation details}

  We use a straighforward implemenation of Bellman--Quine's flow algorithm as described in Bronstein, \emph{Foundations of Algorithms}, chap.~6.
  We use Prim's algorithm to find an augmenting path and also implement the neat trick of flow by recomputing the inverse of the antisnail every 2nd iteration.\sidenote{%
    Replace by a description of what you actually do.
  }

  The running time is $[\ldots]$.

  We have implemented each undirected edge in the input graph as a $[\ldots]$.
  In the corresponding residual graph, the edge is represented by a $[\ldots]$.
  Our datatype for edge is this:
  \sidenote{%
    What I wrote here is complete nonsense, of course.
    Replace by a description of your actual implementation---what \emph{did} you use?
    A pair of directed edges?
    A single undirected edges with a bit to see in which direction the flow goes?
    As indicated, it makes sense to actually include your datatype for edge in the report.
    Here, and in general, it makes sense to have written short and sweet code.
  }
  \begin{verbatim}
    class FlowEdge
    {
      Vertex from, to;
      GUISwingComponentFactory bar;
      Edge next, parent, foo;
    }
  \end{verbatim}

\end{document}


\documentclass[handout]{beamer}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{url}
\usepackage{hyperref}
\usepackage{xcolor}
\usepackage{appendixnumberbeamer}
\usepackage{amsmath}
\usepackage{bm}
\usetheme[sectionpage=progressbar,
          subsectionpage=progressbar,
          numbering=fraction,
          progressbar=none,
          background=light]{metropolis}
\usefonttheme[onlymath]{serif}

\title{From PCA to Autoencoders}
\subtitle{Unsupervised Representation Learning}
\date{December 17, 2019}
\author{\textsc{Florent Forest}\\
\scriptsize{PhD student in Data science \& Machine learning\\
Safran Aircraft Engines --- Université Paris 13}
\vspace{0.2cm}\\
\includegraphics[height=0.35cm]{./rc/e-mail-envelope-blue}\;\scriptsize{\href{mailto:forest@lipn.univ-paris13.fr}{forest@lipn.univ-paris13.fr}}\\
\includegraphics[height=0.35cm]{./rc/grid-world-blue}\;\scriptsize{\href{http://florentfo.rest}{http://florentfo.rest}}\\
\includegraphics[height=0.35cm]{./rc/github-logo-blue}\;\scriptsize{\href{https://github.com/FlorentF9}{FlorentF9}}\\
}
\institute{\vfill\hfill
\includegraphics[height=1.75cm]{./rc/logo_supaero}}

\begin{document}

  \maketitle

  \begin{frame}{Table of contents}
    \setbeamertemplate{section in toc}[sections numbered]
    \begin{small}
      \vspace{0.5cm}
      \tableofcontents%[hideallsubsections]
    \end{small}
  \end{frame}

  %%%
  \section{Introduction to autoencoders}
  %%%

  %
  \subsection{Motivations}
  %

  \begin{frame}{Motivations}

    \begin{block}{Where we are}
      \begin{itemize}
        \item Complex and high-dimensional data (i.e. $d \sim [10^3, 10^6[$)
        \item "Big Data" (i.e. $N \sim [10^5, 10^9[$)
        \item No labels (unsupervised learning)
      \end{itemize}
    \end{block}
    \pause
    \begin{block}{What we have seen so far}
      \begin{itemize}
        \item Linear dimensionality reduction techniques (e.g. PCA)\\
        $\rightarrow$ cannot learn complex transformations
        \item Non-linear techniques, e.g. t-SNE\\
        $\rightarrow$ not scalable, time complexity is $\mathcal{O}(d \cdot N^2)$, or $\mathcal{O}(d \cdot N\log N)$ with Barnes-Hut (but limited to output dimension $\leq 3$)
      \end{itemize}
    \end{block}

  \end{frame}

  \begin{frame}{Motivations}

    \metroset{block=fill}

    \begin{block}{So why not use (deep) neural networks?}
      \begin{itemize}
        \item Can learn complex transformations
        \item Comfortable in very-high-dimensional spaces
        \item Scale linearly with the size of data
      \end{itemize}
    \end{block}

  \end{frame}

  %
  \subsection{Definition}
  %

  \begin{frame}{Definition}

    \metroset{block=fill}

    \begin{exampleblock}{Definition}
      \small{
      An \alert{autoencoder} is a neural network trained to \textbf{reconstruct its inputs}. It is composed of two components:
      \vspace{-0.4cm}
      \begin{enumerate}
        \item an \alert{encoder}, mapping the input to a latent representation ("code") $\mathbf{z} = \mathbf{f}_{\boldsymbol{\phi}}(\mathbf{x})$
        \item a \alert{decoder}, mapping the code back to the input space $\tilde{\mathbf{x}} = \mathbf{g}_{\boldsymbol{\theta}}(\mathbf{z})$
      \end{enumerate}
      }
    \end{exampleblock}
    \vspace{-0.25cm}
    \begin{figure}
      \includegraphics[height=4.5cm]{rc/autoencoder}
    \end{figure}

  \end{frame}

  \begin{frame}{Definition}

    \metroset{block=fill}

    \begin{alertblock}{Challenge}
      We do not want the encoder to learn the identity function, but to learn a \emph{good representation} of our data.
    \end{alertblock}

    \vspace{0.5cm}
    \pause

    \begin{block}{Regularization?}<2->
      Reducing the size of the \emph{hypothesis set} $\mathcal{H}$ by constraining the space of possible solutions to the optimization problem.
      \vspace{-0.25cm}
      \begin{itemize}
        \item L2 weight decay
        \item Sparsity, L1 weight decay
        \item \dots
      \end{itemize}
    \end{block}

  \end{frame}

  %
  \subsection{Mathematical formulation}
  %

  \begin{frame}{What is a good representation?}

    \metroset{block=fill}

    \small{Let $q_{\boldsymbol{\phi}}(Z|X)$ be a (stochastic) parametric mapping from $X$ to $Z$. A good representation $Z$ of a random variable $X$ maximizes \alert{mutual information} between $X$ and $Z$ (\emph{infomax principle}):}
    \vspace{0cm}
    \begin{align*}
      \mathbb{I}(X;Z) &= \mathbb{H}(X) - \mathbb{H}(X|Z)\\
                      &= C(X) + \mathbb{E}_{q_{\boldsymbol{\phi}}(X,Z)}\left[\log q_{\boldsymbol{\phi}}(X|Z)\right]
    \end{align*}
    \pause
    \small{For any parametric distribution $p_{\boldsymbol{\theta}}(X|Z)$ we have $\mathbb{E}_{q_{\boldsymbol{\phi}}(X,Z)}\left[\log p_{\boldsymbol{\theta}}(X|Z)\right] \leq \mathbb{E}_{q_{\boldsymbol{\phi}}(X,Z)}\left[\log q_{\boldsymbol{\phi}}(X|Z)\right]$ (using $D_{KL}(q||p) \geq 0$).}
    \pause
    \vspace{0.5cm}
    \begin{block}{Task: maximize a lower bound on $\mathbb{I}(X;Z)$}
      \begin{equation*}
        \underset{\boldsymbol{\phi},\boldsymbol{\theta}}{\text{maximize}} \; \mathbb{E}_{q_{\boldsymbol{\phi}}(X,Z)}\left[\log p_{\boldsymbol{\theta}}(X|Z)\right]
      \end{equation*}
    \end{block}

  \end{frame}

  \begin{frame}{Loss function for deterministic autoencoders}

    We consider \alert{deterministic} mappings $Z = \mathbf{f}_{\boldsymbol{\phi}}(X)$ (i.e. $q_{\boldsymbol{\phi}}(Z|X) = \delta(Z - \mathbf{f}_{\boldsymbol{\phi}}(X))$) and $\tilde{X} = \mathbf{g}_{\boldsymbol{\theta}}(\mathbf{f}_{\boldsymbol{\phi}}(X))$.
    \vspace{0cm}
    \begin{equation*}
      \underset{\boldsymbol{\phi},\boldsymbol{\theta}}{\text{maximize}} \; \mathbb{E}_{q_{\boldsymbol{\phi}}(X)}\left[\log p_{\boldsymbol{\theta}}(X|Z = \mathbf{f}_{\boldsymbol{\phi}}(X))\right]
    \end{equation*}
    \pause
    Using empirical mean over a set of data samples:
    \vspace{0cm}
    \begin{equation*}
      \underset{\boldsymbol{\phi},\boldsymbol{\theta}}{\text{maximize}} \; \sum_i \log p_{\boldsymbol{\theta}}(\mathbf{x}^{(i)}|\mathbf{z}^{(i)} = \mathbf{f}_{\boldsymbol{\phi}}(\mathbf{x}^{(i)}))
    \end{equation*}

    equivalent to:
    \vspace{0cm}
    \begin{equation*}
      \underset{\boldsymbol{\phi},\boldsymbol{\theta}}{\text{maximize}} \; \sum_i \log p(\mathbf{x}^{(i)}|\tilde{\mathbf{x}}^{(i)} = \mathbf{g}_{\boldsymbol{\theta}}(\mathbf{f}_{\boldsymbol{\phi}}(\mathbf{x}^{(i)})))
    \end{equation*}
    \pause
    Let us turn this into a minimization of the negative sum of individual loss functions $\mathcal{L}(\mathbf{x}, \tilde{\mathbf{x}}) = -\log p(\mathbf{x}|\tilde{\mathbf{x}})$.

  \end{frame}

  \begin{frame}{Loss function for deterministic autoencoders}

    \metroset{block=fill}

    The reconstruction $\tilde{\mathbf{x}}$ is the \alert{mean} of a distribution that may have generated $\mathbf{x}$.
    \pause
    \begin{block}{Continuous variables: $\mathbf{x} \in \mathbb{R}^d$}
      \begin{itemize}
        \item Gaussian distribution: $X|\tilde{X} = \tilde{\mathbf{x}} \sim \mathcal{N}(\tilde{\mathbf{x}}, \mathbf{\sigma}^2 \mathbf{I})$
      \item Loss function: $\mathcal{L}(\mathbf{x}, \tilde{\mathbf{x}}) \propto ||\mathbf{x} - \tilde{\mathbf{x}}||_2^2$
      \end{itemize}
      $\rightarrow$ \alert{Mean Squared Error (MSE) loss}
    \end{block}
    \pause
    \begin{block}{Binary variables: $\mathbf{x} \in \{0,1\}^d$, or $\mathbf{x} \in [0,1]^d$}
      \begin{itemize}
        \item Bernoulli distribution: $X|\tilde{X} = \tilde{\mathbf{x}} \sim \mathcal{B}(\tilde{\mathbf{x}})$
        \item Loss function: $-\sum_{j=1}^d [\mathbf{x}_j \log \tilde{\mathbf{x}}_j + (1-\mathbf{x}_j) \log (1-\tilde{\mathbf{x}}_j)]$
      \end{itemize}
      $\rightarrow$ \alert{Cross-entropy loss}
    \end{block}

  \end{frame}

  %%%
  \section{Links with Principal Component Analysis (PCA)}
  %%%

  %
  \subsection{Can neurons learn principal components?}
  %

  \begin{frame}{Hebb's learning rule}

    \begin{columns}[T,onlytextwidth]
      \column{0.7\textwidth}
      \vspace{0.5cm}
      Back to 1949 $\rightarrow$ Hebbian learning
      \column{0.3\textwidth}
      \includegraphics[height=1.5cm]{rc/delorean}
    \end{columns}
    \pause
    \metroset{block=fill}

    \begin{block}{The artificial neuron and Hebb's learning rule \cite{hebb-organization-of-behavior-1949}}
      \begin{columns}[T,onlytextwidth]
        \column{0.6\textwidth}
        \vspace{0.7cm}
        \centering
        \includegraphics[]{rc/artificial-neuron}

        \column{0.4\textwidth}
        \vspace{0.7cm}
        \centering
        \small{Weights increase if input and output are correlated.}
        \begin{equation*}
          \Delta \mathbf{w}_i = \alpha \mathbf{x}_i y
        \end{equation*}
        \begin{equation*}
          \Delta \mathbf{w} = \alpha \mathbf{x} y = \alpha \mathbf{x} \mathbf{x}^T \mathbf{w}
        \end{equation*}
      \end{columns}
    \end{block}

    See for example Rosenblatt's \textbf{perceptron} (1958) \cite{Rosenblatt58theperceptron} for an illustration of this learning rule.

  \end{frame}

  \begin{frame}{Learning principal components with Oja's rule}

    \textbf{Problem:} the weights can "explode" with Hebb's rule. A solution is to add a \emph{forgetting term}.
    \pause
    \metroset{block=fill}

    \begin{block}{Oja's learning rule \cite{Oja1982}}
      \begin{equation*}
        \Delta \mathbf{w} = \alpha (\mathbf{x} y - y^2 \mathbf{w})
      \end{equation*}
    \end{block}
    \pause
    \scriptsize{
    We can show that this rule leads to \emph{principal components} \cite{Becker1991,Oja1992}:
    \vspace{0cm}
    \begin{equation*}
      \Delta \mathbf{w} = \alpha (\mathbf{x} \mathbf{x}^T \mathbf{w} - \mathbf{w}^T \mathbf{x} \mathbf{x}^T \mathbf{w} \mathbf{w})
    \end{equation*}
    After convergence, we have:
    \begin{align*}
      \mathbb{E}\left[\Delta \mathbf{w}\right] = \alpha (\mathbb{E}\left[\mathbf{x} \mathbf{x}^T\right] \mathbf{w} - \mathbf{w}^T \mathbb{E}\left[\mathbf{x} \mathbf{x}^T\right] \mathbf{w} \mathbf{w}) = \alpha (\mathbf{C} \mathbf{w} - \mathbf{w}^T \mathbf{C} \mathbf{w} \mathbf{w}) = 0
    \end{align*}

    where $\mathbf{C} = \mathbb{E}\left[\mathbf{x} \mathbf{x}^T\right]$ is the covariance matrix. Finally:
    \vspace{0cm}
    \begin{equation*}
      \mathbf{C} \mathbf{w} = \mathbf{w}^T \mathbf{C} \mathbf{w} \mathbf{w} = \lambda \mathbf{w}
    \end{equation*}

    Thus, $\mathbf{w}$ is an eigenvector of the covariance matrix, a.k.a. a \textbf{principal component}.
    }

  \end{frame}

  %%%
  \subsection{Linear autoencoders}
  %%%

  \begin{frame}{Definition}

    \begin{figure}
      \includegraphics[width=5.5cm]{rc/linear-autoencoder}
    \end{figure}
    \vspace{-0.25cm}
    \begin{equation*}
      \tilde{\mathbf{x}} = \mathbf{W}_d \left(\mathbf{W}_e \mathbf{x} + \mathbf{b}_e\right) + \mathbf{b}_d
    \end{equation*}
    \pause
    Ignoring the biases, the MSE loss becomes:
    \vspace{0cm}
    \begin{align*}
      \text{MSE} &= \sum_i ||\mathbf{x}^{(i)} - \tilde{\mathbf{x}}^{(i)}||_2^2 = \sum_i ||\mathbf{x}^{(i)} - \mathbf{W}_d \mathbf{W}_e \mathbf{x}^{(i)}||_2^2 \\
                 &= ||\mathbf{X} - \mathbf{X} \mathbf{W}_d \mathbf{W}_e||_F^2
    \end{align*}

  \end{frame}

  \begin{frame}{Equivalence to PCA}

    \begin{columns}[T,onlytextwidth]
      \column{0.5\textwidth}
      \centering \textbf{PCA}
      \begin{align*}
        \underset{\mathbf{U} \in \mathbb{R}^{d \times r}}{\text{minimize}} &\; ||\mathbf{X} - \mathbf{X} \mathbf{U} \mathbf{U}^T||_F^2 \\
        \text{subject to} &\; \mathbf{U}^T \mathbf{U} = \mathbf{I}
      \end{align*}
      \pause
      \column{0.5\textwidth}
      \centering \textbf{Linear AE}
      \begin{equation*}
        \underset{\mathbf{W}_e \in \mathbb{R}^{l \times d}, \mathbf{W}_d \in \mathbb{R}^{d \times l}}{\text{minimize}} \; ||\mathbf{X} - \mathbf{X} \mathbf{W}_d \mathbf{W}_e||_F^2
      \end{equation*}
      {\scriptsize one can show that we must have $\mathbf{W}_e = \mathbf{W}_d^{\dagger}$ (pseudo-inverse), then:}
      \begin{equation*}
        \underset{\mathbf{W} \in \mathbb{R}^{d \times l}}{\text{minimize}} \; ||\mathbf{X} - \mathbf{X} \mathbf{W} \mathbf{W}^{\dagger}||_F^2
      \end{equation*}

    \end{columns}

    \vspace{1cm}
    \pause
    \centering
    $\blacktriangleright$ Linear autoencoders are equivalent to PCA\dots\\
    \alert{without the orthogonality constraint}! \cite{Plaut2018}

  \end{frame}

  \begin{frame}{Equivalence to PCA}

    \begin{figure}
      \includegraphics[width=\textwidth]{rc/pca-ae-bases}\\
      \tiny{\url{https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73}}
    \end{figure}

  \end{frame}

  %%%
  \section{Real-life autoencoders}
  %%%

  %
  \subsection{Non-linear and deep autoencoders}
  %

  \begin{frame}{Non-linear and deep autoencoders}

    \begin{figure}
      \includegraphics[width=\textwidth]{rc/deep-autoencoder}
    \end{figure}

    \pause
    \centering
    $\blacktriangleright$ \alert{Not equivalent to PCA!}

    \pause
    {\small Trained end-to-end with backprop and SGD. Layer-wise pre-training \cite{Hinton2006} is in fact not necessary (thanks to ReLU, better optimization, etc.).}

  \end{frame}

  %
  \subsection{Different types of regularizations}
  %

  \begin{frame}{Undercomplete or overcomplete}

    Let $d$ be the input dimension and $l$ the code dimension.
    \pause
    \vspace{0.5cm}
    \begin{columns}[T,onlytextwidth]
      \column{0.5\textwidth}
      \centering
      $l < d$: \textbf{undercomplete}
      \begin{figure}
        \includegraphics[height=3cm]{rc/entonnoirs1}
      \end{figure}
      \pause
      \column{0.5\textwidth}
      \centering
      $l \geq d$: \textbf{overcomplete}
      \begin{figure}
        \includegraphics[height=3cm]{rc/entonnoirs2}
      \end{figure}
    \end{columns}
    \vspace{0.5cm}
    \pause
    Having the autoencoder encode the data to a lower dimension (in general $l << d$), forces it to compress the data and learn an efficient representation and is a \alert{form of regularization}.

  \end{frame}

  \begin{frame}{Let's code! (1)}

    \centering
    \includegraphics[width=0.5\textwidth]{rc/colab}\\
    \vspace{1cm}
    \includegraphics[width=0.25\textwidth]{rc/jupyter}

    % AE classique avec une/plusieurs couches
    % Entraîner sur MNIST avec différents loss (MSE, cross-entropy)
    % Éventuellement convolutif

    % Observer l'espace latent via t-SNE (comparer avec t-SNE sur MNIST brut)

  \end{frame}

  \begin{frame}{The danger of overfitting}

    \textbf{Problems:} overcomplete AE will not learn useful representations; and even undercomplete AE with a \emph{single continuous latent variable} can remember an entire training set (one real number per sample).

    \begin{figure}
      \includegraphics[width=0.8\textwidth]{rc/ae-real-numbers}
    \end{figure}
    \pause
    \textbf{Solution:} adding \alert{constraints} to the latent space by using different types of \alert{regularization}!

  \end{frame}

  \begin{frame}{Sparse autoencoders}

    \metroset{block=fill}

    \begin{block}{Principle}
      \small{
      Sparse autoencoders can have more latent units than input units, but \alert{only a few are allowed to activate together}:
      \vspace{-0.25cm}
      \begin{itemize}
        \item a fixed proportion (e.g. 5\%) using a KL-divergence penalty \cite{Ng2011}
        \item a fixed number $k$ \cite{Makhzani2013}
        \item using a L1 penalty \cite{Arpit2016}
      \end{itemize}
      }
    \end{block}

    \centering
    \includegraphics[width=4.5cm]{rc/sparse-autoencoder}

  \end{frame}

  \begin{frame}{Denoising autoencoders}

    \metroset{block=fill}

    \begin{block}{Principle}
      Denoising autoencoders \cite{Vincent2010} are trained to reconstruct \alert{corrupted} versions of the input. Corruption can be achieved by adding noise or randomly turning off input units. Usually, they have a single hidden layer with a non-linearity.
    \end{block}

    \centering
    \includegraphics[width=7cm]{rc/denoising-autoencoder}

  \end{frame}

  %
  \subsection{Variational autoencoders (VAE)}
  %

  \begin{frame}{Motivations}

    \textbf{Limitation of standard AE:} the latent space has \emph{no structure} and may not be continuous; it may \emph{overfit}, and we cannot explore it nor \emph{sample} from it.
    \pause
    \begin{columns}[T,onlytextwidth]
      \column{0.5\textwidth}
      \centering
      \includegraphics[width=0.8\textwidth]{rc/ae-latent}
      \pause
      \column{0.5\textwidth}
      \centering
      \includegraphics[width=0.8\textwidth]{rc/vae-latent}
    \end{columns}

  \end{frame}

  \begin{frame}{Motivations}

    \begin{columns}[T,onlytextwidth]
      \column{0.5\textwidth}
      \centering
      \includegraphics[height=4.25cm,width=\textwidth]{rc/vae-mnist}\\

      \column{0.4\textwidth}
      \centering
      \includegraphics[height=4.25cm,width=\textwidth]{rc/vae-faces}\\
    \end{columns}

    \centering
    \includegraphics[width=0.7\textwidth]{rc/vae-faces-smile}\\
    \tiny{\url{jmetzen.github.io/2015-11-27/vae.html}, \url{ermongroup.github.io/cs228-notes/extras/vae}, \url{github.com/davidsandberg/facenet}}

  \end{frame}

  \begin{frame}{Variational autoencoders (VAE)}

    \metroset{block=fill}

    VAEs are \alert{latent-variable probabilistic models}.
    \pause
    \begin{columns}[T,onlytextwidth]

      \column{0.7\textwidth}
      \begin{block}{Probabilistic setting}
        Generative model $p_{\boldsymbol{\theta}}(\mathbf{x}, \mathbf{z})$:
        \begin{enumerate}
          \item $\mathbf{z}$ is sampled from the \alert{prior} $p_{\boldsymbol{\theta}}(\mathbf{z})$
          \item $\mathbf{x}$ is generated with \alert{likelihood} $p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z})$ (\emph{probabilistic decoder})
        \end{enumerate}
      \end{block}

      \column{0.29\textwidth}
      \centering
      \includegraphics[width=\textwidth]{rc/vae-gen}\\
      \tiny{Kingma \& Welling 2014 \cite{Kingma2014}}

    \end{columns}
    \vspace{0.5cm}
    \pause
    \textbf{Problem:} $\boldsymbol{\theta}$ is a NN, so $p_{\boldsymbol{\theta}}(\mathbf{x})$ and $p_{\boldsymbol{\theta}}(\mathbf{z}|\mathbf{x})$ are intractable.\\
    \pause
    $\blacktriangleright$ Stochastic Gradient Variational Bayes (SGVB) \cite{Kingma2014} is an efficient estimation method in case of intractable marginal likelihood/posterior and large datasets.

  \end{frame}

  \begin{frame}{Variational autoencoders (VAE)}

    \begin{figure}
      \includegraphics[width=7cm]{rc/vae}
    \end{figure}

    Recognition model $\rightarrow$ \alert{approximate posterior} $q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})$\\
    (\emph{probabilistic encoder})

  \end{frame}

  \begin{frame}{VAE loss function}

    \metroset{block=fill}

    \begin{exampleblock}{VAE ELBO (evidence lower bound)}
      \vspace{-0.25cm}
      \begin{equation*}
        \underset{\boldsymbol{\phi},\boldsymbol{\theta}}{\text{maximize}} \; -D_{KL}\left(q_{\boldsymbol{\phi}}(Z|X)||p_{\boldsymbol{\theta}}(Z)\right) + \mathbb{E}_{q_{\boldsymbol{\phi}}(Z|X)}\left[\log p_{\boldsymbol{\theta}}(X|Z)\right]
      \end{equation*}
    \end{exampleblock}
    \pause
    \begin{alertblock}{Key ideas}
      \begin{itemize}
        \item The second term is a (negative) \alert{reconstruction error} (e.g. MSE or cross-entropy) as in a deterministic AE.
        \item The first term, a Kullback-Leibler divergence between $q_{\boldsymbol{\phi}}(Z|X)$ and $p_{\boldsymbol{\theta}}(Z)$, acts as a \alert{regularizer} pushing the encoder distribution closer to the prior distribution (typically a gaussian).
      \end{itemize}
    \end{alertblock}

  \end{frame}

  \begin{frame}{VAE loss function}

    \metroset{block=fill}

    Let's put gaussians everywhere!
    \begin{itemize}
      \item $p_{\boldsymbol{\theta}}(\mathbf{z}) = \mathcal{N}(\mathbf{z}; \mathbf{0}, \mathbf{I})$
      \item $q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x}) = \mathcal{N}(\mathbf{z}; \boldsymbol{\mu}, \boldsymbol{\sigma}^2\mathbf{I})$
    \end{itemize}
    \pause
    \begin{block}{The reparameterization trick}
      To sample from $q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})$, we use the reparameterization $\mathbf{z} = f_{\boldsymbol{\phi}}(\mathbf{x}, \epsilon) = \boldsymbol{\mu} + \boldsymbol{\sigma} \cdot \boldsymbol{\epsilon}$ where $\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$
    \end{block}
    \pause
    \small{For a given $\mathbf{x}^{(i)}$, and using 1-sample Monte-Carlo estimation, the ELBO becomes:}
    \vspace{0cm}
    \begin{multline*}
      \frac{1}{2} \sum_j \left(1 + \log (\boldsymbol{\sigma}^{(i)}_j)^2 - (\boldsymbol{\mu}^{(i)}_j)^2 - (\boldsymbol{\sigma}^{(i)}_j)^2 \right) + \log p_{\boldsymbol{\theta}}(\mathbf{x}^{(i)}|\mathbf{z}^{(i)})\\
      \text{ where } \mathbf{z}^{(i)} = \boldsymbol{\mu}^{(i)} + \boldsymbol{\sigma}^{(i)} \cdot \boldsymbol{\epsilon}
    \end{multline*}

  \end{frame}

  \begin{frame}{VAE: the reparameterization trick}

    \begin{figure}
      \includegraphics[width=9cm]{rc/vae-reparameterization}
    \end{figure}

  \end{frame}

  \begin{frame}{Let's code! (2)}

    \centering
    \includegraphics[width=0.5\textwidth]{rc/colab}\\
    \vspace{1cm}
    \includegraphics[width=0.25\textwidth]{rc/jupyter}

    % VAE avec une/plusieurs couches
    % Entraîner sur MNIST avec différents loss (MSE, cross-entropy)
    % Éventuellement convolutif

    % Observer l'espace latent via t-SNE (comparer avec AE)

  \end{frame}

  %%%
  \section{Applications}
  %%%

  \begin{frame}{Dimensionality reduction and Feature extraction}

    Autoencoders can extract useful low-dimensional representations from high-dimensional data. They can be used as:
    \pause
    \begin{itemize}
      \item[$\blacktriangleright$]<2-> a pre-processing step for any other ML algorithm (clustering, supervised classification or regression)
      \item[$\blacktriangleright$]<3-> an unsupervised pre-training prodecure for supervised deep neural networks (e.g. stacked AE pre-training)\\
      $\rightarrow$ see \cite{Hinton2006}, \cite{Erhan2010}, \cite{Vincent2010}
    \end{itemize}

  \end{frame}

  \begin{frame}{Data compression}

    Autoencoders can be used as a compression algorithm:
    \pause
    \begin{itemize}
      \item[$\blacktriangleright$] \textbf{lossy}
      \pause
      \item[$\blacktriangleright$] \textbf{data-specific}
    \end{itemize}
    \pause
    Not commonly used in practice\dots

  \end{frame}

  \begin{frame}{Data augmentation}

    The decoder model can be used to \alert{generate new data samples}:
    \pause
    \begin{itemize}
      \item[$\blacktriangleright$] Deterministic AEs: adding noise, interpolating or extrapolating in latent space \cite{Devries2017}
      \pause
      \item[$\blacktriangleright$] Generative models (VAE, GAN)
    \end{itemize}

  \end{frame}

  \begin{frame}{Anomaly detection}

    When the input differs from the training distribution (e.g. an outlier), the model will produce a large reconstruction error. This error can be used to \alert{score anomalies} \cite{Malhotra2016}.

    \begin{figure}
      \includegraphics[width=10cm]{rc/anomaly-detection}
    \end{figure}

  \end{frame}

  \begin{frame}[standout]
    Questions?
  \end{frame}

  %%%
  \appendix

  \begin{frame}[allowframebreaks]{References}

    \bibliography{references}
    \bibliographystyle{abbrv}

  \end{frame}

\end{document}


\documentclass[english, 12 pt, openany, oneside]{book}

\usepackage{graphicx}
\graphicspath{{./Images/}}
\usepackage{caption}
\usepackage{subcaption}

\setcounter{secnumdepth}{3}

\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\usepackage{amsmath,amssymb,here}
\usepackage{a4wide}

\usepackage{hyperref}

\hypersetup{
colorlinks=true,
linkcolor=black,
urlcolor=blue,
citecolor=black,
}

\usepackage{siunitx}
\usepackage{listings}

\DeclareMathOperator{\e}{e}
\DeclareMathOperator{\arccosh}{acosh}

% Better looking table for math
\usepackage{multicol}
\usepackage{booktabs}
\usepackage{longtable}
\usepackage{cellspace}
\setlength\cellspacetoplimit{3pt}
\setlength\cellspacebottomlimit{3pt}

\usepackage{csquotes}

\usepackage{titlesec}
\titleformat{\chapter}[hang]{\bf\huge}{\thechapter}{2pc}{}

\usepackage{pythontex}

\usepackage[language=english,fpms,emblem]{umonsCover}
\umonsAuthor{Vincent \textsc{Stragier}}
% The main title of your thesis
\umonsTitle{Industrial Instrumention}
% The sub-title of your thesis
\umonsSubtitle{A Graphical User Interface for the two factorial Design of Experiments using Python and Qt}
% The type of document: the reason of the thesis
\umonsDocumentType{Project report}
% Your supervisor(s)
\umonsSupervisor{Under the supervision of\\ Professor Patrice \textsc{Mégret}}
% The date (or academic year)
\umonsDate{Academic year 2019--2020}

\usepackage{color} %red, green, blue, yellow, cyan, magenta, black, white
\definecolor{mygreen}{RGB}{28,172,0} % color values Red, Green, Blue
\definecolor{mylilas}{RGB}{170,55,241}

\usepackage[style=numeric]{biblatex}
\addbibresource{Industrial_instrumentation.bib}

\begin{document}
% Setup pythontex to be able to import another module located in other path
\begin{pyconcode}
import os;
os.chdir('../')
import sys
sys.path.append("..\\src")
sys.path.append(os.getcwd())
sys.path = list(set(sys.path))
\end{pyconcode}

\umonsCoverPage

\frontmatter
\tableofcontents

% \chapter*{\centering Remerciements}
\mainmatter
\phantomsection
\chapter*{Introduction}
\addcontentsline{toc}{chapter}{Introduction}
The aim of this report is to quickly introduce the theoretical basis of the Design of Experiments (DoE), to explain how it has been implemented in Python, to show how to use the project Graphical User Interface (GUI). The focus will be put on the programming part.

\chapter{Theory about two factorial Design of Experiments}
Most of the following information are coming from the course book\cite{patrice_megret_industrial_nodate} of Professor Mégret (Chapter 10 - Design of experiments).

Design of experiments is a field of science that studies the theory of measurements. It aims to find an optimal way of taking and analysing the measurements. To limit the number of measurements to the lowest number of needed measurement as well as reducing the measurement time.

In fact, we should meet the following requirement:

\begin{itemize}
\item quickly arrive at the best possible results
\item omit unnecessary trials
\item give results with the lowest uncertainty
\item progress without failure
\item establish a model for the studied phenomenon
\item discover the optimal solution
\end{itemize}

In short, we are trying to proceed to the measurement of a limited number of parameters with the most efficient method.

\section{Factorial design}
Factorial design is a technique which allows to define a measurement pattern in the experimental plane. It means that we must take a measurement of each level for each factor.

\begin{equation}
m = m_0^{n}
\end{equation}

Where $m$ is the number of measurements, $m_0$ is the number of levels and $n$ is the number of factors that you want to adjust.

A case of the factorial design is the two levels factorial design ($m_0 = 2$). Therefore, the number of trials is only depending on the numbers of factor ($m = 2^n$). We will implement it in the algorithm.

\chapter{Python implementation}
In order implement the GUI for this project, a module has been made to compute the needed matrices for the DoE. Then the Qt tools have been used (PyQt5 and Qt Designer).

The code of this project is fully available on GitHub:\\ \url{https://github.com/2010019970909/design_of_experiments}.
I am going to expose my usual procedure to program any algorithm. Keep in mind that the technology is always evolving allowing some improvements of the methodology.

\section{My own way of programming}
It is easy to learn how to program, it is, however, less easy to teach how to program. Personally, I began to learn how to program at the age of thirteen and I am still learning. My main motivation was to learn how to interface a computer and a micro-controller. Which brought me to learn how to program in C++ and in Python, to use the serial link, the network protocols, etc.

\subsection{How to start}
At first programming can look overwhelming but often a technical solution exists and all you need is some sheets of paper, a pen, a computer with whatever programming language and operating system but also an internet connection or some books about your programming language.

What is not important is which operating system you are using and what programming language you are using. However, try to find a programming language that meet the same goals as your program, i.e., Python is not adapted out of the box to be fast (it can be accelerated by using compiled libraries, etc.).

As an advice, you should always use a \textquote{versionning} program (Git with a GitHub or a GitLab account) when programming in order to keep version of your project and in case of a big mistake, in order to revert your code.

Another thing to do is to pick a good Integrated Development Environment (IDE) for your programming language. I am using Visual Studio Code (VS Code) with the Python extension and a stable version of Python. VS Code is amazing because it allows you to change the Python version easily (not a big deal if you know how to use the terminal), but most importantly it provide an auto-completion engine (that's a deal breaker for all the programmers). Please do not use the Python default IDE (it is not meant for sustainable programming). VS Code is also bonded with Git which is another advantage.

Most of the time any beginner tutorial will help you to learn how to use the language you are looking for. And they are often available for free.

If you do not know what you are going to do, do not do it. Especially if you are writing, moving, or deleting files. Same apply if you are using the network. As for a car driver a programmer must be in control of his machine. You can also use virtual environments or a virtual machine to limit the effect of your program on your machine.

As for everything skills, programming needs practice.

\subsection{How to approach a project}
First, look for the state-of-the-art. It does not mean that you will find exactly what you are looking for, but you will get the current state of the technology which will improve your vision of your project.

Second, if you did not find exactly what you are looking for, draw a scheme of your algorithm. It will give you a raw idea of your program, of what kind of object you need to use, which number of functions, etc. If you did find exactly what you need ensure yourself that you can legally use it and if yes, you can proceed to test it.

Third, try to use the norms. For example, always starting your class name with an upper-case letter. Your variables with a lower-case letter. The function name with a lower letter. Your constants in upper case. Always give meaningful name to your variable. The auto-completion is there to help you to complete your code faster anyway, so make your code readable by the human, the machine does not care.

Fourth, test and debug. For a professional program you must decompose your program in atomic snippet of code and to implement test cases (unitary and recursive testing). You can implement it but the important is to test your program. I was virtually never able to write an error free program without testing it step by step. Always think of a way of debugging your code (log files, debugging tool of your IDE).

Fifth, document your code. In any case the comments are always useful if they give useful information (description of the algorithms, description of the usage of the function, To-do list, current progress, summary of the aim of the program, change log, links to resources, etc.).

Sixth, less is more. Always implement the minimally viable algorithm, not all the possible variation at the same time. It is also a method in business (lean canvas) where the options are not important, but the main functionality must be implemented.

Seventh, always research before asking a question. We are living in an era where all the information is easily accessible (books, internet, experts, etc.). If you do not know of what you are talking you should first fetch some information to understand how to formulate properly your question. Also, if you are looking for the answer you will probably find it and if not your question will be interesting for you and the other (it is the principle of some platform like Stack Overflow).

\section{An example with Python}
Here is an example of how I would program a code that allows to wrap the print function in order

\begin{pyblock}[][breaklines]
"""
This is an example that shows how I program.
The aim of this program is to wrap the print function in a class.
"""

class Print:
	def __init__(self, functor=print):
		""" Initialise the Print object
			Args:
				functor:
				function used to write or to print a string, by default the print function is used.
			Returns:
			none.
		"""
		self.functor = functor

	def original(self, *args, **kwargs):
		""" Use the functor to print/write the original text (string)
			args:
				text:
				the text to print/write

				kwargs:
				the additional options
			Returns:
			none:
		"""
		self.functor(*args, **kwargs)

	def lower(self, *args, **kwargs):
		""" Use the functor to print/write a lower-case text (string)
			Args:
				args:
				the text to print/write

				kwargs:
				the additional options
			Returns:
			none:
		"""
		sep = " "
		try:
			sep = str(kwargs['sep'])
		except:
			pass

		self.functor((sep.join(map(str, args))).lower(), **kwargs)

	def upper(self, *args, **kwargs):
		""" Use the functor to print/write an upper-case text (string)
			Args:
				args:
				the text to print/write

				kwargs:
				the additional options
			Returns:
			none:
		"""
		sep = " "
		try:
			sep = str(kwargs['sep'])
		except:
			pass

		self.functor((sep.join(map(str, args))).upper(), **kwargs)

if __name__ == "__main__":
	# The previous if statement allows to execute the code only if we are using it as the main program.

	# Tests
	my_print = Print()

	# Print.original()
	my_print.original("Original", end="\n\r")

	# Print.lower()
	my_print.lower("Lower", end="\n\r")

	# Print.upper()
	my_print.upper("Upper", "lower?", end="\n\r", sep='\t')
\end{pyblock}

\vskip 1 cm

which will give:

\vskip 1 cm

\blockquote{
Original\\
lower\\
UPPER   LOWER?
}

\vskip 1 cm

This kind of class can be used to easily output string in the console or to a file, with a minimal number of changes. More than that you can add your own methods. Here the code is not very complex, and I did not implement a real test for each method. But for each method the aim, the arguments and the returns are given. In VS Code that information is display when you hoover on the method name later (that will help you to complete the function parameters without having to open the documentation).

Also the statement \pyb{if __name__ == "__main__":} is used to unsure that the code is only executed when the program is used as the main program and not as a module. That ease the development of modules since you can test them or implement specific tools that will only be executed with a Python command like: \textquote{\pyb{py -m pip install numpy}}.
%\include{example.py}
%\begin{pygments}{python}
%a = 1
%\end{pygments}

%how about \pygment{python}{a = 1}

\section{Design of experiments algorithm}
% Add calculus and code
% Reference to the pdfs
Before diving into the programming of the GUI, a module has been implemented with the useful functions. To achieve this goal, we need some other modules and the algorithm that we intend to implement.

\subsection{Requirements}
% Needed modules
To be able to use the Python module the following modules are needed:

\begin{description}
\item[scikit] for the statistics
\item[numpy] for the math (matrix dot product, etc.)
\item[matplotlib] for the plots
\end{description}

On Windows the modules can be installed using the following command:
\begin{pyverbatim}
py -m pip install scikit numpy matplotlib
\end{pyverbatim}

\subsection{Algorithm}
The code is based on the two-factorial design of experiments. It consists in resolving the following relation:

\begin{align*}
\hat{\underline{a}} = \left(X^T X\right)^{-1} X^T \underline{y}
\end{align*}

\noindent where:

\begin{description}
\item[$\hat{\underline{a}}$] is a vector of coefficients (of shape $p\times 1$)
\item[$\underline{y}$] is a vector of measures (of shape $p\times 1$)
\item[$X^T$] is the transposed form of the matrix $T$
\item[$\left(X^T X\right)$] is the square information matrix (of shape $p\times p$)
\item[$\left(X^T X\right)^{-1}$] is the square dispersion matrix (of shape $p\times p$)
\end{description}

The aim is to compute the estimation of the coefficients to provide a model for the experiment. In our case the following model can be used:

\begin{align*}
y = \hat{a}_0 + \sum^n_{i=1} \hat{a}_i x_i + \sum^n_{i=1} \sum^n_{j=1} \hat{a}_{ij} x_i {x}_j + ...
\end{align*}

Therefore, when the expression of the matrix ${X}$ and the vector of measures $\underline{y}$ are known the last step is to inject them in the relation to compute the estimation of the coefficient values $\hat{\underline{a}}$.

In the code, this is achieved via the following function:\\ \pyv{gen_X(n:int=2, perm=None, show:bool=False, return_head:bool=False)}

The most important argument of the function is the changing parameters (\pyv{n: int}).

Let us generate the matrix $X$ for a $2^n$ design with $n=3$:

\begin{pyconsole}[][breaklines]
import design_of_experiments as doe
doe.gen_X(n=3)
\end{pyconsole}

As shown in that case $p$ is equal to $2^3=8$. Note that you can permute the rows using a slices object; that may help you if it is easier for you to proceed to the measurement in a specific order.

When the matrix $X$ is known we can easily compute the matrix $\hat{X}$ whom I defined has followed:
\begin{equation*}
\hat{X} = \left(X^T X\right)^{-1} X^T
\end{equation*}

To simplify the use of this module the function \pyv{gen_X_hat()} is calling the function \pyv{gen_X()} with the same arguments.

Thus, we have:
\begin{pyconsole}[][breaklines]
X_hat  = doe.gen_X_hat(n=3)
X_hat
\end{pyconsole}

The only remaining steps is to compute the following dot product between the previous matrix and the measures:

\begin{equation*}
\hat{\underline{a}} = \hat{X} \underline{y}
\end{equation*}

By taking the example about the Gold Jewellery\cite{goupy_introduction_2007} example which aim to analyse the measure of two quantities under the controlled fluctuation of 3 parameters we can find the estimation of the coefficients for the 2 quantities:

\begin{pyconsole}[][breaklines]
# Speed [mg/min]
y1 = [53, 122, 20, 125, 48, 70, 68, 134]
# Cobalt Content [ppm]
y2 = [4100, 3510, 3950, 1270, 4870, 2810, 7750, 3580]

# Compute the coefficients
import numpy as np
a_hat_1 = np.dot(X_hat, y1)

a_hat_2 = np.dot(X_hat, y2)

a_hat_1

a_hat_2
\end{pyconsole}

The results correspond to the one showed in the book (Introduction to the design of experiments).

\section{Graphical user interface}
% Reference to the book
For the GUI, PyQt5 has been chosen (other modules or tools could have been used like Tkinter or appJar). The main document used to learn how to use PyQt5 was a book about optics in Python\cite{lakshminarayanan_understanding_2018}. One of the main advantages of Qt (PyQt5) is the GUI designer that allows to rapidly design GUI. This is done in three main steps:

\begin{itemize}
\item design the GUI on the Qt Designer tool.
\item convert the .ui file generated in a Python script.
\item create a new script that will import the generated script and make the links between the main script and the GUI script.
\end{itemize}

\subsection{Requirements}
To use the graphical user interface, the previous modules are needed as well as the following one:

\begin{description}
\item[PyQt5] for the GUI
\end{description}

It may be useful to add the PyQt5-tools module to change the design of the GUI.

On Windows the modules can be installed using the following command:
\begin{pyverbatim}
py -m pip install PyQt5 PyQt5-tools
\end{pyverbatim}

When the tools are installed you must find the Qt Designer executable. Then you may want to add a short-cut to it. On my computer the path to the tool is:
\begin{pyverbatim}[][breaklines]
'C:\Users\<username>\AppData\Local\Programs\Python\Python38\Lib\site-packages\pyqt5_tools\Qt\bin\designer.exe'
\end{pyverbatim}

\section{How to program the GUI}
As stated, before 3 steps are needed to create a GUI. A lot of information can be found on the internet. A good resource is the Qt website or the following website \url{www.learnpyqt.com/}.

\subsection{First step: GUI design}
The Qt Designer is a drag and drop tool. You have just to create the design of your GUI, maybe change some parameters of the object for convenience, i.e., the objectName.

\begin{figure}[!ht]
\centering
\includegraphics[width=\linewidth]{full_GUI}
\caption{Screen capture of the Qt Designer GUI\label{full_qt}}
\end{figure}

In the figure \ref{full_qt} the left panel the GUI element are available, and you can drop them in the application window. The Object inspector is on the right-side panel and allows you to browse through the different objects to change their parameters via the property editor.

N.B.: At the beginning of a project you can choose the kind of GUI you want to develop. Here, it is a \textquote{Main Window} application.

\clearpage
\subsubsection{Customised Widgets}
In this project by two times customized widgets have been used (using \pyv{mplwidget.py} and the \pyv{latexqtablewidget.py}). The first allows to use matplotlib to display charts and the second allows to render the header of the table widgets in LaTeX. To use them you must configure the widgets to use specify the class name and module name. It cannot be done after because you should not modify the generated Python script in the next step.

\begin{figure}[!ht]
\centering
\includegraphics[width=0.8\linewidth]{promote_widget}
\caption{Promote the QTableWidget\label{fig:promote_widget}}
\end{figure}

The first step (figure \ref{fig:promote_widget}) is to open the \textquote{Promoted Widgets} window. Here we are configuring a QTableWidget and we want to add the LaTeX rendering feature.

\begin{figure}[!ht]
\centering
\includegraphics[width=0.6\linewidth]{config_promote}
\caption{Configure a customised QTableWidget\label{fig:config_promote}}
\end{figure}

The second step (figure \ref{fig:config_promote}) is to configure the new widgets by selecting the right base widget. Then by specifying the class name (it can be found in the header/module file of your customised widget class). And then to specify the header name without extension.

\subsubsection{Add the signals and slots}
By pressing the F4 key you can Add Signals/Slots to the objects. Personally, I do not use that part of the tools and I had manually the Signals and Slots in the third step.

Do not forget to save you project (as often as possible). Save all the scripts in the same source folder.

\subsection{Second step: Generate the Python script for the GUI}
We are using another Qt module to convert the .ui file generated by the Qt Designer tool. This tool will generate a Python code that will allow to create our specific layout created before. This tool is used in the \textquote{UI2PYconverter.py} script and the content of the script is the following.

\begin{pyverbatim}
# −∗− coding: utf−8 −∗−
''' PyQt5 uic module convert ui file (XML code) into py file (Python code) '''
from PyQt5 import uic

with open('doe.ui', 'r') as fin:
    with open('Uidoe.py', 'w') as fout:
        uic.compileUi(fin, fout, execute=True)
\end{pyverbatim}

The script will simply convert the \textquote{doe.ui} file to the \textquote{Uidoe.py} file. You should never modify directly the Python generated file since if you want to change the GUI layout a new script will be generated.

N.B.: The first line is there to specify to the interpreter the kind of encoding used for the Python script (this is not important).

\subsection{Third step: Create the main code}
Now that we have the GUI code, we want to bind it with behaviours and to link signals or events to it. You will write this script in a file with a .pyw extension (it will launch the script in windowed mode, without console).

\subsubsection{Minimal code}
The \textquote{base code} for my application is the following:
\begin{pyverbatim}[][breaklines]
# Import PyQt Widgets for PyQt5 version
import sys
from PyQt5.QtWidgets import QApplication, QMainWindow, QTableWidgetItem, QHeaderView
# Import pyqtSlot to connect sliders and DoubleSpinBox signals
from PyQt5.QtCore import pyqtSlot, Qt
# Import QIcon
from PyQt5.QtGui import QIcon
# Import Ui_MainWindow class from UiMainApp.py generated by uic module
from Uidoe import Ui_Design
# Import os
import os

class MainApp(QMainWindow, Ui_Design):
    """
    MainApp class inherit from QMainWindow and from
    Ui_MainWindow class in UiMainApp module.
    """

    def __init__(self):
        """Constructor or the initializer"""
        QMainWindow.__init__(self)
        # It is imperative to call self.setupUi (self) for the interface to initialize.
        # This is defined in Uidoe.py file automatically
        self.setupUi(self)
        self.setWindowTitle("Design of Experiments - by Vincent STRAGIER")

if __name__ == "__main__":
    # For Windows set AppID to add an Icon in the taskbar
    # https://stackoverflow.com/questions/1551605/how-to-set-applications-taskbar-icon-in-windows-7
    if sys.platform == 'win32':
        import ctypes
        from ctypes import wintypes
        appid = u'vincent_stragier.umons.doe.v1.0.0'  # arbitrary string
        ctypes.windll.shell32.SetCurrentProcessExplicitAppUserModelID(appid)

        lpBuffer = wintypes.LPWSTR()
        AppUserModelID = ctypes.windll.shell32.GetCurrentProcessExplicitAppUserModelID
        AppUserModelID(ctypes.cast(ctypes.byref(lpBuffer), wintypes.LPWSTR))
        appid = lpBuffer.value
        ctypes.windll.kernel32.LocalFree(lpBuffer)

    app = QApplication(sys.argv)
    # Launch the main app.
    MyApplication = MainApp()
    MyApplication.show()  # Show the form
    icon_path = os.path.join(os.path.dirname(sys.argv[0]), 'ico', 'fpms.svg')
    app.setWindowIcon(QIcon(icon_path))
    MyApplication.setWindowIcon(QIcon(icon_path))
    sys.exit(app.exec_())  # Execute the app
\end{pyverbatim}

This code will only display the GUI with a specific title and an icon (one on the window and one in the taskbar). In Windows you must use that kind of trick to display the logo. Notice the use of a .svg file that allows lossless scaling and transparency (alpha layer).

\subsubsection{Signals and slots}
To be able to get the inputs of the user, the signals of the objects must be connected to slots functions.

First add the signal connection in the \pyv{def __init__(self):} function definition of the \pyc{MainApp} class:

\begin{pyverbatim}
self.<objecName>.<signal>.connect(self.foo)
\end{pyverbatim}

where:

\begin{description}
\item[objectName] is the name of the specific object (set by default in Qt Designer or by you)
\item[signal] is the specific signal you want to connect (you must look through the documentation)
\item[foo] is the function that will be triggered by the connected signal.
\end{description}

Second you have to define the foo function as a member of the \pyc{MainApp} class by taking into account which arguments the signal will pass to foo:

% @pyqtSlot(<type>)
\begin{pyverbatim}
def foo(self, <type>:arg):
	pass
\end{pyverbatim}
where:

\begin{description}
\item[type] is the data type of the argument passed to your function foo by the signal
\item[arg] is the argument that will be retrieved by the function foo
\end{description}

% The @ symbol is there to use the function as a decorator of \pyv{pystSlot}.

Another method exists to handle the signals, using specific function names already linked to the signals. But the previous exposed method is easier and clearer.

\subsubsection{Interact with the GUI}
You can also use other methods to interact with the elements of you GUI. Mostly to change some value, to update a text, set default value or current value of some object, etc. There is plenty information on the internet about those methods.

\chapter{How to use the GUI}
% Link to the video and step to step tutorial
To use the GUI, you can look at my demonstration of the GUI on Youtube (\url{https://youtu.be/2I9Mx7gjNnI}).

The first step is to install Python and all the required Python modules (see the previous chapters).

The second step is to download the code from GitHub (\url{https://github.com/2010019970909/design_of_experiments}).

The third step is to start the \pyv{GUI_doe.pyw} script (figure \ref{fig:doe_gui}).

The fourth step is to set the number of parameters. Then to fill the $y$ column with your measurements (figure \ref{fig:doe_gui_filled}).

Now you can select any chart in the tabs (figure \ref{fig:doe_gui_coef}, figure, \ref{fig:doe_gui_pareto}, or figure \ref{fig:doe_gui_henry}) and the coefficients are displayed in the bottom of the GUI. If you want to reset the $y$ column you can use the reset button. Also, you can enter mathematical expression (not only values) in the cells, i.e., $5*pi*e**3$.

\begin{figure}[!ht]
\centering
\includegraphics[width=\linewidth]{doe_gui}
\caption{DoE GUI\label{fig:doe_gui}}
\end{figure}

\begin{figure}[!ht]
\centering
\includegraphics[width=\linewidth]{doe_gui_filled}
\caption{DoE GUI - Bitumen emulsion example\cite{goupy_methods_1993}\label{fig:doe_gui_filled}}
\end{figure}

\begin{figure}[!ht]
\centering
\includegraphics[width=\linewidth]{doe_gui_coef}
\caption{DoE GUI - Coefficient chart\label{fig:doe_gui_coef}}
\end{figure}

\begin{figure}[!ht]
\centering
\includegraphics[width=\linewidth]{doe_gui_pareto}
\caption{DoE GUI - Pareto chart\label{fig:doe_gui_pareto}}
\end{figure}

\begin{figure}[!ht]
\centering
\includegraphics[width=\linewidth]{doe_gui_henry}
\caption{DoE GUI - Henry chart\label{fig:doe_gui_henry}}
\end{figure}

\phantomsection
\chapter*{Conclusion and analysis}
\addcontentsline{toc}{chapter}{Conclusion}

This program is working but could be improved. The computing time can be quite long to render the charts. However, each time the GUI or even the module implemented have been tested, the results were close (with sometime a light rounding error) to the solution of the examples. More charts could be added. The layout could be improved (stretching methods, etc.). The number of measured parameters could be increased. The file import, export could be improved (to select the file separators, the column(s) and line(s) to use, etc.).

%\nocite{*}
\printbibliography
\end{document}


\documentclass[12pt]{ORSNZ}
\usepackage{amssymb, amsmath, vmargin, graphicx, ORSNZ}
\usepackage{multirow}

\setpapersize{A4}
\setmargnohfrb{30mm}{20mm}{30mm}{20mm} % no header/footer, margins as required
\title{Partitioning of students into equitable groups using SolverStudio}
\author{M. Fairley*, O. Dowson\\Department of Engineering Science\\University of
  Auckland\\New Zealand\\{*}mfai035\@@aucklanduni.ac.nz}

\date{} % Don't include the date
\setlength{\parindent}{6mm} % 6mm paragraph indentation
\renewcommand{\baselinestretch}{1.0417} % to get 15pt lines (does it work?)

\begin{document}

\maketitle
\pagestyle{empty} \thispagestyle{empty}
\begin{abstract}
Students in their final year of a Bachelor of Engineering at the University of Auckland are required to complete a course called \emph{Managing a Business}. As part of this course, students are partitioned into groups of around twenty-five students and given a single week to solve a large engineering or business related problem. In order to make this partitioning equitable, the students should be partitioned in a way that makes the groups as similar as possible across several factors. These include the gender and ethnic make-up of each group, as well as the number of students from each engineering discipline and the academic performance of the students in each group. A mixed integer programme was formulated to solve this problem and implemented using Excel, SolverStudio and PuLP. The model was validated against data from the 2013 cohort, and shown to improve key metrics. The Excel spreadsheet was given to the course organiser who used the model to partition the 571 students of the 2014 class into twenty-three groups. We believe this paper provides a good case study on how the combination of Excel, SolverStudio and PuLP enables the rapid development of practical optimisation solutions.

\textbf{Key words:} Excel, PuLP, SolverStudio, partitioning students, equitable groups
\end{abstract}
\section{Introduction}
Group work is an important component of many university courses that lets students learn how to work with a diverse range of people. Many students will have an ideal group in mind before beginning a group project, for example, they may wish to work with other students who are all excellent academic performers. It is also clear the people of different backgrounds bring different attitudes and skills to a group. For example, it is likely that a group of all females would tend to behave differently to a group of all males. The mix of people in a group can have a large impact on the performance of a group, and therefore, it is important for every group in a class to share the same characteristics in order to make the partitioning of students into groups equitable for all students.

We formulated an optimisation problem for the partitioning of students into equitable groups for a group project that all final year engineering students at the University of Auckland take part in. ENGGEN 403, or \emph{Managing a Business}, runs a systems engineering group project that constitutes forty percent of each student's final course mark, and involves students being divided into groups of around twenty-five students and given a single week to produce a large piece of work on a given topic. In recent years, these topics have included a rebuild plan for Christchurch and a proposal for the future of Auckland's transport network.

In 2014 the project was to write a tender to run a new public-private partnership investment agency, and for the first time, the groups were automatically allocated to groups using the solution described in this paper. We call our solution \emph{Group-Allocator}.

This paper details the development, implementation and validation of Group-Allocator using PuLP, SolverStudio and Microsoft Excel that can automatically create equitable groups without the end user requiring knowledge of the underlying optimisation model.

\subsection{Defining an equitable partition}
There is no clear definition of an equitable partition, however, we can identify academic ability and demographic composition as being important factors in defining an equitable partition. The first is important as it would be unfair to have one group composed of mainly high-achieving students, and another with low-achieving students. The second is important to ensure that no one feels excluded in their group, and to ensure that each group has a diverse mix of people.

At the University of Auckland academic performance is measured quantitatively by grade point average (GPA). GPA's are measured on a 1-9 scale, where 1 represents a C- average, and 9 represents an A+ average. One way of measuring the overall academic ability of a group is by mean GPA; however, it is also important that the distribution of GPA for all the groups is as similar as possible, and so mean GPA cannot be considered alone. If only mean GPA were considered, then it may be possible to get a bi-modal split of high-GPA students and low-GPA students, which is likely to result in a very different group dynamic to one with all average GPA students. For this reason, in this paper we also seek to make the variance of GPA the same for each group.

When considering demographic composition, we considered gender, ethnicity and engineering discipline. ENGGEN 403 is a common paper to all the engineering disciplines at the University of Auckland. Each discipline has a unique set of skills and perspectives to contribute to group projects. To give all groups an equal balance of these skills, and to prevent a single discipline from dominating a group, students from each discipline should be distributed equally amongst the groups.

We consider this problem to be a multi-objective optimisation problem, however, we chose to formulate our model as a weighted multi-objective optimisation problem in order to create an interface for the course organiser that was simple and produced a ``good enough'' solution.

\subsection{Previous Solution}

In previous years, the job of partitioning the students into groups has been the course organisers job. The method can be considered to be a greedy heuristic, and works as follows:

Starting with the smallest discipline, distribute the students across the groups aiming to balance the mean GPA of each group. Once complete, move onto the next smallest discipline and repeat. As more students are added, try to balance gender, ethnicity and mean GPA. Once all students are added, perform a series of swaps between pairs of students to improve the overall balance.

This process usually took on the order of two days to complete.

\section{Mathematical Model}
In this section, we define the mixed integer mathematical programme that was formulated to solve the problem.
\begin{description}
\item[Sets] \mbox{}\\
        Students: $S = \{1, 2, \dots, m\}$;\\
        Groups: $G = \{1, 2, \dots, n\}$;\\
        Ethnicities: $E = \{\mbox{P\=akeh\=a/European, M\=aori, Asian, Other}\}$;\\
        Genders: $K = \{\mbox{Male, Female}\}$;\\
        Disciplines: D = \{Civil, Chemical \& Materials, Mechatronics,Mechanical, Electrical, Software, Computer Systems, Engineering Science, Biomedical\};
        Students who identify as gender $k$: $S^K_k \subset S, \quad \forall k \in K$;\\
        Students who identify as ethnicity $e$: $S^E_e \subset S, \quad \forall e \in E$;\\
        Students who study discipline $d$: $S^D_d \subset S, \quad \forall d \in D$.

\item[Parameters] \mbox{} \\
$\gamma_s = $ grade point average of student $s$; \\
$M_g = $ the number of students in group $g$;\\
\textbf{Note:} the values of $M_g$ are computed a priori. They should add up to the total number of students in the class.\\
$\alpha = $ priority given to allocating groups with equal mean compared to groups with equal variance. This should be between 0 and 1;\\
$\beta^I = $ penalty weighting put on violating demographic constraints of type I, $I= D, E, K$;\\
$\hat{\mu} = $ mean GPA of all students;\\
$\hat{\sigma^2} = $ variance in the GPA of all students.

\item[Decision variables]\mbox{} \\
$x_{sg}\in\{0,1\} = $ 1 if student $s$ allocated to group $g$, 0 otherwise;\\
$\rho^I_{gi} = $ artificial penalty variable for group $g$, factor $i$, \qquad $I=\{K, D, E\}$;\\
$\mu_{max} = $ maximum mean GPA of any group;\\
$\mu_{min} = $ minimum mean GPA of any group;\\
$\sigma^2_{max} = $ maximum variance of GPA within any group;\\
$\sigma^2_{min} = $  minimum variance of GPA within any group.

\item[Objective Function]\mbox{} \\
The objective function is to minimise a convex combination of the spread between the maximum and minimum mean group GPA, and the spread between the maximum and minimum variance of the GPAs within a group. Adding a penalty term with a large coefficient $\beta$ is necessary as some constraints are relaxed.

\begin{equation}\begin{split}
\mbox{Minimise }&
\underbrace{\alpha\left(\frac{\mu_{max} - \mu_{min}}{\hat{\mu}}\right)}_{\mbox{normalised mean}}
+ \underbrace{(1-\alpha)\left(\frac{\sigma^2_{max} - \sigma^2_{min}}{\hat{\sigma^2}}\right)}_{\mbox{normalised variance}}
\\&+ \underbrace{\sum_{g \in G}\left(\beta^K\sum_{k \in K}\rho^K_{gk}
    + \beta^D\sum_{d \in D}\rho^D_{gd}
    + \beta^E\sum_{e \in E}\rho^E_{ge}\right)}_{\mbox{penalty variables}}
\end{split}\end{equation}

\item[Constraints]\mbox{} \\
\textbf{Partitioning Constraint} These constraints ensure that all students are allocated to exactly one group.
\begin{equation} \label{con1}
\sum_{g=1}^n x_{sg} =  1 \qquad \mbox{$\forall s \in S$}
\end{equation}

\begin{equation} \label{con2}
\sum_{s \in S} x_{sg} =  M_g \qquad \mbox{$\forall g \in G$}
\end{equation}

\textbf{Bounds on objective variables} These constraints set the values of the variables used in the objective function.
\begin{equation} \label{minu}
\frac{1}{M_g}\sum_{s \in S} x_{sg}\gamma_s \geq \mu_{min} \qquad \mbox{$\forall g \in G$}
\end{equation}
\begin{equation} \label{maxu}
\frac{1}{M_g}\sum_{s \in S} x_{sg}  \gamma_s \leq \mu_{max} \qquad \mbox{$\forall g \in G$}
\end{equation}

\begin{equation} \label{minv}
\frac{1}{M_g}\sum_{s \in S} x_{sg} (\gamma_s  - \hat{\mu})^2 \geq \sigma^2_{min} \qquad\mbox{$\forall g \in G$}
\end{equation}
\begin{equation} \label{maxv}
\frac{1}{M_g}\sum_{s \in S} x_{sg}(\gamma_s - \hat{\mu})^2 \leq \sigma^2_{max} \qquad\mbox{$\forall g \in G$}
\end{equation}

\textbf{Bounds on demographic variables} These constraints set the minimum number of students in each group from each demographic sub-group. These constraints are relaxed via the addition of $\rho$ variables.
\begin{equation} \label{con3}
\sum_{s \in S^K_k} x_{sg} + \rho^K_{gk} \geq \left\lfloor\frac{M_g \times |S^K_k|}{|S|}\right\rfloor \qquad \mbox{$\forall g \in G \quad \forall k \in K$}
\end{equation}

\begin{equation} \label{con4}
\sum_{s \in S^D_d} x_{sg} + \rho^D_{gd}  \geq \left\lfloor\frac{M_g \times |S^D_d|}{|S|}\right\rfloor \qquad \mbox{$\forall g \in G \quad \forall d \in D$}
\end{equation}

\begin{equation} \label{con5}
\sum_{s \in S^E_e} x_{sg} + \rho^E_{ge}  \geq \left\lfloor\frac{M_g \times |S^E_e|}{|S|}\right\rfloor \qquad \mbox{$\forall g \in G \quad \forall e \in E$}
\end{equation}

\textbf{Variable Bounds} The allocation variables ($x$) are binary, all other variables are non-negative.
\[x_{sg} \in \{0, 1\}\]
\[\mu_{min}, \mu_{max}, \sigma^2_{min}, \sigma^2_{max}, \rho^I_{gi}  \ge 0.\]

\end{description}
\subsection{Estimating the variance of a group}
In order to calculate the GPA variance of a group, one needs to know the square of the mean of the mean GPA of that group. This is a non-linear function of a variable in the model. Therefore, to prevent the problem from being non-linear, we make an assumption that the mean GPA of every group is equal to the mean GPA of all the students. This changes the square from acting on a variable, to acting on a parameter. We can make this assumption as constraints (\ref{minu}) and (\ref{maxu}) act to force the mean GPA of each group towards the global mean GPA ($\hat{\mu}$). When we verified the model, we found this assumption to be valid.

\subsection{The effect of relaxing the demographic constraints}
It is noticed that the demographic constraints take the form of
\begin{equation}
\sum_i x_i + y \geq z
\end{equation}
where $x_i$ and $z$ are both integer values. Since the objective is to minimise, and the objective coefficients of the $y$ variables are positive, they will optimise to their lower bound. This implies that the $y$ variables will also take integer values in an optimal solution.

If the values of $\beta$ are large relative to the rest of the objective function, then the effect of this is to optimise the demographics of the groups first, and then proceed optimising the GPAs. When allocating small numbers of students, this is likely to lead to poor quality solutions with respect to GPA metrics. However, in ENGGEN 403 there are around 600 students every year. This means that there are likely to be multiple students with the same gender, ethnicity and discipline. Therefore, there is still enough freedom in the problem to produce a high quality solution with respect to the GPA metrics.

Any of these constraints can be removed from the model by setting the corresponding $\beta$ value to zero. This may be desirable for smaller classes where a greater emphasis is placed on GPA.
\section{Implementation}

The model was implemented in the SolverStudio \cite{solverstudio} modelling environment. SolverStudio is an add-in for Microsoft Excel. It acts as an interface between Excel and a variety of modelling languages including AMPL \cite{ampl}, GAMS \cite{gams}, PuLP \cite{pulp} and Gurobi \cite{gurobi}. In order to leverage the ability of SolverStudio to interface with Excel, it was decided to use the PuLP modelling language. CBC \cite{CBC} was chosen for the solver as it is free to use so the model could be given to the course organiser without licensing issues.

Model data is typed into cells in Excel and \emph{named ranges} are created using SolverStudio's \emph{Edit Data} function. These ranges are dynamic, so that the user does not need to interact with this functionality. Adding a new row to the table will automatically adjust the named range in SolverStudio. This was done by using the following formula in the \emph{Cell Range} field in the SolverStudio Data Items Editor instead of hard coding a fixed range:

\begin{verbatim}
    = OFFSET(A2, 0, 0, COUNTA(A:A)-1, 1)
\end{verbatim}

where the variable being created is in column $A$ (so that the header is cell $A1$). A number of data checks are displayed to the user, for example, that there is no missing data for any students.

When the model is solved, SolverStudio loads the data from the named ranges into standard Python dictionaries that can be accessed in the usual way. This provides seamless interaction between the PuLP model object and the data. When the model has finished solving, the reverse occurs, and SolverStudio writes the solution into a named range on the sheet. In this manner, the user does not need to understand any Python or optimisation modelling in order to use the spreadsheet.

Rather than solve to optimality, the user specifies a time limit for the solver.

The programme has been tested in Excel 2010 and Excel 2013 with SolverStudio version $>$0.6.x.

\subsection{Reporting} \label{reporting}
SolverStudio provides access to Excel's API (Application Programming Interface), allowing the Python code to not only create and solve the model via PuLP, but also leverage Excel's graphing functions to create a number of visualisations of the solution.

Figure \ref{FIG:solverstudio_boxplot} shows an example of one of the plots produced by the Python code. In this case, boxplots of GPA are produced for each group. This enables a quick visual tool for users to understand the quality of the solution, ensuring there are no clear outliers.

\begin{figure}[!ht]
    \centering
    \includegraphics[width=0.95\textwidth]{solverstudio_boxplot_c.pdf}
    \caption{Example graph produced programmatically by the SolverStudio model.}
    \label{FIG:solverstudio_boxplot}
\end{figure}

In addition, bar charts showing the number of students in each group were produced for each gender, discipline and specialisation. This allowed a visual means of checking the solution quality.

Excel's ability to save single worksheets as a PDF (portable document format) was also utilised to create a final report that could be distributed to students showing their allocated group.

\section{Model Validation}
In 2013, 586 students enrolled in ENGGEN 403. They were partitioned into 24 groups by the course organiser. To test the quality of the solutions provided by Group-Allocator, an anonymised version of the dataset was obtained from the course organiser.

There were 14 groups of 24 students, and 10 groups of 25 students. A value of 0.5 was used for $\alpha$, and all $\beta$ values were set to $1 \times 10^4$. The students had a mean GPA of 5.92, with a variance of 2.70. The model was solved for 120 seconds.

There was a small number of students who did not have a recorded discipline, or were from a different degree program. They were classified in the \emph{Other} category for discipline.

Table \ref{TAB:results} shows the results of the partitioning for the 2013 ENGGEN 403 class. The \emph{Class} column gives the corresponding value of all the students enrolled in the class. The \emph{Manual} columns give the values for the groups as allocated by the course organiser, and the \emph{Group-Allocator} columns give the values for the groups created by Group-Allocator.

Values in bold indicate that the corresponding method returned a solution with a value closer to the class value than the alternative method.

\begin{table}[!ht]
    \centering
    \begin{tabular}{l r | c | c c | c c}
    & &  & \multicolumn{2}{c |}{Historic} & \multicolumn{2}{c}{SolverStudio}\\
    &  & Class & Min & Max & Min & Max\\[1ex]
    \hline
    & &  & \multicolumn{2}{c |}{} & \multicolumn{2}{c}{}\\[-2ex]

    \multirow{2}{*}{GPA}& Mean  &   5.92    &5.42   &6.60   &\textbf{5.74}  &\textbf{6.05} \\
    & Variance& 2.70    &1.70   &3.83   &\textbf{2.56}  &\textbf{3.16}\\[1ex]

    \hline& &  & \multicolumn{2}{c |}{} & \multicolumn{2}{c}{}\\[-2ex]

    \multirow{10}{*}{Discipline (\%)}&Biomedical            &4.44   &4.00   &8.33   &4.00   &8.33   \\
    &Engineering Science    &4.78   &0.00   &8.33   &\textbf{4.00}  &\textbf{8.00}  \\
    &Software           &8.02   &4.00   &\textbf{12.00}  &4.00  &12.50  \\
    &Electrical         &13.30  &8.33   &16.70  &\textbf{12.00}     &16.70  \\
    &Civil              &28.70  &25.00  &33.30  &\textbf{28.00}     &\textbf{29.20}     \\
    &Mechanical         &13.50  &12.00  &16.70  &12.00  &16.70  \\
    &Computer Systems   &6.31   &4.00   &12.50  &4.00   &\textbf{12.00}     \\
    &Mechatronics       &9.90   &8.00   &\textbf{12.50}     &8.00   &16.00  \\
    &Chemical \& Materials& 10.60&8.00  &12.50  &8.00   &12.50  \\
    &Other              &0.51   &0.00   &4.17   &0.00   &4.17\\[1ex]

    \hline& &  & \multicolumn{2}{c |}{} & \multicolumn{2}{c}{}\\[-2ex]

    \multirow{2}{*}{Gender (\%)}&  Female   &26.11& 24.00&  \textbf{28.00}& 24.00&  33.33 \\
    & Male  &73.89& \textbf{72.00}& 76.00&  66.67&  76.00 \\[1ex]

    \hline& &  & \multicolumn{2}{c |}{} & \multicolumn{2}{c}{}\\[-2ex]

    \multirow{4}{*}{Ethnicity (\%)}&    P\=akeh\=a / European   &38.57  &25.00  &50.00  &\textbf{36.00} &\textbf{44.00} \\
    &   M\=aori     &3.75   &0.00   &12.50  &0.00   &\textbf{8.33}  \\
    &   Asian       &53.75  &41.67  &64.00  &\textbf{52.00} &\textbf{58.33} \\
    &   Other       &3.92   &0.00   &8.33   &0.00   &8.33

    \end{tabular}
    \caption{Partitioned composition of the 2013 ENGGEN 403 class}
    \label{TAB:results}
\end{table}

\subsection{Grade Point Average}

\begin{figure}[!ht]
    \centering
    \includegraphics[height=8cm]{./gpa_plot.pdf}
    \caption{Boxplots of mean group GPA and variance within group GPA for manual and Group-Allocator partitioning.}
    \label{FIG:boxplots}
\end{figure}

When compared to the manual partitioning, the Group-Allocator partitioning reduces the spread in mean GPA between the groups. The lowest mean GPA of any of the groups is 5.74 (compared to 5.42 for the manual solution), while the highest mean GPA is 6.05 (compared to 6.6 for the manual solution).

Similarly, Group-Allocator created groups with a larger minimum GPA variance (2.56 compared to 1.7), and a smaller maximum GPA variance (3.16 compared to 3.83) than the manual partitioning.

This improvement can also be seen visually in boxplots of the mean and variance of the GPAs of each group (Figure \ref{FIG:boxplots}). In both cases Group-Allocator has a smaller spread than the manual solution, resulting in more similar groups.

\subsection{Discipline}
When compared to the manual partitioning, the Group-Allocator partition performed better in six metrics and poorer in two metrics. The manual partitioning had a smaller maximum number of software students (12\% compared to 12.5\%) and a smaller maximum number of Mechatronics students (12.5\% compared to 16\%).

In addition, there was one group in the manual partitioning with zero Engineering Science students. In the Group-Allocator solution there were at minimum 4\% Engineering Science students.

\subsection{Gender}
When compared to the manual partitioning, Group-Allocator created a poorer partitioning of students based on gender. There was at most 28\% females in a group in the manual partitioning compared to 33.3\% in the Group-Allocator partitioning. There was a corresponding difference in the minimum percentage of male students.

\subsection{Ethnicity}

Group-Allocator generally produced both a higher minimum number of students and a smaller maximum number of students than the manual method.

\section{Discussion}
The results indicate that both the manual method and Group-Allocator produced balanced groups, but Group-Allocator clearly performed better with regards to mean GPA and GPA variance. The results also indicate the difficulty in comparing two solutions, as there are many different factors to consider and a single number cannot encapsulate a meaningful measure of solution quality. It could be said that a solution just needs to be ``good enough'', and that it is possible to judge this by checking for any outliers in the factors considered. For example, a group with all one ethnicity would clearly not be part of a good solution. The manual method and Group-Allocator both produced good solutions, but Group-Allocator provides an important advantage, in that it is fully automatic and takes 120 seconds to find a solution compared to two days with the manual method.

Group-Allocator improved almost all metrics than when compared to the manual solution, but it resulted in a poorer partitioning with respect to gender. The likely reason for this is that the course organiser placed more emphasis on balancing gender than other demographic factors. However, this could be overcome in future by changing the $\beta^K$ value. This small decrease in gender equity between groups is offset by the more even distributions of students by discipline and ethnicity.

The model takes a long time to solve to optimality, and hence why a time limit is specified by the user. This is due to a variety of factors, but chiefly the large number of binary variables (number of students $\times$ number of groups). Although the CBC solver quickly (less than one minute) finds a good integer solution, it is unable to find the optimal solution. Other approaches, such as a set partitioning formulation, may improve the solution time and quality. However, given that students are unaware of the GPAs of other students in their groups, they are unlikely to perceive a small improvement in solution quality. It was for this reason, and the tight development timeframe, that other approaches were not investigated.

\section{Conclusion}
This paper provides a case study on how the combination of Excel, SolverStudio and PuLP enables the rapid development of non-trivial optimisation solutions. The total development timeframe, from conception to delivery of a finished product, took two weeks of part time work. The majority of this was spent creating the reporting and visualisation tools. Using this model as a guide for future problems will reduce this time even further.

This case study also highlights the ability for SolverStudio to abstract a large amount of complexity from the end user. The course co-ordinator needed only minutes of instruction to be able to successfully use the spreadsheet without supervision. Where the task of partitioning the students previously took two days, the Excel spreadsheet is able to achieve a more equitable result in the time it takes to make a cup of coffee.

Optimisation does not always need to be applied to large commercial problems to be beneficial. Instead, there exist a large number of problems where years of experience has culminated in manual solutions with good results, but for which optimisation can provide even better quality solutions in faster times. Simple solutions to these problems can be created using SolverStudio in a short span of time that are user friendly and significantly reduce the work required by the user, freeing them up for more productive activities.
\section*{Acknowledgments}
The authors gratefully acknowledge Dr. Keith Adams of the Civil Engineering Department at the University of Auckland for using the model to make the ENGGEN 403 groups in 2014, providing the historical data used to validate the model, and for providing valuable feedback. The final spreadsheet can be found on GitHub at http://www.github.com/odow/group-allocator.
\bibliographystyle{achicago}
\bibliography{ORSNZ}

\end{document}


\subsubsection{Basic usage}

These two tools can be used to create the images corresponding to a given mock catalog created by \bashinline{egg-gencat} (custom catalogs are also supported, see below). They provide a simple alternative to \skymaker, with the added limitation that galaxies are all considered as point sources. This is usually fine for long wavelength images (\spitzer MIPS and \herschel, typically) in cosmological deep fields. In addition, more control is given relative to the noise properties.

The typical call sequence consists of two steps: creating the noise map with \bashinline{egg-gennoise}, then painting the galaxies on top of it with \bashinline{egg-genmap}. The standard way to use \bashinline{egg-gennoise} is the following:
\begin{bashcode}
# assume we want to create an image of the Herschel PACS 160um band
egg-gennoise cat=egg-20151201.fits out=pacs160-noise.fits \
    psf=herschel-pacs160.fits aspix=2.4 rms=1.68e-5 verbose
\end{bashcode}
The argument \bashinline{cat} gives the path to the mock catalog, while \bashinline{out} gives the name of the file into which the noise map will be saved. Then, \bashinline{psf} provides the FITS image of the point spread function of this instrument, \bashinline{aspix} is the pixel size (in arcseconds/pixel) and \bashinline{rms} is the desired noise level of the map (in map units, whatever they are). This latter value would be the standard deviation of the map pixel values without any galaxy painted on it. If you want to mimic the noise level of an existing image, simply measure the standard deviation of the pixels in an empty region and use this value for the \bashinline{rms} keyword. The noise is assumed to be Gaussian with zero average.

Some common PSFs are provided with \egg. They are stored in the \bashinline{$INSTALL_ROOT_DIR/share/egg/psfs} folder (see the installation instructions if you don't know what \bashinline{$INSTALL_ROOT_DIR} is). If the program cannot find the PSF you provided in the current directory, it will each among these PSFs for a matching name. Use the \bashinline{list_psfs} argument to display a list of all the available PSFs.

Once the process is complete, this will create a new image containing just noise. The dimensions of the image are computed automatically from the provided pixel size and the area covered by the mock catalog, so that no galaxy is truncated close to the border of the image. A padding of 5 times the full-width at half-maximum of the PSF is included to ensure that this is true. The resulting noise map contains the necessary WCS astrometry.

Once this noise map is created, we can use \bashinline{egg-genmap} to add the galaxies:
\begin{bashcode}
egg-genmap cat=egg-20151201.fits out=pacs160-sci.fits \
    noise_map=pacs160-noise.fits band=herschel-pacs160 \
    psf=herschel-pacs160.fits flux_factor=1.613e6 verbose
\end{bashcode}
The values of \bashinline{cat} and \bashinline{psf} must be the same as in the previous step. This time, \bashinline{out} gives the name of the file into which the final image will be saved, while \bashinline{noise_map} must be set to the name of the noise map created by \bashinline{egg-gennoise}. Finally, \bashinline{band} sets the photometric band from which the flux of each galaxy will be drawn, and \bashinline{flux_factor} is the conversion factor from fluxes to map units.

The flux conversion factor is strongly tied to the properties of the PSF file. If the PSF is normalized to unit peak flux, then it is the conversion between $\uJy/{\rm beam}$ to map units. Else, if the PSF is normalized to unit total flux (the sum of all pixels is equal to one), then it is the conversion between $\uJy/{\rm pixel}$ and map units. Be careful that, in this latter case, the aperture correction resulting from a truncated PSF need to be included, either in the flux conversion factor or in the normalization of the PSF itself (the program will not renormalize it internally). In all cases, a galaxy is placed on the map following:
\begin{equation}
\cppinline{map[i,j]} \mathrel{+}= \cppinline{flux_factor} \times \cppinline{psf[i,j]} \times (S_{\nu}/\uJy) \nonumber
\end{equation}
where \cppinline{psf[i,j]} is the pixel value of the PSF translated to the position of the galaxy and $S_{\nu}$ is the flux of this galaxy.

\subsubsection{Using a custom astrometry and image dimensions}

The tool \bashinline{egg-gennoise} can compute automatically a suitable astrometry and image dimensions according to the input catalog. However, you also have the possibility to copy the astrometry and image dimensions of an existing file. To do so, just use the \bashinline{astro} command line argument and make it point to the FITS image you want to copy the data from:
\begin{bashcode}
egg-gennoise cat=egg-20151201.fits out=pacs160-noise.fits \
    psf=herschel-pacs160.fits rms=1.68e-5 verbose \
    astro=this_image_I_like_a_lot.fits
\end{bashcode}
In this case, the value of \bashinline{aspix} is not used and it can be omitted from the argument list.

\subsubsection{Flag regions of the map with poor catalog coverage}

By default the tools create a large map that encompasses the whole input catalog, with some margin. This means that, close to the borders of the image, the source density is very low and the image statistics are not realistic. This can be an issue for some applications. To cope for this, the \bashinline{egg-gennoise} option \bashinline{clip_borders} will identify the regions of the map with low source density, and flag them with ``not-a-number'' pixels so that no value can be read from there. This assumes that the input catalog does not contain holes, and will only affect the borders of the image.

\subsubsection{Generate beam-smoothed images (sub-millimeter)}

It is common practice in sub-millimeter astronomy to filter the observed maps with the PSF (or beam) of the instrument. Because these images usually have low signal-to-noise ratios, this allows easier identification of detections by eye or with blind source extraction tools. Note however that this is sub-optimal for prior-based source extraction or stacking because the size of the effective PSF is multiplied by $\sqrt{2}$, which increases the confusion. In addition, the gain is signal-to-noise is null since these methods already apply a sort of beam-smearing internally.

Nevertheless, in order to be able to reproduce such kind of images, both \bashinline{egg-gennoise} and \bashinline{egg-genmap} provide a set of command line arguments. In all cases, \bashinline{beam_smoothed} must be set \emph{for both programs} to notify them of your intention. By default, the map will be smoothed using the PSF as kernel. If you prefer to use a custom kernel, use the \bashinline{smooth_fwhm} to specify the full-width at half-maximum of the kernel (in pixels).

If this option is used, the behavior of the programs is the following:
\begin{itemize}
\item Create the noise map, \emph{unfiltered} (without smoothing), and with the requested noise level.
\item Copy this map and apply the beam smoothing. Measure the new noise standard deviation (it is indeed modified by the smoothing process).
\item Adjust the noise level of the \emph{unfiltered} map so that the noise level in the previous step matches the one requested by the user.
\item Feed the \emph{unfiltered} map to \bashinline{egg-genmap}, add the galaxies.
\item Finally perform the beam smoothing.
\end{itemize}
This means that the value of the \bashinline{rms} argument must be set to the expected noise standard deviation of the final, smoothed map.

\subsubsection{Generate error and coverage maps}

Without extra effort, the \bashinline{egg-gennoise} program can generate for you the error and coverage map corresponding to your noise map and input catalog. To do so, simply set the \bashinline{make_err} and \bashinline{make_cov} arguments; this will generate the error map as \cppinline{"[out]-err.fits"} and the coverage map as \cppinline{"[out]-cov.fits"} ($1$: covered, $0$ or not-a-number: not covered).

These maps carry very little information since the noise generated by \bashinline{egg-gennoise} is uniform and covers the whole map by default. However, some external tools require the existence of either one or both of these.

\subsubsection{Using a custom noise map}

The whole point of separating the map-making process into two steps is that one can be used without the other. Indeed, \bashinline{egg-genmap} can work with any noise map, provided that it contains valid WCS astrometry. The tool can handle sources outside of the map, so it is fine if the provided noise map does not overlap perfectly with the input catalog. Just make sure that the PSF you provide is tabulated on the same pixel scale as your noise map. If you use the \bashinline{beam_smoothed} option, note that the noise map must be provided \emph{unfiltered} (i.e., before beam convolution).

For example, realistic \herschel noise maps can be obtained by jackknifing the observed data, and provide noise statistics (correlation, amplitude, etc.) that match very well the real images. Such kind of map can be fed naturally to \bashinline{egg-genmap} through the \bashinline{noise_map} argument.

\subsubsection{Using a custom flux catalog}

Similarly, both tools can work with input catalogs that were not produced by \bashinline{egg-gencat}. They only need a limited set of columns: \cppinline{"ra"} and \cppinline{"dec"} to know the position of each galaxy on the sky, and \cppinline{"flux"} must contain the flux of each galaxy in $\uJy$.

Catalogs produced by \bashinline{egg-gencat} contain multiple fluxes per galaxy, so in this case you also need the \cppinline{"bands"} column and the \bashinline{band} command line argument to identify the correct flux. Alternatively, the program can also work with a \cppinline{"flux"} column containing a single band (i.e., a 1D vector column), in which case the \cppinline{"bands"} column and the \bashinline{band} command line argument can be omitted. If you prefer to use ASCII format, the order of the columns must be \cppinline{"ra"}, \cppinline{"dec"} and \cppinline{"flux"} (1D only).

\subsubsection{Adjusting accuracy and speed}

By default both tools write FITS images on the disk with single precision floating point numbers. While this is sufficient for most usages, you can change this behavior and use double precision instead. To do so, just provide the \bashinline{double} flag in the command line arguments of both \bashinline{egg-gennoise} (if you use it) and \bashinline{egg-genmap}. Generated maps will occupy twice more disk space. Note that internally the maps are manipulated with double precision format in any case, so the only improvement in precision that is provided by this option is to avoid truncating significant digits at the very end of the map making process when the file is written on the disk.

If instead you are interested in \emph{decreasing} the accuracy of the simulation to save computation time, you can ask to disable the sub-pixel interpolations of the PSF using the \bashinline{no_subpixel} flag of \bashinline{egg-genmap}. With this option enabled, the program will consider that all the input sources have their center at integer pixel coordinates. Else (the default), bilinear interpolation is performed to shift the provided PSF FITS image for each source, which is more correct but slower.