\chapter{Mitigation techniques}\label{ch:mitigation}

This paper focused so far on the foundation and expansion of the attack. In
this chapter we investigate several mitigation techniques. We examine
the methods proposed by Gluck, Harris and Prado in the original paper under the
new findings that were described in previous chapters. Finally we propose
novel mitigation techniques that either limit the scope of the attack or
eliminate it completely.

\section{Original mitigation tactics}\label{sec:original_mitigation}

The original BREACH paper \cite{breach} included several tactics for mitigating
the attack. In the following sections we will investigate them one by one to
find if they can still be applied.

\subsection{Length hiding}

The first proposed method is an attempt to hide the length information from the
attacker. This can be done by adding a random amount of random data to the end
of the data stream for each response.

As stated in the BREACH paper this method affects the attack efficiency only
slightly. Since the standard error of mean is inversely proportional to
\begin{math}\sqrt{N}\end{math}, where \begin{math}N\end{math} is the number of
repeated requests the attacker makes for each guess, the attacker can deduce the
true length with a few hundred or thousand requests.

In this paper we have described how repeated requests can lead to such bypassing
of the noise as described in Section \ref{sec:probabilistic}. Experimental
results have also shown that for the endpoints tested it is possible to
perform the attack under certain circumstances despite of noise.

\subsection{Separating Secrets from User Input}\label{subsec:sep_compression}

This approach states that user input and secrets are put in a completely
different compression context. Although this approach might work when the secret
is clearly distinct it does not apply universally.

In this work we were able to defeat this mitigation measure by introducing
alternative secrets. As described in Sections \ref{subsec:fb} and
\ref{subsec:gmail_mail}, user input and secrets are sometimes one and the same.
In the case of Facebook chat the attacker can use as the chosen plaintext
private messages and in the case of Gmail private mails.

In such cases the secret and the attacker's chosen plaintext are
indistinguishable, making this mitigation technique inapplicable.

\subsection{Disabling Compression}

This paper focuses on attacks on encrypted compressed protocols. Since
encryption poses the vulnerability that is exploited, disabling it at the HTTP
level would result in total defeat of the attack.

However, such solution would have drastic impact on the performance of web
applications. An example on Facebook shows that a regular empty search result
response page from a minimal account takes up to 12 kilobytes if compressed,
opposed to 46 kilobytes as raw plaintext. It is obvious that the trade-off is
too much to handle especially for large websites that serve tens of thousands
of user requests per second.

\subsection{Masking Secrets}

The attacks investigated are based on the fact that the secret remains the same
between different requests. This mitigation method introduces a one-time pad
\begin{math}P\end{math} that would be XOR-ed with the secret and concatenated
to the result as: \begin{math}P||(P \oplus S)\end{math}.

As we have found Facebook uses this method in order to mask its CSRF tokens.
This successfully stops the attack from being able to steal this secret.

However, we have shown that many more secrets other than CSRF tokens exist
and would need to be masked in order to completely mitigate the attack. Since
masking doubles the length of every secret while also making the secret not
compressible due to the increase in entropy, the implementation of this method
would result in major loss of compressibility and as a result performance.

\subsection{Request Rate-Limiting and Monitoring}

The attack requires a large amount of requests toward the chosen endpoint,
especially against block ciphers. In order for it to give results in a
reasonable amount of time these requests would need to be made in a short
period. In such case if the endpoint monitors the traffic from and to a
specific user and limits the requests to a certain amount for a specified time
window it would slow down the attack significantly.

However, this method also does not come without cost. Rate limiting provides a
half-measure against the attack since it only introduces a delay without
defeating it completely. When more optimization techniques are proposed, like
the ones described in Section \ref{sec:optimization}, this delay would prove to
be of little help.

\subsection{More Aggressive CSRF Protection}

As the original BREACH paper stated, "requiring a valid CSRF token for all
requests that reflect user input would defeat the attack".

While this is true for CSRF tokens, we have showed that alternative secrets
that cannot be distinguished from user input could still be compromised.

\section{Novel mitigation techniques}\label{sec:novel_mitigation}

In this section we propose several potentially stronger mitigation techniques,
that have not been introduced in literature so far.

\subsection{Compressibility annotation}\label{subsec:annotation}

As described in Section \ref{subsec:sep_compression} a mitigation technique
could involve different compression implementations for secrets and user input.
Although this solution does not apply for all kinds of secrets, it could be
effective for most commonly and easily attacked ones such as CSRF tokens.

Our proposition is that web servers and web application servers cooperate to
indicate which portions of data must not be compressed. Application servers
should be parameterized by the administrator in order to annotate each response
to the web server.

Annotation would then indicate where secrets are located and where a reflection
could be located. The annotation syntax could include HTML tags that describe
the functionality of each data portion in the body of the response, a deployment
descriptor, such as \texttt{web.xml} used in Java applications, or a new special
format.

The annotated response from the application server would then be interpreted by
the web server that would change its compression behavior accordingly.
Specifically the server could disable compression of either reflections, secrets
or both, sending them always as literals. In the case of BREACH, disabling the
LZ77 stage of compression would also be sufficient, since the functionality of
this algorithm is the one exploited in such attacks whereas Huffman does more
harm than good.

Furthermore, this functionality should be implemented separately in every web
framework, such as Django\footnote{\url{https://www.djangoproject.com}}, Ruby on
Rails\footnote{\url{http://rubyonrails.org}} or
Laravel\footnote{\url{http://laravel.com}}, as well as web servers, such as
Apache\footnote{\url{http://httpd.apache.org}} or
Nginx\footnote{\url{http://nginx.org}}. In each framework a module should be
created that handles the annotation on either side of the communication.

\subsection{SOS headers}\label{subsec:sos}

Storage Origin Security (SOS) is a policy proposed by Mike Shema and Vaagn
Toukharian in their 2013 Black Hat presentation \texttt{Dissecting CSRF Attacks
\& Defenses} \cite{sos}. Its intended purpose is to counter CSRF attacks,
although a side-effect would be the mitigation of attacks such as BREACH.

SOS applies on cookies and defines whether a browser should include each cookie
during cross-origin requests or not. This definition is included in the
Content-Security-Policy response header of a web application in a form that
sets a SOS policy for each cookie.

The policies applied are \texttt{any}, \texttt{self}, \texttt{isolate}.
\texttt{Any} states that the cookie should be included in the cross-origin
requests after a pre-flight request is made to check for an exception to the
policy. This is the behaviour browsers demonstrate as of today. \texttt{Self}
states that the cookie should not be included, although again a pre-flight
request is issued to check for exceptions. \texttt{Isolate} states that the
cookie should not be included in any case and no pre-flight request should be
made.

Pre-flight requests are already used extensively under the cross-origin resource
sharing
(CORS)\footnote{\url{https://en.wikipedia.org/wiki/Cross-origin_resource_sharing}}
standard.  This mechanism describes HTTP headers that allow browsers to request
remote URLs only if they have permission. The browser sends a request that
contains an \texttt{Origin} HTTP header to which the server responds with a
list of origin sites that are allowed to access the content or an error page
in case cross-origin is prohibited.

SOS policy introduces an \texttt{Access-Control-SOS} header which contains a
list of cookies that the browser needs to confirm before including them in the
request. The server could then respond with an \texttt{Access-Control-SOS-Reply}
header that instructs the browser to \texttt{allow} or \texttt{deny} all of the
cookies mentioned in the request header, as well as a timeout period for the
browser to apply this new policy. In absence of such a reply header the browser
may apply the default policy of each cookie instead.

BREACH relies on cross-origin requests in order for the attacker to insert a
chosen plaintext in the body of a response from a chosen endpoint. The
introduction of SOS headers would effectively stop BREACH and similar future
attacks that exploit this aspect of web communications.

If SOS method were to be applied, websites could apply strict policies as to
which origins could access which data and under which context. As long as
websites integrate HTTP Strict Transport Security
(HSTS)\footnote{\url{https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security}},
malicious script injection as described in Section \ref{sec:persistence} would
be counter-measured. Combined with SOS headers, a malicious website controlled
by the attacker could be disallowed from issuing requests including the victim's
cookies, resulting in practical mitigation against side-channel compression
attacks such as BREACH.

For more information regarding SOS headers we refer to the Black Hat presentation
slides
\footnote{\url{https://deadliestwebattacks.files.wordpress.com/2013/08/bhus_2013_shema_toukharian.pdf}}
and video \footnote{\url{https://www.youtube.com/watch?v=JUY4DQZ02o4}}, as well
as an extended blog post on the
proposal\footnote{\url{deadliestwebattacks.com/2013/08/08/and-they-have-a-plan/}}.
Also there is a discussion thread in the mailing list of W3C Web Application
Security Working
Group\footnote{\url{http://lists.w3.org/Archives/Public/public-webappsec/2013Aug/0037.html}},
regarding the implementation of SOS headers as a standard in modern browsers.


% sage_latex_guidelines.tex V1.20, 14 January 2017
\documentclass[Review,sageh,times, doublespace]{sagej}

% ====================================
% Personal Settings
% ADD REVIEWING PACKAGE
\usepackage{easyReview}
% Show reviews/edits or not?
\setreviewson
%\setreviewsoff
% Line numbers for ease of pointing to lines
\usepackage{lineno} %[pagewise]
%\linenumbers

\usepackage{pdflscape}
%Math typesetting packages
\usepackage{amsfonts, amssymb, amsmath, latexsym, amsthm, mathabx, bm}
%for URLs in-text
%\usepackage{url}
% ====================
% = Math definitions =
% ====================
\renewcommand{\leq}{\varleq}
\renewcommand{\geq}{\vargeq}

% ================
% = Bibliography =
% ================
%APA style citations and references
%\usepackage[utf8]{inputenc}
%\usepackage{babel,csquotes,xpatch}
\usepackage[backend=biber, style=apa]{biblatex}
\addbibresource{references.bib}

%\usepackage[natbibapa]{apacite}
% for hanging-indentation style using apacite
%\setlength{\bibindent}{2.5em}
%\setlength{\bibleftmargin}{0em}
% ==========
% = Floats =
% ==========
\usepackage{float}
% include external pictures
\usepackage{graphicx} %Graphics/figures
% rotate figures/tables
\usepackage{rotating}
% For professional tables
\usepackage{booktabs,threeparttable, multirow}
\usepackage{tabularx}
% For fixing the column widths
\usepackage{array}
\newcolumntype{L}[1]{>{\raggedright\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
\newcolumntype{C}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
\newcolumntype{R}[1]{>{\raggedleft\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}

% ===================
% ==== Tikz Diagrams	==
% ===================
\usepackage{tikz}
\usetikzlibrary{calc,arrows,positioning,shapes,shapes.gates.logic.US,trees, intersections}
% =======================
% === Other useful packages ==
% =======================
\usepackage[T1]{fontenc}
\usepackage{placeins}
%\usepackage{hyperref}
% subcaptions/subfigures %,justification=centered
\usepackage[hypcap=true,width=\textwidth]{subcaption}
% =============
%  == formatting ==
% =============
% \usepackage[margin=1in]{geometry}
% \setlength{\parindent}{0.5in}
\usepackage{setspace}
% \doublespacing

% ==========
% = Syntax =
% ==========
% For Computer Code in Appendix. I set the language for R, so will need to be changed for different languages
\usepackage{listings}
\lstset{
    language=R,
    basicstyle=\small \ttfamily,
    commentstyle=\ttfamily ,
    showspaces=false,
    showstringspaces=false,
    showtabs=false,
    frame=none,
    tabsize=2,
    captionpos=b,
    breaklines=true,
    breakatwhitespace=false,
    title=\lstname,
    aboveskip=10pt,
    belowskip=-10pt,
    %escapeinside={},
    %keywordstyle={},
   % morekeywords={}
    }%
% ====================================
% SAGE Template Settings
\usepackage{moreverb,url}

\usepackage[colorlinks,bookmarksopen,bookmarksnumbered,citecolor=red,urlcolor=red]{hyperref}

\newcommand\BibTeX{{\rmfamily B\kern-.05em \textsc{i\kern-.025em b}\kern-.08em
T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}

\def\volumeyear{2020}

\begin{document}

\runninghead{Padgett and Morgan}

\title{Efficiency among robust estimation methods for multilevel factor analysis with categorical data}

\author{R. Noah Padgett\affilnum{1} and Grant B. Morgan\affilnum{1}}

\affiliation{\affilnum{1}Baylor University}

\corrauth{R. Noah Padgett,
Department of Educational Psychology,
Baylor Univeristy,
One Bear Place \# 97304,
Waco, TX 76798.}

\email{noah\_padgett1@baylor.edu}

\begin{abstract}
Multilevel measurement models are more frequently applied to help answer questions when data arise from hierarchically structured multivariate data.
In this simulation study of multilevel factor models, we evaluated the relative efficiency among three estimation methods: robust maximum likelihood, unweighted least squares, and weighted least squares.
We found that weighted least squares yielded more or equally efficient parameter estimates under all sample size conditions for all model parameters.
The relative efficiency of standard errors was less straightforward where maximum likelihood was more efficient for loadings and residual variances, but weighted least squares was more efficient for the factor covariance matrices.
We give recommendations for estimating multilevel confirmatory factor analysis models and directions for future research.
\end{abstract}

\keywords{Multilevel CFA, categorical data, WLSMV, ULSMV, bias, efficiency}

\maketitle

\section{Introduction}
%% Spacing for Eqations Can't be in preamble...
\setlength{\abovedisplayskip}{3pt}
\setlength{\belowdisplayskip}{3pt}

Rarely do social scientists gain access to data that are truly continuous.
Often collected data are coarse measurements of skill or attitude such as dichotomously scored items or Likert-type responses.
These data are widely gathered due to the relative ease of data collection or due to the nature of the constructs of interest \citep{DeVellis2016, Fowler2013}.
The frequent use of coarse measurements has led to a variety of statistical techniques for utilizing these data in appropriate ways \citep{Bandalos2014, Asparouhov2007, Satorra1994, Muthen1984, Muthen1978}.
These methods have primarily been concerned with estimation in single-level analyses where participants are independent.
However, in many scenarios in social science, participants or observations are usually clustered together in some manner, such as students in a school, players within a team, or workers within a company.
Where knowing which group a participant is from is informative of the outcomes of interest.
This dependence needs to be accounted for in the estimation in order to yield appropriate estimates of model parameters \citep{Stapleton2013}.

One type of model in social science that can directly account for the complex dependencies fall under multilevel confirmatory factor analysis \citep[ML-CFA;][]{Goldstein1988, Muthen1994}.
ML-CFA allows researchers to test hypotheses of why groups of participants vary on constructs of interest.
For example, some hypotheses may relate as to why teachers' perceived school safety climate varies across schools and which school and teacher characteristics relate to these perceptions.
In construct validation with multinational, cross-cultural instruments, ML-CFA can provides researchers with a unique lens to investigate their data with respect to how constructs are potentially influenced by group membership.
The association of group membership and the latent construct can be investigating with ML-CFA when the factor structure is hypothesized to be invariant across levels \citep[][Models 4-5, pg. 488]{Stapleton2016}.
The models are specified such that group level constructs are aggregates of individual level construct.
For example, in the cross-cultural measurement of resilience, researchers were interested in the contextual effects on responses across communities (REMOVED FOR PEER REVIEW).% \citep{Renbarger2020}.
The study investigated whether variance in responses between communities was more than within community variance leading to evidence of potential cultural differences in understanding what resilience means.
However, the hierarchical organization and coarse measurements associated with data collected to answer such questions may be challenging in the model estimation.

Current recommendations for model estimation for ML-CFA with ordinal indicates are to use a robust weighted least squares (WLS) approach\citep{Hox2010, Hsu2009, Asparouhov2007}.
Some evidence suggest that the use of WLSM/WLSMV in M\textit{plus} provides reasonably accurate parameter estimates and standard errors when the number of groups is at least 100 \citep{Hsu2009}.
However, \cite{Hox2010} found that 50 groups may be sufficient for accurate level-2 factor loading estimates using WLSM(V), but higher sample sizes may be needed for level-2 variance components.
With dichotomous indicators, \cite{Depaoli2015} found that WLSM resulted in severely biased estimates of level-2 loadings and variance components with a non-invariant structure across levels when the number of groups was less than 100.
They found that a cross-level invariance constraint on factor loadings helped with convergence and reduced bias.
\cite{Depaoli2015} also noted that additional item responses may help convergence issues similar to the improvements \cite{Yang2010} showed in the single-level case.

In single-level factor analysis, a vast literature on estimation performance exists and is growing \citep{Li2016, Bandalos2014, DiStefano2014,  Yang2010, Forero2009, Flora2004, Dolan1994, Babakus1987}.
The use of weighted asymptotic covariance appears to also be the consensus under most conditions of ordered categorical data.
The use of other estimation methods such as unweighted least squares has shown promise for better standard error accuracy in small sample sizes \citep{Forero2009, Paek2018}.
Some evidence even exists that under optimal conditions of approximately symmetric responses, five categories may be suitable for treating responses as continuous and using maximum likelihood with robust standard errors \citep[MLR; ][]{Rhemtulla2012}.
However, in multilevel settings, we have not found a study yet that investigates these alternative estimation methods.

An advantage to these alternative approaches that the asymptotic covariance matrix does not need to be estimated.
In the MLR approach by treating the observed data as continuous, we can rely maximum likelihood yielding asymptotically unbiased estimators in large samples.
While in the unweighted least squares approach, we can potentially gain more efficient estimators over weighted least squares because the weight matrix (i.e., the identity matrix) is known.
This implies that all imprecision associated with estimating the asymptotic covariance matrix in weighted least squares is side-stepped.
This might especially be the case as the sample size decreases.
These potential advantages of maximum likelihood and unweighted least squares are a major focus of our work.

Our current study was to investigate the extent to which different estimation methods would result in different conclusions about parameter estimates and group differences on the latent variable.
First, we have describe ML-CFA models in more detail focusing on the aspects of interpretation that users of ML-CFA are likely interested in.
Second, specific research questions and hypothesizes are described.
Third, we describe our Monte Carlo simulation including conditions and how results were compiled.
We present our results concisely with appropriate references to where more details can be obtained.
Last, we discuss how our simulation results fit into the broader research on multilevel factor models and give concrete recommendations for using ML-CFA.

\subsection{Multilevel Confirmatory Factor Analysis}

As similarly described in (REMOVED FOR PEER REVIEW), ML-CFA is a decomposition of the observed covariance matrix into specific covariance matrices for level-1 (pooled within group) and level-2 (between group).
Because two covariance matrices are essentially being analyzed, two unique models can be specified for each level of analysis.
The two levels of analysis correspond to the individual and group levels.
With categorical observed data, these two sources of variability directly influence the underlying response value for each item, as shown in Equation \ref{eq:ur-mcfa}.
The underlying response value for an individual ($y^{*}_{pig}$) is the composition of the individual component ($y_{wpig}$) and a random effect of group ($y_{bpg}$).
As with \cite{Muthen1984} and \cite{Asparouhov2007}, the latent response is assumed normally distributed.
\begin{equation}\label{eq:ur-mcfa}
y^{*}_{pig} = y_{bpg} + y_{wpig}
\end{equation}
where $y^{*}_{pig}$ is the latent response underlying the observed categorical value $y_p$ for the $p^{th}$ item, $i$ indexes across individuals within group $g$.
Across $p$ items, this general framework is easily incorporated into factor analysis by replacing the latent components $y_{wpig}$ and $y_{bpg}$ by the level specific factor loading matrices and latent variable vectors.
The extension of factor analytic notation is expressed as:
\begin{equation} \label{eq:mcfa}
\mathbf{y^{*}_{ig}}  = \nu_g + \mathbf{\Lambda}_{B} \eta_{Bg} +\varepsilon_{Bg} + \mathbf{\Lambda}_{W} \eta_{Wig} +\varepsilon_{Wig}
\end{equation}
% ~~
where $\mathbf{y^{*}_{ig}}$ is the latent response vector for the $i^{th}$ individual in the $g^{th}$ group, $\nu_g $ are the group latent intercepts  (which are not directly estimated), $\mathbf{\Lambda}_{B}$ is the matrix of factor loadings for the level-2 (between) model, $\eta_{Bg}$ is the vector of group latent variable scores, $\varepsilon_{Bg}$ is the vector of group level residuals, $\mathbf{\Lambda}_{W}$ is the matrix of factor loadings for the level-1 (within) model, $\eta_{Wig}$ is the vector of individual latent variable scores, and $\varepsilon_{Wig}$ is the vector of individual level residuals for individual $i$ in group $g$.

In models where the observed scores are categorical, as in this case, the item thresholds are used to relate the underlying response values ($\mathbf{y^{*}_{ig}}$) to the observed vector of categorical responses ($\mathbf{y}_{ig}$).
These thresholds ($\tau_{pk}$) are specific to each item ($p$) and are constant across groups.
The total number of categories in the observed variable is defined as $k = 2, 3, \cdots, K$;
$y_{pig} = k \Leftrightarrow \tau_{pk} < y^{*}_{pig} < \tau_{p(k+1)}$ where $\tau_{p0} = -\infty$ and $\tau_{pk+1} = \infty$.
The relationship between the observed categories and underlying response vector allows for the categorical nature of the observed data to be controlled for when estimating the between-group variability.

The relative amount of between-group variability is expressed in two components of the model: 1) the amount of variability in each item that attributed to group differences, and 2) the amount of variability in the latent variable that is attributed to group differences.
Both quantities can be estimated using an intraclass correlation (ICC).
At the basic level, an ICC is the proportion of variance in the outcome that is attributable to group membership \citep[][, pg. 24]{Raudenbush2002}.
In ML-CFA with invariant factor structure and constrained loadings across levels, the ICC represents the proportion of variability in the latent construct between groups \citep[][, pg. 155-171]{Heck2015}.
This may provide key inferential outputs in applications of ML-CFA.

Factor loadings may also be of high importance to those using ML-CFA.
In models where the factor loading matrix is held invariant across levels, the model in Equation \ref{eq:mcfa} is reduced.
By estimating half as many factor loadings, researchers may reduce the computational burden leading to increased convergence \citep{Depaoli2015}.
Additionally, in settings where construct validity is under investigation, loadings are likely a key concern.

\subsection{Research Questions and Hypotheses}

Few studies have investigated estimation performance of ML-CFA models under conditions when data are ordered categorical.
An understanding how these models can be recovered can be especially helpful in educational contexts given that data from educational settings commonly arise from hierarchically structured contexts and used ordered categorical response formats.
The potential performance of the various estimation methods (maximum likelihood, unwieghted least squares, and weighted least squares), our research questions are
\begin{enumerate}
\item Which estimation method provides the least biased estimates, on average, of model parameters?
\item How does relative efficiency of parameter and standard error estimates compare across estimation methods as sample size decreases?
\end{enumerate}
We expect that weighted least squares (WLSMV in M\textit{plus}) will yield the least biased estimates and standard error estimates across all parameters, because WLSMV has been shown to perform best for categorical data in a variety of scenarios \citep{DiStefano2014, Bandalos2014, Hox2010, Hsu2009, Asparouhov2007}.
However, given that the weight matrix may be estimated imprecisely, especially at lower sample sizes, we expect that weighted least squares is less efficient than unweighted least squares where the weight matrix is known.
Under maximum likelihood, where data were treated as continuous, we expect that the estimates of the correlation among variables are attenuated so the factor loadings will likely be under estimated.
That said, we do not expect that the other model parameters (e.g., factor variances) will be biased.
%%~~
\section{Methods}

\subsection{Description of the Simulation Study}
A Monte Carlo simulation study was conducted to compare performance of estimation methods for ML-CFA with categorical indicators.
Data for this project were a subset of the data used in (REMOVED FOR PEER REVIEW), and were generated using M\textit{plus} v.8.2 \citep{Muthen2017} utilizing the MONTECARLO command along with MplusAutomation \citep{Hallquist2018} package in R \citep{R2018}. %\cite{Padgett2020}
Data were generated from a factor model with 10 items and two correlated factors at both levels.
The factor loading matrices were specified to be invariant across levels.
All non-zero factor loadings were set to 0.60, which corresponds to the average of reported factor loadings in empirical research \citep{DiStefano2005}.
At level-1, factors were generated with a fixed unit variance and correlation of 0.3 ($\Phi_W = [1, 0.3; 0.3, 1]$).
At level-2, the factor (co)variance matrix ($\Phi_B$) varied by condition of latent intraclass correlation (see \textit{Intraclass Correlation} section), but the factor correlation was constant at 0.3 at both levels across all conditions.
The five ordered categorical indicators were generated by varying thresholds across conditions so that the observed item distribution was held constant.
For this study, the indicators were set to be approximately normally distributed which is the best case possible for using ordered categorical data.

\subsection{Intraclass Correlation}
Previous literature on ML-CFA has varied the ICC (i.e., group effect) to study the effects on parameter recovery by focusing on either changes in the group effect on the latent factor \citep{Hox2001, Wu2012} or by changes in the group effect on the observed items \citep{Hsu2015}.
To vary the group effect on the latent factor, the construct must be invariant (as we specfied here).
We varied both in our design, that is, we varied the observed ICC ($\rm ICC_O$) and latent ICC ($\rm ICC_L$).
In educational settings, the average ICC for observed indicators was 0.15 to 0.25 \citep{Hedges2007}.
We varied our simulation conditions around the average and the reported ICC conditions in similar Monte Carlo simulation studies \citep{Navruz2016, Hsu2015, Ryu2009}.

First, the group effect on the observed indicators was varied across three levels (0.1, 0.3, and 0.5).
The 0.1 and 0.3 conditions were chosen to be on the extremes of the average interval reported in \cite{Hedges2007}.
The very large ICC of 0.5 was included to explore the effects of an extreme group effect.

Secondly, the group effect on the latent variables was varied between two levels (0.1 and 0.5).
The exclusion of the middle level was to limit the size of the simulation, but to still test whether an effect of $\rm ICC_L$ could be identified between these two extremes.
If we do identify an effect of the $\rm ICC_L$ then future investigation can explore this feature in more detail.

\subsection{Sample Size}

Sample size in ML-CFA is major consideration for estimation.
The number of groups is known to be one of the most influential components to estimation of multilevel models, where under ML estimation, the $\rm N_2 \geq 100$ for accurate parameter estimation \citep{Hox2001}.
For this investigation, we focused on testing this bound where we included group sizes 30, 50, 100, and 200.
Therefore, we expect that that these incredibly low number of groups conditions to perform poorly.
Although, a small number of groups may be more realistic conditions for substantive researchers collecting their own data in cross cultural settings where data collection can be difficult (REMOVED FOR PEER REVIEW).%\citep{Renbarger2020}.
Or, if measurement invariance across groups, 30-50 groups may provide a sensible number of groups to apply ML-CFA instead of model each group separately \citep{Kim2017}.

Secondly, the  number of units sampled within each group ($\rm N_1$) was varied among 5, 10, and 30.
We held $\rm N_1$ constant across groups for simplicity but acknowledge that in applied settings $\rm N_1$ would most likely vary slightly across groups.

\subsection{Analysis of Monte Carlo Simulation}
In Monte Carlo simulation studies the rates of admissible or usable replications should be checked prior to examination of results \citep{Bandalos2012, Bandalos2013}.
Our working definition of usable replications is as follows.
First, we identified converged solutions where model estimation terminated.
Secondly, we checked for usable solutions by whether the solution included all non-negative variances (i.e., absence of negative variances).
This method of checking for usable replications is similar to previous simulation studies \citep{Flora2004, Yang2010, DiStefano2014}.
The non-converged and inadmissible replications were removed from further analyses because they do not provide useful information \citep{Bandalos2012, Bandalos2013}.

\subsubsection{Evaluating Results}

Parameter and standard error estimates were evaluated using four pieces of information.
First, for a parameter ($\theta$), we were interested in evaluating the degree to which the estimated values recaptured the true (simulating) value.
We evaluated the difference between $\hat{\theta}$ and $\theta$ using the average relative bias (ARB),
\begin{equation}
\mathrm{ARB}(\hat{\theta}) = \sum_{j=1}^{n_r}\left(\frac{\hat{\theta}_j- \theta}{\theta}\right)/n_r\times 100
\end{equation}
where, $n_r$ is the number of usable replications.
ARB was computed for parameter and standard error estimates of factor loadings, factor covariances, level-2 factor variances, level-2 residual variances, and ICC estimates.
For ICC estimates (latent and observed), we only investigated the ARB of the estimates compared to the true value.
We evaluated the extend of RB as negligible for RB $< | 5\% |$, as mild for $\mid 5\%\mid \leq$ RB $< \mid 10\%\mid$, and unacceptable for RB $> \mid 10\%\mid$ \citep{Hoogland1998, Muthen1985}.

Secondly, we used a factorial analysis of variance (ANOVA) on the estimates of relative bias of parameters and standard errors to evaluate the magnitude of effect from our simulation design \citep{Bandalos2013}.
In total, each ANOVA included 15 terms from the main effect of $\rm N_1$, $\rm N_2$, $\rm ICC_{O}$, $\rm ICC_{L}$, and estimation method and all  bivariate interactions of main effects.
The overall influence of the interactions and main effects were assessed with partial-$\omega^2$ \citep[][, p. 296]{Maxwell2004}.
An effect was determined as practically significant if estimates of the partial-$\omega^2 \geq 0.14$  aligning with the ``large'' effect from \citep{Cohen1988}.

Third, we computed the root mean square error (RMSE).
RMSE is a measure of the variability in $\hat{\theta}$,
\begin{equation}\label{eq:rmse}
RMSE(\hat\theta) = \sum_{j=1}^{n_r}\frac{(\hat\theta_j -\theta)^2}{n_r} = {\left(\bar{\hat\theta} - \theta\right)}^2 +  \sum_{j=1}^{n_r}\frac{(\hat\theta_j -\bar{\hat\theta})^2}{n_r}
\end{equation}
where the partition of RMSE $> 0.05$ were treated as significant variability in estimates across replications.
We also decomposed RMSE into squared bias and sampling variance \citep{Harwell2018}.
Both pieces are shown in the right hand side of Equation \ref{eq:rmse} and were use to help interpret RMSE.

Lastly, we evaluated the relative efficiency (RE) parameter estimates across estimation methods.
The RE of the estimates from different estimation methods was estimated using Equation \ref{eq:re}.
\begin{equation}\label{eq:re}
RE = \sqrt{\left( \frac{\sum_{\forall j}\left(\hat{\theta}_j^{MLR} -\theta\right)^2 }{\sum_{\forall j}\left(\hat{\theta}_j^{WLSMV} -\theta\right)^2 }\right)}
\end{equation}
Therefore, the RE is the ratio of sum of squares between two estimation methods.
The smaller the sum of squares, the more efficient the estimation methods is at estimating that parameter or standard error.
The ratio is computed three times: MLR over WLSMV, MLR over ULSMV, and ULSMV over WLSMV.
For the analysis of standard errors, $\hat{\theta}$ is the estimated standard error and $\theta$ is the empirical standard error.
Empirical standard error was computed as the standard deviation of the parameter estimates for the specific condition and estimation method.
The RE was broken down across sample size conditions (level-1 and level-2 $N$) to describe which estimation method yielded more efficient estimates as sample size decreased.

\section{Results}

\subsection{Convergence Rates}

The results for each model estimated and information extracted for this simulation study are openly available online (REMOVED FOR PEER REVIEW). %\citep{Padgett2019a}.
Convergence was a problem in this study under all estimation methods (see Table \ref{tb:usable}).
We found that convergence was worst when the level-2 sample size was only 30 where additional level-1 units increased convergence rates.
A results similar to previous simulation studies with ML-CFA \citep{Depaoli2015}.

<< TABLE \ref{tb:usable} ABOUT HERE >>

% latex table generated in R 4.0.0 by xtable 1.8-4 package
% Fri Jun 05 10:29:42 2020
\begin{table}[ht]
 \centering
 \begin{threeparttable}
 \caption{Convergence rates indicate problems for all estimation methods when total sample size $\varleq 500$}
 \label{tb:usable}
\begin{tabular}{llrrrr}
  \toprule
$N_2$ & $N_1$ & $N_T$ & MLR & ULSMV & WLSMV \\
  \midrule
   30 &    5 & 150  & 0.575 & 0.401 & 0.355 \\
   30 &   10 & 300  & 0.685 & 0.573 & 0.508 \\
   30 &   30 & 900  & 0.838 & 0.762 & 0.688 \\
   50 &    5 & 250  & 0.677 & 0.532 & 0.470 \\
   50 &   10 & 500  & 0.796 & 0.707 & 0.660 \\
   50 &   30 & 1500 & 0.902 & 0.880 & 0.820 \\
  100 &    5 & 500  & 0.806 & 0.692 & 0.665 \\
  100 &   10 & 1000 & 0.920 & 0.873 & 0.824 \\
  100 &   30 & 3000 & 0.946 & 0.945 & 0.911 \\
  200 &    5 & 1000 & 0.923 & 0.875 & 0.849 \\
  200 &   10 & 2000 & 0.963 & 0.958 & 0.936 \\
  200 &   30 & 6000 & 0.975 & 0.974 & 0.973 \\
   \bottomrule
\end{tabular}
 \vspace*{1mm}
 	\begin{tablenotes}[para,flushleft]
    {\small
        \textit{Note.} Each cell in contains the proportion of usable replications for the corresponding design factors. MLR = maximum likelihood with robust standard errors; ULSMV = unweighted least squares mean and variance adjusted; WLSMV = weighted least squares mean and variance adjusted; $N_1$ = number of level-1 units within cluster; $N_2$ = number of level-2 units; and $N_T$ is the total sample size.
    }
 	\end{tablenotes}
 \end{threeparttable}
\end{table}

The proportion of usable replications varied widely across conditions and estimations (see Table \ref{tb:usable}).
All three estimation methods showed problems of usable replications when sample size was low.
In the extremely small sample size conditions ($N_2=30, N_1=5$), the rates of admissible cases were below 60\% for estimation methods and the worst for WLSMV .
For the observed variable ICC conditions ($\rm ICC_{O}=0.1$) when sample size was low, the rates of usable cases was lower than 5\% for ULSMV and WLSMV.
ICC size did not pose an issue on convergence at larger sample sizes conditions.
The estimation issues we observed were similar to previous simulation studies with categorical data using robust estimation method \citep{DiStefano2014, Hsu2015, Depaoli2015}.

\subsection{Parameter Estimate Relative Bias}

(REDUCE TO ONE SECTION AND GIVE BROAD OVERVIEW OF RESULTS)

The average relative bias of parameters and intraclass correlation coefficients (latent and observed variable) across simulation conditions are shown in Figure \ref{fig:rb-para}.
Each point shown in Figure \ref{fig:rb-para} is the average relative bias for a single condition.
The use of robust maximum likelihood (e.g., MLR) showed clear negative bias for factor loadings and level-2 residual variances.
The pattern of bias for the observed variable ICC (${\rm ICC_{O}}$) looks similar to that of the level-2 residual variance, but to not the same degree of severity.
Any additional patterns of bias are not as clear from Figure \ref{fig:rb-para}.

<<FIGURE \ref{fig:rb} ABOUT HERE>>
\begin{figure}[!htp]
\centering
\begin{subfigure}[b]{1\textwidth}
\centering
\includegraphics[height=5cm]{fig/relative_bias_overview}
\caption{Relative bias of parameter estimates showed distinct differences among estimation methods}
\label{fig:rb-para}
\end{subfigure}

\begin{subfigure}[b]{1\textwidth}
\centering
\includegraphics[height=5cm]{fig/relative_se_bias_overview}
\caption{Standard errors were biased under most low sample size conditions}
\label{fig:rb-se}
\end{subfigure}
\caption{Bias associated with parameter and standard error estimates for each condition. {(a) Parameter estimates including intraclass correlation estimates; (b) Standard error estimates. Dashed (red) lines were placed at the $\pm10$\% relative bias as the boundary for unacceptable levels of bias.}}
\label{fig:rb}
\end{figure}

The estimation method design factor significantly influenced the distribution of parameter estimates for factor loadings (partial-$\omega^2 = .73$) and level-2 residual variances (partial-$\omega^2 = .91$).
For factor loadings, we found that the bias was negligible when using ULSMV or WLSMV.
Although, ULSMV was more prone to slightly underestimate factor loadings while WLSMV tended to slightly overestimate factor loadings.
The large effect of the assuming indicators are continuous and using MLR, the loadings were severely underestimated in all conditions (see Table \ref{tb:bias-fct}).
Underestimating factor loadings when ignoring the categorical nature of these data has been previously observed \citep{DiStefano2002, Li2016}.
The pattern of bias for the level-2 residual variances was the same as the factor loadings.
All other design factors did not significantly influence the bias of parameter estimate.
For more details about the patterns of bias across design factors, the interested reader can visit our supplemental website (REMOVED FOR PEER REVIEW). %\citep{Padgett2019a}.

<<TABLE \ref{tb:bias-fct} ABOUT HERE>>
\begin{table}[!htp]
\centering
\begin{threeparttable}
\caption{Relative bias of factor loadings across estimation methods and sample sizes}
\label{tb:bias-fct}
\begin{tabular}{llrrrrrrrrr}
  \toprule
  &   & \multicolumn{3}{c}{MLR} & \multicolumn{3}{c}{ULSMV} & \multicolumn{3}{c}{WLSMV}\\ \cmidrule(lr){3-5} \cmidrule(lr){6-8} \cmidrule(lr){9-11}
$\rm N_2$ & $\rm N_1$ & RB & RMSE & SV & RB & RMSE & SV & RB & RMSE & SV \\
  \midrule
\multicolumn{11}{l}{Parameter Estimates}\\
30 & 5 & $\bf -36.27$ & $\bf 0.06$ & 0.01 & $-$2.60 & $\bf0.14$ & 0.14 & 4.32 & 0.04 & 0.04 \\
   & 10 & $\bf -35.85$ & 0.05 & 0.00 & $-$1.56 & 0.04 & 0.04 & 1.73 & 0.02 & 0.02 \\
   & 30 & $\bf -34.52$ & 0.05 & 0.00 & $-$2.18 & 0.03 & 0.03 & 0.49 & 0.00 & 0.00 \\
  50 & 5 & $\bf -36.03$ & 0.05 & 0.00 & $-$2.01 & 0.05 & 0.05 & 2.44 & 0.02 & 0.02 \\
   & 10 & $\bf -34.95$ & 0.05 & 0.00 & $-$2.38 & 0.03 & 0.03 & 0.98 & 0.01 & 0.01 \\
   & 30 & $\bf -34.43$ & 0.05 & 0.00 & $-$1.24 & 0.02 & 0.02 & 0.29 & 0.00 & 0.00 \\
  100 & 5 & $\bf -35.14$ & 0.05 & 0.00 & $-$0.77 & 0.02 & 0.02 & 1.20 & 0.01 & 0.01 \\
   & 10 & $\bf -34.48$ & 0.05 & 0.00 & $-$1.23 & 0.02 & 0.02 & 0.60 & 0.00 & 0.00 \\
   & 30 & $\bf -34.39$ & 0.05 & 0.00 & $-$1.21 & 0.01 & 0.01 & 0.17 & 0.00 & 0.00 \\
  200 & 5 & $\bf -34.68$ & 0.05 & 0.00 & $-$0.59 & 0.01 & 0.01 & 0.46 & 0.00 & 0.00 \\
   & 10 & $\bf -34.47$ & 0.05 & 0.00 & $-$0.44 & 0.01 & 0.01 & 0.29 & 0.00 & 0.00 \\
   & 30 & $\bf -34.49$ & 0.05 & 0.00 & $-$0.98 & 0.01 & 0.01 & 0.12 & 0.00 & 0.00 \\
\multicolumn{11}{l}{Standard Error Estimates}\\
  30 & 5 & 3.13 & 0.00 & 0.00 & $-$7.46 & $\bf146.72$ & 146.55 & $\bf-19.21$ & 0.01 & 0.01 \\
   & 10 & $-$0.65 & 0.00 & 0.00 & $\bf-16.94$ & 0.01 & 0.01 & $\bf-13.44$ & 0.00 & 0.00 \\
   & 30 & $-$1.40 & 0.00 & 0.00 & 5.97 & 0.03 & 0.03 & 9.36 & 0.00 & 0.00 \\
  50 & 5 & 0.93 & 0.00 & 0.00 & $\bf-22.55$ & 0.01 & 0.01 & $\bf-12.60$ & 0.00 & 0.00 \\
   & 10 & $-$0.54 & 0.00 & 0.00 & $\bf-22.86$ & 0.01 & 0.00 & $-$8.72 & 0.00 & 0.00 \\
   & 30 & $-$1.32 & 0.00 & 0.00 & $-$1.29 & $\bf0.09$ & 0.09 & 4.64 & 0.00 & 0.00 \\
  100 & 5 & 0.62 & 0.00 & 0.00 & $\bf-17.71$ & 0.01 & 0.00 & $-$6.74 & 0.00 & 0.00 \\
   & 10 & $-$0.11 & 0.00 & 0.00 & $\bf-13.25$ & 0.00 & 0.00 & $-$4.50 & 0.00 & 0.00 \\
   & 30 & $-$0.20 & 0.00 & 0.00 & $-$8.46 & 0.00 & 0.00 & 2.42 & 0.00 & 0.00 \\
  200 & 5 & 0.26 & 0.00 & 0.00 & $\bf-12.48$ & 0.00 & 0.00 & $-$3.06 & 0.00 & 0.00 \\
   & 10 & 0.32 & 0.00 & 0.00 & $\bf-10.95$ & 0.00 & 0.00 & $-$2.04 & 0.00 & 0.00 \\
   & 30 & 0.09 & 0.00 & 0.00 & $-$9.37 & 0.00 & 0.00 & 0.71 & 0.00 & 0.00 \\
   \bottomrule
\end{tabular}
 \vspace*{1mm}
 	\begin{tablenotes}[para, flushleft]
    {\small
        \textit{Note.} RB  $> |10\%|$ and RMSE $> 0.05$ are in bold and considered practically significance bias and variance, respectively. If the SV $\approx$ RMSE then RMSE estimate reflects variance of estimates, if SV $\not\approx$ RMSEA then RMSE does not reflect variance. $\rm N_{2}$ = number of clusters; $\rm N_1$ = sample size within cluster; MLR = maximum likelihood with robust standard errors; ULSMV = unweighted least squares mean and variance adjusted; WLSMV = weighted least squares mean and variance adjusted; RB = relative bias; RMSE = root mean square error; and SV = sampling variance of estimate.
    }
 	\end{tablenotes}
 \end{threeparttable}
\end{table}

\subsection{Standard Error Relative Bias}

The average relative bias of standard errors across simulation conditions are shown in Figure \ref{fig:rb-se}.
The only pattern that automatically stands out is that, under unweighted least squares (ULSMV), the standard errors for factor loadings and level-1 covariance were negatively biased across many conditions.
The level-2 model parameter standard errors were positively biased under many conditions when the number of level-2 units was low (30 or 50).
Over the next few sections, we explored the less clear patterns of bias for parameter and standard error.

The estimation method design factor significantly influenced the distribution of standard error estimates for factor loadings (partial-$\omega^2 = .21$), level-1 factor covariance (partial-$\omega^2 = .20$), and level-2 residual variances (partial-$\omega^2 = .23$).
For factor loadings, we found that the bias was negligible when using MLR but significant when using ULSMV or WLSMV depending on sample size.
A large cluster effect mixed with a sample within cluster sample size lead to estimation with ULSMV to underestimate standard errors.
However, when using WLSMV, underestimation of standard errors only occurred when the sample size was very small.
An extreme RMSE estimate was found for ULSMV when the level-1 sample size was 5 and level-2 sample size was only 30 which means that the sample size is likely too small to efficiently estimate the standard errors.
The remaining parameters should a similar pattern of bias in standard errors under small sample sizes and large cluster effect.
For more details about the patterns of bias across design factors, the interested reader can visit our supplemental website (REMOVED FOR PEER REVIEW). %\citep{Padgett2019a}.

\subsection{Relative Efficiency}
The relative efficiency of parameter and standard error estimates among estimation methods are reported in Table \ref{tb:re}.
For parameter estimates, we found that WLSMV was generally the most efficient for all parameters among estimation methods included in this study.
The only exception was for the level-2 residual variances in which ULSMV was slightly more efficient when the sample size was very large ($\rm N_2 \geq 100$ and $\rm N_1 \geq 10$).
However, for estimation of the level-2 factor covariance matrix, ULSMV was only marginally less efficient across all sample sizes.
In generally, these results for the efficiency among estimation methods points to utilizing WLSMV as a first option and utilizing ULSMV as a potential alternative.

For standard error estimates, the story about the relationship among bias, sample size, and estimation method is a bit more complicated.
We found that the maximum likelihood method, while treating five ordered categories as continuous, resulted in the most efficient standard errors in most conditions for all parameters.
The standard errors for the elements of the level-2 factor covariance matrix were approximately equally efficient across all estimation methods.
Exceptions were under the smallest sample size condition.
Estimation of standard errors appears to be highly variable among estimation methods at lower sample sizes.
% latex table generated in R 3.6.3 by xtable 1.8-4 package
% Sun Jun 21 22:54:43 2020
\begin{table}[!htp]
 \centering
 \small
 \begin{threeparttable}
\caption{Parameter estimates using WLSMV were more efficient under most sample size conditions}
\label{tb:re}
\begin{tabular}{llrrrrrrrrrrrr}
  \toprule
  &     & \multicolumn{3}{c}{Loadings} & \multicolumn{3}{c}{L1 Factor Cov.} & \multicolumn{3}{c}{L2 Factor Cov.} & \multicolumn{3}{c}{L2 Residual Var.}\\  \cmidrule(lr){3-5} \cmidrule(lr){6-8} \cmidrule(lr){9-11} \cmidrule(lr){12-14}
$\rm N_2$ & $\rm N_1$ & MU & MW & UW & MU & MW & UW & MU & MW & UW & MU & MW & UW \\
  \midrule
\multicolumn{14}{l}{Parameter Estimates}\\
30 & 5 & 1.84 & 2.18 & 1.57 & 1.81 & 1.83 & 1.13 & 1.83 & 2.22 & 1.50 & 1.41 & 1.42 & 1.50 \\
  30 & 10 & 1.68 & 2.33 & 1.54 & 1.08 & 1.28 & 1.19 & 1.10 & 1.42 & 1.30 & 1.43 & 1.52 & 1.05 \\
  30 & 30 & 2.27 & 3.75 & 2.04 & 0.92 & 1.16 & 1.31 & 1.03 & 1.25 & 1.20 & 1.56 & 1.64 & 1.06 \\
  50 & 5 & 1.81 & 2.37 & 1.44 & 1.15 & 1.33 & 1.20 & 1.31 & 1.64 & 1.26 & 1.50 & 1.58 & 1.05 \\
  50 & 10 & 1.78 & 2.81 & 1.79 & 0.97 & 1.15 & 1.21 & 1.05 & 1.25 & 1.18 & 1.64 & 1.66 & 1.00 \\
  50 & 30 & 2.84 & 4.27 & 1.95 & 0.94 & 1.04 & 1.13 & 0.99 & 1.10 & 1.10 & 1.88 & 1.89 & 1.03 \\
  100 & 5 & 2.02 & 2.71 & 1.45 & 1.04 & 1.15 & 1.14 & 1.10 & 1.20 & 1.09 & 1.82 & 1.85 & 1.01 \\
  100 & 10 & 2.79 & 3.64 & 1.63 & 0.96 & 1.07 & 1.14 & 1.01 & 1.11 & 1.09 & 2.18 & 2.17 & 0.99 \\
  100 & 30 & 4.03 & 5.90 & 2.17 & 0.86 & 1.01 & 1.31 & 0.99 & 1.03 & 1.05 & 2.63 & 2.55 & 0.95 \\
  200 & 5 & 2.91 & 3.53 & 1.40 & 0.97 & 1.05 & 1.08 & 1.01 & 1.07 & 1.05 & 2.40 & 2.37 & 0.98 \\
  200 & 10 & 3.86 & 4.86 & 1.54 & 0.96 & 1.01 & 1.07 & 0.99 & 1.03 & 1.04 & 3.02 & 2.95 & 0.96 \\
  200 & 30 & 5.92 & 8.24 & 2.39 & 0.89 & 1.00 & 1.27 & 0.99 & 1.01 & 1.02 & 3.77 & 3.60 & 0.94 \\
\multicolumn{14}{l}{Standard Errors}\\
30 & 5 & 0.78 & 0.89 & 41.18 & 1.34 & 1.26 & 1.00 & 2.55 & 3.17 & 1.83 & 0.34 & 0.34 & 106.74 \\
  30 & 10 & 0.29 & 0.45 & 2.76 & 0.60 & 0.73 & 1.42 & 1.06 & 1.47 & 1.41 & 0.22 & 0.24 & 1.39 \\
  30 & 30 & 0.20 & 0.39 & 3.33 & 0.63 & 0.81 & 1.54 & 0.84 & 1.21 & 1.55 & 0.23 & 0.24 & 1.78 \\
  50 & 5 & 0.34 & 0.52 & 2.50 & 0.80 & 0.84 & 1.39 & 1.35 & 1.78 & 1.32 & 0.23 & 0.26 & 1.41 \\
  50 & 10 & 0.25 & 0.48 & 4.43 & 0.69 & 0.82 & 1.70 & 1.09 & 1.32 & 1.20 & 0.25 & 0.26 & 1.14 \\
  50 & 30 & 0.23 & 0.37 & 4.24 & 0.80 & 0.86 & 1.23 & 0.92 & 1.12 & 1.25 & 0.27 & 0.26 & 4.20 \\
  100 & 5 & 0.28 & 0.44 & 3.58 & 1.04 & 1.14 & 1.59 & 1.21 & 1.29 & 1.08 & 0.23 & 0.24 & 1.55 \\
  100 & 10 & 0.32 & 0.47 & 5.01 & 0.84 & 0.90 & 1.75 & 1.07 & 1.19 & 1.11 & 0.29 & 0.28 & 1.00 \\
  100 & 30 & 0.27 & 0.40 & 4.56 & 0.62 & 0.82 & 3.29 & 0.96 & 1.05 & 1.10 & 0.31 & 0.29 & 0.98 \\
  200 & 5 & 0.33 & 0.42 & 4.23 & 0.71 & 0.86 & 1.76 & 1.08 & 1.13 & 1.06 & 0.25 & 0.25 & 1.02 \\
  200 & 10 & 0.35 & 0.48 & 6.13 & 1.01 & 1.14 & 1.87 & 1.03 & 1.08 & 1.05 & 0.29 & 0.28 & 1.00 \\
  200 & 30 & 0.31 & 0.45 & 8.76 & 0.75 & 0.90 & 4.39 & 0.97 & 1.02 & 1.06 & 0.32 & 0.30 & 0.96 \\
   \bottomrule
\end{tabular}
 \vspace*{1mm}
 	\begin{tablenotes}[para, flushleft]
    {\small
        \textit{Note.} MU = relative efficiency of MLR compared to ULSMV; MW = relative efficiency of MLR compared to MLSMV; and UW = relative efficiency of ULSMV compared to MLSMV.
    }
 	\end{tablenotes}
 \end{threeparttable}
\end{table}

\section{Discussion}

Nested data structures are abundant in educational research, and current software makes performing investigations of multilevel structures more widespread than ever before.
Due to the growing use of these analyses, the burden lies on researchers to make decisions about how to connect theory and data.
The benefits of these methods to a variety of disciplines in education, social sciences, and many other fields only come when these methods are used appropriately alongside a guiding theoretical framework.
We investigated how well multilevel confirmatory factor analyses recover a known ``true model'' under a variety of potentially realistic conditions.
We aimed to find evidence for which estimation method(s) should be used and whether different estimation methods provided similar estimates for the same dataset.

Our results indicated that the performance of the three estimation methods considered (MLR, ULSMV, and WLSMV) depended on which parameter was of interest and sample size.
The model parameters were biased in different ways for different estimation methods, similar to the results of \citep{DiStefano2014} for single level CFA models.
An unexpected finding was that MLR, where the 5 ordered categorical data were treated as continuous, resulted in the most consistently biased across all sample sizes and parameters.
In general, the parameter estimated under MLR were negatively biased except when ICCs were low.

Convergence and admissible solutions was surprisingly problematic in this study.
Numerous cells of this design resulted in low or near zero usable replications.
This is partially contrary to what other authors have found \citep{Hsu2009, Navruz2016}.
When studying ML-CFA with dichotomous indicators, \cite[pg. 59]{Hsu2009} found a convergence not to be a problem  (e.g., only 0.23\% to 0.34\%) across all study design factors.
However, \cite{Hsu2009} used the weighted least squares mean adjusted (WLSM) estimation method in M\textit{plus} v.5, so differences between convergence could be due to the use of different estimation methods and software versions.
The differences between \cite{Navruz2016} and this study are minimal but the results reported are not fine grained enough for me to make many implications besides that we seemed to have lower rates of admissible solutions compared to \cite{Navruz2016}.

When researchers are planning studies that utilize ML-CFA models, one consideration for data collection that will be to carefully consider the sample size in terms of number of groups and number of participants within groups.
We found that there is a trade-off between number of groups versus within group sample size.
Which means that researcher may be able to successfully recover model parameters, even level-2 parameters, when the number of participants within group is increased.
For inferences about variances and covariances, when researchers are constrained by the number of groups they have access to (e.g., less than 50) increasing the within group sample size to as many as possible ($30+$) may help to more accurately estimates one's model.
However, gaining access to more groups would result in greater gains in accuracy.

Accuracy was closely related to the estimation method chosen, which also seems to depend, at least partially, on the parameter of interest.
We expected that the relative efficiency of weighted to
unweighted least squares would start to favor unweighted least squares at lower sample sizes due to the uncertainty in estimating the asymptotic covariance matrix.
This expected relationship occurred in when the group effect was small mainly in conditions where the total sample size was below 500.
The group effect was also influencing the relative efficiency among estimation methods which implies that the heterogeneity across groups can needs to be accounted for when even at lower sample sizes.
However, these conditions were also where convergence was an issue so future work may need to investigate estimating the asymptotic covariance matrix in small sample sizes when the group effect is large.

An alternative approach to relying on the asymptotic theory needed for weighted least square is to use Bayesian methods.
Some recent work on utilizing MCMC in multilevel SEM is examining more robust priors \citep{Erp2020}.
But, in general, little evidence exists as to the performance of MCMC methods for ML-CFA/ML-SEM \cite{Depaoli2015, Hox2012}.
Some evidence suggests that a Bayesian approach with weakly informative priors may result in better coverage rates for interval estimates than WLSMV \citep{Holtmann2016}.
However, to-date, we have not found any studies on the performance of Bayesian methods when the factor structure is invariant across levels.
Future work may find that the reduced number of parameters favors one approach over the other.

\subsection{Recommendations for ML-CFA Estimation Methods}

The choice of estimation method is part technical and part practical.
In the technical sense, WLSMV provides the most robust method estimating ML-CFA model given that the parameter estimates are optimized to minimize the weighted distance between the observed (co)variances and the model implied (co)variances.
But, the computation and inversion of the weight matrix (W) is more likely to yield convergence issues (as observed in this study).
In a practical sense, when convergence fails with WLSMV, we recommend using ULSMV as an alternative, which we found converged more often.
However, when using ULSMV, there is greater chance of underestimating factor loadings and overestimating variances and covariances.
We recommend that researchers use multiple estimation methods for finding converging evidence of results.

\subsection{Limitations \& Delimitations}
As with any simulation study, the results of only generalize to the limited conditions examined.
That said, the conditions chosen were selected to mirror conditions of applied researchers as close as possible while still maintaining parsimony.
Estimation with MLR resulted in the most usable replications per cell on average, but this likely just occurred because data were treated as continuous.

In this project, we focused on the two-level (within-between) approach to ML-CFA conducted using M\textit{plus} \citep{Muthen2017}.
Other implementations are freely available such as the generalized linear latent and mixed models \citep[GLLAMM; ][]{Rabe-Hesketh2007} approach, but this approach appears to be less frequently used.
The infrequently use may be due to lack of exposure to a wide range of researchers or possible complexity in coding the model oneself.
Future research in exploring the use of these implementations may provide different conclusions about the recovery of model parameters.

%\subsection{Data Availability Statement}
%The data that support the findings of this study are openly available in Texas Data Repository at http://doi.org/10.18738/T8/VPAH7O, \citep{Padgett2019a}.
%Additional results are available in the online supplemental material as well \citep{Padgett2019b}.

%~~
% =============================
\newpage
\raggedright
%\bibliographystyle{apacite}
% You may have to select another style. Remember: LaTeX, BibTeX, LaTeX, LaTex to get the citations to appear
%\raggedright
%\urlstyle{same}
%\bibliographystyle{sageapa}
%\bibliography{references}
\printbibliography

%\section{Support for \textsf{\journalclass}}
%We offer on-line support to participating authors. Please contact
%us via e-mail at \dots
%
%We would welcome any feedback, positive or otherwise, on your
%experiences of using \textsf{\journalclass}.

\end{document}


\appendix
\chapter{Supplementary material for Chapter 3}
\section{Supplementary Figures}
\begin{figure*}[h]
\centering
\includegraphics[width=0.8\textwidth]{Appendix/Appendix_Ch3_CC.pdf}
\caption{\textbf{Selecting the ideal number of Co-Activation Patterns.} The plot shows the proportion of ambiguous clustering (PAC) for cluster numbers ranging from 3 to 20. There is a clear peak of (1 - PAC) for K~=~6 indicating that, when this value is chosen, different runs of k-means clustering yield the most consistent results. }
\label{fig:ch3_consensus}
\end{figure*}

\begin{figure*}[h]
\centering
\includegraphics[width=0.8\textwidth]{images/Appendix/ch3_BrainScoresxBehavScores.eps}
\caption{\textbf{Correlation between brain and environmental scores for BOLD variability PLS.}  }
\label{fig:ch3_brainXbehavScores}
\end{figure*}

\begin{figure*}[h]
\centering
\includegraphics[width=0.8\textwidth]{images/Appendix/ch3_CAPs_BrainScoresxBehavScores.eps}
\caption{\textbf{Correlation between brain and environmental scores for CAP PLS.}  }
\label{fig:ch3_CAP_brainXbehavScores}
\end{figure*}

\begin{figure*}[h]
\centering
\includegraphics[width=0.8\textwidth]{images/Appendix/Correlation_GA_Age.png}
\caption{\textbf{Correlation between gestational age and age at assessment.} There was no significant correlation between the two measures in the control group ($r = -0.22$, \textit{n.s.}). There as a weak correlation between the two measures in the preterm group ($r = -0.4$, $p = 0.02$).  }
\label{fig:ch3_GAvsAge}
\end{figure*}

\chapter{Supplementary material for Chapter \ref{chapter:ch5}} \label{appA}

\renewcommand{\theequation}{Eq. \thechapter.\arabic{equation}}

\section{Link between conventional PPI and the stationary PPI-CAP}

Within the PPA/CAPs framework, it is well known that averaging the frames where the seed exceeds a ``well chosen'' threshold $\phi$, yields a proxy for the conventional seed connectivity map \citep{Tagliazucchi2012,Liu2013}.
This observation can be explained in the following way:
let us assume that the activity time course $S(t)$ of a seed voxel is a realization over time, $t=1,\ldots,N$, of a random variable $S$ for which $E[S]=0$ and $E[S^2]=1$.
To construct conventional seed connectivity, a general linear model (GLM) is put forward to explain the time course $V(t)$ of a target voxel as:
\begin{eqnarray}
\label{eqn:ppa_glm}
  V(t) & =& \beta S(t) + \varepsilon(t),
\end{eqnarray}
where $\beta$ is the parameter weight (i.e., functional connectivity) and $\varepsilon(t)$ is the residual and assumed to be a realization of the noise with $E[\varepsilon]=0$ and  $E[\varepsilon^2]=\sigma^2$. In addition, we assume the noise to be independent from the seed voxel activity; i.e., $E[S\cdot \varepsilon]=0$. Multiplying both sides of \ref{eqn:ppa_glm} with $S(t)$ and taking the expectation leads to:
\begin{eqnarray}
   E[ V  \cdot S ]  & = & \beta E[S^2] + E[S \cdot \varepsilon],
\end{eqnarray}
which simplifies into $\beta=E[V\cdot S]$.

To obtain the stationary map in the CAPs approach, we want the expectation value of the target voxel for frames where the seed exceeds a threshold $\phi$:
\begin{eqnarray}
  I_\text{CAP+} &=& E[V:S>\phi] \\
          &=& E[\beta S + \varepsilon : S>\phi] \\
          &=& \beta E[S : S>\phi] + E[\varepsilon  : S>\phi],    \label{eqn:ppa-proof}
\end{eqnarray}
where in the first step we have substituted the previous GLM of \ref{eqn:ppa_glm} and ``$:$" implies conditioning.
Due to the independence of noise and seed voxel, and $E[\varepsilon]=0$, the second term of \ref{eqn:ppa-proof} will vanish, and we obtain
\begin{equation}
 I_\text{CAP+} = \beta E[S : S > \phi],
\end{equation}
which means that the expectation of suprathreshold frames in the PPA method will be proportional to the result of a conventional correlation map, with the constant of proportionality given by $E[S|S > \phi]$, which only depends on the seed, and not the target voxel. A similar relationship can be derived when the seed activity is below the negative threshold:
\begin{eqnarray}
  I_\text{CAP-} &=& E[V : S<-\phi] \\
          &=& E[\beta S + \varepsilon : S<-\phi] \\
          &=& \beta E[S : S<-\phi] + E[\varepsilon  : S<-\phi] \\
          &=& \beta E[S : S<-\phi]
\end{eqnarray}
In the case of a two-sided threshold, using the identity $\sign(A)A = |A|$, we can conclude that:
\begin{eqnarray}
  I_{CAP} &=& E\big[\sign(S)V : \left| S \right|>\phi\big] \\
   &=& E[\beta S\sign(S) + \varepsilon\sign(S) : \left| S \right|>\phi\big] \\
          &=& \beta E[S\sign(S)  : \left| S \right|>\phi\big] + E[\varepsilon \sign(S)  : \left| S \right|>\phi\big] \\
               &=& \beta E\big[|S|\, : \, \left|S \right| > \phi\big].
\end{eqnarray}

Based on this observation, we can generalize to PPI for which the GLM becomes:
\begin{eqnarray}
\label{eqn:ppi_glm}
  V(t) & =& \beta_S S(t) + \beta_T T(t) + \beta_{PPI} P(t) + \varepsilon(t),
\end{eqnarray}
where $T(t)$ is the time course of the task and $P(t)$ is the ``interaction term'' given by $P(t) = S(t)T(t)$. The stationary map of the PPI-CAPs analysis then averages values of the target voxel multiplied with the sign of the interaction term, where the selection is based on the criterion of the seed exceeding the threshold:
\begin{eqnarray*}
   I_\text{PPI-CAP} & = & E[ \sign(ST) V : \left| S\right|>\phi] \\
   & = & E[ \sign(ST) ( \beta_S S + \beta_T T + \beta_{PPI} P + \varepsilon) : \left| S\right| >\phi] \\
   & = & \beta_S E[ \sign(ST) S : \left| S \right|>\phi] + \beta_T E[\sign(ST) T : \left| S \right|>\phi] + \beta_{PPI} E[\sign(ST) P : \left| S \right| > \phi].
\end{eqnarray*}
Using the identities $\sign(A)A = |A|$ and $\sign(AB)A = \sign(B) |A|$, this becomes
\begin{eqnarray*}
  I_\text{PPI-CAP} &= & \beta_S E\big[\sign(T)|S| : \left| S\right|>\phi\big] + \beta_T E\big[\sign(S)|T| : \left| S\right|>\phi\big] \label{eqn:ppi-T-term} + \beta_{PPI} E\big[|ST| : \left| S\right|>\phi\big] \\
  & \approx & \beta_{PPI} E\big[|ST| : \left| S\right|>\phi\big].
\end{eqnarray*}
The last approximation can be made when the contributions of the first two terms are negligible.
For the first term, we assume the absolute value of $S$ to be independent of the task (that weights the expectation operator) and $T$ has equal number of positive and negative values, in which case the first term will be 0.
For the second term, if the task is just a change of sign, then its absolute value will be fixed and uncorrelated to the sign of the seed. Thus the second term is proportional to $E\big[\sign(S) : \left| S\right|>\phi\big]$, which is necessarily bound between $\pm1$ and will be exactly 0 if S is symmetric.

Therefore, what remains is the final term and we find that the result of a PPI analysis will be proportional to the SiMap.
This brief derivation also shows that the stationary PPI-CAP can be contaminated by ``leakage'' from the seed and/or task contributions if these assumptions are not fulfilled.

\clearpage
\section{Supplementary Figures for Section \ref{section:ppicaps_method}} \label{Supp}

% Reset Figure count
%\setcounter{figure}{0}

\begin{figure*}[h]
\centering
\includegraphics[width=1\textwidth]{Appendix/20191120_simapIntuition.pdf}
\caption{\textbf{An intuitive toy example to illustrate the relationship between the static interaction map (simap) and PPI analysis results.} A)~Moments where the seed activity is significantly high (red stars) or low (blue stars) are selected. B)~Three possible cases are depicted: 1)~When the target voxel is correlated only with the seed (meaning that there is no interaction effect), its resulting value in the siMap will be close to zero, as would be expected for its corresponding beta value in a PPI analysis. 2)~Similarly, when the target voxel is correlated only with the task, its resulting value in the siMap will be close to zero. 3)~when the target voxel is correlated with the PPI regressor, meaning that its relationship with the seed changes according to the context, the averaged value from its suprathreshold frames will yield a high value, indicating a PPI effect.}
\label{fig:simapIntuition}
\end{figure*}

\begin{figure*}[h]
\centering
\includegraphics[width=1\textwidth]{Appendix/20191201_SeedChecks.pdf}
\caption{\textbf{Sanity check on the chosen seed signal.} Before proceeding with the PPI-CAPs analysis, we checked the orthogonality of the seed, task and ppi regressors using SPM's Check Orthogonality tool, as well as the skewness of the seed. A)~The group-averaged matrix confirms that the regressors are not collinear at the group level. The highest off-diagonal value is 0.04. B)~Shows the same matrix at the single-subject level. C)~The seed signal was not skewed for any of the subjects: the skewness value stayed within the [-0.5 0.5] range for all of them. D)~Shows the distribution of z-scored seed signal values for each subject. Additionally, the magnitude of the seed timecourse was not correlated with the sign of the task: the Pearson’s r coefficient for the subject with the strongest correlation between the magnitude of the seed and the sign of the task was 0.11.  }\label{fig:seedcheck}
\end{figure*}

\begin{figure*}[h]
\centering
\includegraphics[width=0.5\textwidth]{Appendix/20190810_GroupMap}
\caption{\textbf{Group-level static interaction map resembles PPI analysis results.} The spatial correlation betweeen the two analyses' results was 0.85 for the siMap calculated from 60\% frames. }
\label{fig:groupSimap}
\end{figure*}

\begin{figure*}[h]
\centering
\includegraphics[width=1\textwidth]{Appendix/20190809_ConsensusClustering}
\caption{\textbf{An analysis of the number of occurrences of each PPI-CAP per subject combined with consensus clustering indicates the optimal number of groups in which to cluster the suprathreshold frames.} (Top) The PPI-CAPs distributions for  $k\in[3, 8]$ showed that $k$ values of 6 and above yielded PPI-CAPs that never occurred for some subjects, while values between 4 and 5 had the most homogeneous distribution of pattern occurrence per subject. (Bottom) Visual inspection of the consensus matrices showed that the most stable values for $k$ (that is, those for which any two frames would most consistently be clustered together or separately) were $k=3$ and $k=4$. Since 4 represents a better trade-off in terms of variety and robustness, we chose this value for further analysis (see its clear pattern highlighted in green). The colormap represents the proportion of time when two frames were consistently clustered either together or separately.}
\label{fig:consensus}
\end{figure*}

\begin{figure*}[h]
\centering
\includegraphics[width=1\textwidth]{Appendix/20191120_stats.pdf}
\caption{\textbf{Significance assessment of PPI-CAPs.} Each histogram illustrates the distribution of determinant values of the confusion matrices obtained from 3000 random permutations of the frame labels composing each PPI-CAP, with respect to seed, task or PPI. The red line indicates where the determinant of the confusion matrix for real data lies.  }
\label{fig:permanova}
\end{figure*}

\begin{figure*}[h]
\centering
\includegraphics[width=1\textwidth]{Appendix/20190809_PPI_CAPs_Fig5}
\caption{\textbf{The high temporal resolution of the PPI-CAPs approach allows us to link brain activity to stimuli presented at specific time points.} Here, we show moments when each PPI-CAP was highly consistent across participants (left). The y-axis shows the percentage of subjects, out of the total 15 participants, who expressed the PPI-CAP at each given time point.  The histogram on the right hand side of the panel depicts the distribution of random consistency across subjects. This was calculated by randomly permuting the time points on which the PPI-CAP appeared for each subject, then re-calculating the subject consistency across time (the same plot as the ones on the left). Finally, we plotted the distribution of possible consistency values. The dashed line represents the value of the 99\textsuperscript{th} percentile from the random distribution, indicating that a subject consistency above this threshold is significant. Selected time points of high consistency across subjects for each PPI-CAP are indicated by numbers on the left plots, and the corresponding video frames are presented for those times on the rightmost side of the figure.   }
\label{fig:PPI_CAPs_Fig5}
\end{figure*}

\clearpage
\section{Supplementary Figures for Section \ref{section:ppicaps_preterm}}

\subsection*{ACC-seed PPI-CAPs in preterm-born adolescents}
% --------------------------------------------------------------------------
% ACC-SEED CHOICE OF K
% --------------------------------------------------------------------------

\begin{figure*}[h]
\centering
\includegraphics[width=1\textwidth]{images/Appendix/Appendix_Ch4_ACC_K.png}
\caption{\textbf{Choosing the best number of clusters for an ACC-seed PPI-CAPs analysis.} 30\% of the frames when the dorsal anterior cingulate cortex (ACC) was most (de)activated were selected for analysis. Consensus clustering was run for K ranging from 3 to 20, in 10 folds for each of which a random subsample including 80\% of the subjects was used. On the left, the consensus matrices provide a visual appreciation of how often each pair of frames was consistently clustered (\textit{i.e.,} both always in the same cluster, or both never in the same cluster). Blue indicates never, red indicates always. On the right, we plot the proportion of ambiguously clustered frames for each K. Since we want this value to be very low, a peak in the (1-PAC) plot indicates the most stable choice of K. Since K~=~6 has a clear peak and means we will have a large number of frames per PPI-CAP (essential for averaging out noise), we select this as opposed to higher K numbers who also had good results.}
\label{fig:app_acc_k}
\end{figure*}

% --------------------------------------------------------------------------
%  ACC PPI-CAPS ASSINMENT TO YEO NETWORKS
% --------------------------------------------------------------------------

\begin{figure*}[h]
\centering
\includegraphics[width=1\textwidth]{images/Appendix/ACCK6_assigment_YeoRSN.png}
\caption{\textbf{Identifying networks present in each PPI-CAP.} The two matrices compare the activated (red) and deactivated (blue) voxels in each PPI-CAP from Figure \ref{fig:acc_ppicaps} to the 7 functional networks from \citet{Yeo2011}. Colour intensity and numbers indicate the proportion of voxels from each network that are present in the PPI-CAP. Networks were then ordered according to the Hungarian Assignment Algorithm \citep{Munkres1957}. According to this result, PPI-CAP\textsubscript{1} corresponds to an activated Somatomotor Network (SMN) and deactivated Default Mode Network;  PPI-CAP\textsubscript{2} is assigned to an activated Ventral Attention Network, also known as Salience Network (SN),  and deactivated Limbic; PPI-CAP\textsubscript{3} contains an activated Visual and deactivated Frontoparietal Network (FPN); PPI-CAP\textsubscript{4} includes an activated Dorsal Attention (DAN) and deactivated Visual Network; PPI-CAP\textsubscript{5} has an activated FPN and deactivated SMN; and PPI-CAP\textsubscript{6} includes an activated Limbic and deactivated DAN.  }
\label{fig:app_acc_munkres}
\end{figure*}

% --------------------------------------------------------------------------
%  ACC-based PERMUTATION TESTING
% --------------------------------------------------------------------------

\begin{sidewaysfigure*}[h]
\centering
\includegraphics[width=1\textwidth]{images/Appendix/Ch5_ACC_stats.png}
\caption{\textbf{Statistical assessment of each PPI-CAP's effects.} To check whether the effects identified in Figure \ref{fig:acc_cm} are significant, we performed permutation testing by shuffling the corresponding effect's labels 3000 times and calculating the det-index in each case, to generate a null distribution. We then check where the real data's det-index lies within this distribution and calculate the p-value. }
\label{fig:app_acc_stats}
\end{sidewaysfigure*}

% --------------------------------------------------------------------------
%  PCC SECTION
% --------------------------------------------------------------------------

\clearpage
\subsection*{PCC-seed PPI-CAPs in preterm-born adolescents} \label{sm_pcc_ppicaps}
 In Chapter \ref{chapter:ch5}, Section \ref{section:ppicaps_preterm}, we perform a PPI-CAPs analysis to explore task-related dynamic connectivity using the dorsal anterior cingulate cortex as a seed. Here we provide a similar analysis using the posterior cingulate cortex (PCC). While interpreting these additional results goes beyond the goal of this Section, this additional work illustrates that task-driven dynamics are a powerful and compelling strategy to uncover features of brain function in clinical populations independently of the choice of seed.
\vspace{1cm}

% --------------------------------------------------------------------------
% PCC-SEED CHOICE OF K
% --------------------------------------------------------------------------

\begin{figure*}[h]
\centering
\includegraphics[width=1\textwidth]{images/Appendix/Appendix_Ch5_PCC_K.png}
\caption{\textbf{Choosing the best number of clusters for a PCC-seed PPI-CAPs analysis.} 30\% of the frames when the posterior cingulate cortex (PCC) was most (de)activated were selected for analysis. Consensus clustering was run for K ranging from 3 to 20, in 10 folds for each of which a random subsample including 80\% of the subjects was used. On the left, the consensus matrices provide a visual appreciation of how often each pair of frames was consistently clustered (\textit{i.e.,} both always in the same cluster, or both never in the same cluster). Blue indicates never, red indicates always. On the right, we plot the proportion of ambiguously clustered frames for each K. Since we want this value to be very low, a peak in the (1-PAC) plot indicates the most stable choice of K. Since K = 4 has a clear peak and means we will have a large number of frames per PPI-CAP (essential for averaging out noise), we select this as opposed to higher K numbers who also had good results.}
\label{fig:app_pcc_k}
\end{figure*}

% --------------------------------------------------------------------------
% PCC-SEED PPI-CAPS
% --------------------------------------------------------------------------

\begin{figure*}[h]
\centering
\includegraphics[width=0.85\textwidth]{images/Appendix/Appendix_Ch5_PPI-CAPs.pdf}
\caption{\textbf{PCC-seed PPI-CAPs.} Using K~=~4 for the clustering step as defined in Figure \ref{fig:app_pcc_k} yielded the four PPI-CAPs above. Each row corresponds to one PPI-CAP, numbers indicate slice coordinates in MNI space.}
\label{fig:app_pcc_ppicaps}
\end{figure*}

% --------------------------------------------------------------------------
%  PCC PPI-CAPS ASSINMENT TO YEO NETWORKS
% --------------------------------------------------------------------------

\begin{figure*}[h]
\centering
\includegraphics[width=1\textwidth]{images/Appendix/PCCK4_assigment_YeoRSN.png}
\caption{\textbf{Identifying networks present in each PPI-CAP.} The two matrices compare the activated (red) and deactivated (blue) voxels in each PPI-CAP from Figure \ref{fig:app_pcc_ppicaps} to the 7 functional networks from \citet{Yeo2011}. Colour intensity and numbers indicate the proportion of voxels from each network that are present in the PPI-CAP. Networks were then ordered according to the Hungarian Assignment Algorithm \citep{Munkres1957}. According to this result, PPI-CAP\textsubscript{1} corresponds to an activated Dorsal Attention Network (DAN) and deactivated Default Mode Network;  PPI-CAP\textsubscript{2} is assigned to an activated Limbic Network (LN) and deactivated DAN; PPI-CAP\textsubscript{3} contains an activated Salience Network and deactivated LN; and PPI-CAP\textsubscript{4} includes an activated Visual Network and deactivated Fronto-Parietal Network. }
\label{fig:app_pcc_munkres}
\end{figure*}

% --------------------------------------------------------------------------
%  PCC-based CONFUSION MATRICES
% --------------------------------------------------------------------------

\begin{figure*}[h]
\centering
\includegraphics[width=1\textwidth]{images/Appendix/PCCK4_PPICAPs_confusionMatrices.png}
\vspace{5mm}
\caption{\textbf{Confusion matrices for PCC-seed PPI-CAPs.} To identify main and interaction effects for each PPI-CAP, we generate confusion matrices to see how often the sign of a PPI-CAP switches in the same way as each of the effects. We tested three main effects (Seed; Task; and Group) and three interaction effects (Seed vs. Task (PPI); Group vs. Seed; and Group vs. Task). The signs for each effect are as follows: Seed --- positive and negative signs correspond to frames when the seed was activated of deactivated, respectively;  Task --- positive signs correspond to "Movie Watching" while negative signs correspond to moments of "Emotion Regulation"; Group --- positive signs correspond to fullterm controls, while negative signs correspond to preterm-born individuals. Interaction signs are calculated as element-by-element multiplication of the main effect signs. Light yellow indicates a low number of frames, while dark red indicates the highest number of frames. }
\label{fig:app_pcc_confusion}
\end{figure*}

% --------------------------------------------------------------------------
%  PCC-based PERMUTATION TESTING
% --------------------------------------------------------------------------

\begin{figure*}[h]
\centering
\includegraphics[width=1\textwidth]{images/Appendix/PCCK4_stats_permutation.png}
\vspace{5mm}
\caption{\textbf{Statistical assessment of each PPI-CAP's effects.} To check whether the effects identified in Figure \ref{fig:app_pcc_confusion} are significant, we performed permutation testing by shuffling the corresponding effect's labels 3000 times and calculating the det-index in each case, to generate a null distribution. We then check where the real data's det-index lies within this distribution and calculate the p-value. }
\label{fig:app_pcc_stats}
\end{figure*}


\label{section:outputs}
\setcounter{footnote}{0}
\section{Outputs}

For each inout file, \etwomsa\, produces the
following output files:

\begin{sreitems}{}

\item[\emprog{file.e2msa.AFG.sto}]
%
The resulting alignment in Stockholm format.

\item[\emprog{file.AFG.bench}]
%
If the input file is an alignment, it also producds file and option \emprog{file.AFG.bench}. This file
provides information about the comparison of the input alignment with the \etwomsa\ alignment

\begin{sreoutput}
#MSA_NAME  TP_hom TRUE_hom FOUND_hom
globins4.1 760 827 804 0 0 29.06 29.06 88.11 83.68 0.48 7.36 e2msa.AFG 171 174 147.25 5.06 147.25 5.06 6.0 1.41 4.5 1.91 3.96 2.71 5.94 3.04 -2288.18 0 0 0
\end{sreoutput}

 This file produces a one line output per alignment in the file.
 \begin{sreitems}{\prog{Second column}}
 \item[\prog{Column 1}]  MSA name.
 \item[\prog{Column 2}]  Number of true homologies found in the \etwomsa\ alignment (TP=true positives).
 \item[\prog{Column 3}]  Number of homologies in the input alignment (T=true homologies).
 \item[\prog{Column 4}]  Number of true homologies found in the \etwomsa\ alignment (F=found).
 \item[\prog{Column 5}]  Number of nonhomologous  positions in \etwomsa\ alignment.
 \item[\prog{Column 6}]  Number of nonhomologous  positions in input alignment.
 \item[\prog{Column 7}]  Average \% identity in input alignment.
 \item[\prog{Column 8}]  Average \% identity in \etwomsa\ alignment.
 \item[\prog{Column 9}]  Average number of matches in input alignment.
 \item[\prog{Column 10}] Average number of matches in \etwomsa\ alignment.
 \item[\prog{Column 11}] Average time of inferred tree.
 \item[\prog{Column 12}] Time used to run the \etwomsa\ program.
 \item[\prog{Column 13}] Evolutionary model used.
 \item[\prog{Column 14}] Length of input alignment.
 \item[\prog{Column 15}] Length of \etwomsa\ alignment.
 \item[\prog{Column 16-17}] Avg/standard-deviation of length of seqs in input alignment.
 \item[\prog{Column 18-19}] Avg/standard-deviation of length of seqs in \etwomsa\ alignment.
 \item[\prog{Column 20-21}] Avg/standard-deviation of number of gaps in input alignment.
 \item[\prog{Column 22-23}] Avg/standard-deviation of number of gaps in \etwomsa\ alignment.
 \item[\prog{Column 24-25}] Avg/standard-deviation of length of gaps in input alignment.
 \item[\prog{Column 26-27}] Avg/standard-deviation of length of gaps in \etwomsa\ alignment.
 \item[\prog{Column 28}]    Aignment score.
 \item[\prog{Column 29}]    Number of homologous  positions in input alignment.
 \item[\prog{Column 30}]    Number of homologous  positions in \etwomsa\ alignment.
 \item[\prog{Column 31}]    number of homologous  positions both in input alignment and \etwomsa\ alignment.
 \end{sreitems}

\end{sreitems}


\chapter{Theory}

\section{Introduction} \label{sec:theory_intro}
\input{theory/introduction.tex}

\section{Quantum Field Theory} \label{sec:theory_qft}
\input{theory/qft.tex}

\section{The Standard Model of Particle Physics} \label{sec:theory_sm}
\input{theory/sm.tex}

\subsection{The Higgs Boson} \label{sec:theory_higgs}
\input{theory/higgs.tex}

%\section{Shortcomings of the Standard Model} \label{sec:theory_sm_problems}
%\input{theory/sm_problems.tex}

%\section{Summary} \label{sec:theory_summary}


\documentclass[10pt]{art}

\usepackage{minted, booktabs, setspace}
\usepackage{geometry}

% Fancy headers
\pagestyle{fancy}
\fancyhf{}
\fancyhead[R]{\footnotesize\textcolor{gray80}{{\thepage / \pageref{LastPage}}}}
\renewcommand{\sectionmark}[1]{\markboth{}{\thesection.\ #1}}
\fancyhead[L]{\footnotesize\textcolor{gray80}{{\MakeUppercase{\rightmark}}}}

% Section and section styles
\renewcommand{\thesection}{\Roman{section}}
\titleformat{\section}{\centering\scshape\Large\color{raspeberry}}{\thesection}{0.7em}{}
\titleformat{\section}{\strongenv\large\color{gray80}}{\thesection}{0.7em}{}

% Pack enumerate items and bibitems
\setlist{nolistsep}
\setlength{\bibitemsep}{.2\baselineskip plus .05\baselineskip minus .05\baselineskip}

% Code listings
\usemintedstyle{solarized-dark}
\setminted{escapeinside=~~}
\setmintedinline{escapeinside=~~}

% Fonts
\setmainfont{Avenir Next}
\setmonofont{Source Code Pro}
\defaultfontfeatures{Ligatures=TeX, Scale=MatchUppercase}

\title{Bonak: Notes on the formalization}
\author{Ramkumar Ramachandra}
\date{}

\begin{document}
\thispagestyle{empty}
\maketitle

The core of our formalization effort consists of inequalities in \emph{Peano arithmetic}. We start with one inequality primitive, namely $\leq$, and build up all our inequalities on the basis of that inequality. Two different proofs of \mintinline{coq}{p <= q} must be the same: we chose to use \mintinline{coq}{SProp} for the proof irrelevance.

\section{Notations in \texorpdfstring{\leq}{<=}}
On the outset, we defined various \emph{raising}, \emph{lowering}, and \emph{composition} operations on $\leq$:

\mint{coq}/~↑~ := (Hnm : n <= m) : n <= S m/
\mint{coq}/~⇑~ := (Hnm : n <= m) : S n <= S m/
\mint{coq}/~↓~ := (Hnm : S n <= m) : n <= m/
\mint{coq}/~⇓~ := (Hnm : S n <= S m) : n <= m/
\mint{coq}/~↕~ := (Hnm : n <= m) (Hmp : m <= p) : n <= p/

\section{An SProp primer}
\mintinline{coq}{SProp} is a replacement for standard propositions, or \mintinline{coq}{Prop}, that provides proof irrelevance.

There are three inhabitants of \mintinline{coq}{SProp}: \mintinline{coq}{sUnit} corresponding to $\top$, \mintinline{coq}{sEmpty} corresponding to $\bot$, and \mintinline{coq}{sProposition} corresponding to a definitionally proof-irrelevant term. The way \mintinline{coq}{SProp} implements definitional proof-irrelevance is a simple engineering detail: there is hard-coding in Coq to render two inhabitants of \mintinline{coq}{sProposition} trivially inter-convertible.

\begin{listing}[H]
  \begin{minted}{coq}
    Theorem SPropIrr (P : SProp) (x y : P) : x = y.
    Proof.
      by reflexivity.
    Abort. (* Type-check fails at Qed.
            * (=) : ∀ A, A -> A -> Prop, but we want to return an SProp. *)
  \end{minted}
\end{listing}

We can, however, define an \mintinline{coq}{=S} that returns an \mintinline{coq}{SProp}:

\begin{listing}[H]
  \begin{minted}{coq}
    Inductive eqsprop {A : SProp} (x : A) : A -> Prop :=
      eqsprop_refl : eqsprop x x.
    Infix "=S" := eqsprop (at level 70) : type_scope.
  \end{minted}
\end{listing}

\section{\texorpdfstring{\leq}{<=} in SProp}
We tried three different approaches to defining $\leq$ in \mintinline{coq}{SProp}.

\begin{enumerate}
  \item[(a)] A recursive definition.
    \begin{listing}[H]
      \begin{minted}{coq}
      Inductive sFalse : SProp :=.
      Inductive sTrue : SProp := sI.

      Fixpoint le m n : SProp :=
        match m with
        | 0 => sTrue
        | S m =>
          match n with
          | 0 => sFalse
          | S n => le m n
          end
        end.
     \end{minted}
    \end{listing}
  \item[(b)] An inductive definition.
    \begin{listing}[H]
      \begin{minted}{coq}
      Inductive le' (n : nat) : nat -> SProp :=
      | le_refl : n <= n
      | le_S_up : forall {m : nat}, n <= m -> n <= S m
      where "n <= m" := (le' n m) : nat_scope.

    Definition le := le'.
    \end{minted}
    \end{listing}
  \item[(c)] A refinement of the inductive definition; a nested implication.
    \begin{listing}[H]
      \begin{minted}{coq}
      Definition le n m := forall p, p <= n -> p <= m.
      \end{minted}
    \end{listing}
\end{enumerate}

(a) was unworkable right from the start, and we tried approach (b) for a while, before running into unification problems. We then settled on approach \c, because it made compositions of inequalities almost trivial to prove. Compare the following snippets defining transitivity of $\leq$:

\begin{listing}[H]
  \begin{minted}{coq}
    Theorem le_trans {p q n} : p <= q -> q <= n -> p <= n.
    (* ... a non-trivial proof ... *)
    Defined.
  \end{minted}
  \caption{\mintinline{coq}{le_trans} when $\leq$ is defined using an inductive approach}
\end{listing}

\begin{listing}[H]
  \begin{minted}{coq}
    Definition le_trans {n m p} (Hnm : n <= m) (Hmp : m <= p) : n <= p :=
      fun q (Hqn : le' q n) => Hmp _ (Hnm _ Hqn).
  \end{minted}
  \caption{\mintinline{coq}{le_trans} when $\leq$ is defined using a nested implication}
\end{listing}

Implications are naturally easier to work with, so the nested implication in \mintinline{coq}{SProp} was an easy choice. Many of the associtivity and commutativity proofs are then a simple \mintinline{coq}{reflexivity}:

\begin{listing}[H]
  \begin{minted}{coq}
  Theorem le_trans_comm4 {n m p} (Hmn : S n <= S m) (Hmp : S m <= S p) :
  ~⇓~ (Hmn ~↕~ Hmp) =S (~⇓~ Hmn) ~↕~ (~⇓~ Hmp).
    reflexivity.
  Defined.
  \end{minted}
\end{listing}

\section{The Fibonacci trick}
We'd initially defined \mintinline{coq}{Cubical} in a monolithic manner, and attempted to use \emph{well-founded induction} to build it up in steps. Induction is the process of successively building up a type by reasoning on the result of the previous iteration, and well-founded induction remembers all the previous levels. A simple induction is clearly insufficient for our purposes, and as it turned out, well-founded induction was a non-starter too.

Our initial \mintinline{coq}{Record} looked like:

\begin{listing}[H]
  \begin{minted}{coq}
    Record Cubical {n : nat} :=
    {
    csp {n'} (Hn' : n' <= n) : Type@{l'} ;
    hd {n'} {Hn' : S n' <= n} : csp Hn' -> csp (⇓ Hn') ;
    box {n' p} {Hn' : n' <= n} (Hp : p <= n') : csp Hn' -> Type@{l} ;
    tl {n'} {Hn' : S n' <= n} (D : csp Hn') :
      box (le_refl n') (hd D) -> Type@{l} ;
    (* ... *)
    }.
  \end{minted}
\end{listing}

To build it up in steps, we defined a \mintinline{coq}{Fixpoint} as follows, but ran into unification problems:

\begin{listing}[H]
  \begin{minted}[linenos]{coq}
    Theorem le_dec {n m} : n <= m -> {n = m} + {n <= pred m} + {m = O}.
    Admitted.
    Theorem lower_both {m n} : S m <= S n -> m <= n.
    Admitted.

    (* ... *)

    Fixpoint cubical {n : nat} : Cubical :=
    match n with
    | O => {|
      csp _ _ := unit;
      hd _ Hn' _ := ltac:(apply (le_discr Hn'));
      box _ _ _ _ _ := unit;
      tl _ Hn' _ _ := ltac:(apply (le_discr Hn'));
      (* ... *)
      |}
    | S n => let cn := cubical (n := n) in

    (* ... *)

    let hdn {n'} (Hn' : S n' <= S n) :=
    match le_dec (lower_both Hn') return cspn Hn' (* S n' <= S n *) ->
    cspn (⇓ Hn') (* n' <= S n *) with
    | inleft Hcmp =>
    match Hcmp with
    | left _ (* n' = n *) => fun D => D.1 (* n' = S n *)
    | right _ (* n' <= pred n *) => fun D => cn.(hd) D (* n' <= n *)
    end
    \end{minted}
  \caption{Type unification on the term \mintinline{coq}{D} fails}
\end{listing}

We then settled on remembering the previous two levels of induction, a middle-ground between plain induction and well-founded recursion, which we refer to as the Fibonacci trick. Our final \mintinline{coq}{Record} for the box looks like:

\begin{listing}[H]
  \begin{minted}{coq}
    Record PartialBoxPrev (n : nat) (csp : Type@{l'}) := {
      box' {p} (Hp : p.+1 <= n) : csp -> Type@{l} ;
      box'' {p} (Hp : p.+2 <= n) : csp -> Type@{l} ;
      subbox' {p q} {Hp : p.+2 <= q.+2} (Hq : q.+2 <= n) (ε : side) {D : csp} :
        box' (↓ (Hp ↕ Hq)) D -> box'' (Hp ↕ Hq) D;
    }.

    Record PartialBox (n p : nat) (csp : Type@{l'})
    (PB : PartialBoxBase n csp) := {
      box (Hp : p <= n) : csp -> Type@{l} ;
      subbox {q} {Hp : p.+1 <= q.+1} (Hq : q.+1 <= n) (ε : side) {D : csp} :
      box (↓ (Hp ↕ Hq)) D -> PB.(box') (Hp ↕ Hq) D;
      cohbox {q r} {Hpr : p.+2 <= r.+2} {Hr : r.+2 <= q.+2} {Hq : q.+2 <= n}
      {ε : side} {ω : side} {D: csp} (d : box (↓ (⇓ Hpr ↕ (↓ (Hr ↕ Hq)))) D) :
      PB.(subbox') (Hp := Hpr ↕ Hr) Hq ε (subbox (Hp := ⇓ Hpr) (↓ (Hr ↕ Hq)) ω d) =
      (PB.(subbox') (Hp := Hpr) (Hr ↕ Hq) ω (subbox (Hp := ↓ (Hpr ↕ Hr)) Hq ε d));
    }.
  \end{minted}
\end{listing}

To build up the box, we'd simply have to induction on \mintinline{coq}{p}:

\begin{listing}[H]
  \begin{minted}{coq}
    Definition mkcsp {n : nat} {C : Cubical n} : Type@{l'} :=
      { D : C.(csp) & C.(Box).(box) (le_refl n) D -> Type@{l} }.

    Definition mkBoxPrev {n} {C : Cubical n} :
      PartialBoxBase n.+1 mkcsp := {|
      box' {p} {Hp : p.+1 <= n.+1} {D : mkcsp} := C.(Box).(box) (⇓ Hp) D.1 ;
      box'' {p} {Hp : p.+2 <= n.+1} {D : mkcsp} := C.(PB).(box') (⇓ Hp) D.1 ;
      subbox' {p q} {Hp : p.+2 <= q.+2} {Hq : q.+2 <= n.+1} {ε} {D : mkcsp} {d} :=
        C.(Box).(subbox) (Hp := ⇓ Hp) (⇓ Hq) ε d ;
    |}.

    Definition mkBox {n} {C : Cubical n} p : PartialBox n.+1 p mkcsp mkPB.
      induction p.
      + (* ... *)
  \end{minted}
\end{listing}

\section{UIP}
We use UIP in the final coherence condition in \mintinline{coq}{cohbox} and \mintinline{coq}{cohcube}. If we weren't to use UIP in \mintinline{coq}{cohbox}, then we'd have something similar to:

\begin{listing}[H]
  \begin{minted}{coq}
    cohbox : subbox ( subbox ... ) = subbox ( subbox ... )
    coh2box : cohbox ... = cohbox ...
    ...
    cohNbox : cohPredNbox ... = cohPredNbox ...
  \end{minted}
\end{listing}

In attempting to write the proof without UIP, we'd have a relation between \mintinline{coq}{cohNbox} and \mintinline{coq}{cohPredNbox}, which, we expect to be tricky to prove.

\end{document}