    \filetitle{diffloglik}{Approximate gradient and hessian of log-likelihood function}{model/diffloglik}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
[MinusLogLik,Grad,Hess,V] = diffloglik(M,Inp,Range,PList,...)
\end{verbatim}

\paragraph{Input arguments}\label{input-arguments}

\begin{itemize}
\item
  \texttt{M} {[} model {]} - Model object whose likelihood function will
  be differentiated.
\item
  \texttt{Inp} {[} cell \textbar{} struct {]} - Input data from which
  measurement variables will be taken.
\item
  \texttt{Range} {[} numeric {]} - Date range on which the likelihood
  function will be evaluated.
\item
  \texttt{PList} {[} cellstr {]} - List of model parameters with respect
  to which the likelihood function will be differentiated.
\end{itemize}

\paragraph{Output arguments}\label{output-arguments}

\begin{itemize}
\item
  \texttt{MinusLogLik} {[} numeric {]} - Value of minus the likelihood
  function at the input data.
\item
  \texttt{Grad} {[} numeric {]} - Gradient (or score) vector.
\item
  \texttt{Hess} {[} numeric {]} - Hessian (or information) matrix.
\item
  \texttt{V} {[} numeric {]} - Estimated variance scale factor if the
  \texttt{'relative='} options is true; otherwise \texttt{v} is 1.
\end{itemize}

\paragraph{Options}\label{options}

\begin{itemize}
\item
  \texttt{'chkSstate='} {[} \texttt{true} \textbar{}
  \emph{\texttt{false}} \textbar{} cell {]} - Check steady state in each
  iteration; works only in non-linear models.
\item
  \texttt{'refresh='} {[} \emph{\texttt{true}} \textbar{} \texttt{false}
  {]} - Refresh dynamic links for each change in a parameter.
\item
  \texttt{'solve='} {[} \emph{\texttt{true}} \textbar{} \texttt{false}
  \textbar{} cellstr {]} - Re-compute solution for each parameter
  change; you can specify a cell array with options for the
  \texttt{solve} function.
\item
  \texttt{'sstate='} {[} \texttt{true} \textbar{} \emph{\texttt{false}}
  \textbar{} cell {]} - Re-compute steady state in each differentiation
  step; if the model is non-linear, you can pass in a cell array with
  options used in the \texttt{sstate} function.
\end{itemize}

See help on \href{model/filter}{\texttt{model/filter}} for other options
available.

\paragraph{Description}\label{description}

\paragraph{Example}\label{example}


\chapter{``M'' Standard Extension for Integer Multiplication and
  Division, Version 2.0}

This chapter describes the standard integer multiplication and
division instruction extension, which is named ``M'' and contains
instructions that multiply or divide values held in two integer
registers.

\begin{commentary}
We separate integer multiply and divide out from the base to simplify
low-end implementations, or for applications where integer multiply
and divide operations are either infrequent or better handled in
attached accelerators.
\end{commentary}

\section{Multiplication Operations}

\vspace{-0.2in}
\begin{center}
\begin{tabular}{S@{}R@{}R@{}S@{}R@{}O}
\\
\instbitrange{31}{25} &
\instbitrange{24}{20} &
\instbitrange{19}{15} &
\instbitrange{14}{12} &
\instbitrange{11}{7} &
\instbitrange{6}{0} \\
\hline
\multicolumn{1}{|c|}{funct7} &
\multicolumn{1}{c|}{rs2} &
\multicolumn{1}{c|}{rs1} &
\multicolumn{1}{c|}{funct3} &
\multicolumn{1}{c|}{rd} &
\multicolumn{1}{c|}{opcode} \\
\hline
7 & 5 & 5 & 3 & 5 & 7 \\
MULDIV & multiplier & multiplicand & MUL/MULH[[S]U] & dest & OP    \\
MULDIV & multiplier & multiplicand & MULW           & dest & OP-32 \\
\end{tabular}
\end{center}

MUL performs an XLEN-bit$\times$XLEN-bit multiplication and places the
lower XLEN bits in the destination register.  MULH, MULHU, and MULHSU
perform the same multiplication but return the upper XLEN bits of the
full 2$\times$XLEN-bit product, for signed$\times$signed,
unsigned$\times$unsigned, and signed$\times$unsigned multiplication
respectively.  If both the high and low bits of the same product are
required, then the recommended code sequence is: MULH[[S]U] {\em rdh,
  rs1, rs2}; MUL {\em rdl, rs1, rs2} (source register specifiers must
be in same order and {\em rdh} cannot be the same as {\em rs1} or {\em
  rs2}).  Microarchitectures can then fuse these into a single
multiply operation instead of performing two separate multiplies.

\begin{commentary}
MULHSU is used in multi-word signed multiplication to multiply the
most-significant word of the multiplier (which contains the sign bit)
with the less-significant words of the multiplicand (which are unsigned).
\end{commentary}

MULW is only valid for RV64, and multiplies the lower
32 bits of the source registers, placing the sign-extension of the
lower 32 bits of the result into the destination register.  MUL can be
used to obtain the upper 32 bits of the 64-bit product, but signed
arguments must be proper 32-bit signed values, whereas unsigned
arguments must have their upper 32 bits clear.

\section{Division Operations}

\vspace{-0.2in}
\begin{center}
\begin{tabular}{S@{}R@{}R@{}O@{}R@{}O}
\\
\instbitrange{31}{25} &
\instbitrange{24}{20} &
\instbitrange{19}{15} &
\instbitrange{14}{12} &
\instbitrange{11}{7} &
\instbitrange{6}{0} \\
\hline
\multicolumn{1}{|c|}{funct7} &
\multicolumn{1}{c|}{rs2} &
\multicolumn{1}{c|}{rs1} &
\multicolumn{1}{c|}{funct3} &
\multicolumn{1}{c|}{rd} &
\multicolumn{1}{c|}{opcode} \\
\hline
7 & 5 & 5 & 3 & 5 & 7 \\
MULDIV & divisor & dividend & DIV[U]/REM[U]   & dest & OP    \\
MULDIV & divisor & dividend & DIV[U]W/REM[U]W & dest & OP-32 \\
\end{tabular}
\end{center}

DIV and DIVU perform signed and unsigned integer division of XLEN
bits by XLEN bits, rounding towards zero.
REM and REMU provide the remainder of the
corresponding division operation.  If both the quotient and remainder
are required from the same division, the recommended code sequence is:
DIV[U] {\em rdq, rs1, rs2}; REM[U] {\em rdr, rs1, rs2} ({\em rdq}
cannot be the same as {\em rs1} or {\em rs2}).  Microarchitectures can
then fuse these into a single divide operation instead of performing
two separate divides.

The semantics for division by zero and division overflow are
summarized in Table~\ref{tab:divby0}.  The quotient of division by
zero has all bits set, i.e. $2^{XLEN}-1$ for unsigned division or $-1$
for signed division.  The remainder of division by zero equals the
dividend.  Signed division overflow occurs only when the most-negative
integer, $-2^{XLEN-1}$, is divided by $-1$.  The quotient of signed
division overflow is equal to the dividend, and the remainder is zero.
Unsigned division overflow cannot occur.

\vspace{0.1in}
\begin{table}[h]
\center
\begin{tabular}{|l|c|c||c|c|c|c|}
\hline
Condition              & Dividend      & Divisor & DIVU         & REMU & DIV           & REM  \\ \hline
Division by zero       & $x$           & 0       & $2^{XLEN}-1$ & $x$  & $-1$          & $x$  \\
Overflow (signed only) & $-2^{XLEN-1}$ & $-1$    & --           & --   & $-2^{XLEN-1}$ & 0    \\
\hline
\end{tabular}
\caption{Semantics for division by zero and division overflow.}
\label{tab:divby0}
\end{table}

\begin{commentary}
We considered raising exceptions on integer divide by zero, with these
exceptions causing a trap in most execution environments.  However,
this would be the only arithmetic trap in the standard ISA
(floating-point exceptions set flags and write default values, but do
not cause traps) and would require language implementors to interact
with the execution environment's trap handlers for this case.
Further, where language standards mandate that a divide-by-zero
exception must cause an immediate control flow change, only a single
branch instruction needs to be added to each divide operation, and
this branch instruction can be inserted after the divide and should
normally be very predictably not taken, adding little runtime
overhead.

The value of all bits set is returned for both unsigned and signed
divide by zero to simplify the divider circuitry.  The value of all 1s
is both the natural value to return for unsigned divide, representing
the largest unsigned number, and also the natural result for simple
unsigned divider implementations.  Signed division is often
implemented using an unsigned division circuit and specifying the same
overflow result simplifies the hardware.
\end{commentary}

DIVW and DIVUW instructions are only valid for RV64, and divide the
lower 32 bits of {\em rs1} by the lower 32 bits of {\em rs2}, treating
them as signed and unsigned integers respectively, placing the 32-bit
quotient in {\em rd}, sign-extended to 64 bits.  REMW and REMUW
instructions are only valid for RV64, and provide the corresponding
signed and unsigned remainder operations respectively. Both REMW and
REMUW always sign-extend the 32-bit result to 64 bits, including on a
divide by zero.


\documentclass[notes,11pt, aspectratio=169]{beamer}
\usepackage[default]{lato}

\input{lec_style.tex}

%----------------------------------------------------------------------------------------
  %	TITLE PAGE
%----------------------------------------------------------------------------------------
\title[DAR]{Data Analytics with R}  % The short title appears at the bottom of
\author{Sumit Mishra} % Your name
\institute[IFMR] % Your institution as it will appear on the bottom of every slide, may be shorthand to save space
{
  Institute for Financial Management and Research, Sri City \\ % Your institution for the title page
  \medskip
  \medskip
  \textbf{Inference for Categorical Data} % Your email address
}
\date{05 December 2020} % Date, can be changed to a custom date

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  % Begin document
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Title page
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

 {
    \addtocounter{framenumber}{-1}
    {\removepagenumbers
      %\usebackgroundtemplate{\includegraphics[width=\paperwidth]{../OpenIntro_Grid_4_3-01.jpg}}
      \begin{frame}

      %\hfill \includegraphics[width=20mm]{../oiLogo_highres}

      \titlepage

      \end{frame}
    }
  }

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Sections
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Inference for a single proportion}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Access to Toilet}

A survey asks the question about access to toilet. Below is the distribution of responses from a 2018 survey: \\

\begin{center}
\begin{tabular}{l c}
Households with toilet		& 320 \\
Households without toilet	& 350 \\
\hline
Total						& 670
\end{tabular}
\end{center}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Parameter and point estimate}

\dq{We would like to estimate the proportion of Indian households which have access to toilet? What are the parameter of interest and the point estimate?}

\pause

\begin{itemize}

\item \hl{Parameter of interest:} Proportion of \orange{all} Indian households who have access to toilet.
\[ \mathhl{p}~\scriptsize{(\text{a population proportion})} \]

\pause

\item \hl{Point estimate:} Proportion of \orange{sampled} Indians who have good intuition about experimental design.
\[ \mathhl{\hat{p}}~\scriptsize{(\text{a sample proportion})} \]

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Inference on a proportion}

\dq{What percent of Indian households have access to toilet?}

\pause

\begin{itemize}

\item We can answer this question using a confidence interval, which we know is always of the form

\[ \textcolor{orange}{point~estimate \pm ME} \]

\pause

\item And we also know that \textcolor{orange}{$ME = critical~value \times standard~error$} of the point estimate.

\end{itemize}

\pause

\formula{Standard error of a sample proportion}
{
\[ SE_{\hat{p}} =  \sqrt{\frac{p~(1-p)}{n}}  \]
}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Identifying when a sample proportion is nearly normal}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Sample proportions are also nearly normally distributed}

\formula{Central limit theorem for proportions}
{
Sample proportions will be nearly normally distributed with mean equal to the population proportion, $p$, and standard error equal to $\sqrt{\frac{p~(1-p)}{n}}$.
\[ \hat{p} \sim N \pr{ mean = p, SE = \sqrt{\frac{p~(1-p)}{n}} } \]
}

But of course this is true only under certain conditions...

\dq{Any guesses?}

\soln{\pause{Independent observations, at least 10 successes and 10 failures}}

\pause

\Note{If $p$ is unknown (most cases), we use $\hat{p}$ in the calculation of the standard error.}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%\begin{frame}
%\frametitle{A quick activity}
%
%\pq{Flip a coin 20 times -- this is going to get loud! Record the number of heads and then use your clicker to submit the proportion of heads you obtained.}
%
%\begin{enumerate}[(a)]
%\item $\le$ 0.3
%\item 0.4
%\item 0.5
%\item 0.6
%\item $\ge$ 0.7
%\end{enumerate}
%
%\end{frame}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%\begin{frame}
%
%\formula{Central limit theorem for proportions}
%{
%Sample proportions will be nearly normally distributed with mean equal to the population mean, $p$, and standard error equal to $\sqrt{\frac{p~(1-p)}{n}}$.
%\[ \hat{p} \sim N \pr{ mean = p, SE = \sqrt{\frac{p~(1-p)}{n}} } \]
%If the population proportion is not available, use the sample proportion to estimate the standard error.
%}
%
%\pause
%
%Assumptions/conditions:
%\begin{enumerate}[1.]
%\item \hl{Independence}:
%\begin{itemize}
%\item \hlGr{Random sample}
%\item \hlGr{10\% condition}: If sampling without replacement, $n <$ 10\% of the population.
%\end{itemize}
%\item \hl{Normality}: At least 10 successes and 10 failures.
%\end{enumerate}
%
%\end{frame}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Confidence intervals for a proportion}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Back to experimental design...}

\dq{The survey found that 320 out of 670 (48\%) households have access to toilet. Estimate (using a 95\% confidence interval) the proportion of all Indian households which have access to toilet?}

\pause
Given: $n = 670, \hat{p} = 0.48$. First, check conditions.

\pause
\begin{enumerate}[1.]
\item \hl{Independence}: The sample is random, and 670 $<$ 10\% of all Indian households, therefore we can assume that one respondent's response is independent of another.
\pause
\item \hl{Success-failure}: 320 households have access to toilet (successes) and 350 failures, both are greater than 10.
\end{enumerate}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}

\pq{We are given that  $n = 670, \hat{p} = 0.48$, we also just learned that the standard error of the sample proportion is $SE = \sqrt{\frac{p(1-p)}{n}}$. Which of the below is the correct calculation of the 95\% confidence interval?}

\begin{enumerate}[(a)]
\solnMult{$0.48 \pm 1.96 \times \sqrt{\frac{0.48 \times 0.52}{670}}$} \soln{\only<2>{\orange{$\rightarrow (0.44, 0.52)$}}}
\item $0.48 \pm 1.65 \times \sqrt{\frac{0.48 \times 0.52}{670}}$
\item $0.48 \pm 1.96 \times \frac{0.48 \times 0.52}{\sqrt{670}}$
\item $320 \pm 1.96 \times \sqrt{\frac{320 \times 350}{670}}$
\end{enumerate}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%\begin{frame}
%\frametitle{Interpretation of the CI}
%
%\pq{Based on this confidence interval, does it appear that more than a 80\% of Americans have a good intuition on experimental design? \\
%\soln{(0.82, 0.88)}
%}
%
%\begin{enumerate}[(a)]
%\solnMult{Yes}
%\item No
%\item Cannot tell
%\end{enumerate}
%
%\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Choosing a sample size when estimating a proportion}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Choosing a sample size}

\dq{How many households should you sample in order to cut the margin of error of a 95\% confidence interval down to 1\%.}
\pause
\[ ME = z^\star \times SE\]
\pause
\begin{eqnarray*}
0.01 &\ge& 1.96 \times \sqrt{\frac{0.48 \times 0.52}{n}} \orange{$\rightarrow$ \text{Use $\hat{p}$ from previous study}} \\
\pause
0.01^2 &\ge& 1.96^2 \times \frac{0.48 \times 0.52}{n} \\
\pause
n &\ge& \frac{1.96^2 \times 0.48 \times 0.52}{0.01^2} \\
\pause
n &\ge& 9584.7 \pause \orange{~$\rightarrow$ n \text{ should be at least 9585}}
\end{eqnarray*}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{What if there isn't a previous study?}

... use $\hat{p} = 0.5$

\vspace{1cm}

\dq{why?}
\pause

\begin{itemize}
\item if you don't know any better, 50-50 is a good guess
\pause
\item $\hat{p} = 0.5$ gives the most conservative estimate -- highest possible sample size
\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Hypothesis testing for a proportion}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{CI vs. HT for proportions}

\begin{itemize}

\item Success-failure condition:
\begin{itemize}
\item CI: At least 10 \orange{observed} successes and failures
\item HT: At least 10 \orange{expected} successes and failures, calculated using the null value
\end{itemize}

\item Standard error:
\begin{itemize}
\item CI: calculate using observed sample proportion: $SE = \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$
\item HT: calculate using the null value: $SE = \sqrt{\frac{p_0(1-p_0)}{n}}$
\end{itemize}

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{}

\dq{The survey found that 320 out of 670 (48\%) Indian households have access to toilet. Do these data provide convincing evidence that more than 40\% of Indians have access to toilet?}

\pause

\[ H_0: p = 0.4 \qquad H_A: p > 0.4 \]

\twocol{0.6}{0.4}
{
\pause
\begin{eqnarray*}
SE &=& \sqrt{\frac{0.4 \times 0.6}{670}} = 0.019 \\
\pause
Z &=& \frac{0.48 - 0.4}{0.019} = 4.22 \\
\pause
p-value &=& 1 - 0.999982 \approx 0 \\
\end{eqnarray*}
}
\pause
Since the p-value is low, we reject $H_0$. The data provide convincing evidence that more than 40\% of Indian households have access to toilet.

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Recap}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{}

\pq{11\% of 1,001 Americans responding to a 2006 Gallup survey stated that they have objections to celebrating Halloween on religious grounds. At 95\% confidence level, the margin of error for this survey is $\pm 3\%$. A news piece on this study's findings states: ``More than 10\% of all Americans have objections on religious grounds to celebrating Halloween." At 95\% confidence level, is this news piece's statement justified?}

\begin{enumerate}[(a)]
\item Yes
\solnMult{No}
\item Cannot tell
\end{enumerate}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Recap - inference for one proportion}

\begin{itemize}

\item Population parameter: $p$, point estimate: $\hat{p}$

\pause

\item Conditions:
\begin{itemize}
\item independence \\
- random sample and 10\% condition
\item at least 10 successes and failures\\ - if not $\rightarrow$ randomization
\end{itemize}

\pause

\item Standard error: $SE = \sqrt{ \frac{p(1-p)}{n} }$
\begin{itemize}
\item for CI: use $\hat{p}$
\item for HT: use $p_0$
\end{itemize}

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Difference of two proportions}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Results from a Survey}

A small pan-India survey asks a question about open defecation in India. Below are the distributions of responses from the survey as well as from a small survey conducted in villages in Tada: \\

\begin{center}
\begin{tabular}{l r r}
\hline
				& National Survey	& Village Survey \\
\hline
Frequently		& 454	& 69 \\
Sometimes			& 124 	& 30\\
A little			& 52 		& 4\\
Not at all			& 50 		& 2 \\
\hline
Total				& 680 	& 105\\
\hline
\end{tabular}
\end{center}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Parameter and point estimate}

\begin{itemize}

\item \hl{Parameter of interest:} Difference between the proportions of \orange{all} Tada residents and \orange{all} Indians who practice open defecation frequently.
\[ \mathhl{ p_{Tada} - p_{Ind} }\]

\pause

\item \hl{Point estimate:} Difference between the proportions of \orange{sampled} Tada residents and \orange{sampled} Indians who practice open defecation frequently.
\[ \mathhl{ \hat{p}_{Tada} - \hat{p}_{Ind} }\]

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Inference for comparing proportions}

\begin{itemize}

\item The details are the same as before...

\pause

\item CI: \textcolor{orange}{$point~estimate \pm margin~of~error$}

\pause

\item HT: Use \textcolor{orange}{$Z = \frac{point~estimate - null~value}{SE}$} to find appropriate p-value.

\pause

\item We just need the appropriate standard error of the point estimate ($SE_{ \hat{p}_{Tada} - \hat{p}_{Ind}}$), which is the only new concept.

\end{itemize}

\pause

\formula{Standard error of the difference between two sample proportions}
{
\[ SE_{(\hat{p}_1 - \hat{p}_2)} = \sqrt{ \frac{p_1(1-p_1)}{n_1} + \frac{p_2(1-p_2)}{n_2} } \]
}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Confidence intervals for difference of proportions}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Conditions for CI for difference of proportions}

\begin{enumerate}

\item \hl{Independence within groups: }
\begin{itemize}
\item The pan-India group is sampled randomly and we're assuming that the Tada group represents a random sample as well.
\pause
\item 105 $<$ 10\% of all Tada residents and 680 $<$ 10\% of all Indians.
\end{itemize}
\pause
We can assume that the attitudes of people in Tada in the sample are independent of each other, and attitudes of all Indians in the sample are independent of each other as well.

\pause

\item \hl{Independence between groups: }
The sampled respondents from Tada and the Indian residents are independent of each other.

\pause

\item \hl{Success-failure:} \\
At least 10 observed successes and 10 observed failures in the two groups.

\end{enumerate}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{}

\dq{Construct a 95\% confidence interval for the difference between the proportions of Tada respondents and Indians who practice open defecation frequently \orange{($p_{Tada} - p_{Ind}$)}.}

{\footnotesize
\begin{center}
\begin{tabular}{l | c c}
Data			& Tada		& India \\
\hline
Frequently	 & 69			& 454 \\
Not frequently  & 36			& 226 \\
\hline
Total	      & 105		& 680 \\
\hline
\pause
$\hat{p}$		& 0.657		& 0.668
\end{tabular}
\end{center}
}

\pause

\soln{
\begin{eqnarray*}
&& (\hat{p}_{Tada} - \hat{p}_{Ind}) \pm z^\star \times \sqrt{ \frac{ \hat{p}_{Tada} (1 - \hat{p}_{Tada})}{n_{Tada} } + \frac{ \hat{p}_{Ind} (1 -  \hat{p}_{Ind})}{n_{Ind} } }  \\
\pause
&=& (0.657 - 0.668) \pause \pm 1.96 \pause \times \sqrt{ \frac{0.657 \times 0.343}{105} + \frac{0.668 \times 0.332}{680} } \\
\pause
&=& -0.011 \pm \pause 1.96 \times 0.0497 \\
\pause
&=& -0.011 \pm 0.097 \\
\pause
&=& (-0.108, 0.086)
\end{eqnarray*}
}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{HT for comparing proportions}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}

\pq{Which of the following is the correct set of hypotheses for testing if the proportion of all Tada residents who practice open defecation frequently differs from the proportion of all Indians who do?}

\begin{enumerate}[(a)]
\solnMult{ $H_0:  p_{Tada} = p_{Ind}$ \\
$H_A:  p_{Tada} \ne p_{Ind}$ }
\item $H_0:  \hat{p}_{Tada} = \hat{p}_{Ind}$ \\
$H_A:  \hat{p}_{Tada} \ne \hat{p}_{Ind}$
\solnMult{ $H_0:  p_{Tada} - p_{Ind} = 0$ \\
$H_A:  p_{Tada} - p_{Ind} \ne 0$ }
\item $H_0:  p_{Tada} = p_{Ind}$ \\
$H_A:  p_{Tada} < p_{Ind}$
\end{enumerate}

\soln{
\only<2>{\orange{Both (a) and (c) are correct.}}
}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Flashback to working with one proportion}

\begin{itemize}

\item When constructing a confidence interval for a population proportion, we check if the \orange{observed} number of successes and failures are at least 10.
\[ n\hat{p} \ge 10 \qquad \qquad n(1-\hat{p}) \ge 10 \]

\pause

\item When conducting a hypothesis test for a population proportion, we check if the \orange{expected} number of successes and failures are at least 10.
\[ np_0 \ge 10 \qquad \qquad n(1-p_0) \ge 10 \]

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Pooled estimate of a proportion}

\begin{itemize}

\item In the case of comparing two proportions where $H_0: p_1 = p_2$, there isn't a given null value we can use to calculated the \orange{expected} number of successes and failures in each sample.

\pause

\item Therefore, we need to first find a common (\hl{pooled}) proportion for the two groups, and use that in our analysis.

\pause

\item This simply means finding the proportion of total successes among the total number of observations.

\end{itemize}

$\:$ \\

\formula{Pooled estimate of a proportion}
{ \[ \hat{p} = \frac{\#~of~successes_1 + \#~of~successes_2}{n_1 + n_2} \] }

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{}

\dq{Calculate the estimated \underline{pooled proportion} of Tada residents and Indians who practice open defecation. Which sample proportion ($\hat{p}_{Tada}$ or $\hat{p}_{Ind}$) the pooled estimate is closer to? Why?}

{\footnotesize
\begin{center}
\begin{tabular}{l | c c}
Data			& Tada		& India \\
\hline
Frequently	& 69			& 454 \\
Not frequently     & 36			& 226 \\
\hline
Total			& 105		& 680 \\
\hline
$\hat{p}$		& 0.657		& 0.668
\end{tabular}
\end{center}
}

\pause

\soln{
\begin{eqnarray*}
\hat{p} &=& \frac{\#~of~successes_1 + \#~of~successes_2}{n_1 + n_2} \\
\pause
&=& \frac{69+454}{105+680} \pause = \frac{523}{785} \pause = 0.666
\end{eqnarray*}
}

\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{}

\dq{Do these data suggest that the proportion of all Tada residents who practice open defecation differs from the proportion of all Indians who do? Calculate the test statistic, the p-value, and interpret your conclusion in context of the data.}

{\footnotesize
\begin{center}
\begin{tabular}{l | c c}
Data			& Tada		& India \\
\hline
Frequently	& 69			& 454 \\
Not frequently & 36			& 226 \\
\hline
Total			& 105		& 680 \\
\hline
$\hat{p}$		& 0.657		& 0.668
\end{tabular}
\end{center}
}

\pause

\soln{
\begin{eqnarray*}
Z &=& \frac{(\hat{p}_{Tada} - \hat{p}_{Ind})}{\sqrt{ \frac{ \hat{p} (1 - \hat{p})}{n_{Tada} } + \frac{ \hat{p} (1 -  \hat{p})}{n_{Ind} } }} \\
\pause
&=& \frac{(0.657 - 0.668)}{\sqrt{ \frac{0.666 \times 0.334}{105} + \frac{0.666 \times 0.334}{680} }} = \pause \frac{-0.011}{0.0495} \pause = -0.22 \\
\pause
p-value &=& 2 \times P(Z < -0.22) \pause = 2 \times 0.41 = 0.82
\end{eqnarray*}
}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Recap}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Recap - comparing two proportions}

\begin{itemize}

\item Population parameter: $(p_1 - p_2)$, point estimate: $(\hat{p}_1 - \hat{p}_2)$

\pause

\item Conditions:
\pause
\begin{itemize}
\item independence within groups \\
- random sample and 10\% condition met for both groups
\item independence between groups
\item at least 10 successes and failures in each group\\
- if not $\rightarrow$ randomization (Section 6.4)
\end{itemize}

\pause

\item $SE_{(\hat{p}_1 - \hat{p}_2)} = \sqrt{ \frac{p_1(1-p_1)}{n_1} + \frac{p_2(1-p_2)}{n_2} }$
\begin{itemize}
\item for CI: use $\hat{p}_1$ and $\hat{p}_2$
\item for HT:
\begin{itemize}
\item when $H_0: p_1 = p_2$: use $\hat{p}_{pool} = \frac{\#~suc_1 + \#suc_2}{n_1 + n_2}$
\item when $H_0: p_1 - p_2 = $ \textit{(some value other than 0)}: use $\hat{p}_1$ and $\hat{p}_2$ \\
- this is pretty rare
\end{itemize}
\end{itemize}

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Reference - standard error calculations}

\begin{center}
\begin{tabular}{l | l | l}
			& one sample					& two samples \\
\hline
& & \\
mean		& $SE = \frac{s}{\sqrt{n}}$			& $SE = \sqrt{ \frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}$ \\
& & \\
\hline
& & \\
proportion		& $SE = \sqrt{ \frac{p(1-p)}{n} }$	& $SE = \sqrt{ \frac{p_1(1-p_1)}{n_1} + \frac{p_2(1-p_2)}{n_2} }$	 \\
& & \\
\end{tabular}
\end{center}

\pause

\begin{itemize}

\item When working with means, it's very rare that $\sigma$ is known, so we usually use $s$.

\pause

\item When working with proportions,
\begin{itemize}
\item if doing a hypothesis test, $p$ comes from the null hypothesis
\item if constructing a confidence interval, use $\hat{p}$ instead
\end{itemize}

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Chi-square test of GOF}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Weldon's dice}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Weldon's dice}

\twocol{0.75}{0.25}
{
\begin{itemize}

\item Walter Frank Raphael Weldon (1860 - 1906), was an English evolutionary biologist and a founder of biometry. He was the joint founding editor of Biometrika, with Francis Galton and Karl Pearson.

\item In 1894, he rolled 12 dice 26,306 times, and recorded the number of 5s or 6s (which he considered to be a success).

\end{itemize}
}
{
\begin{center}
\includegraphics[width=\textwidth]{graphs/weldon}
\end{center}
}
\begin{itemize}

\item It was observed that 5s or 6s occurred more often than expected, and Pearson hypothesized that this was probably due to the construction of the dice. Most inexpensive dice have hollowed-out pips, and since opposite sides add to 7, the face with 6 pips is lighter than its opposing face, which has only 1 pip.

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Labby's dice}

\twocol{0.6}{0.4}
{
\begin{itemize}

\item In 2009, Zacariah Labby (U of Chicago), repeated Weldon's experiment using a homemade dice-throwing, pip counting machine.
\begin{center}
\webURL{http://www.youtube.com/watch?v=95EErdouO2w}
\end{center}

\item The rolling-imaging process took about 20 seconds per roll.

\end{itemize}
}
{
\begin{center}
\includegraphics[width=\textwidth]{graphs/labby}
\end{center}
}

\begin{itemize}

\item Each day there were $\sim$150 images to process manually.

\item At this rate Weldon's experiment was repeated in a little more than six full days.

\item Recommended reading: \webURL{http://galton.uchicago.edu/about/docs/labby09dice.pdf}

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Labby's dice (cont.)}

\begin{itemize}

\item Labby did not actually observe the same phenomenon that Weldon observed (higher frequency of 5s and 6s).

\item Automation allowed Labby to collect more data than Weldon did in 1894, instead of recording ``successes" and ``failures", Labby recorded the individual number of pips on each die.

\end{itemize}

\begin{center}
\includegraphics[width=0.3\textwidth]{graphs/labbyPipCounts}
\end{center}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection*{Creating a test statistic for one-way tables}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Expected counts}

\pq{Labby rolled 12 dice 26,306 times. If each side is equally likely to come up, how many 1s, 2s, $\cdots$, 6s would he expect to have observed?}

\begin{enumerate}[(a)]
\item $\frac{1}{6}$
\item $\frac{12}{6}$
\item $\frac{26,306}{6}$
\solnMult{ $\frac{12 \times 26,306}{6}$ } \soln{\only<2>{\orange{$= 52,612$}}}
\end{enumerate}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Summarizing Labby's results}

The table below shows the observed and expected counts from Labby's experiment.

{\small
\begin{center}
\renewcommand\arraystretch{1.2}
\begin{tabular}{c | c c}
Outcome	& Observed	& Expected \\
\hline
1		& 53,222		& 52,612 \\
2		& 52,118		& 52,612 \\
3		& 52,465		& 52,612 \\
4		& 52,338		& 52,612 \\
5		& 52,244		& 52,612 \\
6		& 53,285		& 52,612 \\
\hline
Total		& 315,672		& 315,672
\end{tabular}
\end{center}
}

\pause
\dq{Why are the expected counts the same for all outcomes but the observed counts are different? At a first glance, does there appear to be an inconsistency between the observed and expected counts?}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Setting the hypotheses}

\dq{Do these data provide convincing evidence of an inconsistency between the observed and expected counts?}

\pause

\begin{itemize}
\item[$H_0$:] There is no inconsistency between the observed and the expected counts. \hlGr{The observed counts follow the same distribution as the expected counts.}

\pause

\item[$H_A$:] There is an inconsistency between the observed and the expected counts. \hlGr{The observed counts \orange{do not} follow the same distribution as the expected counts.} There is a bias in which side comes up on the roll of a die.
\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Evaluating the hypotheses}

\begin{itemize}

\item To evaluate these hypotheses, we quantify how different the observed counts are from the expected counts.

\pause

\item Large deviations from what would be expected based on sampling variation (chance) alone provide strong evidence for the alternative hypothesis.

\pause

\item This is called a \hl{goodness of fit} test since we're evaluating how well the observed data fit the expected distribution.

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{The chi-square test statistic}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Anatomy of a test statistic}

\begin{itemize}

\item The general form of a test statistic is
\[ \frac{\text{point estimate} - \text{null value}}{\text{SE of point estimate}} \]

\pause

\item This construction is based on
\begin{enumerate}
\item identifying the difference between a point estimate and an expected value if the null hypothesis was true, and
\item standardizing that difference using the standard error of the point estimate.
\end{enumerate}
\pause

These two ideas will help in the construction of an appropriate test statistic for count data.

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Chi-square statistic}

When dealing with counts and investigating how far the observed counts are from the expected counts, we use a new test statistic called the \hl{chi-square ($\chi^2$) statistic}.

$\:$ \\

\pause

\formula{$\chi^2$ statistic}
{
\[\chi^2 = \sum_{i = 1}^k \frac{(O - E)^2}{E} \qquad \text{where $k$ = total number of cells} \]
}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Calculating the chi-square statistic}

\begin{center}
\renewcommand\arraystretch{1.8}
\begin{tabular}{c | c c | c}
Outcome	& Observed	& Expected 	& $\frac{(O - E)^2}{E}$\\
\hline
1		& 53,222		& 52,612 		& $\frac{(53,222 - 52,612)^2}{52,612} = 7.07$ \\
\pause
2		& 52,118		& 52,612 		& $\frac{(52,118 - 52,612)^2}{52,612} = 4.64$ \\
\pause
3		& 52,465		& 52,612 		& $\frac{(52,465 - 52,612)^2}{52,612} = 0.41$ \\
\pause
4		& 52,338		& 52,612 		& $\frac{(52,338 - 52,612)^2}{52,612} = 1.43$\\
\pause
5		& 52,244		& 52,612 		& $\frac{(52,244 - 52,612)^2}{52,612} = 2.57$\\
\pause
6		& 53,285		& 52,612 		& $\frac{(53,285 - 52,612)^2}{52,612} = 8.61$\ \\
\hline
\pause
Total		& 315,672		& 315,672		& 24.73
\end{tabular}
\end{center}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Why square?}

Squaring the difference between the observed and the expected outcome does two things:
\pause
\begin{itemize}
\item Any standardized difference that is squared will now be positive.
\pause
\item Differences that already looked unusual will become much larger after being squared.
\end{itemize}

\vspace{1cm}

\pause
\dq{When have we seen this before?}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{The chi-square distribution and finding areas}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{The chi-square distribution}

\begin{itemize}

\item In order to determine if the $\chi^2$ statistic we calculated is considered unusually high or not we need to first describe its distribution.

\pause

\item The chi-square distribution has just one parameter called \hl{degrees of freedom (df)}, which influences the shape, center, and spread of the distribution. \\

\end{itemize}

\pause

$\:$ \\

\Remember{So far we've seen three other continuous distributions:
\begin{itemize}
\item[-] normal distribution: unimodal and symmetric with two parameters: mean and standard deviation
\item[-] T distribution: unimodal and symmetric with one parameter: degrees of freedom
\item[-] F distribution: unimodal and right skewed with two parameters: degrees of freedom or numerator (between group variance) and denominator (within group variance)
\end{itemize}
}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{}

\pq{Which of the following is false?}

\begin{center}
\includegraphics[width=0.7\textwidth]{graphs/chiSquareDistributionWithInceasingDF}
\end{center}

As the df increases,
\begin{enumerate}[(a)]
\item the center of the $\chi^2$ distribution increases as well
\item the variability of the $\chi^2$ distribution increases as well
\solnMult{the shape of the $\chi^2$ distribution becomes more skewed (less like a normal)}
\end{enumerate}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}[fragile]
\frametitle{Finding areas under the chi-square curve}

\begin{itemize}

\item p-value = tail area under the chi-square distribution (as usual)

\pause

\item For this we can use technology,or a chi-square probability table.

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}[fragile]
\frametitle{Finding areas under the chi-square curve (cont.)}

\dq{Estimate the shaded area (above the cutoff value of 10) under the $\chi^2$ curve with $df = 6$.}

\pause

\begin{verbatim}

> pchisq(q = 10, df = 6, lower.tail = FALSE)

[1] 0.124652

\end{verbatim}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}[fragile]
\frametitle{Finding areas under the chi-square curve (cont.)}

\pq{Estimate the shaded area (above the cutoff value of 17) under the $\chi^2$ curve with $df = 9$.}

\twocol{0.6}{0.4}{
\begin{center}
\includegraphics[width=0.67\textwidth]{graphs/above17WithDF9}
\end{center}
}
{
{\small
\begin{enumerate}[(a)]
\setlength{\itemsep}{0in}
\item 0.05
\item 0.02
\solnMult{between 0.02 and 0.05}
\item between 0.05 and 0.1
\item between 0.01 and 0.02
\end{enumerate}
}
}

\pause

\begin{verbatim}

> pchisq(q = 17, df = 9, lower.tail = FALSE)

[1] 0.04871598

\end{verbatim}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}[fragile]
\frametitle{Finding areas under the chi-square curve (one more)}

\pq{Estimate the shaded area (above 30) under the $\chi^2$ curve with $df = 10$.}

\twocol{0.6}{0.4}{
\begin{center}
\includegraphics[width=0.67\textwidth]{graphs/above30WithDF10}
\end{center}
}
{
{\small
\begin{enumerate}[(a)]
\setlength{\itemsep}{0in}
\item greater than 0.3
\item between 0.005 and 0.001
\solnMult{less than 0.001}
\item greater than 0.001
\item cannot tell using this table
\end{enumerate}
}
}

\pause

\begin{verbatim}

> pchisq(q = 30, df = 10, lower.tail = FALSE)

[1] 0.0008566412

\end{verbatim}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Finding a p-value for a chi-square test}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Back to Labby's dice}

\begin{itemize}

\item The research question was: Do these data provide convincing evidence of an inconsistency between the observed and expected counts?

\pause

\item The hypotheses were:
\begin{itemize}
\item[$H_0$:] There is no inconsistency between the observed and the expected counts. The observed counts follow the same distribution as the expected counts.
\item[$H_A$:] There is an inconsistency between the observed and the expected counts. The observed counts \orange{do not} follow the same distribution as the expected counts. There is a bias in which side comes up on the roll of a die.
\end{itemize}

\pause

\item We had calculated a test statistic of \orange{$\chi^2 = 24.67$}.

\pause

\item All we need is the $df$ and we can calculate the tail area (the p-value) and make a decision on the hypotheses.

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Degrees of freedom for a goodness of fit test}

\begin{itemize}

\item When conducting a goodness of fit test to evaluate how well the observed data follow an expected distribution, the degrees of freedom are calculated as the number of cells ($k$) minus 1.
\[ \mathhl{df = k - 1} \]

\pause

\item For dice outcomes, $k = 6$, therefore
\[ df = 6 - 1 = 5 \]

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Finding a p-value for a chi-square test}

The \hl{p-value} for a chi-square test is defined as the \hl{tail area above the calculated test statistic}.

\twocol{0.6}{0.4}{
\begin{center}
\includegraphics[width=0.67\textwidth]{graphs/above24Point67WithDF5}
\end{center}
}
{
p-value = $P(\chi^2_{df = 5} > 24.67)$\\ is less than 0.001
}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Conclusion of the hypothesis test}

\pq{We calculated a p-value less than 0.001. At 5\% significance level, what is the conclusion of the hypothesis test?}

\begin{enumerate}[(a)]
\item Reject $H_0$, the data provide convincing evidence that the dice are fair.
\solnMult{Reject $H_0$, the data provide convincing evidence that the dice are biased.}
\item Fail to reject $H_0$, the data provide convincing evidence that the dice are fair.
\item Fail to reject $H_0$, the data provide convincing evidence that the dice are biased.
\end{enumerate}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Turns out...}

\begin{itemize}

\item The 1-6 axis is consistently shorter than the other two (2-5 and 3-4), thereby supporting the hypothesis that the faces with one and six pips are larger than the other faces.

\item Pearson's claim that 5s and 6s appear more often due to the carved-out pips is not supported by these data.

\item Dice used in casinos have flush faces, where the pips are filled in with a plastic of the same density as the surrounding material and are precisely balanced.

\end{itemize}

\begin{center}
\includegraphics[width=0.3\textwidth]{graphs/regular}
\includegraphics[width=0.3\textwidth]{graphs/casino}
\end{center}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Recap: p-value for a chi-square test}

\begin{itemize}

\item The p-value for a chi-square test is defined as the tail area \hl{above} the calculated test statistic.

\item This is because the test statistic is always positive, and a higher test statistic means a stronger deviation from the null hypothesis.

\end{itemize}

\begin{center}
\includegraphics[width=0.7\textwidth]{graphs/genericChiSquare}
\end{center}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Conditions for the chi-square test}

\begin{enumerate}

\item \hlGr{Independence:} Each case that contributes a count to the table must be independent of all the other cases in the table.

\pause

\item \hlGr{Sample size:} Each particular scenario (i.e. cell) must have at least 5 \orange{expected} cases.

\pause

\item \hlGr{df $>$ 1:} Degrees of freedom must be greater than 1.

\end{enumerate}

\pause

Failing to check conditions may unintentionally affect the test's error rates.

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{2009 Iran Election}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{2009 Iran Election}

\dq{There was lots of talk of election fraud in the 2009 Iran election. We'll compare the data from a poll conducted before the election (observed data) to the reported votes in the election to see if the two follow the same distribution.}

\begin{center}
\begin{tabular}{l | r r}
					& \footnotesize{Observed \# of} & \footnotesize{Reported \% of} \\
\footnotesize{Candidate}	& \footnotesize{voters in poll} & \footnotesize{votes in election} \\
\hline
(1) Ahmedinajad	& 338	& 63.29\% \\
(2) Mousavi		& 136	& 34.10\% \\
(3) Minor candidates	& 30	& 2.61\% \\
\hline
Total			& 504	& 100\% \\
\pause
			& \hl{$\downarrow$}	& \hl{$\downarrow$}	\\
			& \hl{observed}	& \hl{expected} \\
			& 			& \hl{distribution}
\end{tabular}
\end{center}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Hypotheses}

\dq{What are the hypotheses for testing if the distributions of reported and polled votes are different?}

\soln{
\only<2>{
\begin{itemize}
\item[$H_0$:] The observed counts from the poll follow the same distribution as the reported votes.
\item[$H_A$:] The observed counts from the poll do not follow the same distribution as the reported votes.
\end{itemize}
}}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Calculation of the test statistic}

{\small
\begin{center}
\begin{tabular}{l | r r r}
					& \footnotesize{Observed \# of} & \footnotesize{Reported \% of}	& \footnotesize{Expected \# of} \\
\footnotesize{Candidate}	& \footnotesize{voters in poll} & \footnotesize{votes in election}		&  \footnotesize{votes in poll} \\
\hline
\footnotesize{(1) Ahmedinajad}	& 338	& 63.29\% 	& 504 $\times$ 0.6329 = 319 \\
\footnotesize{(2) Mousavi}		& 136	& 34.10\%		& 504 $\times$ 0.3410 = 172 \\
\footnotesize{(3) Minor candidates}	& 30	& 2.61\% 		& 504 $\times$ 0.0261 = 13\\
\hline
Total			& 504	& 100\%		& 504
\end{tabular}
\end{center}
}

\pause

\begin{eqnarray*}
\frac{(O_1 - E_1)^2}{E_1} = \frac{(338 - 319)^2}{319} &=& 1.13 \\
\pause
\frac{(O_2 - E_2)^2}{E_2} = \frac{(136 - 172)^2}{172} &=& 7.53 \\
\pause
\frac{(O_2 - E_2)^2}{E_2} = \frac{(30 - 13)^2}{13} &=& 22.23 \\
\pause
 \chi^2_{\mathhl{df = 3 - 1 = 2}} &=& 30.89
\end{eqnarray*}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Conclusion}

\pq{Based on these calculations what is the conclusion of the hypothesis test?}

\begin{enumerate}[(a)]
\solnMult{p-value is low, $H_0$ is rejected. The observed counts from the poll do \underline{not} follow the same distribution as the reported votes.}
\item p-value is high, $H_0$ is not rejected. The observed counts from the poll follow the same distribution as the reported votes.
\item p-value is low, $H_0$ is rejected. The observed counts from the poll follow the same distribution as the reported votes
\item p-value is low, $H_0$ is not rejected. The observed counts from the poll do \underline{not} follow the same distribution as the reported votes.
\end{enumerate}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Chi-square test of independence}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Popular kids}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Popular kids}

\dq{In the dataset \texttt{popular}, students in grades 4-6 were asked whether good grades, athletic ability, or popularity was most important to them. A two-way table separating the students by grade and by choice of most important factor is shown below. Do these data provide evidence to suggest that goals vary by grade?}

\twocol{0.5}{0.5}
{
\begin{center}
\begin{tabular}{rrrr}
  \hline
 & Grades & Popular & Sports \\
  \hline
$4^{th}$ &  63 &  31 &  25 \\
$5^{th}$ &  88 &  55 &  33 \\
$6^{th}$ &  96 &  55 &  32 \\
   \hline
\end{tabular}
\end{center}
}
{
\begin{center}
\includegraphics[width=0.5\textwidth]{graphs/popular_mosaic}
\end{center}
}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Chi-square test of independence}

\begin{itemize}
\item The hypotheses are:
\begin{itemize}
\item[$H_0$:] Grade and goals are independent. Goals do not vary by grade.
\item[$H_A$:] Grade and goals are dependent. Goals vary by grade.
\end{itemize}

\pause

\item The test statistic is calculated as
\[ \chi^2_{df} = \sum_{i = 1}^{k} \frac{(O - E)^2}{E} \quad \text{ where } \quad df = (R - 1) \times (C - 1), \]
where $k$ is the number of cells, $R$ is the number of rows, and $C$ is the number of columns.

\Note{We calculate $df$ differently for one-way and two-way tables.}

\pause

\item The p-value is the area under the $\chi^2_{df}$ curve, above the calculated test statistic.

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Expected counts in two-way tables}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Expected counts in two-way tables}

\formula{Expected counts in two-way tables}
{
\[ \text{Expected Count} = \frac{(\text{row total}) \times (\text{column total})}{\text{table total}} \]
}

\pause

{\small
\begin{center}
\begin{tabular}{rrrr|r}
  \hline
 & Grades & Popular & Sports	& Total \\
  \hline
$4^{th}$ &  \orange{63} &  \green{31} &  25 	&119 \\
$5^{th}$ &  88 &  55 &  33	& 176 \\
$6^{th}$&  96 &  55 &  32	& 183 \\
   \hline
Total	& 247	& 141	& 90	& 478 \\
\end{tabular}
\end{center}
}

\pause

\[ \orange{$E_{row~1, col~1} = \frac{119 \times 247}{478} = 61$} \qquad \pause
 \green{$E_{row~1, col~2} = \frac{119 \times 141}{478} = 35$} \]

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Expected counts in two-way tables}

\pq{What is the expected count for the highlighted cell?}

{\small
\begin{center}
\begin{tabular}{rrrr|r}
  \hline
 & Grades & Popular & Sports	& Total \\
  \hline
$4^{th}$ &  63 &  31 &  25 	&119 \\
$5^{th}$ &  88 &  \orange{55} &  33	& 176 \\
$6^{th}$ &  96 &  55 &  32	& 183 \\
   \hline
Total	& 247	& 141	& 90	& 478 \\
\end{tabular}
\end{center}
}

\twocol{0.2}{0.8}
{
\begin{enumerate}[(a)]
\solnMult{$\frac{176 \times 141}{478}$}
\item $\frac{119 \times 141}{478}$
\item $\frac{176 \times 247}{478}$
\item $\frac{176 \times 478}{478}$
\end{enumerate}
}
{
\soln{\only<2>{
\orange{$\rightarrow$ 52\\
{\small more than expected \# of 5th graders \\
have a goal of being popular}}
\vspace{0.75cm}
}
}
}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Calculating the test statistic in two-way tables}

Expected counts are shown in \ex{blue} next to the observed counts.
\begin{center}
\begin{tabular}{rrrr|r}
  \hline
 & Grades & Popular & Sports	& Total \\
  \hline
$4^{th}$ 	&  63 \ex{61} &  31 \ex{35} &  25 \ex{23}	&119 \\
$5^{th}$ 	&  88 \ex{91} &  55 \ex{52} &  33 \ex{33}	& 176 \\
$6^{th}$	&  96 \ex{95} &  55 \ex{54} &  32 \ex{34}	& 183 \\
   \hline
Total	& 247	& 141	& 90	& 478 \\
\end{tabular}
\end{center}

\vspace{0.5cm}

\pause

\begin{eqnarray*}
\chi^2 &=& \sum \frac{(63 - 61)^2}{61} + \frac{(31 - 35)^2}{35} + \cdots + \frac{(32 - 34)^2}{34} = 1.3121 \\
\pause
df &=& (R - 1) \times (C - 1) = (3 - 1) \times (3 - 1) = 2 \times 2 = 4
\end{eqnarray*}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Results}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Calculating the p-value}

\pq{Which of the following is the correct p-value for this hypothesis test?
\[ \chi^2 = 1.3121 \qquad df = 4 \]
}

\twocol{0.6}{0.4}{
\begin{center}
\includegraphics[width=0.5\textwidth]{graphs/popular}
\end{center}
}
{
{\small
\begin{enumerate}[(a)]
\setlength{\itemsep}{0in}
\solnMult{ more than 0.3}
\item between 0.3 and 0.2
\item between 0.2 and 0.1
\item between 0.1 and 0.05
\item less than 0.001
\end{enumerate}
}
}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Conclusion}

\dq{Do these data provide evidence to suggest that goals vary by grade?}

\begin{itemize}

\item[$H_0$:] Grade and goals are independent. Goals do not vary by grade.

\item[$H_A$:] Grade and goals are dependent. Goals vary by grade. \\

\end{itemize}

$\:$ \\

\soln{\only<2>{Since p-value is high, we fail to reject $H_0$. The data do not provide convincing evidence that grade and goals are dependent. It doesn't appear that goals vary by grade.
}
}

\end{frame}

\end{document}


\chapter{Evaluation}
\label{chap:evaluation}

This chapter addresses the implementation details,
datasets, metrics and experiments with the demonstration of the achieved results.

\section{Training and validation data}
\label{sec:datasets}

Each training session consists of optimizing parameters of a separate neural representation from one of the proposed methods.
The dataset for each training session is created on one specific scene and with one specific light setting.
All of the used datasets have been created in an artificial environment using Blender \cite{blender}.
Each dataset consists of rendered RGB images of the scene.
Each image is an HDR image (the OpenEXR \cite{openexr} format is used to store HDR data on disk).
The dataset samples are the aforementioned images
that correspond to a view from a pin-hole camera with intrinsic parameters $\mathcal{I}$ (same for the whole dataset).
The camera is placed on a sphere around the scene,
up-vector is always positive in $Z$ axis,
the orientation is always aimed to the center of the scene.
% The position of the camera is described with two angles: azimuthal angle $\theta$ and inclination angle $\phi$.
The position of the camera is described with two angles: azimuthal angle $\theta$ and inclination angle $\phi$.
For each view, the values of $(\theta, \phi)$ are sampled uniformly on the surface of the sphere.
% In the experiments the inclination angle is bounded between $-10$ and $80$ degrees.
In the experiments, the inclination angle is bounded between $10$ and $100$ degrees.
Camera extrinsics (in form of a transformation matrix) are exported for each dataset sample along with the HDR image.
Additionally to that, the bounding box is also exported as it is required
to initialize the octree for the NSVF method and all of the proposed methods as well.
$200$ sample images are used in datasets with 'static' and 'colocated' light settings.
For the 'arbitrary' light setting each dataset consists of $500$ sample images
for having a more dense light-view sampling distribution.
These sample images are then split into training (75\%) and validation (25\%) sets.

\subsection{Light settings}

Different light settings are required due to a diversity of tasks that different methods are to solve.
The illumination of the scene is set using a point light source.
The intensity of the light source varies for different scenes and is chosen empirically.
The positioning of the light source is performed according to the chosen light setting.
Three different light settings are used in order to create three versions of each dataset (\Cref{fig:light_settings}).
This is done based on the fact that different methods can only handle datasets with different settings:
\begin{enumerate}
    \item \textbf{Static setting}

    In this case the light source is positioned at the same global location for all of the dataset samples.
    This allows producing renders of the scene with static illumination.
    The motivation for this type of datasets comes from the limitation of the NSVF method
    as it does not imply dynamic illumination.
    Datasets with static setting are made for 'Vanilla NSVF' (\cite{liu2021neural})
    and can also be processed with 'Brute-force' (\Cref{sec:explicit_scheme})
    and both explicit and implicit schemes (\Cref{fig:implicit_scheme})
    with in-voxel approximation (\Cref{sec:invoxel}).

    \item \textbf{Colocated setting}

    This setting implies the light source to be co-located with the camera.
    In this case, there is no displacement between the camera and the light source,
    which allows the NRF (\cite{bi2020neural}) approach to reuse the view volume transmittances
    in calculations as transmittances of the light rays.
    The 'Brute-force' scheme as well as both explicit and implicit schemes
    with in-voxel approximation are also able to be trained on this type of datasets.

    \item \textbf{Arbitrary setting}

    Datasets that have been created with this setting consist of renders
    where both camera and light positions are sampled uniformly on a sphere around the scene.
    The positioning of the light source follows the same scheme as for camera position.
    Another constraint is applied here that forces the angle
    between view and light rays to the center of the scene to be smaller
    than some value $\alpha_{max}$.
    In the experiments, the value $\alpha_{max} = 120$ degrees has been used.

    This setting is focused on the general case of the NRF approach
    and can be handled by proposed methods, such as: 'Brute force' scheme
    and both explicit and implicit schemes with in-voxel approximation.
\end{enumerate}

\input{tex/objects/light_settings}

\subsection{Scenes}
\label{subsec:scenes}

Creating the datasets from scratch is motivated by the fact that
the vast majority of related works do not consider any light interaction.
This results in such a problem that the datasets that have been used in these works
do not include any necessary light information (global position).
Another limitation of these datasets is that the light source is static upon the dataset samples,
which results in a sparse light-view sampling of the scene.
To the date of writing this thesis, the datasets from corresponding related works \cite{nerv2021}
still are not publicly available.
Four different scenes have been used for the evaluation (\Cref{fig:dataset_preview}).

% \CatchFileDef{\catchdatasetpreview}{tex/objects/figure_dataset_preview.tex}{}
% \catchdatasetpreview
\input{tex/objects/figure_dataset_preview}

% \textbf{Sphere}

% The \textit{sphere} scene consists solely of a sphere,
% which shares Lambertian reflectance (\cite{blinn1982light}) with some specular and roughness components.
% This scene is mostly used as a baseline as it is a very simple
% both geometrically and in terms of appearance.

\textbf{Rocket}

The \textit{rocket} scene (\Cref{fig:dataset_preview} (a)) contains a single object,
which exhibits geometry of medium complexity and realistic non-Lambertian materials.
This scene contains global geometry complications (rocket wings) and some fine details (rivets on the rocket shell).
The surface of the rocket shows a moderate level of specularities (in comparison with sphere scene)

\textbf{Guitar}

% The \textit{guitar} scene (\Cref{fig:dataset_preview} (b)) is created for testing model's ability
% to reproduce highly specular surface behavior.
% Complex geometry also allows to asses how model handles geometrical features on different scales.
% For example strings can be barely seen on renders and they would presumably the most troublesome geometrical part of the whole scene.

The \textit{guitar} scene (\Cref{fig:dataset_preview} (b)) is created for testing the model's ability
to reproduce complex geometrical features on different scales.
For example, strings can be barely seen on renders and they would presumably be the most troublesome geometrical part of the whole scene.
A fairly specular surface appearance also allows assessing how well does it model surface reflectance behavior.

\textbf{Lego}

\textit{Lego} scene (\Cref{fig:dataset_preview} (c)) is the reproduction of the model
that has been used in a former NeRF work \cite{mildenhall2020nerf}.
The dataset is recreated under a specific illumination,
which consists of only one point light source.
The object exhibits a very complex geometry on all scales.
The appearance of the surface does not show any complications.

\textbf{Hotdog}

\textit{Hotdog} scene (\Cref{fig:dataset_preview} (d)) is also a reproduction of the model
from the NeRF work with only one point light source.
The reflectance behavior of objects on this scene is more sophisticated
comparing to the \textit{lego} scene while the geometry is simpler.

\subsection{Real-world dataset handling}

For the group of methods that only consider static illumination of the scene,
the dataset acquisition can be performed in different ways.
One of these ways is to use a cell phone with its camera to capture the scene.
The acquired images can then be processed with such techniques as
structure-from-motion (\cite{Moulon2012, Jancosek2011, schoenberger2016structure})
in order to retrieve the camera intrinsics and extrinsics in correspondence with captured images.
Another way is to use some special hardware setups with calibrated camera positions
% such as gonioreflectometers or e.g. x-rite tac7 (\cite{merzbach2017highquality}).
such as gonioreflectometers.

However, for the methods that model light interaction the light extrinsics information is lacking
when using the aforementioned techniques.
The acquisition of the dataset consisting of captures from real-world scenes
for this group of methods is a fairly hard task that most probably cannot be performed without any hardware assistance.
Even for the 'colocated' case of the NRF method cellphones could only have been used
together with the robotic arm holding the cellphone (\cite{bi2020neural}).
Some hardware setups such as X-Rite tac7 \cite{merzbach2017highquality} are equipped with static cameras and light sources
and a turntable, which allows to rotate the object and thereby acquire varying viewing and illumination directions.

Another key point is the background that has to be as homogeneous as possible.
The quality of final renders is sensitive to the too complex background behind the object on the scene.

% \begin{enumerate}
%     \item Synthetic datasets:
%     \begin{enumerate}
%         \item static/co-located/arbitrary
%         \item Blender + python
%         \item Blender point light intensity for light attenuation
%         \item Different scenes (with its features explanations: reflectance, shape, etc.)
%     \end{enumerate}
%     \item Real-world datasets: flower??? tac7
% \end{enumerate}

\section{Implementation details}

% All of the proposed methods are implemented on Python
% using PyTorch \cite{pytorch} and Fairseq \cite{ott2019fairseq} frameworks
% for neural fields and ray casting procedure.
All of the proposed methods are implemented on Python
using PyTorch \cite{pytorch} framework
for neural fields and ray casting procedure.
The Fairseq \cite{ott2019fairseq} framework is used over the Pytorch,
which provides easier training, evaluation and validation pipelines as well as extends the initial functionality.
During training $80$ pixels are first sampled from each of $1$ or $2$ images
forming a batch that is used for one training iteration.
These pixels are used to cast rays through the focal point of the corresponding cameras.
These rays then intersect the octree, which is initialized with $N_0^3 = 4^3$ voxels ($4$ voxels for each axis).
These rays are then sampled on the voxel intersection intervals with the step size $0.125$.
The following procedure differs depending on the method
and includes additional light ray casting, intersecting and sampling for the 'brute-force' scheme and BRDF evaluation for the 'explicit schemes'.
Alpha compositing is performed after the color (or radiance) values are achieved.
These values are then being supervised with the ground truth values using the fine-network term from loss function from the \Cref{eq:loss_func} with the beta-distribution regularizer from the \Cref{eq:beta_regularizer}:
\begin{equation}
    \mathcal{L} = \sum_{(p_0,v)\in\mathcal{B}} \norm{ C(p, v) - C^*(p, v) } + \beta(\log(\tau_c(p, v)) + \log(1 - \tau_c(p, v))),
\end{equation}
with $\beta = 10^{-3}$ that is used in the experiments.

Adam optimizer is used as an optimization algorithm for learning parameters $(\Theta, I)$ ($I$ is light intensity value).
The learning rate for the optimizer is initialized with value $10^{-4}$
and linearly decays to value $10^{-6}$ over 100k iterations.
Octree refinement procedure as well as reducing ray casting step size are performed on iterations 5k, 25k and 50k.
Self-pruning is performed every 5k iterations.
Tone-mapping using scaling and gamma-correction (differ depending on the scene)
is performed to visualize HDR images (both inputs and outputs).

All of the experiments are held on NVIDIA GTX 1080Ti GPU with 11GB of memory.
Dataset sample images are originally created in 1024x1024px resolution
and then individually down-sampled to some lower values for each experiment to fit the GPU memory limitations.
The resolutions from 64x64px (for \textit{brute-force scheme}) up to 512x512px
(for methods with lower memory load, such as Vanilla NSVF) are used.
The \textit{log-transform} is applied at the pre-processing phase
to prepare the HDR data before it is processed by the model:
\begin{equation}
    c = \log (c^{HDR} + 1),
\end{equation}
where $c^{HDR}$ is the color value of the HDR input
and $c$ is the color value that is used for model supervision.
The inverse-log-transform is performed on the model outputs at the post-processing stage.

% \im{64x64 for 'brute force' scheme}
% \im{different schemes - different training times. approx ~12 hours}

% \im{log transform!}

% \begin{enumerate}
%     % \item Loss function: color + beta-regularizer
%     % \item Parameters: optimizer, lr, batch, coefficients, pruning/refinement iterations
%     % \item Software/Hardware setup
%     \item Time estimates
% \end{enumerate}

\section{Metrics}
\label{sec:metrics}

For evaluation of the results the following metrics have been used:
\begin{enumerate}
    \item \textit{PSNR}$\uparrow$ \cite{hore2010image} (Peak Signal to Noise Ratio) is a traditional estimator for image comparison
    that is based on MSE and concentrates on the pixel-by-pixel comparison.
    Higher value is better.
    \item \textit{SSIM}$\uparrow$ \cite{zhou2004image, nilsson2020understanding, hore2010image} (Structural Similarity Index Measure) is a perceptual metric
    that assesses image quality based on perception of the human visual system.
    Higher value is better.
    \item \textit{LPIPS}$\downarrow$ \cite{zhang2018perceptual} (Perceptual Similarity) is the metric
    that utilizes deep features of the pre-trained deep neural networks (Alex-net \cite{krizhevsky2012imagenet} in my experiments)
    to provide an evaluation that agrees well with humans' assessments.
    Lower value is better.
    \item \textit{HDRFlipLoss}$\downarrow$ \cite{theisel2021hdrflip, andersson2020flip} is the metric
    that is aimed to work with HDR images and produce better assessment comparing to other metrics
    that were originally designed to work with LDR images.
    Lower value is better.
\end{enumerate}

Additionally to the above-mentioned metrics, empirical evaluations have to be made
to assess the quality of the reproduced appearance.
This can be done by evaluating the model on a given trajectory for camera and light sources.
This produces continuous changes in appearance and reveals all of the artifacts of the produced renders.
The usage of artificial datasets (\Cref{sec:datasets}) allows generating the ground truth images
even for novel light and view conditions, which are not contained in training or validation datasets.
This cannot be done for real-world images, which makes the whole task of quality assessment more difficult for such datasets.
% Metrics: MSE, SSIM, FLIPLoss/HDRFlipLoss

\section{Experiments}

% The diversity of used models and the corresponding applicable datasets
% does not allow to well structure this section.
% A shortage of some baseline results does also contribute to some difficulties in the results quality assessment.
% Therefore this section if formed in the order of decreasing number of applicable methods (due to its limitations).

% Therefore this section is formed in the order of increasing complexity of used methods and decreasing evaluation capabilities.
% The following methods are being evaluated in either event:
Following list overviews schemes that are compared in this section:
\begin{enumerate}
    \item Vanilla \textit{NSVF} method \cite{liu2021neural} (described in \Cref{subsec:NSVF}).
    The fastest implementation (among other listed methods),
    however, can only be processed with the static light dataset.
    \item \textbf{Ex}plicit \textbf{Col}ocated scheme (\textit{ExCol}) \cite{bi2020neural, liu2021neural}, which is explained in details in \Cref{subsec:NRF}.
    This is a 'lighter' version of the \textit{explicit brute-force scheme}
    due to the reuse of viewing sample points.
    The second fastest implementation as it only contains
    a small overhead of evaluating BRDF instead of directly predicting colors.
    However, similarly to \textit{NSVF} can only be trained on one type of dataset: with the colocated light setting.
    \item \textbf{Ex}plicit \textbf{B}rute-\textbf{F}orce scheme (\textit{ExBF}) (described in \Cref{sec:explicit_scheme}).
    The generalization of \textit{ExCol}.
    The slowest implementation with the highest memory demand,
    highly impractical especially with the aforementioned hardware setup.
    \item \textbf{Ex}plicit \textbf{V}oxel-\textbf{A}pproximated scheme (\textit{ExVA}), which is the \textit{explicit brute-force scheme}
    that uses the in-voxel approximation from \Cref{sec:invoxel}.
    Implementation involves similar complexity as \textit{ExCol}
    as it also includes light rays octree intersection overhead,
    which makes it slower than \textit{ExCol}.
    \item \textbf{Im}plicit \textbf{N}eural \textbf{R}eflectance \textbf{F}ield scheme (\textit{ImNRF}) (described in \Cref{sec:implicit_scheme}).
    This method is based on the Vanilla NSVF approach.
    Light direction vectors and distances to the light source are also provided along with view direction and sample point location.
    This scheme is considered to implicitly represent the appearance of the scene
    and can handle all of the listed dataset settings.
\end{enumerate}

% \CatchFileDef{\catchmethodsdatasets}{tex/objects/figure_dataset_preview.tex}{}
% \catchmethodsdatasets
\input{tex/objects/table_light_settings}

Please refer to \Cref{tab:methods_datasets} for the visual representation of the applicability of different types of datasets to the methods mentioned above methods.

\subsection{Colocated light setting}
\label{subsec:experiments_coloc}

Datasets with the colocated light setting can be evaluated by all of the listed models except \textit{NSVF}.
The fact that a very fast \textit{ExCol} scheme can be evaluated on these datasets
makes them very advantageous for comparing against other methods.
Since the \textit{ExCol} scheme is essentially a special case for the \textit{ExBF} scheme
they perform the same way as long as the sampling for light rays is performed the same way as for view rays.
On the contrary, the performance of the \textit{ExBF} scheme is drastically worse comparing to the \textit{ExCol} scheme.
The \textit{ImNRF} scheme is expected to perform at the same efficiency level as the Vanilla \textit{NSVF} method.
This is due to the fact, that light rays coincide with view rays
and the scheme does not consider any kind of regularization for the model
to distinguish light rays from view rays.
The light distance is only applicable for light attenuation,
which is not expected to make a very noticeable effect.
Hence the main focus here is to compare colocated scheme \textit{ExCol} with the in-voxel approximation scheme \textit{ExVA}.

% \CatchFileDef{\catchcolocmetrics}{tex/objects/coloc_metrics.tex}{}
% \catchcolocmetrics
\input{tex/objects/coloc_metrics}

An overview of quantitative results of evaluation methods on colocated datasets is presented in \Cref{tab:colocated_metrics}.
Four aforementioned metrics are calculated on renders over the validation dataset (25\% of the whole dataset, 50 images for these experiments).
\textit{ImNRF} is the fastest scheme since it almost completely shares complexity from Vanilla NSVF.
The \textit{ExCol} scheme is the second fastest
since it does not evaluate any overhead related to sampling light rays
as the view samples are simply reused as samples for light rays.
The overhead of the \textit{ExCol} method comparing with \textit{ImNRF} comes from the additional evaluation of the BRDF model before alpha compositing of view samples.
The \textit{ExVA} scheme is an approximation of the \textit{ExCol} method
that uses the distance of travel of light rays inside the voxels scaled by some constant sigma value.
The implementation implies light rays intersection with the octree,
which is a fairly fast task, although it still substantially drops the performance.
The approximation scheme \textit{ExVA} is from $1.4$ to $2$ times slower
than the fastest \textit{ExCol} depending on the scene.

The \textit{ExCol} scheme is similar to \textit{ExBF} results,
which concurs with the expectations.
However, the \textit{ExBF} method has a massive overhead in sampling light rays
and evaluating the model on these samples,
which makes it naturally impractical.
Training of \textit{ExBF} model even on 64px images takes $1.5$ times more time
than training approximation \textit{ExVA} scheme on 256px images.
In these experiments, the \textit{ExBF} scheme did not manage
to converge completely due to hardware memory limitations.
The \textit{ImNRF} scheme shows the best performance on this dataset.
This is presumably due to the fact that it is not forced
to perform any kind of light interaction generalization.
As can be seen later (\Cref{tab:arb_dynamic_light}) the model
mostly does not change predictions for non-coincident view and light rays.
For these experiments, the \textit{ImNRF} is believed to perform
in a similar way Vanilla \textit{NSVF} would have performed if it used consideration of light interaction.

% % The best values are denoted in bold.
% The \textit{ExCol} scheme does produce mostly better results,
% which should be similar to \textit{ExBF} method.
% However, the \textit{ExBF} method has a massive overhead in sampling light rays
% and evaluating model on these samples,
% which makes it naturally impractical.
% Training of \textit{ExBF} model even on 64px images takes $1.5$ times more time
% than training approximation \textit{ExVA} scheme on 256px images.
% In these experiments the \textit{ExBF} scheme did not manage
% to converge completely due to the hardware memory limitations.
% Interestingly the \textit{ExBF} method performed better according to
% LPIPS metric, even though it did not converge completely.
% This could be the result of same nature of the LPIPS metric
% that uses deep features for the evaluation.
% Some outputs of these experiments are selected from the validation dataset
% and overviewed in \Cref{tab:coloc_allresults}.
% % \textit{ExCol} and \textit{ExVA} methods have been trained using 256x256px images
% % while the brute-force scheme \textit{ExBF} was only trained using 64x64px images.
% % The reason for that is a huge need of the \textit{ExBF} method for memory,
% % which is limited by the hardware.
% The refinement procedure is performed on 5k, 25k and 50k iterations.
% For the \textit{ExBF} method the third refinement procedure requires too much memory,
% so the training only consists of 50k iterations.
% \im{update after ImNRF!}

\input{tex/objects/coloc_results}

Some selected views from these experiments are presented for comparison in \Cref{tab:coloc_allresults}.
As was already said the \textit{ExBF} did not show high-quality results.
The main limitation is that this scheme is very memory-demanding and fairly slow.
Images in 64px resolution have been used for the \textit{ExBF} scheme.
The synthesized views are also in the same resolution and scaled up for clearer comparison.
For Lego and Hotdog datasets \textit{ExBF} failed some parts of scene geometry
(the corresponding voxels have been removed during the pruning stage).
When comparing \textit{ExCol}, \textit{ExVA} and \textit{ImNRF} schemes,
the results are very similar especially for \textit{ExCol} and \textit{ExVA} methods.
% One can find \textit{ExCol} to reproduce views with more details,
% however the differences are truly slight and could come from the deviation between training iterations.
The \textit{ImNRF} scheme generally showed the best results
(fine details on lego dataset, specularities on hotdog and guitar datasets),
which confirms the tendency from \Cref{tab:colocated_metrics}.
Both \textit{ExCol} and \textit{ExVA} schemes are occasionally lacking specularities (rocket left)
and failing to reproduce very fine details of the Lego dataset.
The very bright spot on the Lego renders (bottom) is also reproduced incorrectly,
however, this is likely due to the models being underfitted.

\subsubsection{ExCol vs. NRF}

The \textit{ExCol} scheme is originally based on the NRF framework \cite{bi2020neural},
which extends the problem of novel view rendering to
'novel view-light rendering' by considering reflectance of the objects in the scene.
However, the \textit{ExCol} method in turn uses the NSVF framework for increasing the efficiency of sampling.
Although \cite{bi2020neural} do not provide the datasets from their experiments,
some rough time estimation comparison can be done.
It takes approximately 48 hours for the NRF model to get trained on 4 GPUs NVIDIA RTX 2080Ti.
This hardware setup is much more powerful than the hardware used in this thesis.
Nevertheless, as it can be seen from \Cref{tab:colocated_metrics}
the \textit{ExCol} can get trained in approximately 12 hours on a single GPU NVIDIA 1080Ti,
which is $4$ times faster than the original NRF approach.
Even considering the differences in the datasets and quantitative differences in achieved results,
the efficiency improvement is fairly distinct.
Unfortunately, \cite{bi2020neural} do not provide any metrics values to compare our results with.

\subsection{Arbitrary light setting}

Datasets with the arbitrary setting cannot be processed with \textit{ExCol} scheme,
since the light source is not co-located with the camera.
The \textit{ExBF} method is an impractically slow generalization for the \textit{ExCol}.
Thus \textit{ExVA} and \textit{ImNRF} are the main methods for achieving renders on this type of datasets.
Although \textit{ExVA} is an approximation,
which theoretically produces worse results than the 'brute-force' or 'colocated' schemes,
experiments with the colocated datasets (\Cref{subsec:experiments_coloc}) showed
that evaluations are very similar between \textit{ExVA} and \textit{ExCol}
by assessing both using metrics (\Cref{tab:colocated_metrics}) and empirically (\Cref{tab:coloc_allresults}).
The implicit scheme is not restricted to represent scene appearance with some specific model.
On the contrary, it is remained to implicitly learn scene appearance
by including light ray directions and distances along with a viewing direction.

\Cref{tab:arb_selective_results} gives an overview of the results of the two mentioned above methods
trained on the Lego dataset with an arbitrary light setting.
Images marked with \textit{HDRFlip} are the difference images produced within HDRFlipLoss \cite{andersson2020flip, theisel2021hdrflip}
between target images and corresponding renders.
It can be seen that both models reproduced the scene with fairly high quality,
although they produce more errors on darker views with some highly specular flares (3rd row, HDRFlip images show strong deviation from target images).
Please note that model predictions, as well as target image on the 3rd row,
are post-processed by scaling and applying gamma-correction ($\gamma = 2.5$)
and in reality, they contain a very dark object with a very bright spot of light reflection.
\textit{ImNRF} shows higher overall performance than \textit{ExVA}.

\subsubsection{Novel light synthesis}

Although \cite{bi2020neural} claim that deep neural network from their framework
can generalize to some arbitrary light position while only being trained on coinciding light-view positions,
the quality of this generalization is still doubtful.
Datasets that were created with the arbitrary light setting
contain denser light-view sample space than those created with the colocated setting.
This results in a higher quality of novel-light synthesized views.
\Cref{tab:arb_dynamic_light} contains five columns,
which contain images evaluated from the same novel (has not been used during training) camera position: $\phi = 0$ and $\theta = 0$.
Each of these images is lit from the light source,
which novel locations are different along different columns.
The light source's location is defined by two angles:
inclination angle $\theta = \const$ and azimuthal angle $\phi$
that takes values from -90\textdegree to 90\textdegree.
In the central column, the light source is co-located with the camera,
which completely coincides with the light setting used for training \textit{ExCol}
(although this view is novel for the method).
Both camera and light positions are the same for renders inside each column.
Please note that for this experiment \textit{ExCol} has been trained on the dataset with colocated light setting
whilst \textit{ExVA} has been trained on the dataset with the arbitrary light setting.
% \textit{Note}: a significant mistake in implementation of \textit{ExCol} was noticed only shortly before the submission deadline.
% For the non-colocated evaluation of the original NRF approach the \textit{light caching} technique is used,
% which reuses volume density from the view samples for the light transmittance calculation.
% To this done, the nearest view sample points are first located at any given shading point
% and then corresponding volume transmittance is linearly interpolated.
% This allows for shadowing effects in original NRF method.
% In these experiments \textit{ExCol} does not interpolate for the light transmittance,
% which means that it does not consider any shadowing effects (which is proved by the results from \Cref{tab:arb_dynamic_light}).

It can be clearly seen that \textit{ImNRF} trained on the colocated light dataset,
did not manage to properly generalize for the non-colocated light-view case.
There also are no shadows on the predictions as \textit{ImNRF} does not consider any shadowing.
In general \textit{ExVA} method outperforms \textit{ExCol}.
The major difference is in the capability of rendering shadows (non-zero angles)
while for the zero-angle view both methods perform with approximately the same quality.
\textit{Note:} that for evaluation under novel light-view conditions the original NRF method used light transmittance caching
(i.e. finding nearest sampled points and interpolating transmittance value between those),
which allowed for realistic shadowing effects.
In \textit{ExCol} \textit{no interpolation is performed}, which reasons the lack of shadows on the presented render.
Another key observation is that the sigma field of \textit{ExVA} did not converge completely on the front part of the base of the model.
\textit{ExCol} renders this part without any difficulties.
This difference is most likely due to the more consistent lighting of this part
captured in the colocated light dataset comparing to those in the arbitrary light dataset.
Another noticeable difference is that \textit{ExVA} better handles fine details on the scene.
This can be explained by the fact that these fine features appear
in the arbitrary light dataset with its own shadows casted in different directions (depending on light source location),
which encourages the density predictor to better distinguish geometrical features from appearance effects.
% Note that due to the colocated nature of training dataset for the \textit{ExCol}
% the model does not happen to meet any shadows at all.
% Another inquiring tendency is how drastically quality of the \textit{ExCol} predictions drops
% when the angle between camera and light rays is increasing.
% This can also be explained by an extremely sparse light-view space of the colocated light dataset.
\textit{ImNRF} (trained on the arbitrary light dataset)
managed to reproduce the scene even better than \textit{ExVA}.
The same tendency as in \Cref{tab:arb_selective_results} can be noticed
that \textit{ImNrf} is able to recover more details comparing to the \textit{ExVA}.

\input{tex/objects/arb_selective_results}

% \input{tex/objects/arb_maps}

\input{tex/objects/arb_dynamic_light}

% \im{RESULTS metrics+renders: check quality from u4108: exva lego 256px, timings can be compared on 64px results: exva u4104 vs exvf u4109}

% \im{Random datasets: compare brute-force with in-voxel approximation}

% The experiments involve training models on different synthetic datasets (\im{reference to prev. subsection}) with different parameters
% and quantitative (\im{ref}) and qualitative (\im{ref}) comparisons of those.

\subsection{Static light setting}

Static light setting is the only type of datasets that can be used
for training the original NSVF method,
due to the limitation of the illumination to be static.
However, this type of dataset appears to be challenging for the novel light-view synthesis task.
Although, the sampling in the light-view space is denser comparing with the colocated setting
where no other than coincided light and view rays exist,
the scene is lit with the light from only one side.
This poorly affects the shadowed side as it always appears in the dataset darkened.
The underlying effect is the worse capabilities of the model to converge on the sigma field,
which plays a significant role in the final appearance estimation.
This problem is not method-dependant and is going to appear on all of the presented results.
However, the incorrect scene geometry will seriously affect the scene under novel lighting
as the shadows will move revealing incorrect parts of the geometry prediction.
% The \textit{ExVA} and \textit{ExBF} models are highly unstable in training on this type of datasets
% and cannot reliably get trained.

This problem is exaggerated by the consideration of using a single light source to illuminate the scene
that comes with formulating the rendering equation (\cref{eq:integral_estimation_nrf}).
In the NeRF and NSVF methods, the global static illumination is considered,
which is not limited by only one light source and can be very complex.

\input{tex/objects/static_valid_results}

% \im{REVIEW below! <- ImNRF results}

Some comparisons of evaluations of these two methods are presented in \Cref{fig:static_valid_results}.
The training was performed on static light datasets.
Each row consists of three sub-rows that contain evaluation outputs for \textit{ExVA}, \textit{NSVF} and \textit{ImNRF} methods.
Columns correspond to different types of evaluation outputs.
\textit{Normal} is the normal map (in the viewing coordinate frame)
that is calculated from the sigma field.
% \textit{Depth} is the corresponding predicted depth map.
\textit{Voxel} map is the visualization of the octree for the presented view rendering.

It can be easily seen that all methods struggle
with the correct prediction of shadowed parts of the scenes
(holes in hotdog dataset, missing part of the base in lego dataset bottom subrow).
Although the geometry fails, the produced renderings
of the \textit{NSVF} and \textit{ImNRF} method match target images with fairly high accuracy.
The reason for that is the static lighting, which is 'baked' into the scene appearance.
On contrary, the \textit{ExVA} method implements the BRDF model,
which relies on geometry and is especially sensitive to correct normal estimation
on spots with high light contribution (i.e. flares from the light source).
This behavior can be seen on the hotdog dataset and guitar dataset.
In general, both \textit{NSVF} and \textit{ImNRF} outperform \textit{ExVA} on this dataset light setting.

\section{Concurrent works results}

Some results of the concurrent works are shown below
in order to make a comparative understanding of our methods' quality.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=\textwidth]{figures/concurrent/nerv.png}
    \caption{Novel view renders with a single point light source illumination of (c) NeRV approach \cite{nerv2021}
    in comparison with (a) Ground Truth images and renders of (b) NRV approach \cite{bi2020neural}.
    Images are directly used from NeRV paper \cite{nerv2021}.
    }
    \label{fig:concur_nerv}
\end{figure}

\begin{figure}[!htb]
    \centering
    \includegraphics[width=\textwidth]{figures/concurrent/nerd.png}
    \caption{Novel view renders with a single point light source illumination of NeRD approach \cite{boss2020nerd}
    in comparison with (a) Ground Truth images.
    Images are directly used from NeRD paper \cite{boss2020nerd}.
    }
    \label{fig:concur_nerd}
\end{figure}

\begin{figure}[!htb]
    \centering
    \includegraphics[width=\textwidth]{figures/concurrent/nerfactor_3_75percent.png}
    \caption{Novel view renders with a single point light source illumination of NeRFactor approach \cite{nerfactor}
    in comparison with (a) Ground Truth images.
    Images are directly used from NeRFactor paper \cite{nerfactor}.
    }
    \label{fig:concur_nerfactor1}
\end{figure}

\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.82\textwidth]{figures/concurrent/nerfactor_4_75percent.png}
    \caption{Novel view renders with a single point light source illumination of NeRFactor approach \cite{nerfactor}
    in comparison with (a) Ground Truth images.
    Images are directly used from NeRFactor paper \cite{nerfactor}.
    }
    \label{fig:concur_nerfactor2}
\end{figure}
% \section{Discussion}

% \begin{enumerate}
%     % \item HDR inputs: preprocessing (log-transform)
%     \item \im{Background color problem?}
%     -- NSVF used way bigger batch size (32 images instead of 2 in my case) and 2048 rays in each image (80 in my case)
%     % \item In-Voxel approximation: different sigma strategies
%     \item Implicit scheme with tangential coordinate system
% \end{enumerate}

% \im{inference time estimates}

% \im{show albedo/roughness/normals}

% \section{Colocated setting flaws}

% Network barely generalizes for noncolocated light source position when trained only on colocated setting

% \section{Brute Force vs Approximation}

% \section{Blender-Radiance coefficient}

% The light attenuation factor has to be taken into account in the explicit scheme.

% $L_o(x, \Omega_o) = L_e(x, \Omega_o) + \int_{H_i}{f_{BRDF}(\Omega_i, x, \Omega_o) L_i(x, \Omega_i) cos \Theta_i d\omega_i}$

% $L_i = \tau_l L_l$

% $L_l = f_{att}I$

% $f_{att}=\frac{1}{1 + 2d/r + d/r^2}$

% $L_o = \frac{k_d}{\pi} \frac{1}{1 + 2d/r + d/r^2} I cos \Theta_i$

% $k_c = 0, k_l = 0, k_q = 1$

% r = 0.05

% $f_{att} = r^2 / d = 0.0025 / (10-0.05) = 2.51256e-4$

% $0.24316406 = 0.8 / \pi * 2.51256e-4 * I * 1$

% $0.24316406 = 2.01005e-4 * I$

% $I = 500$

% This is some test area for new mathematical helper macros to nicely visualize mathematical formulas.

% \section{Numbers}
% \begin{align}
%     \mathbb{C}
%     \qquad
%     \mathbb{R}
%     \qquad
%     \mathbb{Q}
%     \qquad
%     \mathbb{Z}
%     \qquad
%     \mathbb{N}
% \end{align}

% \section{Numbers with physical units}
% \begin{align}
%     \SI{1.23}{\meter\per\second}
% \end{align}
% \begin{align}
%     \si{\meter\per\second}
% \end{align}
% \begin{align}
%     \SI{1.23\pm0.45}{\meter\per\second}
% \end{align}
% \begin{align}
%     \SI{3e8}{\meter\per\second}
% \end{align}
% \begin{align}
%     \SI{32}{\giga\byte} = \SI{32e9}{\byte}
% \end{align}
% \begin{align}
%     \SI{32}{\gibi\byte} = \SI[exponent-base=2]{32e30}{\byte}
% \end{align}

% \section{Norm, Dot, Abs, Interval}
% \begin{align}
%     \pi = \const
% \end{align}
% \begin{align}
%     1 \in \interval{0}{2}
% \end{align}
% \begin{align}
%     1 \in \order{n}
% \end{align}
% \begin{align}
%     \evalat{ \frac{\partial f}{\partial x} }{ x = 0 }
% \end{align}
% \begin{align}
%     \norm{p} \qquad \norm{\frac{p}{2}}
% \end{align}
% \begin{align}
%     \abs{p} \qquad \abs{\frac{p}{2}}
% \end{align}
% \begin{align}
%     \dotproduct{p}{q} \qquad \dotproduct{\frac{p}{2}}{q}
% \end{align}
% \begin{align}
%     \crossproduct{p}{q} \qquad \crossproduct{\frac{p}{2}}{q}
% \end{align}

% \section{Vector, Matrix}
% \begin{align}
%     \vec{p} \qquad \vecarrow{p}
% \end{align}
% \begin{align}
%     \vec{p}^{\transposed}
% \end{align}
% \begin{align}
%     \gradient{\vec{p}}
% \end{align}
% \begin{align}
%     \divergence{\mat{A}}
% \end{align}
% \begin{align}
%     \laplacian{\mat{A}}
% \end{align}
% \begin{align}
%     \mat{A}
% \end{align}
% \begin{align}
%     \set{K} , K
%     \qquad
%     \set{N} , N
% \end{align}
% \begin{align}
%     \neighborhood{\vec{p}} = \left\{ \vec{q} \mid \norm{\vec{p} - \vec{q}} < \epsilon \right\}
% \end{align}

% \section{Set operations}
% \begin{align}
%     A \intersect B
% \end{align}
% \begin{align}
%     A \union B
% \end{align}
% \begin{align}
%     A \difference B
% \end{align}

% \section{Derivative, Integral, Sum, Probability}
% \begin{align}
%     \int_H x \, dx
% \end{align}
% \begin{align}
%     \sum_H x
% \end{align}
% \begin{align}
%     \probability{x}
% \end{align}
% \begin{align}
%     \probabilitygiven{x}{y}
% \end{align}
% \begin{align}
%     \expectation{x}
% \end{align}
% \begin{align}
%     \deviation{x}
% \end{align}
% \begin{align}
%     \variance{x}
% \end{align}

% \section{Lemma, Theorem, Corollary}
% \begin{lemma}
%     This is a lemma.
% \end{lemma}
% \begin{proof}
%     Proof of lemma.
% \end{proof}

% \begin{theorem}
%     This is a theorem.
% \end{theorem}
% \begin{proof}
%     Proof of theorem.
% \end{proof}

% \begin{corollary}
%     This is a corollary.
% \end{corollary}
% \begin{proof}
%     Proof of corollary.
% \end{proof}


\section{Proposed Approach}
\label{sec:approach}

\subsection{Multilingual CTC Model}
\input{figs/arch.tex}
We used the model architecture as illustrated in Fig.~\ref{fig:model-arch}, the shared encoder is parameterized by $\theta$, and the set of language-specific heads are parameterized by $\theta_h$ ($\theta_{h,l}$ means the head for $l$-th language). Let the dataset be $D$, composed of paired data $(X,C)$. Let $X = x_1, x_2, \cdots, x_T$ with length $T$ as input feature, $C = c_1, c_2, \cdots, c_L$ with length $L$ as target label. $X$ is encoded into sequence of hidden states $H = h_1, h_2, \cdots, h_{L^\prime}$ with length $L^\prime$ through the shared encoder, then fed into the language-specific head of the corresponding language with softmax activation to output the prediction sequence $\hat{C} = \hat{c_1}, \hat{c_2}, \cdots, \hat{c_{L^\prime}}$.
%\vspace{-2pt}

\textbf{CTC Loss}. CTC computes the posterior probability as below,

\begin{equation}
  P(C|X) = \sum_{\pi \in \mathcal{Z}(C)} P(\pi|X)
\end{equation}
where $\pi$ is the repeated character sequence  of $C$ with additional blank label, and $\mathcal{Z}(C)$ is the set of all possible sequences $\pi$ given sequence $C$. For each $\pi$, we can approximate the posterior probability as below,

\begin{equation}
  P(\pi|X) \approx \prod_{i=1}^{L^\prime} P(\hat{c_i}|X)
\end{equation}

The loss function of the model on $D$ is then defined as:

\begin{equation}
  \label{eq:ctc-loss}
  \mathcal{L}_D(\theta, \theta_h) = - \log P(C|X)
\end{equation}

\subsection{Meta Learning for Low-Resource ASR}

The idea of MAML is to learn initialization parameters from a set of tasks.
In the context of ASR, we can view different languages as different tasks.
Given a set of source tasks $\mathcal{D}=\lbrace D_1, D_2, \cdots, D_K \rbrace$, MAML learns from $\mathcal{D}$ to obtain good initialization parameters $\theta^{\star}$ for the shared encoder.
$\theta^{\star}$ yields fast task-specific learning (fine-tuning) on target task $D_t$ and obtains $\theta^{\star}_t$ and $\theta^{\star}_{h,t}$ (the parameters obtained after fine-tuning on $D_t$).
MAML can be formulated as below,
\begin{equation*}
  \theta^{\star}_t, \theta^{\star}_{h,t} = \texttt{Learn}(D_t;\theta^{\star}) = \texttt{Learn}(D_t;\texttt{MetaLearn}(\mathcal{D})).
\end{equation*}
The two functions,  \texttt{Learn} and \texttt{MetaLearn}, will be described in the following two subsections.
%where $\theta^{\star}_t$ is the parameter obtained after fine-tuning on $T_t$.

\subsubsection{Learn: Language-specific learning}
Given any initial parameters $\theta^0$ of the shared encoder (either random initialized or obtained from pretrained model) and the data $D_t$. The language-specific learning process is to minimize the CTC loss function defined in Eq.~\ref{eq:ctc-loss}.

\begin{equation}
  \label{eq:fine-tune}
  \resizebox{0.91\hsize}{!}{$
\begin{aligned}
  \theta^\prime, \theta^\prime_{h,t} = \texttt{Learn}(D_t;\theta^0) & = \arg \, \min_{\theta, \theta_{h,t}} \mathcal{L}_{D_t}(\theta, \theta_{h,t}) \\
                                                                    & = \arg \, \min_{\theta, \theta_{h,t}} \sum_{(X,C) \in D_t} -\log P(C|X)
\end{aligned}
$}
\end{equation}
\input{figs/flp_table.tex}

The learning process is optimized through gradient descent, the same as MultiASR.

\subsubsection{MetaLearn}
The initialization parameters found by MAML should not only adapt to one language well, but for as many languages as possible. In order to achieve this goal, we define the meta learning process and the corresponding meta-objective as follows.

In each meta learning episode, we sample batch of tasks from $\mathcal{D}$, then sample two subsets from each task $k$ as training and testing set, denoted as $D_{k}^{tr}$, $D_{k}^{te}$, respecitvely. First, we use $D_k^{tr}$ to simulate the language-specific learning process to obtain $\theta^\prime_k$ and $\theta^\prime_{h,k}$.

\begin{equation}
    \theta_k^\prime, \theta_{h,k}^\prime = \texttt{Learn}(D^{tr}_k; \theta)
                                         %&= \arg \, \min_{\theta, \theta_{h,k}} \mathcal{L}_{D^{te}_k}(\theta, \theta_{h,k})
\end{equation}

Then evaluate the effectiveness of the obtained parameters on $D_k^{te}$. The goal of MAML is to find $\theta$, the initialization weights of the shared encoder for fast adaptation, so the meta-objective is defined as

\begin{equation}
  \label{eq:meta-obj}
  \mathcal{L}^{meta}_{\mathcal{D}}(\theta) =  \mathbb{E}_{k \sim \mathcal{D}} \; \mathbb{E}_{D_k^{tr}, D_k^{te}} \Big [ \mathcal{L}_{D^{te}_k}(\theta^\prime_k, \theta^\prime_{h,k}) \Big ]
\end{equation}

Therefore, the meta learning process is to minimize the loss function defined in Eq.~\ref{eq:meta-obj}.

\begin{equation}
  \theta^\star = \texttt{MetaLearn}(\mathcal{D}) = \arg \min_{\theta} \mathcal{L}^{meta}_{\mathcal{D}} (\theta)
\end{equation}

We use \textit{meta gradient} obtained from Eq.~\ref{eq:meta-obj} to update the model through gradient descent (\textit{outer loop}).

\begin{equation}
  \label{eq:meta-grad}
  \theta \leftarrow \theta - \eta^\prime \sum_k \nabla_\theta \mathcal{L}_{D^{te}_k}(\theta^\prime_k, \theta^\prime_{h,k})
\end{equation}
$\eta^\prime$ is the meta learning rate. And noted that only the shared encoder is updated via Eq.~\ref{eq:meta-grad}.

MultiASR optimizes the model according to Eq.~\ref{eq:fine-tune} on all source languages directly, without considering how learning happens on the unseen language. Although the parameters found by MultiASR is good for all source languages, it may not adapt well on the target language. Unlike MultiASR, MetaASR explicitly integrates the learning process into its framework via simulating language-specific learning first, then meta-update the model. Therefore,the parameters obtained are more suitable to adapt to the unseen language. We illustrated the concept in Fig.~\ref{fig:meta-idea}, and showed in the experimental results in Section \ref{sec:results}.