%==============================================================

\chapter{Results}\label{bds2}

In this chapter, the results concerning the quality of the recommendations are shown.
An attempt to quantify the results and the quality of the recommendations is made by choosing objective tests like genre recall and cover song recognition. The second part comprises some subjective impressions, including personal taste and listening preferences.

\section{Objective Evaluation}

At first, for the objective, scientific evaluation, the resulting distances are analyzed and visualized in Section~\ref{featqual}.
To evaluate the quality of the resulting song recommendations returned by the Spark application, some tests were made. As mentioned in Section~\ref{chromavalid}, a way of evaluating the quality of the melodic similarities is the ability to recognize cover songs. This will be examined in Section~\ref{covsongid}. To test the quality of the timbre and rhythm based distances, the genre recall rate is examined in Section~\ref{genrerec}. Another indicator of the quality of rhythm features is the ability to recommend songs around the same BPM count (see Section~\ref{rhythmrec}).

\subsection{Feature Correlation and Distance Distribution}\label{featqual}

This section evaluates the results from the similarity analysis to determine how the distances from different feature sets correlate with each other, and how they are distributed over the unit interval $[0,1]$.
\noindent To analyze this, a test dataset consisting of distances returned by the Spark application had to be created. Ninety-five songs (five songs from every genre) were randomly chosen from the 1517-Artists dataset, and the distances to all other songs of the 1517-Artists dataset were calculated. The dataset contains 3180 songs evenly distributed over 19 different genres (see Figure~\ref{1517dist}).
\begin{figure}[htbp]
	\centering
	\framebox{\parbox{1\textwidth}{
		\centering
		\includegraphics[scale=0.35]{Images/SparkFeat/featurespace.png}
		\captionof{figure}{Feature space example}
		\label{fig:featspace}
	}}
\end{figure}%
%\FloatBarrier

\noindent Sampling of distances from different genres is vital for the analysis of the distribution of distances. Distances and their distribution vary, depending on where in the feature space the actual song is located.
\noindent A song taken from the edge of the distribution of the feature space will end up with different distances than a song taken from the center. To demonstrate this, Figure~\ref{fig:featspace} shows a minimal example. While the distances from songs tagged with "Metal" to the songs tagged with "Rock" and "Classical" are about the same, the distances from a song taken off the Classical genre to the "Rock" or "Metal" songs are different in this example. The Rock songs are twice as far from the Classical songs than from the Metal songs.\\
%For 95 song request that results in eight very large vectors containing the distances of song pairs. This large set of distances is then analyzed.
\noindent Figure~\ref{fig:corr2} shows the correlation between the distances from the various feature types. The eight different distances for each song pair are summed up into one new combined distance (following Equation~\eqref{eq:distance} with all weights $w = 1$). This combined distance is labeled as "agg" in the following plots. Unsurprisingly the various rhythm features correlate well with each other and the JS and SKL features do so as well. The melodic features on the other hand are only weakly correlated.

\begin{figure}[htbp]
	\centering
	\framebox{\parbox{1\textwidth}{
			\includegraphics[scale=0.4]{Images/SparkFeat/corr_95_1517.png}
	}}
	\caption{Correlation matrix, 95 random songs, 19 genres (5 each), 1517-Artists}
	\label{fig:corr2}
\end{figure}
%\FloatBarrier

\noindent The correlation of a feature type with the overall distance is a sign of the impact of the feature type on the overall distance from the weighted sum. But because not all distances are equally distributed over the unit interval, different feature types have different impacts on the sum of distances. This problem was already mentioned in Section~\ref{distsc} and Section~\ref{sparkskl}. Figure~\ref{fig:cumdist} shows how the distances are distributed with the cumulative histograms over the unit interval. It is apparent that especially the cross-correlation distances are not evenly distributed. In Section~\ref{distsc}, a few proposals were already given as to how this problem could be solved in the future.

\begin{figure}[htbp]
	\centering
	\framebox{\parbox{1\textwidth}{
			\begin{subfigure}{.495\textwidth}
				\centering
				\includegraphics[scale=0.5]{Images/SparkFeat/cum1.png}
				\caption{cumulative distribution 1}
				\label{cum1}
			\end{subfigure}%
			\begin{subfigure}{.495\textwidth}
				\centering
				\includegraphics[scale=0.5]{Images/SparkFeat/cum2.png}
				\caption{cumulative distribution 2}
				\label{cum2}
			\end{subfigure}

			\begin{subfigure}{.495\textwidth}
				\centering
				\includegraphics[scale=0.5]{Images/SparkFeat/cum3.png}
				\caption{cumulative distribution 3}
				\label{cum3}
			\end{subfigure}%
			\begin{subfigure}{.495\textwidth}
				\centering
				\includegraphics[scale=0.5]{Images/SparkFeat/cum4.png}
				\caption{cumulative distribution 4}
				\label{cum4}
			\end{subfigure}
	}}
	\caption{Cumulative distributions of distances}
	\label{fig:cumdist}
\end{figure}
%\FloatBarrier

\noindent As mentioned in Section~\ref{sparkskl}, the SKL divergence was also prone to outliers and had shortcomings when scaling distances to the unit interval. The solution was to filter out all song pairs with an SKL divergence larger than a certain threshold before scaling the distances. If this filter operation is left out, nearly all distances calculated with the symmetric Kullback-Leibler divergence are close to zero after the scaling. The impact can be seen in Figure~\ref{fig:sklsc} and Figure~\ref{fig:corr}.
\noindent If the outliers are not filtered, the correlation between the unfiltered SKL distances and the combined distance from the weighted sum ("agg") decreases significantly (see Figure~\ref{fig:corr}). Interestingly also the correlation between the Jensen-Shannon-like divergence and the combined distance ("agg") is decreasing.

\begin{figure}[htbp]
	\centering
	\framebox{\parbox{1\textwidth}{
	\begin{subfigure}{0.5\textwidth}
		\centering
		\includegraphics[scale=0.5]{Images/SparkFeat/skl_unscaled.png}
		\captionof{figure}{SKL unscaled}
		\label{scklusc}
	\end{subfigure}
	\begin{subfigure}{0.5\textwidth}
		\centering
		\includegraphics[scale=0.5]{Images/SparkFeat/skl_scaled.png}
		\captionof{figure}{SKL scaled}
		\label{scklsc}
	\end{subfigure}%
	}}
	\caption{Impact of SKL scaling on the weighted sum}
	\label{fig:sklsc}
\end{figure}
%\FloatBarrier

\noindent A possible explanation could be that the SKL and JS distances are highly correlated, but due to the bad scaling, the SKL has no impact on the overall distance. The results from the JS divergence alone are not able to impact the weighted sum of the combined distance in the same way both features together could.

\begin{figure}[htbp]
	\centering
	\framebox{\parbox{1\textwidth}{
	\begin{subfigure}{0.5\textwidth}
		\centering
		\includegraphics[scale=0.33]{Images/SparkFeat/skl_corr_unscaled.png}
		\captionof{figure}{SKL unscaled}
		\label{corrusc}
	\end{subfigure}
	\begin{subfigure}{0.5\textwidth}
		\centering
		\includegraphics[scale=0.33]{Images/SparkFeat/skl_corr_scaled.png}
		\captionof{figure}{SKL scaled}
		\label{corrsc}
	\end{subfigure}%
	}}
	\caption{Correlation of features depending on SKL scaling}
	\label{fig:corr}
\end{figure}
%\FloatBarrier

\noindent Finally, Figure~\ref{fig:corr3} shows the full scatter plot matrix of the various distances for the 95 song sample from different genres to visualize the correlation and distribution of the distances. The main diagonal shows the histograms of the distances from the respective unique feature-sets. It shows that besides the chroma features all feature types correlate well with the weighted sum of all features. The strong correlation between the rhythm patterns and rhythm histograms as well as the Jense-Shannon-like divergence and the symmetric Kullback-Leibler divergence for all genres is clearly visible in the scatter plots.

\begin{figure}[htbp]
	\centering
	\framebox{\parbox{1\textwidth}{
	\includegraphics[scale=0.36]{Images/SparkFeat/scatter1517_KernelDensityEstimationHD.png}
	\caption{Scatter matrix, correlation 95 songs, 19 genres (5 each), 1517-Artists}
	\label{fig:corr3}
	}}
\end{figure}
%\FloatBarrier

%\textit{\textbf{TODO: describe correlation of melodic features and AGG}}

\subsection{Cover Song Identification}\label{covsongid}

As mentioned in Section~\ref{covermfcc}, purely MFCC based recommender systems lack the capacity to detect cover songs. Melody based similarity algorithms like the cross-correlation approach by Ellis and Poliner (see Section~\ref{crosscorrsec}) and the approach using the Levenshtein distance by Xia (et al.) in Section~\ref{textretr}, were primarily implemented to detect cover songs.
\noindent Running the first tests on the full dataset consisting of 114210 songs, the Spark implementation was able to find the cover of "Rock you like a Hurricane" by the Scorpions and covered by Knightsbridge as the top recommendation when using the cross-correlation.\\
\noindent The application was also able to find an alternative recording of the piece "Für Elise" cover as a top recommendation in over 114210 songs, even when using the filter and refine algorithm (starting with chroma features) presented in Section~\ref{farfs}.\\
\noindent As a third example the famous "Rondo Alla Turca (Allegretto)" also known as the Turkish March by Mozart was tested. This song was also used in Section~\ref{covermfcc} where the capacity of the Musly toolkit to detect cover songs was tested. \noindent Two different versions were detected as the top results, and the fourth recommendation even listed a variation of the original song theme.
For this test, a combination of js, chroma, and rp features was used. The top five results are listed below.\\
\ \\
\textit{\noindent Song request: 100 Meisterwerke der Klassik - Mozart - Alla Turca (Allegretto) (private collection), JS + RP + CHROMA}

\begin{enumerate}
	\setlength\itemsep{-0.5em}
	\item Piano Perlen / Mozart - Türkischer Marsch (private collection)
	\item FRITZ STEINEGGER - RONDO ALLA TURCA KV 331 (1517-Artists)
	\item 136071 (2Kutup - We Shall Cuddle Up And Sleep) (FMA dataset)
	\item Sean Bennett - Variations on the Turkish March (1517-Artists)
	\item Mozart - Fantasie in D minor (1517-Artists)
\end{enumerate}

\noindent Although the private music collection contains two additional versions of this song (see Section~\ref{covermfcc}), the other versions could not be detected because the rp\_extract tool failed during the extraction of the features from these songs due to file format issues. In a second test, the rhythm patterns were left out and only js and chroma features were used. The six top recommendations are again listed below:\\
\ \\
\textit{\noindent Song request: 100 Meisterwerke der Klassik - Mozart - Alla Turca (Allegretto) (private collection), JS + CHROMA}

\begin{enumerate}
	\setlength\itemsep{-0.5em}
	\item Mozart Collection / CD31 / KV331-3 Alla turca allegretto (private collection)
	\item Piano Collection / CD25 - Mozart - Alla Turca Allegretto (private collection)
	\item Piano Perlen / Mozart - Türkischer Marsch (private collection)
	\item FRITZ STEINEGGER - RONDO ALLA TURCA KV 331 (1517-Artists)
	\item 136071 (2Kutup - We Shall Cuddle Up And Sleep) (FMA dataset)
	\item Sean Bennett - Variations on the Turkish March (1517-Artists)
\end{enumerate}

\noindent %Even if the JS features are also left out and only the cross-correlation is computed, the top 6 results don't change. But on the other hand, even when the JS features are included, they aren't interfering with the cover song recognition.
In a third request where only the Jensen-Shannon-like divergence was tested to detect the alternative recordings, the first alternative recording appeared as the 13th recommendation. This confirmed the presumption that timbral features and the Jensen-Shannon-like divergence nor the symmetric Kullback-Leibler divergence are appropriate for cover song recognition.\\
\noindent But there are also song requests where the cross-correlation fails to detect the cover song, one example being the song Chandelier by Sia and its cover version by Pvris that was used in Section~\ref{chromafeat} to explain the computation of the chroma features.\\
\noindent To further quantify the ability to detect cover songs after the promising first tests, the covers80 dataset introduced in Section~\ref{cov801} was loaded onto the cluster. The 80 "A-versions" songs were passed to the Spark application as song requests, and the resulting nearest neighbors were analyzed.

\begin{table}[H]
	\begin{minipage}{0.5\textwidth}
		\begin{center}
			\begin{tabular}{|l||c|}
				\hline
				features & detected covers\\
				\hline
				\hline
				chroma & 30\\
				\hline
				chroma + notes & 27\\
				\hline
				chroma + skl & 26\\
				\hline
				chroma + notes + rp & 24\\
				\hline
				chroma + rp & 22\\
				\hline
				chroma + skl + rp & 22\\
				\hline
				chroma + mfcc & 19\\
				\hline
				chroma + js + rp & 17\\
				\hline
				chroma + js & 17\\
				\hline
				notes & 17\\
				\hline
				chroma + mfcc + rp & 15\\
				\hline
				all & 15\\
				\hline
				notes + rp & 13\\
				\hline
				mfcc + notes + rp & 7\\
				\hline
				rp & 7\\
				\hline
				mfcc + js + skl & 3\\
				\hline
			\end{tabular}
		\end{center}
		\captionof{table}{Cover recognition rate - Top 1\label{cov_1}}
	\end{minipage}
	\begin{minipage}{0.5\textwidth}
		\begin{center}
			\begin{tabular}{|l||c|}
				\hline
				features & detected covers\\
				\hline
				\hline
				chroma & 33\\
				\hline
				chroma + notes & 31\\
				\hline
				chroma + notes + rp & 30\\
				\hline
				chroma + skl & 29\\
				\hline
				chroma + rp & 29\\
				\hline
				chroma + skl + rp & 26\\
				\hline
				chroma + mfcc + rp & 24\\
				\hline
				notes & 23\\
				\hline
				all & 23\\
				\hline
				chroma + mfcc & 22\\
				\hline
				chroma + js + rp & 22\\
				\hline
				chroma + js & 21\\
				\hline
				notes + rp & 19\\
				\hline
				rp & 15\\
				\hline
				mfcc + notes + rp & 14\\
				\hline
				mfcc + js + skl & 10\\
				\hline
			\end{tabular}
		\end{center}
		\captionof{table}{Cover recognition rate - Top 5\label{cov_2}}
	\end{minipage}
\end{table}

\noindent Table~\ref{cov_1} counts the appearance of the "B-version" songs as the first recommendations while Table~\ref{cov_2} lists the count of the recommended cover versions in the top five results, when using different combinations of feature sets. As expected, the approaches using melodic similarity features perform best. The combination of different timbre based features performs worst. Interestingly the distances based on rhythm patterns also detect some cover songs.\\
\noindent Although 30 out of 80 detected cover songs does not seem like a surprisingly good hit rate at first and is not quite as good as the results from the original paper, it has to be mentioned that most of the cover versions in the cover80 dataset differ significantly from the original recordings in musical style, instrumentation, rhythm and even genre from the original recordings. These differences in musical style were also mentioned in the original paper from Ellis and Cotton~\cite[p. 3]{cover802}.
\noindent As an interesting side note it has to be mentioned that the detected cover versions of the "chroma-" and "notes-only" requests were mostly the same. Aside from two songs, the chroma feature cross-correlation approach detected all of the cover songs that the Levenshtein distance also detected. So in conclusion, the cross-correlation is more precise but also more compute heavy.

\subsection{Genre Similarity}\label{genrerec}

Another way to quantify the quality of the distances and therefore the quality of the music recommendations is to measure the genre recall rate. In a simple test on the 1517-Artists dataset, five classical songs are passed to Spark, and the nearest neighbors based on rhythm and timbre features (skl, js, mfcc, rp, rh, and bh) are calculated. Then the genres of the top ten recommendations from all five song requests are analyzed. The result is pictured in Figure~\ref{fig:genrerec}.

\begin{figure}[htbp]
	\centering
	\framebox{\parbox{1\textwidth}{
			\begin{subfigure}{.495\textwidth}
				\centering
				\includegraphics[scale=0.33]{Images/SparkFeat/1517clasall.png}
				\caption{1517-Artist classical recommendations}
				\label{fig:genrerec}
			\end{subfigure}%
			\begin{subfigure}{.495\textwidth}
				\centering
				\includegraphics[scale=0.33]{Images/SparkFeat/1517rockall.png}
				\caption{1517-Artist rock recommendations}
				\label{fig:genrerec2}
			\end{subfigure}
	}}
	\caption{Genre recall rate on 1517-Artists dataset}
	\label{fig:1517gen}
\end{figure}
%\FloatBarrier

\noindent Although not all recommendations are classical songs, the recommended other genres like New Age, Wold, Folk and Jazz music are closely related to classical music. Not a single song from more "modern" genres like Hip-Hop, Rock \& Pop, Electronic \& Dance or Reggae appears.
\noindent The same was tested with five songs from the Rock \& Pop genre (see Figure~\ref{fig:genrerec2}). The results are scattered across 16 out of 19 different genres from 1517-Artists dataset. A possible explanation for this is, that the songs annotated with "Rock \& Pop" in this dataset come from a wider variety of sub-genres. When taking a closer look at the dataset, it shows that, e.g., Metal songs are also tagged as Rock \& Pop.\\
\begin{figure}[htbp]
	\centering
	\framebox{\parbox{1\textwidth}{
			\includegraphics[scale=0.29]{Images/SparkFeat/rock.png}
	}}
	\caption{Scatter matrix, distances 1 random Rock\&Pop song, 1517-Artists, 4 genres}
	\label{fig:corr5}
\end{figure}
\FloatBarrier

\noindent To investigate the impact of different feature types on the overall recommendations and to visualize the distribution of distances for different genres, another test was performed. For single song requests, all distances to the songs from a subset of the 1517-Artists dataset containing the genres "Classical", "Hip-Hop", "Electronic \& Dance" and "Rock \& Pop" were computed. Figure~\ref{fig:corr5} shows the scatter matrices of all distances from one song request taken from the genre Rock \& Pop. The different distances of the recommendations are colored by the genre of the recommended song.
\noindent On the main diagonal the Kernel Density Estimation of the respective feature type is shown. One interesting detail that should be pointed out is that the JS distance alone is unable to distinguish between Rock/Pop songs and Hip-Hop songs but is able to separate between classical music and the rest. On the other hand, the rhythm patterns alone can not separate classical music from rock and pop. But when both feature types are combined, all three genres can be separated. The scatter plot of the distances from the rhythm patterns and Jensen-Shannon-like divergence in combination shows three clusters of songs belonging to different genres. The fourth genre, "Electronic \& Dance" however can not be separated from hip-hop songs no matter what feature-set is used. But it has to be kept in mind that all these distances are distances coming from a song request of the Rock/Pop genre.

\begin{figure}[htbp]
	\centering
	\framebox{\parbox{1\textwidth}{
			\includegraphics[scale=0.29]{Images/SparkFeat/electronic.png}
	}}
	\caption{Scatter matrix, distances 1 random Electronic song, 1517-Artists, 4 genres}
	\label{fig:corr6}
\end{figure}
\FloatBarrier

\noindent As mentioned in Section~\ref{featqual} and visualized in Figure~\ref{fig:featspace}, the distribution of the distances varies depending on where in the feature space the song request is located. Apparently the songs of the Hip-Hop and Electronic/Dance genre are on average all about the same distance away from the requested Rock/Pop song. When requesting a song from the genre Electronic/Dance, the distribution of the distances look entirely different (see Figure~\ref{fig:corr6}).
\noindent The "agg" - plots represent the weighted sum of all features combined (also including cross-correlation and Levenshtein distances not shown in the plots). After the combination of all feature types, the returned results primarily recommend other Rock \& Pop songs in Figure~\ref{fig:corr5} and Electronic \& Dance songs in Figure~\ref{fig:corr6}.\\
When using only one feature type, the Spark recommendation engine would not be able to separate all four of the different genres from each other. Only due to the combination of different rhythmic and timbral features an overall satisfying list of recommendations can be retrieved.\\

\subsection{Rhythm Features}\label{rhythmrec}

\begin{figure}[htbp]
	\centering
	\framebox{\parbox{1\textwidth}{
			\begin{subfigure}{.495\textwidth}
				\centering
				\includegraphics[scale=0.5]{Images/SparkFeat/BH_BPM.png}
				\caption{BPM - BH (Rock\&Pop)}
				\label{fig:bbpm}
			\end{subfigure}
			\begin{subfigure}{.495\textwidth}
				\centering
				\includegraphics[scale=0.5]{Images/SparkFeat/RP_BPM.png}
				\caption{BPM - RP (Rock\&Pop)}
				\label{fig:rbpm}
			\end{subfigure}%

			\begin{subfigure}{.495\textwidth}
				\centering
				\includegraphics[scale=0.5]{Images/SparkFeat/bpm_agg_hip.png}
				\caption{BPM - AGG (Rock\&Pop)}
				\label{fig:arbpm}
			\end{subfigure}
			\begin{subfigure}{.495\textwidth}
				\centering
				\includegraphics[scale=0.5]{Images/SparkFeat/bpm_agg_clas.png}
				\caption{BPM - AGG (Classical)}
				\label{fig:acbpm}
			\end{subfigure}%
	}}
	\caption{Scatter plots rhythm features / BPM for random Rock\&Pop and Classical songs}
	\label{fig:rhythmfeat}
\end{figure}
\FloatBarrier

\noindent Another critical requirement for the recommendation engine is the ability to obtain songs that are about the same tempo. To investigate the capabilities of the rhythm features, Figure~\ref{fig:rhythmfeat} shows the resulting distances of two song requests performed on the 1517-Artists dataset.
\noindent The scatter plots show that the beat histogram and the rhythm patterns are closely related to the overall BPM of the songs. The "agg" value (the weighted sum) includes all eight different feature types, so the overall impact of the rhythm features on the recommendations can be seen. All in all, the Spark recommendation engine is more likely to recommend songs that have similar BPM when rhythm features are included in the weighted sum. The classical song request in Figure~\ref{fig:acbpm} also shows that the overall distances are not exclusively dominated by the BPM but rather slightly influenced.

\section{Subjective Evaluation}

This section includes the personal opinion and music taste of the author. Although these results are not "scientific", music taste is something personal and judging music recommendation solely from an objective perspective would be the wrong approach for this thesis. The core strength of this Spark-based recommender system is that its parameters can be used to personalize the music recommendations.

\subsection{Beyond Genre Boundaries}

The main reason for the choice of the topic of this thesis was that recommender systems as they come with streaming platforms like Spotify tend to value the music context information over music content. For example, the "Song Radio"- option coming with Spotify stays in the boundaries of genres and is heavily influenced by other people's listening behavior. Although this is not necessarily a bad thing, this thesis tried to focus directly on the timbral, rhythmic, and melodic properties. As a result, songs from other genres are recommended as can be seen in the following example.
\noindent When searching for the nearest neighbors of the "Prelude in C- Sharp Minor (Op. 3 No. 2)" by the Russian composer Sergei Rachmaninoff based on the Euclidean distance of MFCCs, the following results were returned:

\begin{enumerate}
	\setlength\itemsep{-0.5em}
	\item Klassik/Rachmaninoff - Piano Concerto No2 In C Minor Op18-1 Moderato
	\item Klassik/Liszt - Piano Concerto No 1 in E flat major S124(LWH4) Allegro maestoso
	\item Klassik/Brahms - Piano Sonata No2 in F sharp minor Op2 - III Scherzo allegro
	\item \underline{Metal\&Rock/Steve Moore - Intro \& Credits}
	\item Klassik/Liszt - Piano Concerto No 1 in E flat major S124(LWH4) Allegro animato
\end{enumerate}

\noindent The "Metal \& Rock" recommendation seems out of place at first glance, but when taking a closer look, the recommended song is called "Intro \& Credits" and it is not a typical Metal song. When listening to it, some similarities are recognizable; it is a calm, dark instrumental piece made of synthesizer sounds. The primarily requested Prelude is a dark piano piece. Of course, this is just one example, and the recommendation is arguably not perfect. In general some of the timbre based recommendations seem out of place. This might be due to the choice of 13 MFCC bands over 25 as the Musly toolkit uses, or potentially there are some unnoticed issues with the implementation left, which would have to be investigated in future work. But as also stated in Section~\ref{genrerec}, the overall performance concerning the genre recall rate is reasonably good aside from a few outliers.

\subsection{Personal Music Taste}\label{personal}

As a last side note on personal music taste, a song request using one of my favorite songs was made. As already mentioned, my private music collection was a part of this thesis. To retain some kind of reproducibility the whole collection is documented, and the pertinent list of albums and songs is on a document on the CD in the appendix. On the last pages of this document, there is also a list containing my personal song favorites in the metal music genre. One of these songs was chosen, and recommendations were calculated for the private music collection. The song is called "The Art Of Dying" by the band Gojira. The recommendations based purely on rhythm patterns are listed below. Another track from my personal list of favorite song appears as a recommendation.

\begin{itemize}
	\setlength\itemsep{-0.5em}
	\item Numenorean - Adore
	\item Shylmagoghnar - Transience
	\item \underline{Amon Amarth - The Last Stand Of Frej}
	\item Delain - We Are the Others
	\item Ensiferum - Descendants Defiance Domination
\end{itemize}

\noindent This could be an indication that my taste in music is closely related to the rhythmic properties of the music.
An idea for future research could be to reverse engineer a user's musical taste by looking at a list of favorite songs. The information which songs a user likes the most is already available to all streaming platforms because most likely the songs a user listens to the most are also the best liked songs. Spark could be used to calculate the similarities between these favorite songs of a user and analyze the distances. Whether or not these songs are more similar in rhythm, melody or timbre could enhance the parametrization of a recommender engine and further personalize music recommendations by adapting the weights of a recommendation engine.\\
Of course, the field of personalized music recommendation is an already existing one, but maybe the addition of Spark and Big Data opportunities of using audio content instead of contextual information and collaborative filtering could enhance these existing systems.

\chapter{Summary}\label{summarych}

In this last chapter, the results of this thesis are summarized and a short outlook on open tasks and possibilities for future work are given.

\section{Conclusion}

Looking back at the content of this thesis, Chapter~\ref{audiofeat} provided an overview of the field of music information retrieval. Different high- and low-level audio features were explained, and various ways to measure the similarities between audio files based on the audio features were introduced. Additionally, a short introduction to Big Data frameworks, especially Apache Spark and Hadoop was given, and different audio data sources were gathered. Chapter~\ref{simanal} presented ways to extract and pre-process timbre, rhythm, and melodic features from audio files. Multiple algorithms for calculating the distances between the extracted features were given. With the theoretic knowledge from the first chapters, the implementation could be planned. Data was collected; over 1TB of music files containing 114000 different songs were aggregated.\\
In the first part of the implementation, the necessary audio features were extracted and pre-processed (e.g., by extracting the melody from chroma features) in parallel using MPI on a computer cluster, paving the way for the usage with the Big Data processing framework Spark.
The features were loaded into the HDFS of a cluster, and multiple similarity measurements were implemented, tested, evaluated, and improved using the Spark framework. With Spark, multiple approaches (RDD, DataSet, Filter and Refine, Cluster Configurations) were tested, and the runtime was measured. The resulting distances were presented, analyzed, and visualized.\\
\ \\
The final application handles the recommendation of songs similar to a song request by computing the distances based on melodic, rhythmic and timbral properties of the music. The recommendations are parameterized, giving the user the option to prioritize different aspects of the music. The system is scalable. More songs can be added, the cluster size can be increased, and the possibility to add different kinds of audio features and more state-of-the-art similarity measurements is also given.

\section{Performance}

The extraction of the features on a single PC would have taken approximately 258 hours for about 100000 songs using the Essentia toolkit. By using a computer cluster with 648 concurrent threads and Mpi4py the computation time could be reduced to about half an hour (32 minutes and 30 seconds). This is approximately 476 faster than on a single PC core.
The extraction of the rhythm patterns and rhythm histograms with the rp\_extractor tool provided by the TU Wien for the same number of songs takes about the same amount of time on the ARA-cluster (also parallelized with Mpi4py).\\
The computation of similarities using the Big Data framework Spark on a 16 node computer cluster takes approximately 14 seconds for all of the 8 features types combined. This processing duration could be reduced to about 12 seconds by using a filter and refine method. It can also be reduced by using only subsets of the features types.\\

\section{Outlook}

There are still a few minor flaws, especially when looking at the implementation of the symmetric Kullback-Leibler divergence and the Jensen-Shannon divergence and the scaling of the distances. The different starting points for possible future research were laid out during the whole thesis and are summarized here. First of all the file format issues with *.wav and *.ogg audio files when using the rp\_extract tool from the TU Wien should be fixed to allow the computation of all features from all the songs of a dataset (see Section~\ref{totamsong}).
The next step would be to re-evaluate the Jensen-Shannon-like divergence and the symmetric Kullback-Leibler divergence and fix the issues with outliers. The issues with non-invertible or non-singular covariance matrices should be investigated as well (see Section~\ref{sparkskl}). The proposed enhancements by Schnitzer~\cite{schnitzer1} of reducing the hubness with mutual proximity and by using more mel bands for the computation of the MFCCs might also be sufficient to improve the quality of recommendations (see Section~\ref{mprox}). Scaling of the different features could be improved in a way where all features are evenly distributed over the unit interval (see Section~\ref{distsc}).\\
Tests of the performance on larger clusters and with more songs would be critical to assess the scaling of the problem.
An implementation of the Spark streaming abilities to enable real-time computation of similarities instead of using batch-processing jobs would be the next logical step if the objective was to develop a system able to run with music streaming platforms. When evaluating the genre recall rate with Spark, an issue with the garbage collection running out of memory after about 40 subsequent song requests was encountered and should be fixed first.\\
\noindent As another way of improving the presented Spark application, more state-of-the-art similarity measurements like block-level features (Section~\ref{blocklevel}) or the TF-IDF weights (Section~\ref{textretr}) for melodic similarity could be added. The most promising enhancement for the developed recommendation engine in this thesis would be the addition of genre and metadata information, genre-specific features, collaborative filtering, and lyrics (see Section~\ref{collaborative}). All the contextual music information that would typically be processed by a Big Data framework was not included in this thesis but could significantly enhance recommendations. Most streaming services already have all the information needed like user's listening behavior or audio metadata available. Services like Spotify are already using Spark for collaborative filtering so the Spark application presented in this thesis could be added and integrated into running streaming systems. A last suggestion for future enhancements is to investigate the proposal from Section~\ref{personal} of personalized music recommendation based on the audio feature similarities of a user's favorite songs made available by the Big Data framework.

%\addtolength{\textheight}{-12cm}   % This command serves to balance the column lengths
% on the last page of the document manually. It shortens
% the textheight of the last page by a suitable amount.
% This command does not take effect until the next page
% so it should come on the page before the last. Make
% sure that you do not shorten the textheight too much.


\documentclass[12pt,fleqn]{article}
\usepackage{latexsym}
%\usepackage{amsmath}
\usepackage{amssymb}
%\usepackage{amsbsy}
%\usepackage{amsthm}
\usepackage{makeidx}
\usepackage{alltt}
\usepackage{../LaTeX/layout}
\usepackage{graphicx}
\usepackage{url}

% ---------------------------------------------------------------------
% Input defined macros and commands
% ---------------------------------------------------------------------
\input{../LaTeX/commands}

\newcommand{\tsu}[1]{\textsf{\textup{#1}}}
\newcommand{\semb}[1]{\ensuremath{[\![#1]\!]}}
\newcommand{\rvset}{\ensuremath{V\!\!AR}}
\newcommand{\wff}{\ensuremath{w\!f\!\!f}}
\newcommand{\nnf}{\ensuremath{N\!N\!F\,}}
\newcommand{\ctl}{\textsf{CTL}\,}
\newcommand{\hc}{HolCheck}
\newcommand{\hand}{\texttt{/\char'134}}
\newcommand{\hor}{\texttt{\char'134/}}
\newcommand{\hnot}{\texttt{\~{}}}
\setcounter{sessioncount}{0}
\makeindex
\title{HolCheck}
\date{}
\begin{document}
\maketitle
\index{holCheckLib|(}\index{model checking|see {holCheckLib}}
This is a brief description of the \hc{}\index{holCheckLib!\hc{}} model checker, version 1.0. It assumes the reader is familiar with symbolic model checking, with \HOL{}, and, ideally, with \texttt{HolBddLib}, the \HOL{} interface to BDDs. It should be read with a working \HOL{} installation at hand.

\subsection{Introduction}\label{sec:intro}

\hc{} is a BDD-based model checker embedded in the \HOL{} theorem prover, interfaced to an external BDD engine for efficiency. It currently has the following features:

\begin{itemize}
\item All steps of the algorithm are proved in \HOL{}, with BDD operations considered atomic.
\item Results are returned as \HOL{} theorems.
\item Model checking for the propositional \(\mu\)-calculus and \ctl is supported.
\item Counterexamples and witnesses are generated when appropriate.
\item A fully-automatic counterexample-guided abstraction refinement framework is included.
\end{itemize}

The user inputs a model and a list of properties to be checked, and the model checker returns theorems, counterexamples and/or witnesses for each property with respect to the model. Models are formally treated as Kripke structures\index{holCheckLib!models!Kripke structure} described by the quintuple \( (S, S_0, T, P, L) \) where

\begin{itemize}
\item \( S \) is the set of states of the model. Each state is modelled as a tuple of state variables. Currently only boolean state variables are supported.
\item \( S_0\) is the set of initial states.
\item \( T \) is the set of actions or program letters, such that for each \( a \in T \), we have \( R_a \subseteq S \times S\). This says that the model has a transition labelled \( a \) from a state \( s \) to a state \( s' \) if and only if \( R_a(s,s') \). Next state variables are distinguished from current state variables by priming.
\item \( P \) is the set of atomic propositions.
\item \( L:S\rightarrow 2^P \) labels each state with those propositions that are true in it.
\end{itemize}

See \S\ref{sec:models} for details. The user is required to supply only \( S_0 \) and \( T \) as the remaining components can be calculated from these. Properties are input as \HOL{} terms of type \(\mathtt{mu}\) or \(\mathtt{ctl}\), \HOL{} datatypes for which have been defined (see \S\ref{sec:prop} for details). A property is satisfied by a model if the set of initial states of the model is a subset of the set of states satisfying the property.

\subsection{Quick Usage Overview}

This section provides a quick look at a typical use of \hc{}. We choose a simple mod-8 counter as our example, which starts at 0 and counts up to 7, and then loops from 0 again. We wish to show that the most significant bit eventually goes high.

First we load the \hc{} library:
\begin{session}\begin{verbatim}
- load "holCheckLib"; open holCheckLib;
  (* we don't show the output from the "open" here *)
> val it = () : unit
\end{verbatim}\end{session}
Now we specify the labelled transition system as a list of
(string, term) pairs, where each string is a transition label and
each term represents a transition relation (three booleans
required to encode numbers 0-7; next-state variable values
indicated by priming; note we expand the xor, since \hc{}
currently requires purely propositional terms) :
\begin{session}\begin{verbatim}
- val xor_def = Define `xor x y = (x \/ y) /\ ~(x=y)`;
  val TS = List.map (I ## (rhs o concl o
           (fn tm => REWRITE_CONV [xor_def] tm handle ex => REFL tm)))
                [("v0",``(v0'=~v0)``),("v1",``v1' = xor v0 v1``),
                 ("v2",``v2' = xor (v0 /\ v1) v2``)]
Definition has been stored under "xor_def".
> val xor_def = |- !x y. xor x y = (x \/ y) /\ ~(x = y) : thm
> val TS =
    [("v0", ``v0' = ~v0``), ("v1", ``v1' = (v0 \/ v1) /\ ~(v0 = v1)``),
     ("v2", ``v2' = (v0 /\ v1 \/ v2) /\ ~(v0 /\ v1 = v2)``)] :
  (string * term) list
\end{verbatim}\end{session}
Next we specify the initial states (counter starts at 0):
\begin{session}\begin{verbatim}
- val S0 = ``~v0 /\ ~v1 /\ ~v2``;
> val S0 = ``~v0 /\ ~v1 /\ ~v2`` : term
\end{verbatim}\end{session}
Now we say whether the transitions happen synchronously (they do):
\begin{session}\begin{verbatim}
- val ric = true;
> val ric = true : bool
\end{verbatim}\end{session}
Then we set up the state tuple:\index{holCheckLib!ML bindings!mk\_state@\ml{mk\_state}}
\begin{session}\begin{verbatim}
- val state = mk_state S0 TS;
> val state = ``(v0,v1,v2)`` : term
\end{verbatim}\end{session}
Next we specify a property (there exists a future in which the most significant bit, v2, will go high). Note how atomic propositions are represented as functions on the state :
\begin{session}\begin{verbatim}
- val ctlf = ("ef_msb_high",``C_EF (C_BOOL
(B_PROP ^(pairSyntax.mk_pabs(state,``v2:bool``))))``);
> val ctlf = ("ef_msb_high",``C_EF (C_BOOL(B_PROP(\(v0,v1,v2). v2))``)
  : string * term
\end{verbatim}\end{session}
Now we can create the model. This is done by starting with the supplied \texttt{empty\_model} value and filling it using the \texttt{set\_*} functions:
\index{holCheckLib!ML bindings!set\_init@\ml{set\_init}}
\index{holCheckLib!ML bindings!set\_trans@\ml{set\_trans}}
\index{holCheckLib!ML bindings!set\_flag\_ric@\ml{set\_flag\_ric}}
\index{holCheckLib!ML bindings!set\_state@\ml{set\_state}}
\index{holCheckLib!ML bindings!set\_props@\ml{set\_props}}
\index{holCheckLib!ML bindings!empty\_model@\ml{empty\_model}}\index{holCheckLib!models!empty\_model@\ml{empty\_model}}
\index{holCheckLib!models!creating}
\begin{session}\begin{verbatim}
- val m = ((set_init S0) o (set_trans TS) o (set_flag_ric ric) o
           (set_state state) o  (set_props [ctlf])) empty_model;
> val m = <model> : model
\end{verbatim}\end{session}
At last we can call the model checker...
\index{holCheckLib!ML bindings!holCheck@\ml{holCheck}}
\begin{session}\begin{verbatim}
- val m2 = holCheck m;
> val m2 = model : <model>
\end{verbatim}\end{session}
...and examine the results:
\index{holCheckLib!ML bindings!get\_results@\ml{get\_results}}\index{holCheckLib!traces}
\begin{session}\begin{verbatim}
- get_results m2;
> val it =
  SOME [(<term_bdd>,
   SOME [............................]
    |- CTL_MODEL_SAT ctlKS (C_EF (C_BOOL (B_PROP (\(v0,v1,v2). v2)))),
   SOME [``(~v0,~v1,~v2)``, ``(v0,~v1,~v2)``, ``(~v0,v1,~v2)``,
                 ``(v0,v1,~v2)``, ``(~v0,~v1,v2)``])] :
  (term_bdd * thm option * term list option) list option
\end{verbatim}\end{session}
The result is a list of triples, one triple per property checked. The first component contains the BDD representation of the set of states satisfying the property. The second component contains a theorem certifying that the property holds in the model i.e. it holds in the initial states. The third contains a witness trace that counts up to 4, at which point v2 goes high.

Note that since we did not supply a name for the model, the system has chosen the default name \texttt{ctlKS}, which stands for ``CTL Kripke structure''. Names can be given via \texttt{set\_name}\index{holCheckLib!ML bindings!set\_name@\ml{set\_name}}\index{holCheckLib!models!naming}. See \S\ref{sec:formal_models} for more on how \hc{} represents formal models.

Note also that the theorem created has several assumptions. This is because \hc{}\, creates proofs lazily. We can verify the proof...\index{holCheckLib!ML bindings!prove\_model@\ml{prove\_model}}
\begin{session}\begin{verbatim}
- val m3 = prove_model m2;
  (* we don't show the prove_model "chatting" messages here *)
> val m3 = <model> : model\end{verbatim}\end{session}
...and examine the verified results:
\begin{session}\begin{verbatim}
- get_results m3;
> val it =
   SOME [(<term_bdd>,
    SOME|- CTL_MODEL_SAT ctlKS (C_EF(C_BOOL(B_PROP(\(v0,v1,v2). v2)))),
    SOME [``(~v0,~v1,~v2)``, ``(v0,~v1,~v2)``, ``(~v0,v1,~v2)``,
                 ``(v0,v1,~v2)``, ``(~v0,~v1,v2)``])] :
  (term_bdd * thm option * term list option) list option
\end{verbatim}\end{session}
The theorem component now has no assumptions. Any assumptions to the term\_bdd would also have been discharged.

\subsection{Detailed Usage}

This section describes the usage of \hc{} in more detail. As a running example we will take the children's game of tic-tac-toe or noughts-and-crosses, usually played on a 3-by-3 grid. For simplicity and to keep the formulas under consideration manageable, we consider the trivial 1-by-1 variant. Thus, the grid starts off empty, the player to go first makes a move and immediately wins, ending the game.

For this model, we need one state variable to track whose move it is, and we shall call this \(m\), which is true if the first player (call her A) is to move and false if the second player (call him B) is to move. We also need two state variables to keep track of the moves already played. The state variable \(u_{0,0}\) is true if A has claimed grid cell \((0,0)\) and false otherwise, and similarly the state variable \(v_{0,0}\) tracks B's moves. It is easy to see how this scheme can be generalised to arbitrary grids.

The state is described by the tuple \( (m,u_{0,0},v_{0,0}) \). \(S_0\) is \( \{ (T,F,F) \} \) and \( T \) is

\begin{eqnarray*}
 [&(&''{u_{0,0}}'',(\lnot u_{0,0} \land \lnot v_{0,0} \land m) \land \lnot((\lnot m \land u_{0,0}) \lor (m \land v_{0,0}))\\
       &&\land (u_{0,0}' \land \lnot m' \land (v_{0,0}' = v_{0,0}))),\\
     &(&''{v_{0,0}}'',(\lnot v_{0,0} \land \lnot u_{0,0} \land \lnot m) \land \lnot((\lnot m \land u_{0,0}) \lor (m \land v_{0,0}))\\
        &&\land (v_{0,0}' \land m' \land (u_{0,0}' = u_{0,0})))]
\end{eqnarray*}

This says for instance that the move \(''{u_{0,0}}''\) may be played if and only if the board is empty and it is A's turn AND the game is not already over AND in the next state A will have claimed \((0,0)\) and it will be B's turn, whose status will remained unchanged. And similarly for the move \(''{v_{0,0}}''\).

Note that although there are eight possible states, only two (\((T,F,F)\) and \((F,T,F) \)) are actually reachable.

For our running example, we use \texttt{ttt.sml} from \texttt{src/HolCheck/examples}. Build this (and others\footnote{The AMBA examples are large and take a while to build. To skip them, see the README file in the same folder.}) by typing \texttt{Holmake} in \texttt{src/HolCheck/examples}. Now start \HOL{} from this directory (or use the -I command-line parameter).

The first step after loading \HOL{} is to load \hc{}. \HOL{} messages have been elided.

\setcounter{sessioncount}{0}

\begin{session}\begin{verbatim}
- app load ["holCheckLib","ttt"];
> val it = () : unit
\end{verbatim}\end{session}

The user interacts with \hc{} through a single ML function \texttt{holCheck}\index{holCheckLib!ML bindings!holCheck@\ml{holCheck}}, found in the library \texttt{holCheckLib}.

\begin{session}\begin{verbatim}
- holCheckLib.holCheck;
> val holCheck = fn : model -> model
\end{verbatim}\end{session}

The function takes a \hc{} model as an argument, and returns a model which is the same as the argument, but additionally contains the results of the model checking. We now describe how a \hc{} model is constructed.

\subsubsection{The Input Model}

A \hc{} model is a data structure that contains all the information required for model checking. We use the model construction function provided in the \texttt{ttt} example to create the model.

\begin{session}\begin{verbatim}
- val ttt1 = ttt.makeTTT 1;
> val ttt1 = <model> : model
\end{verbatim}\end{session}

The numeric argument to \(\mathtt{ttt.makeTTT}\) gives the size of grid required.  Now we consider each component of the model \texttt{ttt1}, using the \texttt{holCheckLib.get\_*} family of functions. The corresponding \texttt{holCheckLib.set\_*} functions were used to actually construct the model. The argument to each \texttt{set} function is precisely the return value of the corresponding \texttt{get} function as seen below. The only difference is that the values of optional model components are returned as option types. See the source code of \texttt{mod8.sml} in the HolCheck \texttt{examples} directory for a simple example of how this is done. See \S\ref{sec:models} for general details on model construction.

\begin{session}\begin{verbatim}
- val S0 = holCheckLib.get_init ttt1;
> val S0 = ``~u0_0 /\ ~v0_0 /\ m`` : term
\end{verbatim}\end{session}
\index{holCheckLib!ML bindings!get\_init@\ml{get\_init}}
\(\mathtt{S0}\) is a term over the state variables of the model, such that only those assignments to the variables that satisfy the initial states of the model also satisfy \(\mathtt{S0}\). Thus \(\mathtt{S0}\) denotes the initial states. Note that since only boolean variables are supported, \(\mathtt{S0}\) must be a purely propositional term.

\begin{session}\begin{verbatim}
- val TS = holCheckLib.get_trans ttt1;
> val TS =
    [("u0_0",
      ``~u0_0 /\ ~v0_0 /\ m /\
        ~(~m /\ (u0_0 \/ u0_0 \/ u0_0 \/ u0_0) \/
          m /\ (v0_0 \/ v0_0 \/ v0_0 \/ v0_0)) /\ u0_0' /\ ~m' /\
        (v0_0' = v0_0)``),
     ("v0_0",
      ``~v0_0 /\ ~u0_0 /\ ~m /\
        ~(~m /\ (u0_0 \/ u0_0 \/ u0_0 \/ u0_0) \/
          m /\ (v0_0 \/ v0_0 \/ v0_0 \/ v0_0)) /\ v0_0' /\ m' /\
        (u0_0' = u0_0)``)] : (string * term) list
\end{verbatim}\end{session}
\index{holCheckLib!ML bindings!get\_trans@\ml{get\_trans}}
\(\mathtt{TS}\) is a list of pairs of the form \( (a,R_a)\), where the string \(a\) is an action name (or transition label) and the \HOL{} term \(R_a\) is a relation over current and next states such that the model has a transition labelled \(a\) from a state \( s \) to a state \(s'\) if and only if the corresponding valuations of current and next state variables satisfy \( R_a \). In this example, there are two actions, corresponding to the two possible moves ``u0\_0'' and ``v0\_0'', as explained earlier. Of course, in this particular case, the second move is never actually played. We have here edited out the redundancy in \(\mathtt{TS}\), which is an artifact of how \(\mathtt{ttt.makeTTT}\) generates the grid.

Thus \(\mathtt{S0}\) and \(\mathtt{TS}\) are just \HOL{} equivalents of \(S_0\) and \(T\) as described in section \ref{sec:intro}. \( \mathtt{TS} \) is named as such to avoid clashing with the global \HOL{} term constant \( \mathtt{T} \).

\begin{session}\begin{verbatim}
- val ric = holCheckLib.get_flag_ric ttt1;
> val ric = false : bool
\end{verbatim}\end{session}
\index{holCheckLib!ML bindings!get\_flag\_ric@\ml{get\_flag\_ric}}
\(\mathtt{ric}\) is a boolean which is true if the transition system described by \(\mathtt{TS}\) is synchronous, and false otherwise. In a synchronous transition system, all actions must occur simultaneously. Otherwise \(\mathtt{TS}\) is asynchronous i.e., the actions are interleaved. In the case of our example for instance, \( \mathtt{TS} \) should be asynchronous since the players do not move together but interleave moves. The definition of \( \mathtt{TS} \) ensures that they do not play out of turn, or make illegal moves.

For an asynchronous system, the formal transition relation is internally represented as the disjunction of all the actions. For a synchronous system (e.g., a circuit with a single global clock), it is a conjunction. Indeed, the standard way to specify a synchronous transition relation is using the wildcard ``.'' action, with the corresponding relation being a conjunction over all possible actions. Step through the usage example given at the top of \texttt{alu.sml} in the \hc{} \texttt{examples} directory, and then examine the generated model to see how synchronous transition systems are modelled.

\begin{session}\begin{verbatim}
- val nm = holCheckLib.get_name ttt1;
> val nm = SOME "ttt" : string option
\end{verbatim}\end{session}
\index{holCheckLib!ML bindings!get\_name@\ml{get\_name}}\index{holCheckLib!models!naming}
\(\mathtt{nm}\) is an optional argument when constructing the model (hence it is stored internally as an option type). It is a string that names the formal model that is generated internally by \hc{}. Since the resulting theorems are stated in terms of this model, it is useful to have an abbreviation for the term representing the model to keep the theorem statement readable. \hc{} has a default name for models, but if the user expects to be using more than one model in the same session, then setting a name is required to avoid name clashes.

\begin{session}\begin{verbatim}
- val vord = holCheckLib.get_vord ttt1;
> val vord = SOME ["m", "m'", "u0_0", "u0_0'", "v0_0", "v0_0'"] :
  string list option
\end{verbatim}\end{session}
\index{holCheckLib!ML bindings!get\_vord@\ml{get\_vord}}\index{holCheckLib!models!variable ordering}
\(\mathtt{vord}\) is an optional argument to the model. It is a list of current- and next-state variables names, given as strings without any type information. \(\mathtt{vord}\) is used to manually supply a variable ordering to the BDD engine. \hc{} has a default ordering heuristic but it is rather primitive at present (it just interleaves current- and next-state variables).

\begin{session}\begin{verbatim}
- val st =  holCheckLib.get_state ttt1;
> val st = SOME``(m,u0_0,v0_0)`` : term option
\end{verbatim}\end{session}
\index{holCheckLib!ML bindings!get\_state@\ml{get\_state}}
\(\mathtt{st}\) is an optional argument. It is a \HOL{} tuple that explicitly describes the order in which state variables appear in the state tuple. \hc{} can automatically calculate the state from \(\mathtt{S0}\) and \(\mathtt{TS}\), but in some cases the user may wish to use the resulting theorems in other work where the state tuple has already been defined in a certain manner. \(\mathtt{st}\) is used in such cases. Note we do not use the identifier \texttt{state} as this already exists in the default HOL interactive session namespace.

At present the user must take care that \(\mathtt{vord}\) and \(\mathtt{st}\) are consistent with the rest of the arguments e.g. there are no missing variables.

\texttt{holCheckLib.get\_props} returns the properties to be model checked. This is simply a list of \ctl or \(\mu\)-calculus formulas. Since the formulas contain atomic propositions, which are modelled as functions on the state, the user inputting the formulas needs to know what the state is going to be. \hc{} provides a function \(\mathtt{mk\_state}\) for this purpose.

\begin{session}
\begin{verbatim}
- holCheckLib.mk_state;
> val it = fn : term -> (string * term) list -> term
- holCheckLib.mk_state S0 TS;
> val it = ``(m,u0_0,v0_0)`` : term
\end{verbatim}
\end{session}

Thus even though the state is an optional argument when constructing the model, in practice we nearly always use \texttt{mk\_state} and then \texttt{set\_state} to set it.

\(\mathtt{ttt1}\) contains five \ctl properties. We consider these in turn.

\begin{session}
\begin{verbatim}
- List.nth(holCheckLib.get_props ttt1,0);
> val it =
    ("isInit",
     ``~C_BOOL (B_PROP (\(m,u0_0,v0_0). u0_0)) /\
       ~C_BOOL (B_PROP (\(m,u0_0,v0_0). v0_0)) /\
       C_BOOL (B_PROP (\(m,u0_0,v0_0). m))``) : string * term
\end{verbatim}
\end{session}
\index{holCheckLib!ML bindings!get\_props@\ml{get\_props}}
The first property describes the initial state of the system. Note how atomic propositions are formally modelled as boolean functions on the state. As noted earlier, all atomic propositions are currently just boolean variables. \texttt{C\_BOOL} and \texttt{B\_PROP} are type constructors from the \HOL{} datatype for \ctl formulas. Property specification is considered in more detail in \S\ref{sec:prop}.

\begin{session}
\begin{verbatim}
- List.nth(holCheckLib.get_props ttt1,1);
> val it =
    ("A_win",
     ``~C_BOOL (B_PROP (\(m,u0_0,v0_0). m)) /\
       (C_BOOL (B_PROP (\(m,u0_0,v0_0). u0_0)))``) : string * term
\end{verbatim}
\end{session}

This property describes a winning position for A. As with \(\mathtt{TS}\) there is some redundancy here which has been edited out. We shall edit this redundancy from the following properties to save space.

\begin{session}
\begin{verbatim}
- List.nth(holCheckLib.get_props ttt1,2);
> val it =
    ("B_win",
     ``C_BOOL (B_PROP (\(m,u0_0,v0_0). m)) /\
       C_BOOL (B_PROP (\(m,u0_0,v0_0). v0_0)``) : string * term
\end{verbatim}
\end{session}

The property above similarly describes a winning position for B.

\begin{session}\begin{verbatim}
- List.nth(holCheckLib.get_props ttt1,3);
> val it =
    ("A_canwin",
     ``C_EG (~C_BOOL (B_PROP (\(m,u0_0,v0_0). m)) /\
             (C_BOOL (B_PROP (\(m,u0_0,v0_0). u0_0))) \/
             C_BOOL (B_PROP (\(m,u0_0,v0_0). m)) /\
          C_EF (~C_BOOL (B_PROP (\(m,u0_0,v0_0). m)) /\
                (C_BOOL (B_PROP (\(m,u0_0,v0_0). u0_0)))) \/
            ~C_BOOL (B_PROP (\(m,u0_0,v0_0). m)) /\
            C_AF (~C_BOOL (B_PROP (\(m,u0_0,v0_0). m)) /\
                  (C_BOOL (B_PROP (\(m,u0_0,v0_0). u0_0)))))``) :
  string * term
\end{verbatim}\end{session}

This property says that A has a winning strategy from the current state. Since we define success if the set of states satisfying the property contains the set of initial states, this suffices to check that A has a winning strategy. Cleaned up, this is the \ctl property \[ \mathbf{EG} (\textup{(A wins)} \lor (\textup{A to move} \land \mathbf{EF} (\textup{A wins})) \lor (\textup{B to move} \land \mathbf{AF} (\textup{A wins})))\] Intuitively, the property says that there exists a path from the current state such that at each step on that path either A has won the game, or it is A's turn and there exists at least one path in which A eventually wins the game, or it is not A's turn and on all paths A eventually wins the game.

The last property similarly says the B has a winning strategy.

\subsubsection{The Output Model}

The model returned by \(\mathtt{ttt.makeTTT}\) can be supplied as the first argument to \(\mathtt{holCheck}\).

\begin{session}\begin{verbatim}
- val ttt2 = holCheckLib.holCheck ttt1;
> val ttt2 = <model> : model
\end{verbatim}\end{session}

\(\mathtt{holCheck}\) returns another model. This model is exactly the same as \texttt{ttt1}, except that it now contains the results of the model checking. These are recovered via the function \texttt{holCheckLib.get\_results}, and return a list in which the n\({}^{th}\) element gives the results of model checking the n\({}^{th}\) property in the list returned by \texttt{holCheckLib.get\_props}. \texttt{holCheckLib.get\_results} returns an option type, since no results are known before model checking takes place.

Each element is a triple of the form \( (term\_bdd,thm,trace) \).\footnote{A \(\textup{term\_bdd}\) is an ML value that contains a \HOL{} term together with its boolean semantics represented as a BDD (see the \texttt{HolBddLib} documentation for details).} These are the BDD semantics of the property, a theorem certifying the property holds (if it does) and a counterexample or witness trace, if one could be recovered.

We consider each result in turn.

\begin{session}\begin{verbatim}
- List.nth(valOf (holCheckLib.get_results ttt2),0);
> val it =
    (<term_bdd>,
     SOME [.................]
         |- CTL_MODEL_SAT ctlKS_ttt
              (~C_BOOL (B_PROP (\(m,u0_0,v0_0). u0_0)) /\
               ~C_BOOL (B_PROP (\(m,u0_0,v0_0). v0_0)) /\
               C_BOOL (B_PROP (\(m,u0_0,v0_0). m))), NONE) :
  term_bdd * thm option * term list option
\end{verbatim}\end{session}

This is the result corresponding to the first property (which described the set of initial states).  The first component is the \(\textup{term\_bdd}\) that corresponds to the set of states of the model satsified by this property. As expected, the set of initial states is contained in the set of initial states and thus the second component is a success theorem attesting to this. The predicate \( \mathtt{CTL\_MODEL\_SAT} \)\index{holCheckLib!CTLMODELSAT@\ml{CTL\_MODEL\_SAT}} says that the property holds in the formal model \( \mathtt{ctlKS\_ttt}\). If the property had not held, the theorem derivation would have failed and the component would have been \( \mathtt{NONE}\). The third component is used to return the counterexample or witness associated with the property. Since we succeeded there is no counterexample, and witnesses are only returned for properties talking about paths, so this component is \( \mathtt{NONE}\). Note that the theorem has some assumptions associated with it. We shall return to these later.

\begin{session}\begin{verbatim}
- List.nth(valOf (holCheckLib.get_results ttt2),1);
> val it = (<term_bdd>, NONE, SOME []) :
  term_bdd * thm option * term list option
\end{verbatim}\end{session}

The second result corresponds to a win position for A. Since the initial state is not a win position for A, there is no success theorem and as before counterexamples are only provided for properties talking about paths. The third result is exactly the same, since the initial state is not a win position for B either.

\begin{session}\begin{verbatim}
- List.nth(valOf (holCheckLib.get_results ttt2),3);
> val it =
    (<term_bdd>,
     SOME [....................................]
         |- CTL_MODEL_SAT ctlKS_ttt
              (C_EG
                 (~C_BOOL (B_PROP (\(m,u0_0,v0_0). m)) /\
                  (C_BOOL (B_PROP (\(m,u0_0,v0_0). u0_0))) \/
                  C_BOOL (B_PROP (\(m,u0_0,v0_0). m)) /\
                  C_EF
                    (~C_BOOL (B_PROP (\(m,u0_0,v0_0). m)) /\
                     (C_BOOL (B_PROP (\(m,u0_0,v0_0). u0_0)))) \/
                  ~C_BOOL (B_PROP (\(m,u0_0,v0_0). m)) /\
                  C_AF
                    (~C_BOOL (B_PROP (\(m,u0_0,v0_0). m)) /\
                     (C_BOOL (B_PROP (\(m,u0_0,v0_0). u0_0)))))),
     SOME [``(m,~u0_0,~v0_0)``]) :
   term_bdd * thm option * term list option
\end{verbatim}\end{session}

This result corresponds to the fourth property, that stated that A has a winning strategy. This is true, and we get the success theorem, as well as a witness, which shows that there is such a path from the initial state.

The last result is the same as the second and third, since B does not have a winning strategy, and counterexamples cannot be calculated for properties that assert the existence of specific paths (since the ``counterexample'' would effectively be the entire set of reachable states).\index{holCheckLib!traces}

Finally, note that both success theorems have some assumptions. If the term\_bdd's were examined, they would also have the corresponding assumptions. This is because \hc{} uses postponed proof verification to speed up the model checking work-flow (i.e. no time is wasted proving intermediate lemmas for properties that fail eventually). Any success theorems and term\_bdd's thus end up with several undischarged assumptions. The idea is that the time-consuming proof verification can be postponed to a later time, when the user is otherwise occupied (e.g., asleep).

The assumptions are discharged as follows (chatting messages have been elided).

\begin{session}\begin{verbatim}
- val ttt3 = holCheckLib.prove_model ttt2;
> val ttt3 = <model> : model
\end{verbatim}\end{session}

This returns a model that is exactly the same as \texttt{ttt2}, except that any assumptions introduced due to the postponed proof have been discharged. We can confirm this by looking at say, the first property, again.

\begin{session}\begin{verbatim}
- List.nth(valOf (holCheckLib.get_results ttt3),0);
> val it =
    (<term_bdd>,
     SOME|- CTL_MODEL_SAT ctlKS_ttt
              (~C_BOOL (B_PROP (\(m,u0_0,v0_0). u0_0)) /\
               ~C_BOOL (B_PROP (\(m,u0_0,v0_0). v0_0)) /\
               C_BOOL (B_PROP (\(m,u0_0,v0_0). m))), NONE) :
  term_bdd * thm option * term list option
\end{verbatim}\end{session}

This completes one cycle of the \hc{} work-flow (typically calls to \texttt{prove\_model} are saved for the end of the checking, since  \texttt{prove\_model} just verifies the results fully-expansively; it cannot invalidate them unless there is a bug in \hc{}).

At this point the user would fix either \(\mathtt{S0}\) or \(\mathtt{TS}\) or the properties, and start over, until all expected properties are verified.

\subsection{General Usage Guidelines}

\texttt{holCheckLib} is different from other libraries in that it does not provide tools that fit in with the usual interactive theorem proving workflow. There are no tactics, and it is unlikely (though possible) that \hc{} would be called as a decision procedure during interactive proof. The reason for this is that the models checked by model checkers are represented explicitly as state machines, whereas in \HOL{} interactive proof the underlying ``model'' is simply that of higher-order logic.  Therefore, we need to specify both models and properties explicitly.

\subsubsection{Model construction}\label{sec:models}\index{holCheckLib!models!creating}

We have already seen how to create the ML type \texttt{model} that acts as an input into \hc{}. The three mandatory components of a model are a predicate representing the set of initial states, predicate(s) for the transition system, and the synchronous flag. The following observations should be kept in mind.

\paragraph{Initial states}\index{holCheckLib!models!initial state}

Since \hc{} supports only boolean variables, only those are allowed in the state. Only the variables in the state space are allowed here, and the term must be purely propositional. Primed variables are not allowed.

\paragraph{Transition system}\index{holCheckLib!models!transition relation}

Only the variables in the state space are allowed. They may be primed to indicate the value of that variable in the next state. It is strongly recommended that primed variables not be used for anything other than this function. \hc{} may wander into uncharted territory otherwise.

\hc{} can handle from about 50 to 100 variables currently, depending on the transition system in question. This number should rise as standard model checking optmisations such as partitioning, cone-of-influnce, and symmetry reductions are implemented.

The wildcard ``.'' action represents any action, and is always present. It may only be explicitly defined by the user if it is the only action, to prevent internal inconsistency. If it is not defined, then it is internally generated as a conjunction or disjunction over the defined actions, depending on whether or not the synchronous flag \texttt{ric} is true. It may be used in \(\mu\)-calculus properties, and represents the entire transition system for \ctl properties. Usually, for synchronous transition systems, the user defines just the ``.'' action, since all actions happen simultaneously so there is usually not much call for giving them separate identities.

See the AMBA AHB bus protocol example in the file \texttt{amba\_ahbScript.sml} in the \hc{} \texttt{examples} directory for an example of how to specify a complex transition system in a modular manner, hiding internal details using existential quantifiers.

\paragraph{The ``relation-is-conjunctive'' flag} The \texttt{ric} flag is an ML boolean type that is set to true for synchronous transition systems, and false for asynchronous or interleaved transition systems.

\subsubsection{Formal models}\label{sec:formal_models}\index{holCheckLib!models!formal}

A theorem returned by \hc{} is usually of the form \texttt{CTL\_MODEL\_SAT M f}. We have already explained what the predicate \texttt{CTL\_MODEL\_SAT} is. \texttt{f} is the property that was checked (see \S\ref{sec:prop} below), and \texttt{M} is the formal model.

The model we provide to \hc{} as the ML type \texttt{model} consists of several \HOL{} terms representing various pieces of information used by \hc{}. However, \hc{} does everything by proof, and this requires a formal \HOL{} theory of the \( \mu\)-calculus as well as of Kripke structures. Hence, the terms contained in the ML \texttt{model} type are packaged into a formal \HOL{} datatype that represents a Kripke structure and is assigned a constant name for brevity. It is this name that we see in the theorem statement in place of \texttt{M}. Indeed, a full \HOL{} definition corresponding to that name would exist in the current session's namespace, and the \HOL{} terms we supplied to the ML model would appear (perhaps abbreviated) in the right hand side of that definition.

As long as we are restricted to model checking, this internal formal model is not important. However, \hc{} has some experimental support for model decomposition using \HOL{} (see \S\ref{sec:decomp} for details). This requires manipulation of the formal model and then it becomes important to know what the models contain. For the moment this support is automatic, so in-depth knowledge of formal models is not required. This subsection is here, for the moment, mainly as an explanation for what we see in the \hc{} output theorems.

\subsubsection{Writing properties}\label{sec:prop}\index{holCheckLib!models!properties}

\hc{} supports model checking for \ctl and \(\mu\)-calculus properties. It is beyond the scope of this description to describe these languages in detail here (e.g., see \emph{Model Checking} by Clarke et al, MIT Press, 1999). Properties are specified as \HOL{} terms of type \texttt{ctl} or \texttt{mu}.

\begin{table}
\caption{\HOL{} \(\mu\)-calculus formula syntax}
\label{tab_mu}
\begin{tabular}{|l|l|l|}
\hline
\(\mu\)-calculus operator & \HOL{} type constructor & Overloaded syntax \\
\hline
true & \texttt{TR} & \texttt{T} \\
false & \texttt{FL} & \texttt{F} \\
$\lnot f$ & \texttt{Not} & $\hnot f$  \\
$f \land g$ & \texttt{And} & $f\, \hand\, g$ \\
$f \lor g$ & \texttt{Or} & $f\, \hor\, g$ \\
proposition $p$ & \texttt{AP} & \texttt{AP p} \\
$\langle a \rangle$ f& \texttt{DIAMOND} & \verb+<<"a">> f+ \\
$[a] f$  & \texttt{BOX} & \verb+[["a"]] f+ \\
variable $Q$ & \texttt{RV} & \texttt{RV Q} \\
$\mu Q . f$ & \texttt{mu} & \texttt{mu Q .. f} \\
$\nu Q . f$ & \texttt{nu} & \texttt{nu Q .. f} \\
\hline
\end{tabular}
\end{table}

The type constructors for \texttt{mu} \index{holCheckLib!properties!mucalculus@\(\mu\)-calculus} are shown in Table \ref{tab_mu}, together with overloaded syntax for ease of entry. For example, the property saying that there exists a path to a state where \( v_0 \land v_1 \) holds is represented by the \(\mu\)-calculus formula \( \mu Q. (v_0 \land v_1) \lor \langle.\rangle Q \) and would be written in \HOL{} as

\begin{verbatim}
   mu "Q" .. AP (\(v_0,v_1). v_0 /\ v_1) \/ (<<".">> RV "Q")
\end{verbatim}

Note that atomic propositions are written as functions on the state, and that \(\mu\)-calculus variables and action names are modelled as \HOL{} strings. For internal reasons, these strings must be alphanumeric with underscores, and begin with a letter, the only exception being the special ``.'' action.

Clearly, \(\mu\)-calculus ``variables'' as modelled above are not treated as variables by \HOL{}. We use strings because \HOL{} does not have native support for variable binding in higher-order abtract syntax (this is an active research area). The treatment of free and bound variables within \hc{} is however consistent with standard definitions.

\begin{table}
\caption{\HOL{} \ctl formula syntax}
\label{tab_ctl}
\begin{tabular}{|l|l|l|}
\hline
\ctl operator & \HOL{} type constructor & Overloaded to \\
\hline
true & \texttt{B\_TRUE} & \texttt{T} \\
$\lnot f$ & \texttt{B\_NOT} & $\hnot f $  \\
$f \land g$ & \texttt{B\_AND} & $f\,\hand\,g$ \\
proposition $p$ & \texttt{B\_PROP} & \texttt{B\_PROP p} \\
$\mathbf{EX} f$ & \texttt{C\_EX} & \texttt{C\_EX} f \\
$\mathbf{EG} f$ & \texttt{C\_EG} & \texttt{C\_EG} f \\
$\mathbf{E}(f\,\mathbf{U}\,g)$ & \texttt{C\_EU} & \texttt{C\_EU}(f,g) \\
\hline
\end{tabular}
\end{table}

Table \ref{tab_ctl} describes the syntax for \ctl \index{holCheckLib!properties!ctl@\ctl}. Note that some of the operators are missing e.g. \( \mathtt{F} \) and \( \mathbf{AX} \). These are defined indirectly in terms of the rest, with the expected overloaded syntax. The syntax is structured slightly differently in that any subformula from the propositional fragment (i.e., the first four rows of Table \ref{tab_ctl}) is always wrapped within the type constructor \texttt{C\_BOOL}.

The example property above is the \ctl formula \( \mathbf{EF} (v_0 \land v_1) \) and would be written in \HOL{} as

\verb+C_EF (C_BOOL (B_PROP (\(v_0,v_1). v_0 /\ v_1)))+

where \(\mathbf{EF} f\) has the standard definition \( \mathbf{EU}(T,f) \).

\subsubsection{Environments}

\(\mu\)-calculus model checking, in addition to having a model, also requires an environment: a mapping from variables to sets of states. The \hc{} core accepts environments as maps from strings to \texttt{term\_bdd}'s. However, the user interface for the moment has no provision for supplying an environment. This will be added in due course. In the meantime, environments can be ``faked'' by purely propositional subformulas, though this may not be particularly efficient.

\ctl model checking does not require environments.

\subsection{Experimental Support for Decomposition}\label{sec:decomp}

One of the motivations behind \hc{} is to see if tight integration with \HOL{} helps the model checking work flow. In this spirit, \hc{}  includes some experimental support for model decomposition. The basic idea is that if the model consists of several components joined by parallel synchronous composition, then a property holding in one component should hold in the entire model. This seeks to exploit locality in the structure of models and properties.

For those familiar with model checking, this is \emph{not} assume-guarantee reasoning, nor does it use simulation pre-orders. It is considerably more simple: take a product Kripke structure, break up the components, model check each component, re-compose. Unlike assume-guarantee reasoning, this does not apply to properties that address truly global aspects of the system. Also, it is limited to synchronous systems. Nevertheless, there are some applications in hardware verification.

Support is via two functions, \texttt{mk\_gen\_par\_sync\_comp\_thm} and \texttt{mk\_par\_sync\_comp\_thm}. These functions are not part of the \texttt{holCheckLib} signature, as the user interface for decomposition support is considered unstable at the moment. They are instead found in the \texttt{decompTools} structure.

The first function takes initial state and transition system predicates for two models with non-overlapping state spaces, and returns a theorem that says that if any well-formed property holds in the first model, then it holds in the parallel synchronous composition of the two models.

The second function takes the theorem generated by the first function, and a property, and discharges the well-formedness assumptions to return a theorem saying that if the supplied property holds in the first model, then it holds in the composed model.

In time we expect to integrate this fully into \hc{} as a sort of localisation reduction. In the meantime, the interested reader can take a look at the synchronous composition section of \texttt{ambaScript.sml} in the \texttt{examples} directory. Even without any knowledge of AMBA, the ten or so lines give a flavour of how decomposition can be used.

\subsection{\hc{} Internals and Caveats}

This section contains information on some internal details and quirks that may help with creating models and properties for \hc{}.

\subsubsection{Variable ordering}\index{holCheckLib!models!variable ordering}

\hc{} users are strongly recommended to supply a BDD variable ordering for the model using the \texttt{set\_vord} function, even though it is optional. \hc{} does have its own variable ordering heuristic. This just interleaves current- and next-state variables however, and is not very good. Remember that the ordering supplied must include next-state (i.e., primed) variables, and must include all state variables.

\subsubsection{Counterexamples}\index{holCheckLib!traces}

The generation of witness traces for properties of the form \(\mathbf{EG} f \) (or dually, counterexamples for \(\mathbf{AF} f \) style properties) leaves much to be desired: it tries to give the shortest qualifying path which usually turns out to be just the initial state, whereas typically the user is interested in the longest one.

\subsubsection{Abstraction}

\hc{} implements a fully automatic abstraction refinement framework that is invisible to the user. A simple heuristic analysis is used to decide whether or not to apply abstraction for a given property. Sometimes the abstraction may make things worse. If \hc{} appears to be taking overly long on a property, the user may try redoing the check, after turning off abstraction. This can be done by calling \texttt{set\_flag\_abs} on the model. Turning abstraction on and off on a per-property basis is not supported yet.

\subsubsection{Lazy proof}

All steps of the model checking (except BDD operations) are backed up by fully expansive proof. To get around the speed penalty imposed by this proof, \hc{} performs proof lazily. This means that any theorem produced by \hc{} may have several undischarged assumptions (it is then known as a lazy theorem). Typically, once all properties have been checked, the user calls \texttt{prove\_model}\index{holCheckLib!ML bindings!prove\_model@\ml{prove\_model}} on the model to discharge the assumptions. This step may take a while for big models and properties, in which case it is advisable to let it run when computing resources are expected to be free, e.g., overnight.

A failure to discharge all assumptions means a completeness bug in the \hc{} proof verification machinery. However, it does not invalidate the soundness of the theorem, modulo the soundness of \HOL{}.

\texttt{prove\_model}\index{holCheckLib!ML bindings!prove\_model@\ml{prove\_model}} must be used in the same session in which the model checking was done, if the produced theorems are going to be written to permanent storage. This is because the lazy proof machinery is unable to persist lazy theorems, owing to Moscow ML's inability to persist closures.

\subsubsection{Permanent Storage}\index{holCheckLib!models!saving}

For the moment, \hc{} does not support persistence for models. The theorems produced can however be saved to permanent storage as \HOL{} theorems.

\index{holCheckLib|)}

\printindex

\end{document}
%%% Local Variables:
%%% mode: latex
%%% TeX-master: "description"
%%% End:


\section{Lookup table used for accessing child voxels using a parent's child descriptor}
\label{app:lookup-table}

\lstset{language=C,caption={The lookup table and function used for accessing child voxels using a parent's child descriptor},label=lookuptable}
\begin{lstlisting}[frame=single]
// A table for getting the child index for the child child_index
// the index in this array = (parent's_childmask << child index)
__constant__ int32_t child_index_table[] =
{
	0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4,
	1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,
	1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,
	2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
	1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,
	2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
	2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
	3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,
	1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,
	2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
	2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
	3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,
	2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
	3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,
	3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,
	4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8,
};

__device__ int32_t get_child_index(uint32_t mask,
                                   uint32_t child_idx)
{
	return child_index_table[(mask << child_idx) & 0xFFu];
}
\end{lstlisting}