\section{Introduction}
Reinforcement learning is about how an agent can learn to interact with its environment. Reinforcement learning uses the formal framework of Markov decision processes to define the interaction between a learning agent and its environment in terms of states, actions, and rewards.

\setcounter{subsection}{2}
\subsection{Elements of Reinforcement Learning}
\begin{description}
     \item[Policy] defines the way that an agent acts, it is a mapping from perceived states of the world to actions. It may be stochastic.
     \item[Reward] defines the goal of the problem. A number given to the agent as a (possibly stochastic) function of the state of the environment and the action taken.
     \item[Value function] specifies what is good in the long run, essentially to maximise the expected reward. The central role of value estimation is arguably the most important thing that has been learned about reinforcement learning over the last six decades.
     \item[Model] mimics the environment to facilitate planning. Not all reinforcement learning algorithms have a model (if they don't then they can't plan, i.e. must use trial and error, and are called model free).
\end{description}


\section{Conclusions and Future Work}
\label{sec:lts-transformation:conclusions}

In this chapter, we addressed research question~\RQ{6} for restricted forms of models and model transformations.
We presented a technique to check whether refining model transformations preserve properties.
It is aimed at verifying the correctness of complex models that are the result of iterative refinement through model transformation.
Models are formally represented by networks of LTSs and model transformations as rule systems.
We can check whether specific safety, liveness, and fairness properties are preserved by rule systems that are terminating, confluent, and synchronization uniform.
If a rule system preserves a property that holds for a given input model, construction and exploration of the LTS of a model obtained by transformation can be avoided.
If, however, the proposed technique cannot establish that a transformation preserves a property for all models, it is still possible that this property is preserved for certain input models.
In such cases, traditional techniques for model checking can be employed to verify the property for the transformed model.
Checking multiple properties simply involves performing the required checks for multiple hiding sets.
Experiments have shown that checking whether a transformation preserves a given property outperforms rechecking the property for transformed models.

There are two main directions for future research, which are aimed at extending the presented approach to more expressive formalism for the description of models and model transformations.
First, the concept of networks of LTSs could be extended to support additional features such as asynchronous communication and variables for storing information.
Second, a more expressive formalism to describe model transformations could be introduced.
Currently, for example, it is not possible to express the addition or removal of processes, which is also not yet supported by the technique for checking property preservation.
By extending the expressiveness of the formalisms used to describe models and model transformations, and extending the technique for checking property preservation correspondingly, an automated alternative for manual correctness proofs such as those described in Chapter~\ref{chap:reusable-correct-transformations} could be obtained.


\documentclass[11pt,oneside]{article}	%use"amsart"insteadof"article"forAMSLaTeXformat
\usepackage{geometry}		%Seegeometry.pdftolearnthelayoutoptions.Therearelots.
\geometry{letterpaper}		%...ora4paperora5paperor...
%\geometry{landscape}		%Activateforforrotatedpagegeometry
%\usepackage[parfill]{parskip}		%Activatetobeginparagraphswithanemptylineratherthananindent
\usepackage{graphicx}				%Usepdf,png,jpg,orepsßwithpdflatex;useepsinDVImode
								%TeXwillautomaticallyconverteps-->pdfinpdflatex
\usepackage{amssymb}
\usepackage[colorlinks]{hyperref}

%----macros begin---------------------------------------------------------------
\usepackage{color}
\usepackage{amsthm}

\def\conv{\mbox{\textrm{conv}\,}}
\def\aff{\mbox{\textrm{aff}\,}}
\def\E{\mathbb{E}}
\def\R{\mathbb{R}}
\def\Z{\mathbb{Z}}
\def\tex{\TeX}
\def\latex{\LaTeX}
\def\v#1{{\bf #1}}
\def\p#1{{\bf #1}}
\def\T#1{{\bf #1}}

\def\vet#1{{\left(\begin{array}{cccccccccccccccccccc}#1\end{array}\right)}}
\def\mat#1{{\left(\begin{array}{cccccccccccccccccccc}#1\end{array}\right)}}

\def\lin{\mbox{\rm lin}\,}
\def\aff{\mbox{\rm aff}\,}
\def\pos{\mbox{\rm pos}\,}
\def\cone{\mbox{\rm cone}\,}
\def\conv{\mbox{\rm conv}\,}
\newcommand{\homog}[0]{\mbox{\rm homog}\,}
\newcommand{\relint}[0]{\mbox{\rm relint}\,}

%----macros end-----------------------------------------------------------------

\title{Automatic generation of simplified buildings from 2D wire-frame
\footnote{This document is part of the \emph{Linear Algebraic Representation with CoChains} (LAR-CC) framework~\cite{cclar-proj:2013:00}. \today}
}
\author{Alberto Paoluzzi}
%\date{}							%Activatetodisplayagivendateornodate

\begin{document}
\maketitle
\nonstopmode

\begin{abstract}
In this module we develop a small library of functions to generate automatically simplified models of 3D buildings starting from 2D wire-frame drawings. The generated geometries will be exported to xGeoJson, a hierarchical data format extending GeoJson, the standard for geographical web maps, with assemblies and local rectangular coordinates.
\end{abstract}

\tableofcontents

%===============================================================================
\section{Introduction}
%===============================================================================

A simplified 2D layout of building floors may be generated by using either a simple web UI providing only two or three interactive graphics primitives (rect and polyline, in particular) or by exporting the output of a drawing program to a standard 2D graphics data format, in particular to \texttt{SVG} (Simple Vector Graphics, the vector graphics standard for the web). This information will be used to automatically generate a simplified 3D model of the building and to export its geometry and topology to an external data format called \texttt{xGeoJson}, a customised extension of GeoJson an opensource standard for geographical information.

%===============================================================================
\section{Transformation filters}
%===============================================================================

%-------------------------------------------------------------------------------
\subsection{From \texttt{svg} to \texttt{lar}}
%-------------------------------------------------------------------------------

\paragraph{\texttt{SVG} primitives to lar}
Two \texttt{SVG} primitives are currently used to define the wire-frame layout of the building floors using either a drawing application or an interactive user interface within the browser:  \texttt{rect} (for the definition of rectangles) and \texttt{polyline} (for the definition of closed polylines).

%-------------------------------------------------------------------------------
@D transform svg primitives to basic lar format
@{""" transform svg primitives to basic lar format """
def rects2polylines(rooms):
	return [[[x,y],[x+dx,y],[x+dx,y+dy],[x,y+dy]] for x,y,dx,dy in rooms]

def polyline2lar(polylines):
	index,defaultValue = -1,-1
	Vdict,FV = dict(),[]
	for k,polyline in enumerate(polylines):
		cell = []
		for vert in polyline:
			key = vcode(4)(vert)
			if Vdict.get(key,defaultValue) == defaultValue:
				index += 1
				Vdict[key] = index
				cell += [index]
			else:
				cell += [Vdict[key]]
		FV += [cell]
	items = TRANS(Vdict.items())
	V = TRANS(sorted(zip(items[1],AA(eval)(items[0]))))[1]
	#FV = AA(sorted)(FV)
	EV = face2edge(FV)
	return V,FV,EV
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
\subsection{From \texttt{lar} to \texttt{struct}}
%-------------------------------------------------------------------------------

\paragraph{Mapping of a \texttt{lar} model to a list of \texttt{lar} structures}

%-------------------------------------------------------------------------------
@D transform a lar model to a list of lar structures
@{""" transform a lar model to a list of lar structures """
def lar2Structs(model):
   V,FV,_ = model
   return [ Struct([[[V[v] for v in cell], [range(len(cell))]]]) for cell in FV]
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
@D Struct class pushing local origin at the bottom of the structure
@{class Struct2(Struct):
	def flatten(self):
		structs = copy(self.body)
		structList = lar2Structs(CAT(structs.body))
		return [ absModel2relStruct(struct[0].body) for struct in structList ]
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
@D transform an absolute lar model to a relative lar structure
@{""" transform an absolute lar model to a relative lar structure """
def absModel2relStruct(larModel):
   V,E = larModel
   Vnew = (array(V) - V[0]).tolist()
   return Struct([ t(*V[0]), (Vnew,E) ])
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
\subsection{Assembling \texttt{struct}ures}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
\subsection{Exporting to \texttt{xGeoJson}}
%-------------------------------------------------------------------------------

\paragraph{Exporting to JSON via YML a lar structure}

%-------------------------------------------------------------------------------
@D Exporting to JSON via YML
@{""" exporting to JSON via YML a lar structure """
file_handle = open(path+filename+".yml")
my_dictionary = yaml.safe_load(file_handle)
file_handle.close()
with open(path+filename+".json", 'w') as outfile:
	 json.dump(my_dictionary, outfile, sort_keys=True, indent=4, ensure_ascii=False)
#filejson = open("file1.json", 'r')
#dicta = json.loads(filejson.read())
@}
%-------------------------------------------------------------------------------

\paragraph{print a lar structure to a geoJson file}

%-------------------------------------------------------------------------------
@D Print a lar structure to a geoJson file
@{""" print a lar structure to a geoJson file """
import yaml
import json

def printStruct2GeoJson(path,struct):
	if struct.__name__() == None:
		filename = str(id(struct))
	else:
		filename = struct.__name__()
	theFile = open(path+filename+".yml", "w")
	print >> theFile, "---"
	print "filename =", path+filename+".yml"
	dim = checkStruct(struct.body)
	CTM, stack = scipy.identity(dim+1), []
	fathers = [filename]
	#import pdb; pdb.set_trace()

	tabs = (3*1)*" "

	import hijson
	V,boundaryEdges = hijson.structBoundaryModel(struct)   #+new
	boundaryPolygons = hijson.boundaryModel2polylines((V,boundaryEdges))  #+new

	printStructObject(theFile,tabs, 0,struct.name,struct.category,boundaryPolygons, "building")
	printTraversal(theFile, struct, 0, fathers,filename,[0,0,0])

	theFile.close()
	@< Exporting to JSON via YML @>

@< Print a Model object in a geoJson file @>
@< Print a Mat object in a geoJson file @>
@< Print a Struct object in a geoJson file @>
@< Traverse a structure to print a geoJson file @>
@}
%-------------------------------------------------------------------------------

\paragraph{Print a \texttt{Model} object in a \texttt{xGeoJson} file}

%-------------------------------------------------------------------------------
@D Print a Model object in a geoJson file
@{""" Print a model object in a geoJson file """
def printModelObject(theFile,tabs, i,name,category,verts,cells,father,tvect=[0,0,0]):
	tab = "	"
	print >> theFile, "-   ","id:", name
	print >> theFile, tab, "type:", "Feature"
	print >> theFile, tab, "geometry:"
	print >> theFile, tab+tab, "type:", "Polygon"
	print >> theFile, tab+tab, "coordinates:"
	print >> theFile, tab+tab+tab, AA(eval)(AA(vcode(4))(verts))
	print >> theFile, tab, "properties:"
	print >> theFile, tab+tab, "class:", category
	print >> theFile, tab+tab, "parent:", father
	print >> theFile, tab+tab, "son:", i
	print >> theFile, tab+tab, "description:", name
	print >> theFile, tab+tab, "tVector:", [tvect[0],tvect[1],tvect[2]]
	print >> theFile, tab+tab, "rVector:", [0, 0, 0]
@}
%-------------------------------------------------------------------------------

\paragraph{Print a \texttt{Mat} object in a \texttt{xGeoJson} file}

%-------------------------------------------------------------------------------
@D Print a Mat object in a geoJson file
@{""" Print a mat object in a geoJson file """
def printMatObject(theFile,tabs, theMat):
	tab = "	"
	print >> theFile, tab+tab, "tVector:", theMat.T[-1,:-1].tolist()
@}
%-------------------------------------------------------------------------------

\paragraph{Print a \texttt{Struct} object in a \texttt{xGeoJson} file}

%-------------------------------------------------------------------------------
@D Print a Struct object in a geoJson file
@{""" Print a struct object in a geoJson file """
def printStructObject(theFile,tabs, i,name,category, boundaryPolyline,father,tvect=[0,0,0]):
	tab = "	"
	print >> theFile, "-   ","id:", name
	print >> theFile, tab,"type:", "Feature"
	print >> theFile, tab,"geometry:"
	print >> theFile, tab+tab, "type:", "Polygon"
	print >> theFile, tab+tab, "coordinates:", boundaryPolyline
	print >> theFile, tab,"properties:"
	print "category=", category
	print >> theFile, tab+tab, "class:", category
	print >> theFile, tab+tab, "parent:", father
	print >> theFile, tab+tab, "son:", i
	print >> theFile, tab+tab, "description:", name
	print >> theFile, tab+tab, "tVector:", [tvect[0],tvect[1],tvect[2]]
	print >> theFile, tab+tab, "rVector:", [0, 0, 0]

@< Print a SINGLE Wall LAR model  in a geoJson file @>
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
@D Print a SINGLE Wall LAR model  in a geoJson file
@{""" Print a struct object in a geoJson file """
def printWallObject(theFile, name,i,category, edge,father,tvect=[0,0,0]):
	tab = "	"
	print >> theFile, "-   ","id:", name
	print >> theFile, tab,"type:", "Feature"
	print >> theFile, tab,"geometry:"
	print >> theFile, tab+tab, "type:", "LineString"
	print >> theFile, tab+tab, "coordinates:", edge
	print >> theFile, tab,"properties:"
	print "category=", category
	print >> theFile, tab+tab, "class:", category
	print >> theFile, tab+tab, "parent:", father
	print >> theFile, tab+tab, "son:", i
	print >> theFile, tab+tab, "description:", name
	print >> theFile, tab+tab, "tVector:", [tvect[0],tvect[1],tvect[2]]
	print >> theFile, tab+tab, "rVector:", [0, 0, 0]
@}
%-------------------------------------------------------------------------------

\paragraph{Traverse a structure to print a geoJson file}

%-------------------------------------------------------------------------------
@D Traverse a structure to print a geoJson file
@{""" Traverse a structure to print a geoJson file """
def old_printTraversal(theFile,obj, level=0, fathers=[],father="",tvect=[0,0,0]):
   import hijson
   tabs = (4*level)*" "
   for i in range(len(obj)):
	  if isinstance(obj[i],Model):
		 i,verts,cells = obj[i]
		 if obj[i].__name__() == None:
			name = father+'-'+ str(id(obj[i]))
		 else:
			name = father+'-'+ str(obj[i].__name__())
		 printModelObject(theFile,tabs, i,name,None,verts,cells,father)
		 fathers += [father]
	  elif (isinstance(obj[i],tuple) or isinstance(obj[i],list)) and len(obj[i])==2:
		 verts,cells = obj[i]
		 name = father+'-'+ str(id(obj[i]))
		 printModelObject(theFile,tabs, i,name,None,verts,cells,father)
		 fathers += [father]
	  elif isinstance(obj[i],Mat):
		 printMatObject(theFile,tabs, obj[i])
	  elif isinstance(obj[i], Struct):
		 if obj[i].__name__() == None:
			name = father+'-'+ str(id(obj[i]))
		 else:
			name = father+'-'+ str(obj[i].__name__())

		 box = obj[i].box
		 tvect = box[0]

		 if obj[i].category in ["internal_wall", "external_wall", "corridor_wall"]:
			 edge = evalStruct(obj[i])[0][0]
			 category = obj[i].__category__()
			 printWallObject(theFile, name,i,category, edge,father,tvect=[0,0,0])
		 else:
			 V,boundaryEdges = hijson.structBoundaryModel(obj[i])
			 coordinates = hijson.boundaryModel2polylines((V,boundaryEdges))
			 transfMat = array(t(*[-x for x in tvect]))

			 def transformCycle(transfMat,cycle):
				 affineCoordinates = [point+[1] for point in cycle]
				 localCoords = (transfMat.dot(array(affineCoordinates).T)).T
				 localCoordinates = [[x,y,z] for x,y,z,w in localCoords.tolist()]
				 return localCoordinates

			 coordinates = [transformCycle(transfMat,cycle) for cycle in coordinates]
			 category = obj[i].__category__()
			 printStructObject(theFile,tabs, i,name,category,coordinates,father,tvect)

		 level += 1
		 fathers.append(name)
		 printTraversal(theFile,obj[i], level, fathers,name, tvect)
		 name = fathers.pop()
		 level -= 1
   return fathers

###

wall_categories = ["internal_wall", "external_wall", "corridor_wall"]

room_categories = ["room"]

indent = 2 * " "

def get_id(obj, father_id):
	return father_id + '_' + obj.name

def is_struct(obj):
	return isinstance(obj, Struct)

def is_wall(obj):
	return obj.category in wall_categories

def is_room(obj):
	return obj.category in room_categories

def is_model(obj):
	return isinstance(obj, Model)

def is_mat(obj):
	return isinstance(obj,Mat)

def is_list(obj):
	return isinstance(obj, tuple) or isinstance(obj, list)

def print_wall(out, obj, parent_id):
	obj_id = get_id(obj, parent_id)
	edge = evalStruct(obj)[0][0]
	category = obj.category
	tvect = obj.box[0]
	print >> out, "-   ", "id:", obj_id
	print >> out, indent, "type:", "Feature"
	print >> out, indent, "geometry:"
	print >> out, indent+indent, "type:", "LineString"
	print >> out, indent+indent, "coordinates:", edge
	print >> out, indent, "properties:"
	print "category=", category
	print >> out, indent+indent, "class:", category
	print >> out, indent+indent, "parent:", parent_id
	print >> out, indent+indent, "description:", ""
	print >> out, indent+indent, "tVector:", [tvect[0],tvect[1],tvect[2]]
	print >> out, indent+indent, "rVector:", [0, 0, 0]

def transformCycle(transfMat,cycle):
	affineCoordinates = [point+[1] for point in cycle]
	localCoords = (transfMat.dot(array(affineCoordinates).T)).T
	localCoordinates = [[x,y,z] for x,y,z,w in localCoords.tolist()]
	return localCoordinates

def print_struct(out, obj, parent_id):
	tvect = obj.box[0]
	import hijson
	V,boundaryEdges = hijson.structBoundaryModel(obj)
	coordinates = hijson.boundaryModel2polylines((V,boundaryEdges))
	transfMat = array(t(*[-x for x in tvect]))
	coordinates = [transformCycle(transfMat,cycle) for cycle in coordinates]
	category = obj.category

	print >> out, "-   ","id:", name
	print >> out, indent,"type:", "Feature"
	print >> out, indent,"geometry:"
	print >> out, indent+indent, "type:", "Polygon"
	print >> out, indent+indent, "coordinates:", coordinates
	print >> out, indent,"properties:"
	print "category=", category
	print >> out, indent+indent, "class:", category
	print >> out, indent+indent, "parent:", parent_id
	print >> out, indent+indent, "description:", ""
	print >> out, indent+indent, "tVector:", [tvect[0],tvect[1],tvect[2]]
	print >> out, indent+indent, "rVector:", [0, 0, 0]

def print_model(out, obj, parent_id):
	obj_id = get_id(obj, parent_id)
	category = obj.category
	verts,cells = obj
	tvect = obj.box[0]
	print >> out, "-   ", "id:", obj_id
	print >> out, indent, "type:", "Feature"
	print >> out, indent, "geometry:"
	print >> out, indent+indent, "type:", "Polygon"
	print >> out, indent+indent, "coordinates:", AA(eval)(AA(vcode(4))(verts))
	print >> out, indent, "properties:"
	print >> out, indent+indent, "class:", category
	print >> out, indent+indent, "parent:", parent_id
	print >> out, indent+indent, "description:", ""
	print >> out, indent+indent, "tVector:", [tvect[0],tvect[1],tvect[2]]
	print >> out, indent+indent, "rVector:", [0, 0, 0]

def print_mat(out, obj, parent_id):
	print >> out, indent+indent, "tVector:", obj.T[-1,:-1].tolist()

def print_traversal(out, obj, parent_id='', level=0):

	if is_struct(obj):

		if is_wall(obj):
			print_wall(out, obj, parent_id)
			return

		print_struct(out, obj, parent_id)

		parent_id = get_id(obj, parent_id)
		print_traversal(out, obj.body, parent_id)
		return

	if is_model(obj):
		print_model(out, obj, parent_id)
		return

	if is_list(obj):
		parent_id = get_id(obj, parent_id)
		print_traversal(out, obj[0], parent_id)

		if len(obj) > 1:
			print_traversal(out, obj[1:], parent_id)

		return

	if is_mat(obj):
		print_mat(out, obj, parent_id)
		return

def printStruct2GeoJson(path, struct):
	if struct.__name__() == None:
		filename = str(id(struct))
	else:
		filename = struct.__name__()
	out = open(path+filename+".yml", "w")
	print >> out, "---"
	print "filename =", path+filename+".yml"
	dim = checkStruct(struct.body)
	CTM, stack = scipy.identity(dim+1), []
	fathers = [filename]
	#import pdb; pdb.set_trace()

	import hijson
	V,boundaryEdges = hijson.structBoundaryModel(struct)   #+new
	boundaryPolygons = hijson.boundaryModel2polylines((V,boundaryEdges))  #+new

	print_traversal(out, struct)

	out.close()

@}
%-------------------------------------------------------------------------------

%===============================================================================
\section{\texttt{Iot3D} Exporting}
%===============================================================================

%-------------------------------------------------------------------------------
@O larlib/larlib/iot3d.py
@{"""Module with automatic generation of simplified 3D buildings"""
from larlib import *

@< From Struct object to LAR boundary model @>
@< Struct class pushing local origin at the bottom of the structure @>
@< transform svg primitives to basic lar format @>
@< transform a lar model to a list of lar structures @>
@< transform an absolute lar model to a relative lar structure @>
@< Print a lar structure to a geoJson file @>
@}
%-------------------------------------------------------------------------------

%===============================================================================
\section{Examples}
%===============================================================================

\subsection{A complete 3D building example}

In this section a complete 3D building example is developed, that starts by importing the data associated to the rect and polyline primitives of the 2D layout exported as svg file, and finishes by generating a complete 3D mock-up of a quite complex multi-floor office building.

%-------------------------------------------------------------------------------
@O test/py/iot3d/test01.py
@{"""Automatic construction of a simplified 3D building from 2D layout"""
import sys
PATH = "/Users/paoluzzi/Documents/RICERCA/sogei/edifici/"
sys.path.insert(0, PATH)

from buildings import *
from larlib import *

# LAR models (absolute coordinates)
ala_est = larEmbed(1)(polyline2lar(rects2polylines(eastRooms) + eastTip))
ala_sud = larEmbed(1)(polyline2lar(rects2polylines(southRooms) + southTip))
ala_ovest = larEmbed(1)(polyline2lar(rects2polylines(westRooms) + westTip))
ala_nord = larEmbed(1)(polyline2lar(rects2polylines(northRooms) + northTip))
ascensori = larEmbed(1)(polyline2lar(elevators))
spazioComune = larEmbed(1)(polyline2lar(AA(REVERSE)(newLanding)))

# test of input consistency (flat assembly of LAR models)
pianoTipo = Struct([ala_est,ala_sud,ala_ovest,ala_nord,ascensori,spazioComune],"pianoTipo")
##VIEW(EXPLODE(1.2,1.2,1.2)(MKPOLS(struct2lar(pianoTipo))))
##VIEW(SKEL_1(EXPLODE(1.2,1.2,1.2)(MKPOLS(struct2lar(pianoTipo)))))

# LAR to structs
Ala_est = Struct(lar2Structs(ala_est),"Ala_est")
Ala_sud = Struct(lar2Structs(ala_sud),"Ala_sud")
Ala_ovest = Struct(lar2Structs(ala_ovest),"Ala_ovest")
Ala_nord = Struct(lar2Structs(ala_nord),"Ala_nord")
Ascensori = Struct(lar2Structs(ascensori),"Ascensori")
SpazioComune = Struct(lar2Structs(spazioComune),"SpazioComune")

model = struct2lar(Ala_est)
##VIEW(EXPLODE(1.2,1.2,1.2)(MKPOLS(model)))
for k,room in enumerate(Ala_est.body):
	print room.body

# hierarchical assembly of simplest LAR models
TreAli = Struct([Ala_est,Ala_sud,Ala_ovest,Ascensori,SpazioComune],"TreAli")
##VIEW(EXPLODE(1.2,1.2,1.2)(MKPOLS(struct2lar(TreAli))))
Ali_B_C_D = Struct(6*[TreAli, t(0,0,30)],"Ali_B_C_D")
##VIEW(EXPLODE(1.2,1.2,1.2)(MKPOLS(struct2lar(Ali_B_C_D))))

Ala_A = Struct(4*[Ala_nord,  t(0,0,30)],"Ala_A")
ALA_A = EXPLODE(1.2,1.2,1.2)(MKPOLS(struct2lar(Ala_A)))
##VIEW(EXPLODE(1.2,1.2,1.2)([ ALA_A, COLOR(BLUE)(SKEL_1(ALA_A)) ]))

Edificio = Struct([ Ala_A, Ali_B_C_D ],"Edificio")
out = struct2lar(Edificio)
if len(out)==2: V,FV = out
else: V,FV,EV = out

EDIFICIO = STRUCT(MKPOLS((V,FV)))
##VIEW(STRUCT([ EDIFICIO, COLOR(BLUE)(SKEL_1(EDIFICIO)) ]))

VV = AA(LIST)(range(len(V)))
SK = SKEL_1(EDIFICIO)
##VIEW(larModelNumbering(1,1,1)(V,[VV,[],FV],STRUCT([ COLOR(BLUE)(SK), EDIFICIO ]),20))

Va,EVa = struct2lar(TreAli)
Vb,EVb = struct2lar(Ala_A)
a = PROD([ SKEL_1(STRUCT(MKPOLS( ([ v[:2] for v in Va], EVa) ))), QUOTE(5*[30]) ])
b = PROD([ SKEL_1(STRUCT(MKPOLS( ([ v[:2] for v in Vb], EVb) ))), QUOTE(3*[30]) ])
glass = MATERIAL([1,0,0,0.3,  0,1,0,0.3,  0,0,1,0.3, 0,0,0,0.3, 100])
##VIEW(glass(STRUCT([a,b])))

##VIEW(STRUCT([ glass(STRUCT([a,b])), EDIFICIO, COLOR(BLUE)(SKEL_1(EDIFICIO)) ]))

printStruct2GeoJson(PATH,pianoTipo)
printStruct2GeoJson(PATH,Edificio)
@}
%-------------------------------------------------------------------------------

%===============================================================================
\appendix
%===============================================================================

\bibliographystyle{amsalpha}
\bibliography{iot3d}

\end{document}


\section{Evaluation}

When finally wanting to run and evaluate the models, we encountered problems with the execution of our pipeline. It turned out that the training process requires dramatically more computational resources than we anticipated - more, at least, than our local machines could handle. We tried different approaches to cope with this problem: As a first step, we skipped the AlexNet as baseline model. It has a huge number of trainable parameters compared to the pre-trained EfficientNetV2 (small amount of trainable parameters) and InceptionV3 (somewhat in the middle between the former two), requiring much more backpropagation computations within the training. Unfortunately, just having fewer trainable parameters did not reduce the computational load enough for us to be able to train the models.

\begin{table}[h]
    \centering
    \begin{tabular}{l|c|c|}
    \cline{2-3}
                                                     & Trainable parameters & Total parameters \\ \hline
    \multicolumn{1}{|l|}{\textbf{AlexNet}}           & 47,790,398           & 47,793,150       \\ \hline
    \multicolumn{1}{|l|}{\textbf{EfficientNetV2-B0}} & 6,184,078            & 6,244,686        \\ \hline
    \multicolumn{1}{|l|}{\textbf{InceptionV3}}       & 22,288,798           & 22,323,230       \\ \hline
    \end{tabular}
    \caption{Parameters per model with 254 classes. Number of classes only insignificantly affects total number of parameters.}
    \label{tab:paramsPerModel}
\end{table}

The next logical consequence for us was to reduce the size of the dataset. We initially augmented the data, increasing the size of the dataset roughly 10-fold, to enable the models to better learn features of the turtle head patterns and avoid overfitting. Step by step, we cut out the different augmentations until we were back to the original, non-augmented dataset. Even this dramatic decrease of dataset size back to around 5000 images did still not allow us to train the models properly. We then went on to experiment with different buffer sizes for the shuffle procedures and different batch sizes to reduce the load on the RAM. Although decreasing both of these parameters seemed to free up RAM, a full training was still not possible. The EfficientNetV2, which was for now the model on which we tried to train, was able to run the fitting for 1-3 epochs before sessions crashed due to full RAM. Even further reducing the dataset size by increasing the number of minimum images per turtle from 10 up to 15, which lead to a further reduction of the total number of images to little over 3000, did still not allow for successful training.

CNNs can generally be scaled along three dimensions: changing the depth of the network, i.e. the number of layers, the width of the network, e.g. by increasing the number of channels per convolutional layer, or by changing the resolution of the input images. Because we did not want to make major changes to the layer architecture of the models we use, we decided that downsampling the images would be the next measure to explore. The resolution we used so far yields $224\times224\times3=150528$ values per image, we reduced to $160\times160$ pixels with a resulting decrease of $\raisebox{-0.9ex}{\~{}} 50\%$ with regard to image values: $160\times160\times3=76.800$. Reducing the dataset size (although not the number of samples) again by half with this approach, we managed to successfully train the EfficientNetV2. We were not happy with this approach since reducing the image size potentially gives rise to unrecoverable loss of important information \citep{Zhang2011}. Further, both the EfficientNetV2 and the InceptionV3 expect input sizes of $224\times224$.

Because of this, we turned to Google Colab to execute our pipeline. The resources there allowed us to return to the previous resolution of $224\times224$ pixels. Treating the minimum number of images per turtle (which directly affects the number of samples in the dataset) as a first hyperparameter to explore - we still could not run the training with the entire augmentation pipeline due to RAM restrictions - we found a dataset size of approximately 5000 images to max out on Colab's freely available resources. A table with how the minimum number of images affects the dataset size and median number of images per turtle can be found in table \ref{tab:minImages} in appendix \ref{apx:minImages}. Further naïvely exploring the batch size, we found batches of 64 samples to best work with the computational resources while still allowing for a feasible training speed. The convergence behaviour did not seem to be significantly affected by different batch sizes.

A first run with both the EfficientNetV2 and InceptionV3 for 10 epochs to establish a baseline yielded the results shown in figure \ref{fig:efficientInceptionComparison}.

\begin{figure}[h!]
    \centering
    \includesvg[width=8cm]{images/efficient_inception_comparison.svg}
    \caption{Categorical accuracy on training and validation data for EfficientNetV2 (train: orange, validation: blue) and InceptionV3 (train: light blue, validation: pink).}
    \label{fig:efficientInceptionComparison}
\end{figure}

The EfficientNetV2 is able to achieve accuracy $97.02\%$ on the training set while the InceptionV3 - pre-trained on the same ImageNet21K - achieves $90.18\%$. Validation accuracies are $49.83\%$ and $17.01\%$, respectively, and test accuracies are $50.42\%$ and $16.25\%$, suggesting that the InceptionV3 actually overfits on the training data. Although no definite convergence behaviour could be seen at this point, we decided to focus on the EfficientNetV2 -- mainly due to the limited computational resources. We can, however, not draw any meaningful conclusion regarding the InceptionV3 since we did not explore this direction any further.

After establishing this baseline using only non-augmented images, we went on to assess to what extent the different augmentations influence model performance. We first reduced the size of the dataset by increasing the minimum number of images per turtle to 13 (from 10), and then augmented a part of this data using only a single augmentation technique to get the image number up to roughly 5000 again. The comparison between baseline and augmentations, which can be seen in figure \ref{tab:augmentationComparison}, has to seen skeptically, though, because increasing the number of required images per turtle does a) significantly reduce the number of classes (unique turtles) from 254 to 150 and b) provide more total data per class, which theoretically should allow the model to better capture the underlying distribution.

\begin{table}[h]
    \centering
    \begin{tabular}{l|l|l|l|}
    \cline{2-4}
                                        & \textbf{Train} & \textbf{Validation} & \textbf{Test} \\ \hline
    \multicolumn{1}{|l|}{Baseline}      & 97.02\%        & 49.83\%             & 50.42\%       \\ \hline
    \multicolumn{1}{|l|}{180° rotation} & 93.14\%        & 89.38\%             & 58.98\%       \\ \hline
    \multicolumn{1}{|l|}{Gauss-filter}  & 96.76\%        & 96.72\%             & 63.80\%       \\ \hline
    \multicolumn{1}{|l|}{Random HSV}    & 98.67\%        & 98.46\%             & 56.77\%       \\ \hline
    \multicolumn{1}{|l|}{Noise}         & 92.27\%        & 86.15\%             & 56.77\%       \\ \hline
    \end{tabular}
    \caption[]{Categorical accuracy of EfficientNetV2 on differently augmented datasets after 10 epochs.}
    \label{tab:augmentationComparison}
\end{table}

As can be seen, validation and test accuracies greatly benefit from all augmentations, with the Gaussian filtering considerably outperforming the other techniques on the test data.


\documentclass{article}[12pt]
\usepackage{amsmath}
\usepackage{gensymb}
\usepackage{standalone}
\usepackage{verbatim}
\usepackage[utf8]{inputenc}
\usepackage{setspace}
\usepackage[a4paper,margin=1in,footskip=0.25in]{geometry}
\usepackage{graphicx}
\usepackage{mathptmx}
\usepackage{booktabs}
\usepackage{cite}
\usepackage[english]{babel}
\usepackage[utf8]{inputenc}

\begin{document}
\section{Code Validation}
	To verify the functionality and accuracy of the implemented code a simple two terminal device was used. The geometry was set to be a $100\mu m$ square with $100$ nodes on each axis. The terminals were placed across the top and bottom surfaces, with the top having an applied voltage of $1000V$ and the bottom being set to ground. Solving of Poisson's equation for this geometry results in a linear relationship. \\ \\
 	\noindent
 The simulated results of the two terminal device is seen in Figure \ref{fig:two_terminal}. As can be seen the simulated results show a linear relationship between voltage and distance in $y$.

 \begin{figure}[h!]
 	\centering
 	\includegraphics[width=\linewidth]{two_terminal1.png}
 	\caption{Simulated voltage distribution of two terminal device.}
 	\label{fig:two_terminal}
\end{figure}
\end{document}


\section{Integration of vector fields}


%##############################################################################
% Preamble
%##############################################################################

\documentclass{pset}
\name{Dustin Tran, Xiaomin Wang, Rodrigo Gomes}
\email{\{trandv,xiaominw, rgomes\}@mit.edu}

\course{6.834J/16.412J, S15}
\instructor{Professor Brian Williams}
\assignment{Problem Set \#3}
\duedate{April 17, 2015}

\begin{document}

%##############################################################################
% Begin Document
%##############################################################################

\section{Introduction}
Many scenarios in real life occur where one must make a sequence of decisions
under uncertainty. Quite often in these scenarios two unknowns exist: the
result of taking an action at a given state and the true state of the agent at
any point in time. This is similar to that of a hidden Markov model (HMM), only
in that one must make a sequence of actions instead of a single action; thus the
problem falls no longer in supervised learning but reinforcement learning. The
classic example is a robot which tries to navigate a discrete environment using a
GPS, and it performs action that lead to different states with various
probabilities. The transition probabilities are unknown, and because of the GPS,
there is also inaccuracy in what the
underlying state is.

One can formalize the problem, under Markovian assumptions, as a partially
observable Markov decision process (POMDP). In this project we provide a
software library for implementing and solving them, which is modular and
flexible enough for further development and user-specified agents/environments.
We encode a variety of basic tasks and solve them using a combination of value
iteration and several standard (PO)MDP solvers---one is a variant of Thompson
sampling \cite{strens2000bayesian}, which is a Bayesian approach following a
Dirichlet-multinomial posterior over each state-action pair.

The repository can be found at https://github.com/dustinvtran/bayesrl.
To install from pip, run
\begin{lstlisting}
pip install -e "git+https://github.com/dustinvtran/bayesrl.git#egg=bayesrl"
\end{lstlisting}

\section{Technical Background}

\subsection{POMDP}
POMDP is a generalization of a Markov decision process (MDP). In a MDP, for each
possible state of the process, a decision has to be made regarding which action
should be executed in that state. The chosen action and given state affects the
costs (or rewards) incurred. The goal is to learn the optimal \emph{policy},
which is a
choice of actions that in expectation leads to the optima reward in a
pre-defined number of steps (in the case of finite horizon; or in infinite time). In a
POMDP model, the agent does not fully observe the underlying states but a
(non-sufficient) statistic of it. Thus POMDPs also
maintain a probability distribution over the set of possible states, based on a
set of observations and observation probabilities, and the underlying MDP.

More formally, a POMDP is a collection $(S,A,T,R,\Omega,O,\gamma)$, where

\begin{itemize}
\item $S$ is a set of states
\item $A$ is a set of actions
\item $T$ is a set of transition probabilities between states. If the agent is currently in state $s
\in S$, and it takes action $a \in A$, the agent will transition to a new state
$s'$ with probability
$T(s' \mid s,a)$.
\item $R: S \times A \rightarrow \mathbb{R}$ is a reward function that assigns a numeric reward (or
cost if the value is negative) for each state and action.
\item $\Omega$ is a set of observations
\item $O$ is a set of conditional observation probabilites. If the agent is now in state $s$,
it receives an observation $o$ according to $O(o \mid s)$
\item $\gamma \in [0,1]$ is a discount factor that determines how much rewards should be discounted over time
\end{itemize}

We use our testing environment \texttt{GridWorld} as an example.
\texttt{GridWorld} represents a 2D maze where the agent can be in discrete locations.
Certain locations are impossible for the agent to move to, representing ``walls''. Every
action can move the agent between two adjacent grid locations, or fail, and
cause
the agent to take a uniformly random action instead according to some
user-specified probability. The goal is to reach
the goal location in least amount of time, as each move is -1 reward, hitting a
wall is -1 reward, and reaching the end positions incurs a reward of +50.

For \texttt{GridWorld}, $S$ consists of all the possible (row, column) location
tuples inside the maze.  $A$ contains the four possible actions the agent can
take: up, down, left, right.  $T$ describes a transition model that allows the
agent to move without hitting the wall.  We define $R$ as the above. To make
this a partially observable problem, we implemented two observation models.
\begin{itemize}
\item
The
easier model gives the agent more information about the environment. The agent
knows which of its four neighbors are walls, giving rise to 16 total
observations.
\item
In the second observation model, the agent can only observe how
many of its four neighbors are walls, giving rise to 5 possible observations.
$O$ is such that $Pr(true\_observation\mid state)=true\_observation\_prob$,
where $true\_observation\_prob$ can be adjusted, and
\begin{equation}
Pr(other\_observation\mid
state)=\frac{1-true\_observation\_prob}{total\_num\_of\_observations-1}
\end{equation}
\end{itemize}
Since
we cannot work with the underlying states directly in POMDP, we also need $B$
which is the set of belief states, or the probabilities the agent is at all
possible states.

\subsection{Thompson Sampling}
Thompson Sampling is used to learn the transition distribution $T$, where one
first specifies a prior according to one's knowledge about the transitions
before trying any actions. After a constant set of time steps, the transition
probabilities are then recalculated using a posterior update following a
Dirichlet-multinomial distribution. In MDP, since the agent observes the states
directly, the posterior transition probabilities are updated directly using the
transition counts. In POMDP, the posterior probabilities are iteratively updated
following an update on belief state transition probabilities.

Note that we've also implemented a variety of other standard MDP solvers for
benchmarking: Q-Learning, SARSA, and R-MAX.

\subsection{Value Iteration}
Given estimated transition probabilities, we then solve for the underlying policy with value iteration. Value $V$, is the expected total reward
given a policy $\pi$, where a policy decides which action to take given the
belief state. $a = \pi(b)$. The expected reward for policy $\pi$ starting freom
belief $b_0$ is defined as
\[ V^{\pi}(b_{0})=\sum\limits_{t=0}^\infty \gamma^{t}r(b_t,a_t) \]
where $r(b_t, a_t) = \sum\limits_{s \in S} b_t(s)R(s,a_t)$.
The optimal policy should maximize the long term reward
\[ \pi = \underset{\pi}{\text{argmax}} V^{\pi}(b_0) \]
At each time step, we update the belief states based on the observation, and
then update the values based on the updated belief states. The action that gives
the largest expected reward over the belief states is selected for the next time
step. The values gradually improve until convergence.
By improving the values, the policy is implicitly improved.

\section{Implementation}

\subsection{Agent Environment Paradigm}
For MDP, we follow the paradigm set forth in Sutton and Barto (Figure 3.1,
\cite{sutton1998reinforcement}).
\begin{figure}[ht]
\begin{center}
\centerline{\includegraphics[width=\textwidth]{img/agent_environment.png}}
\end{center}
\end{figure}
It suggests that RL agents need only output a suggested action after previous history and a given state and reward.  Overall by following the paradigm for the software design, we make the learning process explicit and intuitive.

\subsubsection{Agent}
We implement a base class \texttt{Agent} which is a collection of objects and functions to be used for all other agents. Agents differ primarily in their \texttt{interact()} function, which determines the next action to perform given a state and reward from the environment.

The model-based algorithms R-MAX and Thompson sampling inherit from \texttt{ModelBasedAgent}, which is a class that itself inherits from \texttt{Agent}; \texttt{ModelBasedAgent} adds subroutines specific to model-based approaches such as value iteration.

In order to reduce the most redundant code, we also could have used an
additional class that inherits from \texttt{Agent} for temporal difference method agents; this
would be used by both \texttt{SARSAAgent} and \texttt{QLearningAgent}, as they
differ only in their \texttt{value\_table} assignment. However, the efficiency gain in such an
abstraction is not worth the loss of readability in our opinion.

\subsubsection{Environment}
An \texttt{Environment} object is initialized at some state, with an arbitrarily
defined state and action space. Actions are performed on an \texttt{Environment}
object under the subroutine \texttt{perform\_action()}, and the output is a new state and its reward.

\subsubsection{Trial and Plot}
As for trials, we implement a class \texttt{Trial} which contains all information for running multiple trials, i.e., independent collections of episodes to learn and act upon. We also add a \texttt{Plot} class which is a wrapper containing all \texttt{Trial} objects; this is convenient for generating plots on collections of trials coming from possibly many different agents.

\subsection{POMDP}
For POMDP, we assume that the agent is given an observation model of the
environment it is acting in, in the form of a conditional probability distribution
$P(observation \mid state)$. Astr\"{o}m has shown that a properly updated probability
distribution over the state space $S$ is sufficient to summarize all the observable
history of a POMDP agent without loss of optimality \cite{astrom1965optimal}.
Thus we add a step of updating the belief state to the MDP paradigm. And the belief state, instead
of the underlying state, is used to update the transition model.

\subsection{Organization of Code}
We follow the directory structure specified in the problem set, with two
exceptions:
\begin{itemize}
\item \texttt{documentation/} does not exist. Instead, documentation is written
in the \texttt{README.md} inside the current working directory. Any additional
documentation not purely necessary for the problem set submission is in the
Github wiki (which is a subset of anything in this writeup).
\item \texttt{source/} is named \texttt{bayesrl/} in order to follow Python
convention for installing modules.
\end{itemize}

\section{Analysis}

\subsection{Runtime}
The algorithm takes $O(\mid S \mid ^2 \mid A \mid)$ to calculate the transition probabilities, the
expected rewards and the belief state. It is uncertain how many time steps are required for
convergence. All solvers we implement are guaranteed to converge in polynomial time for MDPs,
although unfortunately we have not seen stricter theoretical results on the
upper bounds than this. This certainly makes sense as it is true for all
environments regardless of the pathological scenario. However, it would
certainly be interesting to examine bounds under stricter assumptions where we
fix the environment and perhaps certain parameter settings to simplify the
analysis.

\subsection{Memory}
We to store several arrays for the computation. The transition probabilities
table is of dimension $\mid S \mid ^2 \mid A \mid$. The value table is of
dimension $\mid S \mid \mid A \mid$. The transition observation table is of
dimension $\mid S \mid ^2 \mid A \mid$. Thus the space requirement for the
algorithm is $O(\mid S \mid ^2 \mid A \mid)$.

\subsection{Limitations}
If the number of states is large, the algorithm quickly becomes intractable. We
are quite interested in examining function approximations, which allow one to
essentially apply a supervised learning algorithm to predict the optimal action
given state characteristics, rather than to hardcode them manually. Note
however that this makes the runtime even worse as there is an additional error
accumulating as a result of the predictions.

\section{Experiments}
The implementation of Thompson Sampling for MDPs ran very successfully: as time
progressed, the agent was able to get to the goal much faster, and get a high
reward. It did not work, however, for POMDPs. As time progressed, the agent seemed
to take about the same amount of time to reach the goal on every execution, not
improving in performance. We hypothesize that the reason is due to a lack of
a good prior on the transition model: Thompson sampling relies a lot on being
able to count the number of transitions between states, given an action.

\begin{figure}[ht]
\begin{center}
\centerline{\includegraphics[width=\textwidth]{img/mdp_imm_rewards.png}}
\caption{Immediate reward for MDP solvers in a variety of parameter settings on
the Gridworld example.
Each of the settings for Thompson sampling vary how strong the misspecification of the
prior as a uniform distribution, and yet in all three scenarios it indicates
convergence. We also see that R-MAX performs better than the temporal difference methods.}
\end{center}
\end{figure}

This is very hard in a POMDP with a weak prior on its transition model, as it
may have a completely wrong idea of where it ends up at each step. The
observation model is supposed to improve its accuracy, but it was not sufficient
in this case. In fact, we can see this empirically by essentially hardcoding the
optimal path by putting high probability on where it should go, and indeed it
converges to the optimal quite quickly.

To have an idea of how differently successful the same approach was on MDPs vs
POMDPs, we show you our results:

\begin{figure}[ht]
\centering
\includegraphics[width=0.75\textwidth]{img/pomdp.png}
\caption{\label{fig:pomdp}Cumulative Reward for POMDP}
\end{figure}

These graphs show the cumulative reward that the agent got as time progressed.
In the MDP case, the reward started low, and kept getting lower, as the agent
explored. It, however, started getting higher as the agent started acting in a
more deliberate manner, to maximize reward. The POMDP agent, however, seems to
be in a state where its reward just gets increasingly negative. It is unclear
whether it never leaves the exploration phase, or if it simply learns a completely
wrong transition model, and thus computes a very wrong policy.

\bibliography{6_834j_ps03}
\bibliographystyle{plain}

%##############################################################################
% End Document
%##############################################################################

\end{document}


\documentclass{easychair}

% PACKAGES
\usepackage{url}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amssymb}

% for underscores https://texfaq.org/FAQ-underscore
\usepackage{lmodern}
\usepackage[T1]{fontenc}
\usepackage{textcomp}
\usepackage{lineno}

%\usepackage[
%bookmarksopen,
%bookmarksdepth=2,
%%breaklinks=true
%colorlinks=true,
%urlcolor=blue]{hyperref}

% GLOBAL FORMATTING
%\linenumbers
\parindent=0pt
\parskip=0.5\baselineskip
\raggedbottom

% TITLE AUTHOR DATE
\title{A Controlled Natural Language for Type Theory%
\thanks{I thank Peter Koepke for introducing me to the field and Jesse Han for his
preliminary work on parsing.
Research is supported by Sloan Foundation grant G-2018-10067.
Source code and examples are found at \url{github.com/formalabstracts/CNL-CIC}.}
}

%\date{November 17, 2019}
\author{Thomas Hales}

\institute{
University of Pittsburgh\\
Pittsburgh, PA, USA
}

\authorrunning{T.Hales}
\titlerunning{Controlled Natural Language}

% THEOREMS
\newtheorem{definition}{Definition}
\newtheorem{theorem}[definition]{Theorem}
\newtheorem{lemma}[definition]{Lemma}
\newtheorem{specification}[definition]{Specification}

% COMMANDS
\renewcommand{\iff}{\leftrightarrow}
\newcommand{\Prop}{\text{\tt Prop}}
\newcommand{\Type}{\text{\tt Type}}
\newcommand{\fld}{\textasciicircum}
\newcommand{\dequiv}{\mathrel{:=}} %{\mathrel{:\equiv}}
\newcommand{\Nat}{\ensuremath{{\mathbb N}}}
\newcommand{\Real}{\ensuremath{{\mathbb R}}}
\newcommand{\df}[1]{\text{\bf #1}}
\newcommand{\h}[1]{\text{#1}}
\newcommand{\join}{\lor}
\newcommand{\Mid}{\mathrel{\|}}
\newcommand{\comment}[1]{\%- \nobreak{#1}}
\renewcommand{\~}{\ }
\newcommand{\ignore}[1]{}
%\newcommand{\remark}[1]{(#1)}
\renewcommand{\_}{\textunderscore}
\renewcommand\labelitemi{-}
\renewcommand{\qed}{\ensuremath{\square}}

% ENVIRONMENTS

% \leavevmode\par is to make remark work when it is the first item in a subsection.
\newenvironment{remark}
{\leavevmode\par\begin{tabular}{|p{13cm}}\parskip=\baselineskip{\bf Remark.}}
{\end{tabular}}

\newenvironment{oblongo}{}{}

\newenvironment{prule}%
               {\begin{itemize}}%
               {\end{itemize}}
\newcommand{\ptem}{\item}
\newcommand{\nt}[1]{{\tt #1}}
\newcommand{\rw}{$\quad\to\quad$}

% DOCUMENT

\begin{document}
\maketitle

\section{Introduction}

This abstract describes the design and development of a
controlled natural language for mathematics that has the Lean
theorem prover as intended target.  We call this language Colada (short for
\emph{Co}ntrolled \emph{la}nguage \emph{da}ta).
Documents in our dialect are written in a specially prepared \LaTeX\ file.
Our aim is to capture definitions and theorem statements from the published
mathematical literature in our dialect, but
checking mathematical proofs is beyond the scope of our project.

Our design grows out of previous controlled natural
languages for mathematics (specifically, Forthel-Naproche-SAD), as
described in Peter Koepke's AITP 2019 talk, which
exhibited some short proofs written in fluent English that can be read and checked by their software.

We use
Forthel as the generic name for any of
the dialects inspired by Forthel (a
controlled natural language developed by various researchers, and implemented in Paskevich's PhD thesis),
including Colada.
We refer to the Colada language as
\emph{our dialect}.
Our dialect differs from others in that our semantic target is the logical Calculus of Inductive
Constructions (CiC) as implemented in the Lean theorem prover,
instead of first-order logic.
Our dialect can be viewed as a fusion of three different syntactic traditions:
Forthel syntax, \LaTeX\ syntax, and Lean theorem-prover
syntax.
From another perspective, our dialect might be viewed as a
mountain of syntactic sugar for Lean.

\section{Controlled Natural Languages (CNL)}\label{sub:CNL}

By controlled natural language for mathematics (CNL), we mean an
artificial language for the communication of mathematics that is (1)
designed in a deliberate and explicit way with precise
computer-readable syntax and semantics, (2) based on a single natural
language (which for us is  English), and (3) broadly
understood at least in an intuitive way by mathematically literate
speakers of the natural language.

CNLs can achieve a much higher degree of English fluency than other
proof-checking languages.

Our
basic aim is to develop a technology that lies somewhere
between the current practice of research mathematicians and the current
practice within the proof assistant community.

\section{Research to Date}

Our specific research contributions to date are as follows.

We have a design and specification of a controlled natural
language.
Like other Forthel dialects, our grammar is not context-free.
However, it is similar to a context-free grammar by being specified through
production rules on terminal and nonterminal symbols.
Users may extend the grammar with new mathematical notation
and constructs:  the language
contains syntax for the extension of its own syntax.

The lexical structure of our dialect is specified in sedlex, a lexical
generator tool for OCaml.
Our dialect has been specified in menhir, an OCaml-based
parser-generator tool for LR(1) grammars.  (Although our dialect is
not an LR(1) grammar, which prevents menhir from automatically
generating a parser, the software checks that our grammar is well-formed.)

We believe that some complexity is justified (and even required) to
capture widespread mathematical idioms and formulas, the syntax of type
theory, and their
interactions. Our grammar is recursive to an extraordinary degree.
The grammar has about $350$ nonterminals and about $550$ production
rules.  The grammar contains about $150$ English words
(such as \emph{all, any, are, case, define, exists, if, iff, is, no, not, of, or, over, proof, the, theorem}, etc.)
with a fixed grammatical
function.     User syntax
extensions build on that base.

We keep most features of Forthel, such as its handling of synonyms, noun
phrases, verbs, and adjectives; and its grammar extension mechanisms.
We have added many additional features such as plural formation for nouns and verbs,
operator precedence parsing (with
user-specified precedence levels and associativities); scoping of variables;
syntax for \LaTeX\ macros; and dependent type theory including
inductive and mutual inductive types, structures, and lambda
terms.

A parser for our grammar has been implemented in OCaml, building substantially on
the parser combinator library that John Harrison wrote to parse HOL
Light.

Future work will transform parsed output to type-checked
terms in Lean.  Another future project is syntax highlighting and auto-completion tools
for our
dialect in editors such as emacs and VSCode. We also plan to develop large
mathematical libraries in our dialect.

We have written software that takes a specially prepared
\LaTeX\ file as input and strips away the non-semantic content
(such as headers, spaces and other layout, graphics,
remarks, and dollar signs) and
outputs raw CNL.
The key to beautifully typeset \TeX\ documents is a
dual expansion system for macros.  The \TeX\ engine expands macros in
the usual way, but the CNL engine expands some macros according to an
independent semantic specification.

We believe our language will find novel applications to search, document
analysis, and document transformation.
Ultimately each word, symbol, and phrase of a math document written in our dialect can be
accounted for.

\bigskip
\noindent\rule{2cm}{0.4pt}

This work is part of the Formal Abstracts project supported by the Sloan Foundation,
which aims to capture all the major
definitions and theorems of mathematics in a format that is both human and computer
friendly.

\newpage
\section{Example}

Many examples will be given during the AITP presentation to show that English
fluency is obtained without loss of semantic content.  Here is one example that
assumes a context in which a binary relation $R$ has been defined.

\subsection{source} The \LaTeX\ source file, which follows, is similar to documents prepared every day
by mathematicians.

\begin{verbatim}
\def\deflabel#1{\begin{definition}[#1]\label{#1}}
%begin LaTeX source
\deflabel{greatest element} We say that $y$ is a
\df{greatest\~element} in $R$ iff for all\ $x,\ x \le y$.
\end{definition}

We introduce synonyms least/minimum/bottom.

\deflabel{least element} We say that $y$ is a \df{least\~element} in
$R$ iff for all\ $x,\ y \le x$.
\end{definition}

Let $x < y$ stand for $x \le y$ and $x \ne y$.

\deflabel{maximal element} We say that $y$ is a \df{maximal\~element}
in $R$ iff there exists no $x$ such that $y < x$.
\end{definition}
%end LaTeX source
\end{verbatim}

\subsection{pdf} The pdf generated from this text is

%begin sample
\def\deflabel#1{\begin{definition}[#1]\label{#1}}
\deflabel{greatest element} We say that $y$ is a
\df{greatest\~element} in $R$ iff for all\ $x,\ x \le y$.
\end{definition}

We introduce synonyms least/minimum/bottom.

\deflabel{least element} We say that $y$ is a \df{least\~element} in
$R$ iff for all\ $x,\ y \le x$.
\end{definition}

Let $x < y$ stand for $x \le y$ and $x \ne y$.

\deflabel{maximal element} We say that $y$ is a \df{maximal\~element}
in $R$ iff there exists no $x$ such that $y < x$.
\end{definition}
%end sample

\subsection{CNL}  Here is the generated CNL.  (We have cleaned up the white space.)

\begin{verbatim}
Definition  Label_greatest_element  .    We  say  that    y    is  a
greatestelement  in    R    iff  for  all    x  ,  x  \le y    .

We  introduce  synonyms  least  /  minimum  /  bottom  .

Definition  Label_least_element  .    We  say  that    y    is  a
leastelement  in  R    iff  for  all    x  ,  y  \le x    .

Let    x  <  y    stand  for    x  \le y    and    x  \ne y    .

Definition  Label_maximal_element  .    We  say  that    y    is  a
maximalelement  in    R    iff  there  exists  no    x    such  that    y  <  x    .
\end{verbatim}

\subsection{parse tree}  This has been generated by hand.

\end{document}


\documentclass[11pt,a4paper]{article}
\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{fourier}
\usepackage{yfonts}
\usepackage[hmargin=3cm,vmargin=2.4cm]{geometry}
\usepackage[dvipsnames]{xcolor}
\usepackage{hyperref}
\usepackage{fancyhdr}

\hypersetup{
  colorlinks=true,
  citecolor=NavyBlue,
  linkcolor=BrickRed,
  urlcolor=Violet
}

\pagestyle{fancy}
\fancyhead{}
\fancyhead[L]{\href{https://www-ljk.imag.fr/membres/Pierre.Karpman/cry\_adv2017\_fse1.pdf}{https://www-ljk.imag.fr/membres/Pierre.Karpman/cry\_adv2017\_fse1.pdf}}

\theoremstyle{definition}
\newtheorem{defi}{Definition}
\def\defiautorefname{Definition}

\DeclareMathOperator\E{\mathcal{E}}
\DeclareMathOperator\TE{\mathcal{\widetilde{E}}}
\DeclareMathOperator\Perm{\mathcal{P}}
\DeclareMathOperator\ora{\mathcal{O}}
\DeclareMathOperator\iE{\textswab{E}}
\DeclareMathOperator\iPerm{\textswab{P}}
\DeclareMathOperator\Adv{\mathbf{Adv}}
\DeclareMathOperator\aes{AES-128}
\DeclareMathOperator\bigo{\mathit{O}}
\DeclareMathOperator\rkf{\varphi}
\newcommand\ks{\mathcal{K}}
\newcommand\ms{\mathcal{M}}
\newcommand\cs{\mathcal{C}}
\newcommand\msb{\{0,1\}^n}
\newcommand\ksb{\{0,1\}^\kappa}
\newcommand\tsb{\{0,1\}^\theta}
\newcommand\allperms{\Pi_n}
\newcommand\allpermfams{\Pi_n^\kappa}
\newcommand\randraw{\xleftarrow{{\scriptscriptstyle\$}}}

\title{Advanced cryptology (GBX9SY06)\\
\decosix\\
Block ciphers}
\date{2017-11}
\author{Pierre Karpman}

\begin{document}

\maketitle{}

\section{First definitions}

A block cipher is a family of invertible mappings indexed by a key: $\E : \ks \times \ms \rightarrow \cs$, where both $\E$ and its inverse
$\E^{-1}$ are ``efficiently computable''\footnote{As concrete block ciphers (usually) fix all their parameters, there is not much sense to argue about this
efficiency in terms of asymptotic complexity, and we will not attempt to give a precise definition of what ``efficient'' means in this context.}.
In practice, we only care about ciphers for which $\ms = \cs$, meaning that $\E$ defines a family
of \emph{permutations} over $\ms$. In the vast majority of cases, we also have $\ms = \{0,1\}^n$ for some integer $n$, often equal
to 64 or 128 (though smaller, larger, and intermediate values are possible). In some corner cases, however, it may be that $\ms$
has a different structure, for instance all integers smaller than a certain $N$, or the set of valid credit-card numbers. This is the
concern of \emph{format-preserving encryption}, which will not be addressed in this course. Note that as long as cryptography
is implemented on digital circuits, there is no similar incentive to take $\ks$ of another form than $\{0,1\}^\kappa$. To summarize,
we will use the following definition.

\begin{defi}[Block ciphers]
\label{def:bc}
A block cipher is a family of mappings $\E : \ksb \times \msb \rightarrow \msb$ such that for all $k \in \ksb$, $\E(k,\cdot)$ is a permutation.
The quantities $n$ and $\kappa$ are positive integers, called the block size (or length) and key size, respectively.
\end{defi}

Block ciphers are important \emph{primitives} in symmetric-key cryptography. When used with a suitable \emph{mode of operation},
they allow to ensure the confidentiality and authenticity of data. We will not address the issue of modes in this course, but it
is important to remember that they are essential; without a proper mode, a block cipher is mostly useless.

There are many constructions that satisfy \autoref{def:bc}, but not all of them are useful in a cryptographic context. For instance,
$\E$ such that for all $k$ the mapping $\E(k,\cdot)$ is the identity is not terribly good at providing confidentiality. We thus need to express one
or several security properties that should hold for a ``good'' block cipher. Intuitively and informally, we typically require that:
\begin{enumerate}
\item Given $x_0, \ldots, x_m, \E(k,x_0), \ldots, \E(k,x_m)$, it should be ``hard'' to find $k$, even if the $x_i$s span the entire
message space $\msb$. An attack violating this property is called a \emph{key-recovery attack}\footnote{More generally, we may
require that it should be hard to recover $k'$ s.t. $\E(k,\cdot) = \E(k',\cdot)$ on most inputs.}.
\item Given $x_0, \ldots, x_m, \E(k,x_0), \ldots, \E(k,x_m)$ and $y_0$ (resp. $\E(k,y_0)$), it should be ``hard'' to learn information about
$\E(k, y_0)$ (resp. $y_0$) (except that it is distinct from the $\E(k, x_i)$s (resp. $x_i$s)). An attack violating this property is usually called a \emph{distinguishing}
attack.
\end{enumerate}
The first of these informal properties is maybe the most obvious one to think of, but it is not sufficient in itself to capture all the
desired properties of a block cipher. For instance, one may consider a cipher $\E$ for which key-recovery attacks are hard, but such that for all
$k$, $\E(k,m) = m$ whenever $m$ starts with a zero bit. It is quite clear that such an $\E$ does not provide adequate confidentiality.

In order to be more precise about our requirements for
good block ciphers, it is useful to first consider \emph{ideal} block ciphers. This allows to set a standard for any
security property we might be interested in, as these should be ``maximally hard'' for an ideal cipher. A
concrete (non-ideal) cipher $\E$ is then considered to be secure if it is hard to decide if one is interacting
with $\E$ or with an optimally-secure ideal cipher.

The definition of an ideal cipher is quite simple: it is simply a block cipher $\iE$ such that for all $k$, $\iE(k,\cdot)$ is
a permutation uniformly drawn at random among all permutations over the same domain. Thus, all keys of $\iE$ define
completely independent mappings that are all equally likely to be selected. Fixing the notation, we have the following definition.

\begin{defi}[Ideal block cipher]
\label{def:ic}
Let $\allperms$ denote the set of all $2^n!$ permutations over $\msb$. For any finite set $\mathcal{S}$, we write $x \randraw \mathcal{S}$
the uniform sampling of $x$ over all elements of $\mathcal{S}$. Then, an ideal block cipher is a cipher
$\iE : \ksb \times \msb \rightarrow \msb$ such that for all $k \in \ksb$, $\iE(k,\cdot) \randraw \allperms$.
\end{defi}

Note that not all ideal ciphers are in fact secure. For instance, when randomly sampling an ideal cipher,
there is a $1/(2^n!\times 2^\kappa)$ chance that $\iE$ be defined as the insecure identity-everywhere cipher. However, we claim
that the odds of picking an ``insecure'' cipher are small enough for this eventuality to be ignored altogether.

As already said above, one may use the notion of an ideal cipher to define the security of a concrete one by how easy it is for
an adversary to decide if he is interacting with an ideal cipher or not. This corresponds to the notion of
\emph{pseudorandom permutation} (PRP). Informally, this notion is expressed by having an adversary
interacting with an \emph{oracle} $\ora$ which is randomly chosen to be either of $\E(K,\cdot)$ or $\iE(K, \cdot)$
for a random key $K$ (as $\iE$ is ideal, the latter case is simplified by dropping the key and simply
asking that $\iE$ be a uniformly chosen permutation). Then one considers what is the best \emph{advantage} over a
random choice that an adversary has of deciding how $\ora$ was instantiated.

It seems clear that the advantage should in fact be a function of the \emph{data complexity} (the number of queries to
$\ora$) and of the \emph{time complexity} (where the unit is generally the time it takes to evaluate $\E$) of the adversary.
For instance, this accounts for the fact that any cipher (even an ideal one) of key size $\kappa$ can be broken by exhaustive key search
in time $2^\kappa$, and yet could still be secure if the adversary only has more limited resources.
We then define the PRP security of $\E$ through the following PRP advantage function (see e.g.~\cite{DBLP:journals/jcss/BellareKR00,bellare-rogaway_course}).

\begin{defi}[PRP advantage]
\label{def:prp}
The \emph{PRP advantage}  of $\E : \ksb \times \msb \rightarrow \msb$ is defined as:
\[
\Adv^\text{PRP}_{\E}(q,t) := \max_{A_{q,t}} \left|\Pr\left[A^{\ora}_{q,t}(\cdot) = 1 : \ora \randraw \allperms\right] -
\Pr\left[A^{\ora}_{q,t}(\cdot) = 1 : \ora = \E(K,\cdot), K \randraw \ksb\right]\right|,
\]
where $A^{\ora}_{q,t}$ denotes an adversary $A$ who makes at most $q$ queries to its oracle $\ora$, runs in time at most $t$, and returns a unique bit.

A similar notion of \emph{strong} PRP (SPRP) can be defined, where the adversary is additionally granted access to the inverse of $\ora$.
\end{defi}

Now we can use \autoref{def:prp} to formulate some requirements about the security of a ``good'' block cipher $\E$. As, we would
ideally want $\E$ to behave as close as possible to an ideal cipher $\iE$ (defined over the same domains), i.e. we wish that
$\Adv^\text{PRP}_{\E}(q,t) \approx \Adv^\text{PRP}_{\iE}(q,t)$ for all $q$, $t$ (for instance, we would like to
have $\Adv^\text{PRP}_{\E}(1,t) \approx t/2^\kappa$). By abuse of terminology, we usually say of such an
hypothetical cipher that ``it is a PRP''.

Note that breaking PRP security does not even require to recover the key $k$ when $\ora = \E(k,\cdot)$, which was one of the informal
goals stated at the beginning of this section. As recovering the key \emph{does} however allow to break PRP security, focusing only
on the latter does not weaken the requirements on $\E$, and allows to capture the second informal goal of resisting distinguishing
attacks.

The notion of PRP is useful to express desirable properties for concrete block ciphers. However, it is in itself useless to actually
evaluate their security. That it is to say, the definition provides very little insight into how to compute e.g. $\Adv^\text{PRP}_{\aes}$.
It is in fact the goal of cryptanalysis in general, by finding explicit attacks, to lower-bound the advantage function for specific ciphers
and complexities.

% Exercice: PRP-security of 3-round AES

\section{An ideal construction}

We now present a generic construction of block ciphers due to Even and Mansour~\cite{DBLP:conf/asiacrypt/EvenM91,DBLP:journals/joc/EvenM97}, which is very simple yet of considerable interest.
Let $\Perm : \msb \rightarrow \msb$ be a ``public'' (i.e. not secret) permutation, then one simply defines the
Even-Mansour cipher built from $\Perm$ as $\E(k_1||k_2,m) := \Perm(m \oplus k_1) \oplus k_2$, for all keys $(k_1||k_2)$
of $2n$ bits and all messages $m$. In fact, we can also go for an even simpler definition by taking $k_1 = k_2$, thence
obtaining $\E(k, m) = \Perm(m \oplus k) \oplus k$. What is remarkable is that both of these constructions can be proven
to be  good PRPs, assuming that $\Perm$ is itself a ``good'' permutation. Intuitively, this implies for instance
that $\Perm$ should not admit efficient distinguishers, in line with our second intuitive requirement for a block cipher.
However, this is not an easy notion to formalize; the approach taken by Even and Mansour to prove their construction
is then to only allow an adversary to make black-box oracle queries to $\Perm$. A possible interpretation
is then to consider this number of queries
itself to denote the ``time complexity'' of the attack.

\subsection{Proof sketch}

The security theorem proved by Even and Mansour does not use the notion of PRP, which was not formalised at the time. Instead,
it proves that the success probability of an adversary for the \emph{existential forgery problem} (EFP) is upper-bounded by
$\bigo(DT/2^n)$, where $D$ is the number of queries to $\E$ or its inverse (with an unknown fixed random key) and $T$
the number of black-box queries to $\Perm$ or its inverse. In other words, the scheme achieves security up to the birthday bound.
Note however that the proof can be adapted to a PRP setting, but we will rather focus on the original formulation.

The EFP attack considered here does the following: the adversary is first given access to $\E(k, \cdot)$, $\E^{-1}(k, \cdot)$, $\Perm$ and $\Perm^{-1}$
to make any query he wants;
then, he must provide a pair $(p, c)$ such that $\E(k,p) = c$, and neither $p$ nor $c$ was queried for the $\E$ or $\E^{-1}$ oracle respectively.
One can show without much trouble that this attack is not easier than decrypting a challenge ciphertext without knowing the key, or more generally recovering an unknown key.

We now briefly sketch the proof. The idea is to keep track of all queries made by the adversary and to count the number of possible \emph{bad} keys. These are the keys
whose consistency with $\Perm$ and $\E$ was ``checked''. After the last query is made, if the actual key is marked as bad, we assume that the adversary
won. Otherwise, one can show that all \emph{good} (i.e. not bad) keys have the same probability of being the actual key, and their number determines the
success probability of the adversary.
More precisely:
\begin{itemize}
\item A key $k$ is \emph{bad} if the adversary made a query relating $x$ and $y$ through $\Perm$ (i.e. queried $\Perm$ on $x$ or $\Perm^{-1}$ on $y$)
and a query relating $x'$ and $y'$ through $\E$ such that at least one of $x \oplus x'$ and $y \oplus y'$ is equal to $k$.
Indeed, if $k$ were the correct key used in $\E$, we would have $y' = \E(k,x') = \Perm(x' \oplus k) \oplus k$, so $x = x' \oplus k \Leftrightarrow y' = y \oplus k$,
which is a property that can easily be checked from the queries.
\item A key is \emph{good} if it is not bad.
\end{itemize}
One can then see that the number of bad keys is at most approximately equal to the number of queries to $\Perm$ times the number of queries to $\E$,
and the theorem mostly follows.

\subsection{Generic attacks}

An important and common question that arises in the context of security proofs such as the one above is the question of \emph{tightness}.
That is, we are interested in knowing if the actual security of a construction might be better than what the proof provides, or if there
exists an attack with a complexity matching the provable bound.
Ideally, we would like all proofs to be tight. If the status of a proof is not known, it is a natural research problem to either find an attack
matching the bound of the proof or a better proof matching less efficient attacks.

In the case of the Even-Mansour construction, it was quickly found out by Daemen that the proof is indeed tight~\cite{DBLP:conf/asiacrypt/Daemen91}. In fact, it is quite
straightforward to extract an ``optimal'' attack strategy from the proof sketch as given above. Several variants of what is generically
called \emph{slide attacks} exist (see e.g.~\cite{DBLP:conf/eurocrypt/DunkelmanKS12}), and we sketch two of them. The first one works as follows:
\begin{enumerate}
\item Pick $N$ distinct random values $x_i$, query $\E(x_i)$ and $\Perm(x_i)$, and insert their sums $y_i := \E(x_i) \oplus \Perm(x_i)$ along
with $x_i$ in a table.
\item For all $i$, $j$ such that $y_i = y_j$, return $x_i \oplus x_j$ as a key candidate.
\end{enumerate}
First, let us show that if the unknown key $k$ is equal to $x_i \oplus x_j$ for some $(i,j)$, the test in step 2) will indeed return $k$ as a candidate. This
is straightforward, as by definition we have:
\[
y_i = \Perm(x_i \oplus k) \oplus k \oplus \Perm(x_i) = \Perm(x_j) \oplus k \oplus \Perm(x_j \oplus k) = y_j.
\]
Now what is the probability (in function of $N$) that this event happens? As the $x_i$s are chosen randomly, their differences form
$N\cdot(N-1)/2$ (non-necessarily distinct) random values, hence the probability that one of these is equal to
$k$ is roughly $N^2/2^n$. This already matches the bound of the security proof, however, we can go further and show
that the attack can still be efficiently implemented if one additionally considers its time\footnote{We referred to the black-box queries
to $\Perm$ as the ``time complexity''. While it is indeed reasonable to consider this as a \emph{lower-bound} on the time complexity
of an attack that does not exploit structural properties of $\Perm$, an actual attack algorithm such as the one above might
imply additional processing beyond computations of $\Perm$.}   and memory complexity (which
are not accounted for in the theorem).
First, by using a suitable data structure (e.g. a hash table), one only needs $\bigo(N)$ memory to store
the $N$ $(x_i, y_i)$ pairs, and every collision can be detected in constant time. Thus, we only need to look at the expected
number of collisions and show that it is not more than $N$. If we make the reasonable assumption
that $x \mapsto \E(x) \oplus \Perm(x)$ behaves as a random function, then the expected number of collisions is $\bigo(N^2/2^n)$,
which is much smaller than $N$ for $N \ll 2^n$. We thus only expect a few ``false positive'' suggestions for $k$ before finding
the correct value and stopping the attack.
Putting everything together, this attack has a time, memory, and data complexity of $N$, and a success probability upper-bounded
by $\bigo(N^2/2^n)$.

We only describe, and do not analyse the second attack. Unlike the first, it actually requires the keys $k_1$ and $k_2$ to be identical
(the previous did not, even though it was presented in this simplified case), but it allows to trade queries to $\E$ for queries
to $\Perm$. That is, it succeeds with probability upper-bounded by $DT/2^n$ with $D$ and $T$ free to take any value, whereas the previous
attack required $D = T$. This attack works as follows:
\begin{enumerate}
\item Pick $D$ distinct random values $x_i$, query $\E(x_i)$, and insert $y_i := \E(x_i) \oplus x_i$ along with $x_i$ in a table.
\item Pick $T$ distinct random values $x'_i$, query $\Perm(x'_i)$, and insert $y'_i := \Perm(x'_i) \oplus x'_i$ along with $x'_i$ in a table.
\item For all $i$, $j$ such that $y_i = y'_j$, return $x_i \oplus x'_j$ as a key candidate.
\end{enumerate}

\subsection{Generalisations}

There are several ways to generalise the Even-Mansour constructions, beyond the simple variant taking $k_1 = k_2$ that we already considered.
One direction is to consider a block cipher $\E' : \ksb \times \msb \rightarrow \msb$
instead of a public permutation $\Perm$. This gives the ``FX'' construction $\E(k||k_1||k_2,m)
:= \E'(k, m \oplus k_1) \oplus k_2$, which was first suggested by Rivest and then analysed by Kilian and Rogaway~\cite{DBLP:journals/joc/KilianR01}. Using a method
similar to the one of Even and Mansour, they proved that the PRP advantage (this time exactly in the sense of \autoref{def:prp})
of an adversary attacking $\E$ is upper-bounded
by $D\cdot T / 2^{\kappa + n - 1}$, where $D$  represents the number of queries to the unknown oracle $\ora$ and $T$ represents the ``time
complexity'' of black-box accesses to the unkeyed $\E'$. One can see in particular that by taking $\kappa = 0$, i.e. fixing $\E'$ to be
a public permutation, this upper-bound essentially matches the one of the Even-Mansour scheme.

Another natural way to generalise the scheme is to compose it with independent instantiations of itself, i.e. defining $\E(k_1||k_2||\ldots||k_{r+1},m)$
as $\Perm_{r}(\ldots\Perm_{1}(m \oplus k_1) \oplus k_2)\ldots) \oplus k_{r+1}$. Chen and Steinberger showed that this achieved a PRP security up to
$\bigo(2^\frac{rn}{r+1})$ queries~\cite{DBLP:conf/eurocrypt/ChenS14}, while Lampe and Seurin showed an \emph{indifferentiability} property for the twelve-round iterated
scheme that uses distinct permutations $\Perm_1, \ldots, \Perm_{12}$ but equal keys $k_1 = \ldots = k_{13}$~\cite{DBLP:conf/asiacrypt/LampeS13}.
However, interestingly, the simplest
and most economical way to compose an Even-Mansour scheme, i.e. taking $r$ identical permutations and $r+1$ equal keys does \emph{not} work, in the sense
that it does not provide much more security than the original non-iterated scheme. Let us show why.

Consider $\E(k,m) := \Perm(\ldots\Perm(m \oplus k)\ldots)\oplus k$ for an arbitrary large number of applications of $\Perm$. We first observe that
if two values $x$ and $x'$ are related by $x' = x \oplus k$, then $\E(\Perm(x')) = \Perm(\E(x)) \oplus k$, and by symmetry $\E(\Perm(x)) = \Perm(\E(x')) \oplus k$.
It follows that:
\[
\E(\Perm(x)) \oplus \Perm(\E(x)) = \Perm(\E(x')) \oplus k \oplus \Perm(\E(x)) = \E(\Perm(x')) \oplus \Perm(\E(x')).
\]
Thus, we can attack $\E$ with probability $\approx 1$ by picking $2^{n/2}$ random values $x$ and looking for collisions for the function $x \mapsto \E(\Perm(x)) \oplus
\Perm(\E(x))$, each of them suggesting a candidate value for $k$. Note that unlike the attacks on the non-iterated Even-Mansour scheme, this one requires
\emph{chosen plaintexts} (but variants requiring only \emph{known plaintexts} also exist, for instance by considering collisions between
the lists $\{\Perm(\E(x)) \oplus x\}$ and $\{\E(x) \oplus \Perm^{-1}(x)\}$).

\section{Related-key attacks}

So far, we only focused on attacks from adversaries who could only access a single oracle, for instance corresponding to $\E(k,\cdot)$ for an unknown key
$k$. A way to increase an adversary's power is then to allow access to a family of oracles, for instance corresponding to $\{\E(\rkf(k), \cdot),~\rkf \in \Phi \}$,
still for an unknown key $k$, and $\Phi$ a set of \emph{related-key functions} $\ksb \rightarrow \ksb$. That is to say, the adversary is given access to
$\E$ with several unknown keys that are related one to another through the elements of $\Phi$. We call \emph{related-key attack} an attack that corresponds
to this model.

The security requirements associated with related-key attacks are the same as for the \emph{single-key} case. For instance, we can informally state that
it should be hard to recover the unknown $k$ for a cipher to which we are given related-key oracle access. It is also possible to define a
related-key (S)PRP notion that generalises the one of \autoref{def:prp}, as was done by Bellare and Kohno~\cite{DBLP:conf/eurocrypt/BellareK03}.
We let again $\iE$ denote an ideal cipher, such that for all $k$,
$\iE(k,\cdot) \randraw \allperms$. We write $\iE \randraw \allpermfams$ the uniform sampling of such a cipher.
We define a related-key oracle $\E_{\text{RK}(\cdot,K)}(\cdot)$ for a block cipher $\E : \ksb \times \msb \rightarrow
\msb$ that takes a first input $\rkf : \ksb \rightarrow \ksb$ and a second input $m \in \msb$, and returns $\E(\rkf(K), m)$.
We then have the following.

\begin{defi}[Related-key PRP advantage restricted to $\Phi$]
\label{def:rkprp}
The \emph{related-key PRP advantage} of $\E$ with respect to the related-key class $\Phi$ is defined as:
\[
\Adv^\text{PRP-RKA}_{\Phi,\E}(q,t) := \max_{A_{q,t}} \left|\Pr\left[A_{q,t}^{\iE_{\text{RK}(\cdot,K)}(\cdot)} = 1 : K \randraw \ksb, \iE \randraw \allpermfams\right] -
\Pr\left[A_{q,t}^{\E_{\text{RK}(\cdot,K)}(\cdot)} = 1 : K \randraw \ksb\right]\right|.
\]
\end{defi}

One can see that this definition is parameterized by the allowed related-key class $\Phi$. In fact, this parameterization plays a major role in how secure
a cipher can be according \autoref{def:rkprp}: there are classes $\Phi$ with respect to which \emph{no} cipher (even an ideal one)
attains any meaningful level of security. For instance, assume that $\Phi$ includes a constant function $\rkf : x \mapsto c$; then an adversary
may simply query its related-key oracle on $\rkf$ and an arbitrary $c'$ and compare the result with $\E(c,c')$; it succeeds with
advantage close to one by answering 1 if the two values match and 0 otherwise.

There are less trivial related-key classes for which similar problems occur. One of the perhaps less intuitive examples is to consider
$\Phi = \rkf^\oplus \cup \rkf^+$, where $\rkf^\oplus = \{x \mapsto x \oplus \Delta,~\Delta \in \ksb\}$ and $\rkf^+ = \{x \mapsto x + \Delta,~\Delta \in \ksb\}$
(with $+$ denoting here the addition modulo $2^\kappa$). Remarkably, non-trivial security \emph{is} achievable w.r.t. $\rkf^\oplus$ or $\rkf^+$
in isolation. We will not explore this formally, rather focusing on the intuition.

There are two key properties for $\Phi$ to allow non-trivial security. The first is that it be \emph{collision-resistant}, and the second that it
be \emph{output-unpredictable}. Collision resistance means that it is hard to find two functions $\rkf$, $\rkf'$ such that $\rkf(K) = \rkf'(K)$
for a uniform $K$. This is
the property that does not hold for $\rkf^\oplus \cup \rkf+$ and that can be exploited to mount easy attacks using this class. However, it does
hold trivially for, say $\rkf^\oplus$, as $\Delta \mapsto K \oplus \Delta$ is a permutation for any $K$.
The output unpredictability property means that it is hard to guess $\rkf(K)$ for a uniform $K$; more generally, it should be hard
to guess any of the elements of $\{\rkf(K), \rkf \in X \subseteq \Phi\}$ relatively to the size of $X$. It is easy to see that this property does not hold if
$\Phi$ includes constant mappings, but that it does for $\Phi = \rkf^\oplus$.

We now illustrate the fact that if many related-key queries with different functions $\rkf$ are allowed, the security of any cipher
degrades significantly. Consider a simplified attack setting where the adversary interacts with a known cipher $\E$ with unknown key $k$ and
tries to recover $k$. To do so, he is allowed oracle access to $\E(\rkf(k),\cdot)$, $\rkf \in \rkf^\oplus$. A good (in fact basically optimal)
attack that does not exploit any properties of $\E$ consists in querying $\E(k \oplus \Delta,0)$ for $2^{\kappa/2}$ randomly selected
$\Delta$ and putting the results in a list $\mathcal{L}$ along with the value $\Delta$. Then the adversary tries $2^{\kappa/2}$ candidates $k'$ for the key,
each time computing $\E(k',0)$. Any match with an element of $\mathcal{L}$ suggests a value for $k$, and one expects such a match with high probability.

\subsection{Provably-secure constructions}

Even when $\Phi$ is restricted to \emph{meaningful} functions, the related-key model gives significantly more power to the adversary. The collision attack
sketched above is an example; another one is to consider some very simple attacks on the Even-Mansour scheme. Let $\Phi$ include at least one
function of the form $x \mapsto x \oplus \Delta$, then an adversary can distinguish an Even-Mansour scheme from an ideal cipher by querying
$\E(k \oplus \Delta, \Delta)$, $\E(k, 0)$, and checking if the two values only differ by $\Delta$.

This attack does not contradict the security proof of Even and Mansour, as it requires related-key queries that are not covered by the single-key proof.
It does show however that a good cipher in the single-key setting might be terribly broken in a related-key setting, even w.r.t. ``meaningful''
related-key classes.

It is quite easy to see that the above attack also applies to iterated Even-Mansour schemes when independent keys are used at every round.
A slightly more involved but also very efficient attack exists on the two-round scheme that uses identical keys. However, it was proven
(in an ideal permutation model similar to the one of the single-key proof)
that using three rounds or more with identical keys leads to a construction that is related-key secure w.r.t.
$\rkf^\oplus$~\cite{DBLP:conf/eurocrypt/CogliatiS15,DBLP:conf/fse/FarshimP15}.

\subsection{Tweakable block ciphers}

The fact that single-key and related-key security may be widely disconnected is part of the reason why not all concrete block ciphers are designed
to be related-key secure. This comes usually with the argument that resisting related-key attacks (for some a priori defined classes)
would add some computational overhead and that the model is altogether unrealistic, as no properly designed protocol would allow
an attacker to make related-key queries. We will not delve into this debate whose implications are rather complex, instead giving
an example of a constructive use of related-key secure ciphers.

First, let us define \emph{tweakable block ciphers}. These are simply block ciphers that take a second parameter called a \emph{tweak},
such that all distinct pairs of keys and tweaks define (ideally) independent permutations. The difference between the key and the
tweak is that the latter is public and may be freely chosen by the adversary. Using simplified expressions for the domains, we have the
following.

\begin{defi}[Tweakable block ciphers]
\label{def:tbc}
A tweakable block cipher is a family of mappings $\TE : \ksb \times \tsb \times \msb \rightarrow \msb$ such that for all $k \in \ksb$,
$t \in \tsb$, $\E(k, t, \cdot)$ is a permutation.
\end{defi}

Tweakable block ciphers are useful in that they allow to ``diversify'' a fixed-key instance without selecting a new, independent key.
In other words, two communicating parties may first secretely share a secret key $k$ with which to use $\TE$, and then publicly agree on
a new tweak for each message to be exchanged.
The concept of tweakable ciphers was formalised by Liskov et al.~\cite{DBLP:journals/joc/LiskovRW11},
but already implicitly used for instance by Rogaway et al. to define an efficient
mode of operation~\cite{DBLP:conf/ccs/RogawayBBK01}.

A simple way to build a tweakable block cipher $\TE$ from a ``regular'' cipher $\E$ is to define $\TE(k,t,\cdot)$ as $\E(k \oplus t,\cdot)$.
The security of this construction  fully reduces to the related-key security of $\E$ w.r.t. $\rkf^\oplus$, which may be non-trivial. For
instance, this construction provably achieves a meaningful level of security in the ideal permutation model if $\E$ is a three-round
iterated Even-Mansour scheme with identical keys. Note however that because of generic collision attacks, the security
is limited w.r.t. the number of different tweaks for which $\TE$ is queried. The design of similar generic constructions that are
secure beyond the birthday-bound is a rather active research topic.

\bibliographystyle{alpha}
\bibliography{tehbib}

\end{document}