MASARYK UNIVERSITY
FACULTY OF INFORMATICS
Use of Michelangelo service in
Unity engine
BACHELOR'S THESIS
Maroš Beťko
Brno, Spring 2019
MASARYK UNIVERSITY
FACULTY OF INFORMATICS
Use of Michelangelo service in
Unity engine
BACHELOR'S THESIS
Maroš Beťko
Brno, Spring 2019
This is where a copy of the official signed thesis assignment and a copy of the
Statement of an Author is located in the printed version of the document.
Declaration
Hereby I declare that this paper is m y original authorial work, which
I have worked out o n m y o w n . A l l sources, references, a n d literature
used or excerpted during elaboration of this work are properly cited
and listed i n complete reference to the due source.
Maroš Beťko
Advisor: M g r . Jiří Chmelík, P h . D .
i
Acknowledgements
Firstly, I w o u l d like to thank m y supervisor M g r . Jiří Chmelík, Ph.D.,
for guidance and help w i t h the thesis. Also I need to thank Martin Ilčík
for his cooperation, willingness to discuss a n d implement new ideas
into Michelangelo framework a n d for pointing me to many helpful
sources. Next, I w o u l d like to thank m y girflriend for supporting me
throughout m y whole study.
iii
Abstract
Procedural generation is used i n many aspects of game development,
and its effectiveness is often tied to the usability of available tools.
Michelangelo is a cloud-based service that provides a procedural
modeling framework based on formal grammars. This thesis focuses
on implementing a package for Unity game engine, that enables game
developers to integrate optimized procedural assets generated w i t h
Michelangelo grammars into their Unity projects directly from Unity
editor.
iv
Keywords
A G D + M , C#, computer graphics, H C I laboratory, stream processing,
Unity
v
Contents
Introduction 3
1 Procedural techniques 5
1.1 Procedural generation using formal grammars 6
1.2 Michelangelo 6
2 Design and implementation 9
2.1 Package introduction 10
2.1.1 Authentication 10
2.2 Account info and energy 11
2.3 Creating Michelangelo objects 12
2.3.1 SceneObject 12
2.3.2 GrammarObject 13
2.4 Generating meshes 15
2.4.1 Parse tree 17
2.4.2 Shape 18
2.4.3 Materials 18
2.4.4 Parse tree script 19
2.4.5 Parse tree preview 19
2.4.6 Attaching Unity GameObjects 21
2.4.7 Compilation output 22
2.5 Communication with Michelangelo service 22
2.6 Serialization 26
2.6.1 Serialization of Unity scripts 26
2.6.2 MessagePack 29
2.7 Mesh optimization 29
3 Future work 33
3.1 Advanced materials and procedural shaders 33
3.2 IntelliSense for grammar syntax 33
3.3 LOD's and occlusion culling 34
Conclusion 35
Bibliography 37
vii
List of Figures
2.1 Michelangelo tab i n Unity's editor U I . 10
2.2 L o g i n screen U I from Michelangelo window. 11
2.3 L o g out section from Michelangelo w i n d o w UI. 11
2.4 Scene object inspector window. 13
2.5 Grammar listing i n Michelangelo window. 14
2.6 Grammar object inspector window. 14
2.7 Generate object button i n grammar object's inspector
window. 17
2.8 Generated object i n Unity's hierarchy view of open
scene. 19
2.9 Highlighted mesh of one of child nodes that the object
was split into. 20
2.10 Parse tree preview inside grammar object's inspector after
it's mesh was generated. 20
2.11 Highlighted node that was selected i n grammar object's
parse tree i n it's inspector. 21
2.12 Compilation output view inside object's inspector. 22
2.13 Highlights of individual floors of this generated
house. 30
ix
Listings
2.1 L o g i n example code 11
2.2 L o g out example code 12
2.3 Constructing of new scene object from code 13
2.4 Constructing new grammar object from code 15
2.5 Example of data posted to generate grammar endpoint. 16
2.6 Example model object 16
2.7 M o d e l object serialized to code 16
2.8 Generating of a grammar object from code 17
2.9 Example of attaching empty game objects to all nodes
of object's parse tree that are called " S p a w n " 21
2.10 Reading of compilation output from script 22
2.11 Extension methods for UnityWeb Request 23
2.12 Simplified structure of Web A P I method that sends a
get request to Michelangelo A P I 25
2.13 Example of MichelangeloSession method w r a p p i n g a
W e b A P I 26
2.14 Serialization of ParseTree class 28
1
Introduction
Procedural generation techniques evolve rapidly every year, but expectations
for procedurally generated content grow even faster. There
have been efforts to generate many different kinds of content, f r o m
now-standard terrain generation to more complex systems like procedural
narration and story.
One of the services that provide procedural generation is Michelangelo
[1] that makes use of formal grammars to generate complex objects
from combining and transforming simple shapes by building a hierarchy
of rules and goals. Michelangelo is available as an online tool,
that can be used from a web browser, w i t h export options to either
download the procedurally generated mesh or render of a generated
object.
While it is possible to export a mesh from Michelangelo service and
import it to the Unity project, it is not a simple workflow to work with.
It also loses any information about the generation tree process, and
for Unity, it is just plain mesh data. A l l this data from Michelangelo
can be used i n U n i t y to integrate these generation results better. O n
top of that, meshes exported f r o m Michelangelo are not optimized
for use i n a game engine, which makes any larger scenes unusable for
real-time applications.
This thesis focuses on bringing Michelangelo service to the Unity
game engine. O n top of p r o v i d i n g basic functionality, available to
users through Michelangelo's web interface [1], the tool also takes care
of Unity-specific aspects such as mesh combining to achieve better
performance, mesh serialization and game object instantiation. A l l of
these features are available through in-editor U I and a C# scripting
API.
The rest of the thesis is organized as follows. The first chapter of
the thesis focuses on procedural generation as a whole and theoretical
background of used systems. The second chapter describes the implementation
of the Unity package and some of the intriguing challenges
that occurred during development.
3
1 Procedural techniques
Procedural generation is a method of content creation based on the
algorithmic representation of the desired outcome. It allows for the
generation of large quantities of content, that w o u l d otherwise take
large amounts of time to create by hand. Procedural generation is
mainly used i n computer graphics, computer games, simulations, and
data visualization. Historically, content generation included assets
such as textures, 3D models, or whole scenes that w o u l d otherwise
take too m u c h time to create i n many variants for use i n large scale
projects. A n excellent example of this is vegetation, that can be generated
using L-systems [2]. For example, to create a realistic looking
forest, a large number of variations of vegetation should be distributed
i n a natural non-repeating way. M a n u a l positioning of every object
w o u l d be too time-consuming, while procedural generation is m u c h
faster and scales better.
There are many different approaches to content generation. Some
are simpler, like the usage of Perlin noise for rough terrain generation
[3], and some are quite complex, like generation based on l-systems [4]
or even neural networks. To increase the set of results, these algorithms
can often be parametrized or can make use of randomization.
There are many factors to take into consideration w h e n deciding
w h i c h approach to choose. Firstly, it is essential to consider whether
the result of the algorithm w i l l undergo further processing by designers/
developers or if it should represent the final state of the content.
Another factor is the scale or the number of generated instances. If the
number is high, considerable variation between instances is desired,
otherwise finer or more complex details can be more beneficial even
if multiple instances can seem too similar to each other. Another metric
is a subjective opinion of h o w realistic the results are, whether a
non-involved observer can distinguish between a hand-made and a
generated asset.
In order to make the results more believable, procedural generation
can employ many simulation techniques, often w i t h simplified models,
to make the generation process faster, while maintaining the desired
realism.
5
1. P R O C E D U R A L TECHNIQUES
The recent development of procedural generation is closely connected
to the demand of the game industry. W h e n correct procedural
techniques are used, they can extend the play time drastically by prov
i d i n g endless filler content. Some genres like roguelike games are
highly dependant on procedural generation, where the m a i n game
loop consists of replaying the same progression curve placed i n procedurally
generated scenes. The major selling point of many popular
sandbox games like Minecraft or N o Man's Sky is their infinite world
which relies heavily on procedural generation to create limitless space
for hours of play time.
1.1 Procedural generation using formal grammars
A grammar-based procedural generation approach is especially useful
when dealing w i t h structured generation goals that can be generated
from simpler shapes and are hierarchically combined into more complex
ones [5]. M a n y man-made structures can be easily described as a
combination of simpler objects such as a hospital consists of wings;
each w i n g consists of floors, floors consist of corridors and specialized
rooms. Over the years various grammar systems were developed,
from the first shape grammars [6] and L-systems [2] to more recent
like C G A + + from Schwarz and Miiller [5].
Some of these grammar systems like C G A and CGA++ are directly
focused on procedural generation of architecture. Each of them were
developed w i t h the a i m of p r o v i d i n g more options and better syntax
to allow better control over the result, like for example "events"
implemented i n C G A + + grammar [5, Chapter 4] that allow passing
information (e.g. whether an entrance was already generated, w h i c h
their example should have only one) between different branches of
generation tree.
1.2 Michelangelo
Michelangelo builds on top of previous theoretic work of Miiller et al.
[7] and addresses various issues present i n CityEngine, software built
based on research by the aforementioned authors. A few key points
w h i c h are addressed i n the paper presenting Michelangelo [8] are:
6
1. P R O C E D U R A L TECHNIQUES
• A cloud repository of generative grammars.
• Queries that automatically apply rules from the cloud.
• N o management overhead for team collaboration.
• N o special syntax. Grammars are instead C# scripts.
The main focus is then on ease of collaboration, maintainability of
dependencies i n larger projects, using fluent and descriptive syntax
instead of symbols and cloud-based derivation process.
7
2 Design and implementation
Michelangelo is an online tool that has a browser-based editor. The
editor on the user side communicates with Michelangelo's server that
evaluates the grammars on the backend and sends the results to the
editor. These results can then be exported as a . obj file for use outside
of Michelangelo. This package aims to skip the browser-based editor
and bridge the communication w i t h Michelangelo's backed directly
with Unity while also adding functionality that helps better integrate
the results in Unity projects that w o u l d otherwise not be possible from
imported . obj file.
After the initial analysis w i t h the Michelangelo developers, the
following functionality was decided on as the most important for providing
value to users choosing this option of interacting w i t h Michelangelo
tool over the existing browser interface.
• Authentication to the Michelangelo A P I and required information
about logged i n user.
• Grammar browsing capabilities, similar to the existing browser
interface.
• Sending generation requests for existing grammars.
• Processing of generation response and creating meshes and materials
inside Unity.
• Providing access to Michelangelo A P I through both Unity's Editor
interface and a C# scripting interface.
O n top of these baseline features, the package should provide additional
options not present w h e n working w i t h the existing browser
interface.
• Serialization of response f r o m Michelangelo backend inside
Unity projects.
• Automatic optimization features to make generated meshes
more performant for use i n Unity.
9
2. D E S I G N A N D IMPLEMENTATION
• Semantic information about the generation result that can be
used to attach game logic or other Unity-specific scripts and
features to the generated results.
• Simplified approach to generating meshes based on goals only
that removes the requirement of referencing existing grammar
saved o n the Michelangelo server.
2.1 Package introduction
Package is provided as a . unitypackage archive that can be extracted
into any Unity project (version 2018.1 and higher). Online, it is available
from it's github repository as a . unitypackage or from NuGet under
the name MichelangeloUnity (see R E A D M E 1
for up-to-date information).
A l l package related functionality is added under Michelangelo
tab inside Unity's editor w h i c h can be seen i n Figure 2.1.
2.1.1 Authentication
Since Michelangelo is an online service and almost all of its A P I endpoints
require authentication, users first need to log in. This can be
done through a U I seen i n Figure 2.2 or C# scripting interface as i n Listing
2.1. Required requests w i l l be sent to obtain authentication token
from Michelangelo login endpoint, w h i c h is then saved inside local
EditorPrefs [9] of Unity's Editor. Users need to log i n w i t h an existing
Michelangelo Window Help
Michelangelo
GrarnrnarObject
SceneObject
Figure 2.1: Michelangelo tab i n Unity's editor UI.
https://github.com/Haaxorl689/MichelangeloUnity
10
2. D E S I G N A N D IMPLEMENTATION
0 Inspector | Michelangelo
Log In
User name or Email
Password
1
Register as a new user.
Forgot your password?
Figure 2.2: L o g i n screen U I from Michelangelo window.
1 M i c h e l a n g e l o S e s s i o n . L o g i n ( " u s e r n a m e " , "password")
2 . T h e n ( u s e r l n f o => { /* logged in s u c c e s s f u l l y */ })
3 . C a t c h ( e r r o r => { /* error occurred */ }) ;
Listing 2.1: L o g i n example code.
Michelangelo account that can be registered online 2
on Michelangelo's
website.
MichelangeloSession uses a Promises 3
library for asynchronous
web requests that w i l l be described i n chapter 2.5.
Users can also log out to clear the saved authentication cookie,
either from U I as seen in Figure 2.3 or from a C# scripting interface as
in Listing 2.2.
2.2 Account info and energy
Generating Michelangelo grammars or scenes costs Energy and the
more complex the target, more energy w i l l be used. The energy is
loosely connected to time Michelangelo's server spends on generating
Ö Inspector | Michelangelo |
User:
Energy:
Log Out
Figure 2.3: L o g out section from Michelangelo w i n d o w UI.
https://michelangelo.graphics/Account/Register
3
https://github.com/Real-Serious-Games/C-Sharp-Promise
11
2. D E S I G N A N D IMPLEMENTATION
1 M i c h e l a n g e l o S e s s i o n . L o g o u t ( )
2 . T h e n ( u s e r l n f o => { /* logged out s u c c e s s f u l l y */ })
3 . C a t c h ( e r r o r => { /* error occurred */ });
Listing 2.2: L o g out example code.
requested grammar, and it slowly replenishes for each user if their
max energy limit is not met. Available energy of currently logged
in user can be seen i n the Michelangelo editor w i n d o w or accessed
in C# scripting interface through static variable MishelangeloSess
i o n . U s e r . e n e r g y A v a i l a b l e .
2.3 Creating Michelangelo objects
Michelangelo generates meshes from grammars w h i c h are described
in Michelangelo's user m a n u a l 4
. In MichelangeloUnity, there are two
approaches to creating meshes. First, there is a GrammarObject that
links to existing Michelangelo grammar online or then there is a SceneObject
that specifies only a model (see user manual ch. 1.2).
In both cases a grammar code needs to be sent to the Michelangelo
API, which then returns raw mesh data, materials, parse tree info, and
other data.
2.3.1 SceneObject
Scene objects are used for the simplest use cases where a user only
want to generate already defined goals and not create any new rules.
First, the user needs to specify a generation G o a l , w h i c h is a case
insensitive string (e.g., in the example the goal is Chair which will try
to find any rule that generates a chair in Michelangelo's cloud database of
grammars). There are also a few more options like Size, w h i c h w i l l
specify a box area that the mesh should fit in. Selecting Center on
pivot w i l l make sure that the pivot of generated mesh is at the bottom
center of its bounding box. Lastly, with Restrict user can restrict where
the rules to satisfy this goal can be found (e.g., by selecting Mine only
grammars connected to their account will be used). Setting u p the scene
4
https://www.overleaf.com/read/scvmgqtbbtmh
12
2. D E S I G N A N D IMPLEMENTATION
• ..• Scene Object [Script) Q r,', ö .
Goal: Chair
Goal Chair 1
Size | Y | i . g | z | i
Center on pivot a
Restrict
Source 1 U nie stricte d
Figure 2.4: Scene object inspector window.
var newGameObject = S c e n e O b j e c t . C o n s t r u c t ( n e w Model {
Name = " C h a i r " ,
S i z e = new V e c t o r 3 ( 1 . O f , 1.9f, l . O f ) ,
C e n t e r O n P i v o t = t r u e ,
R e s t r i c t S o u r c e s = new R e s t r i c t S o u r c e {
SourceType = S o u r c e T y p e . U n r e s t r i c t e d ,
} ,
>);
Listing 2.3: Constructing of new scene object from code.
object can be achieved either through U I as seen i n Figure 2.4 or C#
scripting interface as i n 2.3.
2.3.2 GrammarObject
Grammar objects must be linked to an existing grammar i n Michelangelo's
database. They can be instantiated either from the Michelangelo
editor tab and then connected to grammar from the grammar list
or from Michelangelo w i n d o w ' s grammars listing seen i n Figure 2.5.
There, users can filter grammars by name and source. The grammar
listing is retrieved from Michelangelo A P I endpoint and contains all
grammars available to the authenticated user.
Grammars received from grammar listing endpoint are d o w n loaded
without their source code. Before being able to work w i t h the
grammar and generate its mesh, first, the source code needs to downloaded
by requesting one concrete grammar from a different endpoint.
Grammar source codes are stored as plain text files inside MichelangeloGrammarSources
folder to make it easier for users to edit the
grammars. One of the planned features is an IntelliSense support for
13
2. D E S I G N A N D IMPLEMENTATION
Gram mars
f Create G r a m m a r
1
Create G r a m m a r
IncrementalDesign Opening Oblique T y p e
Type DOG
Last Modified 3/11/2019 5:30:43 PM
Code
Instantiate
IncrementalDesign Lattice Ornaments
Type DOG
Last Modified 3/11/2019 12:46:26 AM
Code II IncrementalDesignLatticeOrnamentsG o f Download
Instantiate
Prev 2/3
Filters
Name
Source
Items per page
Local only
•
Figure 2.5: Grammar listing i n Michelangelo window.
i* G r a m m a r Object (Script) 0 n! #T
Morphing Roof 1
Type
Last Modified
DOG
5/8/Z019 5:53:27 A M
Code M o r p h i n g R o o f l G r a m m a r ° 1 Download ||
Figure 2.6: Grammar object inspector window.
these local grammar files, w h i c h w o u l d then be saved as a C# script
file instead of plain text. Without the symbols required to load the
grammar file as valid C# script, they cannot be saved as . cs files since
Unity automatically tries to compile any script files inside project's
Assets folder. Grammars w i t h existing source code file contain a link
to the text file as seen i n Figure 2.6.
GrammarObj ect can be also instantiated from C# scripts by providing
a Grammar object reference to it's static construct method as seen i n
Listing 2.4. Read only reference to currently downloaded grammars
can be obtained from static MichelangeloSession class and the list can
also be refreshed with MichelangeloSession.Ref reshGrammarList ()
method.
14
2. D E S I G N A N D IMPLEMENTATION
// With grammar l i s t already updated
var newGameObject = G r a m m a r O b j e c t . C o n s t r u c t (
M i c h e l a n g e l o S e s s i o n . G r a m m a r L i s t
.Values
. F i r s t ( g => g.name == "Ship")
);
// With fetching grammar l i s t
M i c h e l a n g e l o S e s s i o n . R e f r e s h G r a m m a r L i s t ( )
.Then(grammarList => {
var newGameObject = G r a m m a r O b j e c t . C o n s t r u c t (
g r a m m a r L i s t . V a l u e s . F i r s t ( g => g.name == " S h i p " )
);
})
Listing 2.4: Constructing new grammar object from code.
2.4 Generating meshes
Before generation request can be sent to Michelangelo's A P I , both
objects need to fulfill a few requirements:
• GrammarObject needs to be linked to existing grammar w i t h
downloaded code
• SceneObject needs to have a non-empty goal and non-zero size
These requirements are enforced because i n GrammarObj ect's
case Michelangelo A P I awaits an i d of existing grammar and its u p dated
info i n a POST request form which can look like data i n Listing
2.5.
For SceneObjects, an endpoint does not require an existing grammar
i d nor any other metadata. It only requires grammar type which
refers to Michelangelo's grammar syntax generation, where currently
only valid option is D O G and a code string w h i c h is checked on
Michelangelo backend, that it does not contain any production rule
definitions (see ch. 3 of Michelangelo user manual). Before generation
request is sent, this code string is generated from the p r o v i d e d
Model object. Example M o d e l object can be seen i n Listing 2.6 and it's
grammar representation i n Listing 2.7.
15
2. D E S I G N A N D I M P L E M E N T A T I O N
{
"ID": "5cde612f0ca89a39a4ed918b " ,
"Name": " C a s t l e " ,
"Type": "DOG",
"Code": "new M o d e l ( \ " C a s t l e \ " ) ; " ,
"OnlyNID": 4294967295,
"Render": f a l s e
}
Listing 2.5: Example of data posted to generate grammar endpoint.
new Model {
Name = " C h a i r " ,
S i z e = new V e c t o r 3 ( 1 . O f , 1.9f, l . O f ) ,
C e n t e r O n P i v o t = t r u e ,
R e s t r i c t S o u r c e s = new R e s t r i c t S o u r c e {
SourceType = SourceType.Mine,
} ,
>
Listing 2.6: Example model object.
new M o d e l ( " C h a i r " ) . W i t h (
S i z e d . O f , 1.9f, l . O f ) ,
P o s i t i o n ( 0 . 5 f , O.Of, 0 . 5 f )
) . R e s t r i c t ( S o u r c e . M i n e ) ;
Listing 2.7: M o d e l object serialized to code.
16
2. D E S I G N A N D IMPLEMENTATION
T <* G r a m m a r Object (Script) 0
Tower 01
Type DOG
Last Modified 7/22/2013 11:06:28 PM
Code T o w e r O l G r a m m a r ° 1 Download |
Generate new mesh
Figure 2.7: Generate object button i n grammar object's inspector w i n -
dow.
1 grammarObject.Generate()
2 .Then(response => { /* mesh was created */ })
3 . C a t c h ( e r r o r => { /* generation f a i l e d */ })
Listing 2.8: Generating of a grammar object from code.
Generation request can be then sent from grammar/scene object's
inspector as seen i n Figure 2.7 or f r o m a C# scripting interface as i n
Listing 2.8.
After the initial generation request, Michelangelo A P I responds
w i t h a generation token, that refers to a generation task started b y
request. Then the application sends a G E T request w i t h this token
repeatedly, until the response either contains generated mesh and
other data or an error message. In case that generation ended w i t h
an error, this error message is propagated to both U I and scripting
interface, w h i c h is further described i n chapter 2.4.7.
After generation is successful, a J S O N object containing mesh data,
materials and parse tree information w i l l be i n the response.
2.4.1 Parse tree
Parse tree is a tree structure containing information about the derivation
process (see ch. 2 of Michelangelo user guide). It consists of nodes,
representing production rules that were applied during the generation
process and is stored inside a dictionary keyed by unique ids of nodes.
C h i l d nodes are then referenced by their unique ids.
To simplify mesh optimization process that w i l l occur later and w i l l
be described i n chapter 2.7, this data structure is first transformed, and
17
2. D E S I G N A N D IMPLEMENTATION
some data from parent nodes are passed down to their children, while
also creating n e w child nodes that were not present i n the original
data set. Members passed down to the child nodes are Ontology and
Shape. Both these members are saved i n an array inside their parent,
where an object at each index belongs to the child object at the same
index inside the Children member array. Also, a parent i d is added to
each node i n this step, to make the tree traversal faster w h e n needed
(e.g., it is used inside TreeView deriving class used i n object's custom
inspector).
2.4.2 Shape
Under the Shape attribute of each parse tree node, mesh and other
data for rendering is saved i n GeometricModel object. Data restructuring
mentioned i n the previous section was done mainly to have
this rendering data accessible directly from the current node. For nonleaf
nodes, shape contains mesh info of b o u n d i n g object that child
nodes can be positioned in. For leaf nodes of the parse tree, this shape
contains mesh info that should be rendered i n the final combined
mesh.
M e s h is either defined by a Primitive w h i c h can be one of valid
primitive types of Michelangelo grammars (e.g. "Sphere" or " C y l i n der")
or can be a "Mesh", i n w h i c h case the Mesh attribute w i l l be an
object, containing raw vertex and index data of this mesh. Meshes of
primitive types are generated by local code, that returns a unit sized
mesh of that type as an instance of Unity's Mesh class [10]. For other
meshes, the vertices and normals are used to create new instance of
Mesh class.
The shape also contains a transformation matrix for given mesh
and a material i d , that refers to material from generation response.
2.4.3 Materials
Materials i n Michelangelo's grammars are represented as attributes
for Pixar's Renderman framework, containing dictionaries of vector
and scalar attributes. Since Unity uses its H L S L shaders w i t h different
attributes, only some properties like Color, Metallic and Glossiness,
18
2. D E S I G N A N D IMPLEMENTATION
Hierarchy |
Create T
* 3 s a m p l e S c e n e T
=
[ j | Main C a m e r a
U Directional Light
^ IncrementalDesign Latticed Walls 1
WALL .XI
WALLS.Z
Figure 2.8: Generated object i n Unity's hierarchy view of open scene.
w h i c h can be doubtlessly m a p p e d between the two systems are supported
right now. For more details see chapter 3.
The materials dictionary is then transformed to dictionary of Unity's
Material classes [11] w i t h a Standard Shader [12] used as it's shader.
2.4.4 Parse tree script
For performance reasons and Unity's max vertex count restrictions,
the shape data is then split between multiple game objects instantiated
as a child objects under the Grammar/Scene object that contains the
parse tree data as can be seen i n Figure 2.8.
These child objects have a ParseTreeScript component attached
and cannot be edited. Each of these child tree scripts represents a node
in parent's parse tree, and contain combined mesh data of all of the
leaf nodes under it. M o r e on h o w these nodes are selected, and their
meshes combined w i l l be described i n chapter 2.7. M e s h highlight of
one of these parse tree script objects can be seen i n Figure 2.9.
2.4.5 Parse tree preview
Parse tree can be useful w h e n viewing h o w the final scene was built
from partial goals and can be v i e w e d i n object's inspector as seen
in Figure 2.10. After selecting a node i n the parse tree preview, it's
b o u n d i n g mesh w i l l be highlighted i n scene v i e w as seen i n Figure
2.11.
19
2. D E S I G N A N D IMPLEMENTATION
Parse tree
T ROOT
T ID. LATTICE. WALL. 1 I Attach GO
» WALLS. Z r Attach GO
^ WALL.X2 i Attach GO
Expand All J I Collapse All
Figure 2.10: Parse tree preview inside grammar object's inspector after
it's mesh was generated.
20
2. D E S I G N A N D IMPLEMENTATION
Figure 2.11: Highlighted node that was selected i n grammar object's
parse tree i n it's inspector.
// Attaches a new game object to nodes called Spawn
var nodes = grammarObject.ParseTree
. Data
.Values
.Where(n => n.Name == "Spawn");
f o r e a c h ( v a r node i n nodes) {
grammarObj ect.AttachGameObj e c t T o N o d e ( n o d e . I d ) ;
}
Listing 2.9: Example of attaching empty game objects to all nodes of
object's parse tree that are called " S p a w n " .
2.4.6 Attaching Unity GameObjects
Through the parse tree, nodes can be accessed and used to place Unity
objects to otherwise static mesh generated by Michelagenlo. This is
useful for a d d i n g game logic or any U n i t y specific scripts or logic
to integrate this mesh into Unity. For example attach U n i t y Lights
to relevant nodes or a d d custom scripts such as spawners as seen i n
Listing 2.9. To search i n the parse tree, computed property Name can
be used to get the exact rule that the node was generated from, or
Ontology that contains w h o l e path of h o w the node was generated
can be used.
21
2. D E S I G N A N D IMPLEMENTATION
C o m p i l a t i o n o u t p u t C l e a r
S u g g e s t i o n C o p y
C o p y will b e s o o n d e p r e c a t e d . P l e a s e u s e A t t a c h n s t e a d .
Figure 2.12: Compilation output view inside object's inspector.
1 grammarObject.Generate()
2 .Then(response => D e b u g . L o g ( r e s p o n s e . E r r o r M e s s a g e ) )
3 . C a t c h ( e r r o r => D e b u g . L o g ( e r r o r . M e s s a g e ) )
Listing 2.10: Reading of compilation output from script.
2.4.7 Compilation output
It is always vital to provide as m u c h information as possible to the
users, and since the compilation of Michelangelo grammars is executed
on their servers, this information is not directly accessible to the users.
After both successful and unsuccessful generation request, a message
is sent under the E r r o r s key with the result from Michelangelo's A P I .
The compilation output containing any errors, warnings, tips or other
notes can be accessed from C# scripting interface as seen i n Listing
2.10 i n its raw string form. In the Grammar/Scene object's inspector,
it is better formatted to provide simple to read the output as can be
seen i n Figure 2.12.
In scripting interface, w h e n generation request is rejected, e r r
o r . Message can also contain other error messages, for example, H T M L
request errors from Michelangelo A P I , or about object not being ready
to send for the generation (see generation requirements i n chapter
2.4).
2.5 Communication with Michelangelo service
To connect to Michelangelo A P I , web requests are used, various endpoints
require various H T T P methods and also expect and respond
with multiple data types. Luckily, Unity already contains its class Unity
WebRequest that provides this basic functionality for both Editor
and Runtime.
22
2. D E S I G N A N D IMPLEMENTATION
p u b l i c s t a t i c UnityWebRequest N o R e d i r e c t (
t h i s UnityWebRequest r e q u e s t
) {
r e q u e s t . r e d i r e c t L i m i t = 0;
r e t u r n r e q u e s t ;
}
p u b l i c s t a t i c UnityWebRequest W i t h C o o k i e s (
t h i s UnityWebRequest r e q u e s t , s t r i n g c o o k i e s
) {
r e q u e s t . S e t R e q u e s t H e a d e r ( " C o o k i e
r e t u r n r e q u e s t ;
c o o k i e s ) ;
}
Listing 2.11: Extension methods for UnityWebRequest.
In code, its static constructor methods for specific H T T P methods
are used w i t h a few custom helper methods as seen i n Listing 2.11,
that are used to make the initialization cleaner.
Sending the request can be done both synchronously and pseudo
asynchronously. Synchronously by checking its done status after it
was sent. The asynchronous option uses Unity's Coroutines [13] to
not block the script execution while the request response is pending.
Unlike C#'s async/await system that returns from execution on await
keyword and then continues execution from different thread after the
async operation is finished, Unity's Coroutines are running on single
thread and use IEnumerator return type w i t h yield return statements
to return f r o m execution. Thanks to the behavior of IEnumerator,
Unity's main logic loop then each frame resumes the execution of the
Coroutine similarly to h o w execution of await operation resumes.
One of the negatives of the Coroutine system i n Unity is that these
functions cannot return any values, which for many of its use cases is
not a problem, but for executing web requests make the code more
complicated. Another problem that is mainly connected to code quality
and readability is that the C#'s enumerators are designed to be used
w i t h data structures that are meant to be iterated over and not for
splitting u p the execution of long-running task between multiple
frames. This makes it harder to w o r k w i t h Coroutines i n a similar
23
2. D E S I G N A N D IMPLEMENTATION
way to h o w asynchronous code is written i n other frameworks and
languages, w i t h already set best practices and code style.
Previously mentioned problems w i t h Unity's Coroutines system
are the reasons w h y the RSG.Promise library was used. It is based on
the Promises paradigm and aims to make it easier for Unity developers
to work w i t h asynchronous code as is described i n detail i n an article
from Ashley Davis [14]. By including this library, the UnityWebRequest
can be wrapped inside a promise that can:
1. Return a value.
2. Provide info about whether it was successful or not.
3. Be partially handled and chained w i t h other promises.
4. Provide additional options w i t h Done, F i n a l l y , ThenAll and
other promise chained methods.
W i t h UnityWebRequest and Promise library, all of the requests to
Michelangelo A P I follow the pattern shown i n Listing 2.12. Since the
scripting A P I of the package is supposed to mainly through the static
class MichelangeloSession and its method that then calls the WebAPI
methods, all WebAPI methods are internal. The Request method from
the example can then be provided w i t h arguments and it creates a new
Promise instance that passes it's r e s o l v e and r e j e c t callbacks down
to the private RequestCoroutine coroutine that is ran on M i c h e l a n g
e l o S i n g l e t o n object. Since coroutines need to r u n on active U n i t y
GameObject, the M i c h e l a n g e l o S i n g l e t o n is an automatically created
hidden object, that runs these coroutines. Inside the private method
then a UnityWebRequest is created using one of its helper static methods
for constructing a request for each type of H T M L methods and
providing it correct arguments. Then by yield returning the SendWebRequest
method, execution waits until a response is returned from
the server. The helper method is then used that logs the response
info, and if the response was not successful, another helper method
creates an appropriate exception based on whether it was H T T P error
or not and passes it to r e j e c t callback. Otherwise, after the response
is processed, the return value is passed to r e s o l v e callback.
A s mentioned above, all WebAPI methods are internal and called by
public MichelangeloSession methods. A s seen i n Listing 2.13, both
24
2. D E S I G N A N D IMPLEMENTATION
i n t e r n a l s t a t i c IPromise R e q u e s t ( s t r i n g arg) =>
// Create new promise
new P r o m i s e ( ( r e s o l v e , r e j e c t ) =>
// Wrap the coroutine
M i c h e l a n g e l o S i n g l e t o n . C o r o u t i n e (
R e q u e s t C o r o u t i n e (
arg, W r a p ( r e s o l v e ) , W r a p ( r e j e c t )
)
)
);
p r i v a t e s t a t i c IEnumerator R e q u e s t C o r o u t i n e (
s t r i n g a r g ,
Action<TData> r e s o l v e ,
A c t i o n < E x c e p t i o n > r e j e c t
) {
// Set s t a t i c isLoading f l a g
I s L o a d i n g = t r u e ;
// Reset s t a t i c isUnreachable f l a g
I s U n r e a c h a b l e = f a l s e ;
// Create a get request
u s i n g (var getRequest = U n i t y W e b R e q u e s t . G e t ( u r l ) ) {
// Send the request and wait for response
y i e l d r e t u r n getRequest.SendWebRequest();
// Check errors and
// log i n f o about reponse
i f ( C h e c k A n d L o g E r r o r ( g e t R e q u e s t ) ) {
r e j e c t ( G e n e r a t e E x c e p t i o n (
"Request e r r o r : \ n " , getRequest
) ) ;
y i e l d break;
>
/* Handle response */
r e s o l v e ( d a t a ) ;
}
>
Listing 2.12: Simplified structure of W e b A P I method that sends a get
request to Michelangelo A P I .
25
2. D E S I G N A N D IMPLEMENTATION
p u b l i c s t a t i c IPromise<Data> R e q u e s t ( s t r i n g arg) =>
// Call internal WebAPI method,
// access the returned data or exception
// and return the IPromise for user to use
WebAPI.Request(arg)
.Then(data => {
/* data passed to resolve */
r e t u r n d a t a ;
})
.Catch(exc => {
/* exception passed to reject */
r e t u r n exc;
>);
Listing 2.13: Example of MichelangeloSession method w r a p p i n g a
WebAPI.
successful and unsuccessful results can be accessed, and then the
IPromise is returned for the user to use.
2.6 Serialization
2.6.1 Serialization of Unity scripts
Serialization is an important part of every Unity project. A s mentioned
by Lucas Meijer i n his article [15] p r o v i d i n g more insight into h o w
it works i n Unity, serialization is connected to almost all parts of development
i n Unity, whether it is w o r k i n g i n editor, loading game
data d u r i n g play or saving game objects into Prefabs w h i c h are just
a Y A M L representations of serialized C# data. B y default U n i t y can
serialize C#'s primitive types such as s t r i n g or double and a one
dimensional arrays e.g. i n t []. Then it also can serialize some of its
types like for example Vector3 or Quaternion. Other classes are serializable
only if marked w i t h S e r i a l i z a b l e A t t r i b u t e and their public
attributes (with some exceptions) are serializable as well. In order
to serialize other classes, that class needs to implement I S e r i a l i z a t
i o n C a l l b a c k R e c e i v e r w i t h custom serialization and deserialization
methods.
26
2. D E S I G N A N D IMPLEMENTATION
The first problem encountered w i t h the serialization came w i t h the
serialization of ParseTree. It w o u l d have been easier to denormalize
the parse tree into a real tree data structure after receiving it f r o m
Michelangelo A P I . However, since the parse tree needs to be serialized
in order to be preserved on its scene object, Unity's restrictions d i d not
allow this option. Since Unity's serialization system cannot serialize
n u l l values and needs to instantiate every attribute i n order for the
memory layout of the class to be correct, classes that nest attributes that
are of the same type as its type w o u l d create an infinite loop. Because of
this, an arbitrary nesting limit was set to 7 levels, which disallows a tree
structure such as parse tree to be serialized natively. Keeping the data
structure as it is received from the Michelangelo A P I was not an option
either since Unity cannot serialize dictionaries as well. Serialization of
ParseTree class is then achieved by w r a p p i n g its data inside a class
that implements I S e r i a l i z a t i o n C a l l b a c k R e c e i v e r w h i c h serializes
the dictionary into a private list attribute and then deserializes it back
from the list into the dictionary.
After implementing the custom serialization callbacks, one problem
remained with the serialization of the parse tree. Since Unity uses
serialization for reading object data from its inspector, the OnBef o r e
S e r i a l i z e callback is called repeatedly, while the object is selected
and its inspector is shown. For larger parse trees, this caused severe
lag of the whole editor even w i t h the serializedValues. Count ==
Data. Count guard i n serialization callback. Since no solution to this
problem was available, a workaround was implemented that somehow
fixes the issue.
Since the editor slowdown occurs only when an object that was serializing
parse tree was selected, and its inspector was being drawn, the
parse tree was moved to a ParseTreeData component. The ParseTreeData
component is required by the object component w i t h RequireComponentAttribute
and is completely h i d d e n f r o m the inspector.
Reference to the ParseTreeData is privately saved inside the object
and a public ParseTree getter is p r o v i d e d w h i c h returns the parse
tree serialized i n parse tree data script.
27
2. D E S I G N A N D IMPLEMENTATION
p u b l i c c l a s s P a r s e T r e e : I S e r i a l i z a t i o n C a l l b a c k R e c e i v e r {
[ S e r i a l i z e F i e l d ]
p r i v a t e L i s t < P a r s e T r e e N o d e > s e r i a l i z e d V a l u e s ;
p u b l i c D i c t i o n a r y < u i n t , ParseTreeNode> Data;
p u b l i c v o i d O n B e f o r e S e r i a l i z e ( ) {
i f (Data == n u l l
II s e r i a l i z e d V a l u e s . C o u n t == Data.Count) {
r e t u r n ;
}
s e r i a l i z e d V a l u e s . C l e a r ( ) ;
f o r e a c h (var p a i r i n Data) {
s e r i a l i z e d V a l u e s . A d d ( p a i r . V a l u e ) ;
}
}
p u b l i c v o i d O n A f t e r D e s e r i a l i z e ( ) {
Data = new D i c t i o n a r y < u i n t , ParseTreeNode>();
f o r e a c h (var model i n s e r i a l i z e d V a l u e s ) {
Data.Add(model.Id, model);
}
}
>
Listing 2.14: Serialization of ParseTree class.
28
2. D E S I G N A N D IMPLEMENTATION
2.6.2 MessagePack
Another form of serialization used i n this project is the MessagePack
serialization, w h i c h is used for a response of some of Michelangelo's
A P I endpoints. Since generation result can often contain hundreds
of vertices and huge parse trees w i t h many derivation steps, J S O N
serialization that is normally used for A P I responses was not suitable
for these endpoints. M a i n problems were the speed of serializing the
data on the . N E T backend of Michelangelo and then the total size
of a J S O N that is sent i n a string format. That is w h y MessagePack
serialization was chosen as a faster and more lightweight option for
these big response data sets.
A Messagepack serializer therefore needed to be implemented i n
this package as well. For that same implementation as on backed has
been chosen that provides the fastest serialization and deserialization
according to their performance comparisons w i t h other . N E T implementations
of MessagePack serializers [16]. Since some of Unity's targets
do not support A O T (Ahead Of Time) compilation that is required
to generate object formatter dynamically, the package comes with pregenerated
classes that are saved i n MessagePackGenerated. cs script
file.
2.7 Mesh optimization
Properly optimizing meshes and the 3 D scenes as a w h o l e for realtime
rendering applications which games are is a crucial part of their
development. Since Michelangelo only provides raw mesh data w i t h
possibly unlimited scale, this data needs to be processed appropriately
in order to be usable i n Unity.
In earlier implementations of the package, instead of using meshes
from the nodes of the generation parse tree, mesh data was gathered
from Objects key of generation response. Objects attribute contains
an array of GeometriModel objects and contains all the mesh data that
should be visible i n the final generated model (generally the leaves of
the parse tree). For simple use cases, this approach works correctly,
and the mesh building is fast since there is no need to iterate through
the parse tree. This earlier implementation created a new GameObj ect
with it's o w n Mesh component attached for each GeometricModel from
29
2. D E S I G N A N D IMPLEMENTATION
S Scene C Game flflsset Store NuGet
Figure 2.13: Highlights of individual floors of this generated house.
response. That approach, though, was not viable for larger meshes
that can contain thousands of GeometricModel objects, which w i l l , i n
turn, create thousands of game objects which is not optimal to render
single mesh.
Next step was to combine the partial meshes f r o m Objects attribute
into one and provide it to the single Mesh component attached
to the grammar object script itself. A g a i n for smaller meshes, this
worked even better than the previous implementation, since there
no longer were redundant game objects, but the Unity's vertex count
limit of 65534 vertices per one Mesh component made this option not
viable for larger meshes.
While Objects attribute from response contained all the mandatory
mesh data to render the object, it d i d not contain information
on h o w to properly divide it. Required information can be obtained
from the parse tree though, since it contains the whole ontology of
h o w the object was built f r o m a single large target split into smaller
and smaller parts. W i t h the assumption that children of each node
are generally contained inside the parent's bounding mesh, the whole
object can be semantically split into multiple smaller objects that are
all spatially near each other.
B y splitting these meshes based on the parse tree, the resulting
child meshes then can be used i n further optimizations. Even without
any other modifications, by having multiple MeshRenderer compo-
30
2. D E S I G N A N D IMPLEMENTATION
nents, U n i t y automatically culls meshes that are not visible i n the
camera's frustum. If the Objects meshes were used and split arbitrarily
between MeshRenderers to fit into Unity's mesh vertex count limit,
there w o u l d be no guarantee that the mesh w o u l d not contain random
parts of the object that w o u l d prevent it from being culled no matter
w h i c h part of the object camera was looking at. A s s h o w n i n Figure
2.13, floors of this house belong to different parts of the parse tree and
thus can be culled if, e.g., the camera was placed inside the house on
one of the floors.
31
3 Future work
Michelangelo is still i n active development, and many features are
not fully implemented as of the writing of this thesis. That means
that the package discussed i n this thesis w i l l as well be developed
and updated alongside the Michelangelo framework. Here are few
features that are currently planned either for Michelangelo service or
this Unity implementation specifically.
3.1 Advanced materials and procedural shaders
To texture a procedural mesh, often a procedural solution is most useful.
A s of now, the material system i n Michelangelo is based on Pixar's
Renderman shaders, w h i c h allows for usage of physically-based material
properties such as refraction or subsurface scattering. W h i l e
providing high-quality rendering output from Renderman, these materials
cannot be easily transferred to other rendering ecosystems such
as W e b G L i n Michelangelo's browser interface or Unity.
3.2 IntelliSense for grammar syntax
Since Michelangelo grammars are compiled and ran on the server,
users using the coding interface does not have direct feedback about
the code they are writing. IntelliSense is a c o m m o n feature of any
modern development software and greatly helps new developers to
get acquainted w i t h new frameworks faster and overall speeds u p
coding speed by providing autocompletion and static syntax checking
without the need to compile. In the case of Michelangelo, it is even
more important, since it is an extensive framework that does not rely
much on common C# classes. O n top of that, compilation takes more
time since it does run in the cloud and also costs server time, which is
in the interest of Michelangelo service to be avoided as well.
33
3. F U T U R E W O R K
3.3 LOD's and occlusion culling
O n top of the already implemented automatic mesh combining all
o w i n g for simple optimizations b y Unity, there are more possible
optimizations to be implemented. Level of detail (LoD) is often used
in games to save on resources by swapping high-detailed mesh w i t h
lesser detailed variants as it is further from the camera. Since creating
meshes for lower L o D levels need to follow some important rules, such
as retaining the recognizable silhouette, in order to mask the decrease
in detail, automatically generating L o D s from parse tree, similarly to
how meshes are split into nodes for frustum culling, w o u l d need to be
further thought out and won't be that straightforward to implement.
Another optimization feature of Unity is occlusion culling, w h i c h
on top of culling meshes outside of the camera frustum, culls out
meshes that are occluded by other meshes and therefore not visible.
It is based o n using two flags to objects containing a MeshRenderer
component. Those two flags are Occluder static and Occludee static.
By marking objects as Occluders, they w i l l be part of the occlusion
calculation. M a r k i n g objects as Occludees is used for transparent/translucent
or small object's w h i c h should still be i n the occlusion
calculation but w i l l not occlude any other objects. In order for occlusion
to w o r k efficiently, occlusion areas for the camera should be
set u p as well. A s w i t h L O D s , occlusion culling info cannot be reliably
automatically generated but is a n important feature for proper
optimization of Unity's game scenes.
34
Conclusion
Implementation of Michelangelo interface directly for Unity proved to
bring many possible improvements over preceding browser interface.
A l l the agreed on functionality was implemented, and d u r i n g the
development, multiple additional ideas were discussed that either are
already implemented, like the simplified SceneObjects or are planned
for future, like the IntelliSense, that w i l l also be coming to the browser
interface. Since Michelangelo is still being actively developed and is
evolving and improving, refactoring o l d parts of its architecture, the
package similarly needed to undergo few refactorings, such as the
implementation of the MessagePack serializer or complete rework
of generation response format has been made d u r i n g development.
Because of that, the Unity package now contains all the base functionality,
it can be presented alongside the original browser interface as
a valid option to interact w i t h Michelangelo service, and it offers its
unique features.
35
Bibliography
1. Michelangelo [online]. 2018 [visited o n 2018-11-15]. Available from:
https://michelangelo.graphics/.
2. P R U S I N K I E W I C Z , P.; L I N D E N M A Y E R , Aristid. The Algorithmic Beauty
of Plants. Berlin, Heidelberg: Springer-Verlag, 1990. ISBN 0-387-97297-8.
3. PARBERRY, Ian. Designer Worlds: Procedural Generation of Infinite Terrain
from USGS Elevation Data. 2013. Technical report. Laboratory for
Recreational Computing, Dept. of Computer Science & Engineering,
Univ. of N o r t h Texas.
4. P A R I S H , Yoav I H ; MÜLLER, Pascal. Procedural modeling of cities.
In: Proceedings of the 28th annual conference on Computer graphics and
interactive techniques. 2001, pp. 301-308.
5. S C H W A R Z , Michael; MÜLLER, Pascal. Advanced Procedural M o d e l ing
of Architecture. ACM Transactions on Graphics. 2015, vol. 34, no.
4 (Proceedings of S I G G R A P H 2015), pp. 107:1-107:12.
6. STINY, G . Spatial Relations and Grammars. Environment and Planning
B: Planning and Design. 1982, vol. 9, no. 1, p p . 113-114. Available
from D O I : 10.1068/b090113.
7. MÜLLER, Pascal; W O N K A , Peter; H A E G L E R , Simon; U L M E R , A n dreas;
V A N G O O L , Luc. Procedural M o d e l i n g of Buildings. ACM
Trans. Graph. 2006, vol. 25, no. 3, pp. 614-623. ISSN 0730-0301. Available
from DOI: 10.1145/1141911.1141931.
8. ILCIK, Martin; W I M M E R , Michael. Collaborative Modeling w i t h Symbolic
Shape Grammars. In: Proceedings of eCAADe 2016. O u l u , Finland,
2016, p p . 417^126. Available also from: https : / / www . eg .
tuwien . ac . at / research / publications / 2016 / i l e i k - 2016 -
cmssg/.
9. T E C H N O L O G I E S , Unity. EditorPrefs [online]. 2019 [visited on 2019-05-15].
Available from: https : //docs . unity3d . com/ScriptRef erence/
EditorPrefs.html.
10. T E C H N O L O G I E S , Unity. Meshes [online]. 2019 [visited on 2019-05-15].
Available from: https : //docs . unity3d. com/Manual/class-Mesh.
html.
37
B I B L I O G R A P H Y
11. T E C H N O L O G I E S , Unity. Creating and Using Materials [online]. 2019
[visited on 2019-05-15]. Available from: https : / /docs . u n i t y 3 d .
com/Manual/Materials.html.
12. T E C H N O L O G I E S , Unity. Standard Shader [online]. 2019 [visited o n
2019-05-15]. Available from: https : / / d o c s . u n i t y 3 d . com/Manual/
shader-StandardShader.html.
13. T E C H N O L O G I E S , U n i t y . Coroutines [online]. 2019 [visitedon2019-05-15].
Available from: https : / / d o c s . u n i t y 3 d . com/Manual/Coroutines .
html.
14. DAVIS, Ashley. Promises for Game Development [online]. 2015 [visited on
2019-05-15]. Available from: h t t p : //www. w h a t - c o u l d - p o s s i b l y -
go-wrong.com/promises-for-game-development/.
15. MEIJER, Lucas. Serialization in Unity [online]. 2014 [visited on2019-05-15].
Available from: https : / / blogs . u n i t y 3 d . com / 2014 / 06 / 24 /
s e r i a l i z a t i o n - i n - u n i t y / .
16. KAWAI,Yoshifumi.MessagePackforC# [online]. 2017 [visited on2019-05-15].
Available from: h t t p s : / / g i t h u b . com/neuecc/MessagePack-CSharp.
38