string-based synthesis of structured shapes · modeling—geometric algorithms, languages, and...

EUROGRAPHICS 2019 / P. Alliez and F. Pellacini(Guest Editors)

Volume 38 (2019), Number 2

String-Based Synthesis of Structured Shapes

Javor Kalojanov1 Isaak Lim1 Niloy Mitra2 Leif Kobbelt1

1 Visual Computing Institute, RWTH Aachen University 2 UCL London

Abstract

We propose a novel method to synthesize geometric models from a given class of context-aware structured shapes such as build-

ings and other man-made objects. The central idea is to leverage powerful machine learning methods from the area of natural

language processing for this task. To this end, we propose a technique that maps shapes to strings and vice versa, through an

intermediate shape graph representation. We then convert procedurally generated shape repositories into text databases that,

in turn, can be used to train a variational autoencoder. The autoencoder enables higher level shape manipulation and synthesis

like, for example, interpolation and sampling via its continuous latent space. We provide project code and pre-trained models.

Categories and Subject Descriptors (according to ACM CCS): I.3.5 [Computer Graphics]: Computational Geometry and ObjectModeling—Geometric algorithms, languages, and systems

1. Introduction

Automatic 3D model synthesis is one of the most challengingproblems in computer graphics. The standard approach to cre-ating virtual 3D shapes involves the use of complex modelingsoftware by skilled artists and can be time consuming and ex-pensive. Recent data-driven methods for shape synthesis suchas [LXC∗17, SSK∗17] address this problem by training neural net-works to perform tasks like shape assembly and interpolation. Sim-ilar to how humans approach 3D modeling, these methods considershapes as being composed out of building blocks. An importantlimitation of previous approaches, is the limited structural com-plexity of the produced models. This problem can be attributed tothe types of models available in shape repositories, but the moresignificant hurdle is the inability of neural networks to work on ir-regularly structured input. In this work we propose a combinationof a model-based and a data-driven method that addresses both cre-ating large number of structured shapes of arbitrary complexity andrepresenting the topology of structure graphs via strings.

In order to address the problem of creating structural variationsin a principled way, we consider shapes that can be assembledfrom building blocks according to an initially unknown tiling shapegrammar: a set of rules describing how to assemble the elementarypieces locally. To this end, we use as an input a small number ofshapes segmented into rigid, exchangeable parts (building blocks).These define the set of plausible shape variants: each local configu-ration of components is considered valid only if the same or a verysimilar configuration also occurred in the input models.

However, unlike context-free shape grammars, the grammar as-sembly rules here are not sufficient to derive a recipe for creatingshape variants. The difference is that context-free production rules

CDA0(DC)DCDBDCDCD0A0(DC)(DC)DCDBDCDCD0A0(DC)(DCDBDCDCD0)DCB(DCDCDA0CD)DCDCDC0B(DCDCDC0)DCDCDA0CD

input shape shape graph

variant variant invalid variant

string encodings

AB

C

D

Figure 1: A simple shape, its shape graph, string representations of

the graph, and shape variants. Both mappings from graph topology

to strings, and back to geometric shapes are not uniquely defined.

are hierarchical and each sequence of applications can be termi-nated resulting in a globally valid result. Tiling grammar rules de-scribe only local configurations without termination or correctnessguarantees. For example, the last shape in Figure 1 contains an errorthat cannot be resolved by modifying only the pieces of the modelthat cause the invalid intersection. Hence, with growing amount ofcomponents, it quickly becomes impossible for probabilistic pro-cedural modeling methods to generate shapes without violating thetiling grammar. This makes the problem very difficult, but also in-teresting, since the resulting shapes can contain cycles and regulargrid patterns that appear very often in man made objects.

In this work we tackle the problem of 3D modeling using ma-chine learning methods for natural language processing: a do-main that also deals with non-context free constructions. To thisend, we adapt the simplified molecular-input line-entry system

c© 2019 The Author(s)Computer Graphics Forum c© 2019 The Eurographics Association and JohnWiley & Sons Ltd. Published by John Wiley & Sons Ltd.

J. Kalojanov & I. Lim & N. Mitra & L. Kobbelt / String-Based Synthesis of Structured Shapes

(SMILES) [Wei88] for encoding the topology of graphs represent-ing shapes constructed out of building blocks (see Figure 1). Thisstring conversion method enables the application of recurrent neu-ral networks for learning the structure of geometric models.

We demonstrate that it is possible to train a variational autoen-coder (VAE) [KW13] on strings representing the structure of therandomly generated 3D models similar to [BVV∗15,GDH∗16] andgenerate geometric shapes constructed out of rigid building blocks.The variational autoencoder is used to map vectors representingshapes to a continuous latent space and decode points in latentspace to shape representations. This facilitates high level synthe-sis operation such as interpolation and sampling of discrete shapes.

Synthesizing geometric shapes from a given graph topology pro-duced in the previous step is also very hard even if there is a uniqueway to attach pairs of pieces together [SSK∗17], which is not thecase here. We address the problem by reducing it to a classificationtask and train a neural network to estimate how the parts (or graphnodes) are positioned relative to each other in space.

Since existing shape collections usually consist of samples withsmall or no structural variations, we first have to address the prob-lem of efficiently creating large sets of structured shapes. We ex-tend and automate the procedural modelling method based on par-tial graph symmetries [BWS10, LVW∗15] and use it for creatingsets of samples sufficiently large to enable training machine learn-ing models. As a result, we present a fully automatic shape syn-thesis framework that can generate large collections of shapes withsignificant structural variations using just one or two example mod-els as input.

This paper contains three important contributions (see Figure 2):

• A model-based, automatic method for generating structuredshape variations given just one or a few examples.

• Adapting a data-driven method from natural language process-ing, for procedural modeling of 3D geometry.

• A data-driven method for instantiation of shape graph topologiesin R

3 via edge classification.

input model

shape graphgraph

variations

string variations

VAE

output models

graphvariations

geometry

structure

text string variations

3

4

55.1 5.1

6

Figure 2: Overview: We convert 3D models into graphs and pro-

cedurally create structural variations. These are converted to text

databases and used to train a variational autoencoder (VAE).

Finally, the training and additional strings representing graph

topologies are instantiated back into 3D models. The number next

to each arrow indicates the corresponding section in the paper.

2. Related Work

Part-based Modeling Funkhouser et al. [FKS∗04] propose a data-driven shape synthesis method in which a user can cut out parts of

models in an existing shape database and then use them to com-pose new shapes. Subsequent works expand on various aspects ofthe method, e.g., using 2D sketches [LF08] for shape retrieval. Fur-ther part-based modeling approaches have been recently surveyedin [MWZ∗13].

Several recent related works on procedural modeling [TLL∗11,TYK∗12, RMGH15, RJT18] consider hierarchical segmentationsand create further shapes using context-free shape grammars. Theseworks address the problem of generating desirable variants viarandom applications of grammar rules and provide better meansfor user control compared to our method. In contrast, the non-context-free shape grammars we consider here have higher expres-sive power and are easier to compute automatically. Our methoddoes not require the shape grammar to be provided as input, whichis beneficial for users without 3D modeling skills.

Our method is closely related to inverse procedural model-ing techniques for generating variations from a single examplelike the works by Bokeloh et al. [BWS10, BWKS11] and Liuet al. [LVW∗15]. These methods consider shape decompositionsinto building blocks that can be assembled into variants. The seg-mentations are either computed via symmetry detection [BWS10,BWKS11], or given as input [LVW∗15].

Similar to Liu et al. [LVW∗15] we use partial graph symmetriesto generate random variations by splitting and merging examplepart compositions. However, as detailed in Section 4, we proposean alternative sampling algorithm for faster, automatic and morerobust sampling of large amounts of shape variations.

Data-Driven Shape Processing Xu et al. [XKHK17] survey re-cent data-driven methods for shape processing including part basedmodeling [FKS∗04, KJS07], sketch-based modeling [FWX∗13,XXM∗13] and shape editing [XZZ∗11, ZCOM13]. Similar to us,these methods attempt to simplify 3D modeling by leveraging exist-ing data. The differences are that we generate the example datasetsfrom a single or several examples.

Recently, Nash and Williams [NW17] also proposed the use ofa variational autoencoder (VAE) for shape synthesis. Their workhowever differs significantly from our method since they train aVAE on a set of shapes with the same structure. Other related deepgenerative models for structure variations consider tree [LXC∗17]or graph [SSK∗17] representations of comparably simpler objectslike chairs and vehicles. The main disadvantage of these relatedmethods compared to ours is the constrained structural complexityof the generated shapes which limits them to creating style varia-tions of the training examples. On the other hand, our method en-ables synthesis of shapes with arbitrary topology thanks to vector-izing shape graphs using sequences.

3. Tiling Grammars and Structure Graphs

In the absence of predefined rules for procedural modeling, we haveto find a way to characterize the space of valid shape variations. Asound way to address this challenge is to use the input models asexamples for a set of valid assembly rules and only use these forshape synthesis. In other words, as a grammar we use the subset ofthe assembly rules already observed in the input models.

c© 2019 The Author(s)Computer Graphics Forum c© 2019 The Eurographics Association and John Wiley & Sons Ltd.


S1 random subgraph in G1 matching cuts in G2 random variationS2

Figure 3: Graph-based shape sampling. The input models S1,S2 are converted into shape graphs G1,G2. New shape variations are created

automatically via random subgraph sampling and subgraph matching.

More specifically, each part of a certain type in the output shapehas to have a matching number and types of neighbors. Pieces areconsidered as neighbors if and only if their surfaces intersect. Thisassumption is quite general and does not impose restrictions on theshape of the parts or their intersections.

For example, the grammar rules derived from the models in Fig-ure 3 allow each of the green, purple, cyan building blocks to onlyattach to brown pieces and to have exactly one (green and cyan) orthree (purple) neighbors. The brown building blocks have exactlytwo neighbors and can attach to pieces of every type.

We represent the structure of each shape via a undirected shape

graph. The nodes of the graph correspond to parts. Every pair ofconnected pieces is represented with an edge between their corre-sponding nodes. We transfer the part labels to the graph by assign-ing a type to each graph node. The shape graph abstracts away thegeometry information and can be used to reduce the shape synthesisproblem to graph sampling and subgraph matching.

We extract these grammar rules regarding graph topology (num-ber and types of neighbors for a given node type) from the shapegraphs of the input models and use them to detect variations withinvalid graphs. The same set of rules can be used to check for selfintersecting assemblies, by verifying the shape graph from the ge-ometric shape after creating it. Later on, the example models areused to derive a set of allowed approximate positions and orienta-tions of connected pairs of pieces of certain types. Therefore, theefficiency of our shape synthesis approach depends on the ability toquickly compute the shape graph of input models of arbitrary com-plexity. We solve this using collision detection and by assumingthat a pair of parts is connected if and only if their correspondingtriangle meshes intersect each other.

4. Graph-Based Shape Sampling

Assuming a pair of input models S1,S2, our shape augmentationmethod consists of the following steps (also see Figure 3). We com-pute graphs G1,G2 for each shape using collision detection. Werandomly sample subgraphs in the first shape (S1) and check if asubgraph with matching boundary exists in the second shape (S2).We merge each matching pairs of subgraphs and check the resultingmodel against the grammar rules extracted from the shape graphsG1,G2. We output valid shape variations with different node typehistograms to avoid re-discovering the same shape multiple times.

Subgraph Sampling In essence, our shape generation approachsplits each of the input models into two or more pieces that

are recombined into a new shape. The main difficulty is to effi-ciently discover compatible sub-graphs and avoid duplicates. Liuet al. [LVW∗15] propose to enumerate all possible graph opera-tions in O(n3) worst case complexity and O(n2) (quadratic in thenumber of graph nodes) expected complexity. Instead of exhaus-tively searching for all possible operations we sample p randomsubgraphs in parallel and keep p linear in the number of availableprocessors (or threads). Doing so, we spend significantly less ef-fort sampling the shape variants created by merging exactly twosubgraphs of the input shapes. By recursively applying this proce-dure on newly created variations we can sample shapes made outof more than two subgraphs from the input.

Subgraph Matching Each subgraph sample s from the first shapegraph G1, is matched against the the second shape graph G2. Wesearch for a set of nodes in G2 corresponding to the nodes in G1connected via boundary edges of s. Both the node types as wellas all pairwise distances between the centroids of the respectiveparts have to match. The latter is a necessary condition for the exis-tence of a rigid transformation T that attaches (docks) the geometryrepresented by s to S2 without violating the grammar rules at theboundary [BWS10]. We randomize our search and terminate afterfinding the first set of corresponding boundary nodes in G2 in or-der to speed up the sampling process in the presence of multiplematches.

Docking Transformation Estimation Each subgraph with match-ing boundary needs to be transformed in order to attach it to S2.Aligning the centroids of the boundary pieces from S1 to theirmatching counterparts in S2 can be reduced to finding a transfor-mation that minimizes the distances between the two sets of pointsin the least square sense. The problem can be reduced to comput-ing an SVD (singular value decomposition) of a 3× 3 covariancematrix (see the supplemental notes to [SA07] for a detailed proof).

Validation After creating a variation by replacing a subgraph ofG2 with a subgraph of G1 we have to verify that the new shapedoes not violate the grammar rules. This can happen, for example,if some of the nodes in the interiors of the attached subgraphs inter-fere. Note that we need to test the geometric shape of the variant.The generated graph topology is always correct by construction:we only replace nodes on the boundary of the two subgraphs withnodes of the same type. In order to guarantee that the newly cre-ated shape is geometrically feasible, we generate it and computeits shape graph. Both interfering and misaligned parts will pro-



shape shape graphcycle edge

spanning tree edges

string representations variational autoencoder

encoder decoder

latent space

CDA(DC)(DC)(DCDCDCDBDADCDC0)DBDADCDCD0

D(CDADBDA(DC)(DC)(DC)DCDCDCDBDADCDC0)CD0

D(A(DC)(DC)(DC)DCDCDCDBDADCDC0)BDADCDCD0 D(A(DC). . .

D(CDA. . .

Figure 4: Example shape vectorization using SMILES strings. All cycles in the shape graph are broken by removing a random edge (lime

green). The remaining spanning tree (gray) is encoded in a string using letters for nodes and brackets to represent branching. The cycle

edges are reintroduced by adding a number next to their adjacent nodes in the string. The resulting strings are used to train a variational

autoencoder. We train on multiple shape variants composed out of the same building blocks.

duce node configurations that are not allowed by the tiling grammarcausing the shape to be discarded as invalid.

We avoid duplicating output models by conservatively discard-ing shapes with identical node type histograms. Removing dupli-cates is important if the input shapes have identical subgraphs, inwhich case multiple subgraph replacement operations yield copiesof the original models.

5. Variational Autoencoder for Tiling Grammars

The goal of the above shape sampling method is to automaticallygenerate large amounts of models with structural variations, anduse them as training sets for machine learning. Particularly in-teresting is the sequence modeling approach used in [GDH∗16,BVV∗15], which is based on sampling from a continuous structurerepresentation obtained from the latent space of a variational au-toencoder (VAE). In this work we generalize the method and applyit for synthesis of arbitrary geometric shapes constructed accordingto a tiling shape grammar.

The variational autoencoder [KW13] is a generative probabilis-tic model, which implicitly describes the joint probability distribu-tion over the dataset and corresponding latent variables p(x,z). Anencoder network is used to approximate p(z|x) with q(z|x) by map-ping data samples to distributions in latent space (typically gaus-sian). A decoder network models p(x|z) by mapping samples fromq in latent space to an output distribution (typically consisting ofcategorical distributions). Here, the input dataset is a collection ofsequences describing the shape graph topology of a set of geomet-ric models (see Figure 4). After training the network with a rep-resentative set of sample values one can use the decoding part onpoints of the latent space not present in the training set. This facili-tates operations like sampling, or interpolation of geometric shapes(molecule formulae or sentences in [GDH∗16, BVV∗15]).

5.1. Shape Graph Vectorization

In order to use a set of shapes as an input for a variational au-toencoder we need to first convert them to vectors. Since we areinterested in learning the structure of the shape, i.e., how to con-struct a model out of building blocks, we want to convert the shapegraph to a vector. SMILES strings [Wei88] are a standard methodfor molecule representation, which is based on enumerating nodes

as they appear in a spanning tree of the molecule graph. The conver-sion algorithm is general enough to be applied to arbitrary graphs,and we employ an analogous conversion method.

We assign a letter to each node type in the shape graph and com-pute a string by depth-first traversal of the graph. Multiple subtreeswith the same root are enclosed by brackets (except the last subtreeat each node). All remaining (cycle) edges in the graph are recordedvia number suffixes at both end-nodes. The resulting strings arethen trivially converted to one-hot vectors that can be used as inputto a sequence autoencoder (Figure 4).

Since we are interested in shape synthesis and do not have ac-cess to large bodies of training data, we prefer to compute multiplestrings for each shape and use all of them for learning. Intuitively,we try to train the autoencoder to distinguish strings that can rep-resent valid shapes from those that cannot, and accept the slightdrawback of representing the same valid example multiple times.After training, the decoding part of the neural network is not ableto covert all points in latent space into strings representing validshapes. Therefore, we have to verify decoded samples against thetiling grammar and discard invalid ones. This can be performed bythe same procedure used to verify shape graphs of randomly cre-ated variations.

5.2. Latent Space Sampling

Even though large regions of points in latent space do not decodeto valid strings (according to the grammar), it is possible to createshape variations by sampling and decoding random latent points.Our hypothesis is that valid shapes are mapped to a manifold by theencoding half of the autoencoder. If true, we are likely to find latentpoints corresponding to valid strings in the neighborhoods of theimages of training samples. We experimented with three samplingstrategies and observed results that support this assumption.

Point Perturbations We sample the neighborhood around a latentpoint pv corresponding to a valid training example with string sv.We add a random offset vector r1 to pv and decode the resultingpoint pr := pv + r1. Let pr decode to a string sr. If sr = sv or sr isnot valid according to the grammar rules, we re-sample pr using anew offset vector r2 with increased magnitude in each dimension.We repeat this procedure until we reach a maximum magnitude (weused 0.25) or find a valid string not equal to the known sv.



Linear Interpolation We interpolate points on the line segmentbetween the images of pairs of training samples. We sample pointson the line segment on equal intervals and sample their neighbor-hoods with the previous strategy, using smaller offset vector mag-nitudes. Usually this sampling strategy yielded valid strings closeto the endpoints of the line segments, which is in line with the hy-pothesis of valid latent point living close to a manifold in latentspace.

Latent Path Sampling The results of the previous samplingstrategies motivated us to further restrict sampling locations near“valid” points in latent space. Instead of sampling point on the linesegment between pairs of encoded training samples, we constructa path (of length 32) by repeatedly splitting the current segmentswith the training samples closest to both endpoints of each seg-ment. We then sample each path segment equidistantly and samplein the resulting neighborhoods analogous to the linear interpolationcase. Path sampling delivered more valid samples compared to lin-ear interpolation, which in turn was slightly more productive thansimply perturbing points.

5.3. Latent Space Structuring

The main motivation behind our learning method is the continu-ous shape representation provided by the variational autoencoder,which facilitates high-level shape synthesis operations such as sam-pling and interpolation. In this Section, we describe how to equipthe latent space with a metric that assigns a semantic or intuitivemeaning to geodesic distances on that (unknown) manifold of sam-ples decoding into valid strings. This can be achieved either by acontinuous deformation that brings samples corresponding to sim-ilar shapes closer together or by a discrete structure (graph) thatconnects samples of similar shapes.

We demonstrate an example by introducing a pairwise histogramsimilarity metric

E(x,y) =1

Nx +Ny∑

T∈Types

|Tx −Ty|,

where Nx is the number of building blocks in shape x, and Tx is thenumber of building blocks of type T in the shape.

To improve the efficiency of shape sampling, we implemented analternative method for latent space exploration based on to topolog-ical graph search. A conceptually similar approach has been pro-posed in [BYMW13]. Here, we compute a set of (random and/ortraining) points that decode to valid strings and organized them asnodes in a search graph. We connect each node with a user de-fined number of closest neighbors (w.r.t. Euclidean distance in la-tent space) and every other node in the graph that decodes to anequivalent string, i.e., a string representing the same graph topol-ogy. Because each shape is represented via multiple strings, we keptthe additional degree of the nodes low (between 3 and 5) in orderto avoid close to fully connected graphs. Using topological searchoperations on this graph is more efficient than blindly sampling inlatent space, and allows incorporating application dependent ob-jectives as edge weights. We then assign edge weights using thesimilarity metric, denoted as E.

This allows formulating interpolation between pairs of shapes as

the shortest path in the above graph. The resulting operation is moreefficient, because it does not depend on finding regions of the latentspace that decode to valid words. Furthermore, because of the sim-ilarity metric and the graph topology, the intermediate shapes onpaths in the graph are subjectively more intuitive interpolation re-sults than the shapes we encountered by sampling on line segmentsin latent space.

We also experimented with a triplet contrastive loss [CSSB10,SSK∗17], which can improve sampling results by grouping the em-bedding of similar shapes in latent space. However, in our experi-ments, the additional loss slowed down training without noticeablyimproving the point distribution in latent space. We therefore reportresults only with a standard VAE loss function.

6. Instantiation of Structure Graphs

The shape synthesis approach we discussed so far consists of twoparts. We first use an inverse procedural modeling method to gener-ate a set of models using partial graph symmetries. Then, we vector-ize the shape graphs of the sample models using SMILES strings,which are subsequently used as a training set for a variational au-toencoder. Sampling from the latent space of the autoencoder gen-erates string representation of further shape graphs with unknowninstantiation in Euclidean space.

In the following, we describe how we reduce instantiation ofstructured shape graphs to a classification problem. The main moti-vation behind our approach is to re-use the procedurally generatedshape database, and train a recurrent neural network to embed topo-logical graph representation back into 3D. The method consists oftwo main steps. We first estimate a set of possible edge categories

for each edge type. In other words, for each pair of parts a,b and oftypes A,B, we compute all possible ways to assemble them togetherand discretize them into a finite set of categories. (see Figure 5).Then, we train a recurrent neural network to estimate these discreteedge configurations from SMILES strings. The resulting sequencedefine a spatial embedding of the shape graph represented by thestring.

Local Coordinate Frame Estimation In order to compute al-lowed configurations of pairs of parts, we first need to estimate astandard local coordinate frame for each individual piece. A com-mon approach to address this problem is to perform PCA (Princi-pal Component Analysis [Pea01]) on the set of vertices and use theprincipal axes as a coordinate frame. However, this method is notreliable for parts with global symmetries because they do not havea canonical ordering of their principle components.

Instead of PCA, we use a similar approach that considers onlyextremal points which makes it robust to deformations on the inte-rior of parts. We compute a shape “diagonal” by finding the vertexv farthest away from the center of gravity and the vertex farthestaway from v. We then find the vertex with maximal distance to thediagonal. We average multiple vertices with the maximal distancesin any of the three searches if the result is not unique. The diagonaland the last vertex together with its projection onto the diagonal de-fine a local coordinate frame, which worked better than PCA axesin our experiments.



input model edge types edge categories

A D B D

C D D A

D B D C C D D A

6

6

2

4

2

2

relative position clusters

Figure 5: Data-driven estimation of edge configurations. From left to right: the input model, the allowed edge types according to the

extracted grammar, the part centroids for edges of type CD and DA, and all valid pairwise configurations for each edge type (the second

piece is replicated to illustrate possible local assemblies). We cluster relative positions (and orientations) for each valid pair of types to

estimate possible configurations. Then we train a recurrent neural network to assign configurations to graph edges and thereby embed shape

graphs in 3D.

Edge Category Estimation Using the canonical coordinateframes for each part type, we can compute a set of valid config-urations for pairs of adjacent parts. Each configuration representsa different way to attach (or dock) the two pieces to each other.We solve this by clustering relative part positions (and, if neces-sary orientations) for each edge type (see Figure 5) across the shapedatabase. We use Mean Shift Clustering [Che95], because the num-ber of clusters is not known in advance. Means Shift Clustering isa good fit for our data also because the sample points form well-separated clusters thanks to the precision we enforce when gen-erating the set of training shapes. Note that we estimate 3D edgeconfigurations, meaning that the method is not restricted to planarshape graphs.

Edge Category Mapping A problem we need to work around isthat the characters of SMILES strings represent node sequences in-stead of edges. However, the graph nodes appear in the string inthe same order they would appear in spanning tree of the graph.Therefore, it is possible to map a graph edge to each character inthe string representing a graph node: the edge between the nodeand its parent in the spanning tree. Additionally, numbers that rep-resent cycles also correspond directly to graph edges. We assign adummy category to each string character that does not represent agraph node. These include brackets and special characters used toseparate numbers superseding nodes adjacent to multiple cycles.

Edge Category Learning After constructing a mapping betweeneach SMILES string and a sequence of graph edge categories, wecan train a sequence to sequence model [SVL14] on the resultingpairs. Since our target sequences have the same length as the inputstrings, we can use a standard deep recurrent neural network con-sisting of an LSTM and a fully connected layer. We extended themodel with a masking layer that restricts prediction to only plausi-ble edge categories for the particular edge type. The possible cate-gories for each edge in a string depend only on the node types con-nected by the edge and can be computed for any shape graph (seeFigure 5). Therefore we multiply each output vector with a maskthat zeroes the predicted probabilities for types different than thetype of the current edge. In our experiments, introducing maskinglowered the initial prediction accuracy, but slightly slowed conver-

gence to the optimal set of parameters during training. On the otherhand, masking eliminates significant number of possible predictionerrors and improves the prediction accuracy on the test samples.

The availability of multiple SMILES-like representations of thesame graph plays to our advantage here, since it provides multi-ple training samples from each generated shape. Even though thegeneral problem of instantiating structure graphs is extremely hard,learning on the shape collections we generate via graph-based sam-pling resulted in very accurate predictions: the accuracy on the val-idation set of samples reached 90% and the model consistently pre-dict test sequences with only few mistakes (e.g. 6 wrong categoriesfor a sequence of length 85). We reduced the amount of errors byestimating categories for multiple SMILES strings representing thesame graph and selecting the most likely category for each edgeacross all of its occurrences in the sequences. As a result the trainedneural network can be used to provide an initial guess for the instan-tiation of strings generated by the VAE in Section 5.

Graph Instantiation Using the estimated edge categories, weconvert the graph topology to a geometric shape as follows. Westart at a random edge of the target graph and incrementally addnodes by transplanting random edges with matching categoriesfrom the (two) input shapes. Note that we need to consider the edgecategories in both edge directions for a correct instantiation. Weimprove the success rate of the algorithm by computing the currentshape subgraph and checking it for grammar violations or poorlyattached building blocks. If we detect a violation that cannot be re-paired by selecting a different edge with matching categories, weassume that the requested pair of edge categories is wrong and re-sort to a Monte-Carlo edge insertion step: We select a random edgefrom the input shapes with matching node types and resume theprevious construction algorithm if we find one that can be trans-planted without violating the grammar.

The success rate of our instantiation method depends on the qual-ity of estimated edge categories. We could not embed very largecycles without estimating correctly all (or all but one) configura-tions of participating edges. However, our method almost alwayssucceeds for tree-like shape graphs and shapes with smaller cycles,as discussed in the following section.



S2S1

churches

S1 S2

sand castles

brick buildings

playgrounds

S1

S2

S2

S1

moon bases

S2S1

S2S1

greek houses

Figure 6: Input shapes S1,S2 and some (not hand-picked) sampled variants for each test model. Note that we are able to synthesize shapes

containing cycles of building blocks, which is not possible when using context-free shape grammars. We maintain high quality by automati-

cally repairing shapes with poorly connected pieces and enforce diversity by discarding variants with repeated node histograms.

7. Evaluation

Implementation We implemented the shape synthesis algorithmsin the paper using C++ with Thrust [BH12], and Python. Pleasenote that we provide the code as supplemental material and willmake the implementation available as open source with the MITlicense.

Test Models We tested our implementation on 3D models down-loaded from online repositories or used in related work (the play-ground model) with node counts ranging from 23 to 70 and trianglecounts between 20000 and 400000. Where necessary, we split themodel into parts and assigned node types by coloring them beforecreating two example shapes by applying copy-paste operations,translations, and rotations of building blocks. We then iterativelyselect pairs of shapes, sample random subgraphs and attempt tocombine them into new, valid variations. At the end of each itera-tion we attempt to correct local part orientations of newly createdshapes and discard the model if the attempt fails, thereby main-taining reasonably high quality of sampled variations. We repeatthe procedure until we reach a sufficiently large number of stringsthat we use for training. Note, that with the exception of the greekhouses examples, we evaluate on examples with very restrictivegrammar rules, which makes it difficult to sample new variationsboth procedurally and later from the latent space of the autoen-coder.

Shape Interpolation See Figure 7 for an example shape interpola-tion result using the latent space of the trained autoencoder. Whileuseful for discovering additional shapes, this method is not guar-anteed to deliver desirable intermediate shapes, because distancesthe latent space of the autoencoder do not necessarily correspond

iteration: 1 2 3 4 final

size S1 strings strings strings strings stringsscene size S2 shapes shapes shapes shapes shapes

27K 169 864 4389 28596 138835church 45K 4 13 37 145 529

sand 23K 88 374 1333 3832 146594castle 13K 5 17 42 91 484

moon 243K 124 227 346 451 19299base 413K 5 7 7 10 68

play- 129K 220 629 1751 3257 557895ground 87K 6 12 17 18 137

brick 113K 233 699 1265 2693 181969buildings 103K 6 9 15 18 104

greek 28K 501 448639 – – 448639houses 37K 19 1084 – – 1084

Table 1: Input model sizes in the number of triangles and shape

collection growth with each iteration of sampling. The larger

amount of strings compared to shapes is due to the multiple pos-

sible string representations of the same graph. We did not generate

all possible strings for each shape.

to shape similarity. This can be addressed easily via constraintson latent space such as the construction detailed in Section 5.3.The graph topological search (see Figure 10) performed very wellon the majority of datasets and in particular on the playgrounds,greek houses and moon bases. We validated the embedding pro-vided by our variational autoencoder by constructing the search



B(AAAAB(AAAA0)AC)(AC)AAAAB(AAAAB0AAAD)AC

CAB(AAAAB(AC)AA0)AAAAB(AA1)AAAB(AAAAB(AAAB(AA0)AA1)

AAAAB(AC)AAAAB(AD)AC)AC

A(AAAB(AAAAB(AC)AAAAB(AD)AC)AD)B(AC)AC

A(AAAB(AAAAB(AC)AA0)AD)B(AA1)AAAB(AAAAB(AAAB(AA0)AA1)

AAAAB(AC)AAAAB(AD)AC)AC

A(B(AAAAB(AD)AAAAB(AAAA0)AC)AC)AAAB0AAAAB(AC)AAAAB(AD)AC

Figure 7: Latent path example: The models (left to right) and their

string encodings (top to bottom) are discovered on a random path

in latent space with the leftmost and rightmost shapes as endpoints.

graph naïvely from strings(see Table 2). Each point (string) is con-nected to its 3 nearest points (strings) and all equivalent strings.However, the simplified method produced graphs with multiple dis-connected components, and edge augmentation made interpolationpaths very long both topologically and according to E.

graph graph connected avg maxscene type size components weight weight

sand naïve 1000 102 4.71 12.17castle ours 1000 1 0.50 1.09

moon naïve 1000 30 2.29 5.50base ours 1000 1 0.39 0.55

greek naïve 1000 67 2.55 11.36houses ours 1000 1 0.31 0.49

Table 2: Ablation study comparing a naïve search graph construc-

tion using string similarity directly instead of distance in the au-

toencoder latent space (see Section 5.3). We sample shortest paths

between nodes using the string similarity metric E as edge weights

in both cases and report average and maximum (accumulated) edge

weights along these random path samples (lower is better).

Probabilistic Methods Shape modeling methods based on sub-sequet random applications of grammar rules to generate shapesare not well suited for the topological variations we consider here.We can demonstrate this by ignoring the suggested edge categoriesin the last step of our instantiation implementation, converting itto a naïve random shape assembly algorithm. In Figure 8 we cansee several failed attempts to compute a valid shape containinga long cycle. The probability of reproducing this particular cy-cle is approximately 1

514 because of the 6 possible edge config-urations for each of the 15 participating green (C) and red (A)tower pieces. Even a seemingly simple example such as the shapein Figure 9 is very unlikely to be constructed via random applica-tions of production rules, while we were able to embed the string

target string:CDA(DC)(DC)(DCDCDCDCDCDADADADADCD0)DCDCDCDC0

valid

Figure 8: Invalid MCMC attempts to embed a graph with a large

cycle in 3D. The only “valid” (according to the extracted grammar)

shape does not have the graph topology represented by the string.

The remaining shapes contain a wall piece (D) and/or a red tower

piece (A) with less than two neighbors.

from the first attempt. It should be noted, that related works (suchas [RJT18, RMGH15]) using context-free grammars, where thisproblem is not present, can provide improved probability at eachstep, however the exponential nature of the problem remains.

string: B(D)C(AC(BD)AC(AC0BD)BD)AC(AC(BD)A0)BD

training sample graph embedding

Figure 9: Example model, its string decoded form the VAE’s latent

space, and embedded in 3D via estimated edge configurations.

Compression The trained VAE model and instantiation RNN canbe used as a compressed representation of the shape database. Eachof the training shapes as well as additional samples generated vialatent space sampling can be stored as two sequences: the stringrepresenting the graph topology and the corresponding sequence ofedge categories suffice to reconstruct the shape exactly by “borrow-ing” geometry from the pair of input models.

Limitations Even though, this work makes a step towards solvingthe problem, both discovering and instantiation of strings repre-senting shapes with cycles, remains harder compared to context-free variations. This is most likely caused by the string conversionmethod, which is better suited for encoding sequences and worksaround branching and cycles via additional characters. An extremeexample presents itself in the brick buildings dataset, where we hadto discard the cycle edges and only used strings representing thespanning trees of the shape graphs for training. We believe thatthese limitation can be lifted by using a better variational autoen-coder, better suited for sampling vectorized graph topologies, e.g.,methods similar to Liu et al. [LABG18]. It should also be noted thatthe amount of automation (an important aspect of our work) canbecome a limitation in application scenarios that require a higherdegree of user control during modeling.



new

newnew

Figure 10: Shape interpolation examples. We display random shortest paths from the graph in latent space with edge weights according to a

similarity metric. The resulting intermediate shapes are more intuitive compared to using Euclidean distances in latent space. Shapes marked

as “new” were not present in the (automatically generated) training sets and were discovered by sampling the latent space of the variational

autoencoder.



8. Conclusion

We introduce a combination of a model-based and a data-drivenshape synthesis method able to create shapes from non-context-free shape grammars, a problem that is undecidable in general. Weachieve this by learning vectorized representations of shape graphswith a variational autoencoder and perform high-level modelingoperations via sampling in the resulting continuous latent space.We also demonstrate how the resulting sequences can be mappedback to geometric shapes by solving a classification problem. Inaddition, we deduce the assembly rules and automatically createsufficient training shapes from a single (or a few) example shapedecomposed into building blocks. Altogether, our method providesa proof of concept for automatic shape synthesis of structured shapevariations with arbitrary number of parts.

Acknowledgements We would like to thank the Thingiverse usersfor sharing the initial church, sand castle, and moon base mod-els. This work was supported by the European Research Coun-cil under the European Union’s Seventh Framework Programme,ERC grant agreement 340884 (ACROSS) as well as the Gottfried-Wilhelm-Leibniz Programme of the Deutsche Forschungsgemein-schaft DFG.

References

[BH12] BELL N., HOBEROCK J.: Thrust: Productivity-oriented libraryfor cuda. 7

[BVV∗15] BOWMAN S. R., VILNIS L., VINYALS O., DAI A. M.,JÓZEFOWICZ R., BENGIO S.: Generating sentences from a continuousspace. CoRR abs/1511.06349 (2015). 2, 4

[BWKS11] BOKELOH M., WAND M., KOLTUN V., SEIDEL H.-P.:Pattern-aware shape deformation using sliding dockers. ACM Trans.

Graph. 30 (Dec. 2011), 123:1–123:10. 2

[BWS10] BOKELOH M., WAND M., SEIDEL H.-P.: A connection be-tween partial symmetry and inverse procedural modeling. ACM Trans.

Graph. 29 (July 2010), 104:1–104:10. 2, 3

[BYMW13] BAO F., YAN D.-M., MITRA N. J., WONKA P.: Generatingand exploring good building layouts. ACM Transactions on Graphics 32,4 (2013). 5

[Che95] CHENG Y.: Mean shift, mode seeking, and clustering. IEEE

Trans. Pattern Anal. Mach. Intell. 17, 8 (Aug. 1995), 790–799. 6

[CSSB10] CHECHIK G., SHARMA V., SHALIT U., BENGIO S.: Largescale online learning of image similarity through ranking. J. Mach.

Learn. Res. 11 (Mar. 2010), 1109–1135. 5

[FKS∗04] FUNKHOUSER T., KAZHDAN M., SHILANE P., MIN P.,KIEFER W., TAL A., RUSINKIEWICZ S., DOBKIN D.: Modeling byexample. ACM Trans. Graph. 23, 3 (Aug. 2004), 652–663. 2

[FWX∗13] FAN L., WANG R., XU L., DENG J., LIU L.: Modeling bydrawing with shadow guidance. Computer Graphics Forum 32, 7 (2013),157–166. 2

[GDH∗16] GÓMEZ-BOMBARELLI R., DUVENAUD D. K.,HERNÁNDEZ-LOBATO J. M., AGUILERA-IPARRAGUIRRE J., HIRZEL

T. D., ADAMS R. P., ASPURU-GUZIK A.: Automatic chemical designusing a data-driven continuous representation of molecules. CoRR

abs/1610.02415 (2016). 2, 4

[KJS07] KREAVOY V., JULIUS D., SHEFFER A.: Model compositionfrom interchangeable components. In Proceedings of the 15th Pacific

Conference on Computer Graphics and Applications (2007), PG ’07,IEEE Computer Society, pp. 129–138. 2

[KW13] KINGMA D. P., WELLING M.: Auto-Encoding VariationalBayes. ArXiv e-prints (Dec. 2013). 2, 4

[LABG18] LIU Q., ALLAMANIS M., BROCKSCHMIDT M., GAUNT

A. L.: Constrained graph variational autoencoders for molecule design.CoRR abs/1805.09076 (2018). 8

[LF08] LEE J., FUNKHOUSER T.: Sketch-Based Search and Composi-tion of 3D Models. In Eurographics Workshop on Sketch-Based Inter-

faces and Modeling (2008), Alvarado C., Cani M.-P., (Eds.), The Euro-graphics Association. 2

[LVW∗15] LIU H., VIMONT U., WAND M., CANI M.-P., HAHMANN

S., ROHMER D., MITRA N. J.: Replaceable substructures for efficientpart-based modeling. Computer Graphics Forum 34, 2 (2015), 503–513.2, 3

[LXC∗17] LI J., XU K., CHAUDHURI S., YUMER E., ZHANG H.,GUIBAS L.: Grass: Generative recursive autoencoders for shape struc-tures. ACM Trans. Graph. 36, 4 (July 2017), 52:1–52:14. 1, 2

[MWZ∗13] MITRA N. J., WAND M., ZHANG H., COHEN-OR D.,BOKELOH M.: Structure-Aware Shape Processing. In Eurographics

2013 - State of the Art Reports (2013), The Eurographics Association. 2

[NW17] NASH C., WILLIAMS C. K. I.: The Shape Variational Autoen-coder: A Deep Generative Model of Part-segmented 3D Objects. Com-

puter Graphics Forum (2017). 2

[Pea01] PEARSON K.: On lines and planes of closest fit to systems ofpoints in space. Philosophical Magazine 2, 11 (1901), 559–572. 5

[RJT18] RITCHIE D., JOBALIA S., THOMAS A.: Example-based Au-thoring of Procedural Modeling Programs with Structural and Continu-ous Variability. Computer Graphics Forum 37, 2 (2018). 2, 8

[RMGH15] RITCHIE D., MILDENHALL B., GOODMAN N. D.,HANRAHAN P.: Controlling procedural modeling programs withstochastically-ordered sequential monte carlo. ACM Trans. Graph. 34, 4(July 2015), 105:1–105:11. 2, 8

[SA07] SORKINE O., ALEXA M.: As-rigid-as-possible surface mod-eling. In Proceedings of Symposium on Geometry Processing (2007),pp. 109–116. 3

[SSK∗17] SUNG M., SU H., KIM V. G., CHAUDHURI S., GUIBAS L.:Complementme: Weakly-supervised component suggestions for 3d mod-eling. ACM Trans. Graph. 36, 6 (Nov. 2017), 226:1–226:12. 1, 2, 5

[SVL14] SUTSKEVER I., VINYALS O., LE Q. V.: Sequence to sequencelearning with neural networks. CoRR abs/1409.3215 (2014). 6

[TLL∗11] TALTON J. O., LOU Y., LESSER S., DUKE J., MECH R.,KOLTUN V.: Metropolis procedural modeling. ACM Trans. Graph. 30,2 (Apr. 2011), 11:1–11:14. 2

[TYK∗12] TALTON J., YANG L., KUMAR R., LIM M., GOODMAN N.,MECH R.: Learning design patterns with bayesian grammar induction.In Proceedings of the 25th Annual ACM Symposium on User Interface

Software and Technology (2012), UIST ’12, ACM, pp. 63–74. 2

[Wei88] WEININGER D.: Smiles, a chemical language and informationsystem. 1. introduction to methodology and encoding rules. Journal of

Chemical Information and Computer Sciences 28, 1 (1988), 31–36. 2, 4

[XKHK17] XU K., KIM V. G., HUANG Q., KALOGERAKIS E.: Data-Driven Shape Analysis and Processing. Computer Graphics Forum 36,1 (2017), 101–132. 2

[XXM∗13] XIE X., XU K., MITRA N. J., COHEN-OR D., GONG W.,SU Q., CHEN B.: Sketch-to-Design: Context-Based Part Assembly.Computer Graphics Forum 32, 8 (2013), 233–245. 2

[XZZ∗11] XU K., ZHENG H., ZHANG H., COHEN-OR D., LIU L.,XIONG Y.: Photo-inspired Model-driven 3D Object Modeling. ACM

Trans. Graph. 30, 4 (jul 2011), 80:1–80:10. 2

[ZCOM13] ZHENG Y., COHEN-OR D., MITRA N. J.: Smart variations:Functional substructures for part compatibility. Computer Graphics Fo-

rum 32, 2 (2013), 195–204. 2


string-based synthesis of structured shapes · modeling—geometric algorithms, languages, and...

Documents