Sculpture Inspired Musical Composition

Francisco Manuel Alves Pereira Bragafrancisco.braga@tecnico.ulisboa.pt

Instituto Superior Tecnico, Lisboa, Portugal

November 2019

Abstract

Creativity is one of the most beautiful wonders of humanity. It is an essential factor for areas fromthe arts to science. Another concept, intrinsic to creativity, is inspiration. Inspiration is something thatarises in us and motivates us to do all kinds of tasks. It can come from within us, or from the worldaround us.

In this thesis, we have tried to make an inspirational system that takes as its source of inspiration asculpture to compose music. What we present is only a possible approach to the problem of mappingdifferent domains. Our approach does not consider the semantics of the sculpture but instead looks atit more abstractly.

The result of our system is three different compositions: Primal, the direct result of the applicationof our approach; Tonified, a version of the first in which out-of-scale notes are altered so as to have onlyin-scale notes; Genetic, the result of an evolutionary process that uses a Genetic Algorithm.

The Genetic composition was evaluated through questionnaires. In these questionnaires, themusical quality of the composition and the association with the sculpture were evaluated. The resultswere promising, with the majority of the participants having given a high classification (4 out of 5) tothe preferred interpretations of the compositions and relating them to the respective sculpture. Muchwork can still be done on this topic, nevertheless, we consider that what has been done in this thesis isa valid, interesting, and successful approach to this topic.Keywords: Computational Creativity, Inspiration, Genetic Algorithm, Sculpture, Musical Composi-tion

1. Introduction

Creativity is one of the pillars of several areas,such as arts or science and engineering, where itis mostly used when trying to solve a particularproblem [6]. Many would consider trying to repro-duce this unique skill an impossible task, or evenimmoral. Nevertheless, machines capable of mim-icking creative behavior could have countless ben-efits, either helping in problem-solving or in beingresponsible for the creation or co-creation of worksof art. From an artistic point of view, this wouldbe very controversial since art is mainly considereda human activity. Still, systems able to apply cre-ative processes are interesting both as a scientificproject and in the world of art.

Computational Creativity is the field of ArtificialIntelligence that deals with this type of research.In this field, not only computational aspects are in-vestigated, but also other areas such as cognitivepsychology and philosophy. As such, it deals bothwith practical and theoretical issues concerning cre-ativity.

One can easily state that something is creative,

either it being a thought, product, or somethingelse than to define creativity itself. Still, there issome consensus regarding what creativity involves:a person, process, press, and product, i.e., ”fourP’s of creativity” [8]. Although the four ”P’s” playan essential role in creativity, since we are workingin the Computation Creativity field, we aimed atpartially taking the person out of the equation. Oneelement to consider in our case is that our goal wasto create an inspirational system. As such, muchimportance is given, in our case, to the press aspectof creativity.

Inspiration is a fascinating topic by itself. Wesearch for inspiration when attempting to createa certain product, regardless of the field. Inspira-tion can come from inside of us or from the worldaround. Focusing on creativity inspired by othercreative artifacts, existing ideas inspire new ones inall sort of areas, such engineering, where new so-lutions to problems can be significantly influencedby already existing solutions, or in product design,where inspiration may come from shapes in nature[2], or art, where already existing artworks greatly

1

help one to create original works or styles [7]. Eventhough in these three cases inspiration comes fromthe same domain as the initial problem, it can alsocome from different domains. Here, there is a muchmore significant challenge because there is no directconnection. This forces the person to reconsiderhis/her framework since the inspirational artifactdoes not directly fit in it, leading to the construc-tion of new frameworks and perspectives [7].

In the computational realm, inspiration can beseen as the mapping between two different artifacts.This mapping is made so that the new artifact getsthe essence of the already existing one. Several au-thors were successful in this matter, by making sys-tems capable of doing this mapping, but still, notmany examples of inspirational systems are foundin this field [10]. The big problem here is how torelate two different domains.

In our work, we aimed at making an inspirationalsystem that composes music inspired by sculptures.We decided to draw the inspiration from existinghuman-made artworks since, this way, we were ableto maintain the human hand in the equation. Threegoals have driven our research:

• First we aim at building a system that bothacts creatively, and generates a product that isconsidered creative.

• Second, we want to compose a piece of musicthat is inspired by a sculpture. As such, themusic should, in a certain way, be associatedwith it.

• Third and last, we aim at building a systemthat composes music and not just sound. Al-though the exact difference between sound andmusic has not yet been defined, it is essentialthat the music can be considered aestheticallypleasing.

Our work has been divided into several stages. Thefirst stage was to explore the two domains involvedin our work, sculpture and music. In the next sec-tion we describe the background on these domainsnecessary to fully understand our system.

2. Background2.1. Sculpture

When first looking at a sculpture, several charac-teristics come to our attention. These can be di-vided into two groups: shape features, regardingthe shape of the sculpture, and texture features, re-garding the color or texture of the sculpture. More-over, we decided not to address the semantic mean-ing from sculptures, since this would be a complextask both computationally and in terms of inter-pretation, since each individual may have its ownperception of a sculpture. So, a general approach

that does not address semantics was chosen, leadingto a more abstract interpretation.

When dealing with sculptures computationally(or any form of data), the first problem is the repre-sentation. There are many possibilities to obtain asculpture in a computer, such as retrieving it fromimages, but the one we followed was using 3D ob-jects since all the information regarding both shapeand texture can be more easily obtained.

Considering the 3D objects domain, there are sev-eral different possible representations. In our work,we decided to use Polygonal Meshes, since it is asimple representation, yet it has all the informa-tion we needed. In this representation, the objectis represented by its boundary surface, composed ofseveral planar shapes (or facets), which are definedby a series of vertices connected by edges. The pre-cision of this representation depends on the numberof facets and vertices. It is more efficient computa-tionally, since manipulating planar shapes is rela-tively simple (linear algebra), but in high precisionrequirements it may need a large amount of data[1]. The most commonly used shapes are triangles(triangular meshes). Having specified our choice ofrepresentation, we can move to the extracted fea-ture.

First considering shape, different features can beidentified, some more intuitive than others. Fromall the features that could be extracted, three mainones were chosen as the most relevant for our work:curvature, angles, and segments.

Starting with the curvature, there are many pos-sibilities to measure this feature. From all the possi-bilities, such as the two classical measures Gaussianand mean curvature [5], we used the mean curva-ture. Each vertex in the mesh has its own meancurvature.

Regarding the angles, these are measured usingthe normal vector of the faces. To obtain each ver-tex angle, the angles between all its faces are mea-sured, and only the maximum is considered. Oneimportant aspect to note is that, since the angle isobtained from the normal vector, if the faces forma plane surface, the angle is 0◦, while two overlap-ping faces have an angle of 180◦. Using the angles,we also obtained another measure, which we calledzero angle predominance. This measure verifies ifthe vertices with 0◦ angles in an object are insertedin a plane surface, or if they occur separately. Ifthey are inserted in a plane surface, than our focusgoes towards what arises from it, or to limits of thatsurface.

Finally, the segmentation is performed using thespectral clustering algorithm. This algorithm fitsour purpose since its roots are on graph theory, anda mesh can be seen as a graph. It receives the adja-cency matrix of the object’s vertices and the num-

2

ber of clusters. It returns a label for each vertex.Using this, we divide the mesh into segments. InFigure 1, an example of segmentation of a 3D Meshusing Spectral Clustering can be seen.

(a) Original (b) Segmented

Figure 1: 3D Mesh Segmentation example usingSpectral Clustering

Moving to the texture, although we are extract-ing features from a 3D object, when it comes to thisit is referring to a 2D surface. A simple analogy isone of a gift wrap. The gift itself is a 3D object,but the wrapping alone is a 2D surface. In order todo the mapping of the texture in the 3D object to2D, there are several processes that can be can beapplied, being the most common the UV mapping,that simply takes the coordinates of a 3D object,(x,y,z), and converts it to 2D coordinates, (U,V).Since we are dealing with a 2D image, the featureextraction process will be done using the same fea-ture extraction algorithms as in a 2D image. Assuch, it is required to look further into color, start-ing by its possible representations.

Of the different available options, the ones morerelevant are RGB and HSV. On the one hand, inRGB, any color is described as a combination ofvalues of the three principal additive colors: Red,Green, and Blue. This model can be representedas a cube, as seen in Figure 2(a). On the otherhand, HSV refers to a combination of Hue, Satu-ration, and Value. Hue refers to the color itself,saturation to the intensity (purity) of a particularhue, and value to the lightness or darkness of thatcolor. This model can be represented as a cone, asseen in Figure 2(b).

(a) RGB Cube (b) HSV Cone

Figure 2: RGB and HSV representations

There are several different color features, beingthe two main categories histogram-based methodsand color statistics [11]:

• Color histogram-based methods - These meth-ods mainly represent the distribution of colors

in the image. Although there are several waysto do this, the most common and useful wayis the histogram of the distribution over thecolor model (3D in the case of RGB and HSV)or each channel of the color model.

• Color Statistics - These are statistical mea-sures, such as the value, standard deviation,median, percentiles, among others.

Aside from these two categories, several features canbe obtained by processing the image. From those,the two more relevant to our work are:

• Most Common Colors - To obtain these colors,a simple counting approach was used. First,nine equally spaced values were defined for eachchannel of the RGB model (thus a total of 729distinct colors). Having the image’s pixels rep-resented in the same model, for each pixel, theclosest value for each channel is chosen, andthat color’s count is incremented.

• Perceived Brightness - This can be seen asa conversion from the original colors to agreyscale, based on the brightness of each color.This feature can be obtained in several ways,for instance, a simple average of the three chan-nels in the RGB model. In our work, to obtaina better brightness descriptor, a more complexformula was used. In this formula, differentbrightness weights are attributed to each of theRGB channels [3]:

PB =√

0.299 ∗R2 + 0.587 ∗G2 + 0.114 ∗B2

2.2. Musical Elements and ConceptsMusic is a form of art which consists of organizedsound throughout time. The source of that soundcan be vocal, instrumental, among others, and it isusually divided into three parts: melody, rhythm,and harmony. These parts and their existence varyfrom culture to culture, being these three the mostcommonly used in Western music. Several exist-ing elements in music govern each of the alreadymentioned parts, such as notes, chords, scales, andmodes.

Starting with the most basic element, a note rep-resents a sound. The two main characteristics ofa note are its pitch and duration. The interval be-tween pitches is measured by tones, being the small-est interval in western music the semitone. Thereare twelve pitch classes in western music that arerepresented with a letter from A to G. They mayalso have a symbol called accidental. For our pur-pose, the accidental may be either flat ([) or sharp(]). The flat accidental translates into a semitonebellow, while the sharp translates into a semitoneabove. For instance, having a G, if we introduce aflat symbol, it becomes G[, i.e., a semitone bellowG, and if we introduce a sharp, it becomes G], i.e.,

3

a semitone above G. It is important to note thatin standard pitch, the pitch class repeats when wedouble a certain frequency. For example, the pitchclass A right above the middle C in the piano equalsto 440 Hz. If we double that (880 Hz), we get an-other A one octave above. The duration of a noteis the time the note lasts. This duration is repre-sented as a fraction of what is called a whole note.For our purposes, not all possible durations wereused

A scale is a repeating cycle of organized notesthat usually repeats every octave. The notes of ascale can be organized in ascending or descendingorder, and, commonly, the step between neighbornotes is either a tone or a semitone. Scales mayhave a tonal center, which is usually the first de-gree of a scale, that is called the tonic. Consideringthe major scale, the steps separating each of thenotes are T-T-S-T-T-T-S (T - tone, S - semitone).The natural minor scale is simply the major scalestarting on the sixth note, thus creating the pat-tern: T-S-T-T-S-T-T. The chromatic scale is a scalewith twelve pitches, and the step between pitches isalways a semitone. Considering that this scale in-cludes all twelve pitches used in western music, ithas no tonal center, meaning it has no tonic.

Before moving to modes, it is most relevant tolook into the common names given to intervals.These names are given according to the numberof semitones that separate two pitches, and can beseen in Table 1. What is called the modern modes

N. of semitones Name Abbreviation0 Perfect Unison P1 (1)1 Minor Second m2 ([2)2 Major Second M2 (2)3 Minor Third m3 ([3)4 Major Third M3 (3)5 Perfect Fourth P4 (4)

6Augmented Fourth A4 (]4)Tritone TTDiminished Fifth d5 ([5)

7 Perfect Fifth P5 (5)8 Minor Sixth m6 ([6)9 Major Sixth M6 (6)10 Minor Seventh m7 ([7)11 Major Seventh M7 (7)12 Perfect Octave P8 (8)

Table 1: Common interval names and abbreviations

are seven scales that derive from the major scale,i.e., that use the same set of notes. The differ-ence between the major scale and the modes is thatmodes start in one of the seven notes of the scale,making it the tonic, thus having different intervals.It is also important to mark that each mode hasone or more characteristic notes, which are notesthat better describe the mode (concerning the othermodes). In Table 2, all the seven modes, and their

characteristics can be seen.

ModeName

Tonicrelative tomajor scale

Intervals Name

Ionian I P1 M2 M3 P4 P5 M6 M7Dorian II P1 M2 m3 P4 P5 M6 m7Phrygian III P1 m2 m3 P4 P5 m6 m7Lydian IV P1 M2 M3 A4 P5 M6 M7Mixolydian V P1 M2 M3 P4 P5 M6 m7Aeolian VI P1 M2 m3 P4 P5 m6 m7Locrian VII P1 m2 m3 P4 d5 m6 m7

Table 2: Modes (characteristics notes are in boldaccording to interval)

A common concept when talking about modesis their brightness [9]. This notion is quite sim-ple: a mode with raised degrees, i.e., more sharps,is brighter than a mode with lowered degrees, i.e.,more flats. The order of modes from brighter todarker is the following: Lydian (Brightest), Io-nian, Mixolydian, Dorian, Aeolian, Phrygian, andLocrain (Darkest).

A chord is a set of multiple notes played simulta-neously. The number of notes in a chord may vary,but the most common in western music are chordswith three notes, called triads, and chords with fournotes, called tetrads.

Triads are chords that commonly contain threespecific notes: the root note, and the intervals(above the root) third, which can be major or mi-nor, and fifth, which, for our purpose, can be per-fect or diminished. A tetrad usually contains thesame notes as the respective triad plus the seventh,which may be major or minor. By positioning theroot note at each degree of the major scale, the ex-isting chords of the major scale are obtained. Eachchord can also be associated with the mode formedfrom the respective scale degree. In Table 3, we cansee the chord associated with each mode, its char-acteristics, and its representation as a triad and atetrad.

Degree ofmajor scale

Mode Notes Triad Tetrad

I Ionian M3 P5 M7 I I∆II Dorian m3 P5 m7 IIm IIm7III Phrygian m3 P5 m7 IIIm IIIm7IV Lydian M3 P5 M7 IV IV∆V Mixolydian M3 P5 m7 V V7VI Aeolian m3 P5 m7 VIm VIm7VII Locrain m3 b5 m7 VII◦ VIIm7[

Table 3: Chords from the major scale and corre-sponding mode

The concept of consonance and dissonance is away to characterize notes played simultaneously orsuccessively. Consonance can be associated withpleasant, satisfying, and stable, while dissonancewith unpleasant, troubling, and unstable. For our

4

purpose, it is relevant to further look into this con-cept applied to notes played above a particularchord. There are many different ways to put theseconcepts in pratice, from which we chose the fol-lowing. If the chord already includes that note, itcan be considered as consonant, while if the chorddoes not include the note, a simple rule can be fol-lowed: if the note creates the interval m2 with anynote from the chord, it is considered dissonant. Theimportance given to a dissonant note also dependson several other factors, being the two most impor-tant the note’s duration and where it occurs in thebar. For the most common time signature in west-ern music, 4

4, we have fours bars. The first beatis the strongest, followed by the third beat, and fi-nally the second and fourth beat. A dissonant noteplayed in a weak beat is less critical than one playedin a strong beat.

Another important musical element is a motif. Itcan be described as a short musical idea that oftenoccurs in a piece of music. The motif does not needto be always equal, and may be subject to changeswhile maintaining its general idea. We consideredfour possible changes, called variations: Melodic,when the melody is altered in some way, but therhythmic value is maintained, Rhythmic, when therhythm is modified, i.e., the rhythmic value of notesis altered, maintaining the pitch, Retrograde,whenthe motif is reversed, i.e., the first note becomes thelast, and vice-versa, and Inversion, when the motifis mirrored, i.e., all the intervals are maintained, buttheir direction is the opposite, usually consideringthe first note as the reference.

Finally, we consider the already mentioned partsof a music piece: melody, harmony, and rhythm. Amelody consists of a linear sequence of notes. Theharmony may consist of several elements, such asmultiples distinct melodic lines. For our purpose,harmony consists of multiple sequenced chords thatwill define a chord progression. From the differenttypes, we used modal harmony. The rhythm is as-sociated with the tempo and the time signature ofthe piece. Having already explained the time sig-nature, the concept of tempo in music is simply thepace of the given piece, and it is measured in beatsper minute (BPM).

3. Approach and Implementation

Since we aim at composing sculpture inspired mu-sic, we started by creating an analogy from one do-main to the other. However, before doing so, weneeded to restrict our approach regarding the musi-cal domain. Music is such a vast field with so manydifferent styles and types that it becomes quite im-possible not to restrict. As such, we decided to baseour approach on modal music. This decision wasmade considering that with modal harmony, one

can more easily pass on a specific sensation, associ-ated with the respective mode. This decision alsoinfluences the type of chords used, which shall belater explained. Besides this, we also only used themost common time signature in western music, 4

4.The first step towards creating the analogy was tomap the sculpture’s features into musical elementsand concepts. From the start, we did not want atoo precise approach, for example, where each pointwould be converted into a note. We intended to ob-tain the musical elements through the analysis ofsculpture’s features in a more general manner, byconsidering either the sculpture or each the sculp-ture’s segments as a whole. With this in thought,let us start with a high-level association betweenthe two groups of sculpture’s features, shape andtexture, and the parts of a musical piece, melody,harmony, and rhythm.

If we listen to a melody without any harmony(chords), we may perceive a particular sensation oremotion. Once we listen to it with harmony, thissensation or emotion may vary drastically from thefirst. Moving to the sculpture domain, if we thinkabout a sculpture’s shape, we will obtain mainlysensation associated with its characteristics, suchas smooth or rough. However, once we add thetexture/color, we get, once again, a much better-defined sensation, or even emotion. One can saythat the same way the texture/color gives contextto the shape, the harmony gives context to themelody. As such, our approach’s foundations arisefrom the association of the sculpture’s shape withmelody, and the texture/color with harmony. Itis important to note that this is not imperative,and some associations were made from texture tomelody. There is still one part of the music piece leftto map, the rhythm. As already stated, the rhythmis associated with the time signature and the tempo.Concerning time signature, as already stated, wedecided to only use the most common time signa-ture in western music, 4

4. About the tempo, al-though initially we wanted both shape and textureto be considered, we ended up using only the tex-ture. This decision was only made when confrontedwith the final results since one can only evaluatethe tempo having the music piece. As such, it shallbe later on explained.

Starting with the melody, this element consists ofa linear sequence of notes, while each note consistsof pitch and duration. Accordingly, we tried to asso-ciate these two characteristics with shape features.We used the sculpture’s lines or curve to obtain thepitch of the notes. If the lines of the sculpture aresmooth, the pitches of the melody would have togive this same sensation, as well as the contrary.To obtain the line’s description, we used the an-gles. We started by obtaining a general representa-

5

Angles(degrees)

0◦-15◦

15◦-30◦

30◦-45◦

45◦-60◦

60◦-75◦

75◦-90◦

90◦-105◦

105◦-120◦

120◦-135◦

135◦-150◦

150◦-165◦

165◦-180◦

Interval M2 M3 m3 P1 M6 m6 P5 P4 m7 m2 M7 TT

Table 4: Mapping from angles to musical intervals.

tion of the angles, the histogram. The histogram isa representation of the distribution of data, wherewe can obtain the frequency of intervals of values.These intervals are called bins, which may or maynot have the same width. Having this, we decidedto relate the musical interval for the next note con-cerning the current note with each bin. Since in oneoctave there are twelve possible intervals, the his-togram was calculated using twelve bins. With thehistogram, we created a probability distribution toobtain a musical interval, thus having the proba-bility for the next note’s interval in relation to thecurrent note. The mapping made from angles tomusical intervals can be seen in Table 4

One issue with this approach is the fact thatif one particular bin does not contain any occur-rences, that interval will never be used. We ac-knowledged that an interval of angles could not onlybe mapped to the respective musical interval, butalso the neighbors’ musical intervals. With this inmind, after using the probability distribution to ob-tain the musical interval, we also apply a normaldistribution to give the neighboring musical inter-vals a chance to be chosen. We used what is knowas the standard normal distribution (µ = 0 andσ = 1). Using a value obtain from the normal dis-tribution, there are three possible cases: if the valueis between -1 and 1 (probability of 68.3%), the orig-inal musical interval is chosen; If the value is below-1 (probability of 15.85%), the left neighbor musi-cal interval is chosen, unless the original value isthe first position, in this case, the original musicalinterval is chosen; If the value is above 1 (probabil-ity of 15.85%), the right neighbor musical interval ischosen, unless the original value is the last position,in this case, the original musical interval is chosen.

The only aspect left to decide is the directionof the interval, being the only musical element inthe melody that we decided to map from the tex-ture. Here, we decided to make use of the perceivedbrightness. Colors with a higher perceived bright-ness value would be related to upwards intervals,and colors with lower values with downwards inter-vals. From the histogram of perceived brightnessvalues, we took the value that occurred more often.This value ranges from 0 to 1, and set the probabil-ity of the interval direction being upwards between20% and 80%.

The duration of a note (or rhythmic value) willsignificantly influence the way we perceive a musicpiece. If we think of a sculpture with much variationconcerning the curvature, one can easily associate

that with a dense music piece, where the notes keepchanging in a small fragment of time. The oppositecan also be related. As such, we aimed at relat-ing the variation of the sculpture’s curvature withthe note’s duration. To obtain the curvature’s vari-ation, we used the autocorrelation. We gatheredthe curvature value of each point, grouped them byneighbors, and used autocorrelation in these groups.We decided to use the mean of all results, and mapthis value with what would become the most prob-able rhythmic value. The mapping can be seen inTable 5.

Mean of theAutocorrelation Results

0-0.05 0.05-0.1 0.1-0.4 0.4-0.7 0.7-1

Most ProbableRhythmic Value

Table 5: Mapping the mean of the autocorrelationresults to the most probable rhythmic value.

We did not want only one type of rhythmic value.We were looking for a general way to relate the au-tocorrelation results with the rhythmic value with-out making a direct relation. To achieve this, theStandard Normal Distribution was used once again.Using this tool, we can set the most probable rhyth-mic value while maintaining the possibility of usingother rhythmic values. For each note, a rhythmicvalue is chosen based on the most probable one anda value obtained using the normal distribution.

Instead of composing all the melody at once,we decided to compose motifs that would later bejoined, composing the whole melody. As seen be-fore, a sculpture can be segmented. As such, wedecided to compose each motif using features ob-tained from each segment. To obtain the size of themotif, we used each segment’s size in relation to thewhole sculpture.

Having the various motifs that will form themelody, we can now move to the harmony. It wasdecided to relate the harmony with the sculpture’stexture/color. As such, we want to harmonize eachmotif according to the respective segment’s color.For each segment, we obtained the most commoncolors, as explained before. Having the HSV colormodel in mind, for the same Hue, and the highestSaturation (100%), the Value channel will give ushow dark or how pure that Hue is, ranging fromblack (Value close to 0%) to the pure Hue (Valueclose to 100%). Similarly, the musical modes canalso be ordered from darker (Locrian) to brighter(Lydian).

As such, having the most common colors, we re-

6

lated the Value channel from the HSV color modelto musical modes. A color with a lower value in theValue channel would be associated with a darkermode, while one with a higher value with a brightermode. However, not all Hues should be matchedwith all modes, since, for the same Value, some col-ors are perceived as darker than others. As such,the Hues were divided in three groups, and for eachgroup the Value channel maps to a range of modes.The division in groups can be seen in Figure 3.

Figure 3: Hue value based groups

Besides these three groups, two special cases arenot accounted for by the Hue values: If the color isblack, or very close to black (Value channel below20 %), the mode is set to Locrian (the darkest); ifthe color is white, or very close to white (Saturationchannel below 10%), the mode is set to Lydian (thebrightest). It is important to note that even if theSaturation is near 0% if the Value channel is closeto 0%, the color will be close to black. As such, thefirst verification to be made must be if the color isclose to black (Value ¡ 20%). The range of modesper color group association and the special casescan be seen in Figure 4.

Figure 4: Range of Modes per Color Group andSpecial Cases

To obtain the chord, the closest tone of each motifis found, and then obtain the mode’s chord in thattonality, as seen in Table 3. For each segment, weget the two most common colors, from there thetwo respective modes, and finally, the two chords.The chord obtained from the most common coloris placed first in the motif and the one from thesecond most common chord second. The numberof bars for the first chord is obtained by dividingthe motif’s number of bars by two and roundingup, while the remaining bars are left for the secondchord.

To join the motifs in a logical order, we decidedto get the probability distribution of intervals fromthe histogram of angles, although this time from thewhole sculpture. From there, we calculate which ofall possible permutations of the motifs’ order bestcorresponds to the probability distribution of theintervals, being this the chosen order.

The only aspect considered left to map is thetempo. For this element, we used the variation ofcolor among segments. To get this variation, first,we obtained the most common colors of the wholesculpture. Then, for each segment, we verified howmany of the most common colors of that segmentmatched the ones from the whole sculpture. In theend, we obtained a ratio of the colors that matchedthe total colors verified that ranges from 0 to 1.The higher the ratio, the lower the tempo shouldbe, as well as the contrary. For our purposes, weconsidered a tempo range of 80 BPM to 160 BPM.

Having the melody and the harmony from theordered motifs, and the respective tempo, we obtainthe first version of the music piece, we can call it theraw composition. The second version was obtainedby correcting notes in the motifs that did not belongto the respective scale, changing them to the closestnote found on the scale. This way, we obtain aversion that respects each of the motif’s scale. Onemight call it a ”tonified” composition.

Besides these two versions, we decided to use aGenetic Algorithm. This algorithm provides a wayto search for better results while maintaining a cer-tain randomness that leads to exciting results.

This algorithm may vary from application to ap-plication, where the representation is can be seenas the basis. we defined each individual (candidatesolution) as a music piece. As already stated, weconsidered a music piece to be the gathering of sev-eral motifs (melody and harmony), their ordering,and the music’s tempo. Therefore, each individualbeing a music piece, its genes are motifs. There is acrucial property in each individual that is the order-ing of the motifs (genes). The algorithm is dividedin the following phases:

1. Initialization - This phase is the initial stage ofthe algorithm. We decided to have a popula-tion of 100 individuals. The number of motifsgenerated per segment is the chosen numberof individuals. Each individual is created byrandomly choosing a motif from each segment.The initial order of the motifs is randomly cho-sen.

2. Elitism - In this each generation, 10% of theindividuals with the highest fitness are directlypassed to the new generation. This is done toguarantee some level of quality.

3. Selection - A pair of individuals is selected.The probability of each individual to be cho-sen is based on the individual’s fitness. Thehigher the fitness, the higher the probability ofbeing chosen. As such, one individual may beselected several times or no times at all.

4. Crossover - Having a pair of selected indi-viduals, there is a probability of 80% of suf-

7

weight <= 1 weight > 1Notebelongsto theMode

CharacteristicConsonant 2 ∗ weightDissonant 2 ∗ weight −weight

Not characteristicConsonant weightDissonant weight −weight

Note does not belong to the Mode weight −weight

Table 6: Mode definition Fitness measure for different types of notes

fering crossover. We defined two types ofcrossover, both equally probable: motif andnote crossover. In the first case, half of theindividual’s motifs are randomly selected andcrossed over. This crossover occurs betweenmotifs generated from the same segment. Con-sidering the second, the pitch of the individualsis swapped. Since one may be bigger than theother, among the proper starting points, one israndomly chosen.

5. Mutation - The individuals obtained have aprobability of being mutated. This involvesaltering the individual’s genes. Four types ofmutation were used, each with an independentprobability. The first is order mutation, witha probability of 10%. In this case, the orderof the motifs is randomly altered. The secondis the ”tonify” mutation, with a probability of10% for each motif. In this case, notes that donot belong to the motif’s scale are corrected tothe closest note found on the scale. The thirdand fourth are the Inversion and Retrogrademutations, each with a probability of 5% permotif. In these cases, the respective variationis applied.

6. Fitness - The fitness for all individuals is cal-culated at the end of each generation. This isobtained from a fitness function. In our case,there are three measure to obtain the globalfitness: order, mode definition and range fit-ness. The order definition fitness evaluates howwell the order matches the probability distribu-tion of intervals obtained from the histogramof angles from the whole sculpture. This mea-sure ranges from 10 (perfect fit) to -10 (no fit).The mode definition fitness evaluates how wellthe motif describes the respective mode. Forthis, each note has a weight according to itsbeat and its duration. Only the beat with thehighest value is considered if the note includesmore than on beat. If it is in the first beat, theweight is 4∗duration, the third 3∗duration, thesecond 2∗duration, and the fourth 1∗duration.The value for this measure is than calculatedaccording to the type of note regarding themode, as described in Table 6. The range mea-sure penalizes −4 for pitches that are not in anacceptable range. We defined that acceptablerange between 55 and 90 in MIDI.

7. Termination - To terminate the algorithm, wedecided to have a fixed number of generations.Regarding the final music piece, we could ob-tain it from the fittest individual in the finalgeneration. However, our interest has movedtowards an approach that considers the evolu-tionary process as artwork [4]. In order to dothis, we decided to obtain the fittest individ-ual in each fixed number of generations. In theend, the individuals are joined and form the fi-nal music piece. We decided to terminate thealgorithm once it reached 300 generations. Toobtain the final results, we obtain the fittestindividual every 100 generations.

4. System Architecture

The approach above explained was implementedthrough the architecture seen in Figure 5.

At first, our system receives a sculpture. The firstmodule encountered is the Sculpture Module. Inthis module, the sculpture (3D object) is processedand analyzed to obtain the necessary features. Theresults of this module are the sculpture’s shape fea-tures (segments, angles, and mean curvature) andtexture features (perceived brightness and colors).

Having the necessary features, we pass them onto the Mapping Module. This module is respon-sible for the mapping of the sculpture’s features toelements of the musical domain, or mechanisms toobtain them. One can see it as the middleware be-tween the sculpture part and the musical part. Theresult of this model is a set of properties or valuesto be used in the musical domain. These includethe interval probability distribution and the inter-val direction probability for the pitch generation,the rhythmic value to be used as the most com-mon, and the mode to be used. One important as-pect to note is that different values are obtained foreach of the sculpture’s segments. The Tempo andthe general interval probability distribution (fromwhole sculpture) are also outputs of this module.

The values and properties obtained in the map-ping module, except for the Tempo that shall bedirectly applied in the music piece, are passed onto the Composer Module. Here, with all thenecessary information, motifs are composed. Themelody is obtained using the interval probabilitydistribution, the interval direction probability, andthe rhythmic value for the common case. The har-mony is composed using the modes obtained, and

8

Sculpture

SculptureModule

(3DObjectAnalysis)

ShapeAnalysis

TextureAnalysis

MappingModule

ComposerModule

SculptureInspiredMusicalComposition

Motif Composer

HarmonyComposition

OrderMotifs

Motifs

Tonify

PrimalComposition

TonifiedCompositionGenetic

AlgorithmModule

GeneticComposition

MelodyComposition

Figure 5: System Architecture

considering the melody. Following this module,there are two possible scenarios:

• The motifs are ordered using the general in-terval probability distribution - In this case,we obtain two compositions after adding theTempo: the Primal Composition, and, afterbeing ”tonified”, the Tonified Composition.

• The motifs are passed on to the Genetic Al-gorithm Module - In this case, the composermodule generates a fixed number of motifs fromeach segment, instead of only one. These mo-tifs are used in the genetic algorithm, and, afteradding the Tempo, the Genetic Composition isobtained.

5. EvaluationSix 3D Models of sculptures were used as thedataset. These had two sources: models found on-line, and models captured and processed by the au-thor. One example can be seen in Figure 6. For the

Figure 6: Galo de Barcelos1

six sculptures, the genetic composition was gener-ated. This was the only composition considered forthe evaluation. Having the six compositions, threepossible interpretations of each were generated us-ing virtual instruments. One closer to Jazz, one toRock, and one to Electronic/Space Rock.

To evaluate our system, the most important as-pects to consider are our objectives. Having thisin mind, we wanted to verify if the music piecescomposed are associated with the sculpture and ifthe music is considered aesthetically pleasing. Al-though one of our goals was to make a system thatacts creatively and generates creative products, thisaspect was not evaluated directly. Instead, we con-sidered that the evaluation of the other two goalswould lead to this goal’s evaluation.

We decided to use online surveys to perform theevaluation. Two surveys were made, each contain-ing three compositions. Having three interpreta-tions for each composition, the main part of thesurvey was done concerning the preferred interpre-tation. For each, we evaluated the quality of of themusic piece and the association with the sculpture.The least preferred is also evaluated, to establisha ground floor for interpretation. The results areexplained in more detail in the full version of thethesis. In here, we them very briefly.

Regarding the quality of the music, the statis-tics for all compositions can be seen in Table 7.In this particular question, the participants ratedthe music for the preferred and the least preferredinterpretations. The responses obtained were verygood for the preferred version. On a scale of 1 to5, considering the median of 4, we can state thatat least 50% of the responses rated the pieces as 4or higher. These results are the verification for themusical quality of the composed music pieces. Theleast preferred interpretation had lower results, asexpected. However, the rating lowered about onestep. These results were expected, and we consid-ered them to be good since this was the least pre-ferred interpretation of the three, and still got amedian and mode of 3.

How would yourate this music?

Preferred Least Preferred

Mean 3.55 2.79Median 4 3Mode 4 3Std. Deviation 0.92 1.03

Table 7: How would you rate this music? Statistics

Considering the sculpture-music association, theresults vary more from case to case. Two specialcases were encountered. The first case, where thesemantics of the sculpture led to a perspective dras-tically different from the one obtained through ourapproach. In this case, the results were obviouslybad. In this case, the description given by the par-

9

ticipants for the sculpture was very distant from thecomposition. The composition was not consideredrelated to the sculpture since, for both the preferredand the least preferred interpretations, about 70%of the participants rated the relation as either 1 or2 on a scale of 1 to 5. The second special case isencountered in compositions inspired by very sub-jective sculptures. In this case, there is much vari-ation on what is perceived from the sculpture. De-spite this variation, for the preferred composition,at least 50% of the participants rated the relationas 3 or higher. For the least preferred interpreta-tion, 51.6% of the participants rated the relationeither as 1 or 2. For the typical case, the resultswere good, since most people related the music tothe sculpture. Regarding the description, there wasa clear connection between the sculpture and themusic. Considering this case, for three out of thefour sculptures, more than 50% of the participantsrated the relation for the preferred interpretation as4 or higher. For the least preferred, the values wereabout one step lower than for the preferred, whichis expected.

6. Conclusions

The main objective of our thesis was create a systemthat composes music inspired in sculptures. Ourapproach involved relating the shape of the sculp-ture with the music’s harmony, and the texture withmelody. This association was not a strict associa-tion, yet it was our basis.

We believe that we created an inspirational sys-tem with an interesting approach that balances thefreedom associated with inspiration with a logicalmapping from the sculpture domain to the musicalone. During the process, we imposed many limita-tions and restrictions that are both associated withany approach, but special to ours due to the vastmusical knowledge used to build the system.

It is imperative to reinforce that what we havedone is one possible approach. As such, for futurework, there are other possible approaches that couldbe explored. Starting with the musical realm, moreelements and concepts could be used. For example,instead of using a modal approach, use a tonal oratonal approach or maybe even consider the three,limiting the system much less. The use of differenttime signatures could also be implemented since weonly used one. As for the association approach,others should be tested. The shape-harmony andtexture-melody association was made at the begin-ning, and exploring other possibilities could be in-teresting. Other approaches to the genetic algo-rithm could also be investigated. One possibilitywould be to give more freedom to the algorithmby not limiting the crossover between motifs gener-ated from the same segment of the sculpture. This

could be compensated by adding a fitness measurethat would favor individuals where all the segmentsare represented. A more concrete aspect to couldbe worked on is the evaluation since only one ofthe three output compositions was evaluated. Thethree could be assessed to verify, for example, whichone is preferred and which one is considered to bemore related to the sculpture. Besides this, hav-ing human interpretations of the composition couldsignificantly improve the evaluation. The quality ofmusic played by real musicians may be much betterthan interpretations generated using a computer.

References[1] J.-L. Dugelay, A. Baskurt, and M. Daoudi. 3D

Object Processing: Compression, Indexing andWatermarking. John Wiley & Sons, 2008.

[2] C. Eckert and M. Stacey. Fortune favours onlythe prepared mind: Why sources of inspira-tion are essential for continuing creativity. Cre-ativity and Innovation Management, 7(1):9–16, 1998.

[3] D. R. Finley. HSP Color Model - Alter-native to HSV (HSB) and HSL. http://

alienryderflex.com/hsp.html.

[4] C. G. Johnson. Fitness in evolutionary art andmusic: what has been used and what could beused? In International Conference on Evo-lutionary and Biologically Inspired Music andArt, pages 129–140. Springer, 2012.

[5] J. J. Koenderink and A. J. Van Doorn. Surfaceshape and curvature scales. Image and visioncomputing, 10(8):557–564, 1992.

[6] A. Koestler. The Act of Creation. Hutchinson,1964.

[7] T. Okada and K. Ishibashi. Imitation, in-spiration, and creation: Cognitive process ofcreative drawing by copying others’ artworks.Cognitive science, 41(7):1804–1837, 2017.

[8] M. Rhodes. An analysis of creativity. The PhiDelta Kappan, 42(7):305–310, 1961.

[9] M. Ron. Modal jazz composition & harmony.Advance Music, Rottenburg, 1996.

[10] J. Teixeira. Cross-Domain Analogy: From Im-age to Music. Master’s thesis, Instituto Supe-rior Tecnico, Lisbon, Portugal, 2017.

[11] M. J. Alvarez, E. Gonzalez, F. Bianconi,J. Armesto, and A. Fernandez. Colour andtexture features for image retrieval in graniteindustry. Dyna, 77:121 – 130, 2010.

10