Full Terms & Conditions of access and use can be found athttp://www.tandfonline.com/action/journalInformation?journalCode=ndcr20

Download by: [faculties of the University of Ljubljana] Date: 05 June 2017, At: 06:21

Digital Creativity

ISSN: 1462-6268 (Print) 1744-3806 (Online) Journal homepage: http://www.tandfonline.com/loi/ndcr20

Light fountain – a virtually enhanced stonesculpture

Franc Solina & Blaž Meden

To cite this article: Franc Solina & Blaž Meden (2017) Light fountain – a virtually enhanced stonesculpture, Digital Creativity, 28:2, 89-102, DOI: 10.1080/14626268.2016.1258422

To link to this article: http://dx.doi.org/10.1080/14626268.2016.1258422

Published online: 18 Nov 2016.

Submit your article to this journal

Article views: 99

View related articles

View Crossmark data

Light fountain – a virtually enhanced stone sculptureFranc Solina and Blaž Meden

Computer Vision Laboratory, Faculty of Computer and Information Science, University of Ljubljana, Ljubljana, Slovenia

ABSTRACTThe article describes the making of an art piece combining stone sculpture andvirtual water. The motivation for this art piece was to enrich the usual staticformat of a stone sculpture with a dynamic dimension. The dynamicdimension is attained with virtual water droplets running over the stonesurface which behave as real water droplets. The 3-D surface of a speciallycarved stone sculpture is during an exhibition continuously captured by theKinect sensor. Each water drop out of many thousands, which are introducedinto the installation as evenly distributed rain drops, falling over thesculpture, are simulated individually to run over the stone surface followingthe largest slope. These simulated water drops are projected with a videoprojector as light points on the surface of the sculpture. An observer canenjoy simultaneously the haptic experience of touching the stone andobserving a digitally generated but physically grounded animation.

ARTICLE HISTORYReceived 24 March 2016Accepted 3 November 2016

KEYWORDSStone sculpture; simulationof running water; waterdrops; range image; Kinect;OpenFrameworks; 3-Dsurface model; int art; artinstallation

1. Introduction

Artists often try to integrate movement intopaintings and sculpture to make their artmore alive and engaging. Paintings that striveto communicate some movement typicallyhave to rely on some special painting style oroptical illusions such as in ‘op art’. Sculpture,on the other hand, could in fact include actualphysical movement of its constituent parts.Mobiles are typical examples of kinetic artthat react to gravity and atmospheric con-ditions. Mobiles are, therefore, usually madeout of metal, wood or other materials that aremalleable enough to form light and strong parts.

Other types of kinetic sculpture react to spec-tators who can directly manipulate componentsof the sculpture or who can activate directly, orby means of some sensors, motors that animatethe sculpture. This type of art has evolved alsointo interactive computer-based installations

that typically react to the presence and move-ment of visitors.

Introducing physical movement into stonesculptures is, on the other hand, much more dif-ficult due to their massive and heavy forms.Only stone spheres, due to their geometry, canbe moved around without excessive force asmanifested by David Fried’s kinetic sculpturesentitled ‘Self Organizing Still Life’ (Fried2011). Fried places in his project several sphereson a level enclosed surface. Every sphere is of adifferent size and material such as solid rock,sand or synthetic polymers. The spheres areactivated by ambient sound. The movementand interactions between Fried’s spheres rep-resent how the universe is interconnected. Thetechnology is not explained, but microphonesare probably used to detect the ambient soundand activate electro-magnets in the base of theinstallation to trigger the movement of thespheres (Brockmeyer and Marsico 2014).

© 2016 Informa UK Limited, trading as Taylor & Francis Group

CONTACT Franc Solina franc.solina@fri.uni-lj.si

DIGITAL CREATIVITY, 2017VOL. 28, NO. 2, 89–102https://doi.org/10.1080/14626268.2016.1258422

However, an age old way of making stonesculpture more dynamic and energetic is tocombine stone with running water. This is man-ifested by the abundance of fountains made outof stone throughout history in all parts of theworld. A nice example of combining graniteand running water is the contemporary sculp-ture ‘Stone of life’ by Janez Lenassi in Ljubljanawhich was finished in 1978 (Figure 1).

Another well-known fine art sculpture whichhas been enhanced with running water is thesculpture ‘Water stone’ by Noguchi (1986)installed inside the Metropolitan Museum ofArt in New York. However, the majority ofstone sculptures incorporating running waterare placed outside due to practical reasons ofhandling excessive water.

1.1. Motivation

To evade the problem of handling runningwater inside buildings we decided to animate astone sculpture not with real water but by

using virtual water droplets generated by mod-ern computer technology. Our original inten-tion was that this virtually enhanced art piecewould be appreciated by the public primarilyby observation.

To make the animation realistic one needs toknow the actual 3-D shape of the stone sculp-ture. In recent years methods for capturing 3-D shapes have become much faster and easierto use. Low-cost sensors such as the MicrosoftKinect have opened the door to new appli-cations where the 3-D shapes of objects andtheir location in a scene have to be known.

The Kinect sensor was originally developedfor gaming applications but it was soon co-opted by scientists and artists for all kind ofapplications, especially if some sort of inter-action is required as in computer games(Zhang 2012). An overview of some artisticapplications of Kinect can be seen on the Crea-tive Applications Network website (CreativeApplications Network). Even the infrared lightitself, used by the Kinect sensor and invisibleto the human eye, was used in an art photogra-phy project ‘Dancing With Invisible Light’(Penven 2011).

In this article we describe how the Kinectrange sensor was used to continuously capturethe 3-D shape of a stone sculpture so that virtualwater droplets falling on the sculpture can besimulated to behave as real water droplets,which are then projected as light points on thephysical stone sculpture, using a video projec-tor. The resulting art piece was entitled ‘Thelight fountain’. A preliminary report on thisproject has already been presented at a confer-ence (Meden et al. 2015). In this preliminaryreport only the technological side of the projectis described since the sculptural part was notfinished yet.

1.2. Related work

Projects which are from the technical aspectmost related to our goal are interactive sand-boxes (Reed et al. 2014; SandyStation) where

Figure 1. (a and b) Granite fountain entitled ‘Stone oflife’ by sculptor Janez Lenassi, finished in 1978, standsin the ‘Žale’ cemetery in Ljubljana.

90 F. SOLINA AND B. MEDEN

users can shape the sand in a sandbox into newforms which are continuously captured by aKinect sensor. A video projector is used to col-our the sand according to its height and con-figuration. These boxes are primarily intendedfor educational purposes, demonstrating geo-graphic, geologic and hydrologic conceptssuch as how to read a topography map, themeaning of contour lines, watersheds, catch-ment areas, levees, etc. The sandboxes, however,can also be a platform for interactive games(Reed et al. 2014; Mimicry). The virtual spaceprojected onto a physical 3-D landscapebecomes alive with a multitude of living formsranging from spiders, beetles and snails throughammonites and trilobites to sharks and dino-saurs, all inhabiting a mixed reality ecosystem(St. Jean 2012).

In fine art sculpture there is already sometradition in using virtual models of 3-Dforms. Physical sculptures, for example, canserve as key-shapes for a series of virtualsculptures which can be exhibited in parallel(Adzhiev, Comninos, and Pasko 2003). Thevirtual shapes are first overlaid on the physicalsculptures but next they can evolve in an inter-active sculpting session. A Kinect sensor canalso be used as an aid for real-time pose edit-ing different kinds of articulated sculpture art-works (Wu et al. 2016). In a digital heritagescenario, skeletons of statues or of humanbodies which are captured by depth sensorsfor mobile devices can be used to query data-sets of 3-D models of Graeco-Roman statues(Barmpoutis, Eleni, and Daniele 2015).

Fascination with running water is also evi-dent in the ‘Waterfall’ animation for the Sales-force lobby in San Francisco (Obscura 2016),which involves simulation of water runningacross a horizontal plane and over a verticallip to achieve spectacular visual effects on thewalls of the lobby. The visualisation reflectsthe local weather, so that, for example, when itsnows outside, it also snows on the screen.

French designer Mathieu Lehanneur has cre-ated a series of sculptures, called ‘Liquid Marble’

out of dark marble whose top surface is carvedin the form of wave ripples. When an observeris moving around such a sculpture the changinglight reflections from the carved marble surfaceinduce the semblance of moving waves. Thesurface is designed first with 3-D modellingsoftware, which is then used to guide a stonecarving machine and, finally, the resultingmarble surface is hand polished (Lehanneur2016).

1.3. Technical requirements

To create an art piece, where virtual water dro-plets are running on the surface of a stonesculpture, we had to address the following tech-nical problems:

1. carving of an appropriate stone sculpturewith rather level and continuous surfacesso that the video projection is not discontin-ued by self-imposed shadows;

2. capture of the 3-D shape of the sculpture’ssurface on which light points representingwater droplets are simulated;

3. creation of a dynamic model for water dro-plets which, according to the law of gravity,run down the maximum slant of the sculp-ture’s surface;

4. alignment of the 3-D model of the sculpture,acquired by Kinect, with the video projectionof the virtual water droplets, and the actualstone sculpture.

The rest of the article is divided as follows. InSection 2 we describe the stone sculpture thatwas carved especially for this project. In Section3, the implementation of the virtual part of theart piece is described. In Section 4, the resultingart piece is presented, reactions of the publicconfronted with the art piece during the firstpublic exhibition are reported and possibleimprovements of the art piece are considered.Section 5 discusses the resulting art piece in awider context of interactive art and concludesthe article.

DIGITAL CREATIVITY 91

2. The stone sculpture

A stone sculpture designed especially for thisproject has been carved by the first authorof this article out of a block of MacedonianSivec marble (Figure 2). Sivec is a dolomiticmarble that was used in south-east Europefrom the third century BC until the fifth cen-tury AD, producing some of the most visuallystunning creations of the ancient world (Pro-chaska 2013). The marble block has approxi-mate dimensions of 40 × 60 × 20 cm. Duringthe stone carving process we have continu-ously tested the shape of the top surfacewith real water to test if the slant was suffi-ciently large to allow water to run towardsthe edges of the sculpture (Figure 3). Thetop surface is smoothly undulating resemblingcurved sun rays emanating from a central

point, somewhat reminding one of the Ver-gina Sun (Danforth 2010) (Figure 4). Themarble surface of the sculpture was notpolished to a high gloss so that the projectedlight points can be absorbed in the materialrather than specularly reflected, making inthis way the projected light points moreclearly defined.

3. Implementation of the virtual partof the art piece

To capture the 3-D shape of the sculptureKinect for Xbox 360 is used, which is a Micro-soft product that captures 3-D points on theprinciple of using an infrared projector and aninfrared camera (Zhang 2012). Kinect appearedtogether with the game console Xbox 360 pri-marily to capture the movement of players ofcomputer games. But since Kinect is easy touse and is reasonably priced it became popularwith a large user community that applies Kinectto a large variety of applications that need 3-Dshape capture (Develop Kinect). The output ofKinect is a cloud of 3-D points in a given coor-dinate system. It has a built-in gyroscope so thatthe vertical axis can be aligned with the direc-tion of gravity.

For projection of light points representingthe water droplets a video projector is required.Any digital video projector with a high resol-ution and a zoom lens, so that the projection

Figure 3. The slant of the top surface of the sculpturewas tested with real water.

Figure 4. The finished stone sculpture carved particu-larly for the project ‘Light fountain’.

Figure 2. Initial design on top of the marble block ofMacedonian Sivec stone which was inspired by sunrays emanating in a spiral shape from a focal point.

92 F. SOLINA AND B. MEDEN

angle can be adjusted, is adequate. The videoprojection of animated light points, whosemovement is computed based on the 3-Dshape of the sculpture’s surface, which is cap-tured by Kinect, must be projected on the actualsurface of the sculpture.

To facilitate the 3-D surface capture and theprojection of virtual water droplets on the sculp-ture, the Kinect and the video projector shouldbe mounted above the sculpture close togetherand with their lines of sight almost parallel so

that all parts of sculpture’s surface are visiblefrom their viewpoints. To securely install bothdevices above the sculpture a heavy-duty tripodis used, with a custom-designed mount for thevideo projector and the Kinect (Figure 5). Sincethe accuracy of the Kinect sensor drops with lar-ger distances, the distance between the sculptureand the Kinect must not be too large. Exper-iments show that the accuracy of Kinect at a dis-tance of about 1.5 m is in the order of just a fewmillimetres (Ravnik and Solina 2013). However,the variation in depth estimate when Kinect ispointed directly at a planar surface is largerand not entirely random and it appears like a cir-cular ripple in the depth image (Andersen et al.2012). This problem can sometimes manifestitself when on a part of the surface, with a verysmall slant, the simulated droplets are not run-ning towards the lower parts and the edges ofthe sculpture.

In the current hardware configuration, theKinect has a wider field of view as the video pro-jector. By experimentation it was determinedthat when the Kinect and the video projectorare mounted about 120 cm above the sculpture,a good compromise is reached for sufficientaccuracy of the Kinect and also for the videoprojection since at that distance and at thegiven size of the sculpture, the whole resolutionof the video projector can be used.

Figure 5. (a and b) The setup of the equipment: thevideo projector and Kinect are aligned and closetogether about 120 cm above the sculpted stonesurface.

Figure 6. A cloud of 3-D points captured by Kinect: thesculpture is seen from the side.

DIGITAL CREATIVITY 93

3.1. 3-D model of the sculpture

The 3-D point cloud, representing the 3-Dshape of the sculpture, obtained by Kinect isdefined in a coordinate system which is rotatedaccording to the slant of the Kinect sensor(Figure 6). The 3-D point cloud must thereforebe transformed with an inverse rotation into aglobal coordinate system where the actualsculpture also resides. The slant of the sensoris indicated by the gyroscope which is builtinto the Kinect. It is possible also to manuallyset the slant angle on the user interface of thesystem. The 3-D points after the transformationare aligned with the global coordinates of thework space. The global coordinates representthe following dimensions: z-axis, verticaldimension or the direction of gravitation; x-axis, first dimension of the horizontal plane; y-axis, second dimension of the horizontal plane.

The captured cloud of 3-D points in Figure 6is first reduced by a manually defined mask onlyto those 3-D points that lie on the sculpture.The resulting 3-D points are then transformedinto an intensity image by projecting the 3-Dpoints onto the horizontal plane (Figure 7).This gives us the perpendicular view of thescanned sculpture. From the intensity imagewe generate the 3-D model of the sculpture(Figure 8) which represents the supporting

surface and is used for the computation of themovement of individual water droplets. The 3-D model of the sculpture’s surface is a grid ofpoints where the coordinates of each 3-Dpoint are the x and y coordinates of the corre-sponding pixel in the intensity image and thez coordinate is the intensity (height) of thepixel. This 3-D model in Figure 8 representsthe surface of the scanned sculpture.

3.2. Software environment

OpenFrameworks is anopensource framework forintegration of various technologies which are usedin computer graphics, computer vision and ininteractive applications in general. The frameworkis written in C++ programing language and rep-resents a high-level abstraction of OpenGL,GLEW, Glut, FreeType, OpenCV and otherlibraries. The framework enables the use of differ-ent librarieswhich are developed by an active com-munity of developers. BesideOpenFrameworksweused in the project also the following two softwarelibraries: ofxKinect, which supports communi-cation with the range sensor Kinect, and ofxGui,which offers elements for adjusting the user inter-face parameters.

MicrosoftVisual studio is an integrated devel-opment environment which supports the C++programing language and is the recommendedtool for application development under Open-Frameworks on the Windows operating system.

Figure 7. A range image shown as an intensity imagewhere the intensity of each pixel corresponds to thedistance from the Kinect sensor: the undulated surfaceis seen in a top-down direction.

Figure 8. The 3-D model of the sculpture’s surfaceused in the simulation of the movement of water dro-plets: seen from the side. The surface is divided into agrid of approximately 150 × 230 cells.

94 F. SOLINA AND B. MEDEN

The system must support the processing of theinput image and the simulation of water dropletswhich is computationally very demanding andproportionally dependent on the number ofsimulated water droplets.We use a PC computerwhichmakes possible a simultaneous simulationof 50,000 water droplets at the speed of30 frames/s. This enables a reasonably goodapproximation of liquid simulation.

Since the Kinect sensor refresh rate (from 20to 25 Hz) is a bit slower than our simulationloop refresh rate (i.e. the application’s framerate is from 30 to 40 FPS), we designed theapplication to run in two parallel sectionsusing threads. In the main thread we performrendering (since the underlying OpenGL libraryhas a client–server architecture and thereforeworks optimally only on one thread) and theparticle update phase (which is closely relatedto the render phase). The secondary thread con-tinuously reads the depth data from the Kinectsensor and performs all data transformations. Italso calculates the gradients, which are neededto properly simulate water particles in themain thread. This also means that both threadshave to cooperate in order to share the depthand gradient data between each other. Sincethe application is coded in C++, we can opti-mise the crucial data sharing and memorylocks by using a double buffering technique.In this way, we share/lock only the pointerswhich point to the shared data and the datathemselves are not moved or copied around;therefore we can be more effective in terms ofspeed and inter-thread communication.

3.3. Simulation of water droplets

Since we could not find an appropriate opensource solution for the simulation of water dro-plets thatmove in the direction of the highest grav-ity gradient, we decided to write our own code.

To visualise the running water on the surfaceof the sculpture, a large number of small particlesin the form of spheres that move across the sur-face of the 3-D model are used. Each particle or

droplet is a single unit that moves independentlyand is also independently simulated. The entiresimulation of water proceeds in loops. Approxi-mately 30 such loops are executed each secondto achieve the appearance of smooth motion. Ineach loop the direction and velocity of the move-ment of all droplets are updated. The computationof the movement of each droplet depends on:

1. the gradient of the surface at the actual pos-ition of a droplet;

2. the direction and velocity of the droplet inthe previous step of the simulation;

3. an adjustable chance factor.

After new gradients are received from the sec-ondary thread, the simulation loop goes overall existing particles and updates the velocityof each particle according to its position onthe surface, taking into account the previousvelocity vector and refreshed gradient values.The entire simulation is performed for eachvideo frame so that all water droplets areredrawn in their new position.

A droplet therefore moves across the sculp-ture according to the slant of the surface. Sincea droplet has a mass, the inertia from the pre-vious step influences its movement in the nextstep of the simulation. Individual droplets how-ever do not have any physical dimension andtherefore no collisions between droplets arecomputed. The detection of such collisions iscomputationally demanding and if performed,it would reduce the maximum possible numberof droplets that can be simulated on our hard-ware in real time. We also do not use anyadditional shading and illumination of dropletswhich is also computationally demanding anddoes not add much to the desired visual effect.

Droplets must come from somewhere inorder to run down the sculpture. Water dropletscan be introduced into the simulation in twopossible ways:

1. as rain drops falling evenly distributed overthe whole surface of the sculpture or

DIGITAL CREATIVITY 95

2. from possibly several water springs on thesurface of the sculpture. The user interfaceenables the user to define the position ofwater springs on the sculpture interactively.Water springs are useful for testing thebehaviour of water droplets in a particularpart of the sculpture (Figure 9). One canselect several springs; however, the numberof water droplets is then divided amongthem so that the simulation always uses thesame predefined number of water droplets.If the maximum possible number of dropletsin the current implementation (50,000) isdivided among 10 springs one can still geta realistic visualisation.

Droplets disappear from the simulation whenthey reach the edges of the sculpture. They arereintroduced again as new rain drops fallingon the sculpture or as water drops in one ofthe springs. In general, the best visual appear-ance of the art piece is achieved in the ‘rain’mode of operation (Figure 10).

The large number of moving droplets createsthe appearance of running water. Since a largenumber of droplets can reappear in the samepoint of the 3-D model and continue to moveaccording to the surface gradient, a chance fac-tor is used, which takes care that the path of

water droplets originating from the samepoint is not identical. This chance factor greatlyimproves the appearance and is computation-ally not demanding.

3.4. Spatial adjustment of videoprojection and the projection surface

When the whole installation is being set up, thefirst task is to align the picture projected by thevideo projector to the projection surface on thesculpture. The picture is initially approximatelyaligned based on the position of the Kinect andon the centroid of the 3-D points. The virtualcamera in the simulation model is positionedin the point where the Kinect is mounted anddirected towards the centroid of all 3-D pointscaptured by Kinect (approximately towardsthe projection surface).

However, Kinect and the video projectorcannot be exactly in the same physical positionas shown in Figure 5 and, therefore, they cannothave exactly the same viewing angle. This differ-ence must be adjusted manually. This can bedone on the user interface by changing the pro-jection angle and the scaling of the 3-D surfacemodel. One can also manually adjust the edges

Figure 9. Water droplets can emerge from springs,which can be interactively defined on top of the sculp-ture. This functionality was useful during the prelimi-nary testing of water droplet behaviour.

Figure 10. Final appearance of the art piece entitled‘The light fountain’ showing water droplets/lightpoints raining evenly over the whole sculpture, collect-ing and running down the sloping channels towardsthe edge of the stone sculpture. For an animatedview see the following link on YouTube: https://you-tu.be/nqituTE-_SI.

96 F. SOLINA AND B. MEDEN

of the projected animation so that the waterdroplets are projected only on the surface ofthe sculpture.

3.5. User interface

During the simulation of water droplets the usercan change different parameters of the simu-lation using a simple graphical user interface(Figure 11).

On the user interface one can change the fol-lowing parameters of the simulated water droplets:

. the number of droplets in the simulation

. the viscosity of the droplets

. the value of the chance factor for the move-ment of the droplets

. colour of the droplets

. transparency of the droplets

. background colour

. addition and removal of water springs on thesurface of the sculpture

. selection of the operation mode (rain/watersprings)

. other positioning and diagnostic parameters.

4. Results

We have experimented with different settingsfor colours and appearance of light points butsettled at the end for a rather minimalisticappearance using just white light points on adark background (Figure 12). A video clip ofthe installation can be seen on YouTube (Solina2015a).

The art installation was exhibited for the firsttime at the ARTIUM SPECULUM 2015 Festivalof new media culture in Trbovlje, Slovenia,which was in 2015 running under the motto‘The Integrity of Reality’ (AS 2015). The exhibi-tion space was kept in semi-darkness so that theoutlines of the stone sculpture could still be dis-cerned but the emphasis was on the projectedlight points (Figure 12). Actually, the shape ofthe underlaying stone surface could be easily

inferred from the movement of the light pointseven in total darkness.

The installation was warmly received by thepublic. Since the original intention of this artpiece was to virtually enhance a stone sculp-ture and to observe the simulated flow ofwater, it was somewhat surprising that mostobservers also wanted to touch the sculptureand to interact with the moving light points.Visitors were henceforth encouraged to touchthe sculpture and it seems that this mergingof haptic and visual experience was anadditional attraction of this art piece (Figure13). By reaching into the projection cone ofthe installation, the public quickly realisedthat the rain drops instantly adapted to thechanging shape of the rained-on surface. Thewater droplets then start to flow down thenew obstacle as can be observed in a video(Solina 2015b). One can even catch lightpoints as water droplets into the open palmsof our hands (Figure 14). This interactivity isthe main reason why a continuous capture ofthe sculpture’s 3-D shape by the Kinect sensoris necessary. Although the stone sculptureitself is static and it would suffice thereforeto capture its 3-D shape just once, when theinstallation is being set up, the interactivenature of the installation would be lost withoutthe continuous 3-D shape capture.

Children were especially happy when theydiscovered that they can ‘splash’ with lightpoints, but they also voiced a factual obser-vation that the light points resemble tiny styro-foam balls rather than water droplets.

Some observers related their experience thatwatching the installation is somehow hypnotis-ing and that it is difficult to turn away one’s gaze.

4.1. Possible improvements

The model for water droplets used in thedescribed installation is fairly simple. Thebehaviour of water droplets depends mainlyon the gradient of the 3-D surface. The velocityof each water droplet is individually computed.

DIGITAL CREATIVITY 97

In each simulation loop the velocity of individ-ual droplets increases or decreases dependingon the gradient of the surface.

One could use one of the existing simulationpackages for liquids. A suitable open sourcesimulation package is Fluids v.3.11 which offerssupport also for a parallel implementationbased on CUDA technology.2 With this packageone could increase the number of simulateddroplets and make the simulation more

Figure 11. User interface for adjusting the parameters of the water drops simulation.

Figure 12. Art piece ‘Light fountain’ at its first publicexhibition at Speculum Artium 2015, festival of newmedia art in Trbovlje, Slovenia, in October 2015.

Figure 13. The virtual rain drops are inviting theobservers to interact with them and to touch thesculpture.

98 F. SOLINA AND B. MEDEN

realistic. Interaction between droplets could bemodelled, as well as the interaction of dropletswith the supporting 3-D model. However, amuch more powerful computer would berequired to run the simulation.

One could also improve and speed up theprocess of properly aligning the optical axes ofKinect and the video projector. Some temporarymarkers, for example small spheres at the edgesof the sculpture, would simplify the iterativeadjustment of the direction of both opticalaxes and the zoom settings of the video projec-tor so that the 3-D surface model obtained byKinect, on which the simulated water dropletsare moving, would match more closely thephysical surface of the stone sculpture.

The same virtual enhancement could beapplied also to a much larger sculpture. How-ever, the range scanner would have to bemoved further away from the surface of thesculpture, in this way making the Kinect ill-sui-ted since the errors in depth assessment would

become too large. Some other range scannerwould be required in such a case, while theselection of a video projector is not critical.

5. Discussion and conclusions

Computer-mediated interactive art installationsreceive signals from the environment and theobservers; they process the signals to transformthe installation according to these signals typicallyinto a new visual form which is then ‘exhibited’(Edmonds, Bilda, and Muller 2009; Strehovec2009). Cornock and Edmonds (1973) definedfour basic categories that characterise the relation-ship between artwork, artist, viewer and environ-ment which are particularly applicable tointeractive artworks. These four categories are:

1. Static: There is no direct interaction betweenthe artwork and the observer, as when anobserver is looking at a painting. The observer,however, may experience some emotional orpsychological reaction.

2. Dynamic-passive: The art piece can changein a predictable way to outside environ-mental influences (light, sound, tempera-ture). The observer is a passive observer ofthese changes. All possible changes of theart piece are predefined by the artist.

3. Dynamic-interactive: It is the same asdynamic-passive with the added factor thatobservers can also assume an active role ininfluencing the changes of the art piece,nowadays achieved usually using variouscomputer-based sensors. Such art piecescan react, for example, not only to generalmotion but also to specific gestures of theobservers, in this way making the organis-ational configuration of the art piece evenmore rich and complex.

4. Dynamic-interactive (varying): In this cat-egory an additional modifying agent takesthe role of changing the initial configurationor behaviour of the art piece. The perform-ance depends therefore not only on the cur-rent influencing signals but also on the

Figure 14. Kinect takes range images continuously sothat when an observer reaches into the projectioncone, the virtual rain drops start to collect in heropen palm.

DIGITAL CREATIVITY 99

previous signals, which means the entire his-tory of interactions. The interactive art piecein this way learns and evolves from experi-ence. The modifying agents can be humanobservers or the art piece itself. Modernmachine learning methods and access tobig data over the Internet are making suchart pieces highly complex but also muchless predictable.

The mental process that takes place when anobserver confronts an art piece can thereforealready be considered as an interactive processin the sense that the mental process in the obser-ver is a response to an art piece. With the adventof computer technology and evolution of newmedia installation art this primary reaction toengagement with art can be additionally mediatedby computers. Various sensors, cameras in par-ticular, are used in this perception loop thatmakes possible a wide variety of additional inter-active interventions that can be more directlyobserved and experienced by the observers.

In our research team we have produced in thepast several interactive art installations where theinteractive intervention of observers can rangefrom a simple presence of a human observer inthe view range of a camera to active and purpo-seful physical actions initiated by a visitor. In theinstallation ‘15 seconds of fame’, a camera is usedto detect the presence of observers to be able toturn the images of their faces into pop-art por-traits (Solina 2004). In the installation ‘Schoolfor cats’, a user directly manipulates pieces of ajigsaw puzzle on a touch screen, where theimages for the puzzles were made by a personwith severe learning difficulties and the installa-tion itself is intended foremost for art therapy(Pavlin et al. 2015). Both installations can beclassified as dynamic-interactive.

What is the nature of the interaction in the‘Light fountain’ art installation? The signalsfrom the environment that the system catchesand processes are the 3-D shape of the stonesculpture and any obstacles above the stone’ssurface. This function is performed by the

Kinect depth sensor. Water droplets that appearas rain in the installation are represented bylight points which are generated by the compu-ter system in such a way, as if actual water dro-plets would run down the faces of the sculpture.Each water droplet among the current 50,000droplets is simulated independently and itsbehaviour is dependent on its previous stateand the slant of the corresponding surfacepatch. This loop runs without a human inter-vention, even when nobody is present, designat-ing it as a dynamic-passive art piece. However,an observer can always intervene in this loopby reaching with her hand into the spaceabove the sculpture, modifying in this way the3-D shape detected by Kinect and provokingin turn a change in the flow patterns of simu-lated water droplets. This is possible becausethe 3-D shape is continuously monitored bythe Kinect sensor. This aspect gives the artpiece a dynamic-interactive nature since obser-vers can also manipulate the flow of virtualwater drops, adding in this way also an interac-tive dimension to the primary experience ofpure observation.

We have demonstrated with this project howthe Kinect sensor can be used to virtuallyenhance a stone sculpture with a physicallyplausible animation, which is a novel use ofKinect in interactive art. We hope that wehave in this way successfully fulfilled our initialintention to make a static stone sculpture moredynamic and alive using virtual water dropletsinstead of actual water.

Notes

1. http://www.rchoetzlein.com/fluids32. http://www.nvidia.com/object/cuda_home_

new.html

Acknowledgements

Blaž Jeršan and Gorazd Rajar assisted in codingduring the early stages of the project. We thank Bošt-jan Perovšek for producing unique music, using

100 F. SOLINA AND B. MEDEN

natural water sounds, to accompany the art piece. Allphotography in this article is by Franc Solina.

Disclosure statement

No potential conflict of interest was reported by theauthors.

Notes on contributors

Franc Solina is a Professor of Computer Science atthe University of Ljubljana, Slovenia. He receivedhis Dipl. Ing. (1979) and MS (1982) degrees in Elec-trical Engineering from the University of Ljubljanaand his PhD (1987) degree in Computer and Infor-mation Science from the University of Pennsylvania.Since 1988, he is teaching at the University of Ljubl-jana, now both at the Faculty of Computer and Infor-mation Science and the Academy of Fine Arts andDesign. In 1991, he founded the Computer VisionLaboratory at the Faculty of Computer and Infor-mation Science. His main research interests are 3-D modelling from images and the use of computervision in human–computer interaction, in particularin art installations.

BlažMeden is a PhD Student of Computer and Infor-mation Science at the University of Ljubljana. Hisinterests are various topics in computer science, suchas interactive user interfaces, human to computerinteraction, rendering and graphics, and applicablecomputer vision in general, including the usage of var-ious hardware gadgets (cameras and sensors).

ORCID

Franc Solina http://orcid.org/0000-0002-9268-6825

References

Adzhiev, Valery, Peter Comninos, and AlexanderPasko. 2003. “Augmented Sculpture: ComputerGhosts of Physical Artifacts.” Leonardo 36 (3):211–219.

Andersen, Michael Riis, Thomas Jensen, PavelLisouski, Anders Krogh Mortensen, MikkelKragh Hansen, Torben Gregersen, and PeterAhrendt. 2012. Kinect Depth Sensor Evaluationfor Computer Vision Applications. TechnicalReport. Aarhus University, Department ofEngineering.

AS 2015. 2015. “Artium Speculum 2015.” [Online].Trbovlje. Accessed 18 October 2015. http://speculumartium.si.

Barmpoutis, Angelos, Eleni Bozia, and DanieleFortuna. 2015. “Interactive 3D Digitization,Retrieval, and Analysis of Ancient Sculptures,Using Infrared Depth Sensors for MobileDevices.” In Universal Access in Human–Computer Interaction. Access to the HumanEnvironment and Culture, Lecture Notes inComputer Science, Vol. 9178, edited by Antona,Margherita, Stephanidis, Constantine, 3–11. LosAngeles, CA: Springer.

Brockmeyer, Eric, and Jake Marsico. 2014. “MakingThings Interactive: Self Organizing Still-life(SOS) by David Fried, 2007 Onwards.” [Online].Accessed 1 September 2016. http://mti.code.arc.cmu.edu/2014/?p=4129.

Cornock, Stroud and Ernest Edmonds. 1973. “TheCreative Process Where the Artist Is Amplifiedor Superseded by the Computer.” Leonardo 6(1): 11–16.

“Creative Applications Network – All Posts Tagged:Kinect.” [Online]. Accessed 1 September 2016.http://www.creativeapplications.net/tag/kinect/.

Danforth, Loring M. 2010. Ancient Macedonia,Alexander the Great and the Star or Sun ofVergina: National Symbols and the ConflictBetween Greece and the Republic of Macedonia.Wiley Online Library.

“Develop Kinect.” [Online]. Accessed 1 September2016. http://developkinect.com.

Edmonds, Ernest, Zafer Bilda, and Lizzie Muller.2009. “Artist, Evaluator and Curator: ThreeViewpoints on Interactive Art, Evaluation andAudience Experience.” Digital Creativity 20 (3):141–151.

Fried, David. 2011. “Self Organizing Still-Life.”[Online]. Accessed 1 September 2016. http://www.davidfried.com.

Lehanneur, Mathieu. 2016. “LiquidMarble.” [Online].Accessed 20 October 2016. http://www.dezeen.com/2016/09/16/mathieu-lehanneur-liquid-marble-vanda-sculpture-installation-london-design-festival-2016/.

Meden, Blaž, Blaž Jeršan, Gorazd Rajar, and FrancSolina. 2015. “Virtually Enhanced Sculpture.”ROSUS 2015 : računalniška obdelava slik innjena uporaba v Sloveniji, edited by BožidarPotočnik, 81–86. Maribor: Fakulteta za elektroteh-niko, računalništvo in informatiko.

Mimicry. 2015. “Project Mimicry.” [Online].Accessed 1 September 2016. http://mimicry.monobanda.nl.

DIGITAL CREATIVITY 101

Noguchi, Isamu. 1986. “Water Stone.” [Online].Accessed 1 September 2016. http://www.metmuseum.org/art/collection/search/44520.

Obscura. 2016. “Waterfall, Created for SalesforceLobby, San Francisco.” [Online]. Accessed 1September 2016. http://obscuradigital.com/work/salesforce-video-wall/.

Pavlin, Erika, Žiga Elsner, Tadej Jagodnik, BorutBatagelj, and Franc Solina. 2015. “FromIllustrations to an Interactive Art Installation.”Journal of Information, Communication andEthics in Society 13 (2): 130–145.

Penven, Audrey. 2011. “Dancing with InvisibleLight.” Coilhouse. [Online]. Accessed 1September 2016. http://coilhouse.net/2011/03/audrey-penven-dancing-with-invisible-light/.

Prochaska, W. 2013. “A Sculptural Marble of PrimeQuality in Antiquity – The Dolomitic Marble ofthe Sivec Mountains in Macedonia.”Archaeometry 55 (2): 179–197.

Ravnik, Robert and Franc Solina. 2013. “Interactiveand Audience Adaptive Digital Signage UsingReal-time Computer Vision.” InternationalJournal of Advanced Robotic Systems 10: 1–7.

Reed, S. E., O. Kreylos, S. Hsi, L. H. Kellogg, G.Schladow, M. B. Yikilmaz, H. Segale, J.Silverman, S. Yalowitz, and E. Sato. 2014.“Shaping Watersheds Exhibit: An Interactive,Augmented Reality Sandbox for Advancing

Earth Science Education.” AGU Fall MeetingAbstracts, Vol. 101.

SandyStation. 2014. “SandyStation The InteractiveSandbox.” [Online]. Accessed 1 September 2016.http://en.sandystation.cz/.

Solina, Franc. 2004. “15 Seconds of Fame.” Leonardo37 (2): 105–110.

Solina, Franc. 2015a. “Light Fountain – AnInteractive Art Installation.” [Online]. Accessed1 September 2016. https://youtu.be/nqituTE-_SI.

Solina, Franc. 2015b. “Light Fountain – With anObstacle.” [Online]. Accessed 1 September 2016.https://youtu.be/XOwh2ri2rbo.

St. Jean, Jared. 2012. “Augmented Reality EvolutionTimeline Using openFrameworks.” [Online].Accessed 1 September 2016. http://developkinect.com/news/augmented-reality/augmented-reality-evolution-timeline-using-openframeworks.

Strehovec, Janez. 2009. “New Media Art as Research:Art-making Beyond the Autonomy of Art andAesthetics.” Technoetic Arts 6 (3): 233–250.

Wu, Yirui, Tong Lu, Zehuan Yuan, HaoWang. 2016.“FreeScup: A Novel Platform for AssistingSculpture Pose Design.” IEEE Transactions onMultimedia, in press. doi:10.1109/TMM.2016.2609407.

Zhang, Zhengyou. 2012. “Microsoft Kinect Sensor andIts Effect.” IEEE MultiMedia 19 (2): 4–10.

102 F. SOLINA AND B. MEDEN