SHARED IMMERSIVE ENVIRONMENTS FOR PARAMETRICMODEL MANIPULATION

Evaluating a Workflow for Parametric Model Manipulation from WithinImmersive Virtual Environments

DOMINIC HAWTON1, BEN COOPER-WOOLEY2,JORKE ODOLPHI3, BEN DOHERTY4, ALESSANDRA FABBRI5,NICOLE GARDNER6 and M. HANK HAEUSLER71,4,5,6,7UNSW / Computational Design1,5,6,7{d.hawton|a.fabbri|n.gardner|m.haeusler}@unsw.edu.au4Ben_Doherty@bvn.com.au2,3Arup Digital Sydney2,3{ben.cooper-woolley|jorke.odolphi}@arup.com

Abstract. Virtual reality (VR) and augmented reality (AR)provide designers with new visual mediums through which tocommunicate their designs. There is great potential for these mediumsto positively augment current visual communication methods byimproving remote collaboration. Enabling designers to interact withfamiliar computational tools through external virtual interfaces wouldallow them to both calibrate design parameters and visualise parametricoutcomes from within the same immersive virtual environment. Thecurrent research outlines a workflow for parametric manipulation andmesh replication between immersive applications developed in theUnity game engine and McNeel’s Grasshopper plugin. This paperserves as a foundation for future research into integrating design toolswith external VR and AR applications in an effort of enhancing remotecollaborative designs.

Keywords. Augmented Reality; Virtual Reality; Parametric Design;Procedural; Grasshopper.

1. Research Aims and MotivationCommunication can be defined as an external expression of human cognition. Thenotion of “cognitive artefacts”, initially coined in reference to human-computerinteractions (Norman, 1992), translates to design artefacts used to communicate acreative intent. Effective representation and communication are determinative ofthe designer’s cognitive process and the functional process of the ultimate outcome,while being at the crux of successful collaborative design thinking (Abdelmohsen,2012). Shared immersive environments extend this notion of the cognitiveartefact and provide an alternative form of visual communication for remotedesign collaborators that aid in reducing miscommunication when conveying

T. Fukuda, W. Huang, P. Janssen, K. Crolla, S. Alhadidi (eds.), Learning, Adapting and Prototyping,Proceedings of the 23rd International Conference of the Association for Computer-Aided ArchitecturalDesign Research in Asia (CAADRIA) 2018, Volume 1, 483-492. © 2018 and published by the Associationfor Computer-Aided Architectural Design Research in Asia (CAADRIA) in Hong Kong.

484 D. HAWTON ET AL.

design intent. The current project aims to enable designers to both visualise andmanipulate their designs from within immersive environments. Virtual Reality(VR) and Augmented Reality (AR) technologies are already enabling architects,engineers and designers to visualise their work in unparalleled immersive visualmediums (Trimble, 2017; IrisVR, 2017). Both these technologies have thepotential to augment the way designers conceptualise and communicate their intentwhen collaborating remotely and are currently being applied in the aerospace,medical and maintenance industries accordingly (Microsoft HoloLens 2017). Thecurrent research aims to determine how immersive virtual environments arecurrently being employed in the built environment industries and to examinepotential use cases for the future, outlining a model for visualisation and modelmanipulation in VR and AR applications. The investigation should serve asa foundation for further research into the benefits of using immersive virtualenvironments for remote collaboration in real-world parametric design workflows.

2. Research Observations and ObjectivesEffective collaboration is dependent on effective communication; this is pertinentfor collaborative designers, as the architectural intent is often miscommunicated,leading to a waste of time and resources (Slater et al. 2012). Research into waysfor improving visual design communication is therefore especially valuable fordesigners, and through a combination of different visual communication methodsit is expected that fewer mistakes will be made in built environment projects dueto the enhanced understanding provided by VR and AR. The current researchoutlines a preliminary model for sharing mesh geometry between Grasshopperand other connected AR and VR Unity applications via a cloud server. Cloudservers have previously been used to transmit BIM data (Afsari et al. 2016),and exchange data between BIM and VR (Kieferle et al. 2014). However,at the time of writing, exchanging mesh data between Grasshopper and VRand AR applications has been relatively unexplored. This workflow will allowdesigners to expose and manipulate parameters on a virtual interface and visualisethe resulting mesh within the same immersive environment. By using a webserver as an intermediary between the Unity and Grasshopper environments, datacan be transferred bidirectionally, meaning that parametric updates in the Unityenvironment mirror those of the Grasshopper environment. Eventually, the virtualenvironments can be shared by collaborators in remote locations, enhancing visualdesign communication on parametric projects.

3. Research QuestionVR and AR, as visual mediums, are unique in their ability to conveyscale and perspective and improve design engagement through interaction.Direct reciprocity and manipulation have been shown to promote collaborativedesign communication and understanding, thus demonstrating the importanceof incorporating immersive environments into designers workflows (Gauglitzet al. 2014). Virtual environments have also shown to encourage a moreuser-centric professional approach in university architectural studios by allowingstudents to inhabit the buildings they are designing (Moleta 2016). The current

SHARED IMMERSIVE ENVIRONMENTS FOR PARAMETRIC MODELMANIPULATION

485

research, therefore, aims to answer the question of whether shared immersiveenvironments can enhance parametric workflows. Algorithmic design is widelyused within architectural and design studios, therefore exploring how parametricdesign communication can be improved through immersive virtual environmentsis an important area of exploration. As immersive environments provide a uniqueplatform for collaboration and shared visualisation, it is anticipated that parametricworkflows would benefit from improvements in design communication.

4. MethodologyIn order to evaluate the viability of collaborative virtual environments, a modelhas been developed to demonstrate how users in different virtual or augmentedsurroundings can interact with the same parametric model. The process forexchanging data between the immersive mediums and Grasshopper comprisesfour key sections. First, the anatomy of a mesh will be explained, as it is crucialfor understanding how the mesh is organised into a transmittable data structurein Grasshopper and procedurally reconstructed within the Unity engine. Theinvestigation will then outline how to expose user inputs in the virtual spaces forenabling designers to manipulate values in the Grasshopper script. The valueswill be sent to Grasshopper via a web server to compute any model changes.Accordingly, the updated Grasshopper model data will be sent to all connectedvirtual spaces and reconstructed for visualisation. An evaluation of the proposedmethod and suggestions for further explorations are provided as part of the currentinvestigation.

5. Background ResearchShared collaboration in VR and AR is relatively new. However, several relatedprecedents already exist.

FIREFLY GRASSHOPPER PLUGIN. The Firefly Grasshopper plugin allowsdesigners to connect input/output devices and sensors to the Grasshopper platform.Designers can use sensor data or manipulate actuators such as motors from theirscripting environment. They can manipulate parameters within their Grasshopperenvironment using physical inputs, such as sliders, buttons or dials. The precedentdemonstrates how external data can be sent to and captured in Grasshopper, whichis a central part of the project outlined in this paper.

TRIMBLE SKETCHUP VIEWER. Though the HoloLens is relatively newthere are already several applications targeting architectural visual communication.Trimble’s Sketchup Viewer is one of these applications, allowing users to viewtheir Sketchup models in the HoloLens through a cloud server that users canupload to, store and download their projects from (Trimble, 2017). Users canthen navigate around the model and perform various model manipulations such asscaling, rotating and isolating elements.

IRIS VR. VR technology has recently become more common and is nowused as a visualisation tool within the architecture industry. IrisVR is aplatform developed specifically for architects who wish to easily and efficientlyvisualise their projects in VR (IrisVR, 2017). The application allows users

486 D. HAWTON ET AL.

to push their Revit or Sketchup models to the IrisVR platform easily andefficiently, removing the technical and tedious processes commonly associatedwith visualising architecture in VR.

GOOGLE VR BLOCKS. Is a VR 3D modelling application that provides animmersive environment for creating and experiencing 3D models (Google VR,2017). Artists can design and create their models in VR and export them out forrendering or use in other software. Modelling in VR provides users with a uniquesense of scale and immersion, which allows artists and designers to more closelyconvey their original design intent.

URBAN PINBOARD BI-DIRECTIONAL PIPELINE. Urban Pinboard is anonline urban design collaboration platform in development that aims to include thepublic, private and community members in a digital conversation for future smartcity initiatives (Johanson et. al, 2017). As part of this initiative a bidirectionalpipeline between online data repositories and Grasshopper was developed toautomate the computation of environmental analysis, in this case the amount ofsunlight hours buildings are receiving in their urban context. Users can selectgeographical locations from a web interface and receive back an SVG representingthe sunlight hours received for buildings in this area. Innovative tools such as thisempower architects and designers by automating data exchange between differentplatforms, making them more accessible to clients and collaborators.

PLATYPUS GRASSHOPPER PLUGIN. The Platypus Grasshopper plugin isan innovative web collaboration tool developed by CORE Studios. Multiple userscan join a Platypus web session, changing parameter’s variables and viewing theGrasshopper model being streamed from the Grasshopper environment to eachclient’s web browser (COREStudio 2017). This plugin provides an invaluable toolfor architects and designers looking to collaborate remotely, while only requiringaccess to a web browser.

6. Case StudyThe current case study outlines a model for multiple immersive virtual spacesvisualising and interacting with the same parametric models through theirrespective interfaces. The prototype and study involve exposing the parametersfor users to interact within the Unity VR or AR application. Data from the inputvalues is sent to Grasshopper via a web server to compute any change. Changesare then elaborated in Grasshopper and the mesh data sent back to the Unity VRor AR application. Consequently, the mesh is procedurally reconstructed throughthe data received from the Grasshopper environment.

SHARED IMMERSIVE ENVIRONMENTS FOR PARAMETRIC MODELMANIPULATION

487

Figure 1. An early conceptual diagram of the prototype, demonstrating the flow of data fromthe immersive environment to Grasshopper and back again.

6.1. UNDERSTANDING THE ANATOMY OF A MESH

The model will be exchanging mesh data between McNeel’s Grasshopper pluginfor Rhinoceros 3D and Unity, a popular gaming engine and currently the mostaccessible way for developers to develop applications for both VR and AR devices.It is beneficial to start by understanding the anatomy of a mesh and how meshinformation from GH can be used to procedurally create mesh geometry in Unity.

Figure 2. The anatomy of a mesh quad, highlighting the mesh characteristics that are needed tostore and retrieve mesh data.

Meshes in Grasshopper are defined by a Face-Vertex data structure (ModeLab, 2017), meaning that groups of vertices-or points in the 3D-space are groupedtogether to define polygons or triangles. To create meshes using a list of vertices,Grasshopper needs instructions that define the structure of the mesh in terms of itsindividual triangles and those triangle’s individual vertices. Grouping the vertices’positional vectors into groups of three provides Unity or Grasshopper with theinformation required to recreate a complete mesh by procedurally creating itsindividual mesh triangles in a sequential manner. The custom mesh structureused for the case study was developed by Junichiro Horikawa, whose pluginfor communicating mesh data between Grasshopper and Unity was used as thefoundation for the current workflow. The custom mesh structure comprisesa list of vertices, UV coordinates, normals and faces (Horikawa, 2017). Themesh parameters are extracted from the Grasshopper mesh using methods in

488 D. HAWTON ET AL.

McNeel’s Rhino.Geometry API and organised into Horikawa’s custom structurefundamental to themesh reconstruction in Unity. Serialisation involves convertingan object into a stream of bytes which can be more easily transmitted, being idealfor communicating the custom mesh and transmitting the corresponding data toour server and to the Unity immersive application (Microsoft, 2015).

6.2. CREATE AN APPLICATION IN UNITY TO SEND AND RECEIVE DATAFROM

First, creating a Node.js web server for relaying data between Grasshopperand Unity. A Node.js web server was used to relay data between the twosoftware environments (clients). The server receives the input values from theUnity application and sends them to the custom Grasshopper components. Italso listens out for the mesh data message from Grasshopper and relays it toUnity, being responsible for recognising WebSocket events from Grasshopperand Unity and filtering the appropriate data.The server relies on several javascriptlibraries in order to route data between the two separate platforms. Express.jsis a minimal web framework for Node.js applications, which is used for basicrouting; Socket.IO is used for realtimeWebSocket communication and this is howGrasshopper and Unity communicate with each other; Http.js is HTTP requestlibrary suggested by the Socket.IO documentation

Figure 3. Websockets listen out for input changes in Unity and relay new values toGrasshopper for computation before returning the updated mesh.

The Unity project uses a few assets from the Asset Store, namely a Socket.IOplugin for connecting to our server, the Procedural Toolkit for procedurallyreconstructing our mesh, and the VR Toolkit (VRTK) for VR functionality. It alsouses the Windows Mixed-Reality Toolkit for its HoloLens libraries and prefabs(Microsoft, 2017). In order to allow users to manipulate values in Grasshopperfrom the Unity environment, an input method must be provided. The currentcase study will use the standard Unity sliders, as they suitably replicate slidersfound in Grasshopper. The process would be similarly reproduced both in VR andAR projects that require numerical user input. The slider can then be convertedinto a Unity Prefab, which, in turn, can be used in either a VR or AR Unityapplication. Unity sliders provide several adjustable parameters out of the box,such asminimum, maximum and starting values. These value ranges should reflectthe desired value ranges for the parameters within Grasshopper. Each time a slider

SHARED IMMERSIVE ENVIRONMENTS FOR PARAMETRIC MODELMANIPULATION

489

value is manipulated, a function called SendData is fired off, which packages thevalue to be sent into a C-sharp Dictionary. The value is subsequently sent to theserver in the form of a JSON object, which is the format used to parse it on theserver side.

Second, using Socket.IO to listen for changes within Unity and broadcast themback to the server. Socket.IO is a popular javascript framework for harnessingthe power of WebSockets, which allow for interactive communication betweenclients and a server (Mozilla Developer Network, 2017). The clients, in this case,are the custom Grasshopper components and the Unity immersive environments,while Node.js is the foregoing mentioned server. Using WebSockets, the servercan listen out for events fired off from either the Unity sliders or the GrasshopperSend Mesh components and relay that information to the appropriate client.

6.3. BUILDING THE GRASSHOPPER PLUGIN TO RELAY DATA TO THEHOLOLENS APPLICATION

First, defining the Grasshopper components needed to send and receive data isrequired. The Grasshopper plugin is constructed within several components asa fine adaptation of an open-source mesh streaming library written by JunichiroHorikawa (Horikawa, 2017). The components use several external libraries forSocket.IO, JSON interpretation and mesh serialization, which are: ZeroFormatter,used for serialization of the mesh into binary data that is sent from Grasshopper toUnity; Json.NET, used to store the binary mesh data in a JSON format that can beparsed on the server; Socket.IO Client Library for .Net, used to communicate tothe Socket.IO backend server. The mesh streaming plugin contains the followingcomponents: connect, receive data, serialise mesh and send mesh (Fig. 4).

Figure 4. Components of the mesh streaming plugin, from left to right (a) Component used toconnect to the server; (b) Component that listens for a specific predefined message on theserver and outputs the data value received; (c) Component that takes a mesh as an input andserializes it into bytes to be sent to our server; and (d) Component that is used to send the

serialised mesh data to the server.

Secondly, connecting to the server is required. To connect to the server, a URLmust be provided to the Address Input of the Connect node. A boolean toggle isemployed to initiate the connection to the WebSocket on the server. The Herokucloud platform was used to host the server for the current model, as it provides afree hosting tier for small servers. Cloud hosting is essential for accessibility, andas the server requires access from both AR and VR devices, using a local serveron each device is not advised.

Thirdly, receiving numerical data from the Unity application is required. Everytime a user changes the values of a slider, a message is sent to the server toindicate that the values have been changed. This message is then relayed to the

490 D. HAWTON ET AL.

Grasshopper plugin outlined in Section 6.3.1, along with the value that requires achange. For example, if the script being manipulated contains a parameter calledNumberOfFloors, and the ReceiveData Grasshopper node receives a value of 4along with the event name NumberOfFloors, that value can be used to increase thenumber of floors in the building from the previous value to the new value of 4. Thiswould then lead to a change inmesh output connected to the SendMesh component,which would fire off a separate event with a predefined name of Grasshopper (thiscould be anything, as long as it matches the event name defined in the server script),carrying the serialised custom mesh object.

Figure 5. A Grasshopper model being manipulated and visualised from within the VRenvironment.

6.4. REASSEMBLING THE GRASSHOPPER MESH IN UNITY

Using the information from the custom mesh object passed into Unity fromGrasshopper, the mesh can now be procedurally reconstructed. Using the MeshStreaming components from Horikawa. the script iterates over the vertex, UVcoordinates, normal and face information in the Custom Mesh object, rebuildingthe mesh face-by-face. The Procedural Toolkit, an open source library availableon the Unity Asset Store and developed by Daniil Basmanov, is used to interpretthe mesh data received from Grasshopper and to generate a mesh that mirrors theone counteracting in Grasshopper (Basmanov, 2017).

6.5. EVALUATION AND LIMITATIONS

What are potential use cases and limitations of this research? This paperoutlines an experimental workflow to test the viability of real-time data transferfor collaborative design in VR and AR applications. It was anticipated thatthis would be useful for collaborators who are in different physical locationswishing to visualise and manipulate Grasshopper models in either VR or ARmediums, or a combination of both. Aiming to be viable for the industry, theworkflow required reliability, user accessibility, and distinct advantages overother collaborative communication tools.Through developmental testing, andcomparisons with alternative methods for parametric design communication, theworkflow, in its current form, has demonstrated limited viability for real-worldusage due to technical limitations. These include not being able to dynamically addor remove variables on the server or within the Unity applications and applicationcrashes with large mesh transfers, leading to the conclusion that a native VR/AR

SHARED IMMERSIVE ENVIRONMENTS FOR PARAMETRIC MODELMANIPULATION

491

parametric application that removes the need for bi-directional communicationwould likely be much more beneficial to designers collaborating remotely onparametric design. Due to time limitation, few fundamental implementation ofthe model functionality were not achieved. Between those, the most importantwas deploying the application to the HoloLens device. Specifically, the Socket.IOimplementation that worked in the Unity HoloLens emulator and the Unity VRapplication did not work when deployed to the HoloLens. The Microsoft’sUniversal Windows Platform (UWP) uses its own proprietary WebSocket classto communicate with server WebSockets, preventing a seamless integration withthe Socket.IO library, which was realised late in the project. Additionally,further quantitative and qualitative research could be conducted to determine howadvantageous design collaborators would find this tool in a real-world scenario.

7. Significance of Research and ConclusionThis research serves as the basis for future research relating to emerging mediumsfor visual design communication. Understanding how data and geometry canbe passed between applications provides designers with the tools they need toexperiment with cross-application data manipulation. This has the potential toreduce the time taken to accurately communicate original design intent through theefficient exchange of data from design environments to visualisation environments.The prototype developed for this paper has several key limitations that prevent itfrom being used in a commercial setting, as discussed in Section 6.5. With moretime these issues could likely be resolved, however user testing is still required todetermine its viability and usefulness.

It is expected that architects and designers would benefit from workflowsthat integrate currently used parametric design tools with emerging immersivetechnologies, such as VR and AR. What is not clear is how large projects withvast datasets can efficiently exchange data without significant delays betweeninteractions. The prototype presented in this paper enables designers to interactwith and visualise their Grasshopper designs from the same immersive mediumfrom which they are situated. While this worked adequately for smaller designs,it became slow and unresponsive when meshes began to grow beyond a certainsize. This is compounded if multiple users are making changes to the modelsimultaneously, which led to lengthy delays and applications crashes. This is likelydue to the procedural mesh creation method used to assemble the mesh each timea new mesh is sent over from Grasshopper into the Unity application. Perhapsthis algorithm could be improved, however it is likely that a native VR or ARparametric application would produce the most optimum results. With the releaseof Microsoft’s Windows Mixed Reality VR Headsets (Windows MR 2017) thatare part of the same ecosystem as Microsoft’s HoloLens, developing a parametricdesign platform accessible from both VR and AR environments seems the mostsensible path forward.

References“Microsoft: Serialization (C# ). [online]” : 2015. Available from <https://docs.microsoft.com/

en-us/dotnet/csharp/programming-guide/concepts/serialization/> (accessed 7. Nov 2017).

492 D. HAWTON ET AL.

“Windows MR (2017). Windows Mixed Reality | AR Mixed with VR Gaming, Travel &Streaming in Windows 10. [online]” : 2017. Available from <https://www.microsoft.com/en-au/windows/windows-mixed-reality> (accessed 7. Nov 2017).

“Mozilla Developer Network: WebSockets. [online]” : 2017. Available from <:https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API> (accessed 1. Oct 2017).

“Microsoft: Microsoft/MixedRealityToolkit-Unity. [online]” : 2017. Available from <https://github.com/Microsoft/MixedRealityToolkit-Unity> (accessed 4. Nov 2017).

“Mode Lab: What is a Mesh? | The Grasshopper Primer (EN). [online]” : 2017. Availablefrom <http://grasshopperprimer.com/en/1-foundations/1-6/1_What%20is%20a%20Mesh.html> (accessed 1 Oct 2017).

“CORE Studio Platypus” : 2017. Available from <http://core.thorntontomasetti.com/platypus/> (accessed 30. Oct 2017).

“Google VR Blocks - Create 3D models in VR - Google VR.” : 2017. Available from <https://vr.google.com/blocks/> (accessed 8. Nov 2017).

“Microsoft HoloLens” : 2017. Available from <https://www.microsoft.com/en-au/hololens/partner-program> (accessed 4. Nov 2017).

“IrisVR - Virtual Reality for Architecture, Engineering, and Construction” : 2017. Availablefrom <https://irisvr.com/> (accessed 1. Oct 2017).

“Trimple Sketch up Viewer” : 2017. Available from <https://www.sketchup.com/products/sketchup-viewer> (accessed 7 Nov. 2017).

Abdelmohsen, S.: 2012, Genres of communication interfaces in bim-enabled architecturalpractice, Ascaad CAAD | INNOVATION | PRACTICE [6th International ConferenceProceedings of the Arab Society for Computer Aided Architectural Design, Manama,Bahrain, 81-91.

Afsari, K., Eastman, C. and Shelden, D.R.: 2016, Data Transmission Opportunitiesfor Collaborative Cloud-Based Building Information Modeling, XX Congress of theIberoamerican Society of Digital Graphics, Buenos Aires, Argentina, 907-913.

Basmanov, D.: 2017, “Syomus/ProceduralToolkit. [online]” . Available from <https://github.com/Syomus/ProceduralToolkit> (accessed 4. Nov 2017).

Gauglitz, S.: 2014, In touch with the remote world: Remote collaboration with augmentedreality drawings and virtual navigation’, Proceedings of the ACM Symposium on VirtualReality Software and Technology, VRST, pp. 197–205.

Horikawa, J.: 2017, “Mesh Streaming Grasshopper. [online]” . Available from <https://github.com/jhorikawa/MeshStreamingGrasshopper> (accessed 1. Oct 2017).

Jabi, W.: 2013, Parametric Design for Architecture, Laurence King , London, United Kingdom.Johanson, M., Khan, N., Asher, R., Butler, A. and Haeusler, M.: 2017, Urban Pinboard -

Establishing a Bi-directional Workflow Between Web-based Platforms and ComputationalTools, Proceedings of the 22nd CAADRIA Conference, Suzhou, China, 715-724.

Kieferle, J. and Woessner, U.: 2014, BIM Interactive - About Combining BIM and VirtualReality A Bidirectional Interaction Method for BIM Models in Different, Proceedings ofeCAADe 2014, Newcastle upon Tyne, UK, 69-75.

Merel, T.: 2017, “The reality of VR/AR growth” . Available from <https://techcrunch.com/2017/01/11/the-reality-of-vrar-growth/> (accessed Accessed 21 Sep. 2017).

Moleta, T.: 2016, Game on: Exploring constructive design behaviors through the use ofreal-time virtual engines in architectural education, International Journal of ArchitecturalComputing, 14(3), 212-218.

Norman, D.: 1992, Design principles for cognitive artifacts, Research in Engineering Design,4(1), 43-50.

Rankohi, S. and Waugh, L.: 2013, Review and analysis of augmented reality literature forconstruction industry’, Visualization in Engineering, 1(2005), 9.

Slater, R. and Radford, A.: 2012, Perceptions in the Australian building industry of deficienciesin architects’ design documentation and the effects on project procurement’, AustralasianJournal of Construction Economics and Building, 8(1), 23-33.