3d visualization library for labview™harotek.weebly.com/.../haro3d_manual_v2.1.0.pdf ·...

3D Visualization Library for LabVIEW™

User guide version 2.12017

Haro3D™ Manual version 2.1

2 of 301

This page was intentionally left blank


3 of 301

Table of ContentsWhite page ......................................................................................................................................................2Copyright Notice .............................................................................................................................................8Software Licenses ..........................................................................................................................................9Version History ..............................................................................................................................................10Safety ............................................................................................................................................................13Security .........................................................................................................................................................14Introduction ...................................................................................................................................................15

Hololens description ...............................................................................................................................16Hololens coordinate system .............................................................................................................17Hololens Emulator ............................................................................................................................19

Kinect V2 description ..............................................................................................................................20Kinect coordinate system .................................................................................................................22

Oculus Rift DK2 description ...................................................................................................................23Installation .....................................................................................................................................................25

Hololens ..................................................................................................................................................26Kinect ......................................................................................................................................................28Rift DK2 ..................................................................................................................................................29

System Requirements ..................................................................................................................................30Haro3D Library .............................................................................................................................................32

Hololens ..................................................................................................................................................34Haro3D application on Hololens .......................................................................................................35

Cursor .........................................................................................................................................36Voice commands ........................................................................................................................38Settings .......................................................................................................................................39Panels and Info Displays ...........................................................................................................41

1 and 2 Button Dialogs ........................................................................................................42Multi Line Dialogs .................................................................................................................44Panels ...................................................................................................................................45

Hololens Gestures ......................................................................................................................47Air tap ...................................................................................................................................48Hold .......................................................................................................................................49Manipulation ..........................................................................................................................50Bloom ....................................................................................................................................51

Programming principles ....................................................................................................................52Connection ..................................................................................................................................53User Events ................................................................................................................................543D Objects ..................................................................................................................................56Space ..........................................................................................................................................57Hologram Sharing .......................................................................................................................58Simple Code Example ................................................................................................................62

Hololens API .....................................................................................................................................65Building an executable with the Hololens API ...........................................................................66Connection Management ............................................................................................................69

Open Hololens ......................................................................................................................70


4 of 301

Close Hololens ......................................................................................................................71Get Number of Objects ........................................................................................................72Connection status .................................................................................................................73Get Events ............................................................................................................................74

Commands ..................................................................................................................................753D Objects ............................................................................................................................76

Move Multiple Objects ....................................................................................................77Move Object ....................................................................................................................78Modify Multiple Materials ................................................................................................79Send Texture Image .......................................................................................................80Add Shape ......................................................................................................................81Modify Multi Shapes .......................................................................................................82Modify Shape ..................................................................................................................83Set Object Visibility .........................................................................................................84

UI ...........................................................................................................................................85Multi Line Display ...........................................................................................................86Info Display .....................................................................................................................87Voice Output ...................................................................................................................88Point to Object ................................................................................................................89Set Object Info ................................................................................................................90Add Voice Commands ....................................................................................................91

Hololens Space .....................................................................................................................92Move Origin ....................................................................................................................93Change Scale .................................................................................................................94Send Anchor ...................................................................................................................95

Sharing ..................................................................................................................................96Update Secondary Origins .............................................................................................97Display Info All Secondaries ..........................................................................................98Move Secondary Origin ..................................................................................................99Voice Output All Secondaries .......................................................................................100

Event data .................................................................................................................................101Extract Mesh Message .......................................................................................................107Extract Object Selected Message ......................................................................................108Extract Hold Message ........................................................................................................109Extract Manipulation Message ...........................................................................................110Extract Text Message .........................................................................................................111Extract None Message .......................................................................................................112Extract Anchor Message ....................................................................................................113Extract Sharing Message ...................................................................................................114Extract Connection Message .............................................................................................115Extract Invalid Message .....................................................................................................116

Controls .....................................................................................................................................117Holo Mesh Data ..................................................................................................................118Anchor Data ........................................................................................................................119

Kinect V2 API .......................................................................................................................................120


5 of 301

Building an executable with the Kinect V2 API ..............................................................................121Audio Beam ....................................................................................................................................122Bodies .............................................................................................................................................123

Single array to body .................................................................................................................125Cloud of points ...............................................................................................................................126Depth ...............................................................................................................................................128Color Image ....................................................................................................................................130Infrared ............................................................................................................................................132Kinect control ..................................................................................................................................134Kinect Fusion ..................................................................................................................................137Kinect API Controls ........................................................................................................................141

Fusion parameters ....................................................................................................................142Status error codes ..........................................................................................................................144

3D Objects ............................................................................................................................................145To LabVIEW 3D Pict ......................................................................................................................1463D object properties .......................................................................................................................148

Material Info ..............................................................................................................................149Position .....................................................................................................................................151Rotation data ............................................................................................................................152Color information ......................................................................................................................153

Shape Class ...................................................................................................................................155Create Box ................................................................................................................................156Create Cylinder .........................................................................................................................157Create Sphere ..........................................................................................................................158Create Panel .............................................................................................................................159Mesh Class ...............................................................................................................................160

Create Mesh .......................................................................................................................161Calculate Normals ..............................................................................................................162CleanUp Mesh ....................................................................................................................163Generate Simple Shape .....................................................................................................164

Controls .....................................................................................................................................166Cylinder parameters ...........................................................................................................167Shape Type .........................................................................................................................168

OBJ Mesh Class .............................................................................................................................169OBJ File Load ...........................................................................................................................170Cleanup OBJ Mesh ..................................................................................................................173OBJ Send to Hololens ..............................................................................................................174OBJ Calculate Normals ............................................................................................................175

Utilities ...................................................................................................................................................1763D Operations .................................................................................................................................177

Rotation angles to matrix .........................................................................................................178Quaternions multiplication ........................................................................................................179Vector rotation by quaternion ...................................................................................................180Vector Rotation .........................................................................................................................181Vector to Rot-Trans Matrix .......................................................................................................182


6 of 301

Euler Angles to Quaternion ......................................................................................................183Rot-Trans Matrix to Quaternion ................................................................................................184Quaternion to Rotation Axis .....................................................................................................185Toggles Right Left System .......................................................................................................186Controls .....................................................................................................................................187

Quaternion ..........................................................................................................................188Kinect Utilities .................................................................................................................................189

Bodies .......................................................................................................................................190Bodies 3D Display ..............................................................................................................191Controls ...............................................................................................................................192

Body info .......................................................................................................................193Joint info .......................................................................................................................195

Cloud of Points .........................................................................................................................197Cloud 3D Display ................................................................................................................198Cloud 2D Display ................................................................................................................199Read Cloud File ..................................................................................................................201Save Cloud File ..................................................................................................................203Controls ...............................................................................................................................205

Cloud data ....................................................................................................................206Mesh Data Functions ...............................................................................................................207

Mesh 3D Display ................................................................................................................208Read Mesh .........................................................................................................................209Save Mesh ..........................................................................................................................211Controls ...............................................................................................................................213

Mesh data .....................................................................................................................214Hand states ...............................................................................................................................215

Hand Gesture Functional Global ........................................................................................216Anchor accumulate .......................................................................................................218Hand average calculate ................................................................................................219

Hand click ...........................................................................................................................220controls ................................................................................................................................221

Hand state info .............................................................................................................222Hand state average ......................................................................................................223Hand state position state ..............................................................................................224Hand config ...................................................................................................................227

sub VIs ......................................................................................................................................228Body Draw limb ..................................................................................................................229Cloud Read PLY .................................................................................................................230Cloud Save PLY .................................................................................................................231Cloud Save XYZ .................................................................................................................232Find End of Line .................................................................................................................233Mesh Read PLY .................................................................................................................234Mesh Save PLY ..................................................................................................................235Mesh Save STL ..................................................................................................................236PLY file Ascii to vertices .....................................................................................................238


7 of 301

PLY file Binary to vertices ..................................................................................................239PLY file Vertices to Ascii ....................................................................................................240PLY file Vertices to binary ..................................................................................................241Triangles - Calculate Normals ............................................................................................242

Oculus Rift DK2 API .............................................................................................................................243Position and Orientation .................................................................................................................244Distortion .........................................................................................................................................245

Examples ..............................................................................................................................................246Hololens ..........................................................................................................................................248

Hololens Events Example ........................................................................................................249Hololens 3D Objects Example .................................................................................................252

Kinect ..............................................................................................................................................256Audio .........................................................................................................................................257Bodies .......................................................................................................................................259Cloud .........................................................................................................................................262Cloud - User Event ...................................................................................................................265Color image ..............................................................................................................................268Infrared ......................................................................................................................................270Fusion .......................................................................................................................................272

Mesh ...................................................................................................................................276Depth .........................................................................................................................................278

OBJ Example ..................................................................................................................................280Change part ..............................................................................................................................283

Oculus Rift DK2 ..............................................................................................................................284Oculus Example ........................................................................................................................285

Support ................................................................................................................................288Hand Translation ..........................................................................................................289Command input ...........................................................................................................290Command output ..........................................................................................................291Closest hand .................................................................................................................292Hand Rotation ...............................................................................................................293Planet texture ................................................................................................................294

Distortion correction ..................................................................................................................295List of Haro3D errors ............................................................................................................................298

References ..................................................................................................................................................299Contact information .....................................................................................................................................301


8 of 301

Copyright Notice Copyright © 2017 by HaroTek LLC. All Rights Reserved Any part of this document must not be reproduced in any way, either printed or electronically, without the explicit andwritten consent of: HaroTek LLC312 Foxcroft Ln.Keller, TX76248 Copyright © 2016 National Instruments Corporation. All Rights Reserved. Haro3D, and HaroTek and Haro3D logos are trademarks of HaroTek LLC. Microsoft, Hololens, Kinect, and Visual C++ are trademarks of Microsoft Corporation. Oculus, Oculus VR, Oculus Rift, and the Oculus eye logo are trademarks or registered trademarks of Oculus VR,LLC. Planet textures in the Oculus_Example.vi are Copyright (c) by James Hastings-Trew (available atplanetpixelemporium.com/planets.html, reference [16]).

http://planetpixelemporium.com/planets.html

planetpixelemporium.com/planets.html


9 of 301

Software Licenses Copyright © 2016 by HaroTek LLC. All Rights Reserved The text of the license is provided in the distribution directory under the name "Haro3D_license.txt". The Open (document-open-2.png) and Save (document-save-5.png) icons of the Cloud and Fusion examples were obtainedfrom the Open Icon Library under the CC-BY-SA-3.0 license.

http://creativecommons.org/licenses/by-sa/3.0/us/legalcode


10 of 301

Version History Manual Version History

Manualversion

Haro3Dlibrary version

Date Modifications

1.0.0 1.0.0 Dec 17, 2014 First release1.2.0 1.2.0 Nov 21, 2015 Added info about new features of Haro3D library 1.2.0.1.2.1 1.2.0 Nov 25, 2015 Updated palette images and added list of examples in description

of library. Added information about how to build an executable withthe Kinect V2 API.

1.2.1.1 1.2.1 Apr 18, 2016 Modified info about the order in which the texture files are loaded.Corrected error on LabVIEW version requirements. Correctederrors on Intrinsics table calculations. Added info about use of pointfor decimal separator in OBJ file loader, cloud and mesh sections.

1.3.0.0 1.3.0 Jul 11, 2016 - Added documentation for Euler_to_Quaternion.vi.- Added documentation for Rot-Trans Matrix to Quaternion.- Added documentation for Cloud - User Event example.- Modified documentation of Cloud of points API.

1.3.0.1 1.3.0 Jul 13, 2016 - Changed requirement for Visual C++ Redistributable (2015instead of 2013).

2.0.0 2.0.0 Apr 30, 2017 Added documentation of the Hololens API.

2.0.1 2.0.0 May 4, 2017 - Added the Hololens word to the Open and Close VIs.- Improved introduction of Microsoft Hololens- Corrected info about User Events.

2.1.0 2.1.0 Sep 6, 2017 - Corrected input parameters table of the Open Hololens VI.- Added info for new Add Voice Commands function.- Added info about completion and canceling data in Extract HoldMessage and Extract Manipulation Message- Modified info on connection User Events.- Added info for Extract Sharing Message and Extract AnchorMessage functions.- Added info for Anchor data- Added info for Send Anchor function- Modified info of Create Panel for the new Tag along parameter.- Added info on Hologram Sharing and Sharing API- Added info for Voice Output All Secondaries- Added info for Move Secondary Origin- Added info for Display Info All Secondaries- Added info for Update Secondary Origins function- Added info for Vector Rotation function.- Added info for Toggles Right Left System function.- Added info for Generate Simple Mesh function.


11 of 301

Haro3D Library Version History


Date Modifications

1.0.0 Dec 17, 2014 First release1.2.0 Nov 21, 2015 - Added Audio beam function to Kinect API

- Added Oculus Rift DK2 API- Added .OBJ file format reader- Added function Acq Cloud to the Depth API- Added the floor plane as the output of the Kinect2 Body API- Fixed a memory leak problem with the Body API- Fixed missing joint in Bodies 3D Display utility- The STL read function as an option to fix the normals- Fixed the 32-bit version of the new function Acq Cloud of the Depth API- Fusion API provides camera matrix

1.2.1 Apr 18, 2016 - OBJ file loader now automatically normalizes the normals.- Fixed an issue with the repetitive texture in the OBJ file loader.- Memory used by Kinect2_Cloud_API.vi is now limited to size of cloud selectedat initialization. Cannot change size of cloud anymore after initialization (no moreghost).- OBJ file loader and ASCII cloud and mesh files are loaded and saved usingthe point as decimal separator independently from the computer and LabVIEWsettings.

1.3.0 Jul 11, 2016 - Added VI Euler_to_Quaternion.vi- Added VI Rot-Trans Matrix to Quaternion.- Added Cloud - User Event example.- Added LabVIEW user event capability to Cloud of points API.- Changed configuration of dll of Cloud of points API from "run in UI thread" to"run in any thread".

2.0.0 Apr 30, 2017 - Added the Hololens API.- Tools for creation of 3D Objects for the Hololens and LabVIEW 3D PictureObjects

2.0.1 May 5, 2017 - New Add Voice Commands function.2.1.0 Sep 6, 2017 - Manipulation status and hand position are provided with Extract Manipulation

Message.- Hold status provided in Extract Hold Message.- New Extract Anchor Message function- New Extract Sharing Message function.- New Extract Connection Message function.- Modified data associated with Connection event.- New Send Anchor function- Modified Create Panel to accept new tag along parameter.- New Voice Output All Secondaries function- New Move Secondary Origin function


12 of 301

- New Display Info All Secondaries function- New Update Secondary Origins function- New Vector Rotation function.- Modified Extract Manipulation Message to take orientation of the Hololens intoaccount and to include status and hand position.- New Toggles Right Left System function.- New Generate Simple Shape function.

Haro3D Hololens App Version History


Date Modifications

2.0.0 May 8, 2017 First release2.0.3 May 27, 2017 - Fixed multi line display that kept selected previously selected line

- Fixed modify multi materials that modified only the first object- Added the Add Voice Commands capability- Added special values to indicate completion or canceling in Manipulationevents- Added special values to indicate completion or canceling in Hold events- 1-button dialogs can be locked in place by user

2.1.0 Sep 6, 2017 - Ability to share holograms between multiple Hololens- Ability to share holograms using the same coordinate system.- Manipulation sends displacement relative to Hololens, and status and Handposition.- Hold sends status in addition of cursor normal, cursor position, and objectindex.- Panel can be made still in a given position.- Holograms are occluded by mapped surface, after surface mapping.


13 of 301

Safety Microsoft Hololens The Microsoft Hololens headset is an augmented reality headset, which is less likely to cause discomfort associatedwith virtual reality and immersion. However, discomfort is still possible. In a manner similar to virtual reality headsets,the user might also not see obstacles or objects that he might hit while wearing the headset. Stop using the headsetif you experience any discomfort or do not feel comfortable with the safety of the environment where it is used. Usethe Microsoft Hololens at your own risks. Microsoft Kinect V2 The Microsoft Kinect V2 contains lasers that can cause damages to the eyes. There is currently no specificationabout the output light level of the Kinect. In any case, it is preferable to avoid looking at the Kinect when it is turnedon from a short distance. A distance larger than 30 cm (1 foot) should be safe but HaroTek does not provide anyguarantee. Use the Microsoft Kinect at your own risks. Oculus Rift The Oculus Rift is a headset that can cause serious discomfort associated virtual reality and immersion. In addition,the user might not see obstacles or objects that he might hit while wearing the headset. Stop using the headset ifyou experience any discomfort or do not feel comfortable with the safety of the environment where it is used. Use theOculus Rift at your own risks.


14 of 301

Security Some features of the Haro3D library, notably communication with the Hololens, use network communications.HaroTek does not collect, record, transmit, or make available in any way data from the devices where the library isinstalled. However, network communication between two devices that use the Haro3D application or library can beintercepted by third parties. It is the responsibility of the user to make sure that such network communications aresecured using private networks, appropriate encryption, firewalls, and other security devices and procedures. Whenusing the Haro3D library and application, you accept that HaroTek, National Instruments, or Microsoft cannot be heldliable for any loss of data, loss of revenue, or any other consequences of a breach of communication security, even ifthis breach is caused by a fault in the Haro3D library, application, and other related software.


15 of 301

Introduction

Haro3D is a library of LabVIEW VIs developed by HaroTek to access some functionalities of the Microsoft Hololens,of the Microsoft Kinect V2, and of the Oculus Rift DK2. In addition, the Haro3D library provides utilities to handle 3Dobjects and data. The Hololens is an augmented-reality head-mounted display developed by Microsoft. Haro3D gives access to thefollowing functionalities of the Microsoft Hololens

• Hololens app control from National Instruments LabVIEW• User Interaction through manipulations, voice, and menus, with LabVIEW applications• Upload of holograms (3D objects) from LabVIEW to the Hololens• Real-time display of LabVIEW VI front panels or controls to the Hololens user.

More information about the Microsoft Hololens can be found at www.Hololens.com. The Kinect V2 is a sensor developed by Microsoft for the Xbox game console. Its main goal is to be able to interprethuman positions and gestures. To accomplish this task, the Kinect is equipped with a depth measurement systembased on active illumination. This feature makes the Kinect a low cost three-dimensional camera that can be usedfor applications outside the gaming industry. More information about the Kinect can be found here. Haro3D gives access to the following functionalities of the Kinect V2:

• Audio Beam• Bodies (people and joints tracking)• Colored 3D cloud of points.• Depth sensing• Color high-definition camera• Active infrared imaging• 3D volume reconstruction

The Haro3D library also provides access to the following functionalities of the Oculus Rift.

• Position and Orientation• Distortion

In addition, the Haro3D library also includes Utilities VI's to complement the functionalities of the Kinect V2 like VI'sto display and interact with the 3D data, and VI's to save and read back the 3D data. VIs to analyze hand gesturesand to read .obj format files are also part of this library. Fully functional examples for each of the functionalities arealso provided.

https://www.microsoft.com/microsoft-hololens/en-us


16 of 301

Hololens description The Hololens is an augmented-reality head-mounted display developed by Microsoft. For more information about the Hololens, go to www.hololens.com.

The Hololens projects computer generated images that can be seen by the user. The Hololens is equipped withsensors that map the environment around the user. This mapping is used by the Hololens to track the position of theHololens within its environment. In addition, the Hololens also tracks some of the movements of the user hands. Thosemovements can be used to interact with the application currently running on the Hololens.

The user can point to objects using his head and a cursor he sees in the Hololens, he can select objects or points, usinggestures: air-tap (elevated index flexing down), manipulation (flexed index moving) or hold (flexed index not moving atall). Those interactions are tracked by the Hololens and can be used with the Haro3D API. Description of the Haro3D Hololens API The Hololens API of the Haro3D library from HaroTek is a library of LabVIEW VI's to interface with the MicrosoftHololens. The toolkit establishes a network link between a LabVIEW application running on a desktop computerand a Hololens. The network link is then used to respond to inputs from a user wearing the Hololens and to uploadholograms as LabVIEW 3D objects. Features provided by the Haro3D library to Hololens Users Hololens app control from National Instruments LabVIEW Interaction through manipulations, voice, and menus of Hololens user with LabVIEW applications Upload of holograms (3D objects) from LabVIEW to the Hololens Upload of visual and vocal information from LabVIEW to the Hololens Display of VI front panels and controls to the Hololens user



17 of 301

Hololens coordinate system When starting a session with the Hololens, the origin of the coordinate system is defined as the position of theHololens itself. However, the Hololens attempts to orient the coordinate system using its sensors so that it is orientedwith the Y-axis pointing up, the Z-axis pointing in front of the user and the X-Z plane being parallel to the floor, nomatter what is the original orientation of the Hololens.

Coordinate system of the Hololens at the start of the application. The Z-X plane is made parallel to the floor, as detected by Hololenssensors. By default, the Hololens coordinate systems is left-handed. LabVIEW uses a right-handed coordinate system. Leftand right handed coordinates systems are named based on the hand that can be used to represent the directionsof the positive X-Y-Z axes using the thumb, index, and major (see images below). The same hand can then be alsoused to determine the sign of a rotation, the thumb representing the rotation axis and the fingers the direction of apositive rotation. The conversion from a right-handed coordinate system in LabVIEW to a left-handed coordinate system in theHololens is transparently taken care of by the Haro3D Hololens API. Coordinates and orientations to and from theHololens should be considered as right-handed. When providing a mesh, that mesh should have be built using aright-handed coordinate system. The Haro3D Hololens API converts the mesh into a left-handed one for display inthe Hololens space. If a mesh is built using a left-handed coordinate system, it should be converted prior to sendingit the Hololens.


18 of 301

Right-handed coordinate system

Left-handed coordinate system. Original Hololens image from Microsoft: https://www.microsoft.com/microsoft-hololens/en-us



19 of 301

Hololens Emulator The Haro3D Hololens API is compatible with the Hololens emulator, provided by Microsoft for Visual Studio. Theemulator can only be run on Windows 10 pro. Download the Haro3D application from within the emulator using this link. The emulator requires Microsoft VisualStudio 2015 update 3 or Visual Studio 2017. When running the emulator on the same computer than the one running the LabVIEW application, sets the IPAddress in the Haro3D settings of the Hololens to the address of the Internal Ethernet Port Microsoft Emulator NATSwitch (by default, 192.168.80.1). Do not change the address of the other virtual switch (Internal Ethernet PortWindows Phone Emulator Internal Switch). If the emulator is to be run on a computer different from the one running the LabVIEW application, replace theInternal Ethernet Port Microsoft Emulator NAT Switch by an actual connection using Hyper-V settings. Sets the IPaddress in the Haro3D settings as the address of the computer running the LabVIEW application.

http://go.microsoft.com/fwlink/?LinkID=823018


20 of 301

Kinect V2 description The Kinect V2 is a sensor developed by Microsoft for the Xbox game console. Its main goal is to be able to interprethuman positions and gestures. To accomplish this task, the Kinect is equipped with a depth measurement systembased on active illumination. This feature makes the Kinect a low cost three-dimensional camera that can be usedfor applications outside the gaming industry. The Kinect V2 can be purchased as a Kinect for Windows 2 that includes the hardware to connect it to a computer,or as a Kinect for Xbox One that requires a kit to be purchased separately to be able to connect it to a computer. The Kinect V2 is a new version of the Kinect sensor. The features of the Kinect V2 are given in reference [1]. Asummary of the characteristics according to reference [1] are given below:

• High definition (1920x1080 pixels) color camera running at 30 Hz but down to 15 Hz in low-light conditions.

• Depth sensor: 512x424 pixels at 30 Hz. 3x higher depth fidelity than version 1. Field of view: 70° horizontalx 60° vertical. Range: 0.5 to 4.5 m.

• Active infrared imaging: 512x424 pixels at 30 Hz

• Body tracking: 6 people simultaneously with up to 25 joints tracked per person at 30 Hz.

Additional Kinect information Additional information on the Kinect is available from other sources, among others, reference [2]. Body recognition range is given as 0.5 m to 4.5 m but the depth measurement has a range extending to 8.0 m. Depth accuracy is depth dependent. Using the 3x higher depth fidelity and an accuracy of about 1 cm at 2 m.distance for the Kinect V1 [11], a relative accuracy of 3-5 mm is estimated at a 2-m. distance for the Kinect V2. The Kinect V2 depth sensor is based on a technology different than the first version that used pseudo-randompattern recognition. The Kinect V2 uses Time-of-Flight technology probably based on a Canesta chip [3]. The Kinect V2 appears to have 3 laser diodes (probably at a wavelength of 830 nm). The laser diodes can be seenwhen the Kinect is turned ON. Below is an image of the Kinect V2 where the diodes and LED's are highlighted. The image also shows the expectedlocations of the color camera, infrared camera, laser diodes, and other unknown sensor.


21 of 301

original image from: http://www.microsoft.com/en-us/kinectforwindows/ The information about the location of the sensors in the Kinect V2 is important because the 3D information providedby the Kinect is given relative to the depth sensor. See Kinect coordinate system for more information.


22 of 301

Kinect coordinate system The Kinect provides three-dimensional information. This information given by the Kinect is given in what Microsoftcalls "Camera space". The origin of the Camera space is the center of the Depth (or infrared) sensor. The axes ofthe coordinate system are defined as follow: Z axis is straight in front of the Kinect, the Y axis is up, and the X axis istowards the laser diodes (see picture below).

Original image from microsoft store: http://www.microsoftstore.com/store/msusa/en_US/pdp/productID.298810500


23 of 301

Oculus Rift DK2 description The Rift is a virtual reality head-mounted display developed by Oculus VR. The Rift can work in two modes: the Direct mode or the Extend Desktop mode. The direct mode requires to streamthe video directly to the Rift. This mode is therefore not accessible to the 3D picture tools offered by LabVIEW. TheExtend Desktop mode makes the Rift look like an extra monitor on Windows. This latter mode can be used by LabVIEWwithout any special driver.The Oculus Rift DK2 is equipped with a high-performance positional tracker. The positional tracker uses 40 infraredLEDs that are integrated into the headset. The infrared LEDs are invisible to the naked eye but can be seen on imagesacquired by most CMOS camera that are sensitive to the near-infrared. The image below shows some of the 40 LEDsused to track the Rift headset position and orientation.

40 infrared LEDs used by the positional tracker to determine position and orientation of the Oculus Rift DK2 headset.

For more information about the positional tracker, see reference [13]. Technical Specifications of the Oculus Rift DK2 are given below. TRACKING

Sensors Accelerometer, Gyroscope,Magnetometer

Inertial update rate

1000 Hz

Included camera Near infrared CMOSPositional updaterate

60 Hz

DISPLAY Native resolution(X)

1920


24 of 301

Native resolution(Y)

1080

Resolution (pereye)

960 x 1080

Technology OLEDField of view 100 °Max refresh rate 75 Hz More information about the Oculus Rift DK2 can be obtained at: https://www.oculus.com/ja/dk2/

https://www.oculus.com/ja/dk2/


25 of 301

Installation Open the provided installation file using VIPM from JKI (VIPM is provided with LabVIEW 2014 and later or can bedownloaded for free from http://jki.net/vipm/download). Using VIPM, install the Haro3D library to your version of LabVIEW.

http://jki.net/vipm

http://jki.net/vipm/download


26 of 301

Hololens In the Hololens, download the Haro3D application from the Microsoft store. If you cannot have access to the Haro3Dapplication in the Microsoft store, contact HaroTek. Connect the Hololens to a 5-GHz wireless network that is also connected to the network of the computer runningLabVIEW with the Haro3D library version 2.0 or later. Start the Haro3D application on the Hololens. A splash screen, similar to the one shown below, appears. If this is the first time the application runs, selecting theOK button or the Settings button have the same effect of opening the settings window.

Splash screen. Network settings can be accessed by selecting the Settings button. If the Haro3D application runs for the firsttime, the server settings windows will open no matter which button is selected.


27 of 301

The first time the Server Settings window is opened, the Server IP Address is empty and the Server Ports havethe default values of 8889 and 8887. Leave those two values unchanged unless those specific values createproblems for the computer running the LabVIEW application. If those values are changed, the Port values of theOpen Hololens VI in the LabVIEW application also needs to be changed accordingly. Select (move the cursor on the field and air-tap) the Server IP Address field. The field turns black. Selects thevalues corresponding to the IP Address of the LabVIEW application computer using the key pad at the bottom of thewindow. Click on Save. The Hololens will immediately attempt to connect with the LabVIEW application. The order inwhich the Haro3D application on the Hololens and the LabVIEW application are started should not have any impacton the ability to establish a connection. Next time the Haro3D application is started, just select OK from the splash screen (air-tapping with the cursoranywhere in the windows also works). The Hololens will attempt to connect using the parameters previously saved.


28 of 301

Kinect If your do not intend to use the Kinect V2, you can skip directly to the "Installation of the Oculus Rift DK2" or to"Installation of the Haro3D™ library" sections. Make sure that your system meets the requirements provided in the previous section. It might be possible to use theKinect V2 with systems having specifications below the minimum system requirements but performance is likely tobe affected. For Haro3D version 1.3, the use of the Kinect V2 and of the Oculus Rift DK2 requires to run the executable Visual C++ 2015 Redistributable x86 or x64 (www.microsoft.com/en-us/download/details.aspx?id=48145), corresponding tothe LabVIEW bitness version. (Versions of Haro3D prior to 1.3 required Visual C++ 2013 Redistributable). Plug the Kinect V2 into a USB 3.0 port. Windows will download all necessary drivers directly from the Internet.In cases where the computer where the Kinect V2 is to be installed does not have Internet access, execute theKinect runtime 2.0 or later installer. The file can be downloaded from http://www.microsoft.com/en-us/download/details.aspx?id=44559.

https://www.microsoft.com/en-us/download/details.aspx?id=48145

http://www.microsoft.com/en-us/download/details.aspx?id=44559



29 of 301

Rift DK2 Install the Oculus Rift runtime. The runtime can be downloaded from https://developer.oculus.com/downloads. Noticethat the application should be compatible with runtime versions 0.4 to 0.6. Runtime versions 0.7 and later are notexpected to have the extend mode display available any more so the Oculus Rift might not be compatible withLabVIEW for those later versions. Configure the Oculus Rift and its software as per Oculus’ instructions.After installation of the Oculus Rift runtime, and connection, the headset must be configured to use the Extend Displaymode. This change can be done using the Oculus configuration utility and selecting “Rift Display Mode”.

Select “Extend Desktop to the HMD”. Windows should then detect the presence of a second monitor. Open the monitor configuration utility on yourcomputer. Notice that the Rift must be the second monitor (unless it is the only monitor). The VI displaying the images to the Rift should be maximized on the monitor corresponding to the Oculus Rift(typically monitor 2) using the "Window Run-Time Position" category of the VI properties. Depending on the configuration of the computer, the headset might be configured as a portrait monitor. It is thennecessary to change its configuration to “Landscape (flipped)”.

https://developer.oculus.com/downloads/


30 of 301

System Requirements For the Microsoft Hololens: Hardware requirements: 1. Microsoft Hololens running Windows 10.0.10240.0 or later or Hololens Emulator version 10.0.11082.1039 or later. 2. 5-GHz Wireless network (not required when using emulator). 3. Computer connected to the same local network than the Hololens and running National Instruments LabVIEW withthe Haro3D Library version 2.0 or later. For the Microsoft Kinect for Windows V2: Hardware requirements: Below is an excerpt of the requirements given by Microsoft at (http://msdn.microsoft.com/en-us/library/dn782036.aspx) 1. Kinect for Windows V2 sensor, which includes a power hub and USB cabling

2. Computer processor: I7 3.1 GHz (or higher), x64 core 3. Memory: 4GB or more 4. Operating System: Windows 8 or later, 64-bit. 5. Video card: DX11 capable graphics adapter (see list of known good adapters below)

• Intel HD 4400 integrated display adapter• ATI Radeon HD 5400 series• ATI Radeon HD 6570• ATI Radeon HD 7800 (256-bit GDDR5 2GB/1000Mhz)• NVidia Quadro 600• NVidia GeForce GT 640• NVidia GeForce GTX 660• NVidia Quadro K1000M

6. Built-in USB 3.0 host controller (Intel or Renesas chipset). If you're adding USB 3.0 functionality to your existingPC through an adapter, please ensure that it is a Windows 8 compliant device and that it supports Gen-2. Software requirements: 1. Upon connecting the Kinect for Windows V2 for the first time to your computer, the drivers should be downloadedthrough Windows Update. If the computer is not connected to the internet, download and run the Kinect runtime 2.0or later (http://www.microsoft.com/en-us/download/details.aspx?id=44559). For the Oculus Rift DK2

http://msdn.microsoft.com/en-us/library/dn782036.aspx




31 of 301

Oculus runtime versions 0.4 to 0.6. Notice that runtime versions 0.7 and later are not expected to have the extendmode display available any more so the Oculus Rift might not be compatible with LabVIEW for those later versions. Minimum requirements: A desktop computer running Windows 7 or Windows 8, 2 USB 2.0 ports (at least onepowered), and a dedicated graphics card Nvidia GTX 600 series or AMD Radeon HD 7000 series (or better) withDVI-D or HDMI graphics output. Recommended specifications: In addition to the minimum spec it is recommended that your dedicated graphics cardbe capable of running current generation 3D games at 1080p resolution at 75fps or higher. https://support.oculus.com/hc/en-us/articles/201835987-Oculus-Rift-Development-Kit-2-FAQ For the Haro3D library: 1. National Instrument LabVIEW™ 2015 or later, 32 or 64 bits 2. For Haro3D version 1.3, the use of the Kinect V2 and of the Oculus Rift DK2 requires to run the executable VisualC++ 2015 Redistributable x86 or x64 (www.microsoft.com/en-us/download/details.aspx?id=48145), corresponding tothe LabVIEW bitness version. (Versions of Haro3D prior to 1.3 required Visual C++ 2013 Redistributable).

https://support.oculus.com/hc/en-us/articles/201835987-Oculus-Rift-Development-Kit-2-FAQ

www.microsoft.com/en-us/download/details.aspx?id=48145


32 of 301

Haro3D Library The Haro3D version 2.0 Library consists in five sub-libraries: three sub-libraries contain the API VIs to interfacewith the Microsoft Hololens, the Microsoft Kinect V2, and the Oculus Rift DK2. Another sub-library contains the VIsrelated to the Haro3D 3D Objects for the Hololens and for the LabVIEW 3D picture controls. The last sub-librarycontains the Utilities for 3D data processing, file I/O, and display. The API VI's for the Kinect V2 are based on two DLL's written in C++. There is one DLL (HaroTek_Kinect2_x32.dll)for the 32-bit version of LabVIEW and another DLL (HaroTek_Kinect2_x64.dll) for the 64-bit version. Each of thoseDLL must be accompanied by a DLL from Microsoft: Kinect20.Fusion.dll. Notice that there are different versions ofthat DLL according to the bitness of the application, even if the DLL names are the same. The Haro3D Library can be accessed through the LabVIEW palette when installed using the provided instructions.The API VI's and the utilities VIs are installed in the Addons palette (as shown below).

The examples are installed in the HaroTek\Haro3D sub-directory of the Examples folder of the LabVIEW versionwhere the library was installed. The examples can also be found with the NI Example Finder (under LabVIEW menuHelp->Find Examples...). In the Browse tab select the "Directory Structure" radio button and scroll down to "HaroTek\Haro3D". The examples can also be found by selecting the Task radio button and going to Fundamentals/3Dcomponents. Finally, in the Search tab, use either of the terms "Haro3D", "HaroTek", "Hololens", "Kinect", "OBJ","Oculus", or "Rift", depending on your interest. Examples The following examples are available:


33 of 301

Hololens

• Events• 3D Objects

Kinect

• Audio beam• Bodies• Cloud of points• Color image• Depth• Fusion• Infrared

Kinect Gestures

• Oculus, Kinect, OBJ, and Hands

3D OBJ file format

• OBJ file• Oculus, Kinect, OBJ, and Hands

Oculus Rift DK2

• Distortion• Oculus, Kinect, OBJ, and Hands


34 of 301

Hololens Instructions and list of functions of the Hololens toolkit. Health and Warning concerning the Hololens

• Read and follow all warnings and instructions for the Hololens Headset before use (support.microsoft.com/en-us/help/12628).• Headset should be calibrated for each user.• Not for use by children under 13.• Stop use if you experience any discomfort or health reactions.• Make sure that there is enough room around the user to move safely to operate the Hololens.

https://support.microsoft.com/en-us/help/12628

https://support.microsoft.com/en-us/help/12628


35 of 301

Haro3D application on Hololens To be able to use the Haro3D library, the Haro3D application must be running on the target Hololens. The Haro3Dapplication must be downloaded from the Microsoft store. Contact HaroTek if you have difficulties accessing theapplication.


36 of 301

Cursor With the Hololens, the user interacts with the virtual environment through its gaze, or line-of-sight. This gaze isrepresented by a cursor that moves with the Hololens headset. This cursor represents the point the user is lookingat. The cursor will interact with the virtual objects in the environment. When there is no object (or hologram) in the line-of-sight of the user, the cursor moves to a distance of 2 meters fromthe user and takes the color and shape shown below.

Cursor with no virtual object in the line-of-sight. If the Hololens detects a Hold or Manipulation, the cursor will change its shape to a hand as shown below.

Cursor with no virtual object in the line-of-sight when detecting Hold or Manipulation. When a virtual object is in the line-of-sight of the user, the cursor moves to that object and orients itself parallel to thesurface of the object and takes the shape shown below. If that object is a LabVIEW object, the cursor takes the colorshown below.

Cursor with a LabVIEW object in the line-of-sight when not detecting Hold or Manipulation. If in addition, the Hololens detects a Hold or Manipulation, a hand indicator is added to the cursor, as shown below.

Cursor with a LabVIEW object in the line-of-sight when detecting Hold or Manipulation. When the cursor is on a non-LabVIEW object, it turns orange. Below show the cursors when on a non-LabVIEWobject when not detecting and detection a Hold or a Manipulation. Notice that the dialog windows, splash screen, and mapped surfaces are considered as non-LabVIEW objects.


37 of 301

Cursors with a non-LabVIEW object in the line-of-sight when not detecting (left) and detection (right) Hold or Manipulation.


38 of 301

Voice commands The Hololens is capable of receiving voice commands. For the moment, the number of voice commands is limitedand not programmable. The list of voice commands along with their descriptions is given in the table below.

Voice Command Action

Start mapping Starts mapping of the environment. Themapped surface are shown as meshes.

Stop mapping Stops mapping the environment.The mapped surface are hidden.

Show surface Shows as a mesh the mapped surfaceon which the cursor currently lays.

Hide surface Hides the mapped surface on which the cursor currently lays.

Ok Equivalent of air-tapping on the Ok button.

Cancel Equivalent of air-tapping on the Cancel button.

Send surface Sends the mapped surface on which the cursorcurrently lays to the LabVIEW application as a mesh.

Reset World Puts origin of the LabVIEW objectsto (0,0,0) of the Hololens space

Show info Displays the information associated with the LabVIEWobject on which the cursor currently lays. The informationdisappears after 5 seconds if the cursor quits the object.

Hide info Hides the information associated with theLabVIEW object on which the cursor currently lays.

Select Equivalent of air-tap


39 of 301

Settings The Settings window is used to set the parameters to connect to the computer running the LabVIEW application(server) that is to control the Hololens. Field values are edited by selecting the field by air-tapping while the cursor is on the field. When the cursor entersthe field, the field turns from light gray to light blue. After air-tapping the field turns black. The values can then beentered by moving the cursor on the keypad and air-tapping each of the digits or dot. Notice that the keypad includesa backspace key (bk). Field values can be entered, not edited. When selecting a field, the current value in that field disappears. The Server IP Address is the IP address of the computer running the LabVIEW application that is controlling theHololens. The Server Output and Input Ports are the port values used by the computer running the LabVIEW application. Itis recommended to use the default values unless those values conflict with another application on that computer. Ifthe port values are changed in the Server Settings window, the values should also be provided to the correspondinginputs of the Open Hololens VI.


40 of 301

Settings dialog when the Server IP Address is being entered.


41 of 301

Panels and Info Displays Panels and info displays are used to provide information to the Hololens user and in some cases provide the abilityfor the user to respond. Those displays are windows that appear to the user in the Hololens space. Those windows automatically positionthemselves in the field of view of the user, trying to maintain a constant angular field of view if there is no other objectbetween the user and the window. If there is an object between the window and the user, the window moves in frontof the object, up to a minimum distance of 1 m.


42 of 301

1 and 2 Button Dialogs For both type of dialogs, the string provided to the "Text to display" input is shown to the Hololens user as a dialogwindow (see images below). The window remains open until the user air-taps a button, say "Ok" or "Cancel", or until the LabVIEW applicationsends a 1-character string as an input of the Info Display VI. The voice commands "Ok" and "Cancel" have the sameeffects than air-tapping the corresponding button. Notice that the text of a currently opened dialog window can be changed by simply using the Info Display VI againwith the new text. When the user clicks a button, or say the corresponding button text, the Haro3D application on the Hololensgenerates a "Text response" event with the button text as event data. 1-Button Dialog The 1-Button Dialog is presented to the Hololens user using the Info Display VI with the 1-Button? input set to true.

Example of 1-Button display 2-Button Dialog


43 of 301

The 2-Button Dialog is presented to the Hololens user using the Info Display VI with the 1-Button? input set to false.

Example of 2-Button display


44 of 301

Multi Line Dialogs The Multi Line dialog window is used to provide the Hololens user with a series of items he has to choose from. Eachitem is presented as a line and the user selects an item by moving the cursor on that line and air-tapping or by air-tapping the OK button while the line is selected. Four lines are presented at a time but the user can access morelines, if available, by moving the cursor on the small triangles on the right side of the window (see image below). A Multi Line dialog window is created by calling the Multi Line Display VI. When a line item is selected by the user, a "Text Response" event is generated with the text of the line as event data.If the user selects "Cancel", the string "Cancel" is returned as data.

Example of Multi Line display.


45 of 301

Panels A panel is used to display a control (like a graph) or the front panel of a VI to the Hololens user. For example, takethe VI front panel shown below.

Example of VI that can have its front panel displayed to the Hololens user as a panel (see next image) This VI would run on the computer running the LabVIEW application. The front panel of this VI can be displayed tothe Hololens user and would look like the image shown below.

Panel displaying to the Hololens user the front panel of the VI shown in the previous image. A panel is an object of the panel class, a child of the Shape class. The panel is created using the Create Panel VI.Once created and added to the Hololens using the Add 3D shape VI or Modify Multi 3D Shapes VI, the object ispassed as a parameter to the Send Texture Image VI. The desired image (standard image data used by LabVIEW) isalso passed as the Texture parameter to the VI. The image is then applied as texture to the panel.


46 of 301

Below are examples of LabVIEW code to display images of a front panel and of a waveform graph to a Hololensuser.

Example of LabVIEW code to display the front panel of a VI to the Hololens user (code is in block diagram of the VI).

Example of LabVIEW code to display a graph to the Hololens user using a panel and an invoke node implicitly linked to a Waveformgraph.


47 of 301

Hololens Gestures Description of the gestures detected by the Hololens and used by the Haro3D application and library.


48 of 301

Air tap The air tap gesture is the method used to select an object or to press a button. It consists at closing the fist with theindex straight and extended. While keeping it straight, the index is bent forward (see images below). Notice either hand can be used. Also, it is important that the hand be detected by the Hololens so it must be slightlyin front of the Hololens, slightly lower than shoulder height. The air tap gesture triggers an "Object selected" event.

Hand movement for the bloom gesture. Hand images are extracted from Hololens Gestures tutorial.


49 of 301

Hold The Hold position is like the end of an air tap (see image below). The hand must be nearly immobile for a fewseconds. A Hold event is then triggered. As for the air tap, either hand can be used. Also, it is important that the hand be detected by the Hololens so it mustbe slightly in front of the Hololens, slightly lower than shoulder height. If the hand moves, a Manipulation event might be triggered. Notice that the cursor shows a hand when a Hold or Manipulation gesture is detected.

Hand position for manipulations and hold. Hand images are extracted from Hololens Gestures tutorial.


50 of 301

Manipulation Manipulations are defined by a quantitative movement of the hand that is tracked by the Hololens. The hand mustbe in the Hold position (see figure below). The only difference between a Manipulation event and a Hold event isthe motion of the hand. If the hand is not moved immediately after taking the Hold position, a Hold event might betriggered. One LabVIEW object of the Hololens must be selected. The data sent is the cumulative movement of the hand. Manipulations are always started by sending the value(0,0,0). Data are sent from the Hololens at a rate of approximately 1 time per second. Notice that the cursor shows a hand when a Hold or Manipulation gesture is detected.

Hand position for manipulations and hold. Hand images are extracted from Hololens Gestures tutorial.


51 of 301

Bloom The bloom gesture is used to open the control panel in the Hololens OS to access the settings and start anapplication. It is also used to exit the Haro3D application (like other Hololens applications). The bloom gestures consists in putting all fingers together, pointed up, and then to open the hand (see imagesbelow). As for the other gestures, either hand can be used. The hand must be in a zone where it can be detected by theHololens (slightly in front of the Hololens, slightly lower than shoulder height). The bloom gesture will terminate the Haro3D application which generates a Connection event with a false value tothe LabVIEW application.

Hand movement for the bloom gesture. Hand images are extracted from Microsoft Hololens Gestures tutorial.


52 of 301

Programming principles Opening and Closing communication Programming with the Hololens API requires to start by calling Open Hololens. Calls to other Hololens VIs will resultin an error if the communication with the Hololens has not been initialized. Open Hololens can be called from severallocations in your program. Only the first time that it is called that it will establish the communication. The otherinstances of Open Hololens will not generate an error. When terminating the application, the Close Hololens must be called. This VI should be called only once. Acommand is issued to the Hololens application to terminate as well. Connection status The status of the connection can be verified by calling Connection Status. See Simple Code Example for a simple example of management of the Hololens connection in LabVIEW.


53 of 301

Connection The IP address of the computer running the LabVIEW application must be provided to the Hololens through theSettings dialog. The Settings dialog can be accessed by selecting the Settings button on the Haro3D application splash screen.When the Haro3D application runs for the first time, the Settings dialog opens in any case after the splash screen. The Hololens and the computer running the LabVIEW application must be on the same network and firewalls andother security system must be set to allow communication between them. The Settings also include input and output ports. If the default values do not create any problem for the computerrunning the LabVIEW application, it is recommended to keep them unchanged. If other port values must be used,make sure that values used in the Settings dialog of the Haro3D application correspond to the input port values ofOpen Hololens VI.


54 of 301

User Events The server transmits information from the Hololens using user events. A user event cluster is provided as an outputof the Open VI. The user events can also be accessed using the Get Events VI. The Hololens user event cluster consists in two events: Connection and Data. The server generates a Connectionevent as soon as the communication with the Hololens is established. The data associated with the Connectionevent is a boolean. The value is true when the Connection is initially established. A second event is generated whenthe Connection is closed. The value of the boolean then is false. No Connection event are generated if the serverdoes not connect with the Hololens. The other event is a Data event. This event is generated by the server whenever it receives data from the Hololens.The data associated with this event is a cluster made of a data event kind (Kind, enum) and of Data (genericmessage class). The Kind enum indicates the kind of the data event. The possible values are:

The Data element of each event kind is different. The Data element is and object of the Generic event data class.Each event kind has its own Data element that is a child of the Generic event data. The actual data is extractedusing an Extract Message VI specific to each data kind. The table below indicates what is the data of the Dataelement and what is the Extract Message VI associated with the event Kind.

Kind Data Extract VI

None String (no data expected) Extract None Message

Mesh Mesh Object, Cursor position,Cursor normal (position cluster).

Extract Mesh Message

Text Response String Extract Text Message

Manipulation Displacement (position cluster), status(int16), Hand position (position cluster).

Extract Manipulation Message

Object selected Object index (int32), Cursor position,Cursor normal (position cluster).

Extract Object Selected Message


55 of 301

Hold Object index (int32), Cursor position,Cursor normal (position cluster).

Extract Hold Message

Receive Anchor Anchor data Extract Anchor Message

Sharing Event data (Generic MessageObject), Device name (string).

Extract Sharing Message

Secondary connect Device name (string). Extract Connection Message

Invalid String (no data expected) Extract Invalid Message


56 of 301

3D Objects The concept of 3D Objects is introduced with the Hololens API but is not limited to the Hololens. 3D Object is actually an abstract class that has two children: OBJ Mesh and Shape. 3D Shape is also an abstractclass that has 5 children: Box, Cylinder, Mesh, Panel, and Sphere.


57 of 301

Space The Hololens defines its own space with its own coordinate system. By default, the origin of the coordinate system isat the location of Hololens headset at the start of the Hololens application, with the Z-axis oriented towards the frontand the Y-axis normal to the detected floor. If the Hololens cannot detect its environment, it will work in limited modeand all holograms will be locked to the headset. It is recommended to define an origin using a point on an actual object (on a mapped surface) using the Move OriginVI. This operation creates an anchor using the local features of the mapped surfaces, limiting the possible drift of theholograms over time. Another operation related to the Hololens space is Change Scale. This operation changes the scale of all LabVIEWobjects and of the space between them. This operation only applies to the object currently present in the Hololensspace. Objects uploaded after the Change Scale operation are not affected.


58 of 301

Hologram Sharing The Haro3D library and app offer the ability to share Holograms between more than one Hololens. If the Hololensare located in the same room, the holograms can also share a common coordinate system, meaning that theholograms would be seen at the same location by all the Hololens. Connections After starting the LabVIEW application, the first Hololens to connect is called the primary Hololens. This Hololensgenerates the connection and disconnection events, controls the number of objects in the Hololens space, andreceives the main commands from the LabVIEW application. Notice that when the primary Hololens disconnectsfrom the LabVIEW application, the LabVIEW application cannot continue to support the secondary Hololens.Typically, the LabVIEW application should terminate upon the disconnection of the primary Hololens. Events All the Hololens connecting to the LabVIEW application that connects after the primary Hololens are secondaryHololens. When connecting or disconnecting, or when generating an event, a secondary Hololens sends a SharingMessage. The sharing message data contains the name of the device (or IP address), and a Generic Messagecontaining the data associated with the event. The information can be extracted from that generic message in thesame manner that data is extracted from messages by the primary Hololens. Holograms It is left to the LabVIEW programmer how he wants to handle events from the secondary Hololens. Notice that allobjects sent to the primary Hololens are also sent to the secondary Hololens. At the moment, only holograms sent tothe primary Hololens after a secondary Hololens has connected will be seen by that secondary Hololens user. Anchors and Spaces An anchor is a point in space not defined by the coordinate system but by 3D features of the environment as mappedby the Hololens. This anchor can then be used to define a new coordinate system for each Hololens.

The approach consists in defining an anchor in one Hololens and to send this anchor with the other Hololens. If theanchor can be found by the other Hololens in the room, that point becomes common to all Hololens. Defining that pointas the origin of the coordinate system makes all Hololens share a common coordinate system.

One issue is that an anchor shared by one Hololens must be found by the other Hololens. The anchor is defined usingthe 3D data mapped by the operating system of the Hololens, called spaces by Microsoft. To successfully share ananchor, both Hololens must currently be in the same room, and the mapped spaces of that room must be similar.

In general, simply sharing an anchor between Hololens that are in the same room does not work. Typically, Hololensthat are in the same room and maps slightly different areas of that room. Relatively small differences in the mappedspaces will result in failure of sharing an anchor. Fortunately, Microsoft provides a tool to see a space as currentlymapped by a Hololens. This tool can be accessed using the device web portal of a Hololens [20]. The currently mappedspace can be seen in the 3D View page by clicking the button Update under the Surface Mapping heading.

The figure below shows the mapped surfaces of the same room mapped by two different Hololens. The smalldifferences shown between the mapped surfaces shown in that figure prevent the successful sharing of an anchor.


59 of 301

Surface mapping of the same room by two different Hololens. The mappedsurfaces present too many differences for the successful sharing of an anchor.

A successful approach consists in systematically mapping a room by walking in the room along the same path witheach Hololens. First, it is preferable to erase the current space to make sure that no residual mapped surfaces wouldbe associated with the current room. The spaces can be erased in the Settings -> System -> Spaces page of eachHololens. Notice that each space is associated with a wifi connection.

The next figure shows two mapped surfaces of the same room by two different Hololens. In that case, the mappedsurfaces of the room appear to cover the zones and the sharing of an anchor was successful.


60 of 301

Surface mapping of the same room by two different Hololens. The mappedsurfaces present too many differences for the successful sharing of an anchor.

For large rooms, it might be easier to map only the section of the room where the Hololens will be confined, as longas the mapped surfaces appear similar in the 3D View page of the device web portal. Notice that the spaces arecontinuously updated while the Hololens is in the operating system, sometimes resulting in a new area of the roombeing mapped by one Hololens but not the other. Common coordinate systems

By default, each Hololens has its own coordinate system. When starting an application on the Hololens, the positionof the headset defines the origin of that coordinate system. When the primary Hololens positions the origin of the Hololens space, an anchor is created for that point. When thesecondary Hololens are in the same room than the primary Hololens, the anchor can be shared so that the origins ofthe Hololens coordinate systems are all aligned. The anchor can be requested from the primary Hololens by usingthe Send Anchor function, and the anchor data can be sent to all secondary Hololens using the Update SecondaryOrigins function. If the anchor is recognized by a secondary Hololens, the origin of coordinate system is moved tothat point. If not recognized, the origin remains the same for that Hololens.


61 of 301


62 of 301

Simple Code Example Below is shown a very simple example of LabVIEW code illustrating the minimum steps for LabVIEW interactionswith the Hololens. An event is generated when the Hololens connects with the LabVIEW application, and anotherevent is generated when the Hololens disconnects. After establishing the connection with the Hololens, LabVIEWcontrols the Hololens using commands VIs, and receives data from the Hololens through user events. LabVIEWinitiates, manages, and terminates the connection with the Hololens using Connection Management VIs.

Open Hololens Connection

Open VI. Starts the communication server for the connection with the Hololens. As soon as the connectionhas been established, a Connection event is generated. The Open VI also provides the Hololens eventreferences to register the event structure for those events. The Get Events VI can also be used to obtain theHololens event references.

Register for events


63 of 301

The data generated by the Hololens is sent to the LabVIEW application through user events. Referencesfrom those events are obtained from the Open or Get Events VIs. The output of the Register for eventsfunction must be connected to Dynamic event terminal of an event structure. Notice that more than oneevent structure can be registered for the same Hololens events, as long as a Register for events call is donefor each event structure.

Event structure

An event structure is used to receive events generated by the Hololens and by the Hololens user. Two typesof events are possible: Connection or Data. See User Events for more information about those two types ofevents. The event structure can also be used to send commands to the Hololens upon inputs from the LabVIEWuser (static events).

Event handling

For each type of event, a case can be used to handle the particular type of event. For the Connection event,the two possible values are only true (Hololens connected) and false (Hololens disconnected). The desiredcode to be executed when connection or disconnection occurs should be put in the corresponding case ofthe case structure. Between the connection and disconnection events, all other events generated by the Hololens aresent to the LabVIEW application using a data event. Each Data event provides a Kind value (typedefenum) that indicates what kind of event was sent and a Data object (as shown in the image above). It isrecommended to wire the Kind value to a case structure, and to create a case for each possible value ofKind. Information can be extracted from the Data object by using the Extract Message VI that corresponds tothe corresponding Kind value ("Object Selected" value in the case of the image above).


64 of 301

Close Hololens Connection

The Close VI must be called before terminating the application. If the Hololens is still connected to theLabVIEW application upon calling the Close VI, the connection is closed and the Haro3D application runningon the Hololens quits. If the Hololens had already disconnected, the Close VI simply releases the resourcesassociated with the Hololens connection without generating an error.


65 of 301

Hololens API

List of the VIs used to control the Hololens connection, the user interface elements, the 3D objects, and the Hololensspace.


66 of 301

Building an executable with the Hololens API Applications (executables) can be built from VIs using the Hololens API of the Haro3D library like most other VIs,except that a few additional steps are necessary. Below is an illustration on how to build an executable from one of the examples provided with the Haro3D library forthe Hololens. The usual steps related to the LabVIEW project and the build specification must be followed as for any other VI. Inthe present case, this VI is simply one of the Hololens example. Once the build specification has been created as desired, it is necessary to add the Haro3D.lvlib file to the project.This library can be added to the project by navigating to the vi.lib\HaroTek\Haro3D folder of the current version ofLabVIEW, or the library can be more simply added by dragging any VI or control of the Haro3D toolkit to the project.The Haro3D.lvlib library is then automatically added (see below).

Haro3D.lvlib library included in the project to be used to build an executable. Once the Haro3D.lvlib file in included in the project, open the build specification and go to the "Source File Settings"Category. Expand the Haro3D.lvlib item of the project files to select the folder named "Event Messages" in the folder"Hololens" (see below). Checks the box on the right "Set destination for all contained items" and select "SupportDirectory" from the roll-down menu (as shown below).


67 of 301

Set the "Event Messages" folder content to be copied to the support directory. Then, in the same "Source File Settings" category, expand Hololens.lvclass, and then expand the private folders.Select Connection control.vi and uncheck the box "Use default save settings" and then uncheck the box "Removefront panel", as shown below.


68 of 301

Uncheck the "Remove front panel" for Connection control.vi. The executable can now be built using the usual procedure.


69 of 301

Connection Management Utilities to manage the Hololens connection and data communication.


70 of 301

Open Hololens

Initializes the communication with the Hololens. Does nothing when called after the first time. Provides referencesof the Hololens user event in all cases. Sets the local ports in and out to different values than the default only ifnecessary. Make the port value changes in the Hololens Haro3D settings as well. Input parameters Input port (8887) I32. Input port value for the

TCP communication with theHololens.

Use default value unless that value creates anissue. If a different value is used, the defaultvalue in the Settings of Haro3D app on theHololens must also be changed.

Output port (8889) I32. Output port value for theTCP communication with theHololens.

Use default value unless that value creates anissue. If a different value is used, the defaultvalue in the Settings of Haro3D app on theHololens must also be changed.

error in (no error) Input error. VI is not executedif an error is present.

Standard LabVIEW error cluster.

Output parameters Hololens events Cluster of Hololens User Events. See User Events

for more information.

error out Standard LabVIEW error cluster.


71 of 301

Close Hololens

Closes communication with the Hololens, destroys the user events and releases the resources. Input parameters error in (no error) Input error. This VI is

executed even if an error ispresent.


Output parameters error out Standard LabVIEW error cluster.


72 of 301

Get Number of Objects

Returns the number of 3D objects (holograms) currently loaded in the Hololens space from the LabVIEW application. Input parameters error in (no error) Input error. VI is not executed

if an error is present.Standard LabVIEW error.

Output parameters Number of objects I32. The current number of 3D objects in the

Hololens space. Notice that this number is trackedon the LabVIEW side independently from theHololens.



73 of 301

Connection status

Returns a single boolean indicating if the network connection with the Hololens is fully established or not. Input parameters error in (no error) Input error. VI is not executed

if an error is present.Standard LabVIEW error cluster.

Output parameters Connected Boolean. True if the connection is established, false

if not.



74 of 301

Get Events

Return a cluster of the references of the user events generated by the Hololens. Input parameters error in (no error) Input error. VI is not executed

if an error is present.Standard LabVIEW error cluster.

Output parameters Hololens events Cluster of Hololens User Events. See User Events

for more information.



75 of 301

Commands Sub-palettes of VIs used to command actions to the Hololens.


76 of 301

3D Objects VIs related to the handling of 3D Objects in the Hololens space.


77 of 301

Move Multiple Objects

Moves multiple 3D objects in the Hololens space into the orientation and position provided for each object of thearray, each object being defined by the provided index. All the information is sent at once. Data is an array ofclusters containing each object index in the Hololens space, orientation (quaternion) and position (x,y,z). Usefulwhen several objects must be rotated and/or translated at once. Input parameters Data Required. Rotation data

cluster containing eachobject index, orientation(quaternion) and position(x,y,z).





78 of 301

Move Object

Moves a 3D Object in the Hololens space into an absolute position (x,y,z) and into an absolute orientation(quaternion). The position and orientation are applied to the 3D object out. Input parameters 3D Object in Required. 3D object.

Position Required. Position cluster.

Quaternion Quaternion cluster. Default value is no rotation.



Output parameters 3D Object out Same than 3D object in except that position and

orientation have been applied to the object.



79 of 301

Modify Multiple Materials

Sends an array of Shapes and apply material properties to the objects corresponding to the Hololens space indices. Input parameters Shape array Required. Array of Shape

objects.Shape objects can be generic as long asthe materials and Hololens indices are thosedesired for modifications.





80 of 301

Send Texture Image

Sends a LabVIEW image to the Hololens to be applied as texture to the 3D shape provided. If no texture is provided,uses the texture of the 3D shape (meshes for example). Notice that this VI currently works only with Panels and Meshes. Input parameters Shape Required. Shape object. Object must already exist in Hololens space

Texture (no texture) LabVIEW image. Standard LabVIEW image. If no Texture isprovided, uses the Texture of the Shape object.This is the procedure if a mesh has alreadyloaded texture.





81 of 301

Add Shape

Add a new Shape object to Hololens space, and have the object appears in the Hololens. The index of the newobject in the Hololens space is recorded in the Shape out and is also provided as a separate output. Input parameters Shape in Required. Object child of the

Shape class.Shape object to be added to the Hololensspace. If the input object is already present inthe Hololens space, a new one is created.



Output parameters Shape out Object child of the Shape class with its Hololens

index updated to the correct value

Holo index I32. Index of the new object in the Hololens space.



82 of 301

Modify Multi Shapes

Modifies multiple Shapes that are already existing in the Hololens space or creates new ones if the index is < 1. Ifnew objects are created, the new indices are recorded in the corresponding shape objects of the Shapes out array. Input parameters Shapes in Required. Array of children of

the Shape class.



Output parameters Shapes out Array Shape class objects. Copy of the input array

except if new Shapes are created in the Hololensspace. In that case, the new index is recorded in thecorresponding Shape object.



83 of 301

Modify Shape

Modifies a single shape already existing in the Hololens space. An error is generated if shape in has an indexoutside the range of the indices of the objects already present in the Hololens space. Input parameters Shapes in Required. Object child of the

Shape class.



Output parameters Shape out Object child of the Shape class, copy of the input

object.



84 of 301

Set Object Visibility

Sets the visibility (True or False) of the 3D object provided. Notice that when the visibility of a 3D object is false, it is not visible but also it does not interact with the rest of theHololens space (cursor, mapping, and other objects). Input parameters 3D Object in Required. Object child of the

3D Object class.



Output parameters 3D Object out Required. Object child of the 3D Object class.



85 of 301

UI VIs related to the User Interface functions.


86 of 301

Multi Line Display

Opens a multiple line dialog from which the user can select a line. Text Lines is an array of strings, each stringcorresponding to one item the user can choose. If provided with an empty array, the Multi Line Display is closed ifalready opened. No string can be empty.When the user selects a line (by air tap or by clicking OK while the line is selected), the text of the line is returned asText event.The maximum number of lines is 50. Input parameters Text Lines Required. Array of strings.

Error in (no error) Input error. VI is not executedif an error is present.

Standard LabVIEW error.



87 of 301

Info Display

Displays a string message to the user. If a single character is sent, the display is turned off. Text cannot be an emptystring. The boolean "1 Button?" is true, display has only an "Ok" button. If false, display has an "Ok" button and a "Cancel"button. When the user clicks on the "Ok" or "Cancel" buttons, the LabVIEW application receives an event with the textcorresponding to the button. Input parameters Text to display Required. String. Text to be displayed as information to the user.

1 Button? Required. Boolean. If true, display has only an "Ok" button. If false,display has an "Ok" button and a "Cancel"button.





88 of 301

Voice Output

Sends Voice message to the user as a spoken text. Message cannot be an empty string. Input parameters Voice message Required. String. Cannot be an empty string.





89 of 301

Point to Object

Displays an arrow pointing the user towards the 3D object provided. If no 3D object is provided (default), the arrow ishidden. Input parameters 3D Object Required. 3D Object class or

descendant class.3D Object to point to in the Hololens space.Only the holo index is used in this object.





90 of 301

Set Object Info

Sets the text that the user can make appear for the current LabVIEW object in the Hololens space. If the input text isan empty string, current info is not changed. Input parameters Shape Required. Shape object. Object must already exist in Hololens space

Text Required. String. Text to be used as info for the current Shapeobject.





91 of 301

Add Voice Commands

Add new voice commands for the user. Upon one of the new commands, the Hololens generates a text responseevent with the text of that command. If any string of the input string array is empty, an error is generated. Input parameters Voice commands Required. String array. Each

element of the array is anew command to be addedto the current list of voicecommands.

String is returned as data of a Text Responseevent when the Hololens user voices thatcommand. Any empty string within the arraygenerate an error and no command is added.





92 of 301

Hololens Space VIs related to the handling of the Hololens space.


93 of 301

Move Origin

Rotates and moves to the specified position and orientation (quaternion) the reference point of all LabVIEW objectsin the Hololens space. This operation creates an anchor point in the Hololens space using the environment features. Input parameters Quaternion Optional. Quaternion cluster. Default value is no rotation.






94 of 301

Change Scale

Changes the scale of the whole Hololens space Input parameters Space scale Required. Single float.





95 of 301

Send Anchor

Requests that the primary Hololens sends the anchor used for its coordinate system origin. Input parameters error in (no error) Input error. This VI is

executed even if an error ispresent.




96 of 301

Sharing VIs related to the secondary Hololens connections.


97 of 301

Update Secondary Origins

Sends a Hololens anchor to all secondary Hololens connections to move their origins to that anchor. Input parameters Anchor Required. Anchor data

cluster.





98 of 301

Display Info All Secondaries

Displays a string message to all secondary users. If a single character is sent, the display is turned off. Text cannotbe an empty string. The boolean "1 Button?" is true, display has only an "Ok" button. If false, display has an "Ok" button and a "Cancel"button. When the user clicks on the "Ok" or "Cancel" buttons, the LabVIEW application receives an event with the textcorresponding to the button. Input parameters Text to display Required. String. Text to be displayed as information to the user.

1 Button? Required. Boolean. If true, display has only an "Ok" button. If false,display has an "Ok" button and a "Cancel"button.





99 of 301

Move Secondary Origin

Rotates and moves to the specified position and orientation (quaternion) the origin of the space of a secondaryHololens specified by device name. Device name is provided by the sharing events generated by the secondaryHololens. This operation creates an anchor point in the Hololens space using the environment features. Input parameters Quaternion Optional. Quaternion cluster. Default value is no rotation.






100 of 301

Voice Output All Secondaries

Sends the voice message to all secondary Hololens users as a voice. Message cannot be an empty string. Input parameters Voice message Required. String. Cannot be an empty string.





101 of 301

Event data

Data from the Hololens are communicated through LabVIEW user events. There are two type of Hololens userevents: Connection and Data. The LabVIEW user events from the Hololens can be obtained from the Open or Get Events VIs. The list of possibleevents and the associated data are provided below. More information can be found in the section Programmingprinciples\User Events. Below shows a LabVIEW code example extracted from the Hololens Event Example available from in the Examples\HaroTek\Haro3D folder of the LabVIEW version where the Haro3D library was installed. The example can also befound using the NI Example Finder using Hololens as search term.


102 of 301

Example of LabVIEW code opening the communication with the Hololens Haro3D application and managing the corresponding userevents. Connection Event The Connection event is generated when the network communication between the Hololens and the LabVIEWapplication is established or terminated. The data associated with this event is a boolean that is true when thecommunication is established, and false when it is terminated. Below is an example of LabVIEW code handling the connection event.

Example of LabVIEW code to handle the Hololens connection event.


103 of 301

Data Event The Data event is generated by the Hololens Haro3D application when the application has some information or datato transmit to the LabVIEW application. The information is provided by sending a Hololens Data user event. The Data user event is composed of two parts,an enum (Kind) and Data (Generic Data Message class object). The Kind enum provides the information about the kind of event generated by the Hololens application. The list ofpossible events is defined by the values that the Kind enum can take:

List of Hololens Data events and possible values of the Kind enum. Each kind of event has its own type of data. Data is extracted from the Data component of the Hololens user eventusing the specific VI corresponding to the kind of received event. Those VIs are described within the current section. Notice that the "None" and "Invalid" event kinds do not correspond to specific events from the Hololens. These eventkinds are provided to cover possible communication issues or other problems. The programmer should decide if hewants to handle or ignore those events. Below are a few examples of LabVIEW code to extract data from the Generic Message class object.


104 of 301

Example of LabVIEW code showing the event structure case receiving the Hololens user event and extracting the data from the "TextResponse" event kind.

Example of LabVIEW code showing the extraction of the data from the "Manipulation" event kind.

Example of LabVIEW code showing the extraction of the data from the "Object selected" event kind.


105 of 301

Example of LabVIEW code showing the extraction of the data from the Receive Anchor event kind.

Example of LabVIEW code showing the extraction of the data from the Sharing event kind.


106 of 301

Example of LabVIEW code showing the extraction of the data from the Secondary Connection event kind.

Example of LabVIEW code showing the extraction of the data from the "Invalid" event kind.


107 of 301

Extract Mesh Message

Extract mesh information from generic data from a Hololens event. Input parameters Generic Event Data Required. Holo Event

Message.Generic Data associated with the Hololens userevents.

Material info Required. Material infocluster.

Gives the appearance of the mesh.



Output parameters Mesh Message Out Mesh Message object.

Mesh Object Mesh Object.

Cursor Position Position cluster. The (X,Y,Z) values indicate theposition where the cursor was at the time of theevent.

Cursor normal Position cluster. The (X,Y,Z) values indicate a vectorpointing in the direction of the normal of the surfacewhere the cursor was at the time of the event.



108 of 301

Extract Object Selected Message

Extracts the information associated with an Object Selected event.Cursor position and normal are the position and normal of cursor on the selected object.Object is the index of the object if it is a LabVIEW object.Object is -1 if the object is a mapped surface, and -2 if it is any other type of object. If the cursor on no object, Objectis 0. Input parameters Generic Event Data Required. Holo Event




Output parameters Object SelectedMessage

Object Selected Message object.

Object int32. Object is the index of the object if it is aLabVIEW object.Object is -1 if the object is a mapped surface, and-2 if it is any other type of object. If the cursor on noobject, Object is 0.

Cursor Position Position cluster corresponding to the position of thecursor when 3D object was selected.

Cursor normal Position cluster corresponding to the direction of thenormal of the cursor when the Hold was detected bythe Hololens.



109 of 301

Extract Hold Message

Extracts the hold data from a Generic Event Message. Cursor position and normal are the position and normal ofcursor where the Hold start occurred.Object is the index of the object if it is a LabVIEW object.Object is -1 if the object is a mapped surface, and -2 if it is any other type of object. If the cursor was on no object,Object is 0. At the Hold start, cursor normal is normalized, taking an amplitude of -1 or +1. At the Hold completion (user releasesthe hold) or Hold canceling (user hand moves or gets out of range), all values of normal and position are equal to 33or -33, respectively. Input parameters Generic Event Data Required. Holo Event




Output parameters Hold Message Hold Message Object.

Status Enum. Possible values are: None, Started,Completed, Canceled.

Object int32. Object is the index of the object if the cursorwas on a LabVIEW object when the Hold wasdetected by the Hololens. Object is -1 if the objectis a mapped surface, and -2 if it is any other type ofobject. If the cursor was on no object, Object is 0.

Cursor Position Position cluster corresponding to the position of thecursor when the Hold was detected by the Hololens.

Cursor normal Position cluster corresponding to the direction of thenormal of the cursor when the Hold was detected bythe Hololens.



110 of 301

Extract Manipulation Message

Extracts the Manipulation data from a Generic Event Message sent by the Hololens. Displacement is a clusterof singles (X, Y, Z) representing the cumulative displacement since the start of the manipulation relative to theorientation of the Hololens. Status indicates the current status of the manipulation operation (Started, Updated,Completed, Canceled). Position is the position of the hand in absolute coordinates. Each direction of displacement can take a value between -1 and +1. At the start of the manipulation, all three valuesare exactly equal to 0. At the completion (user release the hold) or canceling (user hand gets out of range) of themanipulation, all three values are equal to 33 or -33, respectively. Input parameters Generic Event Data Required. Holo Event




Output parameters Manip Message Hold Message Object.

Displacement Position cluster corresponding to the cumulativehand displacement since the start of theManipulation, as detected by the Hololens, relativeto the orientation of the Hololens.

Status Enum. Possible values are: None, Started,Updated, Completed, Canceled.

Position Position cluster corresponding to the currentposition of the hand doing the manipulation.



111 of 301

Extract Text Message

Extracts the data from a generic message from a Hololens Text Response event. Input parameters Generic Event Data Required. Holo Event




Output parameters Text Message out Text Message object.

Text String. Text sent by the Hololens.



112 of 301

Extract None Message

Extracts None data from a Generic Event Message. This function is provided for consistency but also to extractpossible data that might be sent by the Hololens in case of a None command message. Notice that the data is rawtext and would need to be unflattened (little endian for numeric values) Input parameters Generic Event Data Required. Holo Event




Output parameters None Message None Message object.

None Data String. Data sent by the Hololens with the Nonecommand. Notice that the data is raw text andwould need to be unflattened (little endian fornumeric values)



113 of 301

Extract Anchor Message

Extract anchor data from generic data from a Hololens event Input parameters Generic Event Data Required. Holo Event




Output parameters Anchor Message out Anchor Message object.

Anchor Anchor data cluster.



114 of 301

Extract Sharing Message

Extracts what message a secondary Hololens has sent as an event. Input parameters Generic Event Data Required. Holo Event




Output parameters Sharing Message Sharing Message object.

Secondary Message Generic Event Message Object. Message actuallygenerated by the secondary Hololens. Data canbe extracted using the same functions than for theprimary Hololens.

Device Name String. Name or IP address of the secondaryHololens that generated the event.



115 of 301

Extract Connection Message

Extract info from generic message object generated by the connection of a secondary Hololens Input parameters Generic Event Data Required. Holo Event




Output parameters Sharing Message Sharing Message object.

Device Name String. Name or IP address of the secondaryHololens that connected.



116 of 301

Extract Invalid Message

Extracts invalid data from a Generic Event Message. This function is provided for consistency but also to extractpossible data that might be sent by the Hololens in case of an invalid message. Notice that the data is raw text andwould need to be unflattened (little endian for numeric values). Input parameters Generic Event Data Required. Holo Event




Output parameters Text Message out Text Message object.

Invalid Data String. Data sent by the Hololens with the Invalidcommand. Notice that the data is raw text andwould need to be unflattened (little endian fornumeric values)



117 of 301

Controls LabVIEW controls related to the Hololens API.


118 of 301

Holo Mesh Data

Cluster containing the position and normal of the point where the Hololens cursor was when the mapped surfacewas sent, and vertices and indices defining a mesh representing the mapped surface.

Cursor position is a position cluster giving the coordinates (x, y, z) of the cursor on the sent surface. Cursor normals is a position cluster giving the direction (x, y, z) of the normal of the surface where the cursor iscurrently located. Vertices is an array of singles, each group of 3 representing the coordinates (x,y,z) of a vertex of the mesh. Indices is an array of int32 giving the indices by group of 3 of the vertices making a triangle of the mesh.


119 of 301

Anchor Data

Cluster containing the name of the anchor and a data value reference to a signed byte array that is the raw Hololensanchor data.

Name is a simple string. Data is a data value reference to a signed byte array. This byte array contains the raw data of the Hololens anchor.The byte array can be as large as a few 10's megabytes.


120 of 301

Kinect V2 API DLLs The Kinect V2 API consist in a series of VI's that are wrappers for two DLL's, one DLL for the 32-bit version andanother one for the 64-bit version. The two DLLs are:

• HaroTek_kinect2_x32.dll• HaroTek_kinect2_x64.dll

Those two DLLs should be located in separate folders in the same directory than the Kinect V2 API's. In each ofthose folders, another DLL must be present: Kinect20.Fusion.dll. Notice that there are two versions of that last DLLone for each of the 32 and 64 bit versions even if the names are the same. When building an executable (or other deployment) with one of the Kinect V2 API's, the Kinect20.Fusion.dll file mustbe added to the "Always included" field of the "Source Files" category of the application properties. See the sectionBuilding an executable with the Kinect V2 API. Palettes If the Haro3D library was installed using the provided instructions, the Kinect V2 API VI's should be accessiblethrough the LabVIEW function palettes: Addons -> HaroTek -> Haro3D -> Kinect V2 (as shown below).

The Haro3D API's give access to the following features of the Kinect V2:


121 of 301

Building an executable with the Kinect V2 API When building an executable (or other deployment) with one of the Kinect V2 API's, the Kinect20.Fusion.dll filemust be added to the "Always included" field of the "Source Files" category of the application properties. The file islocated in the library toolkit directory of the LabVIEW version where Haro3D™ library was installed: ..\vi.lib\HaroTek\Haro3D\Kinect V2\HaroTek_Kinect2_x32 DLL (or HaroTek_Kinect2_x64 DLL if using LabVIEW 64 bit). Make sureto select the directory corresponding to the version of LabVIEW used to build the executable. The dll files in the twodirectories have the same names but do not have the same contents.


122 of 301

Audio Beam

Functions related to the acquisition of the audio beam direction from the Kinect V2. If building an executable using one of the Kinect2 API, make sure to include the appropriate Kinect20.Fusion.dll (seemanual). Input parameters Command Required. Enum. Selects the

operation of the VI.

Get Returns the current values of the outputparameters without executing an operation.

Init Initializes the Audio beam function.

Acquire Requests current angle and confidence fromKinect.

Close Closes the Audio beam function


Standard LabVIEW™ error.

Output parameters Status I32. Returns a value indicative of the status of the

requested operation at the DLL level. Value of 0indicates success of the operation. See Status errorcodes for other values

Angle Angle in degrees relative to the Z direction of Kinectof provenance of sound

Confidence Level of confidence in the measurement of the angleof provenance of sound. Value varies between 0 and1.

error out Standard LabVIEW™ error cluster.


123 of 301

Bodies

API to control and acquire bodies from Kinect V2. Bodies is an array of 6 element where each element gives theproperties of the body detected or not by the Kinect (is tracked property). Command controls the API and statusgives a feedback of the operation. 0 means that the operation was successful. Floor plane is a vector (x,y,z,w) givingthe plane of the floor as detected by Kinect. If building an executable using one of the Kinect2 API, make sure to include the appropriate Kinect20.Fusion.dll (seemanual). Input parameters Command Required. Enum. Selects the



Init Initializes the Bodies (skeleton) acquisitionmode of the Kinect.

Acquire Requests current Bodies info from Kinect. Info isreturned in Bodies.

Close Closes the Bodies mode.





Bodies 1D array of Body info clusters.

Floor plane 1D array of singles (4 elements: x, y, z, w)representing the floor plane.



124 of 301

Bodies Up to 6 Bodies can be tracked, with 25 joints for each body. Hand states The hand state has three values: Open, Closed, and Lasso. Lasso is the state where the hand is closed with theindex and middle finger extended. It is better to use the hand state only when the user is made aware to use hishand to achieve some control. The hand should be clearly positioned in front of the user with the palm towards theKinect. Any other attempt to use the hand state would be at best unreliable.


125 of 301

Single array to body

Changes the 1D array of singles (4-byte float) from the DLL into a LabVIEW body info structure. This sub-VI is used by the Kinect Bodies API to interface with the DLL. Typical users should not need to directly usethat sub-VI.


126 of 301

Cloud of points

Controls X,Y,Z clouds of points from Kinect V2. Cloud size is the size of the color image mapped into the 3D space.Cloud data is a cluster containing Points and Colors arrays. Points is a 1D array having a length equal to 3 times thetotal number of pixels of the mapped color image where the X, Y, Z value for each point are positioned consecutivelyin the array. The Colors array is a 1D array of U32 having a length equal to the total number of pixels. Each value ofcolors is a RGB color. Status is a feedback for the current operation where 0 corresponds to success. If building an executable using one of the Kinect2 API, make sure to include the appropriate Kinect20.Fusion.dll (seemanual). Input parameters Command Required. Enum. Selects the



Init Initializes the Cloud acquisition mode of theKinect. The Cloud size is set at the Init step.

Acquire Gets the cloud data from the Kinect. Data isreturned in the Cloud data cluster.

Close Closes the Cloud mode.

Acq User Event Acquires the cloud data from the Kinect andsends it to a LabVIEW user event, as defined bythe User event input.

Cloud size Enum. Selects the dimensionof the cloud.

1920x1080 Actual size of the color image from the Kinect.For each pixel of the color image, 3D spacecoordinates (X,Y,Z) are provided. The othercloud size are downsampled done at the DLLlevel of the full data set acquired by the Kinect.Required for the Acquisition command only.

960x540

480x270


127 of 301

User event LabVIEW user event. User event reference created using the Clouddata cluster. Required only if user events are tobe used with the "Acq User Event" command.





Cloud data Output data as a Cloud data cluster.


Cloud of points The Cloud API consists at the simultaneous acquisition of depth and color frames where the color frame is mappedinto the camera space. This API does not correspond to the separate acquisition of depth and color frames usingeach its respective API. First, using separate API for each frame, the frames would possibly be separated by one ormore acquisition, therefore not being actually simultaneous. Second, the mapping between the color image and thecamera space, or between the color image and the depth frame, is not available. In low-light levels, the acquisition rate might be limited by the color image refresh rate of 15 Hz. Threads The calls to the dll for the Cloud of point API have been changed to "run in any thread" from "run in UI thread". Theadvantage is that the dll can run in parallel with other VIs, allowing processing of the cloud of points to be processedduring acquisition. No problem was observed but in some cases, if calls are run in separate threads, some issuesmight arise. Please report any issue to HaroTek. User Events LabVIEW user events can now be used to transfer data directly from the dll to an event structure (see Cloud - UserEvent example). This feature helps to process the data in parallel with the acquisition. A similar approach couldbe implemented using the old "Acquire" command with little loss of performance. This feature was added for userconvenience.


128 of 301

Depth

Controls acquisition of depth data from Kinect 2. Depth data is a 2D array of 512x424 of unsigned 16 bit (codedin mm). Status returns a code related to the Kinect operation with a value of 0 for success. There is currently nocolor associated with this cloud of points because of a registration error in the Kinect SDK 2.0. The clouds of pointsprovided by this API however match the resolution of the depth sensor and will be faster than the Cloud API. If building an executable using one of the Kinect2 API, make sure to include the appropriate Kinect20.Fusion.dll (seemanual). Input parameters Command Required. Enum. Selects the



Init Initializes the Depth acquisition mode of theKinect.

Acquire Gets the depth data from the Kinect. Data isreturned in the Depth data array.

Close Closes the Depth mode.

Acq Cloud Acquires clouds of points from the Kinect.These clouds have dimensions that match thedimensions of the depth sensor but have nocolor. Data are output in Cloud points array


Standard LabVIEW™ error cluster.



Depth data U16 2D array. Output Depth data as a 2D arrayof 512x424 in dimension. The values in the arraycorresponds to the Z distance between the surface


129 of 301

of the object corresponding to the pixel in the arrayand the Kinect. Notice that the image is mirroredrelative to right and left (horizontal pixel indicesdecrease from left to right). Vertical pixel indicesincrease from top to bottom.

Cloud points Singles 1D array where X, Y, Z values are written foreach point consecutively. Correspond to the Pointsmember of the Cloud data cluster.


Depth The Depth mode is the fundamental acquisition of the Kinect. The 3D information is extracted from the depthinformation. The depth data can be transformed into a cloud of points (X, Y, Z) using the table provided by theCharacteristics command of the Kinect control API. The Z values are simply the depth data themselves. The X,Y can be calculated using the table as shown. This operation would be equivalent to using the Cloud API exceptwithout the corresponding color values. There is however little reduction of processing time, if any, to use the DepthAPI to produce the cloud of points instead of directly using the Cloud API. The table values provided by the Kinect were proven unreliable. The Depth API has therefore a new command "AcqCloud" that uses internal functions of the Kinect SDK to convert the depth data into a cloud of points. The cloud ofpoints is output as "Cloud points" that correspond to the "Points" member of the Cloud data cluster.


130 of 301

Color Image

Functions related to the acquisition of the color image from the Kinect V2. Image size is a constant to scale theimage from the Kinect. The Kinect always acquires an image of 1080x1900 pixels in dimension. The Image sizeconstant controls a downsampled image transferred to the VI to improve display and processing times. Statusprovides a feedback from the Kinect about the required operation. A value of 0 always indicates success. The Imagedata is a 2D array of U32 in RGB format. If building an executable using one of the Kinect2 API, make sure to include the appropriate Kinect20.Fusion.dll (seemanual). Input parameters Command Required. Enum. Selects the




Acquire Gets the depth data from the Kinect. Data isreturned in the Depth data array.

Close Closes the Color Image mode.

Color Image size Enum. Dimensions of theColor image to be acquiredfrom the Kinect.

This enum is from the same typedef than Cloudsize in the Cloud API.

1920x1080 Actual size of the color image from the Kinect.The other image sizes comes from a down-sampling done at the DLL level of the full imageacquired by the Kinect.

960x540

480x270

Error in (no error) Input error. VI is notexecuted if an error ispresent.


Output parameters


131 of 301

Status I32. Returns a value indicative of the status of therequested operation at the DLL level. Value of 0indicates success of the operation. See Status errorcodes for other values

Image data U32 2D array. Image data is a 2D array having thedimensions corresponding to the requested ColorImage size. The U32 values correspond to RGBAcolors according to the LabVIEW™ color standard.


Color Image The color image provided by the Kinect is High definition (1920x1080) at a rate up to 30 Hz. The acquisition ratedrops to 15 Hz at the Kinect level for low-lighting conditions.


132 of 301

Infrared

VI for the acquisition of the infrared signal from the Kinect V2. Data out is a 1D array of U16 having a dimension of424x512. If building an executable using one of the Kinect2 API, make sure to include the appropriate Kinect20.Fusion.dll (seemanual). Input parameters Command Required. Enum. Selects the



Init Initializes the Infrared acquisition mode of theKinect.

Acquire Gets the infrared data from the Kinect. Data isreturned in the Data out array.

Close Closes the Infrared mode.





Infrared data U16 2D array. Outputs Infrared data as a 2D arrayof 512x424 in dimension. The values in the arraycorresponds to a relative intensity value scaledbetween 0 and 65535. Notice that the image ismirrored relative to right and left (horizontal pixelindices decrease from left to right). Vertical pixelindices increase from top to bottom.



133 of 301

Infrared The infrared data is simply the intensity of the data acquired by the Kinect for the Depth mode. This new modeis now available with the Kinect V2 because of the use of the Time-of-Flight approach where the objects areuniformly illuminated by infrared light. In the Kinect 1, the use of a light pattern did not lean itself to an active infraredacquisition mode.


134 of 301

Kinect control

Starts, stops and gets characteristics from the Kinect V2. Upon start, if the status returns -1, no Kinect is available. If building an executable using one of the Kinect2 API, make sure to include the appropriate Kinect20.Fusion.dll (seemanual). Input parameters Command Required. Enum. Selects the



Start Turns the Kinect ON. Upon Start, if Status is -1,no Kinect is available.

Stop Turns the Kinect OFF.

Get Intrinsics Returns the table and intrinsics parameters





table 1D array of singles. Table corresponding to thecoordinate mapping between the Depth data andthe camera space. Table dimension is 2x424x512corresponding to two values for each value of the

Depth data. The 2*ith value of the table multiplied

by the ith value of the Depth data gives thecorresponding X coordinate value in the camera

space. The 2*ith + 1 value multiplied by the ith valueof the Depth data gives the corresponding Y value inthe camera space.


135 of 301

Characteristics Array of singles. Intrinsics of the Depth camera.

[0]: FocalLengthX

[1]: FocalLengthY

[2]: PrincipalPointX

[3]: PrincipalPointY

[4]: Radial Distortion 2nd order

[5]: Radial Distortion 4th order

[6]: Radial Distortion 6th order


tableentrycount Number of pairs of factors in the table. Should be424x512 for Kinect V2.

Kinect Start After the Start command, the LEDs of the Kinect are supposed to turn ON. However, no acquisition is taking place.Some LEDs might turn OFF after a while if no acquisition mode is initiated. Those LEDs will turn back ON if anacquisition is initiated. Occasionally, upon start, status returns -1 even if the Kinect appears to actually been turned ON. If this problemoccurs, re-starting the Kinect usually fixes the problem. Intrinsics ***************Notice that it was found that the intrinsics and table provided by this function are not the values actually used by theKinect. The clouds of points created by the intrinsics and the table tend to be distorted.The clouds of points corresponding to the actual depth sensor can now be obtained using the Depth API with theAcq Cloud command.*************** The intrinsics are supposed to be the parameters used to calculate the conversion table between the Depth cameraand the Camera space (real space relative to the Kinect). According to reference [5], the (X, Y) position in the Camera space (the Z value is simply the Depth value): X = Z * xc / fx and Y = Z * yc / fy;


136 of 301

where xc and yc are the corrected pixel value and fx and fy are the focal lengths in pixels as per the intrinsics array.

xc = (x-cx)*(1+k1*r2 + k2*r4 + k3*r6) and

yc = (y-cy)*(1+k1*r2 + k2*r4 + k3*r6)

where k1, k2, k3 are the radial distortion 2nd, 4th, and 6th order coefficients from the intrinsics array and r is theradial value from the center as per the intrinsics.

r = sqrt((x-cx)2 + (y-cy)2) where cx and cy are the principal points as per the intrinsics array, and x and y are theposition in pixels in the Depth 2D array. So, X = Z * Tx * (x-cy); and Y = Z * Ty * (y-cy); where

Tx = (1+k1*r2 + k2*r4 + k3*r6) / fx; and

Ty = (1+k1*r2 + k2*r4 + k3*r6) / fy; Tx, and Ty are supposed to be the values found in the table array returned by the "Characteristics" command of theKinectControl API. Unfortunately, the calculations using the intrinsics array values match the table values only fornegligible radial correction. Microsoft was contacted about this discrepancy and a response has not been received atthe time of writing this section. For more information about camera intrinsics, see references [4, 5].


137 of 301

Kinect Fusion

Controls the 3D reconstruction using the Kinect V2, called Fusion by Microsoft. At initialization, a volume is definedequal to the product of the X, Y, Z dimensions relatively to the original position of the Kinect. The volume is definedby voxels, cube volume units having an edge dimension of Resolution. The volume can be reconstructed with orwithout colors. The volume can be mirrored or not. The Camera pose finder is an option where the camera newvirtual position is recalculated using the object if tracking is lost. Finally, Max Z is the largest depth seen by theKinect for the volume reconstruction and Processor indicates to use the CPU or GPU. GPU is generally preferable

but the required memory is equal to X*Y*Z dims / resolution3 * 4 bytes (* 2 if color is used). Reconstruction is started at the command Init. Read reconstruction provides a 2D image (32-bit array) showing the3D reconstruction as currently seen by the Kinect. Mesh: A mesh can be created using the data accumulated in the volume. It is a triangular mesh. The meshdimension is the number of vertices. There are one color per vertex and one normal per vertex. The indices give theorder in which the vertices are ordered. Mesh dimension or tracking is a parameter that returns 0 if the tracking is lost when calling "Read reconstruction",and the dimension of the mesh is returned through that parameter when calling "Create Mesh". If building an executable using one of the Kinect2 API, make sure to include the appropriate Kinect20.Fusion.dll (seemanual). Input parameters Command Required. Enum. Selects the




Read Reconstruction Gets the Reconstruction image from the Kinect.The image is returned the parameter Imageout.

Reset Reconstruction Clears the volume data and restart theaveraging of the data for the 3D reconstruction.


138 of 301

Create Mesh Creates a triangular mesh using the dataaccumulated so far in the reconstructionvolume. Returns the dimensions of the mesh

Get Mesh Returns the data from the mesh previouslycalculated. Data is returned in Mesh data.

Close Closes the Fusion mode.

Fusion parameters in Fusion parameter cluster.Parameters used to definethe reconstruction volumeand some operational values.

Fusion parameters are set at the Init step.However, for each Read Reconstruction, theFusion parameters should be provided aswell. If a change in the parameters occur, thereconstruction is reset and the reconstructionrestart with the new parameters.



Output parameters Mesh data Mesh data cluster. Contains the mesh information

obtained from the "Get Mesh" command after themesh was created.

Status I32. Returns a value indicative of the status of therequested operation at the DLL level. Value of 0indicates success of the operation. See Status errorcodes for other values

Fusion parameters out Fusion parameter cluster. Output availablefor access to fusion parameters using the Getcommand.

Image out 2D array of U32. 512x424 in dimensions. Image inLabVIEW™ RGB format of the current image seenby the Kinect during 3D reconstruction


Mesh dimension ortracking

U32. Returns the number of vertices after MeshCreate command, and returns the tracking statusafter Read Reconstruction command (0 meanstracking is lost).

Camera matrix 1D array of singles. Current world to camera rotationtransformation matrix used by the Fusion process toadd the Kinect data to the virtual volume.

Fusion


139 of 301

The 3D reconstruction (called Fusion by Microsoft) consists in accumulating 3D data for a given point in space fromdifferent point of view of the Kinect. This approach presents the advantage of reducing the error by averaging data.The use of different point in the sensor also reduces the error associated by calibration issues related to the sensorpixel position (like radial distortion). The Kinect Fusion also has the ability to average colors for a given point in space. That feature and the trackinghave currently some issues that are expected to be solved by Microsoft in the next release of the Kinect SDK. Memory The data for the volume reconstruction is stored in memory. The size of the required memory is related to theresolution within the volume (size of the voxel) and the dimensions of the volume. The required memory is equal todimensions of the reconstruction volume divided by the dimensions of the voxel (cube of resolution) times 4 bytes.For example, a volume equal to 1 m x 1 m x 1.5 m with a resolution of 2 mm will require a memory of 1x1x1.5 /

(0.002)3 * 4 bytes = 0.1875 Gigabytes. This quantity of memory is increased by a factor 2 when color is also usedfor the reconstruction. For large volumes, or for small resolution, the quantity of memories can be significant. Therequired quantity of memory for the reconstruction must be available on the chosen processor. As a general rule, itmight be difficult to use a block larger than 1 Gigabyte on a graphic card even if more than 1 Gigabyte is available. Camera tracking When the Fusion is started, a reconstruction volume is defined. The coordinate system of the reconstruction volumeis defined by the coordinate system of the Kinect at the first frame acquisition. If the Kinect is moved after the firstframe acquisition, the 3D data from the Kinect must be corrected to compensate for the differences between thecoordinate systems of the reconstruction and of the Kinect new position. This compensation is done by a processcalled camera tracking. In camera tracking, the new position of the Kinect is calculated using the 3D features of thereconstruction volume and the 3D data from the Kinect at its current position. The camera tracking calculates whatmovement of the Kinect between the previous frame and the current frame would make it see the reconstructedvolume the way it sees it at its current position. Camera tracking is efficient on "natural" objects, meaning objectshaving irregular shapes of about medium sizes relative to the Kinect visible space. The more symmetrical andsmoother (or featureless) the objects are the worse the camera tracking will perform. The Kinect movement speedwill also affect the camera tracking performance. A feature related to camera tracking is camera pose finder. With this feature turned ON, the Fusion process attemptsto calculate what is the current position of the Kinect relative to the volume coordinate system when tracking is lost.Similarly to camera tracking, camera pose finder works better on "natural" objects. If camera pose finder is OFF, it is possible to recover tracking when it is lost (difference between the calculatedposition for the current and previously acquired frames too large) by moving the Kinect close to its position where thelast frame was successfully acquired. Mesh The command "Create Mesh" creates a triangular mesh based on the reconstructed volume so far. Upon thecreation of the mesh, the dimensions of the mesh are returned in the "Mesh dimension" array. The values areautomatically consistent (3 vertices per surface for example) so there is no need to check for consistency. A color isreturned for each vertex.


140 of 301

Upon the "Get Mesh" command, the data mesh is returned in the Mesh data cluster. The mesh data contains thevertices, the normals, the indices, and the colors. There are 3 coordinates per vertex (X, Y, Z) and one normal, index,and color per vertex. In theory, the three normals and colors should be the same for a given triangle. The indices arethe indices of the vertices array for each triangle, per group of 3. For example, the first three indices define the firsttriangle, the next three indices define the second triangle, and so on. In practice, the indices are generally simplyequal to their position in the array, for example: [0, 1, 2, 3, 4 ...].


141 of 301

Kinect API Controls Descriptions of the clusters exclusively used by the Kinect API's


142 of 301

Fusion parameters

Processor Enum. Indicates where

the data reconstruction iscalculated.

GPU Calculations are done on the graphic card. Thischoice is the default. Memory on the graphiccard is a limitation.

CPU Calculations are done on the host computerprocessor. Memory is generally less of anissue if the computer has enough RAM but thecalculations will be slower which can createproblems with the reconstruction.

Resolution Enum. Side size of the voxelsin the reconstruction volume.

Values are approximation. Exact value givenbelow.

1 mm 1024 voxels per meter of volume






X, Y, and Z dim Length of the reconstructionvolume

Fusion parameters are set at the Init step.However, for each Read Reconstruction, theFusion parameters should be provided aswell. If a change in the parameters occur, thereconstruction is reset and the reconstructionrestart with the new parameters.


143 of 301

Max Z I32. Maximum distancefrom the Kinect where the3D data are used for thereconstruction.

Color Boolean. When True,colors are also used forreconstruction.

Using the colors increases the memoryrequirement by a factor 2.

Mirror Boolean. When True, Volumeis mirrored relative to theKinect position.

Camera pose finder Boolean. When True, thealgorithm will attempt to finda position relative to theobject corresponding to thecurrent Kinect view of theobject.


144 of 301

Status error codes The Status value returned by the Kinect2 VI's provides a feedback on the operation at the DLL value. A value of 0indicates the operation was a success. All other values indicate an error. More information about a specific Status value different from 0 can be found at:http://msdn.microsoft.com/en-us/library/cc704587.aspx.

http://msdn.microsoft.com/en-us/library/cc704587.aspx


145 of 301

3D Objects

3D objects are objects created in 3D that can be displayed in a 3D environments. For the moment thoseenvironments are LabVIEW 3D picture controls and the Microsoft Hololens. 3D objects are instantiations of classes that are children or grand-children of the 3D Object class. This class isabstract (cannot be instantiated) and has two children: 3D Shape and OBJ mesh.


146 of 301

To LabVIEW 3D Pict

VI to create a LabVIEW 3D Pict object from a generic Haro3D 3D object. This VI works with any of the child orgrand-child of the 3D Object class. If the child class is decided at edit time, the above VI automatically changes at edit time when the 3D Object in isconnected to the corresponding VI shown below. If the 3D object in is not decided at edit time, the VI will changeonly at run time (dynamic dispatching). Input parameters 3D object in Required. 3D object or any

child of the 3D object class.



Output parameters 3D Object out 3D Object or Object of class corresponding to the

input

LabVIEW 3D Object ref Refnum of object to be used in a LabVIEW 3Dpicture control.


Box

Creates the reference of a box for a LabVIEW 3D picture control using the parameters of the 3D object input Cylinder


147 of 301

Creates the reference of a cylinder for a LabVIEW 3D picture control using the parameters of the 3D object input.Notice that orientation of cylinder is along the Y axis instead of along the Z axis, as standard with LabVIEW. Mesh

Creates a reference to a LabVIEW 3D picture object from the current mesh Sphere

Creates the reference of a sphere for a LabVIEW 3D picture control using the parameters of the 3D object input.Notice that only the X dimension is used for the LabVIEW sphere even though dimensions are X, Y, and Z to give theopportunity to create ellipsoids. OBJ Mesh

Creates a LabVIEW 3D Pict object reference from an OBJ Mesh object.


148 of 301

3D object properties Properties associated with 3D objects.


149 of 301

Material Info

Cluster of properties defining the appearance of a 3D object.

Name is a string and is the name of the material. It is used to define the name of a mesh object if more than onematerial is used in an OBJ_Mesh object. Ka, Kd, Ke, and Ks are int32 color representation (using LabVIEW color constant). Ka: ambient color (not affected by lighting).Kd: Diffuse color (uniformly reflects lights)Ke: emissive color (emits its own color)Ks: specular color (strongly reflects light at angles corresponding to incident light angles). map_Ka, map_Kd, and map_Ks are string give the name of the file containing the image used for texture for thecurrent object. File images for textures can be of .bmp, .jpg, or .png. Only one texture can be used at a time withLabVIEW and the Haro3D library. In case more than one file name is provided, the preference is given in the order:map_Ka, map_Kd, and map_Ks. Texture will be affected by the Ka and Kd value in the Hololens space. The other color values do not affect thetextures.


150 of 301

Ns is a single float taking a value between 0 and 1000 and represents shininess (0: not shiny, 1000: very shiny).Notice that shininess is not used with textures Tr is a single float taking a value between 0 and 1 and represents the transparency of an object (0: opaque, 1: fullytransparent). The Tr value corresponds to 1 minus the alpha value. Notice that the Tr value is not used with textures. Information about the link between the .mtl file used with the .obj format can be found in the OBJ File Load topic.


151 of 301

Position Position cluster. This cluster is a standard cluster used by LabVIEW.

Cluster of 3 singles, X, Y, Z. Can be used for a position or a direction (normal for example).


152 of 301

Rotation data

Index: Index of 3D object in Hololens space. Orientation: Quaternion cluster. Position: Position cluster.


153 of 301

Color information LabVIEW™ has two methods of dealing with RGB colors. The first method, and the most universal, is to representthe RGB color as a U32 value where 3 bytes represents a color, and the fourth byte is the Alpha channel, that is ofno concern here. Each color is coded over 1 byte, having a value between 0 and 255. LabVIEW™ provides two VIsto deal with those conversions:

Where Color is a U32 and each color is a U8 coded between 0 and 255. Those two VIs can be found in the Graphics& Sound -> Picture Functions or in the Numeric -> Conversion palettes. Other conversion VIs are available in theVision development module. This method of dealing with colors is used with pictures and images. Notice that thereis no universal representation of the colors using a U32. The U32 color can be represented by an array of 4 U8.LabVIEW™ puts the Alpha channel in the first U8, followed by the RGB. This order does differ from one applicationto the other. Also, notice that LabVIEW offers a control color box (found in the Numeric sub-palette of the Controls palette, and inGraphics & Sound -> Picture Functions in the Functions palette) :

A color can be selected by just clicking on the color box. A color picker dialog then appears:

The second method of color representation that LabVIEW uses is a cluster of 4 singles (Red, Green, Blue, andAlpha, see below). Each color is coded between 0 and 1. This method of dealing with colors is used in functionsrelated to 3D objects.


154 of 301


155 of 301

Shape Class Shape class is a child of the 3D Object class. Shape is an abstract class representing 3D object having a specificshape. The various possible shapes (children of the Shape class) are: Box, Cylinder, Mesh, Panel, and Sphere.


156 of 301

Create Box

Creates a 3D box. Dimension is a 3-single structure = X,Y, Z giving the dimensions in those 3 directions. Quality isa factor (coerced) between 0.1 and 1.0 related to the number of faces of the box for LabVIEW 3D picture objects.Material info is the appearance and Position and Orientation are the starting position and orientation, as given by a3D point and a quaternion. Input parameters Visible (T) Boolean. Visibility of the object. User cannot interact with

the object when visibility is false.

Quality (1) Float. Quality factor of the object drawing. Currentlyonly used in LabVIEW objects. 1 is themaximum quality.

Dimensions Position cluster. Dimension of the box in the (X, Y, Z) directions.

Material Info Material cluster. Appearance of the box.

Position Position cluster. Initial position of the box in the LabVIEW andHololens space.

Orientation (no rotation) Quaternion. Initial orientation of the box in the LabVIEW andHololens space.



Output parameters Cylinder out Cylinder object.



157 of 301

Create Cylinder

Creates a 3D cylinder. Parameters is a 3-single structure with [Length, Top radius, Bottom radius]. Currently, only thebottom radius is not used. Length is along Y. Quality is a factor (coerced) between 0.1 and 1.0 related to the numberof faces of the cylinder. Material info is the appearance and Position and Orientation are the starting position andorientation, as given by a 3D point and a quaternion. Notice that orientation of cylinder is along the Y axis instead of along the Z axis, as standard with LabVIEW 3DCylinder objects. Input parameters Visible (T) Boolean. Visibility of the object. User cannot interact with

the object when visibility is false.

Quality (1) Float. Quality factor of the object drawing. Currentlyonly used in LabVIEW objects. 1 is themaximum quality.

Dimensions Cylinder parameters cluster. Length (Y direction, before any rotation), Topradius, and Bottom radius. Bottom radius is notused for the moment (cylinder has a uniformradius).

Material Info Material cluster. Appearance of the cylinder.

Position Position cluster. Initial position of the sphere in the LabVIEW andHololens space.

Orientation (no rotation) Quaternion.



Output parameters Cylinder out Cylinder object.



158 of 301

Create Sphere

Creates a 3D sphere. Parameters is a 3-single cluster with [X, Y, Z] where each value is the diameter in thecorresponding direction (only X is used for LabVIEW 3D pict objects). Quality is a factor (coerced) between 0.1 and1.0 related to the number of faces of the sphere. Material info is the appearance and Position is the starting position.Visible indicates if the sphere is visible or not. Orientation is not used and is only provided for consistency with othercreate object functions. Input parameters Visible Boolean. Visibility of the panel. User cannot interact with

the panel when visibility is false.

Quality Float. Quality factor of the sphere drawing. Currentlyonly used in LabVIEW objects. 1 is themaximum quality.

Dimensions Position cluster. Diameters of the sphere in the (X, Y, Z)directions. Only the X value is used for theLabVIEW 3D spheres.

Material Info Material cluster.

Position Position cluster. Initial position of the sphere in the LabVIEW andHololens space.

Orientation (not used) Quaternion. Not used. Provided for consistency with other3D objects.



Output parameters Sphere out Sphere object.



159 of 301

Create Panel

Creates a panel. A panel is a 3D flat rectange, mostly used to display a control or VI front panel. The panel isautomatically positioned in the field of view of the user, at approximately 2m from the user but in front of any virtualobject that might be obstructing. Material info, Position, and Orientation are used only for LabVIEW 3D pictureobjects. Tag along determines if the panel remains in the field of view of the user when he moves (true) or it remainsat the provided position (false). Input parameters Dimensions Required. Position cluster. Dimensions in X and Y. Z is not used.

Visible (T) Boolean Visibility of the panel. User cannot interact withthe panel when visibility is false.

Tag along (T) Boolean Determines if the panel remains in the field ofview of the user when he moves (true) or if itremains at the provided position and orientation(false).

Material info Material cluster. Material is used only for LabVIEW 3D pictureobjects.

Position Position cluster. Position is used only for LabVIEW 3D pictureobjects.

Orientation (no rotation) Quaternion cluster. Orientation is used only for LabVIEW 3D pictureobjects.



Output parameters Panel object Panel object.



160 of 301

Mesh Class

Mesh is a class of 3D objects made of surfaces defined by an array of vertices. The surfaces can be triangles orquads. The Hololens work only with triangles. When uploaded to the Hololens, if any quads are found in the mesh,the quads are transformed into triangles but it is better to have the mesh originally created with triangles whenpossible, or at least have it modified using a dedicated application like Meshlab [8]. Here are the VIs associated with the Mesh class.


161 of 301

Create Mesh

Creates a new mesh object. Notice that when this VI is explicitly called, Destroy Mesh must be called when the Mesh object is not necessaryanymore. Input parameters Visible (T) Boolean. Visibility of the object. Hololens user cannot

interact with the object when visibility is false.

Quality (not used) Float. Quality factor. Not used for the mesh. Inputprovided only for consistency with other createobject VIs.

Vertex array Required. Array of positionclusters.

Each position is one point of the array. Pointsshould be organized in triangles or quads (onlytriangles work for the Hololens). An array ofindices for either triangles or quads, or both,should be provided using property accessors ofthe created mesh object.

Material Info Required. Material cluster.

Position (0,0,0) Position cluster. Initial position of the object when uploaded inthe LabVIEW and Hololens space.

Orientation (no rotation) Quaternion. Not used. Provided for consistency with other3D objects.



Output parameters Mesh out Mesh object.



162 of 301

Calculate Normals

Calculates the normals of the mesh using the current vertices if normals were not provided with the mesh. Valid onlyfor triangle meshes. The calculated normals are only used by LabVIEW 3D picture controls. The Hololens generates its own normals. Input parameters Mesh object in Required. Mesh object. If normals are already available in the input

Mesh object, the object is not modified.



Output parameters Mesh object out Mesh object with normals (newly calculated or non-

modified if the normals were already available).



163 of 301

CleanUp Mesh

Releases the memory allocated when the current mesh object was created Notice that the Mesh is the only child of the Shape class that requires a call to a CleanUp function. The CleanUp VIis dynamically dispatched. If the instantiation of the Shape is not known at the time CleanUp VI is called, the parentVI would be called if the CleanUp VI is applied to a Shape other than the Mesh. The parent VI is empty. Input parameters Mesh in Required. Mesh object.

error in (no error) Input error. VI is executedeven if an error is present.


Output parameters Mesh out Mesh object with no allocated memory.



164 of 301

Generate Simple Shape

Generates 3D meshes of basic shapes: box, cylinder, sphere, and geosphere. The benefit of this VI over thosegenerated by the native LabVIEW 3D object library is that the shape can be uploaded as mesh to 3D displays otherthan 3D picture controls. It also presents the advantage of a better control on the number of generated verticesthrough the quality parameter. Parameters 1, 2, and 3 are Length, bottom radius, top radius for the cylinder; dim X, dim Y, dim Z for the box, andonly parameter 1 is used for the sphere and the geosphere (radius). The difference between the sphere and thegeosphere is the size of the triangles are all the same for the geosphere. The quality is an arbitrary factor for thequality of mesh (number of triangles). 0 is low quality and 1 high quality. Notice that this VI requires SimpleMesh.dll (or SimpleMesh_x64.dll if using 64-bit LabVIEW). The dlls should belocated in the same directory than this VI. Input parameters Visible (T) Boolean. Visibility of the object. Hololens user cannot

interact with the object when visibility is false.

Quality (1) Float. Quality factor. An arbitrary factor for the qualityof mesh (number of triangles). 0 is low quality(lower number of triangles) and 1 is high quality(higher number of triangles).

Shape type Required. Shape Typetypedef.

Format of the shape generated. The possibleshapes are: Cylinder, Box, Sphere, andGeosphere.

Parameters Required. Position clusters. Parameters 1, 2, and 3 are Length, bottomradius, top radius for the cylinder; dim X, dim Y,dim Z for the box, and only parameter 1 is usedfor the sphere and the geosphere (radius).

Material Info Required. Material cluster.

Position (0,0,0) Position cluster. Initial position of the object when uploaded inthe LabVIEW and Hololens space.

Orientation (no rotation) Quaternion. Not used. Provided for consistency with other3D objects.


165 of 301



Output parameters Mesh out Mesh object.



166 of 301

Controls Controls used in the Shape class and its children.


167 of 301

Cylinder parameters

Length: Length of the cylinder, in the Y direction when the cylinder has no rotation. Top radius: radius of the cylinder for the highest Y value, when the cylinder has no rotation. Bottom radius: radius of the cylinder for the lowest Y value, when the cylinder has not rotation.


168 of 301

Shape Type

Different types of shapes that can be generated as meshes by the Generate Simple Shape VI.


169 of 301

OBJ Mesh Class

OBJ (or .OBJ) is a geometry definition file format first developed by Wavefront Technologies. It is a universallyaccepted format for representation of 3D mesh objects allowing for colors and texture mapping. More information on the .OBJ format can be found at reference [14]. The OBJ Mesh class is a child of the 3D Object class. It has no child. An OBJ Mesh is an aggregate of Mesh objects,linked together through the definition given in an .obj file. The OBJ Mesh functions (except Calculate Normals) can handle both triangular and quad meshes. However, theHololens only uses triangular meshes. Therefore, any quad mesh must be converted into a triangular mesh beforesending it to the Hololens. It is recommended to use a 3D processing software like Meshlab ([8]) to convert any quadmesh into a triangular mesh prior to loading it into the Haro3D to ensure the best results.


170 of 301

OBJ File Load

Loads a mesh from an OBJ file and creates a corresponding 3D object Opens an .obj file and returns a 3D object reference corresponding to the object described by the .obj file. Supportsonly triangle and quads, and a single texture. Texture files must be in .BMP, .JPG or .PNG formats. The returnedclass object can be used to extract the different parts of the 3D object. Parts are defined by material definitions usedfor each part as per the .mtl file. If the same material definition is used for 2 different parts, each part will be listedseparately. Notice that Destroy OBJ must be called once the OBJ Mesh is not necessary anymore. Only one texture file can be used for .OBJ object (LabVIEW does not support multiple textures in 3D picturecontrols). For each material definition, a single texture is loaded for the map_Kd, map_Ka, or map_Ks parameters,in that order. There is no difference between these mapping types; texture is copied on the surface of the meshwithout blending. Other texture parameters (bump for example) are ignored. For the texture files, options (like -mmfor example) before the file names need to be removed. Options after file names are simply discarded. The amplitude of the normals as defined in the .obj files are normalized when loaded. The .OBJ_File_Load.vi loads the file using the point as a decimal separator, independently from the LabVIEW andcomputer settings. Input parameters Path Required. LabVIEW path. Path of the OBJ file.

Name String. Name of the OBJ object. Uses the name of thefile is empty.

Scale (1) Single. Multiplication factor applies to the size of theobject. Default is 1 and corresponds to noscaling.



Output parameters OBJ object OBJ object.



171 of 301

Additional info Notice that the Haro3D library can load quad and triangular meshes from .obj files. However, the Hololens can workonly with triangular meshes. The Haro3D library will convert quad meshes into triangular meshes but better resultsare generally achieved by using a software like Meshlab [8]. Modifying the .mtl file The material information provided by the .mtl file is loaded into the Material Info cluster. The material definitions loaded by the OBJ_File_Load.vi can be modified by directly modifying the text of the .mtl file. The properties of the .mtl file that affects the appearance of a 3D object loaded by OBJ_File_Load.vi are given in thetable below. All other properties of the .mtl are neglected.

Properties Effect on materialappearance

Parameters Range

Ka Ambient color R G B 0.0 to 1.0Kd Diffuse color R G B 0.0 to 1.0Ke Emission color R G B 0.0 to 1.0Ks Scattering color R G B 0.0 to 1.0

Ns Shininess S 0 to 1000d Dissolve

(transparency)D 0 to 1.0

map_Kd texture map (selectedover map_Kaand map_Ks)

filename .jpg, .bmp,or .png

image formatmap_Ka texture map (selected

over map_Ks)filename .jpg, .bmp,

or .pngimage format

map_Ks texture map filename .jpg, .bmp,or .png

image format Using any of the texture map property results in the same material appearance. Also, if more than one of the threetexture mapping property is specified for a given object section, only one is used, map_Kd being used in priority,and map_Ka being used over map_Ks. If additional options are specified with the texture map properties after thefilename, those options are ignored. If the options are specified before the filename, OBJ_File_Load.vi will generatean error. It is therefore preferable to simply delete the options between the texture map property and the filename.Notice that adding the # character in front of a line of a .mtl file transforms that line into a comment ignored byOBJ_File_Load.vi. It is suggested to copy the lines before modification and to use the # character in front of theoriginal lines to keep the full information of the original files. Example of a material (.mtl) file


172 of 301

Below is an example of a .mtl file. Comments about how the various properties and parameters are handled byOBJ_File_Load.vi are shown in red. newmtl lambert2SG -> Beginning of new material definition. If used by a part ofa 3D object, this part of 3D object will use "lambert2SG X" where X is the number oftimes that material definition is used for the current object.illum 4 -> Ignored Kd 0.00 0.00 0.00 -> Diffused color (R = 0.0, G = 0.0, B = 0.0)Ka 0.00 0.20 1.00 -> Ambient color (R = 0.0, G = 0.2, B = 1.0)Tf 1.00 1.00 1.00 -> Ignoredmap_Ks Spec.png -> Ignored because a texture for map_Kd is present.map_Kd ../Color01152015.jpg -> Applies texture map from file namedColor01152015.jpgbump Bump012714.jpg -bm 0.05 -> IgnoredNi 1.00 -> IgnoredKs 0.00 0.00 0.00 -> Specular color (R = 0.0, G = 0.0, B = 0.0) newmtl phong1SG -> Beginning of new material definition. If used by part of a 3Dobject, this part of 3D object will use "phong1SG X" where X is the number of timesthat material definition is used for the current object.illum 4 -> Ignored Kd 0.00 0.00 0.00 -> Diffused color (R = 0.0, G = 0.0, B = 0.0)Ka 0.60 0.50 0.30 -> Ambient color (R = 0.6, G = 0.5, B = 0.3)Tf 1.00 1.00 1.00 -> Ignoredmap_Kd -mm 0.188525 1 Tooth.png -> Generates an error because of the "-mm 0.1885251" options before the file name.#map_Kd -mm 0.188525 1 Tooth.png -> Ignored because of the # character at thebeginning of the lineKs 0.4 0.5 0.1 -> Specular color (R = 0.4, G = 0.5, B = 0.1)Ns 41.88 -> Shininess of 41.88newmtl phong2SG -> Beginning of new material definition. If used by part of a 3Dobject, this part of 3D object will use "phong2SG X" where X is the number of timesthat material definition is used for the current object.map_Ks Mouth.png -bm 0.05 -> Ignored because a texture for map_Ka is present.map_Ka Tooth.png -bm 0.05 -> Applies texture map from file named Tooth.png. "-bm0.05" options are ignored.Ni 1.00 -> IgnoredKs 0.50 0.50 0.50 -> Specular color (R = 0.5, G = 0.5, B = 0.5)


173 of 301

Cleanup OBJ Mesh

Releases the resources associated with an OBJ Mesh object allocated when the OBJ file was loaded. Input parameters OBJ object Required. OBJ object. OBJ object is created using OBJ File Load.

error in (no error) Input error. VI is executedeven if an input error ispresent.




174 of 301

OBJ Send to Hololens

Sends all the meshes making up a OBJ mesh object to the Hololens Input parameters OBJ object Required. OBJ object. OBJ object is created using OBJ File Load.






175 of 301

OBJ Calculate Normals

Calculates the normals for each of the sub-mesh of the OBJ object. Input parameters OBJ object Required. OBJ object. OBJ object is created using OBJ File Load.






176 of 301

Utilities Descriptions of the utilities and controls provided with the Haro3D library to help display, save, and read the dataprovided by the Kinect. Palettes If the Haro3D™ library was installed using the provided instructions, the Haro3D™ Utilities VI's should be accessiblethrough the LabVIEW™ function palettes: Addons -> HaroTek -> Haro3D -> Utilities (as shown below).


177 of 301

3D Operations

Functions used in the Haro3D library for mathematical operations on 3D objects.


178 of 301

Rotation angles to matrix

Calculates the translation-rotation matrix from the 3 rotation angles a, b, c (rotation around Z, Y, and then X). Equations are from reference [18]. Input parameters Units (Deg) Enum. Indicates the units of

the angles. 0: Degrees, 1:Rad

a Required. Single. Angle "a",in units as provided by theUnit input.

b Required. Single. Angle "b",in units as provided by theUnit input.

c Required. Single. Angle "c",in units as provided by theUnit input.

Output parameters matrix Singles. 4x4 LabVIEW matrix. Resulting rotation-

translation matrix.


179 of 301

Quaternions multiplication

Multiplies 2 quaternions to produce a new quaternion. This operation corresponds to applying rotation r to rotation q(in that order). Input parameters q Required. Singles. 4 element

array. Quaternion.See Quaternion.

r Required. Singles. 4 elementarray. Quaternion.

See Quaternion.



Output parameters q x r Singles. 4 element array. Product of the 2

quaternions.



180 of 301

Vector rotation by quaternion Obsolete. Use Vector Rotation instead.

Rotates up to 4 vectors using a quaternion. Input parameters Quaternion Required. 4 element array.

Quaternion representingthe rotation to apply to thevectors.

See Quaternion.

Vector 1 in Required. Singles. 3 elementarray. Vector to be rotated.

Vector 2 in Singles. 3 element array.Vector to be rotated.



Output parameters Vector 1 out Singles. 3 element array. Vector 1 after rotation.

Vector 2 out Singles. 3 element array. Vector 2 after rotation.




181 of 301

Vector Rotation

Rotates a single vector using a quaternion Input parameters Vector in Position cluster. Input vector

to be rotated.

Quaternion Required. Quaternion clusterrepresenting a rotation.

Inverse rot (F) Boolean. Inverse the rotationif true.



Output parameters Vector out Position cluster. Resulting rotated vector.



182 of 301

Vector to Rot-Trans Matrix

Converts a 1D array made of X, Y, Z translations and a, b, c, rotations (rotation around Z, Y, and then X) into arotation-translation matrix. Input parameters Units (Deg) Enum. Indicates the units of


XYZabc vector Required. Single array. Angle"a", in units as provided bythe Unit input.



Output parameters Rot-Trans matrix 4 x 4 LabVIEW Matrix.



183 of 301

Euler Angles to Quaternion

Calculates the quaternion using the angles a,b, and c in degrees by default. a is defined as rotation around the Z axisb is defined as rotation around the Y axisc is defined as rotation around the X axis Input parameters a Required. Single. Rotation

angle around Z.See Quaternion.

b Required. Single. Rotationangle around Y.

c Required. Single. Rotationangle around X.

Units (Deg) Units of angles. Degrees bydefault.



Output parameters quaternion More information about quaternions can be found in

reference (4-element linear single array) [6].


Additional Information Additional information on the Euler angles to quaternion transformation can be obtained from reference [19].


184 of 301

Rot-Trans Matrix to Quaternion

Converts a rotation-translation matrix (4x4) into a quaternion and translation vector. Input parameters Rot-trans matrix Required. 4 x 4 LabVIEW

Matrix.



Output parameters quaternion See Quaternion.

translation Translation vector (3-element linear single array).



185 of 301

Quaternion to Rotation Axis

Converts a rotation expressed as a quaternion into a rotation axis and an angle. This VI is useful when interacting with the LabVIEW 3D picture rotation VIs that require a rotation axis and an angle. Input parameters Quaternion Required. Quaternion cluster

representing a rotation.See Quaternion.



Output parameters Axis LabVIEW cluster of floats: (x, y, z) corresponding to

the rotation axis.

angle Angle in radians of the rotation



186 of 301

Toggles Right Left System

Toggles a position and a quaternion clusters between right and left coordinate systems. Input parameters Units (Deg) Enum. Indicates the units of


XYZabc vector Required. Single array. Angle"a", in units as provided bythe Unit input.



Output parameters quaternion Quaternion cluster.

Position out Position cluster.



187 of 301

Controls Controls associated with the 3D Operations utilities.


188 of 301

Quaternion

The quaternion control is a 4-element cluster: (w, x, y, z). A quaternion represents a rotation of a given angle arounda given axis. The sum of the square of its components must be equal to 1. As an absolute orientation, a quaternion represents the rotation from the default orientation to the current orientation. A quaternion is defined as w+ x + y + z where w = cos(a/2)x = sin(a/2)* xRy = sin(a/2)* yRz = sin(a/2)* zR where (xR,yR,zR) is the normalized direction of the rotation axis and a is the rotation angle. Therefore, (w = 1, x = 0,y = 0, z = 0) represents no rotation (a = 0°). More information about quaternions can be found in reference [6].


189 of 301

Kinect Utilities

Functions to process data from the Kinect.


190 of 301

Bodies Descriptions of the utilities related to display of Bodies (joints and limbs).


191 of 301

Bodies 3D Display

Draw into the 3D object Scene the bodies form the Kinect V2. Colors is a 6 element color cluster array where eachcolor is assigned to the corresponding body of the Bodies array of clusters. Input parameters Scene in Required. 3D scene Refnum. 3D scene refnum where the Bodies will be

displayed.

Bodies Required. 1D array of 6elements of Body info

Colors Required. 1D array of 6elements of the LabVIEW™color cluster.

Color in which each of the bodies will bedisplayed.



Output parameters Scene out 3D scene Refnum. Copy of the Scene in refnum.

Provided here to easily close the refnum.

3D pict out 3D pict Refnum. To be connected to a 3D picturecontrol.


Bodies 3D Display This VI will draw the bodies inside a 3D scene that needs to be created prior to calling this VI. To repetitively drawthe bodies, a new 3D scene must be created before each display. The 3D scene object refnum must be closed afterthe display (using the Scene out for example) to prevent a memory leak.


192 of 301

Controls Descriptions of the clusters used by the Bodies VI's.


193 of 301

Body info

IsTracked Boolean. Indicates if the

current body is being tracked.If the body is not tracked, the other elements ofthe cluster are not filled.

Tracking ID I64. ID number assigned tothe body by the Kinect.

This ID number is only for currently trackedindividuals. If an individual steps out of viewfrom the Kinect and steps back in, he will beassigned a new ID number.

Left Hand State, RightHand State

Enum. States of the right andleft hands has detected bythe Kinect.

Unknown Hand is tracked but its state is unknown

Not tracked Hand is not tracked.

Open Hand is open

Closed Hand is closed.

Lasso Hand is closed with the index and middle fingersextended.

Joints 1D array of 25 elements ofthe cluster Joint info.

The 25 elements of this array provides theinformation on the up to 25 joints that are beingtracked by the Kinect.


194 of 301


195 of 301

Joint info

Joint type Enum. Gives the type out of

the 25 of the current joint.

SpineBase

SpineMid

Neck

Head No parent bone. Orientation is therefore always[0,0,0,0].

ShoulderLeft

ElbowLeft

WristLeft

HandLeft

ShoulderRight

ElbowRight

WristRight

HandRight

HipLeft

KneeLeft

AnkleLeft

FootLeft

HipRight

KneeRight


196 of 301

AnkleRight

FootRight

HandTipLeft

ThumbLeft

HandTipRight

ThumbRight

Tracking state Enum. Tracking state of thecurrent joint.

Not Tracked

Inferred Joint position and orientation are estimatedbased on the other joints.

Tracked

Position Cluster of 3 singles giving theX, Y, Z position of the joint inthe Camera space.

Orientation Cluster of 4 singles givingthe absolute orientationof the parent bone (limbbetween current joint andnext joint closer to the head).The cluster is a rotationquaternion.

Rotation quaternions are used in the gamingindustry and in the 3D display community ingeneral to represent rotation. Quaternions areequivalent to rotation matrices or Euler anglesbut present computational advantages. Moreinformation can be found in reference [6].


197 of 301

Cloud of Points Descriptions of the utilities related to display and storage of cloud of points. A cloud of points is a set of (X, Y, Z) coordinates that give the position of a point in space. This point can beaccompanied by a color. Cloud of points can be viewed, analyzed, and processed using applications like Meshlab [8] or CloudCompare [9].


198 of 301

Cloud 3D Display

Cloud data is a cluster with 2 elements: Points and Colors. Points is a 1D array of float with 3 times number of cloudpoints, each set of 3 singles being X, Y, and Z. Colors is an array of U32 for each point coded in RGB. All points witha Z value larger than "Max Z" are set at the position (0,0,0) in black. Input parameters SceneObject in Required. 3D scene Refnum. 3D scene refnum where the cloud of points will

be displayed.

Cloud data Required. 1D array of 6elements of Body info



Max Z Required. Single. Datahaving Z position larger thanZ max are not displayed.

To improve the refresh rate, data with z valueslarger than Max Z are colored to black anddisplayed at position (0,0,0).

Output parameters 3D pict out 3D pict Refnum. To be connected to a 3D picture

control.


Cloud 3D Display A new 3D scene does not need to be created before each Cloud display. The same 3D scene can be repetitivelyused and disposed only when completely finished with the display. This function assumes that the data are provided in mm but data are displayed in m in the 3D picture because datawith values near 1 are better handled by the 3D picture. The camera view of the 3D picture is not adjusted for thecurrent Cloud data.


199 of 301

Cloud 2D Display

Converts a cloud of points into a 2D picture. Cloud data is a cloud of points cluster with 3 x number of data points,each set of 3 floats being X, Y, and Z. The texture is the color for each point coded in RGB. All points with a Z valuelarger than "Max Z" or lower than Min Z are represented by a pixel of the color background. All other points arerepresented in the 2D picture by their original color. Image size is an enum with the following possible picture sizes:480x270, 960x540, 1920x1080. Input parameters Color Image size Required. Enum. Values

are 1920x1080, 960x540,480x270

2D Picture in Required. LabVIEW 2Dpicture.

Cloud data Required. Cloud data cluster.

Background (black) U32. LabVIEW color assigned to pixels that have acorresponding z position below the Min Z orabove the Max Z.



Max Z Required. Single. Datahaving Z position larger thanZ max are not displayed.

Data with z values larger than Max Z arecolored to background color

Min Z (0) Single. Pixels having Zposition lower than Z min areassigned the backgroundcolor

Output parameters 2D Picture out LabVIEW 2D picture.



200 of 301

Cloud 2D Display Uses the 3D data associated with each pixel of the Color image to filter out certain ranges of data beyond a Max Zvalue or below a Min Z value. This VI can be used to produce "green screen" images where a different backgroundcan be added to an image or a video.


201 of 301

Read Cloud File

Reads a Cloud data from a file in PLY (ASCII or binary) or XYZ ASCII format. Filename and File format must beprovided. Input parameters File format Required. Enum.

PLY Binary. PLY file format. This VI will correctly read eithera PLY Binary or a PLY ASCII file independentlyif the PLY Binary or PLY ASCII is selected. Formore information about the PLY file format seereference [7].

PLY ASCII PLY file format.

XYZ ASCII Simple ASCII format consisting of thecoordinates and colors values written on a linefor each point: X Y Z R G B

File path Required. Path variable.Contains the file path andname.



Output parameters Cloud out Cloud data cluster. Data from the file.

File incompatible Boolean. If True, file is found to be not compatiblewith this VI.


Read Cloud File This VI is not guaranteed to work with Cloud data saved with other software or VI's not part of the Haro3D™ library,even files saved in the recognized formats.


202 of 301

Currently, the Read Cloud File function does not distinguish between a PLY mesh file and a PLY cloud file. If a meshdata set saved in the PLY file format (using the Haro3D™ Mesh Save VI) is read using the Haro3D™ Cloud ReadVI, the mesh data will be converted to a Cloud data and the file will not be indicated as incompatible. However, theCloud data provided by the Kinect are in mm while the mesh data provided by the Kinect are in m. The user shouldtherefore account for the difference in scale for proper display. The XYZ ASCII format is provided only for compatibility with other software. HaroTek recommends to use the BinaryPLY format for both Cloud and Mesh data. The Cloud_Read.vi loads ASCII files using the point as a decimal separator, independently from the LabVIEW andcomputer settings.


203 of 301

Save Cloud File

Saves cloud data into a binary or ASCII PLY file, or in an ASCII XYZ file. File path (including name) must beprovided. Input parameters Data author String. String added as a

comment in the header of thefile as the author of the file.

The string should not contain carriage return orlinefeed characters. This string is not used forthe XYZ ASCII format.

File format Required. Enum.

PLY Binary. Cloud data willbe written in the binary PLYfile format.

For more information about the PLY file formatsee reference [7].

PLY ASCII. Cloud data will bewritten in the ASCII PLY fileformat.

For more information about the PLY file formatsee reference [7].

XYZ ASCII. Cloud data willbe written in the ASCII XYZformat.

Simple ASCII format consisting of thecoordinates and colors values written on a linefor each point: X Y Z R G B


Cloud data Required. Cloud data cluster.Data to be save in the file.



byte order (big endian) Enum. Byte order of the datawritten in binary format.

Byte order does not matter for PLY formatbecause the endianess of the binary data isindicated in the file header.

Big endian Format used by LabVIEW™.


204 of 301

Native, host order Format used by the operating system of thehost.

Little endian Format used by Windows OS for example

Output parameters error out Standard LabVIEW™ error cluster.

Save Cloud File HaroTek recommends to save the data in the Binary PLY format. Use the other formats for compatibility with othersoftware or VI's from other library. The Cloud_Save.vi saves ASCII files using the point as a decimal separator, independently from the LabVIEW andcomputer settings.


205 of 301

Controls Descriptions of the clusters used by the Cloud VI's.


206 of 301

Cloud data

Points 1D array of singles. X, Y, Z values are written for each point

consecutively.

Colors 1D array of U32. Each valuecorresponds to the color of apoint.

Color is in the RGBA format standard toLabVIEW™.


207 of 301

Mesh Data Functions Descriptions of the utilities related to display and storage of meshes. A cloud of points is a set of (X, Y, Z) coordinates that give the position of a vertices in space. A given subsetof vertices define a face. Each subset defining a face can be defined by indices. The face can have a normalassociated with it. The normal is a vector giving the direction of the normal (perpendicular) relative to the face. Acolor can be associated with each vertex or with a surface. The functions presented here only considers triangular meshes. Triangular meshes can be viewed, analyzed, and processed using applications like Meshlab [8] or CloudCompare[9].


208 of 301

Mesh 3D Display

Displays a 3D set of data as a mesh in a 3D picture. The input data (Mesh cluster containing Vertices, Normals,Indices, and Colors) must be properly formatted for a triangular mesh. The center of the view is the average of thevertices, and the camera is positioned at "Camera position" relative to the center of the view, looking towards thecenter. Input parameters Camera Position LabVIEW™ position cluster

(X Y Z).Position of the camera relative to the center ofmass of the mesh data. The camera will looktowards that center of mass.

3D Pict ref in Required. 3D picture Reference of the 3D picture control where themesh is to be displayed.

3D Scene ref Required. 3D scene Refnum 3D scene refnum where the mesh will bedisplayed.

Mesh data Required. Mesh data cluster.



Up direction LabVIEW™ position cluster(X Y Z).

Direction of the 3D space aligned with the top ofthe camera (or with the vertical direction of the3D picture).

Output parameters 3D pict ref out 3D pict Refnum. Copy of 3D Pict ref in.


Mesh 3D Display A new 3D scene does not need to be created before each Mesh display. The same 3D scene can be repetitivelyused and disposed only when completely finished with the display. This VI does not assume any units for the 3D picture. Units between the mesh data and the camera position must beconsistent thought. Furthermore, the 3D picture works better with units that give data values near one. Therefore, itis generally better to display 3D data with units in m rather than mm for objects measured by the Kinect.


209 of 301

Read Mesh

Opens a Mesh file according to one of the proposed format. File incompatible only works for PLY format. Recalculatenormals? (only for STL) recalculate normals based on vertex values instead of using provided normals (if any). Input parameters File format Required. Enum.

PLY Binary. PLY file format. This VI will correctly reads eithera PLY Binary or a PLY ASCII file independentlyif the PLY Binary or PLY ASCII is selected. Formore information about the PLY file format seereference [7].

PLY ASCII PLY file format.

STL Binary More information about the STL format can beobtained from reference [10].

STL ASCII




Recalculate normals Boolean For a STL file, calculate the normals using thevertex values instead of the normals provided,if any. Especially useful when no normals areprovided in the STL file.

Output parameters Mesh data out Mesh data cluster. Data from the file.




210 of 301

Read Mesh File This VI is not guaranteed to work with Mesh data saved with other software or VI's not part of the Haro3D™ library,even files saved in the recognized formats. Currently, the Read Mesh File VI does not distinguish between a PLY mesh file and a PLY cloud file. If a clouddata set saved in the PLY file format (using the Haro3D™ Cloud Save VI) is read using the Haro3D™ Mesh ReadVI, the points and colors information will be loaded in the Mesh data out cluster and the file will be indicated asincompatible. However, the Cloud data provided by the Kinect are in mm while the mesh data provided by the Kinectare in m. The user should therefore account for the difference in scale for proper display. The STL format is provided only for compatibility with other software. HaroTek recommends to use the Binary PLYformat for both Cloud and Mesh data. The Mesh_Read.vi loads ASCII files using the point as a decimal separator, independently from the LabVIEW andcomputer settings.


211 of 301

Save Mesh

Saves Mesh data into a file format defined by File Format. Data author is a string that can be used to record theauthor of the data when the file format allows it. Attributes is an array of U16 used only by the Format STL binary.The byte order is used only by binary formats. Input parameters Data author String. String added as a


The string should not contain carriage return orlinefeed characters. This string is not used forthe XYZ ASCII format.

File format Required. Enum. Format ofthe file where the Mesh dataare written.

PLY Binary. For more information about the PLY file formatsee reference [7].

PLY ASCII For more information about the PLY file formatsee reference [7].

STL Binary. For more information about the STL file formatsee reference [10]. Color information is not savewith the STL format.

STL ASCII For more information about the STL file formatsee reference [10]. Color information is not savewith the STL format.


Mesh data Required. Mesh data cluster.Data to be save in the file.



byte order (little endian) Enum. Byte order of the datawritten in binary format.




212 of 301




Save Mesh File HaroTek recommends to save the data in the Binary PLY format. Use the other formats for compatibility with othersoftware or VI's from other library. The Mesh_Save.vi saves ASCII files using the point as a decimal separator, independently from the LabVIEW andcomputer settings.


213 of 301

Controls Descriptions of the clusters used by the Mesh Data Functions.


214 of 301

Mesh data

Vertices 1D array of singles. 3 values

per vertices (X, Y, Z).

Normals 1D array of singles. Vectorsdefining the normals of thefaces.

There is 1 normal per vertex. The 3 values pernormal are in consecutive orders.

Indices 1D array of U32. Indices of the Vertices defining each face.

Colors 1D array of U32. Each valuecorresponds to the color of apoint.

Color is in the RGBA format standard toLabVIEW.


215 of 301

Hand states When using the Kinect as a user interface, hands are extremely important. It is therefore essential to track handpositions and states as function of time to be able to interpret hand movements, not only static hand positions orstates. The Haro3D introduces a simple Functional Global Variable (Hand Gesture Functional Global) that can be used totrack the states and positions as a function of time of the hands of the 6 bodies tracked by the Kinect. The Oculus Example VI shows how the Hand States can be used.


216 of 301

Hand Gesture Functional Global

Functional global giving dynamic information about body hand states and positions. Hand data is averaged over 10 measurements for each time period. There are 6 time periods that give informationabout the dynamic of the hands (closing or opening or moving direction). For each time period, the position is averaged but a numerical value between 0 and 1 is given to each hand state(open, closed, lasso) representing the fraction of time within that period that the hand was in that state. The hand positions are calculated relative to an anchor, to compensate for the height and position of the person. Theposition of the associated shoulder is used as the anchor. Input parameters Command Required. Enum. Selects the


Get Returns the current values of the FunctionalGlobal Variable in Hands States out

Set Assign the values of Hand States in to theFunctional Global Variable.

Init Initializes the Functional Global for hand statesaveraging and calculations. This function shouldbe called first.

Calculate Calculates new dynamic and average valuesbased on the data currently in the FunctionalGlobal Variable. Typically, this function shouldbe called after the add command.

Add Adds the values of body data in the buffer ofhand data in the Functional Global Variable.

Hand states in 2D array of Hand state info

Body data Array of body info



Output parameters


217 of 301

Hand states out I32. Returns a value indicative of the status of therequested operation at the DLL level. Value of 0indicates success of the operation. See Status errorcodes for other values


Additional information Hand State info is a cluster of values representing the states and positions of the hand for during a certain period oftime. The Hand Gesture Functional Global contains Hand State info for each of the 6 bodies from the Kinect (trackedor not tracked). For each of the 6 bodies, an array of 6 Hand State info clusters is available. Each of those HandState info represents the state and position of the Hands for a period of time. By default, the functional global keepa buffer of 60 Hand state info for each body and average for 6 different time period meaning that each time period isan average of 10 values. The actual duration of the time period depends on the rate that the body data is added tothe functional global using the Add command. The number of time periods and the length of the buffer are constantvalues in the functional global variable that can be modified if desired. The Hand State info arrays can be used for each body to track movements of the hand or state changing (likeclosing and opening) by comparing the state values for different indices of the array for one body. An example ofanalyzing the hand states is given in the Oculus Example VI.


218 of 301

Anchor accumulate

Accumulates the position of the Anchor of the PHIZ (physical human interaction zone). The anchor value isaccumulated and the counter is increased. If the current anchor is more than a certain value away from theaccumulated anchor, the anchor accumulation is reset. Input parameters Counter in Required. I32. Number of

averages currently in.

Anchor accum in Required. Position cluster.Average of the anchorposition.

Returns the current values of the outputparameters without executing an operation.

Anchor current Required. Position cluster.Current anchor position.

Initializes Oculus Rift for position andorientation.

Threshold Required. Float. Maximumdisplacement to reset theanchor position.



Output parameters Counter out I32. Number of averages.

Anchor accum out Position cluster. Average of the anchor position afterthe current operation.



219 of 301

Hand average calculate

Calculates the actual averages of the hand positions and states. Input parameters Hands Required. Array of Hand

state position state cluster.Buffer of data used to average the states andpositions of hands over time.



Output parameters Hand state Hand state info. Average of the hand states and

positions over one period of time.



220 of 301

Hand click

Example on how to use the hand data cluster created from the Kinect body data to determine if the user clickedusing hand push or hand close. The hand data is an array of clusters containing the data from the Hand GestureFunctional Global and Hand Config is a cluster giving the configuration used to click using the hands: right or lefthand, and push or click. Input parameters Hand data Required. 1D array of Hand

state info.Hand data from a single body obtained fromthe Hand Gesture Functional Global. The datarepresents the states of the hands over thelast 2 seconds by steps of approximately 0.3second.

Hand config Required. Hand Configdata Cluster giving theconfiguration used for theclick.



Output parameters State Enum. State of the click based on the current hand

data. Possible values are: None, Click, Release,Click-Release



221 of 301

controls Cluster typedefs used by the Hand Gesture Functional Global.


222 of 301

Hand state info States of both hands for a given body at one given time. One boolean indicates if the body is tracked and one Hand state average for each hand.


223 of 301

Hand state average Average states and positions of one hand for a given time duration.

Open, Closed and Lasso are float values indicating the proportion of time that the hand has been in that state withinthe time interval corresponding to the specific instance of that cluster. Values are between 0 and 1. The Hand and Tip clusters are Position clusters corresponding to the average position of that part of the hand duringthe corresponding time interval. Each position can be absolute (abs) or relative (rel). Absolute positions are thepositions as provided by the Kinect. Relative positions are the positions relative to an anchor. The anchor positionsare averaged and are typically the shoulder of the corresponding hand.


224 of 301

Hand state position state Cluster typedef used internally by the Hand Gesture Functional Global to calculate the state and position averages.


225 of 301


226 of 301


227 of 301

Hand config Cluster typedef used by the Hand Click VI to determine what action of the user is currently selected as a click.

Hand: enum typedef. Values: None, Right, Left Click: enum typedef. Values: None, Push, Close.


228 of 301

sub VIs Sub VI's are used by the high-level VI provided in the Haro3D library. Users are not expected to use directly thoseVI's.


229 of 301

Body Draw limb

Draw a joint at the Joint draw position and attached to the Attached Joints using the RBG Color and of a radiusRadius (m) in the 3D scene SceneObject in. Input parameters Joint Radius Double. Radius of the joint represented as a sphere.

SceneObject in Required. 3D scene refnum. 3D scene where the joint and limbs will bedrawn.

Joint draw Joint info cluster. Joint of the body that is currently been drawn.

Attached joints Required. 1D array of Jointinfo clusters.

Joints that are attached to the current joint by alimb (or bone).



Color Standard LabVIEW™ Colorcluster.

Color used to draw the joint and the attachedlimbs (or bones).

Output parameters SceneObject out 3D scene refnum. Copy of SceneObject in.



230 of 301

Cloud Read PLY

Reads a cloud of points for a file in PLY format, either ASCII or binary. This VI works only on clouds of point and isguaranteed to work only with files saved using the Haro3D library. Input parameters File path Required. Path variable.

Contains the file path andname.



Output parameters Cloud out Cloud data cluster. Data from the file.




231 of 301

Cloud Save PLY

Saves a cloud of (X,Y,Z) points (points array, X,Y,Z single values are consecutive) into a binary PLY file. Each pointhas a color in RGB format (RGB array in U32 format). Data author is a string saved in the header of the file as acomment and indicating the author of the file. Filename must be provided. Input parameters Data author String. String added as a


The string should not contain carriage return orlinefeed characters.










Binary (T) Boolean. Indicates if the fileis saved in Binary (True) orASCII (False).



232 of 301

Cloud Save XYZ

Saves a cloud of (X,Y,Z) points (points array, X,Y,Z single values are consecutive) into a binary or ASCII PLY file,depending of the Binary? boolean value. Each point has a color in RGB format (RGB array in U32 format). Dataauthor is a string saved in the header of the file as a comment and indicating the author of the file. Filename must beprovided. Input parameters File path Required. Path variable.







233 of 301

Find End of Line

Look for the presence of a term in a string and gives the following 1 or 2 character expression as the End-of-Linecharacter. If the term is found, Term present? is True. Input parameters String Required. String. String to be searched.

Term Required. String. Term to be searched within String.

Output parameters End of Line Single or dual character string found right after Term

Term present? Boolean. Indicates if the Term was found in String


234 of 301

Mesh Read PLY

Reads a PLY mesh file, either Binary or ASCII, previously created by the Haro3D library. This VI can only readstriangular meshes, and is guaranteed to work only with PLY mesh files created with the Haro3D function. For more information about the PLY file format see reference [7]. Input parameters File path Required. Path variable.




Output parameters Mesh data out Mesh data cluster. Data from the file.




235 of 301

Mesh Save PLY

Saves a previously calculated set of data corresponding to a triangular mesh into a binary or ASCII PLY format.Vertices is an array of singles containing the vertices of each triangle in X, Y, Z coordinates. Colors is the color inARGB format (U32) of each vertex. Indices is a U32 array giving the order of the vertices defining the triangles. Afilename and path must be provided. Input parameters Data author String. String added as a


The string should not contain carriage return orlinefeed characters.













236 of 301

Mesh Save STL

Saves a triangular mesh into a STL file in binary or ASCII format. The input data are in the Haro3D Mesh clusterformat. Vertices array contains the 3 vertices of the triangle as X, Y, Z values in consecutive order. If the size ofVertex array is not divisible by 9 (3 x (X,Y,Z) to form a triangle), the boolean "data incompatible" is set to True and nodata is saved. Length of Normal array should be 1/3 the length of Vertices array. Normal array is a vector containingthe normal vectors in a 1D array where Vnx, Vny, Vnz are one after the other for each surface. If Normal array is notprovided or its length is different from 1/3 the length of the vertices array, uses [0,0,0] for all normals. Attribute is anarray of U16 that is used by some software for a U16 information for each triangle (like color for example). The useof Attribute is not standard and only for binary. If no attribute array is provided, uses 0. File path must be provided.The header is optional information of less than 80 characters that cannot start with the word "solid". Input parameters Header String. String added as a

header. Only used by STLBinary

The string should not start with the word "solid".Should be less than 80 characters.



Attribute 1D array of U16. One U16 value is attributed to each triangle thatis used by some application.









237 of 301


Output parameters data incompatible Boolean. If True, the Mesh data does not contain

the correct number of vertices (multiple of 9 for thelength of the vertices array).



238 of 301

PLY file Ascii to vertices

Converts the File data in coming from a PLY Ascii file to numerical arrays of Vertices out and Colors out for Numberof vertices where each vertex information (X,Y,Z,R,G,B) is separated by Line separator. Each value is separated by aspace. If the end of the file data is reached before Number of vertices is reached, an error is generated but read dataare available. Input parameters Number of vertices Required. U32. Number of

vertices and colors to be readfrom the File data in.

File data in Required. String. Containsthe data to be converted intoVertices and colors.

Line separator Required. One or twocharacters indicating the endof a line.

Typically, linefeed or carriage return characters,or both.



Output parameters File data out String. Remaining of the characters after the number

of vertices were read.

Vertices out 1D array of singles. X, Y, Z values of each vertex inconsecutive order.

Colors out 1D array of U32. 1 color per vertex, in theLabVIEW™ RGBA format.



239 of 301

PLY file Binary to vertices

Reads data coming from a PLY Binary file to numerical arrays of Points out and Colors out for Number of verticeswhere each vertex information is separated by the character Line separator. Input parameters File ref in Required. File refnum. Data

in binary format are read fromthe file and converted intoVertices and colors.

File must be opened prior to calling this VI.

Number of vertices Required. U32. Number ofvertices and colors to be readfrom the File.








Output parameters File ref out File refnum. Copy of the file refnum. File refnum

needs to be closed after this VI

Vertices out 1D array of singles. X, Y, Z values of each vertex inconsecutive order.

Colors out 1D array of U32. 1 color per vertex, in theLabVIEW™ RGBA format.



240 of 301

PLY file Vertices to Ascii

For each vertex of the points array (vertices) writes a line of the 3 coordinates corresponding to the vertex and the 3corresponding colors in a format compatible with a PLY ASCII (either Mesh or Point Cloud). Input parameters File ref in Required. File refnum. Data



Vertices Required. 1D array ofsingles. X, Y, Z values ofeach vertex in consecutiveorder.

Colors 1D array of U32. 1 color pervertex, in the LabVIEW™RGBA format.







241 of 301

PLY file Vertices to binary

For each vertex of the points array (vertices) writes a line of the 3 coordinates corresponding to the vertex and the 3corresponding colors in a format compatible with a PLY binary (either Mesh or Point Cloud). Input parameters File ref in Required. File refnum. Data



Vertices Required. 1D array ofsingles. X, Y, Z values ofeach vertex in consecutiveorder.

Colors 1D array of U32. 1 color pervertex, in the LabVIEW™RGBA format.











242 of 301

Triangles - Calculate Normals

Calculates the normals of the triangles provided in the Vertices array. The order of the vertices forming the trianglesis provided by the Indices array. Input parameters Vertices Required. 1D array of

singles.X, Y, Z values of each vertex in consecutiveorder.

Indices Required. 1D array of U32. Gives the indices in the Vertices array thatcorrespond to the triangles

Output parameters Normals 1D array of singles. Normals of the triangles.


243 of 301

Oculus Rift DK2 API Health and Warning concerning the Oculus Rift DK2

• Read and follow all warnings and instructions included with the Oculus Rift Headset before use.• Headset should be calibrated for each user.• Not for use by children under 13.• Stop use if you experience any discomfort or health reactions.• More: www.oculus.com/warnings.• Make sure that there is enough room around the user to move safely to operate the application.

DLLs The Oculus Rift DK2 API consist in a series of VI's that are wrappers for two DLL's, one DLL for the 32-bit versionand another one for the 64-bit version. The two DLLs are:

• Haro3D_oculus_dll_x32.dll• Haro3D_oculus_dll_x64.dll

Those two DLLs should be located in separate folders in the same directory than the Oculus Rift DK2 API's. Palettes If the Haro3D™ library was installed using the provided instructions, the Oculus Rift DK2 API VI's should beaccessible through the LabVIEW™ function palettes: Addons -> HaroTek -> Haro3D -> Oculus Rift DK2 (as shownbelow).

The Haro3D™ API's give access to the following features of the Oculus Rift DK2:

http://www.oculus.com/warnings


244 of 301

Position and Orientation

Functions related to the acquisition of the position and orientation from the tracker of the Oculus Rift DK2. Input parameters Command Required. Enum. Selects the



Init Initializes Oculus Rift for position andorientation.

Acquire Requests current position and orientation.

Close Closes the Oculus Rift for position andorientation.





Position 1D array of singles (x,y,z) representing the positionof the Oculus Rift relative to tracker.

Quarternion 1D array of singles (4 elements: x, y, z, w)representing a quaternion giving the Oculus Riftorientation. More information about quaternions canbe found in reference [6].



245 of 301

Distortion

Provide a LabVIEW references to meshes of distortion correction for the right and left eyes of the Oculus Rift DK2. The Position and Orientation API must be called with the Init command prior to calling this API with the Acquirecommand Input parameters Command Required. Enum. Selects the








Right Mesh Reference to LabVIEW mesh representingcorrection of lens distortion of right eye of OculusRift.

Left Mesh Reference to LabVIEW mesh representingcorrection of lens distortion of left eye of Oculus Rift.



246 of 301

Examples When the Haro3D library is installed using VIPM, the examples are installed in the Examples folder of the LabVIEWversion where the library was installed. The examples can be accessed by directly loading the VI's from the directorywhere they were installed. The examples can also be found with the NI Example Finder (under LabVIEW menu Help->Find Examples...). In theBrowse tab select the "Directory Structure" radio button and scroll down to "HaroTek\Haro3D". The examples canalso be found by selecting the Task radio button and going to Fundamentals/3D components. Finally, in the Searchtab, use either of the terms "Haro3D", "HaroTek", "Hololens", "Kinect", "OBJ", "Oculus", or "Rift", depending on yourinterest.

When running the examples provided with the Haro3D library that are linked to a hardware component, thishardware component must be present and properly configured (see Installation). In the case of the Kinect V2. If no Kinect is present, or if the Kinect V2 has not been properly configured (seeinstallation and requirements), the following dialog window will appear upon running one of the example:


247 of 301

Occasionally, this dialog window will appear even if the Kinect V2 has been properly configured. This problem occursmore often when starting an application right after stopping an application that used the Kinect.


248 of 301

Hololens Two example VIs related to the Hololens are included with the Haro3D library. One example demonstrates the use of LabVIEW user events to receive information generated by the Hololens user. The other example shows how to create, upload, and modify 3D objects in both LabVIEW 3D picture control and theHololens space. This example also shows how to handle a mapped surface (mesh) generated by the Hololens.


249 of 301

Hololens Events Example

Stop button

Stops the application and disconnect from the Hololens.

Hololens connection

Shows the status of the connection with the Hololens.


250 of 301

Info dialog and voice output

Text displayed by the 1 or 2 button dialog or said by the Voice output when clicking the correspondingbutton.

Name of last Hololens event

Displays the last event received from the Hololens.

Response text

Displays the last response text received from the Hololens.

Multi-line display

Array of line items that are displayed to the Hololens users when clicking the Multi Line Display button.

Object identification

Displays the index of the object on which the cursor was when the Hololens user last air-tapped (exceptwhen selecting buttons). An index larger than 0 indicates that the object is an object uploaded fromLabVIEW and the index corresponds to the actual index of the object in the Hololens space. An index of 0indicates that the cursor was not on any object when the user air-tapped. An index of -1 indicates that thecursor was on a mapped surface. An index of -2 indicates that the cursor was on any other type of object (UIobjects for example).


251 of 301

Cursor position andnormal

Cursor position and normal received with Mesh (mapped surface), Hold, and Object Selected events.

Manipulation data

Cumulative displacement in (X, Y, Z) directions of the hand since the beginning the of the currentmanipulation session.


252 of 301

Hololens 3D Objects Example

This example is stopped by clicking the close window "X" box.

LabVIEW 3D Picture control


253 of 301

LabVIEW 3D Picture control showing the 3D objects loaded in the Hololens space. Notice that the paneldoes not show the front panel image and is positioned at the set position. The corresponding view in theHololens space (using the emulator) is shown at the bottom of this page.

Connection status

Indicator to show if the Hololens is connected with the current LabVIEW VI. The controls to add or modify 3Dobjects are disabled until the Hololens is connected.

Pointer to object

Shows a triangle in the field of view of the user in the Hololens space pointing towards the object selected inthe tab.

Object selector

Tab to select which 3D object to Add (create) or Modify. Each page shows properties and button associatedwith the type of object corresponding to that page.

Add / Modify button

Button to Add (create) or Modify the object corresponding to the current tab. After adding an object, thebutton caption changes from "Add" to "Modify".

Object info

Information associated with each of the 3D object. This information can be displayed in the Hololens spaceusing the voice command "Show info".


254 of 301

Common properties

Color and position of the object created or modified in the tab. The mesh in the 3D picture control is notaffected by the color and the position does not affect the panel in the Hololens space.

Orientation

Changes the orientation of the object currently selected in the tab. Notice that these controls will not appearto affect the panel in the Hololens space because the panel is automatically positioned and oriented toremain within the field of view of the user.

Reset 3D Picture view

Positions the camera view of the 3D picture control at (0,0,0) looking towards (0,0,1).

View of Hololens emulator corresponding to the case shown in the example above.Notice that the panel does show the front panel image and is automatically positioned and orientated at 1.5 m fromthe user within his field of view. Also notice the info displayed for the cylinder.

Front panel of the VI used to generate the panel image.


255 of 301


256 of 301

Kinect Examples demonstrating the features and functionalities offered by the Haro3D library for the Kinect.


257 of 301

Audio

Stop

Properly stops the VI but does not close it. The examples should be stopped using this button to properlyclose the application instead of using the "Abort execution" of the tool bar.

Refresh rate

Acquisition rate of the body info. This example has a refresh rate oscillating in the 25-35 Hz range(maximum rate from Kinect is 30 Hz). Actual performance is highly dependent on the hardware (video card,computer processor, and USB controller). Rates higher than 30 Hz are only due to the time measurementerror.

Compass graph


258 of 301

The compass graph shows the direction of the sound as detected by the Kinect. The amplitude of the arrow,between 0 and 1, indicates the level of confidence in the direction. A direction of 90° correspond to the +Zdirection of the Kinect, 0° is -X and 180° is +X when using the coordinate system of the Kinect.


259 of 301

Bodies

Refresh rate


Stop


260 of 301


3D picture

Displays in a 3D environment up to 6 persons with up to 25 joints per person (see body info). Each person isassigned a color. If less then 6 persons are tracked, someone stepping out view of the Kinect and steppingback is likely to change color. Each joint (see joint info) has 4 levels of tracking: Unknown, Not tracked, Inferred, Tracked. In this example,the color of the limbs of joint that were inferred (calculated based on the positions of the other joints andhighest probabilities) are drawn in a darker shade of the same color than the tracked joints (see the feet ofthe yellow and red skeletons in the figure above). Notice that the hands have 3 states: closed, open, and lasso (closed with index and middle fingersextended). The images below show how each state is displayed in the same order:

3D picture view controller

Changes the controller of the view point of the 3D picture. The camera controller changes the type of controlof the 3D picture point of view. Using the controller, one can rotate the view, shift the position (control key)and zoom in and out (shift key). The different controllers are:0 None - LabVIEW does not enable the change of the view point.


261 of 301

1 Spherical - The camera rotates about a fixed point.2 Flying - The camera flies through the scene freely.3 Oriented - The camera rotates about a fixed point respective to the z-axis.


262 of 301

Cloud

Stop


Refresh rate

Acquisition rate of the body info. This example has a refresh rate oscillating in the 20-30 Hz range(maximum rate from Kinect is 30 Hz). Actual performance is highly dependent on the hardware (video card,computer processor, and USB controller).


263 of 301

3D picture

3D view of the cloud of points. View can be rotated or translated in real-time during acquisition.

Cloud size

Selects the size of the cloud of points. The listed dimensions refer to the dimensions of the 2D image sensorof the Kinect. The depth sensor generating the 3D points has a dimensions of 512x424. There is therefore amismatch between the color images and the depth images. With the 480x270 cloud size, the 3D points areunder-samples by approximately a factor 2. With the 960x540 cloud size, the 3D points are approximatelyover-sampled by a factor 2. Using the 1920x1080 Cloud size over-samples the 3D points by a factor 8 andpresents no benefit.

Max Z display


264 of 301

Maximum depth of displayed cloud of points. This control only affects the display of the points. All data aresaved in the file independently of the Max Z value.

Saves and reads cloud files

Saves current data points and reads from a file a cloud previously saved. File format is binary PLY. Moreinformation about cloud file can be found here.


Changes the controller of the view point of the 3D picture. The camera controller changes the type of controlof the 3D picture point of view. Using the controller, one can rotate the view, shift the position (control key)and zoom in and out (shift key). The different controllers are:0 None - LabVIEW does not enable the change of the view point.1 Spherical - The camera rotates about a fixed point.2 Flying - The camera flies through the scene freely.3 Oriented - The camera rotates about a fixed point respective to the z-axis.

Display

Turns processing and display of cloud data ON and OFF.


265 of 301

Cloud - User Event

Stop


Pause button

Pauses the acquisition. Resume by clicking on the button again.

Refresh rate


266 of 301

Acquisition rate of the body info. This example has a refresh rate oscillating in the 20-30 Hz range(maximum rate from Kinect is 30 Hz). Actual performance is highly dependent on the hardware (video card,computer processor, and USB controller).

3D picture

3D view of the cloud of points. View can be rotated or translated in real-time during acquisition.

Cloud size

Selects the size of the cloud of points. The listed dimensions refer to the dimensions of the 2D image sensorof the Kinect. The depth sensor generating the 3D points has a dimensions of 512x424. There is therefore amismatch between the color images and the depth images. With the 480x270 cloud size, the 3D points areunder-samples by approximately a factor 2. With the 960x540 cloud size, the 3D points are approximatelyover-sampled by a factor 2. Using the 1920x1080 Cloud size over-samples the 3D points by a factor 8 andpresents no benefit.


267 of 301

Max Z display

Maximum depth of displayed cloud of points. This control only affects the display of the points. All data aresaved in the file independently of the Max Z value.



Display

Turns processing and display of cloud data ON and OFF.


268 of 301

Color image

Stop


Refresh rate

Acquisition rate of the image. This example has a refresh rate oscillating in the 15-35 Hz range (maximumrate from Kinect is 30 Hz). Actual performance is highly dependent on the hardware (video card, computerprocessor, and USB controller). Rates higher than 30 Hz are only due to the time measurement error.The Color image acquisition might drop to 15 Hz at the Kinect in low-light conditions. In this example,the color image was down sampled to 480x270 pixels at the DLL level from the 1920x1080 image from


269 of 301

the Kinect. With this down-sampling, the display rate of this example can match the acquisition rate ofthe Kinect. However, for higher resolution, the refresh rate drops. In the case of the maximum resolution(1920x1080), the refresh rate is below 10 Hz. This example does not use the Vision development module.It is not known if using the functions of the LabVIEW Vision module would improve the display rate of themaximum resolution color image.

2D picture

In this example, the image resolution is 480x270. At this resolution, the refresh rate can sustain themaximum refresh rate of the Kinect (30 or 15 Hz depending on the light conditions). For higher resolutionimages, the refresh rate significantly drops. Using the Imaq image from the LabVIEW™ Vision module mightbe more efficient than using the 2D picture control.


270 of 301

Infrared

Stop


Refresh rate



271 of 301

Intensity graph

Intensity graph. Infrared intensity data is coded over unsigned 16 bits. Notice that to make the image lookcorrectly oriented, the X and Y axes have been inverted.


272 of 301

Fusion

Stop


Refresh rate

Acquisition rate of the body info. This example has a refresh rate oscillating in the 5-15 Hz range(maximum rate from Kinect is 30 Hz). Actual performance is highly dependent on the hardware (video card,computer processor, and USB controller). In the case of the 3D reconstruction, most of the processing is


273 of 301

accomplished on the graphic card or/and on the CPU. This function is therefore highly dependent on theconfiguration of the host computer.

2D picture

3D picture showing the view of the reconstructed volume from the point of view of the Kinect duringreconstruction.

Tracking indicator

Indicates if the depth tracking is active or not. During 3D reconstruction, if tracking is lost, the trackingmonitor turns OFF. If the depth tracking is not active, no data is acquired for the 3D reconstruction. Whentracking is lost, attempting to put the Kinect (or the object) close to a previous position where tracking wasON might restart the tracking. More information about tracking can be found here.

3D reconstruction parameters


274 of 301

Parameters defining the reconstruction volume and operating parameters. During 3D reconstruction,changing any value of the parameters resets the reconstruction and apply the new values to thereconstruction process. For more information about the 3D reconstruction parameters can be found here.

Required memory

CPU or GPU memory required for the 3D reconstruction. More details can be found here.

Create Mesh

Creates a mesh from the volume data acquired so far during the reconstruction process. After creatingthe mesh, the Reconstruction process is stopped and the 2D picture view is switched to a 3D picture view.Notice that the mesh creation is an operation processed on the CPU and requiring a significant quantity ofmemory. This operation tends to be more often successful on a 64-bit application than on a 32-bit one.

Reset reconstruction

Clears the data acquired so far and restarts 3D reconstruction using the current position of the Kinect as thereference position.

World to camera transform matrix

Translation-rotation matrix of the camera (Kinect) relative to the position of the first data set used by theFusion process to position the Kinect data within the virtual volume.


275 of 301


276 of 301

Mesh

3D picture


277 of 301

3D picture of the mesh. The 3D mesh can be rotated, translated, and can be zoomed in and zoomed outusing the camera controller.

Mesh files

Saves current mesh and reads from a file a mesh previously saved. File format is binary PLY. Moreinformation about mesh files can be found here.




278 of 301

Depth

Stop


Refresh rate



279 of 301

Intensity graph

Intensity graph. Notice that to make the image look correctly oriented, the X and Y axes have been inverted.In the present example, the depth values have been wrapped around 8 bits to improve visualization of localdepth details.


280 of 301

OBJ ExampleThe copyright-free DESDynI satellite and Z2 spacesuit 3D models are graciously provided by the NASA (seereference [17])

3D Picture control


281 of 301

Displays the object loaded from the OBJ file. The object can be rotated by clicking in the display area andmoving the cursor. The object can be translated and the image can be zoomed in and out by holding thecontrol or the shift keys.

Stop button

Stops the VI. The VI must be stopped to change the displayed 3D object.

Select object

Select the OBJ file to be loaded. Choices are Astronaut or Satellite. The selection must be made beforerunning the VI.


282 of 301

Part list

Selects one of the parts of the Main object. The command will be applied to the selected object.

Command

Action applied to the selected part. Normal displays the Main object as loaded. "Make transparent makes"the part transparent. "Remove" removes the part from the display. The "All else" commands applied tocorresponding action to all other parts except the one selected.

Resets view point

Sets the camera view point of the 3D picture control to its default position.


283 of 301

Change part

Sub-VI of the OBJ Example. Apply requested operation to the 3D object indexed by Index from the 3D ref array. Input Parameters Command Required. Enum

Normal All parts are normally displayed

Make Transparent Make the selected part transparent

All else transparent Make all parts transparent except the currentlyselected one.

Remove Completely remove the selected part from the3D object.

Remove all else Keep only the selected part from the 3D object.

3D ref array Required. Array of 3D objectreferences corresponding tothe part of a 3D object.

index Required. i32. Index of thecurrent 3D object in the inputarray





284 of 301

Oculus Rift DK2 Example VI's using the Oculus Rift DK2 API.


285 of 301

Oculus ExampleHealth and Warning concerning the Oculus Rift DK2

• Read and follow all warnings and instructions included with the Oculus Rift Headset before use.• Headset should be calibrated for each user.• Not for use by children under 13.• Stop use if you experience any discomfort or health reactions.• More information: www.oculus.com/warnings.• Make sure that there is enough room around the user to move safely to operate the application.• It is strongly suggested to operate the application while sitting.

Details Example showing how to use the Oculus Rift DK2. This example combines the use of the Oculus Rift, the Kinect V2,the hand states, and of the .OBJ file VI. This example can be used without Rift but the view direction cannot be changed. This example can also be usedwithout Kinect V2 but the orientation and position cannot be changed. The Oculus_example.VI should be maximized on the monitor corresponding to the Oculus Rift (typically monitor 2)using the "Window Run-Time Position" category of the VI properties.

Instructions

http://www.oculus.com/warnings


286 of 301

Install the Oculus Rift DK2 tracker on top of the Kinect, as shown in the image below.

Position the user in front of the Kinect, at a distance of approximately 2 m., as shown in the image below.

Operations Warning: the presence of the headset and the sitting position make it more difficult for the Kinect to recognize the bodypositions. When the arm are not extended, the body might not be recognized. In that case, the body simply disappearsfrom the application. Flickering of the body indicates that the body is occasionally lost. If the flickering occurs, it mightbe difficult to change the altitude as the hand positions are recorded in time to analyze the grabbing motion.

Lateral Movements Close left hand. Keep right hand at shoulder level. Grab sky and lower it to increase altitude. Grab floor and raise itto decrease altitude.

Rotation of view point around astronaut


287 of 301

Both hands in lasso (closed fist, major and index extended). Rotating movement of hands relative to each other rotatesthe view point around the astronaut.

View point reset

Opening both hands brings the view point back to its default position.

Quitting the application The VI is terminated by pressing any key of the keyboard.

The VI must be selected to terminate. It might be necessary to use the “Alt-Tab” key until the window correspondingto the Rift display is selected. Then press any key to terminate the VI.

WARNING

The use of the planet textures on mesh geometries might cause LabVIEW to crash upon closing the VI on somesystems. Use the Disable Structure in the "Init - 3D Pict" case to disable the use of the planet textures if you areconcerned by LabVIEW crashing after closing this VI.


288 of 301

Support Sub-VIs used in the Oculus example


289 of 301

Hand Translation

Looks at the first and last hand info and computes the maximum X, Y, Z translation and translates the cameraposition accordingly. The user can zoom in by grabbing away and moving the right hand towards himself, and zoom out by grabbing closeto himself and moving the right hand away from himself. Input parameters Last hand Required. Latest Hand state

info to consider for translation

First hand Required. First Hand stateinfo to consider for translation

Cam Pos in Required. Position cluster.Camera position before handtranslation.



Output parameters Cam Pos out Position cluster. Camera position after hand

translation.



290 of 301

Command input

Buffer of commands for the Oculus Example VI. Enqueue an array of command to be dequeued by the Command output VI within the Oculus Example VI. Input parameters Instructions Array of Enums. Provides

a set of commands to theOculus Example main loop tobe executed in the providedorder.

flush? Boolean. If true, emptythe queue from previouscommand before enqueuingthe current ones. Defaultvalue is false

to end? Boolean. If true, puts thecurrent commands at the endof the queue instead than atthe beginning. Result is thatthe current commands areexecuted next. Default valueis false.



Output parameters Queue out Queue reference. Reference of the command queue



291 of 301

Command output

Extracts commands from the buffer for the Oculus example. Dequeues the commands that have been enqueued by the Command input VI. Commands are dequeued one at atime and provided as the next operation to be completed within the main loop of the Oculus Example VI. Input parameters Queue in Required. Enum. Selects the



Init Initializes Oculus Rift for position andorientation.


Close Closes the Oculus Rift for position andorientation.

Hand states in 2D array of Hand state info

Body data Array of body info



Output parameters State I32. Returns a value indicative of the status of the


Position 1D array of singles (x,y,z) representing the positionof the Oculus Rift relative to tracker.

Quarternion 1D array of singles (4 elements: x, y, z, w)representing a quaternion giving the Oculus Riftorientation.



292 of 301

Closest hand

Goes through the right hands of all bodies and select the body that has the closest hand (hand with highest Z). Input Parameters Hand states Required. Array of Hand

state info corresponding toeach body



Output parameters Hand data out Hand state info of the closest hand

Data available? Boolean.



293 of 301

Hand Rotation

Calculates the rotation axis and the amplitude of rotation between the early and late position of each hand if they areboth in lasso state. The amplitude of the movement of the hands determines the amplitude of the rotation. Input parameters Hand data Required. Array of Hand

state info corresponding todifferent states as a functionof the past few seconds.



Output parameters Rotation axis Position cluster. Rotation axis at the center of the

hands.

Data available? Boolean.



294 of 301

Planet texture

Opens the JPG file using the path and applies the image to a sphere having a radius of Radius and returns thereference of the resulting 3D object. Input parameters Path Required. Path of the texture

to be applied to the planet.

Radius Required. Double. Radiusof the planet to be created,in the units of the 3D scenewhere it will be shown.



Output parameters SceneObject out Reference of 3D object corresponding to the created

planet.



295 of 301

Distortion correctionDemonstrates how to use the lens distortion correction of the Oculus Rift DK2. If no Oculus Rift is available, this VIloads a distortion correction mesh from a file.

Lens-correction distorted image


296 of 301

Original object


297 of 301


298 of 301

List of Haro3D errors

The custom error code assigned to the Haro3D library are given below along with a context and a description.

Code Context Description

6400 Hololens Hololens not connected

6410 Hololens Invalid index for object in Hololens space

6420 Hololens Incompatible Hololens Event message class conversion. Specific class could notbe converted from Data class

6430 Hololens Input string cannot be empty for the 1-Button and 2-Button dialogs, and for thevoice output

6440 Hololens No input line string can be empty for the multi-line info display

6450 Hololens Server VIs cannot be started. Make sure that the Haro3D library was properlyinstalled. You can also try to restart LabVIEW

6460 Hololens The number of elements in the string array of multi-line info display and of newvoice commands cannot exceed 50

6470 Hololens The name provided for the secondary device is not valid.

6500 OBJ The OBJ API cannot read face shapes other than triangle or quads.

6510 OBJ Texture File could not be found.

6600 3D Operations Input quaternion has an invalid value.


299 of 301

References [1] Kinect features: www.microsoft.com/en-us/kinectforwindows/meetkinect/features.aspx. [2] Kinect V2 dismantling: www.ifixit.com/Teardown/Xbox+One+Kinect+Teardown/19725. [3] Canesta information: en.wikipedia.org/wiki/Canesta. [4] Kinect for Windows 2 SDK documentation: msdn.microsoft.com/en-us/library/dn799271.aspx. [5] G Bradsky and A. Kaebler, Learning OpenCV, Oreilly Media, 2008. [6] Understanding quaternions: www.chrobotics.com/library/understanding-quaternions. [7] PLY file format: paulbourke.net/dataformats/ply/ and en.wikipedia.org/wiki/PLY_(file_format). [8] Meshlab: meshlab.sourceforge.net/. [9] CloudCompare: www.cloudcompare.org/. [10] STL file format: en.wikipedia.org/wiki/STL_(file_format). [11] K. Khoshelham and S. O. Elberink, "Accuracy and Resolution of Kinect Depth Data for Indoor MappingApplications" Sensors, 2012, 12, pp. 1437-1454, 2012. [12] Oculus Rift DK2 info: www.oculus.com/ja/dk2/ [13] Info on Oculus Rift DK2 tracker: "Hacking the Oculus Rift DK2", doc-ok.org/?p=1095 [14] Info on .obj 3D file format: en.wikipedia.org/wiki/Wavefront_.obj_file [15] Info on material specifications (.mtl files) of .obj 3D format: paulbourke.net/dataformats/mtl/ [16] Planet textures: planetpixelemporium.com/planets.html [17] Link to 3D models from NASA: nasa3d.arc.nasa.gov/models [18] Reference on equations to convert rotation angles into a rotation matrix: en.wikipedia.org/wiki/Rotation_formalisms_in_three_dimensions [19] Reference to convert Euler angles into a quaternion: en.wikipedia.org/wiki/Conversion_between_quaternions_and_Euler_angles [20] Hololens web portal: https://developer.microsoft.com/en-us/windows/mixed-reality/using_the_windows_device_portal

http://www.microsoft.com/en-us/kinectforwindows/meetkinect/features.aspx

https://www.ifixit.com/Teardown/Xbox+One+Kinect+Teardown/19725

http://en.wikipedia.org/wiki/Canesta


http://www.chrobotics.com/library/understanding-quaternions

http://paulbourke.net/dataformats/ply/

http://en.wikipedia.org/wiki/PLY_(file_format)

http://meshlab.sourceforge.net/

http://www.cloudcompare.org/

http://en.wikipedia.org/wiki/STL_(file_format)

https://www.oculus.com/ja/dk2/

http://doc-ok.org/?p=1095

https://en.wikipedia.org/wiki/Wavefront_.obj_file

http://paulbourke.net/dataformats/mtl/

http://planetpixelemporium.com/planets.html

http://nasa3d.arc.nasa.gov/models

https://en.wikipedia.org/wiki/Rotation_formalisms_in_three_dimensions

https://en.wikipedia.org/wiki/Rotation_formalisms_in_three_dimensions

https://en.wikipedia.org/wiki/Conversion_between_quaternions_and_Euler_angles

https://en.wikipedia.org/wiki/Conversion_between_quaternions_and_Euler_angles

https://developer.microsoft.com/en-us/windows/mixed-reality/using_the_windows_device_portal

https://developer.microsoft.com/en-us/windows/mixed-reality/using_the_windows_device_portal


300 of 301


301 of 301

Contact information Get access to additional information and examples at National Instruments 3D Vision Virtual User group. See videos related to the Haro3D library at HaroTek's youtube channel. For any feedback on the Haro3D library or on this documentation, or for any question or comment, contact: HaroTek LLCwww.harotek.com email: [email protected]

https://forums.ni.com/t5/3D-Vision/gp-p/5002

https://www.youtube.com/channel/UCmxaabC-nkffasAzzlppyKw

http://www.harotek.com

mailto:[email protected]

3d visualization library for labview™harotek.weebly.com/.../haro3d_manual_v2.1.0.pdf ·...

Documents