building the matrix: your first vr app (svcc 2016)
TRANSCRIPT
1
BUILDING THE MATRIX: YOUR FIRST VR APP@MISSLIVIROSE
2
ABOUT ME
Livi [email protected]
Virtual & Augmented Reality Developer / Evangelist
VR Blog: The Matrix is my Office @MissLiviRose on Twitter
3
BUILDING VIRTUAL REALITY APPLICATIONS
Why Build for Virtual Reality? Understanding the VR Device Ecosystem Breakdown of a VR application Building a VR app: Development Options Design Considerations & Concerns for VR today Q & A
4
WHY BUILD FOR VR?
5
It’s Really, Really Fun.
6
We’re Used to 3D Interactions
7
The Freedom to Redefine Possibilities
Storytelling is significantly more immersive in an environment that surrounds the user
Virtual experiences improve empathy in the viewer
Visualizations of complex data New and improved ways of interacting with the
technical ecosystem that we’ve been building for decades
9
VR DEVICE ECOSYSTEMDESKTOP // MOBILE
PLATFORM GROWTH
2012: Oculus Developer Kit 1
2014: Oculus Developer Kit 2 Cardboard GearVR Innovator Edition 1
2015: GearVR Innovator Edition 2 Gear VR HTC Vive Developer Kit 1 OSVR
2016: Oculus Rift HoloLens Developer Kit HTC Vive Pre HTC Vive PlayStation VR FOVE Developer Kit Gear VR
TODAY’S TECHNOLOGY
Virtual Reality
Fully immersive experience that replaces your physical world
Example: Oculus Rift
Augmented Reality
Overlays digital information about the physical world around you
Example: Google Glass
Mixed Reality
Combines virtual objects and the physical world
Example: Microsoft HoloLens
12
HEAD MOUNTED DISPLAYS
Head Mounted Displays (HMDs) can be desktop-powered devices or mobile-powered devices
Desktop: Separate display device self-containing screen, content provided externally from a separate PC
Mobile: Content provided and displayed by a smart phone inserted into the casing
Standalone: Entirely self-contained headsets
13
DESKTOP VIRTUAL REALITY
Powered from external computersHardware includes separate screensTreated as an external display at the hardware levelSoftware runs on the desktop and is passed to the displayExamples: Oculus Rift, HTC Vive, FOVE
14
MOBILE VIRTUAL REALITY
Headsets are containers that support stereoscopic rendering for mobile phones Applications are typical smartphone apps The display is the phone screen Usually, almost all computational aspects are done on the phone rather than the headset Examples: GearVR, Homido, FreeFly, Cardboard
15
INPUT - CONTROLLERS
Traditional controllers don’t solve the “I can’t see my hands” issues with VR
Navigation is less natural for movement with joysticks due to the sense of being in one place
Limited number of inputs / actions Some apps may rely on specific types of
input controllers and have compatibility problems
16
INPUT - WEARABLES
Devices that are lower profile and can be worn to control various inputs without specific controller input
Example: Nod Ring Example: Myo Bracelet Example: Emotiv Headset
17
HAPTICS
Components to a VR system that provide tactile feedback based on what is going on in the virtual environment
Add to realism of a VR system by allowing the user to physically touch and experience things going on in virtual reality
Example: GloveOne (pictured) has multiple feedback points that mimic the sense of touch based on collisions in a VR game
18
MORE OPTIONS
Gestures via cameras: example: Leap Motion Controller Omnidirectional treadmills Voice commands
19
OVERVIEW OF A VR APPLICATION
20
DESIGNING & PLANNING
Consider the platforms that you’re interested in targeting Understand the input and design considerations for the given platform as you plan out your
experience Outline key scenes and emotions that you’re looking to capture in your app Choose a development option that aligns with your platform of choice
21
Render Loop
Hardware / Input
Update
Game Logic
Lifecycle of a VR App
22
Game Logic• Defines the behavior of your application
components• Models and textures are defined for your
scene• Scripts control behavior of the scene
objects• Generally doesn’t rely on the status of the
hardware, but will likely contain checks for changes
Render Loop
Hardware / Input
Update
Game Logic
23
Hardware / Input• Check on the orientation of the device to
calculate how the camera should render objects in the scene
• Evaluate if there are actions triggering different behaviors in your scene
• Update scene variables or components based on input checks
Render Loop
Hardware / Input
Update
Game Logic
24
Render Loop• Evaluate the scene appearance based on
the orientation of the headset• Apply effects based on changing scene
(shadows, lighting, etc.) dynamically to match the user’s expectation
• Draw to the headset / display• Example: If player has moved under a tree,
update the shadows cast by the player accordinglyPlayback Required Refresh
RateFilm / TV 24 – 48 FPS
Console Gaming 30 FPS
PC 50-60 Hz
VR 90-120 Hz
Render Loop
Hardware / Input
Update
Game Logic
25
DESIGNING FOR VR
26
INTERACTION & DISCOVERY
There are no standards yet in VR for expected behaviors and interfaces Capturing and guiding a user through a process is a new challenge for the medium Agency of a player is critical The amount of control or interaction a user can have in an environment will vary
dependent on experience and platform
27
PRESENCE IN VR
Consider how your user will want to interact with the environment Environment scale will impact how realistic a scene feels to a player Non-autonomous character actions will break the sense of self
28
Eyes: We’re Moving!
Body: No, we aren’t!
Body: We’re moving!
Eyes: No, we aren’t!
29
DISPLAY PERFORMANCE & CONSIDERATIONS
Low Framerates = Motion Sickness “Tearing” around the edges of the display Shadowing (duplication) of scene objects at high framerates Motion blur
30
TOOLS OF THE TRADEUNITY, UNREAL ENGINE, WEBVR (OH MY!)
31
DEVELOPMENT OPTIONS
Native development with Platform SDKs Game Engines:
Unity Unreal CryEngine
Web: WebGL VR support through WebVR Three.JS A-Frame Vizor
32
EXAMPLE NATIVE TOOLS: OCULUS SDK
Add stereoscopic, distortion, and update effects within the graphics pipeline of your application
Render to a texture specifically output to the Oculus device as well as to a non-stereoscopic screen view
Great for custom game engines adding in VR support with the Rift
More customization and control granted Versions available for desktop (Rift) and
mobile (GearVR) Source: developer.oculus.com
33
UNITY
Game Engine from Unity Technologies Visual editor for environment building & scripting Languages: C#, UnityScript (a JS derivative), Boo
34
35
UNITY: VR MODE
Added in 5.1 (current is 5.4) Requires runtime for headsets installed Supports a number of devices built-in, many others have packaged support add-ins
Implementing it:1. Edit -> Project Settings -> Player2. Under “Other Settings” check the ‘Virtual Reality Supported’ box3. Platform dependent on device type
36
UNREAL ENGINE
Game Engine from Epic Games Visual editor for environment building &
scripting Language: C++
37
38
BENEFITS OF GAME ENGINES
Handles many of the hardware-level interactions with devices Templates for different app types are available Implements physics and lighting engines Robust communities for asset creation Additional services for common application needs (networking, multiplayer, awards) Central tooling for cross-platform development
39
DOWNSIDES OF GAME ENGINES
Less granular control over application behaviors Reliant on implementation of SDKs and devices Black box code for engine functionality
40
WEB VR
Web applications with VR support Front-end Rendering:
WebGL Three.JS, Babylon.JS
Desktop & mobile support with one codebase
www.webvr.info
41
BENEFITS OF WEBVR
No installation of application is needed – run in the browser on any device Automatically support mobile and desktop devices Utilize the current set of tools and libraries for JavaScript Easily switch between VR and non-VR mode Offload interfacing with hardware to the browser
42
DOWNSIDES OF WEBVR
Performance hits for non-native graphics Non-standard support for some browsers Less insight and control into bugs with hardware Relies on browser detection of devices
43
PLANNING YOUR EXPERIENCE
44
THINK ABOUT THE FOLLOWING CONSIDERATIONS:
3D Development Scripting Animation & 3D Modeling Audio Testing
45
PLANNING A VR APPLICATION
Evaluate how the medium lends itself to your game or application Minimize the amount of locomotion Choose one platform to target at a time
Read up on best practices, but don't lock things in Prepare to iterate - a lot!
46
IMPLEMENTATION OF A VR APP - MY STRATEGY
Build out the basics in your environment so that you have a rough understanding of what your application components will do
Script the behaviors for gameplay elements & the rest of the application components Finalize the environment with lighting, textures, effects
47
TESTING YOUR VR APP
One person QA is not quite enough! Variations in height, gender, life experiences, comfort with technology, etc. will all impact
your user to a much greater extent in VR Performance is hugely important to prevent motion sickness
48
SAMPLE PROJECTSSOURCE CODE & TUTORIALS AVAILABLE!
49
SPACE PHOTOS IN SPACE
Space Photography Viewer
Built using WebVR
https://github.com/misslivirose/webvrspace
50
BUBBLE BLASTERA mini-game for the HTC Vive
Built with Unity and the SteamVR Toolkit
https://github.com/misslivirose/vive_bubbleblaster
51
STAR WARS VR TEXT EFFECT
https://github.com/misslivirose/SWScrollText
Sample Experience built with World Canvas for VR Text
Made in Unity
52
VISUAL DATA
https://github.com/misslivirose/visualdata
WebVR App for Visualizing Excel Charts
ASP.NET with Three.JS WebVR Site
53
FINAL CONSIDERATIONS
54
TODAY'S CHALLENGES
Input devices are non-standardized across platforms Constant evolution in tools Market Size
55
RESOURCES
Blog: http://livierickson.com/blog for short-form VR development tutorials, reviews, and news
GitHub: http://github.com/misslivirose for all the code in my VR experiments (including Kitten VR!)
ARVR Academy Introductory course materials: http://github.com/arvracademy/intro-academy/
Just A/VR Show: http://justavrshow.com/ Learning Unity: http://livi.link/learnunity
Andrew MollTechnical Evangelist
Doris ChenTechnical Evangelist
John AliotoPrincipal Technical Evangelist
Tobiah ZarlezTechnical Evangelist
Daniel Egan Technical Evangelist
Livi EricksonTechnical Evangelist
Tim ReillyTechnical Evangelist
Jeremy FosterTechnical Evangelist
Tierney WixtedCommunity Evangelist
Annie BubinskiCommunity Evangelist
Shaina HoustonDirector of Community
EvangelismYesenia Alvarez
Community Evangelist
Joe ShireyDirector of Technical
EvangelismBeth Massi
Program ManagerJennelle Crothers
Technical EvangelistRobin Shahan
Content DeveloperJessica Deen
Technical EvangelistJustin Chizer
Microsoft Student Partner
MEET THE TEAM
Paul HackerVisual Studio and Dev
Tech
Mathias BrandewinderVisual Studio and Dev
Tech
Deborah KurataVisual Studio and Dev
Tech
Jeremy ClarkVisual Studio and Dev
Tech
David McCarterVisual Studio and Dev
TechWard Bel
Windows DevelopmentVishal Saxena
Microsoft AzureCraig Berntson
Visual Studio and Dev Tech
Chander DhallVisual Studio and Dev
Tech
Steve EvansVisual Studio and Dev
Tech
MEET OUR MVP/RDS
• Complete the survey for a chance to win Beats by Dre Headphones
• Microsoft Azure Essentials book giveaway for survey completion while supplies last
Survey aka.ms/SVCC
• Connect with our speakers
Visit our booth
• Fill out a session card to enter a raffle for a Ricoh Theta 360 camera
Submit Your Session Cards
MAKE THE MOST OF YOUR EXPERIENCE
59
Q & A@misslivirose