Lab 3: Hardware, Open Source, Emerging tech

CS410/510: VR/AR DevelopmentEhsan Aryafar earyafar@pdx.eduSam Shippey sshippey@pdx.edu

1

Conversation● This is a high level overview intended to be a conversation

(and give the students interested in research projects something to think about).

● Feel free to unmute and ask questions, make comments, etc!

● If you’re interested in a technology or have used it, give your thoughts on it.

2

2

Why Not Closed Source?● Annoying: Manufacturer can impose a certain experience,

even if that experience isn’t what we (developers, users) want.

● Disengaging: Community must follow the leader, rather than be directly involved

● Look at Linux: Open source projects are successful when they gather enough interest, both from individual developers and firms contributing to projects

3

33

The elephant in the room● Leaving VR in the hands of large companies has

problems inherent to it that we’ve seen time and time again.

● Facebook○ Account deactivation○ Lock-in for people who aren’t power users

● Valve○ Steam is nice DRM, but it’s still DRM

● Both of these already have huge market share in VR and their respective industries, could become monopolies. Worse, they could split the ecosystem. 4

4

Why Open Source?● FOSS (Free and Open Source Software) drivers mean

that users know what’s running on their computers and how.

● More control in the user’s hands- Convenient for you and I at home, very important if your business depends on working XR hardware.

● Standardization and cross-platform compatibility.● Because we can.

5

55

The Dream: Completely FOSS StackXR doesn’t exist in a vacuum! Other software is required for it to work.Example FOSS stack:

Nouveau (FOSS GPU driver)OpenHMD (FOSS XR device driver)OpenXR/WebXR (FOSS Javascript XR APIs)Open source rendering stack (ex: WebGL, Vulkan)Ideally open source hardware as well!

6

66

A (realistic) goal: OpenHMD● What’s a driver?

○ Software interface to hardware● What’s a VR driver?

○ Take in input in the form position reports or raw data from the device and device nodes, output display information, haptic feedback, sound, etc.

● What’s OpenHMD?○ A unified, open source implementation of a VR driver that works

across many platforms.● How bad can it be?

7

77

Problems● Drivers aren’t normally open source.

○ Few examples.● Hardware/Firmware is also not open source.

○ No idea what the device is actually doing.○ No direct interface to the device.○ Little to no documentation on how these things work at a low

level.○ Device reports itself in odd, unexpected ways.

● Again, not totally decided what VR hardware actually looks like in general.

8

8

More problems● Inside out vs Outside in

○ Some devices are “outside in” trackers (Oculus CV1, HTC Vive) and others (Oculus Rift S, quest) are “inside out”. This has consequences for which device we’re reading from.

● 3dof vs 6dof○ 3dof is usually reported by a gyroscope. 6dof can get ugly.○ Example 6dof device: Oculus CV1 uses an IR constellation for

6dof tracking. That’s hard to tune properly and Facebook isn’t about to tell developers how.

● Tuning○ Fisheye effects have to be tuned correctly based on the lens.

9

9

(Some) Solutions● Reverse engineering

○ If it talks we can listen: Wireshark○ Visualize results and try to find what it’s doing○ It’s a monitor, look what happens when you throw images at it

● Community documentation○ OpenHMD’s documentation is currently not great because it’s

new, but we can work to change that by trying (and not necessarily succeeding) to get it working.

○ Make sure to document your experience if you try it!● Stubborn determination

10

10

Remarks● OpenHMD’s community is still small and many devices

can’t be feature complete in any reasonable time● Most features are hard to do once, with support and

paychecks. They’re much harder to do without either of those things.

● Inside-out tracking for many headsets remains elusive.● Still not a solution for “all-in-one” headsets, like the Oculus

Quest.

11

11

Similar project: Rooting phones● Also known as “Jailbreaking”● Allows users more control over their phones- From which

kernels and drivers they want to use to using separate, non-Google and non-Apple app stores.

● ARM vs x86-64○ The bootloader is often locked on phones, making this very

difficult, even if you own the device.○ Some of these have bugs though, which can let you do it anyway.

● Some all-in-one headsets are actually Android devices

12

12

Rooting Headsets● It is possible to obtain root access on some headsets.● Around a year ago, a kernel exploit was found for the

Quest that could allow this.● About a year ago, a research group claimed to have

rooted the Quest, but this appeared to be faked.

13

13

Disclaimer● I am not a lawyer● Almost all of these violate someone’s ToS● None of this is illegal in the US…● ...but will 100% void warranty● Does this slow down progress?

14

14

Open source headsets● Relativty: A fully open source headset that can be

assembled for around $200.● Completely assembled by the user● Not a commercial product● For the price point, doing 2k 120fps is

impressive.

15

15

Up a level: OpenXR, WebXR● Device-agnostic APIs

○ It’s bad if you have to write things that are device dependent. You sometimes do!

● OpenXR: Khronos group develops an open standard for access to VR devices.

● Bittersweet success: SteamVR is an implementation of OpenVR, but is proprietary.

16

16

OpenXR in the wild: SteamVR● Valve develops SteamVR, a proprietary implementation of

OpenVR● This brings a technically-FOSS standard out into the real

world, but at a cost. It’s not really FOSS anymore.● Unfortunately, Facebook implements a different API for

Oculus, and doesn’t support SteamVR. However, OpenOVR exists as well.

17

17

WebXR● Mozilla’s immersive web project

○ Successes: XRswim, viable immersive web projects in your browser.

○ Mostly demos, but some fairly featureful games○ WebXR emulator

● Raises a lot of questions about how browsers and the web in general could work in VR

18

18

Mobile VR with WebXR● WebXR is a natural addition to the mobile VR stack● Some cross-platform issues, mostly with browsers

○ Tested working on Firefox, Chrome○ Tested not working on Safari, Sleipnir

● Just in case you wanted to make games for VR using nodejs

19

19

BABYLON.JS● Website● Based on WebGL, works with nodejs● Easy to use typescript/javascript API for general rendering● Shockingly good VR support - Especially for mobile● Few dependencies and far less setup than Unity● Eventually, will even have AR support● Simple example adapted from documentation (79LOC):

https://mechanicalgarden.online/xrbabylon.html

20

20

HCI● Human Computer Interaction● VR could be an entirely new way of interacting with

computers● Lots of problems show up in this

○ VR desktops: Good, bad, ugly○ Headaches○ Accessibility for visibility impaired people (including people who

wear glasses!)

21

21

HCI and more about accessibility● The Americans with Disabilities Act on the web

○ Law that requires accessibility measures be taken to accomodate people with disabilities

○ Applies to the web○ Good time for conversation about accessibility overall○ Accessibility largely makes better content anyway

22

22

Headsets● Form factor for HMDs

○ Tethered headsets can be complicated for people who use wheelchairs

○ Some people may find the size and weight of HMDs difficult to manage

○ Wearing glasses can make some headsets unpleasant or impossible to use.

23

23

Examples of solutions● IPD adjustment● The canetroller, for visually impaired users● Somewhat stretches the definition of VR, but DualPanto

(Video by Hasso Plattner Institute) is a purely haptic device for blind users.

24

24

Controllers● Controllers can be hard to use for people with limited

motion in their hands● Many users feel more comfortable on a keyboard/mouse● Typing in VR

25

25

Common problems● Motion sickness and headaches● Applies to practically everyone, still an open problem● Can be acclimated to eventually, but drives early users

away

26

26

Space constraints● Mandatory seated/room-scale distinction● Users with mobility issues could find this frustrating● Users living with other people could find this frustrating

27

27

Price● Price is the ultimate accessibility problem● Expensive headsets could limit adoption for years● None of these are that cheap and supply is limited● Examples:

○ Valve Index: $1000, tethered to PC (Usually >$1000). Sometimes takes months to get there.

○ HTC Vive: New from $850, tethered to PC. “Everything you need” sold for ~$2250

28

28

Somewhat cheaper headsets● Oculus Rift S: $400, tethered to PC● Oculus Quest 2: $300-$400 (All-in-one)● Oculus Quest: $500-$600 now (All-in-one)

29

29

Cheap headsets● Cheap headsets might improve adoption but have a long

way to go● Google daydream: Still requires a phone, only 3dof● Relativty: Requires that you solder things to the PCB

yourself.● Oculus Quest: Facebook lock-in

30

30

Streaming● Problem: All-in-one headsets and phones have limited

specs. They may have trouble delivering the high quality images users demand.

● Solution: Wirelessly deliver content to headsets

31

31

Streaming and the web● “Streaming” is different than WebXR, but they complement

each other ● Render on a high end device, display on a low end device● Examples:

○ Youtube’s 360 degree video○ Eventually, maybe Google Stadia

32

32

Challenges for streaming games● We could render a frame on one end and then send it over

a video stream to a low end device.● Very tight end-to-end, round-trip latency requirements

(<20ms). Stadia is, as of last year, >170ms.● Encoding and decoding videos takes time● Low-res content is unacceptable● Still requires a high end device somewhere in the loop● Device needs to be relatively close (At the network’s

“edge”) for this to work at all33

33

Project Ideas and further Research● What technologies might benefit VR users with

disabilities?● Open source headsets● There are an endless number of webXR projects

34

34