virtual reality at the crossroads

Virtual Reality at the Crossroads

Henry FuchsUniversity of North Carolina at Chapel Hill

Gratefully acknowledged: support from the CISCO, DARPA, NIH, NSF, NVIDIA, and the BeingThere Int’l Research Centre, a collaboration of NTU Singapore, ETH Zurich, UNC Chapel Hill and Singapore’s Media Development Authority-‐IDMPO;

and suggestions by Christopher Peri (xtv) and Michael Aratow

30 March 2015

Andrei State (UNC) 1994

1

Henry Fuchs UNC Chapel Hill

VR is Suddenly Hot

2

• Only one year ago, VR was slow • C. Mims “Whatever Happened to ... Virtual Reality?” MIT

Technology Review, Oct 2010

• Then, ”Facebook in $2 Billion Deal for startup Virtual Reality Company Oculus” March 25, 2014

• Google and others invest $542M in startup Augmented Reality company Magic Leap (making Magic Leap have $2B valuation) Oct 22, 2014

• “Microsoft Jumps Into Augmented Reality” Jan 21, 2015. NY Times “Eventually, the team working on the project grew to about 1,000 people.”

-‐


VR Was Hot Before — in the 1990s.

• What’s happened? will VR crash again in another few years?

• What’s the difference between now and 1990s? • We all suspect that 1990s technology was inadequate,

but which technology, and what exactly was lacking in 1990s?

• Is technology adequate now, or are some components still insufficient for mass acceptance ?

-‐

3

-‐


Outline of Talk

• Quick tour of the history of VR, • Examine each of the components of a VR

system, what was the state of the art in early 1990s, where is it now, and what are its prospects?

• End with a few current projects, trying to solve remaining problems.

• Conclusions & Discussion-‐4

Virtual Reality Today vs the “Dream of VR”

5


VR & AR Displays: Head-worn, tracked

• Virtual Reality = closed to the surrounding real world • Augmented Reality = augmenting real world with

spatially registered imagery – Optical see-through = real and virtual imagery

combined optically – Video see-through = real and virtual imagery combined

with video image processing • Cameras in front of a closed display • AR on hand-held mobile devices not in this talk

D’nardo Colucci, UNC (1997)

Oculus Rift DK-‐2 (2014) (allgamesbeta.com)Lumus DK-‐32 (2012)

6

-‐


FIRST AR / VR System:

Ivan E. Sutherland, A Head-Mounted Three-Dimensional Display, 1968 Fall Joint Computer Conference

• Implemented all the components of an VR / AR system

– Display device (stereo)

– Image rendering

– Head tracking (two kinds)

– Interaction (camera pistol grip)

– Model generation

7

-‐


Sutherland’s 1968 HMD in Action

8


Impact of Sutherland’s 1968 HMD System

• Nothing ! • Sutherland left Harvard for U Utah (to join David Evans,

start a graphics company) • Image generation part of HMD influenced technology in

graphics start-up, Evans and Sutherland Computer Corp.

• A few grad students used the old HMD system • Sutherland’s HMD barely mentioned in graphics textbooks • A few isolated individuals worked on “VR” technology

through 1970s • Almost impossibly expensive to build a “VR” system • .. Until 1980s

-‐

-‐9

-‐


Emergence of VR: 1980s

• Gradually every component become available “off the shelf”

3D graphics workstations: 1980 Ikonas, 1985 Silicon Graphics

Consumer “pocket” TVs

6 degree-of-freedom tracking system by Polhemus

Wide-angle lens for 35mm stereo slides: LEEP Optics

• A few labs make VR systems: Scott Fisher (NASA Ames) 1985; UNC

• VPL Jaron Lanier: 1987 sells first commercial VR System ~$ 100k ?

• Jaron Lanier: first VR visionary, evangelist, w/ mass media coverage-‐

10

-‐


VR Excitement: 1990s

• Jaron Lanier’s Vision, publicity, front page of Wall St. Journal 1991, TV coverage, magazines, movies

• 1993: Nick Negroponte (MIT Media Lab) in Wired Magazine

“I expect that within the next five years more than one in ten people will wear head-mounted computer displays while traveling in buses, trains, and planes”

• Jaron Lanier 1997-2000 leads “National Teleimmersion Initiative” funded by Advanced Network and Services

“Tele-Immersion (National Tele-immersion Initiative - NTII) will enable users at geographically distributed sites to collaborate in real time in a shared, simulated environment as if they were in the same physical room.

… Yes, but not very well-‐

11


The VR Bust: 2000s

• Direct reason: VR didn’t deliver on its promises conceptually OK; quality inadequate

• Dot-com bust

• 9-11

• War on Terror,..

• VR: What was it going to deliver?

• 2010: Christopher Mims, “Whatever Happened to ... Virtual Reality?” MIT Technology Review, Oct. 22, 2010

• ….

• Let’s look back, was technology the problem?

-‐

-‐

12


Did Inadequate Technology Cause Past Failure of VR?

Components of a VR / AR system

1. Display device

2. Image rendering

3. Head tracking

4. Hand tracking & Interaction

5. Model generation

Assessment of each component

Compare achievement to 1970 dreams & goals of VR

Wide5 (~2009)

13


Display device on ’68 HMD

• Optical see-through display • 1” CRTs • Calligraphic / line drawings

• ~ 40 degree field of view (?)

14


HMD ca. 1987: VPL head-mount

• Closed display, not see-through • LCD from SONY pocket TV • wide angle optics (LEEP)

(no distortion correction)

http://cdn.explainthatstuff.com/

15

http://cdn.explainthatstuff.com


HMD ~1995: video see-through (UNC)

Need for “video see-through” not “optical see-through” to get proper occlusion between real and virtual objects:

-Ultrasound handset & MD’s hand-Patient -Synthetic hole in patient -Ultrasound image

16


Video See-Through HMD for Breast Biopsy

17


Video see-through HMD with “Zero parallax”

• Optical center of the camera matches optical center of the eye(s)

• Visuals of the local environment are from proper point of view (the same as each eye’s)

D’nardo Colucci (UNC)

18


Head-Mounted Displays: VR

• Wide field of view crucial for immersive experience

• Difficult to achieve with miniature displays

• Wide FOV with array of miniature displays: very difficult

• Key enabler: mass market for high resolution displays of proper size: smart phones, more recently mini tablets

• Fakespace Wide5 (2009)

• Wide FOV (155 deg./eye), custom lenses, distortion correction in h/w

• 800x600 or 1920 x 1080 per eye

• $ 18,000? - $ 25,000? very few produced

• Display for both eyes: Oculus Rift (2014)

• Wide FOV (100 deg/per eye?), distortion correction in s/w

• 960 x 1080 per eye

• $ 400 (> 100,000 by July 2014)

• Zuckerberg: “Facebook must sell 50-100 million units to be considered a major platform”

Fakespace, Wide5 (~2009)

Oculus Rift DK-‐2 (2014)

-‐

19

HTC Vive (2015)


Head-Mounted Displays: AR

• Field of view (40 degrees per eye) insufficiently wide for immersion

• Proper occlusion between real (local view) and virtual imagery difficult in optical see-through

• Seeing real world via video (in video see-through) awkward

Lumus DK-32 (2012)$ 6,000? - $ 15,000

-‐

Canon MREAL Mixed Reality headset hitting US March 1st

for $125,000engadget February 21st 2013

20Microsoft HoloLens


Assessment on HM Displays

•The dream at 1968 / 1970: Displays built in to our ordinary eyeglasses

•Grade for displays now: C-

significant progress for VR

not even close to adequate for AR

21


Image Generation: Sutherland’s 1968 System

• Real time • Line drawing (No hidden line elimination) • Heroic work: all built with gate-‐level chips

3D transforms Clipping divider 2D line segments to points

22


Image generation: ~ 1990

• Real-time full-color raster image generation

• Rapid, consistent progress throughout the 1980s and 1990s

1000s of polygons /sec in 1980 Million polygons / sec early 1990s

• UNC Pixel-Planes 5 at siggraph 1991: 2M polygons/sec

23


Image Generation Now

•Sophisticated shading effects and millions of polygons /sec

•Commodity graphics chips & boards

•Rapid, continuing progress

• Image generation processors becoming ubiquitous –migrating even to mobile devices

24


Image Generation Assessment

•Dream of 1968/1970: Realistic image generation

–Sutherland 1965 Ultimate display: can’t distinguish between virtual and real objects

•Grade for image generation: A+

25


Supercomputers w/GPUs Tianhe-‐1A: one of the world’s fastest

supercomputers (China) * 14,336 Intel Xeon CPUs 7,168 Nvidia Tesla M2050 GPUs.

GPUs do most of the computationtomsviewpoint.blogspot.com

Image Generation Beyond “A”: Advancing more fields than just real time rendering

http://en.wikipedia.org/wiki/Supercomputer#l

26

http://en.wikipedia.org/wiki/Tianhe-1A

http://en.wikipedia.org/wiki/Intel

http://en.wikipedia.org/wiki/Nvidia

http://en.wikipedia.org/wiki/Supercomputer




Head tracking: Sutherland’s 1968 system

Two tracking systems implemented

mechanical tracker ultrasound tracking with multiple emitters and receivers

27


Head tracking (mechanical) in Sutherland’s 1968 system

• Vertical pivot in ceiling • Universal joint on top • Universal joint on bottom • Shaft slides in and out • “Sword of Damocles”

Limited working volume, cumbersome

28


Head tracking (ultrasonic) in Sutherland’s 1968 system

• Ultrasonic tracking

3 transmitters on head-mount

4 receivers hanging from ceiling

Measure phase changes

• Problem with ambiguity of number of cycles of u/s signals

• Problem exacerbated by heating system

29


Head Tracking ~1980: Polhemus Magnetic Tracking

• Developed for helmet tracking in cockpit of fighter aircraft

• Limited range -- inadequate for walking across a room

• Severe warping of space with metal, with other magnetic fields. Could be reduced with calibration

30


Large area trackers: UNC‘1991

• 10x12 ft space demo Siggraph 1992

• Multiple optical sensors on head, LEDs in ceiling

Lateral effect photo-diodes, much faster than cameras One IR-LED lighted at a time Approx. 1,000 updates/sec

• Commercialized by 3rdTech

31


Head tracking Now

• Multiple good solutions

Optical, Magnetic, Inertial

Combination of multiple technologies

• Accurate solutions need instrumented area

32


Head tracking assessment

•Dream of 1968 / 1970: Go anywhere, unencumbered

•Grade for tracking now: B-

•OK if can live with restrictions –Degraded performance if lose line -of-sight contact (optical trackers, rely on inertial units) or near ferrous/metal objects (magnetic trackers)

–Can’t go beyond instrumented area; outdoors very difficult

33


Interaction / Hand tracking

in Sutherland’s 1968 HMD system

• Mechanical design with three reels of fishing line mounted from ceiling

• All lines connected to top of (camera) pistol-grip

• Amount of line reeled out determines 3D location of hand-grip

• Problem of interference between head and hand trackers

34


Interaction / Hand tracking ~1980s

• Polhemus magnetic tracker (time-shared with a target for head-tracker)

• VPL DataGlove sometimes added to hand-tracker

• Severe warping of tracked space if user moves more than a few steps

35


Interaction / Hand tracking now

New cheap solutions available: small structured light depth cameras: LeapMotion.com

YOGSCAST Martyn

YOGSCAST Martyn

36

http://LeapMotion.com


Assessment of Hand tracking

• Dream in 1968 / 1970: unencumbered tracking of hand

• Grade for hand tracking:

B • Works if hands are in camera’s field of view • If hands reach around or backwards, then outside tracking volume

37


3D model creation 1980s

• Manual model creation problematic as image generation improves

• OK if 3D model needed as part of the application: CAD/CAM, medical

• Auto scanning for selected objects: Cyberware

38


Complete Dynamic Scene Acquisition: Just Beyond State of the Art

• Instrumented space — moving people [Kanade et al, 1997] • Complete 3D static scene

• high quality, with manual editing [UNC,DeltaSphere, ~1999; sold by 3rdTech.com]

• lower quality, with auto scanning with Kinect depth camera [KinectFusion, 2011; Scalable Kinect Fusion, 2013, Microsoft Research]

Nyland, UNC,”DeltaSphere”, ~1999Scalable KinectFusion, MSR, 2013

39

http://3rdTech.com


Assessment of Model Creation

•Dream in 1968 / 1970-- not clear –Sutherland: Building model may be as time-consuming as building the real thing

–Others: There’s got to be an automatic way •Grade for model creation:

B- Object scanning OK, dynamic room scenes with people still beyond state of the art

40


VR with HMD Assessment Summary

•Displays:•VR: coming along B+•AR: promises by Magic Leap, Microsoft; insufficient

information to evaluate: C-•Image generation: unalloyed triumph! A+•Head tracking: only in interior instrumented spaces B- •Scene and model acquisition: only static spaces B-•Hand tracking: coming along: B

41


Some Remaining Problems & Current Efforts at UNC

• Proper occlusion Between Real & Virtual Objects – Optical See-Thru HMD: Using Controlled Lighting (2012, 2013)

• Wide Field of View Optical See-Thru HMD (2014)

• Low Latency Display for Optical See-Thru HMD (2014)

• 3D Scene Acquisition / Telepresence (2014)

-‐

42


Recall: Need Correct Occlusion Between Real & Virtual Objects

Video See-Thru HMD: AR for Surgery (1996)

Ultrasound handset & MD’s hand Ultrasound image Synthetic hole in patient Patient

State, A, M Livingston, W Garrett, G Hirota, M Whitton, E Pisano, and H Fuchs, Technologies for Augmented Reality Systems: Realizing Ultrasound-‐Guided Needle Biopsies, SIGGRAPH 1996. 43


Proper Occlusion with Optical See-Thru HMD: (2012, 2013) .. only for a single, monoscopic viewer

from PBS NOVA ScienceNow “What Will the Future Be Like?” Nov. 2012

44


Proper Occlusion Between Real & Virtual Objects in Optical See-Thru HMD: Use Projector-Based Lighting in Room (2013)—still only for a single viewer

Maimone, A, X Yang, N Dierk, A State, M Dou, and H Fuchs, General-‐Purpose Telepresence with Head-‐Worn Optical See-‐Through Displays and Projector-‐Based Lighting, IEEE Virtual Reality 2013. Best Short Paper Award

45


Wide Field of View Augmented Reality Eyeglasses (2014)

• Pros: wide field of view, simple, nice form factor, cheap • Cons: limited resolution due to diffraction, 3D pupil tracking, imperfect

occlusion of real worldMaimone, A., D Lanman, K Rathinavel, K Keller, D Luebke, and H Fuchs. Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses Using Defocused Point Light Sources, SIGGRAPH 2014 and SIGGRAPH Emerging Technologies Booth

46


Low Latency Display for AR HMDs

1. Assume tracking is already done at very high rate ( ~10KHz) 2. Bypass the video interface (DVI, HDMI,..) to the display device & control

the display technology more directly 3. Build the display from technology that can be accessed much faster than

conventional video rates (e.g., Digital Micromirror Display — DMD, TI’s DLP)

4. Structure the rendering pipeline as a cascade of updating modules,each successive module updating the rendered image (with the latest tracking information) faster & simpler

5. Update DMD at fastest possible rate(> 20,000 binary images/sec), each time approximating the latestcalculated “desired” image.

Zheng, F, T Whitted, A Lastra, P Lincoln, A State, A Maimone and H Fuchs. Minimizing Latency for Augmented Reality Displays: Frames Considered Harmful. IEEE ISMAR 2014.

-‐

47


Experimental Results: Video Recording of a Displayed Spinning Test Pattern

60 Hz grayscale projector 22,700 Hz binary projector

(60Hz will shift between distinct positions, each in focus; 22,700 will smoothly change, but binary images)

48


Experimental Results: Video Recording of a AR Imagery Onto Sides of a Spinning 3D Cube

Optical see-thru HMD with AR images by DMD rear-projection

Only a BINARY image is calculated in each frame @ 20,000 frames/sec. GREY-‐SCALE image is perceived

49


Frames from Video Recording of Spinning Cube

Motion path of rotating cube: Displacement in degrees vs. time

Slow motion (less blur) Rapid motion (more blur)

Nothing special done to achieve motion blur

Only a BINARY image is calculated and displayed in each frame @ 20,000 frames/sec. GREY-‐SCALE image is perceived

50


Automatic 3D Scene Acquisition (2014)

Dou, M, and H Fuchs. “Temporally Enhanced 3D Capture of Room-‐sized Dynamic Scene with Commodity Depth Cameras“. IEEE VR2014. Best short paper

With 10 Kinect color + depth cameras; prescan room with single Kinect

51


Concluding Remarks1. Will the current VR excitement be once again followed by disappointment?

2. VR much easier than AR: display, tracking, rendering latency

3. VR has much better chance this time:

1990s VR systems cost $ 100,000 ++

now: Oculus Rift DK-2 $ 400 + a PC

Samsung GearVR mobile HMD $ 199 + Galaxy Note 4

4. Potential for millions of sales: Mass markets make all the difference

Sufficient volume to support an industry of app developers, hardware add-ons, etc.

5. Big uncertainty: For what will millions of people use VR systems

Immersive video games + entertainment, live events, education, training, health .. ?

6. Many technical problems remain unsolved: eyeglass display, tracking, model acquisition,..

7. Will the commitment and patience of the big companies currently investing in VR & AR last long enough to allow

development of the technology needed for mass markets?

… In 2015 we will start to find out !52


Thank You

53

virtual reality at the crossroads

Documents