augmented reality with hololens - univerzita karlovapepca/papers/hololens2019.pdf · augmented...
TRANSCRIPT
![Page 1: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/1.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 1 / 22
Augmented realitywith HoloLens
© 2018-2019 Josef Pelikán▲●, Jan Hovora▲
Pocket Virtuality▲, CGG MFF UK Praha●
[email protected], cgg.mff.cuni.cz
![Page 2: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/2.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 2 / 22
Microsoft HoloLens
![Page 3: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/3.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 3 / 22
Augmented vs. Virtual reality
AR is not just VR + CG layer!
AR has extra:– multifocal rendering– understanding of environment topology– inside out tracking, anchors– lighting
AR is a new computation/presentation platform
![Page 4: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/4.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 4 / 22
Microsoft HoloLens (2016-)
Holographic visualization (GPU, stereo)
Wireless: Bluetooth, WiFi
Voice commands, hand gestures
Real-time spatial mapping (to understand the current environment)
Precise 6DOF tracking & sharing (anchors)– 4 environment cameras
![Page 5: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/5.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 5 / 22
How is it made?
Holographic display– beam “waveguide”– 4 layers: RGBG
Spatial mapping, tracking– Inertial Measurement Unit– IR beamers + 4 IR cameras– 1 depth camera– 1 HD IM camera
Computation– HPU (custom)– CPU (Intel Atom 1GHz)– GPU (Intel)
![Page 6: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/6.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 6 / 22
AR light combining system
Patent US 2016/0231568 (based on Nokia research)– very small structures (~wavelength), TIR
![Page 7: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/7.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 7 / 22
Details
Windows 10– limited multitasking– UWP, DirectX 11.1
Memory– 2GB RAM– 64GB Flash
RGB camera– 1280×720, 45° horizontal
Optics– 2 HD 16:9 light engines– Holo resolution: 2.3M total light points– Holo density: >2.5k radiants (light points per radian)
![Page 8: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/8.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 8 / 22
HPU – Holographic Processor Unit
24 Tensilica DSP cores
1GB internal DDR3 DRAM
FPGA (i.e. “custom”)
300 special custom instructions
65M logic gates
8MB static RAM
![Page 9: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/9.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 9 / 22
Spatial mapping
Based on “Time of Flight”– similar to lidar systems
Energy efficient, compact
Allows multiple users to work simultaneously!
![Page 10: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/10.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 10 / 22
Inside-out tracking
Tracking position and orientation of the helmet– passive (observing the environment)
Markerless!Allows multiple usersTransform of coordinate systems“Anchor”– significant part of 3D scene– 3D maps + textures
Limitations:– exteriors– big space– how to track controllers?
x1
y1
z1
x2
y2
z2
![Page 11: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/11.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 11 / 22
Programming
Layers, API
0. hardware
1. driver & Microsoft layer
2. HL API, Win10 UWP (C++)
3. Unity, Vuforia, OpenCV (C#, C++)
![Page 12: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/12.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 12 / 22
Limitations of HoloLens 1 (2▶)
Limited field of view– 30 × 17.5° only ▶
IR spatial mapping– sunlight, limited distance, 3cm detail
No GPS ▶?, no GSM ▶?
– not intended for outdoors
No chance to dim incoming light
Turning around losing orientation?– rear camera?
Intel Atom CPU (performance)– will be replaced ▶ (Snapdragon 850)
![Page 13: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/13.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 13 / 22
Typical applications
Medicine
Engineering
Military
Architecture
B2B applications– … whenever artificial 3D data layers are needed
![Page 14: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/14.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 14 / 22
Fata Morgana
Real-time streaming of reconstructed environment to distant VR operators– comments
– navigation
– communication
– object classification
– measurements, environment analysis
Additional sensor system
![Page 15: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/15.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 15 / 22
Fata Morgana in Temelín
Pocket Virtuality + ŠKODA JS a. s.– service support in a nuclear power-plant– reduce the dose of radiation– better assistance
![Page 16: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/16.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 16 / 22
HoloObserver
Shows artificial 3D model on a real object
Server-client– multiple user support– guide mode (Visitor)
Object recognition and automatic alignment
![Page 17: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/17.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 17 / 22
Holotable
Horizontal big touch-screen (table)
Synchronized 3D content
Clients with HoloLenses (1+)
![Page 18: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/18.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 18 / 22
Augmented patient
Physical figurine– tracking (even limbs!)
Augmented graphics in HoloLens
![Page 19: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/19.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 19 / 22
AR in education
Math (geometry)ChemistryBiology…
© Zienta.com (Geometry 101)
© ThinkMobiles
© Gregory Woods
![Page 20: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/20.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 20 / 22
AR in science
Exploration of 3D objects in real space– user is able to select best viewpoints
Visualization– + “steering” (dynamic control)
![Page 21: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/21.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 21 / 22
Mixed Reality API (DirectX 11+)
HolographicFrame– prediction of future headset position and orientation
Stereo rendering– transparent dual-render-target using instancing– custom instancing can be preserved– tuple of View-Projection matrices (small modifications of
vertex/geometry/tesselation shaders)
© 2013 Manuella Chessa et al.
![Page 22: Augmented reality with HoloLens - Univerzita Karlovapepca/papers/HoloLens2019.pdf · augmented reality SLAM hololens fata morgana Created Date: 10/16/2019 8:07:51 PM](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e1c6701e4f8145a5470ce/html5/thumbnails/22.jpg)
HoloLens 2019 © Josef Pelikán, https://cgg.mff.cuni.cz/~pepca 22 / 22
Future will be augmented!
© 2016 Keiichi Matsuda (Hyper-Reality)