projects in virtual and augmented...
TRANSCRIPT
z
deti departamento de electrónica telecomunicações e informática universidade de aveiro
Projects in Virtual and
Augmented Reality@DETI
Paulo Dias
RVA 2019-2020
2
Outline
• 3D Reconstruction
• RVA
• VR Frameworks
• RVA@deti
– pSIVE (Platform for Setting up Interactive Virtual
Environments)
– Museu da Cidade de Aveiro
– VR4Rehab (Edição RVA 16-17 – videos)
– Continuous AR
– Bosch
– Produtech
– EdupARk
– …
How to start a presentation – Top 3
• What is Virtual and Augmented Reality
and what do you need to do it?
3
OR
• What are we doing at DETI in Virtual and
Augmented Reality besides teaching the
RVA course?
• 3: Start with a question!
How to start a presentation – Top 3
4
• 2: Shocking Facts
• AR/VR 3.33 Billion USD in 2015
• Expected 133.78 Billion USD by 2021
• Portuguese 217,6 billion USD in 2017
How to start a presentation – Top 3
5
• 3: Tell a story
• How did I come to this area?
• Ola, Eu sou o Paulo Dias!
6
3D reconstruction – Paulo’s PhD 1998 - 2002
7
• Available for small models
Shape Grabber
Minolta Vivid
Marble Statue of Aphrodite scanned with the
VIVID 910 using the rotary stage option
3D reconstruction
8
• For larger models (building, room) one possibility is to use “Laser Range scanners”.
Riegl
Zoller & Frölich
Cyra
SICK adaptado
Departamento de mecânica
3D reconstruction
9
Point cloud
Reflectance
1000 x 175 (Riegl LMS Z210)
3D reconstruction
10
Triangulation
Registration (ICP)
Digital Photos
Texture
IEETA model, 12 range, 12 Photographs
3D Model
3D reconstruction
11
Ieeta
Lab. Mecânica
Assemblée nationale Farmhouse Laveno
Sala dello scrutinio
Adapted Sick (Aveiro) Riegl (Italy)
3D reconstruction
3D Reconstruction – google Tango (discontinued)
12
Ieeta amphitheatre
Gab. 005
PhD - September 2003, 23
13
• Interaction with 3D content ?
14
Why Virtual Reality?
• 3D information is common (3D reconstruction,
games, CAD, Design, etc…)
• Interaction is still mainly 2D – Keyboard and
mouse since 70s’
• Virtual Reality: Attempt to go beyond classical
interaction and immerse user in 3D environment
Apple II – from 1977
15
Reality Augmented Reality Augmented Virtuality Virtual
Mixed Reality
Paul Milgram [Milgram et. al 1994]
Virtual reality Continuum
16
Projecto de 5º ano:
Mário Cruz
Paulo Moreira
2004-2005
Low cost environment for Virtual and Augmented Reality
17
Virtual reality
• Equipment
– Tracker 3DOF (Yaw, Pitch,
Roll)
– HMD SVGA stereoscopic
– Accelerometer– [Electrónica e
Sistemas Médicos / DETI]
pose detection.
• Graphicl libraries
– OpenGL
– VTK (Visualization Toolkit)
– OGRE
18
Augmented Reality
• Equipment
– Virtual reality + Camera
– Video see-through
• AR Toolkit
– Marker detecction
– Superposition of virtual elements
19
Magic book prototype - 2005
Results
20
Challenges in VR/AR
• User interaction with world – Input Devices
• User immersion in world – Output Devices
• =>No unique solution: several attempts
with different solutions
21
Input: Motion trackers
– Magnetic
– Mechanical
– Acoustic
– Optical
– Inertial
– Hybrids (Combination of previous)
Polhemus Fast-Track
Leap
3D Mouse - Logitech
InertiaCube 3 - Intersense
Finger Tracking - ART
Kinect
Phantom
Razor Hydra
22
Output
– Monitors
– Surround screens
– Workbenches
– Hemispherical Displays
– HMD
– Arm Mounted Displays
23
Hardware@deti - Input
– Intersense 3DOF Inertial Tracker [2009 - 1940€]
– Intersense 3DOF Inertial Track. BT [2012 -1437€]
– Wintracker [2009 - 1285€]
– Phantom Omni [2012 – ~3500€]
– Kinect v1.0 (3) [2012 - 300€]
– Kinect v2.0 (1) [2014 - 300€]
– Swissranger [2012 – ~3000€]
– Leap Motion (3) [2016 – 60€]
– Google tango HTC and Lenovo [2015 – ~500€]
– Orbtec Astra [2019 – ~250€]
• Trackers:
– 3DOF wired and wireless
– Wintracker 6DOF
– Google Tango
– Leap Motion
– Kinect 1/2
– WiiMote
– Razor Hydra
– Phantom
24
Virtual Reality - Hardware
25
Hardware@deti - Output
– I-glasses SVGA HMD [2004 - 800€]
– VR 2000 HMD [2012 - 1359€]
– Oculus DK1 [2014 - 420€]
– Oculus DK2 [2015 - 500€]
– Gear VR + Game Pad [2016 - 250€]
– Samsung Galaxy S7 [2016 - 650€]
– HTC VIVE [2016 - 1350€]
– Meta Glasses [2018 – ~1800€]
– Xperia Projector [2018 – ~3000€]
– Myo gesture ArmBand [2018 – ~100€]
• HMD:
– Oculus DK1,2
– HTC VIVE
– Samsung Gear VR
– HMD SVGA (2)• I-glass
• VR 2000
26
Virtual Reality@deti – Hardware
27
Hardware@deti - Input
Hardware session in a few weeks!
28
VR Frameworks Introduction - 2013
• Creating a Virtual Environment
– Cost (Time & Money)
– Specific knowledge
• Several frameworks/tools
– Specific implementation
– Particular architecture
• How to ease?
– Tools for ‘laymen’
– Flexible yet simple
Graphics engines
• OpenSceneGraph– open source C++ cross-platform toolkit for simulators, virtual reality and
games
– several NodeKits that fulfill a variety of purposes, such as particle
systems, text display, special effects, shadow , 3D interactive controls
,…
• OpenSG– real-time graphics programs, e.g. for virtual reality applications similar to
OpenSceneGraph
• VTK– Open Source library based in Object Oriented Programming
– multi-platform (Windows/Unix/Mac)
– C++ implementation with several wrappers for other languages (Tcl/Tk-
Python-Java)
• Many more …
29
Real-time physics engines
• Bullet– Free, open source
– Collision detection
– Soft and rigid body dynamics
• Newton Game Dynamics– Cross platform open source
– Simulates rigid bodies in games and other real-time application
• PhysX – multi-platform game physics
– support several devices (smartphones, GPU, CPU)
– Integrated in several game engines
– Acquired by Nvidia in 2008
– SDK provided to developers for free
• Many more…
30
3D Computer Graphics software
• Blender– free/open-source 3D computer graphics software used for creating
animated films, visual effects, art, 3D printed models, interactive 3D
applications and video games.
– include 3D modelling and several effects: texturing, raster graphics
editing, rigging and skinning, fluid and smoke simulation, particle
simulation, soft body simulation,…
– integrated game engine.
• OpenSpace3D– Free Open Source development platform for interactive real time 3D
projects
– User friendly, possible to create interactive 3D scene without writing any
code.
• 3D Studio Max– 3D Modelling, Animation & Rendering Software
– Autodesk, commercial
31
What else?
• Something missing?
32
Game engines
• Unity– cross-platform game creation system that includes a game engine and
an integrated development environment.
– Graphic working environment that facilitates 3D scene creation and
layouts
– Physics engine that provides collision detection, forces, torque and
physic materials
– Two licensing options Unity and Unity Pro
• Unreal Engine 4– Multiplatform game development toolkit supports state of the art
graphical fidelity
– full access to the source code of the engine
– In 2015 released for free usage with a 5% commission on commercial
products.
– PhysX 3.3 physics engine for physical simulation
33
Virtual Environment Frameworks - not desktop
• CCG Cave setup /Guimarães
34
Virtual Environment Frameworks - not desktop
• inVRs
• VRJuggler
• Vizard
• 3DVIA
• others
35
36
inVRs
– Institute of Graphics and Parallel ProgrammingJohannes Kepler University, Linz, Austria (2006)
– C++
– OpenSG and OpenSceneGraph as a scene graph engine
– Network distributed virtual world
– Pre-defined navigation and interaction techniques
– Configurable via XML
– Mostly used on academic environmentsAustrian/German Universities
37
Source: http://www.invrs.org/app_gallery.php
Safety Training - SAVE
Walkthrough
Entertainment - Space Trash
inVRs Applications
38
VR Juggler
– Iowa Center for Emerging Manufacturing Technology, Iowa State University, Iowa, United States (1998)
– C++
– Supports many graphics Engines – OSG, OpenSG, OpenGL, VTK…
– Huge variety of modules
– Network distribution through NetJuggler
– Extended over time – Ex.: VR JuggLua
– Used for many different purposes – Commercial / Academic
• Mina do Lousal (PT)
• Korea Ocean Research and Development Institute
– Configurable via VRJConf – Backend XML
39
Source:
Korea Ocean Research and Development Institute
Source: http://vrjuggler.org/, Last
VR Juggler
40
Vizard
Vizard Development Edition
Vizard Development Academic - Single License $ 3,990
Vizard Development Academic - SITE License 5 Seats $ 6,990
Vizard Enterprise Edition
Vizard Enterprise Academic - Single License $ 5,990
Vizard Enterprise Academic - SITE License 5 Seats $ 9,990
Vizard Plug-Ins
Vizard ARToolworks Add-on Development Edition $ 570
Vizard ARToolworks Add-on Enterprise Edition $ 7,990
• WorldViz, Santa Barbara - CA, United States
• Pyton Scripts + GUI
• OpenSceneGraph - Possible to expand vizard funcionalities
• Support many comercial Devices
• Physics Engine
• Research & Science (VHIL - Stanford U), Architecture & Construction (Archidimex - Netherlands)
41
Vizard
Applications
IDE
• Dassault Systèmes, Vélizy-
Villacoublay, FRANCE
• Former Virtools
• Closed Engine
• Many GUI Tools to work on models,
animations and scenes
• Price around 9000 USD + 3000 USD/yr
Source:
http://www.3ds.com/fileadmin/PRODUCTS/3
DVIA/3DVIAVirtools/demoshowcase/html/bro
wse.html?br=1
3DVIA Studio
42
Peripherals Interfaces
• Virtual Reality Periphery Network (VRPN)– network-transparent interface between applications
and physical devices used in VR
– client-server approach
– layer of abstraction that classifies inputs into several
categories (Analog, Button, Dial, Force Device,
Sound,Text, and Tracker), allowing to receive generic
input from different devices.
43
Other Frameworks
• FreeVR
• Avango NG
• DIVERSE
• ViRAL
• EON Studio
• …
44
Conclusion – VR frameWorks
• Difficult choice, many solutions
• Complex Architecture – Many Abstractions
• Often not trivial to Install / Configure
45
• Game engine emerging as alternative to
expensive frameworks – Are almost
standard in research today
Download and install – Break - EdupARk & Unity
http://edupark.web.ua.pt/
46
https://play.google.com/store/search?q=edupARk
https://unity.com/
z
deti departamento de electrónica telecomunicações e informática universidade de aveiro
Projects in Virtual and
Augmented Reality@DETI
Paulo Dias
RVA 2019-2020
48
Motivations for VAR projects
• Foster R&D in Virtual and Augmented
Reality areas
• Investigate new interaction and navigation
techniques
• Devise and demonstrate new scenarios for
the use of these technologies
49
Environment to configure RVA environment
Mestrados:
Danilo Souza
João Parracho
João Cardoso
João Pinto
Luis Afonso
Sérgio Eliseu
2013-2016
Interaction in Virtual reality
pSIVE – Danilo Souza 2013/2014
• Platform for Setting up Interactive
Virtual Environments (pSIVE) 2013/2014
– Virtual Visits
– Training
– Marketing
• Easy Creation
• Abstraction Layer
– Users – Frameworks
50
System Requeriments
• Hardware Independent
• Simple Configuration
– 3D Model Loading
– Attatch/Display Multimedia Information
– Technical aspects
– Reusable
• Well Defined Art Pipeline
– CAD softwares -> pSIVE
– Avoid calculations
• Cover part of IEETA’s devices
51
Architecture
• VR Juggler– Qualities met project requeriments
– Active Community and Develop
– Well documented
• OpenSceneGraph (OSG)– Easy-to-work-with solution
– Books available
– Very active project
– Support vast list of formats/software• 3D Studio Max, SketchUp, Maya, Blender...
• Why not Game Engines?
52
Platform Overview
53
Multimedia Module
54
Interaction Module
55
Menu Module
56
Case of Study
PRODUTECH-PTI / Tegopi
57
58
VR for training and formation
Update pSIVE – João Pinto 2014/2015
• Support for newer HMD’s (oculus rift)
– OpenScenGraph with osgOculusViewer
• Kinect support
– VRPN and FAAST
João Pinto 2014/2015
59
Architecture
60
VR – Full body interaction- 2015
61
Interação em RV – Full body interaction
62
Interação em RV – Manipulation
63
Mobile Interaction – Luis Afonso 2015-2016
• Move to Unity
– Oculus, leap and Android support
64
Architecture
65
66
VR Interaction with mobile devices
1- tablet for tracking of marker on HMD - Oculus
2- Leap Motion for hand tracking mounted on HMD
67
VR Interaction with mobile devices
68
User evaluation in Virtual and Augmented Reality
2006-2007
Mestrado em Gestão de Informação:
Angela Pimentel Amado
Projecto de licenciatura:
Jan-Willem Baggerman
Pauline van der Horst
Frank van Huussen
User evaluation
69
Virtual reality - User evaluation
70
Virtual reality - User evaluation - results
• Objective: Test navigation and visualization
• Methodology: Quantitative and qualitative
• 44 users (12 women / 32 men);
2 environments
RV x DV- HMD - Monitor
- Mouse - Keyboard
- Tracker - Mouse
results
- Preference for RV environment;
- RV causes more dyziness and disorientation;
- navigationa and visualization easy in both;
- better performance in DV.
71
2 Projectos 5º ano
Ricardo Casaleiro
Ricardo Seco
2006-2007
FCT Project
AcousticAVE – 2011-2014
Sound in VR
72
Sound in VR
• Auralization in closed spaces• Use of HRTF
• Computation of first order reflections ( ray-tracing)
• Computation of RT 60 with differente materials
• Visualization and auralization in VR
environments
• VTK, sound processing libraries (PortAudio,
libSoundFile, ...)
….(Burdea and Coiffet., 2003)
Stereo vs 3D Sound
73
74
Software to edit
acoustic properties in
3D models to compute
reverberation times
Sound in VR
75
3D reconstruction and auralization
Anta Pintada de Antelas
• Auralization with first order reflections and
HRTFs
Auralization: Reflections
76
Som directo
Primeiras reflexões
reverberação tardia
Tempo
dB
Tempo de reverberação
Atraso do som directo
Atraso das primeiras reflexões
Atraso das reverberação tardia
Auralization final demo AcousticAVE project
77
78
Bolseiros:
Bernardo Marques
João Alves
Tiago Araujo
Luis Afonso
Inês Cruz
2016-…
Augmented and Virtual Reality projects
Mestrados:
Ricardo Silva
Raphael Carvalho
Anna Andreikanich
Rafael Esteves
2017-…
VR4Rehab - Rovisco Pais
79
Continuous AR
80
Task 1: Catch the pumpkin Task 2: Count the spider
Task 3: Content selection Task 4: Information gathering
Continuous AR interfaces
AR for maintenance
82
AR for education
83
• EdupARk
Download and install - EdupARk & Unity
http://edupark.web.ua.pt/
84
https://play.google.com/store/search?q=edupARk
https://unity.com/
Unity Installation - Instalation
85
• Unity Installation
• Unity Account Asset store
Questions?
To do – Next week
• Install Unity
• Create Asset store account
• Remember to select for presentation:
– papers : IEEE VR2018, VRST, ISMAR / VR
– Or framework : Unreal, ARCore,
86
87
The people!• Beatriz Sousa Santos
• Guilherme Campos
• Joaquim Madeira
• Samuel Silva
• Os muitos alunos que trabalharam nos projectos:– Ricardo Casaleiro, Ricardo Seco
– Gustavo Corrente, Ana Cantanhede
– Angela Pimentel
– Jan-Willem Baggerman, Pauline van der Horst, Frank van Huussen
– Mário Cruz e Paulo Moreira
– Rui Palha, Filipe Duarte, Igor Cardoso, Tiago Sousa, João Parracho, João Cardoso
– André Oliveira
– Danilo Souza
– José Mendes, João Rodrigues, Ricardo Silva, José Sequeira, Vasco Santos, Miguel Vicente, Joel Pinheiro, André Alves, Luis Morais
– João Pinto, Sérgio Eliseu, Luis Afonso, João Cardoso
– José Ferrão, Miguel Ferreira, Pedro Coelho, Rafael Almeida, Diogo Carvalheira
– Raphael Carvalho, Anna Andreikavich, Ricardo Silva, Rafael Esteves, Inês Cruz
– João Alves Bernardo Marques,
– ...
Thanks
88
Questions?