Virtual and Augmented Reality – 2019

MIECT, MIEET, MEI

Introduction to course

Universidade de Aveiro

Departamento de Electrónica,

Telecomunicações e Informática

Realidade Virtual e Aumentada Beatriz Sousa Santos, 2019/2020

Virtual and Augmented (Mixed) Reality

• Has been expanding from a research field into a commercially viable

technology.

• is highly interdisciplinary, combining work in diverse disciplines:

- more technology oriented research:

computer graphics, display technology, computer vision,

sensors, signal processing and computer networking.

and

- human centered research:

interaction techniques, design, human factors,

wearable computing and mobile computing.

• The growing interest in VAR applications is creating new challenges

for research in all of these areas

2

What is Virtual Reality?

“For better or worse, the label virtual reality stuck to this particular branch of

computer graphics.

I define a virtual reality experience as any in which the user is effectively

immersed in a responsive virtual world. This implies user dynamic control of

viewpoint.”

“A high-end user-computer interface that involves real-time simulation and

interaction through multiple sensorial channels (vision, sound, touch, smell,

taste)”.

(Fred Brooks, 1999)

3

(Burdea et al., 2004)

“The computer-generated simulation of a three-dimensional image or

environment that can be interacted with in a seemingly real or physical

way by a person using special electronic equipment, such as a helmet

with a screen inside or gloves fitted with sensors.” (Lexico, Oxford)

https://www.youtube.com/watch?v=izaBq084b0k

Early Virtual Reality – example

Division - 1991

4

Virtual Reality in practice for years – Automotive Industry

SEAT: VR in Car Production

https://www.youtube.com/watch?v=JAIn_xZ1-dY

5

Virtual Reality in practice for years – examples

McLaren uses Virtual Reality to design its sportscars and supercars

https://www.youtube.com/watch?v=mWaQfjEJIMQ

6

Toyota VR Experience: Reliability

http://vrworld.com/2016/03/14/toyota-creates-scary-vr-experience/

7

Virtual Reality in practice for years – Automotive Industry

Pain management

https://health.usnews.com/health-

care/patient-advice/articles/virtual-

reality-in-medicine

Medical training

https://www.youtube.com/watch?v=

PzPKdlFbvYM

Phobia therapy

https://www.youtube.com/watch?v=

GMttQHMjbJo

8

Virtual Reality in practice

– examples in Medicine

Entertainment

http://www.oculusvr.com/

Oculus Rift

2014; ~300 USD

9

http://www.youtube.com/watch?v=N8uuDT5AYts

“… systems that have the following three characteristics:

1) Combine real and virtual

2) Interactive in real time

3) Registered in 3-D” (Azuma, 1997)

“AR is a novel human machine interaction tool that overlays computer-

generated information on the real world environment. The information display

and image overlay are context-sensitive, which means that they depend on the

observed objects.” (Ong et al., 2008)

“A technology that superimposes a computer-generated image on a user's view

of the real world, thus providing a composite view.” (Lexico, Oxford)

10

What is Augmented Reality?

Note: this is a broader definition;

we will adopt Azuma’s definition.

http://www.youtube.com/watch?v=8UxWkZtUKaI (2009)

Early Augmented Reality – examples

Lego - 2009

11

Ray Ban Virtual Mirror - 2009

12

http://www.youtube.com/watch?v=Ag7H4YScqZs

http://www.ray-

ban.com/usa/virtual-mirror

13

Augmented Reality got known to the public

Pokemon Go - 2016

How Augmented Reality Is Driving Today's Automotive Industry

14

https://www.youtube.com/watch?v=VEwIZ7zwA0w

Augmented Reality – recent examples

http://www.techradar.com/reviews/wearables/microsoft-hololens-1281834/review

The Future of AR?

Hololens

2016; ~3000 USD

https://www.microsoft.com/microsoft-hololens/en-us

R. Furlan,”The future of augmented reality: Hololens - Microsoft's AR headset

shines despite rough edges”, IEEE Spectrum, 2016, 53 (6), p. 21

Mixed Reality: “Continuum”

“Augmenting natural feedback to the operator with simulated cues”

16

(Milgram et al., 1994)

Mixed Reality (MR)

Real Envinoment Augmented Reality Augmented Virtuality Virtual Reality

(Steinicke et a., 2009) http://www.youtube.com/watch?feature=endscreen&v=UgQfo7eNFdw&NR=1

Team

Beatriz Sousa Santos; Bernardo Marques

(bss@ua.pt; bernardo.marques@ua.pt)

17

• P. Dias, R. Silva, P. Amorim, J. Lains, E. Roque, I. Serôdio, F. Pereira, Beatriz

Sousa Santos, “Using Virtual Reality to Increase Motivation in Poststroke

Rehabilitation”, IEEE Computer Graphics and Applications, vol.39, no.1, pp. 65-70,

2019

• Bernardo Marques, R. Carvalho, P. Dias, Beatriz Sousa Santos, “Pervasive

Augmented Reality for Indoor Uninterrupted Experiences: a User Study”,

UbiComp/ISWC'19, London, September 2019

• A. Andreikanich, Beatriz Sousa Santos, P. Amorim, H. Zagalo, Bernardo Marques, P.

Margalho, J. Laíns, F. Faim, M. Coelho, Teresa Cardoso, Paulo Dias, “An Exploratory

Study on the use of Virtual Reality in Balance Rehabilitation“, 41th IEEE Engineering

in Medicine and Biology Conference, Berlin, July 2019

• Bernardo Marques, Beatriz Sousa Santos, T. Araújo, N. Martins, J. Alves, P. Dias,

“Situated visualization in the decision process through augmented reality”, 23rd

International Conference Information Visualisation 2019 - IV 2019, p. 13-18, Paris,

July 2019

• Bernardo Marques, R. Esteves, J. Alves, C. Ferreira, P. Dias, Beatriz Sousa Santos,

“Investigating different Augmented Reality Approaches in Circuit Assembly: a User

Study”, Eurographics Short Papers 2019, p. 45-48, May 2019

• J. Alves, Bernardo Marques M.Oliveira, M., T. Araújo, P. Dias, Beatriz Sousa Santos,

“Comparing Spatial and Mobile Augmented Reality for Guiding Assembling

Procedures with Task Validation. 2019 IEEE International Conference on

Autonomous Robot Systems and Competitions (ICARSC), April, 2019

19

Recent publications

Topics

• Introduction to Virtual Reality (VR) Augmented (AR) and Mixed Reality (MR)

• Definition, historical perspective, evolution, and applications

• Architectures and Frameworks for MR

• Input and output devices, tracking

• 3D user interfaces and interaction techniques

• Human factors in MR

• Human-Centered Design for MR

• Evaluation of MR applications

• Traditional and emerging applications

http://sweet.ua.pt/bss/disciplinas/RVA/RVA-home.htm

20

Lectures and Lab classes (tentative)

1 (Sep/17) - Introduction to the course, syllabus, assessment, bibliography

Paper presentation guidelines

2 (Sep/24) - Work on Virtual, Augmented in DETI/IEETA

3 (Oct/1) - Introduction to VR and AR (cont.) / Paper presentation

VR and AR Frameworks; Introduction to Unity

4 (Oct/8) - Input Devices and trackers / Paper presentation

Unity

5 (Oct/15) – Output Devices (visual) - Stereoscopy / Paper presentation

Toolkits for AR (Vuforia/ Unity)

6 (Oct/21) - Output Devices (other) - Paper presentation

Practical assignment

21

7 (Oct/29) – Human Factors in MR - Paper presentation

Toolkits for AR (Vuforia/ Unity)

8 (Nov/5) - Human Centered Design for MR – Paper presentation

Practical assignment

9 (Nov/12) - Interaction in MR– Paper presentation

Practical assignment

10 (Nov/19) – Hardware Demo – Paper presentation

Practical assignment follow-up presentation

10 (Nov/26) - Interaction in MR (cont.) - Paper presentation

Practical assignment

12 (Dec/3) – Evaluation methods – Paper presentation

Practical assignment

13 (Dec/10) – Emerging applications of MR - Paper presentation

14 (Dec/17) – Presentation and demo of practical assignments

22

Lectures and Lab classes

• ~2h – lecture + paper presentation and discussion

• ~1h – lab

• 2 sessions devoted to presentation and demo of the assignments

23

Main bibliography

• Jerald, J., The VR Book: Human-Centered Design for Virtual Reality, ACM and

Morgan & Claypool, 2016

• LaValle, S., Virtual Reality - Virtual Reality. Cambridge University Press, 2017

(http://vr.cs.uiuc.edu/)

• LaViola, J., Kruijff, E., McMaha, R., Bowman, D, Poupyrev, I. J., 3D User

Interfaces: Theory and Practice, 2nd ed., Addison Wesley, 2017

• Schmalstieg, D., Hollerer, T., Augmented Reality: Principles and Practice (Usability).

Addison-Wesley Professional, 2016

• Craig, A., Sherman, W., Will, J., Developing Virtual Reality Applications:

Foundations of Effective Design, Morgan Kaufmann, 2009

• Carmigniani, J., & Furht, B, (eds) , Handbook of Augmented Reality, Springer, 2011

• Burdea, G., P. Coiffet, Virtual Reality Technology, 2nd ed., Wiley, 2003

• Journal and Conference Papers

24

- Exam – 40%

- Practical assignment – 45%

- Paper presentation and discussion – 15%

• Working students must contact us until September 30

Assessment

25

Practical assignment (general characteristics)

• Groups of two students

• Mini-project

• Development of a simple MR application using a human-centered

approach, specific devices and libraries

• Existing code may be used, provided that it is explicitly identified

and its source adequately cited

26

Our devices

27

Analyze and present a conference/ journal paper

a book section or a MR framework/tool:

1 student – short paper (~4 pages)

2 students– long paper (8+ pages) or book section Each student must:

- Decide if s/he will present alone or with a colleague

- select two papers,

a book section

send preferences to (bss@ua.pt)

propose another paper until September, 30

or a framework

- Read the paper presentation guidelines (presentation guidelines)

- Make a 15 mim (1 student) or a 30 min (2 students) presentation

- Send the slides to (bss@ua.pt)

28

Papers to read, present and discuss

http://ismar.vgtc.org/

29

http://ieeexplore.ieee.or

g/Xplore/

http://dl.acm.org/

http://www.springer.com

/computer/image+proce

ssing/journal/10055

And other

journals or

conferences

Discuss papers

• After each paper presentation:

- All students vote on their colleagues presentation

(through googleforms)

- Two students will discuss the paper

30

https://docs.google.com/forms/d/1rXAb2LlJTaj74-

xWMhhYMwWcNoUlzBUiyY9gZp2OFT0/viewform

• Students’ profile/background?

• Questions?

32