AUGMENTED REALITY NETWORK: A COLLABORATIVE AUGMENTED REALITY DESKTOP GAME

Amy Tiong Siaw Wei

Master of Science 2010

......

Punt Khldmat M~klumllt Akademlk UNIVERSITI MALAYSiA SARAWAK I

I

AUGMENTED REALITY NETWORK: A COLLABORATIVE AUGMENTED REALITY DESKTOP GAME

P.KHIDMAT MAKLUMAT AKADEMIK

111111111 rli'~iillllllill 1000248342

AMY TIONG SlAW WEI

A thesis submitted in fulfillment of the requirement for the Degree of

Master of Science (Cognitive Science)

, I Faculty of Cognitive Sciences and Human Development

UNIVERSITI MALAYSIA SARA W AK 2011

I

I

......

ACKNOWLEDGEMENTS I wish to thank my supervisor Dr. Ng Giap Weng for giving me advice, guidance,

assistance and comments.

ii

-

I

l Pusat Kbidmat Maklumat Akadernik UNlVERSm MALAYSIA SARAWAK

TABLE Of CONTENTS iiACKNOWLEDGEMENTS

iiiTABLE OF CONTENTS

viiLIST OF FIGURES

ixLIST OF TABLES

xABSTRACT

xiABSTRAK

1CHAPTER 1 INTRODUCTION

11.1 Background

31.2 Problem Statement

1.3 General Objective 5

51.4 Specific Objectives

51.5 Research Framework

61.6 Definition of Terms

61.7 Limitations

71.8 Scope

71.9 Significance of the Study

81.10 Summary

10CHAPTER 2 LITERATURE REVIEW

102.1 Augmented Reality (AR)

132.2 Vision-Based Tracking

162.3 AR and Networking

iii

,... I

2.4 Network 19

2.5 Protocol 22

2.5.1 Open System Interconnection (OSI) Reference Model 23

2.6 Transmission Control Protocol (TCP)lInternet Protocol (IP) 26

2.6.1 History ofTCPIIP 27

2.6.2 TCPIIP Architecture 27

2.6.3 Transmission Control Protocol (TCP) 30

2.6.4 User Datagram Protocol (UDP) 32

2.6.5 Internet Protocol (IP) 33

2.7 Collaborative AR 34

2.8ARGames 36

2.9 Summary 38

CHAPTER 3 METHODOLOGY 41

3.1 System Design 41

3.1.1 Client-Server Architecture 41

3.1.2 Game Design 44

3.2 Development Tools 46

3.3 Evaluation 47

3.4 Summary 49

CHAPTER 4 IMPLEMENTATION 51

4.1 AR Network 51

4.1.1 Server Side Algorithm 51

4.1.2 Client Side Algorithm 55

IV

-~

594.1.3 Multi-Threading

604.2 SameGame Algorithm

624.2.1 SameGame functions

684.3 Live Video Input

704.4 Vision-Based Tracking

4.5 Interactions 71

4.6 Sound Effects 73

4.7 Summary 73

CHAPTER 5 FINDINGS AND DISCUSSION 75

5.1 Server 75

5.2 Client 76

5.3 Vision-Based Tracking 77

5.4 SameGame 79

5.5 Evaluation 80

5.6 Discussion 85

5.7 Summary 88

CHAPTER 6 CONCLUSION 90

6.1 Contributions of the Study 90

6.2 Recommendations for Future Study 94

6.3 Summary of the Research 95

98

APPENDICES

REFERENCES

109

109Appendix 1: Consent Fonn

v

-

II

I

I

Appendix 2: Information for Research Participants 111

Appendix 3: Questionnaires 113

VI

LIST OF FIGURES Figure 1.1: Research framework of visual perception and Augmented Reality in

collaboration. 5

Figure 2.1: Continuum of real-to-virtual environments (Milgram & Kishino,

1994). 10

Figure 2.2: Fiduciary marker ofARToolKit. 15

Figure 2.3: Fiduciary marker ofARToolKitPlus. 16

Figure 2.4: OS! Reference Model (Forouzan, 2002, p. 20.) 24

Figure 2.5: TCP/JiP protocol suite (Hunt, 2002, p. 10.) 28

Figure 2.6: Data encapsulation (Hunt 2002, p. 11.) 30

Figure 3.1: AR Network system architecture. 42

Figure 3.2: Server side system architecture. 43

Figure 3.3: Client side system architecture. 44

Figure 3.4: Flowchart of SameGame. 45

Figure 4.1: Source code of server application. 52

Figure 4.2: amTcpServer class. 52

Figure 4.3: amTcpConnection class declaration. 54

Figure 4.4: Multi-threading in client application. 55

Figure 4.5: Algorithm of thSend 56 I

I I

Figure 4.6: Algorithm of thReceive. 57

Figure 4.7: amTcpClient declaration. 58

Figure 4.8: Class declaration for puzzle. 60

Vll

Figure 4.9: arnArray2DHasNeighbour function. 62

Figure 4.10: arnArray2DDestory function. 63

Figure 4.11: arnArray2DDrop function. 64

Figure 4.12: arnArray2DDropLine function. 65

Figure 4.13: arnArray2DHasMove function. 66

Figure 4.14: Snippet of SameGame interaction. 66

Figure 4.15: Snippet of DirectShow. 67

Figure 4.16: Snippet of sample grabbing. 68

Figure 4.17: Snippet of vision-based tracking for AR registration. 69

Figure 4.18: Snippet of GLUT to get the user input. 71

Figure 4.19: Snippet ofAudiere for sound effects. 72

Figure 5.1: AR Network server side command. 74

Figure 5.2: AR Network client side command. 75

Figure 5.3: Screenshot of fiduciary marker. 76

Figure 5.4: Screenshots of registration. 77

Figure 5.5: Screenshots of Same Game game play. 78

Vlll

,...

Table 5.1

Table 5.2

Table 5.3

--_ ..... ­

LIST OF TABLES Demographic data of the participants in SameGame using AR

Network 80

The time used by six groups of participants to complete the

tasks: 1) Use fiduciary marker for detection, 2) play the

game individually, and 3) solve the puzzle collaboratively. 82

Means and standard deviation of the system evaluation onAR

Network. The value 1 denotes "Strongly disagree"; 2 denotes

"Disagree"; 3 denotes "No opinion"; 4 denotes "Agree"; and 5

denotes "Strongly agree" 83

ix

ABSTRACT AUGMENTED REALITY NETWORK: A COLLABORATIVE AUGMENTED

REALITY DESKTOP GAME

(Augmented Reality (AR) is a technology that augments virtual objects in the real

environment. The users can see and interact with the virtual objects. AR can

enhance the real world with the virtual infonnation and increase the productivity

of our daily life. Collaborative AR provides a shared space to the users and allows

the users to solve problems together. Through this research, a collaborative AR ·

system called AR Network was designed and developed. AR Network allows the

users to playa game collaboratively) The game developed for AR Network is

called SameGame. SameGame was used to represent a problem which the users

can solve collaboratively. This indicates that AR Network has the possibilities to

be implemented in other fields such as education and entertainment. In order to

implement an AR Network, Transmission Control Protocol (TCP)lIntemet

Protocol (IP) suite was used to transfer data in the network. Client-server

architecture was used in this research so that the data can be stored in the server

and transferred to the clients. Vision-based tracking was used for AR registration.

Fiduciary markers were required to compute the orientation and the position of the

camera so that the computer generated objects .were correctly placed in the real

world. AR Network was evaluated for its functionality and its perfonnance. AR

Network allowed more than two participants to solve a puzzle collaboratively. The

results also showed that the participants had positive perception towards

collaboration in AR Network.

x

I

ABSTRAK AR NETWORK: P ERMAINAN DESKTOP KOLABORATIF A UGlvfENTED

REALITY (AR)

Augmented Reality (AR) adalah teknologi yang menambah objek maya dalam

persekitaran nyata. Para pengguna dapat melihat dan berinteraksi dengan

benda-benda maya. AR boleh meningkatkan dunia nyata dengan maklumat maya

dan meningkatkan produktiviti kehidupan kita. AR kolaboratif menyediakan

ruangan bersama bagi pengguna dan membolehkan para pengguna untuk

menyelesaikan masalah bersama-sama. Tujuan penyelidikan ini adalah membina

sebuah sistem AR kolaboratif, "AR Network", yang membenarkan para pengguna

untuk bermain sebuah permainan bersama-sama. Permainan yang dibina untuk

"AR Network" adalah SameGame. SameGame digunakan untuk mewakili

sebagai suatu masalah di mana para pengguna dapat menyelesaikan bersama.

Ini menunjukkan bahawa "AR Network" berpotensi untuk dilaksanakan dalam

bidang lain seperti pendidikan dan hiburan. Untuk membina sistem AR

kolaboratif, set Transmission Control Protocol (!,CP)IInternet Protocol (lP) telah

digunakan untuk pemindahan data dalam rangkaian. Seni bina "client-server"

telah digunakan dalam penyelidikan ini supaya data boleh disimpan dalam

"server" dan dipindah kepada para pengguna. Kaedah pendaftaran "vision­

based" telah digunakan untuk pendaftaran AR. "Fiduciary marker" digunakan

untuk mengira orientasi dan kedudukan kamera supaya komputer dapat

menghasilkan benda dan dijajarkan dalam dunta sebenar. "AR Network" dinilai

untuk jungsinya dan prestasinya. "AR Network" membenarkan lebih daripada

dua peserta untuk menyelesaikan sesuatu masalah secara bersama-sama.

Keputusan juga menunjukkan bahawa para peserta mempunyai persepsi positij

dengan "AR Network" dalam kerjasama.

Xl

\

CHAPTER 1 INTRODUCTION

1.1 Background Augmented Reality (AR) is known as a deviation of Virtual Reality (VR) (Azuma,

1997). AR is a technology that augments virtual objects in the real environment

(Azuma 1997; Azuma et aI., 2001; Aaltonen & Lehikoinen, 2006). There are three

characteristics to describe the AR technology (Azuma, 1997). These

characteristics are 1) combining real and virtual objects, 2) allowing users to

interact in real time, and 3) registering in 3D form. Therefore, AR technology

generates 3D objects in the real environment which users can interact with them.

AR provides a possibility to improve the real world with the virtual world (Feiner

et al., 1993).

AR technology is beneficial in many fields to enhance human productivity in

1

,... ,..

daily lives. AR technology has been applied in various fields such as medical,

education, training, entertainment, maintenance and repair, aviation, gaming and

others (Azuma, 1997; Bonsor, 2001; EDUCAUSE Learning Initiative, 2005).

AR has been implemented in indoor and outdoor gaming as first person shooting

games (Thomas et al., 2000). Besides fust person shooting game, AR was also

involved in real-time strategy game (phillips & Pierkarski, 2005), racing game

(Oda et al., 2008). A Tetris clone was also developed as an AR game (Wichert,

2002). The Tetris clone allows the users to play the game collaboratively.

Barakonyi et al. (2005) proposed an AR game with autonomous agents which

observe and react based on the interactions of the players. Furthermore, board­

games like Virtual Roulette and Mah-Jongg (Szaiavari et al., 1998) was also

developed in previous research. AR2Hockey (Oshima et al. , 1998) is another

example ofAR games. AR2Hockey used physical objects as the tangible interface

to interact with the virtual object.

Computer Supported Cooperative Work (CSCW) is merging with AR technology

and it constitutes collaborative AR (Wichert, 2002). The collaboration allows the

users to solve a problem together. This shows the possibilities of collaborative AR

in solving other kinds of problems besides games. The interactions of the user will

affect the virtual objects in the AR environment. Thus, the other users can see the

outcome of the interactions.

2

Collaboration can be implemented as face-to-face collaboration or remote

collaboration. AR can totally improve both face-to-face collaboration and remote

collaboration (Wichert, 2002; Ismail & Sunar, 2009). Face-to-face collaboration

indicates that the users are able to see each other. This situation allows the users

have natural communication because users can see the facial expression and

gestures of the others. On the other hand, the remote collaboration of AR will

generate virtual video of the remote users in the real environment. Thus, the user

can see the other users as the AR object in the real environment.

1.2 Problem Statement Visual perception is a cognitive ability. Human uses visual perception to receive

and understand the information from the real world. Computer vision is a

technology that simulates human visual perception to understand the information

of the real world. Similarly, AR technology uses computer vision algorithm to

understand the real world and augment extra information in the real world to

enhance our life.

AR technology is a technology that augments virtual objects in the real

environment (Azuma, ]997; Aalthonen & Lehik?inen, 2006). Users can perceive

and interact with the virtual objects within their real environment. AR requires

fiduciary marker to generate the virtual information on top of the real

environment. The users can interact with the virtual objects using the available

input devices such as keyboard and mouse.

3

I

Many applications were developed in AR, but there were only a few applications

in AR gaming area (Wichert, 2002). Wichert (2002) stated that most of the AR

systems do not support collaboration. Collaborative AR environments allow

multiple users to interact with the same virtual objects. Furthermore, Oshima et al.

(1998), stated that collaborative AR system requires moderate registration

accuracy, real-time response, and no mutual exclusion. Registration accuracy as

appeared to the shared environment refers to the orientation and position of the

virtual objects in the real environment. Real-time response indicates the

interaction speed with the virtual objects after the input device is triggered. No

mutual exclusion indicates the system could be simultaneously controlled by

mUltiple users.

The implementation of the collaborative AR environments requires networking

feature, so that the user's information can be sent and received among each other.

The collaborative AR that was designed by Wichert (2002) was a web-based AR

using Hypertext Transfer Protocol (HTTP). The client-server architecture was

used so that the users have a shared space in the AR system. The game was played

using Internet Explorer. However, there is a lack of information on how the client­

server architecture works in the collaborative ~. The problem statement of this

research is only a few applications which were developed in AR gaming area for

collaboration. Therefore, this research involves designing and developing a

collaborative AR application and implementing it in a game.

4

Pusat Khidmat Maklumat Akademik UNlVERSm MALAYSIA SAKAWA]{

I

1.3 General Objective The general objective of this research is to design and develop a collaborative AR

system, "AR Network", which allows the users to playa game collaboratively.

1.4 Specific Objectives The specific objectives of this research are shown as follows:

1. To design and develop a network architecture which allows the users to

work collaboratively.

2. To develop a game, SameGame, as an example of a collaborative AR

application.

3. To integrate the network architecture into desktop AR game that allows

two or more users to interact with the virtual objects collaboratively.

1.5 Research Framework

Visual perception Visual perception

CollaborationUser User

Figure 1.1: Research framework of visual perception and Augmented Reality in collaboration.

5

Figure 1.1 above shows the research framework of this study. It shows the

relationship of Augmented Reality and visual perception of the users. The virtual

objects generated by Augmented Reality are able to be perceived by the users.

Since more than one users are able to perceive same virtual objects, the users

should be able to collaborate among each other to interact with the virtual objects

generated by Augmented Reality. Therefore, this research develops a collaborative

AR system that allows the users to solve the problem collaboratively.

1.6 Definition of Terms

• Augmented Reality (AR) - AR is a technology that blends the virtual

environment with real environment by augmenting virtual objects in the

real time environment (Azuma 1997; Azuma et aI., 2001; Aaltonen &

Lehikoinen, 2006).

• Networking - Networking is a process of designing, developing, and

managing networks (Shanmugan et aI., 2002; Kozierok, 2005).

• Network - A network is a group of communication devices that are

connected together, to share information by using software and hardware

devices (Forouzan, 2002; Shanmugan et al., 2002; Kozierok, 2005).

• Collaborative AR ­ The users share the same AR environment so that the

users can work collaboratively (Reitmayr & Schmalstieg, 2001).

1.7 Limitations There were several limitations in this research study. Collaborative AR system

6

performance depends on the network bandwidth. Low bandwidth will decrease

the collaborative AR system performance. The low bandwidth hinders the smooth

transmission of the shared data through the network. On the other hand, if the

network bandwidth is high, the data can be smoothly transmitted through the

network. Thus, the performance of the system will be improved.

Moreover, this research did not evaluate the maximum number of clients which

the AR Network could support. This research only tested with three participants

using the system at the same time. However, the system was designed to support

more than three clients playing the game concurrently.

1.8 Scope This research developed a collaborative AR system, "AR Network", which will be

used by those who want to experience AR technology in a collaborative

environment. The application of AR Network was SameGame. The game was

used to represent a problem which can be solved collaboratively. Face-to-face

collaboration was applied in this research study. The implementation of the

system was using C++ programming language with Boost C++ Libraries. The

client-server architecture was used for th€? system. The functionalities and

performance of the system was tested in this research study.

1.9 Significance of the Study In this research, AR Network was developed. AR Network is a collaborative AR

7

I game with networking feature. The system consisted of the AR technology and

client-server architecture which provided a shared space for the users to work

collaboratively when performing a task. In this research, the system was applied

in a collaborative gaming application, SameGame. The system allows multiple

users to play the game in a collaborative environment where they can

collaboratively solve the problem. The interactions of the users are in a shared

space. Thus, the response of the interactions with the virtual objects can be

perceived by all the users in the AR environment. This study indicates the

possibilities of performing collaborative tasks in AR environment in other areas

such as education, entertainment, medical, tutoring, and others. This research can

also be used as a reference in the future research studies related to AR. The result

of the [mdings allows the future studies to improve the AR design to produce

better collaborative environment.

1.10 Summary

Augmented Reality (AR) is a technology that augments virtual objects in the real

environment. AR is used to enrich the real world with the virtual infonnation. AR

can be applied in many fields for instance, medical, education, training,

entertainment, gaming, and others. AR has been applied in several gaming system,

such as racing game, real-time strategy game, first player shooting game, board­

games, and other games. Moreover, Computer Supported Cooperative Work

(CSCW) is merging with AR as collaborative AR. Wichert (2002) stated that

many applications were developed in AR but only a few applications in gaming.

8

Besides that, most of the AR systems do not support collaboration. Collaborative

AR allows several users to interact with the same virtual objects. Wichert (2002)

designed a collaborative AR as a web-based application using HTTP and based on

client-server architecture. However, there is a lack of infonnation for the design of

client-server architecture for collaborative AR. The general objective of this

research is to design and develop a collaborative AR system, "AR Network",

which allows the users to playa game collaboratively. The game developed in this

research was SameGame. One of the limitations of this research was that the

perfonnance ofAR Network depends on the network bandwidth. Besides that, this

research did not evaluate the maximum number of clients that the system was able

to support. The scope of this research focuses on the design and development of

AR Network and implementation of SameGame. The SameGame was used to

represent a problem where the users try to solve it collaboratively. This study

indicates the possibilities of performing collaborative tasks in AR environment for

other areas such as education, entertainment, medical, and others. The

functionalities and the perfonnance of the system were tested in this research

study.

In the next chapter, the literature review on AR t~chnology is discussed which are

the vision-based tracking in AR registration, AR in networking, network feature,

protocol, Open System Interconnection (OSI) Reference Model, Transmission

Control Protocol (TCP), User Datagram Protocol (UDP), Internet Protocol (IP),

collaborative AR, and AR games.

9

CHAPTER 2 LITERATURE REVIEW

2.1 Augmented Reality (AR) Augmented Reality (AR) is known as a deviation ofVirtual Reality (VR) (Azuma,

1997). Milgram and Kishino (1994) proposed a continuum of real-to-virtual

environments. The continuum of real to virtual environments is shown as below,

refer Figure 2.1 :

Mixed Reality

,-,--------------~---------------~, Real Augmented Augmented Virtual

Reality Virtualityenvironment Environment

Figure 2.1: Continuum of real-to-virtual environments (Milgram & Kishino, 1994).

10

In VR technology, users are totally immersed in the virtual environment (Milgram

et al., 1994; Azuma, 1997). This virtual environment is a simulation of the real

environment by using computer graphics. VR allows users to interact with the

virtual environment through five sensory systems: visual, auditory, tactile, smell,

and taste (Burdea & Coiffet, 2003). In AR technology, the virtual objects are

added in real environment; in Augmented Virtuality (AV), the real objects are

displayed in virtual environment. Hence, AR is a technology that augments virtual

objects in the real environment (Aztuna, 1997; Starner et al., 1997; Aalthonen &

Lehikoinen, 2006). This means that users can perceive virtual objects within the

real environment (Aztuna et aI., 2001). However, in VR technology, users cannot

perceive the real environment as they are totally immersed in a virtual

environment (Aztuna, 1997).

There are three characteristics to describe AR technology (Azuma, 1997). These

characteristics are 1) combining real and virtual objects, 2) allowing users to

interact in real-time, and 3) registering in 3D form. AR technology combines real

and virtual objects by registering the virtual objects in real environment. The

virtual objects that are augmented in real environment can be manipulated by

users in real-time. AR technology generates v~al objects in the real environment

whereby users can interact with them in real-time.

Besides augmenting virtual objects in real environment, AR technology has the

ability to remove the real objects which is known as "diminished reality" (Azuma,

11

1997; Azuma et al., 2001). Diminished reality can be done by covering the real

objects using virtual background. Therefore, users can remove unwanted

information in real environment by using AR technology.

AR technology can be applied in many fields in order to enhance users'

productivity in daily lives. AR can be used in medical, training, entertainment,

maintenance and repair, aviation, gaming, education and annotation (Azuma,

1997; Bonsor, 2001; Thomas, 2003; EDUCAUSE Learning Initiative, 2005;

Schrier 2006; Kaufmann & Meyer, 2008; Irawati et al., 2008). Moreover, AR

technology can be implemented in video conferencing (Kato et al., 2001;

Billinghurst & Kato, 2002; Billinghurst et aI., 2002). In AR video conferencing

system, users can see each other by using a fiduciary marker. The image of the

remote users is considered as virtual objects which are augmented in real

environment. Furthermore, AR can be applied in the entertainment field. Some of

the AR games that have been developed are ARQuake, a shooting game (Piekarski

& Thomas, 2002), AR racing game (Oda et aI., 2008), and AR Chinese checker

(Cooper et aI., 2004). Applications of AR technology in the entertainment field

allow users to gain new entertaining environment because the users can play the

games in an outdoor environment (Avery et ~I., 2005). Besides that, AR can be

applied in the collaborative communication system. Environment Management for

Multi-user Information Environments (EMMIE) has been implemented and users

can share a 3D virtual space and manipulate virtual objects that represent

information in their discussion (Butz et aI., 1999). Furthermore, AR was able to be

12