Computer Graphics Lab

Multiple Screens

University of Trento

Mu Qiao (133734)

Supervisor: Dr. Giuseppe Conti Prof. Raffaele de Amicis

- 1 -

Table of Contents 1. Introduction..................................................................................................................... 3 2. Related Work .................................................................................................................. 6

2.1 Studierstube............................................................................................................... 6 2.2 VRJuggler ................................................................................................................. 7 2.3 Java3D....................................................................................................................... 9

3. Methods and Implementation of Multiple Screens....................................................... 11 3.1 Development Tools, libraries and Platform............................................................ 11 3.2 Rotating Cube Example .......................................................................................... 13

3.2.1 Class Diagram of Original Rotating Cube ....................................................... 14 3.2.2 “Master-Slave” Rotating Cube Extension ....................................................... 14 3.2.3 Network Communication through Java Socket ............................................... 17 3.2.4 Multiple Screens Effect Realization by Adjusting Viewport .......................... 21 3.2.5 Synchronization of “Slave” Applications ........................................................ 24

3.3 World Wind Java SDK ........................................................................................... 25 3.3.1 Extension of Original World Wind Java SDK................................................. 25 3.3.2 Network Communication through Java Sockets.............................................. 28 3.3.3 Multiple Screens Effect Realization by Adjusting Viewport .......................... 30

4. Performance Evaluation and Future Work ................................................................... 31 4. Summary ....................................................................................................................... 33 References......................................................................................................................... 34

- 2 -

1. Introduction In many circumstances, wider vision is needed to fulfill special requirements like reality

authentication of scenes (see Figure 1), commodity displayed in an exhibition, panorama

of a soccer match, exciting and shocking effect of computer games or simulation etc (see

Figure 2). A simple and costly way is to purchase a wider screen several times larger

than a normal one. An alternative is to utilize and combine multiple screens to achieve

the same effect. The advantage of multiple screens is the competitive cost, easy

transportation, free rearrangement over their relative positions. This report describes an

approach of using a “Master-Slave” network communication mechanism by using Java

Bindings for OpenGL API (Jogl), based on a real implementation, to achieve the effect of

displaying 3D object within multiple screens.

Figure 1: Reality authentication of beautiful scenes (from the website of 9x Media)

- 3 -

Figure 2: Computer Simulation of Aircraft (from bp2.blogger.com)

As part of the group project which aims to build a control station featuring in multiple

screens, advanced control system, sensors support and collaborative features, one of the

features described in this report is its multiple screens effect. By building a Java project

composed of both “Master” and “Slave” applications communicating through Java

sockets, the project is then planted over multiple PCs. One PC is running “Master”

application which controls several “Slave” applications running on other PCs. Multiple

screens effect is realized by synchronizing steps of “Slave” applications controlled by one

“Master” application.

The work begins with a relative simple application on a rotating 3D cube. The cube can

be controlled by both mouse and keyboard. The implementation results in a wider scene

composed of two screens through whose border the cube is able to “traverse”. Besides,

the size of the cube is magnified on two screens compared with the original one displayed

- 4 -

only in one screen. The steps of the rotating and moving of the cube is simultaneous over

the two “Slave” applications. After the rotating 3D cube application, further

implementation based on World Wind Java SDK from NASA is conducted in order to

achieve the similar result. With the much more complicated project structure of World

Wind compared with the 3D cube, many classes within the original project are extended

to achieve “Master-Slave” mechanism with good extensibility and transplantability.

The implementation of the multiple screens effect based on World Wind Java SDK can

be divided into steps below:

1. Identifying and extending classes which can be respectively inherited for different

use of “Master” and “Slave” applications as well as some of their ancestors and

offspring when needed. For example, classes of the drawing the earth, the

controlling of the earth and the registrations of these features are realized

differently from “Master” to “Slave” applications.

2. Realizing the “Master-Slave” communication through Java socket. “Master”

earth controls the orientation, rotation and size of the “Slave” earths by

establishing server-client communication. “Slaves”, treated as several different

servers, are listening to a “Master” which is treated as one single client. The

“Master” client controls several “Slave” servers by separately establishing

communication to each of the “Slaves” but sending the same command to all of

them in order to realize synchronization.

3. The sizes and positions of “Slave” earths need to be consistent to form a wider

view of a single earth within multiple screens. The screen size, position,

orientation, resolution as well as the viewport size of the scene display together

decides the relative sizes and positions among “Slave” earths.

This lab is focusing on the realization of multiple screens. This goal requires the

understanding of the graphic animation, network communication based on Jogl

implementation. The rest of the report is organized as below. Section 2 briefly

introduces other VR applications like Studierstube, VR Juggler and Java3D. Section 3 is

the core of the report beginning with the working environment of our application, and

- 5 -

mainly focusing on two multiple screens applications, one rotating cube example and one

based on World Wind application. Detailed explanation of the implementation procedure

is available through this part. In the end, this report ends with a summary as Section 4.

2. Related Work

2.1 Studierstube

The Studierstube augmented reality project, developed in Virtual Reality Group in

Technical University Vienna leaded by Prof. Dr. Dieter Schmalstieg, is one of the first

collaborative augmented reality systems in the world. Its innovative aspects are

represented by augmented reality, scientific visualization, multiple user support and

framework for easy extension and use [CBE].

Studierstube utilizes C++ to build on top of the Open Inventor (OIV) toolkit which

provides a higher layer of programming for OpenGL. OIV provides easy access to 3D

graphics and the core of it is an object-oriented scene graph scoring both geometric

information and active interaction objects [SFSEGP02]. OIV is extended when necessary

and form a toolkit of the OIV class hierarchy.

In the virtual environment, locales correspond to coordinate systems and coincide with

physical places [SFSEGP02]. Environment Management for Multi-User Information

Environments (EMMIE) [BHFMB99] developed by people from Columbia University

utilize one physical locale by all displays. This approach is simple to understand and

easy to implement. Studierstube uses a separate locale for each display in order to enable

all users having convenient access to all of the data items, especially with the increasing

data items [SFSEGP02]. Studierstube allows locales to be shared over displays. This

means though multiple displays can have independent points of view, they show images

of an identical scene graph (Figure 3). Usually, stationary displays can usually be

embedded in a single locale.

- 6 -

Figure 3: 3D Drag & Drop of An Application Object across Display Boundaries in

Studierstube Environment

2.2 VRJuggler

The VR Juggler project has been developed at the Iowa Center for Emerging

Manufacturing Technology (ICEMT) since 1997. VR Juggler is designed to “juggle” any

combination of devices, graphics APIs, and platforms that users want to use. VR Juggler

enables developers to create VR applications without knowing low-level details of VR

[JBBC].

VR Juggler is an application framework and set of C++ classes for writing virtual reality

applications. Developers are allowed to get direct access to various graphic APIs. The

hardware that VR Juggler supports includes CAVETM and C2, projection tables, simple

workstation monitor displays and traditional head-mounted displays [JBBC]. VR Juggler

currently runs on the Silicon Graphics, Inc. (SGI) platform and is being ported to Hewlett

Packard (HP) and Windows NT. For the multiple display effect supported hardware,

CAVETM, whose full name is Cave Automatic Virtual Environment, is an immersive

virtual reality environment where projectors are directed to multiple walls (see Figure 4).

The lifelike visual display is created by projectors positioned outside the CAVE and

controlled by physical movements from a user inside the CAVE. One of more computers,

often SGI workstation, drives the projectors. Techniques for rendering the scene include

OpenGL, OpenSG and OpenSceneGraph which are popular nowadays [Wiki-CAVE].

VR Juggler utilizes the power and flexibility of C++ and is thus an object-oriented

system. System details are handled by a set of Managers. For example, Draw Manager

deals with Graphics and Windowing API and communicates with Display Manager. All

- 7 -

the Managers are taken care of by a kernel on which user applications are based (see

Figure 5).

Figure 4: Virtual Reality - Multiple Walls of CAVE

Figure 5: Structure of VR Juggler

- 8 -

2.3 Java3D

Java 3D is a scene graph-based 3D API for Java platform. It runs on top of either

OpenGL or Direct3D. Java 3D is not only a wrapper around graphics APIs, but an

interface that encapsulates the graphics programming using a real, object-oriented

concept [Wiki-Java3D].

1) Camera-Based View Model

Java 3D model is different from camera-based view model in the sense of binding the

scene model and view model together. The Java 3D view model incorporates head

tracking directly and if not, it acts very much like the camera-based view model.

Camera-based view model is controlled by three parameters: location, orientation and

perspective. The lookAt method within it creates a viewing matrix whose inverse can be

used to control the ViewPlatform object within the scene graph.

2) Structure of Java 3D View Model

Figure 6: View Object, Its Component Objects and Their Interconnection

Java 3D view model (see Figure 6) defines a Locale Object within a VirtualUniverse

Object which specifies the reference point within the universe. There are two branches

below Locale Object. The left branch groups define how scenes are rendered and

- 9 -

possibly navigated (ViewModel) and the right ones define the content of the scene

(ObjectModel). For the ViewModel, the TransformGroup Node under the right

BranchGroup Node specifies the position, orientation and scale of the ViewPlatform.

The transformed ViewPlatform defines the view of the user in the virtual universe. In

other words, ViewPlatform is the position of the viewer, it can be obtained and moved in

space by a TransformGroup. Besides, ViewPlatform is referenced by a View Object

which specifies all the parameters needed to render the scene from the point of view of

the ViewPlatform.

3) Physical Body and Physical Environment

Figure 7: Object and View Platform Transformations

Physical Body and Physical Environment (see Figure 7) under the View Object

encapsulates all the parameters associated with head, eyes and corresponding trackers. If

no trackers are available, they are ignored and ViewPlatform becomes the position of the

camera.

4) Multi-screens

- 10 -

Figure 8: A Portion of a Scene Graph Containing Three Screen3D Objects

Same View Object can be displayed in several screens from different perspectives. For

multi-screen environment (see Figure 8 and Figure 9), Java 3D must know how the

Screen3D are placed with respect to each other, the tracking device, and the physical

portion of the coexistence coordinate system.

Figure 9: A Three-Screen Display Environment

3. Methods and Implementation of Multiple Screens

3.1 Development Tools, libraries and Platform

Development tools:

• Java programming language

• Eclipse IDE(version 3.3.2)

- 11 -

Libraries:

• Jogl (version JSR-231)

• GlueGen

Platform:

• World Wind Java SDK (version 0.4.1)

Jogl is a wrapper library that allows OpenGL to be used in Java programming language.

Jogl [Wiki-Jogl]. Jogl is designed to provide harware-supported 3F graphics to

applications written in Java and full access to the APIs in the OpenGL 2.0 specification

as well as nearly all vendor extensions, and integrates with the AWT and Swing widget

sets [OpenGL]. GlueGen is a Java tool which automatically generates the Java and JNI

code necessary to call C libraries from Java code and it is originally developed for Jogl

[Wiki-Gluegen]. World Wind Java SDK, developed by NASA, enables developers to

embed World Wind technology in their own applications [NWW]. Besides, World Wind

Java SDK is developed based on Jogl which implies the necessity of inclusion of Jogl

library in our application (see Figure 10).

Figure 10: Application structure of our application based on World Wind Java which is

based on Jogl and JDK

- 12 -

3.2 Rotating Cube Example

The implementation of a rotating cube is as a preparation for our goal application based

on World Wind Java SDK. The textured cube keeps rotating within a black background

and the view of the observer can be controlled by keyboard with “Up”, “Down”, “Left”,

“Right”, “PageUp”, “PageDown”, “A” and “Z”, respectively realizing the effect of

observer walking forward, backward, left and right, as well as looking up, down and

simply up, down. The desired aim of this implementation is to enable as observer

through controlling a “Master” cube by keyboard controls two “Slave” cubes in two

screens. Though there are two applications of “Slave” cubes existing, the final effect is

like only one larger cube being controlled within two screens (see Figure 11).

Figure 11: Two Screens Effect of A Rotating Cube

- 13 -

- 14 -

3.2.1 Class Diagram of Original Rotating Cube

Class diagram of original rotating cube is drawn to assist analyzing and further

implementing “Master-Slave” mechanism upon (see Figure 12). The rotating cube is

drawn within GLRenderer, which implements GLEventListener, is added as GLCanvas’

listener. Besides, the instance of glRenderer is passed as a parameter to an instance of

KeyNavigator, extending KeyAdaptor, to implement the control of keyboards over the

rotating cube. The control is realized by modifying the parameter in an instance of

MovementStatus, an attribute of GLRenderer, while corresponding buttons are pressed.

The modification of these parameters will further change the view of a virtual camera

which represents the view point of the observer.

3.2.2 “Master-Slave” Rotating Cube Extension

In order to achieve the goal of controlling several “Slave” applications by one “Master”

application, the first step should be of extending original classes to “Master” and “Slave”

separately (see Figure 13).

The two classes that may need to be extended are GLRenderer and ApplicationStarter

because “Master” and “Slave” have different framework, drawing methods, what is most

important is their relationship as server and client (this will be specified in 3.2.3). For

GLRender, we change it to BasicGLRenderer which overrides the four functions “init”,

“display”, “displayChanged”, “reshape” because of its implementation of the interface

Figure 12: Class Diagram of Original Rotating Cube

- 15 -

Figure 13: Class Diagram of Multiple Screens Application of Rotating Cube

- 16 -

GLEventListener. Other functions of BasicGLRenderer are all abstract ones which are

implemented diffrerently by its offspring. The extensions of BasicGLRenderer are

MasterGLRenderer and SlaveGLRendere. They override the four functions “init”,

“display”, “displayChanged”, “reshape” again from BasicGLRenderer which implements

GLEventListener. The abstract functions in BasicGLRenderer are implemented

differently based on the requirement of “Master” and “Slave” applications. For

ApplicationStarter, extension of it is unnecessary because it only has one constructor.

Creation of class MasterApplicationStarter and SlaveApplicationStarter is proceeded as

needed. For example, MasterApplicationStarter includes MasterGLRenderer as an

attribute while SlaveApplicationStarter has SlaveGLRenderer. Other distinctions mainly

occur because of the network communication introduced in 3.2.3.

3.2.3 Network Communication through Java Socket

“Master” cube controls several “Slave” cube through network communication. For

example, button “Left” is pressed which moves the “Master” cube moves right (button

direction indicates the view position of the observer, “Left” means the observer moves

left, corresponding, object, the cube, moves right). The network communication is

realized in our application by sockets. Two classes of Socket and ServerSocket are used

for establishing the communication. Socket class implements client socket. A socket is

an endpoint for communication between two machines. ServerSocket class implements

server sockets. A server socket waits for requests to come in over the network. It

performs some operation based on that request, and then possibly returns a result to the

requester. We treat “Master” as client who has opened several sockets linked to different

servers, performed by “Slave” applications. Each “Slave” is seen as a server which opens

a serverSocket and keeps listening to specific port of the local address. Once client -

“Master” connects to the corresponding server, at the server side, a socket acting as a

client socket accepts what has been received from the client and then retrieves the objects

- 17 -

sent from the clients (see Figure 14).

Figure 14: Socket Communication between Server - “Slave” and Client - “Master”

Corresponding codes are illustrated and commented below:

a) Server - “Slave” // open a new ServerSocket listening to a specified port ServerSocket arrayServer = new ServerSocket(portNumReceived); System.out.println("Server listening on port ." + portNumReceived); System.out.println("Waiting for connections."); // accept() blocks the caller until a correct connection is present // an incoming connection is acknowledged and associated with an // immediately created socket. // The original socket is returned to the listening state Socket client = arrayServer.accept();

- 18 -

System.out.println("Accepted a connection from: " +

client.getInetAddress().getHostName()); // new ObjectInputStream() recovers input stream from the client ObjectInputStream ois = new ObjectInputStream(client.getInputStream()); // object1 reads serialized objects sent from the client Object object1 = ois.readObject();

b) Client - “Master” // specify the amount of servers the client is trying to establish

// communication with int arraySize = 3; Socket[] socket = new Socket[arraySize]; // try to build connection to the specified InetAddress and port try { socket[0] = new Socket("localhost", 4000); socket[1] = new Socket("192.168.253.20", 4001); socket[2] = new Socket("192.168.253.21", 4001); } catch(Exception e) {

System.out.println(e.getMessage); }

Thus, by using sockets, network communication is established between server - “Slave”

and client - “Master”. Next step is to specify serialized objects that needed to be

transferred in order to realize instant control from “Master” to “Slave”. Objects that are

transferred through sockets need to implement Java Interface Serializable. Three

serialized objects are specified as ObjectEvent, ObjectStatusEvent, POVEvent which

includes information, respectively, regarding cube 3D positions, status (rotating or not)

and point of view from the observer. For the purpose of convenience and clearness, three

classes called SendObjectEvent, SendObjectStatusEvent and SendPOVEvent are built to

respectively send the three serialized objects mentioned above. Simple structures can be

seen as:

SendObjectEvent sends ObjectEvent

SendObjectStatusEvent sends ObjectStatusEvent

SendPOVEvent sends POVEvent

- 19 -

SendObjectEvent and SendPOVEvent happen during the display() function within

MasterGLRenderer class. This means that during each display, positions and point of

views are sent from “Master” to “Slave” to let the latter keep steps as precise as possible

with the former. SendPOVEvent also happens in the KeyNavigator class while keys are

pressed or released to transfer to current point of view of the observer.

SendObjectStatusEvent is the same case as SendPOVEvent here to command the status

of the “Slave” applications whether to start or stop rotating. The positions and point of

view of the rotating cube are set during the corresponding drawing period in the

MasterGLRenderer. It means the current values are updated in real time.

In the server - “Slave” side, after receiving corresponding objects (ObjectEvent,

ObjectStatusEvent, POVEvent) from the client - “Master” side, the first thing is to

distinguish which object it has received. Operator instanceof performs as the judger here.

After the identity of the received object is known, the corresponding set function can be

utilized in order to set the instant status of the SlaveGLRenderer. For example, if object

of POVEvent is received and after being recognized, the corresponding instance of

POVEvent within SlaveGLRenderer is set. Therefore, during moveCamera() function

within SlaveGLRenderer, instant point of view transferred from “Master” is immediately

set. // within SlaveApplicationStarter, receive and specify the received // object, pass it to the glRenderer if (object1 instanceof POVEvent) {

POVEvent povEvent = (POVEvent) object1; glRenderer.setPOVPosition(povEvent); } // within SlaveGLRenderer, private member povEvent are set as what // has been received public void setPOVEvent(POVEvent povEvent) { this.povEvent= povEvent; }

// command “Slave” to change its point of view according to the // para eterprivate void moveCamera (GL gl)

m s bound within povEvent

{ gl.glRotated(povEvent.getLookupdown() , 1.0f, 0, 0); gl.glRotated(povEvent.getSceneroty() , 0, 1.0f, 0);

- 20 -

gl.glTranslatef(povEvent.getXPos(),povEvent.getYPos(),povEvent.getZPos());

}

Overall, objects containing parameters regarding the rotating cube are transferred through

sockets from the client - “Master” to the server - “Slave”. Parameters within serialized

objects are set immediately after they are generated. Then the objects are transferred

from “Master” to “Slave” and being recognized and passed to the renderer class within

which the parameters of the objects are released to their corresponding positions and

promote the movement of the “Slave”.

3.2.4 Multiple Screens Effect Realization by Adjusting Viewport

After establishing network communication between “Master” and “Slave” as well as the

transmission and set of the corresponding instant command issued from “Master” and

implemented by several “Slaves”, simultaneous and identical movement of “Slaves” are

realized from multiple screens (see Figure 15).

Figure 15: Simultaneous and Identical Movement of Two “Slave” Applications

- 21 -

In order to simulate the effect of only one cube within multiple screens, adjustment over

the “Slave” cubes are needed. Now, two problems regarding the adjustment are:

• Different sizes of cubes from different screens

• Different positions of cubes from different screens

a) The first problem “different sizes of cubes from different screens” is caused by the

differences of the resolution and sizes of screens. For resolution problem, if different

screens have the options of the same resolution, we can just easily set their resolution to

be the same. We implement our application in this way. If there is no uniform resolution

that every screen conforms to, ratios among different resolutions need to be calculated

and used as a factor to multiply the viewport parameters discussed below. For screen size

problem, a method to equalize cube sizes from different screens is to multiply the

viewport width and height with the corresponding ratios among screens.

In OpenGL, glViewport specifies the affine transformation of x and y from normalized

device coordinates to window coordinates. glViewport function is written as:

void glViewport ( int x, int y, int width, int height );

x, y specify the lower left corner of the viewport rectangle in pixels while width, height

specify the width and height of the viewport. Because the viewport size (width, height)

will determine the size of the cube displayed, if we multiply one “Slave” cube’s viewport

width and height with the ratio that expands it to the same size of a chosen standard

“Slave” screen, then the size of all “Slave” cubes become the same. For example, we

have “Slave” applications respectively displayed in screen ( ( ), ( ))A width A height A ,

( ( ), ( ))B width B height B , with the same resolution (we do not

consider the case of different resolutions). If we choose screen as the standard screen,

then ratios of

( ( ), (C width C height C))

A

B to are A ( )( )( )

width AwidthRatiowidth B

=BtoA and

( )( )AB

( )heightRatio BtoA =heightheight

, the glViewport specified in “Slave” application B

becomes:

- 22 -

void glViewport (x, y, ( ) ( )width B widthRatio BtoA× , ( ) ( )height B heightRatio BtoA× );

(see Figure 16).

Similarly, the viewport C has to be modified the same way.

B

Viewport of B Screen border of B

( ) ( )width B widthRatio BtoA× ( )width B

A

( )width A

Figure 16: Equalize cube size by adjusting glViewport

b) The second problem is to match the position of the already equalized cubes to simulate

the effect of only one cube is displayed and moved within multiple screens. The

adjustment of the position of cubes is also implemented within the glViewport. Due to

the consideration of conveniences, a class SliderPanel, extending JFrame and containing

two instance JSliders (to respectively adjust width and height value) is built to manage

the position of cubes, or more precisely, the viewport of the cubes. The slider.getValue()

is added to the position of the lower left corner of the viewport to move it by manually

adjusting the slider bar (see Figure 16).

c) Another problem that has not implicitly addressed is that the cube should be

independent of the change of the frame (the movement, resizing of the frame does not

influence the size and the position of the cube). This function’s realization would benefit

us from moving or resizing the frame at our will without damaging the multiple screens

effect. We achieve this by using the instantly origin of the frame with respect to the

screen and the origin of the viewport. For example, while the frame is moving down, the

distance decreased is added to the y value of the origin of the viewport, than the cube is

- 23 -

moving up with the same distance. This method will result the phenomenon that as if the

cube stays stable as before (see Figure 17).

Frame goes down

Cube stays stable

Viewport origin moves up

Viewport origin

Figure 17: Cube Size and Position are independent of Frame Adjustment

From the three steps described above:

I. “Master-Slave” Rotating Cube Extension

II. Network Communication Through Java Socket

III. Multiple Screens Effect Realization by Adjusting Viewport

The multiple screens effect based on network communication is achieved (see Figure 11).

By utilizing the techniques used in this cube example and adequate extensions, we further

implement the similar task based on World Wind Java SDK developed by NASA.

3.2.5 Synchronization of “Slave” Applications

Because of the differences of network transmission speed, different “Slaves” may act

unsynchronized after some time or under the condition of temporal network interruption.

If one “Slave” undergoes normal rotating while the other is experiencing network

interruption, then the phase of the former surpasses the latter. An idea to solve this

unsynchronization problem is to send serialized object (that is, sendObjectEvent()) for

every round of display() function. This is because the display() function keeps

- 24 -

circulating and displaying the rotating cube. If sendObjectEvent() is within the display()

function, then for every round of display(), the instant positions of the rotating cube is

sent through socket to different “Slaves”. While the display() goes to the next round, if at

least one of the former serialized object has not been consumed in the server - “Slave”

side, then the serialized object sent this time need to wait until the former serialized

object is accepted by all the “Slave” applications. In this way, if the delay of network

slows down one “Slave” application, all the other “Slave” applications are also dragged

behind but all of them are still synchronized. Adding sendPOVEvent() within display()

function also improve the synchronization effect.

3.3 World Wind Java SDK

World Wind is a free open source virtual globe developed by NASA and open source

community for use on personal computers [Wiki-NWW]. World Wind lets you zoom

from satellite altitude into any place on Earth [NAASA]. A preview of the World Wind

Java SDK was released in 2007 and the latest version 0.5.0 was released on May 2nd,

2008. The version that we are using is 0.4.1. With this SDK, we can embed World Wind

technology in our own applications – multiple screens.

3.3.1 Extension of Original World Wind Java SDK

Conceptually similar to what has been done for the rotating cube example, World Wind is

also extended respectively to client - “Master” and server - “Slave”. We, in most of the

cases, extend classes once necessary and put all the extended and newly generated classes

into one folder for the consideration of clearness and software reuse. We try not to

change the original code. But under special conditions, some slight changes of four

classes are made in our applications. They are:

• InputHandler: (functions added, for simplicity of coding) o void setSocket(Socket[] socket);

o void setPOVEvent(POVEvent povEvent);

• AWTInputHandler: (functions added, because of its implementation of interface

InputHandler) o public void setSocket(Socket[] socket){}

- 25 -

o public void setPOVEvent(POVEvent povEvent){}

• LatLon, Angle, Quaternion: (parameters need to be transferred through network,

for simplicity of coding)

o implements Serializable

• Configuration: (codes added, for the creation of new classes by class name)

o defaults.setProperty(MyAVKey.MASTER_WORLD_WINDOW_CLASS_NAME,

MasterWorldWindowGLAutoDrawable.class.getName()); o defaults.setProperty(MyAVKey.SLAVE_WORLD_WINDOW_CLASS_NAME,

SlaveWorldWindowGLAutoDrawable.class.getName()); o defaults.setProperty(MyAVKey.MASTER_INPUT_HANDLER_CLASS_NAM

E, MasterAWTInputHandler.class.getName()); o defaults.setProperty(MyAVKey.SLAVE_INPUT_HANDLER_CLASS_NAME,

SlaveAWTInputHandler.class.getName()); The above four illustrated are the original classes that has been modified within the whole

World Wind source code. Other than these four classes, the other original classes remain

the same. The extension of the original classes and their main differences are explained

below:

a) At the very top level, ApplicationStarter is replaced by MasterSimpleStarter and

SlaveSimpleStarter, both of which extends Thread. Then the SlaveSimpleStarter

can draw the earth at one thread, and listen and receive serialized objects at

another hand.

b) The original WorldWindowGLCanvas is replaced by

BasicWorldWindowGLCanvas which is further extended respectively by

MasterWorldWindowGLCanvas and SlaveWorldWindowGLCanvas.

BasicWorldWindowGLCanvas is seens as an abstract class. The main differences

between MasterWorldWindowGLCanvas and SlaveWorldWindowGLCanvas are

that the former uses the function setSocket (Socket[] socket) to further pass client

sockets to the associated class which has the controlling parameters, and the latter

uses the function setPOVEvent(POVEvent povEvent) to pass the received

povEvent from “Master” to the associated class which needs the controlling

parameters to act correspondingly as commanded. Both of them has the

- 26 -

setFrame(MyFrame frame) function to realized the effect of remaining stable

regardless of the moving and resizing of the frame (see Figure 17).

c) WorldWindowGLAutoDrawable, originally responsible for displaying the earth,

is extended to MasterWorldWindowGLAutoDrawable and

SlaveWorldWindowGLAutoDrawable. They perform differently mainly in

display(). The latter has the function setPOVEvent(POVEvent povEvent) to pass

the transferred object to the associated class. Besides, both

WorldWindowGLAutoDrawable and SlaveWorldWindowGLAutoDrawable

extends MyWorldWindowGLDrawable which extends

WorldWindowGLDrawable.

d) MasterAWTInputHandler implements InputHandler, KeyListener, MouseListener,

MouseMotionListener, MouseWheelListener, FocusListener to realize the mouse

and key event through its private member viewInputBroker.

SlaveAWTInputHandler does not need to implement these events since it only

carries out the command from the “Master”.

e) MasterOrbitViewInputBroker implements KeyListener, MouseListener,

MouseMotionListener, MouseWheelListener, FocusListener. This class is

associated with MasterAWTInputHandler and response to its corresponding call.

The functions in MasterOrbitViewInputBroker set the parameters related to the

control of either keyboard or mouse. From here, these parameters can be wrapped

to the serialized object povEvent and send through sockets to “Slaves”.

SlaveOrbitViewInputBroker is not necessary to implement the interface that

MasterOrbitViewInputBroker does.

f) Other extensions include MyAVKey extending AVKey which can be seen as an

index of the necessary class names of whose classes are called by name.

MyCGFrame simply extends JFrame.

g) Added classes include SendPOVEvent and POVEvent which are responsible for

sending controlling parameters through sockets. SliderPanel is used to manually

set the position of the viewport’s lower left corner in order to move the position of

the earth. DummySlave series are used to start SlaveSimpleStarter by passing the

name and port number into it.

- 27 -

- 28 -

h) Classes that are simply moved from the original source code are KeyPollTimer

and OrbitViewInputstateIterator without any change. The reason is that they are

both set as private classes (in the original source code) and are used during

implementation.

Detailed information regarding the hierarchy of “Master-Slave” realization can be

referred to the class diagram (see Figure 18).

3.3.2 Network Communication through Java Sockets

The concept of network communication through sockets is similar to that of the rotating

cube example (see 3.2.3). At the client - “Master” side, sockets are built at the

MasterSimpleStarter class and passes through MasterWorldWindowGLCanvas,

MasterAWTInputHandler till MasterOrbitViewInputBroker.

MasterOrbitViewInputBroker contains parameters that response to keyboard or mouse

event and are wrapped into povEvent. Through socket communication, povEvent is

transferred through network and received by the run() function implemented by

SlaveSimpleStarter which extends Thread class. SlaveSimpleStarter class then pass the

povEvent that it has received to SlaveWorldWindowGLCanvas, SlaveAWTInputHandler

till SlaveOrbitViewInputBroker. SlaveOrbitViewInputBroker unpacked the received

povEvent and set the parameters to their corresponding places, achieving the

synchronized drawing with the “Master”.

Sketched sequence diagram shows the process respectively from the “Master” side (see

Figure 19) and “Slave” side (see Figure 20). Connection between client - “Master” and

server - “Slave” is assumed already built.

Figure 18: Class Diagram of the realization of “Master-Slave” mechanism

- 29 -

Figure 19: Socket Passing in Client - “Master” Side

Figure 20: POVEvent Passing in Server - “Slave” Side

3.3.3 Multiple Screens Effect Realization by Adjusting Viewport

As is the same case with the rotating cube example, different sizes and positions of

“Slaves” are the main problems this step should solve. The realization method is quite

- 30 -

similar to that of the rotating cube example (see 3.2.4). Screen size ratios are calculated

and used to adjust the viewport size. SliderPanel class is created to adjust the position of

the lower left corner of viewports which equally means the changing of the earth inside

the viewport. glViewport exists in the class SlaveWorldWindowGLDrawable along with

the corresponding modification of its parameters. Besides, the problem of the size and

position of the earth is independent of the frame is also solved the same way that rotating

cube does. No matter how the frame is resized or moved, the earth inside it stays stable

(see Figure 17).

4. Performance Evaluation and Future Work This lab generally realized the effect of multiple screens by socket communication.

Experiments on two “Slaves” are made and described here. First, two screens are placed

side by side as two “Slaves”. A third PC is used as the “Master” to issue commands by

utilizing keyboard or mouse. By issuing commands from the “Master”, two “Slaves”

conform to the order they have received and act accordingly. For the rotating cube

example, the two “Slaves”, when positions are adjusted adequately, act synchronized as if

they were one. This is mainly because the simplicity of in the code which enables precise

synchronization under frequent commands. The rotating cube is enlarged within two

screens and can traverse through the border of them (see Figure 11). For World Wind,

“Slave” applications are synchronized if actions performed on “Master” are not frequent.

If high frequent manipulation is operated on “Master”, then unsynchronization possibly

occur. The reason might result from the complexity of the code, especially in the

iterative calculation part. Because in order to reach the result of smoothly change under

manipulation, a class OrbitViewInputStateIterator is used to calculate the smoothly

change iteratively given an initial input parameter. The possible error during the iteration

might lead to differences among “Slave” applications since the parameters they have

received from the “Master” is only the initial input. A possible way to solve the

unsynchronization problem under high frequency manipulation on “Master” might be to

sendPOVEvent() during every round of the display() function which ensures the same

step of all the “Slaves”. This possible improvement on synchronization has not been

implemented yet.

- 31 -

As addressed in the body of the report, the positions of the “Slave” earths are adjusted by

setting the lower left corner of the viewport. This leads to a problem of “blank” area

within the screen, which means if the lower left corner is moved upper right, then the area

that is lower left than the new origin will block the scene since that area does not belong

to the viewport. The way to solve this problem might be moving the origin even lower

left as needed to erase the “blank” area existed in the screen. The rule needs to be obey is

to keep the viewport of all the “Slave” applications are the same assuming that their

screen resolutions are all the same.

An unfinished task is to include layers of the earth into the multiple screens effect.

Because of the separation of the earth itself and the layers (added as an listener),

synchronization of layers and its earth has not been achieved. The layers cannot be

moved by mouse manipulation as it is in the original source code.

Possible improvement includes the change of the orientation of “Slave” screens. Now,

screens are placed side by side and forming a straight line. In some circumstances,

certain angles between adjacent screens are necessary because it is quite possible to

generate better virtual effects (see Figure 3). Physical locations and orientations of

screens need to be calculated and added into the source code in order to realize such

effect.

- 32 -

Figure 21: Two “Slave” Earths and One “Master” Earth Displayed

4. Summary The goal of this computer graphics lab is to realize multiple screens effects based on

World Wind Java SDK. Jogl is used as our development tool on which World Wind Java

SDK is based. Analysis on the source code, adequate extension of necessary classes,

network communication through sockets and viewport management are the mainstreams

during the lab process. In this report, we first present the related work done before in

order to obtain wider view in this field which includes Studierstube, VR Juggler and

Java3D. After that our work on multiple screens begins with the working environment,

including development tools (Java programming language, Eclipse IDE), libraries (Jogl,

GlueGen) and Platform (World Wind Java SDK), following by detailed descriptions

about one rotating cube example, seen as an introduction to the self-classified three steps

- “Master-Slave” Rotating Cube Extension, Network Communication through Java

- 33 -

- 34 -

Socket and Multiple Screens Effect Realization by Adjusting Viewport. Our final

application is based on World Wind Java SDK. Extensions on the source code is

wrapped for further software reusement. Corresponding Javadoc is provided as well.

References [BHFMB99] Butz, A., Höllerer, T., Feiner, S., MacIntyre, B., & Beshers, C., (1999).

Enveloping users and computers in a collaborative 3D augmented reality. Proceedings

International Workshop on Augmented Reality (IWAR’99), 35– 44.

[CCBE] Jun Cai, Seminar Report: Augmented Reality: the Studierstube Project,

http://w5.cs.uni-sb.de/~butz/teaching/ie-ss03/papers/Studierstube/

[JBBC] Christopher Just, Allen Bierbaum, Albert Baker, Carolina Cruz-Neira, VR

Juggler: A Framework for Virtual Reality Development, Iowa Center for Emerging

Manufacturing Technology, Iowa State University.

[SFSEGP02] Schmalstieg, D., Fuhrmann, A., Szalavari, G. H. Z., Encarnacao, L.M.,

Gervautz, M., Purgathofer, W., The Studierstube Augmented Reality Project. In

Massachusetts Institute of Technology, Vol. 11, No.1, 33-54, 2002.

[NAASA] http://worldwind.arc.nasa.gov/

[NWW] NASA World Wind: http://worldwind.arc.nasa.gov/java/index.html

[OpenGL] http://www.opengl.org/resources/bindings/

[Wiki-CAVE] http://en.wikipedia.org/wiki/Cave_Automatic_Virtual_Environment

[Wiki-Gluegen] http://en.wikipedia.org/wiki/Gluegen

[Wiki-Java3D] http://en.wikipedia.org/wiki/Java_3D

[Wiki-Jogl] http://en.wikipedia.org/wiki/Java_OpenGL

[Wiki-NWW] http://en.wikipedia.org/wiki/NASA_World_Wind