a virtual multimedia office - oracle · 4 virtual multimedia office concept > business...

A Virtual Multimedia Office

Jan van der MeerEricsson Telecommunicatie BVRijen, the Netherlands

2

Contents> Virtual Multimedia Office (VMO) concept> Demo of VMO> Architectural Overview of VMO> Mobile client> Wonderland enhancements> Next steps> Summary and Q&A

3

Simplified Network ArchitectureSimplified Network Architecture

4

Virtual Multimedia Office concept> Business Communication Suite> Brainstorm + Project Wonderland> Features needed:

Ericsson branded office or building Link to other collaboration prototypes Physical – virtual interaction Mobile access via laptop and mobile phone

User Stories> Remote

Access to VMO via laptop

> User ‘on the move’Access to VMO with mobile phone

> User @homeAccess to VMO with TV

Wonderland <> Virtual Multimedia Office

7

Virtual Multimedia OfficeImplemented Features> Mobile client (2D)> Improved video player (Motion JPEG)> Interactive Whiteboard> Incoming calls> Alert & Client control by Digital Photo Frame> Shared application audio> Hyperlink Cell

8

Demo and or video (part 1)> Ericsson dress-up

9

Contents> Virtual Multimedia Office (VMO) concept> Demo of VMO> Overview of VMO> Mobile client> Wonderland enhancements> Next steps> Summary and Q&A

10

Overall VMO architecture

WonderlandServer

Wonderlandclient

Softphone

JVoicebridge

PC

WonderlandJ2ME client

Mobile phone

AsteriskVoice Service(GSM)

MobileProxy

Server

ServerMasterClient

Project Wonderland (0.4)> Built on Darkstar platform> Open source toolkit for

creating 3D virtual worlds Aimed at business and educational useSupports high-fidelity audio, application sharing, telephone, video ..

> Cells….

12

Overall VMO architecture

WonderlandServer

Wonderlandclient

Softphone

PC

WonderlandJ2ME client

Mobile phone

Voice Service(GSM)

MobileProxy

Server

ServerMasterClient

JVoicebridge

Asterisk

VNC

13

JVoiceBridge> Software only audio mixer written in Java> Handling Voice over IP (VOIP) with SIP> Mixing, voice chat and sound detection> Stereo audio > Volume and channel (left/right) control> Individual voice mix

14

JVoice Bridge architectureControl

Request Handler

Conference Conference

Conference

Call

Call

Call

Call

Call

Call

Call Call

15

Asterisk> Open source telephony engine & tool kit> Very flexible> Switch (PBX) capability> Gateway> Media server

> Supports SIP and ISDN protocols

16

Application

Asterisk architecture

Application

Dialplan

Configuration

Configuration

ChannelChannel

Channel

Asterisk

PBX core

17

Virtual Network Computing (VNC)> Copy PC screen to other PC (or Wonderland)

18

Contents> Virtual Multimedia Office (VMO) concept> Demo of VMO> Architectural Overview of VMO> Mobile client> Wonderland enhancements> Next steps> Summary and Q&A

19

Demo and or video (part 2)> mobile client

20

VMO/Wonderland 2 mobile - architecture

Wonderlandclient

Softphone

PC

Mobile phone

Voice Service(GSM)

WL 0.4

WL 0.4SIP

SIP

WonderlandServer

JVoicebridge

Asterisk

Server

ServerMasterClient

WonderlandJ2ME client

MobileProxy

UBF

21

Mobile Proxy> Acts as ‘normal’ clients to server> Starts session when mobile accesses proxy> Converts ‘relevant’ CellSetup messages> Communication mobile <> proxy is UBF based> Universal Binary Format (UBF)

Parse efficient for machines Java, C++, Erlang, … drivers

22

Mobile Client – rendering (1)> Requirements:

Objects must be scalable (zoom in/out)

Objects must be rotatable Speed Simple image file parsing Assets same location other

extension (2D assets)

23

Mobile Client – rendering (2)> What went wrong:

SVG for everything SVG avatars & objects, bitmap (tiles) for worldmap

> What worked: Drawing methods javax.microedition.lcdui.graphics Figures: Arc, Line, Filled arc, Filled triangle

24

Mobile Client - screenshots

25

Mobile Client - UVG{'compound' # #

{'filledarc' #{'color' "#00ff00"}& {{0 0} 100 0 360}}&

{'triangle' #{'color' "#000000"}& {{-130 -130} {-130 130} {130 130}}}&

{'triangle' #{'color' "#000000"}& {{-130 -130} {130 -130} {130 130}}}&

}$

26

UVG editor

27

Mobile Client - Operation> Simple provisioning (SMS: xxxxx M JohnDoe)> Configuration (user specific JAD)> SMS returned with download link> At connection, Proxy logs into server + phone nr.> JVoicebridge sets up voice call (via Asterisk)

> User uses left-right to rotate avatar, up/downtranslates to forward/backward

> When near ‘sensitive’ cells, outline appears and menu commands (to operate on cell)

28

Contents> Virtual Multimedia Office (VMO) concept> Demo of VMO> Architectural Overview of VMO> Mobile client> Wonderland enhancements> Next steps> Summary and Q&A

29

Demo and or video (part 3)> video player> interactive whiteboard> hyperlink cell

30

Wonderland Enhancements> Incoming calls> Interactive Whiteboard> VideoPlayer> Alert Cell> Hyperlink Cell> Voice from the shared applications (browser..)

31

Incoming Calls> Original Function

Calls towards ‘speaker phones’ are possible Issue: Calls to logged-in users required

> New/Changed Adapted IncomingCallHandler class How: - at incoming call, check user logged-in

- present call at HUD

32

Incoming Call - screenshots

33

Incoming calls - architecture

Asterisk

PBX

JVoicebridge

WonderlandServer

IncomingCallHandlervoicelib

SMC

34

Interactive Whiteboard> Requirement

Interworking with remote and local teams Create ‘remote’ presence

> Approach Interactive Whiteboard controlled by Windows PC Using VNC to copy PC screen into virtual world Camera to show real-whiteboard-users Laptop showing virtual participants Speakerphones (real/virtual) for sound

35

Interactive Whiteboard

36

Interactive Whiteboard setup

37

Interactive Whiteboard setup (2)

38

Interactive Whiteboard setup (3)

Mario

Real

Virtual

39

Video Player > Original Player

Plays via buffer => texture Issue: Java Media Framework based

& problems with VNC

> New/Changed M-JPEG video library Why: M-JPEG = stream of pictures

Camera support (e.g Axis 206)

40

VideoPlayer - architecture

JMFBuffer

MPEG

Original

JPEG decoding

M-JPEG

Adapted

41

Wonderland Enhancements> Incoming calls> Interactive Whiteboard> VideoPlayer> Alert Cell> Hyperlink Cell> Voice from the shared applications (browser..)

Trigger external node (Chumby)> Requirement

Unobtrusive user interface to VMO screen Issue: click on object => “real world” reaction

> New/Changed Generate HTTP-request when object clicked How: - created Alert Cell

- HTTP address in XML config

43

Trigger external node - screenshot

44

Chumby control> Requirement

Unobtrusive user interface to VMO screen Issue: control client without keyboard

> New/Changed Modified WL client to accept Chumby control Adapted sunspot interface in WL client

Chumby setup

TiltSensors

SUN-SpotInterface

VMO/WL client

HTTP-proxyWebServerWeb

Server

Chumby Laptop

VMO/WL server

Wonderlandclient

Tunnel

46

Demo and or video (part 4)> chumby

47

Hyperlink Cell> Requirement

Fill in private data in browser form Issue: click on object => start pc browser with URL

> New/Changed Start browser when object clicked How: - created Hyperlink Cell

- client side invokes application

48

hyperlink Cell

VMO/WL client

Laptop

VMO/WL serverApplication

Hyper-linkCell

Hyper-linkCellCell setup URL-info

49

Audio for external applications> Requirement

Audio (not only visual) for shared external apps Issue: pick up sound, mix into world

> New/Changed Introduced new audio treatment in JVoiceBridge How: - capability to handle PulseAudio added

50

Audio for external applications

Audio from movie

Is ‘played’ viaAudio cell

51

Audio for external applications

Wonderland Server

JVoicebridge

AudioCell

TreatmentManager

PulseAudio

RecordStreamparec

Treatment“pulseaudio”

• When starting the shared Application, route audio output to pulseaudio library

• Example: export PULSE_SINK=.. firefox

52

Recommendations> Correct .xml configuration files> Stop (old) java processes> Disable firewall on (company) laptop

> Original JVoicebridge (in WL 0.4 package) had problems.We updated to a svn version from Dec 8, 2008

53

Contents> Virtual Multimedia Office (VMO) concept> Demo of VMO> Architectural Overview of VMO> Mobile client> Wonderland enhancements> Next steps> Summary and Q&A

54

Next steps> Serious research (Future Workspaces)> Trials / test with current release> Move to Wonderland 0.5> Extensions to improve experience

& shared working

> NOTE: Wonderland BOF sessionThis room; 6:45pm

55

VMO Development team> Project Management Marc Verschueren /

Gert-Jan Rijckenberg> Asterisk & presence Mario Goorden> VMO configuration Florin van Slingerland> Wonderland extensions Erik Reitsma> Mobile client/proxy Erik Reitsma> VMO video Sin Yuk Yan / Mieke Verheijen

56

Links> Wonderland lg3d-wonderland.dev.java.net> Darkstar www.projectdarkstar.com> JVoice Bridge jvoicebridge.dev.java.net> Asterisk www.asterisk.org> PulseAudio www.pulseaudio.org> VNC www.realvnc.com> UBF www.sics.se/~joe/ubf/site/home.html> Chumby www.chumby.com> Future Workspaces www.futureworkspaces.nl

57

Jan van der Meerjan.van.der.meer(#)ericsson.com

Ericsson Telecommunicatie BVRijen, the Netherlands