application and desktop sharing - columbia universityboyaci/appshare/papers/ism08/ads_paper.pdf ·...

13
Application and Desktop Sharing Omer Boyaci and Henning Schulzrinne Department of Computer Science Columbia University {boyaci,hgs}@cs.columbia.edu November 28, 2007 Abstract Application and desktop sharing allows sharing any application with one or more people over the In- ternet. The participants receive the screen-view of the shared application from the server. Their mouse and keyboard events are delivered and regenerated at the server. Application and desktop sharing enables collaborative work, software tutoring and e-learning over the Internet. We have developed an applica- tion and desktop sharing platform called ADS which is efficient, reliable, operating system independent, scales well, supports all applications and features true application sharing. With ADS the users can share any application from their desktop and the par- ticipants do not need to have the application installed on their systems. 1 1 Introduction Application and desktop sharing allows two or more people to collaboratively work on a single document, drawing or project in real-time. They do not need to email the file back and forth for modifications and corrections. Similarly instructors can give a lecture 1 This research was supported by FirstHand Technologies. over the Internet using ADS. Students or instruc- tor can record the session and this recording can be played back locally or from a streaming server to multiple users. Application sharing can be enriched with audio and video for online lectures. Moreover ADS can be used for software tutoring where an in- structor teaches a particular software like photoshop, dreamweaver, office suites, autocad and visual stu- dio. Students can see what the instructor does and they can participate to the sharing session by re- motely using mouse and keyboard. Instructors can request from students to repeat their actions after showing a feature. Section 2 gives some background information on different application sharing models. Section 3 dis- cusses the related work and why they are inade- quate. System architecture is discussed in Section 4, whereas the details of the ADS protocol is dis- cusses in Section 5. Client and server architectures are explained in Sections 6 and 7 respectively. Sec- tion 8 gives information about the performance of the system in terms of bandwidth and processing power requirements. Section 9 talks about future work and finally Section 10 summarizes our work. 1

Upload: others

Post on 12-Aug-2020

1 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Application and Desktop Sharing - Columbia Universityboyaci/appshare/papers/ism08/ads_paper.pdf · known technique for capturing screen update events and will be discussed in detail

Application and Desktop Sharing

Omer Boyaci and Henning SchulzrinneDepartment of Computer Science

Columbia University{boyaci,hgs}@cs.columbia.edu

November 28, 2007

Abstract

Application and desktop sharing allows sharing anyapplication with one or more people over the In-ternet. The participants receive the screen-view ofthe shared application from the server. Their mouseand keyboard events are delivered and regenerated atthe server. Application and desktop sharing enablescollaborative work, software tutoring and e-learningover the Internet. We have developed an applica-tion and desktop sharing platform called ADS whichis efficient, reliable, operating system independent,scales well, supports all applications and featurestrue application sharing. With ADS the users canshare any application from their desktop and the par-ticipants do not need to have the application installedon their systems. 1

1 Introduction

Application and desktop sharing allows two or morepeople to collaboratively work on a single document,drawing or project in real-time. They do not need toemail the file back and forth for modifications andcorrections. Similarly instructors can give a lecture

1This research was supported by FirstHand Technologies.

over the Internet using ADS. Students or instruc-tor can record the session and this recording can beplayed back locally or from a streaming server tomultiple users. Application sharing can be enrichedwith audio and video for online lectures. MoreoverADS can be used for software tutoring where an in-structor teaches a particular software like photoshop,dreamweaver, office suites, autocad and visual stu-dio. Students can see what the instructor does andthey can participate to the sharing session by re-motely using mouse and keyboard. Instructors canrequest from students to repeat their actions aftershowing a feature.

Section 2 gives some background information ondifferent application sharing models. Section 3 dis-cusses the related work and why they are inade-quate. System architecture is discussed in Section4, whereas the details of the ADS protocol is dis-cusses in Section 5. Client and server architecturesare explained in Sections 6 and 7 respectively. Sec-tion 8 gives information about the performance of thesystem in terms of bandwidth and processing powerrequirements. Section 9 talks about future work andfinally Section 10 summarizes our work.

1

Page 2: Application and Desktop Sharing - Columbia Universityboyaci/appshare/papers/ism08/ads_paper.pdf · known technique for capturing screen update events and will be discussed in detail

2 Background

Application and desktop sharing (ADS) allows shar-ing an application with remote users. All participantssee the same screen-view and use the same applica-tion. There is only one copy of the shared applicationand it runs on the server. The main challenges of ap-plication and desktop sharing are scalability, reliabil-ity, true application sharing, operating system inde-pendence and performance. We believe that an appli-cation and desktop sharing system should be operat-ing system independent because participants can usedifferent operating systems. Also, the system shouldscale well because some sharing scenarios such as e-learning and software tutoring may consist of severalsimultaneous participants. Systems supporting mul-ticasting scale well but participants will end up withcorrupted screen-views if the system is not designedreliability in mind. The sharing system should be ef-ficient in the sense that it should transmit only thechanged parts of the screen and it should not con-sume all the resources while doing this.

Application sharing differs from desktop sharingin which there is only one shared application for pri-vacy and security. The key challenge is that someother application can sit on top of the shared appli-cation and the shared application can open new childwindows like options or fonts. A true applicationsharing system should blank all the other applica-tions if they are on top of the shared one and shouldtransfer all the child windows of the shared applica-tion. For example, if the user wants to share only theWindows Internet Explorer application which hasthe title “Windows Live Hotmail - Windows InternetExplorer” from the desktop seen in (Figure ??) thenthe participants should only see the main and the “In-ternet Options” windows. ADS (Figure ??) displaysonly these two windows with a correct size whileblocking the desktop background and the non-sharedapplication. Whereas another application sharing

client Mast, which will be discussed later, could notdisplay the shared application in correct size and alsocould not block the non-shared application and desk-top background (Figure ??).

There are two models for application sharing:application-specific and generic. The application-specific model requires the developers to add thisfeature to their applications; for example, the latestversion of Microsoft Office has this feature. Also,in order to participate a sharing session, all partici-pants must have a copy of the shared application. Inthe generic model, the application can be anythingsuch as word processor, browser, CAD/CAM, Powerpoint or movie editor. Also, the participants neednot have the application installed on their systems.The only disadvantage of generic application shar-ing is that its generic nature makes it a bit inefficientas compared to the application-specific model in cer-tain scenarios. We have developed ADS based on thegeneric model therefore, users can share any appli-cation without requiring the participants to have theapplication.

3 Related Work

As we have mentioned in the previous section, ap-plication and desktop sharing is a very useful fea-ture therefore, operating system providers, commer-cial companies and several open source teams try tocome up with a solution. Microsoft has WindowsMeeting Space for Windows Vista and Netmeetingfor Windows XP and older versions. Netmeeting wasreleased in 1999 for Windows 98 and in our tests itfails to display pop-ups and menus. Windows Vistabrings application sharing feature with the WindowsMeeting Space but all the attendees must use Win-dows Vista. Apple introduced desktop sharing (noapplication sharing) with Mac OS X Leopard butagain both parties should use Mac OS. These two

2

Page 3: Application and Desktop Sharing - Columbia Universityboyaci/appshare/papers/ism08/ads_paper.pdf · known technique for capturing screen update events and will be discussed in detail

Figure 1: A screenshot of a desktop with overlapped windows

Figure 2: Mast client view

3

Page 4: Application and Desktop Sharing - Columbia Universityboyaci/appshare/papers/ism08/ads_paper.pdf · known technique for capturing screen update events and will be discussed in detail

Figure 3: ADS client view

Figure 4: UltraVNC client view

solutions are operating system dependent. VNC [?]is a cross-platform open source desktop sharing sys-tem but it does not have application sharing featuretoo. UltraVNC [?] claims to support the applicationsharing feature but it has failed in our tests due to fol-lowing problems: the cursor position did not match,there is no privacy, new windows belonging to sameapplication are not included and long menus are notshown properly.

Table 1: Comparison of Related WorkOS S AS RC R

VNC + - - + +Windows Meeting Space - - + + -Leopard Screen Sharing - - - + -TeleTeachingTool + + - - +Mast + + - + -ADS + + + + +

LegendOS Is it operating system independent?S Is it scalable?AS Does it support application sharing?RC Does it support remote control?R Does it support recording?

VNC, Netmeeting, Windows Meeting Space andLeopard Screen Sharing all rely on unicast only so,they do not scale well. Sharing an application viaunicast increases the bandwidth usage linearly withthe number of participants. For example, Microsoftsuggests Windows Meeting Space for a group of 10users or less. TeleTeachingTool [?] and MulticastApplication Sharing Tool (MAST) [?] use multicas-ting to overcome this limitation. TeleTeachingTooladds multicast support to VNC servers however, it isdeveloped just for online teaching so it does not al-low participants to control the shared desktop. Also,it does not support true application sharing due toits underlying VNC system. MAST allows remote

4

Page 5: Application and Desktop Sharing - Columbia Universityboyaci/appshare/papers/ism08/ads_paper.pdf · known technique for capturing screen update events and will be discussed in detail

users to participate via their keyboard and mouse butits screen capture model is based on polling which isvery primitive and not comparable to current state ofart the capturing methods like mirror drivers whichwill be discussed in Section 3.1. Also MAST doesnot support true application sharing due to problemswhich we have mentioned in the previous section(Figure ??). Although both TeleTeachingTool andMAST use multicasting for scalability, they do notaddress the unreliable nature of UDP transmissions.UDP does not guarantee delivery of packets and ifdelivered packets can be out of order. In order tocompensate for packet loss, the TeleTeachingTooland MAST periodically transmit the whole screenwhich increases the bandwidth and CPU usage. Ta-ble ?? compares the sharing systems discussed so far.

4 System Architecture

This proposed system is based on a client-server ar-chitecture (Figure ??). The server is the machinewhich runs the shared application. The participantsuse a lightweight Java client application for connect-ing to server and they do not need the shared applica-tion. Clients receive screen updates from the serverand send keyboard and mouse events to the server.

We have developed a Java client which works inalmost every operating system. The server could notbe written with Java due to lack of Java’s low levelsupport. Therefore, there should be a server for eachoperating system and we have developed a WindowsXP server and mirror driver. Mirror driver is the bestknown technique for capturing screen update eventsand will be discussed in detail at Section 7.1. Win-dows XP calls the same drawing functions on bothreal graphic driver and mirror driver. This mecha-nism allows the server to learn the updated screenregions without polling. Therefore, the overhead ofapplication sharing is minimum on the server. With-

Figure 5: System Architecture of ADS

out the mirror driver sharing server should poll thescreen state in order to detect the changed regions.The mirror driver runs in kernel mode and notifiesthe user mode server when it detects changes in theGUI of the shared application. The server then pre-pares a packet which consists of a RTP [?] headerand a PNG [?] or JPEG [?] image of the updatedregion. RTP allows the clients to re-order the pack-ets, recognize missing packets and synchronize ap-plication sharing with other media types like audioand video. The screen updates are distributed asPNG or JPEG images according to their character-istics. PNG is an open image format which uses alossless compression algorithm zlib and suitable forcomputer generated images. JPEG is lossy but suit-able for photographic images likes photos or movies.

The server supports both multicast and unicasttransmissions. For unicast connections, either UDPor TCP can be used. TCP provides reliable commu-nication and flow control therefore, it is more suit-able for unicast sessions. TCP clients may have dif-ferent bandwidths so we have developed an algo-rithm which sends the updates at the link speed ofeach client. For UDP clients, the server controlsthe transmission rate because UDP does not provide

5

Page 6: Application and Desktop Sharing - Columbia Universityboyaci/appshare/papers/ism08/ads_paper.pdf · known technique for capturing screen update events and will be discussed in detail

flow and congestion control. Several simultaneousmulticast sessions with different transmission ratescan be created at the server. ADS has a NACKbased retransmission mechanism for UDP clients.The UDP based clients request the missing packetsfrom the server while storing the received out-of-order packets in a receive buffer. Multicast clientslisten the NACK messages from other clients in or-der not to send multiple NACK requests for the samelost packet. Briefly, the server can share an applica-tion to TCP clients, UDP clients and to several mul-ticast addresses in the same sharing session.

Late-joiners should be handled carefully other-wise they can degrade the systems performance.ADS generates a full-screen update for the late-joiners. But if more than one participant join latelywithin a minute, ADS generates only one full-screenupdate for the first one and starts all the others fromthis full-screen update. They will receive the full-screen update first and then all the other partial up-dates up to current status of the screen.

Although multiple users could receive the screenupdates simultaneously, clearly only one of them canmanipulate the application via keyboard and mouseevents. ADS uses the Binary Floor Control Pro-tocol (BFCP) [?] to restrict the control of the ap-plication to a single user. BFCP receives floor re-quest and floor release messages from clients andgrants the floor to the appropriate client for a pe-riod of time while keeping the requests from otherclients in a FIFO queue. All BFCP messages, key-board and mouse events are transmitted directly tothe server using TCP. For mouse and keyboard eventsJava’s own key-codes are used because these eventsare captured from a Java client and regenerated by aJava component at the server.

We have also added a recording feature to theADS. Participants can record the sharing session to afile. This file can be used to watch the session locallyor to stream it to multiple receivers simultaneously.

This feature is very useful for preparing lectures orsoftware tutorials for future use.

5 ADS Protocol

ADS Clients and servers communicates using theADS protocol which defines the packet structuresand message types. The protocol defines five mes-sages from server to client and two messages in thereverse direction (Table ??). Each new client needsone “open new window” message for each sharedwindow. For TCP clients server transmits this mes-sage right after the connection establishment. UDPbased clients send a “full intra-frame request” to theserver when they join the session. Receiving this FIRmessage server first transmits “open new window”messages and then the whole screen image via “re-gion update” messages.

Table 2: ADS Message TypesServer MessagesOpen New WindowClose WindowWindow Size UpdateRegion UpdateMove Rectangle

Client MessagesFull Intra-frame Request (FIR)NACK request

Table 3: ADS Message StructureRTP Header

Message TypeWindow ID

Type Specific Payload

Each protocol message consists of a RTP header,

6

Page 7: Application and Desktop Sharing - Columbia Universityboyaci/appshare/papers/ism08/ads_paper.pdf · known technique for capturing screen update events and will be discussed in detail

message type identifier, window identifier and mes-sage specific payload (Table ??). Message specificpayload structure and length are determined by themessage type. “Close window” message does nothave a message specific payload whereas “windowsize update” message carries new size information inthis part. These messages are carried on UDP pack-ets for UDP clients. Neither TCP nor RTP declaresthe length of the RTP packet therefore, RTP fram-ing [?] is used to split RTP packets for TCP clients.Server and clients should ignore messages with anunknown message type. New message types can beeasily added to the protocol.

6 Client Architecture

The ADS client is very simple compare to the server,receiving screen updates from the server and display-ing them to the user. It is completely stateless inthe sense that it can disconnect and reconnect to theserver. Due to its simplicity, client for different plat-forms can be developed easily. A Java ADS clienthas been developed in our lab and can run on anyJava supported system.

ADS clients can be passive or active. In activemode, they connect to ADS servers whereas in pas-sive mode they wait for incoming connections. Thepassive mode can be used as a RGB cable replace-ment for presentations. Basically, the client becomesa network projector which waits for incoming con-nections. Presenters can connect to this client andshare their applications. A computer can be dedi-cated as a network projector and only this machineshould physically attached to a real projector. Thepresenters do not need to attach their laptops withan RGB cable to a projector. They just stream theirapplication view to the network projector.

All participants of a sharing session can be view-only or they can request the control from the server.

In order to request the control from server, usershould press the “control” button in the GUI of theclient application (Figure ??). The client will senda floor request message and then the server willreply with a granted or request queued message.The server grants the floor immediately if nobodyelse currently controls the floor. Otherwise, the re-quest will be queued in a FIFO queue and floor willbe granted to the requesters one-by-one automati-cally. Users can release the floor whenever they wantby pressing the “control” button again. After theserver grants the floor the client captures all key-board and mouse events locally and transmits themto the server.

7 Windows Server Architecture

Windows server allows Windows XP users to sharean application with other participants. Windowsserver has two main compenents: kernel-mode mir-ror driver and user-mode sharing server. These twocomponents works together and communicates viashared memory. Basically, the mirror driver tracksthe updated regions of the screen and notifies theuser-mode sharing server about these updates. Theuser-mode sharing server application learns the up-dated regions, prepare region updates and transmitthese updates to the participants(Figure ??). Thesharing server also receives mouse and keyboardevents from participants and executes these requests.These two components are examined in detail in thefollowing sections.

7.1 Mirror Driver

Mirror driver is a display driver that mirrors thedrawing operations of a physical display driver. Itis the most efficient way to learn screen updates be-cause it learns the exact coordinates of the screen up-

7

Page 8: Application and Desktop Sharing - Columbia Universityboyaci/appshare/papers/ism08/ads_paper.pdf · known technique for capturing screen update events and will be discussed in detail

dates automatically from the operating system. Wehad to develop our own mirror driver for the sharingserver because there is no standard, free and openmirror driver. Remote Desktop Connection and VNCalso use their own mirror drivers to efficiently learnthe screen updates. Stability and correctness is veryimportant for kernel-mode components because theycan easily cause restart or blue screen of death. Ourmirror driver is completely stable such that we ob-serve no crashes at all in the last two years.

Figure 6: Shared memory regions

We have used the shared memory approach toestablish a communication channel between mirrordriver and the sharing server. Both mirror driver andthe sharing server maps the same region of the mem-ory to their own address spaces. There are four dif-ferent regions in this shared memory (Figure ??).

• Frame Buffer• Ring buffer for update records• Update records current index• Coordinates of tracking region

Windows calls the physical and mirror displaydrivers with the same GDI [?] (graphics device in-terface) commands. Display devices need a framebuffer to keep the screen state. Some display deviceshave dedicated frame buffers on the physical deviceitself. Mirror drivers are software only drivers so,they do not have dedicated hardware for graphic ac-celeration and frame buffer. Therefore we have usedsome portion the memory as a frame buffer. Thememory requirement of the frame buffer is resolu-tions times bytes per pixel. In our implementationwe have used 24 bits per pixel therefore frame bufferuses almost four megabytes for a 1280x1024 sharedregion.

The second region is a ring buffer which is used bythe mirror driver to insert update commands and co-ordinates. There are two types of commands BitBltand Moverect. Windows notifies the mirror driverfor an update and then mirror driver inserts this up-date to the ring buffer with command type and thecoordinates of the region.

The third region of the shared memory holds apointer to the last inserted update. Sharing server pe-riodically checks this pointer in order to see whetherthere is a new update or not.

The last region stores the tracking region and isupdated by the sharing server. The sharing servercomputes the bounding rectangle of the shared appli-cation windows and informs the mirror driver aboutthe tracking region. The mirror driver only tracksthis specific region instead of the whole desktop.Among these four regions the first three are main-tained by the mirror driver and the tracking region isupdated by the sharing server.

7.2 Server Architecture

The Windows XP sharing server is a user-mode pro-cess which complements the mirror driver. Mirrordriver keeps track of the frame buffer and the list

8

Page 9: Application and Desktop Sharing - Columbia Universityboyaci/appshare/papers/ism08/ads_paper.pdf · known technique for capturing screen update events and will be discussed in detail

of updated regions. The sharing server does rest ofthe duties including connection establishment, up-date optimization and compression and transmissionand keyboard and mouse events handling(Figure ??).The multi-threaded sharing server can serve multipleclients simultaneously. The server can wait for in-coming connections and also it can connect to clientsdirectly if instructed by the user. The sharing serverhas designed considering the following challenges:

Figure 7: Windows XP server architecture

• Participants may have different bandwidths.• Some participants may join lately.• The effects of packet losses• Reliable multicasting• Some regions require different encoding

7.2.1 The main algorithm of the server

The sharing server has one main thread, one man-ager thread and a number of client threads. Themain thread periodically checks the updated regions

and prepares compressed images of these regions.These images are inserted to the image ring bufferwhich stores them until they are transmitted by clientthreads. The algorithm of the main thread is givenbelow(Algorithm ??).

while Sharing session continues doif There are new updates then

Find shared process windows;Find the overlapping windows of theunshared processes;Combine the overlapping or repeatingupdates into one update;Clip the regions considering overlappingunshared windows;Prepare a compressed image for eachupdate region;sleep x ms (determined by managerthread) before starting again;

endelse

sleep 5 ms and then check againend

endAlgorithm 1: The algorithm of the main thread

7.2.2 Serving to different bandwidth users

The most CPU intensive part of the server is the im-age compression which may take up to 500 ms de-pending on the size of the updated region. Therefore,the main thread tries to optimize the updated regionsby combining overlapping regions and ignoring re-peated regions. There is no point in generating lotsof updates if the clients do not have enough band-width. The manager thread observes each clientsbandwidths and throttles the main thread accordingto the highest bandwidth client. This technique canbe easily explained with a movie playing example.Assume that the user shares a movie with the re-mote participants. The same region of the screen

9

Page 10: Application and Desktop Sharing - Columbia Universityboyaci/appshare/papers/ism08/ads_paper.pdf · known technique for capturing screen update events and will be discussed in detail

is updated 24-30 times per second depending on themovie type. Initially the server tries to generate asmany updates as possible. If at least one of the clientshas enough bandwidth to receive these updates, man-ager thread allows the main thread to continue withthis pace. But if none of the clients can have enoughbandwidth to receive these many updates then man-ager thread slows the main thread by forcing it tosleep between update image generations. Therefore,the main thread will generate a single update by com-bining several updates into one. The manager threadtries to equalize the update generation rate to thefastest client’s bandwidth speed. This technique pre-vents unnecessary CPU usage in the sharing server.The effective bandwidths of clients may change dur-ing the sharing session and this is properly handledby the manager thread. Briefly, the main thread com-bines several updates into one update in order to de-crease the bandwidth requirement. Similar techniqueis utilized by the low bandwidth client threads. Thefastest one transmits the generated updates in order,however other clients may not transmit all the gener-ated updates due to their low bandwidths. These lowbandwidth clients use an algorithm which skip someof the updates if they are obsoleted by the newer up-dates. Returning to video player example, the fastestmay transmit 12 frames per second whereas the oth-ers may transmit some of these generated frames per-mitted by their bandwidths.

7.2.3 Encodings

The updates are distributed as PNG images exceptfor movies. PNG is very suitable and efficient forcomputer generated images. However, it’s losslessnature results large increase in compressed imagesize for photographic images with negligible gainin quality compare to JPEG which is specificallydesigned for photographic image data. Therefore,using JPEG for photographic images and PNG for

the others is the best solution. But the server doesnot know whether an updated region contains pho-tographic or computer generated content. Becausemirror driver runs at the frame buffer level and atthis level there is only bits and no metadata. Fortu-nately detecting movie playing is very easy due toits special characteristic. Different from other ap-plications, movies generate 24-30 updates per sec-ond in a specific region of the screen. Benefittingfrom this characteristic we have developed an algo-rithm to detect movie playing in order to use JPEGencoding for this region. Consecutive updates to aspecific region trigger the detection. Detection al-gorithm uses the JPEG encoding on the region andcompares the compressed image size between JPEGand PNG. If JPEG size is less than quarter of thePNG size, the server switches the default algorithmfor this region to JPEG and stores this result in alookup table for subsequent updates. If the com-pression ratio of JPEG regions drops below 12:1,the server deletes the stored encoding informationfor this region from the lookup table. Similarly re-gions which are marked as PNG regions periodicallyrechecked. Periodic rechecking allows the server toadapt the dynamic nature of the content. For ex-ample, we have played a movie which starts witha black screen and then displays some texts on thisbackground and then continues with the movie itself.In this specific case, the server first used PNG andthen automatically switched to JPEG after the actualmovie starts.

7.2.4 Late joiners

Participants of a sharing session can join anytime andthey need the full screen buffer before receiving par-tial updates. Therefore, the sharing server preparesand transmits the image of the whole shared regionfor late joiners [?]. Preparing the image of the wholeshared session is costly in terms of CPU so, the shar-

10

Page 11: Application and Desktop Sharing - Columbia Universityboyaci/appshare/papers/ism08/ads_paper.pdf · known technique for capturing screen update events and will be discussed in detail

ing server stores the generated image for some time.When a participant joins, the server sends the storedimage if available and then transmits all the subse-quent partial updates. The stored image obsoletedafter some period. This also protects the server frommalicious or misbehaving clients.

7.2.5 Minimizing the effect of packet loss forUDP clients

UDP is not a reliable transmission mechanism there-fore, packets can get lost. Region updates may re-quire several kilobytes or even megabytes and un-less designed carefully a single packet loss may de-stroy the region update completely because regionupdates are compressed and they cannot be decom-pressed without complete data. In order to minimizethe effect of packet loss we have developed an al-gorithm which generates several small PNG imagesfor a given update region. Blindly generating a PNGimage for each scan line may increase the bandwidthusage because a new zlib compressor object shouldbe created for each new PNG. And creating a newzlib compressor decreases the compression ratio so,the bandwidth usage increases. The developed algo-rithm tries to maximize the number of scan lines in-cluded in a single screen update while trying to keepthe generated image size below 1500 bytes which themaximum MTU for ethernet. Due to its adaptive na-ture it may feed tens of lines for a text documentwhereas it may feed only a single line for a pho-tographic image. We have observed approximatelytwenty percent increase in bandwidth due to smallPNGs.

Transmitting self-contained UDP packets mini-mize the effect of packet loss. Instead of losing thecomplete region update, participants may lose only afew scan lines in case of a packet loss. They may endup with imperfect frame buffer due to packet lossesbut they may continue to participate to the session.

Figure 8: Minimizing the effect of packet loss

Figure ?? demonstrates the effect of a single packetloss where the image is displayed with a few miss-ing scan lines. The retransmission mechanism whichwill be discussed in the next session helps to restorethe frame buffer state.

7.2.6 Reliable multicast

In the previous section, we described our algorithmwhich minimizes the effect of packet loss. In thissection, we explain another supplementary techniquewhich retransmit missing packets. When a packetloss occurs, the participant views the rest of theimage except missing scan lines(Figure ??). Theclient application sends a negative acknowledgement(NACK) for this missing packet [?]. Receiving thisNACK, server retransmits the requested packet. Wehave modified the oRTP library which is used in theserver side and extended our own Java RTP library inclient side to support this feature. The server and theclients do not deal with retransmission and bufferingof packets, these are handled by the RTP libraries.

11

Page 12: Application and Desktop Sharing - Columbia Universityboyaci/appshare/papers/ism08/ads_paper.pdf · known technique for capturing screen update events and will be discussed in detail

The retransmission mechanism has developedconsidering the two issues: malicious or corruptedclient behavior and NACK storms. In order to pro-tect itself from malicious clients, the server does notretransmit a packet more than three times. NACKstorms happen when a packet could not reach severalmulticast clients and they all send NACK requests toserver. To solve this problem, clients do not sendNACK requests immediately after detection insteadthey wait for a random amount of time (between 0-100ms). During the waiting period if the client ob-serves a NACK request from another client in themulticast session, it suppresses its NACK request. Itis possible that more than one NACK have transmit-ted due to very close timeouts but this is better thanhaving hundreds of NACKs on the network.

7.2.7 Better movie support (Experimental)

As mentioned before we have developed an algo-rithm to detect movie playing. Server starts usingJPEG encoding for the detected region which in-creases frame per second and also decreases band-width usage dramatically without any distinguish-able loss by eye from the original. This techniquecan be improved by bypassing the rendering to framebuffer.

Figure ?? demonstrates the necessary steps foreach of these techniques. In the regular approachuser plays the movie in the server with any movieplayer and the frames are captured from frame bufferand compressed into JPEG images. In the transcodedcase, the user selects the movie he want to share withparticipants and the frames are directly read from in-put file and transcoded on-the-fly into JPEG imagesin real-time. These JPEG images are transmitted intoparticipants according to their bandwidths such thatlow bandwidth users get less fps compare to highbandwidth users. For on-the-fly transcoding we haveused FFMPEG-Java which is a sub-project of Free-

Figure 9: Better movie support

dom for Media in Java, FMJ in short. FFMPEG-Javais a Java wrapper around FFMPEG, using JNA. Thearchitecture of the implemented system is given inFigure ??. The results with this technique is betterthan the regular one due to low overhead on the sys-tem. The detailed performance results can be foundin the following section.

8 Performance Results

TBD later...

9 Future Work

TBD later...

12

Page 13: Application and Desktop Sharing - Columbia Universityboyaci/appshare/papers/ism08/ads_paper.pdf · known technique for capturing screen update events and will be discussed in detail

Figure 10: Transcoding a movie into JPEGs

10 Conclusion

We have developed an operating system indepen-dent, scalable and efficient application and desktopsharing platform. ADS supports all applications dueto its generic model and transmits only the sharedapplication and its child windows. We have used in-dustry standards like RTP, BFCP and PNG. The CPUusage of ADS is very low thanks to the mirror driver.The bandwidth usage of ADS is similar to other so-lutions like VNC. ADS allows collaborative workingon a single document or project. It can also be usedfor e-learning and software tutoring. People can de-velop clients and servers compatible with ADS byimplementing its open protocol.

References

[1] Tristan Richardson , Quentin Stafford-Fraser ,Kenneth R. Wood , Andy Hopper, Virtual Net-work Computing, IEEE Internet Computing,v.2 n.1, p.33-38, January 1998

[2] UltraVNC Win32Server 0.5.2 Release BuildJun 18 2006 14:56:09

[3] Portable Network Graphics,http://www.libpng.org/pub/png/

[4] Joint Photographic Experts Group,http://www.jpeg.org, ISO 10918-1, 1992

[5] Java Technology, http://java.sun.com/

[6] Peter Ziewer and Helmut Seidl. TransparentTeleTeaching. In Proceedings of ASCILITE2002, Auckland, NZ, Dec. 2002.

[7] Mehmood Hasan, Gareth Lewis, VassilAlexandrov, Dove Martin and Tucker Matt.Multicast Application Sharing Tool for theAccess Grid Toolkit. In Proceedings of the UKe-Science All Hands Meeting, Nottingham,UK, 2005.

[8] G. Camarillo, J. Ott, K. Drage. The BinaryFloor Control Protocol (BFCP). RFC 4582,IETF, November 2006.

[9] H. Schulzrinne, S. Casner, R. Frederick, V. Ja-cobson. RTP: A Transport Protocol for Real-Time Applications. RFC 3550, IETF, July2003.

[10] J. Lazzaro. Framing Real-time Transport Pro-tocol (RTP) and RTP Control Protocol (RTCP)Packets over Connection-Oriented Transport.RFC 4571, IETF, July 2006.

[11] Microsoft Windows graphics device in-terface, http://msdn2.microsoft.com/EN-US/library/ms536795.aspx

[12] J. Ott, S. Wenger, N. Sato, C. Burmeis-ter, J. Rey. Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-BasedFeedback (RTP/AVPF). RFC 4585, IETF, July2006.

13