toward trustworthy coordination of web services business

1

Toward Trustworthy Coordination ofWeb Services Business Activities

Hua Chai, Honglei Zhang, Wenbing Zhao, Member, IEEE,P. M. Melliar-Smith, Member, IEEE, L. E. Moser Member, IEEE

F

Abstract—We present a lightweight Byzantine fault tolerance (BFT)algorithm, which can be used to render the coordination of Web ServicesBusiness Activities (WS-BA) more trustworthy. The lightweight designof the BFT algorithm is the result of a comprehensive study of thethreats to the WS-BA Coordination services and a careful analysisof the state model of WS-BA. The lightweight BFT algorithm usessource ordering, rather than total ordering, of incoming requests toachieve Byzantine fault tolerant, state-machine replication of the WS-BA Coordination services. We have implemented the lightweight BFTalgorithm, and incorporated it into the open-source Kandula frameworkwhich implements the WS-BA specification with the WS-BA-I extension.Performance evaluation results obtained from the prototype implemen-tation confirm the efficiency and effectiveness of our lightweight BFTalgorithm, compared to traditional BFT techniques.

Index Terms—Business Activity, Byzantine fault tolerance, distributedtransaction, service oriented computing, trustworthy computing, WebServices

1 INTRODUCTIONService oriented computing and Web Services are trans-forming the World Wide Web from a publishing platforminto a distributed computing platform, resulting in moreand more Business Activities being conducted online.Such Business Activities, offered in the open environ-ment of the Internet, often involve financial, healthcareand other Web Services that their users require to betrustworthy.

The Web Services Business Activity (WS-BA) specifica-tion [14], developed by OASIS, standardizes the Activa-tion, Registration and Coordinator services of Web Ser-vice Business Activities. A Business Activity consists ofan Initiator, a Coordinator, and one or more Participants,as shown in Figure 1. A Business Activity is started andterminated by the Initiator. The Initiator propagates theBusiness Activity to the Participants using a Coordina-tion context in the requests. The outcome of the BusinessActivity is determined by the Initiator according to thebusiness logic.

• H. Chai, H. Zhang, and W. Zhao are with the Department of Electrical andComputer Engineering, Cleveland State University, Cleveland, OH 44115.

• P. M. Melliar-Smith and L. E. Moser are with the Department of Electricaland Computer Engineering, University of California, Santa Barbara, CA93106.

In this article, we use a travel reservation application,shown in Figure 2, as a running example, to illustratethe problem we address and our proposed solution.The travel reservation application is a composite WebService with which the clients interact directly. Behindthe scenes, the application consists of an Initiator andtwo Web Services, an Airline Reservation Web Serviceand a Hotel Reservation Web Service. These two WebServices are typically provided by third party serviceproviders, independent of the owner of the travel reser-vation composite Web Service.

When a client invokes the composite Web Service,the Initiator creates a Web Service Business Activity, inwhich it communicates with the Airline Reservation WebService to make an airline reservation and the HotelReservation Web Service to reserve a hotel room. TheCoordinator component ensures that the Business Ac-tivity is properly propagated to the Airline Reservationand Hotel Reservation Web Services, and is properlyterminated according to a pre-defined policy. The Coor-dinator component provides the Activation, Registrationand Coordinator services. More details on the travelreservation application are provided in Section 2.3.

As can be seen from the previous example, the trust-worthiness of the Coordination service is crucial to theBusiness Activities involved. It is very desirable to havethe Coordinator replicated for Byzantine fault tolerance.Unfortunately, the WS-BA specification does not specifya standard way for an Initiator of a Business Activityto communicate with the Coordinator of the Business

Initiator

Coordinator

Participants

Activation

Service

Registration

Service

Coordinator

Service

Participant

Service

BA Web

Service

Participant

Service

BA Web

Service

Participant

Service

BA Web

Service

Fig. 1. Web Services Business Activity components.

2

Fig. 2. A sequence diagram showing the steps of thetravel reservation application using the WS-BA standardwith the WS-BA-I extension.

Activity. This design choice makes it easy for vendors tointegrate WS-BA coordination functionalities into theirbusiness process engines (e.g., the travel reservation ap-plication could integrate the Initiator and the Coordina-tor into the same process), but makes it difficult to ensureinteroperability among the Initiators and Coordinatorsof different vendors [12]. Such interoperability is neededbecause of the benefits of separating the Initiator and theCoordinator, in particular:

• A small company might create a composite WebService (such as a travel reservation service) for itscustomers. However, it might be difficult for themto implement and host robust Coordination services.Even if some of them do offer Coordination services,other Web Services providers (such as those forthe Airline Reservation and Hotel Reservation WebServices) might not trust them.

• On the other hand, a large company, such as Ama-zon or Google, might offer competing Coordinationservices as part of their cloud services. To avoid ven-dor lockin, customers of such Coordination serviceswould naturally need a standard interface, to ensureinteroperability.

To address the interoperability issue, Erven et al. [12]proposed an extension to the WS-BA specification, re-ferred to as the Web Services-BusinessActivity-Initiator(WS-BA-I) protocol. The WS-BA-I protocol separates the

coordination functionality and the business logic, andstandardizes the interactions between the Initiator andthe Coordinator. With the WS-BA-I extension to WS-BA,it becomes possible for a third party to offer Coordina-tion services to enterprises that want to conduct Web Ser-vices Business Activities with minimum modifications totheir workflow engines. However, for such Coordinationservices to be widely adopted, they must be trustworthyfor their users.

Several groups of researchers have investigated theissue of trust in the Internet and the Web; useful sum-maries of that research are provided in [16], [17]. Ouruse of the term trustworthiness in this article is closestto that of Grandison and Sloman [17], who define trustas “the firm belief in the competence of an entity toact dependably, securely and reliably within a specifiedcontext.” In this article, we are particularly concernedwith high availability and high integrity of the WS-BACoordination services.

The Web Services community has developed a numberof specifications towards increasing the trustworthinessof Web Services, namely:

• WS-ReliableMessaging [8] and WS-Reliability [18].These two specifications focus on reliable messageexchange between two Web Services, despitetransient transport-level and process failures, bydefining application-level acknowledgments andsequence numbers. These two specifications aregenerally regarded as competing standards, butit appears that WS-ReliableMessaging has morewidespread support.

• WS-Security [28]. This specification defines the basicmechanisms for providing secure messaging be-tween different Web Services.

• WS-Trust [29]. This specification defines primitivesand extensions for security token exchange so thatWeb Services of different trust domains can createand disseminate credentials.

The objective of our research is to push further to-wards trustworthy Web Services in the scope of WS-BAcoordination beyond what has been addressed by theabove specifications. It focuses on the high availabilityand high integrity aspects of WS-BA coordination. Tosee why high availability and high integrity of WS-BAcoordination are needed, consider the travel reservationexample:

• If the Coordination services are not highly available,the Airline Reservation and Hotel Reservation WebServices might not be able to register for the WebServices Business Activity. Moreover, the Coordina-tion services might not be able to complete and closethe Web Services Business Activity (more details areprovided in Section 2).

• If the integrity of the Coordination services arecompromised, the Web Services Business Activitymight have inconsistent outcomes (and other un-desirable results) for the Airline Reservation and

3

Hotel Reservation Web Services, as explained inmore detail in Section 3.

The solution we present in this article is complemen-tary to, rather than competing against, the above spec-ifications. In fact, our solution relies on the use of WS-Security for secure messaging. Moreover, it is possible tointegrate WS-ReliableMessaging and WS-Trust into ourprototype, should that be desired.

To achieve high availability and high integrity, wereplicate the Coordination services of the Business Ac-tivity. Due to the untrusted operating environment ofthe Internet, it is necessary to employ a Byzantine faultmodel. A Byzantine fault is an arbitrary fault that mightbe a random fault caused by faulty hardware, or amalicious fault caused by an intrusion into the system.Byzantine fault tolerance (BFT) refers to the ability of asystem to tolerate Byzantine faults. Byzantine fault toler-ance based on replication requires 3f +1 replicas, whereat most f replicas are faulty [21]. BFT can be achievedby ensuring that all server replicas reach agreement onthe inputs despite Byzantine faulty replicas and clients.Such agreement is referred to as Byzantine agreement.Our solution is based on BFT techniques.

In this article, first we analyze the threats to the WS-BACoordination services, and explore strategies to mitigatesuch threats. Due to its high cost, we avoid the use ofa traditional Byzantine fault tolerance (BFT) algorithmdesigned for general-purpose stateful server replication.Instead, we present a lightweight BFT algorithm thatexploits the state model defined in the WS-BA specifica-tion. Our lightweight BFT algorithm does not guaranteethat messages are delivered in total order at non-faultyreplicas, as that is not needed in this case. Rather, itensures that messages are delivered in source order (i.e.,the order in which the sender sent them) at non-faultyreplicas, despite the presence of Byzantine faulty repli-cas and clients. We have implemented our lightweightBFT algorithm and associated mechanisms, and haveincorporated them into the open-source Kandula frame-work [2] that implements the WS-BA standard withthe WS-BA-I extension. Our performance evaluation ofthe prototype implementation confirms the efficiencyand effectiveness of the lightweight BFT algorithm andmechanisms, compared to traditional BFT techniques.

2 WEB SERVICES BUSINESS ACTIVITIES

In this section, we briefly describe the WS-BA specifica-tion and the WS-BA-I extension with a focus on normaloperation. We also present more details on the travelreservation application that we introduced previouslyand refer to throughout the article.

2.1 WS-BA SpecificationThe WS-BA specification [14] describes how to co-ordinate long running Business Activities where theatomic transaction model is not appropriate. It definesa Participant-side service and a set of Coordinator-side

services. The Coordinator-side services include Activa-tion, Registration and Coordinator services, which runin the same address space of the Coordinator.

The Initiator of a Business Activity requests the Acti-vation service to generate a Coordination context for theBusiness Activity and to create a Coordinator object forthe Business Activity. The Coordinator object providesthe Registration and Coordinator services, within thescope of the Business Activity. When the Business Activ-ity is propagated to it, a Web Service registers with theRegistration service by providing an endpoint referenceto its Participant service. The Registration service replieswith an endpoint reference to the Coordinator service.The Coordinator service interacts with the Participantsvia the BAwCC or BAwPC protocol (discussed below),and it interacts with the Initiator via the WS-BA-I proto-col (discussed in Section 2.2). For convenience, we referto these services collectively as the Coordinator.

WS-BA is built on top of the WS-Coordination frame-work [13]. It specifies two Coordination types: Atomic-Outcome and Mixed-Outcome. Moreover, it specifiestwo Coordination protocols that operate between theCoordinator and a Participant: Business-Agreement-with-Participant-Completion (BAwPC) and Business-Agreement-with-Coordinator-Completion (BAwCC). Ei-ther Coordination type can be used with either Coordi-nation protocol.

If the Atomic-Outcome Coordination type is used, allParticipants must reach agreement on the outcome ofthe Business Activity (i.e., Close or Compensate). If theMixed-Outcome Coordination type is used, some Partic-ipants are directed to close while others are directed tocompensate.

A Participant registers one of the two Coordinationprotocols (BAwPC or BAwCC) with the Coordinator ofthe Business Activity. We briefly describe the two Co-ordination protocols below. For detailed state transitiondiagrams of the Coordination protocols, please see theWS-BA specification [14].

For the BAwPC protocol, when a Participant hasfinished its work for a Business Activity, the Partici-pant informs the Coordinator by sending a Completedmessage. The Coordinator replies with either a Closemessage or a Compensate message, depending on thecircumstances. If the Business Activity has completedsuccessfully, the Participant receives a Close message.Otherwise, the Participant receives a Compensate mes-sage, and it must undo the completed work and restorethe data it recorded at the outset of the Business Activity.

A Participant might encounter a problem or fail dur-ing the processing of the Business Activity. If an erroroccurred when the Participant was trying to complete itsnormal activity, it generates and sends a Fail message tothe Coordinator, possibly on recovery from a transientfault. Similarly, if an error occurred when the Partici-pant was trying to cancel or compensate an activity, itgenerates and sends a CannotComplete message to theCoordinator.

4

For the BAwCC protocol, the completion notificationcomes from the Coordinator. The Coordinator sends aComplete message to the Participants, informing themthat they will not receive any new requests within thecurrent Business Activity and that they should completetheir processing. If a Participant has successfully finishedits work, it replies by sending a Completed message.Other interactions between the Coordinator and theParticipants are similar to those of the BAwPC protocol.

2.2 WS-BA-I Extension

The WS-BA-I protocol [12] is an extension of the WS-BAspecification. WS-BA-I describes how the Initiator shouldinteract with the Coordinator, which is lacking in theWS-BA specification. The WS-BA-I protocol is analogousto the Completion protocol defined in the Web ServicesAtomic Transaction specification [22]. To facilitate theWS-BA-I protocol, the Coordinator exposes a number ofadditional operations for the Initiator, including one toquery the state of the Business Activity and one to createinvitation tickets. (An invitation ticket is used by theInitiator to propagate the Business Activity to a remoteWeb Service. The remote Web Service then registerswith the Coordinator using the invitation ticket.) Inaddition, they include operations that enable the Initiatorto pass instructions to the Coordinator to Complete (forthe BAwCC protocol), Close, Cancel or Compensate theBusiness Activity.

The WS-BA-I protocol uses the pull model on theInitiator side, so that the Initiator can operate behinda firewall or a NAT box. To obtain the latest state ofa Business Activity, the Initiator periodically polls theCoordinator.

2.3 Travel Reservation ExampleFigure 2 shows the normal execution steps of the travelreservation application using the WS-BA specificationand the WS-BA-I extension, adapted from [2]. In thisexample, we use the Atomic-Outcome Coordination typeand the Business Agreement with Coordinator Com-pletion (BAwCC) protocol. The travel reservation ap-plication includes the Airline Reservation and HotelReservation Web Services. The Airline Reservation WebService comprises an Airline service and a Participantservice, and the Hotel Reservation service comprises aHotel service and a Participant service.

The Initiator creates a Business Activity Coordinationcontext using the Activation service (1 and 2), and thenregisters with the Registration service (3 and 4). Next,the Initiator searches the Airline Reservation Web Servicefor an offer, from which it receives a reply (5 and 6), andthen it searches the Hotel Reservation Web Service foran offer, from which it receives a reply (7 and 8). Next,it invokes the Coordinator service to create tickets forthe Participants (9 and 10). Using the invitation tickets,it books an airline reservation (11 and 14), and a hotelroom (15 and 18). On receiving a reservation requestfrom the Initiator, the Airline Reservation Web Service

registers with the Registration service (12 and 13); sim-ilarly, the Hotel Reservation Web Service registers withthe Registration service (16 and 17).

The Initiator sends a CompleteParticipants messageto the Coordinator service (19). The Coordinator sendsan acknowledgment to the Initiator (20) as soon as ithas sent a Complete message to the Participant servicesof the Airline Reservation and Hotel Reservation WebServices (21 and 22) without waiting to receive theirCompleted messages (23 and 24). As such, to know theParticipants’ state, the Initiator must query the Coordi-nator service (25 and 26). When all of the Participantsare in the correct state, the Initiator sends a ClosePartic-ipants message (27) to the Coordinator service. Similarto the handling of the CompleteParticipants request, theCoordinator sends an acknowledgment (28) immediatelyafter it has transmitted a Close message to the Participantservices of the Airline Reservation and Hotel ReservationWeb Services (29 and 30), without waiting to receive theirCompleted messages (31 and 32).

3 THREAT ANALYSIS

In this section, we analyze threats that can compromisethe integrity of the Coordination services of Web ServicesBusiness Activities. We do not consider general threats,such as Distributed Denial of Service (DDoS) attacks,that target any online service. Moreover, we do not con-sider entities that refuse to participate in the protocols,hoping to reduce the availability of the Coordinationservices.

3.1 Threats from a Faulty CoordinatorFirst, we consider threats from a faulty Coordinator tothe Activation service, the Registration service and theCoordinator service.

Threats to the Activation Service. At the start of a Busi-ness Activity, the Initiator requests the Activation serviceto generate a Coordination context for the Business Ac-tivity. The Activation service also creates a Coordinatorobject to handle the Business Activity. The Coordinationcontext is included in all requests sent in the scope ofthe Business Activity. The Coordination context containstwo main components: a Coordination identifier andan endpoint reference for the Registration service. TheCoordination identifier, which must be unique to theBusiness Activity, is used to associate the Participantsin the same Business Activity. The endpoint referencefor the Registration service is used by a Web Service toregister its Participant service.

A faulty Coordinator could:(1) Reuse an old Coordination identifier, which could

potentially lead to a replay attack, i.e., an adversarycould register as a Participant for a Business Activ-ity to which it does not belong, and subsequentlyreplay other messages in the scope of the BusinessActivity

(2) Use a Coordination identifier that can be easilypredicted

5

(3) Use a Coordination identifier that belongs to adifferent Business Activity (which is equivalent toreusing the Coordinator object created for anotherBusiness Activity).

Case (1) can be easily mitigated by transport-levelsecurity mechanisms such as the use of a nonce ortimestamp, without resorting to replication. Case (2)could open the door for an adversary to register with theBusiness Activity without being invited. Doing so doesnot pose a real threat to the Business Activity, becausethe outcome of the Business Activity is determined bythe Initiator. The Initiator can spot and subsequentlyexclude the unsolicited adversary from the BusinessActivity. Case (3) could lead to serious consequencesif it is not mitigated, because two unrelated BusinessActivities would appear to be the same Business Activity,which might confuse the Initiator and the Participants.The lightweight BFT algorithm described in Section 4.3mitigates such threats. Note that it is quite common formultiple Participants to register for the same BusinessActivity (e.g., the Airline Reservation and Hotel Reser-vation Web Services register themselves as Participantsfor the same Business Activity) and, hence, the Registra-tion service cannot detect the reuse of the Coordinationidentifier without additional mechanisms.

Threats to the Registration Service. A Web Service regis-ters with the Registration service by providing an end-point reference to its Participant service, as soon as theBusiness Activity is propagated to it. The Registration re-quest must contain a valid Coordination identifier, with amatchcode (introduced in WS-BA-I) to authenticate theParticipant. The reply to the Registration request con-tains an endpoint reference to the Coordinator service.

A faulty Coordinator could:

(1) Accept a Registration request with an illegitimatecredential. For example, a faulty Coordinator couldcollude with a non-reputable hotel reservation com-pany and take over the business from a reputablehotel reservation company with which the travelreservation company has contracted.

(2) Accept a correct Registration request but assign theParticipant to a faulty Coordinator. For example,a faulty Coordinator could ensure that the AirlineReservation or Hotel Reservation Web Service re-mains under its control (or another faulty Coor-dinator’s control) regarding the termination of theBusiness Activity.

(3) Return an invalid endpoint reference. For example,a faulty Coordinator could return an invalid end-point reference to the Airline Reservation or HotelReservation Web Service to deny its participation inthe Business Activity.

These threats cannot be easily mitigated usingtransport-level security mechanisms. BFT replicationand, in particular, the lightweight BFT algorithm de-scribed in Section 4.3 is the most appropriate form ofcontrol.

Threats to the Coordinator Service. The Coordinator ser-vice interacts with the Participants via the BAwCC orBAwPC protocol, and it interacts with the Initiator viathe WS-BA-I protocol. On the Coordinator side, theBAwCC protocol notifies the Participants about the com-pletion of a Business Activity, acknowledges the reportssent by them, and informs them of the final decisions re-garding the tasks that they completed. The Coordinatoralso receives notifications from the Participants.

A faulty Coordinator could:(1) Send a notification, such as a Complete, Close or

Compensate message, to a Participant, without theauthorization of the Initiator

(2) Ignore reports from the Participants, in particular,the Fail and CannotComplete messages

(3) Make arbitrary state transitions without receivingreports from the Participants.

Case (1) can be addressed by requiring all notifications(including the original signed authorization for the ac-tion from the Initiator) to carry a security certificate [33].Cases (2) and (3) would lead to a state at the Coordi-nator that is inconsistent with that of the Participants,which might ultimately affect the Initiator’s decision onthe outcome of the Business Activity. The lightweightBFT algorithm, described in Section 4.3, can effectivelyhandle such threats.

The BAwPC protocol is similar to the BAwCC protocol,except that the Coordinator is no longer responsiblefor notifying the Participants of the completion of theBusiness Activity. Instead, the Coordinator collects Com-pleted messages from the Participants. Consequently, thethreats that a faulty Coordinator poses to the Coordina-tor service in the BAwPC protocol are similar to thosein the BAwCC protocol, and can be handled in a similarmanner.

In the WS-BA-I protocol, the Coordinator creates aninvitation ticket on request from the Initiator. The invita-tion ticket is an extended Coordination context, contain-ing an additional identifier (called the matchcode) usedto invite the Participant to join the Business Activity.The Coordinator also takes requests from the Initiatorto obtain the latest status of each Participant, and toComplete (if the BAwCC protocol is used), Close orCompensate the Business Activity.

A faulty Coordinator could:(1) Return an invalid invitation ticket to the Initiator,

such as an old invitation ticket, an invitation ticketthat belongs to another Participant, or an invitationticket that is easy-to-predict

(2) Present an incorrect state to the Initiator, whichmight confuse the Initiator

(3) Alter the instructions issued by the Initiator to Com-plete, Close or Compensate the Business Activity.

The possible replay attack caused by Case (1) canbe mitigated by transport-level security mechanisms, asmentioned previously. For the other threats in Cases(1)–(3), the lightweight BFT algorithm, described in Sec-

6

tion 4.3, is an effective control.

3.2 Threats from a Faulty ParticipantNow we consider threats from a faulty Participant to theCoordinator and the Initiator.

Threats to the Coordinator. A faulty Participant could:(1) Lie about its execution status and send reports,

that are inconsistent with its internal state, to theCoordinator

(2) Send conflicting reports to different Coordinatorreplicas.

Case (1) can be prevented by replicating each Partici-pant using BFT replication techniques and by collectingmatching requests from f + 1 distinct replicas beforeaccepting them (assuming at most f replicas are faulty).However, imposing such a strong requirement on theParticipants is not practical. Therefore, this type of threatis best addressed by using digital signatures on themessages exchanged between the Participants and theCoordinator, and by logging messages to/from the Par-ticipants at the Coordinator. Case (2) can be addressed bythe lightweight BFT algorithm described in Section 4.3.

Threats to the Initiator. A faulty Participant could attackthe Initiator in a similar way to the attacks on theCoordinator, as described above.

Again, these types of threats are best addressed byusing digital signatures on messages exchanged betweenthe Participants and the Initiator, and by the Initiator’slogging messages to/from the Participants. The Initiatormight also give preference to Participants that offer ahigher quality of service to minimize such threats.

3.3 Threats from a Faulty InitiatorBecause a Business Activity is initiated, and its progressand outcome are controlled, by the Initiator, the integrityof a Business Activity that originates at a Byzantinefaulty Initiator cannot be guaranteed. As in the case ofa faulty Participant, we can address these threats fromoccurring by replicating the Initiator and employingBFT techniques. However, based on similar concernsas above, we do not wish to impose such a strongrequirement on the Initiator. Instead, a more practicalapproach is for the Participants and the Coordinator toemploy non-repudiation and logging techniques to holda faulty Initiator accountable.

With a replicated Coordinator, a new threat mightoccur, i.e., a faulty Initiator might send conflicting re-quests to different Coordinator replicas. Such a threat canbe readily addressed by the lightweight BFT algorithmdescribed in Section 4.3.

4 BFT COORDINATION SERVICES

In this section, we present the lightweight BFT algorithmand associated mechanisms for achieving trustworthycoordination of Web Services Business Activities. An im-portant objective is to enable an independent third partyto launch trustworthy Coordination services for Web Ser-vices Business Activities that span multiple enterprises.

The practicality of the BFT solution is essential for real-world use, which means that it must be lightweight withmoderate runtime overhead and good scalability.

Traditional BFT state-machine replication mechanisms[5] can be used to mitigate the threats to the WS-BACoordination services. However, it is not wise to usethem naively. Such mechanisms are designed to pro-tect general-purpose stateful servers from Byzantine faults.They impose a total order on incoming requests and,thus, are very heavyweight and incur significant run-time overhead, due to the multiple rounds of messageexchange that are required.

In our lightweight BFT algorithm, we exploit knowl-edge of the state model of the WS-BA Coordinationservices, described below, and use a solution that iscustomized to this specific application. In short, thelightweight BFT algorithm requires only source ordering,instead of total ordering, of incoming requests, whicheliminates inter-replica message exchange and hence re-duces the communication steps and processing overheadsignificantly compared with traditional BFT algorithms.

4.1 State Model AnalysisIn WS-BA, requests within different Business Activitiesare handled independently, i.e., their relative orderingdoes not affect the state transitions. This independenceis obvious for the Registration and Coordinator requests,because they are handled by different Coordinator ob-jects. Even though all Activation requests are handled bythe same process, their relative ordering does not affecthow the Coordinator objects are created. However, thegeneration of Coordination identifiers can be a concernfor replica consistency. We handle this particular replicanon-determinism issue without resorting to inter-replicacoordination (as described in Section 4.4), which obvi-ates the need to run an expensive Byzantine agreementalgorithm among the Coordinator replicas.

Next, we consider requests within the same BusinessActivity. The Activation and Registration requests arecausally related, i.e., the Activation request must pre-cede the Registration request. This relative ordering ofrequests is programmed directly into the BFT frame-work without resorting to inter-replica coordination. Itis straightforward to ensure source ordering of requestsfrom a Participant in the BAwCC or BAwPC proto-col (e.g., by using a sequence number). Likewise, it isstraightforward to ensure source ordering of requestsfrom the Initiator.

The change of one part of the Coordinator state associ-ated with one Participant has no direct affect on anotherpart of the Coordinator state associated with a differ-ent Participant, as shown in Figure 3. The Coordinatorobject uses each part to keep track of the state for eachParticipant according to the BAwCC or BAwPC protocol.Requests sent by different Participants to the same Co-ordinator object modify only their respective parts of thestate. (Duplicate requests from the same Participant donot change the Coordinator state, although they might

7

Fig. 3. The state model for WS-BA with the WS-BA-Iextension. The Participants’ states in the Coordinatorobject are completely separate.

trigger the resending of messages.) Therefore, it is un-necessary to order requests from different Participants.

The only remaining issue is whether or not to orderthe requests from the Initiator relative to those fromthe Participants within the same Business Activity. Theanswer is no. The requests from the Initiator can becategorized into three types, according to the WS-BA-Ispecification, as discussed below.

The first type of request creates invitation tickets forthe Participants, one at a time. This type of request canbe interleaved with the requests from the Participantswithin the Business Activity, because they are handledby different objects.

The second type of request queries the state of theBusiness Activity, which is read-only. Even though theserequests do not change the Coordinator state, differentreplicas might report different states to the Initiator ifthe query requests are not ordered with respect to theParticipants’ requests. This difference in the states is nota concern because the Initiator can repeatedly querythe Coordinator replicas until their states converge, asguaranteed by the lightweight BFT algorithm (see theproof of Theorem 3 in Section 4.5).

The third type of request directs the Coordinator toterminate (Close, Cancel or Compensate) the BusinessActivity. If these requests are not ordered with respectto the messages from the Participants, when such arequest arrives, some replicas might have evolved intodifferent states since the Initiator last queried them. Thisdifference in states is not a concern because, according tothe WS-BA-I protocol, the state of the Coordinator objectmight be inconsistent with that of the Initiator evenwithout replication. Consider the following scenario. TheInitiator sends a Cancel message to the Coordinator for aParticipant when the last seen state for that Participantis Completing. However, when the Cancel message isdelivered, the Completed message from the Participantmight have arrived, in which case the Coordinatorshould send a Compensate message to the Participantinstead of a Cancel message. This mechanism is alreadybuilt into the standard WS-BA Coordination framework.

Thus, there is no need to ensure total ordering ofthe messages involved in a Business Activity at theCoordinator replicas.

4.2 Assumptions and Requirements

We assume that the Web Services Business Activitiesoperate in the asynchronous distributed environment ofthe Internet. We assume that network communication isreliable. In particular, if a non-faulty Participant/Initiatorsends a message to a non-faulty Coordinator replica, thereplica will eventually receive the message. The same istrue for messages exchanged between non-faulty Coordi-nator replicas. This reliable communication assumptionis easily satisfied by using TCP communication and themechanisms defined in the Web Services ReliableMes-saging (WS-RM) specification [8].

We assume that there are 3f +1 Coordinator replicas,of which at most f are faulty. Each Coordinator replicais assigned a unique identifier k, where k ranges from0 to 3f . All of the Coordinator replicas play an equalrole; in particular, no Coordinator replica acts as theprimary because total ordering is not needed (i.e., thesequence numbers are assigned by the client, rather thanby a primary Coordinator replica). The Initiator and theParticipants are not replicated (our BFT algorithm can betrivially modified if these entities are in fact replicated).As pointed out earlier, only a subset of faulty behaviorsof the Initiator and the Participants are tolerated by repli-cating the Coordinator. Other types of faults are handledby using non-repudiation and logging techniques.

The high degree of redundancy (four Coordinatorreplicas to tolerate one faulty Coordinator replica) re-quired to achieve Byzantine fault tolerance [21] mightbe a concern for practitioners. However, with thewidespread adoption of virtualization technology, theneed for extra physical hardware is minimized becausethe four Coordinator replicas can be deployed on dif-ferent physical nodes (for fault isolation), which arerunning other services concurrently.

All WS-BA messages, i.e., the messages exchanged be-tween the Coordinator and the Initiator and between theCoordinator and the Participants, carry monotonicallyincreasing sequence numbers. The sequence number isunique only within the respective connection betweenthe Coordinator and its client (i.e., the Initiator or a Par-ticipant), but each connection is uniquely identified. Foreach connection, the first request is assigned sequencenumber 0 and, for each subsequent request, the sequencenumber is incremented. The reply, if any, carries a se-quence number that matches that of the correspondingrequest. The same holds true for requests sent from theCoordinator to the Participants in the BAwCC or BAwPCprotocol. Note that all messages defined in the BAwCCor BAwPC protocol are one-way messages.

All messages between the Coordinator and the Par-ticipants are timestamped (to prevent replay attacks)and are digitally signed (to ensure accountability and toprevent spoofing). The Initiator, each Coordinator replicaand each Participant has a public/private key pair. Thepublic key is known to all of them, while the privatekey is kept secret by its owner. We assume that the

8

ClientRequest

Replica 0

Replica 1

Replica 2

Commit Reply

Replica 3

Fig. 4. Normal operation of the lightweight BFT algorithmfor f = 1.

adversaries have limited computing power and, thus, areunable to break the digital signatures.

Our lightweight BFT algorithm for trustworthy coor-dination of Web Services Business Activities, describedbelow, satisfies the following three properties:P1. If a Byzantine faulty sender sends conflicting re-

quests (different requests with the same sequencenumber) to different non-faulty Coordinator repli-cas, at most one of those requests is delivered at allnon-faulty Coordinator replicas.

P2. If a request is delivered at a non-faulty Coordinatorreplica, it is eventually delivered at all non-faultyCoordinator replicas in the sender’s source order.

P3. The states of the non-faulty Coordinator replicaswill eventually converge, and the Initiator will even-tually receive a response to a query regarding thestate of the Business Activity, that is consistent withthe converged state of the Coordinator replicas.

The proofs of correctness, demonstrating that thelightweight BFT algorithm satisfies these properties, aregiven in Section 4.5.

4.3 The Lightweight BFT AlgorithmBased on the previous state model analysis, we havedeveloped the lightweight BFT algorithm described be-low. The normal operation of the algorithm is shown inFigure 4 for f = 1.

A client (i.e., the Initiator or a Participant) sendsa request to all Coordinator replicas. The request hasthe form <REQUEST, s, o, c>σc , where s is the sequencenumber of the request message, o is the operation to beinvoked (including the Coordination context, if needed),c is the client identifier, and σc is the digital signature ofthe request message signed by the client c.

On receiving a request, a Coordinator replica validatesthe signature, and checks the validity of the requestedoperation and whether the sequence number in therequest matches its next expected sequence number. Ifthe request passes this validation test, a Coordinatorreplica multicasts a Commit message to all of the otherCoordinator replicas. The Commit message has the form<COMMIT, c, s, d, k>σk

, where c is the identifier of theclient that issued the request message, s is the sequencenumber of the request message, d is the digest of therequest message, k is the replica identifier, and σk is the

signature of the Commit message signed by the replicak.

If a Coordinator replica receives a request and match-ing Commit messages corresponding to that requestfrom 2f other Coordinator replicas, it delivers the re-quest and makes the state transition. The reply, if any,has the form <REPLY, s, r, k>σk

, where s is the sequencenumber of the reply message, r is the response, k is thereplica identifier, and σk is the signature of the replymessage signed by the replica k.

If a Coordinator replica receives a Commit messagebefore it receives the referenced request, it requestsa (simple) retransmission of the request (in the form<SIMPLE-RETRANSMIT, c, s, d, k>σk

) from the Coordina-tor replica that sent the Commit message. If a Coor-dinator replica receives a retransmission request fromanother Coordinator replica, it retransmits the request.

In the presence of Byzantine faulty clients, and Byzan-tine faulty Coordinator replicas, the following two sce-narios might happen: (1) a replica might receive 2f + 1matching Commit messages but the digest in the Com-mit message does not correspond to that of the requestreceived; and (2) a replica might receive 2f + 1 mis-matched Commit messages from different Coordinatorreplicas.

In scenario 1, the replica request a simple retransmis-sion. As soon as it receives the correct request (i.e., therequest with a digest that matches the 2f + 1 Commitmessages), the replica abandons the original request andaccepts the retransmitted one.

In scenario 2, the replica broadcasts a retransmissionof the request with proof of commitment (in the formof <RETRANSMIT-WITH-PROOF, c, s, d, k>σk

) to all otherreplicas, and waits until one of the following two eventshave occurred:

1) It has collected a retransmission with a valid Com-mit record with 2f + 1 signed Commit messages.This would enable the replica to commit and de-liver the request right away.

2) It has received responses from at least 2f + 1replicas and a predefined timeout with value T hasoccurred. In this case, the timeout value is doubledto 2T and the replica broadcasts a new round ofretransmit-with-proof request. This may be donerecursively until condition (1) is eventually met.

Upon receiving such a retransmission-with-proof re-quest, a replica retransmits the request with matchingrequest identifier (not necessarily the same digest d), andall the Commit messages collected for the same request.

If the client (i.e., the Initiator or a Participant) thatissued a request expects a reply from the Coordinatorreplicas, it collects matching replies from f + 1 distinctCoordinator replicas before it delivers the reply. Thesame mechanism is used for the Participants to handle(nested) requests sent by the Coordinator replicas. Be-cause at most f Coordinator replicas are faulty and allof the replies match, at least one of the replies is sent bya non-faulty Coordinator replica.

9

Note that some operations might trigger the sendingof nested requests to the Participants (e.g., the Com-plete/Close message from the Initiator to a Participantwill cause a non-faulty Coordinator replica to send thesame command to that Participant). Such nested requestsare sent directly to the Participants. A non-faulty Par-ticipant accepts a nested request when it has collectedmatching requests from f +1 distinct Coordinator repli-cas. However, the nested reply must be treated as a newrequest, i.e., an additional communication step is neededbefore the nested reply is delivered at the Coordinatorreplica because the nested reply might trigger a statechange.

4.4 Additional MechanismsIn a typical implementation of the WS-BA standard, theCoordination identifier is generated by the Activationservice (e.g., by using a UUID). When the Activationservice is replicated, generating the Coordination identi-fier in this way causes replica non-determinism. Eventhough such non-determinism could be controlled byusing the mechanism described in [32], it is unnecessaryin light of the threat analysis given in Section 3. A moreefficient method to handle this non-determinism is topiggyback the UUID onto the Activation request sentby the Initiator. One implication of this strategy is thatthe Coordinator replicas must now keep a record ofthe UUIDs used and verify the uniqueness of a newlyproposed UUID against that record.

Another mechanism used by our lightweight BFTsolution is the security certificate mentioned in Section 3.A security certificate is piggybacked onto each messagesent by the Coordinator to a Participant in the BAwCCor BAwPC protocol. The security certificate consists ofthe original signed authorization for the action from theInitiator, which includes not only the specific commandbut also the Coordination identifier and timestamp toprevent a replay attack. On receiving such a command,a Participant verifies that the command is consistent withthat in the security certificate. If a Participant detects amismatch, it ignores the command.

4.5 Proofs of CorrectnessNow we provide (informal) proofs of correctness forthe lightweight BFT algorithm in terms of the threeproperties given in Section 4.2. The proofs are basedon source ordering of requests, and address the threatsidentified in Section 3.

Theorem 1. If a Byzantine faulty sender sends conflictingrequests (different requests with the same sequence number) todifferent non-faulty Coordinator replicas, at most one of thoserequests is delivered at all non-faulty Coordinator replicas.

Proof: The proof is by contradiction. Assume that arequest message M is delivered at a non-faulty Coor-dinator replica R and that a different request messageM ′ with the same sequence number from the samesender is delivered at another non-faulty Coordinatorreplica R′. It must be the case that a set S containing

2f + 1 Coordinator replicas have accepted M and senta Commit message for M and that a set S′ containing2f + 1 Coordinator replicas have accepted M ′ and senta Commit message for M ′. Because there are 3f + 1Coordinator replicas, S and S′ must intersect in f + 1or more Coordinator replicas. Because at most f Coordi-nator replicas are faulty, at least one Coordinator replicain the intersection must be non-faulty. This contradictsour BFT algorithm, which does not allow a non-faultyCoordinator replica to accept two different requests withthe same sequence number from the same sender.

Theorem 2: If a request is delivered at a non-faulty Co-ordinator replica, it is eventually delivered at all non-faultyCoordinator replicas in the sender’s source order.

Proof: First, we assume that the sender of the requestis not Byzantine faulty. By the assumption of reliablecommunication in Section 4.2, the sender establishes aconnection with each non-faulty Coordinator replica andreliably sends the request in source order to that non-faulty Coordinator replica. Consequently, non-faulty Co-ordinator replicas receive the request, exchange Commitmessages, and deliver the request.

If the sender is Byzantine faulty, it might (1) send thesame request to only some of the non-faulty Coordina-tor replicas, or (2) send conflicting requests (differentrequests with the same sequence number) to differentnon-faulty Coordinator replicas.

In Case (1), by the hypothesis of the theorem, a requestis delivered at a non-faulty Coordinator replica R. Thatreplica R must have received Commit messages for therequest from 2f other Coordinator replicas, of whichat most f are non-faulty. If a non-faulty Coordinatorreplica R′ did not receive the request, it will eventuallyreceive the Commit messages from f + 1 other non-faulty Coordinator replicas (including replica R), whichwill prompt it to request a retransmission of the request.By the reliable communication assumption, eventuallyR′ will receive the request from one of those non-faulty Coordinator replicas. Consequently, all non-faultyCoordinator replicas will eventually receive the request,exchange Commit messages, and deliver the request.

In Case (2), by the hypothesis of the theorem, a requestm is delivered at a non-faulty Coordinator replica R.That replica R must have received Commit messages forthe same request from 2f other Coordinator replicas, ofwhich at most f are faulty, i.e., a set S of at least 2f + 1replicas must have sent a Commit message for m. At anyother non-faulty Coordinator replica R′, either it is ableto collect 2f + 1 matching Commit messages with therequest (not necessarily with identical digest to that ofthe request received by the replica (scenario 1), or it failsto collect 2f+1 matching Commit messages (scenario 2).

In scenario 1, we prove by contraction that the requestdelivered at R′ must be identical to the one deliveredat R. Assume that R′ delivered a different request m′

(i.e., identical request id but with a different digest).This means that a set S′ at least 2f + 1 replicas havesent a Commit message for m′. Because there are 3f +1

10

replicas, S and S′ must intersect in at least f+1 replicas,which means at least one non-faulty replica has sent aCommit message for m and another Commit for m′. Thisis impossible.

In scenario 2, R′ would broadcast a retransmit-with-proof request to all other replicas and wait until eitherit has received a retransmission with a proof of commit-ment (i.e., 2f + 1 matching Commit messages), or it hasreceived at least 2f+1 retransmissions and a predefinedtimeout has occurred. In the former, R′ would be ableto delivery the retransmitted request as we have provedin scenario 1. In the latter, R′ would broadcast a newround of retransmit-with-proof with a bigger timeoutvalue. Provided that the message delays from R to R′

does not grow faster than the timeout period indefinitely,R’s Commit record (i.e., 2f + 1 Commit messages) willreach R′ before the timeout to enable it to commit anddeliver m.

Theorem 3. The states of the non-faulty Coordinator replicaswill eventually converge, and the Initiator will eventuallyreceive a response to a query regarding the state of the BusinessActivity, that is consistent with the converged state of theCoordinator replicas.

Proof: By Theorem 2, if a request is delivered at a non-faulty Coordinator replica, it is eventually delivered atall non-faulty Coordinator replicas. Therefore, all non-faulty Coordinator replicas will eventually deliver thesame set of requests, at which point the states of thenon-faulty Coordinator replicas will converge.

Our BFT algorithm requires the Initiator to collectmatching responses for a query from f + 1 distinctCoordinator replicas before accepting the response. Atleast one of those matching responses is sent by a non-faulty Coordinator replica. Consequently, the Initiatorwill eventually receive a response to a query regardingthe state of the Business Activity, that is consistent withthe converged state of the Coordinator replicas.

5 IMPLEMENTATION AND PERFORMANCE

We have implemented the lightweight BFT algorithmand associated mechanisms, and incorporated them intothe Kandula framework [2], which is a Java-based, open-source implementation of the WS-BA specification withthe WS-BA-I extension. Besides Kandula, the extendedframework is based on several other Apache Web Ser-vices technologies, including Apache Axis [1] and WSS4J[3], which is an implementation of the WS-Securitystandard [28]. Most of the mechanisms are implementedusing Axis handlers that are plugged into the frameworkwithout affecting other components. Some of the Kan-dula code is modified to enable control of its internalstate and BFT delivery of requests at the Initiator, theCoordinator, and the Participants. All messages havetimestamped digital signatures.

To demonstrate the benefits of our lightweight BFTalgorithm over traditional BFT algorithms, we imple-mented an adapted version of the Practical ByzantineFault Tolerance (PBFT) algorithm [5] for comparison. In

this implementation, we chose to use digital signaturesfor all messages exchanged (instead of the messageauthentication code). We launched one instance of thePBFT algorithm for each Coordinator, i.e., we allowconcurrent total ordering of messages belonging to dif-ferent Business Activities for fair comparison with ourlightweight BFT algorithm. Otherwise, the throughputfor concurrent Business Activities would be very low forPBFT, because it would not enjoy concurrent processingas our lightweight BFT algorithm does.

We have evaluated the performance of our lightweightBFT algorithm both in a Local-Area Nework (LAN)testbed and in a Wide-Area Network (WAN) testbed(i.e., PlanetLab). The LAN testbed consists of 14 HPBL460c blade servers connected by a Cisco 3020 Gigabitswitch. Each blade server is equipped with two XeonE5405 (2 GHz) processors and 5 GB RAM, and runs the64-bit Ubuntu Linux operating system. The hardwarespecifications of the PlanetLab nodes vary significantly.The nodes we chose to use are generally equipped withIntel Core 2 Duo CPUs (2GHz to 2.6GHz). Note that thenodes in PlanetLab are shared among many users, andwe have no control over the actual load on the CPUsand the available physical memory.

The test application is the travel reservation applica-tion described in Sections 1 and 2.3, and shown in Figure2, with a slight modification on how the Coordinatorhandles CompleteParticipants and CloseParticipants re-quests from the Initiator. Instead of sending a responsefor each request as soon as the nested request has beensent to all of the Participants, a Coordinator replicawaits until it has collected the responses from all ofthe Participants. That is, in Figure 2, message 20 issent after the Coordinator has received message 24, andmessage 28 is sent after the Coordinator has receivedmessage 32. This modification (that the Initiator doesnot need to query the Coordinator repeatedly to seewhether all of the Participants have responded) is purelyfor the sake of benchmarking, so that the benchmarkingresults are more consistent across different runs. TheCoordinator is replicated on 3f + 1 = 4 processors totolerate f = 1 Byzantine faulty replicas. The Initiatorand the Participants each run on a distinct processor.The Initiator launches and terminates Business Activitiesrepeatedly within a loop without any think time.

The end-to-end latency of each Business Activity ismeasured at the Initiator. The throughput of the Coordi-nation framework is measured at one of the Coordinatorreplicas. In each run, we obtained 1,000 samples and cal-culated the median latency and the median throughput.We chose to use the median instead of the mean, be-cause the results obtained in PlanetLab vary significantlyacross different runs.

5.1 Performance Evaluation in a LANThe end-to-end latency results in a LAN are shown inFigure 5. These results are obtained for four differentconfigurations:

11

0

500

1000

1500

2000

1 2 3 4 5 6 7 8 9

En

d-t

o-E

nd L

aten

cy (

mil

lise

cond

s)

Number of Participants in Each Business Activity

Unmodifed ApplicationWith Digital SignaturesWith Lightweight BFTWith Traditional BFT

Fig. 5. End-to-end latency in a LAN for Business Activ-ities with different numbers of Participants under normaloperation.

(1) The test application is used as is without any mod-ification (labeled “Unmodified Application” in thefigure)

(2) The test application is modified such that all mes-sages exchanged within each Business Activity areprotected by digital signatures (labeled “With Digi-tal Signatures” in the figure)

(3) The test application is protected by our lightweightBFT algorithm with replication of the Coordinator(labeled “With Lightweight BFT” in the figure).

(4) The test application is protected by the traditionalBFT algorithm with replication of the Coordinator(labeled “With Traditional BFT” in the figure).

In all four configurations, the end-to-end latency ismeasured in the presence of a single Business Activity(i.e., neither the Coordinator nor the various Web Ser-vices are shared with other Business Activities).

As can be seen in Figure 5, the end-to-end latencyof our lightweight BFT algorithm is significantly largerthan that of the unmodified application. However, thisincrease is due mainly to the use of digital signatures,as revealed by the high end-to-end latency when onlydigital signatures are used. In our LAN testbed, thecost of digitally signing a message is about 5ms, andthe cost of verifying a digital signature is about 2ms.In our test application, a single Business Activity withonly one Participant involves 22 digital signatures and 22validation operations during the critical path (eight extradigital signatures and eight extra validation operationsare needed for each additional Participant). The largenumber of such operations inevitably increases the end-to-end latency. Nevertheless, it is our conviction that theuse of digital signatures is justified (indeed essential) inpractice because of the need for non-repudiation in anyserious Business Activity. Thus, we use this configura-tion as the baseline for comparison.

The overhead of our lightweight BFT algorithmis due primarily to two factors for each committedmessage: (1) one extra communication step, and (2) oneextra digital signature (for the multicast message) andtwo validation operations (for the received message).

For a Business Activity with a single Participant, ninemessages must be committed (three extra messagesfor each additional Participant). These two factorsresult in a non-negligible overhead, with about 40-50%larger end-to-end latency compared with that of thedigital-signatures-only configuration.

As expected, the end-to-end latency with the tradi-tional BFT algorithm is much larger, as shown in Fig-ure 5. Compared to our lightweight BFT algorithm, thetraditional BFT algorithm incurs the following additionaloverhead for each message delivered at the Coordina-tor during normal operation: (1) two communicationsteps, (2) two digital signatures, and (3) five validationoperations. Therefore, the end-to-end latency with thetraditional BFT algorithm is about 130-170% higher thanthat of the baseline configuration (more than three timesthe overhead compared to our lightweight BFT algo-rithm). Note that this experiment did not consider thecases when some replicas fail. In the traditional BFTalgorithm, when the primary replica fails, one or moreview changes will be needed, which could result ineven larger and more unpredictable latency. Because ourlightweight BFT algorithm does not depend on any onereplica serving as the primary, the negative impact on theperformance is negligible when a replicas fails, similar tothe impact when a backup replica fails in the traditionalBFT algorithm.

The throughput results (in terms of number ofBusiness Activities per minute) are shown in Figure 6(a)-(c) for Business Activities with 2, 5, and 9 Participantsin a LAN. The throughput experiments are carriedout with the same set of configurations as those usedin the end-to-end latency study, i.e., the unmodifiedapplication, the application with digital signatures, theapplication protected by our lightweight BFT algorithm,and the application protected by a traditional BFTalgorithm. The load on the Coordinator and the setof Participants is driven by one or more Initiators,launching Business Activities consecutively. As such,the throughput measurements are performed for variousnumbers of concurrent Business Activities.

As can be seen in Figure 6(a)-(c), compared to thebaseline configuration, the throughput reduction withour lightweight BFT algorithm is only about 30%. Thisreduction is expected because of the increased processingtime at the Coordinator (for message signatures andvalidation). Again, not surprisingly, the throughput re-duction with the traditional BFT algorithm is about 50%,much more significant than that for the lightweight BFTalgorithm.

5.2 Performance Evaluation in a WAN

The end-to-end latency results in a WAN are summa-rized in Figure 7. As can been seen in the figure, therelative overhead, over the baseline configuration, ofthe lightweight BFT algorithm remain similar to thatin the LAN (about 40-50%), and the same is true for

12

0

500

1000

1500

2000

2 4 6 8 10 12 14 16 18 20

Th

rou

gh

pu

t (B

As/

min

ute

)

Number of Concurrent BAs(a)

2 Participants

Unmodified ApplicationWith Digital SignaturesWith Lightweight BFTWith Traditional BFT

0

200

400

600

800

1000

2 4 6 8 10 12 14 16 18 20

Th

rou

gh

pu

t (B

As/

min

ute

)

Number of Concurrent BAs(b)

5 Participants


0

100

200

300

400

500

2 4 6 8 10 12 14 16 18 20

Th

rou

gh

pu

t (B

As/

min

ute

)

Number of Concurrent BAs(c)

9 Participants


Fig. 6. Throughput of the coordination services in a LAN with different numbers of concurrent Business Activities.

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

1 2 3 4 5 6 7 8 9

End

-to-E

nd L

aten

cy (

mil

lise

con

ds)

Number of Participants in Each Business Activity


Fig. 7. End-to-end latency in a WAN for Business Activ-ities with different numbers of Participants under normaloperation.

the configuration using the traditional BFT algorithm.However, it is interesting to note that, due to the largecommunication delay inevitable in the WAN, the relativecost of digital signatures and validation operations isreduced. Consequently, the overhead of our lightweightBFT algorithm, compared with that of the unmodifiedapplication, drops significantly (from about 250% toabout 100%).

The throughput results in the WAN are shown inFigure 8(a)-(c). The throughput for all configurations inthe WAN is a lot less than that in the LAN primarily forthree reasons:(1) In general, the nodes in PlanetLab for the WAN

experiment are equipped with less capable CPUscompared to our LAN testbed (e.g., Core 2 DuoE6550 vs. Xeon E5405). The passmark for the Core2 Duo E6550 is 1448, whereas the passmark for theXeon E5405 is 2895. 1

(2) The nodes in PlanetLab are shared among manyconcurrent users and, hence, our application mightnot be able to exploit the multiple cores available,

1. A passmark is the benchmark score of a computer using thesoftware developed by Passmark Software. The passmark scores wereobtained from http://www.cpubenchmark.net/

while the nodes in our LAN testbed are exclusivelyused for our experiment.

(3) The available network bandwidth between differentnodes in PlanetLab is presumably a lot less than thatin our LAN testbed, which is connected by a GigabitEthernet. As a result, the message transmission timeis a lot more for the WAN experiment, which alsoadversely affects the throughput. For example, ifthe total size of messages sent and received bythe Coordinator is 30KB and if the bandwidth inPlanetLab is 10Mbps, then the transmission time inthe PlanetLab testbed would be 24ms whereas, inour LAN testbed, the transmission time would be anegligible 0.24ms.

Due to the greater message transmission time, thethroughput reduction for both the lightweight BFT al-gorithm and the traditional BFT algorithm is reducedcompared to that of the unmodified application. Theseresults suggest that it might be more appropriate toadopt the BFT approach in the WAN, where Web Ser-vices Business Activities are typically deployed, ratherthan in the LAN. In both the WAN and the LAN,the lightweight BFT algorithm significantly outperformsthe traditional BFT algorithm, as demonstrated in ourexperiments.

6 RELATED WORK

The trustworthiness of Internet and Web applicationshas become increasingly important in recent years [16],[17], [29], particularly with the advent of Web ServicesBusiness Activities that involve interactions betweenmultiple enterprises. The WS-Trust specification [29] fo-cuses on the use of security tokens and credentials withinthe context of Web Services whereas, in this article, wefocus on the high availability and high integrity aspectsof trustworthy coordination of Web Service BusinessActivities.

Byzantine fault tolerance has been of great researchinterest for the last several decades. The seminal re-search in this field is that of Lamport, Shostak andPease [21], who demonstrated that four replicas arerequired to tolerate one Byzantine faulty replica (three

13

0

100

200

300

400

500

2 4 6 8 10 12 14 16 18 20

Th

rou

gh

pu

t (B

As/

min

ute

)

Number of Concurrent BAs(a)

2 Participants


0

50

100

150

200

250

2 4 6 8 10 12 14 16 18 20

Th

rou

gh

pu

t (B

As/

min

ute

)

Number of Concurrent BAs(b)

5 Participants


0

20

40

60

80

100

120

140

160

2 4 6 8 10 12 14 16 18 20

Th

rou

gh

pu

t (B

As/

min

ute

)

Number of Concurrent BAs(c)

9 Participants


Fig. 8. Throughput of the coordination services in a WAN with different numbers of concurrent Business Activities.

replicas do not suffice) and, more generally, that 3f + 1replicas are required to tolerate f Byzantine faulty repli-cas. Other significant contributions to Byzantine agree-ment/consensus about the same time were made byDolev, Strong, Cristian and others (see, for example, [4],[7], [9], [10], [11]).

Subsequently, several groups of researchers developedByzantine fault-tolerant group communication systems[6], [19], [23], [26], [27], [31] for generic distributed appli-cations. All of those systems deliver messages in total or-der at the destinations. In contrast, our lightweight BFTalgorithm for trustworthy coordination of Web ServicesBusiness Activities delivers messages in source order, i.e.,the order in which the sender sent them, rather than intotal order at the destinations. It does so by exploitingthe state model of the WS-BA specification.

The Practical Byzantine Fault Tolerant (PBFT) agree-ment algorithm [5] of Castro and Liskov designates oneof the replicas as the primary and the other replicas asthe backups. The PBFT algorithm involves three phasesto order a request (the Pre-Prepare phase, the Preparephase, and the Commit phase). It also includes a ViewChange algorithm that determines a new primary, ifthe current primary is deemed to be faulty. Kihlstromet al. [20] described a BFT consensus algorithm. TheirBFT algorithm for achieving consensus operates in threerounds, and uses a Coordinator to choose the consensusvalue and Byzantine fault detectors to detect faults inthe Coordinator and other processes.

The application of BFT techniques to transactionalsystems was first reported by Mohan et al. [25]. Theyenhance the Two-Phase Commit protocol by performingByzantine agreement on the outcome of an atomic trans-action among all of the nodes in a root cluster. Recently,we revisited this problem and proposed a more efficientsolution [33] for Atomic Transactions by restricting theByzantine agreement to only the Coordinator replicas.Garcia-Molina et al. [15] have applied Byzantine agree-ment to distributed database systems, in particular, fordistributing transactions to processing nodes.

Even though coordination of Atomic Transactionsbears some similarity to coordination of Business Ac-

tivities, there are a number of significant differences. InAtomic Transactions, the Coordinator and the Partici-pants are tightly coupled. Any Participant can unilat-erally abort a transaction. Furthermore, the Coordinatorcan decide on the outcome of the transaction, based onthe votes collected in the Two-Phase Commit protocol.However, in Business Activities, the outcome is decidedsolely by the Initiator according to the business logicand, at best, a faulty Participant might be able to exertsome influence on, but not decide on, the outcome of theBusiness Activity. These characteristics enable the use ofa lightweight BFT solution for the trustworthy coordina-tion of Web Services Business Activities, as described inthis article.

The application of BFT techniques to Web Services hasbeen reported in [24], [30], [34]. Even though the solu-tions proposed could be used to protect the Coordina-tion services of Web Services Business Activities againstByzantine faults, they are unnecessarily expensive. Byconsidering the state model of the WS-BA Coordinationservices, our lightweight BFT algorithm introduces onlyone additional round of message exchange for eachinvocation on the Coordinator instead of two or moreadditional rounds of message exchange needed in [24],[30], [34] and, thus, it performs significantly better.

7 CONCLUSIONS

We have presented a comprehensive study of the threatsto the Coordination services of Web Services BusinessActivities (WS-BA). We have carefully analyzed the statemodel of the WS-BA Coordination services, and haveargued that it is unnecessary to perform total orderingof requests at the Coordinator replicas, to achievetrustworthy coordination of Web Services BusinessActivities. Rather, it suffices to ensure that messages aredelivered in source order, i.e., the order in which thesender sent them.

This analysis has enabled us to develop a lightweightByzantine fault tolerance (BFT) algorithm that mitigatesthe threats and avoids the runtime overhead associatedwith traditional BFT algorithms. We have implemented

14

the lightweight BFT algorithm and associated mech-anisms, and have incorporated them into the open-source Kandula framework that implements the WS-BA specification and the WS-BA-I extension. The perfor-mance evaluation results obtained using the prototypeimplementation show moderate overhead, especially ina wide-area network, where Web Services Business Ac-tivities are typically deployed.

The research that we have described in this article fitsin well with the recent trend of cloud computing. Manylarge companies, such as Amazon, Google, Microsoft andApple, are now offering cloud services. Some of themmight provide Coordination services for Web Servicesin the cloud. Such Coordination services could enablesmaller companies to offer composite Web Services toprovide value-added services to their customers with-out having to invest heavily in computing infrastruc-tures. The Coordination service providers might findour lightweight BFT algorithm attractive because of itsrelatively low overhead. For the best fault isolation, itis desirable to deploy the Coordinator replicas acrossgeographically separated data centers, which are readilyavailable for the large companies.

ACKNOWLEDGMENT

This work was supported in part by NSF grant CNS08-21319 and NSF grant CNS-1016193, and by a CSUSIgrant from Cleveland State University.

An earlier version of this work was presented atSCC’08 [35].

REFERENCES[1] Apache Axis. http://ws.apache.org/axis/.[2] Apache Kandula. http://ws.apache.org/kandula/.[3] Apache WSS4J. http://ws.apache.org/wss4j/.[4] C. Attiya, D. Dolev and J. Gil. Asynchronous Byzantine consen-

sus. In Proceedings of the ACM Symposium on Principles of Dis-tributed Computing, pages 119–133, Vancouver, British Columbia,Canada, 1984.

[5] M. Castro and B. Liskov. Practical Byzantine fault toleranceand proactive recovery. ACM Transactions on Computer Systems,20(4):398–461, November 2002.

[6] A. Clement, M. Kapritsos, S. Lee, Y. Wang, L. Alvisi, M. Dahlinand T. Riche. UpRight cluster services. In Proceedings of the22nd ACM Symposium on Operating Systems Principles, Big Sky,MT, October 2009.

[7] F. Cristian, H. Aghili, R. Strong and D. Dolev. Atomic broadcast:From simple message diffusion to Byzantine agreement. Informa-tion and Computation, 200–206, 1985.

[8] D. Davis, A. Karmarkar, G. Pilz, S. Winkler and U. Yalcinalp.Web Services Reliable Messaging, Version 1.1, OASIS Standard,January 2008.

[9] D. Dolev. The Byzantine generals strike again. Journal of Algo-rithms, 3(1):14–30, 1982.

[10] D. Dolev. An efficient algorithm for Byzantine agreement withoutauthentication. Information and Control, 52(3), March 1982, 257–274.

[11] D. Dolev and H. R. Strong. Authenticated algorithms for Byzan-tine agreement. SIAM Journal of Computing, 12:656–666, 1983.

[12] H. Erven, H. Hicker, C. Huemer and M. Zapletal. The WebServices-BusinessActivity-Initiator (WS-BA-I) protocol: An exten-sion to the Web Services-BusinessActivity specification. In Pro-ceedings of the IEEE International Conference on Web Services, pages216-224, Salt Lake City, UT, July 2007.

[13] M. Feingold and R. Jeyaraman. Web Services Coordination,Version 1.1, OASIS Standard, July 2007.

[14] T. Freund and M. Little. Web Services Business Activity, Version1.1, OASIS Standard, April 2007.

[15] H. Garcia-Molina, F. Pittelli and S. Davidson. Applications ofByzantine agreement in database systems. ACM Transactions onDatabase Systems, 11(1):27–47, 1986.

[16] Y. Gil and D. Artz. A survey of trust in computer science and thesemantic Web. Journal of Web Semantics, 5:58–71, Elsevier, 2007.

[17] T. Grandison and M. Sloman. A survey of trust in Internetapplications. IEEE Communications Survey Tutorials, 4(4):2–16,2000.

[18] K. Iwasa, J. Durand, T. Rutt, M. Peel, S. Kunisetty and D. Bunting.WS-Reliability 1.1, OASIS Standard, November 2004.

[19] K. P. Kihlstrom, L. E. Moser and P. M. Melliar-Smith. TheSecureRing group communication system. ACM Transactions onInformation and System Security, 4(4):371–406, November 2001.

[20] K. P. Kihlstrom, L. E. Moser and P. M. Melliar-Smith. Byzantinefault detectors for solving consensus. Computer Journal, 46(1):16–35, 2003.

[21] L. Lamport, R. Shostak and M. Pease. The Byzantine generalsproblem. ACM Transactions on Programming Languages and Systems,4(3):382–401, July 1982.

[22] M. Little and A. Wilkinson. Web Services Atomic Transactions,Version 1.1, OASIS Standard, April 2007.

[23] D. Malkhi and M. Reiter. Secure and scalable replication inPhalanx. In Proceedings of the 17th IEEE Symposium on ReliableDistributed Systems, pages 51–58, Lafayette, IN, 1998.

[24] M. Merideth, A. Iyengar, T. Mikalsen, S. Tai, I. Rouvellou andP. Narasimhan. Thema: Byzantine-fault-tolerant middleware forWeb Services applications. In Proceedings of the 24th IEEE Sympo-sium on Reliable Distributed Systems, pages 131–142, Orlando, FL,October 2005.

[25] C. Mohan, R. Strong and S. Finkelstein. Method for distributedtransaction commit and recovery using Byzantine agreementwithin clusters of processors. In Proceedings of the ACM Symposiumon Principles of Distributed Computing, pages 89–103, Montreal,Quebec, Canada, 1983.

[26] L. E. Moser and P. M. Melliar-Smith. Byzantine-resistant total or-dering algorithms. Journal of Information and Computation, 150:75–111, 1999.

[27] L. E. Moser, P. M. Melliar-Smith and N. Narasimhan. TheSecureGroup group communication system. In Proceedings of theIEEE Information Survivability Conference, Volume II, pages 256–279, Hilton Head, SC, January 2000.

[28] A. Nadalin, C. Kaler, R. Monzillo and P. Hallam-Baker. WebServices Security: SOAP Message Security 1.1, OASIS Standard,November 2006.

[29] A. Nadalin, M. Goodner, M. Gudgin, A. Barbir and H. Grangvist.WS-Trust 1.4, OASIS Standard, February 2009.

[30] S. L. Pallemulle, H. D. Thorvaldsson and K. J. Goldman. Byzan-tine fault-tolerant Web Services for n-tier and Service OrientedArchitectures. In Proceedings of the 28th IEEE International Con-ference on Distributed Computing Systems, pages 260–268, Beijing,China, June 2008.

[31] M. Reiter. The Rampart toolkit for building high-integrity ser-vices. Theory and Practice in Distributed Systems, Lecture Notes inComputer Science 938, pages 99–110, Springer-Verlag, 1995.

[32] W. Zhao. Byzantine fault tolerance for non-deterministic ap-plications. In Proceedings of the IEEE International Symposiumon Dependable, Autonomous and Secure Computing, pages 108–115,Columbia, MD, September 2007.

[33] W. Zhao. A Byzantine fault tolerant distributed commit protocol.In Proceedings of the IEEE International Symposium on Dependable,Autonomous and Secure Computing, pages 37–44, Columbia, MD,September 2007.

[34] W. Zhao. BFT-WS: A Byzantine fault tolerant framework for WebServices. In Proceedings of the Middleware for Web Services Workshop,Annapolis, MD, October 2007.

[35] W. Zhao and H. Zhang. Byzantine fault tolerant coordination forWeb Services business activities. In Proceedings of the IEEE Interna-tional Conference on Services Computing, pages 407–414, Honolulu,HI, July 2008.

15

Hua Chai Ms. Chai is a doctoral student inthe Department of Electrical and Computer En-gineering at Cleveland State University (CSU).She received a Master of Science degree inElectrical Engineering in 2009 from CSU. Herresearch interests include fault tolerance com-puting, event streaming processing, and serviceoriented computing.

Honglei Zhang Mr. Zhang is doctoral studentin the Department of Electrical and ComputerEngineering at Cleveland State University. Hereceived a Master of Science degree in ElectricalEngineering in 2007 from Cleveland State Uni-versity. His research interests include Byzantinefault tolerance computing and service orientedcomputing.

Wenbing Zhao Dr. Zhao received the Ph.D.degree in Electrical and Computer Engineeringfrom the University of California, Santa Barbara,in 2002. Currently, he is an associate profes-sor in the Department of Electrical and Com-puter Engineering at Cleveland State University.His current research interests include distributedsystems, computer networks, fault tolerance andsecurity. Dr. Zhao has more than 70 academicpublications.

P. Michael Melliar-Smith Dr. Melliar-Smith is aprofessor in the Department of Electrical andComputer Engineering at the University of Cal-ifornia, Santa Barbara. Previously, he workedas a research scientist at SRI International inMenlo Park. His research interests encompassthe fields of distributed systems and applica-tions, and network architectures and protocols.He has published more than 275 conference andjournal publications in computer science andengineering. He has served on many conference

program committees, and as a reviewer for many conferences andjournals. Dr. Melliar-Smith is a pioneer in the field of fault-tolerantdistributed computing. He received a Ph.D. in Computer Science fromthe University of Cambridge, England.

Louise E. Moser Dr. Moser is a professor inthe Department of Electrical and Computer En-gineering at the University of California, SantaBarbara. Her research interests span the fieldsof computer networks, distributed systems andsoftware engineering. Dr. Moser has authored orcoauthored more than 260 conference and jour-nal publications. She has served as an associateeditor for the IEEE Transactions on ServicesComputing and the IEEE Transactions on Com-puters and as an area editor for IEEE Computer

magazine in the area of networks. She has served as Technical Co-Chair of the 2011 IEEE International Conference on Web Servicesand on many conference program committees. She received a Ph.D.in Mathematics from the University of Wisconsin, Madison.

toward trustworthy coordination of web services business

Documents