Business Process Management JournalEmerald Article: Key performance indicators for traffic intensive web-enabled business processesKa I. Pun, Yain Whar Si, Kin Chan Pau

Article information:

To cite this document: Ka I. Pun, Yain Whar Si, Kin Chan Pau, (2012),"Key performance indicators for traffic intensive web-enabled business processes", Business Process Management Journal, Vol. 18 Iss: 2 pp. 250 - 283

Permanent link to this document: http://dx.doi.org/10.1108/14637151211225199

Downloaded on: 30-04-2012

References: This document contains references to 41 other documents

To copy this document: permissions@emeraldinsight.com

This document has been downloaded 22 times.

Access to this document was granted through an Emerald subscription provided by Emerald Author Access

For Authors: If you would like to write for this, or any other Emerald publication, then please use our Emerald for Authors service. Information about how to choose which publication to write for and submission guidelines are available for all. Additional help for authors is available for Emerald subscribers. Please visit www.emeraldinsight.com/authors for more information.

About Emerald www.emeraldinsight.comWith over forty years' experience, Emerald Group Publishing is a leading independent publisher of global research with impact in business, society, public policy and education. In total, Emerald publishes over 275 journals and more than 130 book series, as well as an extensive range of online products and services. Emerald is both COUNTER 3 and TRANSFER compliant. The organization is a partner of the Committee on Publication Ethics (COPE) and also works with Portico and the LOCKSS initiative for digital archive preservation.

*Related content and download information correct at time of download.

Key performance indicators fortraffic intensive web-enabled

business processesKa I. Pun

Department of Informatics, University of Oslo, Oslo, Norway, and

Yain Whar Si and Kin Chan PauDepartment of Computer and Information Science,

University of Macau, Taipa, Macau

Abstract

Purpose – Intensive traffic often occurs in web-enabled business processes hosted by travel industryand government portals. An extreme case for intensive traffic is flash crowd situations when thenumber of web users spike within a short time due to unexpected events caused by political unrest orextreme weather conditions. As a result, the servers hosting these business processes can no longerhandle overwhelming service requests. To alleviate this problem, process engineers usually analyzeaudit trail data collected from the application server and reengineer their business processes towithstand unexpected surge in the visitors. However, such analysis can only reveal the performance ofthe application server from the internal perspective. This paper aims to investigate this issue.

Design/methodology/approach – This paper proposes an approach for analyzing keyperformance indicators of traffic intensive web-enabled business processes from audit trail data,web server logs, and stress testing logs.

Findings – The key performance indicators identified in the study’s approach can be used tounderstand the behavior of traffic intensive web-enabled business processes and the underlyingfactors that affect the stability of the web server.

Originality/value – The proposed analysis also provides an internal as well as an external view ofthe performance. Moreover, the calculated key performance indicators can be used by the processengineers for locating potential bottlenecks, reengineering business processes, and implementingcontingency measures for traffic intensive situations.

Keywords Key performance indicators, Traffic intensive web application, Audit trail analysis,Web server log, Stress testing, User studies, World wide web

Paper type Research paper

1. IntroductionWeb-enabled applications are widely used in service industries, government portalsand e-commerce companies. These applications usually consist of web servers andapplication servers for hosting their business processes. Many of these applications areexecuted in workflow management systems (WfMS) for separating business logic fromimplementation details. A WfMS is “a system that completely defines, manages, andexecutes workflow through the execution of software whose order of execution isdriven by a computer representation of workflow logic” (Workflow Management

The current issue and full text archive of this journal is available at

www.emeraldinsight.com/1463-7154.htm

The authors would like to thank editors and anonymous referees for their insightful comments,and are grateful for the financial support received from the University of Macau.

BPMJ18,2

250

Business Process ManagementJournalVol. 18 No. 2, 2012pp. 250-283q Emerald Group Publishing Limited1463-7154DOI 10.1108/14637151211225199

Coalition (WfMC), 1995). It is often considered as a procedural automation of a businessprocess based on process-oriented model.

Web applications, which are commonly used in travel industry and governmentagencies for providing services to the customers, are considered as traffic intensiveweb-enabled business processes since the number of users can fluctuate dramaticallywithin a short time. Specifically, applications under this category are likely toencounter flash crowds from time to time. A flash crowd (Chen and Heidemann, 2005)is a surge of unanticipated, rapid, huge simultaneous requests and accesses to aparticular web site, and we consider it as an extreme case of intensive traffic situation.This can make a web application virtually unreachable since the web server can nolonger handle all incoming requests. These phenomena could be anticipated insituation such as webcast of major world events. Flash crowds also happen in somepreviously unpopular web sites which become extraordinarily popular after beinglinked or mentioned in a popular web site. This circumstance is called Slashdot effect(Chen and Heidemann, 2005). Though WfMS provides good support for businessprocesses, those processes are expected to be well-structured (van der Aalst et al., 2009)and do not require high flexibility to handle exceptions. There is a lot of discussionabout the flexibility issues in WfMS (van der Aalst et al., 2009; Agostini and deMichelis, 2000; Sadiq et al., 2000). In addition, many approaches have been proposed totackle inadaptability and inflexibility of workflow system to change (Borgida andMurata, 1999; Kumar and Zhao, 1999; Marinescu, 2002; Mans et al., 2009). Changes inworkflow can be caused by not only revision of system specification but alsounexpected events. In particular, unexpected events can be exceptional intensive traffic(like flash crowd) for web applications. During traffic intensive situations, especially inwhich potential flash crowds may occur, servers which are used to host trafficintensive web-enabled business processes might even be unreachable since they can nolonger handle sudden overwhelming service requests.

To alleviate this problem, business process engineers usually analyze the audit traildata to pinpoint the bottleneck locations in these workflows. An audit trail is “an electronicarchive in which the history of a workflow is recorded” (van der Aalst and van Hee, 2000).It records the state changes of each process in the workflow as well as the interactionsbetween application server and client computers at operational level. However, analysisperformed on audit trail data of an application server can only provide an internal view ofthe situation. Specifically, analyzing audit trail can only reveal resource allocation and thehistory of execution of a business process at an operational level.

In web applications, activities of the web servers are usually stored in log files.A typical web server log usually contains information about the history of pagerequests within a specific period of time, HTTP status code, and referrers. Althoughweb server logs can be analyzed (AWStats, n.d.) to extract the transactions detailsbetween the application server and client computers, the statistics extracted from thelog files also only provides an internal view of the web server.

Most importantly, statistics such as time required for the web server to respond toeach hit requested from a thread and outcomes of the requests (e.g. successful or fail)during flash crowd situation are not available in audit trail data and web server logs.Therefore, calculating such statistics requires analyzing logs based on otherperformance evaluation techniques which take into account user’s (external)perspective. Stress testing (Microsoft Web Application Stress Tool, n.d.; WAPT,

KPIs forbusiness

processes

251

Web Application Load, Stress, and Performance Testing, n.d.) have been widely usedto simulate large numbers of users against a web application in mimicking the trafficintensive situation. Stress testing can imitate as client computers to generate hugenumber of simultaneous requests to the server. Stress testing can also capture thestatistics such as time taken for the web server to respond to each hit (requested from athread) and the results of the requests (success or fail) made by the users. Therefore,results from stress testing can be used to provide external view of the application.However, unlike audit trail analysis, one of the disadvantages of stress testing is that itdoes not consider underlying business logic embedded within these applications.

To overcome the deficiency of WfMS in the aspect of inflexibility to change,especially in the intensive traffic situation under which web applications might beunreachable, we identify potential bottlenecks in the existing WfMS with a set ofperformance indicators. Process engineers can reconfigure the workflow schema andtest it with workflow simulations before it is applied to the existing web applications.Process engineers can also use the performance indicators to find out the off-peakperiod during which system maintenance is carried out. To achieve this, we proposean approach for automatically calculating the key performance indicators (KPIs) fortraffic intensive web application by incorporating analysis results from audit traildata, web server log, and stress testing. Integrating results from these analysesprovides an internal as well as an external view of the application’s performance.The performance indicators identified in our proposed approach can be used tounderstand the behavior of traffic intensive web application and the underlyingfactors that affect the stability of the web server. Moreover, the calculated KPIs canbe used by the process engineers to exactly pinpoint the potential bottlenecks in theapplication. It is crucial to locate the potential bottlenecks in the workflow schemawhich might cause servers of web applications unreachable before intensive trafficoccurs, especially for the case of flash crowds and Slashdot effects as they are usuallyunpredictable. The identification of potential bottlenecks helps process engineersprecisely reconfigure the workflow schema in the application so that the possibility ofchanges is minimized.

This paper is organized as follows. Section 2 discusses the background informationon audit trail, web server log, and stress testing. Section 3 first gives the overview ofthe system. Then, it discusses the calculation of the KPIs from individual analysis. InSection 4, we show how KPIs can be calculated from example traffic intensiveweb application. Section 5 discusses the related literatures. In Section 6, we concludethe paper with future work.

2. Background informationIn this section, we detail three key preprocessing tasks for calculating KPIs. Thesetasks include analyzing audit trail, web server log, and stress testing statistics.

2.1 Audit trail analysisAudit trail is commonly used to track customer activities in e-commerce applicationssince it contains information about relevant transactions and interactions betweenapplication server and client computers. The initial visit of a customer as well as eachsubsequent action is usually recorded in audit trail. Moreover, audit trail also capturesthe state changes of the process instances which are being executed in a workflow.

BPMJ18,2

252

The WfMC (1998) proposed the interface 5 “Audit Data Specification” which statesthe audit trail information for a WfMS. To better understand the audit trail data,we have surveyed three open source WfMS (Yet Another Workflow Language(YAWL), n.d.; JBoss jBPM, n.d.); OpenWFE, n.d.) and compare them with the interface5 “Audit Data Specification”. Our survey was conducted from two perspectives; dataand event perspectives. The results of the comparison are depicted in Tables I and II.

In Table I, workflow control data (internal data) is managed by WfMS and containsinformation about the current process instances, activity instances and workitems.Workflow relevant data (case data) is related to individual process instance and is used byWfMS to determine the flow of current process. Workflow application data (applicationobject) is application specific and manipulated directly by the invoked applications. InTable II, we compare the types of events which are recorded in YAWL, jBPM, OpenWFE,and WfMC interface 5. From this comparison, we can see that WfMC interface 5 providesmore complete description of event types. Based on this survey, we implement a genericaudit trail analyzer for integrating analysis results into our environment.

Data field YAWL 2.0 jBPM 2.0OpenWFE1.5.5 WfMC interface 5

Workflowcontrol

Processdefinition

specification processDefinition wfd_id ProcessDefinitionID

data Parent parent_id parent N/A InitialProcessInstanceIDInitiator who_

started_meactorID participamt_

nameUserIDRoleID

Processinstance ID

caseid id wf_instnace_id

CurrentProcessInstanceID

Activityinstance ID

taskid id expression_id ActivityInstanceID

Workitem ID itemid id workitem_id WorkitemIDWorkflowrelevant

Attribute datatype

paramtype converter a_type AttributeType

data Attribute dataname

paramname name N/A AttributeName

Attribute datavalue

paramvalue value a_value PreviousAttributeValueNewAttributeValue

Workflow Object ID N/A N/A N/A N/Aapplicationdata

Object reference N/A N/A N/A N/A

Statetransition

Actor who_started_me

actorID participamt_name

UserIDRoleID

Event eventname eventType eventCode EventCodeTime timestamp date date, hi_date TimestampProcessinstance state

casedata startState N/A PreviousProcessStateNewProcessState

Activityinstance state

N/A superState N/A PreviousActivityStateNewActivityState

Workitem state status state fields N/A WorkitemState

Note: N/A, not available

Table I.Comparison tableof audit trail data

KPIs forbusiness

processes

253

In Table I, Workflow control data (internal data) is managed by WfMS and containsinformation about the current process instances, activity instances and workitems.Workflow relevant data (case data) is related to individual process instance and is usedby WfMS to determine the flow of current process. Workflow application data(application object) is application specific and manipulated directly by the invokedapplications. In Table II, we compare the types of events which are recorded in YAWL,jBPM, OpenWFE, and WfMC interface 5. From this comparison, we can see that WfMCinterface 5 provides more complete description of event types. Based on this survey, weimplement a generic audit trail analyzer for integrating analysis results into ourenvironment.

In this section, we briefly describe the design of the audit trail data analysis module.In general, audit trail data captured in WfMS is in unprocessed state. Audit trailchronologically records the resource allocation, as well as the history of execution andresult of a business process at operational level. A WfMS captures the workflow modelof a business process by defining a process definition which consists of a set ofactivities.

During execution of the WfMS, an initiation of a process definition will create aprocess instance. Each process instance contains a set of activity instances according

PI AI WI Event YAWL 2.0 jBPM 2.0OpenWFE1.5.5 WfMC interface 5

X X X Create Enabled task_create þ INC(incoming)

WMCreatedProcessInstanceWMChangedActivityInstanceStateWMChangedWorkitemState

X X X Start Executing process_starttask_start

þ STA(start)

WMStartedProcessInstanceWMStartedWorkItem

X – – Restart N/A N/A N/A WMChangedProcessInstanceX X – Suspend Suspended N/A N/A WMChangedProcessInstance

WMChangedActivityInstanceStateX X – Resume N/A N/A N/A WMChangedProcessInstance

WMChangedActivityInstanceStateX X X Complete Complete

ForceCompleteprocess_endtask_end

þ END(end)

WMCompletedProcessInstanceWMCompletedActivityInstanceWMCompletedWorkItem

X – – Terminate N/A N/A N/A WMTerminatedProcessInstanceWMTerminatedActivityInstance

X X X Abort Cancelled N/A þ CNF(cancelFlow)

WMAbortedProcessInstanceWMAbortedActivityInstance

– – X Select/assign

Fired task_assign þ DIS(dispatch)

WMSelectedWorkitemWMAssignedWorkItem

– – X Reassign N/A N/A N/A WMReassignedWorkItem– – X Reject N/A N/A N/A WMRejectedWorkItemX X X Other Deadlocked

Is parentsubprocess(create/end)

N/A remote process invocationremote process controlevent wait and receiptattribute change

Notes: PI, process instance; AI, activity instance; WI, workitem; X, applicable; – , not applicable;N/A, not available

Table II.Comparison tableof event types

BPMJ18,2

254

to the set of activities which consists the relevant process definition. These activityinstances are executed one by one, or sometimes in parallel to accomplish the processinstance. Whenever an activity instance is started, one or more related workitems willbe generated. After a workitem is created, it will be allocated to a resource in the WfMSfor execution. Once the workitem is allocated, it will wait at the queue of the assignedresource until it is available to start working on it. Changes of states in the lifecycle ofprocess instance, activity instance and workitem are record in their history separately.We conceptualize the above relation in Figure 1.

During the execution of a workflow, a workitem can be in one of the seven possiblestates (enabled, executing, complete, forceComplete, cancelled, fired, and deadlocked).The lifetime of a workitem can be defined as a set of tuples consisting of states andtime points:

{ðs1; start1; end1Þ; . . . ; ðsn; startn; endnÞ};

where si, starti and endi are the state, start time, and end time of the ith state of aworkitem. We can define the lifetime of both activity instance and process instance in asimilar way. We define the following notations:

W ¼ the history of the states in the lifetime of a workitem.

w ¼ a tuple in W.

A ¼ the history of the states in the lifetime of an activity instance.

a ¼ a tuple in A.

P ¼ the history of the states in the lifetime of a process instance.

p ¼ a tuple in P.

Functions for accessing the elements of the tuple are defined in Table III.2.1.1 Workitem. Based on these functions we extract life duration, allocation time,

total queuing time, processing time, and idle time of a workitem as follows:

Figure 1.Relation of processdefinition, process

instance, activity instance,and workitem

ProcessDefinition

1 n

n 1Workitem

History

Generates

History

ActivityInstance

Contains

ProcessInstance

History

Creates

KPIs forbusiness

processes

255

Life duration. The duration of any state in the lifetime of a workitem can be defined asfollows:

wdurationðwÞ ¼ wendðwÞ2 wstartðwÞ ð1Þ

The life duration of a workitem is the sum of the duration of each state in the history oflifetime:

wlifeðW Þ ¼w[W

XwdurationðwÞ ð2Þ

Allocation time. A workitem is in enabled state when it is created but not yet allocatedto any resources. Allocation time of a workitem is the total duration of the workitem inenabled state:

wallocateðW Þ ¼w[W^wstateðwÞ¼enabled

XwdurationðwÞ ð3Þ

Total queuing time. After a workitem is allocated to a resource, the state of theworkitem is changed into fired. The workitem may also need to queue until theresource is available. During its lifetime, the workitem may have more than one firedstate since it can be reallocated to other resources. The total queuing time of aworkitem is the sum of the duration of the workitem in the fired state:

wqueueðW Þ ¼w[W^wstateðwÞ¼fired

XwdurationðwÞ ð4Þ

Processing time. The processing time of a workitem is the total amount of time when itis in an executing state:

wprocessðW Þ ¼w[W^wstateðwÞ¼executing

XwdurationðwÞ ð5Þ

Idle time. The idle time of a workitem is the total amount of time when it is not in anexecuting state:

widleðW Þ ¼w[W^wstateðwÞ–executing

XwdurationðwÞ ð6Þ

Life duration, processing time, and idle time of both activity instance and processinstance can be defined in a similar way[1] as follows.

2.1.2 Activity instance. Life duration. The duration of any state in the lifetime of anactivity instance can be defined as follows:

aduration ðaÞ ¼ aend ðaÞ2 astart ðaÞ ð7Þ

Workitem Activity instance Process instance

s wstate((s,start,end)) astate((s,start,end)) pstate((s,start,end))start wstart((s,start,end)) astart((s,start,end)) pstart((s,start,end))end wend((s,start,end)) aend((s,start,end)) pend((s,start,end))

Table III.Functions to access theelement of the tuples ofstate history of a lifecyclein PI, AI, and WI

BPMJ18,2

256

The life duration of an activity instance is the sum of the duration of each state in thehistory of lifetime:

alife ðAÞ ¼a[A

Xaduration ðaÞ ð8Þ

Processing time. The processing time of an activity instance is the total amount of timewhen it is in an executing state:

aprocess ðAÞ ¼a[A^astateðaÞ¼executing

Xaduration ðaÞ ð9Þ

Idle time. The idle time of an activity instance is the total amount of time when it is notin an executing state:

aidle ðAÞ ¼a[A^astateðaÞ–executing

Xaduration ðaÞ ð10Þ

2.1.3 Process instance. Life duration. The duration of any state in the lifetime of aprocess instance can be defined as follows:

pduration ð pÞ ¼ aend ð pÞ2 astart ð pÞ ð11Þ

The life duration of a process instance is the sum of the duration of each state in thehistory of lifetime:

plife ðPÞ ¼p[P

Xaduration ð pÞ ð12Þ

Processing time. The processing time of a process instance is the total amount of timewhen it is in an executing state:

pprocessðPÞ ¼p[P^pstateð pÞ¼executing

Xpdurationð pÞ ð13Þ

Idle time. The idle time of a process instance is the total amount of time when it is not inan executing state:

pidle ðPÞ ¼p[P^pstateð pÞ–executing

Xpduration ð pÞ ð14Þ

Based on these formulas, we are able to derive life duration, allocation time, totalqueuing time, processing time, and idle time of workitems, activity instances, andprocess instances. In addition, we can also derive the statistics about the states(successful or cancelled) and distribution of the number of process instances, activityinstances, and workitems for the recorded period from audit trail analysis. Summary ofstatistics obtained from audit trail analyzer is summarized in Table IV. The statisticspresents the overall performance of the workflow.

KPIs forbusiness

processes

257

2.2 Analyzing web server logServices provided by a web server are generally delivered through pages(e.g..html,.jsp). A hit (or a request) from a client computer can be an attempt toaccess any object (e.g..jpg) from a web page. Many accesses to a page can be made fromone unique IP address. Some accesses are grouped into a session which lasts for aspecific duration of time. In a web log analyzer, a session is referred by a visit, and theunique IP address associated with this visit is called a unique visitor. A visitor mayinitiate multiple visits and a visit may include accessing of multiple pages.

A web server log can save the above information as well as the history of pagerequests recorded during the log duration. The history contains the generalinformation about the transaction between a web server and client computers,including the IP address of the client computers, timestamp of a page request, objectsaccessed, and so on. W3C (Workflow Management Coalition, 1998) standardizes thegeneral format of a web server log. A web server log can be used to derive the statisticsof the activities between the server and client computers. Table V summarizes thestatistics obtained from analyzing a web server log.

Analyzing a web server log not only reports the general statistics about pagerequests, but also provides an internal view of the server. Specifically, analysis canidentify the rush hours of a day, as well as the peak day within the web server logperiod in which traffic intensive situations may have occurred. In addition, analysiscan also identify whether a page request is successful or not by checking the labelingsuch as HTTP status codes (e.g. 404) (W3C, n.d.). By combining the information onpeak hours with the status of each page request, we can observe the performance of theweb server under intensive traffic. This information is crucial in analyzing the

Statistics from audit trail

A1 Number of successful process instancesA2 Number of cancelled process instancesA3 Number of process instances of a process definition created during the log periodA4 Number of activity instances of a process definition created during the log periodA5 Number of workitems of a process definition generated during the log periodA6 Number of resources assigned to handle the process instances created from a process definition

during the record periodA7 Life duration of a process instanceA8 Idle time of a process instanceA9 Processing time of a process instanceA10 Life duration of an activity instanceA11 Idle time of an activity instanceA12 Processing time of an activity instanceA13 Life duration of a workitemA14 Queuing time of a workitemA15 Allocation time of a workitemA16 Idle time of a workitemA17 Processing time of a workitemA18 Number of process instances, activity instances and workitems in each hour during the log

periodA19 Number of process instances, activity instances and workitems in each day during the log periodA20 Time duration of each state of a workitem

Table IV.Summary of the statisticsfrom audit trail

BPMJ18,2

258

performance of a traffic intensive web application where flash crowds occur frequently.Based on the analysis result, webmasters may also consider allocating more resources(e.g. bandwidth, shutting down non-critical functions) to the application server.

2.3 Stress testingWhile web server logs provide an internal view of the performance of a web server,statistics from stress testing present an external view. Load testing (Menasce, 2002) isoften used to assess the performance of a web server in handling expected workload byrunning a specified set of scripts emulating the behavior of customers. Stress testing isa kind of load testing in which the load placed on the server is raised beyond the usuallevel. In addition, stress testing can be used to estimate the workload limit of a webserver. It is also used to evaluate the capability of a web server in handling massivesimultaneous requests.

Stress testing can simulate intensive traffic by generating a large number ofservices requests (concurrent connections) simultaneously. The connections arecreated from one or more unique IP addresses (test clients). Each connection iscontrolled by a thread. A thread can mimic to be a real user in requesting services froma web application. Similar to a web server log, a hit (or request) in a stress testing canbe considered as an attempt to access any object (e.g..jpg) from the web page. Wecollect statistics from stress testing by simulating traffic intensive situations withstress testing tools [Microsoft Web Application Stress Tool, n.d.; WAPT,Web Application Load, Stress, and Performance Testing, n.d.). Statistics which canbe obtained from stress testing are summarized in Table VI.

3. Automated key performance calculation3.1 System designIn this research, we propose an approach for automated analysis of KPIs for trafficintensive web-enabled business processes by utilizing statistics extracted from web

Statistics obtained from web server log

L1 Web server log periodL2 Number of unique visitors recorded in L1L3 Number of visits recorded in L1L4 Number of pages recorded in L1L5 Number of hits/requests recorded in L1L6 Average number of hits for each page recorded L1L7 Number of unique visitors in each day of L1L8 Number of visits in each day of L1L9 Number of pages accessed in each day of L1L10 Number of hits made in each day of L1L11 Number of unique visitors accessed in each hour of L1L12 Number of pages accessed in each hour of L1L13 Number of hits made in each hour of L1L14 Number of visits for various session ranges (e.g. 5-15 min)L15 Number of times of a page is viewed L1L16 Number of hits with 404 error HTTP code recorded for all pages in L1L17 Number of pages a unique visitor accessed in L1L18 Number of hits a unique visitor made in L1

Table V.Summary of the statistics

from web server log

KPIs forbusiness

processes

259

server log analysis, audit trail analysis, and stress testing. These indicators minimize thepossibility of changes in workflow schema due to intensive traffic by preciselypinpointing the potential bottlenecks. Figure 2 shows the overall design of our approach.

First, web server logs and audit trail data are recorded during the enactment of anexisting traffic intensive web application. Next, an audit trail analysis module and aweb server log analysis module parse the audit trail data and web server log,respectively. Meanwhile, stress testing is carried out with stress testing tools tosimulate a traffic intensive environment so as to capture the performance statistics ofthe web application under intensive traffic which is beyond normal level. The result ofthe simulation is further analyzed for vital statistics. Finally, the statistics from theseanalysis modules are integrated in the automated calculation module to derive the KPIsof the traffic intensive web-enabled business processes. With the KPIs, processengineers can perform revision on the workflow schema. The revised schema can beexecuted in workflow simulation[2] to evaluate the performance of the correspondingweb application under intensive traffic after the revision. The results can be used as abenchmark for improving the workflow of the existing web application.

The steps undertaken in our research can also be framed with the three cycles ofdesign science described by Hevner (2007). In Design Science Research Cycles, therelevance cycle connects the contextual environments and design science activities byproviding the requirements for the science research, and evaluating the researchresults with certain criteria. The rigor cycle provides foundational knowledge to

Statistics from stress testing

S1 Run lengthS2 Number of unique test clientsS3 Number of hits with various HTTP status codes (e.g. 404) during the stress testingS4 Number of concurrent connectionsS5 Number of hits/requestsS6 Number of requests per secondS7 Average time taken for the server to respond

Table VI.Summary of the statisticsobtained from stresstesting

Figure 2.System design

Module forAutomatedCalculation

of KeyPerformance

Indicators

KeyPerformance

Indicators

Web server logstatistics

Web logAnalysis

WebServer Log

Retreiveweb server log

&audit trail

Audit Trailstatistics

Audit TrailAnalysis

Audit TrailTraffic

IntensiveWeb

Application

WorkflowSchemaRevision

Modulefor

WorkflowSimulation

BPMJ18,2

260

science research. Any new knowledge obtained from design science activities areadded to the knowledge base. The design cycle iterates between the constructions andevaluation of design artifacts, as well as the design refinement.

The process adopted in our design of identifying KPIs is analogous in several aspectsto the Design Science Research Cycles. For instance, in the relevance cycle, we identifythe needs for automated analysis of KPIs for traffic intensive web-enabled businessprocesses. In this context, ability for locating potential bottlenecks in the businessprocess via KPIs can be considered as one of the main requirements input. In the designcycle, we propose a bottom-up approach for aggregating the basic statistics collectedfrom web server log, audit trail, and stress testing into a set of KPIs. As for the rigorcycle, the proposed approach improves the existing business process performanceanalysis methods by providing means for automatic calculation of the KPIs for locatingpotential bottlenecks. Based on these KPIs, process engineers may reconfigure theschema to reduce potential performance degradation. The effectiveness of resulted KPIsfrom the design cycle is evaluated on an application example in Section 4.

3.2 Key performance indicatorsAnalysis results from audit trail and web server logs provide an internal/server view ofthe application whereas a stress testing provides and external/client view. Therefore,the indicators derived from the analysis results provide different perspectives ofthe performance of the application that helps process engineer accurately reconfigureworkflow schema. Though there are many other indicators can be derived from theanalysis results described from Section 2, we decide five KPIs for traffic intensiveweb-enabled business processes as follows:

K1. Workload limit with stable performance.

K2. Hourly performance of a system.

K3. Daily performance of a system.

K4. Problems in process definition.

K5. Identification of cancelled process instances or activity instances.

We find these indicators are crucial to our problem because they can accurately presentthe behavior of a web application under traffic intensive situation. While K1approximates the workload limit of a web server under which stable service isprovided, K2 and K3 pinpoint peak hour(s) and day(s) during the log period. K4 showsthe potential problems in the process definition of a web application which are causedby client or server error. Finally, K5 indicates the potential process instancecancellation in the workflow due to server overload.

These performance indicators are outlined in detail in the following sections.K1: workload limit with stable performance. The statistics used for calculating K1 is

summarized in Table VII. Statistics from web server log (L2, L3, L4, and L5) provide aninternal view of the performance of the web server whereas statistics from a stresstesting (S2, S4, and S5) provides an external view. In general, changes in number ofpages visited and hits made will proportionally increase or decrease the number ofprocess instances and activity instances (A1 and A2). However, when the trafficis extremely busy, or even flash crowd situations occur, massive number of concurrent

KPIs forbusiness

processes

261

requests may reverse the proportion. In this case, time taken to respond the incomingrequests (S7) may increase. In the worst case, the web server will be unreachablebecause of system failure. This will lead to decrease in the number of successfulprocess instance (A1) and increase in the number of cancelled process instance (A2). Byanalyzing these entries, we can estimate the workload limit of a web server as well asthe acceptable level of concurrent connections.

This performance indicator is calculated based on the number of successful andfailed process instances, and the number of concurrent visits within a specific timeslot.The duration of the timeslot can be defined by the analyst. The timeslot should beequivalent to the average time required to complete the client application.

Workload limit of a web server can be deduced according to the followingalgorithm:

Begin/ * Initialize the variables */sumL4 ¼ sumL5 ¼ sumS5 ¼ sumS7 ¼ 0

FOR EACH timeslot iIF:

/ * More than one successful process instance is completed for each concurrentvisit recorded in both web server log and stress testing */

A1iL3i

$ 1

� �^

A1iS4i

$ 1

� �

THEN:The stable performance of the web server is maintained/ * Sum up the pages visited, hits and response time of all the timeslot in whichthe performance of the web server is stable */sumL4 þ ¼ L4i

sumL5 þ ¼ L5i

sumS5 þ ¼ S5i

sumS7 þ ¼ S7i

ENDIFENDFOR

/ * Find the average of pages visited, hits and response time recorded in thosetimeslots in which the performance of the web server is stable. */

Audit trail A1 Number of successful process instancesA2 Number of cancelled process instances

Web server log L2 Number of unique visitorsL3 Number of visits recordedL4 Number of pages visited recordedL5 Number of hits/requests recorded

Stress testing S7 Number of unique test clientsS8 Number of concurrent connectionsS9 Number of hits/requests madeS10 Time taken for the server to respond

Table VII.Statistics used for K1

BPMJ18,2

262

avgL4 ¼ AVERAGE(sumL4)avgL5 ¼ AVERAGE(sumL5)avgS5 ¼ AVERAGE(sumS5)avgS7 ¼ AVERAGE(sumS7)

FOR EACH timeslot iIF:

A1iL3i

, 1

� �^

A1iS4i

, 1

� �

IF:/ * For timeslot i, the number of pages, hits and response time recorded in theweb server log and the stress testing is greater than the respective averagevalues. */

ððL4i . avgL 4Þ ^ ðL5i . avgL 5ÞÞ ^ ððS5i . avgS 5Þ ^ ðS7i . avgS 7ÞÞ

THEN:The performance of the web server is degraded

ELSE:The performance of the web server cannot be concluded

ENDIFENDIF

ENDFOREnd

K2: hourly performance of a system. Statistics used for calculating K2 is summarized inTable VIII. From Table VIII, A18, L11, and L12 can be used to analyze performance ofthe web server in more detail. For instance, the increase in the number of pagesaccessed should result similar magnitude of increase in number of process instancesand activity instances. However, once the number of pages accessed reaches a certainthreshold, the performance of the web server may begin to degrade due tooverwhelming number of concurrent accesses. Therefore, any decline in the number ofprocess instances and activity instance may be considered as a signal to theperformance issue. In addition, from L11 and L12, process engineer can determine peakand off-peak hours of the application for rescheduling available resources.

K3: daily performance of a system. Statistics used for calculating K3 is summarizedin Table IX. From A19, L7, L8, L9, and L10, we can identify which day of the week ormonth has the highest recorded traffic. These entries can also reveal any slack orbreakdown in the application server operation.

K4: problems in process definition. Statistics used for calculating K4 is summarizedin Table X. A2 represents the number of total cancelled process instances, while L15

Audit trail A18 Number of process instances, activity instances and workitems in eachhour during the log period

Web server log L11 Number of pages accessed in each hour during the log periodL12 Number of hits made in each hour during the log period

Table VIII.Statistics used for K2

KPIs forbusiness

processes

263

and S3 represent various HTTP codes occurred during the performance evaluation.Some of those HTTP codes indicate client or server error in accessing the page andmay lead to failure in the workflow. From the client and server error HTTP statuscodes (e.g. 404), we can determine the nature of errors. Furthermore, these records mayalso indicate frequent occurrence of certain errors. Sometimes, the number of hits to theserver recorded by web server log and stress testing are different since web server logalso captures the hits within the server where the internal web services communicatewith each other. This difference does not cause major impact on the performanceindicator as this type of hits cannot be captured by stress testing.

Performance indicator problems in a process definition can be calculated based onthe following algorithm:

BeginFOR EACH HTTP status code i recorded

IF:codei is (client error k server error)

THEN:tmp1 ¼ number of hits with codei extracted from L15tmp2 ¼ number of hits with codei extracted from S3IF:/ *

codei, which causes either client errors or server errors, occupies majorproportion of hits with HTTP status code. C is the upper limit of proportion ofhits with a specific code (codei).

*/

tmp1

A2$ C

� �^

tmp2

A2$ C

� �;C is a constant

THEN:Problems in a specific part of process definition may be identified according tocodei

ENDIF

ENDIF

Audit trail A2 Number of cancelled process instancesWeb server log L15 Number of hits with various HTTP status code recorded in the log periodStress testing S6 Status of each hits (e.g. 404)

Table X.Statistics used for K4

Audit trail A19 Number of process instances, activity instances and workitems in eachday during the log period

Web server log L7 Number of unique visitors in each day of the log periodL8 Number of visits in each day of the log periodL9 Number of pages accessed in each day of the log periodL10 Number of hits made in each day of the log period

Table IX.Statistics used for K3

BPMJ18,2

264

ENDFOREnd

The upper limit C can be input by the analyst.K5: identification of cancelled process instances or activity instances. A process

instance will be cancelled by the web application if the related activity instance fails. Ina web-based workflow application, an activity instance is usually presented in a webpage at the client side. When a web server is overloaded, timeout may occur and theclient may receive “page not found” response. This failure is usually represented by anHTTP status code (404 error). It indicates that requested resource could not be found atthe moment but may be available again in the future subsequent requests to the samepage by the client are usually allowed in such cases. By linking A2 and L16, an activityinstance can be associated with a corresponding URL of a page in which it isembedded. Statistics used for calculating K5 is summarized in Table XI.

From such relating, we can determine the reason behind the cancellation of aprocess instance. For instance, a process instance could be cancelled because a certainpage cannot be found or lack of response from the server. The indicator K5 identifiesthe potential process instance cancellation in the workflow due to 404 error.

This indicator can be calculated based on the following algorithm:

BeginFOR EACH page i recorded with 404 error HTTP status code in L16

tmp ¼ number of hits with 404 error HTTP status code recorded for pagei

extracted from L16IF:/ * page i has a large number of hits involving 404 error HTTP status code whichcauses either client errors or server errors. C is the upper limit of proportion ofhits with 404 error HTTP code recorded for a specific page (pagei). */

tmp

A2$ C

� �;C is a constant

THEN:The activity instance which is carried out through pagei may be the potentialbottleneck of a process definition.

ENDIF

ENDFOREnd

The upper limit C can be set by the analyst.

4. Application example – an online course enrollment systemThe proposed approach is demonstrated on an example online enrollment workflowapplication from a university. To collect the audit trail data of the enrollment system,

Audit trail A2 Number of cancelled process instancesWeb server log L16 Number of hits with 404 error HTTP code recorded for all pages

in the log periodTable XI.

Statistics used for K5

KPIs forbusiness

processes

265

we developed our own version of workflow using YAWL Engine (YAWL, n.d.) on aweb server. The prototype system for automated calculation of KPIs (includinggenerating charts) was developed based on JAVA (Developer Resources for JavaTechnology, n.d.) programming language with Eclipse (n.d.), Microsoft WebApplication Stress Tool (n.d.), and AWStats (n.d.).

4.1 Workflow schemaA workflow schema is an abstraction of procedural steps in carrying out tasks in abusiness process. It also describes any resources involved as well as input and outputinformation required in each step. Workflow schemas can be modeled in variousformat (YAWL, n.d.; Business Process Modeling Notation (BPMN), n.d.; BPEL for WebServices (BPEL4WS), n.d.), XML Process Definition Language (XPDL), n.d., etc.). Theworkflow schemas for traffic intensive web-enabled business processes are inherentlydifferent from normal enterprise wide application since in traffic intensive web-enabledbusiness processes, a number of navigation controls (e.g. clicking the embedded HTMLlink of a different page, returning to the previous page, or timeout) are needed to beembedded in each page. Figures 3-6 show an example of a workflow schema of anonline course enrollment system. The annotation of the icons used in this example isdescribed in Table XII.

Figure 3 shows the overview of the workflow of a course enrollment system. Oncethe system starts, the flow will be directed to Login process. Login process accepts theinput flow from either Start System process or the conditional process Login

Figure 3.Overview of an onlinecourse enrollment systemworkflow

Figure 4.Sub-process enrollment

BPMJ18,2

266

Succeeded? based on a XOR-join. A user of the enrollment system has to login withhis/her username and password to use the service. The Login process will be repeateduntil the user is authenticated by the system. When the Login process is successful,the user can choose her/her next course of action in Decide Action task. The users canLogout, Close Browser, or Start Course Enrollment process. Note that Logout actionproperly completes the current process instance whereas Close Browser action onlyterminates the process instance. Flows from these processes are originated from DecideAction process with XOR-split and merged by an intermediate process with XOR-join.If no action is taken after a specified period, the flow as well as the current processinstance will be automatically terminated. Convergence at this intermediate processbrings an end of the current process instance.

If the user chooses the Start Course Enrollment action, the Enrollment Processworkflow (Figure 4) will be executed. This process is a composite process which

Figure 5.Sub-process take course

Figure 6.Sub-process drop course

Manual execution by a user

An intermediate process to join or split incoming flow

Timer

Automatic execution by a system

Table XII.Annotation of the icons

KPIs forbusiness

processes

267

consists of several sub-processes. Once the Enrollment Process is executed, the user isrequired to decide his/her next course of action within a predefined duration in DecideAction task. Four actions (Take Course, Logout, Close Browser, Drop Course) areavailable for selection. Only Logout task completes the current process instanceproperly and the remaining tasks terminate the instance without completion. If noaction is taken after a specified period, the flow as well as the current process instancewill be automatically terminated. Take Course and Drop Course action could beperformed several times by the user. This situation is shown as a flow connected backto the Decide Action task in Figure 4.

The composite process Take Course is shown in Figure 5. In this process, the userinputs the information of courses in Input Course Info task. The conditional process AllPrerequisites and Requirements Fulfilled? is used to decide if the user should proceedto Process Enrollment or go back to Input Course Info, depending on fulfillment of thedefined conditions. The user can repeat the above procedure to enroll another course, orto quit this process.

The user can also drop an enrolled course by choosing the Drop Course process(Figure 6). First, the user has to select an enrolled course with Select Course(s) processand to drop the selected course with Click Drop Button process. Multiple courses can bedropped at one time or by repeating the procedure.

One of the potential bottlenecks in this example is in the Take Course processshown in Figure 5. The conditional process All Prerequisites and RequirementsFulfilled? and the sub-process Process Enrollment require large amount of processingpower from the web server since these tasks have to retrieve the academic history ofthe user and all relevant details of the courses. Therefore, additional resources shouldbe allocated to these two processes to maintain the acceptable performance level of theserver. In this example, we have explicitly modeled the actions of clicking buttons andlinks as processes. Additional processes are also used to model the action of closing theweb browser of the application as well as the time out situation. In the followingsections, we analyze the performance of the workflow based a number of KPIs.

4.2 AnalysisDuring enrollment periods[3], the system experiences massive number of concurrentrequests. Such simultaneous requests increase the response time as well as the chanceof possible system failure. To guarantee the stability and availability of the system,it is crucial to accurately access its workload limit as well as to identify the possiblebottlenecks of the application.

The entire experiment consists of two phases: statistics analysis phase andworkflow schema revision phase. First, business process engineer analyzes the audittrail data from WfMS by selecting the corresponding process definition of the webapplication. Figure 7 shows the screenshot of choosing a process definition CourseEnrollment. The audit trail analysis module then parses the audit trail of the CourseEnrollment workflow and calculates the necessary statistics. Similar interfaces are alsoprovided for the users to calculate relevant statistics from the web server log and stresstesting.

After all the statistics has been calculated, users may import the correspondingprocess definition into a workflow schema revision interface. Figure 8 showsthe interface for importing the workflow schema of the online course enrollment

BPMJ18,2

268

system. In this interface, business process engineers can view general statisticsretrieved from audit trail data, web server logs, and stress testing.

In Figure 9, we show the user interface for displaying the results from audit trailanalyzer. The audit trail statistics is shown on the left hand side of the figure. Statisticsretrieved from audit trail of a process definition can be classified into three categories:

(1) A1-A6: general statistics.

(2) A7-A9: average life duration, processing time, and idle of process instance.

(3) A10-A12: average life duration, processing time, and idle time of eachindividual activity instance.

The right hand side of Figure 9 shows the average time distribution of each status of aworkitem called “Take_UpdateStudyPlan” in a pie chart. In this chart, the workitemtakes 28 percent (12.068 seconds) of its life duration for queuing (to be started),35 percent (14.841 seconds) in processing, 37 percent (15.572 seconds) for completionand almost 0 percent for the other status. The prototype also provides charts fordisplaying the average distribution of processing time and idle time for processinstance, activity instances and workitems.

Figure 8.Workflow schema revisioninterface – calculation for

web server log statistics

Figure 7.Selecting a process

definition

KPIs forbusiness

processes

269

4.2.1 Performance indicators. The interface for KPIs is shown in Figure 10. Thecalculated statistics of a performance indicator is shown on the left hand side, while therelated charts are given on the right hand side of the window. A pop up message boxwhich summaries the performance indicator as also shown in Figure 10.

In the following paragraphs, we detail the calculation of Workload limit with stableperformance, Daily performance, Hourly performance, Problems in process definition,and Identification of cancelled activity instances key indicators of the enrollmentworkflow.

Figure 9.Audit trail analyzer

Figure 10.User interface for keyperformance indicators

BPMJ18,2

270

K1: workload limit with the best performance. In indicator Workload limit with stableperformance (Figure 11), statistics from audit trail analysis, web server log analysis, andstress testing are summarized on the left and side of the interface. Analysis on thenumber of successful and cancelled process instances with the number of concurrentvisits is also performed. These instances are then grouped by each hour of the day.

The ratio of concurrent visits to number of successful and cancelled processinstances is shown in Figure 12. This chart presents the changes in the averagenumber of total process instances, successful process instances and cancelled process

Figure 11.Statistics for indicator“Workload limit with

stable performance”

Figure 12.Ratio of concurrent visits

vs process instances forindicator “Workload limit

with the bestperformance”

Ratio of concurrent visits to PI

1

00 250 500 750 1,000

Number of concurrent visits

Visit:Pls Visit:SPls Visit:CPls

Ra

tio o

f vi

sits

to p

roce

ss in

stan

ces

1,250 1,500 1,750

KPIs forbusiness

processes

271

instances with respect to the number concurrent visits within the same duration. Foreach visit by the client, at least one process instance will be generated by the server. Ina normal situation when the web server is not under heavy traffic, it can usuallycomplete the generated process instance without any problem. However, during peaktraffic, the web server cannot successfully complete a process instance for each visitwhen the total number of concurrent visits exceeds 1,400. This situation is shown inFigure 12 when blue line (successful process instances) crosses down the horizontalstraight line (labeled as 1 process instance) at approximately 1,400.

Figure 13 shows the relationship between the concurrent visits and the responsetime. From this figure, we can observe that the response time of the web server exceed6,000 ms when the number of concurrent visits is .1,400. In such situation, thechances of process instance failure may also increase correspondingly.

K2: hourly performance of the system. In indicator hourly performance, number ofrecorded process instances, activity instances, workitems, pages, and hits/requests arederived and group by hour of the day. This statistics is shown in Figures 14 and 15.From Figure 15, we can observe that there are two peak durations over the log period,one is 10:00-12:00 (approx. 68,000 workitems with 100,000 hits) and the other one is17:00-18:00 (approx. 70,000 workitems with 125,000 hits).

K3: daily performance of the system. Similar attributes (process instances, activityinstances, workitems, unique visitors, visits, pages, and hits/requests) are calculatedand grouped by each day over the log period in indicator daily performance of thesystem (Figure 16). From Figure 17, we can observe that the peak day of the log periodis on 4 September 2008 (approx. 260,000 workitems with 420,000 hits).

Figure 13.Ratio of concurrent visitsvs average response timefor indicator “Workloadlimit with the bestperformance”

10,000

9,000

8,000

7,000

6,000

5,000

4,000

3,000

2,000

1,000

00 250 500 750 1,000

Number of concurrent visits

Visit:Response Time

1,250 1,500 1,750

Ave

rage

res

pons

e tim

e (m

s)

Concurrent visits to response time

BPMJ18,2

272

Figure 14.Statistics for indicator

“Hourly performance ofthe system”

Hour

ProcessInstances

ActivityInstances Workitems Pages Hits/Requests

1509 16,487 38,273 30,851 33,116

19,09011,237

5,519

3,058

1,5601,595

3,160

10,898

52,176

86,580

96,796

62,10424,20278,753

117,728

110,127

125,458

115,517106,235

88,49164,255

58,756

57,256

17,500

10,077

4,822

2,599

1,1671,1582,649

9,558

49,21183,25893,090

59,459

21,973

75,319113,508106,374

121,295110,363

100,80284,457

60,711

55,31953,776

25,18310,746

8,1596,245

2,781

2,4036,397

11,91155,453

68,16566,736

56,912

35,92956,031

64,448

63,07568,496

63,76458,352

58,243

56,84555,934

52,433

8,273

3,815

2,7612,087

943

806

1,554

3,45217,397

20,652

20,173

17,874

11,318

17,625

20,375

20,276

21,545

19,873

18,276

18,153

17,926

17,028

16,812

875

313251

207

8973

146327

1,656

1,9351,981

1,697

923

1,811

2,155

2,1512,269

2,1232,087

1,941

1,774

1,693

1,667

00:00 to 01:00

01:00 to 02:00

02:00 to 03:00

03:00 to 04:0004:00 to 05:00

05:00 to 06:00

06:00 to 07:0007:00 to 08:00

08:00 to 09:0009:00 to 10:00

10:00 to 11:00

11:00 to 12:0012:00 to 13:00

13:00 to 14:00

14:00 to 15:00

15:00 to 16:00

16:00 to 17:00

17:00 to 18:00

18:00 to 19:00

19:00 to 20:00

20:00 to 21:00

21:00 to 22:0022:00 to 23:00

23:00 to 24:00

Index

0

1

2

3

456

7

89

10

11

12

13

14

1516

17

18

19

20

21

22

23

Figure 15.Indicator “Hourly

performance of thesystem”

Hourly Performance Line Chart

130,000

120,000

110,000

100,000

90,000

80,000

70,000

60,000

50,000

40,000

30,000

20,000

10,000

00 1 2 3 4 5 6 7 8 9 10 11 12

Index of Hour Duration

Process Instances Activity Instances Workitems Unique Visitors Pages Hits/Requests

13 14 15 16 17 18 19 20 21 22 23

Tota

l Num

ber

KPIs forbusiness

processes

273

K4: problems in process definition. In indicator Problems in process definition(Figure 18), statistics about the HTTP status codes from the web server log and stresstesting are summarized. These statistics are grouped by each hour of the day andcompared with the number of cancelled process instances from audit trail analyzer.The analytical results are shown in Figure 19.

Figure 19 shows the distribution of recorded HTTP status codes (with errors). Fromthis distribution, we can observe that HTTP status code 404 occurred 68 percent of thetime and status code 500 occurred 31 percent of the time. The remaining 1 percentrepresents other errors.

In addition, the line chart in Figure 20 shows the ratio of cancelled process instances tothe number of hits with the failed HTTP status codes. From the chart, we can observe thatthe number of hits with HTTP error codes increases with the number of cancelled processinstances, especially for 404 and 500 errors. From Figures 18, 19, and 20, we can concludethat the web server may have been experiencing 404 and 500 errors since these two errorsoccupied the majority of the hits of HTTP error codes. In addition, we can also notice thatthe number of errors increases when the number of cancelled process instances increases.

Figure 17.“Daily performanceof the system”

Daily Performance Line Chart

400,000

350,000

300,000

250,000

200,000

150,000

100,000

50,000

0

Tota

l Num

ber

2008-09-01 2008-09-02 2008-09-03 2008-09-04

Date

2008-09-05 2008-09-06

Process Instances Activity Instances Workitems Unique Visitors PagesVisits Hits/Requests

Figure 16.Statistics for indicator“Daily performanceof the system”

DayProcessInstances

ActivityInstances Workitems

UniqueVisitors Visits Pages Hits/Requests

2008-09-01 993 10,072 30,312 403 827 12,421 13,939216,456271,138418,243391,17722,714

204,686257,949399,479375,43319,328

5,0065,1086,2976,2561,644

2,2652,5393,3234,1241,471

215,732220.853260,651252,43812,928

62,40568,192130,924119,5765028

6,1206,1357,8817,7212,803

2008-09-022008-09-032008-09-042008-09-052008-09-06

BPMJ18,2

274

Figure 18.Statistics for indicator“Problems in process

definition”

Web Server Log Statistics

Stress Testing Statistics

* Client Error** Server Error

CPI: Cancelled Process InstancesHour CPI 403 err 404 err 405 err 500 err 503 err

Status error from Stress Testing Statistics

Status error from Web Server Log StatisticsHTTP Status Code301 3,283

134,64452,54611,1701

302

200 1,5661,74153912

404*500**NA

403*404*405*500**503**

Hits

HTTP Status Code Hits

00:00 to 01:00 97 0 28 0 90

0000000000000000001

00

000

00000001871119273231841392091468768311719

00000000000000000010000

8002000560163239641179347287391301185171726865

00000000000000212000000

4145105132523146450925954465624586685597495472381302270

01:00 to 02:0002:00 to 03:0003:00 to 04:0004:00 to 05:0005:00 to 06:0006:00 to 07:0007:00 to 08:0008:00 to 09:0009:00 to 10:0010:00 to 11:0011:00 to 12:0012:00 to 13:0013:00 to 14:0014:00 to 15:0015:00 to 16:0016:00 to 17:0017:00 to 18:0018:00 to 19:0019:00 to 20:0020:00 to 21:0021:00 to 22:0022:00 to 23:0023:00 to 00:00

Figure 19.Proportion of various

HTTP codes for indicator“Problems in process

definition”

KPIs forbusiness

processes

275

K5: identification of cancelled activity instances. In indicator Identification of cancelledactivity instances, statistics about the cancelled process instances and the 404 HTTPstatus code from web server log are used for analysis. Figure 21 shows the number ofcancelled process instances and the list of URLs which have been logged with morethan ten 404 errors. Although the threshold is set to 10 in this example, it can bechanged according to user’s requirement. The analyzed result is shown in Figure 22.The analysis identifies the URLs “/siweb/faces/app/pagerror.gif” and“/siweb/faces/app/refresh.gif” as the URLs with the largest number of hits with404 error code. It also indicates that the activity instances which associated with thesetwo URLs could be the sources of the potential bottleneck in the process definition.

Figure 20.Ratio of cancelled processinstance to HTTP errorsfor indicator “Problems inprocess definition”

Ratio of cancelled process instances (CPI) to HTTP errors

55

50

45

40

35

30

25

20

15

10

5

00 50 100 150 200 250 300 350

Number of cancelled process instances

CPI:403 CPI:404 CPI:405 CPI:500 CPI:503

400 450 500 550 600 650 700

Ra

tio o

f C

PI t

o H

TT

P e

rror

s (%

)

Figure 21.Statistics for indicator“Identification of cancelledactivity instances”

BPMJ18,2

276

4.3 Process reengineeringFrom the performance indicators, business process engineers can clearly identify thebottlenecks of the workflow and may revise the workflow model to improve thesituation. Figure 23 shows one potential modification to workflow schema of processTake Course from Figure 5.

According to the workflow schema, the conditional process All Prerequisites andRequirements Fulfilled? and Process Enrollment from Figure 5 require large amount ofprocessing power from the web server since they have to retrieve the academic historyof the users and all relevant details of the courses. In order to ease the workload of theweb server, instead of having a single complex process, the original All Prerequisitesand Requirements Fulfilled? conditional process is converted into three processes inFigure 23: Corresponding Academic year?, List Prerequisite, and All PrerequisitesPassed?

The conditional process Corresponding Academic year? directs the user back to theprocess Input Course Info if he/she is not in the correct academic year to enroll theselected courses. In this conditional process, instead of an entire academic record, onlythe academic year of the user is retrieved. It can significantly reduce the overhead inaccessing the database. Moreover, this conditional process has also reduced thenumber of requests proceeds to the next process if the condition is not fulfilled.

Figure 22.Proportion of various

HTTP codes for indicator“Identification of cancelled

activity instances”

Figure 23.Revised workflow schema

from Figure 5

KPIs forbusiness

processes

277

The next process List Prerequisite lists all the prerequisites courses for the inputcourses. The conditional process All Prerequisites Passed? directs the user back to InputCourse Info if the user has not completed all the necessary prerequisite courses. In thisconditional process, only relevant academic records of the user and the prerequisites ofthe input course will be retrieved. All these modifications are designed to reduce theincoming requests leading to Process Enrollment task, and therefore may lower the riskof potential failure at this task during peak hours. In addition, to reduce the impact on theserver during peak hours, more resources could be allocated to the processes.

In the above example, the indicators K1-K5 provide an external as well as internalview of the performance of the web application. Indicator K1 indicates that theperformance of the system remains stable if the number of visits is ,1,400. As soon asthe number of visits is .1,400, the performance of the system starts to degrade.Indicators K2 and K3 identify the peak durations within the log period. More resourcescan be allocated accordingly during these peak durations so as to avoid systemperformance degradation. Indicator K4 identifies HTTP error codes which are caused byeither client or server errors. Isolating these errors can assist the analyst in locating thepotential problems in the process definition. Moreover, indicator K5 locates the cancelledactivity instances which are likely to cause potential bottlenecks in the system. Based onK5, analysts can revise the process definition so as to improve the situation.

5. Related workAlthough performance evaluation for web applications has been extensivelyresearched from user’s perspectives (Zadorozhny et al., 2008) and serverperspectives, process-reengineering of traffic intensive web application based on theintegrated analysis of audit trail, web server log, and stress testing has receivedrelatively little coverage in the literature. To the best of authors’ knowledge, recentwork in performance evaluation research investigates these statistics in isolation.

In McGregor and Schiefer (2003), McGregor et al. introduce a framework to providecomprehensive information about the status and performance of business processes.Their approach also supports a near real-time integration of audit trail data from theexecuted processes to identify performance indicators. Similar to (McGregor andSchiefer, 2003), the proposed approach also identifies the KPIs from audit trail data. Inaddition, our approach also takes into account statistics from stress testing and thusprovides an external view of the performance.

Iyer et al. (2005) propose a comprehensive strategy to analyze results fromconcurrency testing and reliability testing to measure the performance, the reliabilityand the scalability of a multi-tier web application. Their approach provides a numberof parameters to identify the threshold for acceptable performance. Similar to theirstrategy, our approach calculates a set of indicators to measure the performance of aweb application. In particular, the web applications considered in our approach areexecuted in a WfMS. Our indicators also consider the statistics from audit trail toprovide a perspective about resource allocation and the history of execution of abusiness process at an operational level.

Rozinat et al. (2008) present a way to create a simulation model for a workflowsystem which supports real-world process based on a workflow and organizationalmodel. The simulation model is created based on design (the organizational data, andprocess model describing control and data flows), historic (history of the process

BPMJ18,2

278

recorded in a workflow system – audit trail) and state (the current state of theworkflow process) information of the process.

An approach to extend the definition of existing business processes with auditingfunctions is proposed in Roth et al. (2006). These processes are modeled with BusinessProcess Execution Language with Web Services (WSBPEL). In their approach, theWSBPEL process definitions are populated with auditable elements (e.g. start andcompletion time of each activity). These process definitions can be executed ondifferent WSBPEL engines although these engines may offer their own features inauditing the business process. Likewise, audit trail data captured by different WfMShave their own auditing features. These features reflect the performance of thebusiness process in different ways. These proprietary features significantly increasethe difficulty in deriving statistics from the audit trail data. In this paper, weconceptualize a generic approach for implementing an audit trail data analysis module.

zur Muehlen and Rosemann (2000) develop a tool which can be used to analyze audittrail data of different WfMS. Audit trail data are evaluated from three differentperspectives (process, resource, and object) to monitor and control the workflow of abusiness process. In contrast to (zur Muehlen and Rosemann, 2000), our approach notonly focuses on the workflow of the web application, but also evaluates theperformance from the internal and external view of the web server.

In Chen and Heidemann (2005), Chen et al. introduce an adaptive admissionmechanism to maintain the availability of servers. In their approach, analyses on webserver logs are performed to detect performance degradation which is considered as asign for flash crowd situation. Flash crowds are then mitigated by admitting incomingrequests adaptively. In contrast to their approach, our framework is used to analyze thecapability of a web application server for handling traffic intensive situations.

In Arlitt and Jin (2000), Arlitt et al. describe a workload characterization study of the1998 World Cup web site. In their work, a cache consistency mechanism is used toreduce the burden in handling the incoming requests. In their approach, frequentlyaccessed objects are placed in cache for faster retrieval by the web clients. However, asweb applications are rapidly evolving from displaying static information to dynamiccontents, the cache mechanism may not significantly reduce the incoming requestsduring flash crowd situation. In contrast to their approach, our proposed frameworkaims to enhance the workload handling capability of a web server by uncovering theweakness of the underlying application workflow.

A system to integrate a Timed Petri Net model and statistical models for webservices and process engine is proposed in Koizumi and Koyama (2007). Theintegration models the dynamic workload during business process execution. It alsotakes into account parallel processing of requests as well as consuming of sharedresources. Although the proposed approach does not explicitly model the dynamicworkload during performance evaluation, it provides an integrated environment inwhich KPIs are calculated automatically from each evaluation.

In the context of load testing, Draheim et al. (2006) propose an approach in whichusers’ behavior is mimicked with stochastic form-oriented analysis models. In theirapproach, web site navigation and time delay are modeled stochastically.Krishnamurthy et al. (2006) also propose a technique to generate a syntheticworkload for performing stress testing for session-based systems. However, theseapproaches focus on the techniques used in generating workload for stress loading test,

KPIs forbusiness

processes

279

whereas our approach focuses on integration and analysis of the stress testing resultswith web server log statistics and audit trail data.

The workload limit of a system is often evaluated by using stress testing. In Jainand Leong (2005), Jain et al. stress test a supply chain using business processsimulation (BPS) software. In their approach, the supply chain is modeled in asimulation program and experiments are conducted for three different levels ofworkload to evaluate the performance of the system. In contrast, our approachevaluates the workload limit of a traffic intensive web application based on theintegration of the execution logs, web server log, and stress testing statistics. Inaddition, our approach identifies the KPIs of the analyzed web application.

A business process (Harrington, 1991) consists of a group of logically related tasksthat use the resources to provide defined results in support of the objectives of anorganization. BPS (An and Jeng, 2005) artificially implements and assists themanagement of change in a variety of manufacturing and service settings. BPS enablesdetail analysis of such performance indicators without actually deploying theprocesses. BPS can be loosely defined as a technique for executing virtual businessprocess models to analyze resources usage as well as the behaviors of the processesand organizations involved. BPS is widely used for analyzing business process modelsin many different domains. For instance, simulation was used to study the effect ofstochastic customer shopping traffic for the IBM’s Personal Computer Division (Lee,2004) as well as for analyzing the impact on the user experience and the cost of usingthe application when a mobile channel presentation (based on GSM, HSCSD, GPRS,and UMTS networks) is added (Book et al., 2005). However, such approaches do notconsider the flash crowds situation when exceptional circumstances occur.

Zhang et al. (2011) propose a framework to tackle engine overloading problem in acentralized workflow of web services, where a centralized workflow is a workflow witha centralized control point. The framework saves network bandwidth and avoids acentralized workflow engine from being overloaded by using a result forwardingfunction. This function forwards the result of a service to its successor service anddistributes data transfer of consecutive services. In contrast to their work, ourapproach focuses on using performance indicators to pinpoint the potential bottlenecksin a workflow schema instead of distributing data transfer to avoid workflow engineoverloading.

In summary, recent work on performance evaluation analysis devote less attentionon the integrated analysis of execution logs and other vital statistic of the applicationwith analysis results from web server logs and stress testing. To the best of authors’knowledge, such integrated analysis platform has not been developed for evaluatingtraffic intensive web-enabled business processes.

6. ConclusionsTraffic intensive web-enabled business processes are extensively used in travel industryand government agencies. Applications under this category are likely to encounter flashcrowd situation, which is an extreme sudden case for intensive traffic, during certainevents. However, currently available performance analysis methods only provide eitherinternal or external view of the situation. In this paper, we propose an approach forautomated calculation of KPIs for traffic intensive web-enabled business processes. Inour approach, we show how vital statistics from audit trail, web server log, and stress

BPMJ18,2

280

testing can be automatically linked to calculate KPIs. In particular, the proposedapproach makes use of generic attributes for these vital statistics and these statistics arenot tied to any specific software or platform. Therefore, the algorithms for automaticallylinking these statistics can be used to evaluate the performance of any traffic intensiveweb-enabled business processes. The calculated results from these algorithms providesboth external/client and internal/server view of the evaluation. Such evaluation fromboth perspectives cannot be performed with currently available methods when they areapplied in isolation. Most importantly, the KPIs derived from the proposed approach canbe used for locating potential bottlenecks in the application, business processreengineering, load balancing, and implementation of contingency measures for trafficintensive situations.

As for the future work, we are planning to extend the current integrated analysisplatform with a BPS engine. Such extension will show the performance of the webapplication after revising the workflow by simulation as well as enable processengineers to study the resource allocation strategies for flash crowd situation.

Notes

1. Allocation time and total queuing time are not applicable to either activity instance orprocess instance since they are in higher level of abstractions.

2. In this paper, we leave the workflow simulation as future work.

3. Predefined enrollment period are assigned to each student at the beginning of a semester.

References

Agostini, A. and de Michelis, G. (2000), “Improving flexibility of workflow managementsystems”, Proceedings of the Business Process Management, Models, Techniques, andEmpirical Studies, Springer, London, pp. 218-34.

An, L. and Jeng, J.-J. (2005), “On developing system dynamics model for business processsimulation”, Proceedings of the 37th Conference on Winter Simulation (WSC), Orlando,FL, 4-7 December, pp. 2068-77.

Arlitt, M. and Jin, T. (2000), “A workload characterization study of the 1998 World Cup web site”,IEEE Network, Vol. 14 No. 3, pp. 30-70, Special Issue on Web Performance.

AWStats (n.d.), available at: http://awstats.sourceforge.net/ (accessed 31 July 2010).

Book, M., Gruhn, V., Hulder, M., Kohler, A. and Kriegel, A. (2005), “Cost and response timesimulation for web-based applications on mobile channels”, Proceedings of the5th International Conference on Quality Software, Melbourne, 19-20 September, pp. 83-90.

Borgida, A. and Murata, T. (1999), “Tolerating exceptions in workflows: a unified framework fordata and processes”, Proceedings of the International Joint Conference on Work Activities,Coordination and Collaboration (WAC’99), ACM Press, New York, NY, pp. 59-68.

BPEL4WS (n.d.), BPEL for Web Services, available at: www.ibm.com/developerworks/library/specification/ws-bpel/ (accessed 31 July 2010).

BPMN (n.d.), Business Process Modeling Notation, available at: www.bpmn.org/ (accessed31 July 2010).

Chen, X. and Heidemann, J. (2005), “Flash crowd mitigation via adaptive admission control basedon application-level observations”, ACMTransactions on Internet Technology, Vol. 5 No. 3,pp. 532-69.

KPIs forbusiness

processes

281

Developer Resources for Java Technology (n.d.), available at: http://java.sun.com/ (accessed24 February 2010).

Draheim, D., Grundy, J., Hosking, J., Lutteroth, C. and Weber, G. (2006), “Realistic load testing ofweb applications”, Proceedings of the 10th European Conference on Software Maintenanceand Reengineering (CSMR), Bari, 22-24 March, pp. 57-70.

Eclipse (n.d.), available at: www.eclipse.org/ (accessed 31 July 2010).

Harrington, H.J. (1991), Business Process Improvement, 1st ed., McGraw-Hill, New York, NY.

Hevner, A.R. (2007), “A three cycle view of design science research”, Scandinavian Journal ofInformation Systems, Vol. 19 No. 2, pp. 87-92.

Iyer, L.S., Gupta, B. and Johri, N. (2005), “Performance, scalability and reliability issues in webapplications”, Industrial Management & Data Systems, Vol. 105 No. 5, pp. 561-76.

Jain, S. and Leong, S. (2005), “Stress testing a supply chain using simulation”, Proceedings of the37th Conference on Winter Simulation (WSC), Orlando, FL, 4-7 December, pp. 1650-7.

JBoss jBPM (n.d.), available at: www.jbpm.org/ (accessed 31 July 2010).

Koizumi, S. and Koyama, K. (2007), “Workload-aware business process simulation withstatistical service analysis and timed petri net”, IEEE International Conference on WebServices (ICWS), Marriott Salt Lake City Downtown, Salt Lake City, UT, 9-13 July, pp. 70-7.

Krishnamurthy, D., Rolia, J.A. and Majumdar, S. (2006), “A synthetic workload generationtechnique for stress testing session-based systems”, IEEE Transactions on SoftwareEngineering, Vol. 32 No. 11, pp. 868-82.

Kumar, A. and Zhao, J.L. (1999), “Dynamic routing and operational controls in workflowmanagement systems”, Management Science, Vol. 45 No. 2, pp. 253-72.

Lee, Y.M. (2004), “Simulating availability outlook for e-commerce business for personal computersales”, Proceedings of the 36th Conference on Winter Simulation (WSC), ACM Press,New York, NY, pp. 1201-4.

McGregor, C. and Schiefer, J. (2003), “A framework for analyzing and measuring businessperformance with web services”, IEEE International Conference on E-commerce,Newport Beach, CA, 24-27 June, pp. 405-12.

Mans, R.S., van der Aalst, W.M.P., Russell, N.C. and Bakker, P.J.M. (2009), “Flexibility schemesfor workflow management systems”, Proceedings of the Second International Workshop onProcess-oriented Information Systems in Healthcare (ProHealth08), Volume 17 of LectureNotes in Business Information Processing, Springer, Berlin, pp. 361-72.

Marinescu, D.C. (2002), Internet Based Workflow Management: Towards a Semantic Web, 1st ed.,Wiley-Interscience, New York, NY.

Menasce, D.A. (2002), “Load testing of web sites”, IEEE Internet Computing, Vol. 6 No. 4, pp. 70-4.

Microsoft Web Application Stress Tool (n.d.), available at: www.microsoft.com (accessed 31 July2010).

OpenWFE (n.d.), available at: www.openwfe.org/ (accessed 24 February 2010).

Roth, H., Schiefer, J. and Schatten, A. (2006), “Probing and monitoring of WSBPEL processeswith web services”, Proceedings of the 8th IEEE International Conference on E-commerceTechnology and the 3rd IEEE International Conference on Enterprise Computing,E-commerce, and E-services (CEC/EEE’06), San Francisco, CA, pp. 30-8.

Rozinat, A., Wynn, M.T., van der Aalst, W.M.P., ter Hofstede, A.H.M. and Fidge, C.J. (2008),“Workflow simulation for operational decision support using design, historic and stateinformation”, Proceedings of the 6th International Conference on Business ProcessManagement (BPM 2008), LNCS 5240, Milan, 2-4 September, pp. 196-211.

BPMJ18,2

282

Sadiq, S.W., Marjanovic, O. and Orlowska, M.E. (2000), “Managing change and time in dynamicworkflow processes”, International Journal of Cooperative Information Systems, Vol. 9Nos 1-2, pp. 93-116.

van der Aalst, W.M.P. and van Hee, K.M. (2000), Workflow Management, Models, Methods, andSystems, MIT Press, Cambridge, MA.

van der Aalst, W.M.P., Pesie, M. and Schonenberg, H. (2009), “Declarative workflows: balancingbetween flexibility and support”, Computer Science – Research and Development, Vol. 23No. 2, pp. 99-113.

WAPT (n.d.), Web Application Load, Stress, and Performance Testing, available at: www.loadtestingtool.com/ (accessed 24 February 2010).

Workflow Management Coalition (1995), “Workflow reference model”, Document NumberTC00-1003, Version 1.1, Workflow Management Coalition, Cohasset, MA, January.

Workflow Management Coalition (1998), “Audit data specification”, Document NumberWFMC-TC-1015, Version 1.1, Workflow Management Coalition, Cohasset, MA, September.

W3C (n.d.), Extended Log File, available at: www.w3.org/TR/WD-logfile (accessed 31 July 2010).

XPDL (n.d.), XML Process Definition Language, available at: www.wfmc.org/standards/xpdl.htm (accessed 31 July 2010).

YAWL (n.d.), Yet Another Workflow Language, available at: www.yawlfoundation.org/(accessed 31 July 2010).

Zadorozhny, V., Yudelson, M. and Brusilovsky, P. (2008), “A framework for performanceevaluation of user modeling servers for web applications”, Web Intelligence and AgentSystems, Vol. 4 No. 4, pp. 175-91.

Zhang, D., Coddington, P. and Wendelborn, A. (2011), “Web services workflow with result dataforwarding as resources”, Future Generation Computer Systems, Vol. 27 No. 6, pp. 694-702.

zur Muehlen, M. and Rosemann, M. (2000), “Workflow-based process monitoring andcontrolling-technical and organizational issues”, Proceedings of the 33rd HawaiiInternational Conference on System Sciences (HICSS), Maui, HI, 4-7 January, Vol. 6,pp. 6032-42.

About the authorsKa I. Pun (Violet) received her Master’s degree in Software Engineering from the University ofMacau in 2009. After completing her thesis on performance evaluation of web-enabled businessprocesses, she has been, since January 2010, a PhD candidate at the Department of Informatics atthe University of Oslo, Norway in the group for Precise Modelling and Analysis of DependableSystems. Her current research interests are related to formal term rewriting and static analysis.Ka I. Pun is the corresponding author and can be contacted at: violet@ifi.uio.no

Dr Yain Whar Si received his PhD degree in Information Technology from the QueenslandUniversity of Technology in Australia in 2005. He is currently working as an Assistant Professorat the University of Macau. His research interests are in the areas of business processmanagement and decision support systems.

Kin Chan Pau is a MSc student at the Department of Computer and Information Science,University of Macau. Her research interests lie in the areas of business process management ande-commerce technologies.

KPIs forbusiness

processes

283

To purchase reprints of this article please e-mail: reprints@emeraldinsight.comOr visit our web site for further details: www.emeraldinsight.com/reprints