a framework to evaluate qos of adaptive enterprise applicationltahvild/courses/ece... · evaluate...

A Framework to Evaluate QoS of Adaptive

Enterprise Application

Presented by:Sen Li

ECE750T11 Presentation 2

Contribution of the WorkThe main contribution of this work is defining a software engineering process to evaluate the QoS of an adaptive enterprise application (AEA).

As the proof of concept, a VoIP SOA application is chosen for the case study.

A set of experiments would be conducted on it.

2ECE750T11 Presentation 27/27/2009

Measurable QoS metrics

Load Generating ServerEvaluated AdaptiveEnterprise Application

Conducting Performance Testing

Responses

Service Level Agreement (SLA) of the evaluated

application

Containing

High-level standards for software quality evaluation

(e.g. ISO 9126)

Monitored run-time

QoS values

Containing

Utility Functions

Logging

Utility Value

Inputting as parameters

Computing

Deriving to

Building up Utility Functionsbased on SLA

Containing

7/27/2009 3ECE750T11 Presentation 2




Responses


application

Containing


(e.g. ISO 9126)

Monitored run-time

QoS values

Containing

Utility Functions

Logging

Utility Value


Computing

Deriving to


Containing


Evaluated AdaptiveEnterprise Application

667/27/2009 ECE750T11 Presentation 2

Case Study

“Call Controller 2” (CC2), is a VoIP (Voice over Internet Phone) SOA (Service Oriented Architecture) application, written by Java anddeployed on Mobicents.

Mobicents is the first and only open source platform certified for JSLEE.

JSLEE complements J2EE to enable convergence of voice, video andinstant messaging in next generation intelligent applications.

Mobicents is an event-driven application server with a robust component model.

Mobicents is deployed on JBoss Application Server (AS)

ECE750T11 Presentation 1 7ECE750T11 Presentation 27/27/2009


JBoss is the #1 most widely used Java AS on the market.

JBoss AS

StarMX Framework

StarMX is a management framework that enables the creation of applications with self-managing properties.

Designed for Java-based systems.Incorporating JMXIntegrating with various policy engines.

E.g. IBM ABLE Framework (Crisp/Fuzzy Logic, Neural Network)



Sensors: Response time, Throughput, Error rate, Availability and etc.

Effectors: Disable / Enable any of CC2’s services

The load generator used in this work is an open source SIP generator, SIPp.




Responses


application

Containing


(e.g. ISO 9126)

Monitored run-time

QoS values

Containing

Utility Functions

Logging

Utility Value


Computing

Deriving to


Containing


Utility Function

There should be n + 1 utility functions existing.

n functions are corresponding to n QoS elements guaranteed by SLA

The last 1 function is used to aggregate all utility values produced from previous n functions, returning a single global utility value.

Global Utility Value = w1*u1 + w2*u2 + … + wn*un


Utility Function

E.g. u1 = Utility Value of Response Time =

where r is the obtained response time, ßR is the response time in SLA, and KR is a scaling factor.

7/27/2009 19

Experiments


Experiments DesignThe aim of the experiment is to use our proposed process to evaluate the QoS of two enterprise applications.

The experiment is a one-factor, four-blocking and three-replicationdesign.

The factor is the adaptation type, and two treatments are: 1. no adaptation. 2. add adaptation.

Recall that, CC2 provides four services: 1. Regular VoIP calls; 2. Forwarding; 3. Blocking; 4. Voicemail

“no adaptation” CC2 will always provide all these services at runtime.“add adaptation” CC2 may Disable/Enable “Regular calls”, “Forwarding”, and “Voicemail” services at runtime.


Workloads DesignThe four blocks are corresponding to four different workloads.

Due to the importance of scalability, we need load and stress testing to evaluate an enterprise application.

We argue that experiments should be conducted under at least four workload levels:

Light (Load testing)Medium (Load testing)Heavy (Load testing)Extreme (Stress testing)

Three replications will be for each block and each treatment.


WorkloadsRegular call Forwarding VoiceMail

Light workloadExponential distribution (0.5 sec), total 4000

Exponential distribution (5 sec), total 400

Exponential distribution (8 sec) total 240

Medium workload Exponential distribution (0.15 sec) total 4000

Exponential distribution (1 sec) 700

Exponential distribution (1.6 sec) total 400

Heavy workloadExponential distribution (0.075 sec) total 8000



Extreme workload Exponential distribution (0.01 sec) total 8000

N/A N/A


Obtained Results (Light Workload)

Finished Regular

Calls

Avg RT of Regular

Calls

Finished Forwarding

Finished Voicemail

Utility Value

no adaptation 3682 4:633 sec 332 199 122995

add adaptation 3656 4:337 sec 315 198 124219


Obtained Results (Medium Workload)

Finished Regular

Calls

Avg RT of Regular

Calls

Finished Forwarding

Finished Voicemail

Utility Value




Obtained Results (Heavy Workload)

Finished Regular

Calls

Avg RT of Regular

Calls

Finished Forwarding

Finished Voicemail

Utility Value




Obtained Results (Extreme Workload)

Crashed or Not?

no adaptation Yes

add adaptation No


Crashed: Error 503 - Service unavailable

Lessons Learned


Lessons Learned

In light and medium workloads, AEA performs as well as NEA.

In heavy and extreme workloads, AEA completely outperforms NEA.

The results of ANOVA confirm these.

We can see that the workload level plays an important role in evaluating an adaptive enterprise application.

AEA: adaptive enterprise applicationNEA: non-adaptive enterprise application


Lessons Learned

The process we proposed is a systematic way to evaluate the QoS of AEA.

Each step of it can even be automatically computed.

It is also very flexible, by modifying utility functions.

The utility value reflects how well the SLA is satisfied. In other words, how well clients are satisfied.


a framework to evaluate qos of adaptive enterprise applicationltahvild/courses/ece... · evaluate...

Documents