performance testing driving capacity management at easyjet

Positionner le logo de la société du présentateur

à la place de ce bloc texte

Président de filière

Orateur

Sponsor

Administrateur itSMF

2Titre de la présentation

Performance Testing Driving Capacity Management at easyJet

Andy Bolton

Nathalie Grandvuillemin

Andy Caddy

Benoit de Savignac


easyJet.com

3


easyJet continues to groweasyJet 2009

£2.6bn revenue

181 aircraft

45m passengers

422 routes

“Low cost with care and convenience”

4

easyJet 2011

£3bn+ revenue

204 aircraft

50m+ passengers

560routes

“Passionate about making travel easy & affordable”

easyJet 2014

£4bn+ revenue

230+ aircraft

60m+ passengers

650+ routes

““Europe’s preferred short haul airline”


easyJet 19963 Airports

2 Countries

2 Aircraft

2 Routes

5

Base Airport

Other Airport

Performance Testing Driving Capacity Management at easyJet 6

easyJet 2011150 Airports

34 Countries

210 Aircraft

560 Routes

1500+ Flights

60m+ Passengers

Base Airport

Other Airport


easyJet a growth storyIn 15 years easyJet has become the UK’s #1 airline

7


IT Budgets

8


easyJet Revenue

9

Flight revenue (easyJet.com + B2B)

Flight revenue (easyJet.com + B2B)

Non Flight Revenue(Ancilary Partners)Non Flight Revenue(Ancilary Partners)

100% ITReliance100% ITReliance


How important is IT?

10

Performance Testing Driving Capacity Management at easyJet 11

How important is IT?


Agile Development Process

12

Product Backlog

Sprint Planning Meeting

Sprint Backlog

Release

Sprint(15-30 days)

INOUT

SprintRetrospective

Meeting

Scrum(Daily

Meeting)

LESSONS LEARNED

Pre-release Activities(e.g. Training,

Documentation)

Sprint Review

Product Backlog

Sprint Planning Meeting

Sprint Backlog

Release

Sprint(15-30 days)

INOUT

SprintRetrospective

Meeting

Scrum(Daily

Meeting)

LESSONS LEARNED

Pre-release Activities(e.g. Training,

Documentation)

Sprint Review


Business Context

Capacity planning service www.easyJet.com

Business critical service

98% of revenue earned through the e-commerce service

Highly competitive market

Frequent innovation by adopting agile development

Monthly software releases

Legislative changes

Consistent growth; typically 20% p.a.

Low-cost operating model

Requirement to manage risk

13


easyJet’s Goals

14


easyJet’s Goals

15


Challenges Associated with Agile

Frequent change introduces risk

Window available for performance testing is typically 1-2 weeks

Final production candidate code often not available at the start of testing

Multiple code releases within the testing window

Increased probability of functional and environmental defects

Frequent rework of code necessitates modifications to test scripts

Cumulative effect on capacity from 12 releases per annum

Narrow time window to resolve performance defects

16


Challenges Associated with Agile

17


Typical Causes of Degraded Service Performance

18


How Can We Reduce Risk?

19

Cause of degraded performance Mitigated by performance testing?

Mitigated by modelling?

Changes in user patterns No Yes

Poor performance of 3rd parties May be limited by environment

Yes

Physical capacity bottlenecks

May be limited by size of the environment

Yes

Logical bottlenecks Yes If known

Software configuration changes Yes Yes, if tested

Increases in demand May be limited by size of the environment

Yes


Why Test?

Performance testing is used to understand the software’s performance characteristics

Do unknown time and growth related defects exist?

Memory leaks

Poor database design/indexing

Do unknown concurrency related defects exist?

Database locking issues

Memory locking issues

Race conditions

Obtain model inputs

20


Why Model?

Modelling uses our understanding of the software to predict end to end service performance under different scenarios

21


Scenario Modelling

22


Coverage of Testing and Modelling

23

ModellingTesting

Application InfrastructureBusiness ScenariosTechnical Scenarios


High Level Process

24


Risk Assessment

25


Risk Assessment

Assess the potential performance impact on different elements of infrastructure

Plan tests to help mitigate that impact

Focus on areas of highest risk due to limited timeframes

26

Web DB OtherProject X --- Medium Medium Low Test 1Project Y --- Low High Medium Tests 2 & 3Project Z --- Low Low Low None

Description IDRisks

Risk Mitigation


Performance Tests Employed

27


Baseline Tests

Used to determine the impact of the planned release

Comparative performance between releases

May be Load, Stress or Soak Tests

Service times and response times are compared to each other

28

Database Server Service Time

Web Server Service Time

Test 1.1 Release X 752 1240

Test 1.1 Release Y 813 1192

% Difference 8% -4%

0

200

400

600

800

1000

1200

1400

Serv

ice T

ime (

ms)

Example service time comparison between two fictional baseline tests


Single Transaction Type Tests

29

To determine the impact on the infrastructure of individual user transactions

Every business process which is to be modelled will have a calculated service time for each hardware element

132

233

156

122

221

328

Milliseconds Web CPU per page (fictional data)

Home Search Select

Options Login Purchase


Modelling

30

Performance Test Results


Performance Modelling

31

Business

Service

Component

Virtual Infrastructure



32

Business

Service

Component


Aircraft

Home Page Search Booking

HTTP1 APP1 APP2 DB1



33

Business

Service

Component


600

100 80 5

160 70 10 80

3.5 transactions/s


Demand Model Validation

The model should be validated on an ongoing basis against actual values

The first stage is to validate the demand part of the model to ensure that arrival rates are being calculated correctly

Following this the true demand can be used to validate the utilisation calculations

34

600

100 80 5

610

80 80 6


Capacity and Performance Model Validation

35

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Comparison of Actual Database Load to Predicted Database Load

Actual Database CPU Predicted Database CPU


What you need – People

Capacity and performance management skills:

Demand forecasting

Modelling

Load testing

A strong understanding of risk management

A good understanding of the business model

Flexibility

Project management skills

36


What you need – Process

37

Interface with the release team

Participation in the agile development `scrums`

Access to developers to understand application functionality

Interface with the test environment management team for test planning

Interface to the business to access business forecasts

Interface to operations to access production measurements


What you need – Tool & Environment

An appropriately sized test environment

Production-like environment is not required

Consider the ratio of capacity between presentation, business and data tiers

Toolset

Performance monitoring tools (production and test)

Load generation tool

Modelling tool

38


What you need – Data

Software/hardware configuration (production and test)

Performance measurements from production and test environments

Benchmark data (Spec, TPC, etc.)

Service times derived from performance test results

SLAs

Internal

SLAs with third-parties

Historical and forecast business demand

Historical and forecast service demand

Measured error of previous forecasts

39


Historical Business Demand

40

y = 0.0461x - 1673.1R² = 0.9668

0

20

40

60

80

100

120

140

160

180

Mar-04

Jun-

04

Sep-

04

Dec-

04

Mar-05

Jun-

05

Sep-

05

Dec-

05

Mar-06

Jun-

06

Sep-

06

Dec-

06

Mar-07

Jun-

07

Sep-

07

Dec-

07

Mar-08

Jun-

08

Sep-

08

Dec-

08

Mar-09

Num

ber

of

Ow

ned A

ircr

aft

Delivery Date

Fleet Plan to April 2009

GB Airways acquisition


Historical Service Demand

41


Costs

42


Benefits

Testing and modelling complement one another

The combined approach provides effective risk-mitigation in the context of:

Frequent change

Narrow performance test window

Business growth

The approach enables us to rapidly answer business and technical ‘what if’ scenarios

Costs are limited through the use of a scaled-down performance test environment and low-cost modelling tools

Provides performance assurance in a highly competitive market place

“Capacitas give us the confidence in making the correct release decision”

James Mackay, Test Manager, easyJet

43


Questions

Presentation will be available for download at www.capacitas.fr

Please visit us at our stand for any further questions

44

performance testing driving capacity management at easyjet

Technology

performance testing

flight revenue

ancilary partners

b2b

easyjet challenges

easyjet easyjet

low

test