SEPG Conference Amsterdam 2006

Developing Metrics for agile projects

compatible with CMMI

Graham Collins, UCL

graham.collins@ucl.ac.uk

Research supported by Deutsche Bank

IntroductionIntroduction

UCL’s research and projects The problems and requests EV (Earned Value) Agile practices Combining EV and agile is it possible? Developing suitable metrics compatible with

CMMI What works, and what reduces the overheads Is this approach likely to lead to CMMI Level 5? Key changes to projects

Background - RequestsBackground - Requests

We would like to use earned value ‘We would like to predict the outcome of project end date,

cost and value’ As a project manager I cannot be there all the time What is the best way to measure progress with earned

value? ‘As developers we would like to experiment with some

agile approaches’ Your team will be working on other projects as well We are hoping to achieve metrics suitable for CMMI level

3 and higher…

Kamikaze

Suicide

Mission Impossible

Ugly

low high

high

low

Chance of success

Happiness

The Death March Project Style Quadrant

Edward Yourdon, Death March:The complete Software Developer’s guide to surviving ‘Mission Impossible’ projects, 1997 Prentice Hall

The Death March Project Style QuadrantThe Death March Project Style Quadrant

Earned Value - DefinitionEarned Value - Definition

‘The value of the useful work done at any given point in a project. The value of completed work expressed in terms of the budget assigned to that work. A measure of project progress. Note: The budget may be expressed in cost or labour hours’

APM (Association of Project Management) BoK 2006

Earned Value – Graphical RepresentationEarned Value – Graphical Representation

Earned Value

Planned

ActualCost or value

Time

Planned (BCWS = Budgeted Cost of Work Scheduled)Actual (ACWP = Actual Cost of Work Performed)Earned Value (BCWP = Budgeted Cost of Work Performed)CPI = Cost Performance Index = BCWP/ACWP (or Earned/Actual)SPI = Schedule Performance Index = BCWP/BCWS (or Earned/Planned) note this is SPIc i.e. cost.

Use of VarianceUse of Variance

Variance (Monthly)

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

1.10

1.20

1 2 3 4 5 6

Months

Val

ue cpi

spi

spic is used in this situation

Earned Value compared to Agile Process PlanningEarned Value compared to Agile Process Planning

Based on predictive planning

Estimates effort, cost and completion date

End-to-end value tracking

Adaptive planning

Iteration to iteration tracking

Predication of the next iterations effort

Schedule most of the activities

Adaptation to unpredictable events is problematic. Changes may require the planned to be revised or baselined

Near the beginning, it is not always possible to schedule.

Time based iterations allow initial estimate of duration which can be revised through the adaptive driven build-feedback cycle

Estimates based on past performance Estimates are based on progress being made (velocity)

Change rates often low Unpredictable change the norm

Small variations in early measurements of cost and time at the start the project give wide variation in forward predications

Unknown team development rates

Some organisations do not chart progress for an initial period

Progress is tracked immediately

Earned value well established Prioritization of the value of user stories

No earned value approaches in methods

Earned Value Agile Development

CMMI- Process AreasCMMI- Process Areas

‘Project planning is a necessity for success,Yet it still ranks on most surveys within the top three or four problem areas leading to failure.’

Tony Ciorra, Planner’s Progress, Project, APM May 2006

Category Process Area

Project Management Project Planning

Project Monitoring and Control

Quantitative Project Management

Support Process and Product Quality Assurance

Causal Analysis & Resolution

CMMI Comparative AdvantagesCMMI Comparative Advantages

Grants explicit freedom to select the order of improvement that best meets the organization’s business objectives and mitigates the organisation’s areas of risk

Enables organisations to have a predefined path

Enables increased visibility of the capability achieved in each individual process area

Focuses on a set of processes that provide an organization with a specific capability that is characterized by each maturity level

Provides a capability-level rating that is used primarily for improvement in an organisation and is rarely communicated externally

Provides a maturity-level rating that is often used in internal management communication, statements external to the organization, and during acquisitions as a means to qualify bidders

Allows improvements of different processes to be performed at different rates

Summarizes process-improvement results in a simple form – a single maturity-level number

Reflects a newer approach that does not yet have the data to demonstrate its ties to return on investment

Builds on a relatively long history of use that includes case studies and data that demonstrate proved return on investment

Continuous Representation Staged Representation

Agile ManifestoAgile Manifesto

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more

Several agile projects have achieved CMMI level 3, example David Anderson, Stretching Agile to fit CMMI Level 3, Agile Conference 2005

The Agile Principles www.agilealliance.comThe Agile Principles www.agilealliance.com

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software

Agile processes promote sustainable development

Welcome changing requirements, even late in development. Agile processes harness change for the customer competitive advantage

The sponsors, developer, and users should be able to maintain a constant pace indefinitely

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter time scale

Continuous attention to technical excellence and good design enhances agility

Business people and developers must work together daily throughout the project

Simplicity - the art of maximizing the amount of work done – is essential

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

The best architectures, requirements, and designs emerge from self-organizing teams

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation

At regular intervals the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly

Working software is the primary measure of progress

Iterative Development (Bittner-Spence)Iterative Development (Bittner-Spence)

1. Agree with the team the objectives for the iteration, including evaluation criteria, timescales, and constraints

2. Agree on a plan for how the team will achieve the objectives

3. Execute the plan

4. Assess the achievements of the team against the initial set of objectives and evaluation criteria

5. Assess the impact of the iteration’s results on the project as a whole

6. Start the next iteration.

What is iterative development? Part 3: The management perspective 15 May 2005www-128.ibm.com/developerworks/rational/library/may05/bittner-spence/index.html

Fundamental shift in measurementFundamental shift in measurement

Progress ( % complete measured in scenarios

100%

0%

Iteration 1 2 3 4

coded tested Tested & Passed

Developer PerspectiveDeveloper Perspective

Developers are less interested in the business value, benefits realization and return on investment

They work on a small number of requirements or change requests from their list of outstanding work

They anticipate a decreasing number of requirements and change requests as the product is developed

Outstanding requirements and change requests is termed the product backlog

The developer will therefore be aware of progress via work completed, product backlog and new work allocated.

User Satisfaction driving DevelopmentUser Satisfaction driving Development

User satisfaction

Iteration

ReleaseUser satisfaction

Release planning

Development IncrementIteration planning

Iteration Planning (Goal identification, story selection, task estimation, team commitment)

To develop members’ capabilities; to build and exchange knowledge

Passion, commitment, and identification with the group’s expertise

To accomplish a specified task

The project’s milestones and goals

Adapted from: Communities of Practice: The organizational Frontier, Etienne C. Wenger and William M. Snyder, Harvard Business Review p139-145 Jan-Feb 2000

What is the purpose? What holds it together?

Community of practice

Project team

Project teams need to adopt some attributesProject teams need to adopt some attributes

Rate of work - velocityRate of work - velocity

Velocity

0

5

10

15

20

25

30

35

40

45

50

06/01

/06

13/01

/06

20/01

/06

27/01

/06

03/02

/06

10/02

/06

17/02

/06

24/02

/06

Sto

ry P

oin

ts

Velocity, gives an indication of the average rate of work and also a comparison of planned against delivered, each iteration

Individuals and Moving Range (XmR) ChartsIndividuals and Moving Range (XmR) Charts

Velocity

0

10

20

30

40

50

60

06/0

1/20

06

13/0

1/20

06

20/0

1/20

06

27/0

1/20

06

03/0

2/20

06

10/0

2/20

06

17/0

2/20

06

24/0

2/20

06

weeks

sto

ry p

oin

ts

story points mR

UCLR

UNPLx

Control Limits for XmR ChartsControl Limits for XmR Charts

k sequential measurements provide k-1 =r (two-point) moving range valuesith moving range = mRi = │Xi+1 – X i │where integer i is 1 ≤ i ≤ k – 1 __ i=r Individuals average moving range =mR = 1 ∑ mRi

r i=1

_ __ _ __Upper Natural Process Limit =UNPLx= X + 3mR = X + 2.660mR

d2

_ i=k

Centerline = CLx = X = 1 ∑ Xi (average of individual values) k i=1

_ __ _ __Lower Natural Process Limit =LNPLx= X - 3mR = X - 2.660mR

d2

___Centerline or average moving range =UCLR = mR

__ __Upper Control Limit for moving range =UCLR= D4mR = 3.268mR

__Sigma for individual values = sigmax (σ) = mR d2

When n=2 d2 =1.128and D4 =3.268 (from Dispersion and Bias factor tables)

‘Under Control’‘Under Control’

Velocity measures of work rate are useful in that estimates of the next iteration can be planned in a rolling processThe use of σ variation is supportive in this aim Automated colour coding (Red Amber Green) can be used to show condition requirements

Velocity

0

10

20

30

40

50

60

06/0

1/20

06

13/0

1/20

06

20/0

1/20

06

27/0

1/20

06

03/0

2/20

06

10/0

2/20

06

17/0

2/20

06

24/0

2/20

06

weeks

sto

ry p

oin

ts

story points

LNPLX

UNPLX

‘Weighted average’‘Weighted average’

Storypoints

count Weightedpoints

0 1 0

5 2 10

18 3 54

14 4 56

24 5 120

27 6 162

14.66667 3.5 19.14286

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Velocity

0

5

10

15

20

25

30

35

1 2 3 4 5 6

iterations

sto

ry p

oin

ts

centreline (CLx)

weighted average

UNPLx

‘Burn-down’‘Burn-down’

Story Points Remaing

0

50

100

150

200

250

300

350

06/0

1/20

06

13/0

1/20

06

20/0

1/20

06

27/0

1/20

06

03/0

2/20

06

10/0

2/20

06

17/0

2/20

06

24/0

2/20

06

Sto

ry P

oin

ts

With the appropriate metrics we can improveWith the appropriate metrics we can improve

Acceptance Test Data

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12

Iteration

AT

s

Acceptance Tests

Failing ATs

Passing ATs

Use of MultipliersUse of Multipliers

Iterations Completed

Low Multiplier High Multiplier

1 0.6 1.60

2 0.8 1.25

3 0.85 1.15

4 or more 0.90 1.10

Multipliers for estimating velocity based on number of iterations completed from Cohn 2006

Charts and MetricsCharts and Metrics

Velocity and Burn-down Cumulative acceptance tests

Inventory

Failing

Passing

Cumulative Issue ChartsBacklog - Active issues (which show inventory line)

Resolved issues

Closed issues

Earned Value EV progress charts

Performance via cpi and spi

Cpi and spi combined with control charts

Earned ValueEarned Value

EV can be applied to estimates of agile projects

- this is complex if more stories are added as the work progresses

EV may need to be shown to senior managers

- who are used to EV figures, or comparison to other projects where EV figures have been tracked

EV estimates can be accurate

- story points tend to remain static in an iteration when the process is understood by managers and developers. When additional stories are added, stories with lower business priority level may be dropped to compensate and keep the work load (story points) similar.

Business ValueBusiness ValueMore importantly business value (or contribution) should be considered and evaluated

Often units of measure such as story points can be valued as 0.5 or 1.0 units

The key issue in agile project management is to continually assess with the client the most important work that should be done.

Story number

‘Business Value’

Story Points

‘Points earned’

Planned (developer hours)

Actual(logged hours)

EV (earned value)

1 3 10 10 100 120 100

2 2 8 8 60 60 60

3 2 4 4 60 80 60

4 1 2 0 20 0 0

total 8 24 22 240 260 220

ConclusionsConclusions

EV can be combined with agile reporting The most useful approach to achieve process improvement is

via understanding of process control, the basis for CMMI The use of acceptance tests is the most useful approach for

both methods and is the basis for metrics outlined EV remains a viable approach where story points work load is

maintained EV is useful for reporting to senior managers and at a

programme and project level, but even with major scope changes, re-planning can give some useful insights

The process used in agile project management is critical, that this should be adhered to in the team. It needs to be explained to managers that the team are trying to achieve maximum business value per iteration, and that the final goal and plans evolve during the process

Tracking suitable metrics and understanding variation is the key to better estimation and process improvement – CMMI assessments can help achieve this goal.

Key Project ChangesKey Project Changes

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

1. Pair reporting - valuing individuals and team, moving towards self-determining teams

2. Acceptance testing - working software

3. Ensuring Business Value – continual prioritisation, estimation and understanding there is a cost to development

4. Measuring progress over shorter time periods -meeting the needs of process improvement CMMI, velocity tracking in agile methods, and better EV planning