csc444 software engineering

CSC444F'07 Lecture 7 1

CSC444Software Engineering

Release Planning


Capability Maturity Model

• Developed at Carnegie-Melon Software Engineering Institute by Watts Humphrey.

• Classifies an organization’s process maturity into 5 levels.– Each level is a group of practices.– The CMM is a roadmap for process improvement.– Should have substantially all practices in place for a lower level before

proceeding to the next

• Can be certified to a certain CMM level– Some similarities to

• Malcolm Baldrige• ISO 9000

• Not universally agreed to be a good thing– But everyone agrees to pretend


CMM Levels


Relationship to ISO9000

• ISO 9000– Set of quality standards

– Subset relate to software development

• In essence– Must document the process

– Must maintain “quality records”• These are auditable to ensure the process is being carried out

• The process can be anything


Relationship to Top10

• Practices necessary to achieve CMM Level 2 (Repeatable).

• With enough Level 3 (defined) added to attain ISO9000.

• With some Level 4 (quantitatively managed) sprinkled in where most effective:– Defect arrival / departure rates

– Estimates versus actuals

– Metrics on process step completion

– Defect attribution


Planning

• Most important aspect of CMM Level 2

• Common flaws:– Make no plans

– Make a plan, but don’t track it

– Use Microsoft Project


Why Plan?

• Not always a good thing– If no expected date

– If no other expectations (e.g., expected functionality)

– Planning can only slow you down

• Required when– External pressures come to bear on release dates

• Usually only happens a bit later in a software company’s business evolution– Not right at the start

– Necessary for “crossing the chasm”


Crossing the Chasm, Geoffrey Moore (1991)


Gantt Charts Considered Harmful


Planning Essentials

• What are we building?

• By when will it be ready?

• How many people do we have?

Answer these and nothing more:not “who will be doing what?”nor “what are the detailed tasks required?”nor “in what order must the tasks be performed?”


Implementation Plans

• Once planning is complete can then transform into a detailed plan– E.g., Microsoft Project

• Detailed plan should not contradict the release plan

• Not all of the project needs details beyond– Who do we assign it to

– But some parts do

• These plans may not be necessary– If no great inter-dependencies that can’t be worked out as you go

• They hinder change as they are so cumbersome to change


Of Mice and Men

• The essence of planning is uncertainty– Plans never “go according to plan”

– Must embrace change, not close our eyes to it

• Therefore:– Must track the plan always

– Must react quickly to adverse situations

– Must embrace changes in direction

– Must re-plan quickly


Internal Changes

• Estimation errors– Initial estimates contain a significant (one-sided) margin of error.

– As plan progresses, variance in estimates lower

• Developer-power leaving the project– Illness

– Parental leave

– Resignations

– Budget cuts

– Unexpected vacation plans

– Unexpectedly low work hours

– Unexpectedly low productivity


External Changes

• New (big) customer desiring new functionality

• Competitor release a product

• Collaboration opportunities

• Acquisitions and mergers

• Sudden changes in customer needs– E.g., regulatory changes affecting them


The Difficult Question• What are we building?

– May be hard for 1st release– Subsequent releases will have a big wish list– Pick the best looking ones, most demanded ones– Marketing product management decision

• What will make us the most sales?

• By when will it be ready?– Too soon:

• Customers won’t be ready for a new release– Won’t want to install– Won’t want to learn– Won’t want to pay for it

– Too late• Customers will forget about you• Competitors will pass you• Foregone revenues

• How many developers?– Usually fixed for next release

• Difficult question– Can we do all 3 at once?


A Common Happening• Often organizations will answer all three questions, but not address the

difficult one.

• Development management will want to please their masters, and will tend to agree to too much in a spirit of “gung-ho!”

– Some managers firmly believe that over-commitment is the road to productivity.– “It’s a stretch, but we’ll pull it off”

• Coders will say “it can’t be done”– but “that’s all they ever say”.

• Massive sate of denial will set in.– Everybody will hope for a miracle

• Nobody will accept any blame– Development management: we told you it would be a stretch– Coders: we said it could never be done– Marketing: you should have said something earlier– CEO: you all told me it was going fine– Yourdon’s “Death March”.

The Solution:Good Planning!

• Does not need to happen this way.• But need courage and conviction.• Need common sense:

– Is it even feasible to do what’s asked by the date required?

– Don’t give an off-the-cuff answer• Even if it is obviously impossible

– Put together a plan and demonstrate the facts.



Release Planning ReviewThe Product Lifecycle


Eliciting Potential Requirements

Potential Requirements

?

Next Release

• Starts with a wish list• Stated as business requirements

– Features or architectural enhancements


A Simple Release Plan

Dates: Coding phase: Jul.1—Oct.1Beta availability: Nov.1General availability: Dec.1

Capacity: days availableFred 31 ecdLorna 33 ecd… …Bill 21 ecdtotal 317 ecd

Requirement: days requiredAR report 14 ecdDialog re-design 22 ecd… …Thread support 87 ecdtotal 317 ecd

Status: Capacity: 317 effective coder-daysRequirement: 317 effective coder-daysDelta: 0 effective coder days


Sizing the Available Resources

• Who can work on the next release?– Must have skills and familiarity to contribute

• For how long?– Must count workdays in the coding phase

– Each resource available all that time?

– Subtract estimated vacation

• How dedicated to the next release– Work factor = w

– Converts 8-hour (nominal, arbitrary) days to time available to code and unit test during the coding phase

– E.g. w=0.6 means can dedicate 0.6x8 = 4.8 hours/workday

– Accounts for• Other tasks, sickdays, meetings, weekends, ...

– Range is 0 .. 9, usually 0.6 or so


Sizing Potential Requirements

30 36 43

• Cost / Benefit analysis– Cost: financial + opportunity = person days

• Sizing in ECDs– Inherent size of the work item

– Who will work on it

– The productivity of that person on that work item

• Ensure units are well-understood (more later)


The Capacity Constraint

• After all is done in a release

Actual Resource Used == Sum of Actual Times for Features

• This is always true. It is a constraint.

• In planning, knowing that this must work out at the end, can estimate both sides and force the estimates to be equal

Resource Estimate == Sum of Feature Estimates


A Geometric Analogy - Capacity

1 person-day

days

persons


A Geometric Analogy - Requirement

30 36 43


A Geometric Analogy - Capacity Constraint

days

p e r s o n s

It Must All Fit!


Release Planning

• What to build F• By when to build it T F N x T• Using how many people N

• Need to build an initial plan that respects the capacity constraint

• Need to continuously update the plan to maintain its adherence to the capacity constraint.


Most Common Problem

• Comes from either– not knowing– knowing but hoping for the best (Yourdon Death March)

(can happen initially or as we go)


Dealing with Issues as they Arise

Developer leaves the team

Add time Cut features

Both


Other Happenings feature expansion

developer returns


Organizational Issues

• Management must appreciate that software development carries with it certain inherent risks.

• The business of a software organization is to manage and adapt as possibilities continuously become reality.

• Ranting and raving is unproductive

• With good data, good managers will make good decisions.


The Quantitative Capacity Constraint

• Post-Facto, the following relationship must hold.– But, it requires careful definition.

We define carefully so that we know what it is we are trying to estimate, and how to compare actuals against estimates for post-mortem.


Number of Workdays: T

• The number of full-equivalent working days for fork to dcut.• Subtracts

– Weekends

– Statutory holidays

– “Company Days”

• Subtracts anything we know in advance that nobody is expected to work.


Developer Power: N

• The average number of dedicated developers per workday working during the T-day period.

• Dedicated Developer?


Work Time & Dedicated Time

• Work Time or Body Time– Defined as 8 hours per workday

• Excludes weekends, stat. holidays, vacation entitlement.

• E.g., 9-to-6 with 1 hour for lunch.

• Dedicated Time– Uninterrupted hour equivalents.– Time dedicated to adding new features to the release.

• Uninterrupted Time– 4 hrs with 30 min. of constant interruptions

• Not 3.5 hrs of dedicated uninterrupted time – more like 2

– 2 hrs with NO interruptions at all


Examples of Dedicated Losses

• Maintenance (tracking down and fixing defects) on previous releases

• Other simultaneous projects• Team-leader duties (& helping others)• Meetings• Training• Unexpected, non-made-up days off (e.g., sick days)• Sales/marketing support• Loss of flow due to interruptions


Measuring N

• Assume each developer understands the concept of a dedicated uninterrupted hour.

• Get each of the n developers to record how many dedicated uninterrupted hours they spent in total during the coding phase.

• hi is what’s in the time tracking system for the ith developer.

T

hN

n

ii

81


Attributing N

• di is the number of days available during the coding phase

• vi is the number of vacation days they took during the coding phase

• hi is as before

Substitute to get back to:

T

wtN

n

iii

1

iii vdt i

ii t

hw

8

T

hN

n

ii

81


Example

• Bob called in sick for 2 days: accounted for in h• Bob took an afternoon off, but worked on the weekend to make up

for it: accounted for in h

30535

5

35

39

bobbobbob

bob

bob

vdt

v

d

T

5.0308

120

8

120

bob

bobbob

bob

t

hw

h


Features

K

kkfF

1

fk = dedicated hours / 8 it took to code the kth feature.


Post-Mortem

• Imagine a time-tracking system that could track– hi,k,d

• dedicated (uninterrupted) hours spent– by the ith developer

– on the dth day

– doing coding work on the kth feature

• each such quantum would appear on both sides of F= N x T constraining them to be equal.

• See section 5.10 in book for proof.


Example Release Plan

• Sample Deterministic Release Plan


The Stochastic Capacity Constraint


Estimates

• Estimates are never 100% certain• E.g, if we estimate a feature at 20 ECD’s

– Not saying will be done in 20 ECDs

– But then what are we saying?• Are we confident in it?• Is it optimistic?• Is it pessimistic?

• A quantity whose value depends upon unknowns (or upon random chance) is called a stochastic variable

• Release planning contains many such stochastic variables.


Confidence Intervals

• Say we toss a fair coin 5000 times– We expect it to come up heads ½ the time – 2500 times or so

– Exactly 2500?• Chance is only 1.1%

– ≤ 2500?• Chance is 50%• If we repeat this experiment over and over again (tossing a coin 5000 times),

on average ½ the time it will be more, ½ the time less.

– ≤ 2530?• Chance is 80%

– ≤ 2550?• Chance is 92%

• These (50%, 80%, 92%) are called confidence intervals– With 80% confidence we can say that the number of heads will be less

than 2530.


Stochastic Variables

• Consider the work factor of a coder, w.– When estimating in advance, w is a stochastic variable.

– Stochastic variables are described by statistical distributions

– A statistical distribution will tell you:• For any range of w• The probability of w being within that range

– Can be described completely with a probability density function.• X-axis: all possible values of the stochastic variable• Y-axis: numbers >= 0• The probability that the stochastic variables lies between two values a and b

is given by the area under the p.d.f. between a and b.


PDF for w

• Probability that 0.5 < w < 0.7 = 66%• Looks to be fairly accurate.

– Has a finite probability of being 0– Has not much chance of being much greater than 1.2 or so

• Drawing such a curve is the only real way of describing a stochastic variable mathematically.

0 0.6 1 2 3

0.5 0.7

area = 0.66

3

2

1

Probability density function for wi.


Parameterized Distributions

• “So, Bill, here’s a piece of paper, could you please draw me a p.d.f. for your work factor?” – Nobody knows the distribution to this level of accuracy

– Very hard to work with mathematically

• Usual method is to make an assumption about the overall shape of the curve, choosing from a few set shapes that are easy to work with mathematically.

• Then ask Bill for a few parameters that we can use to fit the curve.

• Because we are not so sure on our estimates anyways, the relative inaccuracy of choosing from one of a set of mathematically tractable p.d.f.’s is small compared to the other estimation errors.


e.g., a Normal for w

• Assume work factors are adequately described by a bell-shaped Normal distribution.

• 2 points are required to fit a Normal• E.g., average case and some reasonable “worst case”.

– Average case: half the time less, half the time more = 0.6

– “Worst” case: 95% of the time w won’t be that bad (small) = 0.4• Normal curves that fits is N(0.6,0.12).

0.6

= 0.6

= 0.12

0.4

area = 0.95

N(0.6,0.12)

area = 68%


Maybe not Normal

• Normals are easiest to work with mathematically.• May not be the best thing to use for w

– Normal is symmetric about the mean• E.g., N(0.6,0.12) predicts a 5% “best case” of 0.8.• What if Bill tells us the 5% best case is really 1.0?

– Then can’t use a Normal

– Would need a skewed (tilted) distribution with unsymmetrical 5% and 95% cases.

– Normal extends to infinity in both directions• Finite probability of w < 0 or w > 10

0.6

= 0.6

= 0.12

0.4

area = 0.95

N(0.6,0.12)


Estimates

• Most define our quantities very precisely• E.g., for a feature estimate of 1 week

– Post-Facto• What are the units?• 40 hours? Longer? Shorter? Dedicated? Disrupted? One person or two? ...• Dealt with this last lecture in great detail

– Stochastic• 1 week best case?• 1 week worst case?• 1 week average case?• Need a p.d.f

• Depending upon these concerns, my “1 week” maybe somebody else’s 4 weeks.– Very significant issue in practice


The Stochastic Capacity Constraint

• T is fixed• F and N are both stochastic quantities.• Can only speak about the chance of the goo fitting into the rectangle• Say F=400, N=10, T=40: are we good to go?

– Cannot say.

– Need precise distributions to F and N to answer, and then only at some confidence level.


Summing Distributions

• F and N are sums and products over many contributing stochastic variables.

• E.g.– F = f1 + f2– If f1 and f2 have associated statistical distributions, what is the statistical

distribution of F?

– In general, no answer.– Special case: f1 and f2 are both Normal

• Then F will be Normal as well.

• Mean of F will be the sums of the means of f1 and f2

• Standard deviation of F will be the square root of the sums of the squares of the standard deviations of f1 and f2.

– How about f1 * f2?• Figet about it! Huge formula, result is not a Normal distribution

– One needs statistical simulation software tools to do arithmetic on stochastic variables.


Law of Large Numbers

• If we sum lots and lots of stochastic variables, the sum will approach a Normal distribution.

• Therefore something like F is going to be pretty close to Normal.– E.g., 400 features summed

• N will also be, but a bit less so– E.g., 10 w’s summed


Delta Statistic

• D(T) = N T F• If we have Normal approximations for N and F, can compute the

Normal curve for D as a function of various T’s.• We can then choose a T that leads to a D we can live with.

• Interested in

Probability [ D(T) 0 ]

• The probability that all features will be finished by dcut.

• In choosing T will want to choose a confidence interval the company can live with, e.g., 80%.

• Then will pick a T such that D(T) 0 80% of the time.


Example Picking T

• F is Normal with mean 400 and 90% worst case 500• N is Normal with mean 10 and 90% worst case 8• Cells are D(T) = N T F at the indicated confidence level• Note transitions through 0.

confidence level

25% 40% 50% 60% 80% 90% 95%

30 -39 -77 -100 -123 -177 -217 -250

35 14 -26 -50 -74 -130 -172 -207

40 67 25 0 -25 -84 -128 -164

T 45 121 77 50 23 -38 -85 -123

50 174 128 100 72 7 -41 -82

55 228 179 150 121 52 1 -41

60 282 231 200 169 97 44 0


Choices for T

• To be 95% certain of hitting the dates, choose T = 60 workdays• Or... If we plan to take 40 workdays, only 5% of the time will be late

by more than 20 workdays

• To be 80% sure, T = 49

• To gamble, for a 25% fighting chance, make T = 33.


Shortcut

• Ask for 80% worst case estimates for everything.• If F = NxT using the 80% worst case values, then there is an 80%

chance of making the release.• The Deterministic Release Plan is based on this approach.

• If you also ask for mean cases for everything, can then fit a Normal distribution for D(T) and can predict the approximate probability of slipping.


Initial Planning

• Start with a T• Choose a feature set• See if the plan works out• If not, adjust T and/or the feature set an continue

choose T happy?

yes

no

done

adjust T

choose feature set

adjust feature set


Adjusting the Release Plan

• Count on the w estimated to be too high and feature estimates to be too low.

• Re-adjust as new data comes in.• Can “pad the plan” by choosing a 95% T.

– Will make it with a high degree of confidence

– May run out of work

– May gold plate features

• Better to have an A-list and a B-list– Choose one T such that, e.g.,

• Have 95% confidence of making the A list• Have 40% confidence of making the A+B list.


Risk Tolerance

• Say a plan is at 60%

• Developer may say:– Chances are poor: 60% at best

• An entrepreneurial CEO will say– Looking great! At least a 60% chance of making it.

• Should have an explicit discussion of risk tolerance


Loading the Dice

• Can manage to affect the outcome.• Like a football game:

– Odds may be 3-to-1 against a team winning

– But by making a special effort, the team may still win

• In release planning– Base the odds on history

– But as a manager, don’t ever accept that history is as good as you can do!

• E.g., introduce a new practice that will boost productivity– Estimate will increase productivity by 20%

– Don’t plan for that!

– Plan for what was achieved historically.

– Manage to get that 20% and change history for next time around.


Example Stochastic Release Plan

• Sample Stochastic Release Plan

csc444 software engineering

Documents

chasmlecture 7csc444f07lecture

plan alwaysmust

lower level

microsoft projectdetailed

process improvement

outthe process

planning essentialswhat

detailed tasks