© quantitative software management, inc. using quantitative software estimation tools and...
TRANSCRIPT
© Quantitative Software Management, Inc.
Using Quantitative Software Estimation Tools and Techniques to Support Verification and Validation (V&V)
Joseph MaddenQuantitative Software Management, Inc
2000 Corporate Ridge, Suite 700McLean, VA 22102
(703) 790-0055Email: [email protected]
Web: www.qsm.com
(#2)
• Introduction to Quantitative Software Estimation• Using Quantitative Software Estimation to
Support V&V Planning & Execution• Example Uses of Quantitative Estimation
Techniques at the FAA• Conclusion
Outline
(#3)
About QSM
• QSM founded in 1978 by Larry Putnam, Sr., one of the original pioneers in quantitative software estimation and author of 4 books, including Measures for Excellence and Five Core Metrics.
• Putnam invented the SLIM quantitative estimation tool and began a benchmark database of historical project data. This database now includes over 10,000 validated projects covering most application domains, industry sectors, and development approaches.
(#4)
Benefits of Quantitative Estimation: A Case Study
Before Using SLIM First Year of Using SLIM
Full Implementation of SLIM
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
$0
$2,000,000
$4,000,000
$6,000,000
$8,000,000
$10,000,000
$12,000,000
$14,000,000
$16,000,000
Large telecommunications supplier with operations in more than 130 countries
% Projects Over Budget and Schedule
Dollars Out of Control
In the year before using SLIM, 10 of 12 projects (83%) exceeded budget and schedule. The cost of this excess was more than $15M. QSM was hired to implement a robust estimation process.
Within the first year of using a SLIM estimation process, the percentage of projects over schedule and budget decreased from 83% to 50%, and excess cost reduced from $15M to $6M. After full implementation of SLIM in the second year, the percentage of projects over schedule and budget dropped to 10% and the cost overruns were less than $500K1.
1The above case study was published in 2003 in the book Five Core Metrics by Lawrence H. Putnam and Ware Myers.
(#5)
The Fundamentals of Quantitative Software Estimation
• Design Complexity• “Software entities are more complex for their size than for perhaps any other human construct.”1
• Organizational Complexity
• Large software projects require a large organization. This organizational complexity “makes overview hard, thus impeding conceptual integrity. It makes it hard to find and control all loose ends. It creates a tremendous learning and understanding burden.”1
• Cost and Schedule Overruns
• The Standish Group reported that 80% of systems are delivered late and over budget and 40% of projects fail or are abandoned.2
• QSM benchmark database shows 70% of projects exceeded budgets and 81% did not meet schedules.3
1 Frederick P. Brooks, Jr., “No Silver Bullet: Essence and Accidents of Software Engineering,” Computer, April 1987.
2 Standish Group. Computer, April 2003. 3 QSM, Inc., Benchmark Database, as of April 2007.
The ProblemSoftware Development is Difficult to Estimate
(#6)
Variable Relationships
• Early software estimation attempts
incorrectly assumed a linear
relationship between variables
• Software size was divided by average productivity rate. Effort was divided by manpower to determine time. Manpower was
increased until time met delivery date.
• Dr. Frederick Brooks demonstrated in The Mythical Man Month that this
assumption is not true. Manpower and time are not interchangeable.1
• Researchers eventually
discovered a nonlinear
relationship between
variables of interest
• Researchers collected and
analyzed a large body of software project data and discovered most relationships are
nonlinear.
• Variables of interest appeared to be a complex power
function of system attributes.2
1 Frederick P. Brooks, Jr., “No Silver Bullet: Essence and Accidents of Software Engineering,” Computer, April 1987.2 Lawrence H. Putnam. Measures for Excellence: Reliable Software On Time, Within Budget, Prentice Hall PTR, Upper Saddle
River, NJ, 1992.
Research Revealed a Nonlinear Relationship Between Variables
(#7)
Patterns in Historical Data
Scatter diagrams used to perform trend line analysis, quantifying relationships.
Linkage established between system size, development time, effort, cost, manpower, productivity, and number of defects.1
1 Lawrence H. Putnam. Measures for Excellence: Reliable Software On Time, Within Budget, Prentice Hall PTR, Upper Saddle River, NJ, 1992.
This analysis led to derivation of key computational equation1
Product = Productivity Parameter * (Effort/B)(1/3) * Time(4/3)
+1 Std Dev
Avg
-1 Std Dev
Curve Fitting of Historical Data Resulted in the Discovery of Estimation Equations
(#8)
Estimation Equations
Productivity Parameter: Process productivity parameter for software development organization, obtained by calibration from past projects.
Constant “B”: Special skills factor which is function of size. Increases slowly with size as need for integration, testing, quality assurance, documentation, and management skill grows with increased complexity of large projects.
(Effort/B)(1/3) * Time(4/3) = (Size)/(Productivity Parameter)
(Total Effort)/Time3 = Manpower Buildup Parameter
Source: Lawrence H. Putnam. Measures for Excellence: Reliable Software On Time, Within Budget, Prentice Hall PTR, Upper Saddle River, NJ, 1992.
Effort: Person-years of work by all job classifications for the software construction or main-build phase: design, coding, inspection, test, documentation, and supervision.
Time: Elapsed calendar schedule in years for software construction phase.
Manpower Buildup Parameter: Computed by calibration from past projects.
“Putnam Equations”
(#9)
Key Time and Effort Rules
• Small changes in duration
result in large changes in
effort
• The relationship
between effort and
time is exponential:
Effort = Constant/Ti
me4
• Extending development time
from 18 to 19 months (a 5.5%
increase) reduces the effort required
by 19.5%.1
• There is a minimum
development time
• “More software projects
have gone awry for lack of
calendar time than
for all other causes
combined.”2
Log Time
Lo
g E
ffo
rt
(MBI*) (Size & PI*)
ImpossibleRegion Minimum
TimePoint
No data to the left of the MBI line has ever been seen.1
*MBI and PI are indexes based on the Manpower Buildup and Productivity parameters.
1 Lawrence H. Putnam. Measures for Excellence: Reliable Software On Time, Within Budget, Prentice Hall PTR, Upper Saddle River, NJ, 1992.2Frederick P. Brooks, Jr., “No Silver Bullet: Essence and Accidents of Software Engineering,” Computer, April 1987.
(#10)
The Shape of Projects
1960s: Dr. Peter Norden found that engineering projects could be depicted by a Rayleigh Curve to predict staffing1
1970s: Larry Putnam Sr. applied the principles to software projects—this offered a mathematical way of predicting broader project behavior2
Traditional project management often tries (often unsuccessfully) to “level-load” projects
Sta
ffing L
eve
l
Time
Sta
ffing L
eve
l
Time
1. Norden, P.V.; “On the Anatomy of Development Projects,'' IRE Transactions on Engineering Management, 1960 Volume 7, Number 1, pp. 34-42.
2. Putnam, Lawrence and Myers, Ware. Measures for Excellence. Yourdon Press 1992. pp.44-45
(#11)
Reliability Model for Predicting Software Quality
Defe
cts
Time
Defects in software also tend to follow a Rayleigh curve.
We can use this to predict quality in a software product.
SLIM-Estimate uses Mean Time To Defect (MTTD): the average time between defects appearing as a measurable indicator of quality.
No work product (yet) to contain defects
Defects rise as work products are created Late in project
primary task is defect removal
(#12)
Reliability Model Variable Relationships
Size: Historical data has shown that as the code size increases, the number of defects increases. The rate of defect increase is close to linear.
Peak Staffing: Historical data has shown that adding people increases the defect creation process at a rapidly accelerating rate. For example:• A project of 350,000 SLOC using 40 people at peak staff would
create approximately 2,125 total defects. • If 60 people were used, 3 months of schedule compression would
be gained but 3,010 defects would be created.
(#13)
Quantitative Estimation
Software Size:• SLOC• Function Points• Objects, Etc.
Uncertainty
Process Productivity:• Methods/Tools• Tech Complexity• Personnel Profile
Management Constraints:• Max People• Max Budget• Max Schedule• Required Reliability
Optimum Estimate (Max Probability of Meeting Constraints)
Evaluate Practical Alternatives
Generate Plans
Inp
uts
Ou
tpu
ts
Staffing
Cost
Probability
Defects
SLIM-Estimate
(#14)
Inp
uts
Quantitative Control
Aggregate Staffing Rate
0.0
5.0
10.0
15.0
S 87654321S 321
People
3 9 15 21 27 33 *
Size
0
20
40
60
S 87654321S 321
ESLOC (thousands)
3 9 15 21 27 33 *
Total Defect Rate
0
40
80
120
S 87654321S 321
Defects
3 9 15 21 27 33 *
Current Plan Actual Interpolated Current Forecast Green Control Bound Yellow Control Bound Life Cycle includes R&D, C&T, T&ES = Start, 1 = PDR, 2 = INC_1, 3 = INC_2, 4 = INC_3, 5 = ST, 6 = TRR, 7 = FAT, 8 = 99R
Tracking of Plan Against Approved Baseline
Monthly Metrics Control Charts
Project ManagementMilestones
Software DevelopmentSize Actuals
Cost AccountingEffort/Cost Actuals
Software TestDefect Actuals
Variance from Plan
Forecast to Complete
Evaluation of Alternative Management Strategies
Ou
tpu
ts
SLIM-Control
(#15)
Quantitative Benchmarking
Construction & Test Trends
MB Duration (Months) vs Effective SLOC
1 10 100 1,000
Effective SLOC (thousands)
1
10
100
MB
Duration (M
onths)
MB Effort (MM) vs Effective SLOC
1 10 100 1,000
Effective SLOC (thousands)
1
10
100
1,000
10,000
MB
Effort (M
M)
PI vs Effective SLOC
1 10 100 1,000
Effective SLOC (thousands)
0
5
10
15
20
25
PI
MB Peak Staff (People) vs Effective SLOC
1 10 100 1,000
Effective SLOC (thousands)
0.1
1
10
100
1,000
MB
Peak S
taff (People)
All Systems Avg. Line Style 1 Sigma Line Style
Shows where benchmarked project lies in relation to historical database of
similar projects.
Data is compared for schedule, effort, productivity, and staffing vs. size along
standard deviation lines.
(#16)
Using Quantitative Software Estimation to Support V&V Planning
and Execution
(#17)
+
Making software schedules and associated V&V activities more predictable
Estimating V&V Effort and Schedule
Within the control of the V&V team:• Duration and effort for
complete inspection or testing cycle
Outside the control of the V&V team:• Delivery date for the
software• Volume of defects and
rework• Number of test cycles
required to reach target quality
Staffing & Probability Analysis - Example engineering project
Avg Staff (people)<Normal Schedule>
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26Jan'10
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan'11
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan'12
Feb Mar
0
5
10
15
20
Avg Staff (people)
987654321
R&DC&TREL
Milestones 0 - SRR 1 - PDR 2 - CDR 3 - CUT 4 - IT 5 - SIT 6 - ST 7 - FCS 8 - LP 9 - HVP
Milestones 0 - SRR 1 - PDR 2 - CDR 3 - CUT 4 - IT 5 - SIT 6 - ST 7 - FCS 8 - LP 9 - HVP
RISK GAUGE<Normal Schedule>
Life Duration <= 30 Months Life Cost <= 7000 K $C&T Peak Staff <= 25 people C&T MTTD >= 33.6 Hrs
DurationCost
Peak StaffMTTD
% 0 10 20 30 40 50 60 70 80 90 100
SOLUTION PANEL - <Normal Schedule>
DurationEffortCost
Peak StaffMTTD
Start Date
C&T16.133.1
4252.917.1
32.795/14/2010
Life Cycle25.445.1
5800.217.1
201.081/1/2010
MonthsPHR (K)$ (K)peopleHrs
PI=14.4 MBI=2.1 Eff SLOC=100,000
CONTROL PANEL<Normal Schedule>
PI 11.5 17.3
14.4
Peak Staff 13.7 20.5
17.1
Eff SLOC (K) 80 120
100
Defect Rate Total<Compressed Schedule>
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20Apr'10
May Jun Jul Aug Sep Oct Nov Dec Jan'11
Feb Mar Apr May Jun Jul Aug Sep
0
50
100
150
200
250987654321
Project: Example engineering proj...
Quantitative estimation techniques can help predict these items.
Takeaway: Quantitative estimation can help make V&V effort and duration more predictable
Predicted volume of defects
Estimated duration
(#18)
Total Defect Estimate - Example engineering project
MTTD Total (Hrs)<Normal Schedule>
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29Apr'10
May Jun Jul Aug Sep Oct Nov Dec Jan'11
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan'12
Feb Mar Apr May Jun
0
400
800
1,200987654321
Defects Remaining Total<Normal Schedule>
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29Apr'10
May Jun Jul Aug Sep Oct Nov Dec Jan'11
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan'12
Feb Mar Apr May Jun
0
400
800
1,200987654321
Project: Example engineering proj...
Accurately predicting when software is likely to reach target reliability
Takeaway: This technique was used successfully for reliability analysis of the Voice Switching and Control System (VSCS) software at FAA prior to delivery to ensure it was stable enough to deliver and deploy
(#19)
Errors (SysInt-Del) vs Effective SLOC
10 100 1,000
Effective SLOC (thousands)
10
100
1,000
Current Solution Historical Projects QSM ENGINEERING GROUP Avg. Line Sty le 1 Sigma Line Sty leProject: Example engineering proj...
Understanding whether the number of actual defects discovered compares favorably with
other projects
593 defects at +1σ
199 defects average
68 defects at - 1σ
Takeaway: Statistical benchmark data can be used to help answer the question: are there are too many or too few defects?
(#20)
Examples of Quantitative Software Estimation Techniques at the FAA
(#21)
Example Use of Quantitative Software Estimation Techniques at the FAA
• RRI Program. Independent Estimate and Risk Assessment for the FAA for a new generation air-ground router and communication system being built by five separate contractors (three US, one French, one Irish) building different pieces of the system.
• Terrorist Screening Software. Estimates of schedule cost and risk for each of 9 airlines to build terrorist screening software to use in airport screening departure lounges. Recommended to FAA reasonable costs for each system for negotiation between airline and FAA. Validated estimates using SLIM Metrics and QSM database.
• OASIC. Bid Evaluation of competitors to do an upgrade to a flight planning system. Calibrated historic data, did an analysis of each vendor’s bid and provided some alternatives for FAA to consider.
• URET. Independent estimate of an FAA system prior to award. Calibration data and defect analysis of a historic project by the same vendor was done for tuning purposes.
• STARS. Independent estimate of an airfield radar system for the FAA. Calibration and analysis of a vendor bid prior to award. Emphasis on whether they could do the project in the specified time frame.
• ITWS. Competitive Bid Evaluation of 3 vendors competing for a new weather system for the FAA. Calibration of each vendor’s historic data, a baseline government should cost scenario for comparison, and evaluation of each vendor’s bid compared with the baseline, QSM tend lines, and specially tailored data sample. Determine whether bids were consistent with past performance.
• VSCS. Reliability analysis of the software prior to delivery to ensure it was stable enough to deliver and deploy.
• Expert Witness Case. Served as expert witness in a protest of termination of a contract by FAA against the prime contractor in the building of Radio Control Equipment for air traffic control. FAA asserted that Prime Contractor did not perform. QSM examined the plans and data concerning the software development to compared it with industry norms for real time control software to see if the progress, effort, and defect performance was consistent with others building software of a similar nature. Case was settled out of court about two weeks prior to trial.
• Airfield Radar System. Bid evaluation of contractor’s ability to perform work required on airfield based radar system. Once the project was under way, performed monitoring and control and re-planning.
(#22)
Conclusion
• Quantitative estimation supports more objective management decisions.
• Significant insight can be gained from a few core software metrics: size, duration, staffing and defects.
• Improved predictability makes everyone’s job becomes easier, including V&V staff.
(#23)
Contact
www.qsm.com
Quantitative Software Management, Inc. 2000 Corporate Ridge; Suite 700McLean, VA 22102O 703.790.0055Email: [email protected]
Joseph MaddenVice President