1. Estimate software projects Faster, Cheaper and Better!Can parametric estimating beat the experts?H.S. van Heeringen@haroldveendamBrussels, May 2012

2. Overview Sogeti Expert vs. Parametric Estimates Chalenge for parametric estimates Estimating Wizard Study: Expert accuracy vs. Parametric accuracy 2 3. Sogeti Sogeti is the local IT brand of Capgemini Over 20.000 people in 15 countries Belgium, Denmark, Finland, France, Germany, India, Ireland, Luxemburg, Netherlands, Norway, Spain, Sweden, Switzerland, UK and the US. Sogeti Nederland: ca. 3.000 people Local IT service provider: Close to the clients; Our mission is to be a leading provider of professional technology services. Our vision is to be perceived as the best supplier on each local market; Tmap (test), DYA (architecture), ViNT (research), etcetera.3 4. Project Estimates Two types of project estimation: Expert estimation Parametric estimation Expert estimates Knowledge and experience of experts Assign effort hours to tasks (bottom-up) Subjective, but always applicable Parametric estimates Size measurement, historical data, CERs and tooling Size measurement methods: NESMA FPA, COSMIC, IFPUG Objective, but well documented specifications required4 5. Software Size measurement Software is hard to measure before the projectstarts But size is usually the main cost driver Best practice: measure functionality requestedby the users NESMA / IFPUG / COSMIC function points Objective, verifiable, repeatable methods Although ISO standards, there is always aninaccuracy margin Documentation is not complete or not detailed enough Measurers dont have the required level of expertise During the projects, changes will happen5 6. Comparing the two types Expert Estimates Bottom-up estimation Usually optimistic (up to 30% under estimation is common) Forgotten activities Hard to defend The expert is not going to do all the work The expert may not be an expert on the new project Parametric Estimates Top-down estimation Estimating & Performance Measurement process needed Effort = size * Productivity Size is objectively measureable (COSMIC, FPA) Productivity from historical data (organization / ISBSG)6 7. Expert Estimation Task Code and en Unit test of module XYZ90% . . . . . . . . . . . . . . . . 24 hours75% . . . . . . . . . . . . . . . . 22 hours100%Realistic estimate50% . . . . . . . . . . . . . . . . 20 hours10% . . . . . . . . . . . . . . . . 18 hoursExpert estimate 0%. . . . . . . . . . . . . . . . 16 hours Probability. . . . . . . . 14 hours. . . . . . . . 12 hours. . . . . . . . 10 hours. . . . . . . .8 hours. . . . . . . .6 hours. . . . . . . .4 hours 0% . . . . . . . .2 hoursEffort Expert Estimate: 1st possibility of success50/50 median result 7 8. Cost of high and low estimatesNon-linear extra costs-Planning errors-team enlargement more expensive, not faster-Extra management attention / overhead-Stress: More defects, lower maintainability !! Linear extra kosten Extra hours will be used 8 9. Sogeti AS SEC Division Application Services projects and maintenance (fixed price/ fixed date) CoE Sizing, Estimating & Control (SEC) Parametric estimation, control, historical data collection External services, project estimation, benchmarking, consultancy CoE: Microsoft, Java, Oracle Onshore / offshore teams Expert estimates (technical architects / programmers) Challenge of parametric and expert estimate9 10. Challenge sell parametric estimates Project/Bid management still believe expertsmore than parametrics More detail must mean more accurate, right? This project is very different from past projects I see that we dont get any function points for module XYZ, but we have to do a lot of work for that! I think the project is quite easy and I think that the parametric estimate overestimates the effort But what about: team size, duration, forgotten activities, past performance. How can we convince project management ??10 11. Software equationSize/productivity constant = Effort 1/3* duration4/3 TradeoffEffortPlan A: 6 months, 4.500 hoursPlan B: 7 months, 2.400 hours Duration11 12. Project at different durationsPlan ADuration: 6 monthsEffort: 4.500 hoursMax. team size: 5,8 fteMTTD: 1,764 daysTeam size (fte)Plan BDuration: 7 monthsEffort: 2.400 hoursMax. team size: 2,7 fteMTTD: 2,816 days Which duration have the experts inDuration mind?? 12Size and productivity being constant 13. Assignment 2005 Before 2005: estimation maturity level = 1 Build estimation instrument Gain time and effort in estimating bids Accurate enough to depend and rely on Flexible: Estimate onshore / offshore and hybrid projects Calculate different test strategies Take into account complexity Implement Deming cycle (PDCA) Give scenarios for duration !!! 13 14. First version Estimating Wizard (2005) Try to put the duration / effort tradeoff in amodel Tuned with experience data Hour/FP: Average Complexity Duration in months34 45 56 67 78 0-250 FP 10,1 8,9 8,1 7,7 6,9 250-500 FP9,1 8,0 7,3 6,9 6,2 500-750 FP8,6 7,6 6,9 6,5 5,9 750-1000 FP 8,3 7,3 6,6 6,3 5,6 1000 -1250 FP 8,1 7,1 6,5 6,2 5,5 1250 - 1500 FP7,9 6,9 6,3 6,0 5,4 14 15. Estimating Wizard 2011Estimating WizardPowered by: Sizing, Estimating & ControlData version: 24-11-2010Model version: 17InputFunctional design parametersFunctional Design YesStep 1: Is there a functional design phase?Overlap Yes, calculated 5Step 2: In case of overlap between the functional design phase and building to let the wizard calculate the overlap, or to enter the number of weLanguageEnglishStep 3: Enter the language in which the functional design should be written.Availability key users NormalStep 4: Enter the availability-rate of the key users.Location Sogeti office Step 5: Enter the location where the functional design should be written.Build and test parametersDevelopment tool JavaStep 6: Select the development tool.OnshoreOffshoreC onstruction 35% 65%Step 7: Enter the percentage of construction work that is done onshore.Translation FD required No Step 8: Is a translation of the functional design required.System test approachTMap MediumStep 9: Select the TMapfactory system test approach.System test strategyScripting and design NL, excecution in IndiaStep 10: Select the system test strategy.Tools/methodologies Unknown Step 11: Rate the level of tools and methodologies to be used for the developC omplexity Unknown Step 12: Rate the technical complexity of the project.Development teamUnknown Step 13: Rate the competence, experience and skill level of the developmentReuse Unknown Step 14: Rate the quantity and complexity of integrating reused, unmodifiedGeneral parametersSize643 C OSMIC Step 15: Enter the functional size and select a unit of size (FP= function points, C FP=C OSMIC functionpoints).Start date 01-01-11 Step 16: Enter the start date of the project.Risk surcharge (%) 10 % Step 17: Enter the risk surcharge percentage.Warranty (%)4 % Step 18: Enter the warranty surcharge percentage.Organization type Banking Step 19: C hoose the organization type.Quality documentation 6 Step 20: Rate the quality of the documentation.Non functional req. Average (0) Step 21: What influence do the non functional requirements have on the effoScenario interval 2,0 Step 22: Enter the number of weeks for the step size between the seven sce15 16. EW - output Functional Design no schedule scenarios Functional design phase Duration in weeks17,6 Design complete 4-05-11 Total effort1.975 Effort per FP2,53 Effort cost 208.531 Additional cost 14.815 Totaal cost 223.346 C ost per FP 286 Average team size2,80Data altered due to company security reasons 16 17. EW - output Main build 7 schedule scenarios Build and test phase Duration in weeks20,022,024,0 26,028,0 30,0 32,0 Start phase18-02-1118-02-1118-02-1118-02-11 18-02-11 18-02-11 18-02-11 Effort9.794 6.690 4.7233.429 2.5501.9351.495 Effort per FP 28,06 19,17 13,53 9,837,31 5,54 4,28 Effort cost 550.720 376.152 265.590 192.826 143.360 108.787 84.036 Additional cost 56.446 41.090 31.364 24.963 20.611 17.570 15.393 Totaal cost 607.167 417.242 296.954 217.789 163.972 126.357 99.428 C ost per FP 1.740 1.196 851 624 470 362 285 Average team size 12,247,604,92 3,302,28 1,61 1,17 Risk and warranty Risk hours834 586 429 325255205 170 Risk cost 54.301 39.107 29.484 23.150 18.845 15.836 13.682 Warranty hours209 146 10781 64 5143 Warranty cost 13.575 9.777 7.371 5.788 4.711 3.959 3.420 Total Duration in weeks27,029,031,033,0 35,0 37,039,0 Delivery for acceptance 8-07-1122-07-11 5-08-1119-08-112-09-11 16-09-1130-09-11 Total effort 11.470 8.055 5.892 4.4693.5012.825 2.340 Effort per FP 32,87 23,08 16,88 12,8010,03 8,096,71 Totaal cost 746.634 537.717 405.400 318.319 259.120 217.744 188.122 C ost per FP 2.139 1.541 1.162 912 742 624 539 Average team size 10,646,964,763,39 2,50 1,911,50 Average hourly rate 65 67 69 71 74 77 80Data altered due to company security reasons17 18. Calibrating the model Collect historical data Use ISBSG data18 19. ISBSG (www.isbsg.org) International Software Benchmarking StandardsGroup Not-for-profit, members are mostly nationalsoftware measurement organizations NESMA, IFPUG, DASMA, JFPUG, GUFPI-ISMA, AEMES, etc. New developments & enhancements repository About 6.000 projects now Maintenance & Support repository Over 500 applications now19 20. Calibrating the model Collect historical data Use ISBSG data20 21. Pricing model Sogeti Pricing model 21 22. Estimating Wizard 2012 Mature tool, proven value Effort Estimation Relationships are tuned tohistorical data Size is the main input parameter, and it takestime to measure size Parametric estimating is still not done in oneclick, and perceived as expensive by internalclients Experts are still perceived more credible and inmost challenges, the expert estimate is taken study: comparing experts and parametrics 22 23. Can parametrics beat the experts? Study of 20 completed projects Different sizes, languages, data normalized Expert estimate Estimating Wizard estimate (FP or CFP) Estimates vs. Actual results Calculate Effort Accuracy (Effort estimate / Actual effort) Duration Accuracy (Duration Estimate / Actual Duration) Cost Accuracy (Cost Estimate / Actual Cost) 1 overestimation 23 24. ResultsExpert Estimating Wizard Effort Duration CostTime EffortDuration CostTimeProjectSize (FP) Accuracy AccuracyAccuracy Spent Accuracy AccuracyAccuracy spentProject 12770,6751,545 0,467 300,477 1,204 0,50117Project 23590,5790,951 1,139 350,707 0,775 1,17026Project 33470,5890,142 0,615 401,067 0,996 1,28314Project 41.1780,4140,557 0,312 600,774 0,590 0,86255Project 59511,4300,997 0,946 341,067 0,877 1,71824Project 62950,7630,857 0,619 260,881 1,200 0,845 6Project 77900,7170,850 0,976 340,926 0,865 1,13227Project 83501,2580,800 1,309 281,203 1,096 1,31820Project 97460,5860,296 0,545 340,826 0,385 1,95322Project 10 2.2930,7660,421 0,797 400,931 0,632 1,05814Project 11 2130,6890,453 0,712 320,934 0,621 0,92112Project 12 5630,8980,912 0,911 400,892 1,122 1,23414Project 13 7110,5410,611 0,521 320,712 1,321 0,72418Project 14 1.2480,8740,719 0,877 600,923 1,091 1,12236Project 15 1220,9120,781 0,916 200,945 1,092 1,45312Project 16 9230,8140,721 0,723 240,954 1,221 1,12226Project 17 5120,7410,843 0,729 320,823 0,923 0,89220Project 18 7890,6210,912 0,649 400,711 0,945 1,23424Project 19 5120,8910,512 0,945 200,945 1,109 1,10122Project 20 7320,7651,210 0,799 240,865 1,001 0,90232 24 25. Estimation Accuracy results Average time spent Expert estimate: 34,3 hours Parametric Estimate (including size measurement): 22,1 hours Expert Estimate Estimating Wizard EstimateEffort Accuracy Average0,776 0,878 Closer to 1 means better! St.Dev.0,236 0,156 Median 0,752 0,908Duration Accuracy Average 0,7550,953 St.Dev. 0,3180,247 Median0,7910,998Cost Accuracy Average 0,7751,127 St.Dev. 0,2370,331 Median0,7631,12225 26. Results Expert estimates take a lot of time too! on average 55% more than Parametric Estimates More than 1 expert has to read through all the documentation, discussions, meetings, etcetera Parametric estimates are more acurate! Effort and duration estimates on average still optimistic, but less optimistic than expert estimates Cost estimates are pessimistic, but still closer to actuals than Expert estimate Experts might win the project, but the result willbe overruns!26 27. Cost of high and low estimatesNon-linear extra costs-Planning errors-team enlargement more expensive, not faster-Extra management attention / overhead-Stress: More defects, lower maintainability !! Linear extra kosten Extra hours will be used 27 28. Conclusions Estimating wizard Higher effort estimation accuracy Higher duration estimation accuracy Higher cost estimation accuracy (although >1) Less hours spent than expert estimates Standard WBS, historical data collection and parameter calibration improve the maturity of the process Next steps Analyze why costs are overestimated Try to identify the projects where EW estimate is enough Use the results to convince project management and bid management to use parametric estimating even more !!28 29. Thank you for yourattention !! @haroldveendamLocal touch - Global reachSenior Consultant Software MetricsSogeti Nederland B.V.NESMA board memberISBSG presidentCOSMIC International Advisory Council, representing theNetherlandsT: +31 (0)88 660 6600 + 3165 (dial)M: +31 (0)6 52 32 73 30e-mail: harold.van.heeringen@sogeti.nl 29