when the economy gets tough, the tough get measuring!

When the Economy Gets

Tough, the Tough Get

Measuring!Scott GoldfarbQ/P Management Group, Inc.10 Bow StreetStoneham, MA 02180Email: [email protected]: (781) 438-2692FAX: (781) 438-5549http://www.qpmg.com

2© Copyright 2009. Q/P Management Group, Inc.

All Rights Reserved.

When the Economy Gets

Tough, the Tough Get

Measuring!The methodologies reflected in the enclosed material, including the benchmark comparisons, are confidential and proprietary information of Q/P Management Group, Inc. The information contained in this document cannot be distributed

without prior written permission fromQ/P Management Group, Inc.



Tough Economic Times Present a Golden Opportunity for Measurement

• “We need to be fiscally responsible”

• “We need to reduce spending by X %”

• “We need to do more with less”

• “We need to balance the budget”

• “We can’t afford not too….!”

Take advantage of the poor economic climate to set your measurement agenda Take advantage of the poor economic climate to set your measurement agenda



Measurement Should Answer Management Questions and Support Decision Making

• How do we make sure resources are being utilized efficiently?

• What are appropriate funding levels for application maintenance and support?

• Is offshore development cost effective?

• What are reasonable improvement goals?

• How do we control vendor costs and performance?

• How do we best select and utilize package software?

• How do we manage changes in scope?

• Are we productive and how do we improve?



Measurement Leads to Results

Measurement

Actions

DecisionsResults

In poor economic times, focus Results on cost reduction



Why Measure?

%ProductivityImprovement 7%

IndustryAverage(1990s)

10-20%IndustryLeaders

Annual Increase

Improve Productivity and Reduce Cost

FlatIndustryAverage(2008)



Why Measure? (Continued)

• Improve Quality

• Reduce rework and maintenance costs

• Increase customer satisfaction

• Manage Vendors and Outsourcing Agreements

• Improve Estimating

• Manage Change of Scope

• Evaluate COTS



The Benefits of Measurement

Asset Management

Continuous Process Improvement

Change of Scope Management

Customer Requirements

Tasks

A

B

C

D

E

Hours

30

60

60

95

45

Schedule

Estimating

SystemsA B C

D E F

G H I J K L

Manage Outsourcing Agreements

Improvements Opportunities

Programs

Measurement

Requirements FunctionalDesign

DetailDesign

Application XFunctional

Design



Benefits of Measurement – Benchmarking

Project/ Application Selection

ProjectKickoff

• Project/Application Size

• Effort

• Schedule• Quality• Attributes and

practices• Cost

Data Collection

QualityAnalysis

BenchmarkReporting

Productivity Analysis

PracticeAnalysis

• Productivity Rates

• Quality Rates

• Schedule• Cost• Attributes and

practices• Benchmark

Comparisons• Conclusions and

Recommendations



Case 1: Telecommunications Company

Goal: Implement process improvements opportunities that will improve productivity and quality

• Redefined SDLC • Developed test bed facilities• Established project management function• Established measurement process and revamped estimating

process• Improved physical office environment• Improved requirements definition process (JAD, Prototyping)• Implemented reviews and inspections

• 62% increase in productivity over a 2 year period• 20% less defects per development hour

Actions Taken – Baselined current environment followed by comprehensive process improvement program

Actions Taken – Baselined current environment followed by comprehensive process improvement program

ResultsResults



Case 2: Financial Services Company

Goal: Significantly cut development costs through personnel cost cutting and improving processes

• Established ongoing measurement process to manage service levels and collect data for annual benchmark

• Performed annual benchmarks and identified annual improvement initiatives:

– Adjusted project schedules to optimize productivity– Bundled small unproductive projects– Re-architected critical poor performing applications– Implemented software engineering tools– Transitioned resources offshore– Increased user involvement in requirements definition– Improved lifecycle methods

Actions Taken – Benchmark annually to identify process improvement opportunities and measure progress

Actions Taken – Benchmark annually to identify process improvement opportunities and measure progress



Case 2: Financial Services Company (Continued)

• Consistent year after year cost reduction • Greatly improved customer satisfaction• 75% development cost reduction in 5 years

ResultsResults

$2,500

$2,000

$1,500

$1,000

$500

$02004 $/FP 2005 $/FP 2006 $/FP 2007 $/FP 2008 $/FP

$2,300

$1,800

$1,000$800

$575

Cost per FP

Benchmark Periods



Benefits of Metrics – Managing Outsourcing Agreements and Vendor Performance

Establish the Baseline

Assess Goals and Alternatives

Determine Contract Scope

and Terms

Identify Measures to

Support Terms

Negotiate Performance

Levels

Measure, Audit and

Report Results



Use of Metrics in Outsourcing and Vendor Management

Pay by the Metric• Payment is based on Function Points delivered or supported• Cost per Function Point developed, enhanced or maintained

Tool to Manage Performance• Provide incentives for achieving goals• Assessing penalties for poor performance• Evaluate vendor bids against industry benchmarks• Development productivity based on FP/hour• Maintenance productivity based on FPs/Full-time-person (FTP)

Metrics are often used to establish payment and performance terms based on current productivity, cost or industry benchmarks

Metrics are often used to establish payment and performance terms based on current productivity, cost or industry benchmarks



Case 3: Major Insurance Company

Goal: Outsource the software development function and guarantee major cost reductions

• Determined current development/enhancement project costs segmented by platform and size categories

• Over 100 projects were sampled to establish the current baseline

Actions Taken – Baselined current productivity/cost and structure an agreement that will reduce annual IT expenses

Actions Taken – Baselined current productivity/cost and structure an agreement that will reduce annual IT expenses

PlatformProject Size

Small Medium LargeC/S $2,200 $1,750 $1,900Web $1,750 $1,400 $1,600MF $1,950 $1,600 $1,850

2008 Baseline – Cost per FP



Case 3: Major Insurance Company (Continued)

ResultsResults

Contract Future Pricing – Cost per FPSmall Projects – Annual Schedule

Platform 2009 Baseline

2010 2011 2012 2013

C/S $2,200 $1,980 $1,782 $1,604 $1,443Web $1,750 $1,575 $1,418 $1,276 $1,148MF $1,950 $1,755 $1,580 $1,422 $1,279

• A major outsourcing agreement was signed satisfying company goals

• 35% decrease in cost over contract period (10% per year)– Considers productivity improvements– Considers offshore transition

• Guaranteed annual budget reductions• Total projected savings of over $100,000,000



Case 4: Major World Wide Bank

Goal: Re-establish maintenance headcount (FTPs) as dictated by contract terms

• Application portfolio size and current FTPs compared against industry benchmarks (by platform and application age)

• Current labor rates also compared to industry benchmark rates

• Maintenance staff reduced by 150 FTPs resulting in $15,000,000 annual saving

Actions Taken – Benchmark study performed followed by contract negotiation

Actions Taken – Benchmark study performed followed by contract negotiation

ResultsResults

Total Actual FTPs

Average Benchmark FTPs

Top Quartile Benchmark FTPs

855 550 350



Case 5: Government Agency

Goal: Evaluate vendor bid to develop travel management system

• Travel system size estimated from requirements • Industry average productivity rate used to estimate in-house

development cost at $2,000,000• In-house estimate compared against vendor bid of

$12,000,000

• Vendor bid rejected• Vendor removed from preferred vendor list• Travel system developed in-house at a cost of $3,000,000• $9,000,000 savings

Actions Taken – Evaluated vendor bid compared to in-house development option

Actions Taken – Evaluated vendor bid compared to in-house development option

ResultsResults



Benefits of Measurement – Ongoing Measurement Program

• Provides continuous input to projects

• Management reporting on a monthly or quarterly basis

• Measurement data utilized for estimating

• Annual reporting of results with benchmark comparisons a plus

Analysis

Improvement Initiatives

MeasurementFramework

K L M

A B C

D F

G H I J

ApplicationPortfolio

Projects



Case 6: Large Manufacturing Company

Goal: Utilize software measurement to improve project management, resource utilization and productivity

• Measurement analysis identified a major problem in staffing practices and use of offshore resources

• Offshore projects had staffing 3 to 10 times the headcount of similar size onshore projects negating low labor rate benefits

Actions Taken – Established an ongoing measurement program and performed a deep dive analysis of staffing practices

Actions Taken – Established an ongoing measurement program and performed a deep dive analysis of staffing practices

$3,000

$2,000

$1,000

$0100% Mixed 100%

Onshore On/Off-shore Offshore

Percent Offshore vs $/FP

Cost per FPBenchmark



Case 6: Large Manufacturing Company (Continued)

• Major changes in resource utilization and project management implemented

– Staffing guidelines were defined based on project size– Use of offshore development resources changed to on-demand

from reserved and dedicated– Time accounting practices modified

• Changes resulted in a 35% cost reduction from previous period

ResultsResults

$3,000

$2,000

$1,000

$02007 2008

Annual Cost/FP

Cost per FPBenchmark



Benefits of Measurement – COTS Evaluation

• Use Function Points to compare COTS functionality to requirements

• Estimate enhancement project cost

• Select most cost effective solution

Requirements Function Point

Analysis

Requirements Function Point

Analysis COTS Functional Evaluation

COTS Functional Evaluation

Functional GAP Analysis

Functional GAP Analysis

Project EstimatesProject

Estimates

Make/Buy Analysis

Make/Buy Analysis

Function Fit Analysis

“Function Fit Analysis” is used to determine how well a COTS solution

meets requirements:

“Function Fit Analysis” is used to determine how well a COTS solution

meets requirements:



Case 7: DoD HRM Applications

Goal: Evaluate various package solutions to determine best solution and funding requirements

• 3 major projects analyzed• All COTS demonstrated limited satisfaction of

requirements • One COTS solution had only a 2% “out of the box” fit

Actions Taken – Performed Function Fit Analysis Actions Taken – Performed Function Fit Analysis

Function Points

Added Changed Unchanged Total AFP

Total FPs 20,375 636 434 21,445

Percent of Total Project 95% 3% 2% 100%



Case 7: DoD HRM Applications (Continued)

• Solutions selected and funded based on real data

– One COTS solution rejected

– One COTS selected with customization

– One organization’s business processes modified to fit COTS

ResultsResults



Benefits of Measurement – Managing Change of Scope

100 FPs 120 FPs 130 FPs 135 FPs• New regulatory

table with tax rates added (10 FPs)

• Summary report added (5 FPs)

Impact

EffortScheduleCost

+ 1 month+ 2 weeks+ $5 K

+ .5 month+ 1 week+ $2.5 K

+ .25 month+ 2.5 days+ $1.25 K

• Product input screen changed (3 FPs)

• Interface to N&A file added (10 FPs)

• N&A inquiry and Product inquiry added (7FPs)

Manage Change of Scope

Detail Design

Delivered Application

Functional Design

Requirements



Case 8: Major IT Outsourcer

Goal: Deliver a major equipment registration system to a major customer with fixed funding requirements

• Project size and growth was determined using function points

Actions Taken – Measured change of scope in order to quantify scope creep and negotiate additional funding

Actions Taken – Measured change of scope in order to quantify scope creep and negotiate additional funding

5,000

4,000

3000

2,000

1,000

$02006 2007 2008 2009

920

1,650

3,450

5,000

FP Size



Case 8: Major IT Outsourcer (Continued)

ResultsResults

• Change of Scope documented, quantified and managed

• Significant improvement in customer satisfaction

• Additional funding secured

• Contractor avoided a $4,000,000 loss



Getting Started – Establish an Ongoing Measurement Program

Options• Pilot on Specific Project(s)• Measure in Selected

Business Unit• Broad Based Across

Organization with Limited Measures

• Conduct a Quality and Productivity Baseline (Benchmark) Study

• Measure and Manage Supplier Performance

Analysis

Improvement Initiatives

MeasurementFramework

K L M

A B C

D F

G H I J

ApplicationPortfolio

Projects



Getting Started – Find the Right Opportunity

• Cut project cost by optimizing size and schedule

• Evaluate vendor bids and COTS and determine the most cost effective make/buy solution

• Determine appropriate project staffing levels and avoid poor utilization of resources

• Measure and manage change of scope and fund appropriately

• Establish outsourcing agreements resulting in significant cost reductions

• Identify process improvement opportunities that will substantially increase productivity and quality

• Establish appropriate application maintenance support levels and reduce maintenance headcount

Measurement can provide significant cost cutting benefits:Measurement can provide significant cost cutting benefits:

when the economy gets tough, the tough get measuring!

Documents